Is Your Website Readable by AI Crawlers?

If AI assistants cannot read your website, they cannot quote it. And a surprising number of modern websites are close to unreadable for them, even though they look fine in a browser.

Why a good-looking site can look empty

Many websites are built so that the server sends a nearly blank page, and JavaScript in your browser then fills in the text. Google runs that JavaScript. Many AI crawlers do not. They fetch the raw response and read what is there. If the raw response is an empty container and a script tag, that is all they see.

You can test it yourself. Open your page, right-click and choose "View page source" (not "Inspect"). Search for a sentence from the body of the page. If you cannot find it, a crawler that does not run scripts cannot either.

We hit this on our own site. Our build loads each page in a headless browser and saves the rendered HTML, so the text and structured data are in the response a crawler receives. Web developers call this prerendering or server-side rendering, and a developer can usually add it to an existing site.

The checklist

1. Text must be in the raw HTML

Headings, body copy, FAQs and contact details should appear in "View page source". Important content hidden behind tabs, accordions or "read more" scripts is at risk.

2. Use real links

Make sure pages are reachable through ordinary links. A blog list that only appears after a script runs can hide your articles from crawlers. Make sure articles also appear in your sitemap.

3. Check robots.txt

The robots.txt file tells crawlers what they may fetch. Reputable AI crawlers say they respect it, and each company names its own crawlers in its documentation. Two things to get right:

• Do not block by accident. Security plugins and default CMS settings sometimes block unfamiliar crawlers. Read your file and confirm it says what you intend.
• Decide deliberately. Some firms block AI training crawlers, or block everything except search-related crawlers. That is a legitimate business choice, particularly for firms with confidential or premium content. Make the decision on purpose, and check each AI company's current documentation for the names of its crawlers, since they change.

Remember that robots.txt is a request, not a security control. Never rely on it to protect private information. Anything confidential belongs behind a login.

4. Watch your firewall and bot protection

Services such as Cloudflare can challenge or block automated visitors. Strict settings can block legitimate crawlers along with the bad ones. Review them with whoever manages your security, and keep the protections that stop real attacks. Our post on ClickFix is a reminder of why you do not simply switch protection off.

5. Consider an llms.txt file

An llms.txt file is a plain-text summary of your site, written for language models. It is a proposed convention and not a guaranteed ranking factor. No major assistant has promised to rely on it, so treat it as cheap and harmless, not as a shortcut. If you add one, keep it factual and accurate and update it when your services change.

6. Add structured data honestly

Structured data (schema markup) describes your business, articles and FAQs in a machine-readable way. Use it for facts that are true and visible on the page: your name, address, services and the questions you answer. Do not mark up reviews or credentials you do not have.

Keep it secure

An AI-friendly website is still a website that has to be patched. An out-of-date plugin that lets attackers inject a fake security check will do more damage to your reputation than any SEO gain. Review ASD's questions in Is your website patched?

Where to go next

For the bigger picture, read AI search for small business. If you want a hand checking what a crawler sees on your site, contact us, and see how this fits with the Essential Eight.