A technical checklist for AI visibility
A technical review for AI visibility comes down to three questions: can the crawler fetch the page, can it read the content without executing JavaScript, and does it understand what the page is about. Below are the checks for each, in the order worth doing them, and most take under a minute.
What should be checked, and with what?
Six checks, the tool for each, and what a pass looks like. Most take under a minute. This site runs the same list on every build, and the criteria below come from there rather than from a recommendation.
| Check | Tool | A pass looks like |
|---|---|---|
| Rendering | curl without JavaScript | Headings and body copy in the response |
| Crawler access | robots.txt plus CDN settings | AI crawlers explicitly admitted |
| Structured data | Rich Results Test, Schema Validator | Right type per page, no dead URLs |
| Page structure | Manual review | One h1, no skips, self-referencing canonical |
| Indexing signals | Manual review | Noindex pages carry no canonical or hreflang |
| Crawler visits | Server log | GPTBot and ClaudeBot actually appear |
Can the content be read without JavaScript?
Run curl against your own page and read the response. If headings, body copy and links are there, the foundation is sound. If what comes back is an empty element filled in by a script, several AI crawlers see nothing. This is the single most important check and the one that most often reveals the problem.
Does robots.txt admit the right crawlers?
Fetch your robots.txt and look for Disallow rules hitting the whole site, and for blocks on named AI crawlers. Also check your security vendor or CDN is not blocking them a layer above, which is common and invisible in robots.txt.
Is structured data present and correct?
Run the page through the Rich Results Test and the Schema Markup Validator. Existing is not enough, it has to be right. Common errors I see:
- Everything marked as WebPage, which is formally correct and says nothing.
- Properties that do not belong to the type, for instance inLanguage on a Service.
- Images and URLs pointing at pages that do not exist.
- Every page declaring its own organisation instead of referencing the same one.
Does the page structure hold together?
One h1 per page, no skipped heading levels, canonical addresses pointing at themselves. Pages that are noindex should carry neither canonical nor hreflang, because those are indexing signals contradicting the directive. Cheap to check and surprisingly often wrong.
What does the server log say?
The log shows which crawlers actually fetch your pages and how often, and it rarely matches what anyone assumes. Filter on GPTBot, ClaudeBot, PerplexityBot and Google-Extended. If you find no visits at all, the question is not how to optimise the content but why the crawler never arrives.
How are you actually described in AI answers?
Ask twenty questions about your category to ChatGPT, Perplexity and Google AI Overviews. Save the answers with the date. Repeat monthly. It is blunt and cannot be meaningfully automated, but it is the only measurement showing whether the work achieved anything.
Common questions
Which check should I run first?
Fetch the page with curl and read the response. If the body copy is not there, nothing else on the list matters, because several AI crawlers do not run JavaScript and therefore never see the content. It is also the check that most often reveals the problem on modern sites built as single-page applications.
How often should the review be repeated?
At every launch and quarterly after that. Rendering and crawler access change when somebody adjusts a CDN setting or adds a security layer, and that happens without anyone thinking about it. This site runs the checks automatically on every build and refuses to build if a security header or a redirect has disappeared.
What if the server log shows no AI crawlers at all?
Then the question is not how to optimise the content but why the crawler never arrives. Check in order: robots.txt, then the bot settings at your CDN or firewall, then whether the site responds at all to a request without a browser user-agent. The block is almost always in the middle layer rather than in robots.txt.
