How to get cited by ChatGPT and Perplexity
To be cited by a language model, three things have to hold: the content must be in the server response without requiring JavaScript, the crawler must be admitted in robots.txt, and the passage must survive being lifted out of the page. The third is what separates most sites, because a model rarely cites a whole page, it cites a single paragraph.
Can the AI crawlers even read your page?
Fetch your own page without executing JavaScript and read what comes back. If the body copy is there, the foundation is sound. If the response is an empty element filled in by a script, several AI crawlers see nothing at all. Google can render JavaScript, but several of the crawlers feeding language models cannot. This is step one and it is surprisingly often the only step needed.
Are you letting the right crawlers in?
Many default setups and security vendors block AI crawlers by default. If you want citations, they need to be explicitly admitted in robots.txt. The crawlers worth knowing right now:
- GPTBot and OAI-SearchBot, for ChatGPT.
- ClaudeBot and Claude-SearchBot.
- PerplexityBot.
- Google-Extended, which governs use in Google’s AI answers separately from ordinary search.
- Applebot-Extended and Meta-ExternalAgent.
Which crawlers are actually fetching?
They are not one group but several, and they are governed separately in robots.txt. Admitting Googlebot says nothing about the others.
| Crawler | Feeds | Runs JavaScript |
|---|---|---|
| GPTBot | ChatGPT, training and index | No |
| OAI-SearchBot | ChatGPT search results | No |
| ClaudeBot | Claude | No |
| PerplexityBot | Perplexity | Partially |
| Google-Extended | Google AI answers, separate from search | Via Googlebot |
| Googlebot | Ordinary search and AI Overviews | Yes |
How do you write a quotable passage?
This is the change with the highest return and the lowest cost. A paragraph starting "This means that..." and pointing backwards becomes unintelligible once extracted. Open instead by restating the subject and giving the answer immediately. Write "AI automation means that..." rather than "It means that...". It feels over-explicit while writing and reads completely normally on the page.
What structure helps the machine choose?
Headings that are actual questions, an answer directly beneath the heading, lists where the content is genuinely a list. Schema.org in JSON-LD states who stands behind the content and which questions the page answers. It is the same data that powers rich results, and what models lean on when judging whether a source is credible enough to cite.
How do you measure something analytics cannot see?
Traffic from AI answers often arrives without a referrer and therefore never appears as a channel in analytics. Two things are still measurable. The server log shows which AI crawlers actually fetch your pages and how often. And you can ask the models the same twenty questions about your category every month and keep the answers. The second is blunt, but it is the only thing that shows whether you are actually mentioned.
Should you block the AI crawlers instead?
It depends what you sell. A publisher whose business is its content has a real reason to block. A services company that wants to be recommended does not, because a blocked site cannot be cited. For most B2B companies, AI answers are a channel rather than a leak.
Common questions
How do I know whether AI crawlers can read my page?
Fetch the page without executing JavaScript, with curl for instance, and read what comes back. If headings, body copy and links are in the response, the foundation is sound. If what returns is an empty element filled in by a script, several AI crawlers see nothing at all. It takes thirty seconds and is the single most important check.
How long before you appear in AI answers?
Considerably longer than in ordinary search, and no date can be promised. The crawlers refetch at different intervals, and being cited also requires the model to judge you a reasonable source for that particular question. Count in months rather than weeks, and measure by asking the same questions at intervals.
Do we need llms.txt to get cited?
No. llms.txt is a proposal rather than an established standard, and support varies between vendors. It takes half an hour to generate from data you already have, so it is cheap to add, but the larger effect comes from content being present in the HTML and written in self-contained passages.
Is Cloudflare or our firewall blocking the AI crawlers?
Often yes, by default, and it is invisible in robots.txt because the block happens a layer above. It is the most common explanation I see for a site not being fetched at all despite a correct-looking robots.txt. Check the bot settings at your CDN or security vendor, then verify in the server log that the visits actually arrive.
