What an AI crawler is
An AI crawler is an automated agent that fetches web content for AI systems, for training data, for retrieval indexes, or live at answer time. The distinction matters: blocking a training crawler affects future model knowledge, while blocking a retrieval crawler like PerplexityBot removes you from today's answers. Most visibility-minded sites allow both; the decision should be deliberate either way.
GPTBot and the robots.txt rules
GPTBot is OpenAI's crawler, identified by the GPTBot user agent and documented at openai.com's GPTBot page, which also lists its IP ranges for verification. To allow it, either say nothing, absence of a rule permits crawling, or be explicit: User-agent: GPTBot, Allow: /. To block: User-agent: GPTBot, Disallow: /. Partial rules work like any robots.txt entry, so you can disallow private paths while leaving marketing content open. OpenAI also operates OAI-SearchBot for search features and ChatGPT-User for on-demand fetches; treat them as separate user agents with separate rules.
Markdown and content negotiation: what is real
A frequent question is whether AI crawlers like GPTBot support content negotiation for markdown, requesting a cleaner text version of a page via Accept headers. As of early 2026, the major crawlers fetch standard HTML and do not reliably negotiate markdown alternates, so serving clean, parseable HTML remains the practical answer. The adjacent emerging conventions, llms.txt as a curated content map, and markdown mirrors of key pages, are inexpensive to add and some retrieval systems already read them, but treat them as progressive enhancement, not a substitute for well-structured HTML.
The allow-or-block decision
If AI visibility matters to your growth, the default is allow: assistants cannot recommend content their crawlers never saw. The legitimate exceptions are paywalled or licensed content, and competitive data you never want in training sets. Audit your robots.txt now, many sites blocked AI crawlers wholesale in 2023 via CMS defaults and forgot, and verify with each vendor's published IP ranges rather than trusting user-agent strings alone. Aethon's free audit includes a crawler-access check alongside the answer measurement, since a blocked GPTBot explains a lot of empty ChatGPT answers.
