Last verified: October 2026.
Two small files on your website affect how AI tools treat it. robots.txt can stop AI companies collecting your content, and llms.txt is meant to help AI tools find their way around. We don’t recommend spending time on llms.txt yet, because the big AI companies don’t use it. Keep letting AI crawlers in through robots.txt unless they are slowing your site down or pushing up your hosting bill.
What each file is
These files sit at fixed addresses on your site and are written for automated programs, usually called crawlers or bots.
robots.txt (yoursite.org/robots.txt) tells crawlers which parts of your site they may visit, and can give different rules to different crawlers by name. Search engines have followed it for decades, and the major AI companies say they follow it too. It works on trust, so a badly behaved crawler can ignore it.
llms.txt (yoursite.org/llms.txt) is a short summary of your site written for AI models, with links to your most important pages. The proposal is meant to help AI tools find their way around a site quickly.
llms-full.txt holds the full text of your pages in one file. Documentation platforms such as Mintlify generate it for software documentation. In our review of animal advocacy organization websites, almost none had one.
llms.txt: no sign that AI companies read it
A tidy summary of your site could help an AI tool describe your work accurately, but there is no evidence the major AI companies read these files.
- Google’s John Mueller wrote in June 2025: “FWIW no AI system currently uses llms.txt.”
- Google’s guide to appearing in its AI search features, last updated in July 2026, says you can keep one if other services use it, but it “will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.”
- Chrome’s Lighthouse website checker lists llms.txt as an “emerging convention” but calls providing one “optional at the moment.”
- We found no statement from OpenAI or Anthropic saying their products read llms.txt.
In our own testing, ChatGPT and Gemini only looked at an advocacy organization’s llms.txt when we told them to.
So we don’t recommend writing or maintaining an llms.txt file for now. If your website platform already makes one, there is no need to remove it. We’ll update this guide if llms.txt becomes useful.
robots.txt: AI companies say they follow it
The major AI companies publish the names of their crawlers and explain how to block each one in robots.txt. Anthropic, for example, says its crawlers honor “industry standard directives in robots.txt.” Each company runs separate crawlers for separate jobs, so you can block one job and allow another.
| Company | Name in robots.txt | What blocking it does |
|---|---|---|
| OpenAI | GPTBot | Your content isn’t used to train OpenAI’s models. |
| OpenAI | OAI-SearchBot | Your site won’t appear in ChatGPT search answers. |
| Anthropic | ClaudeBot | Your future content is left out of Anthropic’s training data. |
| Anthropic | Claude-SearchBot | Anthropic won’t index your site for Claude’s search results. |
Google-Extended | Your content isn’t used to train Gemini or to ground its answers. Google Search is not affected. |
Blocking has limits:
- It only affects what is collected from now on. Content already gathered stays in existing training data.
- It may not stop ChatGPT from opening a page when a person asks it to. OpenAI says that for these requests, “robots.txt rules may not apply.”
- Not every company plays by the rules. In August 2025, Cloudflare reported that Perplexity used undeclared crawlers to get around robots.txt blocks. Perplexity disputed this.
Should you let AI crawlers in or block them?
The case for letting them in. AI models learn from huge amounts of text gathered from the web, and we want pro-animal content to be part of it, so that models answer questions about factory farming and animal welfare accurately and fairly. Each advocacy site that blocks AI crawlers keeps its future content out. Letting search crawlers in also means tools like ChatGPT search and Claude can find and cite your work. For most organizations, this is the right choice.
The case for blocking them. AI crawlers can request many pages very quickly, which on some sites raises hosting costs or slows the site for human visitors. The Wikimedia Foundation, which runs Wikipedia, found that bots caused at least 65% of its most resource-heavy traffic. If your hosting bill jumps or your site slows down, and your hosting dashboard or server logs show AI crawlers are the cause, blocking them is a reasonable response.
If you do block, keep it narrow:
- Block only the crawlers causing the load.
- Consider slowing a crawler down instead. Anthropic’s crawlers support a
Crawl-delayline, which sets a minimum gap in seconds between their requests. - Blocking
Google-Extendedwon’t reduce load. Google collects pages with its normal search crawler, so blocking it changes how your content is used but not how often Google visits. - Never block every crawler with
User-agent: *andDisallow: /. That also removes your site from Google Search.
These lines block OpenAI’s training crawler and ask Anthropic’s training crawler to wait 10 seconds between requests, leaving both companies’ search crawlers alone:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Crawl-delay: 10
Check what your site does today
In our review of animal advocacy organization websites, more than 95% let AI crawlers in, so most organizations don’t need to change anything. A quick check still helps, because some sites block AI crawlers, possibly by accident.
Open yoursite.org/robots.txt in your browser. A crawler’s name, such as GPTBot or ClaudeBot, followed by Disallow: / means that company is blocked. A “page not found” error means you have no robots.txt file, and every crawler is allowed.
Of the 79 websites in our review that blocked AI crawlers, 58 used a one-click setting in Cloudflare or Squarespace, and only 21 had written their own rules. If your site blocks AI crawlers and nobody remembers choosing to, check those settings or ask whoever manages your website.
This guide draws on work in collaboration with Stray Dog Institute and Kyle Behrend.

