Psyche Creative — Connecting Humanity to Healing
Journal/GPTBot and Other AI Crawlers Explained: What to Allow on a Therapy Website
SEO & AI SearchNov 2026·10 min read·By Adam Watters

GPTBot and Other AI Crawlers Explained: What to Allow on a Therapy Website

Open the robots.txt file of almost any large website now and you'll find a list of names that didn't exist a few years ago: GPTBot, OAI-SearchBot, Google-Extended, PerplexityBot, ClaudeBot. Some publishers block them all. Some allow them all. Many small business owners don't know the file exists. For a therapist, the decision touches two things you care about: being findable in the places people now ask for recommendations, and having a say over how your writing is used. This guide explains what each crawler does, what blocking it actually changes, and how we handle it on the sites we build.

GPTBotAI Crawlersrobots.txtGoogle-ExtendedAI SearchTechnical SEO
GPTBot and Other AI Crawlers Explained: What to Allow on a Therapy Website
Quick answer

GPTBot is OpenAI's crawler for collecting content to train its models. It's separate from OAI-SearchBot, which powers ChatGPT search, and from ChatGPT-User, which fetches pages when a person asks. Other AI crawlers split the same way: some gather training data, some build search indexes, some act for a user. You control them in robots.txt. Blocking a training crawler keeps your words out of future training; blocking a search crawler can keep you out of that tool's answers.

What is a crawler, and what is robots.txt?

A crawler (or bot) is a program that visits web pages and reads them. Search engines have used crawlers for decades to build their indexes. AI companies now run their own, for training models, for powering AI search, or for fetching a page when someone asks an assistant to read it.

Robots.txt is a small public text file at the root of your website (yoursite.co.uk/robots.txt) that tells crawlers which parts of the site they may visit. Each crawler identifies itself with a name, called a user agent, and the file can give different rules to different names. The major companies say their crawlers respect it. It is a request, though, and it isn't a security measure: a badly behaved bot can ignore it, and anything that must stay private shouldn't be on a public page in the first place.

Three kinds of AI crawler: training, search and user-triggered

The single most useful thing to understand is that AI crawlers do different jobs, and blocking one doesn't block the others.

  • —Training crawlers collect public web content that may be used to train future AI models. Blocking them means your content shouldn't be included in future training. It doesn't remove anything already learned.
  • —Search crawlers build the index an AI search tool uses to find and cite pages when answering questions. Blocking them can mean your site isn't shown or cited in that tool's answers.
  • —User-triggered fetchers visit a page because a person asked an assistant to look at it, for example by pasting in a link. They behave more like a browser acting on someone's behalf.

GPTBot, Google-Extended and the main AI crawlers compared

These are the crawlers that matter most for a UK therapy website at the time of writing (November 2026). Companies add and rename crawlers from time to time, so it's worth checking each company's own documentation before making changes.

The main search and AI crawlers, and what blocking each one changes (November 2026)
Crawler (user agent)CompanyWhat it doesWhat blocking it changes
GooglebotGoogleCrawls pages for Google Search, including AI Overviews and AI Mode.Removes you from Google Search entirely, AI features included. Almost never what a practice wants.
Google-ExtendedGoogleA robots.txt control, not a separate crawler. It governs whether content Google crawls can be used to train Gemini models and for grounding in Gemini apps.Opts your content out of that Gemini use. Google says it doesn't affect Search ranking or inclusion in AI Overviews.
BingbotMicrosoftCrawls pages for Bing, whose index also feeds Microsoft Copilot and is one of the sources behind ChatGPT search.Removes you from Bing, and with it much of your reach in Copilot and ChatGPT search.
GPTBotOpenAICollects content that may be used to train OpenAI's models.Your content shouldn't be used in future OpenAI training. It doesn't remove you from ChatGPT search.
OAI-SearchBotOpenAIBuilds the index used to show and link websites in ChatGPT search answers.OpenAI says blocked sites won't be shown in ChatGPT search answers, though they may still appear as plain navigational links.
ChatGPT-UserOpenAIFetches a page when a ChatGPT user or custom GPT asks to read it.Limited. OpenAI's documentation has said that because these visits are started by a person, robots.txt rules may not apply to them in the same way.
PerplexityBotPerplexityIndexes pages so Perplexity can find and cite them in answers. Perplexity says it isn't used to train foundation models.Your pages are less likely to be found and cited in Perplexity answers.
ClaudeBotAnthropicCollects public web content that may be used to train Anthropic's Claude models. Anthropic also documents separate agents for search (Claude-SearchBot) and user requests (Claude-User).Your content shouldn't be used in future Claude training. Search and user fetches are controlled by their own names.

The pattern to take from this: training crawlers and search crawlers are separate decisions. A practice can choose not to contribute its writing to model training while staying fully visible in search and AI search, or the other way round.

What does blocking GPTBot actually change?

Blocking GPTBot asks OpenAI not to use your site's content in training future models. That's all. It doesn't remove your site from ChatGPT search, because that uses OAI-SearchBot and Bing's index. It doesn't erase anything a model has already learned. And it doesn't stop a ChatGPT user pasting your page address into a conversation.

For most therapists, GPTBot is the crawler people are thinking of when they say they'd like to 'block AI'. It's a reasonable choice to make, as long as you know it's a choice about training rather than about visibility.

Robots.txt examples

Robots.txt is written as short groups of lines. Each group names a user agent, then says what it may or may not visit. 'Disallow: /' means the whole site; 'Allow: /' means everything is permitted. Here are some common setups, shown line by line.

Allow everything (our default)

  • —User-agent: *
  • —Allow: /
  • —Sitemap: https://www.example.co.uk/sitemap.xml

Stay in search and AI search, opt out of AI training

  • —User-agent: GPTBot
  • —Disallow: /
  • —User-agent: Google-Extended
  • —Disallow: /
  • —User-agent: ClaudeBot
  • —Disallow: /
  • —User-agent: *
  • —Allow: /
  • —Sitemap: https://www.example.co.uk/sitemap.xml

Opt out of one AI search tool

  • —User-agent: PerplexityBot
  • —Disallow: /

Two cautions. First, never write 'User-agent: *' followed by 'Disallow: /' unless you mean to block every search engine; we see this left over from development more often than you'd think. Second, robots.txt controls crawling. For finer control over how Google shows your content, including in AI Overviews, Google supports page-level settings such as nosnippet and max-snippet, which limit how much text it can show.

Privacy and consent considerations for therapists

Therapists rightly think harder about this than most businesses. A few points help keep the decision in proportion.

  • —AI crawlers read public pages, the same pages Google and Bing already read. They can't see your client notes, booking system, contact form submissions or anything behind a login, any more than a search engine can.
  • —Client information should never be on a public page, whatever the crawler settings. Robots.txt is not a privacy tool and shouldn't be treated as one.
  • —Don't list private areas in robots.txt to 'hide' them. The file is public, so a Disallow line pointing at a private folder advertises that it exists. Private areas need proper access control.
  • —The real question is about your published writing: your articles, your approach page, your thoughts on trauma or grief. Whether you're comfortable with that informing AI models is a values question, and either answer is legitimate.
  • —If you write about clinical themes using composite or illustrative examples, make sure they are fully non-identifiable. That matters for any public page, and AI training only raises the stakes.

I'd separate the two questions. Do you want people who ask an AI assistant for help to be able to find you? Most therapists do. Do you want your writing used to train models? That's a personal call, and we'll set the site up whichever way you choose.

Adam Watters, Founder, Psyche Creative

Our default, and how a client can opt out

By default, the sites we build allow search engines and AI crawlers, including GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot and Google-Extended. We do that because more people now ask AI tools for help finding services, and a practice that blocks those tools can't be found there. We think that's the sensible default for a business that wants enquiries.

It's your site and your writing, so it's your decision. If you'd prefer to opt out of AI training, or of a particular service, tell us and we'll change the robots.txt to match, explain exactly what that changes, and note it in your site documentation so it isn't accidentally reversed later.

How we build this into every website we make

  • —A clean robots.txt that allows search and AI crawlers by default, points to your XML sitemap and is adjusted to your preferences on request.
  • —Server-rendered pages with the full content in the HTML, so crawlers that run little or no JavaScript still read everything.
  • —Canonical URLs, clean URLs and per-page meta titles and descriptions, so whichever system reads the page gets a clear picture.
  • —Google Search Console and Bing Webmaster Tools verified on launch, since Bing's index feeds Copilot and ChatGPT search.
  • —An llms.txt file with your key facts. Google has said it doesn't use llms.txt; other tools may, and it costs nothing to keep accurate.
  • —Analytics that track visits from AI tools such as chatgpt.com, perplexity.ai, copilot.microsoft.com and gemini.google.com, so you can see whether they matter for your practice.

For the plain-text file some AI tools read, and the honest truth about it, see our guide. Read: llms.txt Explained →

Robots.txt is one line on a longer technical checklist worth running on any therapy website. Read: Technical SEO Checklist for Therapist Websites →

Not sure what your current site's robots.txt says, or whether it's blocking something it shouldn't? We'll check it on a free call. Get in touch →

Frequently asked questions

What is GPTBot?

GPTBot is OpenAI's web crawler for collecting public content that may be used to train its AI models. It's separate from OAI-SearchBot, which powers ChatGPT search, so blocking GPTBot doesn't remove your site from ChatGPT search results.

Will blocking Google-Extended hurt my Google rankings?

Google says it won't. Google-Extended only controls whether your content is used for Gemini model training and grounding in Gemini apps. Google Search, including AI Overviews, is governed by Googlebot.

Can AI crawlers see my clients' information?

No. Crawlers only read public web pages. Client notes, booking systems, form submissions and anything behind a login aren't visible to them, just as they aren't visible to Google. Client information should never be on public pages in any case.

Should a therapist block AI crawlers?

It depends on what you want. If you'd like to be findable in AI search tools, allow their search crawlers. If you'd rather your writing wasn't used to train models, you can block training crawlers such as GPTBot, Google-Extended and ClaudeBot while staying visible in search.

Does robots.txt guarantee a bot won't read my site?

No. It's a public request that the major companies say they honour, not an enforcement mechanism. Anything that must stay private needs a login or should not be published.

How do I see my website's robots.txt?

Add /robots.txt to the end of your domain in a browser, for example yoursite.co.uk/robots.txt. If you see 'Disallow: /' under 'User-agent: *', your site is asking all search engines not to crawl it, and that's worth fixing quickly.

AW
Written by
Adam Watters
Founder, Psyche Creative · Digital marketing for UK therapy practices

Adam works with UK therapy practices, EMDR trainers and mental health organisations on websites, social media and digital campaigns. He started Psyche Creative to give clinicians a marketing partner who understands the sector — and won't compromise on professional standards to get results.

Found this useful?

Let's talk about your practice.

If this article resonated, a free 30-minute call is the natural next step.

Book a Free Call →