Last Updated on July 10, 2026
Published by Whito Research · Last updated 2 July 2026 · Data checked July 2026
Key facts
- We audited 49 UK business websites in June 2026: 21 national brands and 28 independent local firms across 8 sectors.
- Only 1 of the 49 (Rightmove) actively blocks any AI crawler. 92% give no instruction to AI crawlers at all, which means they allow them by default.
- 18% had an llms.txt file, the emerging “summary for AI” standard. Two thirds of those were created automatically by the website platform, not by the business.
- The most AI-ready sites were not the big brands. Three independent firms on Wix (a Nottingham gym, a Glasgow restaurant, a Bristol vet) expose a live endpoint that lets AI agents read their details and book appointments. The owners almost certainly do not know.
There is a debate going round about whether businesses should “block AI” from their websites. Newspapers block it. Some big publishers block it. So the question lands on the desk of every small business owner: should I let ChatGPT and Google read my site, or should I lock the door?
It is the wrong question. And we can prove most UK businesses have already answered it without realising.
We went and looked. In June 2026 we checked the actual instructions that 49 UK business websites give to AI crawlers. Not opinions. The files themselves. What we found is that the door is already wide open almost everywhere. The thing nobody has done is make the room worth walking into.
Key statistics from this study (cite these)
- 2%: only 1 of 49 UK business websites audited (Rightmove) actively blocks any AI crawler. Source: Whito, The AI Door Test, June 2026.
- 92%: 45 of 49 give AI crawlers no instruction at all in robots.txt, which allows them by default.
- 18%: 9 of 49 publish an llms.txt file, but 6 of those 9 were generated automatically by the website platform or SEO plugin, not by the business.
- 3 of 28 independent UK businesses audited expose a live Model Context Protocol (MCP) endpoint that lets AI agents read their details and begin a booking, all on the Wix platform.
- 0 of 49 small or independent businesses had made a deliberate, owner-led decision to either block or formally welcome AI crawlers.
The mistake almost everyone is making
Most owners think the AI question is about access: do I let it in or keep it out. That feels like the big decision, so it gets the attention.
The real question is comprehension. Once an AI system is on your site, can it understand what you do, who you serve, what you charge and how to recommend you? Access is a switch. Comprehension is structure. And structure is the part everyone is ignoring.
Think of it like a shop. The access debate is about whether to unlock the front door. Useful, but trivial. What actually decides whether anyone buys is whether the shelves are labelled, the prices are visible and a stranger can find what they need in ten seconds. Most UK businesses have unlocked the door and left the shelves in a heap.
What we did
We built a sample of 49 UK business websites. 21 were well known national brands (John Lewis, Tesco, Monzo, Specsavers, Screwfix and similar). 28 were independent local businesses we found through ordinary searches: plumbers, dentists, accountants, salons, vets, solicitors, gyms and restaurants, spread across London, Manchester, Birmingham, Leeds, Bristol, Nottingham and Glasgow.
For each one we read two files that decide how AI sees a website:
robots.txt is the instruction sheet every crawler reads first. It is where a business would block or allow AI bots like GPTBot (ChatGPT), Google-Extended (Gemini), ClaudeBot and PerplexityBot.
llms.txt is newer. It is a plain summary of a site written for AI, listing the key pages so a model does not have to guess. Think of it as a contents page handed straight to the machine.
The full dataset is published below so anyone can repeat the check.
Definitions: the files and bots that decide what AI sees
robots.txt is a plain text file at the root of a website that tells automated crawlers which parts of the site they may visit. It is where a business would allow or block AI crawlers.
llms.txt is a newer plain text file that summarises a website for large language models, listing the key pages so a model does not have to infer them. It is the AI equivalent of a contents page.
Model Context Protocol (MCP) endpoint is a live connection point that lets an AI agent query a website directly for current information and perform actions such as starting a booking, without scraping the page.
The main AI crawlers a UK business would see are GPTBot and OAI-SearchBot (OpenAI and ChatGPT), Google-Extended (Google Gemini and AI Overviews), ClaudeBot (Anthropic), PerplexityBot (Perplexity) and CCBot (Common Crawl).
Finding one: nobody is blocking AI
Of 49 UK business websites, exactly one blocks any AI crawler. Rightmove tells GPTBot to stay out of most of the site. That is the entire list.
Everyone else falls into two groups. A small number have written an explicit “yes, AI is welcome” instruction. The other 45, which is 92% of the sample, say nothing about AI at all. Saying nothing is not neutral. A crawler that finds no rule against it treats itself as allowed. Silence is a yes.
| What the website tells AI crawlers | Sites | Share |
|---|---|---|
| Nothing at all (allowed by default) | 45 | 92% |
| Explicitly welcomes AI | 1 | 2% |
| AI bots named by the platform, left allowed | 2 | 4% |
| Actively blocks an AI crawler | 1 | 2% |
So the “should I block AI?” argument is, for almost every UK business, already settled. You are in. The decision was made for you by the default. Spending another minute on it is wasted energy.
Finding two: the room is empty
If access is solved, the next question is whether AI can make sense of the site once inside. The clearest single signal is the llms.txt file: a deliberate summary that tells a model what matters.
9 of the 49 sites had one. That is 18%. On the surface, better than expected. Underneath, the picture is different. Of those 9 files, 6 were generated automatically by the website platform or an SEO plugin. The business did not write them and in most cases does not know they exist. Only 3 looked like a deliberate choice, and all 3 were large retailers (Argos, ASOS and MoneySuperMarket) with teams paid to think about this.
Who has an llms.txt, and did they mean to?
40 sites
6 sites
3 sites
This matches what we found in our earlier audit of UK websites, where 96% had no structured FAQ data for search engines or AI to read. The pattern is consistent. The doors are open. The shelves are bare. And where they are not bare, it is usually because a tool tidied them automatically, not because anyone made a decision.
Finding three: the most AI-ready businesses are the small ones
This is the part we did not expect.
The national brands have budgets, agencies and SEO teams. Yet the most advanced AI access in the entire sample belonged to three independent local firms: a gym in Nottingham, a trattoria in Glasgow and a vet in Bristol. All three are built on Wix. And Wix has started publishing an llms.txt on these sites that does more than summarise. It hands AI a live connection point, an endpoint that lets an AI agent read the business details and even start booking an appointment, without scraping anything.
Read that again. A 24/7 AI assistant can, in principle, query that Nottingham gym for its opening hours and membership options and walk a customer towards booking, on the gym’s behalf, around the clock. The gym did not build it. Wix switched it on. The owner almost certainly has no idea it is there.
The lesson is not “use Wix”. The lesson is that AI-readiness has nothing to do with your size or your budget. It is decided by the platform you happen to sit on and whether you have done the basic structural work. A plumber on the right setup can be more findable by AI than a national chain. That is a rare moment where the small business has the advantage, if it acts.
What each website platform does for AI by default
The single biggest factor in whether a UK business is readable by AI was not its budget or its size. It was the platform its website sits on. Here is what we observed each platform doing on its own, before the owner changes anything.
| Platform | What it does for AI readability by default |
|---|---|
| Wix | Publishes an llms.txt and a live MCP endpoint, so AI agents can read details and begin a booking. The most AI-ready setup in the sample. |
| Yoast SEO (recent versions) | Generates an llms.txt summary automatically. |
| All in One SEO | Generates an llms.txt summary automatically. |
| Squarespace | Lists every known AI crawler by name in robots.txt, but leaves them allowed unless the owner switches a block on. |
| WordPress with no SEO plugin, Webflow, most custom builds | Nothing specific. AI is allowed in by default, but gets no summary and no structured help. |
The takeaway for an owner is simple. Before paying for anything, find out what your platform already does, then fill the gap it leaves.
Why this matters for what your business can actually do
None of this would matter if people had stopped using search to find businesses. The opposite is happening. They are using it more, just differently.
In an April 2026 survey of 2,564 UK consumers, 78% said they now use AI chat tools such as ChatGPT. More than half said AI had helped them discover a brand they would never have found otherwise, and nearly half said they would consider buying from a brand they had never heard of if AI recommended it. At the same time, the number of Google searches that end without a single click to a website has climbed to roughly 6 in 10, because the answer now appears on the results page itself.
Put those two trends together and the shift is clear. A growing share of buying decisions is being shaped before anyone reaches your website. The AI is doing the shortlisting. If it cannot understand your site, you are not on the shortlist, and you will never see the lost enquiry in your analytics because it never arrived.
That is the revenue line. Not vanity reach. Quotes you do not get asked for. Bookings that go to the firm the AI could actually read. This is the same pattern we tracked in our work on the state of AI adoption among UK businesses: the tools have moved faster than the habits.
What to do about it, in order
The temptation is to chase the shiny end of this: MCP endpoints, AI agents, automation. Do not. That is scale work, and scale work on weak foundations is wasted money. Fix the structure first, in this order.
| Stage | What to fix |
|---|---|
| Start | Make the basics legible. State plainly, in text, what you do, where you work, who you serve and what it costs. Put your real answers to common customer questions on the page as words, not in a PDF or an image. If a stranger cannot understand your business in ten seconds, neither can a machine. |
| Build | Add structure a machine can read. A proper FAQ section, clear headings, and schema markup so search engines and AI can lift your facts cleanly. Check what your own website already tells AI. Read your robots.txt and look for an llms.txt. You may find your platform has made decisions for you. |
| Scale | Only now, consider the live connections: agent endpoints, automated booking, richer feeds. These multiply a strong foundation. They do nothing for a weak one. |
How we ran this
Between 20 and 26 June 2026 we requested the robots.txt and llms.txt files for all 49 sites and recorded, for each, whether any AI crawler was named, whether any was blocked, and whether an llms.txt existed and what produced it. We classed a site as “blocking AI” only where it explicitly disallowed a named AI crawler from its main content. The sample is a deliberate spread of brands and independents, not a random national sample, so treat the percentages as a clear signal rather than a precise national figure. The point holds either way: the access debate is over, and the structure work has barely begun.
The takeaway
Stop asking whether to let AI in. You already have. Start asking whether it can understand you once it is there. The businesses that win the next few years will not be the ones with the biggest budgets. They will be the ones whose websites are built so a machine can read them, summarise them and recommend them without having to guess. Structure before scale. As ever.
Frequently asked questions
Should my small business block AI from my website?
For almost all UK businesses, no. The benefit of being understood and recommended by AI tools that 78% of UK consumers now use far outweighs the downside. Our audit found only 1 in 49 UK business sites blocks any AI crawler. The bigger risk is being invisible, not being read.
What is an llms.txt file and do I need one?
It is a plain text summary of your website written for AI models, listing your key pages so a model does not have to work them out. It is not essential yet, but it helps. Many website platforms and SEO plugins now create one automatically, so check whether you already have one at yourdomain.co.uk/llms.txt.
How do I know if AI can already read my website?
Visit yourdomain.co.uk/robots.txt to see what instructions you give crawlers, and yourdomain.co.uk/llms.txt to see if a summary exists. Then check that your core facts, services, locations, prices and answers to common questions, appear as actual text on your pages rather than buried in images or PDFs.
Is appearing in AI answers actually worth anything?
It is increasingly where the decision happens. Around 6 in 10 Google searches now end without a click because the answer shows on the page, and roughly half of UK AI users say they would consider a brand they had never heard of if AI suggested it. If AI cannot understand your site, you are left off the shortlist before a customer ever reaches you.
About this data
This study audited the publicly served robots.txt and llms.txt files of 49 UK business websites in June 2026, recording which AI crawlers each site allows or blocks. No private data was accessed.
Cite this research
Whito Research (2026). The AI Door Test: We Checked What 49 UK Business Websites Tell AI (Only 1 Blocks It). Whito. https://whito.co.uk/research/ai-door-test-uk-business-websites/
Key finding: Of 49 UK business websites audited in June 2026, only one blocks AI crawlers; 92 per cent allow them by default and three expose live machine-readable endpoints.
This is original Whito research. You are welcome to reuse these figures with a link to this page as the source.
