domain,retailer,names_ai_agents_in_robots_txt,ai_agents_named,treatment_of_named_ai_agents,checked
www.amazon.co.uk,Amazon UK,yes,"GPTBot; CCBot; PerplexityBot; Google-Extended; ClaudeBot; meta-externalagent; meta-externalfetcher; Bytespider; Claude-User; Claude-SearchBot; Claude-Web; Perplexity-User; ChatGPT-User; OAI-SearchBot","Every named AI agent is given Disallow: / (full block)",2026-10-16
www.ebay.co.uk,eBay UK,yes,"GPTBot; OAI-SearchBot; ChatGPT-User; PerplexityBot; ClaudeBot; Claude-User; Claude-SearchBot; anthropic-ai; Amazonbot; Bytespider; meta-externalagent; CCBot","GPTBot, PerplexityBot, ClaudeBot, anthropic-ai, Amazonbot, Bytespider, meta-externalagent and CCBot are blocked site-wide with three narrow carve-outs; OAI-SearchBot, ChatGPT-User, Claude-User and Claude-SearchBot get near-standard access (192 disallow rules and 5 allows, against 197 disallow rules and 8 allows for all other visitors)",2026-10-16
www.tesco.com,Tesco,no,none,"No AI-specific rules; one User-agent: * group (26 disallow rules) applies to everything",2026-10-16
www.sainsburys.co.uk,Sainsbury's,no,none,"No AI-specific rules; one User-agent: * group (83 disallow rules) applies to everything",2026-10-16
www.argos.co.uk,Argos,no,none,"No AI-specific rules; groups for *, Googlebot, Googlebot-Image and Bingbot only",2026-10-16
www.currys.co.uk,Currys,no,none,"No AI-specific rules; one User-agent: * group applies to everything",2026-10-16
www.johnlewis.com,John Lewis,no,none,"No AI-specific rules; one User-agent: * group applies to everything",2026-10-16
www.next.co.uk,Next,no,none,"No AI-specific rules; 15 groups name ad, social and image bots (AdsBot-Google, TikTokBot, Pinterestbot and others) but no AI crawler or agent",2026-10-16
www.asda.com,Asda,no,none,"No AI-specific rules; one User-agent: * group (9 disallow rules) applies to everything",2026-10-16
www.marksandspencer.com,Marks and Spencer,no,none,"No AI-specific rules; one User-agent: * group (139 disallow rules) applies to everything",2026-10-16
www.boots.com,Boots,no,none,"No AI-specific rules; groups for * and AdsBot-Google only",2026-10-16
www.asos.com,ASOS,no,none,"No AI-specific rules; groups for *, AdsBot-Google, Twitterbot and Snapbot only",2026-10-16
www.very.co.uk,Very,no,none,"No AI-specific rules; groups for * and eight Baidu crawler variants only",2026-10-16
www.screwfix.com,Screwfix,no,none,"No AI-specific rules; one User-agent: * group (350 disallow rules) applies to everything",2026-10-16
www.diy.com,B&Q,no,none,"No AI-specific rules; groups for *, AdsBot variants, Yandex and Pinterestbot only",2026-10-16
www.ocado.com,Ocado,no,none,"No AI-specific rules; groups for * and OneTrustBot only",2026-10-16
www.jdsports.co.uk,JD Sports,no,none,"No AI-specific rules; one User-agent: * group (38 disallow rules) applies to everything",2026-10-16
www.dunelm.com,Dunelm,no,none,"No AI-specific rules; groups for *, Googlebot-Image and AdsBot variants only",2026-10-16
ao.com,AO,no,none,"No AI-specific rules; groups for * and GoogleOther only",2026-10-16
www.wickes.co.uk,Wickes,no,none,"No AI-specific rules; one User-agent: * group (34 disallow rules) applies to everything",2026-10-16