What share of sites name AI crawlers in robots.txt, and what does Shopify's default robots.txt say about them?
GPTBot is named in the robots.txt files of 4.5% of desktop sites in the Web Almanac 2025, and the default robots.txt on Shopify stores, checked October 18, 2026, names none of the five crawlers. Updated October 18, 2026.
By Martin DejnickiUpdated
Short answer
On desktop, GPTBot is named in the robots.txt files of 4.5% of sites, more than any other AI crawler, in the HTTP Archive Web Almanac 2025. Of the other four crawlers in Deploi's dataset, ClaudeBot is named by 3.6%, CCBot by 3.5%, Google-Extended by 3.4% and PerplexityBot by 2.8%. Shopify's default robots.txt names none of the five, checked on October 18, 2026. Naming a crawler does not mean blocking it.
GPTBot is named in the robots.txt of 4.5% of desktop sites, more than any other AI crawler, and Shopify's default file names none of the five
Share of desktop sites whose robots.txt names each AI crawler, web-wide, Web Almanac 2025; Shopify's default file, as stores serve it, names none
- Shopify's default robots.txt names none of these five crawlers.
- Shares are web-wide desktop figures from the Web Almanac 2025; no Shopify-only share is claimed.
- Naming a crawler is not the same as blocking it: a file can name a bot to allow it as well as to disallow it.
- Shopify's documentation does not list its default robots.txt groups; the default file served by Shopify stores checked on October 18, 2026 names none of the five crawlers.
- The Web Almanac figure names ten AI-related user agents, and the chart shows the five in Deploi's dataset; PetalBot, Amazonbot and FacebookBot are among the others, and on mobile PetalBot is named more often than GPTBot.
Source: HTTP Archive, Web Almanac 2025, SEO chapter; robots.txt files served by Shopify stores, checked October 18, 2026. Read on .
What this means for your decision
- Deciding whether to block AI crawlers? Start from Shopify's default. The robots.txt that Shopify stores serve, checked on allbirds.com and gymshark.com on October 18, 2026, names none of GPTBot, ClaudeBot, CCBot, Google-Extended or PerplexityBot, so a store that wants to treat one of them differently has to name it in its own file.
- Comparing with other sites? GPTBot is named in the robots.txt files of 4.5% of desktop sites in the Web Almanac 2025 and PerplexityBot in 2.8%, so most sites leave these crawlers unnamed.
- Reading a share as a block rate? Do not. A file can name a crawler to allow it as well as to disallow it, and the dataset holds no split between the two.
- Looking for a Shopify-only figure? The shares are web-wide. The dataset holds no Shopify-only share, only what Shopify's default file names, and a store's own file can differ from the default.
Which AI crawlers do robots.txt files name most often?
On desktop, GPTBot is the AI crawler named most often in robots.txt, by 4.5% of sites in the HTTP Archive Web Almanac 2025. Of the other four crawlers in Deploi's dataset, ClaudeBot is named by 3.6%, CCBot by 3.5%, Google-Extended by 3.4% and PerplexityBot by 2.8%. The shares are web-wide and count sites whose robots.txt names each crawler's user agent, whether to allow or to disallow it.
Each share is the portion of all desktop sites in the crawl whose robots.txt names the crawler's user agent. The Almanac figure names ten AI-related user agents, and the dataset holds five. PetalBot, Amazonbot and FacebookBot are among the others, and on mobile PetalBot is named more often than GPTBot.
Does naming an AI crawler mean blocking it?
No. A robots.txt file can name an AI crawler to allow it as well as to disallow it, so the Web Almanac 2025 shares of 2.8% to 4.5% of desktop sites count names, not blocks. The dataset holds no split between allow and disallow, so a share of sites that block AI crawlers cannot be taken from it.
A crawler that a file does not name falls under the file's group for all user agents, when the file has one.
What does Shopify's default robots.txt say about AI crawlers?
Shopify's default robots.txt names none of the five AI crawlers: GPTBot, ClaudeBot, CCBot, Google-Extended and PerplexityBot. Deploi checked the file served by Shopify stores, including allbirds.com and gymshark.com, on October 18, 2026. Shopify's documentation does not list its default groups, so the check rests on the live files.
A store's file can differ from the default if the merchant has changed it. The check covers the default as served on the day.
The data
Each chart on this page is drawn from the numbers below. Units are in the column headers.
| Row key | Share of desktop sites naming the bot (%) | Named in Shopify's default file |
|---|---|---|
| GPTBot | 4.5 | no |
| ClaudeBot | 3.6 | no |
| CCBot | 3.5 | no |
| Google-Extended | 3.4 | no |
| PerplexityBot | 2.8 | no |
Download AI crawlers named in robots.txt files, and Shopify's default robots.txt as CSV · Dataset page
Methodology
Reading the chart: GPTBot is named in the robots.txt of 4.5% of desktop sites, more than any other AI crawler, and Shopify's default file names none of the five
- Shopify's default robots.txt names none of these five crawlers.
- Shares are web-wide desktop figures from the Web Almanac 2025; no Shopify-only share is claimed.
- Naming a crawler is not the same as blocking it: a file can name a bot to allow it as well as to disallow it.
- Shopify's documentation does not list its default robots.txt groups; the default file served by Shopify stores checked on October 18, 2026 names none of the five crawlers.
- The Web Almanac figure names ten AI-related user agents, and the chart shows the five in Deploi's dataset; PetalBot, Amazonbot and FacebookBot are among the others, and on mobile PetalBot is named more often than GPTBot.
Caveats: AI crawlers named in robots.txt files, and Shopify's default robots.txt
- Shares are web-wide desktop figures from the Web Almanac 2025; no Shopify-only share is claimed.
- The Web Almanac figure names ten AI-related user agents and the dataset holds five. The other five are PetalBot (4.0% of desktop sites), Amazonbot (3.3%), FacebookBot (2.9%), ChatGPT-User (2.8%) and Meta-ExternalAgent (2.8%). On mobile, PetalBot (4.4%) is named more often than GPTBot (4.2%).
- Naming a crawler is not the same as blocking it: a file can name a bot to allow it as well as to disallow it. The dataset holds no allow or disallow split.
- Shopify's documentation does not list its default robots.txt groups. The Shopify column rests on the default file served by Shopify stores, including allbirds.com and gymshark.com, on October 18, 2026.
- A store's file can differ from the default if the merchant has changed it.
Deploi's charts and compilation are licensed CC BY 4.0. The underlying facts belong to their sources, which are named on every page. How Deploi Shopify Data works.
Sources
Tier A sources are primary documents: filings, official releases and official price pages. How sources are ranked.
- Tier BSEO | 2025 | The Web Almanac by HTTP Archive
HTTP Archive (Web Almanac) · HTML · Read on
- Tier AGymshark robots.txt (Shopify default groups, observed)
Gymshark (served by Shopify) · TEXT · Read on
- Tier Aallbirds.com/robots.txt (live Shopify-generated default robots.txt, example store), retrieved 2026-10-18
Shopify Inc. · TEXT · Read on
Cite and reuse
Deploi's charts and compilation are licensed CC BY 4.0. The underlying facts belong to their sources, which are named on every page.
APA
Deploi. (2026). What share of sites name AI crawlers in robots.txt, and what does Shopify's default robots.txt say about them? [Web page]. https://deploi.ca/shopify/data/ai-crawlers-blocked-in-robots-txt Retrieved October 18, 2026.
BibTeX
@misc{deploi_ai_crawlers_blocked_in_robots_txt_2026,
author = {{Deploi}},
title = {What share of sites name AI crawlers in robots.txt, and what does Shopify's default robots.txt say about them?},
year = {2026},
url = {https://deploi.ca/shopify/data/ai-crawlers-blocked-in-robots-txt},
note = {Accessed 2026-10-18}
}Embed this chart
Embed code
<iframe src="https://deploi.ca/embed/shopify/data/ai-bots-named-in-robots-txt" title="GPTBot is named in the robots.txt of 4.5% of desktop sites, more than any other AI crawler, and Shopify's default file names none of the five" loading="lazy" style="width:100%;height:550px;border:0"></iframe> <p style="font:12px/1.4 Arial,sans-serif;margin:4px 0 0">Source: <a href="https://deploi.ca/shopify/data/ai-crawlers-blocked-in-robots-txt#chart-ai-bots-named-in-robots-txt">Deploi, GPTBot is named in the robots.txt of 4.5% of desktop sites, more than any other AI crawler, and Shopify's default file names none of the five</a></p>
Keep the credit line under the embed. It is the license condition.
Questions
What percentage of sites block GPTBot?
The data show naming, not blocking. GPTBot is named in the robots.txt files of 4.5% of desktop sites in the HTTP Archive Web Almanac 2025, and a file can name a crawler to allow it as well as to disallow it.
Which AI crawler is named most in robots.txt files?
On desktop, GPTBot, named in the robots.txt files of 4.5% of sites in the HTTP Archive Web Almanac 2025. Of the other four crawlers in Deploi's dataset, ClaudeBot is named by 3.6%, CCBot by 3.5%, Google-Extended by 3.4% and PerplexityBot by 2.8%. On mobile, PetalBot, which the dataset does not hold, is named more often than GPTBot.
Does Shopify's default robots.txt block AI crawlers?
It names none of the five. The default robots.txt served by Shopify stores, checked October 18, 2026, does not name GPTBot, ClaudeBot, CCBot, Google-Extended or PerplexityBot, so it neither allows nor blocks them by name.
Can I see what share of Shopify stores block AI crawlers?
Not on this page. The Web Almanac 2025 shares, 2.8% to 4.5% of desktop sites for the five crawlers, are web-wide, and no Shopify-only share is claimed.
More on Performance and technology
- How do the agentic commerce protocols compare on launch dates, backers, surfaces and merchant requirements?
UCP, ACP, Shopify Catalog and Shopify Agentic Storefronts launched between May 21, 2025 and January 11, 2026; ACP listed 5 spec versions and UCP 4 on October 18, 2026. Updated October 19, 2026.
- Which Shopify deprecations and deadlines land in the next 12 months, and which have already passed?
Five dated Shopify deadlines in Deploi's dataset fall in the next 12 months, from January 16 to October 16, 2027; the other 12 have passed. Updated October 19, 2026.
- Do Hydrogen storefronts pass Core Web Vitals more often than Shopify sites overall?
In HTTP Archive's September 2026 mobile data, 50% of Hydrogen origins pass the Core Web Vitals assessment against 76% of all detected Shopify origins, computed by Deploi. Updated October 18, 2026.
- How many sites publish an llms.txt file, and who is publishing them?
Valid llms.txt files appeared on 2.13% of desktop sites and 2.1% of mobile sites in the Web Almanac 2025 crawl, and the Almanac traces 39.6% of llms.txt files to All in One SEO. Updated October 18, 2026.
- How many sites run Hydrogen on Shopify, and is the share rising?
Hydrogen runs on 0.34% of Shopify sites in HTTP Archive's September 2026 crawl, up from 0.19% in January 2025, computed by Deploi. Updated October 18, 2026.
- How many themes does the Shopify Theme Store list, and how many are free?
The Shopify Theme Store lists 1,326 themes, and only 24 are free, according to its own filter counts on October 18, 2026. Updated October 18, 2026.
Changes
- · AI crawlers named in robots.txt files, and Shopify's default robots.txt
First release. Shares read from the Web Almanac 2025 SEO chapter and the default robots.txt of Shopify stores checked on October 18, 2026.