logo

AI crawlers named in robots.txt files, and Shopify's default robots.txt

Version 1. 5 rows, 3 columns.

By Martin DejnickiUpdated

Short answer

The dataset gives the share of desktop sites whose robots.txt names each of five AI crawlers, GPTBot, ClaudeBot, CCBot, Google-Extended and PerplexityBot, from the HTTP Archive Web Almanac 2025 SEO chapter. It also records whether the default robots.txt served by Shopify stores names each crawler, checked on allbirds.com and gymshark.com on October 18, 2026. Shares are web-wide.

GPTBot is named in the robots.txt of 4.5% of desktop sites, more than any other AI crawler, and Shopify's default file names none of the five

Share of desktop sites whose robots.txt names each AI crawler, web-wide, Web Almanac 2025; Shopify's default file, as stores serve it, names none

GPTBot 4.5%ClaudeBot 3.6%CCBot 3.5%Google-Extended 3.4%PerplexityBot 2.8%ClaudeBotCCBotGoogle-ExtendedPerplexityBotGPTBot3.6%3.5%3.4%2.8%4.5%

  • Shopify's default robots.txt names none of these five crawlers.
  • Shares are web-wide desktop figures from the Web Almanac 2025; no Shopify-only share is claimed.
  • Naming a crawler is not the same as blocking it: a file can name a bot to allow it as well as to disallow it.
  • Shopify's documentation does not list its default robots.txt groups; the default file served by Shopify stores checked on October 18, 2026 names none of the five crawlers.
  • The Web Almanac figure names ten AI-related user agents, and the chart shows the five in Deploi's dataset; PetalBot, Amazonbot and FacebookBot are among the others, and on mobile PetalBot is named more often than GPTBot.

Source: HTTP Archive, Web Almanac 2025, SEO chapter; robots.txt files served by Shopify stores, checked October 18, 2026. Read on .

About this dataset

The dataset gives the share of desktop sites whose robots.txt names each of five AI crawlers, GPTBot, ClaudeBot, CCBot, Google-Extended and PerplexityBot, from the HTTP Archive Web Almanac 2025 SEO chapter. It also records whether the default robots.txt served by Shopify stores names each crawler, checked on allbirds.com and gymshark.com on October 18, 2026. Shares are web-wide.

Version
1
Coverage in time
Web Almanac 2025 crawl; Shopify default file checked October 18, 2026
Coverage in space
Worldwide (websites on the open web; no country or platform split)
Last updated
October 18, 2026
License
CC BY 4.0. Deploi's compilation is CC BY 4.0. The underlying figures belong to the HTTP Archive, the Web Almanac authors and Shopify Inc. and are cited to the sources listed.

Variables

The CSV header uses the column name.

Variables in AI crawlers named in robots.txt files, and Shopify's default robots.txt
Column nameLabelTypeUnitMeaning
rowKeyRow keystringStable key of the row; matches provenance rowKey.
desktopSharePctShare of desktop sites naming the botnumber%Share of all desktop sites in the Web Almanac 2025 crawl whose robots.txt names the crawler's user agent
inShopifyDefaultNamed in Shopify's default filestringtextyes or no: whether the default robots.txt served by Shopify stores names the crawler, from the live files of allbirds.com and gymshark.com checked on October 18, 2026

The data

AI crawlers named in robots.txt files, and Shopify's default robots.txt
Row keyShare of desktop sites naming the bot (%)Named in Shopify's default file
GPTBot4.5no
ClaudeBot3.6no
CCBot3.5no
Google-Extended3.4no
PerplexityBot2.8no

Download the full dataset as CSV

Where each value comes from

Every value, the document it was read from, the exact place in it and the date it was read. Computed values show their formula.

Show all 10 traced values
Provenance of each value
RowFieldValueSourceWhere in the sourceRead onMethod
GPTBotShare of desktop sites naming the bot4.5 %SEO | 2025 | The Web Almanac by HTTP Archive
gptbot: 4.5% (desktop), 4.2% (mobile) is up from 2.9% (desktop) and 2.7% (mobile) in 2024, representing a ~55% increase.
html
GPTBotNamed in Shopify's default filenoallbirds.com/robots.txt (live Shopify-generated default robots.txt, example store), retrieved 2026-10-18User-agent: Pinteresttext
ClaudeBotShare of desktop sites naming the bot3.6 %SEO | 2025 | The Web Almanac by HTTP Archiveclaudebot: 3.6% (desktop), 3.4% (mobile) is up from 1.9% (desktop) and 1.6% (mobile) in 2024, nearly doubling.html
ClaudeBotNamed in Shopify's default filenoallbirds.com/robots.txt (live Shopify-generated default robots.txt, example store), retrieved 2026-10-18User-agent: Pinteresttext
CCBotShare of desktop sites naming the bot3.5 %SEO | 2025 | The Web Almanac by HTTP Archiveccbot: 3.5% (desktop), 3.2% (mobile) is up from 2.7% (desktop) and 2.4% (mobile) in 2024.html
CCBotNamed in Shopify's default filenoallbirds.com/robots.txt (live Shopify-generated default robots.txt, example store), retrieved 2026-10-18User-agent: Pinteresttext
Google-ExtendedShare of desktop sites naming the bot3.4 %SEO | 2025 | The Web Almanac by HTTP Archivegoogle-extended appears in 3.4% of desktop sites and 3.0% of mobile sites.html
Google-ExtendedNamed in Shopify's default filenoallbirds.com/robots.txt (live Shopify-generated default robots.txt, example store), retrieved 2026-10-18User-agent: Pinteresttext
PerplexityBotShare of desktop sites naming the bot2.8 %SEO | 2025 | The Web Almanac by HTTP Archiveperplexitybot appears in 2.8% of desktop sites and 2.7% of mobile sites.html
PerplexityBotNamed in Shopify's default filenoallbirds.com/robots.txt (live Shopify-generated default robots.txt, example store), retrieved 2026-10-18User-agent: Pinteresttext
Methodology

Caveats: AI crawlers named in robots.txt files, and Shopify's default robots.txt

  • Shares are web-wide desktop figures from the Web Almanac 2025; no Shopify-only share is claimed.
  • The Web Almanac figure names ten AI-related user agents and the dataset holds five. The other five are PetalBot (4.0% of desktop sites), Amazonbot (3.3%), FacebookBot (2.9%), ChatGPT-User (2.8%) and Meta-ExternalAgent (2.8%). On mobile, PetalBot (4.4%) is named more often than GPTBot (4.2%).
  • Naming a crawler is not the same as blocking it: a file can name a bot to allow it as well as to disallow it. The dataset holds no allow or disallow split.
  • Shopify's documentation does not list its default robots.txt groups. The Shopify column rests on the default file served by Shopify stores, including allbirds.com and gymshark.com, on October 18, 2026.
  • A store's file can differ from the default if the merchant has changed it.

Deploi's charts and compilation are licensed CC BY 4.0. The underlying facts belong to their sources, which are named on every page. How Deploi Shopify Data works.

Sources

Tier A sources are primary documents: filings, official releases and official price pages. How sources are ranked.

Cite and reuse

Deploi's charts and compilation are licensed CC BY 4.0. The underlying facts belong to their sources, which are named on every page.

APA

Deploi. (2026). AI crawlers named in robots.txt files, and Shopify's default robots.txt (Version 1) [Data set]. https://deploi.ca/shopify/data/datasets/ai-crawler-robots Retrieved October 18, 2026.

BibTeX

@misc{deploi_ai_crawler_robots_2026,
  author = {{Deploi}},
  title = {AI crawlers named in robots.txt files, and Shopify's default robots.txt},
  year = {2026},
  version = {1},
  url = {https://deploi.ca/shopify/data/datasets/ai-crawler-robots},
  note = {Accessed 2026-10-18}
}

Downloads

  • Data: AI crawlers named in robots.txt files, and Shopify's default robots.txt CSV

Changes

  1. · AI crawlers named in robots.txt files, and Shopify's default robots.txt
    First release. Shares read from the Web Almanac 2025 SEO chapter and the default robots.txt of Shopify stores checked on October 18, 2026.
Deploi is not affiliated with Shopify. Questions about this data: contact@deploi.ca.