Free Shopify Tool

Shopify Robots.txt + AI Crawler Checker

Paste your robots.txt contents and instantly see which crawlers are allowed or blocked: Googlebot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and 6 more. Checks Shopify-specific patterns including /policies/ disallow and missing Sitemap directives. 100% browser-side, no signup, no network calls.

Your robots.txt file lives at yourstore.com/robots.txt. Open that URL in any browser, select all the text, and paste it into the box below. The file is auto-generated by Shopify from a Liquid template called robots.txt.liquid. If your theme does not include that template, Shopify falls back to a built-in default that allows every crawler, including all AI bots.

This tool parses what you paste and reports the access status for 11 major crawlers: two search engines (Googlebot and Bingbot) and nine AI crawlers covering ChatGPT, Claude, Perplexity, Gemini, Common Crawl, TikTok, Apple Intelligence, and Amazon. For each blocked AI crawler it notes the real-world cost. The tool also runs three Shopify-specific checks: whether a Sitemap directive is present, whether /policies/ is disallowed, and whether the file is empty or rule-free.

Everything runs in your browser. Nothing is sent anywhere. No account required.

Want the full audit?

Get a personalised PDF report + 15-min strategy slot

This free tool shows the headline issue. I'll send you the complete 25-point audit for your store as a polished PDF: every fix ranked by revenue impact, plus a free 15-minute strategy call to walk through it. No pitch, zero commitment.

  • 25 points across PDP, collection, cart, checkout, mobile, CWV, app stack, SEO
  • Prioritised action list by projected dollar impact
  • 15-min Zoom to walk through the top 3 findings live

I reply within 24 hours. Prefer to talk? Book a call.

FAQ

Frequently Asked Questions

Where do I find my Shopify robots.txt?

Visit yourstore.com/robots.txt in any browser. You will see a plain-text file. Select all (Cmd+A on Mac, Ctrl+A on Windows), copy, and paste it into the textarea above. Shopify generates this file from a Liquid template called robots.txt.liquid. If your theme does not include that file, Shopify uses a built-in default that allows all crawlers.

Are AI crawlers blocked on Shopify by default?

No. Shopify's default robots.txt allows all crawlers, including GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. If you have not customised robots.txt.liquid in your theme, every AI crawler can access your store. Blocking them requires adding a specific User-agent block to robots.txt.liquid. This tool will tell you whether each AI crawler is currently blocked or allowed based on your actual file.

Should I block AI crawlers on my Shopify store?

It depends on your goals. Blocking training crawlers like GPTBot and CCBot prevents your content from being used in future model training, but it also prevents those models from citing your products in answers and recommendations. Blocking ChatGPT Search's crawler (OAI-SearchBot) or PerplexityBot prevents your store from appearing in AI search results. For most Shopify merchants, allowing AI crawlers is the better choice for brand visibility in AI-powered search. The decision should be deliberate, not an accident.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot is OpenAI's training crawler: it collects data to train future versions of ChatGPT. Blocking it prevents your content from appearing in model training data but does not directly affect ChatGPT search results. OAI-SearchBot is OpenAI's search crawler: it powers real-time retrieval inside ChatGPT Search (the search mode within the ChatGPT interface). Blocking OAI-SearchBot means your products will not appear when users search for them inside ChatGPT. They are two separate bots with two separate business impacts.

What is the /policies/ disallow and why does the tool flag it?

Some Shopify robots.txt configurations include a Disallow: /policies/ line. Shopify ships this by default in certain setups to discourage indexation of generic policy pages like /policies/refund-policy and /policies/privacy-policy, which add little SEO value. The tool flags it so you are aware it is there. It is usually intentional and harmless for SEO, but if your store has unique policy content you want indexed, you may want to remove it.

Need a developer to actually fix what this tool finds?

I rebuild Shopify Markets currency setups, eliminate hardcoded prices, and ship clean Liquid that converts across every region. Book a free 30-minute strategy call or read the Everly case study that inspired this tool.