Free tool
AI robots.txt generator
robots.txt tells each crawler what it may visit. AI companies run separate crawlers for search answers, for visits a user asks for and for model training. Pick a preset or set each crawler, then copy the rules into your robots.txt.
Rules to add to robots.txt
Add these rules to your existing robots.txt. On Shopify, edit robots.txt.liquid in your theme. Cloudflare's bot settings work separately: check them too.
Checking…
How it works
- 01
Pick a preset
Most stores want to appear in AI answers. Whether to allow model training is a business choice.
- 02
Adjust each crawler
Every crawler is listed with its owner and what it does.
- 03
Copy the rules
Then check the result with the AI bot access checker.
What you get
-
Search crawlers
OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot, Applebot.
-
User-requested visits
ChatGPT-User, Perplexity-User, Claude-User.
-
Training crawlers
GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent.
Quelta product videos
Turn the open questions into product videos
Quelta makes product-accurate videos from your product photos, for a whole catalog: the product looks the same in every shot. Each video answers one buyer doubt.
See product videosStart with a pilot
Tell us about your catalog
Send a link to your catalog or website and tell us roughly how many videos you need a month. We start with a pilot on 3–5 of your products, so you can judge the videos on your own catalog.
Send a link to your catalog
Your website, a product feed or your store on a marketplace.
We make a pilot on 3–5 of your products
Product-accurate videos of your real products, for the markets and platforms you need.
A monthly contract for the volume you need
New videos for the catalog and new arrivals every month, in one style.
We don’t copy other people’s videos. Every video is new and about your product. We keep the mechanics that sell (script, pacing, delivery), not anyone’s footage, people or music.
Contact us
Made with our engineTests, not client projects
Questions and answers
Will blocking training crawlers hurt my visibility in AI search?
It doesn't remove you from AI search results, because those use separate crawlers. Whether training data influences future model answers is not something any company publishes.
Do all crawlers obey robots.txt?
The major AI companies say their crawlers follow it. User-requested agents like ChatGPT-User may not apply robots.txt, because a person asked for the page. For hard blocks, use your firewall or CDN.
Where is robots.txt on Shopify?
Shopify generates it; you can customize it with a robots.txt.liquid template in your theme.