Contact us
Product Huntlaunching December 5 Follow the launch

Free tool

AI robots.txt generator

robots.txt tells each crawler what it may visit. AI companies run separate crawlers for search answers, for visits a user asks for and for model training. Pick a preset or set each crawler, then copy the rules into your robots.txt.

Start from a preset

Rules to add to robots.txt

 

Add these rules to your existing robots.txt. On Shopify, edit robots.txt.liquid in your theme. Cloudflare's bot settings work separately: check them too.

How it works

  1. 01

    Pick a preset

    Most stores want to appear in AI answers. Whether to allow model training is a business choice.

  2. 02

    Adjust each crawler

    Every crawler is listed with its owner and what it does.

  3. 03

    Copy the rules

    Then check the result with the AI bot access checker.

What you get

  • Search crawlers

    OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot, Applebot.

  • User-requested visits

    ChatGPT-User, Perplexity-User, Claude-User.

  • Training crawlers

    GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent.

Quelta product videos

Turn the open questions into product videos

Quelta makes product-accurate videos from your product photos, for a whole catalog: the product looks the same in every shot. Each video answers one buyer doubt.

See product videos

Start with a pilot

Tell us about your catalog

Send a link to your catalog or website and tell us roughly how many videos you need a month. We start with a pilot on 3⁠–⁠5 of your products, so you can judge the videos on your own catalog.

  1. Send a link to your catalog

    Your website, a product feed or your store on a marketplace.

  2. We make a pilot on 3⁠–⁠5 of your products

    Product-accurate videos of your real products, for the markets and platforms you need.

  3. A monthly contract for the volume you need

    New videos for the catalog and new arrivals every month, in one style.

We don’t copy other people’s videos. Every video is new and about your product. We keep the mechanics that sell (script, pacing, delivery), not anyone’s footage, people or music.

Contact us

Prefer to talk first? Yegor, Co-founder. A 30-minute video call about your catalog.

Book a call

Made with our engineTests, not client projects

Questions and answers

Will blocking training crawlers hurt my visibility in AI search?

It doesn't remove you from AI search results, because those use separate crawlers. Whether training data influences future model answers is not something any company publishes.

Do all crawlers obey robots.txt?

The major AI companies say their crawlers follow it. User-requested agents like ChatGPT-User may not apply robots.txt, because a person asked for the page. For hard blocks, use your firewall or CDN.

Where is robots.txt on Shopify?

Shopify generates it; you can customize it with a robots.txt.liquid template in your theme.