robots.txt & llms.txt Generator
This generator builds a robots.txt file that controls which search engines and AI crawlers can read your site, plus an llms.txt file that summarizes your site for language models. Choose to allow AI search bots like OAI-SearchBot and PerplexityBot while blocking training bots if you prefer, then copy or download both files.
Upload both files to your site's root folder (the same place as wp-config.php). robots.txt is a request, not a lock: well-behaved crawlers follow it, others may not.
How it works
- Enter your website and sitemap URLs.
- Choose whether to allow search engines, AI search assistants and AI training crawlers, and list any private paths.
- Add a short summary and your key pages for llms.txt, then copy or download both files and upload them to your site's root folder.
Why it matters for AI search
AI assistants can only cite pages they are allowed to fetch. Many sites block AI search bots by accident while trying to opt out of model training. Separating the two lets you stay visible in AI answers while keeping control of your content.
Frequently asked questions
What is the difference between AI search bots and AI training bots?
AI search bots such as OAI-SearchBot, PerplexityBot and Claude-SearchBot fetch pages to answer a user's question and can cite you. Training bots such as GPTBot, ClaudeBot and Google-Extended collect content to train models. Blocking training bots does not remove you from AI search.
What is llms.txt?
llms.txt is a proposed standard: a Markdown file at the root of your site that summarizes who you are and links to your most important pages. It is not an official ranking factor, and support varies by AI company, but it is quick to add.
Is robots.txt enough to stop a crawler?
No. robots.txt is a request that reputable crawlers follow. It does not block access technically. Use authentication or firewall rules for content that must stay private.
Where do I upload the files?
Upload robots.txt and llms.txt to the root folder of your domain so they load at yoursite.com/robots.txt and yoursite.com/llms.txt. On WordPress that is the folder containing wp-config.php.
Related tools
FAQPage, HowTo, Article, Organization and LocalBusiness JSON-LD with live preview.
See your Google result and pixel width before you publish.
Want us to fix this for you?
Book a free 15-minute call. We'll review your results and tell you what we'd fix first, with no obligation.
Book a free 15-minute call