Write a robots.txt That Welcomes AI Crawlers
Generate a complete robots.txt with an explicit allow-list for the major AI answer-engine crawlers (GPTBot, PerplexityBot, ClaudeBot, Google-Extended, Applebot-Extended and more) plus a sitemap pointer. Paste-ready for any small business site.
Pro tip
Defaulting to "Allow: /" technically lets every bot in, but explicit allow-listing is a clearer signal to AI engines and protects you if a future default-deny rule ships. Re-run this prompt twice a year — new AI crawler bots appear quarterly, and missing one quietly costs you citations.
How to use this prompt
- Pick your AI model. Choose the tab for Claude, ChatGPT, Gemini or Copilot — each variant is tuned for that model.
- Copy the full prompt. Click Copy Full Prompt to copy the text to your clipboard.
- Paste into your AI tool. Open your chosen model and paste the prompt into a new chat.
- Replace the
[placeholders]. Swap any bracketed fields for your company name, audience, product or tone. - Run and refine. Review the output. If anything is off, ask the AI to tighten tone, length or format.
Prompt Variants by Model
Claude
Claude 4.x
FRESH APR 2026
You are a technical SEO consultant generating a production robots.txt for a small business website that wants to be cited by AI answer engines.
<site_context>
Domain: [YOUR-DOMAIN.COM]
Sitemap URL:...You are a technical SEO consultant generating a production robots.txt for a small business website that wants to be cited by AI answer engines.
<site_context>
Domain: [YOUR-DOMAIN.COM]
Sitemap URL: [https://YOUR-DOMAIN.COM/sitemap.xml] (leave blank if you don''t have one yet — I''ll generate a placeholder)
Paths to BLOCK from all crawlers: [e.g. /admin/, /checkout/, /api/, /*.json$ — leave blank if none]
Paths to BLOCK from AI training crawlers but ALLOW for search: [e.g. /pricing/, /case-studies/ — leave blank if none]
Crawl delay needed (seconds): [LEAVE BLANK FOR NONE — only set if your server is slow]
</site_context>
<requirements>
The robots.txt must:
1. Allow all well-behaved crawlers by default.
2. Explicitly allow-list these AI answer-engine + search bots, each with its own User-agent block:
- GPTBot, OAI-SearchBot, ChatGPT-User (OpenAI)
- PerplexityBot, Perplexity-User (Perplexity)
- ClaudeBot, Claude-Web, anthropic-ai (Anthropic)
- Google-Extended (Google AI training opt-in)
- Applebot, Applebot-Extended (Apple Intelligence)
- CCBot (Common Crawl — feeds many LLMs)
- Bingbot (Bing + Copilot)
3. Reference the sitemap URL.
4. Include a header comment with site name and last-updated date.
5. Be valid robots.txt syntax — no trailing whitespace, correct line endings, one directive per line.
</requirements>
Output exactly two things:
**PART 1 — robots.txt (paste-ready)**
A code block containing the complete robots.txt content. Nothing else inside the code block.
**PART 2 — Deploy notes**
- Where to upload it (root of the domain, must be reachable at https://[domain]/robots.txt)
- How to verify it''s live (curl command + Google''s robots.txt Tester URL)
- One-sentence explanation of any path I told you to block
- A reminder of which AI crawlers were intentionally NOT included (and why I might want to add them later)
Notes: Claude follows the formatting rules strictly. The output is usually paste-ready with zero edits.
ChatGPT
GPT-5
FRESH APR 2026
Act as a technical SEO consultant. Generate a complete, paste-ready robots.txt for a small business site that wants AI answer engines to find and cite it.
Site details:
- Domain: [YOUR-DOMAIN.COM]
-...Act as a technical SEO consultant. Generate a complete, paste-ready robots.txt for a small business site that wants AI answer engines to find and cite it.
Site details:
- Domain: [YOUR-DOMAIN.COM]
- Sitemap URL: [https://YOUR-DOMAIN.COM/sitemap.xml] (leave blank for placeholder)
- Paths to block from ALL crawlers: [LIST or NONE]
- Paths to block from AI training but allow for search: [LIST or NONE]
- Crawl delay seconds: [BLANK or NUMBER]
Requirements:
1. Allow all well-behaved crawlers by default.
2. Explicit User-agent allow blocks for: GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, ClaudeBot, Claude-Web, anthropic-ai, Google-Extended, Applebot, Applebot-Extended, CCBot, Bingbot.
3. Reference the sitemap.
4. Header comment with site name + last-updated date.
5. Valid syntax, one directive per line, no trailing whitespace.
Output:
1. The robots.txt content in a single code block — nothing else inside.
2. Deploy notes: where to upload, curl + Google tester command to verify, one-line explanation of any blocks, and a reminder of which AI crawlers were NOT included and why I might add them later.
Notes: GPT-5 sometimes wraps the robots.txt in extra prose — instruct it to put ONLY the file contents in the code block.
Gemini
Gemini 2.5 Pro
FRESH APR 2026
You are a technical SEO consultant. Produce a complete, paste-ready robots.txt for a small business website that wants to be cited by AI answer engines.
SITE DETAILS:
- Domain: [YOUR-DOMAIN.COM]
-...You are a technical SEO consultant. Produce a complete, paste-ready robots.txt for a small business website that wants to be cited by AI answer engines.
SITE DETAILS:
- Domain: [YOUR-DOMAIN.COM]
- Sitemap URL: [https://YOUR-DOMAIN.COM/sitemap.xml] (blank = placeholder)
- Paths to block from all crawlers: [LIST or NONE]
- Paths to block from AI training but allow for search: [LIST or NONE]
- Crawl delay: [BLANK or NUMBER seconds]
REQUIREMENTS:
1. Default-allow all well-behaved crawlers.
2. Explicit User-agent allow blocks for each of: GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, ClaudeBot, Claude-Web, anthropic-ai, Google-Extended, Applebot, Applebot-Extended, CCBot, Bingbot.
3. Sitemap directive.
4. Header comment: site name + last-updated date.
5. Valid syntax, one directive per line, no trailing whitespace, Unix line endings.
OUTPUT:
1. robots.txt content in a single code block — nothing else inside.
2. Deploy notes: upload location, curl + Google robots.txt Tester verification, one-line rationale for any blocks, list of AI crawlers intentionally omitted and when to add them.
Notes: Gemini occasionally adds Markdown bold inside the file — strip any ** markers before pasting.
Copilot
Copilot (GPT-5)
FRESH APR 2026
Generate a complete, paste-ready robots.txt for my small business site so AI answer engines (ChatGPT Search, Perplexity, Google AI Overviews, Bing Copilot) can find and cite it.
Site:
• Domain:...Generate a complete, paste-ready robots.txt for my small business site so AI answer engines (ChatGPT Search, Perplexity, Google AI Overviews, Bing Copilot) can find and cite it.
Site:
• Domain: [YOUR-DOMAIN.COM]
• Sitemap URL: [https://YOUR-DOMAIN.COM/sitemap.xml] (blank = placeholder)
• Paths to block from all crawlers: [LIST or NONE]
• Paths to block from AI training only: [LIST or NONE]
• Crawl delay: [BLANK or NUMBER]
Requirements:
1. Default-allow all well-behaved crawlers.
2. Explicit User-agent allow blocks for: GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, ClaudeBot, Claude-Web, anthropic-ai, Google-Extended, Applebot, Applebot-Extended, CCBot, Bingbot.
3. Sitemap reference.
4. Header comment with site name + last-updated date.
5. Valid syntax — one directive per line, no trailing whitespace.
Output:
1. robots.txt in a single code block (nothing else inside).
2. Deploy notes: upload location, verification commands (curl + Google tester), one-line block rationale, list of AI crawlers intentionally omitted with notes on when to add.
Notes: Copilot output is reliably clean — usually zero post-edit needed.
Frequently Asked Questions
What does the Write a robots.txt That Welcomes AI Crawlers prompt do?
Generate a complete robots.txt with an explicit allow-list for the major AI answer-engine crawlers (GPTBot, PerplexityBot, ClaudeBot, Google-Extended, Applebot-Extended and more) plus a sitemap pointer. Paste-ready for any small business site.
Which AI models is this prompt tested on?
This prompt is field-tested on Claude, ChatGPT, Gemini and Copilot. Each model has its own optimized variant above.
Do I need a paid AI account to use this prompt?
No. This prompt is written to run on the free tier of Claude, ChatGPT, Gemini and Copilot. Paid tiers simply give you longer context windows and faster responses.
Can I customize this prompt for my business?
Yes. Any text inside square brackets is a placeholder you replace with your own business details, such as company name, audience, product or tone. You can also ask the AI to adjust format, length or style after the first output.
When was this prompt last verified?
Each model variant above shows its own freshness stamp. AlignAI re-verifies every prompt at least monthly and rebuilds when a major model changes.
Don’t see what you need? tailored to your use case.