🤖

Robots.txt Generator

Generate robots.txt files with presets for blocking AI bots, search engines, and custom rules.

Quick Presets

User-Agent Rules

User-Agent

Paths

Sitemap URL (optional)

Crawl-delay (optional, in seconds)

Generated robots.txt

User-agent: *
Allow: /

About Robots.txt Generator

Free robots.txt generator with quick presets and full customization. Create a properly formatted robots.txt file for your website in seconds. Choose from presets like Allow All, Block All, Block AI Bots (GPTBot, ChatGPT-User, CCBot, Bytespider), or Standard recommended settings. Add custom rules for any user-agent with multiple Allow and Disallow paths. Include your sitemap URL and optional crawl-delay. See a live preview of your robots.txt file as you configure it, then copy to clipboard or download as a file. Essential for controlling how search engine crawlers and AI bots access your website content.

Frequently Asked Questions

What is a robots.txt file?

A robots.txt file is a text file placed at the root of your website (e.g., example.com/robots.txt) that tells search engine crawlers and bots which pages they can and cannot access. It follows the Robots Exclusion Protocol standard.

How do I block AI bots from crawling my site?

Add Disallow rules for AI bot user-agents: GPTBot (OpenAI), ChatGPT-User (ChatGPT browsing), CCBot (Common Crawl/Anthropic training data), and Bytespider (ByteDance). This tool has a one-click 'Block AI Bots' preset that sets this up for you.

Does robots.txt guarantee bots won't access my site?

No. robots.txt is a voluntary standard - well-behaved bots (Google, Bing, most AI crawlers) respect it, but malicious bots may ignore it. For sensitive content, use server-side authentication or access controls instead.

Where should I place my robots.txt file?

The robots.txt file must be placed at the root of your domain (e.g., example.com/robots.txt). It will not work in subdirectories. Search engines only look for it at the root URL. The file should be plain text with UTF-8 encoding.

Is this robots.txt generator free?

Yes, completely free with no signup. Choose from presets or create custom rules, add your sitemap URL, preview the output, and copy or download the file instantly. All processing happens in your browser.

THE RIGHT BOT

You just told the bad bots to leave. Hire the one that pays off.

Intellure is an AI employee that runs customer chat, sales, and follow-up across WhatsApp, Instagram, and your site. Fully managed, $299 a month.

Get Intellure

About Robots.txt Generator

Frequently Asked Questions

What is a robots.txt file?

How do I block AI bots from crawling my site?

Does robots.txt guarantee bots won't access my site?

Where should I place my robots.txt file?

Is this robots.txt generator free?

Generated robots.txt

About Robots.txt Generator

Frequently Asked Questions

Related Tools

Meta Description Generator

Meta Tag Generator

Keyword Density Checker

Open Graph Generator

Sitemap Generator

Schema Markup Generator (JSON-LD)

You just told the bad bots to leave. Hire the one that pays off.

Generated robots.txt

About Robots.txt Generator

Frequently Asked Questions

Related Tools

Meta Description Generator

Meta Tag Generator

Keyword Density Checker

Open Graph Generator

Sitemap Generator

Schema Markup Generator (JSON-LD)

You just told the bad bots to leave. Hire the one that pays off.