robots.txt Generator

Tell crawlers where they can go, then test a URL against your rules.

Start

Load a preset

All crawlers may

Paths

Disallow one per line

Allow (exceptions)

Crawl delay in secondsBing and others respect it. Google ignores it.

Sitemaps

Block specific crawlers

robots.txt

Test a URL

robots.txt is public and only asks well-behaved crawlers to stay out. It is not security: never list private paths here, and protect them properly instead. Save the file as robots.txt in your site's root folder.