Robots.txt Checker

Analyze robots.txt file, validate rules, and check sitemap directives. Free online tool.

Enter URL here or click to paste

About this tool

Robots.txt Checker is a free online tool for validating and analyzing your robots.txt file. The robots.txt file controls which pages search engine bots can crawl and index. This tool helps you understand your crawl rules, identify errors, and ensure Googlebot and other crawlers can access important pages. Perfect for SEO professionals, webmasters, and developers managing site crawlability.

How to use

1Enter your website URL or paste robots.txt content directly
2Click "Check Robots.txt" and wait for validation
3View all allow/disallow rules for different search engine bots
4Check for syntax errors or problematic rules
5Get suggestions for improving your robots.txt configuration

Specifications and limits

Validates robots.txt syntax according to REP (Robots Exclusion Protocol) standard. Checks: User-agent rules, Disallow/Allow paths, Crawl-delay, Request-rate, Sitemap directives. Tests specific URLs against rules. Shows which bots are affected by each rule. Processing speed: instant.