Tool

Robots.txt Validator

Fetch, inspect, and test robots.txt rules for any domain. Validate whether specific URLs are allowed or blocked for any crawler, and check which page resources are affected.

Fetch Robots.txt

Enter any URL to fetch and parse its domain's robots.txt file.

URL