Robots.txt Tester
Check whether robots.txt rules allow a crawler to visit your URL.
No files selected
Paste the robots.txt content, enter the URL to test, and choose the crawler’s user-agent. The tool evaluates the supplied rules; it does not fetch the site’s current robots.txt. Up to 1,000,000 characters. Review the settings before running.
Your result
How to use Robots txt tester
- Enter your text in the input field.
- Review the settings, then choose Run Robots txt tester.
- Review the result, then use Copy result or a download link when available.
What to expect
Check whether robots.txt rules allow a crawler to visit a URL. Paste the robots.txt content, enter the URL to test, and choose the crawler’s user-agent. The tool evaluates the supplied rules; it does not fetch the site’s current robots.txt.