{
  "title": "Robots txt tester",
  "human_url": "https://happytails.ai/en/web-tools/robots-txt-tester/",
  "agent_url": "https://happytails.ai/agents/en/web-tools/robots-txt-tester/",
  "language": "en",
  "markdown_url": "https://happytails.ai/en/web-tools/robots-txt-tester.md",
  "content": "Robots txt tester\nCheck whether robots.txt rules allow a crawler to visit a URL.\nHow to use Robots txt tester\nEnter your text in the input field.\nReview the settings, then choose Run Robots txt tester.\nReview the result, then use Copy result or a download link when available.\nWhat to expect\nCheck whether robots.txt rules allow a crawler to visit a URL. Paste the robots.txt content, enter the URL to test, and choose the crawler’s user-agent. The tool evaluates the supplied rules; it does not fetch the site’s current robots.txt.",
  "content_format": "text/plain",
  "visibility": "public",
  "links": [],
  "tool": {
    "id": "robots-txt-tester",
    "category": "web-tools",
    "built": true,
    "requirements": "Paste-first rules evaluation by user-agent and URL; show winning rule.",
    "implementation": {
      "fields": [
        {
          "key": "url",
          "label": "URL to test",
          "value": "https://happytails.ai/path",
          "type": "text"
        },
        {
          "key": "agent",
          "label": "Crawler user agent",
          "value": "Googlebot",
          "type": "text"
        }
      ],
      "note": "Paste robots.txt content; this tool does not fetch or change the remote file.",
      "experience": {
        "automatic": false,
        "single": false,
        "label": "Your text",
        "placeholder": "Type or paste your input…",
        "help": "Up to 1,000,000 characters. Review the settings before running.",
        "action": null
      },
      "mode": "explicit",
      "interactive": false
    },
    "execution": "POST /api/v1/tools/robots-txt-tester/run",
    "api": {
      "id": "robots-txt-tester",
      "name": "Robots txt tester",
      "category": "web-tools",
      "description": "Check whether robots.txt rules allow a crawler to visit a URL.",
      "notes": "Paste robots.txt content; this tool does not fetch or change the remote file.",
      "human_path": "/en/web-tools/robots-txt-tester/",
      "method": "POST",
      "endpoint": "/api/v1/tools/robots-txt-tester/run",
      "schema_url": "/api/v1/tools/robots-txt-tester",
      "input_schema": {
        "type": "object",
        "additionalProperties": false,
        "properties": {
          "input": {
            "type": "string",
            "maxLength": 1000000,
            "default": "",
            "description": "Plain text input. File tools use files instead unless otherwise documented."
          },
          "options": {
            "type": "object",
            "additionalProperties": false,
            "properties": {
              "url": {
                "type": "string",
                "description": "URL to test",
                "default": "https://happytails.ai/path"
              },
              "agent": {
                "type": "string",
                "description": "Crawler user agent",
                "default": "Googlebot"
              }
            }
          }
        }
      },
      "output_schema": {
        "type": "object",
        "required": [
          "tool",
          "result"
        ],
        "properties": {
          "tool": {
            "type": "string"
          },
          "result": {
            "type": "object",
            "required": [
              "text",
              "files"
            ],
            "properties": {
              "text": {
                "type": [
                  "string",
                  "null"
                ]
              },
              "data": {
                "description": "Parsed JSON when the textual result is JSON."
              },
              "name": {
                "type": "string"
              },
              "mime": {
                "type": "string"
              },
              "files": {
                "type": "array",
                "items": {
                  "type": "object",
                  "required": [
                    "name",
                    "mime",
                    "bytes",
                    "base64"
                  ],
                  "properties": {
                    "name": {
                      "type": "string"
                    },
                    "mime": {
                      "type": "string"
                    },
                    "bytes": {
                      "type": "integer"
                    },
                    "base64": {
                      "type": "string",
                      "contentEncoding": "base64"
                    }
                  }
                }
              }
            }
          }
        }
      },
      "processing": "Server; same transformation code as browser tools",
      "limits": {
        "request_bytes": 8388608,
        "input_characters": 1000000,
        "files_bytes": 5242880,
        "max_files": 0,
        "timeout_seconds": 12
      }
    },
    "agent_processing": "Server; same transformation code as browser tools",
    "browser_agent": {
      "supported": true,
      "name": "run_current_tool",
      "discovery": "WebMCP on the human page in a supporting browser",
      "verification": "See audit/API-AUDIT.md; availability is not verification"
    }
  },
  "markdown": "# Robots txt tester\n\n- Human page: https://happytails.ai/en/web-tools/robots-txt-tester/\n- Markdown: https://happytails.ai/en/web-tools/robots-txt-tester.md\n- Structured JSON: https://happytails.ai/agents/en/web-tools/robots-txt-tester/index.json\n- Visibility: public\n\n## Tool capabilities\n\n| Property | Value |\n| --- | --- |\n| Tool ID | `robots-txt-tester` |\n| Category | web-tools |\n| Implementation | Registered implementation. Registration alone is not a production-quality guarantee. |\n| Browser execution | Use the visible action or interactive controls. |\n| Browser processing | Browser; see tool notes for network behavior. |\n| HTTP API | /api/v1/tools/robots-txt-tester/run |\n| Browser-agent interface | run_current_tool |\n\n### Implementation notes\n\nPaste robots.txt content; this tool does not fetch or change the remote file.\n\n### Inputs and settings\n\nText input: Your text.\n\nExample or placeholder shown in the interface (not a validated fixture):\n\n```text\nType or paste your input…\n```\n\nUp to 1,000,000 characters. Review the settings before running.\n\n| Setting | Meaning | Control type | Default | Constraints |\n| --- | --- | --- | --- | --- |\n| `url` | URL to test | text | \"https://happytails.ai/path\" |  |\n| `agent` | Crawler user agent | text | \"Googlebot\" |  |\n\n### HTTP contract\n\nAPI calls process supplied data on the server. The website origin is `https://happytails.ai`. Server API hosting is not connected on the public website yet. Use your local server origin to test these requests.\n\n```http\nPOST https://happytails.ai/api/v1/tools/robots-txt-tester/run\nContent-Type: application/json\n```\n\n[Detailed operation reference](https://happytails.ai/en/api/tools/robots-txt-tester.md) · [Shared API guide](https://happytails.ai/en/api.md)\n\n#### Limits\n\n| Limit | Value |\n| --- | --- |\n| request_bytes | 8388608 |\n| input_characters | 1000000 |\n| files_bytes | 5242880 |\n| max_files | 0 |\n| timeout_seconds | 12 |\n\n#### Complete input schema\n\n```json\n{\n  \"type\": \"object\",\n  \"additionalProperties\": false,\n  \"properties\": {\n    \"input\": {\n      \"type\": \"string\",\n      \"maxLength\": 1000000,\n      \"default\": \"\",\n      \"description\": \"Plain text input. File tools use files instead unless otherwise documented.\"\n    },\n    \"options\": {\n      \"type\": \"object\",\n      \"additionalProperties\": false,\n      \"properties\": {\n        \"url\": {\n          \"type\": \"string\",\n          \"description\": \"URL to test\",\n          \"default\": \"https://happytails.ai/path\"\n        },\n        \"agent\": {\n          \"type\": \"string\",\n          \"description\": \"Crawler user agent\",\n          \"default\": \"Googlebot\"\n        }\n      }\n    }\n  }\n}\n```\n\n#### Complete output schema\n\n```json\n{\n  \"type\": \"object\",\n  \"required\": [\n    \"tool\",\n    \"result\"\n  ],\n  \"properties\": {\n    \"tool\": {\n      \"type\": \"string\"\n    },\n    \"result\": {\n      \"type\": \"object\",\n      \"required\": [\n        \"text\",\n        \"files\"\n      ],\n      \"properties\": {\n        \"text\": {\n          \"type\": [\n            \"string\",\n            \"null\"\n          ]\n        },\n        \"data\": {\n          \"description\": \"Parsed JSON when the textual result is JSON.\"\n        },\n        \"name\": {\n          \"type\": \"string\"\n        },\n        \"mime\": {\n          \"type\": \"string\"\n        },\n        \"files\": {\n          \"type\": \"array\",\n          \"items\": {\n            \"type\": \"object\",\n            \"required\": [\n              \"name\",\n              \"mime\",\n              \"bytes\",\n              \"base64\"\n            ],\n            \"properties\": {\n              \"name\": {\n                \"type\": \"string\"\n              },\n              \"mime\": {\n                \"type\": \"string\"\n              },\n              \"bytes\": {\n                \"type\": \"integer\"\n              },\n              \"base64\": {\n                \"type\": \"string\",\n                \"contentEncoding\": \"base64\"\n              }\n            }\n          }\n        }\n      }\n    }\n  }\n}\n```\n\n#### Error and retry handling\n\nFailures return a non-200 status and `error.code` plus `error.message`. Correct invalid input before retrying; retry server-busy responses with bounded backoff. Returned files contain Base64 data, not persistent download URLs. Decode and inspect the output before treating conversion as successful. See the shared API guide for the complete status and timeout rules.\n\n### Browser-agent workflow\n\n1. Open the human page in a browser that supports WebMCP.\n2. Discover the interface exposed by that page; use its actual schema.\n3. Select files on the page first when the tool requires files.\n4. Call `run_current_tool` with valid input and settings.\n5. Inspect the returned result or error and the displayed output. Retrieve files from the displayed download links.\n\nCalls do not automatically copy or download results. Browser permissions still apply. Interface availability alone is not proof of successful execution.\n\n### Planning specification\n\nPaste-first rules evaluation by user-agent and URL; show winning rule.\n\nThis describes intended requirements. Use the implementation notes, interface schema and observed output to determine current support.\n\n### Complete machine-readable capability record\n\n```json\n{\n  \"id\": \"robots-txt-tester\",\n  \"category\": \"web-tools\",\n  \"built\": true,\n  \"requirements\": \"Paste-first rules evaluation by user-agent and URL; show winning rule.\",\n  \"implementation\": {\n    \"fields\": [\n      {\n        \"key\": \"url\",\n        \"label\": \"URL to test\",\n        \"value\": \"https://happytails.ai/path\",\n        \"type\": \"text\"\n      },\n      {\n        \"key\": \"agent\",\n        \"label\": \"Crawler user agent\",\n        \"value\": \"Googlebot\",\n        \"type\": \"text\"\n      }\n    ],\n    \"note\": \"Paste robots.txt content; this tool does not fetch or change the remote file.\",\n    \"experience\": {\n      \"automatic\": false,\n      \"single\": false,\n      \"label\": \"Your text\",\n      \"placeholder\": \"Type or paste your input…\",\n      \"help\": \"Up to 1,000,000 characters. Review the settings before running.\",\n      \"action\": null\n    },\n    \"mode\": \"explicit\",\n    \"interactive\": false\n  },\n  \"execution\": \"POST /api/v1/tools/robots-txt-tester/run\",\n  \"api\": {\n    \"id\": \"robots-txt-tester\",\n    \"name\": \"Robots txt tester\",\n    \"category\": \"web-tools\",\n    \"description\": \"Check whether robots.txt rules allow a crawler to visit a URL.\",\n    \"notes\": \"Paste robots.txt content; this tool does not fetch or change the remote file.\",\n    \"human_path\": \"/en/web-tools/robots-txt-tester/\",\n    \"method\": \"POST\",\n    \"endpoint\": \"/api/v1/tools/robots-txt-tester/run\",\n    \"schema_url\": \"/api/v1/tools/robots-txt-tester\",\n    \"input_schema\": {\n      \"type\": \"object\",\n      \"additionalProperties\": false,\n      \"properties\": {\n        \"input\": {\n          \"type\": \"string\",\n          \"maxLength\": 1000000,\n          \"default\": \"\",\n          \"description\": \"Plain text input. File tools use files instead unless otherwise documented.\"\n        },\n        \"options\": {\n          \"type\": \"object\",\n          \"additionalProperties\": false,\n          \"properties\": {\n            \"url\": {\n              \"type\": \"string\",\n              \"description\": \"URL to test\",\n              \"default\": \"https://happytails.ai/path\"\n            },\n            \"agent\": {\n              \"type\": \"string\",\n              \"description\": \"Crawler user agent\",\n              \"default\": \"Googlebot\"\n            }\n          }\n        }\n      }\n    },\n    \"output_schema\": {\n      \"type\": \"object\",\n      \"required\": [\n        \"tool\",\n        \"result\"\n      ],\n      \"properties\": {\n        \"tool\": {\n          \"type\": \"string\"\n        },\n        \"result\": {\n          \"type\": \"object\",\n          \"required\": [\n            \"text\",\n            \"files\"\n          ],\n          \"properties\": {\n            \"text\": {\n              \"type\": [\n                \"string\",\n                \"null\"\n              ]\n            },\n            \"data\": {\n              \"description\": \"Parsed JSON when the textual result is JSON.\"\n            },\n            \"name\": {\n              \"type\": \"string\"\n            },\n            \"mime\": {\n              \"type\": \"string\"\n            },\n            \"files\": {\n              \"type\": \"array\",\n              \"items\": {\n                \"type\": \"object\",\n                \"required\": [\n                  \"name\",\n                  \"mime\",\n                  \"bytes\",\n                  \"base64\"\n                ],\n                \"properties\": {\n                  \"name\": {\n                    \"type\": \"string\"\n                  },\n                  \"mime\": {\n                    \"type\": \"string\"\n                  },\n                  \"bytes\": {\n                    \"type\": \"integer\"\n                  },\n                  \"base64\": {\n                    \"type\": \"string\",\n                    \"contentEncoding\": \"base64\"\n                  }\n                }\n              }\n            }\n          }\n        }\n      }\n    },\n    \"processing\": \"Server; same transformation code as browser tools\",\n    \"limits\": {\n      \"request_bytes\": 8388608,\n      \"input_characters\": 1000000,\n      \"files_bytes\": 5242880,\n      \"max_files\": 0,\n      \"timeout_seconds\": 12\n    }\n  },\n  \"agent_processing\": \"Server; same transformation code as browser tools\",\n  \"browser_agent\": {\n    \"supported\": true,\n    \"name\": \"run_current_tool\",\n    \"discovery\": \"WebMCP on the human page in a supporting browser\",\n    \"verification\": \"See audit/API-AUDIT.md; availability is not verification\"\n  }\n}\n```\n\n## Page guide\n\nCheck whether robots.txt rules allow a crawler to visit a URL.\n\n## How to use Robots txt tester\n\n1. Enter your text in the input field.\n2. Review the settings, then choose Run Robots txt tester.\n3. Review the result, then use Copy result or a download link when available.\n\n## What to expect\n\nCheck whether robots.txt rules allow a crawler to visit a URL. Paste the robots.txt content, enter the URL to test, and choose the crawler’s user-agent. The tool evaluates the supplied rules; it does not fetch the site’s current robots.txt.\n"
}