{
  "title": "Semantic deduplication",
  "human_url": "https://happytails.ai/en/data-tools/semantic-deduplication/",
  "agent_url": "https://happytails.ai/agents/en/data-tools/semantic-deduplication/",
  "language": "en",
  "markdown_url": "https://happytails.ai/en/data-tools/semantic-deduplication.md",
  "content": "Semantic deduplication\nLocal MiniLM embeddings propose similarity groups.\nHow to use Semantic deduplication\nChoose a file in CSV.\nReview the settings, then choose Run Semantic deduplication.\nReview the result, then use Copy result or a download link when available.\nWhat to expect\nLocal MiniLM embeddings propose similarity groups. Local MiniLM embeddings propose similarity groups. At most 500 rows with 1,000 characters per text value. All rows are retained for review; no automatic deletion.",
  "content_format": "text/plain",
  "visibility": "public",
  "links": [],
  "tool": {
    "id": "semantic-deduplication",
    "category": "data-tools",
    "built": true,
    "requirements": "Group near-duplicate text rows in uploaded CSV; set similarity threshold; review proposed groups before export; never automatically discard records based only on embeddings.",
    "implementation": {
      "fields": [
        {
          "key": "column",
          "label": "Text column name",
          "type": "text",
          "value": "text"
        },
        {
          "key": "threshold",
          "label": "Similarity threshold",
          "type": "number",
          "value": 0.85,
          "min": 0.5,
          "max": 1
        }
      ],
      "files": true,
      "multiple": false,
      "input": false,
      "processing": "Server required",
      "native": true,
      "extra": true,
      "maxFileBytes": 20971520,
      "accept": ".csv",
      "note": "Local MiniLM embeddings propose similarity groups. At most 500 rows with 1,000 characters per text value. All rows are retained for review; no automatic deletion.",
      "experience": {
        "automatic": false,
        "single": false,
        "label": "Your text",
        "placeholder": "Type or paste your input…",
        "help": "Up to 1,000,000 characters. Review the settings before running.",
        "action": null
      },
      "mode": "explicit",
      "interactive": false
    },
    "execution": "POST /api/v1/tools/semantic-deduplication/run",
    "api": {
      "id": "semantic-deduplication",
      "name": "Semantic deduplication",
      "category": "data-tools",
      "description": "Local MiniLM embeddings propose similarity groups.",
      "notes": "Local MiniLM embeddings propose similarity groups. At most 500 rows with 1,000 characters per text value. All rows are retained for review; no automatic deletion.",
      "human_path": "/en/data-tools/semantic-deduplication/",
      "method": "POST",
      "endpoint": "/api/v1/tools/semantic-deduplication/run",
      "schema_url": "/api/v1/tools/semantic-deduplication",
      "input_schema": {
        "type": "object",
        "additionalProperties": false,
        "properties": {
          "input": {
            "type": "string",
            "maxLength": 1000000,
            "default": "",
            "description": "Plain text input. File tools use files instead unless otherwise documented."
          },
          "options": {
            "type": "object",
            "additionalProperties": false,
            "properties": {
              "column": {
                "type": "string",
                "description": "Text column name",
                "default": "text"
              },
              "threshold": {
                "type": "number",
                "description": "Similarity threshold",
                "default": 0.85,
                "minimum": 0.5,
                "maximum": 1
              }
            }
          },
          "files": {
            "type": "array",
            "maxItems": 1,
            "items": {
              "type": "object",
              "required": [
                "name",
                "base64"
              ],
              "additionalProperties": false,
              "properties": {
                "name": {
                  "type": "string",
                  "maxLength": 200,
                  "description": "Filename only, no path."
                },
                "mime": {
                  "type": "string",
                  "maxLength": 150
                },
                "bytes": {
                  "type": "integer",
                  "minimum": 0,
                  "description": "Optional decoded byte count; must match content if supplied."
                },
                "base64": {
                  "type": "string",
                  "contentEncoding": "base64",
                  "description": "File bytes as padded base64. Use the tool-specific limits.files_bytes value for the total decoded input size."
                }
              }
            }
          }
        }
      },
      "output_schema": {
        "type": "object",
        "required": [
          "tool",
          "result"
        ],
        "properties": {
          "tool": {
            "type": "string"
          },
          "result": {
            "type": "object",
            "required": [
              "text",
              "files"
            ],
            "properties": {
              "text": {
                "type": [
                  "string",
                  "null"
                ]
              },
              "data": {
                "description": "Parsed JSON when the textual result is JSON."
              },
              "name": {
                "type": "string"
              },
              "mime": {
                "type": "string"
              },
              "files": {
                "type": "array",
                "items": {
                  "type": "object",
                  "required": [
                    "name",
                    "mime",
                    "bytes",
                    "base64"
                  ],
                  "properties": {
                    "name": {
                      "type": "string"
                    },
                    "mime": {
                      "type": "string"
                    },
                    "bytes": {
                      "type": "integer"
                    },
                    "base64": {
                      "type": "string",
                      "contentEncoding": "base64"
                    }
                  }
                }
              }
            }
          }
        }
      },
      "processing": "Server; self-hosted native engine, no external conversion service",
      "limits": {
        "request_bytes": 31457280,
        "input_characters": 1000000,
        "files_bytes": 20971520,
        "max_files": 1,
        "timeout_seconds": 90
      }
    },
    "agent_processing": "Server; self-hosted native engine, no external conversion service",
    "browser_agent": {
      "supported": true,
      "name": "run_current_tool",
      "discovery": "WebMCP on the human page in a supporting browser",
      "verification": "See audit/API-AUDIT.md; availability is not verification"
    }
  },
  "markdown": "# Semantic deduplication\n\n- Human page: https://happytails.ai/en/data-tools/semantic-deduplication/\n- Markdown: https://happytails.ai/en/data-tools/semantic-deduplication.md\n- Structured JSON: https://happytails.ai/agents/en/data-tools/semantic-deduplication/index.json\n- Visibility: public\n\n## Tool capabilities\n\n| Property | Value |\n| --- | --- |\n| Tool ID | `semantic-deduplication` |\n| Category | data-tools |\n| Implementation | Registered implementation. Registration alone is not a production-quality guarantee. |\n| Browser execution | Use the visible action or interactive controls. |\n| Browser processing | Server required |\n| HTTP API | /api/v1/tools/semantic-deduplication/run |\n| Browser-agent interface | run_current_tool |\n\n### Implementation notes\n\nLocal MiniLM embeddings propose similarity groups. At most 500 rows with 1,000 characters per text value. All rows are retained for review; no automatic deletion.\n\n### Inputs and settings\n\nNo free-text input is required. Use the file inputs and/or settings below.\n\nAccepted browser files: `.csv`. Choose one file.\n\nBrowser file-input limit: 20971520 bytes in total. HTTP limits can differ.\n\n| Setting | Meaning | Control type | Default | Constraints |\n| --- | --- | --- | --- | --- |\n| `column` | Text column name | text | \"text\" |  |\n| `threshold` | Similarity threshold | number | 0.85 | min=0.5 max=1 |\n\n### HTTP contract\n\nAPI calls process supplied data on the server. The website origin is `https://happytails.ai`. Server API hosting is not connected on the public website yet. Use your local server origin to test these requests.\n\n```http\nPOST https://happytails.ai/api/v1/tools/semantic-deduplication/run\nContent-Type: application/json\n```\n\n[Detailed operation reference](https://happytails.ai/en/api/tools/semantic-deduplication.md) · [Shared API guide](https://happytails.ai/en/api.md)\n\n#### Limits\n\n| Limit | Value |\n| --- | --- |\n| request_bytes | 31457280 |\n| input_characters | 1000000 |\n| files_bytes | 20971520 |\n| max_files | 1 |\n| timeout_seconds | 90 |\n\n#### Complete input schema\n\n```json\n{\n  \"type\": \"object\",\n  \"additionalProperties\": false,\n  \"properties\": {\n    \"input\": {\n      \"type\": \"string\",\n      \"maxLength\": 1000000,\n      \"default\": \"\",\n      \"description\": \"Plain text input. File tools use files instead unless otherwise documented.\"\n    },\n    \"options\": {\n      \"type\": \"object\",\n      \"additionalProperties\": false,\n      \"properties\": {\n        \"column\": {\n          \"type\": \"string\",\n          \"description\": \"Text column name\",\n          \"default\": \"text\"\n        },\n        \"threshold\": {\n          \"type\": \"number\",\n          \"description\": \"Similarity threshold\",\n          \"default\": 0.85,\n          \"minimum\": 0.5,\n          \"maximum\": 1\n        }\n      }\n    },\n    \"files\": {\n      \"type\": \"array\",\n      \"maxItems\": 1,\n      \"items\": {\n        \"type\": \"object\",\n        \"required\": [\n          \"name\",\n          \"base64\"\n        ],\n        \"additionalProperties\": false,\n        \"properties\": {\n          \"name\": {\n            \"type\": \"string\",\n            \"maxLength\": 200,\n            \"description\": \"Filename only, no path.\"\n          },\n          \"mime\": {\n            \"type\": \"string\",\n            \"maxLength\": 150\n          },\n          \"bytes\": {\n            \"type\": \"integer\",\n            \"minimum\": 0,\n            \"description\": \"Optional decoded byte count; must match content if supplied.\"\n          },\n          \"base64\": {\n            \"type\": \"string\",\n            \"contentEncoding\": \"base64\",\n            \"description\": \"File bytes as padded base64. Use the tool-specific limits.files_bytes value for the total decoded input size.\"\n          }\n        }\n      }\n    }\n  }\n}\n```\n\n#### Complete output schema\n\n```json\n{\n  \"type\": \"object\",\n  \"required\": [\n    \"tool\",\n    \"result\"\n  ],\n  \"properties\": {\n    \"tool\": {\n      \"type\": \"string\"\n    },\n    \"result\": {\n      \"type\": \"object\",\n      \"required\": [\n        \"text\",\n        \"files\"\n      ],\n      \"properties\": {\n        \"text\": {\n          \"type\": [\n            \"string\",\n            \"null\"\n          ]\n        },\n        \"data\": {\n          \"description\": \"Parsed JSON when the textual result is JSON.\"\n        },\n        \"name\": {\n          \"type\": \"string\"\n        },\n        \"mime\": {\n          \"type\": \"string\"\n        },\n        \"files\": {\n          \"type\": \"array\",\n          \"items\": {\n            \"type\": \"object\",\n            \"required\": [\n              \"name\",\n              \"mime\",\n              \"bytes\",\n              \"base64\"\n            ],\n            \"properties\": {\n              \"name\": {\n                \"type\": \"string\"\n              },\n              \"mime\": {\n                \"type\": \"string\"\n              },\n              \"bytes\": {\n                \"type\": \"integer\"\n              },\n              \"base64\": {\n                \"type\": \"string\",\n                \"contentEncoding\": \"base64\"\n              }\n            }\n          }\n        }\n      }\n    }\n  }\n}\n```\n\n#### Error and retry handling\n\nFailures return a non-200 status and `error.code` plus `error.message`. Correct invalid input before retrying; retry server-busy responses with bounded backoff. Returned files contain Base64 data, not persistent download URLs. Decode and inspect the output before treating conversion as successful. See the shared API guide for the complete status and timeout rules.\n\n### Browser-agent workflow\n\n1. Open the human page in a browser that supports WebMCP.\n2. Discover the interface exposed by that page; use its actual schema.\n3. Select files on the page first when the tool requires files.\n4. Call `run_current_tool` with valid input and settings.\n5. Inspect the returned result or error and the displayed output. Retrieve files from the displayed download links.\n\nCalls do not automatically copy or download results. Browser permissions still apply. Interface availability alone is not proof of successful execution.\n\n### Planning specification\n\nGroup near-duplicate text rows in uploaded CSV; set similarity threshold; review proposed groups before export; never automatically discard records based only on embeddings.\n\nThis describes intended requirements. Use the implementation notes, interface schema and observed output to determine current support.\n\n### Complete machine-readable capability record\n\n```json\n{\n  \"id\": \"semantic-deduplication\",\n  \"category\": \"data-tools\",\n  \"built\": true,\n  \"requirements\": \"Group near-duplicate text rows in uploaded CSV; set similarity threshold; review proposed groups before export; never automatically discard records based only on embeddings.\",\n  \"implementation\": {\n    \"fields\": [\n      {\n        \"key\": \"column\",\n        \"label\": \"Text column name\",\n        \"type\": \"text\",\n        \"value\": \"text\"\n      },\n      {\n        \"key\": \"threshold\",\n        \"label\": \"Similarity threshold\",\n        \"type\": \"number\",\n        \"value\": 0.85,\n        \"min\": 0.5,\n        \"max\": 1\n      }\n    ],\n    \"files\": true,\n    \"multiple\": false,\n    \"input\": false,\n    \"processing\": \"Server required\",\n    \"native\": true,\n    \"extra\": true,\n    \"maxFileBytes\": 20971520,\n    \"accept\": \".csv\",\n    \"note\": \"Local MiniLM embeddings propose similarity groups. At most 500 rows with 1,000 characters per text value. All rows are retained for review; no automatic deletion.\",\n    \"experience\": {\n      \"automatic\": false,\n      \"single\": false,\n      \"label\": \"Your text\",\n      \"placeholder\": \"Type or paste your input…\",\n      \"help\": \"Up to 1,000,000 characters. Review the settings before running.\",\n      \"action\": null\n    },\n    \"mode\": \"explicit\",\n    \"interactive\": false\n  },\n  \"execution\": \"POST /api/v1/tools/semantic-deduplication/run\",\n  \"api\": {\n    \"id\": \"semantic-deduplication\",\n    \"name\": \"Semantic deduplication\",\n    \"category\": \"data-tools\",\n    \"description\": \"Local MiniLM embeddings propose similarity groups.\",\n    \"notes\": \"Local MiniLM embeddings propose similarity groups. At most 500 rows with 1,000 characters per text value. All rows are retained for review; no automatic deletion.\",\n    \"human_path\": \"/en/data-tools/semantic-deduplication/\",\n    \"method\": \"POST\",\n    \"endpoint\": \"/api/v1/tools/semantic-deduplication/run\",\n    \"schema_url\": \"/api/v1/tools/semantic-deduplication\",\n    \"input_schema\": {\n      \"type\": \"object\",\n      \"additionalProperties\": false,\n      \"properties\": {\n        \"input\": {\n          \"type\": \"string\",\n          \"maxLength\": 1000000,\n          \"default\": \"\",\n          \"description\": \"Plain text input. File tools use files instead unless otherwise documented.\"\n        },\n        \"options\": {\n          \"type\": \"object\",\n          \"additionalProperties\": false,\n          \"properties\": {\n            \"column\": {\n              \"type\": \"string\",\n              \"description\": \"Text column name\",\n              \"default\": \"text\"\n            },\n            \"threshold\": {\n              \"type\": \"number\",\n              \"description\": \"Similarity threshold\",\n              \"default\": 0.85,\n              \"minimum\": 0.5,\n              \"maximum\": 1\n            }\n          }\n        },\n        \"files\": {\n          \"type\": \"array\",\n          \"maxItems\": 1,\n          \"items\": {\n            \"type\": \"object\",\n            \"required\": [\n              \"name\",\n              \"base64\"\n            ],\n            \"additionalProperties\": false,\n            \"properties\": {\n              \"name\": {\n                \"type\": \"string\",\n                \"maxLength\": 200,\n                \"description\": \"Filename only, no path.\"\n              },\n              \"mime\": {\n                \"type\": \"string\",\n                \"maxLength\": 150\n              },\n              \"bytes\": {\n                \"type\": \"integer\",\n                \"minimum\": 0,\n                \"description\": \"Optional decoded byte count; must match content if supplied.\"\n              },\n              \"base64\": {\n                \"type\": \"string\",\n                \"contentEncoding\": \"base64\",\n                \"description\": \"File bytes as padded base64. Use the tool-specific limits.files_bytes value for the total decoded input size.\"\n              }\n            }\n          }\n        }\n      }\n    },\n    \"output_schema\": {\n      \"type\": \"object\",\n      \"required\": [\n        \"tool\",\n        \"result\"\n      ],\n      \"properties\": {\n        \"tool\": {\n          \"type\": \"string\"\n        },\n        \"result\": {\n          \"type\": \"object\",\n          \"required\": [\n            \"text\",\n            \"files\"\n          ],\n          \"properties\": {\n            \"text\": {\n              \"type\": [\n                \"string\",\n                \"null\"\n              ]\n            },\n            \"data\": {\n              \"description\": \"Parsed JSON when the textual result is JSON.\"\n            },\n            \"name\": {\n              \"type\": \"string\"\n            },\n            \"mime\": {\n              \"type\": \"string\"\n            },\n            \"files\": {\n              \"type\": \"array\",\n              \"items\": {\n                \"type\": \"object\",\n                \"required\": [\n                  \"name\",\n                  \"mime\",\n                  \"bytes\",\n                  \"base64\"\n                ],\n                \"properties\": {\n                  \"name\": {\n                    \"type\": \"string\"\n                  },\n                  \"mime\": {\n                    \"type\": \"string\"\n                  },\n                  \"bytes\": {\n                    \"type\": \"integer\"\n                  },\n                  \"base64\": {\n                    \"type\": \"string\",\n                    \"contentEncoding\": \"base64\"\n                  }\n                }\n              }\n            }\n          }\n        }\n      }\n    },\n    \"processing\": \"Server; self-hosted native engine, no external conversion service\",\n    \"limits\": {\n      \"request_bytes\": 31457280,\n      \"input_characters\": 1000000,\n      \"files_bytes\": 20971520,\n      \"max_files\": 1,\n      \"timeout_seconds\": 90\n    }\n  },\n  \"agent_processing\": \"Server; self-hosted native engine, no external conversion service\",\n  \"browser_agent\": {\n    \"supported\": true,\n    \"name\": \"run_current_tool\",\n    \"discovery\": \"WebMCP on the human page in a supporting browser\",\n    \"verification\": \"See audit/API-AUDIT.md; availability is not verification\"\n  }\n}\n```\n\n## Page guide\n\nLocal MiniLM embeddings propose similarity groups.\n\n## How to use Semantic deduplication\n\n1. Choose a file in CSV.\n2. Review the settings, then choose Run Semantic deduplication.\n3. Review the result, then use Copy result or a download link when available.\n\n## What to expect\n\nLocal MiniLM embeddings propose similarity groups. Local MiniLM embeddings propose similarity groups. At most 500 rows with 1,000 characters per text value. All rows are retained for review; no automatic deletion.\n"
}