TheCrawler

Indexed by io.github.manchittlab

About

Universal web scraper with LLM-ready markdown, RAG chunking, PDF/DOCX support.

Use this server

Add it to your MCP client. The catalogue lists this config verbatim from its source — it does not run, download, or vouch for the server. Review the command before you run it.

Transportstdio (runs a local command)

Claude Code

claude mcp add thecrawler -- npx thecrawler

Cursor

Add to Cursor

If the button doesn't open Cursor, use the JSON below — Cursor accepts the same mcpServers config.

VS Code

code --add-mcp '{"name":"thecrawler","command":"npx","args":["thecrawler"]}'

JSON

{
  "mcpServers": {
    "thecrawler": {
      "command": "npx",
      "args": [
        "thecrawler"
      ]
    }
  }
}