TRAWL: A Self-Hosted Challenge-Aware Scraping Engine

What it is for

TRAWL is an AGPL-3.0 self-hosted web-scraping engine designed for ordinary requests as well as sites that present JavaScript challenges or CAPTCHAs. Its FlareSolverr-compatible /v1 endpoint is intended to fit tools such as Prowlarr, Jackett, and Sonarr. A native scraping endpoint returns richer metadata.

The README describes a four-tier path: a plain HTTP request, a cached browser session, a fresh challenge solve, and a residential proxy. It also documents Redis-backed session caching, Camoufox Firefox, and dedicated flows for Cloudflare, Akamai Bot Manager, and Imperva/Incapsula. Listed CAPTCHA-related capabilities include Turnstile, reCAPTCHA, hCaptcha, GeeTest, ALTCHA, and Friendly Captcha. These are documented capabilities, not a guarantee that every target site will work.

Who may benefit

TRAWL is most relevant to operators already running Prowlarr or another *arr service who want a self-hosted component for protected indexer workflows. Automation users can also examine its /mcp endpoint. In v1.5.0, the project expanded MCP with tools for reading pages, scraping, screenshots, and browser diagnostics. The documentation says these tools load known public URLs and do not provide web search or ranking, so TRAWL is not a general-purpose search service.

It may also suit users comfortable maintaining Redis, browser runtimes, container networking, and certificate trust. Community app catalogs for TrueNAS and Unraid provide installation paths, although the supplied sources do not establish the exact maintenance scope or behavior of those integrations.

Limitations before adoption

The project describes challenge and CAPTCHA handling as “best effort.” README claims about being “2–6x faster” and having a higher success rate are project statements, not independent test results. The supplied sources do not confirm minimum hardware, sustained concurrency, or a site-by-site compatibility matrix.

Residential proxies appear as part of the escalation path, but the sources do not confirm that proxy capacity is bundled with TRAWL, nor do they establish pricing or legal boundaries. Operators must assess target-site terms, applicable law, and data-handling obligations themselves. CAPTCHA flows may also depend on third-party services or optional local Whisper components, so availability must be checked against the chosen configuration.

Deployment and upgrade considerations

The README’s basic deployment uses Docker Compose and Redis. It says the browser pool may take 15–30 seconds to warm on first boot. If the challenge-aware HTTP/HTTPS MITM proxy is enabled, clients must trust TRAWL’s generated and persisted root CA. This materially expands the trust boundary: the project warns that the proxy can impersonate hosts to clients trusting that CA and should be limited to localhost or a private Docker network.

Version 1.5.0 requires migration attention. Legacy environment-variable aliases were removed, and an empty or unset REDIS_URL explicitly disables the Redis-backed cache. Before upgrading, review the configuration migration documentation and verify Redis connectivity, browser processes, proxy certificates, and downstream *arr clients.

Sources

• GitHub repository metadata
• Project README
• Official release
• Official project page

Sources checked: 2026-09-15