2026.10.19
Crawlee: Building Web Crawling Workflows in Node.js
Crawlee gives Node.js projects a common interface for HTTP and browser crawling, with queues and storage. Before adoption, review the Node.js requirement, browser dependencies, local storage defaults, and target-site rules.
What the project is for
Crawlee is a web scraping and browser automation library for Node.js, intended for JavaScript and TypeScript projects. Rather than being only an HTTP request utility, it organizes URL scheduling, page handling, and result storage into a crawler workflow. Project materials list link crawling, data extraction, and downloading website files such as HTML, PDF, JPG, and PNG among its uses. They also name data extraction for AI, LLM, RAG, or GPT work. That does not mean Crawlee automatically cleans data, labels it, or builds a knowledge base.
Who it suits
Crawlee suits developers already working with Node.js who need to turn collection scripts into recurring crawler jobs. For static pages, HTTP crawling can work with Cheerio, JSDOM, or raw HTTP. For pages that depend on JavaScript rendering, the same interface can be used with Playwright or Puppeteer. Its CLI, configurable routing, hooks, retries, and persistent request queue are also relevant where URLs must be scheduled over time.
Known capabilities
The README lists a shared interface for HTTP and headless-browser crawling, along with a persistent URL queue that supports breadth-first and depth-first crawling. For output, Crawlee offers pluggable storage for tabular data and files. Its browser tooling supports JavaScript rendering, screenshots, and both headless and headful operation. The project also lists proxy rotation, session management, error handling, retries, and automatic scaling with available system resources.
By default, data is written to ./storage under the current working directory, and configuration can change that location. The README also lists deployable Dockerfiles. The repository is licensed under Apache License 2.0; users should still assess the license text in light of their own distribution, dependency, and compliance context.
Limits and deployment considerations
The runtime requirement comes first: the README requires Node.js 22.13 or later. If PlaywrightCrawler is selected, Playwright is not bundled with Crawlee and must be handled as an additional dependency. Browser-based crawling also means browser installation and process management need to be included in the deployment design. The supplied sources provide no minimum CPU, memory, disk, or concurrency specification, so they cannot support infrastructure sizing.
The default local ./storage path is useful for getting started, but a production deployment should define where data persists, how it is backed up, and how capacity is managed. Queues, result files, and retries should be planned alongside logging and recovery procedures. Page structure, request cadence, authentication, and network conditions at the target site can affect collection results. The README mentions proxy rotation, session management, and browser-fingerprint-related features, but these are not a guarantee of access or results for every site.
The official v3.18.2 release notes record a fix for browser-pool handling when a page close does not settle, plus a fix related to timed-out task context in the core pool loop. When upgrading, teams can pin the version in use and validate browser dependencies, output formats, and any existing Apify SDK integration.
Sources
Sources checked: 2026-10-19