apify /crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.
★ 9,675⑂ 833👁 44⑃ 1,725● 636▤ 82
+3
日增长
9,675
stars
833
forks
44
watchers
1,725
prs
636
issues
82
releases
README
A web scraping and browser automation library
Crawlee covers your crawling and scraping end-to-end and helps you build reliable scrapers. Fast.
Your crawlers will appear almost human-like and fly under the radar of modern bot protections even with the default configuration. Crawlee gives you the tools to crawl the web for links, scrape data and persistently store it in machine-readable formats, without having to worry about the technical details. And thanks to rich configuration options, you can tweak almost any aspect of Crawlee to suit your project's needs if the default settings don't cut it.
👉 View full documentation, guides and examples on the Crawlee project website 👈
We also have a TypeScript implementation of the Crawlee, which you can explore and utilize for your projects. Visit our GitHub repository for more information Crawlee for JS/TS on GitHub.
Installation
We recommend visiting the Introduction tutorial in Crawlee documentation for more information.
Crawlee is available as crawlee package on PyPI. This package includes the core functionality, while additional features are available as optional extras to keep dependencies and package size minimal.
To install Crawlee with all features, run the following command:
sh
python -m pip install 'crawlee[all]'
Then, install the Playwright dependencies:
sh
playwright install
Verify that Crawlee is successfully installed:
sh
python -c 'import crawlee; print(crawlee.__version__)'
For detailed installation instructions see the Setting up documentation page.
With Crawlee CLI
The quickest way to get started with Crawlee is by using the Crawlee CLI and selecting one of the prepared templates. First, ensure you have uv installed:
sh
uv --help
If uv is not installed, follow the official installation guide.
Then, run the CLI and choose from the available templates:
sh
uvx 'crawlee[cli]' create my-crawler
If you already have crawlee installed, you can spin it up by running:
sh
crawlee create my-crawler
Examples
Here are some practical examples to help you get started with different types of crawlers in Crawlee. Each example demonstrates how to set up and run a crawler for specific use cases, whether you need to handle simple HTML pages or interact with JavaScript-heavy sites. A crawler run will create a storage/ directory in your current working directory.
BeautifulSoupCrawler
The BeautifulSoupCrawler downloads web pages using an HTTP library and provides HTML-parsed content to the user. By default it uses ImpitHttpClient for HTTP communication and BeautifulSoup for parsing HTML. It is ideal for projects that require efficient extraction of data from HTML content. This crawler has very good performance since it does not use a browser. However, if you need to execute client-side JavaScript, to get your content, this is not going to be enough and you will need to use PlaywrightCrawler. Also if you want to use this crawler, make sure youTop issues
#925Add get_run_state method to BasicCrawler10 comments · 2025-01-21#2242Keep a single `EventManager` per process (event loop)9 comments · 2026-09-19#2020Docs search is broken on crawlee.dev/python4 comments · 2026-07-05#1842Possible deadlock in `SitemapRequestLoader`4 comments · 2026-04-08#1600Pass router as request_handler to crawler - is this intended?4 comments · 2025-12-05#1953Feature Request: Add serpbase.dev as a URL discovery source for research-style crawls3 comments · 2026-06-08#1744Redesign browser pool and the whole browsers subpackage3 comments · 2026-02-12#1253Importing the crawler class takes over 2 seconds3 comments · 2025-06-17
Discussions
#2333Showcase: ATS Zero-Browser Gateway — Fast protocol-level scraper on Apify0 comments · 2026-10-15#2281Using crawlee-python with a fingerprint-patched Chromium (tested on 1.10.4, Windows, 0 failed sannysoft checks)0 comments · 2026-10-06#2256python/docs/introduction0 comments · 2026-09-25#2180We’re building a context-aware code review tool and would love some feedback0 comments · 2026-08-20#2028How to keep cookies on request retry?3 comments · 2026-07-06#2019Search on your docs page doesn't seem to work2 comments · 2026-07-04#2006Realistic Firefox binary to pair with browser_type='firefox'6 comments · 2026-07-02#1967add_requests with a token that has an expiry time1 comments · 2026-06-15
Latest release
1.11.2 v1.11.2 2026-11-101.11.2 (2026-11-10) 🐛 Bug Fixes - Stop truncating fractional robots.txt crawl-delay to an integer (2392) (fa386bb) by @RaphaelFakhri - Fit the default desired_concurrency within min_concurrency and max_concurrency (2397) (b71d5af) by @PrashanthLande - Pass the crawler state to the rendering type detection run of AdaptivePlaywrightCrawler (2417) (4d3f989) by @Mantisus - Accept None for proxy_configuration in BasicCrawlerOptions (2416) (16858fd) by @vdusek - file-system: Stop KVS pagination from skipping or repeating keys (2431) (7d271c0) by @peterolkhov