yujiosaka /headless-chrome-crawler

Distributed crawler powered by Headless Chrome

★ 5,631⑂ 404👁 112⑃ 256● 135▤ 20
JavaScriptMITCreated 2017-12-02
下载:1.8.01.8.0 ↓1.8.02018-06-11
+2
日增长
5,631
stars
404
forks
112
watchers
256
prs
135
issues
20
releases

README

Headless Chrome Crawler   

 API | Examples | Tips | Code of Conduct | Contributing | Changelog

Distributed crawler powered by Headless Chrome



 Features

Crawlers based on simple requests to HTML files are generally fast. However, it sometimes ends up capturing empty bodies, especially when the websites are built on such modern frontend frameworks as AngularJS, React and Vue.js.

Powered by Headless Chrome, the crawler provides simple APIs to crawl these dynamic websites with the following features:

 Distributed crawling
 Configure concurrency, delay and retry
 Support both depth-first search and breadth-first search algorithm
 Pluggable cache storages such as Redis
 Support CSV and JSON Lines for exporting results
 Pause at the max request and resume at any time
 Insert jQuery automatically for scraping
 Save screenshots for the crawling evidence
 Emulate devices and user agents
 Priority queue for crawling efficiency
 Obey robots.txt
 Follow sitemap.xml
 [Promise] support

 Getting Started

 Installation

sh
yarn add headless-chrome-crawler
 or "npm i headless-chrome-crawler"


 Note: headless-chrome-crawler contains Puppeteer. During installation, it automatically downloads a recent version of Chromium. To skip the download, see Environment variables.

 Usage

js
const HCCrawler = require('headless-chrome-crawler');

(async () = {
  const crawler = await HCCrawler.launch({
    // Function to be evaluated in browsers
    evaluatePage: (() = ({
      title: $('title').text(),
    })),
    // Function to be called with evaluated results from browsers
    onSuccess: (result = {
      console.log(result);
    }),
  });
  // Queue a request
  await crawler.queue('https://example.com/');
  // Queue multiple requests
  await crawler.queue(['https://example.net/', 'https://example.org/']);
  // Queue a request with custom options
  await crawler.queue({
    url: 'https://example.com/',
    // Emulate a tablet device
    device: 'Nexus 7',
    // Enable screenshot by passing options
    screenshot: {
      path: './tmp/example-com.png'
    },
  });
  await crawler.onIdle(); // Resolved when no queue is left
  await crawler.close(); // Close the crawler
})();


 Examples

 Priority queue for crawling efficiency
 Emulate device and user agent
 Redis cache to skip duplicate requests
 Export a CSV file for crawled results
 Conditionally saving screenshots

See here for the full examples list. The examples can be run from the root folder as follows:

sh
NODE_PATH=../ node examples/priority-queue.js


 API reference

See here for the API reference.

 Debugging tips

See here for the debugging tips.

 FAQ

 How is this different from other crawlers?

There are roughly two types of crawlers. One is static and the other is dynamic.

The static crawlers are based on simple requests to HTML files. They are generally fast, but fail scraping the contents when the HTML dynamically changes on browsers.

Dynamic crawlers based on PhantomJS and Selenium work magica

Top issues

Latest release

1.8.0 1.8.0 2018-06-11
Added

- Set previousUrl to onSuccess argument.
- Set options, depth, previousUrl to errors.
- Support customCrawl for HCCrawler.connect() and HCCrawler.launch()'s options.
- Add Dockerfile and tips for using Docker.

 Changed

- Drop newpage event.
- Update Puppeteer version to 1.5.0.

 Fixed

- Fix a bug of not marking skipped requests correctly.
- Fix requestfinished event's argument as described in the API reference.

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON