Url, allowedDomains, deniedDomains, timeout, priority, delay, retr圜ount, retryDelay, jQuery, device, username, password, evaluatePage The following options are passed to nnect().Īlso, the following options can be set as default values when crawler.queue() are executed. This method connects to an existing Chromium instance.

returns: > Promise which resolves to HCCrawler instance.

onError(error) Function to be called when request fails.

links List of links found in the requested page.

screenshot Buffer with the screenshot image, which is null when screenshot option not passed.

result The result resolved from evaluatePage() option.

ok whether the status code in the range 200-299 or not.

onSuccess(response) Function to be called when evaluatePage() successes.

options crawler.queue()'s options with default values.

You can also return false if you want to skip the request.

preRequest(options) Function to do anything like modifying options before each request.

persistCache Whether to clear cache on closing or disconnecting from the Chromium instance, defaults to false.

cache A cache object which extends BaseCache's interfaces to remember and skip duplicate requests, defaults to a SessionCache object.

exporter An exporter object which extends BaseExporter's interfaces to export results, default to null.

maxRequest Maximum number of requests, defaults to 0.

maxConcurrency Maximum number of pages to open concurrently, defaults to 10.```jsconst HCCrawler = require('headless-chrome-crawler') To skip the download, see Environment variables. During installation, it automatically downloads a recent version of Chromium. Note: headless-chrome-crawler contains Puppeteer.

```yarn add headless-chrome-crawler or 'npm i headless-chrome-crawler' Save screenshots for the crawling evidence.Insert jQuery automatically for scraping.Pause at the max request and resume at any time.Support CSV and JSON Lines for exporting results.Breadth-first search (BFS) to automatically follow links.Powered by Headless Chrome, the crawler provides simple APIs to crawl these dynamic websites with the following features: However, it sometimes ends up capturing empty bodies, especially when the websites are built on such modern frontend frameworks as AngularJS, React and Vue.js. Distributed crawler powered by Headless ChromeĬrawlers based on simple requests to HTML files are generally fast.

Free download puppeteer github