postlight /parser
📜 Extract meaningful content from the chaos of a web page
★ 5,788⑂ 525👁 90⑃ 515● 220▤ 23
+0
日增长
5,788
stars
525
forks
90
watchers
515
prs
220
issues
23
releases
README
Postlight Parser - Extracting content from chaos
[![Apache License][license-apach-badge]][license-apach] [![MITC License][license-mit-badge]][license-mit]
[license-apach-badge]: https://img.shields.io/badge/License-Apache%202.0-blue.svg?style=flat-square
[license-apach]: https://github.com/postlight/mercury-parser/blob/master/LICENSE-APACHE
[license-mit-badge]: https://img.shields.io/badge/License-MIT%202.0-blue.svg?style=flat-square
[license-mit]: https://github.com/postlight/mercury-parser/blob/master/LICENSE-MIT
Postlight's Parser extracts the bits that humans care about from any URL you give it. That includes article content, titles, authors, published dates, excerpts, lead images, and more.
Postlight Parser powers Postlight Reader, a browser extension that removes ads and distractions, leaving only text and images for a beautiful reading view on any site.
Postlight Parser allows you to easily create custom parsers using simple JavaScript and CSS selectors. This allows you to proactively manage parsing and migration edge cases. There are many examples available along with documentation.
How? Like this.
Installation
bash
If you're using yarn
yarn add @postlight/parser
If you're using npm
npm install @postlight/parser
Usage
javascript
import Parser from '@postlight/parser';
Parser.parse(url).then(result = console.log(result));
// NOTE: When used in the browser, you can omit the URL argument
// and simply run Parser.parse() to parse the current page.
The result looks like this:
json
{
"title": "Thunder (mascot)",
"content": "... Thunder is the stage name for the...",
"author": "Wikipedia Contributors",
"date_published": "2016-09-16T20:56:00.000Z",
"lead_image_url": null,
"dek": null,
"next_page_url": null,
"url": "https://en.wikipedia.org/wiki/Thunder_(mascot)",
"domain": "en.wikipedia.org",
"excerpt": "Thunder Thunder is the stage name for the horse who is the official live animal mascot for the Denver Broncos",
"word_count": 4677,
"direction": "ltr",
"total_pages": 1,
"rendered_pages": 1
}
If Parser is unable to find a field, that field will return null.
parse() Options
Content Formats
By default, Postlight Parser returns the content field as HTML. However, you can override this behavior by passing in options to the parse function, specifying whether or not to scrape all pages of an article, and what type of output to return (valid values are 'html', 'markdown', and 'text'). For example:
javascript
Parser.parse(url, { contentType: 'markdown' }).then(result =
console.log(result)
);
This returns the the page's content as GitHub-flavored Markdown:
json
"content": "...Thunder is the stage name for the..."
Custom Request Headers
You can include custom headers in requests by passing name-value pairs to the parse function as follows:
javascript
Parser.parse(url, {
headers: {
Cookie: 'name=value; name2=value2; name3=value3',
'User-Agent':
'Mozilla/5.0 (iPhone; CPU iTop issues
#605Mercury Reader doesnt send articles to amazon account9 comments · 2021-01-25#425Invalid decoded text8 comments · 2019-05-24#591Send to kindle - images not sent7 comments · 2020-11-08#715Send to Kindle EPUB title issues5 comments · 2022-11-23#471Impossible to run `preview` because of linting errors5 comments · 2019-08-15#516gq.com page truncated4 comments · 2019-12-06#708Fix CI3 comments · 2022-11-01#675Header cleaning is a bit too aggressive3 comments · 2022-07-27