Skip to main content

HTML::ArticleExtractor - Article Scraper

Scraper overview​

Scraper overviewHTML::ArticleExtractorHTML::ArticleExtractor collects articles from web pages.

It works using the @mozilla/readability module, which is built into A-Parser and collects core data such as: title, content with and without HTML formatting, and article length.

It is based on the Net::HTTPNet::HTTP scraper, which allows it to support its functionality. It supports multi-page scraping (pagination). It has built-in tools to bypass CloudFlare protection and also the option to choose Chrome as the engine for scraping emails from pages where data is loaded by scripts.

Capable of reaching speeds up to 200 requests per minute, – which is 12 000 links per hour.

Collected data​

  • Article title - $title
  • HTML string of the processed article content - $content
  • Text content of the article (all HTML removed) - $textContent
  • Article length in characters - $length
  • Article description or a short excerpt from the content - $excerpt
  • Author metadata - $byline
  • Site name - $siteName

Capabilities​

  • Multi-page scraping (pagination support)
  • Supports gzip/deflate/brotli compression
  • Detection and conversion of website encodings to UTF-8
  • CloudFlare protection bypass
  • Engine selection (HTTP or Chrome)
  • Ability to set article length
  • Scraping articles with and without HTML tags

Use cases​

  • Collecting ready-made articles from any websites

Queries​

Specify links to the pages from which you need to scrape articles as queries, for example:

https://a-parser.com/docs/
https://lenta.ru/articles/2021/09/11/buran/
https://www.thetimes.co.uk/article/the-russian-banker-the-royal-fixers-and-a-500-000-riddle-vvgc55b2s

Output results examples​

A-Parser supports flexible result formatting thanks to the built-in Template Toolkit, which allows it to output results in any format, as well as structured ones like CSV or JSON

Possible settings​

note

Common settings for all scrapers Supports all settings of the Net::HTTPNet::HTTP scraper.