Scraping Archive.org with A-Parser: Extract Thousands of Articles from the Wayback Machine

[Support Anastasia]

Scraping Archive.org with A-Parser: Extract Thousands of Articles from the Wayback Machine

Executive Summary

Search engines only index the live web, so historical articles and older pages disappear over time: according to Pew Research Center, 38% of sampled web pages from 2013 were unavailable by October 2023.

The Wayback Machine (Archive.org) is one of the main archives of the web. Traditional crawling is inefficient, so professional extraction relies on two core mechanisms:

• CDX Server API: Instantly queries the archive index, returning up to 5,000 unique URLs in 2–7 seconds (~350 KB JSON payload) without downloading heavy HTML.
• Clean id_ Mode: Direct retrieval of raw original HTML stripped of Wayback toolbars, analytics counters, and injected archive scripts.

In this guide, we break down two practical scraping workflows in A-Parser:
1. A No-Code Pipeline chaining built-in scraping modules Net::HTTP ➔ HTML::ArticleExtractor for rapid export to JSONL.
2. A Standalone TypeScript Scraper (JS::WaybackCDX) featuring multi-level subqueries (this.query.add), 15–50 concurrent worker threads, URL canonicalization, Mozilla Readability content extraction, and automated export to Markdown (.md) with YAML Frontmatter (benchmarked at 131 articles/min — 600 articles in 9 minutes).

---

1. WHY HISTORICAL CONTENT VANISHES — AND WHERE TO FIND ARCHIVED COPIES

Modern search engines are built to surface the current, live web, not preserve digital archives. When a site rebrands, revamps its pricing, or prunes older posts, search engines update their indexes and remove old URLs from search results. (Google recently discontinued its cached webpage feature entirely.)

As a consequence, valuable historical data disappears from the public web. In Pew Research Center's study "When Online Content Disappears", 38% of sampled web pages from 2013 were unavailable by October 2023.

The Wayback Machine (Archive.org) is one of the main sources of historical copies. It preserves snapshots of public web pages and lets you check what publications, prices, or product descriptions looked like on a given date. Availability and completeness depend on what the archive captured.

What You Can Do with Wayback Machine Data:

1. Populating Knowledge Bases and RAG Systems:
   Large Language Models and Retrieval-Augmented Generation (RAG) systems require clean, structured text free of technical clutter. Unlike raw HTML, where scripts, banners, and counters can make up 70–85% of the volume, saving articles directly as Markdown with metadata reduces text volume fourfold and provides a useful source of historical data for RAG systems.
2. Populating PBNs and Drop-Domain Sites with Content:
   To restore the historical topical authority of a purchased drop domain, you need its original content. A-Parser downloads the site's complete archive of publications, including headlines, dates, and authors. The resulting Markdown files can be imported directly into WordPress or any other CMS, saving hundreds of hours of manual rewriting.
3. Tracking Price Trends and Testing Market Hypotheses:
   Track how the actual prices of goods, real estate, or cars changed over the past 10–15 years. Analysts and research agencies use historical price and news snapshots to build accurate forecasting models without distorting the data.
4. Competitive Intelligence and Auditing Offer Evolution:
   Analyze how market leaders changed their pricing, positioning, and editorial policies over the last 5–10 years. The archive reveals the real history of competitors' product experiments: which offers took off and which failed and were removed.
5. Finding Deleted Information and Pages That Are No Longer Public:
   Many platforms used simpler interfaces and open data structures in their early versions. Web archives allow investigators to find deleted journalistic investigations, closed sections, and older publications that disappeared from search results.

In the walkthrough below, we break down two approaches in A-Parser: a visual No-Code workflow and a ready-made TypeScript scraper. Both approaches separate snapshot discovery from content extraction, outputting clean Markdown and JSONL.

---

2. THE ARCHITECTURE OF WAYBACK SCRAPING: CDX API & CLEAN ID_ MODE

Crawling the Wayback Machine by following on-page links is inefficient: you will waste hours traversing archive menus, pagination, and broken links.

A practical extraction pipeline relies on two fast operations:

1. Instant URL Discovery via the CDX Server API — Retrieve a lightweight JSON list of saved pages without downloading heavy HTML.
2. Fetching Snapshots in Clean id_ Mode — Download raw original snapshot HTML stripped of Wayback toolbars, analytics widgets, and injected scripts.

2.1. CDX API: Instant Snapshot Indexing vs. Blind Crawling

The CDX Server API is the search index powering the Wayback Machine. Instead of downloading terabytes of web pages, you send a single HTTP request and receive a precise list of all available archived pages from the site for the requested years.

Example of an optimized CDX index query:

https://web.archive.org/cdx/search/cdx?url=smashingmagazine.com/&matchType=prefix&collapse=urlkey&output=json&fl=timestamp,original&filter=statuscode:200&filter=mimetype:text/html&from=2020&to=2021&limit=5000

Key CDX Query Parameters Explained:

Parameter	Example Value	Purpose & Technical Function
url + matchType=prefix	smashingmagazine.com/	Discovers all pages under a domain or specific directory path (/blog/, /articles/).
filter=statuscode:200	statuscode:200	Excludes 404, 500, and 301 responses, capturing only successfully archived pages.
filter=mimetype:text/html	text/html	Filters out images, CSS, JavaScript, and binary PDF assets at the API level.
collapse=urlkey	urlkey	Core Deduplication Filter: Collapses hundreds of repeated snapshots of the same URL down to a single unique record.
fl	timestamp,original	Field selector: returns only the timestamp and original URL, reducing payload size by up to 80%.
from / to	2020 / 2021	Restricts snapshot capture dates to the specified year range.
limit	5000	Limits batch size to prevent socket timeouts on large domains.

Performance Benchmark: A single CDX query for smashingmagazine.com or theverge.com returns 5,000 unique URLs in 2–7 seconds, with a payload footprint of just ~350 KB.

2.2. Clean id_ Mode: Stripping Wayback Toolbars, Scripts, and Boilerplate

When you browse an archived page through a standard browser, Archive.org injects a large toolbar, JavaScript tracking snippets, and rewrites internal URLs.

If you pass this bloated HTML directly into a content extraction library (Mozilla Readability, Trafilatura), archive boilerplate will contaminate your extracted article body.

To request the raw, pristine snapshot HTML as originally served by the target server, simply append the id_ flag immediately after the 14-digit timestamp:

STANDARD REPLAY URL (Contains injected Wayback toolbars and scripts):
https://web.archive.org/web/20210301134743/http://www.theverge.com/2011/06/09/google-voice-skype-imessage-and-the-death-of-the-phone-number/
  └── Injects iframe toolbars, analytics counters, and modified DOM trees.

CLEAN REPLAY URL WITH id_ (Pristine original HTML only):
https://web.archive.org/web/20210301134743id_/http://www.theverge.com/2011/06/09/google-voice-skype-imessage-and-the-death-of-the-phone-number/
  └── Delivers the untouched original HTML snapshot, ready for downstream text extraction.

Production Tip: While id_ mode strips the archive's UI toolbar, external third-party assets (such as off-site CDN fonts or deleted JavaScript widgets) may not be stored in the archive. Therefore, the content extraction engine must parse the DOM hierarchy and text nodes directly rather than depending on client-side JS rendering.

---

3. APPROACH #1: THE NO-CODE PIPELINE (NET::HTTP ➔ HTML::ARTICLEEXTRACTOR)

For code-free extraction, A-Parser chains two built-in scraping modules into a two-stage pipeline: Stage 1 queries the CDX API directly without proxies, while Stage 2 fetches the clean id_ replay pages and extracts the main content in parallel; proxies can be used at this stage.

How the Two-Stage Workflow Operates:

1. Stage 1 (Net::HTTP): Queries the CDX API with useproxy disabled, filters for HTTP 200 HTML snapshots, and appends the id_ modifier to generate clean replay links. The resulting URLs are saved to an intermediate queue file cdx_clean_links.txt.
2. Stage 2 (HTML::ArticleExtractor): Consumes the URL queue, downloading and parsing pages concurrently using the configured proxy pool. Built-in Mozilla Readability isolates the core article text and returns the title, author, length, HTML, and plain text. The result is saved as JSONL.

The preset streams one structured JSON line per successfully processed article. It preserves the complete $p1.json object, including the query URL in query.query, title title, author byline, length length, clean text textContent, HTML content, website name siteName, HTTP status code, and success flag success:

{"query":{"query":"https://web.archive.org/web/20210301134743id_/http://www.theverge.com/2011/06/09/google-voice-skype-imessage-and-the-death-of-the-phone-number/"},"title":"iMessage, Skype, Google Voice, and the death of the phone number","byline":"Nilay Patel","length":6840,"textContent":"Extracted clean article body...","siteName":"The Verge","code":200,"success":1}

Performance Benchmark: No-Code Pipeline (Smashing Magazine, The Verge, TechCrunch)

We tested the Net::HTTP ➔ HTML::ArticleExtractor preset across 3 major media domains with 15 concurrent worker threads using the configured proxy pool:

• Input Queries:
  

  smashingmagazine.com 2021 200
  www.theverge.com 2021 200
  techcrunch.com 2019 200

• Stage 1 (CDX Discovery): 600 unique id_ links (200 per domain) gathered in just 6 seconds, with 0 errors.
• Stage 2 (ArticleExtractor): Parallel extraction with 15 concurrent worker threads using the configured proxy pool at an average throughput of ~80–90 pages per minute.
• Data Quality: 100% successful requests (totalFail = 0); articles were automatically cleaned of archive boilerplate via Mozilla Readability, preserving titles, authors, lengths, and text.
• Final Output: theverge_articles_clean.jsonl containing structured JSON lines.

Download the Ready-to-Use Preset: preset.txt

Quick Setup Guide:

1. Navigate to Task Editor ➔ Task ➔ select Import Preset from the dropdown and paste the code from preset.txt.
2. In the Queries list field, specify your target domains with the year and article limit separated by spaces:
   

   smashingmagazine.com 2021 200
   www.theverge.com 2021 200
   techcrunch.com 2019 200

3. Click Add task. The built-in Query Builder will parse the domain, year, and limit, execute the CDX discovery query, save the resulting id_ URLs, and start a second task to process them in parallel.

---

4. APPROACH #2: STANDALONE TYPESCRIPT SCRAPER (JS::WAYBACKCDX)

Built-in scraping modules solve separate tasks: downloading pages, extracting content, or retrieving search results. In practice, a full workflow may need URL selection, deduplication, load distribution, and convenient output.

A-Parser supports custom JavaScript and TypeScript scrapers. You define the rules, while A-Parser handles multi-threading, networking, proxies, queues, and data storage.

For this article, we prepared JS::WaybackCDX — a full-featured TypeScript scraper that combines the entire workflow in one run. It searches the CDX API, downloads pages in parallel through the internal subquery queue (this.query.add), cleans them with Mozilla Readability, normalizes URLs to a canonical form, and saves ready-to-use Markdown articles to disk.

Date ranges, article limits, and output formats are configured in a simple interface without touching code. The same workflow can be run for different sites without changing the code:

Under the Hood: Consolidating an Entire Scraping Stack into One Script

Building an equivalent production pipeline from scratch typically requires writing a custom crawler, spinning up Redis for subquery queues, orchestrating proxy pools, and managing a database for deduplication.

In JS::WaybackCDX, this entire process is packaged into a single, self-contained TypeScript script:

1. Instant CDX Discovery & Dynamic Subqueries (this.query.add):
   The scraper queries the lightweight CDX index through a proxy when enabled, filters out service pages, and registers links in A-Parser's internal queue. This lets you use any number of worker threads while viewing dynamic per-article progress in the UI.
2. Smart URL Canonicalization & Noise Filtering:
   The same material can appear under different protocols (http/https), with or without a www prefix, or as a non-content URL. The scraper canonicalizes URLs, rejects service sections (/wp-admin/, /tag/, /category/, media files), and keeps only unique article pages.
3. Ready-to-Use Markdown (.md) with YAML Frontmatter for RAG and LLMs:
   Every article is cleaned with Mozilla Readability and saved as an individual .md file with structured YAML frontmatter (title, author, archive date, original URL). These files can be loaded directly into vector databases, Obsidian, or fine-tuning datasets without additional preprocessing.

Example of a Generated .md Article File:

---
title: "How To Communicate Design Decisions To Clients? — Smashing Magazine"
author: "About The Author"
site_name: "Smashing Magazine"
length: 9301
timestamp: "20210211022848"
original_url: "http://www.smashingmagazine.com/2008/07/how-to-communicate-design-decisions-to-clients/"
archive_url: "https://web.archive.org/web/20210211022848id_/http://www.smashingmagazine.com/2008/07/how-to-communicate-design-decisions-to-clients/"
---

# How To Communicate Design Decisions To Clients? — Smashing Magazine

**Author:** About The Author 
**Source:** Smashing Magazine 
**Archive Replay:** https://web.archive.org/web/20210211022848id_/http://www.smashingmagazine.com/2008/07/how-to-communicate-design-decisions-to-clients/

---

Brian Armstrong is an entrepreneur who also enjoys studying design. He writes about topics such as UI design, building web companies, and how to quit your ...

Performance Benchmark: JS::WaybackCDX (Smashing Magazine, The Verge, TechCrunch)

We benchmarked the TypeScript scraper on real-world media archives with 15 concurrent worker threads using the configured proxy pool:

• Input Queries:
  

  smashingmagazine.com 2021 200
  www.theverge.com 2021 200
  techcrunch.com 2019 200

• CDX Index Retrieval: Snapshots for all 3 domains were retrieved through proxies and filtered, with automatic handling of 502/504 gateway errors.
• Multi-Level Queueing (this.query.add): 600 subqueries (200 per domain) were dynamically enqueued — worker threads extracted content concurrently while the UI counter updated in real time.
• Articles Saved to Disk: Exactly 600 accepted Markdown articles (200 per domain).
• Content Cleaning & Quality: Text extracted via Mozilla Readability; noise and thin content discarded; totalFail = 0 (100% success rate).
• Total Elapsed Time: 9 minutes 08 seconds for all 600 articles (average throughput: 131 articles per minute).
• Disk Output: Directory results/wayback-md-94/ containing individual .md files (YAML Frontmatter + clean text body).

Download the Ready-to-Use Scraper (Single File): https://files.a-parser.com/img/WaybackCDX.ts[/URL
Installation Guide: Two Simple Setup Methods
Method #1: Install via Web UI (Parsers Editor)

1. • In the A-Parser main navigation, open Parsers Editor.
   • In the parser creation panel, configure:
     • Parser Type: select TypeScript.
     • Parser Name (Mandatory): enter WaybackCDX
       (Critical: The parser class is registered internally as JS_<DirectoryName>. The name WaybackCDX must match the directory name exactly, including capitalization, or A-Parser will not load the preset settings and task override fields.)
   • Click Create. The editor will open the newly generated WaybackCDX.ts.
   • Select and delete all boilerplate code, then paste the full source code from [URL=http://s3.a-parser.com/articles/archiveorg/WaybackCDX.ts]WaybackCDX.ts

1. .
2. Click Save (Ctrl + S). A-Parser will automatically compile the TypeScript into a working JavaScript module.

Method #2: Install from the File System

1. Download the single file WaybackCDX.ts.
2. Place it into the directory files/parsers/WaybackCDX/ inside your A-Parser installation (create the WaybackCDX folder if it does not exist).
3. Refresh the A-Parser page or restart the service — the scraper JS::WaybackCDX will immediately appear in your available parsers list.

How to Run the Extraction Task:

1. Open the Task Editor and select JS::WaybackCDX from the parsers dropdown.
2. In the Queries list field, enter your target domains (you can include a year and limit separated by spaces):
   

   smashingmagazine.com 2021 200
   www.theverge.com 2021 200
   techcrunch.com 2019 200

3. Parser Settings (Available in Preset and Task Overrides ️):
   • fromYear and toYear — Snapshot year range (defaults to 2020–2026);
   • maxArticles — Maximum accepted articles per domain (default: 500);
   • minLength — Minimum length of cleaned article text in characters (default: 10);
   • markdownFiles — Automatically save individual .md files with YAML frontmatter to disk (default: enabled);
   • cdxUseProxy — Route CDX discovery queries through proxies (disabled by default for a fast direct connection; page content extraction always uses your proxy pool).
4. Results Format: Set to $p1.articles.format('$json\n') (or keep $p1.preset).
5. Click Add task. The scraper will instantly retrieve the CDX snapshot index, distribute parallel page extraction across worker threads, and save clean Markdown articles to results/wayback-md-<taskId>/.

---

5. COMPARISON MATRIX: WHICH APPROACH SHOULD YOU CHOOSE?

Criterion	Approach #1: No-Code Pipeline	Approach #2: JS/TS Scraper
Setup Complexity	Import preset and configure two linked task stages	Load custom scraper file and configure the limits in the UI
Execution Flow	Stage 1 saves links and launches Stage 2	Single unified task manages discovery and parallel extraction
Intermediate Data	Saves intermediate .txt link list to disk	Adds work to an in-memory subquery queue (this.query.add)
Deduplication & Filtering	Basic URL deduplication at the CDX level	Comprehensive: CDX urlkey + URL canonicalization + filtering service paths and tag pages
Extraction Speed	~80–90 articles/min across 15 threads	131 articles/min across 15 threads (600 articles in 9 min)
Output Formats	JSONL, CSV, TXT	Markdown with YAML Frontmatter, JSONL, Raw HTML
Limit Controls	Configured in the raw CDX query string	Dedicated GUI settings & task overrides

---

6. CONCLUSION: THE ARCHITECTURAL FLEXIBILITY OF A-PARSER

This Wayback Machine implementation showcases A-Parser's core architectural strength: it isn't a rigid black box with fixed functionality, but a scalable platform for data of any volume and complexity.

You choose the exact level of control your workflow demands:

• No-Code Level (Speed & Simplicity):
  Combine dozens of built-in scraping modules (Net::HTTP, HTML::ArticleExtractor, search engines, social networks) into linked workflows using the visual Query Builder. This lets you automate 90% of typical SEO and analytics tasks without writing a single line of code.
• Low-Code / TypeScript Level (Customizability):
  When standard capabilities aren't enough, write custom scrapers in JavaScript or TypeScript. You do not need to run Redis, configure worker pools, or write proxy-rotation logic — A-Parser handles the underlying network infrastructure and multi-threading, letting developers focus on business logic.

Whether you are building historical datasets for LLMs and RAG systems, restoring lost site content, or conducting large-scale competitor audits — A-Parser provides production-grade speed and reliability right out of the box.

---

Recommended Stack for Getting Started:

• A-Parser Pro / Enterprise
  Tool for parallel data collection and content extraction. Running a custom JavaScript/TypeScript scraper requires a license with support for custom scrapers; current terms are listed on the purchase page.
• A-Parser Unlimited Proxies
  Proxies help you load hundreds of archived pages quickly and work around rate limits.

---

Useful Links and Resources:

• Official A-Parser Website
• Buy A-Parser License (Pro / Enterprise)
• Documentation: JavaScript / TypeScript Scraper Development
• HTML::ArticleExtractor Documentation
• Net::HTTP Scraper Documentation