1. Home
  2. Data
  3. Wayback Machine Scraper
  4. Guide
  5. How to scrape articles from the web archive...

How to scrape articles from the web archive (Wayback Machine)

4 min readWayback Machine Scraper

To get the text of an archived article you need two things: the list of its snapshots, and the page behind each one. The Wayback Machine gives you both for free, the list through its CDX API and the page through the snapshot address, but joining them for many pages takes a script or a tool.

Not affiliated with the Internet Archive.

1. By hand on the Wayback Machine website

  1. Go to web.archive.org, enter the article's address and pick a date on the calendar. Blue dots are captures that loaded normally.
  2. Select the text and copy it.
  3. To see which addresses of a site were archived, open https://web.archive.org/*/example.com/*.

Where it stops: one page at a time, menus and footers included.

2. The CDX API: the list of snapshots

The CDX server is the Wayback Machine's index. It is free and needs no key. Open this in a browser:

https://web.archive.org/cdx/search/cdx?url=archive.org/about/&from=2020&to=2021&filter=statuscode:200&filter=mimetype:text/html&collapse=digest&output=json&limit=5
  • url and matchType: this example leaves matchType out, which means exact: one address. prefix is a folder, host is one host, domain adds its subdomains.
  • from and to: dates of 1 to 14 digits (yyyyMMddhhmmss), both inclusive.
  • filter: a field and a pattern. Here: status 200 only, HTML only.
  • collapse=digest: drops a capture whose content matches the capture before it.

It returns one row per capture with seven columns: urlkey, timestamp, original, mimetype, statuscode, digest, length. There is no page text in the list.

3. The page as archived: the id_ address

A snapshot's address is https://web.archive.org/web/<timestamp>/<original>. Put id_ straight after the timestamp and the Wayback Machine returns the page as it was captured, without its toolbar and without rewritten links. The Archive's API pages do not document this form. Its forum does.

https://web.archive.org/web/20150101013824id_/http://www.example.com/

That is HTML. You still have to strip the tags.

4. The Availability API: is there a copy near this date?

https://archive.org/wayback/available?url=example.com&timestamp=20060101 answers in JSON with the closest snapshot: its address, timestamp and status. One snapshot per question: good for checking dead links, not for a site's history.

Where the free route stops for a non-programmer: someone has to write the loop that reads the list, fetches each page and strips the HTML. The Internet Archive also blocks clients that ask too fast with a 429 "too many requests" error.

5. An Apify actor that does the loop

Internet Archive & Wayback Machine Scraper is an actor we make (details on this site). It reads the CDX list, fetches each id_ page and turns the HTML into plain text.

  1. Open the actor on Apify and sign in.
  2. Mode (mode): choose "Wayback Machine snapshots". The default is archive.org search.
  3. URLs or domains (urls): the article's address, or a site such as example.com.
  4. Match type (matchType): exact for one page, prefix for a folder such as example.com/blog/, host for every page on the host, domain to add subdomains.
  5. HTTP status filter (statusFilter): 200. MIME type filter (mimeFilter): text/html.
  6. Collapse (collapse): "Content changes only", so unchanged copies are not downloaded again. Narrow the period with From date (from) and To date (to), as YYYY, YYYY-MM or YYYY-MM-DD.
  7. Max snapshots per URL (maxSnapshotsPerUrl): 100 by default, up to 100,000. It counts per line you entered, so for a whole site it caps the whole site.
  8. Tick Download page text (fetchContent). Max text characters (maxContentChars) cuts each text at 20,000 characters by default. 0 means no cut.
  9. Start the run, then export the rows as CSV, Excel or JSON.

One row holds original_url, captured_at, status_code, mimetype, digest, snapshot_url and raw_snapshot_url, plus page_title, text, text_length and content_error when the text is on.

Cost: USD 0.50 per 1,000 rows on Apify's free plan, plus Apify's small start fee for each run. The text does not change the price per row. Apify's free plan includes USD 5 of credit a month (apify.com/pricing, read 21 November 2026).

Limits

  • It reads what archive.org holds. A page that was never archived, or that the Archive excludes, returns no rows.
  • The text is plain text taken from the archived HTML. It includes menu and footer words, not the article alone. Scripts are not run, so a page built by JavaScript can come back nearly empty. PDFs and images are listed, not read.
  • There is no "one row per page" setting. A page that changed five times gives five rows.
  • With the text on, each snapshot takes about 1.5 seconds. Archive.org can be slow or refuse requests. The actor waits and retries. A page that still fails gets a reason in content_error.
  • The text belongs to its authors. Respect copyright, the original site's terms and the Internet Archive's terms.

Questions

Can it recover a page that was deleted? Only if the Wayback Machine captured it before it went. Check the calendar or the Availability API first.

Do I need an API key? Not for archive.org. You need an Apify account to run the actor.

Open the actor on Apify

Details and price


More guides