Skip to main content

SE::Wikipedia::Article - Wikipedia article scraper

Overview of the scraper​

Wikipedia article scraper. The input is a link to an article (or a short format lang:title), and the output is the full text of the article in plain text, a short description, canonical link, thumbnail and original image, categories, language versions, Wikidata ID, and revision metadata.

Data is retrieved via the official MediaWiki Action API (prop=extracts|info|pageimages|...). The language is determined from the URL or the query prefix. Wikipedia redirects are handled automatically (e.g., USA → United States).

To search for articles by keywords, use the SE::Wikipedia scraper.

A-Parser functionality allows you to save scraping settings for future use (presets), set a scraping schedule, and much more. Saving results is possible in any form and structure you need, thanks to the built-in powerful Template Toolkit.

Collected data​

Data is collected via the API https://{lang}.wikipedia.org/w/api.php.

  • Article title (title)
  • Short description (description)
  • Full article text in plain text (text)
  • Canonical link (link)
  • Image preview (thumb)
  • Original image (image)
  • Page ID (pageid)
  • Article size in bytes (size)
  • Edition language (lang)
  • Wikidata ID (wikidata)
  • Last revision time (timestamp)
  • Revision ID (revid)
  • List of categories (categories)
  • Language versions of the article (langlinks: lang, title, link)
  • Raw API response JSON (data)

Capabilities​

  • Parsing of desktop, mobile, and index.php Wikipedia links
  • Short query format lang:title
  • Automatic redirect following
  • Full plain text of the article (without HTML/Markdown)
  • Categories and language links with API limit max (up to 500)
  • Support for proxies and general A-Parser settings

Use Cases​

  • Exporting article texts for NLP, RAG, and knowledge bases
  • Collecting descriptions and images for cards and bots
  • Obtaining categories and language versions of a page
  • Enriching SE::Wikipedia results with full article text

Queries​

Supported query formats:

  1. Standard link:
https://en.wikipedia.org/wiki/Albert_Einstein
https://ru.wikipedia.org/wiki/Ефремов,_Иван_Антонович
  1. Mobile link:
https://en.m.wikipedia.org/wiki/Albert_Einstein
  1. Link of type index.php:
https://en.wikipedia.org/w/index.php?title=Albert_Einstein
  1. With section anchor (#... is ignored):
https://en.wikipedia.org/wiki/Albert_Einstein#Scientific_career
  1. Short format lang:title:
en:Albert Einstein
en:Efremov, Ivan Antonovich
eo:Esperanto

URL-encoding, _ in title, redirects, and language codes with hyphens (be-tarask) are also correctly processed.

Output results examples​

A-Parser supports flexible result formatting thanks to the built-in Template Toolkit, which allows it to output results in any form, as well as in structured formats like CSV or JSON.

Default output​

Result format:

$link\n$title\n$description\n$text\n

Example result:

https://en.wikipedia.org/wiki/Albert_Einstein
Albert Einstein
German-born theoretical physicist (1879–1955)
Albert Einstein (14 March 1879 – 18 April 1955) was a German-born theoretical physicist best known for developing the theory of relativity. Einstein also made important contributions to quantum theory. His mass–energy equivalence formula E = mc2, which arises from special relativity, has been called "the world's most famous equation". He received the 1921 Nobel Prize in Physics for "his services to theoretical physics, and especially for his discovery of the law of the photoelectric effect".
Born as a subject to the Kingdom of Württemberg, part of the German Empire, Einstein moved to Switzerland in 1895, forsaking his citizenship the following year. In 1896, at the age of seventeen, he enrolled in the mathematics and physics teaching diploma program at the Swiss federal polytechnic school in Zurich, graduating in 1900. He acquired Swiss citizenship a year later, which he kept for the rest of his life, and afterwards secured a permanent position at the Swiss Patent Office in Bern. In 1905, he submitted a successful PhD dissertation to the University of Zurich. In 1914, he moved to Berlin to join the Prussian Academy of Sciences and the Humboldt University of Berlin, becoming director of the Kaiser Wilhelm Institute for Physics in 1917; he also became a Prussian and consequently also German citizen again. In 1933, while Einstein was visiting the United States, Adolf Hitler came to power in Germany. Horrified by the Nazi persecution of his fellow Jews, he decided to remain in the US, and was granted American citizenship in 1940. On the eve of World War II, he endorsed a letter to President Franklin D. Roosevelt alerting him to the potential German nuclear weapons program and recommending that the US begin similar research, later carried out as the Manhattan Project.
...

Result format:

$title\t$description\t$link\n

Example result:

Albert Einstein German-born theoretical physicist (1879–1955)   https://en.wikipedia.org/wiki/Albert_Einstein

CSV output​

Result format:

[% tools.CSVline(title, description, link, pageid, wikidata, lang, size) %]

Initial text:

Title,Description,Link,Page ID,Wikidata,Language,Size

Example result:

"Albert Einstein","German-born theoretical physicist (1879–1955)",https://en.wikipedia.org/wiki/Albert_Einstein,736,Q937,en,238698

Categories​

Result format:

$categories.format('$name\n')

Example result:

1879 births
1955 deaths
19th-century American Jews
19th-century German Jews
19th-century Swiss Jews
20th-century American engineers
20th-century American inventors
20th-century American letter writers
20th-century American male writers
20th-century American physicists

Result format:

$langlinks.format('$lang\t$title\t$link\n')

Example result:

ab  Einstein, Albert    https://ab.wikipedia.org/wiki/%D0%95%D0%B8%D0%BD%D1%88%D1%82%D0%B5%D0%B8%D0%BD,_%D0%90%D0%BB%D0%B1%D0%B5%D1%80%D1%82
af  Albert Einstein https://af.wikipedia.org/wiki/Albert_Einstein
gsw Albert Einstein https://als.wikipedia.org/wiki/Albert_Einstein
am  አልበርት አይንስታይን   https://am.wikipedia.org/wiki/%E1%8A%A0%E1%88%8D%E1%89%A0%E1%88%AD%E1%89%B5_%E1%8A%A0%E1%8B%AD%E1%8A%95%E1%88%B5%E1%89%B3%E1%8B%AD%E1%8A%95
de  Albert Einstein https://de.wikipedia.org/wiki/Albert_Einstein
ru  Einstein, Albert    https://ru.wikipedia.org/wiki/%D0%AD%D0%B9%D0%BD%D1%88%D1%82%D0%B5%D0%B9%D0%BD,_%D0%90%D0%BB%D1%8C%D0%B1%D0%B5%D1%80%D1%82

JSON dump​

Result format:

[%
obj = {};
obj.title = title;
obj.description = description;
obj.link = link;
obj.pageid = pageid;
obj.wikidata = wikidata;
obj.lang = lang;
obj.json _ "\n"
%]

Example result:

{"wikidata":"Q937","link":"https://en.wikipedia.org/wiki/Albert_Einstein","lang":"en","pageid":"736","title":"Albert Einstein","description":"German-born theoretical physicist (1879–1955)"}

Possible settings​

note

General settings for all scrapers

The scraper has no unique options in the UI: the language and title are taken from the query. Only general settings (proxies, retries, etc.) are used.