Skip to main content

SE::Wikipedia - Wikipedia search scraper

Overview of the scraper​

Wikipedia search results scraper. It allows searching for encyclopedia articles by keywords via the official MediaWiki Action API and obtaining structured results: links, titles, snippets, short descriptions, image previews, page IDs, and article sizes.

All Wikipedia language editions are supported (en, ru, de, ja, simple, be-tarask, etc.). Up to 20 results per page (MediaWiki extracts API limit), pagination — up to 50 pages. Duplicates between pages are automatically filtered out.

A-Parser functionality allows saving scraping settings for future use (presets), setting up scraping schedules, and much more. Saving results is possible in any form and structure you need, thanks to the built-in powerful Template Toolkit.

To get the full text of an article, use the SE::Wikipedia::Article scraper.

Collected data​

Data is collected via the API https://{lang}.wikipedia.org/w/api.php (generator=search).

  • Total number of results (totalcount)
  • Article link
  • Title (anchor)
  • Snippet (plain text extract)
  • Short description (description)
  • Image preview (if available)
  • Page ID
  • Article size (bytes)

Capabilities​

  • Search across any Wikipedia language edition (list of editions)
  • Snippet length configuration (up to 1200 characters — MediaWiki TextExtracts maximum)
  • Pagination up to 50 pages × 20 results
  • Automatic article deduplication between search result pages
  • Support for proxies and general A-Parser settings

Use cases​

  • Collecting databases of Wikipedia article links
  • Selecting relevant articles by keywords for content and SEO
  • Obtaining snippets and descriptions for chatbots and reference systems
  • Further article parsing via SE::Wikipedia::Article

Queries​

Search phrases should be specified as queries, for example:

Albert Einstein
Ivan Yefremov
Berlin
東京
Kyiv

The search language is set in the scraper settings (Wikipedia language), not in the query.

Query substitutions​

You can use built-in macros to multiply queries.

Output results examples​

A-Parser supports flexible result formatting thanks to the built-in Template Toolkit templating engine, which allows it to output results in any form, as well as in structured formats like CSV or JSON.

Link list (default)​

Result format:

$serp.format('$link\n')

Example result:

https://en.wikipedia.org/wiki/Albert_Einstein
https://en.wikipedia.org/wiki/Hans_Albert_Einstein
https://en.wikipedia.org/wiki/Religious_and_philosophical_views_of_Albert_Einstein
https://en.wikipedia.org/wiki/Albert_Brooks
https://en.wikipedia.org/wiki/Einstein_family

Result format:

$serp.format('$link\t$anchor\t$snippet\n')

Example result:

https://en.wikipedia.org/wiki/Albert_Einstein   Albert Einstein Albert Einstein (14 March 1879 – 18 April 1955) was a German-born theoretical physicist best known for developing the theory of relativity. Einstein also made important contributions to quantum theory. His mass–energy equivalence formula E = mc2, which...
https://en.wikipedia.org/wiki/Hans_Albert_Einstein  Hans Albert Einstein    Hans Albert Einstein (May 14, 1904 – July 26, 1973) was a Swiss-American engineer, the second child and first son of physicists Albert Einstein and Mileva Marić. He was a professor of hydraulic engineering at the University of California, Berkeley from...
https://en.wikipedia.org/wiki/Religious_and_philosophical_views_of_Albert_Einstein  Religious and philosophical views of Albert Einstein    Albert Einstein's religious views have been widely studied and often misunderstood. Albert Einstein stated "I believe in Spinoza's God". He did not believe in a personal God who concerns himself with fates and actions of human beings, a view which he...

CSV output​

Result format:

[% FOREACH item IN p1.serp; tools.CSVline(item.link, item.anchor, item.snippet, item.description, item.image, item.pageid, item.size); END %]

Initial text:

Link,Title,Snippet,Description,Image,Page ID,Size

Example result:

https://en.wikipedia.org/wiki/Albert_Einstein,"Albert Einstein","Albert Einstein (14 March 1879 – 18 April 1955) was a German-born theoretical physicist best known for developing the theory of relativity. Einstein also made important contributions to quantum theory. His mass–energy equivalence formula E = mc2, which...","German-born theoretical physicist (1879–1955)",https://upload.wikimedia.org/wikipedia/commons/thumb/2/28/Albert_Einstein_Head_cleaned.jpg/330px-Albert_Einstein_Head_cleaned.jpg?utm_source=en.wikipedia.org&utm_campaign=api&utm_content=thumbnail,736,238698
https://en.wikipedia.org/wiki/Hans_Albert_Einstein,"Hans Albert Einstein","Hans Albert Einstein (May 14, 1904 – July 26, 1973) was a Swiss-American engineer, the second child and first son of physicists Albert Einstein and Mileva Marić. He was a professor of hydraulic engineering at the University of California, Berkeley from...","Swiss-American engineer and educator",https://upload.wikimedia.org/wikipedia/commons/c/c9/Hans-Albert-Einstein-1953.jpg?utm_source=en.wikipedia.org&utm_campaign=api&utm_content=thumbnail_unscaled,1373258,14371

JSON dump​

Result format:

[% FOREACH item IN p1.serp;
    '{"link":"' _ item.link _ '","anchor":"' _ item.anchor _ '","snippet":"' _ item.snippet _ '"}\n';
END %]

Example result:

{"link":"https://en.wikipedia.org/wiki/Albert_Einstein","anchor":"Albert Einstein","snippet":"Albert Einstein (14 March 1879 – 18 April 1955) was a German-born theoretical physicist best known for developing the theory of relativity. Einstein also made important contributions to quantum theory. His mass–energy equivalence formula E = mc2, which..."}
{"link":"https://en.wikipedia.org/wiki/Hans_Albert_Einstein","anchor":"Hans Albert Einstein","snippet":"Hans Albert Einstein (May 14, 1904 – July 26, 1973) was a Swiss-American engineer, the second child and first son of physicists Albert Einstein and Mileva Marić. He was a professor of hydraulic engineering at the University of California, Berkeley from..."}

Possible settings​

note

General settings for all scrapers

Parameter nameDefault valueDescription
Pages count5Number of search pages (from 1 to 50). Up to 20 results per page
Wikipedia languageenWikipedia language edition (en, ru, de, ja, simple, be-tarask, etc.). List of editions
Snippet length250Snippet length in characters (150–1200). API TextExtracts maximum is 1200
tip

The Wikipedia API may return overlapping results during deep pagination. The scraper removes duplicates by pageid and logs how many were filtered out.