跳转到主要内容

SE::Wikipedia - Wikipedia 搜索爬虫工具

爬虫工具概览​

Wikipedia 搜索结果爬虫工具。允许通过官方 MediaWiki Action API 按关键词搜索百科全书文章，并获取结构化结果：链接、标题、摘要、简短描述、图像预览、页面 ID 和文章大小。

支持所有语言版本的 Wikipedia（en, ru, de, ja, simple, be-tarask 等）。每个结果页最多包含 20 条结果（MediaWiki API extracts 限制），翻页最多支持 50 页。页面间的重复内容会自动过滤。

A-Parser 的功能允许保存数据抓取设置以供后续使用（预设），设置数据抓取计划等等。得益于内置强大的模板引擎 Template Toolkit，可以将结果保存为您需要的任何形式和结构。

如需获取文章全文，请使用 SE::Wikipedia::Article 爬虫工具。

采集数据​

数据通过 API https://{lang}.wikipedia.org/w/api.php (generator=search) 进行采集。

  • 结果总数 (totalcount)
  • 文章链接
  • 标题 (anchor)
  • 摘要 (plain text extract)
  • 简短描述 (description)
  • 图像预览（如果有）
  • Page ID
  • 文章大小（字节）

功能​

  • 搜索任何语言版本的 Wikipedia（版本列表）
  • 设置摘要长度（最高 1200 字符 —— MediaWiki TextExtracts 的最大值）
  • 翻页支持高达 50 页 × 20 条结果
  • 自动对搜索结果页之间的文章进行去重
  • 支持代理和 A-Parser 的通用设置

应用场景​

  • 采集 Wikipedia 文章链接库
  • 为内容创作和 SEO 根据关键词筛选相关文章
  • 为聊天机器人和参考系统获取摘要和描述
  • 通过 SE::Wikipedia::Article 进一步解析文章

查询​

需要指定搜索短语作为查询，例如：

Albert Einstein
Ivan Efremov
Berlin
東京
Kyiv

搜索语言在爬虫工具设置中指定（Wikipedia language），而不是在查询中指定。

查询占位符​

您可以使用内置宏来批量生成查询。

结果输出示例​

A-Parser 支持灵活的结果格式化，这得益于内置的模板引擎 Template Toolkit，使其能够以任意形式以及结构化形式（如 CSV 或 JSON）输出结果。

链接列表（默认）​

结果格式：

$serp.format('$link\n')

结果示例：

https://en.wikipedia.org/wiki/Albert_Einstein
https://en.wikipedia.org/wiki/Hans_Albert_Einstein
https://en.wikipedia.org/wiki/Religious_and_philosophical_views_of_Albert_Einstein
https://en.wikipedia.org/wiki/Albert_Brooks
https://en.wikipedia.org/wiki/Einstein_family

结果格式：

$serp.format('$link\t$anchor\t$snippet\n')

结果示例：

https://en.wikipedia.org/wiki/Albert_Einstein   Albert Einstein Albert Einstein (14 March 1879 – 18 April 1955) was a German-born theoretical physicist best known for developing the theory of relativity. Einstein also made important contributions to quantum theory. His mass–energy equivalence formula E = mc2, which...
https://en.wikipedia.org/wiki/Hans_Albert_Einstein  Hans Albert Einstein    Hans Albert Einstein (May 14, 1904 – July 26, 1973) was a Swiss-American engineer, the second child and first son of physicists Albert Einstein and Mileva Marić. He was a professor of hydraulic engineering at the University of California, Berkeley from...
https://en.wikipedia.org/wiki/Religious_and_philosophical_views_of_Albert_Einstein  Religious and philosophical views of Albert Einstein    Albert Einstein's religious views have been widely studied and often misunderstood. Albert Einstein stated "I believe in Spinoza's God". He did not believe in a personal God who concerns himself with fates and actions of human beings, a view which he...

输出为 CSV​

结果格式：

[% FOREACH item IN p1.serp; tools.CSVline(item.link, item.anchor, item.snippet, item.description, item.image, item.pageid, item.size); END %]

起始文本：

链接,标题,摘要,描述,图像,Page ID,大小

结果示例：

https://en.wikipedia.org/wiki/Albert_Einstein,"Albert Einstein","Albert Einstein (14 March 1879 – 18 April 1955) was a German-born theoretical physicist best known for developing the theory of relativity. Einstein also made important contributions to quantum theory. His mass–energy equivalence formula E = mc2, which...","German-born theoretical physicist (1879–1955)",https://upload.wikimedia.org/wikipedia/commons/thumb/2/28/Albert_Einstein_Head_cleaned.jpg/330px-Albert_Einstein_Head_cleaned.jpg?utm_source=en.wikipedia.org&utm_campaign=api&utm_content=thumbnail,736,238698
https://en.wikipedia.org/wiki/Hans_Albert_Einstein,"Hans Albert Einstein","Hans Albert Einstein (May 14, 1904 – July 26, 1973) was a Swiss-American engineer, the second child and first son of physicists Albert Einstein and Mileva Marić. He was a professor of hydraulic engineering at the University of California, Berkeley from...","Swiss-American engineer and educator",https://upload.wikimedia.org/wikipedia/commons/c/c9/Hans-Albert-Einstein-1953.jpg?utm_source=en.wikipedia.org&utm_campaign=api&utm_content=thumbnail_unscaled,1373258,14371

导出为 JSON​

结果格式：

[% FOREACH item IN p1.serp;
    '{"link":"' _ item.link _ '","anchor":"' _ item.anchor _ '","snippet":"' _ item.snippet _ '"}\n';
END %]

结果示例：

{"link":"https://en.wikipedia.org/wiki/Albert_Einstein","anchor":"Albert Einstein","snippet":"Albert Einstein (14 March 1879 – 18 April 1955) was a German-born theoretical physicist best known for developing the theory of relativity. Einstein also made important contributions to quantum theory. His mass–energy equivalence formula E = mc2, which..."}
{"link":"https://en.wikipedia.org/wiki/Hans_Albert_Einstein","anchor":"Hans Albert Einstein","snippet":"Hans Albert Einstein (May 14, 1904 – July 26, 1973) was a Swiss-American engineer, the second child and first son of physicists Albert Einstein and Mileva Marić. He was a professor of hydraulic engineering at the University of California, Berkeley from..."}

可用设置​

备注

所有爬虫工具的通用设置

参数名称默认值描述
Pages count5搜索页数（1 到 50）。每页最多 20 条结果
Wikipedia languageenWikipedia 语言版本（en, ru, de, ja, simple, be-tarask 等）。版本列表
Snippet length250摘要长度（字符数，150–1200）。API TextExtracts 最大值为 1200
提示

Wikipedia API 在深翻页时可能会返回重叠的结果。爬虫工具会根据 pageid 删除重复项，并在日志中记录筛选掉的数量。