跳转到主要内容

Cloudflare::Radar - 来自 Cloudflare Radar 的域名公开数据

img

Cloudflare Radar 爬虫工具概览​

Cloudflare::Radar 爬虫工具从 Cloudflare Radar Web 界面获取有关域名的公开信息:域名卡片、WHOIS、DNS 记录、后量子 TLS 支持、访客国家分布、URL 扫描表和排名历史。

得益于内置强大的 Template Toolkit 模板引擎,您可以按需保存所需格式和结构的结果,该引擎允许对结果应用额外逻辑并以各种格式输出数据,包括 JSON、SQL 和 CSV。

采集数据​

  • $asn — ASN 编号
  • $asn_name — 自治系统名称
  • $asn_country — ASN 国家
  • $dnssec — DNSSEC
  • $created_date, $updated_date, $expiration_date — WHOIS 日期
  • $registrar — 注册商
  • $pq — 后量子 TLS 支持
  • $kexName — 密钥交换名称 (KEX)
  • $categories 数组:
    • $categories.$i.name — 网站类别名称
  • $nameservers 数组:
    • $nameservers.$i.name — NS 服务器
  • $status 数组:
    • $status.$i.status — WHOIS 中的域名状态
  • $records 数组:
    • $records.$i.type — DNS 记录类型
    • $records.$i.name — 名称
    • $records.$i.data — 值 (RDATA)
  • $geo 数组(访客分布):
    • $geo.$i.code — 国家代码
    • $geo.$i.value — 来自 Radar 的数值
  • $scans 数组(域名的 URL 扫描):
    • $scans.$i.uuid — UUID
    • $scans.$i.url — URL
    • $scans.$i.time — 时间
    • $scans.$i.isMalicious — 恶意标记
  • $rankHistory 数组:
    • $rankHistory.$i.date — 日期
    • $rankHistory.$i.rank — 排名

应用场景​

  • 按域名确定网站类别
  • ASN 和 DNSSEC 摘要
  • WHOIS:日期、注册商、NS、状态
  • DNS 记录列表
  • 检查后量子 TLS 和 KEX
  • 访客地理分布、URL 扫描、域名排名历史

查询​

查询时需要提供域名列表,例如:

a-parser.com  
yandex.ru
google.com
vk.com
facebook.com
youtube.com

结果输出示例​

A-Parser 支持灵活的结果格式化,这得益于内置的 Template Toolkit 模板引擎,使其能够以任意形式以及结构化形式(如 CSV 或 JSON)输出结果。

默认输出​

结果格式:

$query: $categories.format('$name, ')\n

包含类别及其描述的结果示例:

a-parser.com: Business, Business & Economy, 
yandex.ru: News & Media, Entertainment,
vk.com: Social Networks, Society & Lifestyle,
youtube.com: Video Streaming, Entertainment,
facebook.com: Social Networks, Society & Lifestyle,
google.com: Search Engines, Technology,

输出到 CSV 表格​

结果格式:

[% FOREACH categories;
tools.CSVline(name, desc);
END %]

结果示例:

Business,"Sites related to business."
"Business & Economy","Sites that are related to business, economy, finance, education, science and technology."
"Social Networks","Sites that facilitate interaction and networking between people."
"Society & Lifestyle","Sites related to lifestyle that are not included in other categories like fashion, food & drink etc."
"Social Networks","Sites that facilitate interaction and networking between people."
"Society & Lifestyle","Sites related to lifestyle that are not included in other categories like fashion, food & drink etc."
"Search Engines","Sites that allow users to search for content using keywords."
Technology,"Sites related to technology that are not included in the science category."
"News & Media","Sites related to news and media."
Entertainment,"Sites related to entertainment that are not includeded in other categories like Comic books, Audio streaming, Video streaming etc."

将结果转储为 JSON​

通用结果格式:

[% IF notFirst;
",\n";
ELSE;
notFirst = 1;
END;

obj = {};
obj.query = query;
obj.categories = [];

FOREACH item IN p1.categories;
obj.categories.push({
name = item.name
desc = item.desc
});
END;

obj.json %]

起始文本:

[

结束文本:

]

结果示例:

[{"query":"yandex.ru","categories":[{"desc":"Sites related to news and media.","name":"News & Media"},{"desc":"Sites related to entertainment that are not includeded in other categories like Comic books, Audio streaming, Video streaming etc.","name":"Entertainment"}]},{"query":"google.com","categories":[{"desc":"Sites that allow users to search for content using keywords.","name":"Search Engines"},{"desc":"Sites related to technology that are not included in the science category.","name":"Technology"}]},{"query":"a-parser.com","categories":[{"desc":"Sites related to business.","name":"Business"},{"desc":"Sites that are related to business, economy, finance, education, science and technology.","name":"Business & Economy"}]}]
提示

要在任务编辑器中使用“Prepend text”和“Append text”选项,需要激活“More options”。

可选设置​

参数名称默认值描述
Bypass CloudFlare☑自动绕过 CloudFlare 验证
Bypass CloudFlare with Chrome Max Pages10通过 Chrome 绕过 CF 时的最大页面数
Bypass CloudFlare with Chrome Headless☑如果启用此选项,通过 Chrome 绕过 CF 时将不显示浏览器界面
Use session☑保存良好的会话,从而能够以更少的错误实现更快速的数据抓取。