Пресет Определяем CMS для 1000000 доменов за 15 часов 1.0

странно, у меня такая же проблема, и добавления днс серваков не помогло (добавил штук 10 топовых)
конфиг похожий, правда апарсер стоит на дебиане, а не винде
по скоростям совсем всё плохо
500 потоков - 4000
1000 потоков - 5500
1500 потоков - 4500 (даже медленее работает)

ретрайс 1, таймаут 5 сек, без проксей, чекаю простенькой регулярой
канал, оперативка, проц, всё свободно.. походу буду запускать несколько копий, обидно что придётся разбивать задания и потом соеденять заноово результаты

 

получилось, скорость на каждом апарсере выходит по 4500, т.е. в сумме 9000, можно ещё 3ью запустить, но это совсем неудобно..
кстати я понял проблему, там хоть проц и свободный, одно ядро всегда забито на 100%, оно видимо основное и отвечает как раз за переключение потоков?

у меня есть ещё другой сервер, где более мощный проц, так там тоже задание упирается уже в скорость 10к, повышение потоков не помогает и опять таки, всё потому что 1 ядро забито на максимум

как я понимаю, в 64х битной версии этой проблемы уже не будет?

 

Ребят, какие сервера днс прописываете для увеличения скорости?

 

Wappalyzer обновил базу CMS до 202 штук.

 

Еще есть одна особенность, если скажем у сайта будет 301 редирект на папку то ничего не будет.
http://domain.com 301 редирект или meta-refrash http://domain.com/blog или http://domain.edu
То не будет следования правилам редиректа и определения.

 

Имхо пока лучше сервиса не нашёл по определению CMS

Код:

https://toolbar.netcraft.com/site_report?url=https://a-parser.com/

 

Можно обновить Wappalyzer?
Wappalyzer recognises 1,238 technologies in 65 categories

 

Обновленный файл всегда можно найти на github

 

Не знаю с чем может быть связано, ну сам плагин Wappalyzer определяет на много точнее, чем a-parser. (файл js обновлял самым новым).
Бывает смотришь результаты чека с а-парсера на определенный двиг с пометкой unknown, а встроенный плагин в браузер показывает что это wordpress!
Потом опять, суешь домены в а-парсер, прогоняешь и на выходе unknown

Если нужно отчекать домены на определенный движок, то лучше самому составить регулярку для поиска, имхо, получится самый лучший результат!

И где там можно увидеть CMS? Ты ничего не путаешь?

 

у меня такие чудеса только когда сервис под CF и прокси там забанены.

 

Насчет этого знаю, еще некоторые UA могут давать такой результат, по этому лучше использовать реальный UA.
Самое интересное, что эти же домены отлично парсятся Net::HTTP.

 

Как по мне признаки/методы используемые в а-парсере для определения цмс не достаточные.
Возьмем на примере smf.

Спойлер: a-parser

Код:

    "SMF": {
      "cats": [
        2
      ],
      "html": "credits/?\" title=\"Simple Machines Forum\" target=\"_blank\" class=\"new_win\">SMF ([0-9.]+)</a>\\;version:\\1",
      "icon": "SMF.png",
      "implies": "PHP",
      "js": {
        "smf_": ""
      },
      "website": "http://www.simplemachines.org"
    },

Спойлер: GSA Platform Identifier

Код:

[setup]
enabled=1
default checked=0

engine type=Forum
description=Registers on this forum and leaves a URL on your profile page.
dofollow=1
anchor text=1
creates own page=1
uses pages=0

page must have1=Powered by SMF|powered by Simple Machines|>Simple Machines</a>|"http://www.simplemachines.org/about/smf/license.php"|var smf_theme_url = "|var smf_charset = "

;options to search for thies sites....

search term="Powered by SMF"|"powered by Simple Machines"|inurl:"/index.php?action=register"|"Powered by SMF" inurl:"register.php"|"Please note that with each post, your IP address is recorded"|"Listen to the letters / Request another image"|"Most users online today" "Most users online ever"

add keyword to search=2

;if enabled we will extract keywords from meta-keywords and later try to find new targets
extract keywords=0

posted domain check=1

;-------------------------------------------------------------------------
;the variables we have to define for this engine (if not already defined before)
;name, keyword, url, e-mail are already defined by default


[Website_title]
type=text
allow html=0
must be filled=1
hint=Thats usually a anchor text, so please make sure it holds an important keyword.
max length=80
min length=4
replace=,; -

[Your E-Mail]
type=email
static=1


[URL]
type=url

[Anchor_text]
type=text
alternate data=%spinfile-generic_anchor_text.dat%


[About_Yourself]
type=memo
allow html=1
html to bbs=1
allow return=1
must be filled=0
hint=Please fill this with some information about yourself. Use the spin syntax heavily here.
custom mode=1
auto modify=1
default=%file-about_yourself.dat%
auto add anchor url=1
bbs supports=b,i,u,s,pre,left,right,center,font,size,color,img,url,email,ftp,glow,shadow,move,sup,table,li,hr,list,

[Forum_nick]
type=login
must be filled=1
hint=The login for the forums. Use numbers and letters only.
min length=4
static=1

[Forum_password]
type=password
must be filled=1
hint=A password used to log into the forum. Use
min length=8
static=1

[Forum_Subject]
type=text
allow html=0
must be filled=1
default={Just {want|wanted} to say {Hello|Hi}{!|.}|I am the new {guy|girl|one}|Im {happy|glad} I {finally|now} {registered|signed up}}
hint=A topic of your new thread.

[Forum_Comment]
type=memo
allow html=1
html to bbs=1
must be filled=1
hint=Use spin syntax here to not get duplicate content.
default=%file-comments_forum.dat%
allow return=1
custom mode=1
auto modify=0
html line break=0
auto add anchor url=2
bbs supports=b,i,u,s,pre,left,right,center,font,size,color,img,url,email,ftp,glow,shadow,move,sup,table,li,hr,list,

[Twitter_URL]
type=text
hint=Enter the Twitter URL you might use for the projects Website. Leave this empty if you don't have one.
must be filled=0
allow html=0

[Facebook_URL]
type=text
hint=Enter the Facebook URL you might use for the projects Website. Leave this empty if you don't have one.
must be filled=0
allow html=0
;---------------------------------------------------------------------------

[userid]
type=extract
front=name="userID" value="
back="

[userid2]
type=extract
front1=sa=summary
front2=?action=profile;u=
back1="
back2=;

[userid3]
type=extract
front=area=forumprofile;u=
back1=;
back2='
back3="

[signature]
type=extract
input=%file-auto_anchor.dat%
html to bbs=1
allow html=1
bbs supports=b,i,u,s,pre,left,right,center,font,size,color,img,url,email,ftp,glow,shadow,move,sup,table,li,hr,list,

[forum_post_url]
type=extract
find link=%forum_subject%
find url=*/index.php/topic,[1-9]*.0.html|*/index.php?topic=[1-9]*.0

[REGISTER_STEP1]
page must have1=!Die Registrierung ist momentan deaktiviert.
page must have2=!Sorry, registration is currently disabled.
page must have3=!you are banned from using this forum!
page must have4=!An Error Has Occurred!
page must have5=!registratie is momenteel uitgeschakeld

find link=REGISTER|registrieren
find url=*action=register|*action=register&*
alternative url=./index.php?action=register

form id=creator|registration
form name=creator*|*I*accept*the*terms*of*the*agreement*|*Acepto*los*tйrminos*del*acuerdo*y*soy*mayor*
form url=*action=register2|*action=register

submit success=Merci de vous кtre inscrit.|Gracias por registrarte.|Hvala vam na registriranju.|Thank you for registering|The admin must approve your registration|you will receive an email shortly advising you of the admins decision.|Danke fьr die Registrierung.|Der Administrator muss Deine Registrierung genehmigen|Du wirst in Kьrze eine E-Mail mit der Entscheidung erhalten|You will receive an email shortly with a link to activate your membership|Please check your email, and follow the instruction on the email|Total time logged in:|>Logout<|action=logout|Show unread posts since last visit|>Logout<|action=logout|>log out<|Dieser Benutzername ist schon vergeben|Could not find phrase 'registration_complete'.|We need to confirm your email address.|To complete the subscription process, please click the link in the email we just sent you.|O seu Registro foi efetuado com sucesso|You are now a member of the Forum.|Danke fьr die Anmeldung.|Ce nom est dйjа utilisй par un autre membre.|please click the link in the email we just sent you.|This name is already in use by another member.|is being used by a registered member already.
submit failed=is a Spam, please contact forum administrator.|Sorry, registration is currently disabled.|An avatar is required|is a Spam, please contact forum administrator|An Error Has Occurred!|Bots are not allowed to register.|Ce nom est dЪjУ utilisЪ par un autre membre|Du darfst nicht mehrere Benutzerkonten zur gleichen Zeit fьr den gleichen Computer registrieren|Ein Fehler ist aufgetreten!|Invalid Registration Password|Please confirm that you are human in order to proceed with registration|Please type the correct result|Require a member to select a location"|The letters you typed don't match the letters that were shown in the picture|You did not answer the verification questions correctly|You did not complete the anti-bot clock|You did not complete the anti-bot puzzle|You did not complete the spam-bot puzzle|You did not fill in the|Your email was entered incorrectly|Your password must contain a mix of upper and lower case letters, as well as digits.|The following errors were detected in your registration|Sono stati riscontrati i seguenti errori nella tua registrazione.|<div class="register_error">|Registration cannot continue|Error: please enter a valid email address.|Internal Server Error|Service Temporarily Unavailable|You're a spam bot. Shame on you.|Ocorreu um erro!
submit success skip verify=Total time logged in:|>Logout<|action=logout|Show unread posts since last visit|>log out<|Dieser Benutzername ist schon vergeben|Ce nom est dйjа utilisй par un autre membre.|This name is already in use by another member.|is being used by a registered member already.

set unknown variable=%leave%
set unknown variable condition=name="passwrd2"

verify submission=1
verify by=email
verify interval=30
verify timeout=60
first verify=5
verify on unknown status=1

aim=%leave%
allow_email=1
anti_spam_ver_resp*=%question%
anti_spam_yogolas=%question%
areyouhuman=Yes
assyria=%question%
authenticate=Password|Contrase*|passwd
bday1=%random-1-29%
bday2=%random-1-12%
bday3=%random-1980-1990%
captcha=%captcha%
cat=%random_option%
city=%columnspinfile-address_data.dat-3%
clean=%leave%
company=%leave%
country=%columnspinfile-address_data.dat-1%|%random_option%
cp=%leave%
customfield[areyou]=%question%
customfield[characte]=%leave%
customfield[city]=%columnspinfile-address_data.dat-3%
customfield[country]=%columnspinfile-address_data.dat-1%|%random_option%
customfield[courses]=%leave%
customfield[cust_addres]=%columnspinfile-address_data.dat-5%
customfield[cust_advanc]=%random_option%
customfield[cust_affili]=%leave%
customfield[cust_bbpin]=%leave%
customfield[cust_blackb]=%leave%
customfield[cust_branch]=%leave%
customfield[cust_busine]=%leave%
customfield[cust_campus]=%leave%
customfield[cust_city]=%columnspinfile-address_data.dat-3%
customfield[cust_compan]=%leave%
customfield[cust_company]
customfield[cust_contac]=%spinfile-lnames.dat%
customfield[cust_countr]=%columnspinfile-address_data.dat-1%|%random_option%
customfield[cust_divisi]=%leave%
customfield[cust_electr]=%leave%
customfield[cust_facebo]=%facebook_url%
customfield[cust_family]=%leave%
customfield[cust_family0]=%leave%
customfield[cust_family1]=%leave%
customfield[cust_family2]=%leave%
customfield[cust_first]=%spinfile-names.dat% %spinfile-lnames.dat%
customfield[cust_firstn]=%spinfile-names.dat%
customfield[cust_firstn0]=%spinfile-names.dat%
customfield[cust_flight]=%leave%
customfield[cust_fullna]=%spinfile-names.dat% %spinfile-lnames.dat%
customfield[cust_gamert]=%leave%
customfield[cust_gender]=%random_option%
customfield[cust_graphi]=%leave%
customfield[cust_iamint]=%random_option%
customfield[cust_iypnum]=%leave%
customfield[cust_jabber]=%leave%
customfield[cust_jbb]=%leave%
customfield[cust_klass]=%random_option%
customfield[cust_lastna]=%spinfile-lnames.dat%
customfield[cust_locati]=%leave%
customfield[cust_mainch]=%leave%
customfield[cust_member]=%leave%
customfield[cust_memory]=%leave%
customfield[cust_middle]=%leave%
customfield[cust_minecr]=%leave%
customfield[cust_mobile]=%leave%
customfield[cust_mother]=%leave%
customfield[cust_myspac]=%leave%
customfield[cust_name]=%spinfile-names.dat%
customfield[cust_namn]=%random_option%
customfield[cust_nickna]=%forum_nick%
customfield[cust_number]=%leave%
customfield[cust_occupa]=%leave%
customfield[cust_os]=%leave%
customfield[cust_ossyst]=%leave%
customfield[cust_pa]=%question%
customfield[cust_phone]=%columnspinfile-address_data.dat-6%
customfield[cust_phonen]=%columnspinfile-address_data.dat-6%
customfield[cust_proces]=%leave%
customfield[cust_railwa]=%leave%
customfield[cust_realna]=%spinfile-names.dat% %spinfile-lnames.dat%
customfield[cust_refere]=%leave%
customfield[cust_region]=%leave%
customfield[cust_ride]=%leave%
customfield[cust_skype]=%leave%
customfield[cust_state]=%columnspinfile-address_data.dat-2%
customfield[cust_stateh]=%leave%
customfield[cust_states]=%leave%
customfield[cust_status]=%leave%
customfield[cust_steami]=%leave%
customfield[cust_street]=%columnspinfile-address_data.dat-5%
customfield[cust_suffix]=%leave%
customfield[cust_surnam]=%spinfile-lnames.dat%
customfield[cust_twitte]=%twitter_url%
customfield[cust_union]=%leave%
customfield[cust_unit]=%leave%
customfield[cust_vehicl]=%leave%
customfield[cust_vehicl0]=%leave%
customfield[cust_vehicl1]=%leave%
customfield[cust_year]=%random-1970-1980%
customfield[cust_yearof]=%random-1970-1980%
customfield[cust_yearso]=%leave%
customfield[cust_yourco]=%leave%
customfield[cust_yourna]=%spinfile-names.dat% %spinfile-lnames.dat%
customfield[cust_yourri]=%leave%
customfield[cust_youtub]=%leave%
customfield[cust_zipcod]=%columnspinfile-address_data.dat-4%
customfield[doyouh]=yes
customfield[face_pro]=%facebook_url%
customfield[facebook]=%facebook_url%
customfield[firstna]=%spinfile-names.dat%
customfield[guild]=%leave%
customfield[invited]=%leave%
customfield[lastnam]=%spinfile-lnames.dat%
customfield[location]=%columnspinfile-address_data.dat-3%
customfield[phonenu]=%columnspinfile-address_data.dat-6%
customfield[rank]=%leave%
customfield[skype]=%leave%
customfield[state]=%columnspinfile-address_data.dat-2%
customfield[tel]=%columnspinfile-address_data.dat-6%
customfield[twitter]=%twitter_url%
customfield[year]=%random-1970-1980%
default_options*=%leave%
default_options[linkedin]=%leave%
deviantart=%leave%
email=%your e-mail%
emailRepeat=%your e-mail%
facebook=%facebook_url%
fec_ncto=%random_option%
file=%leave%
first_name=%spinfile-names.dat%
gamer_psn=%leave%
gamer_steam=%leave%
gamer_xbgt=%leave%
gamer_xfire=%leave%
gender=%random_option%
googleplus=%leave%
hide_email=%leave%
hideEmail=1
human=%random_option%
icq=%leave%
id_group=%random_option%
invite=%leave%
join_reason=%leave%
last_name=%spinfile-lnames.dat%
linkedin=%leave%
listas=%random_option%
location=%columnspinfile-address_data.dat-3%
msn=%leave%
myspace=%leave%
name=%spinfile-names.dat% %spinfile-lnames.dat%
ocupacion=%random_option%
openid_identifier=%leave%
openid_url=%leave%
pais=%random_option%
passwrd1=%Forum_password%
passwrd2=%Forum_password%
phone_number=%columnspinfile-address_data.dat-6%
phone=%columnspinfile-address_data.dat-6%
postal_code=%columnspinfile-address_data.dat-4%
postcode=%columnspinfile-address_data.dat-4%
private_key=%captcha%
q1=%leave%
q2=%question%
referredby=%leave%
refferedbyname=%spinfile-names.dat%
regagree=1
register_vv[code]=%captcha% action=verificationcode
register_vv?q?*=%question%
reguser=%forum_login%
rnd_ver_resp*=%question%
rnd_ver_resp_1=%question%
rnd_ver_resp_2=%question%
sector=%random_option%
sexo=%random_option%
skip_coppa=1
skype=%leave%
street_address=%columnspinfile-address_data.dat-5%
swallow=%question%
twitter=%twitter_url%
txtDOB_day=%random_option%
txtDOB_month=%random_option%
txtDOB_year=%random_option%
user=%Forum_nick%
username=%forum_nick%
visual_verification_code=%captcha% action=verificationcode
website_title=%leave%
website_url=%leave%
websiteTitle=%spinfile-names.dat% %spinfile-lnames.dat%
websiteUrl=%url%
yim=%leave%
youtube=%leave%


[REGISTER_STEP2]
form id=registration
form name=*Register
form url=&action=register2
optional form=1

set unknown variable=%leave%
set unknown variable condition=name="passwrd2"
 
[LOGIN_STEP1]
find link=Login
find url=*action=login*
alternative url=./index.php?action=login

form id=frmLogin
form name=frmLogin*
form url=*action=login2*

submit success=>Logout<|action=logout|Show unread posts since last visit|>log out<
submit failed=That username does not exist.|Your email address needs to be validated before you can login|Benutzername nicht vorhanden|<b style="color: red;">|An Error Has Occurred!|Your account is still awaiting admin approval.|<p class="error">

seconds to wait before submission=5

verify on unknown status=1

user=%Forum_nick%
password=%Forum_password%
passwrd=%Forum_password%
cookielength=%leave%
cookieneverexp=0
authenticate=Password|Contrase*|passwd
openid_identifier=%leave%

[STEP1]
link type=Profile-Contextual

find link=Profile
find url=*action=profile*
alternative url=./index.php?action=profile
just download=1

submit success=Your profile has been updated successfully|value="%url%"
submit failed=An Error Has Occurred!|The following BBC is not allowed within your signature|The following errors occurred when trying to save your profile

seconds to wait before submission=5

verify submission=1
verify by=url
verify url=./index.php?action=profile&u=%userid%%userid2%%userid3%
verify interval=1
verify timeout=1
verify on unknown status=1
verify search detail url=0

[STEP2]
find link=*Profile Information|Forum Profile Information|Forum Profile
find url=*action=profile*sa=forumProfile*|*sa=forumProfile*|*area=forumprofile*|*action=profile;area=forumprofile*
alternative url=./index.php?action=profile&sa=forumProfile&u=%userid%%userid2%%userid3%

seconds to wait before submission=5

form name=creator*

variable must be used=url
set unknown variable=%leave%

websiteTitle=%website_title%
websiteUrl=%url%
website_title=%website_title%
website_url=%url%
customfield[aboutme]=%about_yourself%
signature=%signature%
country_flag=%random_option%
customfield[cust_aboutm]=%about_yourself%
personalText=%about_yourself%
msn=%leave%
aim=%leave%
icq=%leave%
yim=%leave%
bday1=%random-10-29%
bday2=%random-10-12%
bday3=%random-1980-1990%
avatar_choice=%leave%
personal_text=%about_yourself%
userpicpersonal=%leave%
attachment=%leave%
customfield[cust_bio]=%about_yourself%
customfield[cust_google]=%leave%
usertitle=%leave%
customfield[cust_2go]=%leave%
customfield[face_pro]=%facebook_url%
gender=%leave%

[EXTRA_STEP1]
link type=Forum Post

submit success=%forum_subject%
submit failed=An Error Has Occurred!|The following BBC is not allowed within your signature|The following errors occurred when trying to save your profile

seconds to wait before submission=5

verify submission=1
verify by=url
verify url=%forum_post_url%
verify interval=1
verify timeout=1
verify on unknown status=1
verify search detail url=0


find link=Home
find url=*index.php
alternative url=./index.php
just download=1

[EXTRA_STEP2]
find url=*/index.php/board,[0-9]*.[0-9]*.html|*/index.php?board=[0-9]*.[0-9]*
find link=*Intro*|*hello*|*welcome*|*off*|*Other*|*General*|*misc*
alternative url=./index.php?board=2.0
just download=1

[EXTRA_STEP3]
find link=New Topic
find url=*/index.php?action=post*
alternative url=./index.php?action=post;board=2.0

form id=postmodify
form name=postmodify*Post|postmodify*|*Post
form url=*/index.php?action=post*|*action=post*

set unknown variable=%leave%
variable must be used=url

subject=%forum_subject%
icon=%random_option%
sel_face=%facebook_url%
sel_size=%leave%
sel_color=%leave%
message=%forum_comment%
notify=0
lock=0
goback=1
ns=0
attachment[]=%leave%

[DELETE_STEP1]
;delete topic
find link=Remove Topic
find url=*action=removetopic*
optional find url link=1
just download=1


[DELETE_STEP2]
;delete profile
find link=*Profile Information|Forum Profile Information|Forum Profile
find url=*action=profile*sa=forumProfile*|*sa=forumProfile*|*area=forumprofile*|*action=profile;area=forumprofile*
alternative url=./index.php?action=profile&sa=forumProfile&u=%userid%%userid2%%userid3%

seconds to wait before submission=5

form name=creator*

set unknown variable=%leave%

websiteTitle=%empty%
websiteUrl=%empty%
website_title=%empty%
website_url=%empty%
customfield[aboutme]=%empty%
signature=%empty%
country_flag=%leave%
customfield[cust_aboutm]=%empty%
personalText=%empty%
msn=%leave%
aim=%leave%
icq=%leave%
yim=%leave%
bday1=%leave%
bday2=%leave%
bday3=%leave%
avatar_choice=%leave%
personal_text=%empty%
userpicpersonal=%leave%
attachment=%leave%
customfield[cust_bio]=%empty%
customfield[cust_google]=%leave%
usertitle=%leave%
customfield[cust_2go]=%leave%
customfield[face_pro]=%empty%
gender=%leave%
*url*=%empty%
*about*=%empty%

 

Такие списки сайтов не сохранял, тк пришел к выводу для себя что мне проще и гораздо эффективнее парсить сайты а не чекать их на цмс, хотя по логике чекинг по применяемым ресурсам менее затратен и должен быть более эффективнее (не нужны прокси, не нужно дергать поисковики, отсутствие капч...). Чекал по несколько дней, днсы шустрые находил и прописывал их в программе, но выхлоп совершенно не тот что ожидался. Допускаю что что-то неправильно делал. Потом как то попался на глаза триал GSA Platform Identifier, хоть то же не идеально, но уже повеселее. В принципе Вы можете сами скачать триальную версию (программа дает запустить себя несколько раз по полчаса) и провести объективный тест драйв: один и тот же список доменов прогнать в обоих программах при одинаковом числе потоков и сравнить полученные результаты.

 

Пожалуйста, подскажите.
Мне надо определить системы аналитики на сайтах, выбрал категорию - Analytics - https://take.ms/vBr45
Сайт для теста: https://www.ivi.ru/
Проверил в коде, есть система аналитики Google Tag Manager

Но в результате получаю https://www.ivi.ru/ - unknown
Или А-Парсер не видит, то, что находится в скриптах? например, https://take.ms/kSx1R