Ошибка [1.2.482] SE::AOL не энкодит query, пустые результаты

[siventes]

Версия v1.2.480
В случае с отсутствием энкодинга параметров запроса, добавить нечего.
С пустыми результатами - скорее всего парсер неправильно "детектит" "плохой" прокси. По каким то причина пачка пустых результатов именно при старте парсинга, дальше реже.
на 7К результатов, пришлось 1.3К "пустых", что совсем немало.

 

[siventes]

Отредактировать первое сообщение не вышло, не понравилось движку форума.
Пресет:

Спойлер

eyJwcmVzZXQiOiJTRTpBT0wiLCJ2YWx1ZSI6eyJwcmVzZXQiOiJTRTpBT0wiLCJw
YXJzZXJzIjpbWyJTRTo6QU9MIiwiZGVmYXVsdCIseyJ0eXBlIjoib3ZlcnJpZGUi
LCJpZCI6InVzZXByb3h5IiwidmFsdWUiOnRydWV9LHsidHlwZSI6Im92ZXJyaWRl
IiwiaWQiOiJwcm94eXJldHJpZXMiLCJ2YWx1ZSI6IjIwMCJ9LHsidHlwZSI6Im92
ZXJyaWRlIiwiaWQiOiJwYWdlX2NvdW50IiwidmFsdWUiOjV9LHsidHlwZSI6Im92
ZXJyaWRlIiwiaWQiOiJwcm94eWJhbm5lZGNsZWFudXAiLCJ2YWx1ZSI6IjAifSx7
InR5cGUiOiJvdmVycmlkZSIsImlkIjoidGltZW91dCIsInZhbHVlIjoiMzAifSx7
InR5cGUiOiJvdmVycmlkZSIsImlkIjoicmVxdWVzdGRlbGF5IiwidmFsdWUiOiIw
In0seyJ0eXBlIjoib3ZlcnJpZGUiLCJpZCI6ImRvbWFpbiIsInZhbHVlIjoic2Vh
cmNoLmFvbC5jb20ifSx7InR5cGUiOiJvdmVycmlkZSIsImlkIjoiZXh0cmFxdWVy
eSIsInZhbHVlIjoiIn1dXSwicmVzdWx0c0Zvcm1hdCI6IlslXG5saW1pdD0yO1xu
cmVzdWx0LmFkc19jb3VudD1wMS5hZHMuc2l6ZTtcbnJlc3VsdC5xdWVyeT1wMS5x
dWVyeS5xdWVyeTtcbnJlc3VsdC5xdWVyeV9sdmw9cXVlcnkubHZsO1xucmVzdWx0
LnNlcnA9W107XG5yZXN1bHQucmVsYXRlZD1bXTtcbnJlc3VsdC5hZHM9W107XG5y
ZXN1bHQudG90YWxjb3VudD1wMS50b3RhbGNvdW50O1xuXG5GT1JFQUNIIGkgPSBw
MS5yZWxhdGVkO1xuXHRrZXk9aS5yZWxhdGVkIHwgdHJpbTtcblx0cmVzdWx0LnJl
bGF0ZWQucHVzaChrZXkpO1xuXHRJRiBxdWVyeS5sdmwgPCAgbGltaXQ7XG5cdFx0
dG9vbHMucXVlcnkuYWRkKGtleSk7XG5cdEVORDtcbkVORDtcbkZPUkVBQ0ggaSA9
IHAxLnNlcnA7XG5cdHNuaXBwZXQgPSBpLnNuaXBwZXQgfCB0cmltO1xuXHRzbmlw
cGV0ID0gc25pcHBldC5yZXBsYWNlKCdcXG4nLCAnJyk7XG5cdGFuY2hvciA9IGku
YW5jaG9yIHwgdHJpbTtcblx0YW5jaG9yID0gYW5jaG9yLnJlcGxhY2UoJ1xcbics
ICcnKTtcblx0cmVzID0ge1xuXHQgICBzbmlwcGV0ID0+IHNuaXBwZXQsXG5cdCAg
IGFuY2hvciA9PiBhbmNob3IsXG5cdCAgIGxpbmsgPT4gaS5saW5rLFxuICAgIH07
XG5cdHJlc3VsdC5zZXJwLnB1c2gocmVzKTtcbkVORDtcbkZPUkVBQ0ggaSA9IHAx
LmFkcztcblx0c25pcHBldCA9IGkuc25pcHBldCB8IHRyaW07XG5cdHNuaXBwZXQg
PSBpLnNuaXBwZXQucmVwbGFjZSgnXFxuJywgJycpO1xuXHRhbmNob3IgPSBpLmFu
Y2hvciB8IHRyaW07XG5cdGFuY2hvciA9IGFuY2hvci5yZXBsYWNlKCdcXG4nLCAn
Jyk7XG4gICAgcmVzID0ge1xuXHQgICBzbmlwcGV0ID0+IHNuaXBwZXQsXG5cdCAg
IGFuY2hvciA9PiBhbmNob3IsXG4gICAgICAgcG9zaXRpb24gPT5pLnBvc2l0aW9u
LFxuICAgICAgIGxpbmsgPT4gaS5saW5rLFxuICAgIH07XG5cdHJlc3VsdC5hZHMu
cHVzaChyZXMpO1xuRU5EO1xucmVzdWx0Lmpzb24oKSB8IHRyaW07XG5cIlxcblwi
O1xuJV0iLCJyZXN1bHRzU2F2ZVRvIjoiZmlsZSIsInJlc3VsdHNGaWxlTmFtZSI6
IiRkYXRlZmlsZS5mb3JtYXQoKS50eHQiLCJhZGRpdGlvbmFsRm9ybWF0cyI6W10s
InJlc3VsdHNVbmlxdWUiOiJubyIsInF1ZXJpZXNGcm9tIjoiZmlsZSIsInF1ZXJ5
Rm9ybWF0IjpbIiRxdWVyeSJdLCJ1bmlxdWVRdWVyaWVzIjp0cnVlLCJzYXZlRmFp
bGVkUXVlcmllcyI6ZmFsc2UsIml0ZXJhdG9yT3B0aW9ucyI6eyJvbkFsbExldmVs
cyI6ZmFsc2UsInF1ZXJ5QnVpbGRlcnNBZnRlckl0ZXJhdG9yIjpmYWxzZSwicXVl
cnlCdWlsZGVyc09uQWxsTGV2ZWxzIjpmYWxzZX0sInJlc3VsdHNPcHRpb25zIjp7
Im92ZXJ3cml0ZSI6ZmFsc2UsIndyaXRlQk9NIjpmYWxzZX0sImRvTG9nIjoibm8i
LCJrZWVwVW5pcXVlIjoiTm8iLCJtb3JlT3B0aW9ucyI6ZmFsc2UsInJlc3VsdHNQ
cmVwZW5kIjoiIiwicmVzdWx0c0FwcGVuZCI6IiIsInF1ZXJ5QnVpbGRlcnMiOltd
LCJyZXN1bHRzQnVpbGRlcnMiOlt7InNvdXJjZSI6WzAsWyJyZWxhdGVkIiwicmVs
YXRlZCJdXSwidHlwZSI6ImRlY29kZUh0bWwiLCJhcnJheSI6InJlbGF0ZWQiLCJ0
byI6InJlbGF0ZWQifSx7InNvdXJjZSI6WzAsWyJyZWxhdGVkIiwicmVsYXRlZCJd
XSwidHlwZSI6InJlbW92ZUh0bWwiLCJhcnJheSI6InJlbGF0ZWQiLCJ0byI6InJl
bGF0ZWQifSx7InNvdXJjZSI6WzAsWyJzZXJwIiwic25pcHBldCJdXSwidHlwZSI6
ImRlY29kZUh0bWwiLCJhcnJheSI6InNlcnAiLCJ0byI6InNuaXBwZXQifSx7InNv
dXJjZSI6WzAsWyJzZXJwIiwic25pcHBldCJdXSwidHlwZSI6InJlbW92ZUh0bWwi
LCJhcnJheSI6InNlcnAiLCJ0byI6InNuaXBwZXQifSx7InNvdXJjZSI6WzAsWyJz
ZXJwIiwiYW5jaG9yIl1dLCJ0eXBlIjoiZGVjb2RlSHRtbCIsImFycmF5Ijoic2Vy
cCIsInRvIjoiYW5jaG9yIn0seyJzb3VyY2UiOlswLFsic2VycCIsImFuY2hvciJd
XSwidHlwZSI6InJlbW92ZUh0bWwiLCJhcnJheSI6InNlcnAiLCJ0byI6ImFuY2hv
ciJ9XSwiY29uZmlnT3ZlcnJpZGVzIjpbXSwicnVuVGFza09uQ29tcGxldGUiOm51
bGwsInVzZVJlc3VsdHNGaWxlQXNRdWVyaWVzRmlsZSI6ZmFsc2UsInJ1blRhc2tP
bkNvbXBsZXRlQ29uZmlnIjoiZGVmYXVsdCIsInRvb2xzSlMiOiIiLCJwcmlvIjo1
LCJyZW1vdmVPbkNvbXBsZXRlIjpmYWxzZSwiY2FsbFVSTE9uQ29tcGxldGUiOiIi
LCJxdWVyaWVzRmlsZSI6WyJxdWVyaWVzL3NyYy9qb2JfMV9maWx0ZXJlZF9hbGwu
dHh0Il0sImNhdGVnb3J5IjpbXX19

Пустые результаты:

Спойлер

{"ads":[],"related":[],"query":"find jobs in my area","ads_count":"","totalcount":"none","serp":[],"query_lvl":0}
{"ads":[],"related":[],"query":"best way to find jobs in los angeles","ads_count":"","totalcount":"none","serp":[],"query_lvl":0}
{"ads":[],"related":[],"query":"how to find remote jobs at startups","ads_count":"","totalcount":"none","serp":[],"query_lvl":0}
{"ads":[],"related":[],"query":"find tutoring jobs near me","ads_count":"","totalcount":"none","serp":[],"query_lvl":0}
{"ads":[],"related":[],"query":"find odd jobs near me","ads_count":"","totalcount":"none","serp":[],"query_lvl":0}
{"ads":[],"related":[],"query":"get jobs near me","ads_count":"","totalcount":"none","serp":[],"query_lvl":0}
{"ads":[],"related":[],"query":"find jobs close to me","ads_count":"","totalcount":"none","serp":[],"query_lvl":0}
{"ads":[],"related":[],"query":"find cash jobs near me","ads_count":"","totalcount":"none","serp":[],"query_lvl":0}
{"ads":[],"related":[],"query":"find jobs near me for 15 year olds","ads_count":"","totalcount":"none","serp":[],"query_lvl":0}

 

[Support]

Пустые результаты:

Запустите задание с включенным логом и покажите лог по одному из "пустых" результатов.

 

[siventes]

А разделе "Parser Test" никак не удается пока воспроизвести проблему, чтобы дать исходный код страницы. Сейчас перепишу результаты, чтобы получить исходники html для понимания что это такое вообще.
Несколько логов, что я просмотрел, выглядят примерно одинаково:

Thread #73 07/05 14:46:21 - 07/05 14:46:28 SE::AOL::0(Complete): find jobs applied linkedin
07/05 14:46:21
Parser SE::AOL::0 parse query find jobs applied linkedin
07/05 14:46:21
Use proxy socks://ip:port
07/05 14:46:21
Parse page 1
07/05 14:46:27
GET(1): https://search.aol.com/aol/search?q=find jobs applied linkedin&ei=UTF-8&btf= - 200 OK (23.16 KB)
07/05 14:46:27
Decode from utf-8(meta http-equiv)
07/05 14:46:27
Results count not found
07/05 14:46:27
Total found 0 results
07/05 14:46:27
No more results

 

[siventes]

К сожалению, не могу предоставить нужные данные для дебага, так как в парсере куда то "пропал" раздел сырых данных - $p1.data.
Прошу починить это в первую очередь в целях отладки.
Result format:

$p1.json();

Результат:

Спойлер

{"related":[{"related":"jobs hiring near me"},{"related":"best job for you test"},{"related":"why are you looking for new opportunities"},{"related":"what job is right for you"},{"related":"find jobs near my location"},{"related":"finding employment opportunities"},{"related":"looking for a job online"},{"related":"post a job for free"}],"success":1,"info":{"success":1,"retries":1,"stats":{"success":1,"retries":1,"proxiesUsed":1,"requests":1,"queries":1}},"query":{"first":"find jobs for me","num":0,"lvl":0,"prev":"","threadId":"0","query":"find jobs for me","queryUid":"9e39a4","orig":"find jobs for me"},"serp":[{"link":"https://www.monster.com/jobs/search/","snippet":"Explore all the new jobs in the US from entry level to management roles. Browse by location, industry or company. Find the right job for you and build a career.","anchor":"Job Search: Find Job Openings | Monster.com"},{"link":"https://www.whatcareerisrightforme.com/","snippet":"Career Aptitude Test Take our free career test to determine what jobs are best suited to your skills and interests.. The test is composed of four brief sections covering your: SKILLS: What are your personal strengths and weakness? INTERESTS: What sorts of activities and subjects do you find interesting? STYLE: What style of work environment are you looking for?","anchor":"What Career is Right For Me?"},{"link":"https://www.monster.com/","snippet":"Monster is your source for jobs and career opportunities. Search for jobs, read career advice from Monster's job experts, and find hiring and recruiting advice.","anchor":"Monster Jobs - Job Search, Career Advice & Hiring ..."},{"link":"https://www.carecareers.com.au/page/career-quiz","snippet":"job search Will this suit me? Looking for more flexibility, greater challenges? More rewarding work? The care sector could be for you. Find out who it suits. Take our Career Quiz! Try our Career Quiz - it's a fun and easy way to see what career is right for you. Let Employers Find You ...","anchor":"Career Quiz | Care Careers"},{"link":"http://time.com/4343767/job-personality-work/","snippet":"Feeling the itch to change jobs, or even careers? Before you take the plunge, try this quiz to see which job best fits your personality. Find Out What Job Best Matches Your Personality","anchor":"Find Out What Job Best Matches Your Personality - time.com"},{"link":"https://www.roberthalf.com/blog/job-market/find-me-a-job-how-a-recruiting-agency-can-help","snippet":"Expert help in finding a job. Working with a recruiter is almost as simple as saying, \"Please find me a job.\" Once you reach out, a recruiter can act as your liaison to hiring managers, share industry information and career resources that support your job hunt, and even coach you on the finer points of assembling your job-search materials.","anchor":"Find Me a Job! How a Recruiting Agency Can Help | Robert Half"},{"link":"https://www.careerbuilder.com/jobs-full-time","snippet":"Search CareerBuilder for Full Time Jobs and browse our platform. Apply now for jobs that are hiring near you.","anchor":"Full Time Jobs - Apply Now | CareerBuilder"},{"link":"https://www.payscale.com/data-packages/best-jobs","snippet":"Basically, you're constantly trying to find out what job is best for you, no matter how old or how experienced you are. That's why PayScale created our Best Job for Me quiz. Learn About Jobs That ...","anchor":"PayScale's Best Jobs Tool - What Job Is Best For You?"},{"link":"https://www.indeed.com/find-jobs.jsp","snippet":"Find jobs. indeed.com one search. all jobs.","anchor":"Find jobs | Indeed.com"},{"link":"https://www.indeed.com/","snippet":"With Indeed, you can search millions of jobs online to find the next step in your career. With tools for job search, resumes, company reviews and more, we're with you every step of the way.","anchor":"Job Search | Indeed"}],"totalcount":728000000};

 

[Support]

Добавлена проверка контента, теперь при получении страницы, отличной от страницы с результатами (или сообщением об отсутствии результатов), парсер будет повторять попытку.
Также добавлена возможность выводить контент страниц $pages.$i.data
Запросы энкодились и ранее.

 

[siventes]

Добавлена проверка контента, теперь при получении страницы, отличной от страницы с результатами (или сообщением об отсутствии результатов), парсер будет повторять попытку.
Также добавлена возможность выводить контент страниц $pages.$i.data
Запросы энкодились и ранее.

Судя по логу в предыдущем моем сообщении, или запросы не энкодились или лог "врет".

GET(1): https://search.aol.com/aol/search?q=find jobs applied linkedin&ei=UTF-8&btf= - 200 OK (23.16 KB)

 

[Support]

 

[siventes]

Понял, спасибо, в заблуждения я был введен потому что в других парсерах все строчки в логах с энкодом, в этом - нет.
За обновление спасибо, потестирую.