Версия v1.2.480 В случае с отсутствием энкодинга параметров запроса, добавить нечего. С пустыми результатами - скорее всего парсер неправильно "детектит" "плохой" прокси. По каким то причина пачка пустых результатов именно при старте парсинга, дальше реже. на 7К результатов, пришлось 1.3К "пустых", что совсем немало.
Отредактировать первое сообщение не вышло, не понравилось движку форума. Пресет: Спойлер eyJwcmVzZXQiOiJTRTpBT0wiLCJ2YWx1ZSI6eyJwcmVzZXQiOiJTRTpBT0wiLCJw YXJzZXJzIjpbWyJTRTo6QU9MIiwiZGVmYXVsdCIseyJ0eXBlIjoib3ZlcnJpZGUi LCJpZCI6InVzZXByb3h5IiwidmFsdWUiOnRydWV9LHsidHlwZSI6Im92ZXJyaWRl IiwiaWQiOiJwcm94eXJldHJpZXMiLCJ2YWx1ZSI6IjIwMCJ9LHsidHlwZSI6Im92 ZXJyaWRlIiwiaWQiOiJwYWdlX2NvdW50IiwidmFsdWUiOjV9LHsidHlwZSI6Im92 ZXJyaWRlIiwiaWQiOiJwcm94eWJhbm5lZGNsZWFudXAiLCJ2YWx1ZSI6IjAifSx7 InR5cGUiOiJvdmVycmlkZSIsImlkIjoidGltZW91dCIsInZhbHVlIjoiMzAifSx7 InR5cGUiOiJvdmVycmlkZSIsImlkIjoicmVxdWVzdGRlbGF5IiwidmFsdWUiOiIw In0seyJ0eXBlIjoib3ZlcnJpZGUiLCJpZCI6ImRvbWFpbiIsInZhbHVlIjoic2Vh cmNoLmFvbC5jb20ifSx7InR5cGUiOiJvdmVycmlkZSIsImlkIjoiZXh0cmFxdWVy eSIsInZhbHVlIjoiIn1dXSwicmVzdWx0c0Zvcm1hdCI6IlslXG5saW1pdD0yO1xu cmVzdWx0LmFkc19jb3VudD1wMS5hZHMuc2l6ZTtcbnJlc3VsdC5xdWVyeT1wMS5x dWVyeS5xdWVyeTtcbnJlc3VsdC5xdWVyeV9sdmw9cXVlcnkubHZsO1xucmVzdWx0 LnNlcnA9W107XG5yZXN1bHQucmVsYXRlZD1bXTtcbnJlc3VsdC5hZHM9W107XG5y ZXN1bHQudG90YWxjb3VudD1wMS50b3RhbGNvdW50O1xuXG5GT1JFQUNIIGkgPSBw MS5yZWxhdGVkO1xuXHRrZXk9aS5yZWxhdGVkIHwgdHJpbTtcblx0cmVzdWx0LnJl bGF0ZWQucHVzaChrZXkpO1xuXHRJRiBxdWVyeS5sdmwgPCAgbGltaXQ7XG5cdFx0 dG9vbHMucXVlcnkuYWRkKGtleSk7XG5cdEVORDtcbkVORDtcbkZPUkVBQ0ggaSA9 IHAxLnNlcnA7XG5cdHNuaXBwZXQgPSBpLnNuaXBwZXQgfCB0cmltO1xuXHRzbmlw cGV0ID0gc25pcHBldC5yZXBsYWNlKCdcXG4nLCAnJyk7XG5cdGFuY2hvciA9IGku YW5jaG9yIHwgdHJpbTtcblx0YW5jaG9yID0gYW5jaG9yLnJlcGxhY2UoJ1xcbics ICcnKTtcblx0cmVzID0ge1xuXHQgICBzbmlwcGV0ID0+IHNuaXBwZXQsXG5cdCAg IGFuY2hvciA9PiBhbmNob3IsXG5cdCAgIGxpbmsgPT4gaS5saW5rLFxuICAgIH07 XG5cdHJlc3VsdC5zZXJwLnB1c2gocmVzKTtcbkVORDtcbkZPUkVBQ0ggaSA9IHAx LmFkcztcblx0c25pcHBldCA9IGkuc25pcHBldCB8IHRyaW07XG5cdHNuaXBwZXQg PSBpLnNuaXBwZXQucmVwbGFjZSgnXFxuJywgJycpO1xuXHRhbmNob3IgPSBpLmFu Y2hvciB8IHRyaW07XG5cdGFuY2hvciA9IGFuY2hvci5yZXBsYWNlKCdcXG4nLCAn Jyk7XG4gICAgcmVzID0ge1xuXHQgICBzbmlwcGV0ID0+IHNuaXBwZXQsXG5cdCAg IGFuY2hvciA9PiBhbmNob3IsXG4gICAgICAgcG9zaXRpb24gPT5pLnBvc2l0aW9u LFxuICAgICAgIGxpbmsgPT4gaS5saW5rLFxuICAgIH07XG5cdHJlc3VsdC5hZHMu cHVzaChyZXMpO1xuRU5EO1xucmVzdWx0Lmpzb24oKSB8IHRyaW07XG5cIlxcblwi O1xuJV0iLCJyZXN1bHRzU2F2ZVRvIjoiZmlsZSIsInJlc3VsdHNGaWxlTmFtZSI6 IiRkYXRlZmlsZS5mb3JtYXQoKS50eHQiLCJhZGRpdGlvbmFsRm9ybWF0cyI6W10s InJlc3VsdHNVbmlxdWUiOiJubyIsInF1ZXJpZXNGcm9tIjoiZmlsZSIsInF1ZXJ5 Rm9ybWF0IjpbIiRxdWVyeSJdLCJ1bmlxdWVRdWVyaWVzIjp0cnVlLCJzYXZlRmFp bGVkUXVlcmllcyI6ZmFsc2UsIml0ZXJhdG9yT3B0aW9ucyI6eyJvbkFsbExldmVs cyI6ZmFsc2UsInF1ZXJ5QnVpbGRlcnNBZnRlckl0ZXJhdG9yIjpmYWxzZSwicXVl cnlCdWlsZGVyc09uQWxsTGV2ZWxzIjpmYWxzZX0sInJlc3VsdHNPcHRpb25zIjp7 Im92ZXJ3cml0ZSI6ZmFsc2UsIndyaXRlQk9NIjpmYWxzZX0sImRvTG9nIjoibm8i LCJrZWVwVW5pcXVlIjoiTm8iLCJtb3JlT3B0aW9ucyI6ZmFsc2UsInJlc3VsdHNQ cmVwZW5kIjoiIiwicmVzdWx0c0FwcGVuZCI6IiIsInF1ZXJ5QnVpbGRlcnMiOltd LCJyZXN1bHRzQnVpbGRlcnMiOlt7InNvdXJjZSI6WzAsWyJyZWxhdGVkIiwicmVs YXRlZCJdXSwidHlwZSI6ImRlY29kZUh0bWwiLCJhcnJheSI6InJlbGF0ZWQiLCJ0 byI6InJlbGF0ZWQifSx7InNvdXJjZSI6WzAsWyJyZWxhdGVkIiwicmVsYXRlZCJd XSwidHlwZSI6InJlbW92ZUh0bWwiLCJhcnJheSI6InJlbGF0ZWQiLCJ0byI6InJl bGF0ZWQifSx7InNvdXJjZSI6WzAsWyJzZXJwIiwic25pcHBldCJdXSwidHlwZSI6 ImRlY29kZUh0bWwiLCJhcnJheSI6InNlcnAiLCJ0byI6InNuaXBwZXQifSx7InNv dXJjZSI6WzAsWyJzZXJwIiwic25pcHBldCJdXSwidHlwZSI6InJlbW92ZUh0bWwi LCJhcnJheSI6InNlcnAiLCJ0byI6InNuaXBwZXQifSx7InNvdXJjZSI6WzAsWyJz ZXJwIiwiYW5jaG9yIl1dLCJ0eXBlIjoiZGVjb2RlSHRtbCIsImFycmF5Ijoic2Vy cCIsInRvIjoiYW5jaG9yIn0seyJzb3VyY2UiOlswLFsic2VycCIsImFuY2hvciJd XSwidHlwZSI6InJlbW92ZUh0bWwiLCJhcnJheSI6InNlcnAiLCJ0byI6ImFuY2hv ciJ9XSwiY29uZmlnT3ZlcnJpZGVzIjpbXSwicnVuVGFza09uQ29tcGxldGUiOm51 bGwsInVzZVJlc3VsdHNGaWxlQXNRdWVyaWVzRmlsZSI6ZmFsc2UsInJ1blRhc2tP bkNvbXBsZXRlQ29uZmlnIjoiZGVmYXVsdCIsInRvb2xzSlMiOiIiLCJwcmlvIjo1 LCJyZW1vdmVPbkNvbXBsZXRlIjpmYWxzZSwiY2FsbFVSTE9uQ29tcGxldGUiOiIi LCJxdWVyaWVzRmlsZSI6WyJxdWVyaWVzL3NyYy9qb2JfMV9maWx0ZXJlZF9hbGwu dHh0Il0sImNhdGVnb3J5IjpbXX19 Пустые результаты: Спойлер {"ads":[],"related":[],"query":"find jobs in my area","ads_count":"","totalcount":"none","serp":[],"query_lvl":0} {"ads":[],"related":[],"query":"best way to find jobs in los angeles","ads_count":"","totalcount":"none","serp":[],"query_lvl":0} {"ads":[],"related":[],"query":"how to find remote jobs at startups","ads_count":"","totalcount":"none","serp":[],"query_lvl":0} {"ads":[],"related":[],"query":"find tutoring jobs near me","ads_count":"","totalcount":"none","serp":[],"query_lvl":0} {"ads":[],"related":[],"query":"find odd jobs near me","ads_count":"","totalcount":"none","serp":[],"query_lvl":0} {"ads":[],"related":[],"query":"get jobs near me","ads_count":"","totalcount":"none","serp":[],"query_lvl":0} {"ads":[],"related":[],"query":"find jobs close to me","ads_count":"","totalcount":"none","serp":[],"query_lvl":0} {"ads":[],"related":[],"query":"find cash jobs near me","ads_count":"","totalcount":"none","serp":[],"query_lvl":0} {"ads":[],"related":[],"query":"find jobs near me for 15 year olds","ads_count":"","totalcount":"none","serp":[],"query_lvl":0}
А разделе "Parser Test" никак не удается пока воспроизвести проблему, чтобы дать исходный код страницы. Сейчас перепишу результаты, чтобы получить исходники html для понимания что это такое вообще. Несколько логов, что я просмотрел, выглядят примерно одинаково: Код: Thread #73 07/05 14:46:21 - 07/05 14:46:28 SE::AOL::0(Complete): find jobs applied linkedin 07/05 14:46:21 Parser SE::AOL::0 parse query find jobs applied linkedin 07/05 14:46:21 Use proxy socks://ip:port 07/05 14:46:21 Parse page 1 07/05 14:46:27 GET(1): https://search.aol.com/aol/search?q=find jobs applied linkedin&ei=UTF-8&btf= - 200 OK (23.16 KB) 07/05 14:46:27 Decode from utf-8(meta http-equiv) 07/05 14:46:27 Results count not found 07/05 14:46:27 Total found 0 results 07/05 14:46:27 No more results
К сожалению, не могу предоставить нужные данные для дебага, так как в парсере куда то "пропал" раздел сырых данных - $p1.data. Прошу починить это в первую очередь в целях отладки. Result format: Код: $p1.json(); Результат: Спойлер {"related":[{"related":"jobs hiring near me"},{"related":"best job for you test"},{"related":"why are you looking for new opportunities"},{"related":"what job is right for you"},{"related":"find jobs near my location"},{"related":"finding employment opportunities"},{"related":"looking for a job online"},{"related":"post a job for free"}],"success":1,"info":{"success":1,"retries":1,"stats":{"success":1,"retries":1,"proxiesUsed":1,"requests":1,"queries":1}},"query":{"first":"find jobs for me","num":0,"lvl":0,"prev":"","threadId":"0","query":"find jobs for me","queryUid":"9e39a4","orig":"find jobs for me"},"serp":[{"link":"https://www.monster.com/jobs/search/","snippet":"Explore all the new jobs in the US from entry level to management roles. Browse by location, industry or company. Find the right job for you and build a career.","anchor":"Job Search: Find Job Openings | Monster.com"},{"link":"https://www.whatcareerisrightforme.com/","snippet":"Career Aptitude Test Take our free career test to determine what jobs are best suited to your skills and interests.. The test is composed of four brief sections covering your: SKILLS: What are your personal strengths and weakness? INTERESTS: What sorts of activities and subjects do you find interesting? STYLE: What style of work environment are you looking for?","anchor":"What Career is Right For Me?"},{"link":"https://www.monster.com/","snippet":"Monster is your source for jobs and career opportunities. Search for jobs, read career advice from Monster's job experts, and find hiring and recruiting advice.","anchor":"Monster Jobs - Job Search, Career Advice & Hiring ..."},{"link":"https://www.carecareers.com.au/page/career-quiz","snippet":"job search Will this suit me? Looking for more flexibility, greater challenges? More rewarding work? The care sector could be for you. Find out who it suits. Take our Career Quiz! Try our Career Quiz - it's a fun and easy way to see what career is right for you. Let Employers Find You ...","anchor":"Career Quiz | Care Careers"},{"link":"http://time.com/4343767/job-personality-work/","snippet":"Feeling the itch to change jobs, or even careers? Before you take the plunge, try this quiz to see which job best fits your personality. Find Out What Job Best Matches Your Personality","anchor":"Find Out What Job Best Matches Your Personality - time.com"},{"link":"https://www.roberthalf.com/blog/job-market/find-me-a-job-how-a-recruiting-agency-can-help","snippet":"Expert help in finding a job. Working with a recruiter is almost as simple as saying, \"Please find me a job.\" Once you reach out, a recruiter can act as your liaison to hiring managers, share industry information and career resources that support your job hunt, and even coach you on the finer points of assembling your job-search materials.","anchor":"Find Me a Job! How a Recruiting Agency Can Help | Robert Half"},{"link":"https://www.careerbuilder.com/jobs-full-time","snippet":"Search CareerBuilder for Full Time Jobs and browse our platform. Apply now for jobs that are hiring near you.","anchor":"Full Time Jobs - Apply Now | CareerBuilder"},{"link":"https://www.payscale.com/data-packages/best-jobs","snippet":"Basically, you're constantly trying to find out what job is best for you, no matter how old or how experienced you are. That's why PayScale created our Best Job for Me quiz. Learn About Jobs That ...","anchor":"PayScale's Best Jobs Tool - What Job Is Best For You?"},{"link":"https://www.indeed.com/find-jobs.jsp","snippet":"Find jobs. indeed.com one search. all jobs.","anchor":"Find jobs | Indeed.com"},{"link":"https://www.indeed.com/","snippet":"With Indeed, you can search millions of jobs online to find the next step in your career. With tools for job search, resumes, company reviews and more, we're with you every step of the way.","anchor":"Job Search | Indeed"}],"totalcount":728000000};
Добавлена проверка контента, теперь при получении страницы, отличной от страницы с результатами (или сообщением об отсутствии результатов), парсер будет повторять попытку. Также добавлена возможность выводить контент страниц $pages.$i.data Запросы энкодились и ранее.
Судя по логу в предыдущем моем сообщении, или запросы не энкодились или лог "врет". Код: GET(1): https://search.aol.com/aol/search?q=find jobs applied linkedin&ei=UTF-8&btf= - 200 OK (23.16 KB)
Понял, спасибо, в заблуждения я был введен потому что в других парсерах все строчки в логах с энкодом, в этом - нет. За обновление спасибо, потестирую.
