Lots of "Wait" Proxy Threads

[scrapefun]

I just upgraded to 1.1.496 from 1.1.486 and for some reason I'm getting a lot of wait proxy threads to the point where after about 10 minutes or so all threads are "waiting".

I think I've set all the new proxy checker settings correctly.

Under "config presets" I've selected the proxy checker to use and I've also used the override preset option for the NET:Http parser to use the correct proxy checker. And the only proxy checker "enabled" is the one I want to use.

I've rolled back to 1.1.486 and everything works fine so I must not have things set correctly in the latest version.

It wouldn't have anything to do with using tools.query.add would it? It would know to use the same proxy checker for queries configured through tools.query.add?

 

[Support]

In newer versions (starting from 1.1.491):

1. Select Proxy Checker in menu left.
2. Add new Proxy checker and/or enable existing.
3. If necessary - edit and save settings of this proxy checker.
4. Open the folder that indicated in Working path.
5. Specify proxy sources that you want to use in this Proxy checker: links in sites.txt, list of proxy in proxy.txt.
6. Go back to the parser, select Settings - Config Presets, edit existing or create new, by selecting one or more Proxy checkers.
7. Now you can use the created Config preset, with specified Proxy checkers in their tasks. You can also override the proxy checker in each parser using Override function - Proxy Checker.

Directly to work tools.query these innovations do not apply. All added queries will be processed with the proxies that are specified in the parser/task.

 

[scrapefun]

That's how I've got things setup so I'm not sure what the issue is.

In the log file when running version 1.1.496 and starting a task it always says this:
Apr 27 07:58:15.62481 starting task #265 threads: 500, proxies: undef

I tried using both Google and Net::HTTP parsers and also tried deleting custom proxy checker presets and only use default.

If I revert back to 1.1.486 in the log it will show the number of proxies. I assume "undef" means it's not finding any proxies but I've tried every combination of settings i can think of and it always comes back with "undef"

When I use the task tester it is using the proxies in my proxy file.

 

[Support]

proxies: undef

This is not a problem. In this version, so the log is written, it will be fixed in future versions.

Does the problem from the 1st post decided?

 

[scrapefun]

yes, still have same problem. The task starts fine and then after 2-3 minutes wait threads start and eventually all threads waiting.

I had been using the "no check proxies" option enabled but I'm now testing with it disabled and it's been running about 10 minutes at the moment and not having the problem. To soon to tell for sure but that seems to have made a difference.

 

[Support]

It seems that all proxies are banned. Try to set the Proxy ban time 0

 

[scrapefun]

Proxy ban time is already set to 0. It's been running fine with the checking option enabled (the default setting). But enabling "no check proxies" threads go back to waiting almost immediately. Same proxy lists.

Then if I switch to older version (1.1.486) and use same list there are no waiting threads. Weird

 

[Forbidden]

Please post sreenshots when it in waiting threads state:

• screenshot of task in queue + info(mouse over on "i")
• screenshots of task settings + parsers settings
• screenshot of proxy checker tab

thanks

 

[scrapefun]

Below are the screenshots. I can see just from mouse over on "i" that the problem is banned proxies but since proxy ban is 0 should it be banning any?

Here is the job queue for the same same task, settings and proxies using version 1.1.486. In this case no proxies are banned as expected:

The proxies I am using are backconnect so a ip/port that might be bad or blocked might be fine a few minutes later as new IPs are rotated to the various ports.

I also tried it with a proxy ban and the banned numbers never decrease even when the time for banned proxies to become "live" again has passed. But in version 1.1.486 with proxy ban enabled you can watch the banned numbers fluctuate up and down as banned proxies are released.

 

[Forbidden]

Fixed in 1.1.498, thanks you for detailed info!

 

[scrapefun]

Great! Glad I could help in some small way.

Thanks for the great support and software