Redis API Questions

[scrapefun]

A few questions about the Redis API:

1) When setting threads using {num:1:150}, and I have a config preset for 150 threads set for the task, does Redis API use the preset's settings? or does it use the default settings for the threads presets?

2) How do retries work? I have a Google preset I use with Redis API set to 25 retries, and the redis preset also set to 25 retries? Does the Redis preset override the retries of the Google preset?

3) Any optimizations tips for getting the most speed? I'm using Kvrocks. Also I have multiple instances of A-parser running on the same server (Kvrocks a separate server)

 

[Support]

The Documentation contains information about which parameters are used from where:

• all settings, except for useproxy, proxyChecker, and proxybannedcleanup are taken from the preset of the called scraper + overrideOpts
• settings useproxy, proxyChecker, and proxybannedcleanup are taken from the preset API::Server::Redis + overrideOpts

1) When setting threads using {num:1:150}, and I have a config preset for 150 threads set for the task, does Redis API use the preset's settings? or does it use the default settings for the threads presets?

The task with API::Server::Redis will be launched in 150 threads, the rest of the parameters will be used according to the information above.

2) How do retries work? I have a Google preset I use with Redis API set to 25 retries, and the redis preset also set to 25 retries? Does the Redis preset override the retries of the Google preset?

According to the information above, the request retries parameter (proxyretries) will be taken from the preset of the called scraper.

3) Any optimizations tips for getting the most speed? I'm using Kvrocks. Also I have multiple instances of A-parser running on the same server (Kvrocks a separate server)

There are no special tips or recommendations. You should experiment and monitor the server load based on your tasks. If you have multiple instances of A-Parser, you can process queries from a single queue using the same Redis Queue Key.

 

[scrapefun]

Thanks for the reply. Still not clear on one thing.

When I set the thread count using: {num:1:150}

Does it use the default settings for threads?





If so, how can I change the default settings? Specifically, "Reuse proxy between retries":


Would I use the configPreset parameter so the options I want for threads are used?

 

[Support]

{num:1:150}

This is just a quick way to set the number of queries equal to the number of threads. In API::Server::Redis, each thread waits indefinitely for tasks to appear in the Redis queue. So you need to run it this way.

Does it use the default settings for threads?
If so, how can I change the default settings? Specifically, "Reuse proxy between retries":

The task with API::Server::Redis uses the threads config that you have selected. Accordingly, you need to select the threads config with the settings you need (having previously created it).

Would I use the configPreset parameter so the options I want for threads are used?

No, this parameter is not supported in queries via the Redis API.

 

[scrapefun]

Thanks, this helped a lot.