1. Вступайте в наш Telegram чат: https://t.me/a_parser Нас уже 2600+ и мы растем!
    Скрыть объявление

Улучшение Custom Proxy Delay

Тема в разделе "Отклоненные задачи", создана пользователем scrapefun, 24 фев 2015.

  1. scrapefun

    scrapefun A-Parser Enterprise License
    A-Parser Enterprise

    Регистрация:
    24 фев 2015
    Сообщения:
    184
    Симпатии:
    34
    It would be great if we could set how long to wait after a proxy has been used before it can be used again.

    If you wait 30-45 seconds between each query per proxy you can pretty much scrape Google indefinitely without proxy ban (with dedicated/private proxies). So to be able to control this would really enhance the software.

    Thanks!
     
  2. Forbidden

    Forbidden Administrator
    Команда форума A-Parser Enterprise

    Регистрация:
    9 мар 2013
    Сообщения:
    3.339
    Симпатии:
    1.799
    After review of code I can confirm that use Max threads per proxy + Request delay(in parsers settings) is suitable for this case

    It's work like this:
    lock proxy for thread
    wait some time before do request
    release proxy lock
     

Поделиться этой страницей