Ошибка вылет апарсера при использовании парсера гугла и капмонстра

вобщем в 250 потоков парсит нормально, стоит выставить 300 и больше - вылет. раньше, при парсинге без капмонстра не вылетало, ставил 700 потоков (такое кол-во потоков потому что прокси паблик и отрабатывают далеко не каждый раз). версия апарсера 1.1.666, win7x64, настройки антикапчи вот

Спойлер: настройка капмонстра

Спойлер: лог

Dec 12 18:02:15.54201 Start ProxyChecker proxies
Dec 12 18:02:16.9691 A-Parser v1.1.666-windows started, tasks: 2 / 497, memory total: 0 MB, available: 0 MB
Dec 12 18:02:17.22399 starting task #1836 threads: 300
Dec 12 18:02:21.30634 loading before not loaded iterator
Dec 12 18:02:21.30767 resuming queries/q.txt at 205 line
[2016-12-12 18:02:39] ERROR Frame is too large: 4740180
[0.401] ERROR Frame is too large: 4740180
IO error: config/unique/1836_global/003223.ldb: Too many open files at build/core.to_build.pl line 19426.
Dec 12 18:53:26.87168 Start ProxyChecker proxies
Dec 12 18:53:28.39877 A-Parser v1.1.666-windows started, tasks: 2 / 497, memory total: 0 MB, available: 0 MB
Dec 12 18:53:28.65197 starting task #1836 threads: 300
Dec 12 18:53:31.65796 loading before not loaded iterator
Dec 12 18:53:31.6593 resuming queries/q.txt at 205 line
[2016-12-12 18:53:47] ERROR Frame is too large: 4740180
[1.604] ERROR Frame is too large: 4740180
[6.166] ERROR Frame is too large: 4740180
[9.840] ERROR Frame is too large: 4740180
[13.445] ERROR Frame is too large: 4740180
IO error: config/unique/1836_global/003206.ldb: Too many open files at build/core.to_build.pl line 19426.
Dec 12 18:58:49.27212 Start ProxyChecker proxies
Dec 12 18:58:50.76221 A-Parser v1.1.666-windows started, tasks: 2 / 497, memory total: 0 MB, available: 0 MB
Dec 12 18:58:51.01259 starting task #1836 threads: 300
Dec 12 18:58:53.99939 loading before not loaded iterator
Dec 12 18:58:54.00085 resuming queries/q.txt at 205 line
IO error: config/unique/1836_global/003218.ldb: Too many open files at build/core.to_build.pl line 19426.
Dec 12 19:00:54.41828 Start ProxyChecker proxies
Dec 12 19:00:55.84236 A-Parser v1.1.666-windows started, tasks: 2 / 497, memory total: 0 MB, available: 0 MB
Dec 12 19:00:56.08942 starting task #1836 threads: 300
Dec 12 19:00:59.02955 loading before not loaded iterator
Dec 12 19:00:59.03085 resuming queries/q.txt at 205 line
[2016-12-12 19:01:38] ERROR Frame is too large: 4740180
[4.626] ERROR Frame is too large: 4740180
write_file 'config/tasks/1836.db~' - sysopen: Too many open files at build/core.to_build.pl line 23435
Dec 12 19:05:56.39455 Start ProxyChecker proxies
Dec 12 19:05:57.87664 A-Parser v1.1.666-windows started, tasks: 2 / 497, memory total: 0 MB, available: 0 MB
Dec 12 19:05:58.13668 starting task #1836 threads: 300
Dec 12 19:06:01.11682 loading before not loaded iterator
Dec 12 19:06:01.11827 resuming queries/q.txt at 205 line
IO error: config/unique/1836_global/003217.ldb: Too many open files at build/core.to_build.pl line 19426.

возможно поможет этот скрин, задание парсится долго, соответственно в нём большие цифры результатов

 

только 1 это задание. запустил, выставил 350 потоков - завис апарсер. запустил второй раз - вылетел. лог второго запуска:

Спойлер: лог вылета

Dec 13 20:21:41.38331 Start ProxyChecker proxies
Dec 13 20:21:42.9014 A-Parser v1.1.666-windows started, tasks: 2 / 497, memory total: 0 MB, available: 0 MB
Dec 13 20:21:43.15725 starting task #1836 threads: 350
Dec 13 20:21:46.5376 loading before not loaded iterator
Dec 13 20:21:46.53882 resuming queries/dorks/wp-dorks.txt at 205 line
IO error: config/unique/1836_global/003437.ldb: Too many open files at build/core.to_build.pl line 19426.

 

обновится - это ж надо продлить обновления ))) щас попробую
скрины настроек:




капмонстр:

 

попробовал на крайней версии - то же самое, однако заметил штуку, может поможет - выставил 400 потоков, однако после старта задания "Активных потоков" стало не 400, а 300 (проксей 1700+ рабочих, не в них дело) и дальше 300 не сдвигалось, парсинг пошёл, на капмонстр капчи ушли и продолжали идти ещё секунд 10, после чего апарсер вылетел.

Спойлер: лог с -morelogs

Dec 15 01:24:41.81897 [master] create pool of 7 threads
Dec 15 01:24:41.82457 [master] Start ProxyChecker proxies
Dec 15 01:24:42.89003 [master] load task 1850
Dec 15 01:24:42.90193 [master] load task 1836
Dec 15 01:24:42.90348 [master] load task 1849
Dec 15 01:24:42.90432 [master] load task 1848
Dec 15 01:24:42.90563 [master] load task 1847
Dec 15 01:24:42.90662 [master] load task 1838
Dec 15 01:24:42.90860 [master] load task 1839
Dec 15 01:24:42.90951 [master] load task 1830
Dec 15 01:24:42.91261 [master] load task 1831
Dec 15 01:24:42.91465 [master] load task 1829
Dec 15 01:24:42.91562 [master] load task 1828
Dec 15 01:24:42.91653 [master] load task 1825
Dec 15 01:24:42.91748 [master] load task 1822
Dec 15 01:24:42.91843 [master] load task 1821
Dec 15 01:24:42.91979 [master] load task 1820
Dec 15 01:24:42.92144 [master] load task 1818
Dec 15 01:24:42.92266 [master] load task 1817
Dec 15 01:24:42.92385 [master] load task 1816
Dec 15 01:24:42.92500 [master] load task 1805
Dec 15 01:24:42.92597 [master] load task 1798
Dec 15 01:24:42.92689 [master] load task 1795
Dec 15 01:24:42.92783 [master] load task 1792
Dec 15 01:24:42.92887 [master] load task 1791
Dec 15 01:24:42.92985 [master] load task 1783
Dec 15 01:24:42.93102 [master] load task 1782
Dec 15 01:24:42.93205 [master] load task 1779
Dec 15 01:24:42.93302 [master] load task 1729
Dec 15 01:24:42.93453 [master] load task 1728
Dec 15 01:24:42.93603 [master] load task 1723
Dec 15 01:24:42.93746 [master] load task 1722
Dec 15 01:24:42.93880 [master] load task 1719
Dec 15 01:24:42.94048 [master] load task 1718
Dec 15 01:24:42.94245 [master] load task 1690
Dec 15 01:24:42.94350 [master] load task 1689
Dec 15 01:24:42.94462 [master] load task 1688
Dec 15 01:24:42.94570 [master] load task 1687
Dec 15 01:24:42.94689 [master] load task 1685
Dec 15 01:24:42.94811 [master] load task 1684
Dec 15 01:24:42.94972 [master] load task 1683
Dec 15 01:24:42.95087 [master] load task 1682
Dec 15 01:24:42.95182 [master] load task 1681
Dec 15 01:24:42.95277 [master] load task 1680
Dec 15 01:24:42.95374 [master] load task 1679
Dec 15 01:24:42.95471 [master] load task 1678
Dec 15 01:24:42.95571 [master] load task 1677
Dec 15 01:24:42.95687 [master] load task 1676
Dec 15 01:24:42.95811 [master] load task 1675
Dec 15 01:24:42.95906 [master] load task 1674
Dec 15 01:24:42.95996 [master] load task 1672
Dec 15 01:24:42.96090 [master] load task 1671
Dec 15 01:24:42.96182 [master] load task 1670
Dec 15 01:24:42.96272 [master] load task 1669
Dec 15 01:24:42.96364 [master] load task 1668
Dec 15 01:24:42.96462 [master] load task 1666
Dec 15 01:24:42.96558 [master] load task 1665
Dec 15 01:24:42.96646 [master] load task 1664
Dec 15 01:24:42.96735 [master] load task 1663
Dec 15 01:24:42.96823 [master] load task 1662
Dec 15 01:24:42.96916 [master] load task 1661
Dec 15 01:24:42.97008 [master] load task 1653
Dec 15 01:24:42.97102 [master] load task 1658
Dec 15 01:24:42.97200 [master] load task 1659
Dec 15 01:24:42.97312 [master] load task 1655
Dec 15 01:24:42.97408 [master] load task 1646
Dec 15 01:24:42.97516 [master] load task 1650
Dec 15 01:24:42.97604 [master] load task 1649
Dec 15 01:24:42.97691 [master] load task 1648
Dec 15 01:24:42.97781 [master] load task 1642
Dec 15 01:24:42.97882 [master] load task 1641
Dec 15 01:24:42.97987 [master] load task 1639
Dec 15 01:24:42.98088 [master] load task 1640
Dec 15 01:24:42.98191 [master] load task 1638
Dec 15 01:24:42.98327 [master] load task 1637
Dec 15 01:24:42.98426 [master] load task 1608
Dec 15 01:24:42.98531 [master] load task 1603
Dec 15 01:24:42.98629 [master] load task 1602
Dec 15 01:24:42.98728 [master] load task 1601
Dec 15 01:24:42.98830 [master] load task 1600
Dec 15 01:24:42.98958 [master] load task 1574
Dec 15 01:24:42.99085 [master] load task 1573
Dec 15 01:24:42.99189 [master] load task 1572
Dec 15 01:24:42.99321 [master] load task 1571
Dec 15 01:24:42.99423 [master] load task 1570
Dec 15 01:24:42.99529 [master] load task 1569
Dec 15 01:24:42.99632 [master] load task 1568
Dec 15 01:24:42.99735 [master] load task 1567
Dec 15 01:24:42.99833 [master] load task 1566
Dec 15 01:24:42.99934 [master] load task 1558
Dec 15 01:24:43.00038 [master] load task 1562
Dec 15 01:24:43.00137 [master] load task 1560
Dec 15 01:24:43.00236 [master] load task 1555
Dec 15 01:24:43.00344 [master] load task 1559
Dec 15 01:24:43.00440 [master] load task 1557
Dec 15 01:24:43.00550 [master] load task 1541
Dec 15 01:24:43.00675 [master] load task 1561
Dec 15 01:24:43.00786 [master] load task 1556
Dec 15 01:24:43.00886 [master] load task 1554
Dec 15 01:24:43.00985 [master] load task 1553
Dec 15 01:24:43.01085 [master] load task 1547
Dec 15 01:24:43.01184 [master] load task 1545
Dec 15 01:24:43.01285 [master] load task 1552
Dec 15 01:24:43.01391 [master] load task 1548
Dec 15 01:24:43.01491 [master] load task 1551
Dec 15 01:24:43.01590 [master] load task 1550
Dec 15 01:24:43.01688 [master] load task 1549
Dec 15 01:24:43.01789 [master] load task 1543
Dec 15 01:24:43.01888 [master] load task 1534
Dec 15 01:24:43.01986 [master] load task 1544
Dec 15 01:24:43.02088 [master] load task 1546
Dec 15 01:24:43.02192 [master] load task 1540
Dec 15 01:24:43.02382 [master] load task 1539
Dec 15 01:24:43.02500 [master] load task 1536
Dec 15 01:24:43.02606 [master] load task 1537
Dec 15 01:24:43.02711 [master] load task 1542
Dec 15 01:24:43.02810 [master] load task 1538
Dec 15 01:24:43.02939 [master] load task 1535
Dec 15 01:24:43.03045 [master] load task 1533
Dec 15 01:24:43.03153 [master] load task 1532
Dec 15 01:24:43.03255 [master] load task 1531
Dec 15 01:24:43.03353 [master] load task 1530
Dec 15 01:24:43.03451 [master] load task 1503
Dec 15 01:24:43.03548 [master] load task 1502
Dec 15 01:24:43.03645 [master] load task 1501
Dec 15 01:24:43.03742 [master] load task 1500
Dec 15 01:24:43.03838 [master] load task 1498
Dec 15 01:24:43.03949 [master] load task 1497
Dec 15 01:24:43.04065 [master] load task 1499
Dec 15 01:24:43.04167 [master] load task 1488
Dec 15 01:24:43.04270 [master] load task 1484
Dec 15 01:24:43.04376 [master] load task 1494
Dec 15 01:24:43.04483 [master] load task 1495
Dec 15 01:24:43.04602 [master] load task 1493
Dec 15 01:24:43.04750 [master] load task 1476
Dec 15 01:24:43.04879 [master] load task 1491
Dec 15 01:24:43.05009 [master] load task 1492
Dec 15 01:24:43.05121 [master] load task 1487
Dec 15 01:24:43.05229 [master] load task 1490
Dec 15 01:24:43.05350 [master] load task 1489
Dec 15 01:24:43.05469 [master] load task 1485
Dec 15 01:24:43.05609 [master] load task 1482
Dec 15 01:24:43.05787 [master] load task 1486
Dec 15 01:24:43.05900 [master] load task 1480
Dec 15 01:24:43.06013 [master] load task 1483
Dec 15 01:24:43.06118 [master] load task 1478
Dec 15 01:24:43.06226 [master] load task 1468
Dec 15 01:24:43.06348 [master] load task 1481
Dec 15 01:24:43.06471 [master] load task 1479
Dec 15 01:24:43.06583 [master] load task 1477
Dec 15 01:24:43.06688 [master] load task 1473
Dec 15 01:24:43.06793 [master] load task 1471
Dec 15 01:24:43.06903 [master] load task 1475
Dec 15 01:24:43.07014 [master] load task 1474
Dec 15 01:24:43.07165 [master] load task 1472
Dec 15 01:24:43.07303 [master] load task 1469
Dec 15 01:24:43.07436 [master] load task 1470
Dec 15 01:24:43.07549 [master] load task 1467
Dec 15 01:24:43.07658 [master] load task 1466
Dec 15 01:24:43.07766 [master] load task 1463
Dec 15 01:24:43.07872 [master] load task 1464
Dec 15 01:24:43.07988 [master] load task 1465
Dec 15 01:24:43.08102 [master] load task 1461
Dec 15 01:24:43.08209 [master] load task 1462
Dec 15 01:24:43.08359 [master] load task 1456
Dec 15 01:24:43.08462 [master] load task 1455
Dec 15 01:24:43.08564 [master] load task 1454
Dec 15 01:24:43.08656 [master] load task 1450
Dec 15 01:24:43.08747 [master] load task 1449
Dec 15 01:24:43.08835 [master] load task 1427
Dec 15 01:24:43.08979 [master] load task 1447
Dec 15 01:24:43.09103 [master] load task 1424
Dec 15 01:24:43.09204 [master] load task 1446
Dec 15 01:24:43.09335 [master] load task 1445
Dec 15 01:24:43.09438 [master] load task 1444
Dec 15 01:24:43.09537 [master] load task 1441
Dec 15 01:24:43.09638 [master] load task 1440
Dec 15 01:24:43.09741 [master] load task 1426
Dec 15 01:24:43.09841 [master] load task 1400
Dec 15 01:24:43.09968 [master] load task 1411
Dec 15 01:24:43.10075 [master] load task 1391
Dec 15 01:24:43.10189 [master] load task 1390
Dec 15 01:24:43.10318 [master] load task 1388
Dec 15 01:24:43.10439 [master] load task 1381
Dec 15 01:24:43.10573 [master] load task 1380
Dec 15 01:24:43.10736 [master] load task 1379
Dec 15 01:24:43.10880 [master] load task 1377
Dec 15 01:24:43.10998 [master] load task 1373
Dec 15 01:24:43.11143 [master] load task 1375
Dec 15 01:24:43.11296 [master] load task 1374
Dec 15 01:24:43.11452 [master] load task 1363
Dec 15 01:24:43.11596 [master] load task 1360
Dec 15 01:24:43.11729 [master] load task 1359
Dec 15 01:24:43.11848 [master] load task 1349
Dec 15 01:24:43.11967 [master] load task 1352
Dec 15 01:24:43.12110 [master] load task 1351
Dec 15 01:24:43.12231 [master] load task 1350
Dec 15 01:24:43.12367 [master] load task 1345
Dec 15 01:24:43.12530 [master] load task 1344
Dec 15 01:24:43.12642 [master] load task 1343
Dec 15 01:24:43.12744 [master] load task 1340
Dec 15 01:24:43.12855 [master] load task 1342
Dec 15 01:24:43.12966 [master] load task 1338
Dec 15 01:24:43.13076 [master] load task 1335
Dec 15 01:24:43.13178 [master] load task 1329
Dec 15 01:24:43.13277 [master] load task 1333
Dec 15 01:24:43.13382 [master] load task 972
Dec 15 01:24:43.13481 [master] load task 974
Dec 15 01:24:43.13584 [master] load task 973
Dec 15 01:24:43.13686 [master] load task 968
Dec 15 01:24:43.13786 [master] load task 971
Dec 15 01:24:43.13892 [master] load task 970
Dec 15 01:24:43.14015 [master] load task 969
Dec 15 01:24:43.14138 [master] load task 1025
Dec 15 01:24:43.14268 [master] load task 1022
Dec 15 01:24:43.14397 [master] load task 1021
Dec 15 01:24:43.14523 [master] load task 1020
Dec 15 01:24:43.14665 [master] load task 1327
Dec 15 01:24:43.14778 [master] load task 1019
Dec 15 01:24:43.14964 [master] load task 1018
Dec 15 01:24:43.15139 [master] load task 1323
Dec 15 01:24:43.15285 [master] load task 1023
Dec 15 01:24:43.15411 [master] load task 1024
Dec 15 01:24:43.15549 [master] load task 1320
Dec 15 01:24:43.15666 [master] load task 1319
Dec 15 01:24:43.15779 [master] load task 1317
Dec 15 01:24:43.15927 [master] load task 1315
Dec 15 01:24:43.16036 [master] load task 1314
Dec 15 01:24:43.16143 [master] load task 1313
Dec 15 01:24:43.16245 [master] load task 1006
Dec 15 01:24:43.16372 [master] load task 1004
Dec 15 01:24:43.16499 [master] load task 986
Dec 15 01:24:43.16630 [master] load task 987
Dec 15 01:24:43.16727 [master] load task 984
Dec 15 01:24:43.16834 [master] load task 983
Dec 15 01:24:43.16941 [master] load task 982
Dec 15 01:24:43.17047 [master] load task 967
Dec 15 01:24:43.17146 [master] load task 966
Dec 15 01:24:43.17276 [master] load task 965
Dec 15 01:24:43.17420 [master] load task 963
Dec 15 01:24:43.17571 [master] load task 964
Dec 15 01:24:43.17672 [master] load task 962
Dec 15 01:24:43.17768 [master] load task 979
Dec 15 01:24:43.17857 [master] load task 961
Dec 15 01:24:43.17958 [master] load task 960
Dec 15 01:24:43.18060 [master] load task 959
Dec 15 01:24:43.18157 [master] load task 957
Dec 15 01:24:43.18257 [master] load task 955
Dec 15 01:24:43.18358 [master] load task 956
Dec 15 01:24:43.18460 [master] load task 944
Dec 15 01:24:43.18567 [master] load task 954
Dec 15 01:24:43.18677 [master] load task 958
Dec 15 01:24:43.18797 [master] load task 953
Dec 15 01:24:43.18946 [master] load task 952
Dec 15 01:24:43.19162 [master] load task 951
Dec 15 01:24:43.19298 [master] load task 948
Dec 15 01:24:43.19407 [master] load task 946
Dec 15 01:24:43.19511 [master] load task 947
Dec 15 01:24:43.19614 [master] load task 945
Dec 15 01:24:43.19718 [master] load task 943
Dec 15 01:24:43.19823 [master] load task 942
Dec 15 01:24:43.19923 [master] load task 976
Dec 15 01:24:43.20033 [master] load task 941
Dec 15 01:24:43.20134 [master] load task 950
Dec 15 01:24:43.20233 [master] load task 949
Dec 15 01:24:43.20339 [master] load task 940
Dec 15 01:24:43.20492 [master] load task 938
Dec 15 01:24:43.20675 [master] load task 936
Dec 15 01:24:43.20881 [master] load task 939
Dec 15 01:24:43.20990 [master] load task 975
Dec 15 01:24:43.21107 [master] load task 937
Dec 15 01:24:43.21211 [master] load task 935
Dec 15 01:24:43.21318 [master] load task 934
Dec 15 01:24:43.21425 [master] load task 933
Dec 15 01:24:43.21531 [master] load task 932
Dec 15 01:24:43.21642 [master] load task 931
Dec 15 01:24:43.21782 [master] load task 930
Dec 15 01:24:43.21895 [master] load task 928
Dec 15 01:24:43.21990 [master] load task 926
Dec 15 01:24:43.22082 [master] load task 914
Dec 15 01:24:43.22190 [master] load task 910
Dec 15 01:24:43.22308 [master] load task 902
Dec 15 01:24:43.22415 [master] load task 894
Dec 15 01:24:43.22518 [master] load task 892
Dec 15 01:24:43.22618 [master] load task 767
Dec 15 01:24:43.22750 [master] load task 831
Dec 15 01:24:43.22912 [master] load task 830
Dec 15 01:24:43.23082 [master] load task 766
Dec 15 01:24:43.23213 [master] load task 827
Dec 15 01:24:43.23386 [master] load task 826
Dec 15 01:24:43.23551 [master] load task 714
Dec 15 01:24:43.23718 [master] load task 761
Dec 15 01:24:43.23831 [master] load task 760
Dec 15 01:24:43.23957 [master] load task 757
Dec 15 01:24:43.24078 [master] load task 759
Dec 15 01:24:43.24188 [master] load task 758
Dec 15 01:24:43.24297 [master] load task 756
Dec 15 01:24:43.24438 [master] load task 755
Dec 15 01:24:43.24603 [master] load task 754
Dec 15 01:24:43.24772 [master] load task 750
Dec 15 01:24:43.24932 [master] load task 752
Dec 15 01:24:43.25057 [master] load task 751
Dec 15 01:24:43.25182 [master] load task 744
Dec 15 01:24:43.25352 [master] load task 725
Dec 15 01:24:43.25473 [master] load task 724
Dec 15 01:24:43.25583 [master] load task 722
Dec 15 01:24:43.25690 [master] load task 721
Dec 15 01:24:43.25795 [master] load task 720
Dec 15 01:24:43.25901 [master] load task 719
Dec 15 01:24:43.26011 [master] load task 718
Dec 15 01:24:43.26118 [master] load task 716
Dec 15 01:24:43.26222 [master] load task 710
Dec 15 01:24:43.26336 [master] load task 702
Dec 15 01:24:43.26466 [master] load task 706
Dec 15 01:24:43.26572 [master] load task 700
Dec 15 01:24:43.26680 [master] load task 693
Dec 15 01:24:43.26783 [master] load task 692
Dec 15 01:24:43.26885 [master] load task 686
Dec 15 01:24:43.26984 [master] load task 678
Dec 15 01:24:43.27079 [master] load task 677
Dec 15 01:24:43.27176 [master] load task 672
Dec 15 01:24:43.27278 [master] load task 671
Dec 15 01:24:43.27379 [master] load task 670
Dec 15 01:24:43.27479 [master] load task 669
Dec 15 01:24:43.27581 [master] load task 668
Dec 15 01:24:43.27680 [master] load task 665
Dec 15 01:24:43.27791 [master] load task 664
Dec 15 01:24:43.27936 [master] load task 612
Dec 15 01:24:43.28090 [master] load task 663
Dec 15 01:24:43.28192 [master] load task 656
Dec 15 01:24:43.28295 [master] load task 655
Dec 15 01:24:43.28400 [master] load task 654
Dec 15 01:24:43.28509 [master] load task 653
Dec 15 01:24:43.28613 [master] load task 652
Dec 15 01:24:43.28720 [master] load task 651
Dec 15 01:24:43.28825 [master] load task 650
Dec 15 01:24:43.28930 [master] load task 649
Dec 15 01:24:43.29033 [master] load task 648
Dec 15 01:24:43.29134 [master] load task 647
Dec 15 01:24:43.29235 [master] load task 646
Dec 15 01:24:43.29335 [master] load task 645
Dec 15 01:24:43.29437 [master] load task 644
Dec 15 01:24:43.29537 [master] load task 643
Dec 15 01:24:43.29640 [master] load task 642
Dec 15 01:24:43.29739 [master] load task 641
Dec 15 01:24:43.29839 [master] load task 640
Dec 15 01:24:43.29939 [master] load task 639
Dec 15 01:24:43.30037 [master] load task 638
Dec 15 01:24:43.30136 [master] load task 637
Dec 15 01:24:43.30235 [master] load task 636
Dec 15 01:24:43.30349 [master] load task 635
Dec 15 01:24:43.30450 [master] load task 634
Dec 15 01:24:43.30547 [master] load task 633
Dec 15 01:24:43.30641 [master] load task 632
Dec 15 01:24:43.30734 [master] load task 631
Dec 15 01:24:43.30828 [master] load task 630
Dec 15 01:24:43.30926 [master] load task 629
Dec 15 01:24:43.31021 [master] load task 628
Dec 15 01:24:43.31117 [master] load task 627
Dec 15 01:24:43.31211 [master] load task 626
Dec 15 01:24:43.31312 [master] load task 625
Dec 15 01:24:43.31409 [master] load task 624
Dec 15 01:24:43.31537 [master] load task 623
Dec 15 01:24:43.31661 [master] load task 622
Dec 15 01:24:43.31827 [master] load task 621
Dec 15 01:24:43.32034 [master] load task 620
Dec 15 01:24:43.32207 [master] load task 619
Dec 15 01:24:43.32408 [master] load task 617
Dec 15 01:24:43.32626 [master] load task 616
Dec 15 01:24:43.32830 [master] load task 615
Dec 15 01:24:43.33001 [master] load task 614
Dec 15 01:24:43.33154 [master] load task 613
Dec 15 01:24:43.33340 [master] load task 611
Dec 15 01:24:43.33569 [master] load task 610
Dec 15 01:24:43.33792 [master] load task 609
Dec 15 01:24:43.33987 [master] load task 608
Dec 15 01:24:43.34220 [master] load task 607
Dec 15 01:24:43.34425 [master] load task 606
Dec 15 01:24:43.34588 [master] load task 605
Dec 15 01:24:43.34706 [master] load task 604
Dec 15 01:24:43.34861 [master] load task 602
Dec 15 01:24:43.35051 [master] load task 601
Dec 15 01:24:43.35166 [master] load task 599
Dec 15 01:24:43.35280 [master] load task 598
Dec 15 01:24:43.35392 [master] load task 597
Dec 15 01:24:43.35504 [master] load task 596
Dec 15 01:24:43.35662 [master] load task 595
Dec 15 01:24:43.35774 [master] load task 575
Dec 15 01:24:43.35921 [master] load task 573
Dec 15 01:24:43.36032 [master] load task 572
Dec 15 01:24:43.36136 [master] load task 555
Dec 15 01:24:43.36251 [master] load task 551
Dec 15 01:24:43.36354 [master] load task 529
Dec 15 01:24:43.36463 [master] load task 522
Dec 15 01:24:43.36586 [master] load task 514
Dec 15 01:24:43.36688 [master] load task 531
Dec 15 01:24:43.36791 [master] load task 518
Dec 15 01:24:43.36894 [master] load task 521
Dec 15 01:24:43.37025 [master] load task 515
Dec 15 01:24:43.37168 [master] load task 547
Dec 15 01:24:43.37335 [master] load task 517
Dec 15 01:24:43.37441 [master] load task 520
Dec 15 01:24:43.37540 [master] load task 516
Dec 15 01:24:43.37645 [master] load task 546
Dec 15 01:24:43.37746 [master] load task 545
Dec 15 01:24:43.37905 [master] load task 537
Dec 15 01:24:43.38064 [master] load task 544
Dec 15 01:24:43.38176 [master] load task 542
Dec 15 01:24:43.38332 [master] load task 540
Dec 15 01:24:43.38469 [master] load task 519
Dec 15 01:24:43.38567 [master] load task 538
Dec 15 01:24:43.38662 [master] load task 541
Dec 15 01:24:43.38760 [master] load task 530
Dec 15 01:24:43.38857 [master] load task 543
Dec 15 01:24:43.38958 [master] load task 539
Dec 15 01:24:43.39106 [master] load task 533
Dec 15 01:24:43.39215 [master] load task 535
Dec 15 01:24:43.39354 [master] load task 536
Dec 15 01:24:43.39462 [master] load task 525
Dec 15 01:24:43.39562 [master] load task 534
Dec 15 01:24:43.39658 [master] load task 532
Dec 15 01:24:43.39757 [master] load task 524
Dec 15 01:24:43.39859 [master] load task 523
Dec 15 01:24:43.39959 [master] load task 527
Dec 15 01:24:43.40057 [master] load task 528
Dec 15 01:24:43.40157 [master] load task 513
Dec 15 01:24:43.40253 [master] load task 526
Dec 15 01:24:43.40357 [master] load task 478
Dec 15 01:24:43.40464 [master] load task 483
Dec 15 01:24:43.40564 [master] load task 486
Dec 15 01:24:43.40663 [master] load task 481
Dec 15 01:24:43.40763 [master] load task 480
Dec 15 01:24:43.40860 [master] load task 482
Dec 15 01:24:43.40956 [master] load task 479
Dec 15 01:24:43.41051 [master] load task 371
Dec 15 01:24:43.41155 [master] load task 372
Dec 15 01:24:43.41259 [master] load task 369
Dec 15 01:24:43.41365 [master] load task 364
Dec 15 01:24:43.41473 [master] load task 367
Dec 15 01:24:43.41581 [master] load task 365
Dec 15 01:24:43.41690 [master] load task 368
Dec 15 01:24:43.41796 [master] load task 366
Dec 15 01:24:43.41913 [master] load task 363
Dec 15 01:24:43.42032 [master] load task 356
Dec 15 01:24:43.42151 [master] load task 346
Dec 15 01:24:43.42268 [master] load task 355
Dec 15 01:24:43.42394 [master] load task 354
Dec 15 01:24:43.42513 [master] load task 358
Dec 15 01:24:43.42630 [master] load task 357
Dec 15 01:24:43.42771 [master] load task 362
Dec 15 01:24:43.42895 [master] load task 360
Dec 15 01:24:43.43022 [master] load task 361
Dec 15 01:24:43.43135 [master] load task 347
Dec 15 01:24:43.43249 [master] load task 359
Dec 15 01:24:43.43366 [master] load task 353
Dec 15 01:24:43.43481 [master] load task 349
Dec 15 01:24:43.43592 [master] load task 352
Dec 15 01:24:43.43703 [master] load task 350
Dec 15 01:24:43.43818 [master] load task 351
Dec 15 01:24:43.43930 [master] load task 348
Dec 15 01:24:43.44048 [master] load task 343
Dec 15 01:24:43.44161 [master] load task 345
Dec 15 01:24:43.44270 [master] load task 344
Dec 15 01:24:43.44382 [master] load task 340
Dec 15 01:24:43.44496 [master] load task 337
Dec 15 01:24:43.44610 [master] load task 335
Dec 15 01:24:43.44731 [master] load task 338
Dec 15 01:24:43.44881 [master] load task 342
Dec 15 01:24:43.45037 [master] load task 341
Dec 15 01:24:43.45177 [master] load task 339
Dec 15 01:24:43.45297 [master] load task 330
Dec 15 01:24:43.45412 [master] load task 334
Dec 15 01:24:43.45531 [master] load task 336
Dec 15 01:24:43.45651 [master] load task 329
Dec 15 01:24:43.45772 [master] load task 331
Dec 15 01:24:43.45893 [master] load task 332
Dec 15 01:24:43.46032 [master] load task 333
Dec 15 01:24:43.46150 [master] load task 228
Dec 15 01:24:43.46355 [master] load task 285
Dec 15 01:24:43.46494 [master] load task 274
Dec 15 01:24:43.46604 [master] load task 241
Dec 15 01:24:43.46899 [master] load task 239
Dec 15 01:24:43.47173 [master] load task 236
Dec 15 01:24:43.47298 [master] load task 237
Dec 15 01:24:43.47411 [master] load task 235
Dec 15 01:24:43.47523 [master] load task 229
Dec 15 01:24:43.47633 [master] load task 232
Dec 15 01:24:43.47742 [master] load task 231
Dec 15 01:24:43.47853 [master] load task 230
Dec 15 01:24:43.47987 [master] load task 191
Dec 15 01:24:43.48114 [master] load task 227
Dec 15 01:24:43.48242 [master] load task 201
Dec 15 01:24:43.48353 [master] load task 200
Dec 15 01:24:43.48461 [master] load task 199
Dec 15 01:24:43.48567 [master] load task 198
Dec 15 01:24:43.48674 [master] load task 197
Dec 15 01:24:43.48784 [master] load task 193
Dec 15 01:24:43.48894 [master] load task 194
Dec 15 01:24:43.49002 [master] load task 196
Dec 15 01:24:43.49112 [master] load task 192
Dec 15 01:24:43.49221 [master] load task 184
Dec 15 01:24:43.49339 [master] A-Parser v1.1.707-windows-x86 started, tasks: 2 / 497, memory total: 0 MB, available: 0 MB
Dec 15 01:24:43.76601 [master] load task 1836
Dec 15 01:24:43.76764 [master] starting task #1836 threads: 400
Dec 15 01:24:45.14816 [worker/-9520] started
Dec 15 01:24:45.32917 [worker/-9584] started
Dec 15 01:24:45.52418 [worker/-5168] started
Dec 15 01:24:47.15827 [worker/-8296] started
Dec 15 01:24:47.34128 [worker/-5816] started
Dec 15 01:24:47.52029 [worker/-10008] started
Dec 15 01:24:47.70931 [worker/-5472] started
Dec 15 01:24:48.41234 [master] using existing iterator
Dec 15 01:24:48.41266 [master] loading before not loaded iterator
Dec 15 01:24:48.41377 [master] resuming queries/q.txt at 205 line
Dec 15 01:24:48.41416 [master] register unique 1836_iterator undef bloom 10 max_open_files 1000
Dec 15 01:24:48.43037 [master] iterator check ok
Dec 15 01:24:48.43064 [master] new results collector
Dec 15 01:24:48.43099 [master] register unique 1836_global undef bloom 10 max_open_files 1000
Dec 15 01:24:48.46759 [master] results collector check ok
Dec 15 01:24:48.48010 [master] start child d631
Dec 15 01:24:48.48043 [master] started
IO error: config/unique/1836_global/003519.ldb: Too many open files at build/core.to_build.pl line 21392.

хотя без -morelogs тоже выдаёт ту же самую ошибку крайней строкой, номер бд только различается от раза к разу

 

спарсило ещё 20к запросов, теперь уже и на 250 потоках вылетает с той же ошибкой. чувствую к концу задания буду в 10 потоков парсить )))

попробую купить ваших проксей, попарсить в 100 потоков, думаю на них 100 потоков будут примерно равнятся 400-500 потоков на пабликах...

 

проблема частично решилась, с wifi переключился на кабель и уже более 300 потоков стало возможно ставить. при 500 было норм, но при 600 уже вылетало. вобщем не знаю, я задания по-другому организовал (разбил файл запросов на много частей), теперь всё норм в 1000+ потоков.