подтормаживает а-парсер при парсинге релейтедов с гугла

[DeLaKroiX]

использую парсинг релейтедов с проксями с гугла, потоков выставил 950(пробовал и 450, результат тот же), чек прокси 400 потоков, парсинг происходит так:
начинается с маленькой текущей скорости, потом она повышается до примерно одинаковой величины(для 450 и для 950 потоков величина соответственно разная), потом интерфейс подвисает секунд на 10-30 и скорость сбрасывается до начальной и всё начинается по-новой. при подвисании интерфейса все кнопки тыкаются нормально, но ничего не происходит, если в это время нажать на сохранение чего-либо, то сохраняться оно будет ровно до конца этого тормоза. подскажите пожалуйста, что это может быть ? могу показать тимвьювером.

 

[Forbidden]

покажи последние 10-20 строчек aparser.log во время выполнения задания

 

[Forbidden]

и скрин настроек задания

 

[DeLaKroiX]

это то, что в логе, такое я уже давно заметил, но тормоза заметил вот только вчера:
[59.285] ERROR Frame is too large: 4740180
[59.978] ERROR Frame is too large: 4740180
[2016-08-24 17:39:58] ERROR Frame is too large: 4740180
периодически встречается вот такая строка:
Aug 24 17:00:52.03156 GC takes 4.27501249313354

пардон, забыл сказать, версия апарсера была 613, были проблемы, обновился до последней, сейчас последняя, проблемы остались.

задание:

 

[DeLaKroiX]

решил скинуть побольше лога отдельным сообщением, чтобы было заметно, а не редактировать предыдущее. циферки в квадратных скобках от строки к строке увеличиваются, возможно это имеет какое-то значение, потому что в предыдущем посте этого не видно:

Спойлер: больше лога

[0.269] ERROR Frame is too large: 4740180
[0.692] ERROR Frame is too large: 4740180
[0.929] ERROR Frame is too large: 4740180
[1.178] ERROR Frame is too large: 4740180
[1.581] ERROR Frame is too large: 4740180
[1.986] ERROR Frame is too large: 4740180
[2.100] ERROR Frame is too large: 4740180
[2.280] ERROR Frame is too large: 4740180
[2.425] ERROR Frame is too large: 4740180
[3.067] ERROR Frame is too large: 4740180
[3.091] ERROR Frame is too large: 4740180
[3.734] ERROR Frame is too large: 4740180
[3.887] ERROR Frame is too large: 4740180
[3.968] ERROR Frame is too large: 4740180
[4.664] ERROR Frame is too large: 4740180
[4.997] ERROR Frame is too large: 4740180
[5.634] ERROR Frame is too large: 4740180
[5.794] ERROR Frame is too large: 4740180
[5.916] ERROR Frame is too large: 4740180
[6.423] ERROR Frame is too large: 4740180
[6.709] ERROR Frame is too large: 4740180
[6.793] ERROR Frame is too large: 4740180
[7.364] ERROR Frame is too large: 4740180
[7.692] ERROR Frame is too large: 4740180
[8.140] ERROR Frame is too large: 4740180
[8.691] ERROR Frame is too large: 4740180
[9.077] ERROR Frame is too large: 4740180
[9.191] ERROR Frame is too large: 4740180
[9.214] ERROR Frame is too large: 4740180
[9.452] ERROR Frame is too large: 4740180
[9.456] ERROR Frame is too large: 4740180
[10.421] ERROR Frame is too large: 4740180
[10.611] ERROR Frame is too large: 4740180
[10.842] ERROR Frame is too large: 4740180
[11.757] ERROR Frame is too large: 4740180
[11.774] ERROR Frame is too large: 4740180
[12.591] ERROR Frame is too large: 4740180
[13.373] ERROR Frame is too large: 4740180
[13.666] ERROR Frame is too large: 4740180
[14.072] ERROR Frame is too large: 4740180
[14.447] ERROR Frame is too large: 4740180
[14.995] ERROR Frame is too large: 4740180
[15.373] ERROR Frame is too large: 4740180
[15.455] ERROR Frame is too large: 4740180
[15.620] ERROR Frame is too large: 4740180
[16.172] ERROR Frame is too large: 4740180
[16.201] ERROR Frame is too large: 4740180
[16.481] ERROR Frame is too large: 4740180
[17.163] ERROR Frame is too large: 4740180
[17.493] ERROR Frame is too large: 4740180
[18.240] ERROR Frame is too large: 4740180
[18.414] ERROR Frame is too large: 4740180
[18.747] ERROR Frame is too large: 4740180
[19.038] ERROR Frame is too large: 4740180
[19.088] ERROR Frame is too large: 4740180
[19.353] ERROR Frame is too large: 4740180
[19.642] ERROR Frame is too large: 4740180
[19.825] ERROR Frame is too large: 4740180
[20.137] ERROR Frame is too large: 4740180
[20.961] ERROR Frame is too large: 4740180
[21.067] ERROR Frame is too large: 4740180
[21.967] ERROR Frame is too large: 4740180
[22.230] ERROR Frame is too large: 4740180
[22.905] ERROR Frame is too large: 4740180
[22.930] ERROR Frame is too large: 4740180
[23.133] ERROR Frame is too large: 4740180
[24.023] ERROR Frame is too large: 4740180
[24.366] ERROR Frame is too large: 4740180
[24.703] ERROR Frame is too large: 4740180
[25.292] ERROR Frame is too large: 4740180
[25.753] ERROR Frame is too large: 4740180
[25.867] ERROR Frame is too large: 4740180
[26.470] ERROR Frame is too large: 4740180
[27.095] ERROR Frame is too large: 4740180
[27.159] ERROR Frame is too large: 4740180
[27.607] ERROR Frame is too large: 4740180
[27.841] ERROR Frame is too large: 4740180
[27.845] ERROR Frame is too large: 4740180
[28.300] ERROR Frame is too large: 4740180
[28.335] ERROR Frame is too large: 4740180
[28.428] ERROR Frame is too large: 4740180
[28.605] ERROR Frame is too large: 4740180
[28.717] ERROR Frame is too large: 4740180
[29.281] ERROR Frame is too large: 4740180
[29.519] ERROR Frame is too large: 4740180
[30.013] ERROR Frame is too large: 4740180
[30.016] ERROR Frame is too large: 4740180
[30.094] ERROR Frame is too large: 4740180
[30.167] ERROR Frame is too large: 4740180
[30.543] ERROR Frame is too large: 4740180
[31.086] ERROR Frame is too large: 4740180
[31.246] ERROR Frame is too large: 4740180
[31.617] ERROR Frame is too large: 4740180
[32.157] ERROR Frame is too large: 4740180
[32.834] ERROR Frame is too large: 4740180
[32.912] ERROR Frame is too large: 4740180
[33.715] ERROR Frame is too large: 4740180
[34.033] ERROR Frame is too large: 4740180
[34.256] ERROR Frame is too large: 4740180
[34.534] ERROR Frame is too large: 4740180
[34.672] ERROR Frame is too large: 4740180
[35.246] ERROR Frame is too large: 4740180
[35.533] ERROR Frame is too large: 4740180
[36.232] ERROR Frame is too large: 4740180
[36.319] ERROR Frame is too large: 4740180
[36.442] ERROR Frame is too large: 4740180
[36.559] ERROR Frame is too large: 4740180
[36.935] ERROR Frame is too large: 4740180
[37.518] ERROR Frame is too large: 4740180
[37.521] ERROR Frame is too large: 4740180
[37.646] ERROR Frame is too large: 4740180
[37.700] ERROR Frame is too large: 4740180
[38.417] ERROR Frame is too large: 4740180
[38.468] ERROR Frame is too large: 4740180
[38.680] ERROR Frame is too large: 4740180
[39.084] ERROR Frame is too large: 4740180
[39.150] ERROR Frame is too large: 4740180
[39.169] ERROR Frame is too large: 4740180
[39.573] ERROR Frame is too large: 4740180
[40.074] ERROR Frame is too large: 4740180
[40.297] ERROR Frame is too large: 4740180
[40.454] ERROR Frame is too large: 4740180
[40.901] ERROR Frame is too large: 4740180
[41.091] ERROR Frame is too large: 4740180
[41.729] ERROR Frame is too large: 4740180
[41.857] ERROR Frame is too large: 4740180
[42.135] ERROR Frame is too large: 4740180
[42.669] ERROR Frame is too large: 4740180
[42.765] ERROR Frame is too large: 4740180
[42.853] ERROR Frame is too large: 4740180
[42.961] ERROR Frame is too large: 4740180
[43.555] ERROR Frame is too large: 4740180
[43.940] ERROR Frame is too large: 4740180
[43.983] ERROR Frame is too large: 4740180
[44.386] ERROR Frame is too large: 4740180
[44.696] ERROR Frame is too large: 4740180
[44.756] ERROR Frame is too large: 4740180
[45.076] ERROR Frame is too large: 4740180
[45.147] ERROR Frame is too large: 4740180
[45.292] ERROR Frame is too large: 4740180
[45.392] ERROR Frame is too large: 4740180
[45.697] ERROR Frame is too large: 4740180
[46.151] ERROR Frame is too large: 4740180
[46.467] ERROR Frame is too large: 4740180
[46.946] ERROR Frame is too large: 4740180
[47.299] ERROR Frame is too large: 4740180
[47.798] ERROR Frame is too large: 4740180
[48.026] ERROR Frame is too large: 4740180
[48.297] ERROR Frame is too large: 4740180
[49.683] ERROR Frame is too large: 4740180
[49.787] ERROR Frame is too large: 4740180
[50.287] ERROR Frame is too large: 4740180
[50.964] ERROR Frame is too large: 4740180
[51.063] ERROR Frame is too large: 4740180
[51.127] ERROR Frame is too large: 4740180
[51.179] ERROR Frame is too large: 4740180
[52.130] ERROR Frame is too large: 4740180
[52.387] ERROR Frame is too large: 4740180
[52.986] ERROR Frame is too large: 4740180
[53.545] ERROR Frame is too large: 4740180
[53.678] ERROR Frame is too large: 4740180
[54.464] ERROR Frame is too large: 4740180
[54.491] ERROR Frame is too large: 4740180
Aug 24 17:06:51.29689 GC takes 3.06770133972168
[58.417] ERROR Frame is too large: 4740180
[58.480] ERROR Frame is too large: 4740180
[58.490] ERROR Frame is too large: 4740180
[59.547] ERROR Frame is too large: 4740180
[2016-08-24 17:06:53] ERROR Frame is too large: 4740180
[0.100] ERROR Frame is too large: 4740180
[0.326] ERROR Frame is too large: 4740180
[1.232] ERROR Frame is too large: 4740180
[1.544] ERROR Frame is too large: 4740180
[3.491] ERROR Frame is too large: 4740180
[3.847] ERROR Frame is too large: 4740180
[4.165] ERROR Frame is too large: 4740180
[4.917] ERROR Frame is too large: 4740180
[5.076] ERROR Frame is too large: 4740180
[5.176] ERROR Frame is too large: 4740180
[5.748] ERROR Frame is too large: 4740180
[5.759] ERROR Frame is too large: 4740180
[6.694] ERROR Frame is too large: 4740180
[7.139] ERROR Frame is too large: 4740180
[7.188] ERROR Frame is too large: 4740180
[7.724] ERROR Frame is too large: 4740180
[8.096] ERROR Frame is too large: 4740180
[8.474] ERROR Frame is too large: 4740180
[8.811] ERROR Frame is too large: 4740180
[9.222] ERROR Frame is too large: 4740180
[9.581] ERROR Frame is too large: 4740180
[9.972] ERROR Frame is too large: 4740180
[10.014] ERROR Frame is too large: 4740180
[10.304] ERROR Frame is too large: 4740180
[10.984] ERROR Frame is too large: 4740180
[11.155] ERROR Frame is too large: 4740180
[11.174] ERROR Frame is too large: 4740180
[11.811] ERROR Frame is too large: 4740180
[12.197] ERROR Frame is too large: 4740180
[12.591] ERROR Frame is too large: 4740180
[13.088] ERROR Frame is too large: 4740180
[13.159] ERROR Frame is too large: 4740180
[13.470] ERROR Frame is too large: 4740180
[13.614] ERROR Frame is too large: 4740180
[13.713] ERROR Frame is too large: 4740180
[14.174] ERROR Frame is too large: 4740180
[14.216] ERROR Frame is too large: 4740180
[14.843] ERROR Frame is too large: 4740180
[14.940] ERROR Frame is too large: 4740180
[15.578] ERROR Frame is too large: 4740180
[15.716] ERROR Frame is too large: 4740180
[16.164] ERROR Frame is too large: 4740180
[17.197] ERROR Frame is too large: 4740180
[17.327] ERROR Frame is too large: 4740180
[17.573] ERROR Frame is too large: 4740180
[17.831] ERROR Frame is too large: 4740180
[18.536] ERROR Frame is too large: 4740180
[18.703] ERROR Frame is too large: 4740180
[18.983] ERROR Frame is too large: 4740180
[19.202] ERROR Frame is too large: 4740180
[19.851] ERROR Frame is too large: 4740180
[20.081] ERROR Frame is too large: 4740180
[20.087] ERROR Frame is too large: 4740180
[20.345] ERROR Frame is too large: 4740180
[20.654] ERROR Frame is too large: 4740180
[21.303] ERROR Frame is too large: 4740180
[21.374] ERROR Frame is too large: 4740180
[21.639] ERROR Frame is too large: 4740180
[22.094] ERROR Frame is too large: 4740180
[22.893] ERROR Frame is too large: 4740180
[23.563] ERROR Frame is too large: 4740180
[23.658] ERROR Frame is too large: 4740180
[23.714] ERROR Frame is too large: 4740180
[23.956] ERROR Frame is too large: 4740180
[24.339] ERROR Frame is too large: 4740180
[24.469] ERROR Frame is too large: 4740180
[24.925] ERROR Frame is too large: 4740180
[25.364] ERROR Frame is too large: 4740180
[25.983] ERROR Frame is too large: 4740180
[26.109] ERROR Frame is too large: 4740180
[26.829] ERROR Frame is too large: 4740180
[26.881] ERROR Frame is too large: 4740180
[27.653] ERROR Frame is too large: 4740180
[27.782] ERROR Frame is too large: 4740180
[27.911] ERROR Frame is too large: 4740180
[28.184] ERROR Frame is too large: 4740180
[28.807] ERROR Frame is too large: 4740180
[29.225] ERROR Frame is too large: 4740180
[29.292] ERROR Frame is too large: 4740180
[29.350] ERROR Frame is too large: 4740180
[29.532] ERROR Frame is too large: 4740180
[30.128] ERROR Frame is too large: 4740180
[30.256] ERROR Frame is too large: 4740180
[30.399] ERROR Frame is too large: 4740180
[31.059] ERROR Frame is too large: 4740180
[31.296] ERROR Frame is too large: 4740180
[31.644] ERROR Frame is too large: 4740180
[31.806] ERROR Frame is too large: 4740180
[32.049] ERROR Frame is too large: 4740180
[32.566] ERROR Frame is too large: 4740180
[32.771] ERROR Frame is too large: 4740180
[33.405] ERROR Frame is too large: 4740180
[33.597] ERROR Frame is too large: 4740180
[33.656] ERROR Frame is too large: 4740180
[34.187] ERROR Frame is too large: 4740180
[34.451] ERROR Frame is too large: 4740180
[34.951] ERROR Frame is too large: 4740180
[35.364] ERROR Frame is too large: 4740180
[35.954] ERROR Frame is too large: 4740180
[36.232] ERROR Frame is too large: 4740180
[36.607] ERROR Frame is too large: 4740180
[37.251] ERROR Frame is too large: 4740180
[37.382] ERROR Frame is too large: 4740180
[37.743] ERROR Frame is too large: 4740180
[37.968] ERROR Frame is too large: 4740180
[38.060] ERROR Frame is too large: 4740180
[38.543] ERROR Frame is too large: 4740180
[38.556] ERROR Frame is too large: 4740180
[38.667] ERROR Frame is too large: 4740180
[38.715] ERROR Frame is too large: 4740180
[39.878] ERROR Frame is too large: 4740180
[39.879] ERROR Frame is too large: 4740180
[40.414] ERROR Frame is too large: 4740180
[40.674] ERROR Frame is too large: 4740180
[40.776] ERROR Frame is too large: 4740180
[40.825] ERROR Frame is too large: 4740180
[40.935] ERROR Frame is too large: 4740180
[40.995] ERROR Frame is too large: 4740180
[41.535] ERROR Frame is too large: 4740180
[42.197] ERROR Frame is too large: 4740180
[42.686] ERROR Frame is too large: 4740180
[42.878] ERROR Frame is too large: 4740180
[43.507] ERROR Frame is too large: 4740180
[43.618] ERROR Frame is too large: 4740180
[43.892] ERROR Frame is too large: 4740180
[44.019] ERROR Frame is too large: 4740180
[44.268] ERROR Frame is too large: 4740180
[44.405] ERROR Frame is too large: 4740180
[44.932] ERROR Frame is too large: 4740180
[45.165] ERROR Frame is too large: 4740180
[45.963] ERROR Frame is too large: 4740180
[45.976] ERROR Frame is too large: 4740180
[46.326] ERROR Frame is too large: 4740180
[46.629] ERROR Frame is too large: 4740180
[46.948] ERROR Frame is too large: 4740180
[47.315] ERROR Frame is too large: 4740180
[47.358] ERROR Frame is too large: 4740180
[47.580] ERROR Frame is too large: 4740180
[48.056] ERROR Frame is too large: 4740180
[48.221] ERROR Frame is too large: 4740180
[48.341] ERROR Frame is too large: 4740180
[48.639] ERROR Frame is too large: 4740180
[48.782] ERROR Frame is too large: 4740180
[48.815] ERROR Frame is too large: 4740180
[49.122] ERROR Frame is too large: 4740180
[49.181] ERROR Frame is too large: 4740180
[49.443] ERROR Frame is too large: 4740180
[49.655] ERROR Frame is too large: 4740180
[49.956] ERROR Frame is too large: 4740180
[50.286] ERROR Frame is too large: 4740180
[50.338] ERROR Frame is too large: 4740180
[50.845] ERROR Frame is too large: 4740180
[51.427] ERROR Frame is too large: 4740180
[52.457] ERROR Frame is too large: 4740180
[52.643] ERROR Frame is too large: 4740180
[52.864] ERROR Frame is too large: 4740180
[53.344] ERROR Frame is too large: 4740180
[53.420] ERROR Frame is too large: 4740180
[53.496] ERROR Frame is too large: 4740180
[54.068] ERROR Frame is too large: 4740180
[54.102] ERROR Frame is too large: 4740180
[54.262] ERROR Frame is too large: 4740180
[54.270] ERROR Frame is too large: 4740180
[54.285] ERROR Frame is too large: 4740180
Aug 24 17:07:51.12743 GC takes 2.84459590911865
[57.761] ERROR Frame is too large: 4740180
[57.816] ERROR Frame is too large: 4740180
[57.827] ERROR Frame is too large: 4740180
[57.859] ERROR Frame is too large: 4740180
[58.175] ERROR Frame is too large: 4740180
[58.721] ERROR Frame is too large: 4740180
[58.965] ERROR Frame is too large: 4740180
[59.185] ERROR Frame is too large: 4740180
[59.268] ERROR Frame is too large: 4740180
[59.468] ERROR Frame is too large: 4740180
[59.471] ERROR Frame is too large: 4740180
[59.508] ERROR Frame is too large: 4740180
[59.862] ERROR Frame is too large: 4740180
[2016-08-24 17:07:53] ERROR Frame is too large: 4740180
[0.037] ERROR Frame is too large: 4740180
[0.088] ERROR Frame is too large: 4740180
[0.172] ERROR Frame is too large: 4740180

upd. только что понаблюдал - подвисание происходит тогда, когда записывается строка без циферок, а с временем/числом

 

[Forbidden]

Попробуй отключить уникализацию запросов и результатов, а также фильтр, и посмотри сохранится ли проблема. Если проблемы с подвисанием не будет(лог тут не причем), то следующим шагом будет проверить по отдельности с фильтром и с уникализацией. Если дело в уникализации - то скорее всего узкое место - диск(медленный, или с ошибками)

 

[DeLaKroiX]

хорошо, я попробую, но апарсер у меня на ссд стоит, так что проблема диска тут скорее всего исключается. по результатам отпишусь

 

[DeLaKroiX]

вобщем не знаю, запустил парсинг линков с гугла, не релейтедов - проблемы не наблюдается. возможно потому что линков напарсило 60к, а релейтедов на момент тормозов было 1.8кк и уникализация подтормаживала всё это дело