использую парсинг релейтедов с проксями с гугла, потоков выставил 950(пробовал и 450, результат тот же), чек прокси 400 потоков, парсинг происходит так: начинается с маленькой текущей скорости, потом она повышается до примерно одинаковой величины(для 450 и для 950 потоков величина соответственно разная), потом интерфейс подвисает секунд на 10-30 и скорость сбрасывается до начальной и всё начинается по-новой. при подвисании интерфейса все кнопки тыкаются нормально, но ничего не происходит, если в это время нажать на сохранение чего-либо, то сохраняться оно будет ровно до конца этого тормоза. подскажите пожалуйста, что это может быть ? могу показать тимвьювером.
это то, что в логе, такое я уже давно заметил, но тормоза заметил вот только вчера: [59.285] ERROR Frame is too large: 4740180 [59.978] ERROR Frame is too large: 4740180 [2016-08-24 17:39:58] ERROR Frame is too large: 4740180 периодически встречается вот такая строка: Aug 24 17:00:52.03156 GC takes 4.27501249313354 пардон, забыл сказать, версия апарсера была 613, были проблемы, обновился до последней, сейчас последняя, проблемы остались. задание:
решил скинуть побольше лога отдельным сообщением, чтобы было заметно, а не редактировать предыдущее. циферки в квадратных скобках от строки к строке увеличиваются, возможно это имеет какое-то значение, потому что в предыдущем посте этого не видно: Спойлер: больше лога [0.269] ERROR Frame is too large: 4740180 [0.692] ERROR Frame is too large: 4740180 [0.929] ERROR Frame is too large: 4740180 [1.178] ERROR Frame is too large: 4740180 [1.581] ERROR Frame is too large: 4740180 [1.986] ERROR Frame is too large: 4740180 [2.100] ERROR Frame is too large: 4740180 [2.280] ERROR Frame is too large: 4740180 [2.425] ERROR Frame is too large: 4740180 [3.067] ERROR Frame is too large: 4740180 [3.091] ERROR Frame is too large: 4740180 [3.734] ERROR Frame is too large: 4740180 [3.887] ERROR Frame is too large: 4740180 [3.968] ERROR Frame is too large: 4740180 [4.664] ERROR Frame is too large: 4740180 [4.997] ERROR Frame is too large: 4740180 [5.634] ERROR Frame is too large: 4740180 [5.794] ERROR Frame is too large: 4740180 [5.916] ERROR Frame is too large: 4740180 [6.423] ERROR Frame is too large: 4740180 [6.709] ERROR Frame is too large: 4740180 [6.793] ERROR Frame is too large: 4740180 [7.364] ERROR Frame is too large: 4740180 [7.692] ERROR Frame is too large: 4740180 [8.140] ERROR Frame is too large: 4740180 [8.691] ERROR Frame is too large: 4740180 [9.077] ERROR Frame is too large: 4740180 [9.191] ERROR Frame is too large: 4740180 [9.214] ERROR Frame is too large: 4740180 [9.452] ERROR Frame is too large: 4740180 [9.456] ERROR Frame is too large: 4740180 [10.421] ERROR Frame is too large: 4740180 [10.611] ERROR Frame is too large: 4740180 [10.842] ERROR Frame is too large: 4740180 [11.757] ERROR Frame is too large: 4740180 [11.774] ERROR Frame is too large: 4740180 [12.591] ERROR Frame is too large: 4740180 [13.373] ERROR Frame is too large: 4740180 [13.666] ERROR Frame is too large: 4740180 [14.072] ERROR Frame is too large: 4740180 [14.447] ERROR Frame is too large: 4740180 [14.995] ERROR Frame is too large: 4740180 [15.373] ERROR Frame is too large: 4740180 [15.455] ERROR Frame is too large: 4740180 [15.620] ERROR Frame is too large: 4740180 [16.172] ERROR Frame is too large: 4740180 [16.201] ERROR Frame is too large: 4740180 [16.481] ERROR Frame is too large: 4740180 [17.163] ERROR Frame is too large: 4740180 [17.493] ERROR Frame is too large: 4740180 [18.240] ERROR Frame is too large: 4740180 [18.414] ERROR Frame is too large: 4740180 [18.747] ERROR Frame is too large: 4740180 [19.038] ERROR Frame is too large: 4740180 [19.088] ERROR Frame is too large: 4740180 [19.353] ERROR Frame is too large: 4740180 [19.642] ERROR Frame is too large: 4740180 [19.825] ERROR Frame is too large: 4740180 [20.137] ERROR Frame is too large: 4740180 [20.961] ERROR Frame is too large: 4740180 [21.067] ERROR Frame is too large: 4740180 [21.967] ERROR Frame is too large: 4740180 [22.230] ERROR Frame is too large: 4740180 [22.905] ERROR Frame is too large: 4740180 [22.930] ERROR Frame is too large: 4740180 [23.133] ERROR Frame is too large: 4740180 [24.023] ERROR Frame is too large: 4740180 [24.366] ERROR Frame is too large: 4740180 [24.703] ERROR Frame is too large: 4740180 [25.292] ERROR Frame is too large: 4740180 [25.753] ERROR Frame is too large: 4740180 [25.867] ERROR Frame is too large: 4740180 [26.470] ERROR Frame is too large: 4740180 [27.095] ERROR Frame is too large: 4740180 [27.159] ERROR Frame is too large: 4740180 [27.607] ERROR Frame is too large: 4740180 [27.841] ERROR Frame is too large: 4740180 [27.845] ERROR Frame is too large: 4740180 [28.300] ERROR Frame is too large: 4740180 [28.335] ERROR Frame is too large: 4740180 [28.428] ERROR Frame is too large: 4740180 [28.605] ERROR Frame is too large: 4740180 [28.717] ERROR Frame is too large: 4740180 [29.281] ERROR Frame is too large: 4740180 [29.519] ERROR Frame is too large: 4740180 [30.013] ERROR Frame is too large: 4740180 [30.016] ERROR Frame is too large: 4740180 [30.094] ERROR Frame is too large: 4740180 [30.167] ERROR Frame is too large: 4740180 [30.543] ERROR Frame is too large: 4740180 [31.086] ERROR Frame is too large: 4740180 [31.246] ERROR Frame is too large: 4740180 [31.617] ERROR Frame is too large: 4740180 [32.157] ERROR Frame is too large: 4740180 [32.834] ERROR Frame is too large: 4740180 [32.912] ERROR Frame is too large: 4740180 [33.715] ERROR Frame is too large: 4740180 [34.033] ERROR Frame is too large: 4740180 [34.256] ERROR Frame is too large: 4740180 [34.534] ERROR Frame is too large: 4740180 [34.672] ERROR Frame is too large: 4740180 [35.246] ERROR Frame is too large: 4740180 [35.533] ERROR Frame is too large: 4740180 [36.232] ERROR Frame is too large: 4740180 [36.319] ERROR Frame is too large: 4740180 [36.442] ERROR Frame is too large: 4740180 [36.559] ERROR Frame is too large: 4740180 [36.935] ERROR Frame is too large: 4740180 [37.518] ERROR Frame is too large: 4740180 [37.521] ERROR Frame is too large: 4740180 [37.646] ERROR Frame is too large: 4740180 [37.700] ERROR Frame is too large: 4740180 [38.417] ERROR Frame is too large: 4740180 [38.468] ERROR Frame is too large: 4740180 [38.680] ERROR Frame is too large: 4740180 [39.084] ERROR Frame is too large: 4740180 [39.150] ERROR Frame is too large: 4740180 [39.169] ERROR Frame is too large: 4740180 [39.573] ERROR Frame is too large: 4740180 [40.074] ERROR Frame is too large: 4740180 [40.297] ERROR Frame is too large: 4740180 [40.454] ERROR Frame is too large: 4740180 [40.901] ERROR Frame is too large: 4740180 [41.091] ERROR Frame is too large: 4740180 [41.729] ERROR Frame is too large: 4740180 [41.857] ERROR Frame is too large: 4740180 [42.135] ERROR Frame is too large: 4740180 [42.669] ERROR Frame is too large: 4740180 [42.765] ERROR Frame is too large: 4740180 [42.853] ERROR Frame is too large: 4740180 [42.961] ERROR Frame is too large: 4740180 [43.555] ERROR Frame is too large: 4740180 [43.940] ERROR Frame is too large: 4740180 [43.983] ERROR Frame is too large: 4740180 [44.386] ERROR Frame is too large: 4740180 [44.696] ERROR Frame is too large: 4740180 [44.756] ERROR Frame is too large: 4740180 [45.076] ERROR Frame is too large: 4740180 [45.147] ERROR Frame is too large: 4740180 [45.292] ERROR Frame is too large: 4740180 [45.392] ERROR Frame is too large: 4740180 [45.697] ERROR Frame is too large: 4740180 [46.151] ERROR Frame is too large: 4740180 [46.467] ERROR Frame is too large: 4740180 [46.946] ERROR Frame is too large: 4740180 [47.299] ERROR Frame is too large: 4740180 [47.798] ERROR Frame is too large: 4740180 [48.026] ERROR Frame is too large: 4740180 [48.297] ERROR Frame is too large: 4740180 [49.683] ERROR Frame is too large: 4740180 [49.787] ERROR Frame is too large: 4740180 [50.287] ERROR Frame is too large: 4740180 [50.964] ERROR Frame is too large: 4740180 [51.063] ERROR Frame is too large: 4740180 [51.127] ERROR Frame is too large: 4740180 [51.179] ERROR Frame is too large: 4740180 [52.130] ERROR Frame is too large: 4740180 [52.387] ERROR Frame is too large: 4740180 [52.986] ERROR Frame is too large: 4740180 [53.545] ERROR Frame is too large: 4740180 [53.678] ERROR Frame is too large: 4740180 [54.464] ERROR Frame is too large: 4740180 [54.491] ERROR Frame is too large: 4740180 Aug 24 17:06:51.29689 GC takes 3.06770133972168 [58.417] ERROR Frame is too large: 4740180 [58.480] ERROR Frame is too large: 4740180 [58.490] ERROR Frame is too large: 4740180 [59.547] ERROR Frame is too large: 4740180 [2016-08-24 17:06:53] ERROR Frame is too large: 4740180 [0.100] ERROR Frame is too large: 4740180 [0.326] ERROR Frame is too large: 4740180 [1.232] ERROR Frame is too large: 4740180 [1.544] ERROR Frame is too large: 4740180 [3.491] ERROR Frame is too large: 4740180 [3.847] ERROR Frame is too large: 4740180 [4.165] ERROR Frame is too large: 4740180 [4.917] ERROR Frame is too large: 4740180 [5.076] ERROR Frame is too large: 4740180 [5.176] ERROR Frame is too large: 4740180 [5.748] ERROR Frame is too large: 4740180 [5.759] ERROR Frame is too large: 4740180 [6.694] ERROR Frame is too large: 4740180 [7.139] ERROR Frame is too large: 4740180 [7.188] ERROR Frame is too large: 4740180 [7.724] ERROR Frame is too large: 4740180 [8.096] ERROR Frame is too large: 4740180 [8.474] ERROR Frame is too large: 4740180 [8.811] ERROR Frame is too large: 4740180 [9.222] ERROR Frame is too large: 4740180 [9.581] ERROR Frame is too large: 4740180 [9.972] ERROR Frame is too large: 4740180 [10.014] ERROR Frame is too large: 4740180 [10.304] ERROR Frame is too large: 4740180 [10.984] ERROR Frame is too large: 4740180 [11.155] ERROR Frame is too large: 4740180 [11.174] ERROR Frame is too large: 4740180 [11.811] ERROR Frame is too large: 4740180 [12.197] ERROR Frame is too large: 4740180 [12.591] ERROR Frame is too large: 4740180 [13.088] ERROR Frame is too large: 4740180 [13.159] ERROR Frame is too large: 4740180 [13.470] ERROR Frame is too large: 4740180 [13.614] ERROR Frame is too large: 4740180 [13.713] ERROR Frame is too large: 4740180 [14.174] ERROR Frame is too large: 4740180 [14.216] ERROR Frame is too large: 4740180 [14.843] ERROR Frame is too large: 4740180 [14.940] ERROR Frame is too large: 4740180 [15.578] ERROR Frame is too large: 4740180 [15.716] ERROR Frame is too large: 4740180 [16.164] ERROR Frame is too large: 4740180 [17.197] ERROR Frame is too large: 4740180 [17.327] ERROR Frame is too large: 4740180 [17.573] ERROR Frame is too large: 4740180 [17.831] ERROR Frame is too large: 4740180 [18.536] ERROR Frame is too large: 4740180 [18.703] ERROR Frame is too large: 4740180 [18.983] ERROR Frame is too large: 4740180 [19.202] ERROR Frame is too large: 4740180 [19.851] ERROR Frame is too large: 4740180 [20.081] ERROR Frame is too large: 4740180 [20.087] ERROR Frame is too large: 4740180 [20.345] ERROR Frame is too large: 4740180 [20.654] ERROR Frame is too large: 4740180 [21.303] ERROR Frame is too large: 4740180 [21.374] ERROR Frame is too large: 4740180 [21.639] ERROR Frame is too large: 4740180 [22.094] ERROR Frame is too large: 4740180 [22.893] ERROR Frame is too large: 4740180 [23.563] ERROR Frame is too large: 4740180 [23.658] ERROR Frame is too large: 4740180 [23.714] ERROR Frame is too large: 4740180 [23.956] ERROR Frame is too large: 4740180 [24.339] ERROR Frame is too large: 4740180 [24.469] ERROR Frame is too large: 4740180 [24.925] ERROR Frame is too large: 4740180 [25.364] ERROR Frame is too large: 4740180 [25.983] ERROR Frame is too large: 4740180 [26.109] ERROR Frame is too large: 4740180 [26.829] ERROR Frame is too large: 4740180 [26.881] ERROR Frame is too large: 4740180 [27.653] ERROR Frame is too large: 4740180 [27.782] ERROR Frame is too large: 4740180 [27.911] ERROR Frame is too large: 4740180 [28.184] ERROR Frame is too large: 4740180 [28.807] ERROR Frame is too large: 4740180 [29.225] ERROR Frame is too large: 4740180 [29.292] ERROR Frame is too large: 4740180 [29.350] ERROR Frame is too large: 4740180 [29.532] ERROR Frame is too large: 4740180 [30.128] ERROR Frame is too large: 4740180 [30.256] ERROR Frame is too large: 4740180 [30.399] ERROR Frame is too large: 4740180 [31.059] ERROR Frame is too large: 4740180 [31.296] ERROR Frame is too large: 4740180 [31.644] ERROR Frame is too large: 4740180 [31.806] ERROR Frame is too large: 4740180 [32.049] ERROR Frame is too large: 4740180 [32.566] ERROR Frame is too large: 4740180 [32.771] ERROR Frame is too large: 4740180 [33.405] ERROR Frame is too large: 4740180 [33.597] ERROR Frame is too large: 4740180 [33.656] ERROR Frame is too large: 4740180 [34.187] ERROR Frame is too large: 4740180 [34.451] ERROR Frame is too large: 4740180 [34.951] ERROR Frame is too large: 4740180 [35.364] ERROR Frame is too large: 4740180 [35.954] ERROR Frame is too large: 4740180 [36.232] ERROR Frame is too large: 4740180 [36.607] ERROR Frame is too large: 4740180 [37.251] ERROR Frame is too large: 4740180 [37.382] ERROR Frame is too large: 4740180 [37.743] ERROR Frame is too large: 4740180 [37.968] ERROR Frame is too large: 4740180 [38.060] ERROR Frame is too large: 4740180 [38.543] ERROR Frame is too large: 4740180 [38.556] ERROR Frame is too large: 4740180 [38.667] ERROR Frame is too large: 4740180 [38.715] ERROR Frame is too large: 4740180 [39.878] ERROR Frame is too large: 4740180 [39.879] ERROR Frame is too large: 4740180 [40.414] ERROR Frame is too large: 4740180 [40.674] ERROR Frame is too large: 4740180 [40.776] ERROR Frame is too large: 4740180 [40.825] ERROR Frame is too large: 4740180 [40.935] ERROR Frame is too large: 4740180 [40.995] ERROR Frame is too large: 4740180 [41.535] ERROR Frame is too large: 4740180 [42.197] ERROR Frame is too large: 4740180 [42.686] ERROR Frame is too large: 4740180 [42.878] ERROR Frame is too large: 4740180 [43.507] ERROR Frame is too large: 4740180 [43.618] ERROR Frame is too large: 4740180 [43.892] ERROR Frame is too large: 4740180 [44.019] ERROR Frame is too large: 4740180 [44.268] ERROR Frame is too large: 4740180 [44.405] ERROR Frame is too large: 4740180 [44.932] ERROR Frame is too large: 4740180 [45.165] ERROR Frame is too large: 4740180 [45.963] ERROR Frame is too large: 4740180 [45.976] ERROR Frame is too large: 4740180 [46.326] ERROR Frame is too large: 4740180 [46.629] ERROR Frame is too large: 4740180 [46.948] ERROR Frame is too large: 4740180 [47.315] ERROR Frame is too large: 4740180 [47.358] ERROR Frame is too large: 4740180 [47.580] ERROR Frame is too large: 4740180 [48.056] ERROR Frame is too large: 4740180 [48.221] ERROR Frame is too large: 4740180 [48.341] ERROR Frame is too large: 4740180 [48.639] ERROR Frame is too large: 4740180 [48.782] ERROR Frame is too large: 4740180 [48.815] ERROR Frame is too large: 4740180 [49.122] ERROR Frame is too large: 4740180 [49.181] ERROR Frame is too large: 4740180 [49.443] ERROR Frame is too large: 4740180 [49.655] ERROR Frame is too large: 4740180 [49.956] ERROR Frame is too large: 4740180 [50.286] ERROR Frame is too large: 4740180 [50.338] ERROR Frame is too large: 4740180 [50.845] ERROR Frame is too large: 4740180 [51.427] ERROR Frame is too large: 4740180 [52.457] ERROR Frame is too large: 4740180 [52.643] ERROR Frame is too large: 4740180 [52.864] ERROR Frame is too large: 4740180 [53.344] ERROR Frame is too large: 4740180 [53.420] ERROR Frame is too large: 4740180 [53.496] ERROR Frame is too large: 4740180 [54.068] ERROR Frame is too large: 4740180 [54.102] ERROR Frame is too large: 4740180 [54.262] ERROR Frame is too large: 4740180 [54.270] ERROR Frame is too large: 4740180 [54.285] ERROR Frame is too large: 4740180 Aug 24 17:07:51.12743 GC takes 2.84459590911865 [57.761] ERROR Frame is too large: 4740180 [57.816] ERROR Frame is too large: 4740180 [57.827] ERROR Frame is too large: 4740180 [57.859] ERROR Frame is too large: 4740180 [58.175] ERROR Frame is too large: 4740180 [58.721] ERROR Frame is too large: 4740180 [58.965] ERROR Frame is too large: 4740180 [59.185] ERROR Frame is too large: 4740180 [59.268] ERROR Frame is too large: 4740180 [59.468] ERROR Frame is too large: 4740180 [59.471] ERROR Frame is too large: 4740180 [59.508] ERROR Frame is too large: 4740180 [59.862] ERROR Frame is too large: 4740180 [2016-08-24 17:07:53] ERROR Frame is too large: 4740180 [0.037] ERROR Frame is too large: 4740180 [0.088] ERROR Frame is too large: 4740180 [0.172] ERROR Frame is too large: 4740180 upd. только что понаблюдал - подвисание происходит тогда, когда записывается строка без циферок, а с временем/числом
Попробуй отключить уникализацию запросов и результатов, а также фильтр, и посмотри сохранится ли проблема. Если проблемы с подвисанием не будет(лог тут не причем), то следующим шагом будет проверить по отдельности с фильтром и с уникализацией. Если дело в уникализации - то скорее всего узкое место - диск(медленный, или с ошибками)
хорошо, я попробую, но апарсер у меня на ссд стоит, так что проблема диска тут скорее всего исключается. по результатам отпишусь
вобщем не знаю, запустил парсинг линков с гугла, не релейтедов - проблемы не наблюдается. возможно потому что линков напарсило 60к, а релейтедов на момент тормозов было 1.8кк и уникализация подтормаживала всё это дело
