Node.js High Memory Usage?

scrapefun

A-Parser Enterprise License
A-Parser Enterprise
For the past 2-3 weeks I've noticed A-parser performance will slowly get worse until rebooting server and then after reboot performance would improve 40-50%

Today, I noticed Node.js was using over 8GB memory for each A-parser instance with 4 instances of A-parser running on the server using around 32GB of memory.

Instead of rebooting, I just restarted each A-parser instance and performance improved 40-50% again and Node.js memory usage dropped considerably.

I'm using NET::HTTP and Google Modern parsers with A-parser versions 1.2.1222 and 1.2.1226 . The instances using NET::HTTP seem to cause Node.js to consume more memory but Google Parser also causes Node.js to use very high memory.

Not sure if this is a bug/memory leak or just normal behavior but I didn't always have this problem when running the same number of instances/threads/parsers/etc...

LWR_Recording.png



After restarting A-parser:

LWR_Recording.png
 
Last edited:
Please send us the presets on which this problem occurs.
And also tell the OS version and after what time of work of these presets there is an increase in the amount of consumed memory.
 
1) Windows Server 2012 R2. I am using multiple server providers and they all have this issue so I don't think it is server related. Also tried disabling malware/virus protection but same result.

2) Tasks run fine for 4-6 hours after restarting A-parser before memory usage gets so high performance begins to slow down. Memory usage slowly increases as soon as A-parser restarts and tasks start running again. Also, pausing or stopping tasks has no effect on memory usage.

3) There are 4-5 presets I use and this happens with all of them. But they all use either Google Modern or NET::HTTP. I have included 2 samples below.

4) Attached is a portion of a log file from when I pause the currently running task to successfully restarting A-parser. Maybe it will help.


eJyNVm1z2jgQ/isaTRuSG2qbJuFyZJiblAvXXhPIBTL9ABlGtRfQIUtGkkNylP/e
lTDG6VyT8zCMtdpnXx7trrymlpmFudFgwBraGq1p5t9pi/Yuh62Pw+ENrdOMaQPa
7Y9oD2xrJ09gynJhaX1N7VMGCFIPoDVPADd5guuZUklH+fUDEzmqjN5H0f3m54gc
Pb1jM5B2j6HX6l8uBAtPg4gcfuEyUStDekPSiILonKCgeXJOHpsnR+QiywR8ga+f
uQ1Pj38Njpvk8PPH4fVVnQi+APInxAt1RDpzrVIIz86CKDg5fn8SNM6aZMCmTPMC
Rl+IMdPq8UmD1RxMJcpGFEWvwr4yKSGJBTCZZxXwi0jLU1B5lZHmi/pIone2B1id
wwsADcscjE1AsKf/G5T30JkjoaArmF+eYTLLlTQ7SOy0O0ra58c7t6kg3EJqYpVB
e0zH1C+dEVzNrc1aYWgQnbJA6Vk4AKbj+S0YrD5zg9WCAMHkDJVBjimaZknCnWsm
aGv9zO81s/F86MNrbDb393VM3tvpKp0yV/mjt+RTl2SNgMupCkwex2AMabdJ43ws
xxY3EmYZmRAMFAUUpZe9P/D/bnBJ0PwcXxOuSdsv0Io9RHb1UyDzlITktyg6Im/v
ael4wB5gqNDxlAvYi7u46rEUXo8I97YOxINw0gilBJ+xrUREiueVyArNbUJjW8N2
zCaarcLcvKthzg4+IbXQvW+xi5UTBO4Qa+eI2+a2P4Atr9vpYbHIJlPGRXg3mLxB
GsHlHEy9yuFRYB9dXfw03cjHNKZvdp4r/G/dul/lTO8kX/oSkwp3HApbtou9v2fb
m9qd/cj5XiI/ZWrfSK45evgvjmuuMg2W5mq1CnDUzTCVWAX5IjS+QH9ftj1Njp+D
bGXa0cFMtHNzgHS3T6PaPnCMOPeh/r0NcduwdWqwNLpIFyTlxpQJgzvYH5pZpftF
h2GVK3khxBU8gCjxPuQPORcJzu+LKWI+FbjS0DOV/o8mNiWVVUc4CVYaIyiN+NWH
/nUhQFSirtRsR7zgKbe4Nh2VS+sHS50uALLyfHpOLVUaSjeF5cI7XlEZSDdD9g2C
s74UPcvCX2RG5Tp2pv0WqhQjyeDYlrNBJrgrNRyUrOCDnjsl5YpgsXK3G9f0frOv
pYp1FMZKTvmsX8zEQqhzOcQLtS87KsV7yBEkcyHqbiDf7rv6whSn6Rb7TH8Ed7wL
DKy8ZzE8Jcxfg23OmeYY7akLMMUjqXotTMZMiLvbq+oOrfSB9z7aLUNkwnIEhgt4
WimdmAk26WXvOPJt6XLGfp0pJNMnu9xVJKWOpeIjofyWWFc+FVrrDZbEPziqvY5z
7DRQhvwZPG83i78DIGrZAg==



eJx9VW1v2zYQ/isC0QLb4Mpyhw2oin5I3XhtkcRZHH+yA4GVTjZrimRIyo5h+L/v
SOotXVrEQMzjcy+8e+7xiVhqduZWgwFrSLo6EeW/k5T8I+WGQ3YtC9CCjIii2oB2
oBVZXKZpuE/TDmDB2MzkmirQZHQi9qgA48g9aM0KQAQr8Ky0fDpOt5DvHIzsKa8d
7A9y/rlPISvKxAB9OBzijS8gzmVc737lvOEDx+XiV9DtEAqCnB8eRgQbUnNrZlJX
1DXmlZrEim7AxElcUEtJB1nQPdxLhJSMQ2+e4emGVi4mdlBlmh7Gq9fRl1mEkZgo
ZWzqPAdjog8fokn0+gEvl4vL6Jra7ftoTWrzZk0yf0S4/e2xBn2MRV1F4+hdkvwe
ZWsydgh/4U7x1lZ8TUKoy5tP+AXLoUXBLJOC8vCWMEtfUkmxxmIc/sX2ySL8pyUm
GO6Vz7Vei2EC9xl0bCnYo++kkHjjHBiYmZZV3yEfpe3sioSoBEPU3vff4EPSknID
I2KwwTNf4483zIKmVuq5ci9E+4lIccH5FeyB9zAf/2PNOHLWXJTo9KVxfBky/1+M
c/e8YSrk0UFjDV0Uf/o4v+69CnklN20zOKuYxbOZylo4UiVo3AGormc3DlZJDV2a
JnKTHXdWgXCk7Yl2oXrTs2dgb/uxPDfmUpRsM28WoUXW4h6FYS6mslIc3LtEzTmO
xcBdT+oL04zBHfoCf3Se+hRujaGk6Ou0Qkpuvi5CqUozXJq/XIEVdnKYtQmZU86X
d1fDGzKglM++ao9jVCrL0HG8g+NB6sJky0V2efP2nSe2ezO1sJFINP/Yx5ZKBPe9
07lOE08vqR1an8ldegq6psE2sSZJ4maKDfMXJLW6btlVdlIS+D4iwRIG1EhMIy3e
+xsVAoqcAxW1athiWQWydui/E08BjGVsAZweG8QzpXUKiySEPcvBj8LsrFRe1zeQ
BxZOHDHFzuCLnBVnkoy8PoR2485aFym8xNldDXiH02l/IIS0JQYrWpQoVb9ajYy/
IN+YWIckqNZepUdOjL0I453EieESBMTSMp7dwZQqm2/p26z7xerppdvbu24efw7N
t9TggP1YmtJwTvhnwqaF0gsp7D3dwSLYOyjK6GegYYOI7+m3epMJKeCJGWuy8Mp+
YeHJahoG7Sh2Ri34bm4Dyxx1HcfQhhsY0kzO/wGIoJWb
 

Attachments

Last edited:
Also specify in how many threads you run these tasks and send files with queries for each of these tasks (you can send it to PM).
 
250-300 threads. Sending sample now. You can use this query file for either task preset.
 
Problem appears to be fixed! Running 6 hours with no increase in memory usage. Thanks for the help. Performance is much better!
 
Back
Top