1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
  2. Join our Telegram chat: https://t.me/a_parser_en
    Dismiss Notice

Node.js High Memory Usage?

Discussion in 'A-Parser Support Forum' started by scrapefun, Jun 7, 2021.

  1. scrapefun

    scrapefun A-Parser Enterprise License
    A-Parser Enterprise

    Joined:
    Feb 24, 2015
    Messages:
    145
    Likes Received:
    20
    For the past 2-3 weeks I've noticed A-parser performance will slowly get worse until rebooting server and then after reboot performance would improve 40-50%

    Today, I noticed Node.js was using over 8GB memory for each A-parser instance with 4 instances of A-parser running on the server using around 32GB of memory.

    Instead of rebooting, I just restarted each A-parser instance and performance improved 40-50% again and Node.js memory usage dropped considerably.

    I'm using NET::HTTP and Google Modern parsers with A-parser versions 1.2.1222 and 1.2.1226 . The instances using NET::HTTP seem to cause Node.js to consume more memory but Google Parser also causes Node.js to use very high memory.

    Not sure if this is a bug/memory leak or just normal behavior but I didn't always have this problem when running the same number of instances/threads/parsers/etc...

    [​IMG]


    After restarting A-parser:

    [​IMG]
     
    #1 scrapefun, Jun 7, 2021
    Last edited: Jun 7, 2021
  2. Support

    Support Administrator
    Staff Member A-Parser Enterprise

    Joined:
    Mar 16, 2012
    Messages:
    3,401
    Likes Received:
    1,835
    Please send us the presets on which this problem occurs.
    And also tell the OS version and after what time of work of these presets there is an increase in the amount of consumed memory.
     
  3. scrapefun

    scrapefun A-Parser Enterprise License
    A-Parser Enterprise

    Joined:
    Feb 24, 2015
    Messages:
    145
    Likes Received:
    20
    1) Windows Server 2012 R2. I am using multiple server providers and they all have this issue so I don't think it is server related. Also tried disabling malware/virus protection but same result.

    2) Tasks run fine for 4-6 hours after restarting A-parser before memory usage gets so high performance begins to slow down. Memory usage slowly increases as soon as A-parser restarts and tasks start running again. Also, pausing or stopping tasks has no effect on memory usage.

    3) There are 4-5 presets I use and this happens with all of them. But they all use either Google Modern or NET::HTTP. I have included 2 samples below.

    4) Attached is a portion of a log file from when I pause the currently running task to successfully restarting A-parser. Maybe it will help.


    eJyNVm1z2jgQ/isaTRuSG2qbJuFyZJiblAvXXhPIBTL9ABlGtRfQIUtGkkNylP/e
    lTDG6VyT8zCMtdpnXx7trrymlpmFudFgwBraGq1p5t9pi/Yuh62Pw+ENrdOMaQPa
    7Y9oD2xrJ09gynJhaX1N7VMGCFIPoDVPADd5guuZUklH+fUDEzmqjN5H0f3m54gc
    Pb1jM5B2j6HX6l8uBAtPg4gcfuEyUStDekPSiILonKCgeXJOHpsnR+QiywR8ga+f
    uQ1Pj38Njpvk8PPH4fVVnQi+APInxAt1RDpzrVIIz86CKDg5fn8SNM6aZMCmTPMC
    Rl+IMdPq8UmD1RxMJcpGFEWvwr4yKSGJBTCZZxXwi0jLU1B5lZHmi/pIone2B1id
    wwsADcscjE1AsKf/G5T30JkjoaArmF+eYTLLlTQ7SOy0O0ra58c7t6kg3EJqYpVB
    e0zH1C+dEVzNrc1aYWgQnbJA6Vk4AKbj+S0YrD5zg9WCAMHkDJVBjimaZknCnWsm
    aGv9zO81s/F86MNrbDb393VM3tvpKp0yV/mjt+RTl2SNgMupCkwex2AMabdJ43ws
    xxY3EmYZmRAMFAUUpZe9P/D/bnBJ0PwcXxOuSdsv0Io9RHb1UyDzlITktyg6Im/v
    ael4wB5gqNDxlAvYi7u46rEUXo8I97YOxINw0gilBJ+xrUREiueVyArNbUJjW8N2
    zCaarcLcvKthzg4+IbXQvW+xi5UTBO4Qa+eI2+a2P4Atr9vpYbHIJlPGRXg3mLxB
    GsHlHEy9yuFRYB9dXfw03cjHNKZvdp4r/G/dul/lTO8kX/oSkwp3HApbtou9v2fb
    m9qd/cj5XiI/ZWrfSK45evgvjmuuMg2W5mq1CnDUzTCVWAX5IjS+QH9ftj1Njp+D
    bGXa0cFMtHNzgHS3T6PaPnCMOPeh/r0NcduwdWqwNLpIFyTlxpQJgzvYH5pZpftF
    h2GVK3khxBU8gCjxPuQPORcJzu+LKWI+FbjS0DOV/o8mNiWVVUc4CVYaIyiN+NWH
    /nUhQFSirtRsR7zgKbe4Nh2VS+sHS50uALLyfHpOLVUaSjeF5cI7XlEZSDdD9g2C
    s74UPcvCX2RG5Tp2pv0WqhQjyeDYlrNBJrgrNRyUrOCDnjsl5YpgsXK3G9f0frOv
    pYp1FMZKTvmsX8zEQqhzOcQLtS87KsV7yBEkcyHqbiDf7rv6whSn6Rb7TH8Ed7wL
    DKy8ZzE8Jcxfg23OmeYY7akLMMUjqXotTMZMiLvbq+oOrfSB9z7aLUNkwnIEhgt4
    WimdmAk26WXvOPJt6XLGfp0pJNMnu9xVJKWOpeIjofyWWFc+FVrrDZbEPziqvY5z
    7DRQhvwZPG83i78DIGrZAg==



    eJx9VW1v2zYQ/isC0QLb4Mpyhw2oin5I3XhtkcRZHH+yA4GVTjZrimRIyo5h+L/v
    SOotXVrEQMzjcy+8e+7xiVhqduZWgwFrSLo6EeW/k5T8I+WGQ3YtC9CCjIii2oB2
    oBVZXKZpuE/TDmDB2MzkmirQZHQi9qgA48g9aM0KQAQr8Ky0fDpOt5DvHIzsKa8d
    7A9y/rlPISvKxAB9OBzijS8gzmVc737lvOEDx+XiV9DtEAqCnB8eRgQbUnNrZlJX
    1DXmlZrEim7AxElcUEtJB1nQPdxLhJSMQ2+e4emGVi4mdlBlmh7Gq9fRl1mEkZgo
    ZWzqPAdjog8fokn0+gEvl4vL6Jra7ftoTWrzZk0yf0S4/e2xBn2MRV1F4+hdkvwe
    ZWsydgh/4U7x1lZ8TUKoy5tP+AXLoUXBLJOC8vCWMEtfUkmxxmIc/sX2ySL8pyUm
    GO6Vz7Vei2EC9xl0bCnYo++kkHjjHBiYmZZV3yEfpe3sioSoBEPU3vff4EPSknID
    I2KwwTNf4483zIKmVuq5ci9E+4lIccH5FeyB9zAf/2PNOHLWXJTo9KVxfBky/1+M
    c/e8YSrk0UFjDV0Uf/o4v+69CnklN20zOKuYxbOZylo4UiVo3AGormc3DlZJDV2a
    JnKTHXdWgXCk7Yl2oXrTs2dgb/uxPDfmUpRsM28WoUXW4h6FYS6mslIc3LtEzTmO
    xcBdT+oL04zBHfoCf3Se+hRujaGk6Ou0Qkpuvi5CqUozXJq/XIEVdnKYtQmZU86X
    d1fDGzKglM++ao9jVCrL0HG8g+NB6sJky0V2efP2nSe2ezO1sJFINP/Yx5ZKBPe9
    07lOE08vqR1an8ldegq6psE2sSZJ4maKDfMXJLW6btlVdlIS+D4iwRIG1EhMIy3e
    +xsVAoqcAxW1athiWQWydui/E08BjGVsAZweG8QzpXUKiySEPcvBj8LsrFRe1zeQ
    BxZOHDHFzuCLnBVnkoy8PoR2485aFym8xNldDXiH02l/IIS0JQYrWpQoVb9ajYy/
    IN+YWIckqNZepUdOjL0I453EieESBMTSMp7dwZQqm2/p26z7xerppdvbu24efw7N
    t9TggP1YmtJwTvhnwqaF0gsp7D3dwSLYOyjK6GegYYOI7+m3epMJKeCJGWuy8Mp+
    YeHJahoG7Sh2Ri34bm4Dyxx1HcfQhhsY0kzO/wGIoJWb
     

    Attached Files:

    #3 scrapefun, Jun 8, 2021
    Last edited: Jun 8, 2021
  4. Support

    Support Administrator
    Staff Member A-Parser Enterprise

    Joined:
    Mar 16, 2012
    Messages:
    3,401
    Likes Received:
    1,835
    Also specify in how many threads you run these tasks and send files with queries for each of these tasks (you can send it to PM).
     
  5. scrapefun

    scrapefun A-Parser Enterprise License
    A-Parser Enterprise

    Joined:
    Feb 24, 2015
    Messages:
    145
    Likes Received:
    20
    250-300 threads. Sending sample now. You can use this query file for either task preset.
     
  6. Support

    Support Administrator
    Staff Member A-Parser Enterprise

    Joined:
    Mar 16, 2012
    Messages:
    3,401
    Likes Received:
    1,835
    Ok, let us test this problem.
     
    scrapefun likes this.
  7. Support

    Support Administrator
    Staff Member A-Parser Enterprise

    Joined:
    Mar 16, 2012
    Messages:
    3,401
    Likes Received:
    1,835
    @scrapefun Try 1.2.1239, the memory leak is most likely fixed.
     
    scrapefun likes this.
  8. scrapefun

    scrapefun A-Parser Enterprise License
    A-Parser Enterprise

    Joined:
    Feb 24, 2015
    Messages:
    145
    Likes Received:
    20
    Problem appears to be fixed! Running 6 hours with no increase in memory usage. Thanks for the help. Performance is much better!
     
    Support likes this.

Share This Page