Suspect memory leak in v6.10.1

[stefanosala]

• Version: v6.10.1
• Platform: Linux f227befc-7ce9-41a4-9b09-363eac11b499 3.13.0-112-generic Added DS_Store to gitignore #159-Ubuntu SMP Fri Mar 3 15:26:07 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
• Subsystem:

Hi there, we noticed a weird behaviour after upgrading to v6.10.1 from v6.10.0. Basically we see memory accumulating until Heroku restarts the process.

You can better notice the behaviour in this graph:

v1314 is the upgrade to v6.10.1, v1315 is the rollback to v6.10.0.

We start the process with this command: NODE_ICU_DATA=node_modules/full-icu node --max-old-space-size=900 lib/processes/oneInstance.js.

Help? :) Please let me know if and how we can provide more useful info.

Thanks a lot.
Stefano

[stefanosala]

We had a similar behaviour when we tried node v7, I'm not sure if that helps :)

[cjihrig]

@stefanosala do you have any code you can share?

[stefanosala]

@cjihrig we haven't isolated the issue yet, I'm sorry :(

[joyeecheung]

You mentioned this behavior also presents in v7, can you provide the specific version of that release? Cross comparing it with #11759 we might be able to find the related commit.

[lpinca]

Possibly related: #12019.

[stefanosala]

Yep, we had the same issue with v7.6.0.

[SuperTux88]

I have the same behaviour after upgrading to 6.10.1 with camo, started with this command: /usr/bin/node /srv/camo/node_modules/.bin/coffee server.coffee.

[dvfeinblum]

I also see similar issues. My child processes are utilizing more ram than they are provisioned (using the --max_old_space_size flag) and I'm seeing a linear increase in memory usage.

Edit: Bit more information. I am using child_process.exec() to fire off my workers and switching to 6.9 has fixed the problem. Memory is no longer increasing linearly and I'm only using 15gb as opposed to maxing out the 30 I have at my disposal. Scripts also seem to be respecting the 1.5 gb I'm giving each of them now.

[kmikiy]

I've also had the same issue
6.10.0 works fine

6.10.1 starts leaking

7.5.0 also works fine

7.6.0 and up starts leaking

Same code was running all 4 times, only changed the node version.

[joyeecheung]

Excluding benchmark, doc, meta, test, tools and build, these are the PRs both added in v6.10.1 and v7.6.0 (obtained by scripting the OP of #11185 and #11759, I might be missing something though..)

• crypto: Remove expired certs from CNNIC whitelist Distrust certs issued after 00:00:00 Oct. 21, 2016 by StartCom and WoSign  #9469
• crypto: add cert check issued by StartCom/WoSign Distrust certs issued after 00:00:00 Oct. 21, 2016 by StartCom and WoSign  #9469
• dgram: remove this aliases dgram: remove this aliases #11243
• lib: replace \u2019 with regular ascii quote src: support UTF-8 in compiled-in JS source files #11129
• readline: update 6 comparions to strict readline: update 6 comparisons to strict #11078
• src: support UTF-8 in compiled-in JS source files src: support UTF-8 in compiled-in JS source files #11129
• src: fix delete operator on vm context src: fix delete operator on vm context #11266
• src: unconsume stream fix in internal http impl src: unconsume stream fix #11015
• stream: move legacy to lib/internal dir stream: move legacy to lib/internal dir #8197
• timer: remove duplicated word in comment remove duplicated word in comment #11323

[MylesBorins]

@joyeecheung thank you so much for compiling that list

@nodejs/lts @nodejs/collaborators do any of the above prs stand out to you?

[addaleax]

src: unconsume stream fix in internal http impl #11015

Seems like the most likely candidate from the list by far … /cc @Kasher @jasnell @indutny

[addaleax]

Maybe strike that, I’ve just seen #12089, that looks like a good candidate for fixing the leak.

[joegoldbeck]

We ran into a memory leak in 4.8.1 (was not there in 4.8.0). These may be related.

[shigeki]

I confirmed that #12089 fixes a memory leak bug and I'm sure that it resolves this issue.

[MylesBorins]

As it would appear that the memory leak is also appear on v4.x I've gone ahead and made a list of commits that appears in v4.8.1, v6.10.1, and v7.6.0. It is much smaller

• crypto: Remove expired certs from CNNIC whitelist (Shigeki Ohtsu) Distrust certs issued after 00:00:00 Oct. 21, 2016 by StartCom and WoSign  #9469
• crypto: add cert check issued by StartCom/WoSign (Shigeki Ohtsu) Distrust certs issued after 00:00:00 Oct. 21, 2016 by StartCom and WoSign  #9469
• readline: update 6 comparions to strict (Umair Ishaq) readline: update 6 comparisons to strict #11078 not likely to cause leak
• src: unconsume stream fix in internal http impl (Roee Kasher) src: unconsume stream fix #11015

edit: @nodejs/streams do you think #11015 could have caused a leak?

[shigeki]

TLS and HTTP client memory leaks were fixed in a6f9494.
I'm sure it fixes this. Please test it.

I apology my bug in my commit and thanks for everyone helping and identifying the issues.

[stefanosala]

@shigeki thanks a lot and thanks to everybody else involved, we appreciate a lot! :)

I'll test the new release asap.

[dvfeinblum]

Yeah, thanks a bunch for fixing this! We'll give this patch a go next time we run our job.

[stefanosala]

Stats are looking good with v7.8.0, thanks again!