I've decided to try to bisect it a bit. Since v0.166.4 is from 30 Aug 2023 and v10.9.0 is from 27 Feb 2025, which means the middle happens to be on 2024-05-29:

It appears v0.179.13 is free from the memory leak based on one day of metrics:

Nest closest release between that and v10.9.0 is v3.0.0 from 2024-10-07.

Deployed: https://ci.infra.status.im/job/status-go/job/deploy-notify-prod/17/

It appears v3.0.0 is free from the memory leak based on one day of metrics:

Nest closest release between that and v10.9.0 is v8.0.0 from 2024-12-23.

And v8.0.0 seems to have the memory leak:

So we have to go back in time between v8.0.0 and v3.0.0. For example v3.7.0 from 2024-11-19.

It seems v3.7.0 is fine:

Next should be v7.1.0 from 2024-12-06.

Memory leak present in v7.1.0:

Next up is v6.0.0.

And v6.0.0 is also leaking:

We need to go back to v4.0.0.

And v4.0.0 is a dud too:

Next up v3.10.0.

v3.10.0 is fine:

So there's no more releases to check, just individual commits.

And 5fa57c8 is broken too:

We're getting closer, f80042c is broken:

We're down to 6 commits:

 > g sl v3.10.0~1..f80042c
f80042c5c 2024-11-22 Sale Djenic          > chore(wallet)_: ens resolver identified under ens api  (origin/test-mem-leak, test-mem-leak)
c5dede93b 2024-11-22 Alex Jbanca          > feat(browser connect)_: Implementing signTypedData_V4 
c5c28cc56 2024-11-22 Patryk Osmaczko      > chore(logging)_: switch to zap.Logger as central logger 
d18dfeffc 2024-11-22 Patryk Osmaczko      > chore(logging)_: introduce geth adapter 
ed36d6054 2024-11-22 Alex Jbanca          > fix(BC)_: Fix personal_sign RPC event processing 
3912e7645 2024-11-22 Patryk Osmaczko      > chore(logging)_: add benchmark for custom zap core 
eae034aea 2024-11-22 Patryk Osmaczko      > feat(logging)_: introduce custom zap.Core enabling runtime changes 
3824691f7 2024-11-22 frank                > chore_: remove CLI option  MEDIA_HTTPS (#6112)  (tag: v3.10.0)

So there's a 50% chance it's @osmaczko commit 😂

And ed36d60 is clean:

So we have 3 left, and c5c28cc5687963491ce2815c4523ce6248ff1adb should tell us which one it is.

The c5c28cc commit has the memory leak:

Which means it's either that commit, or d18dfef.

And it turns out that d18dfef has no memory leak:

Which means I'm reverting to c5c28cc to verify last time it's the cause.

Thanks @jakubgs for blame. I'll try to fix ASAP, sorry for inconvenience 😓

@jakubgs could you please do a test against #6449? Branch is on top of c5c28cc

Done: https://ci.infra.status.im/job/status-go/job/deploy-notify-prod/55/

@jakubgs The neatness of your investigation is one for the books and deserves a round of applause, thank you! 👏

Looks promising, will give it few more hours.

Looks good except for the large interim spike.

Such spikes seem to appear before the upgrade to v10.9.0, so I consider it "normal."

Indeed, it does look better. I have no idea what this spike is tho. But if it's only temporary we can live with it, until it becomes too big of a problem. Thanks for looking into the issue.