Expensive string parse in Lua resulting in diff consuming too much memory

[Hugobsb]

The Neogit diff just freezed my Linux here in a way that I had to press the power button to shutdown (I don't have the SysRq keys enabled to try a kernel shutdown, maybe I should try it later. It can also be Linux misconfiguration).

I also tried it in my Mac M1 and it freezed the NeoVim process, I had to manually kill it.

That said, I would like to know if someone can elaborate on my discussion here about it.

In my case I have an edge case example of a really large package-lock.json file changes.

See the output of git diff --stat:

Unfortunately, I can't share the package-lock.json file here, but I think anyone would be able to reproduce the issue by creating a large file diff as such.

Notes: I didn't open an issue because I would like to discuss first.

---

Edit: I updated the title to be more precise on what was happening.

[Hugobsb]

I mean, I can simply use git diff instead, it's not about it. The minimum I achieve here is to not accidentally freeze my entire computer by hitting Tab by accident in a file that is too much large for Neogit to diff.

[CKolkey]

#1427 :)

[Hugobsb]

Hey, @CKolkey! I'm sorry for the delay, thank you very much for the responses on this.

It seems to be comically fast now. I tested it again with another package-lock.json as below (I lost the original one :c)

Here is the profile.log. I am sending a screenshot to make the visualization easier:

[CKolkey]

No worries, mate. I was a bit surprised to hear about the regression, since I had tested this with a 100k change a while back, and that loaded fine. The culprit was a classic lua fail: string concatenation via .. in a tight loop. Every single time triggers a rehashing, which gets very expensive... At least it was a simple fix :)

The __index call in your profile is the actual loading/parsing of the diff, so at least now it's spending time where I would expect it to be

[Hugobsb]

Really nice. Quite interesting though, I will definitely search more about it. I'm marking the discussion as resolved :D

[CKolkey]

I recommend this: https://www.lua.org/gems/sample.pdf - I think I've read it every 6 months or so for the last two years haha