Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

core/state, tests: fix memory leak via fastcache #28387

Merged
merged 3 commits into from
Oct 20, 2023

Conversation

holiman
Copy link
Contributor

@holiman holiman commented Oct 20, 2023

This change fixes a memory leak, when running either state-tests or blockchain-tests, we allocate a 1MB fastcache during snapshot generation. fastcache is a bit special, and requires a Reset() (it has it's own memory allocator).

The 1MB was hidden here and here respectively.

EDIT:

This PR also disables the long-running tests on windows/appveyor, by using the -short flag to tests. This disables the following statetests:

	st.slow(`^stAttackTest/ContractCreationSpam`)
	st.slow(`^stBadOpcode/badOpcodes`)
	st.slow(`^stPreCompiledContracts/modexp`)
	st.slow(`^stQuadraticComplexityTest/`)
	st.slow(`^stStaticCall/static_Call50000`)
	st.slow(`^stStaticCall/static_Return50000`)
	st.slow(`^stSystemOperationsTest/CallRecursiveBomb`)
	st.slow(`^stTransactionTest/Opcodes_TransactionInit`)

And these blockchain tests:

	// Slow tests
	bt.slow(`.*bcExploitTest/DelegateCallSpam.json`)
	bt.slow(`.*bcExploitTest/ShanghaiLove.json`)
	bt.slow(`.*bcExploitTest/SuicideIssue.json`)
	bt.slow(`.*/bcForkStressTest/`)
	bt.slow(`.*/bcGasPricerTest/RPC_API_Test.json`)
	bt.slow(`.*/bcWalletTest/`)

Copy link
Member

@MariusVanDerWijden MariusVanDerWijden left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@holiman
Copy link
Contributor Author

holiman commented Oct 20, 2023

Interesting!
It passed on appveyor/ubuntu, but window/386

ok  	github.com/ethereum/go-ethereum/eth/gasprice	1.040s
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x0 pc=0x78d247]
goroutine 34 [running]:
github.com/VictoriaMetrics/fastcache.(*Cache).HasGet(0x0, {0x0, 0x0, 0x0}, {0xadef7e4, 0x20, 0x20})
	C:/Users/appveyor/go/pkg/mod/github.com/!victoria!metrics/fastcache@v1.12.1/fastcache.go:174 +0x37
github.com/ethereum/go-ethereum/core/state/snapshot.(*diskLayer).AccountRLP(0xaf47aa0, {0x0, 0xbf, 0x49, 0xf4, 0x40, 0xa1, 0xcd, 0x5, 0x27, ...})
	C:/projects/go-ethereum/core/state/snapshot/disklayer.go:115 +0x197
github.com/ethereum/go-ethereum/core/state/snapshot.(*diskLayer).Account(0xaf47aa0, {0x0, 0xbf, 0x49, 0xf4, 0x40, 0xa1, 0xcd, 0x5, 0x27, ...})
	C:/projects/go-ethereum/core/state/snapshot/disklayer.go:81 +0x30
github.com/ethereum/go-ethereum/core/state.(*StateDB).getDeletedStateObject(0xaf6a280, {0x71, 0x56, 0x2b, 0x71, 0x99, 0x98, 0x73, 0xdb, 0x5b, ...})
	C:/projects/go-ethereum/core/state/statedb.go:569 +0x10d
github.com/ethereum/go-ethereum/core/state.(*StateDB).getStateObject(...)
	C:/projects/go-ethereum/core/state/statedb.go:550
github.com/ethereum/go-ethereum/core/state.(*StateDB).GetNonce(0xaf6a280, {0x71, 0x56, 0x2b, 0x71, 0x99, 0x98, 0x73, 0xdb, 0x5b, ...})
	C:/projects/go-ethereum/core/state/statedb.go:293 +0x39
github.com/ethereum/go-ethereum/core.(*StateTransition).preCheck(0xadefda0)
	C:/projects/go-ethereum/core/state_transition.go:272 +0x81
github.com/ethereum/go-ethereum/core.(*StateTransition).TransitionDb(0xadefda0)
	C:/projects/go-ethereum/core/state_transition.go:363 +0x62
github.com/ethereum/go-ethereum/core.ApplyMessage(0xaf123c0, 0xaccdb90, 0xacb35d8)
	C:/projects/go-ethereum/core/state_transition.go:181 +0x50
github.com/ethereum/go-ethereum/core.precacheTransaction(0xaccdb90, 0x1348a20, 0xacb35d8, 0xaf6a280, 0xaf6e240, 0xaf123c0)
	C:/projects/go-ethereum/core/state_prefetcher.go:92 +0x20c
github.com/ethereum/go-ethereum/core.(*statePrefetcher).Prefetch(0xac8f7d0, 0xaf523c0, 0xaf6a280, {{0x0, 0x0}, 0x0, 0x0, {0x0, 0x0, 0x0}}, ...)
	C:/projects/go-ethereum/core/state_prefetcher.go:71 +0x55c
github.com/ethereum/go-ethereum/core.(*BlockChain).insertChain.func3({0xc144ad210a62ad18, 0x4027a85, 0x1348860}, 0xaf523c0, 0xaf6a280)
	C:/projects/go-ethereum/core/blockchain.go:1799 +0x73
created by github.com/ethereum/go-ethereum/core.(*BlockChain).insertChain in goroutine 24
	C:/projects/go-ethereum/core/blockchain.go:1798 +0x258a
FAIL	github.com/ethereum/go-ethereum/eth/protocols/eth	0.154s

Ah, the precacher is using the same snap structure, and we niled the cache

@holiman holiman added this to the 1.13.5 milestone Oct 20, 2023
@holiman holiman merged commit c1d5a01 into ethereum:master Oct 20, 2023
devopsbo3 pushed a commit to HorizenOfficial/go-ethereum that referenced this pull request Nov 10, 2023
This change fixes a memory leak, when running either state-tests or blockchain-tests, we allocate a `1MB` fastcache during snapshot generation. `fastcache` is a bit special, and requires a `Reset()` (it has it's own memory allocator). 

The `1MB` was hidden [here](https://github.com/ethereum/go-ethereum/blob/master/tests/state_test_util.go#L333) and [here](https://github.com/ethereum/go-ethereum/blob/master/tests/block_test_util.go#L146) respectively.
devopsbo3 added a commit to HorizenOfficial/go-ethereum that referenced this pull request Nov 10, 2023
devopsbo3 added a commit to HorizenOfficial/go-ethereum that referenced this pull request Nov 10, 2023
Dergarcon pushed a commit to specialmechanisms/mev-geth-0x2mev that referenced this pull request Jan 31, 2024
This change fixes a memory leak, when running either state-tests or blockchain-tests, we allocate a `1MB` fastcache during snapshot generation. `fastcache` is a bit special, and requires a `Reset()` (it has it's own memory allocator). 

The `1MB` was hidden [here](https://github.com/ethereum/go-ethereum/blob/master/tests/state_test_util.go#L333) and [here](https://github.com/ethereum/go-ethereum/blob/master/tests/block_test_util.go#L146) respectively.
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants