Implement flat storage for state dumps #8322
Assignees
Comments
|
On the meeting today, we decided that state dumps are important for parameter estimation, because there we need to restore "real" state of filled DB. Due to the same reason, we don't want to make dumps more compact by storing only the last state. So it makes sense just to add contents of DBCol::FlatState* columns to the dump file. I think we don't need deltas though and we can set flat storage heads to genesis block hashes, because Pointers:
|
near-bulldozer bot
pushed a commit
that referenced
this issue
Jan 25, 2023
Extend state dump to include FlatState column. State dump format is changed to support multiple columns by including column index along with every key-value pair. This introduces an overhead of 1 byte per entry which should be negligible. While we are changing the format, also add an end of file mark so that we can properly detect truncated files. Previously, if we were very unlucky and the file was truncated just at a record border we wouldn’t notice it when reading the file. With explicit end of file marker we can detect it now. Co-authored-by: Anton Puhach <anton@near.org> Issue: #8322
ppca
pushed a commit
to ppca/nearcore
that referenced
this issue
Jan 30, 2023
Extend state dump to include FlatState column. State dump format is changed to support multiple columns by including column index along with every key-value pair. This introduces an overhead of 1 byte per entry which should be negligible. While we are changing the format, also add an end of file mark so that we can properly detect truncated files. Previously, if we were very unlucky and the file was truncated just at a record border we wouldn’t notice it when reading the file. With explicit end of file marker we can detect it now. Co-authored-by: Anton Puhach <anton@near.org> Issue: near#8322
near-bulldozer bot
pushed a commit
that referenced
this issue
Feb 6, 2023
Part of #8322. ### Testing In order to make sure that flat state is actually used I've made the following change to [`get_ref`](https://github.com/near/nearcore/blob/633a0443a57f8970bc0703f4eb61ccc4325d3020/core/store/src/trie/mod.rs#L951): ``` if matches!(mode, KeyLookupMode::FlatStorage) && !is_delayed { if let Some(flat_state) = &self.flat_state { let flat_result = flat_state.get_ref(&key); + println!("flat ok"); assert_eq!(result, flat_result); } } ``` and then checked the output of the following cmd: ``` cargo run --package runtime-params-estimator --features "required nightly" --bin runtime-params-estimator -- --accounts-num 2000 --additional-accounts-num 2000 --iters 1 --warmup-iters 1 --metric time ``` Please note that we cannot enable flat state assert as part of `sanity_check` test, see [zulip discussion](https://near.zulipchat.com/#narrow/stream/295306-pagoda.2Fcontract-runtime/topic/parameter.20estimator/near/324929122) for more context.
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
We have an option to initialize node from a state dump, see genesis-tools/README.md. This is used to start node which already has a large state. This state is stored in
state_dumpfile which currently stores serializedDBCol::State. It is used in params-estimator and some nayduck tests (state_sync_massive.py and state_sync_massive_validator.py) to check that state sync works.We need to support flat storage there as well, I have two options in mind:
state_dumpfile format by storing only KV pairs - and then construct Trie and Flat Storage from itDBCol::FlatState*to this file as wellSome TODOs:
genesis_state_from_dump) and then makes some state readsProgress:
The text was updated successfully, but these errors were encountered: