Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Prevent nodes from trying to use the broken state sync mechanism #8719

Closed
exalate-issue-sync bot opened this issue Mar 14, 2023 · 1 comment · Fixed by #8730
Closed

Prevent nodes from trying to use the broken state sync mechanism #8719

exalate-issue-sync bot opened this issue Mar 14, 2023 · 1 comment · Fixed by #8730
Assignees
@nikurt
Labels
GroomedA

Comments

@exalate-issue-sync
Copy link

No description provided.

@exalate-issue-sync
Copy link
Author

Nikolay Kurtov commented:

for reference #7285 , but there are many more random reports of issues that a block is missing. I suspect that users have already learned to download snapshots

@nikurt nikurt mentioned this issue Mar 15, 2023
Merged
near-bulldozer bot pushed a commit that referenced this issue Mar 23, 2023
@nikurt
feat: Disable state sync by default because it's unreliable (#8730)
348d651 
Add a config option `state_sync_enabled`. Default value is `false`.

If the option is `true`, state sync works as usual, but prints this message every 10 seconds:
```
Mar 15 07:31:11 nikurt-4 neard[48582]: 2023-03-15T07:31:11.352458Z  WARN stats: The node is syncing its State. The current implementation of this mechanism is known to be unreliable. It may never complete, or fail randomly and corrupt the DB.
Mar 15 07:31:11 nikurt-4 neard[48582]: Suggestions:
Mar 15 07:31:11 nikurt-4 neard[48582]: * Download a recent data snapshot and restart the node.
Mar 15 07:31:11 nikurt-4 neard[48582]: * Disable state sync in the config. Add `"state_sync_enabled": false` to `config.json`.
Mar 15 07:31:11 nikurt-4 neard[48582]: A better implementation of State Sync is work in progress.
```

If the option is `false`, the node proceeds to download and apply blocks.
Started a node from a month-old snapshot. After about 24 hours of running I see this, which confirms that the Block Sync is enabled and State Sync wasn't enabled:
```
Mar 15 07:32:27 nikurt-3 neard[36289]: 2023-03-15T07:32:27.565700Z  INFO stats: #118440788 Downloading blocks 4.70% (1951625 left; at 118440788) 23 peers ⬇ 638 kB/s ⬆ 86.9 kB/s 2.00 bps 70.7 Tgas/s CPU: 62%, Mem: 4.27 GB
```

Fix #8719
posvyatokum pushed a commit that referenced this issue Mar 26, 2023
@nikurt @posvyatokum
feat: Disable state sync by default because it's unreliable (#8730)
91d546b 
Add a config option `state_sync_enabled`. Default value is `false`.

If the option is `true`, state sync works as usual, but prints this message every 10 seconds:
```
Mar 15 07:31:11 nikurt-4 neard[48582]: 2023-03-15T07:31:11.352458Z  WARN stats: The node is syncing its State. The current implementation of this mechanism is known to be unreliable. It may never complete, or fail randomly and corrupt the DB.
Mar 15 07:31:11 nikurt-4 neard[48582]: Suggestions:
Mar 15 07:31:11 nikurt-4 neard[48582]: * Download a recent data snapshot and restart the node.
Mar 15 07:31:11 nikurt-4 neard[48582]: * Disable state sync in the config. Add `"state_sync_enabled": false` to `config.json`.
Mar 15 07:31:11 nikurt-4 neard[48582]: A better implementation of State Sync is work in progress.
```

If the option is `false`, the node proceeds to download and apply blocks.
Started a node from a month-old snapshot. After about 24 hours of running I see this, which confirms that the Block Sync is enabled and State Sync wasn't enabled:
```
Mar 15 07:32:27 nikurt-3 neard[36289]: 2023-03-15T07:32:27.565700Z  INFO stats: #118440788 Downloading blocks 4.70% (1951625 left; at 118440788) 23 peers ⬇ 638 kB/s ⬆ 86.9 kB/s 2.00 bps 70.7 Tgas/s CPU: 62%, Mem: 4.27 GB
```

Fix #8719
nikurt added a commit that referenced this issue Apr 14, 2023
@nikurt
feat: Disable state sync by default because it's unreliable (#8730)
459495b 
Add a config option `state_sync_enabled`. Default value is `false`.

If the option is `true`, state sync works as usual, but prints this message every 10 seconds:
```
Mar 15 07:31:11 nikurt-4 neard[48582]: 2023-03-15T07:31:11.352458Z  WARN stats: The node is syncing its State. The current implementation of this mechanism is known to be unreliable. It may never complete, or fail randomly and corrupt the DB.
Mar 15 07:31:11 nikurt-4 neard[48582]: Suggestions:
Mar 15 07:31:11 nikurt-4 neard[48582]: * Download a recent data snapshot and restart the node.
Mar 15 07:31:11 nikurt-4 neard[48582]: * Disable state sync in the config. Add `"state_sync_enabled": false` to `config.json`.
Mar 15 07:31:11 nikurt-4 neard[48582]: A better implementation of State Sync is work in progress.
```

If the option is `false`, the node proceeds to download and apply blocks.
Started a node from a month-old snapshot. After about 24 hours of running I see this, which confirms that the Block Sync is enabled and State Sync wasn't enabled:
```
Mar 15 07:32:27 nikurt-3 neard[36289]: 2023-03-15T07:32:27.565700Z  INFO stats: #118440788 Downloading blocks 4.70% (1951625 left; at 118440788) 23 peers ⬇ 638 kB/s ⬆ 86.9 kB/s 2.00 bps 70.7 Tgas/s CPU: 62%, Mem: 4.27 GB
```

Fix #8719
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees

@nikurt nikurt

Labels
GroomedA
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: Disable state sync by default because it's unreliable nikurt/nearcore
1 participant
@nikurt