Enhance P2P Streams with Reserved Stream Replacement

[GheisMohammadi]

This PR introduces a new feature to the Harmony P2P stream management system: a reserved stream list. The reserved stream list acts as a backup pool of pre-established streams, ensuring that peer connections can be replaced immediately whenever a stream is removed due to invalid data, connectivity issues, or other disruptions. This improvement eliminates the need to go through the discovery process to find new peers, resulting in faster recovery and maintaining stable network operations.

The core functionality allows the stream manager to replace removed streams with available streams from the reserved list. By doing so, the system ensures that the number of connected peers remains sufficient to support syncing and consensus processes without interruptions. This mechanism is particularly advantageous in larger networks with numerous peers, where traditional peer discovery methods may take longer and introduce delays. This approach not only reduces the risk of disruptions in syncing and consensus but also ensures that the network remains resilient and responsive.

This PR lays the groundwork for future optimizations, such as dynamically adjusting the size of the reserved list based on network conditions or further refining stream management to handle edge cases more effectively.

[sophoah]

i was testing this one by doing :

1. make debug
2. kill one of the harmony process
3. restart the harmony process

at restart i faced the below error

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1b53c78]

goroutine 1942 [running]:
github.com/torquem-ch/mdbx-go/mdbx.openCursor.func1(0xc00137b260?, 0x69, 0x441e65?)
        /home/soph/go/pkg/mod/github.com/torquem-ch/mdbx-go@v0.27.10/mdbx/cursor.go:69 +0x18
github.com/torquem-ch/mdbx-go/mdbx.openCursor(0x0, 0x69)
        /home/soph/go/pkg/mod/github.com/torquem-ch/mdbx-go@v0.27.10/mdbx/cursor.go:69 +0x5b
github.com/torquem-ch/mdbx-go/mdbx.(*Txn).OpenCursor(...)
        /home/soph/go/pkg/mod/github.com/torquem-ch/mdbx-go@v0.27.10/mdbx/txn.go:597
github.com/ledgerwatch/erigon-lib/kv/mdbx.(*MdbxTx).stdCursor(0xc00137b1a0, {0x25f53e5, 0xd})
        /home/soph/go/pkg/mod/github.com/ledgerwatch/erigon-lib@v0.0.0-20230607152933-42c9c28cac68/kv/mdbx/kv_mdbx.go:1061 +0x125
github.com/ledgerwatch/erigon-lib/kv/mdbx.(*MdbxTx).RwCursor(0xc00137b1a0, {0x25f53e5, 0xd})
        /home/soph/go/pkg/mod/github.com/ledgerwatch/erigon-lib@v0.0.0-20230607152933-42c9c28cac68/kv/mdbx/kv_mdbx.go:1048 +0xb9
github.com/ledgerwatch/erigon-lib/kv/mdbx.(*MdbxTx).Cursor(...)
        /home/soph/go/pkg/mod/github.com/ledgerwatch/erigon-lib@v0.0.0-20230607152933-42c9c28cac68/kv/mdbx/kv_mdbx.go:1052
github.com/ledgerwatch/erigon-lib/kv/mdbx.(*MdbxTx).statelessCursor(0xc00137b1a0, {0x25f53e5, 0xd})
        /home/soph/go/pkg/mod/github.com/ledgerwatch/erigon-lib@v0.0.0-20230607152933-42c9c28cac68/kv/mdbx/kv_mdbx.go:905 +0xa5
github.com/ledgerwatch/erigon-lib/kv/mdbx.(*MdbxTx).Put(0x2392b00?, {0x25f53e5?, 0xb?}, {0xc00178e6b0, 0xb, 0x10}, {0xc00178e698, 0x8, 0x8})
        /home/soph/go/pkg/mod/github.com/ledgerwatch/erigon-lib@v0.0.0-20230607152933-42c9c28cac68/kv/mdbx/kv_mdbx.go:915 +0x3c
github.com/harmony-one/harmony/api/service/stagedstreamsync.SaveStageProgress({0x7f084c7c2518, 0xc00137b1a0}, {0x25f1222?, 0x2d40328?}, 0xa0?, 0x60)
        /home/soph/harmony/harmony3/api/service/stagedstreamsync/stages.go:54 +0xf0
github.com/harmony-one/harmony/api/service/stagedstreamsync.(*StageState).Update(...)
        /home/soph/harmony/harmony3/api/service/stagedstreamsync/stage.go:89
github.com/harmony-one/harmony/api/service/stagedstreamsync.(*StageBlockHashes).runBlockHashWorkerLoop(0xc0005477a0, {0x2d17528, 0xc001371bd0}, {0x2d40328, 0xc00137b1a0}, 0xc000b526e0, 0xc0017a82c0, {0xc1d85697f3667bc5, 0xbb4928b, 0x41395e0}, ...)
        /home/soph/harmony/harmony3/api/service/stagedstreamsync/stage_blockhashes.go:242 +0x32d
github.com/harmony-one/harmony/api/service/stagedstreamsync.(*StageBlockHashes).Exec(0xc0005477a0, {0x2d17528, 0xc001371bd0}, 0x0?, 0x0, 0xc0017a82c0, {0x2d28da8?, 0xc00182d2b0?}, {0x0, 0x0})
        /home/soph/harmony/harmony3/api/service/stagedstreamsync/stage_blockhashes.go:131 +0x6f3
github.com/harmony-one/harmony/api/service/stagedstreamsync.(*StagedStreamSync).runStage(0xc000ce0e08, {0x2d17528, 0xc001371bd0}, 0xc0018a44b0, {0x2d28da8, 0xc00182d2b0}, {0x0, 0x0}, 0x1, 0x0)
        /home/soph/harmony/harmony3/api/service/stagedstreamsync/staged_stream_sync.go:601 +0x154
github.com/harmony-one/harmony/api/service/stagedstreamsync.(*StagedStreamSync).Run(0xc000ce0e08, {0x2d17528, 0xc001371bd0}, {0x2d28da8, 0xc00182d2b0}, {0x0, 0x0}, 0x49?)
        /home/soph/harmony/harmony3/api/service/stagedstreamsync/staged_stream_sync.go:463 +0x2ef
github.com/harmony-one/harmony/api/service/stagedstreamsync.(*StagedStreamSync).doSyncCycle(0xc000ce0e08, {0x2d17528, 0xc001371bd0})
        /home/soph/harmony/harmony3/api/service/stagedstreamsync/syncing.go:459 +0xcd
github.com/harmony-one/harmony/api/service/stagedstreamsync.(*StagedStreamSync).doSync(0xc000ce0e08, {0x2d17528, 0xc00111b9a0}, 0x1)
        /home/soph/harmony/harmony3/api/service/stagedstreamsync/syncing.go:375 +0x4f6
github.com/harmony-one/harmony/api/service/stagedstreamsync.(*Downloader).loop(0xc000a6cf00)
        /home/soph/harmony/harmony3/api/service/stagedstreamsync/downloader.go:241 +0x1bb
github.com/harmony-one/harmony/api/service/stagedstreamsync.(*Downloader).Start.func1()
        /home/soph/harmony/harmony3/api/service/stagedstreamsync/downloader.go:104 +0x26
created by github.com/harmony-one/harmony/api/service/stagedstreamsync.(*Downloader).Start in goroutine 1
        /home/soph/harmony/harmony3/api/service/stagedstreamsync/downloader.go:102 +0x4f

[GheisMohammadi]

The issue you mentioned seems related to stage_blockhashes but stage_blockhashes.go belongs to PR #4824 and is not merged yet.

[sophoah]

if it is not merged how do i have the error ?

[sophoah]

i did it twice and faced the same error twice. Maybe you can try on your end ?

[GheisMohammadi]

i did it twice and faced the same error twice. Maybe you can try on your end ?

here is the staged sync folder in dev branch and there is no such file there:
https://github.com/harmony-one/harmony/tree/dev/api/service/stagedstreamsync

[sophoah]

ok my bad i mixed the PR the error above is for #4824

[sophoah]

@Frozen kindly review/approve, thanks