Skip to content

kad: Providers part 7: better types and public API, public addresses & known providers #246

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom
Merged
merged 43 commits into from
Sep 30, 2024
Merged

kad: Providers part 7: better types and public API, public addresses & known providers #246

merged 43 commits into from
Sep 30, 2024

Conversation

dmitry-markin
Copy link
Collaborator

@dmitry-markin dmitry-markin commented Sep 18, 2024 •
edited
Loading

This PR introduces the following changes:

  1. Introduces a better type ContentProvider without a burden of KademliaPeer.
  2. Simplifies public litep2p content providers API.
  3. Uses PublicAddresses API when advertising local providers.
  4. Adds locally known providers to discovered providers when performing GET_PROVIDERS request.
  5. Emits IncomingProvider event when remote node registers as a provider via ADD_PROVIDER request.

It may be easier to review this PR through individual commits. The commits of interest are:

  1. Introduce ContentProvider type
  2. Use PublicAddresses API
  3. Use locally known providers when performing `KademliaHandle::get_providers() request
  4. Emit IncomingProvider event

All other commits are merge commits from upstream branches.

Builds upon #245.

@dmitry-markin
Introduce ADD_PROVIDER query
1048320
@dmitry-markin
Merge remote-tracking branch 'origin/master' into dm-provider-queries
2ab84ae
@dmitry-markin
Execute ADD_PROVIDER query
778078b
@dmitry-markin
Introduce local providers in MemoryStore
323a26f
@dmitry-markin
Add provider refresh interval to Kademlia config
8946809
@dmitry-markin
Merge remote-tracking branch 'origin/master' into dm-republish-providers
474a58a
@dmitry-markin
Move FuturesStream to a separate file
d02acee
@dmitry-markin
Refresh providers: dry-run without network queries
5432c9b
@dmitry-markin
Remove try_get_record() and other try_...() non-async methods
feb493d
@dmitry-markin
Move query ID generation from KademliaHandle to Kademlia
982c73d
@dmitry-markin
Republish providers
68c7a87
@dmitry-markin
Merge remote-tracking branch 'origin/master' into dm-republish-providers
f9ed3e7
@dmitry-markin
Merge remote-tracking branch 'origin/master' into dm-add-providers
764a3f2
@dmitry-markin
Use getter TransportService::local_peer_id() instead of accessing d…
5e97484 
…irectly
@dmitry-markin
Use getter TransportService::local_peer_id() instead of accessing d…
7caf290 
…irectly
@dmitry-markin
Introduce GET_PROVIDERS query
60b8fb6
@dmitry-markin
Update get_providers() to receive query ID via oneshot channel
d340c35
@dmitry-markin
Make lines fit into 100 characters
10b96db
@dmitry-markin
Introduce GetProvidersContext & GetProvidersConfig
6146540
@dmitry-markin
Implement GET_PROVIDERS query
e167419
@dmitry-markin
Merge returned provider addresses and deduplicate records
eb6eee8
@dmitry-markin
Merge branch 'dm-add-providers' into dm-republish-providers
8314d95
@dmitry-markin
minor: fix log target
974eac5
@dmitry-markin
Revert "Remove try_get_record() and other try_...() non-async met…
e895a44 
…hods"

This reverts commit feb493d.
@dmitry-markin
Revert "Move query ID generation from KademliaHandle to Kademlia"
ce18594 
This reverts commit 982c73d.
@dmitry-markin
Use AtomicUsize to generate QueryId in both KademliaHandle and …
0fcd621 
…`Kademlia`
@dmitry-markin
Merge branch 'dm-republish-providers' into dm-get-providers
5f26161
@dmitry-markin
Use open_substream_or_dial() to add provider records to peers
57f17ce
@dmitry-markin
Merge remote-tracking branch 'origin/master' into dm-add-providers
8594103
@dmitry-markin
Fix refresh when we are the only provider
4861959
@dmitry-markin dmitry-markin added the enhancement New feature or request label Sep 18, 2024
lexnv
lexnv approved these changes Sep 20, 2024
View reviewed changes
Copy link
Collaborator

@lexnv lexnv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Will probably need to take a quick look at the first PR in the series after all others are merged into it 🙏

@dmitry-markin
Copy link
Collaborator Author

Will probably need to take a quick look at the first PR in the series after all others are merged into it 🙏

My idea was to merge PRs one-by-one into master. When the first PR is merged, GitHub will automatically set the target branch for the second PR to master, and so on.

@lexnv
Copy link
Collaborator

lexnv commented Sep 23, 2024

Sounds like a plan! 👍

@dmitry-markin dmitry-markin mentioned this pull request Sep 27, 2024
Merged
Base automatically changed from dm-stop-providing to master September 30, 2024 14:44
@dmitry-markin dmitry-markin merged commit 1a0325c into master Sep 30, 2024
8 checks passed
@dmitry-markin dmitry-markin deleted the dm-improve-providers-api branch September 30, 2024 17:02
@lexnv lexnv mentioned this pull request Nov 1, 2024
Merged
lexnv added a commit that referenced this pull request Nov 4, 2024
@lexnv @dmitry-markin
chore: Release litep2p version 0.8.0 (#280)
af0535c 
## [0.8.0] - 2024-11-01

This release adds support for content provider advertisement and
discovery to Kademlia protocol implementation (see libp2p
[spec](https://github.com/libp2p/specs/blob/master/kad-dht/README.md#content-provider-advertisement-and-discovery)).
Additionally, the release includes several improvements and memory leak
fixes to enhance the stability and performance of the litep2p library.

### Added

- kad: Providers part 8: unit, e2e, and `libp2p` conformance tests
([#258](#258))
- kad: Providers part 7: better types and public API, public addresses &
known providers ([#246](#246))
- kad: Providers part 6: stop providing
([#245](#245))
- kad: Providers part 5: `GET_PROVIDERS` query
([#236](#236))
- kad: Providers part 4: refresh local providers
([#235](#235))
- kad: Providers part 3: publish provider records (start providing)
([#234](#234))

### Changed

- transport_service: Improve connection stability by downgrading
connections on substream inactivity
([#260](#260))
- transport: Abort canceled dial attempts for TCP, WebSocket and Quic
([#255](#255))
- kad/executor: Add timeout for writting frames
([#277](#277))
- kad: Avoid cloning the `KademliaMessage` and use reference for
`RoutingTable::closest`
([#233](#233))
- peer_state: Robust state machine transitions
([#251](#251))
- address_store: Improve address tracking and add eviction algorithm
([#250](#250))
- kad: Remove unused serde cfg
([#262](#262))
- req-resp: Refactor to move functionality to dedicated methods
([#244](#244))
- transport_service: Improve logs and move code from tokio::select macro
([#254](#254))

### Fixed

- tcp/websocket/quic: Fix cancel memory leak
([#272](#272))
- transport: Fix pending dials memory leak
([#271](#271))
- ping: Fix memory leak of unremoved `pending_opens`
([#274](#274))
- identify: Fix memory leak of unused `pending_opens`
([#273](#273))
- kad: Fix not retrieving local records
([#221](#221))

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
@lexnv lexnv mentioned this pull request Nov 4, 2024
Merged
github-merge-queue bot pushed a commit to paritytech/polkadot-sdk that referenced this pull request Nov 5, 2024
@lexnv @dmitry-markin
litep2p: Update litep2p to v0.8.0 (#6353)
94389a9 
This PR updates litep2p to the latest release.

- `KademliaEvent::PutRecordSucess` is renamed to fix word typo
- `KademliaEvent::GetProvidersSuccess` and
`KademliaEvent::IncomingProvider` are needed for bootnodes on DHT work
and will be utilized later


### Added

- kad: Providers part 8: unit, e2e, and `libp2p` conformance tests
([#258](paritytech/litep2p#258))
- kad: Providers part 7: better types and public API, public addresses &
known providers ([#246](paritytech/litep2p#246))
- kad: Providers part 6: stop providing
([#245](paritytech/litep2p#245))
- kad: Providers part 5: `GET_PROVIDERS` query
([#236](paritytech/litep2p#236))
- kad: Providers part 4: refresh local providers
([#235](paritytech/litep2p#235))
- kad: Providers part 3: publish provider records (start providing)
([#234](paritytech/litep2p#234))

### Changed

- transport_service: Improve connection stability by downgrading
connections on substream inactivity
([#260](paritytech/litep2p#260))
- transport: Abort canceled dial attempts for TCP, WebSocket and Quic
([#255](paritytech/litep2p#255))
- kad/executor: Add timeout for writting frames
([#277](paritytech/litep2p#277))
- kad: Avoid cloning the `KademliaMessage` and use reference for
`RoutingTable::closest`
([#233](paritytech/litep2p#233))
- peer_state: Robust state machine transitions
([#251](paritytech/litep2p#251))
- address_store: Improve address tracking and add eviction algorithm
([#250](paritytech/litep2p#250))
- kad: Remove unused serde cfg
([#262](paritytech/litep2p#262))
- req-resp: Refactor to move functionality to dedicated methods
([#244](paritytech/litep2p#244))
- transport_service: Improve logs and move code from tokio::select macro
([#254](paritytech/litep2p#254))

### Fixed

- tcp/websocket/quic: Fix cancel memory leak
([#272](paritytech/litep2p#272))
- transport: Fix pending dials memory leak
([#271](paritytech/litep2p#271))
- ping: Fix memory leak of unremoved `pending_opens`
([#274](paritytech/litep2p#274))
- identify: Fix memory leak of unused `pending_opens`
([#273](paritytech/litep2p#273))
- kad: Fix not retrieving local records
([#221](paritytech/litep2p#221))

See release changelog for more details:
https://github.com/paritytech/litep2p/releases/tag/v0.8.0

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
lexnv added a commit to paritytech/polkadot-sdk that referenced this pull request Nov 15, 2024
@lexnv @dmitry-markin
litep2p: Update litep2p to v0.8.0 (#6353)
1cac380 
This PR updates litep2p to the latest release.

- `KademliaEvent::PutRecordSucess` is renamed to fix word typo
- `KademliaEvent::GetProvidersSuccess` and
`KademliaEvent::IncomingProvider` are needed for bootnodes on DHT work
and will be utilized later

- kad: Providers part 8: unit, e2e, and `libp2p` conformance tests
([#258](paritytech/litep2p#258))
- kad: Providers part 7: better types and public API, public addresses &
known providers ([#246](paritytech/litep2p#246))
- kad: Providers part 6: stop providing
([#245](paritytech/litep2p#245))
- kad: Providers part 5: `GET_PROVIDERS` query
([#236](paritytech/litep2p#236))
- kad: Providers part 4: refresh local providers
([#235](paritytech/litep2p#235))
- kad: Providers part 3: publish provider records (start providing)
([#234](paritytech/litep2p#234))

- transport_service: Improve connection stability by downgrading
connections on substream inactivity
([#260](paritytech/litep2p#260))
- transport: Abort canceled dial attempts for TCP, WebSocket and Quic
([#255](paritytech/litep2p#255))
- kad/executor: Add timeout for writting frames
([#277](paritytech/litep2p#277))
- kad: Avoid cloning the `KademliaMessage` and use reference for
`RoutingTable::closest`
([#233](paritytech/litep2p#233))
- peer_state: Robust state machine transitions
([#251](paritytech/litep2p#251))
- address_store: Improve address tracking and add eviction algorithm
([#250](paritytech/litep2p#250))
- kad: Remove unused serde cfg
([#262](paritytech/litep2p#262))
- req-resp: Refactor to move functionality to dedicated methods
([#244](paritytech/litep2p#244))
- transport_service: Improve logs and move code from tokio::select macro
([#254](paritytech/litep2p#254))

- tcp/websocket/quic: Fix cancel memory leak
([#272](paritytech/litep2p#272))
- transport: Fix pending dials memory leak
([#271](paritytech/litep2p#271))
- ping: Fix memory leak of unremoved `pending_opens`
([#274](paritytech/litep2p#274))
- identify: Fix memory leak of unused `pending_opens`
([#273](paritytech/litep2p#273))
- kad: Fix not retrieving local records
([#221](paritytech/litep2p#221))

See release changelog for more details:
https://github.com/paritytech/litep2p/releases/tag/v0.8.0

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
# for free to join this conversation on GitHub. Already have an account? # to comment
Reviewers

@lexnv lexnv lexnv approved these changes

Assignees
No one assigned
Labels
enhancement New feature or request
Projects
Networking
Status: Done ✅
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@dmitry-markin @lexnv