Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Allow specification of metadata to be fetched for Websocket Queries #234

Merged
merged 1 commit into from
Dec 3, 2021

Conversation

silvanheller
Copy link
Member

Introduces the MetadataAccessSpecification, of which a list can be handed to Cineast. * for either domain or keys mean all elements should be fetched. For backwards-compatibility, null is treated as *, *, which means all metadata is fetched.

Implementation is unit-tested.

Closes #166

As a simple comparison, on the LSC 21 dataset, this reduces metadata fetch time for a tag query sushi_bar (5k results) from approx. 2.5 to 2 seconds:

(old)

[11-18 19:19:04.449][DEBUG][metadata-load-write-1db170ea-e7db-4117-b49c-d1bd4afc0a4a] AbstractMetadataReader - Performed metadata lookup for 6671 ids in 2439 ms. 92879 results.
[11-18 19:19:04.514][TRACE][metadata-ws-write-queryId0] AbstractWebsocketMessageHandler - Serialization for QR_METADATA_S in 44 ms
[11-18 19:19:04.656][TRACE][metadata-ws-write-queryId0] AbstractWebsocketMessageHandler$1 - metadata-ws-write-queryId0: Successfully wrote message QR_METADATA_S in 187 ms

(new)

[11-18 19:15:10.310][DEBUG][metadata-load-write-c6917953-51e3-400b-b06a-ea492ce166ba] AbstractMetadataReader - Performed metadata lookup for 6671 ids in 1970 ms. 39914 results.
[11-18 19:15:10.372][TRACE][metadata-ws-write-queryId0] AbstractWebsocketMessageHandler - Serialization for QR_METADATA_S in 44 ms
[11-18 19:15:10.435][TRACE][metadata-ws-write-queryId0] AbstractWebsocketMessageHandler$1 - metadata-ws-write-queryId0: Successfully wrote message QR_METADATA_S in 108 ms

For larger result-sets (i.e. hitting the config limit, currently 20k) it goes from 9.4 to 6 seconds + there are (obviously) gains in serialization speed.

(old)

[11-18 19:20:46.498][DEBUG][metadata-load-write-5066239b-fbe4-4bcb-9af6-033e51b4249c] AbstractMetadataReader - Performed metadata lookup for 20000 ids in 9473 ms. 315522 results.
[11-18 19:20:46.600][TRACE][metadata-ws-write-queryId3] AbstractWebsocketMessageHandler - Serialization for QR_METADATA_S in 8 ms
[11-18 19:20:46.620][TRACE][metadata-ws-write-queryId3] AbstractWebsocketMessageHandler$1 - metadata-ws-write-queryId3: Successfully wrote message QR_METADATA_S in 28 ms
[11-18 19:20:46.664][TRACE][metadata-ws-write-queryId2] AbstractWebsocketMessageHandler - Serialization for QR_METADATA_S in 72 ms
[11-18 19:20:46.664][TRACE][metadata-ws-write-queryId0] AbstractWebsocketMessageHandler - Serialization for QR_METADATA_S in 72 ms
[11-18 19:20:46.823][TRACE][metadata-ws-write-queryId1] AbstractWebsocketMessageHandler - Serialization for QR_METADATA_S in 231 ms
[11-18 19:20:46.935][TRACE][metadata-ws-write-queryId2] AbstractWebsocketMessageHandler$1 - metadata-ws-write-queryId2: Successfully wrote message QR_METADATA_S in 343 ms
[11-18 19:20:47.054][TRACE][metadata-ws-write-queryId2] AbstractWebsocketMessageHandler$1 - metadata-ws-write-queryId0: Successfully wrote message QR_METADATA_S in 463 ms
[11-18 19:20:47.170][TRACE][metadata-ws-write-queryId2] AbstractWebsocketMessageHandler$1 - metadata-ws-write-queryId1: Successfully wrote message QR_METADATA_S in 578 ms

(new)

[11-18 19:23:36.383][DEBUG][metadata-load-write-6437f6c0-f2bc-4f2f-8653-45a3615d17ab] AbstractMetadataReader - Performed metadata lookup for 20000 ids in 6278 ms. 134793 results.
[11-18 19:23:36.486][TRACE][metadata-ws-write-queryId1] AbstractWebsocketMessageHandler - Serialization for QR_METADATA_S in 50 ms
[11-18 19:23:36.534][TRACE][metadata-ws-write-queryId0] AbstractWebsocketMessageHandler - Serialization for QR_METADATA_S in 98 ms
[11-18 19:23:36.541][TRACE][metadata-ws-write-queryId1] AbstractWebsocketMessageHandler$1 - metadata-ws-write-queryId1: Successfully wrote message QR_METADATA_S in 105 ms
[11-18 19:23:36.674][TRACE][metadata-ws-write-queryId0] AbstractWebsocketMessageHandler$1 - metadata-ws-write-queryId0: Successfully wrote message QR_METADATA_S in 238 ms

@sauterl
Copy link
Collaborator

sauterl commented Nov 18, 2021

Generally speaking I like this PR very much, there are a few minor engineering points I think are to be improved.

Copy link
Collaborator

@sauterl sauterl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Thanks @silvanheller for the PR

@silvanheller silvanheller merged commit 6037f63 into dev Dec 3, 2021
@silvanheller silvanheller deleted the feature/md-spec-166 branch December 3, 2021 08:22
silvanheller added a commit that referenced this pull request Jun 10, 2022
Allow specification of metadata to be fetched for Websocket Queries

Former-commit-id: 6037f63
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants