Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

[FEATURE] Late Filtering Operator for Queries #121

Open
ppanopticon opened this issue Nov 22, 2024 · 1 comment
Open

[FEATURE] Late Filtering Operator for Queries #121

ppanopticon opened this issue Nov 22, 2024 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@ppanopticon
Copy link
Member

Description

In the XReco context (and potentially for some other applications) we require late-stage filtering for queries. This type of filter removes Retrievables from a result stream generated by some upstream Retriever.

## Solution
I'd propose to have a LateFilter that can do late-stage filtering of Retrievables based on some boolean predicate. The way this filter works is roughly as follows:

  • It fetches the necessary field from the schema.
  • It performs a comparison of the field's value with a predicate.
  • It removes all Retrievables that don't match the predicate.
  • As a side-effect, the LateFilterOperator can append the fetched field to the Retrievable (similar to the FieldLookup).

For the sake of simplicity, a first implementation compares a single field to a single predicate. Logical combinations (AND, OR) are not supported. There are some implementation details to consider and optimise here:

  • Batch processing vs. stream processing for large result-sets
  • Attribute matching for structs

## Dependencies
No know dependencies.

Context

It is currently not in scope to push-down these filters and combine them with the retrievable operation (which could be done, e.g., in PostgreSQL). This would constitute a next step.

@net-cscience-raphael
Copy link
Contributor

See #123

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants