Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Refactor ImporterFactory for multiple files into a finder #1142

Open
Tracked by #780
manasaV3 opened this issue Sep 16, 2024 · 0 comments
Open
Tracked by #780

Refactor ImporterFactory for multiple files into a finder #1142

manasaV3 opened this issue Sep 16, 2024 · 0 comments

Comments

@manasaV3
Copy link
Contributor

Motivation

The MultiSourceFileFinder is a special case of the DefaultImporterFactory which has a different behaviour than other ImporterFactories. It was created to handle when we have multiple files that need to be fetched for a single entity, ie all the files retrieved in a single source entry are will be passed as input for processing the said entity.

This is used for the alignment importer, where we have multiple different files that need to be fetched for a single alignment entity.

This is a low-lift solution and can be a little opaque for the config creator on the behaviour of how the files are processed. This should be refactored to be a finder which supports all the different cases such as source_glob, source_multi_glob, literal, etc. That will make it more intuitive.

Definition of Done

  • There is a new finder for this specific usecase
  • All the existing configs that utilize this ImporterFactory have been updated to the new finder.
  • The existing MultiSourceFileFinder has been sunset
@manasaV3 manasaV3 added backend P0 Must have maintenance P1 Should have and removed P0 Must have labels Sep 16, 2024
# for free to join this conversation on GitHub. Already have an account? # to comment
Projects
None yet
Development

No branches or pull requests

1 participant