Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Make .get_storage() caching listing lazy #317

Closed
Tracked by #39
ilongin opened this issue Aug 19, 2024 · 0 comments · Fixed by #976
Closed
Tracked by #39

Make .get_storage() caching listing lazy #317

ilongin opened this issue Aug 19, 2024 · 0 comments · Fixed by #976
Assignees
@ilongin

Comments

@ilongin
Copy link
Contributor

ilongin commented Aug 19, 2024

This is follow up for PR #294 where listing is saved (cached) inside .from_storage() method itself which means it's not lazy unlike all the other DataChain steps.
We should think how to postpone caching / saving listing when all chain steps are applied.

Ideas:

  1. Make some generic step that accepts function (closure) in DatasetQuery
  2. Create .terminate() on DataChain class
  3. ...
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees

@ilongin ilongin

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Making listing lazy in DatasetQuery iterative/datachain
1 participant
@ilongin