Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Add efficient storage for sharing between all interpreters in a pipeline #92

Open
jesper-friis opened this issue Dec 16, 2022 · 1 comment

Comments

@jesper-friis
Copy link
Contributor

jesper-friis commented Dec 16, 2022

Motivation
We cannot assume that the different initialize() and get() methods in a pipeline are called from the same interpreter - at least that is not the case in AiiDA. Hence, all initialize() and get() methods should fetch the collection from the storage and save them back again if anything has been changed.

The current solution in PR #95 that stores the collection in the data cache works, but is very inefficient.

Task
To support such a common storage:

  • Figure out the "correct" way to refer to the specific storage used load/store the collection. This should be shared between all interpreters that are spawned for the same pipeline.
  • From OTELib, It would be useful if the storage to use could be configured in the client (e.g. via arguments to otelib.OTEClient() or dedicated methods). This would probably require some dedicated REST endpoints in oteapi-services with admin permissions.
  • Ideally the changes can be implemented in the utility functions for retrieving and updating the collection and will not affect any code using them.
@CasperWA
Copy link
Contributor

Please update the title of this issue.

@jesper-friis jesper-friis changed the title Add a Add efficient storage for sharing between all interpreters in a pipeline Jan 1, 2023
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants