Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Refactoring the data management #96

Open
jbusecke opened this issue Apr 10, 2024 · 0 comments
Open

Refactoring the data management #96

jbusecke opened this issue Apr 10, 2024 · 0 comments

Comments

@jbusecke
Copy link
Contributor

Our data management is in need for a refactor to

  • support independent information (requirements.txt, meta.yaml) for each feedstock
  • align with the release version of the Carbonplan Catalog

I have been pondering some of the choices to make and wanted to discuss them a bit more widely:

  1. Monorepo or feedstock repos?
    There is a fundamental question if we want to keep all feedstocks in a single repo (this one) or have a repo for each feedstock.
  • Monorepo keeps management of code/secrets 'compact'
  • But it also allows for less flexibility (swapping secrets, config files etc)
  1. "Interface" to the catalog
    I had discussed with @norlandrhagen that it would be great to have a fully self-contained zarr store with metadata as the ideal case to hand of to the catalog layer. The logic building the catalog could simply have a list of 'registered' zarr stores which need to contain a bunch of extra metadata (I have been tinkering with this quite a bit in https://github.com/leap-stc/proto_feedstock
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant