request review of metadata

[MichaelTiemannOSC]

Please review and comment on the metadata implementation here:

https://github.com/os-climate/wri-gppd-ingestion-pipeline/blob/master/notebooks/WRI-gppd-ingest.ipynb

Relates to: os-climate/os_c_data_commons#48

[MichaelTiemannOSC]

I have pushed a new branch that adapts the pyarrow ideas that Vincent shared yesterday: https://github.com/os-climate/wri-gppd-ingestion-pipeline/tree/metadata-v1

Please have a look and comment.

[caldeirav]

I have now moved the metadata implementation to DBT pipelines - as OpenMetadata is able to ingest metadata from catalog.json which is generated and versioned when generating DBT documentation

[MichaelTiemannOSC]

I just reviewed the notebook and see that it is unchanged since October 2021. It needs a complete overhaul in terms of credentials.env variable names, use of osc-ingest-tools, among other things. What's the best way to both true up this notebook to modern standards and true up to new DBT+OpenMetadata?

[caldeirav]

I suggest deprecating the older notebook as I am essentially rebuilding the pipeline from scatch. But keep it around so you can have a look when I complete the end-to-end flow, as you may want to make some functional changes (note: most of the data processing that was in the notebook should be in DBT now).

I have already checked in the notebooks for extraction and loading, with the data transformation now being shifted to DBT together with metadata ingestion.