Derived Catalog: test for all needed variables and skip if existing

[aulemahal]

Change Summary

When updating datasets with derived variable calculations, the current code calls the derived variable as soon as at least one dependency is present in the dataset. It also overwrites variables if they are already present.

This PR ensures that all dependencies are present before computing the derived cat. What would have resulted in an error before now is silently skipped.

This PR also skips the calculation if a variable of the same name already exists. The use case here is when a variable is missing on some datasets but not others. For example "tas", it's better to use a native "tas" than one derived from "tasmin" and "tasmax".

Related issue number

Checklist

• Unit tests for the changes exist
• Tests pass on CI
• Documentation reflects the changes where applicable

[andersy005]

This PR also skips the calculation if a variable of the same name already exists.

I'm curious... what should happen if the user wants to override the existing variable from the catalog i.e. they want to use their own derived variable?

[aulemahal]

Well, you're right that this wouldn't work anymore, and some might see that as a problem.
My view was that the derived catalog is there to compensate for missing variables, rather than to change their definition. Thus the change felt justified (and is useful is my use of intake-esm). However, this is only my view. If you feel the previous behaviour was better, I can remove the added line.

[andersy005]

Thus the change felt justified (and is useful is my use of intake-esm).

I agree with you.

However, this is only my view. If you feel the previous behaviour was better, I can remove the added line.

How about we keep your changes as the default behavior and maybe allow users to override this behavior via a flag/argument passed to the Registry.register()?

intake-esm/intake_esm/derived.py

Lines 89 to 94 in 82692af

	def register(
	self,
	func: typing.Callable,
	*,
	variable: str,
	query: typing.Dict[pydantic.StrictStr, typing.Union[typing.Any, typing.List[typing.Any]]],

I imagine an argument along these lines

 prefer_derived : bool, optional (default=False)
           Specify whether to use a "derived variable" regardless if 
           a variable by the same name is found in the catalog.

Let me know what you think... 👍🏽 for merging this as is and addressing my suggestion in future PR if need be.

[aulemahal]

I agree with your idea! Added in upcoming commit.

[andersy005]

Awesome!! Thank you!