Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

inital draft "equalise_cubes" operation to assist merge/concatenate #6248

Closed
Tracked by #5375
pp-mo opened this issue Dec 10, 2024 · 4 comments · Fixed by #6257
Closed
Tracked by #5375

inital draft "equalise_cubes" operation to assist merge/concatenate #6248

pp-mo opened this issue Dec 10, 2024 · 4 comments · Fixed by #6257
Assignees

Comments

@pp-mo
Copy link
Member

pp-mo commented Dec 10, 2024

Scoping notes, drawn from #5375

craft the initial operation, implementing only certain operations
= why not equalise_attributes/unify_time_coords for now, as these already exist ?


Similarity of proposed "operations"

All of the proposed "operations" follow a pattern -
(which matches iris.util.equalise_attributes or iris.util.unify_time_coords)

  • they take a list of cubes + return a list of corresponding modified cubes
  • equalise or remove various metadata elements to make all the cubes mergeable/concatenable
  • return the same cubes, if operation is not needed (or not possible)
  • won't make further changes, if re-applied
  • can work out whether changes are needed (e.g. remove elements only if they disagree)
  • equalise all the cubes they are given,
    intending that they will all (potentially) combine into a single result cube

Proposed new API form

We could write all these as additional iris.util routines, in the style of equalise_attributes/unify_time_coords
- but we think it is probably better to have a single function
-- say, "equalise_cubes" --
with keyword controls for the various different operations

  • NOTE: assumes ideal = all ops work independently, and independent of order of application

Grouping of input cubes ?

Optionally, the top-level "equalise_cubes" function could identify the "groups" of cubes (which would naturally be chosen by merge/concat),
so as to allow the operation to be applied to (for instance) all cubes from a load.

  • most of the suggested operations seem to require no additional context or arguments,
    • so you either run that operation, or you don't
    • but SOME it seems might need additional per-operation difference
      = e.g. "Remove scalar coordinates" (effectively using "new_axis"??)
      • might need to specify which
      • and, as for the time-dependent-orography problem, possibly which other aux-coords need to map the new dim)
  • actual grouping is determined by merge/concat implementations
    -- but we suspect that the definition of this may be subtly different between merge + concatenate_ ???
@pp-mo
Copy link
Member Author

pp-mo commented Dec 10, 2024

Some practical realisations ...

(1) the "grouping" idea isn't really a requirement for direct user usage, i.e. when calling merge/concatenate.
- But it is really needed if you want to integrate into load operations

(2) effectively, many possible "operation" options will inevitably affect the "grouping" operation : e.g. equalise attributes --> must ignore attributes when grouping. Likewise if ignoring cell-methods, units, etc

@trexfeathers
Copy link
Contributor

I don't think this should be assigned to me

@pp-mo pp-mo assigned stephenworsley and unassigned trexfeathers Dec 17, 2024
@bjlittle
Copy link
Member

@pp-mo Note that the common metadata API has convenience behaviour that may help here i.e., difference and combine

@pp-mo
Copy link
Member Author

pp-mo commented Dec 18, 2024

@pp-mo Note that the common metadata API has convenience behaviour that may help here i.e., difference and combine

The current implementation I'm proposing has, I hope, found a way of cutting through the potential complexity of this : it's now only interested in metadata equality, based on a subset of fields.
FWIW, I convinced myself that this was "good enough" for the purpose, and doesn't need to align exactly with whatever merge/concatenate consider relevant.

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants