Tricky containers: many executables across distinct versions

[marcodelapierre]

Hi @vsoch,

so after thinking that MPI + containers was a bumpy road, I met ... OpenFoam! (which is also an MPI package, though irrelevant here)

At Pawsey we build custom OpenFoam containers for our researchers, as the stock images from the developers only come with OpenMPI, and we need ...MPICH (eheh).
I have just updated the images for the upcoming supercomputer, and I was starting to write the SHPC recipes, when....

Two interesting things!

1. Each OpenFoam version comes with 300+ executables; well, I said, it will be a longer SHPC recipe....but...
2. Distinct OpenFoam versions ship with different sets of these 300+ executables (as in, some executable names are different)

So, how to tackle this? -- brainstorming welcome!

Let me share my early thoughts, staring from point 2.
So I was thinking, how about enabling a container recipe to have sections that can be specific to specific tags, e.g.:

docker: vanessa/salad
[..]
latest:
  2: [..]
tags:
  1: [..]
  2: [..]

1:
  aliases:
    salad: /code/salad1

2:
  aliases:
    salad: /code/salad2

Doesn't have to be only for aliases, I guess it could also be other properties such as features, envs...
As a side note, we'd need to make sure that the latest tag is handled properly (but I suspect it would work out of the box right?).

And then there's still point 1. left.
The specific case of OpenFoam would still imply a 3000+ lines long recipe, so I was wondering whether it would make sense to implement a second feature, to allow the recipe include other yamls in the same dir, or subdirs. In this way, the recipe contents can be chunked down into many files, for improved readability and maintenance.

What do you think?

[marcodelapierre]

Well, here is a better way to write the YAML example above:
(tag_properties , I am not sure it's the best name for these tag-specific configurations)

docker: vanessa/salad
[..]
latest:
  2: [..]
tags:
  1: [..]
  2: [..]

tag_properties:
- tag: 1
  aliases:
    salad: /code/salad1
  features:
    gpu: true

- tag: 2
  aliases:
    salad: /code/salad2
  env:
    dressing: mayo

[vsoch]

that is an interesting idea - would this work for the issue that you have? Would the files be "too big" ?

[vsoch]

It would be a "load on demand" sort of deal - given that the user has selected a specific tag, we'd then index it there. I'd even say:

tag_properties:
tag1:
  aliases:
    salad: /code/salad1
  features:
    gpu: true

tag2:
  aliases:
    salad: /code/salad2
  env:
    dressing: mayo

to enforce uniqueness.

[muffato]

Being able to override properties for each tag in container.yaml would be fantastic. Different versions of a software can pack different executables.

[marcodelapierre]

Yes, the idea above would work.
Chopping the "too big" files would be a plus, but this is something manageable any way. (doom-scrolling, SHPC edition! 😆 )

[marcodelapierre]

Hi @vsoch , what do you think about this issue?
Do you think it is time consuming to implement? I don't have energies to work on it, but in case you have, I would be happy to test it with a good user case, the OpenFoam containers I probably mentioned above. :-)
2 containers, 4-5 tags per container, about 200 aliases per tag...!

[vsoch]

Hmm let me think more about it - I think our assumption here is that a container namespace (with different tags) is adhering to a possibly growing but generally consistent set of commands. It sounds like openfoam doesn't adhere to that, and each tag might be considered a separate container? I'm thinking the simplest thing would be to allow another metadata file type in the directory to indicate command groups. Named by tag? Or just something else? If we name by tag, then you can quickly check 1:1 if a tag file is there with commands and load it. But then if there are similar tags, that means a lot of redundancy. We could also have named files and use the main container.yaml as a lookup, and if a version isn't in the lookup we use the default set in the container.yaml (otherwise the tag).

This might be a lot to ask - but could you list out these openfoam container tags (and point me to where inside the executables are) so I can play around?

[marcodelapierre]

I like a lot the 2nd option you describe, with the named files and the fallback base template.
It is not a lot to ask, because... I already have a setup for the OpenFoam containers (currently glued with bash .. shame on me!)

To make things funnier, as I mentioned above, there are 2 distinct open-source project for openfoam, so here are the 2 sets of recipes for SHPC for a few versions:

https://github.com/PawseySC/pawsey-spack-config/tree/main/setonix/shpc_registry/quay.io/pawsey/openfoam

https://github.com/PawseySC/pawsey-spack-config/tree/main/setonix/shpc_registry/quay.io/pawsey/openfoam-org

In each directory, there is a base container template (without aliases), and then there is an alias subdirectory with the aliases for each version, which I just concatenate before running SHPC:
https://github.com/PawseySC/pawsey-spack-config/blob/main/setonix/setup_scripts/run_install_shpc_openfoam.sh#L58-L59

(There is a lot of redundancy, but at least these lists can be easily and automatically generated by inspecting each container)

Let me know if you need more details!

[vsoch]

That should be good! I'll start on something and ping you if I run into any questions.

[vsoch]

All set! #557

I'm not sure if the pawsey containers are too big to pull, but I just added the "container-recipe" label to take a shot anyway. I also had to tweak our test to limit to files named container.yaml (it was trying, and failing to test the new alias files!)

Let me know what you think when you have some time! I think I really like this simple design - it can be easily extended given any other attributes that get out of hand.

[marcodelapierre]

I'm not sure if the pawsey containers are too big to pull

They're huge lol!

Thanks, I will have a look as soon as I can.