OCI-SIF data container creation command #2872
Milestone
Comments
This was referenced May 3, 2024
|
We might wish to use the |
dtrudg
added a commit
to dtrudg/singularity
that referenced
this issue
May 23, 2024
Add a new command, `singularity data package`, that will create an OCI-SIF data container that 'packages' a host file or directory. The OCI-SIF data container consists of an OCI image, with a custom config mediaType. At present it will always hold a single layer, in squashfs format, containing the file / directory that was packaged. We currently use a flow that: * Reads the file / directory into a tar stream. * Presents the tar stream as a ggcr layer. * Mutates the layer into squashfs format. * Writes the image containing the mutated layer into an OCI-SIF. Arguably, given we are producing OCI-SIF with only a squashfs layer, we could directly call out to mksquashfs and then insert the layer from that - without going through a tar reader. We might adopt this optimisation in future, but for now we follow similar patterns to OCI-SIF container image handling, where we start with a tar layer and mutate. We probably need to consider what we do with layer format on push / pull, e.g. sylabs#2874 so let's keep tar in mind for now. Note that the command is using `singularity package <src> <dst>` argument order. This is different from our `build <dst> <src>`... but is generally more intuitive? Thoughts? Closes sylabs#2872
dtrudg
added a commit
to dtrudg/singularity
that referenced
this issue
May 23, 2024
Add a new command, `singularity data package`, that will create an OCI-SIF data container that 'packages' a host file or directory. The OCI-SIF data container consists of an OCI image, with a custom config mediaType. At present it will always hold a single layer, in squashfs format, containing the file / directory that was packaged. We currently use a flow that: * Reads the file / directory into a tar stream. * Presents the tar stream as a ggcr layer. * Mutates the layer into squashfs format. * Writes the image containing the mutated layer into an OCI-SIF. Arguably, given we are producing OCI-SIF with only a squashfs layer, we could directly call out to mksquashfs and then insert the layer from that - without going through a tar reader. We might adopt this optimisation in future, but for now we follow similar patterns to OCI-SIF container image handling, where we start with a tar layer and mutate. We probably need to consider what we do with layer format on push / pull, e.g. sylabs#2874 so let's keep tar in mind for now. Note that the command is using `singularity package <src> <dst>` argument order. This is different from our `build <dst> <src>`... but is generally more intuitive? Thoughts? Closes sylabs#2872
dtrudg
added a commit
to dtrudg/singularity
that referenced
this issue
May 23, 2024
Add a new command, `singularity data package`, that will create an OCI-SIF data container that 'packages' a host file or directory. The OCI-SIF data container consists of an OCI image, with a custom config mediaType. At present it will always hold a single layer, in squashfs format, containing the file / directory that was packaged. We currently use a flow that: * Reads the file / directory into a tar stream. * Presents the tar stream as a ggcr layer. * Mutates the layer into squashfs format. * Writes the image containing the mutated layer into an OCI-SIF. Arguably, given we are producing OCI-SIF with only a squashfs layer, we could directly call out to mksquashfs and then insert the layer from that - without going through a tar reader. We might adopt this optimisation in future, but for now we follow similar patterns to OCI-SIF container image handling, where we start with a tar layer and mutate. We probably need to consider what we do with layer format on push / pull, e.g. sylabs#2874 so let's keep tar in mind for now. Note that the command is using `singularity package <src> <dst>` argument order. This is different from our `build <dst> <src>`... but is generally more intuitive? Thoughts? Closes sylabs#2872
dtrudg
added a commit
to dtrudg/singularity
that referenced
this issue
May 23, 2024
Add a new command, `singularity data package`, that will create an OCI-SIF data container that 'packages' a host file or directory. The OCI-SIF data container consists of an OCI image, with a custom config mediaType. At present it will always hold a single layer, in squashfs format, containing the file / directory that was packaged. We currently use a flow that: * Reads the file / directory into a tar stream. * Presents the tar stream as a ggcr layer. * Mutates the layer into squashfs format. * Writes the image containing the mutated layer into an OCI-SIF. Arguably, given we are producing OCI-SIF with only a squashfs layer, we could directly call out to mksquashfs and then insert the layer from that - without going through a tar reader. We might adopt this optimisation in future, but for now we follow similar patterns to OCI-SIF container image handling, where we start with a tar layer and mutate. We probably need to consider what we do with layer format on push / pull, e.g. sylabs#2874 so let's keep tar in mind for now. Note that the command is using `singularity package <src> <dst>` argument order. This is different from our `build <dst> <src>`... but is generally more intuitive? Thoughts? Closes sylabs#2872
dtrudg
added a commit
to dtrudg/singularity
that referenced
this issue
May 23, 2024
Add a new command, `singularity data package`, that will create an OCI-SIF data container that 'packages' a host file or directory. The OCI-SIF data container consists of an OCI image, with a custom config mediaType. At present it will always hold a single layer, in squashfs format, containing the file / directory that was packaged. We currently use a flow that: * Reads the file / directory into a tar stream. * Presents the tar stream as a ggcr layer. * Mutates the layer into squashfs format. * Writes the image containing the mutated layer into an OCI-SIF. Arguably, given we are producing OCI-SIF with only a squashfs layer, we could directly call out to mksquashfs and then insert the layer from that - without going through a tar reader. We might adopt this optimisation in future, but for now we follow similar patterns to OCI-SIF container image handling, where we start with a tar layer and mutate. We probably need to consider what we do with layer format on push / pull, e.g. sylabs#2874 so let's keep tar in mind for now. Note that the command is using `singularity package <src> <dst>` argument order. This is different from our `build <dst> <src>`... but is generally more intuitive? Thoughts? Closes sylabs#2872
cstangeyanez
pushed a commit
to cstangeyanez/singularity
that referenced
this issue
Jun 4, 2024
Add a new command, `singularity data package`, that will create an OCI-SIF data container that 'packages' a host file or directory. The OCI-SIF data container consists of an OCI image, with a custom config mediaType. At present it will always hold a single layer, in squashfs format, containing the file / directory that was packaged. We currently use a flow that: * Reads the file / directory into a tar stream. * Presents the tar stream as a ggcr layer. * Mutates the layer into squashfs format. * Writes the image containing the mutated layer into an OCI-SIF. Arguably, given we are producing OCI-SIF with only a squashfs layer, we could directly call out to mksquashfs and then insert the layer from that - without going through a tar reader. We might adopt this optimisation in future, but for now we follow similar patterns to OCI-SIF container image handling, where we start with a tar layer and mutate. We probably need to consider what we do with layer format on push / pull, e.g. sylabs#2874 so let's keep tar in mind for now. Note that the command is using `singularity package <src> <dst>` argument order. This is different from our `build <dst> <src>`... but is generally more intuitive? Thoughts? Closes sylabs#2872
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
A new command e.g.
singularity dataor a subcommand onsingularity sif(TBD) should allow the creation of an OCI-SIF image that only holds data. This OCI-SIF image:The text was updated successfully, but these errors were encountered: