Usage of quick_digest to track installed files doesn't work on Docker

[kit-ty-kate]

Currently Docker doesn't store the nanosecond part of mtime of files in-between commands (see moby/moby#40004). However opam does use and store the nanosecond part and thinks the file changed and thus doesn't remove the files when instructed to.

Example (taken from the docker issue above):

docker build --no-cache .                                                                                                                                                                                                        [4.07.1]
Sending build context to Docker daemon  2.048kB
Step 1/4 : FROM ocaml/opam2
 ---> 2dae3f7f8c97
Step 2/4 : RUN touch test.ml
 ---> Running in e454ba2f273e
Removing intermediate container e454ba2f273e
 ---> 252832be85a4
Step 3/4 : RUN opam exec -- ocamlopt test.ml && stat test.cmx
 ---> Running in aa36444a3335
  File: test.cmx
  Size: 167       	Blocks: 8          IO Block: 4096   regular file
Device: 3eh/62d	Inode: 26825932    Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/    opam)   Gid: ( 1000/    opam)
Access: 2019-09-27 21:56:21.377654651 +0100
Modify: 2019-09-27 21:56:21.373654605 +0100
Change: 2019-09-27 21:56:21.373654605 +0100
 Birth: -
Removing intermediate container aa36444a3335
 ---> fec7beec4029
Step 4/4 : RUN stat test.cmx
 ---> Running in 9921d4dc9239
  File: test.cmx
  Size: 167       	Blocks: 8          IO Block: 4096   regular file
Device: 3eh/62d	Inode: 26826002    Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/    opam)   Gid: ( 1000/    opam)
Access: 2019-09-27 21:56:21.000000000 +0100
Modify: 2019-09-27 21:56:21.000000000 +0100
Change: 2019-09-27 21:56:21.789659327 +0100
 Birth: -
Removing intermediate container 9921d4dc9239
 ---> 02d86804a92a
Successfully built 02d86804a92a

This has caused problem to the CI several times (e.g. ocaml/opam-repository#14938 (comment), ocaml/opam-repository#14685 (comment), …)

[AltGr]

Have you tried setting OPAMPRECISETRACKING=1 ? It should switch to using checksums instead ; these were only disabled by default for speed reasons.

[AltGr]

Big thumbs up for digging up this issue too 👍

[kit-ty-kate]

Have you tried setting OPAMPRECISETRACKING=1 ? It should switch to using checksums instead ; these were only disabled by default for speed reasons.

It does work! Thank you! This issue is now mitigated in the opam-repository CI. It might be worth adding it by default to the docker images as well. What do you think @avsm?

[kit-ty-kate]

cc @talex5 as well for the ocurrent images

[ejgallego]

cc @erikmd is this relevant to Coq's images? See also coq-community/bignums#36

[kit-ty-kate]

@ejgallego I believe it is

[cfcs]

Isn't this technically speaking a bug in Docker?

[RalfJung]

From what I gather the issue is that nanoseconds in timestamps are not preserved, and opam doesn't like that?

It's not just Docker, we had issues on CI for more than a year until we realized that the CI cache likely does not preserve nanoseconds and hence we had to set OPAMPRECISETRACKING and re-generate all caches.

One thing that would help is for opam to detect that the timestamp is almost right, and say as much in the error/warning that it shows, and tell users about the env var as a possible fix.

[dra27]

Possible scheme for this discussed last Friday which @AltGr might get to for 2.2.0 (:wink:):

• Always compute checksums on installation of files, so have the .changes file always include a checksum and a timestamp
• On scanning, use the timestamp first (unless OPAMPRECISETRACKING=true, obviously). If the timestamp differs, then use the checksum. If the checksum is correct, update the timestamp.
• For docker containers, it might be worth adding a subcommand or option to opam clean or opam config which force rescans all .changes files. This would allow a Dockerfile to do RUN opam config check-changes (or something) as a separate step from RUN opam install 300-packages. Another option for Docker might be to have a configuration option in .opam/config which explicit disables comparison of nanoseconds.