Stale files in opam-managed directories

[RalfJung]

From what I read in the docs, I expected opam 2 would automatically track which files are part of which package, and handle uninstallation. So when we switched to opam 2, I removed all mentions of uninstallation from our opam files.

But something has gone very wrong, because I just did a package upgrade which renamed a file, and now I have both the old and new files in _opam/lib/coq/user-contrib/iris/heap_lang/. This breaks everything. Looks to me like a serious bug somewhere in opam where it fails to properly clean up on package upgrades?

I'll have to fix this now by manually deleting files so I can keep working, but I'll make a tarball of that switch in case you need me to take a look at some parts of it. I'm afraid I don't know how to reproduce this... here's the log from the upgrade that included the rename:

[coq-iris-examples-builddep.dev] no changes from file:///home/r/Dokumente/Unisachen/iris/coq-users/examples/build-dep
The following actions will be performed:
  ↗ upgrade   coq-iris                   dev.2020-05-26.0.e3f87a95 to dev.2020-05-26.1.d80e7abf [required by coq-iris-examples-builddep]
  ↻ recompile coq-iris-string-ident      dev                                                    [uses coq-iris]
  ↻ recompile coq-iris-examples-builddep dev*
===== ↻ 2   ↗ 1 =====
Do you want to continue? [Y/n] y

<><> Gathering sources ><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
[coq-iris-examples-builddep.dev] no changes from file:///home/r/Dokumente/Unisachen/iris/coq-users/examples/build-dep
Processing  3/3: [coq-iris.dev.2020-05-26.1.d80e7abf: git] [coq-iris-string-ident.dev: git]
[coq-iris-string-ident.dev] no changes from git+https://gitlab.mpi-sws.org/iris/string-ident#v8.11
[coq-iris.dev.2020-05-26.1.d80e7abf] synchronised from git+https://gitlab.mpi-sws.org/iris/iris.git#d80e7abf681813798157a247899ecccddc3638f3

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
⊘ removed   coq-iris-examples-builddep.dev
⊘ removed   coq-iris-string-ident.dev
⊘ removed   coq-iris.dev.2020-05-26.0.e3f87a95
∗ installed coq-iris.dev.2020-05-26.1.d80e7abf
∗ installed coq-iris-string-ident.dev
∗ installed coq-iris-examples-builddep.dev
Done.

This switch is old-ish, and the coq-iris package in question has been upgraded many many times. When it was first installed, it was an opam 1 package definition installed into the opam 2 switch. Only later we upgraded the package definition to opam 2.

# opam config report
# opam-version      2.0.7 
# self-upgrade      no
# system            arch=x86_64 os=linux os-distribution=debian os-version=testing
# solver            builtin-mccs+glpk
# install-criteria  -removed,-count[version-lag,request],-count[version-lag,changed],-changed
# upgrade-criteria  -removed,-count[version-lag,solution],-new
# jobs              8
# repositories      4 (http), 3 (local), 2 (version-controlled) (default repo at 263e96c5)
# pinned            14 (rsync)
# current-switch    /home/r/Dokumente/Unisachen/iris/coq-users/examples

[rjbou]

Thanks for the report!

You read well the doc, because at each install, opam does a snapshot before and after install in order to know what has been installed, and then what to remove at uninstall.
This feature is present since 2.0-alpha (cf. #2568). Here what is important is the opam versionsthat did the install first, not the opam file version. It is that opam version that generates the changes file. Do you remember which one was it? Because even is the switch is oldish, it was upgraded when using newer opam versions, and each install with opam >=2.0 should have generated a corresponding change file.

Tested with opam/master & opam 2.0.7, the removal is well done

$ opam show foo -f install:
["mkdir" "-p" "%{lib}%/foo"]
["touch" "%{lib}%/foo/foo.%{version}%"]

$ opam install foo.0.1 -y
The following actions will be performed:
  ∗ install foo 0.1

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
∗ installed foo.0.1
Done.

$ find _opam/lib
_opam/lib
_opam/lib/foo
_opam/lib/foo/foo.0.1
_opam/lib/stublibs
_opam/lib/toplevel

$ opam upgrade -y
The following actions will be performed:
  ↗ upgrade foo 0.1 to 0.4

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
⊘ removed   foo.0.1
∗ installed foo.0.4
Done.

$ find _opam/lib
_opam/lib
_opam/lib/foo
_opam/lib/foo/foo.0.4
_opam/lib/stublibs
_opam/lib/toplevel

[RalfJung]

I am pretty sure I only started using opam 2.0 when there was the first release candidate. So the first install should have a sufficiently new opam already.

But back then, the package was still in opam 1 format -- not sure if that makes any difference.

[rjbou]

Packages format shouldn't have any incidence on the changes files, if opam is >= 2.0.

[dra27]

We'll proceed with the hunch that this is the same issue as #4419!

[RalfJung]

Yeah that sounds plausible.

[dra27]

Closing this one in favour of just keeping #4419 open for further tracking.