pin local source tree does not pick up locally modified extra-files

[jberdine]

1. place an opam directory in the root of a local source tree of a package
2. copy the opam file and files directory from opam-repository into the local opam directory
3. modify the opam/opam file and something in opam/files/
4. pin the local directory

What I see is the updated opam file being taken into account, but the extra files are taken from the opam-repository rather than using the local version. (The build fails without the local changes to the extra-files.)

I encountered this when trying to pin a modified llvm (master plus this diff). The package is the llvm ocaml bindings, and to install the opam package with my local changes I need to modify files/META.patch. See also discuss.

[rjbou]

I'll need more information on this one. I reproduced your use case, and extra-files get updated, not taken from repository.
If llvm is git pinned, did you committed your patch modifications?

Hint for further investigation: (you can share your full log if more convenient)

• opam internal overlay directory contains opam specific pinning files.

$ pwd
llvm/_opam/.opam-switch/overlay/llvm/files

$ grep -C 1 debuginfo META.patch 

package "debuginfo" (
    requires = "llvm"
--
    description = "Debug information in LLVM IR form"
-    archive(byte) = "llvm_debuginfo.cma"
-    archive(native) = "llvm_debuginfo.cmxa"
+    archive(byte, -llvm.static) = "shared/llvm_debuginfo.cma"
+    archive(native, -llvm.static) = "shared/llvm_debuginfo.cmxa"
+    archive(byte, llvm.static) = "shared/llvm_debuginfo.cma"
+    archive(native, llvm.static) = "shared/llvm_debuginfo.cmxa"
 )

• As patches and extra files are retrieved before build process, you can check by looking at --debug output:

01:26.158  SYSTEM  [log-6425-b76d46] (in 0.033s) cp llvm/_opam/.opam-switch/overlay/llvm/files/link-META.patch llvm/_opam/.opam-switch/build/llvm.8.0.0/link-META.patch
01:26.187  SYSTEM  [log-6425-ad6566] (in 0.028s) cp llvm/_opam/.opam-switch/overlay/llvm/files/install.sh llvm/_opam/.opam-switch/build/llvm.8.0.0/install.sh
01:26.205  SYSTEM  [log-6425-8b8a2d] (in 0.018s) cp llvm/_opam/.opam-switch/overlay/llvm/files/fix-shared.patch llvm/_opam/.opam-switch/build/llvm.8.0.0/fix-shared.patch
01:26.222  SYSTEM  [log-6425-dc6e87] (in 0.017s) cp llvm/_opam/.opam-switch/overlay/llvm/files/META.patch llvm/_opam/.opam-switch/build/llvm.8.0.0/META.patch
01:26.238  SYSTEM  [log-6425-48c407] (in 0.016s) cp llvm/_opam/.opam-switch/build/llvm.8.0.0/fix-shared.patch /home/rjbou/.opam/log/processed-patch-6425-ebe0e0

• if you run on verbose mode -vv, it is possible to see what has been taken into account for the build/install in the remaining opam internal build directory.

$ pwd
llvm/_opam/.opam-switch/build/llvm.8.0.0
$ git status
[...]
Untracked files:
  (use "git add <file>..." to include in what will be committed)
	META.patch
	fix-shared.patch
	install.sh
	link-META.patch
[...]

$ grep -C 3 debuginfo META.patch
+    archive(native, llvm.static) = "static/llvm_executionengine.cmxa"
 )

package "debuginfo" (
    requires = "llvm"
    version = "@PACKAGE_VERSION@"
    description = "Debug information in LLVM IR form"
-    archive(byte) = "llvm_debuginfo.cma"
-    archive(native) = "llvm_debuginfo.cmxa"
+    archive(byte, -llvm.static) = "shared/llvm_debuginfo.cma"
+    archive(native, -llvm.static) = "shared/llvm_debuginfo.cmxa"
+    archive(byte, llvm.static) = "shared/llvm_debuginfo.cma"
+    archive(native, llvm.static) = "shared/llvm_debuginfo.cmxa"
 )

 package "ipo" (

[rjbou]

ping @jberdine - what is the status of this issue ?

[jberdine]

(Sorry I am slow @rjbou, it's just that testing this takes a lot of context switching.)

First, thanks a lot for your hints and suggestions, they have been very helpful for me to understand what is going on underneath.

I am not sure how to proceed. It might be that this use case is just hard to support smoothly. That is, for a package where the opam metadata is not in the source repo, but only in opam-repo, then the extra files are not going to be committed, so git-pinning a local clone of the package doesn't have a smooth way of using updated extra files. For my particular use case, it would be convenient if opam looked in the locally git-pinned directory for metadata files even if they aren't committed. But I don't know if there are other bad consequences of doing that. Also, for prioritization purposes, in this particular case it seems that I can work around this difficulty by rearranging things and using a directory pin instead of git. But if the package's build system wasn't flexible enough to allow moving things out from under the package source directory, I'd be stuck.

[rjbou]

It's my turn to feel sorry, I don't identify the issue :)
Let's simplify it, i created a simple test case:

• add a package with extra-files, on a repo
• create a local version of this package, git versioned directory, and modify the extra file
  → test install from repo & install from local pin ✔️

You can find the material in this gist
Can you confirm that it is your process ?

it would be convenient if opam looked in the locally git-pinned directory for metadata files even if they aren't committed

opam 1.2 was mixing directory & git pinning. With opam 2.0, we choose to separate those two kind for simplicity & consistency. There is some options you can use like --working-dir that takes all you working directory (committed and uncommitted files) as source.

If you want to share the modified extra files (as said in the discuss post), why don't you version them?

[dra27]

Cross referencing #4652 which came to the conclusion of "won't fix"