Could not update local repository: processed-patch failed

[madroach]

On OpenBSD I regularly get this error when updating from my local opam repository:

<><> Updating package repositories ><><><><><><><><><><><><><><><><><><><><><><>
[local] synchronised from file:///home/madroach/src/opam-repository
[ERROR] Could not update repository "local": "/usr/bin/patch -p1 -i /home/madroach/opam/log/processed-patch-63718-42bb52" exited with code 2

I can work around this problem by removing and re-adding the repository.

# opam config report
# opam-version      2.0.0~rc3 
# self-upgrade      no
# system            arch=x86_64 os=openbsd os-distribution=openbsd os-version=6.3
# solver            builtin-mccs+glpk
# install-criteria  -removed,-count[version-lag,request],-count[version-lag,changed],-changed
# upgrade-criteria  -removed,-count[version-lag,solution],-new
# jobs              1
# repositories      1 (local)
# pinned            0
# current-switch    default

[AltGr]

Thanks for reporting. If you have tested an earlier beta or rc, it would be useful to know whether the error is new or wals already there? Thanks.

[madroach]

I'm not 100% sure, but I think I saw it also with rc2, but did not test any earlier version. 1.2 had no problems updating repositories.

[madroach]

OK, I found the problem. Question remains on how to fix it.
Half of the problem is that OpenBSD patch will always save backups.
This behaviour can be turned of with this patch:

Index: src/core/opamSystem.ml
--- src/core/opamSystem.ml.orig
+++ src/core/opamSystem.ml
@@ -1131,7 +1131,9 @@ let patch ~dir p =
      raise Not_found);
   let p' = temp_file ~auto_clean:false "processed-patch" in
   translate_patch ~dir p p';
-  make_command ~name:"patch" ~dir "patch" ["-p1"; "-i"; p'] @@> fun r ->
+  make_command
+    ~env:Array.(append default_env [|"POSIXLY_CORRECT=yes"|])
+    ~name:"patch" ~dir "patch" ["-p1"; "-i"; p'] @@> fun r ->
     if not (OpamConsole.debug ()) then Sys.remove p';
     if OpamProcess.is_success r then Done None
     else Done (Some (Process_error r))

The other half of the problem is OpenBSD patch not supporting binary diffs.
A binary diff is generated when diff is run over .git/index.
Reading opamLocal.ml I suspect the diff is not supposed to diff the .git subdirectory, since its excluded when using rsync, but still copied when using plain cp, which also means copying a whole lot more of date.
Maybe this needs fixing too? Always use rsync to exclude VCS files?
For now I'll patch OPAM to use gpatch on OpenBSD.

[AltGr]

This issue was actually reproduced on Linux too. In that case, it's due to several factors:

• stale binary files in ~/.opam/repo/*/ (these happen here due to git, but may happen due to an upgrade from 1.2 as well)
• the patch internally generated by opam being rewritten by 9ec2458 , which apparently breaks binary patches.

This combination results in a dire breakage on the 1.2.2→2.0.0~rc3 upgrade path, so it's quite critical.

Thanks for reporting: this will need to be fixed before 2.0.0 proper.
Note: can you confirm that OpenBSD's patch doesn't support binary content ? In which case working around 9ec2458 wouldn't be enough to fix for you (it would be for the 1.2.2→2.0.0~rc3 upgrade).

[AltGr]

Note: if you are in the broken state after the 1.2.2 update:
Symptoms:

[ERROR] Could not update repository "default": "/usr/bin/patch -p1 -i xxx/processed-patch-20636-b1a82b" exited with code 1
then
The following dependencies couldn't be met:
  - ocaml-system → ocaml
      unknown package

Solution:

$ rm -f ~/.opam/repo/*/index.tar.gz*
$ opam update

[madroach]

I can confirm OpenBSD's patch is broken for binary content.
OpenBSD's patch currently shows undefined behaviour on binary patches. I just proposed a patch which would cause OpenBSD's patch to error out on binary content, which would be better than undefined behaviour, obvioursly.

[madroach]

Can the diff be confined to the "interesting" files? Like so:

for x in repo version packages
do
  diff -aurN "opam/repo/local/$x" "opam/repo/local.new/$x"
done >diff

[AltGr]

Ok, thanks. We'll probably require gpatch for the 2.0.0 release, but work on a better solution afterwards.

[johnwhitington]

Has this been fixed or worked around yet?

I am writing the new up-and-running page for ocaml.org, so I copied the instructions from opam.ocaml.org to use brew install gpatch prior to brew install ocaml on MacOS. Do I still need this?

[johnwhitington]

Patch underway: Homebrew/homebrew-core#64301

[rjbou]

Yes, gpatch is required for open bsd & free bsd, see init defaults

>  opam init --show-default-opamrc
opam-version: "2.0"
[...]
required-tools: [
 [...]
 "patch" {} {!(os = "openbsd" | os = "freebsd")}
 "gpatch" {} {os = "openbsd" | os = "freebsd"}
[...]

[kit-ty-kate]

This was fixed in 2.1.6 by #5893