Feature request: filter repository with a set of package invariants

[hannesm]

Dear all,

my usecase is the following: I have a private opam repository and would like to easily remove packages that are not installable since e.g. the OCaml version isn't met (or some other filter). I came across "opam admin filter", but am not sure how to use it.

An example from the real world:

• remove all "ocaml-freestanding" packages (since this isn't relevant anymore)
• restrict OCaml to >= 4.14.0 & < 5
• only keep mirage=4.4.1

Using these three things as input and an opam repository, I'd like to run the solver and get as output which packages are fine to remove (since they'll never be installed anyways - for example since the ocaml version is not met).

Thanks a lot for your continued hard work on opam :D

[kit-ty-kate]

mmh, indeed opam admin filter doesn’t seem to be the most useable outside of very small examples and even seems buggy in places from what i’ve tried (e.g. opam admin filter --recursive --or --depends-on ocaml.4.14.0 --required-by ocaml.4.14.0 doesn’t really do what the documentation says it should do), and --installable and --coinstallable-with are more than extremely slow.

In the meantime, in case it’s useful for you, here is a script i’ve made yesterday that scans all the repositories you have installed for the current switch, and according to the list of packages returned by opam list --available --installable --short --all-versions (but you can use any other command ofc), will create a new repository in /tmp/tmp-opam-repo with only those packages:

mkdir /tmp/tmp-opam-repo
cd /tmp/tmp-opam-repo
repos=$(opam repo list -s)
for pkg in $(opam list --available --installable --short --all-versions) ; do
  path="packages/$(echo "$pkg" | cut -d. -f1)/$pkg"
  for repo in $(echo "$repos") ; do
    opamfile="/home/opam/.opam/repo/$repo/$path/opam"
    if test -f "$opamfile" ; then
      mkdir -p "$path"
      cp "$opamfile" "$path/"
      cp -r "$(dirname "$opamfile")/files/" "$path/"
      break
    fi
  done
done
echo 'opam-version: "2.0"' > repo

[hannesm]

Thanks for your reply @kit-ty-kate and further discussions. I came up with the following:

#!/bin/sh
set -e

RM="git rm -rf --ignore-unmatch"
OCAML_VERSION="4.14.1"
MIRAGE_VER="4.4.1"
MIRAGE_CRYPTO_VER="0.11.2"
DUNE_VER="3.11.1"
BRANCH="master"
OUR_BRANCH="pruned-opam"

if [ $(git branch --show-current) != $BRANCH ]; then
    echo "you're not on the branch $BRANCH!"
    exit 1
fi

if [ $(git branch --list | grep -c $OUR_BRANCH) -eq 1 ]; then
    echo "branch '$OUR_BRANCH' already exists"
    exit 1
fi

git checkout -b $OUR_BRANCH

# as a first step, we have some orphaned packages that we can just remove
cd packages
$RM solo5-bindings* solo5-kernel* ocaml-freestanding nocrypto jbuilder mirage-xen-* mirage-types mirage-types-lwt

# we are also only interested in one ocaml version
find ocaml -type d -mindepth 1 -maxdepth 1 | grep -v $OCAML_VERSION | xargs $RM
find ocaml-variants -type d -mindepth 1 -maxdepth 1 | grep -v $OCAML_VERSION | xargs $RM

# let's only consider the latest mirage/mirage-runtime/functoria/functoria-runtime
find mirage mirage-runtime functoria functoria-runtime -type d -mindepth 1 -maxdepth 1 | grep -v $MIRAGE_VER | xargs $RM

# let's only consider the latest mirage-crypto-*
find mirage-crypto mirage-crypto-* -type d -mindepth 1 -maxdepth 1 | grep -v $MIRAGE_CRYPTO_VER | xargs $RM

# also only latest dune
find dune dune-* -type d -mindepth 1 -maxdepth 1 | grep -v $DUNE_VER | xargs $RM

cd ..

git commit -m "removed unneeded packages"

# let's remove obsolete packages
for pkg in $(opam admin check --obsolete --short); do p=$(echo $pkg | cut -d '.' -f 1); if [ $p != "ocaml-config" ] && [ $p != "ocamlbuild" ]; then $RM packages/$p/$pkg; fi; done
for pkg in $(opam admin check --obsolete --short); do p=$(echo $pkg | cut -d '.' -f 1); if [ $p != "ocaml-config" ] && [ $p != "ocamlbuild" ]; then $RM packages/$p/$pkg; fi; done
for pkg in $(opam admin check --obsolete --short); do p=$(echo $pkg | cut -d '.' -f 1); if [ $p != "ocaml-config" ] && [ $p != "ocamlbuild" ]; then $RM packages/$p/$pkg; fi; done

git commit -m "removed obsolete packages"

# only take installable stuff into account
for pkg in $(opam admin check --installability --short); do p=$(echo $pkg | cut -d '.' -f 1); $RM packages/$p/$pkg; done

git commit -m "removed uninstallable packages"

# a second round of obsolete packages
for pkg in $(opam admin check --obsolete --short); do p=$(echo $pkg | cut -d '.' -f 1); if [ $p != "ocaml-config" ] && [ $p != "ocamlbuild" ]; then $RM packages/$p/$pkg; fi; done
for pkg in $(opam admin check --obsolete --short); do p=$(echo $pkg | cut -d '.' -f 1); if [ $p != "ocaml-config" ] && [ $p != "ocamlbuild" ]; then $RM packages/$p/$pkg; fi; done
for pkg in $(opam admin check --obsolete --short); do p=$(echo $pkg | cut -d '.' -f 1); if [ $p != "ocaml-config" ] && [ $p != "ocamlbuild" ]; then $RM packages/$p/$pkg; fi; done

git commit -m "removed obsolete packages again"

# second round of installable packages
for pkg in $(opam admin check --installability --short); do p=$(echo $pkg | cut -d '.' -f 1); $RM packages/$p/$pkg; done

git commit -m "removed uninstallable packages again"

git checkout $BRANCH

This uses opam admin check --installability and opam admin check --obsolete (note below) with a handcrafted list of old things (and a handcrafted list of which version of which package to retain) -- obviously not generally usable (and only for OCaml 4.14.1). The script runs on my (old) laptop in roughly 15 minutes. There's no active opam switch required.

Numbers:
full opam-repository: 29686 opam files
after this script: 7167 opam files

Removals in details:

• first commit (hardcoded packages and versions): 1985 files removed
• second commit (obsolete): 10667 files removed
• third commit (installability): 11478 files removed
• fourth commit (obsolete): 571 files removed
• fifth commit (installability): 35 files removed

Certainly there are tradeoffs when to run what phase, but the above script was time-wise pretty well (better than in other combinations).

There's a considerable speed improvement for all opam commands when using the stripped-down version of the repository. I'll likely test this further in more environments.

NOTE: opam admin check --obsolete: this unfortunately removes ocaml-config.2 and ocamlbuild.0.14.2, which leads to a opam repository that isn't very nice. That's why I guard those two packages in the loop. Also, the obsolete isn't idempotent (so running it multiple times leads to further results).

Since I manually remove the ocaml versions where I have barely any interest, there's no need to use opam admin filter --coinstallable-with anymore.

EDIT: I pushed the branch generated by the script above to https://github.com/hannesm/opam-repository.git#pruned-opam so you can play around with it :)