Add ability to override specific package manifest without editing files on disk

[MatthewDolan]

Problem Statement

We have a few use cases where we want to test a library published by hermit in another repository before publishing it (CI on each pull request for the package). We implement this today by creating a new, local hermit-packages directory @ bin/hermit-packages and then edit bin/hermit.hcl to point to that local hermit-packages as a source. Roughly:

cat <<EOF > bin/hermit-packages/example.hcl
description = "Description for example"
binaries = ["example"]
channel "unstable" {
  update = "5m"
  source = "git@github.com:cashapp/example.git#$EXAMPLE_VERSION"
}
EOF

echo 'sources = ["env:///bin/hermit-packages", "https://github.com/cashapp/hermit-packages.git"]' >> bin/hermit.hcl

hermit install example@unstable

This works, but it feels hacky, it leaves behind a dirty git repo at the end of the build, doesn't work if the code is read-only, and it can be unintuitive to follow along with the edits to disk as a part of CI and reconstruct what the files will look like at the end of the end.

Proposed Solution Options

The proposed solution consists of two parts:

1. Add a way to configure sources using command arguments / environment variables. To avoid having to edit bin/hermit.hcl. Roughly:

HERMIT_ADDITIONAL_SOURCES='...' hermit ...

Add these additional sources after builtin:/// and before the sources specified in bin/hermit.hcl.

2. Add a way to specify sources without having to read them from disk. There are two rough proposals out there:

a. Use data uri scheme. Roughly:

example_config=$(cat <<EOF
description = "Description for example"
binaries = ["example"]
channel "unstable" {
 update = "5m"
 source = "git@github.com:cashapp/example.git#$EXAMPLE_VERSION"
}
EOF
)

HERMIT_ADDITIONAL_SOURCES="data:application/json;base64,$(echo "{\"example\":$(echo "$example_config" | jq -R -s '.')}"| base64)" hermit ...

Which results in a source which looks something like this:

data:application/json;base64,eyJleGFtcGxlIjoiZGVzY3JpcHRpb24gPSBcIkRlc2NyaXB0aW9uIGZvciBleGFtcGxlXCIKYmluYXJpZXMgPSBbXCJleGFtcGxlXCJdCmNoYW5uZWwgXCJ1bnN0YWJsZVwiIHsKIHVwZGF0ZSA9IFwiNW1cIgogc291cmNlID0gXCJnaXRAZ2l0aHViLmNvbTpjYXNoYXBwL2V4YW1wbGUuZ2l0I21haW5cIgp9CiJ9Cg==

Bonus points for (2.a.), it's a valid URI and can be put into some browsers to render the json.

b. Use a special inline://.... Roughly:

HERMIT_ADDITIONAL_SOURCES='inline:{"example":{"description":"Description for example", ..., "channel": {"unstable": {"update: "5m", ...}}' hermit ...

Bonus points for (2.b.), it's way more readable, but is not a valid URI (it has invalid characters like spaces).

Still need to decide on (2.a.) or (2.b.) but going to roughly do both versions and see how the code looks on either side.

(Some discussion occurred outside of Github, but capturing here to memorialize.)