Internal `repos-config` new syntax #6393

rjbou · 2025-02-20T19:02:08Z

It is quite difficult and misleading to repository related information in the repos-config file (like in #4933 and #6283).
The current format contain a list with a some optional elements, which complicated the parsing

repositories: [ <string> { <URL> } { {<string>}+ } { <int> } ... ]: lists the configured repository idents, their URLs and trust anchors. The format is similar to repositories: from opamrc, except that the itself is optional.

The PR implements a new syntax for this file: each repository is a section with it's own defined labeled fields:

repo <string> {
  url: <URL>
  fingerprints:  <string>
  quorum: <int>
}

Adding then a new field is quite easy.

This change implies an hard upgrade (no best effort loading if read only) to update the opam root.

TODO:

update doc
update changes

kit-ty-kate

I'd be interested in a crash course on OpamPp at the next meeting to be able to understand the new Repos_configSyntax module better but overall otherwise that looks fine to me.

We might also want to discuss removing the soft/hard upgrade distinction.

kit-ty-kate · 2025-02-21T11:18:11Z

src/format/opamFile.ml

+      (fun ~pos:_ repos ->
+         let repos, errs = List.split repos in
+         let repos = List.filter_map Fun.id repos in
+         let map = OpamRepositoryName.Map.of_list repos in
+         let errs = List.flatten errs in
+         map, errs)
+      (fun (map, _errs) ->
+         OpamRepositoryName.Map.bindings map
+         |> List.map (fun x -> Some x, []))


These two functions can be done more efficiently in one fold each

kit-ty-kate · 2025-02-21T11:24:24Z

src/state/opamFormatUpgrade.ml

-       (List.map (fun (_, r, u) -> r, (u,None)) prio_repositories));
+       (List.map (fun (_, repo_name, repoc_url) ->
+            repo_name,
+            OpamFile.Repos_config.{repoc_url; repoc_trust = None})


It's probably best to use the proper naming syntax instead of the local open one to avoid confusion in the future between values and fields:

Suggested change

OpamFile.Repos_config.{repoc_url; repoc_trust = None})

{OpamFile.Repos_config.repoc_url; repoc_trust = None})

kit-ty-kate · 2025-02-21T11:26:39Z

src/state/opamFormatUpgrade.ml

+            let OpamFile.Repos_config_Legacy.{repoc_url; repoc_trust} =
+              old_repo
+            in
+            OpamFile.Repos_config.{repoc_url; repoc_trust})


Suggested change

let OpamFile.Repos_config_Legacy.{repoc_url; repoc_trust} =

old_repo

in

OpamFile.Repos_config.{repoc_url; repoc_trust})

let {OpamFile.Repos_config_Legacy.repoc_url; repoc_trust} =

old_repo

in

{OpamFile.Repos_config.repoc_url; repoc_trust})

kit-ty-kate · 2025-02-21T11:27:10Z

src/state/opamRepositoryState.ml

@@ -164,10 +164,10 @@ let load lock_kind gt =
         load with best-effort (read-only)"
      (OpamVersion.to_string (OpamFile.Config.opam_root_version gt.config))
      (OpamVersion.to_string (OpamFile.Config.root_version));
-  let mk_repo name (url, ta) = {
+  let mk_repo name OpamFile.Repos_config.{repoc_url; repoc_trust} = {


Suggested change

let mk_repo name OpamFile.Repos_config.{repoc_url; repoc_trust} = {

let mk_repo name {OpamFile.Repos_config.repoc_url; repoc_trust} = {

kit-ty-kate · 2025-02-21T11:27:29Z

src/state/opamRepositoryState.ml

@@ -280,7 +280,9 @@ let write_config rt =
  OpamFile.Repos_config.write (OpamPath.repos_config rt.repos_global.root)
    (OpamRepositoryName.Map.filter_map (fun _ r ->
         if r.repo_url = OpamUrl.empty then None
-         else Some (r.repo_url, r.repo_trust))
+         else
+           Some OpamFile.Repos_config.{ repoc_url = r.repo_url;


Suggested change

Some OpamFile.Repos_config.{ repoc_url = r.repo_url;

Some { OpamFile.Repos_config.repoc_url = r.repo_url;

kit-ty-kate · 2025-02-21T18:32:11Z

src/state/opamFormatUpgrade.ml

+       (OpamFile.make
+          (OpamFile.filename f)));


Suggested change

(OpamFile.make

(OpamFile.filename f)));

(OpamFile.make (OpamFile.filename f)));

kit-ty-kate · 2025-02-21T18:33:30Z

tests/reftests/repository.test


 # Packages matching: any
 # No matches found
 ### sh -c "rm OPAM/repo/state-*.cache"
 ### opam list -A
 [WARNING] Errors in ${BASEDIR}/OPAM/repo/repos-config, some fields have been ignored:
-            - At ${BASEDIR}/OPAM/repo/repos-config:2:16-2:23::
-              expected url
+            - In ${BASEDIR}/OPAM/repo/repos-config:


what is the reason we're loosing the position here?

kit-ty-kate · 2025-02-21T20:19:59Z

src/format/opamFile.ml

@@ -1386,7 +1386,7 @@ module ConfigSyntax = struct
  let internal = "config"
  let format_version = OpamVersion.of_string "2.1"
  let file_format_version = OpamVersion.of_string "2.0"
-  let root_version = OpamVersion.of_string "2.2"
+  let root_version = OpamVersion.of_string "2.4"


Suggested change

let root_version = OpamVersion.of_string "2.4"

let root_version = OpamVersion.of_string "2.4~alpha1"

rjbou added 5 commits February 17, 2025 17:17

version sett form opam root version

5884834

remove an upgrade test as it make no more sense

3cf5c86

New repo config file format

f6a1278

add 2.4 test

f865a06

fixup! New repo config file format

0e49b8f

rjbou requested review from kit-ty-kate and dra27 February 20, 2025 19:02

kit-ty-kate added this to the 2.4.0~alpha1 milestone Feb 20, 2025

rjbou mentioned this pull request Feb 21, 2025

New OpamFile.Repos_config type ocaml-opam/opam-rt#82

Open

kit-ty-kate reviewed Feb 21, 2025

View reviewed changes

kit-ty-kate linked an issue Feb 21, 2025 that may be closed by this pull request

Update internal repos-config file format #6327

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Internal `repos-config` new syntax #6393

Internal `repos-config` new syntax #6393

rjbou commented Feb 20, 2025

kit-ty-kate left a comment

kit-ty-kate Feb 21, 2025 •

edited

Loading

kit-ty-kate Feb 21, 2025

kit-ty-kate Feb 21, 2025

kit-ty-kate Feb 21, 2025

kit-ty-kate Feb 21, 2025

kit-ty-kate Feb 21, 2025

kit-ty-kate Feb 21, 2025

kit-ty-kate Feb 21, 2025

	OpamFile.Repos_config.{repoc_url; repoc_trust = None})
	{OpamFile.Repos_config.repoc_url; repoc_trust = None})

	let mk_repo name OpamFile.Repos_config.{repoc_url; repoc_trust} = {
	let mk_repo name {OpamFile.Repos_config.repoc_url; repoc_trust} = {

	Some OpamFile.Repos_config.{ repoc_url = r.repo_url;
	Some { OpamFile.Repos_config.repoc_url = r.repo_url;

	(OpamFile.make
	(OpamFile.filename f)));
	(OpamFile.make (OpamFile.filename f)));

	let root_version = OpamVersion.of_string "2.4"
	let root_version = OpamVersion.of_string "2.4~alpha1"

Internal repos-config new syntax #6393

Are you sure you want to change the base?

Internal repos-config new syntax #6393

Conversation

rjbou commented Feb 20, 2025

kit-ty-kate left a comment

Choose a reason for hiding this comment

kit-ty-kate Feb 21, 2025 • edited Loading

Choose a reason for hiding this comment

kit-ty-kate Feb 21, 2025

Choose a reason for hiding this comment

kit-ty-kate Feb 21, 2025

Choose a reason for hiding this comment

kit-ty-kate Feb 21, 2025

Choose a reason for hiding this comment

kit-ty-kate Feb 21, 2025

Choose a reason for hiding this comment

kit-ty-kate Feb 21, 2025

Choose a reason for hiding this comment

kit-ty-kate Feb 21, 2025

Choose a reason for hiding this comment

kit-ty-kate Feb 21, 2025

Choose a reason for hiding this comment

Internal `repos-config` new syntax #6393

Internal `repos-config` new syntax #6393

kit-ty-kate Feb 21, 2025 •

edited

Loading