Allow to define dependencies as "development only", regardless how the package is built

[jchavarri]

At the moment, there is the option of using {dev} variable to define "development dependencies" in a package X. However, this doesn't work when the applications or packages that depend on X are not building it from a release archive.

As a more concrete example, as an author of a package, I would like to define ocamlformat as a development dependency of my package for those that want to contribute and develop features on it, but not impose ocamlformat as a dependency of those users that depend on my package, regardless how they build it (from a release archive or otherwise).

Maybe there could be a new option --recommended-dev to support this use case, or a with-dev variable, similar to with-test and with-doc.

[dra27]

From dev meeting: {with-dev} and --with-dev are getting the vote! It's mildly confusing to have {dev} and {with-dev} but the fact that the CLI and the variable are the same should help mitigate.

[rjbou]

Do we need this variable only in packages filters (depends:, depopts, etc.) or commands too (install:, build:, etc.)?

[kit-ty-kate]

Only depends and depopts would suffice I’d say. Though whatever is the most consistent with the rest (I see no issues also allowing it for install and build). Having it in fields such as post-messages could be useful too

[rjbou]

I don't see issues on adding them for install or build, more wondering if it makes sense (do we want to add build commands regarding the setup?), and will it interact / be understood regarding the already present dev var.
Good point for *-messages!

[dra27]

OCaml has developer-mode settings in configure, for example, it’d be handy to be able to turn them on (or, rather, prevent them from being turned off, which is the usual opam default)

[Et7f3]

On some reason project the build command is wrapped with refmterr a tool that convert errors messages in reason syntax and propose fix (a bit less useful since OCaml has better error message). So having two build command can be handy. Also dev might be tempted to use --profile dev When building with dune. The same manner you activate -Wall -Werror on a C project.

I don't see reason to have install dependent of this variable. Maybe more sanity check when installing in dev mode.

[rjbou]

backported comment from PR #5016 (comment)

I guess I'm a bit late to the party but I think --with-tools is a pretty confusing name. Packages do install tools or not (I have many packages that optionally install command line tools if cmdliner is installed). I can already see people trying to install them using this command line switch. Maybe --with-dev-tools ?
(Personally I'm not entirely convinced by the actual need for the initial request, I'm not sure what's wrong with tagging these dependencies with dev, it looks like micro management of dependencies)
In fact I think for-dev would be a much better and less confusing name.

/cc @dbuenzli

[kit-ty-kate]

Reopening as this should be closed by #5214, not #5160

[dbuenzli]

@kit-ty-kate asked me to comment on #5214 but I'm going to do this here.

1. The change in preposition from with to for in for-dev is unfortunate and will lead to confusions – I say it even more as I suggested it.
2. I remain doubtful that this the right way of handling the initial request. Introducing a new field for that seems a better idea to me.

Regarding 1. I think there's quite a bit of potential confusion on opam pin. For example if I do opam pin add PKG --for-dev people might think it implies --dev. Same on opam install PKG --for-dev, I would rather think that this will pin to dev and install dev dependencies while IIUC this will simply install PKG and its for-dev dependencies (should the option be with-for-dev ?).

Regarding 2. I think it boils down to define what the semantics of the depends: field should be. Up to now this field was always about how to build the package (including its tests and documentation). Now it seems that people also want to use this field to encode developer tool preferences.

If you ask me, depends: is not the place to put these things, I think they should go into a separate field. depends: should remain about building.

If we take debian as an example it has a richer vocabulary to express package relationships with Enhances, Suggests or Recommends. I don't think opam has to copy that design to its full extent but at least it suggests that lumping everything into depends: may not be a great idea.

So here is my suggestion:

1. Add a new dev-setup: [ <filtered-package-formula> ... ] field which allows to list these packages recommended for development.
2. Add a --{with,only}-dev-setup flag to install these.

[kit-ty-kate]

Thank you for the review!

• I like the with-dev-setup idea very much (better than for-dev i must say, it keeps the with-* theme 😅).

• However the second suggestion (adding a new dev-setup field) is good on paper but comes with some issues. For example:

  • This is not backward compatible with opam 2.0, so in theory we would have to wait opam 3.0 to add it (which we are waiting to also add more breaking changes so it would delay the release of the feature quite a while)
  • In theory users might also want to do something special with their regular build dependencies or build steps in dev-setup mode (for whatever reason) and thus having a with-dev-setup variable available in depends and build could be useful. This duplicates where users could add their dev-setup dependencies (either in dev-setup or in depends with with-dev-setup) and removes a bit of the usefulness of having a separate field.

So for these reason, I'm in favour of changing with-tools/for-dev in #5214 to with-dev-setup as-is, without adding a new field.

[dbuenzli]

• In theory users might also want to do something special with their regular build dependencies or build steps in dev-setup mode (for whatever reason) and thus having a with-dev-setup variable available in depends and build could be useful.

Sometimes it's better to not let people do what they could want in theory. It's always possible to add arbitrary complexity to a system and programmers seem especially keen to do so in their builds. Without a concrete use case I wouldn't let the design be guided by this "theory". Besides in the case you mention it doesn't seem horrible to then simply add the dependency as dev dependency.

I think having a clear separation between depends and dev-setup is also good for the curation of the opam-repository, there's already enough noise to sift through.