Feature request: Store multiple registered packages in a single Git repository #1251
Comments
|
As far as installing package goes, this already works. Where is the support lacking? Real question: what doesn't work when you try it? |
|
I've set up a working example.
Minimum working exampleThe Git repository The registry
|
|
Thanks @DilumAluthge, that helps push this forward considerably. So the question with Pkg.develop is: What do we want this to do? What should developing a package that is not at the root of its repo do? For Registrator, I think it's pretty straightforward: we should look for project files anywhere in the repo. However, if there's more than one, which one is supposed to be registered? I had originally envisioned that Registrator would work by just looking for commits changing the version number in any project file and then considering that commit to be a trigger for registering the new version of the package where the project file lives. @nkottaary, @KristofferC, any thoughts? |
For developing packages that are not at the root of the repo, we could require that the user provide the path (relative from the repo root) to the package. This could be a new field in Pkg.develop(Pkg.PackageSpec(name = "Foo", subdirectory = "myfolder"))
Pkg.develop(Pkg.PackageSpec(name = "Bar", subdirectory = "otherfolder/subfolder"))Alternatively, it could be a keyword argument to Pkg.develop("Foo"; subdirectory = "myfolder")
Pkg.develop("Bar"; subdirectory = "otherfolder/subfolder")
For registrator, if the project file is not located in the project root, we could similarly require the user to supply the location of the package within the repo. For example, this would register And this would register |
It should be relatively easy to find the subdirectory by looking for the correct Edit: I guess just a URL is not enough, but url + name? |
|
Thanks for the excellent summary @DilumAluthge. I personally like the explicitness of providing the relative path to the package in question. Like you said, this could be used both for Registrator and |
|
Aside from the keyword Alternatively, what if used URL fragment identifiers here? I.e. We could interpret the fragment as a path first, look at that path for a project file; failing that, if the name doesn't have |
|
We kinda used I think it is fine to start simple with only providing the |
|
How about using the query string? (I'm borrowing the idea from pip https://pip.pypa.io/en/stable/reference/pip_install/#vcs-support ) |
|
This is a feature we'd also like to have. For DataKnots we'll be growing an ecosystem of adapters of various sorts. Each one of it has glue code, and it'd be rather silly to have a unique repository for each since it complicates releases and makes it harder to find what you are after. Has there been any progress lately on making this work? This topic was asked last year on the forums. |
|
Marking for triage to be discussed on the next Pkg call so that we can make a decision here. This shouldn't be too hard to do, we just need to decide how to express it. |
|
Hurray... there is a WIP, #1422 ;) |
|
And hurray, #1422 was merged! Progress! I am really hoping this can make it into the next LTS |
|
Also, referencing pull request #11169 |
|
Implemented |
Why would an Ad library has to be tagged in lock step with Flux? And why would stuff start to fail if only one of these are tagged. This just looks like bad developer practice and has nothing to do with Pkg. Define a proper API and expose it and don't break stuff all the time? |
How do the CI triggers work? And are the dependencies per sub-package? |
LocalRegistry made JuliaRegistries/General#16663 easy. It might be ideal to give Registrator the ability to register multiple subdirs, but this is the least of my worries.
While I'd certainly like to make the act of splitting or merging seamless, my biggest concern is whether we've really thought through the ongoing development cycle. As I pointed out in #1251 (comment), suppose When I submit this for CI, I'm reasonably sure Pkg is going to barf, because it has no idea where it will get v2 of EDIT: I don't think I even needed to go to 2.0.0, I think the same thing would happen if some whiz-bang new (but non-breaking) feature were added to SubPkgA and so everyone bumps to 1.9.0. |
🤦 . It's following all of the good developer practices that were laid out for the auto-merge era. It gave an update to Adapt 2.0, an update to Zygote that wants that, and then when Pkg decides to serve that, everything else falls back to a version that was either before it used Adapt or before they used good versions, so you get a version of Tracker that brings the whole world down until all of the Adapt 2.0 updates go through, and any time you add a package that bounds Adapt 2.0 Pkg finds this as a valid solution. So why does everything need to move in lock step? Because our good developer practices easily leads to these situations 🤷 . I can know that this nonsense leads to 2% of DiffEq failing tests, but instead, good developer practice means that 100% of DiffEq/Turing/Pumas/etc. is unusable with the current standard registry. I think the most salient issue of the current situation is that good developer practices is unable to decouple the issues of Flux from the rest of SciML because of the hard bounds. A "meh" solution isn't even possible without coordinating 50 other packages outside SciML to join in, so instead every time Flux makes a move we just have to sit back and wait for the whole ecosystem to sort itself out again. |
Right now I run all tests as part of the "outer" package (SnoopCompile is not huge). There, my intention is to have everything advance in lock-step. With JuliaImages (which is bigger, though not as big as your ecosystem) I might want to have the ability to run a subset of tests, not sure. We've often thought about running tests for "meta" packages when updating focused packages and vice versa, e.g., JuliaImages/Images.jl#468. I seem to remember others but can't find them now, maybe @johnnychen94 knows. Even though a CI run of everything would take quite a long time, it would have advantages.
Yes, in that SnoopCompile repo you can see each subpackage (just a subdirectory of the top-level repo) has its own Project.toml file. Most dependencies of the sub-packages are not declared as direct dependencies of the wrapper package. |
|
@timholy Do you mean JuliaImages/Images.jl#805 (and probably the docs JuliaImages/Images.jl#843) |
|
I attempted to explain a package-dependency use case for DataKnots -- basically, we have glue packages we'd like in a monorepo. We would like a top-level documentation and CI tests inclusive of all optional modules (JSON, XML, SQL, etc.); this top-level need isn't a Julia package, or at least not one you'd want to install. It would be nice to be able to commit the entire ecosystem with a single version, even though they are modeled as separate Julia packages. Tim, is this similar to your situation? |
|
Here's another example, one which you're familiar with @KristofferC. There's a big rewrite of Revise brewing in timholy/Revise.jl#497. Because Revise is used by lots of people who seem to have very different workflows, I'd really like to get some beta testers before inflicting it on the world. Revise depends on CodeTracking, JuliaInterpreter, and LoweredCodeUtils, and to make the rewrite possible all of them need changes. Because we have upper bounds---and because I know the difficult process of getting testers will be even harder if I have to tell people to Except it doesn't. In addition to timholy/Revise.jl#508, there's also the issue that Pkg's resolver notices the fact that it can increase the version on two packages, CodeTracking and LoweredCodeUtils, if it lowers the version of Revise to something that depends on neither. So if you happen to Updating `/tmp/pkgs/environments/v1.4/Project.toml`
[da1fd8a2] + CodeTracking v1.0.0
[6f1432cf] + LoweredCodeUtils v1.1.2
[295af30f] + Revise v1.0.3Revise 1.0.3 is really old. Now, if you only That why git branches feel like the right way to develop new features; if Revise, JuliaInterpreter, Debugger, CodeTracking, and LoweredCodeUtils were all part of one giant git repo, I could just say "check out this one branch" and everything is done. Then when it's time to release I can release them all, but until then I don't need to trouble the rest of the world with compatibility issues. So I'm really excited about this new feature Pkg devs have developed! I am just pointing out that I think we're still missing some tooling to make this work properly. |
Yes, one of them, particularly our JuliaImages ecosystem. Images is basically a meta-package. There's also the issue of our documentation, which is a separate repo and deploys docs for the whole ecosystem. We've wrestled a lot with questions like "should focused packages host their own docs, or should the whole ecosystem be documented as a whole?" Again, one of the really huge things about this new development is we won't have to make that choice any more: if all the focused packages are in the same github repo as the meta package, then there's really only one documentation site. |
|
I still have not understood how this discussion relate to the possibility of having multiple packages in the same repo. It seems like all the same constraints apply to having the packages in multiple repos. |
|
There are two separate points:
Based partly on the confusion of folks who understand Pkg better than I do, I've reconsidered how I'm thinking now, and I think the answer to the first one is "no." But, it's not how I thought about it when starting out to split SnoopCompile. The "right" way, as I think you're espousing, to do it is to (1) duplicate the code you need to create SubPkgA (important: do not delete the copy that lives in the old monolithic BigPkg), get it through CI, and register it; (2) after 3 days, duplicate the code you need to create SubPkgB, get it through CI, and register it; ... (n) after 3 days, delete the duplicated code from BigPkg, make it depend on all the subpackages, and register the new version. This works. But for me that was counterintuitive because my first thought was to aim for the final state I wanted, not craft the strategy in terms of how to get it past CI & Pkg registration. And of course it's very slow (but this is no different from doing it in separate repos). The answer to the second question is "yes, we're definitely failing to take advantage of a huge opportunity to do better." To explain this point clearly, I've submitted #1874. |
|
It sounds like people have different views as to why sub-packages are good to have. The following describes my own use case: Why do I want all sub-packages in a git repo to have a single "repo-wide" version number?Because every "project" in that repo should be somewhat related, and would often build on its neighboring packages. So, it appears I have much in common with @timholy in this respect. Side-benefit of registering all subpackages simultaneouslyIf everything got registered simultaneously in one PR to the GeneralRegistry, could I not trigger testing of all packages from the root package, by "adding" sub-packages to the "test" "[target]" on that root Project.toml file? That should install their dependencies correctly, and make the CI test environment ready for the sub-packages as well. So why not just make a sigle package?(Question from @fredrikekre #1251 (comment)) Well, since the package system pulls in (downloads, builds, etc) ALL dependencies of my monolithic package, any user of my monolithic package would be forced to suffer long download, build, and compile times. I'm also pretty sure the memory footprint and time-to-first-execution of the resulting application will blow up, as well. So I really just need these sub-packages to limit the dependency graph without resorting to having to manage a huge amount of independent Git repos, commits, and PRs to GeneralRegistry. I intend on achiving this goal by sepearating out code that "requires" external packages to said sub-packages. MY QUESTION: Could this somehow be done from the "targets" section of Project.toml????? I mean, the "test" target dependencies only just kick in when you run test, right? Maybe we could do something similar so that |
|
Registering multiple packages in a repo with a single command just requires some additions to the registration infrastructure. I think a globbing syntax like Bumping all project versions could be done with some script. There still doesn't seems to be a need to tweak anything in Pkg for this. |
|
@KristofferC: You might be right. I guess my complaint a the moment stem from the fact that I seem to be unable to develop my new multi-package module locally on my computer. That's why I described my use case above. I keep running into issues where the package manager appears to need my module to be registered in
(I am using Julia 1.5-rc1) |
If there is a bug in the subdir functionality we want to find and fix it before 1.5 is released. What are the exact steps to reproduce it? |
I don't get that. Just dev all the packages. Just like always. Could you make a MWE that shows the problem? |
|
Yes, I saw that dev all package one-liner that @timholy posted on the other issue. I haven't tried that yet. Note that it does seem counter-intuitive to have to dev all packages when you technically shouldn't be aware that your repo hosts multiple packages in the currently supported workflow. I'll try creating a MWE for you, though. I got the vague impression it was already known. Sorry. |
|
Ok. I built a sample multi-package repo on Github, to illustrate my MWE. You can add Now, if I want to modify one of the sub-packages, I would logically do the following: But in Julia 1.5.0-rc1.0, I get the following error: So it "dev"s the entire repo under the name |
|
Thanks, I can reproduce this. So the scenario here is developing an unregistered subdir package by name, which is known to Pkg after having been added by URL. Clearly the code is not doing the right thing here and I've opened #1925 to track it. Meanwhile, the easiest workaround is probably to develop the subdir URL shown by |
|
Thanks @GunnarFarneback. Yes that appears to work! |
Currently, a single Git repository can only contain one registered package, which must live in the top folder of the Git repository.
I would like to request the ability to store more than one registered package in the same Git repository.
For example, I might have two packages
FooandBarthat I store in a single GitHub repositoryDilumAluthge/FooBar.git. In this case, the contents ofhttps://github.com/DilumAluthge/FooBar.gitmight look something like this:For example, the contents of
https://github.com/DilumAluthge/FooBar.gitmight instead look something like this:Possibly related issues/pull requests:
Allow packages to live in a directory of repo #691: "Allow packages to live in a directory of repo"
Proposal for "sub-projects". #1233: Proposal for "sub-projects"
The text was updated successfully, but these errors were encountered: