Renv installation fails with dependency only available in Archive

[bentilley]

Hello! Thanks for the great package 🎉

Description of Issue

I'm looking for some guidance on how renv handles dependencies stored in the archive of a repo. Specifically we are using Artifactory, and have a package ABC that isn't available at /src/contrib on the repo, it's only available at /src/contrib/Archive. This file doesn't appear in the PACKAGES index for the repo.

From checking through the renv source, it seems that renv uses this PACKAGES index exclusively to determine what is available on the repo. This means that renv::install("ABC", repos="artifactory.example.com/repo") fails with Error: package 'ABC' is not available.

If I do renv::install("ABC@0.1.2", repos="artifactory.example.com/repo") (i.e. requesting the specific version that is in the archive. Then this succeeds. From checking the source, this seems to be because specific version installations elicit an additional renv heuristic check in the /Archive location.

So that's good, that we can explicitly fetch the package. However, what I don't know how to handle is when I try to install a package MainPackage that depends on the archived package ABC. In this situation, when I run renv::install("MainPackage", repos="artifactory.example.com/repo"), renv goes to fetch the package dependencies and in the end errors with Error: package 'ABC' is not available.

It looks to me that during the dependency fetching, renv is only consulting the PACKAGES index, and therefore doesn't know that the ABC package is available in the archive.

Questions

1. Is this a bug with our repo? i.e. Should the archived package appear in the PACKAGES index somehow? (I should note that the same package is available on public CRAN, and on CRAN it also doesn't appear in the PACKAGES index, which makes me think that this is not the case. However, I'm relatively new to the R eco system, so def wouldn't put it past myself simply having completely the wrong end of the stick about how these things should be working!)
2. Should renv be looking further afield to determine the available packages on a repo? I've noticed while exploring this issue that cranlike repos provide an archive.rds file available at /src/contrib/Meta/archive.rds, should renv also consult this and add the data to its list of index DB's when determining what packages are available?
3. If the answer to 2 is "no", then should renv also be performing its own archive heuristic check (i.e. looking in /Archive) for packages that don't appear in it's index DB's?

Thank you for any insight you can shed on this 🙏 . renv is great!

[kevinushey]

Is this a bug with our repo? i.e. Should the archived package appear in the PACKAGES index somehow? (I should note that the same package is available on public CRAN, and on CRAN it also doesn't appear in the PACKAGES index, which makes me think that this is not the case. However, I'm relatively new to the R eco system, so def wouldn't put it past myself simply having completely the wrong end of the stick about how these things should be working!)

I don't think so; this is how CRAN repositories behave as well. Indeed, as you say, once a package has been archived, it's no longer declared or listed in the PACKAGES index, even though older versions are actually available in the Archive.

Should renv be looking further afield to determine the available packages on a repo? I've noticed while exploring this issue that cranlike repos provide an archive.rds file available at /src/contrib/Meta/archive.rds, should renv also consult this and add the data to its list of index DB's when determining what packages are available?

That's an excellent idea; I hadn't realized that this file is available! I could extend renv to make use of this file when attempting to resolve packages.

If the answer to 2 is "no", then should renv also be performing its own archive heuristic check (i.e. looking in /Archive) for packages that don't appear in it's index DB's?

I think (2) is the right approach, so I'll try to explore that when I'm next working on renv.

[kevinushey]

As far as escape hatches go, I believe you should be able to manually install the archived package, and then install the main package, e.g.

renv::install("ABC@1.2")
renv::install("MainPackage")

Does that work?

[bentilley]

Hey @kevinushey , thank you very much for the detailed reply 🙏 I had a couple of fires to fight, so just managed to come back to this.

renv::install("ABC@1.2")
renv::install("MainPackage")

Does that work?

Yes, this works 👍 And was the escape hatch I used to setup our original renv.lock file. What still breaks though is doing a renv::restore() from a clean library. As in that case when renv checks the dependencies of MainPackage it throws the same error as according to the dbs, the ABC dependency can't be found.

That's an excellent idea; I hadn't realized that this file is available! I could extend renv to make use of this file when attempting to resolve packages.

I also did know about this file until I went down the rabbit hole on this one 😄 ! I might also have some time to work on this at work in the next few weeks, so I'll update here if I make any progress / have any questions. Thank you for confirming that you also think that making use of this information is the correct way to address this.

[thefabnab]

@kevinushey I saw that you've added this to the v1.1.0 milestones for renv and that you have an experimental branch working on this feature. Any insight on the status of the feature?

[kevinushey]

No more insight yet; I just need to confirm whether or not this change would really be safe to add to renv -- I want to make sure it's not accidentally going to cause unexpected trouble for users down the line.

[chicotobi]

Hey @kevinushey

would be interested in the status here. I am using renv in restoring environments in automated pipelines, so manual interaction (like renv::install) is not really an option.

Our internal RSPM also did archive some packages (I am not so familiar with the reasons why and how and for which versions this happens?) and now renv::restore fails within the pipeline for lock-files that contain these packages in "older" (in this case Nov'23, so 15 months) versions.

Any ideas? Can we override this behavior with something like renv::restore(with_archived = TRUE) or a sys env variable?

Best wishes and thanks for the awesome package!

[kevinushey]

I've just merged #1771, which tries to make use of the repository's src/contrib/Meta/archive.rds file when looking for archived packages. Are you able to install the development version of renv and take a look? (I'm not sure if this file is provided or maintained by RSPM, but it should at least be available on public CRAN instances.)

If not, can you please file a new issue with more details? It would help to know which packages in particular are affected, so I can test.

[chicotobi]

I installed and tried - it seems renv_available_packages_latest is not called from renv::restore(). It's difficult to see at which point we expect that this function is called. There are a lot of nested calls to debug. Should I look in renv_retrieve_impl_one? But both compat and uselatest evaluate to FALSE, so renv_available_packages_latest is not called there.

But calling of the functions at least shows:

> renv:::renv_available_packages_latest("packagefoo")
$Package
[1] "packagefoo"

$Version
[1] "2.3.4"

$OS_type
[1] NA

$NeedsCompilation
[1] "no"

$Repository
[1] "REPOFOO_STAGING"

$Source
[1] "Repository"

attr(,"url")
[1] "http://urlfoobar.com:4242/staging/latest/src/contrib"
attr(,"type")
[1] "source"
attr(,"name")
[1] "REPOFOO"
> renv:::renv_available_packages_latest_archive("packagefoo")
$Package
[1] "packagefoo"

$Version
[1] "2.3.3"

$Source
[1] "Repository"

$Repository
[1] "REPOFOO_STAGING"

But in my renv I have 2.3.0 - which is of course older than the latest in the archive, but also still available under "archived" in the Package Manager.

Sorry I can't provide an example for a package - it only happened just for our internal packages.