migration from poetry to uv

[king-11]

make use of standard keys for project and dependeny specification

• provide sources to run uv build so that it can refer local packages
• using hatchling for build as is stock build option
• use optional-dependencies.dev for dev-dependencies
• add hatch targets for packages and includes where unclear
• update Makefiles to use uv when running commands that access python dependencies

Closes #7665

Important

25.05 FREEZE MAY 12TH: Non-bugfix PRs not ready by this date will wait for 25.08.

RC1 is scheduled on May 23rd, RC2 on May 26th, ...

The final release is on MAY 29TH.

Checklist

Before submitting the PR, ensure the following tasks are completed. If an item is not applicable to your PR, please mark it as checked:

• The changelog has been updated in the relevant commit(s) according to the guidelines.
• Tests have been added or modified to reflect the changes.
• Documentation has been reviewed and updated as needed.
• Related issues have been listed and linked, including any that this PR closes.

[king-11]

pending changes:

• ci.yaml
• macos.yaml
• setup.sh

[vincenzopalazzo]

What are the benefit of uv over poetry?

[cdecker]

It's much faster (5-10 seconds compared to minutes using poetry). While that doesn't sound too different, it means that we don't create long lived venvs that may drift from the metadata, causing fewer "it worked on my machine" kind of errors. It can also be used as a shebang, automatically wrapping for example a plugin, downloading dependencies on first invocation, creating the venv and only then passing control to python. This means we can make plugins self-contained and single file if we wanted to.

The speed, which unlocks all of these use cases, is done via caching, and interesting speedup ideas, that are quite fun to read up on. And finally, you don't need a system-wide python installation, since uv can also download and manage python versions (prevents you from accidentally polluting the system python and potentially break your OS).

[cdecker]

Great initiative @king-11 , I had a branch with some initial steps, but you are much further along, so I'd rather help you than double-tracking

[king-11]

Thanks @cdecker i will get the last few things in, just raised a draft PR given I saw some interest around it and wanted to have a pre-ready change look available.

[king-11]

@cdecker do you have any suggestions I am currently encountering an issue where running tests shows pyln path not found.

After these changes I am setting things up as

uv venv --python 3.12
source .venv/bin/activate
uv sync --all-extras
uv run pytest tests/test_renepay.py

I ran the debug command

uv pip list

it shows pyln-testing is installed the only hunch I have here is maybe I need to set that as editable 🤔

[cdecker]

Yeah, workspace-related options take a bit of getting used to.

For packages that we'd like uv to pull in from the repo, rather than the PyPI repository, we need to tell uv about the dependency:

[project]
...
dependencies = [
  "pyln-testing",
  ...
]

Then we need to tell uv that there is a member gl-testing in the workspace:

[tool.uv.workspace]
members = [
  "pyln-testing",
  ...
]

And then finally we need to tell uv to prefer the local copy from the repo, rather than the PyPI repo:

[tool.uv.sources]
pyln-testing = { workspace = true }

At least these are the three steps that would work for me whenever I ran into this issue.

[king-11]

Got it thanks

[king-11]

Thanks for the workspace tip @cdecker that helped but one more thing was needed that is __init__.py files with PATH override for namespace sharing of pyln took some help from our ai friends going to get working on ci now I think

[king-11]

The flake8 fails for auto generated files not sure why but I have ran locally and tests are working fine now. Can you use some guidance on how to resolve the flake9 issue

[king-11]

@cdecker can you review now I have some changes in local for reckless they are around the case that we shouldn't create an additional venv while running reckless from one that should fix the remaining test failures. cc: @endothermicdev for reckless change recommendation

I have updated the readmes as well.

[cdecker]

Please remove commit cf8588c. The whole idea of reckless managing venvs is to guarantee isolation. By reusing a random venv that is inherited from outside, we may end up polluting a user's venv, and multiple plugins with different dependencies will clobber each other, causing hard-to-debug failures.

The overhead, and additional disk space is intentional, because it buys us isolation, that we cannot get otherwise./

[king-11]

By reusing a random venv that is inherited from outside, we may end up polluting a user's venv, and multiple plugins with different dependencies will clobber each other, causing hard-to-debug failures.

Got it now am more clear on the intent the issue is if there is an existing venv we don't know with what name the folder might be it can be .venv or something else so will have to add some sort of randomisation while creating venv with pip. Poetry should be fine as they use random location.

[king-11]

@cdecker I think this is good now there was this one sed change I made in PR for mac compatibility that was messing the tests everywhere but removed the reckless change wasn't required it doesn't conflict with uv venv as the directory is different so we are good I think

[rustyrussell]

Ack 873cd0e

While ambitious, now is the time to put this in. I will be delighted to leave poetry behind us, and I like some of the other fixes in here too.

[cdecker]

@king-11 this is an amazing PR, thank you so much for purging poetry from the project (it should likely stay in reckless since we'd like to allow downstream devs to chose their own tooling). Also great work on hunting down those path issues and the cryptography/LICENSE issue 👏

Just some minor clarifications, and minor nits that can be addressed in a separate PR. This is gtg for me 🚀

[cdecker]

Can you explain what this __init__.py file does? It seems like it changes its own import path somehow? And if the problem is pyln-grpc-proto why aren't we just fixing that?

[king-11]

It allows for packages to create pyln.proto kind of packages by extending the parent pyln. Otherwise only one of them i.e. pyln-grpc-proto overwrites the pyln path

[cdecker]

I'm pretty sure that only the pytest.ini in the repo root is being loaded, and this should have no effect.

[king-11]

🤔 there is no pytest.ini in repo root though

[cdecker]

Do we even still use the system python? I think uv will pull down a standalone python binary, completely ignoring the system one.

[king-11]

I think uv will only do it if we explicitly ask it to create python venv with a different python version that what is available in system