Adding an editable install function for the build-system API

[adamboche]

What's the problem this feature will solve?

Currently, pip does not support editable installs for build systems using pyproject.toml
instead of setup.py.

For example, in a project that uses Poetry, I get an error when I try to install editable.

venv/bin/pip install -e /path/to/project
ERROR: File "setup.py" not found. Directory cannot be installed in editable mode: /path/to/project
(A "pyproject.toml" file was found, but editable mode currently requires a setup.py based build.)

The pip install -e <path> command is useful in cases where I want to make frequent updates to a package while using it from another environment.

A proposal

PEP 517 defines a build-system API that provides several hooks, such as build_wheel() and
build_sdist(), and some additional optional hooks. I propose that specification be
expanded to include an optional function:

def editable_install(config_settings: Dict[str, Union[str, List[str]]]) -> None: ...
    """Build an egg-link file in the current site-packages directory."""

The function should create an egg-link file as described in the setuptools docs.

An alternative specification could instead return path to the package source's parent directory, and pip could build the egg-link itself.

def editable_install(
    config_settings: Dict[str, Union[str, List[str]]]
) -> Dict[str, str]:
    """Return a mapping from the package name to the parent directory of its source."""    

A third would be to leave it up to the build system how it wants to implement editable installs, but I'm not sure what that would mean in practice.

Alternative Solutions

1. Workaround by build systems.

Poetry could create a setup.py for use by pip, which users could check into version
control. This means the build backend is once again coupled to setuptools, which is
contrary to the intent of PEP 517.

2. Workaround by users.

Another workaround is to activate a virtualenv, cd to the Poetry project's directory, and run
poetry install there. With a virtualenv activated, Poetry will detect the environment by
checking $VIRTUALENV_ENV and install into that environment rather than creating a new
virtualenv like it normally would. If a virtualenv's pip is used without activating, it might be detectable using sys.real_prefix, but then the user's intent is less clear about whether Poetry should create a new virtualenv.

This workaround is non-obvious and would require explanation for each user. It would be
easier if we could just use pip install -e /path/to/project like normal.

3. Workaround by pip.

An alternative solution is to just use the regular installation code path but pass -e in
the config_settings dict. However, if the build system does not support that option, it
would probably fail silently by installing in non-editable mode. It is better to be
informed that the editable build failed, which means there should be an explicit API for
pip to check. If the builder does not provide the editable_install() function, pip can
raise an exception itself.

Additional context

• PEP 517
• Pip discussions "conflict with the backend dependencies" error running pip install in editable mode with pyproject.toml / PEP-517 enabled project #6314 (comment), Deprecate legacy setup.py bdist_wheel mechanism for building projects #6334
• Poetry discussion PEP-517 and pip install --editable with pip 19 python-poetry/poetry#1279
• Current handling of editable installs Fix editable-mode logic when pyproject.toml present #6370

---

/cc @sdispater (Poetry) @takluyver (Flit) in case you have thoughts on this.

[chrahunt]

Given that we can't make a unilateral decision, and this would need input and discussion from a larger audience, it may be more effective to start or find a discussion on discuss.python.org to get feedback. There are several there already, but I couldn't find an authoritative-looking thread where people are following up on the PEP517 note regarding editable installs.

[adamboche]

For future readers: The most relevant discussion I can find is https://discuss.python.org/t/specification-of-editable-installation/1564.

[pradyunsg]

See pypa/packaging-problems#256 (comment)

[pradyunsg]

To reiterate what @chrahunt said, this is a multi project, interoperability specification. Thus, this needs much broader discussion and Packaging category on discuss.python.org is likely the best place for moving this forward.

There are a lot of the details here to figure out so, know that it'll take a fair amount of effort and time to get this done. That's just the nature of a packaging ecosystem that's almost completely maintained by folks on a volunteer basis.

[chrahunt]

Since there's no action for pip here I will close this issue. Anyone interested should participate in the discussions linked above.