Determine what should be done about --(install|global)-option with Wheels

[dstufft]

Currently pip has --install-option and --global-option to make it possible to pass flags to the setup.py invocation. However these currently get ignored completely when installing from Wheel. This previously wasn't a big deal because commonly these options are only used for compiled packages and those often don't have Wheels on the platforms you might need these options on. However now in pip 7.0 via #2618 we now attempt to automatically build wheels for all projects. This means that, assuming we can build a wheel, these options are just completely ignored.

So what do we do with these options? Do we make their use imply --no-use-wheel? Do we attempt to convert them? Get rid of them?

[qwcode]

I had previously opened #1716 for this. I'll close that.

[qwcode]

the only extra tidbit from #1716 is just the awareness that install-options are recognized when placed in distutils config files.

[pfmoore]

The main use I've made of --install-option (actually --global-option) is the obscene hack:

pip wheel --global-option "--with-libyaml" --global-option "build_ext" --global-option "-DYAML_DECLARE_STATIC" --global-option "-LC:\Work\Scratch\pyyaml\yaml-0.1.5\win32\vs2008\Output\Release\lib" --global-option "-IC:\Work\Scratch\pyyaml\yaml-0.1.5\include"  .\PyYAML-3.10.tar.gz

This lets me add build options to the extension build (by injecting an explicit build_ext step into the setup command). The reason for doing this is to avoid having to download and edit the source distribution just to add some settings into setup.cfg.

Maybe having a means to add section.key=value items into the source archive's setup.cfg when doing the build would be a more general solution, that is explicitly a "build step" option.

Something like --build-setting [build_ext]define=YAML_DECLARE_STATIC, maybe? Then there's no implication that it has an effect when installing from wheel, it's clearly a build-only option.

(Whether this would be viable depends on what uses other people have for --install-option and --global-option...)

[xavfernandez]

According to the recently merged #2537 , people seem to have a need for these options.

An implied --no-use-wheel when --install-option or --global-option are used seems easy enough.

[rbtcollins]

So the problem (and its not unique to the wheel autobuilding) is scoping those options.

We can and should use the options from requirements file binding to pass through to the wheels. But the top level --install-option is only defined as applying to the thing being installed, isn't it? Or is it global?

Anyhow, if someone can define 'right' here, I'll happily write the code for it.

Remember too that folk can always use 'pip wheel' and then -f wheelhouse to get exactly what they want.

[dstufft]

Scoping is certainly a problem, and I've thought before that maybe we should remove those options and instead we should do something like --build-option:myproject="--bar" or something that handles the scoping.

There's also the question of what --install-option and --global-option even means in the future (tm) where we're trying to move to a situation where we only ever install from wheels and the only thing we use a source distribution for is building a wheel. In that scenario there is nothing to pass a --install-option too (and ideally anything people are using it for we'd either explicitly say we don't support, or we'd get a better, generic option for) and --global-option doesn't really make much sense because if you're only invoking a source distribution to build a wheel there is no global, only build.

On the other hand, we might not want to define the API for --build-option util the metabuild system is done so we can implement it in terms of that (e.g. if the API for metabuild is CLI based, we'd want to pass --foo options, if it's Pythonic "call this function" based, we'd want some way to translate CLI opts into Python opts).

In the short term it might just make the most sense that using --install-option and/or --global-option disables the automatic wheel cache and then open up a new issue to deprecate and replace those with a --build-option.

[rbtcollins]

@dstufft and I discussed this on IRC, he says that the resolution is: "--install-option, --global-option, --build-option will all disable the consumption of wheels, and any autobuilding of wheels." And that follow on work can be done to assess what we really want/need longer term.

[qwcode]

the question of what --install-option and --global-option even means in the future (tm)
where we're trying to move to a situation where we only ever install from wheels

why doesn't --install-option make sense when installing from wheels?
regardless of the format, you might want to set different paths for the install.

feels like a loss for wheel installs to have less flexibility than sdists.
(see https://github.com/twang817/pip/commit/e3338f7b087a9aba7e9033aa9078971fc33fc547 as an example of a pip user hacking this themselves, and wanting us to add this support)

how would this get sorted out in the "longer term"?

[dstufft]

Because --install-option, --global-option, and --build-option is conceptually "pass this option to the setup.py execution", it's documented that way and I'm sure people are using it that way. When we're installing from a Wheel we have no setup.py to execute and thus nothing to pass options to. Further more as we move to the meta build system we (most likely) aren't even going to be guaranteed to have a setup.py that exists for even a source distribution.

Longer term I think that the way to solve this is something like that:

• Figure out what people are using --install-option for that makes sense at all for wheels (one example I've seen is to be able to direct scripts to a different location) and expose a top level pip flag that supports that same use case. When installing from a source distribution we'll pass those options to setup.py and when installing from a wheel we'll take them into consideration for computing paths.
• Figure out what people are using --global-option for, and if it still makes sense in the new world order. I don't know off the top of my head what people are using this for, so I can't speak to what this would look like.
• Figure out the meta build system and what API is available for pip to call to pass options into the build system, once that is nailed down figure out a way that we can expose those build options to both pip wheel and pip install.

[dstufft]

In short, the basic problem with those options is that they are explicitly tied to executing setup.py and one of their use cases is for people being able to pass random arbitrary options to their setup.py invocations.

[qwcode]

these all seem like valid uses of install-options that could be applied to wheels.

 --home                               (Unix only) home directory to install
                                       under
  --install-base                       base installation directory (instead of
                                       --prefix or --home)
  --install-platbase                   base installation directory for
                                       platform-specific files (instead of --
                                       exec-prefix or --home)
  --root                               install everything relative to this
                                       alternate root directory
  --install-purelib                    installation directory for pure Python
                                       module distributions
  --install-platlib                    installation directory for non-pure
                                       module distributions
  --install-lib                        installation directory for all module
                                       distributions (overrides --install-
                                       purelib and --install-platlib)
  --install-headers                    installation directory for C/C++
                                       headers
  --install-scripts                    installation directory for Python
                                       scripts
  --install-data    

yes, these options are conceptually tied to setuptools, but don't have to be literally. we'd map them over to our distutils scheme utility.

for sure, it's weird to use setuptools options when there's no "setup.py" involved, but the reality is things are fractured right now, and maybe it's ok to break a conceptual line, to give people support for install flexibility in the mean time.

to be clear, I'm ok with the approach of ignoring them for wheels for now, but just making it clear, that we could do otherwise, if a lot of people wanted this.