Do not use the upstream Pipenv

[fridex]

The recent issues on upstream Pipenv caused s2i builds to always fail:

New python executable in /opt/app-root/src/.local/venvs/pipenv/bin/python3
--
  | Also creating executable in /opt/app-root/src/.local/venvs/pipenv/bin/python
  | Installing setuptools, pip, wheel...done.
  | Collecting pipenv
  | Downloading https://files.pythonhosted.org/packages/eb/64/9b2747d54f2008ac3dfe86c0b1c8ec126042726fd8a540d5208d26732701/pipenv-2018.7.1-py3-none-any.whl  (5.0MB)
  | Collecting virtualenv (from pipenv)
  | Downloading https://files.pythonhosted.org/packages/b6/30/96a02b2287098b23b875bc8c2f58071c35d2efe84f747b64d523721dc2b5/virtualenv-16.0.0-py2.py3-none-any.whl  (1.9MB)
  | Collecting certifi (from pipenv)
  | Downloading https://files.pythonhosted.org/packages/df/f7/04fee6ac349e915b82171f8e23cee63644d83663b34c539f7a09aed18f9e/certifi-2018.8.24-py2.py3-none-any.whl  (147kB)
  | Collecting pip>=9.0.1 (from pipenv)
  | Downloading https://files.pythonhosted.org/packages/c2/d7/90f34cb0d83a6c5631cf71dfe64cc1054598c843a92b400e55675cc2ac37/pip-18.1-py2.py3-none-any.whl  (1.3MB)
  | Collecting virtualenv-clone>=0.2.5 (from pipenv)
  | Downloading https://files.pythonhosted.org/packages/6d/c2/dccb5ccf599e0c5d1eea6acbd058af7a71384f9740179db67a9182a24798/virtualenv_clone-0.3.0-py2.py3-none-any.whl
  | Collecting setuptools>=36.2.1 (from pipenv)
  | Downloading https://files.pythonhosted.org/packages/96/06/c8ee69628191285ddddffb277bd5abdf769166e7a14b867c2a172f0175b1/setuptools-40.4.3-py2.py3-none-any.whl  (569kB)
  | Installing collected packages: virtualenv, certifi, pip, virtualenv-clone, setuptools, pipenv
  | Found existing installation: pip 9.0.1
  | Uninstalling pip-9.0.1:
  | Successfully uninstalled pip-9.0.1
  | Found existing installation: setuptools 28.8.0
  | Uninstalling setuptools-28.8.0:
  | Successfully uninstalled setuptools-28.8.0
  | Successfully installed certifi-2018.8.24 pip-18.1 pipenv-2018.7.1 setuptools-40.4.3 virtualenv-16.0.0 virtualenv-clone-0.3.0
  | ---> Installing dependencies via pipenv ...
  | Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead.
  | Installing dependencies from Pipfile.lock (c00e44)...
  | Traceback (most recent call last):
  | File "/opt/app-root/src/.local/bin/pipenv", line 11, in <module>
  | sys.exit(cli())
  | File "/opt/app-root/src/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
  | return self.main(*args, **kwargs)
  | File "/opt/app-root/src/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 697, in main
  | rv = self.invoke(ctx)
  | File "/opt/app-root/src/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
  | return _process_result(sub_ctx.command.invoke(sub_ctx))
  | File "/opt/app-root/src/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
  | return ctx.invoke(self.callback, **ctx.params)
  | File "/opt/app-root/src/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
  | return callback(*args, **kwargs)
  | File "/opt/app-root/src/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/cli.py", line 435, in install
  | selective_upgrade=selective_upgrade,
  | File "/opt/app-root/src/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/core.py", line 1943, in do_install
  | pypi_mirror=pypi_mirror,
  | File "/opt/app-root/src/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/core.py", line 1322, in do_init
  | pypi_mirror=pypi_mirror,
  | File "/opt/app-root/src/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/core.py", line 807, in do_install_dependencies
  | pypi_mirror=pypi_mirror,
  | File "/opt/app-root/src/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/core.py", line 1375, in pip_install
  | package_name.split('--hash')[0].split('--trusted-host')[0]
  | File "/opt/app-root/src/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 704, in from_line
  | line, extras = _strip_extras(line)
  | TypeError: 'module' object is not callable

The upstream issue can be found at pypa/pipenv#2924 A brief summary - Pipenv relies on some logic present in pip and with a new pip release where the shared logic changed, Pipenv broke. Even though the patch to address the issue was already merged into Pipenv master today, we are still unable to build any s2i based project that is using Pipenv as a bugfix release was not not issued by upstream since Friday when this issue occurred.

As Pipenv is already packaged as an RPM it could be worth to consider adding Pipenv to the base image from Fedora repositories so s2i does not rely on the latest Pipenv that can introduce such issues.

[omron93]

I hit this issue too.

@mcyprian @torsava ?

[pkubatrh]

@torsava How about not upgrading dependencies to latest versions when installing pipenv?

[hakkeroid]

As the pipenv situation didn't improve after the latest release I fixed this issue for now by overriding the assemble file. If someone wants to follow along simply copy the respective assemble file of the python version you use into your project and change the function install_pipenv like so:

function install_pipenv() {
  if [ ! -z ${PIN_PIPENV_VERSION} ]; then
    pipenv_req="pipenv==${PIN_PIPENV_VERSION}"
  else
    pipenv_req="pipenv"
  fi
  if [ ! -z ${PIN_PIP_VERSION} ]; then
    pip_req="pip==${PIN_PIP_VERSION}"
  else
    pip_req="pip"
  fi
  echo "---> Installing $pipenv_req packaging tool with $pip_req ..."
  VENV_DIR=$HOME/.local/venvs/pipenv
  virtualenv_bin "$VENV_DIR"
  $VENV_DIR/bin/pip --isolated install -U $pipenv_req $pip_req
  mkdir -p $HOME/.local/bin
  ln -s $VENV_DIR/bin/pipenv $HOME/.local/bin/pipenv
}

Now provide a specific pipenv + pip version via PIN_PIP_VERSION and PIN_PIPENV_VERSION environment variables to the build container.

Note: I used the prefix PIN_ to prevent issues with naming clashes. Namely PIP_VERSION.

If you have a better idea, let me know. :)

[torsava]

Apologies, I'm currently swamped with a different project. How urgent is this?

[GrahamDumpleton]

FWIW, you can't use RPM version of pipenv. Has to be a version which is installed in the virtual environment created for the image. That is, it doesn't use system or SCL versions of Python packages.

[frenzymadness]

IFAIK this is no longer a problem but I agree that it might be handy for future cases to be able to pin a specific pip/pipenv version.

Could you please open a PR or should somebody else take over it?

[hakkeroid]

@frenzymadness My proposal isn't tested (it was good enough for my situation though) and it has to be adapted to all assemble files I guess. However, feel free to take over. Although I worry that it will take a while to get merged, as @torsava already mentioned beeing occupied elsewhere.

[torsava]

A PR for pinning the version would be great. If anyone can create it, we'll get it merged.

[frenzymadness]

I would like to implement it but I need help. I completely understand the problem and how we can solve it but, there are two copies of pip installed in the image:

• one is in the main virtual environment and is used by application itself for installation of its dependencies
• and the second one is in the special virtual environment created just for pipenv itself

Implementation of PIP_PIPENV_VERSION is quite simple. But what PIN_PIP_VERSION should mean? Should it has an impact on the version of pip installed in venv with pipenv only or should both pip copies should be updated to that version? How to communicate this with users?

[hakkeroid]

@frenzymadness (from my memories) Usually it is only required to pin the system's pip version. That is because when pipenv is installed it is (or was) conflicting with the existing pip due to some upstream changes. What pipenv is then installing inside the virtualenv shouldn't matter anymore. However, I cannot recall why I pinned pip inside pipenv as well. Maybe it was due to some strange behavior.

one is in the main virtual environment and is used by application itself for installation of its dependencies

I think this depends on the variable ENABLE_PIPENV. So if it is enabled then the application is installed inside the virtualenv created by pipenv. Otherwise the system's pip or python is used.

How to communicate this with users?

Good question. To keep it simple I suggest to only be able to pin the system's pip or pipenv. At the end I assume most users would pin pip only, because

1. their application requires that, or
2. the upstream conflict with pipenv happens.

In case of (1) if users don't use pipenv, then pinning the system's pip makes sense. If they do use pipenv though, they probably should pin pip inside of the Pipfile instead. So the system's pip is unimportant and only used to install pipenv itself. However, this is done automatically anyway.
Only in case (2), when the automatic install fails or the resulting virtualenv is broken, then they need to fiddle around with the system's pip and for that need to understand the relationship between the system's pip, pipenv and the virtualenv's pip.

I hope that helps! :)