WSL : Python executable coming from outside... and failing.

[asmodehn]

Issue description

Attempting to use pipenv to create a virtualenv on Windows(10) Subsystem Linux (Ubuntu distro), pipenv fails during the virtualenv creation. The issues seems to be that pipenv decides to use a python executable from windows, instead of the one from the WSL distro.

The flow I used to install pipenv on WSL was :

• install python3.7 via apt,
• install python3-pip via apt,
• install latest pip in user env via pip3 install --user --upgrade pip,
• install pipenv in user env via pip install --user pipenv. Note user shell must be restarted to get the correct pip.
  Note I also have Python 3.7.2 installed on Windows.

Expected result

Pipenv decides to use python 3.7.1 coming from WSL environment, and environment is created with success :

alexv@AlexV-Win10-Oryx:/mnt/c/Users/AlexV/Documents/GitHub/functional-python/01$ pipenv install
Creating a virtualenv for this project…
Pipfile: /mnt/c/Users/AlexV/Documents/GitHub/functional-python/01/Pipfile
Using /usr/bin/python3.7 (3.7.1) to create virtualenv…
⠋ Creating virtual environment...Using base prefix '/usr'
New python executable in /mnt/c/Users/AlexV/Documents/GitHub/functional-python/01/.venv/bin/python3.7
Also creating executable in /mnt/c/Users/AlexV/Documents/GitHub/functional-python/01/.venv/bin/python
Installing setuptools, pip, wheel...
done.
Running virtualenv with interpreter /usr/bin/python3.7

✔ Successfully created virtual environment!

Actual result

alexv@AlexV-Win10-Oryx:/mnt/c/Users/AlexV/Documents/GitHub/functional-python/01$ pipenv install -v
Creating a virtualenv for this project…
Pipfile: /mnt/c/Users/AlexV/Documents/GitHub/functional-python/01/Pipfile
Using /mnt/c/Program Files/Python37/python.exe (3.7.2) to create virtualenv…
⠹ Creating virtual environment...Running virtualenv with interpreter /mnt/c/Program Files/Python37/python.exe

✘ Failed creating virtual environment
Traceback (most recent call last):
  File "/home/alexv/.local/bin/pipenv", line 10, in <module>
    sys.exit(cli())
  File "/home/alexv/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/alexv/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/alexv/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/alexv/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/alexv/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/alexv/.local/lib/python3.6/site-packages/pipenv/vendor/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/home/alexv/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/alexv/.local/lib/python3.6/site-packages/pipenv/vendor/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/alexv/.local/lib/python3.6/site-packages/pipenv/cli/command.py", line 254, in install
    editable_packages=state.installstate.editables,
  File "/home/alexv/.local/lib/python3.6/site-packages/pipenv/core.py", line 1741, in do_install
    pypi_mirror=pypi_mirror,
  File "/home/alexv/.local/lib/python3.6/site-packages/pipenv/core.py", line 574, in ensure_project
    pypi_mirror=pypi_mirror,
  File "/home/alexv/.local/lib/python3.6/site-packages/pipenv/core.py", line 506, in ensure_virtualenv
    python=python, site_packages=site_packages, pypi_mirror=pypi_mirror
  File "/home/alexv/.local/lib/python3.6/site-packages/pipenv/core.py", line 935, in do_create_virtualenv
    extra=[crayons.blue("{0}".format(c.err)),]
pipenv.exceptions.VirtualenvCreationException: Failed to create virtual environment.

Steps to replicate

Pipfile :

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
pytest = "*"

[requires]
python_version = "3.7"

Workaround

Running instead :
pipenv install --python=/usr/bin/python3.7
fixes the problem by specifying which python needs to be used.

---

$ pipenv --support

Pipenv version: '2018.11.26'

Pipenv location: '/home/alexv/.local/lib/python3.6/site-packages/pipenv'

Python location: '/usr/bin/python3'

Python installations found:

• 3.7.2: /mnt/c/Program Files/Python37/python.exe
• 3.7.2: /mnt/c/Program Files/Python37/pythonw.exe
• 3.7.1: /usr/bin/python3.7
• 3.7.1: /usr/bin/python3.7m
• 3.6.7: /usr/bin/python3
• 3.6.7: /usr/bin/python3.6m

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.7',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.4.0-17763-Microsoft',
 'platform_system': 'Linux',
 'platform_version': '#253-Microsoft Mon Dec 31 17:49:00 PST 2018',
 'python_full_version': '3.6.7',
 'python_version': '3.6',
 'sys_platform': 'linux'}

System environment variables:

• LS_COLORS
• HOSTTYPE
• LESSCLOSE
• LANG
• DIRENV_DIR
• OLDPWD
• DIRENV_DIFF
• VIRTUAL_ENV
• USER
• DIRENV_WATCHES
• PWD
• HOME
• NAME
• XDG_DATA_DIRS
• PIPENV_ACTIVE
• SHELL
• TERM
• SHLVL
• LOGNAME
• PATH
• WSLENV
• LESSOPEN
• BASH_FUNC_show_virtual_env%%
• _
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PIP_SHIMS_BASE_MODULE
• PIP_PYTHON_PATH
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

• PIPENV_ACTIVE: 1

Debug–specific environment variables:

• PATH: /bin:/home/alexv/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/mnt/c/Program Files/Python37/Scripts:/mnt/c/Program Files/Python37:/mnt/c/Windows/System32:/mnt/c/Windows:/mnt/c/Windows/System32/wbem:/m nt/c/Windows/System32/WindowsPowerShell/v1.0:/mnt/c/Windows/System32/OpenSSH:/mnt/c/ProgramData/chocolatey/bin:/mnt/c/Program Files/NVIDIA Corporation/NVIDIA NvDLISR:/mnt/c/Program Files (x86)/NVIDIA Corporation/PhysX/Common:/mnt/c/Program Files/Git/cmd:/mnt/c/Users/A lexV/AppData/Roaming/Python/Python37/Scripts:/mnt/c/tools/Cmder:/mnt/c/tools/direnv:/mnt/c/tools/Syncthing:/mnt/c/Users/AlexV/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/AlexV/AppData/Local/GitHubDesktop/bin:/snap/bin
• SHELL: /bin/bash
• LANG: C.UTF-8
• PWD: /mnt/c/Users/AlexV/Documents/GitHub/functional-python/01
• VIRTUAL_ENV: ``

---

Contents of Pipfile ('/mnt/c/Users/AlexV/Documents/GitHub/functional-python/01/Pipfile'):

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
pytest = "*"

[requires]
python_version = "3.7"

Contents of Pipfile.lock ('/mnt/c/Users/AlexV/Documents/GitHub/functional-python/01/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "828b8ad012f4c8773e6e61e3ac2be0ffcd7540fd7ed175a8355676c8e31c4d3d"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.7"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "atomicwrites": {
            "hashes": [
                "sha256:0312ad34fcad8fac3704d441f7b317e50af620823353ec657a53e981f92920c0",
                "sha256:ec9ae8adaae229e4f8446952d204a3e4b5fdd2d099f9be3aaf556120135fb3ee"
            ],
            "version": "==1.2.1"
        },
        "attrs": {
            "hashes": [
                "sha256:10cbf6e27dbce8c30807caf056c8eb50917e0eaafe86347671b57254006c3e69",
                "sha256:ca4be454458f9dec299268d472aaa5a11f67a4ff70093396e1ceae9c76cf4bbb"
            ],
            "version": "==18.2.0"
        },
        "more-itertools": {
            "hashes": [
                "sha256:38a936c0a6d98a38bcc2d03fdaaedaba9f412879461dd2ceff8d37564d6522e4",
                "sha256:c0a5785b1109a6bd7fac76d6837fd1feca158e54e521ccd2ae8bfe393cc9d4fc",
                "sha256:fe7a7cae1ccb57d33952113ff4fa1bc5f879963600ed74918f1236e212ee50b9"
            ],
            "version": "==5.0.0"
        },
        "pluggy": {
            "hashes": [
                "sha256:8ddc32f03971bfdf900a81961a48ccf2fb677cf7715108f85295c67405798616",
                "sha256:980710797ff6a041e9a73a5787804f848996ecaa6f8a1b1e08224a5894f2074a"
            ],
            "version": "==0.8.1"
        },
        "py": {
            "hashes": [
                "sha256:bf92637198836372b520efcba9e020c330123be8ce527e535d185ed4b6f45694",
                "sha256:e76826342cefe3c3d5f7e8ee4316b80d1dd8a300781612ddbc765c17ba25a6c6"
            ],
            "version": "==1.7.0"
        },
        "pytest": {
            "hashes": [
                "sha256:41568ea7ecb4a68d7f63837cf65b92ce8d0105e43196ff2b26622995bb3dc4b2",
                "sha256:c3c573a29d7c9547fb90217ece8a8843aa0c1328a797e200290dc3d0b4b823be"
            ],
            "index": "pypi",
            "version": "==4.1.1"
        },
        "six": {
            "hashes": [
                "sha256:3350809f0555b11f552448330d0b52d5f24c91a322ea4a15ef22629740f3761c",
                "sha256:d16a0141ec1a18405cd4ce8b4613101da75da0e9a7aec5bdd4fa804d0e0eba73"
            ],
            "version": "==1.12.0"
        }
    },
    "develop": {}
}

[patrickayoup]

I had the same issue on WSL, I noticed something interesting.
My Pipfile requires 3.7 but I didn't have 3.7 installed on the subsystem. After installing it, pipenv install worked without any issue and didn't try to go to the windows executable anymore.

[tony]

pipenv install --python=/usr/bin/python3.7 does fix for me

pipenv is also pulling up python.exe for me

Config / settings

Ubuntu WSL, Windows 1903, 
❯ sudo lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.2 LTS
Release:        18.04
Codename:       bionic

❯ pip --version
pip 19.1.1 from /home/x/.local/lib/python3.6/site-packages/pip (python 3.6)

❯ pipenv --version
pipenv, version 2018.11.26

❯ python --version
Python 2.7.15rc1

❯ python3 --version
Python 3.6.7

❯ python3.7 --version
Python 3.7.1

I also bumped into this issue from inside a WSL environment.

I have python 3.7 installed, but pip is installing to my python3.6 site packages by default.

I only specify python 3 in my Pipfile:

[requires]
python_version = "3"

Error

Error output:
['Traceback (most recent call last):\n', '  File "/home/x/.local/lib/python3.6/site-packages/pipenv/vendor/vistir/misc.py", line 160, in _create_subprocess\n    combine_stderr=combine_stderr)\n', '  File "/home/x/.local/lib/python3.6/site-packages/pipenv/vendor/vistir/misc.py", line 134, in _spawn_subprocess\n    return subprocess.Popen(cmd, **options)\n', '  File "/usr/lib/python3.6/subprocess.py", line 709, in __init__\n    restore_signals, start_new_session)\n', '  File "/usr/lib/python3.6/subprocess.py", line 1344, in _execute_child\n    raise child_exception_type(errno_num, err_msg, err_filename)\n', "OSError: [Errno 8] Exec format error: '/mnt/c/Users/tony/AppData/Local/Microsoft/WindowsApps/python.exe'\n", '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', '  File "/home/x/.local/lib/python3.6/site-packages/pipenv/vendor/vistir/contextmanagers.py", line 150, in spinner\n    yield _spinner\n', '  File "/home/x/.local/lib/python3.6/site-packages/pipenv/vendor/vistir/misc.py", line 314, in run\n    write_to_stdout=True\n', '  File "/home/x/.local/lib/python3.6/site-packages/pipenv/vendor/vistir/misc.py", line 162, in _create_subprocess\n    sys.stderr.write("Error %s while executing command %s", exc, " ".join(cmd._parts))\n', 'TypeError: write() takes 2 positional arguments but 4 were given\n']
Traceback (most recent call last):
  File "/home/x/.local/lib/python3.6/site-packages/pipenv/vendor/vistir/misc.py", line 160, in _create_subprocess
    combine_stderr=combine_stderr)
  File "/home/x/.local/lib/python3.6/site-packages/pipenv/vendor/vistir/misc.py", line 134, in _spawn_subprocess
    return subprocess.Popen(cmd, **options)
  File "/usr/lib/python3.6/subprocess.py", line 709, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.6/subprocess.py", line 1344, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 8] Exec format error: '/mnt/c/Users/tony/AppData/Local/Microsoft/WindowsApps/python.exe'
During handling of the above exception, another exception occurred:

[thernstig]

I ran into this as well today. Very strange that it looks for the Python in the Windows path.

[frostming]

The python version on Windows is higher than that on WSL, making it come before the latter one..

Specify python path explicitly can fix the problem:

$ pipenv --python /usr/bin/python3

[thernstig]

@frostming I am unsure why you closed this, can you elaborate? This is a real concern that you most likely will get a large influx of issues on. This is a serious bug that will hit anyone developing in WSL.

In the Windows 10, May 2019 update (19.03) python.exe and python3.exe are found by default, but they do not point to a real Python executable. I.e. a user have these without having installed Python in Windows. Read this announcement https://devblogs.microsoft.com/python/python-in-the-windows-10-may-2019-update/ and more specifically this quote:

Finally, with the May 2019 Windows Update, we are completing the picture. While Python continues to remain completely independent from the operating system, every install of Windows will include python and python3 commands that take you directly to the Python store page.

I get this failure even though I have only installed python into the WSL distribution, and my project resides in the /home/username/ dir.

11:56:55 computer1 ~/code/scrapy (master)$ which python
/usr/bin/python
11:57:01 computer1 ~/code/scrapy (master)$ which python3
/usr/bin/python3
11:57:04 computer1 ~/code/scrapy (master)$ printenv PATH
/home/username/code/scrapy/.venv/bin:/home/username/.pyenv/shims:/home/username/.pyenv/bin:/home/username/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/mnt/c/Program Files/WindowsApps/CanonicalGroupLimited.UbuntuonWindows_1804.2019.521.0_x64__79rhkp1fndgsc:/mnt/c/WINDOWS/system32:/mnt/c/WINDOWS:/mnt/c/WINDOWS/System32/Wbem:/mnt/c/WINDOWS/System32/WindowsPowerShell/v1.0/:/mnt/c/WINDOWS/System32/OpenSSH/:/mnt/c/Users/username/AppData/Local/Microsoft/WindowsApps:/snap/bin

This completely breaks working in WSL and automatic installing like with pyenv, or using direnv with layout_pipenv. And the user on the 2019 May update has not even installed in Windows.

[frostming]

@thernstig I didn't say this is not an issue. Let's track it in #3807

There should be a fix in pythonfinder projects and the vendored library.

[shamoons]

How do you maintain multiple Python versions? Specifically 3.6 and 3.7?

[yuri-stepanov]

@shamoons I personally use pyenv for that. (https://github.com/pyenv/pyenv)

[mjherich]

I got annoyed from having to specify my python version every time so I just removed %USERPROFILE%\AppData\Local\Microsoft\WindowsApps from PATH in Windows. Now pipenv never gets confused with the python.exe that Micrsoft put in there.

[danmichaelo]

I got annoyed from having to specify my python version every time so I just removed %USERPROFILE%\AppData\Local\Microsoft\WindowsApps from PATH in Windows. Now pipenv never gets confused with the python.exe that Micrsoft put in there.

Also works to just disable the app execution alias (Settings > Apps > Apps & features > Manage app execution aliases): https://superuser.com/a/1461471/56154