Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Windows Support #2658

Open
jsirois opened this issue Feb 4, 2025 · 5 comments
Open

Windows Support #2658

jsirois opened this issue Feb 4, 2025 · 5 comments
Assignees
@jsirois
Labels
enhancement feature request in progress

Comments

@jsirois
Copy link
Member

jsirois commented Feb 4, 2025 •
edited
Loading

This tracks the second major effort at bringing Windows support to Pex - the 1st being in 2022. Notes will go here, issues may be broken out for actual issues.

The big change from the 1st effort will be to do this incrementally with a dedicated red CI shard that has diagnostics to quantify the redness. Hopefully watching the redness go from ~100% to 0% will be the difference that helps get this over the line.
@jsirois jsirois self-assigned this Feb 4, 2025
@jsirois
Copy link
Member Author

jsirois commented Feb 4, 2025

Since Python 3.9 is now the current last supported release from CPython (see: https://devguide.python.org/versions/#python-release-cycle), I'll be starting there.

@jsirois
Copy link
Member Author

jsirois commented Feb 4, 2025 •
edited
Loading

Known major issues:

jsirois added a commit to jsirois/pex that referenced this issue Feb 5, 2025
@jsirois
Checkpoint 1.
8f3f59e 
Working:
+ `py -mpex cowsay -ccowsay -- -t Moo!`
+ `py -mpex cowsay -ccowsay --venv -- -t Moo!`
+ `py -mpex cowsay -ccowsay -ocowsay.pex`
+ `py -mpex cowsay -ccowsay --venv -ocowsay.pex`
+ `py -mpex cowsay`
+ `py -mpex.cli cache {prune,purge}`

Next up:
+ Figure out tox on Windows.
+ Set up failing CI.

Work towards pex-tool#2658
jsirois added a commit to jsirois/pex that referenced this issue Feb 5, 2025
@jsirois
Checkpoint 1.
9077c1b 
Working:
+ `py -mpex cowsay -ccowsay -- -t Moo!`
+ `py -mpex cowsay -ccowsay --venv -- -t Moo!`
+ `py -mpex cowsay -ccowsay -ocowsay.pex`
+ `py -mpex cowsay -ccowsay --venv -ocowsay.pex`
+ `py -mpex cowsay`
+ `py -mpex.cli cache {prune,purge}`

Next up:
+ Figure out tox on Windows.
+ Set up failing CI.

Work towards pex-tool#2658
jsirois added a commit to jsirois/pex that referenced this issue Feb 6, 2025
@jsirois
Add Windows support in key critical areas.
0a2a7cc 
Although Pex still does not support Windows, this is work towards pex-tool#2658.

In particular, support is added for:
+ Shared and exclusive file locks.
+ Versions os os.link, os.symlink and os.rename that either work with
  Windows or work with most modern Windows setups.
+ Executable file detection that works for Windows.
+ Partial support for Windows venvs by accomodating its Scripts dir and
  console script style (.exe stubs).

Informal testing shows the following work:
+ `py -mpex cowsay`
+ `py -mpex cowsay -ccowsay {,--venv} -- -t Moo!`
+ `py -mpex cowsay -ccowsay {,--venv} -ocowsay.pex`
+ `py -mpex cowsay -ccowsay -ocowsay.pex --scie {eager,lazy}`
+ `py -mpex.cli cache {prune,purge}`

Important non-working things include:
+ The developer system: `tox ...` needs work most fundamentally.
+ Venv scripts / venv scies: Need support for Windows .exe stub scripts.
@jsirois jsirois mentioned this issue Feb 6, 2025
Merged
jsirois added a commit that referenced this issue Feb 6, 2025
@jsirois
Add Windows support in key critical areas. (#2663)
5b06cce 
Although Pex still does not support Windows, this is work towards #2658.

In particular, support is added for:
+ Shared and exclusive file locks.
+ Versions of `os.link`, `os.symlink` and `os.rename` that either work
  with Windows or work with most modern Windows setups.
+ Executable file detection that works for Windows.
+ Partial support for Windows venvs by accommodating its `Scripts` dir
  and console script style (`.exe` stubs).

Informal testing shows the following work:
+ `py -mpex cowsay`
+ `py -mpex cowsay -ccowsay {,--venv} -- -t Moo!`
+ `py -mpex cowsay -ccowsay {,--venv} -ocowsay.pex`
+ `py -mpex cowsay -ccowsay -ocowsay.pex --scie {eager,lazy}`
+ `py -mpex.cli cache {prune,purge}`

Important non-working things include:
+ The developer system: `tox ...` needs work most fundamentally.
+ Venv scripts / venv scies: Need support for Windows `.exe` stub
  scripts.
jsirois added a commit to jsirois/pex that referenced this issue Feb 6, 2025
@jsirois
Windows Build Fixes.
5cdcff4 
Re-jigger tox and git to support Windows development / CI. As part of
this, fix `pex.pyenv` to work with `pyenv-win`.

More work towards pex-tool#2658.
@jsirois jsirois mentioned this issue Feb 6, 2025
Merged
jsirois added a commit that referenced this issue Feb 7, 2025
@jsirois
Windows Build Fixes. (#2664)
39e8778 
Re-jigger configuration for tox and git to support Windows dev / CI.
As part of this, fix `pex.pyenv` to work with `pyenv-win`.

More work towards #2658.

CF: tox-dev/tox#3136 (comment)
jsirois added a commit to jsirois/pex that referenced this issue Feb 7, 2025
@jsirois
Add a system to produce Windows script .exes.
6b8df1b 
Use the stubs generated by the uv project for this. The lineage extends
back to distutils through the Posy project. The API is setup and
manually tested to produce working windows scripts and the packaging
system is updated to embed these stubs in the Pex PEX, wheel and scies.
Actually using the API to create scripts when creating venvs on Windows
is still left to do.

Work towards pex-tool#2658.
@jsirois jsirois mentioned this issue Feb 7, 2025
Merged
jsirois added a commit to jsirois/pex that referenced this issue Feb 8, 2025
@jsirois
Consolidate relative symlink creation.
15cb0b8 
This is work towards pex-tool#2658 which will need to tackle falling back from
symlinks when there are insufficient permissions to create them.
@jsirois jsirois mentioned this issue Feb 8, 2025
Merged
jsirois added a commit that referenced this issue Feb 8, 2025
@jsirois
Consolidate relative symlink creation. (#2667)
a16c3de 
This is work towards #2658 which will need to tackle falling back from
symlinks when there are insufficient permissions to create them.
jsirois added a commit that referenced this issue Feb 8, 2025
@jsirois
Add a system to produce Windows script .exes. (#2665)
b94a7ea 
Use the stubs generated by the uv project for this. The lineage extends
back to distutils through the Posy project. The API is setup and
manually tested to produce working Windows scripts and the packaging
system is updated to embed these stubs in the Pex PEX, wheel and scies.
Actually using the API to create scripts when creating venvs on Windows
is still left to do.

Work towards #2658.
jsirois added a commit to jsirois/pex that referenced this issue Feb 9, 2025
@jsirois
Use Windows scripts stubs.
0f4fc3b 
More work towards pex-tool#2658.
@jsirois jsirois mentioned this issue Feb 9, 2025
Merged
jsirois added a commit that referenced this issue Feb 9, 2025
@jsirois
Use Windows scripts stubs. (#2668)
9a22ec6 
More work towards #2658.
jsirois added a commit to jsirois/pex that referenced this issue Feb 9, 2025
@jsirois
Initial Windows CI setup w/o caching.
247c66a 
More work towards pex-tool#2658.
@jsirois jsirois mentioned this issue Feb 9, 2025
Merged
jsirois added a commit that referenced this issue Feb 9, 2025
@jsirois
Add a Windows test report workflow.
7d38a4f 
This should allow highlighting Windows test status during the bring-up
period.

More work towards #2658.
jsirois added a commit to jsirois/pex that referenced this issue Feb 9, 2025
@jsirois
Initial Windows CI setup w/o caching.
5fb7e77 
More work towards pex-tool#2658.
jsirois added a commit that referenced this issue Feb 22, 2025
@jsirois
Initialize fingerprints db differently on Windows. (#2695)
75fb09c 
More work towards #2658.
@jsirois jsirois mentioned this issue Feb 24, 2025
Merged
jsirois added a commit to jsirois/pex that referenced this issue Feb 24, 2025
@jsirois
Shard Windows tests 4 ways.
ca701b3 
Windows is currently extremely slow; help it out.

More work towards pex-tool#2658.
@jsirois jsirois mentioned this issue Feb 24, 2025
Merged
jsirois added a commit to jsirois/pex that referenced this issue Feb 24, 2025
@jsirois
Fix docs server for Windows.
f89afaa 
A bit hacky, but this seems to work well. Hack tracked in pex-tool#2699.

More work towards pex-tool#2658.
@jsirois jsirois mentioned this issue Feb 24, 2025
Merged
jsirois added a commit that referenced this issue Feb 24, 2025
@jsirois
Shard Windows tests 4 ways. (#2698)
8013e01 
Windows is currently extremely slow; help it out.

More work towards #2658.
jsirois added a commit to jsirois/pex that referenced this issue Feb 24, 2025
@jsirois
Skip tests that use pexpect on Windows.
a18d4b5 
More work towards pex-tool#2658.
jsirois added a commit that referenced this issue Feb 24, 2025
@jsirois
Fix docs server for Windows. (#2700)
40a6c21 
A bit hacky, but this seems to work well. Hack tracked in #2699.

More work towards #2658.
@jsirois jsirois mentioned this issue Feb 24, 2025
Merged
jsirois added a commit that referenced this issue Feb 25, 2025
@jsirois
Skip tests that use pexpect on Windows. (#2701)
c4d3cde 
More work towards #2658.
jsirois added a commit to jsirois/pex that referenced this issue Feb 25, 2025
@jsirois
Fix script detection for Windows.
8ac8534 
More work towards pex-tool#2658.
@jsirois jsirois mentioned this issue Feb 25, 2025
Merged
jsirois added a commit that referenced this issue Feb 25, 2025
@jsirois
Use a devpi host Windows can handle. (#2697)
5fd4478 
Windows will serve from 0.0.0.0 but does not like connecting to 0.0.0.0 
as a TCP client.

More work towards #2658.
jsirois added a commit to jsirois/pex that referenced this issue Feb 25, 2025
@jsirois
Fix script detection for Windows.
8ef88f7 
More work towards pex-tool#2658.
jsirois added a commit that referenced this issue Feb 26, 2025
@jsirois
Fix script detection for Windows. (#2702)
6cc6f1e 
More work towards #2658.
@jsirois
Copy link
Member Author

jsirois commented Feb 26, 2025

Ok, the basic machinery is now solid: https://github.com/pex-tool/pex/runs/37826704273

Currently at 740 passed, 849 failed and 148 skipped.

jsirois added a commit to jsirois/pex that referenced this issue Feb 26, 2025
@jsirois
Fix test_inject_python_args for Windows.
6a6f94a 
More work towards pex-tool#2658.
@jsirois jsirois mentioned this issue Feb 26, 2025
Merged
@jsirois
Copy link
Member Author

jsirois commented Feb 26, 2025

745 passed, 844 failed and 148 skipped

jsirois added a commit that referenced this issue Feb 26, 2025
@jsirois
Fix test_inject_python_args for Windows. (#2703)
c1a20f2 
More work towards #2658.
jsirois added a commit to jsirois/pex that referenced this issue Feb 27, 2025
@jsirois
Fix cryptography builds on Windows.
76f1cdb 
More work towards pex-tool#2658.
@jsirois jsirois mentioned this issue Feb 27, 2025
Merged
jsirois added a commit to jsirois/pex that referenced this issue Feb 27, 2025
@jsirois
Skip tests that use pyenv Python 2.7 on Windows.
f3e30c4 
More work towards pex-tool#2658.
@jsirois jsirois mentioned this issue Feb 27, 2025
Merged
jsirois added a commit that referenced this issue Feb 27, 2025
@jsirois
Skip tests that use pyenv Python 2.7 on Windows. (#2705)
ff17647 
More work towards #2658.
@jsirois
Copy link
Member Author

jsirois commented Feb 27, 2025

745 passed, 802 failed and 190 skipped

jsirois added a commit to jsirois/pex that referenced this issue Feb 27, 2025
@jsirois
Fix cryptography builds on Windows.
1c38748 
More work towards pex-tool#2658.
jsirois added a commit that referenced this issue Feb 28, 2025
@jsirois
Fix cryptography builds on Windows. (#2704)
5880b9b 
More work towards #2658.
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees

@jsirois jsirois

Labels
enhancement feature request in progress
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jsirois