provide pep425 platform tag for AIX #6922
Comments
ghost
added
the
|
This may be better in pypa/packaging as we're planning to replace |
chrahunt
added
S: awaiting response
|
OK. I forked packaging, and shall add a PR there as well. But - how do I test this? And, will pypa/packaging (do not know how to make it blue, etc.) also replace pep425tags in pypa/wheel (where I am also submitting a change asap). If yes - wonderful - only one change (to make for the future) - but how to test now? |
no-response
bot
removed
the
S: awaiting response
Are you asking how to write a test? Or ... |
|
No. not write a test. Not sure what I would test. How to test using Tag, actually - although, perhaps I should/could ask the same there. My struggle is that I also have this same 'chunk' added into pypa/wheel (about to post that PR as well). To date, the testing I have been doing involves using pypa/pip and pypa/wheel, both patched, using the devpi (https://pypi.org/project/devpi/) platform to test upload and download of wheels. I have this much working - before I opened a new issue/PR - to be sure I had the minimum working. A new piece I also want to see happening is when an AIX system, at a higher, better later, platform tag level than any published wheel - accepts the published wheel - as it should be binary compatible. I have been doing this (providing pre-packaged modules and Python) - packaged on AIX 6.1 with "users" of the packaging running these, unmodified on AIX 7.1 and AIX 7.2. I want, now that it is clearly doable (including virtualenv environments), to make packaging available using Python (and pypi repository) because it is superior to platform software management tools. (On AIX these are installp or RPM). So, back to testing - how do I test using Tag() rather than pep425tags.get_platform() ? |
Generally speaking, whenever there is a change to the behavior of a function, there should usually be a unit test testing that change IMO. And in cases where the behavior change depends on a particular platform, it's best if the test can be written to run on all platforms (especially if the platform isn't part of the CI grid). That may mean patching and/or mocking certain functions.
Since pip isn't even using |
|
May be related to #5453. |
|
Maybe I do not understand how you use the word behavior as in the context of change behavior. I would not see this as a behavior change - only more specific output string. Without the change (and the accompanying one in pypa/wheel) I can create wheels - but the tag (coming from distutils.itil/get_platform(), or sys.get_platform() is insufficient. Including AIX 6.1 TL7 - which is the version of AIX 6.1 I build on - there are three TL (technology levels) (7, 8 and 9) to be considered. For AIX 7.1 there are currently 6 TL0..TL5), and for AIX 7.2 that will soon be 4 (TL3 is soon to be released)). So, back to simplicity - if I package on AIX 6.1 TL7 and load on AIX 6.1 TL9, both as 32 or 64 bit, the current tagging could work. However, if I packaged on AIX 6.1 TL9 that current tagging would try to load the tagged wheel, but it would not be binary compatible. AIX provides binary compatibility from old to new, not both directions. A 'behavior change I would like to get included, but separate from this tag change, is that pypa tools would accept a properly tagged AIX 6.1TL7 wheel on e.g., AIX 61.TL9, or AIX 7.2 TL0, while and AIX 7.1 TL3 system would reject a wheel tagged as coming from AIX 7.1 TL4. In short, the current proposal merely resolves the inadequacy of distutils.util.get_paltform() (and sys.get_platform()) to adequately tag wheels intended for AIX. A 'bug' that has been in the system since at least 2007 when IBM adopted the "TL-SP" methodology and also adequately labeled the filesets (using the M digit of VRMF) to identify the Technology Level. |
By "change in behavior," I was making a distinction between PR's that don't change anything for the end user (e.g. refactoring PR's) versus PR's that do change something (e.g. bug fixes, feature additions, etc). In the case of PR #6923 that you proposed, what it does is change the return value of |
|
Ah :bigsmile:, yes, something simple I might even be able to do. FYI - in case you are wondering why I do not use "oslevel -s" to get the values. On this server, the man pages, and X11 rte is behind (not the base kernel |
OK cool, so there's a forward compatible ABI.
I'm confused. IIUC, you're the primary distributor of AIX builds for Python, and have contributed compatibility patches upstream as well. Is there some reason you can't patch the Python that you distribute, to return the appropriate value from distutils/sys? Or even contribute that patch upstream to CPython? My main concern is that while you know AIX very well, I have no clue what that platform is or how it works and honestly, no interest in learning about it. Having more bits of code in pip, for which we have exactly 0 expertise or significant understanding among the maintainers, is not a scenario I'd want to be in. To be clear, I'm not opposed to AIX support -- I have a really strong preference for not being responsible for things I don't understand. :) |
|
re: a test - I am not familiar with pytest. How do I get verbose output similar to python -m test -v test_xyz I am also trying options python setup.py test seems equivalent with python setup.py -s tests that seems to only run tests.unit.test_vcs - HOW to get more! Michael |
Yes, unofficial since 1998-1999, official since 2007 (and AIX 6.1).
We may be in Python4 before that would ever get added 😄 and :sigh:
I am a primary packager. IBM itself prefers to package using gcc and RPM - and has shown no interest in the Python way.
While I could patch sys.get_platform() and distutils.util.get_platform() to get around the PR process in 4 repositories (cpython, pip, wheel, packaging) I do not feel it is the right way to be doing this.
I understand. I do not like to flame anyone, but IBM should have made some kind of support for Python possible 'decades' ago. It breaks my heart to see it ignored, and as a consequence, the MANY hacks VARs and customers apply to get anything done. Worse, it could become a reason to leave AIX. So, if you say no - I may just use the idea you gave me (patch Cpython outside of official sources). What I did not mention above is that it seems to take forever and a day to get a PR for AIX considered. There are people, who when approached individually, look more closely - and some progress is made. Everyone is a volunteer - and I do not want to be pinging, ttt posts, etc.. As much as possible, for better and worse, I am trying to be part of the process - not outside of it.
Understood. As my understanding of Python grows I identify where AIX support is lacking. Wheels (and pip) are great. Off topic: one of the things I liked about perl is CPAN. Wheels are so much better. But it must be done right - or the forward compatible ABI cannot work properly. p.s. Someone checked in PyPi for me - there are zero AIX wheels/eggs etc. in PyPi, so we have no pip/setuptools/wheel dependencies to worry about. There is light in the tunnel! |
Hmm... I feel that might actually be the right way. I'll optimistically ping @ncoghlan, @pfmoore and @dstufft since they all likely have a better understanding of this stuff than me. |
I don't know if have already, but if you haven't, please look at https://pip.pypa.io/en/latest/development/getting-started/. That should contain the answer to your question. To answer your question more directly, you can do: This will invoke pytest like: That should do what I think you want to do. |
I agree. I also have zero knowledge of AIX, and no real interest in, or need for, knowing more. However, I do think the fact that you're patching multiple places to work around a limitation in the stdlib distutils function says something about where the fix should go. Having said that, it will also make a difference what versions of Python you want to support here. There is a near-zero chance, I suspect, that a change like this would be viewed as a bugfix for CPython, so you'd be targeting Python 3.9+, which may limit your audience. So from a purely practical point of view, making the change in the packaging tools may be your only option. However, I share @pradyunsg's reluctance to take OS-specific patches in pip. I've seen the discussions you have had in getting AIX fixes into CPython, and while I have sympathy with you over the difficulties you have encountered, I also understand why the core team don't want to be left supporting platform-specific code that only you understand. If you put yourself on the critical path for AIX on pip, setuptools and wheel as well, the problem gets even worse. One possible workaround for the issue might be if you were to create a (pure Python) |
|
I can understand why the bitness should be in the ABI tag, but I'm not sure about the build week and TL number. Wouldn't that imply a specific package would need to be uploaded for each client, or that each client would need to include their latest and all prior versions in their list of applicable tags? |
|
On 26/08/2019 17:01, Paul Moore wrote:
> Hmm... I feel that might actually be the right way.
I agree. I also have zero knowledge of AIX, and no real interest in, or need for, knowing more. However, I do think the fact that you're patching multiple places to work around a limitation in the stdlib distutils function says something about where the fix should go.
Having said that, it will also make a difference what versions of Python you want to support here. There is a near-zero chance, I suspect, that a change like this would be viewed as a bugfix for CPython, so you'd be targeting Python 3.9+, which may limit your audience. So from a purely practical point of view, making the change in the packaging tools may be your only option.
Would be 'nice' if it was in 3.8 also officially - in
distutils.util.get_platform() - perhaps calling a pure-python _aix.py
(as I did also in Lib/ctypes).
Same for sys.get_platform() - perhaps the _aix.py could go in "sys" and
distutils.util could import that - only code in one place.
As to Python3.7 - I could patch that manually before packaging - and
refer to the official code - that it is there, etc..
However, I share @pradyunsg's reluctance to take OS-specific patches in pip. I've seen the discussions you have had in getting AIX fixes into CPython, and while I have sympathy with you over the difficulties you have encountered, I also understand why the core team don't want to be left supporting platform-specific code that only you understand. If you put yourself on the critical path for AIX on pip, setuptools and wheel as well, the problem gets even worse.
Maybe 'packaging' will fix more things. But I shall try patching
cpython, installing, and testing with default pip and wheel. The strange
issue with wheel is that it uses a special feature that apparantly calls
dist.utils.get_platform() so that when the wheel pep425 code goes
looking, it sees it is already set, so does not call the OS specific
code. Again, if the cpython code was giving a good tag - that would not
be needed either.
Please let me emphasize - with no patches - pip and wheel do what they
are suppossed to do, but the tag is not sufficient.
I'll reply to your second mail re: technology and build_date. Those are
vital to permitting forward compatible ABI.
… One possible workaround for the issue might be if you were to create a (pure Python) `aix-support` library, that contained helper utilities for adding AIX support to pip etc. Then you could maintain and support that library, and the packaging tools could depend on it. If there's any issue with AIX, we could direct the problem at your library, and all we'd need to do would be to update our dependencies once you release a fix. There would likely be an initial patch to the packaging tools to call your library, but ongoing support would be handled by you, not by pip. Also, that sort of approach could be used as a model for any future requests for support of less common platforms.
|
|
On 27/08/2019 02:26, Christopher Hunt wrote:
I can understand why the bitness should be in the ABI tag, but I'm not sure about the build week and TL number. Wouldn't that imply a specific package would need to be uploaded for each client, or that each client would need to include their latest and all prior versions in their list of applicable tags?
I either wrote earlier, or intended to send a larger text regarding VRMF
and build-date. Here is an excerpt:
While one might think you can update (migrate) from any level of AIX 6.1
to any level of AIX 7.1
this is not the case. There is a final value, not usually mentioned,
known as the "build-date".
For an AIX system to update and maintain binary compatibility for
applications the new OS level
must satisfy all of the following AIX0=>AIX1 (V0 <= V1, R0 <= R1, M0 <=
M1, Builddate0 <= Builddate1)
e.g., these two official AIX releases:
6100-07-01-1141
6100-06-12-1339
Although AIX 6.1 TL6 appears to be 'younger' that AIX 6.1 TL7 - which is
true 'feature wise', that is what a TL specifies - the build-dates are
not. the ABI forward compatibility is not supported. To update
6100-06-12-1339 I would need, at a minimum - 6100-07-08-1339. All VRM-BD
numbers are increased or equal (the SP numbers, just as the F in VRMF)
do not assist with maintaining ABI forward compatibility.
I hope this answers your question re: TL and builddate.
Note: because of the difference in features I do not want a (oslevel -s)
tag of
6100-07-08-1339 and 6100-06-12-1339 to be compressed to 6100-1339
To compress the tag - rather than so many parts - just go with:
6106-1339 and 6107-1339, etc..
This is regarding 'tag' generation - and again, pip and wheel will work
- asis - with these tags. A feature change is adding the logic to see
that a wheel packaged with tag 6107-1339 can be loaded by 6108-1339
(6100-08-03-1339).
Thanks for asking!
Michael
|
I really have no interest in understanding the complexities of AIX. But do you understand that if I run pip on a machine that is tag 6108-1339, in order to be able to load wheels tagged with 6107-1339, pip has to know that (a) that older tag exists, and (b) it's compatible with a 6108-1339 system? See I'd strongly recommend that you test any proposed changes by trying to install a wheel generated on an older version of AIX onto a system that runs a newer version. If that's something you already do, and you have this covered, then that's fine. I'm not looking at any code you've submitted, just going off your explanations (which I could easily have misunderstood, as they include a lot of things I'm not even trying to understand 🙂) |
This was what I was trying to say, thank you. Indicating that all the old build numbers are compatible means that list has to be kept somewhere. I can't see how it could be generated. |
I replied by e-mail, re: how to extract a list from AIX of potential tags (oslevel -s -q - with q for query). I have not researched how .../+simple/package works. Being naive I thought it generated a query to ask what is there, and use what is available as it's list. I am taking no concept of older tags to no concept of other tags, only my current tag. re: macos, if I understand the basics correctly, the build system has some defines in the build that it uses to determine it's oldest level possible (or is that required level, not sure if macOS is actually ABI forward compatible. My wife complains that she old software no longer works after an update - and I stay away from macOS because of similar concerns. And, likewise, I am no longer interested in figuring it out (did that back in macOS v2 through v7 days). So, thanks for the hints and suggestions. p.s., to someone else - yes, I am testing - incrementally. Now I'll go a step back to see if patches to pip and wheel can be skipped for the 'basics', and then look at writing an _aix.py module for the .get_supported() piece. Note: if I slow in responses - it is because my real job is getting busy and time is limited. Clearly not because I am losing interest. I consider this quite important and I want to see it done correctly. |
|
As to the best start... I have a potential PR for CPython that "fixes" sysconfig.get_platform() and distutils.util.get_platform(). With stock pip and wheel, pip debug gives expected output, as does pip wheel cffi |
I'll also investigate this. Currently working with Cpython/Lib/_aix_support (using CPython/Lib/_osx_support as an example) |
|
Just noting that I'm actually the one that nudged @aixtools towards finding an approach that would allow the use of properly tagged AIX wheels on existing versions of Python, rather than relying solely on standard library updates that would only be available on Python 3.9+. While it may end up making sense to structure the work as "That is the proper fix for Python 3.9+,this is a backport of equivalent behaviour to earlier versions", I'm not entirely convinced of that - platform tags should be able to naturally evolve based on platform update cycles, not standard library baseline API updates (for example, see the multitude of challenges around the various incarnations of the manylinux spec). |
|
@ncoghlan Understood, but I still don't think it's a good idea to embed specialised platform knowledge for an obscure platform like AIX into tools like pip - hence my suggestion of some sort of |
|
@pfmoore Yeah, I posted my comment before I had fully caught up on the rest of the thread. Your So what I'm starting to wonder is whether there may be value in having some kind of naming convention for support libraries that adapt
Precedent: |
Yes, I think there is. This is a fairly interesting idea that I think I'll bring up on the |
|
The PR I shall be submitting for consideration consists of an additional env variable (for the builddate of the built python - as this sets the base value for any packaging - same, in purpose, as MACOSX_DEPLOYMENT_TARGET. This would not actually be referenced by anything until someone gets around to writing aix_packaging.get_platforms(). The other change is Lib/_aix_support.py and changing the code for aix from/to: # fall through to standard osname-release-machine representation
elif osname[:3] == "aix":
- return "%s-%s.%s" % (osname, version, release)
+ from _aix_support import get_platform as aix_platform
+ return aix_platform()
elif osname[:6] == "cygwin":Should we 'switch' the main discussion of "pypa/packaging"? @ncoghlan - does it make good sense to hurry with an issue and PR on CPython? I had investigated pypa/* first, to see also what could be done outside of Cpython. However, except for get_platform() - which needs some specifics to be able to support multiple tags (equality already works) - just having this patch in permits pip download and pip wheel to work without modification to the pypa tools. |
|
@aixtools We've already missed the feature window for 3.8, but having a PR in review for 3.9 would likely help frame the backport discussions for the PyPA tooling. Edit: I originally wrote "accepted for 3.9" above, but changed it as I think the PR will be valuable for discussion purposes even while it's in review, as it makes the scope of the platform support changes clear. |
|
I'll make a PR for 3.9 then. So wish it could have been seen as a fix rather than as a feature. It is something that has been overlooked since "forever". Having taken additional time I believe it is a better tag - at least I think it is easier to read. And I think I'll drop the b from b32|b64 so the fields just split into digits. I'll add the PR reference here, when finished. |
|
PR posted: python/cpython#15678 |
|
FYI: Two months down the road. No action yet on the Phyton3-3.9 PR yet, but it does work. Roughly speaking - a wheel built with v3.6.9 needs to message that the wheel is not supported when the Python version changes (e.g. v3.7.5), but still be accepted if the only change is the level of AIX. I am going to get started with the There were several references to taking this to a different area. For the short term, I'll be busy here - as I understand the basics. |
|
Question for someone who understands the way 'packaging' thinks... In The first line for 'match' is some magic I do not readily follow (re.compile.match(arch) - abbreviated form) but assume it is looking for some generic starting point, and if found has something 'magical', if I understood the comments "get_platform()" was re-written in this file so that it returns the system running Python, rather than the system/platform that built Python. And a list is generated appending the actual arch (which I have understood to be a label for 32-bit or 64-bit. So, it generates a list with "major == 10", and minor is the range[0..minor] and the processor name (ppc, ppc64, intel, x86_64, fat, etc) is taken care of. So, is a list, such as: build: 6106-1043-64 or root@x068:[/data/prj/python/git/python3-3.6.9]python3 x1.py what the 'packaging' system is expecting? As reference - see the "71XX" and the "61XX" as the major part, and 64 (in this case) as the arch part - and the numbers in between as minor - just not sequential. FYI: the numbers come from the system, not a hard-coded table. Any feedback is welcome - before I try and splice this into pep425tags. |
aixtools
changed the title
|
renamed the issue - should this be a new issue instead? Phase 1: is to get a proper tag for AIX - this is straightforward. Phase 2: will be dealing with "platform" tags that AIX can provide. The goal is similar to the manylinux and macos multi-platform support (e.g., PEP513 for manylinux1, unsure if there is an INFO PEP for macos multi-platform support). re: Phase1: I have a patched CPython I am using for Phase1 - and pip and wheel work asis for exact matches. Re: Phase2: testing - to validate it will work - I rename a wheel e.g., mv cryptography-2.8-cp36-cp36m-AIX_5307_0747_32.whl cryptography-2.8-cp36-cp36m-AIX_7105_1731_32.whl. After this I am able to load and use the wheel that was built on AIX 5.3 even though the system loading it is from AIX 7.1. NOTE - please - the wheel is not changed, only it's file name. Note also: via my portals AIX admins are downloading, _mv_ing wheels and installing them successfully. bdist - better wheels - are a tremendous boon to admins who either may not, or cannot, install and use a compiler. I'll be starting a new PR. I am ready to write some documentation or what's new - but would like some assistance with choosing a name for the News entry. |
Use a |
|
so news/6922.feature ? I'll start with that in any case. Thx. |
|
OK. Not sure where the rock is, and where the hard place is - but I feel stuck between the two. From my perspective, this discussion had the PR7375 as it's focus, but it was closed in preference of the PR in pypa/packaging pypa/packaging#190. However, the related discussion pypa/packaging#188 sends us back here. I would like to see a discussion somewhere. Please specify! |
Environment
Description
pip._internal.pep425tags.get_platform() removes dependency from distutils.util.get_platform()
Just as for macos, the value returned by distutils is insufficient for portable packaging,
as it lacks sufficient details.
Expected behavior
When AIX is the platform, return a platform in the form:
AIX_V_R_M_YYWW_bNN
where: V=version; R=release; M=TechLevel; YYWW represents the builddate (Year (19 for 2019); WW is week of year); bNN - where NN is 32|64 for 32 or 64 bit mode)
How to Reproduce
Output
*** Without: ***
pip version: pip 19.2.2 from /opt/lib/python3.7/site-packages/pip (python 3.7)
sys.version: 3.7.4 (default, Aug 14 2019, 12:40:12) [C]
sys.executable: /opt/bin/python3.7
sys.getdefaultencoding: utf-8
sys.getfilesystemencoding: iso8859-1
locale.getpreferredencoding: ISO8859-1
sys.platform: aix6
sys.implementation:
name: cpython
Compatible tags: 20
cp37-cp37m-aix_6_1
cp37-abi3-aix_6_1
cp37-none-aix_6_1
cp36-abi3-aix_6_1
cp35-abi3-aix_6_1
cp34-abi3-aix_6_1
cp33-abi3-aix_6_1
cp32-abi3-aix_6_1
py3-none-aix_6_1
*** With: ***
(build) root@x066:[/home/root]pip debug
WARNING: This command is only meant for debugging. Do not use this with automation for parsing and getting these details, since the output and options of this command may change without notice.
pip version: pip 19.3.dev0 from /home/root/build/lib/python3.7/site-packages/pip-19.3.dev0-py3.7.egg/pip (python 3.7)
sys.version: 3.7.4 (default, Aug 14 2019, 12:40:12) [C]
sys.executable: /home/root/build/bin/python
sys.getdefaultencoding: utf-8
sys.getfilesystemencoding: iso8859-1
locale.getpreferredencoding: ISO8859-1
sys.platform: aix6
sys.implementation:
name: cpython
Compatible tags: 20
cp37-cp37m-AIX_6_1_7_1415_b32
cp37-abi3-AIX_6_1_7_1415_b32
cp37-none-AIX_6_1_7_1415_b32
cp36-abi3-AIX_6_1_7_1415_b32
cp35-abi3-AIX_6_1_7_1415_b32
cp34-abi3-AIX_6_1_7_1415_b32
cp33-abi3-AIX_6_1_7_1415_b32
cp32-abi3-AIX_6_1_7_1415_b32
py3-none-AIX_6_1_7_1415_b32
cp37-none-any
The text was updated successfully, but these errors were encountered: