pythemis: Extra effort to locate Themis Core library #915
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ilammy
added
W-PyThemis 🐍
vixentael
approved these changes
Apr 17, 2022
Instead of each submodule figuring out where the native library is, just initialize it one place and pull it from there.
|
Rebased with no changes, swizzled out the first commit which got already merged into |
vixentael
approved these changes
Apr 18, 2022
Lagovas
reviewed
Apr 18, 2022
Lagovas
approved these changes
Apr 18, 2022
3 tasks
That is, "libthemis" ABI version 0.
First of all, this makes it possible for users to avoid installing
development package of Themis on Linux. That is, only "libthemis",
without "libthemis-dev" or "libthemis-devel".
Second, it makes sure that if Themis 0.XX and Themis 2.YY are
co-installed, then this PyThemis will load Themis 0.XX library.
By default, find_library() favors the latest version, and that
could cause ABI mismatches.
On Linux, this is how 'real' binaries load libraries via run-time
dynamic linker-loader: they remember "libthemis.so.0" soname, ask
ld to load that, and if it fails then your binary fails to start.
On macOS, applications typically remember absolute path to the library,
which would be something like this for default Homebrew install:
/usr/local/opt/libthemis/lib/libthemis.0.dylib
However, Homebrew on Apple Silicon installs libraries in /opt/homebrew
instead of /usr/local, so look there as well.
Existing behavior of using find_library('themis') is used as a fallback
on Linux and macOS. For now, this is a genuine fallback with a warning.
PyThemis 0.15 will be a test-bed for this change, and if it works fine
then some later version might load *only* versioned libraries, assuming
libthemis.so.2 would be an actual risk.
Other systems -- like Windows -- keep using find_library(). I'm not
quite sure with the story about ABI versioning on Windows specifically.
The recommended approach there is to use absolute paths, partly because
of how easy DLL hijacking is there. Anyhow, ignore this issue for now,
and just keep using whatever we were using.
Turns out, if find_library() cannot find a library it simply returns
None and then LoadLibrary() does not raise an exception for that,
instead loading a "None" library, lol. That won't do.
We had people run into situations where native library could not be
found and then the interpreter reports errors like
AttributeError: dlsym(RTLD_DEFAULT, themis_gen_sym_key): symbol not found
which confusingly *look like* some issues with library symbols.
|
Updated the last couple of commits to follow #917's approach. |
vixentael
approved these changes
Apr 21, 2022
Not at all 👍 |
5 tasks
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Put in some extra effort to locate the right native library.
Look for
libthemis.so.0orlibthemis.0.dylibfirst.Instead of looking for some Themis Core, look for the one with the proper ABI version in it. First of all, this finally allows users to install only
libthemispackage, withoutlibthemis-dev. Second, this prevents PyThemis from accidentally loading Themis 1.x or 2.x libraries, should they ever be co-installed in the system.find_library('themis')code path remains as a fallback on Linux and macOS, and it's also used on Windows by default. No idea whether PyThemis works there, but this PR should at least not make it worse.Look in some "standard" search paths as well.
PyThemis expects Themis Core to be installed in one of the paths dynamic linker would be looking in. Normally, it all works out, but
on some 🌈 ｓｐｅｃｉａｌ 🦄 systemsHomebrew on Apple M1 installs libraries into/opt/homebrew, which – surprise! – is not in standard search paths when non-Homebrew Python is used.While arguably it's the user environment issue, Python developers are not very fond of debugging native library loading and that's PyThemis fault for requiring a native library in the first place, so let's at least attempt to make things "just work" before giving up.
So on both Linux and macOS we'd be looking in
/usr/local/libwheremake installputs Themis Core, and on macOS we'd be looking in/opt/homebrew/libjust in case (on Intel CPUs Homebrew installs to/usr/local/lib).Related issues:
Checklist