installation issue: can't find pythemis on macOS M1 with non-brew way of installing python

[AikakiAdmin]

Describe the bug
I am trying to use Themis using python on MacOS
I installed brew first and followed the instructions. but I am not able to run the code and following error is occurring everytime, what could be be cause of this behaviour.?

Error
File "securecell.py", line 4, in
master_key = GenerateSymmetricKey()
File "/Users/aikaki/Documents/Python Programs/themis/venv/lib/python3.8/site-packages/pythemis/skeygen.py", line 71, in GenerateSymmetricKey
res = themis.themis_gen_sym_key(None, byref(key_length))
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/ctypes/init.py", line 378, in getattr
func = self.getitem(name)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/ctypes/init.py", line 383, in getitem
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(RTLD_DEFAULT, themis_gen_sym_key): symbol not found

Expected behavior
I was expecting the code to run smoothly

Environment (please complete the following information):

• OS: [macOS Monterey version 12.13.2]
• Hardware: [M1 chipset]
• Themis version: [0.14.0]
• Installation way:
  -(https://docs.cossacklabs.com/themis/languages/python/installation/)

[vixentael]

thank you @AikakiAdmin, we will take a look on M1 installation.

maybe openssl installation path? 🤔
cc @ilammy

[vixentael]

@AikakiAdmin could you please provide step by step commands and their outputs that you are running?

We have tested the following on M1 with macOS 12.3.1 (21E258) according to the installation instructions.

Installing themis

brew tap cossacklabs/tap
brew install libthemis

==> Downloading https://github.com/cossacklabs/themis/archive/0.14.0.tar.gz
==> Downloading from https://codeload.github.com/cossacklabs/themis/tar.gz/refs/tags/0.14.0
                                                      #   -=#=- #   #         
==> Reinstalling cossacklabs/tap/libthemis 
==> make install
🍺  /opt/homebrew/Cellar/libthemis/0.14.0: 34 files, 671.8KB, built in 8 seconds
==> Running `brew cleanup libthemis`...

pip install pythemis

DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Collecting pythemis
  Downloading pythemis-0.14.0.tar.gz (12 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: six in /opt/homebrew/lib/python3.9/site-packages (from pythemis) (1.16.0)
Building wheels for collected packages: pythemis
  Building wheel for pythemis (setup.py) ... done
  Created wheel for pythemis: filename=pythemis-0.14.0-py2.py3-none-any.whl size=15946 sha256=d7f3186ea4cc994246e5fb19b6cd163544d8a45ea4d88be10900252fcee4efd9
  Stored in directory: /Users/arturhil/Library/Caches/pip/wheels/54/14/6a/618dc12818f899bec8a0d7f1494f279a53329feae52da7ada2
Successfully built pythemis
Installing collected packages: pythemis
  DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Successfully installed pythemis-0.14.0

Generating keys

python3

from pythemis.skeygen import GenerateSymmetricKey
master_key = GenerateSymmetricKey()
print(master_key)

b'\xcb\x1e\x1d\xca\x9dU\xca\x1b\xac\x94\xd7\x18\x81\xd6Fd\xc7\x84#R\x7f\xe0\xe2o\xf2\xbd\xca["\xd4v\x0e'

that's working on our side. Let's try to find the difference.

[AikakiAdmin]

aikaki@Vijays-MacBook-Pro documents % brew tap cossacklabs/tap
==> Auto-updated Homebrew!
Updated 1 tap (homebrew/core).
==> Updated Formulae
Updated 1 formula.

==> Tapping cossacklabs/tap
Cloning into '/opt/homebrew/Library/Taps/cossacklabs/homebrew-tap'...
remote: Enumerating objects: 89, done.
remote: Counting objects: 100% (2/2), done.
remote: Total 89 (delta 1), reused 1 (delta 1), pack-reused 87
Receiving objects: 100% (89/89), 20.62 KiB | 879.00 KiB/s, done.
Resolving deltas: 100% (32/32), done.
Tapped 1 formula (16 files, 49.9KB).
aikaki@Vijays-MacBook-Pro documents % brew install libthemis
==> Downloading https://github.com/cossacklabs/themis/archive/0.14.0.tar.gz
Already downloaded: /Users/aikaki/Library/Caches/Homebrew/downloads/35b0334bda822db50514b9a9aad69cc14d96a708105b0c31edefb8a9a6375e0b--themis-0.14.0.tar.gz
==> Installing libthemis from cossacklabs/tap
==> make install
🍺 /opt/homebrew/Cellar/libthemis/0.14.0: 34 files, 671.8KB, built in 3 seconds
==> Running brew cleanup libthemis...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see man brew).aikaki@Vijays-MacBook-Pro documents % pip3 install pythemis
Defaulting to user installation because normal site-packages is not writeable
Collecting pythemis
Using cached pythemis-0.14.0.tar.gz (12 kB)
Preparing metadata (setup.py) ... done
Requirement already satisfied: six in /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages (from pythemis) (1.15.0)
Building wheels for collected packages: pythemis
Building wheel for pythemis (setup.py) ... done
Created wheel for pythemis: filename=pythemis-0.14.0-py2.py3-none-any.whl size=15928 sha256=d0332faea884cf12778e0c62f2c531879b33147ec821baf3e57a152bd25f73f9
Stored in directory: /Users/aikaki/Library/Caches/pip/wheels/21/b0/0d/767b69bf9f45e0606ba0b04f8cea0e0bfef38f47a5b47c312b
Successfully built pythemis
Installing collected packages: pythemis

[AikakiAdmin]

It seems you installed later version of the python using brew. My python site package in not writeable

[ilammy]

Okay... so the library seems to be at least loaded, right?

Could you please do

import ctypes.util

print(ctypes.util.find_library('libthemis'))

which is how PyThemis locats the native library. This should print out the following path:

/opt/homebrew/lib/libthemis.dylib

(which is normally a symlink into homebrew's cellar)

If it's something else, this means Python is probably loading not the library it should load.

However, if it is that path, let's check the library symbols:

nm -gU /opt/homebrew/lib/libthemis.dylib

if there is no _themis_gen_sym_key there, then indeed it's not there.

---

At the moment I can't find any known issues in Python library loading & name resolution in ctypes, but this could very well be something linked to that.

[AikakiAdmin]

code is returning None, which mean it is not getting the library. so a path problem.

ikaki@Vijays-MacBook-Pro themis % nm -gU /opt/homebrew/lib/libthemis.dylib
0000000000001f74 T _secure_comparator_append_secret
0000000000001f90 T _secure_comparator_begin_compare
0000000000001eec T _secure_comparator_create
0000000000001f40 T _secure_comparator_destroy
0000000000002380 T _secure_comparator_get_result
00000000000020c4 T _secure_comparator_proceed_compare
0000000000004b78 T _secure_session_cleanup
0000000000005738 T _secure_session_connect
0000000000005088 T _secure_session_create
0000000000004bc8 T _secure_session_destroy
0000000000005108 T _secure_session_generate_connect_request
0000000000005b40 T _secure_session_get_remote_id
0000000000004bfc T _secure_session_init
0000000000005970 T _secure_session_is_established
0000000000006528 T _secure_session_load
0000000000005980 T _secure_session_receive
000000000000645c T _secure_session_save
0000000000005844 T _secure_session_send
00000000000060ec T _secure_session_unwrap
0000000000005fd8 T _secure_session_wrap
00000000000034bc T _themis_gen_ec_key_pair
00000000000033f8 T _themis_gen_rsa_key_pair
0000000000003688 T _themis_gen_sym_key
00000000000034d4 T _themis_get_asym_key_kind
0000000000003590 T _themis_is_valid_asym_key
0000000000001860 T _themis_secure_cell_decrypt_context_imprint
0000000000001594 T _themis_secure_cell_decrypt_seal
0000000000001740 T _themis_secure_cell_decrypt_seal_with_passphrase
0000000000001820 T _themis_secure_cell_decrypt_token_protect
0000000000001844 T _themis_secure_cell_encrypt_context_imprint
00000000000014ac T _themis_secure_cell_encrypt_seal
0000000000001644 T _themis_secure_cell_encrypt_seal_with_passphrase
00000000000017fc T _themis_secure_cell_encrypt_token_protect
0000000000003890 T _themis_secure_message_decrypt
00000000000036b8 T _themis_secure_message_encrypt
000000000000397c T _themis_secure_message_sign
0000000000003b9c T _themis_secure_message_unwrap
0000000000003a10 T _themis_secure_message_verify
0000000000003abc T _themis_secure_message_wrap
aikaki@Vijays-MacBook-Pro themis %

[AikakiAdmin]

library is present

aikaki@Vijays-MacBook-Pro documents % cd /opt/homebrew/lib/
aikaki@Vijays-MacBook-Pro lib % ls
libsoter.0.dylib libspatialindex.6.1.1.dylib libspatialindex_c.6.1.1.dylib libthemis.0.dylib pkgconfig
libsoter.a libspatialindex.6.dylib libspatialindex_c.6.dylib libthemis.a
libsoter.dylib libspatialindex.dylib libspatialindex_c.dylib libthemis.dylib
aikaki@Vijays-MacBook-Pro lib %

[AikakiAdmin]

added path also but same output "None" for the code you suggested. Any idea how libraries are searched in Mac?

aikaki@Vijays-MacBook-Pro ~ % echo $PATH
/opt/homebrew/bin:/opt/homebrew/sbin:/Library/Frameworks/Python.framework/Versions/3.10/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/homebrew/lib/

[AikakiAdmin]

This problem is resolved when installed python using brew. My confusion is if I need to package Themis in docker, what step I need in Dockerfile

aikaki@Vijays-MacBook-Pro themis % python3 securecell.py
/opt/homebrew/lib/libthemis.dylib

[ilammy]

Uh-huh... Thanks, noted your observation. I believe we should add some check for None later, so that this situation produces a more helpful error message. Or just hardcode some extra paths to search in, so that it "just works".

I guess, homebrew's Python has homebrew paths baked into it, while with non-brew Python you wouldn't get /opt/homebrew/lib in your search paths. On x86, I think, it kinda works out because /usr/local/lib is in standard search paths or something. So even if you install Python from a different source, it still works. But not on M1 where /opt in not in the defaults.

---

Any idea how libraries are searched in Mac?

There is DYLD_LIBRARY_PATH environment variable with extra search paths, Python should respect it. (Or LD_LIBRARY_PATH on Linux.)

You can also tweak the native library search paths via sys.path variable as well, IIRC.

[ilammy]

if I need to package Themis in docker, what step I need in Dockerfile

Like, with macOS in Docker?.. 🤔

If you use a Linux-based image for Docker, there are Themis packages for some distros (Fedora, CentOS, Ubuntu, Debian – not for Alpine though, that one you'd likely need to build from source).

If you use Python from the distro's package repositories, Themis packages should work with it fine: yum/apt/rpm them after adding Cossack Labs' repositories.

[AikakiAdmin]

I already built a docker image which uses fast api frame work and I want add some encryption as part of APIs. I am in love with Themis for its simplicity of use and light weight. Let me see how we can go to next step. For now issue is closed.

…

On Sat, 16 Apr 2022 at 7:21 PM, Oleksii Lozovskyi ***@***.***> wrote: if I need to package Themis in docker, what step I need in Dockerfile Like, with *macOS* in Docker?.. 🤔 If you use a Linux-based image for Docker, there are Themis packages for some distros <https://docs.cossacklabs.com/themis/installation/installation-from-packages/> (Fedora, CentOS, Ubuntu, Debian – not for Alpine though, that one you'd likely need to build from source). If you use Python from the distro's package repositories, Themis packages should work with it fine: yum/apt/rpm them after adding Cossack Labs' repositories. — Reply to this email directly, view it on GitHub <#913 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AN67P53HABTJW3PN3MJL2TDVFLAXBANCNFSM5TQAMW2Q> . You are receiving this because you were mentioned.Message ID: ***@***.***>