Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Wrong libprotobuf Version Dependency? #210

Closed
astrostu opened this issue Sep 30, 2024 · 12 comments · Fixed by #211
Closed

Wrong libprotobuf Version Dependency? #210

astrostu opened this issue Sep 30, 2024 · 12 comments · Fixed by #211
Assignees
Labels

Comments

@astrostu
Copy link

I'm doing a new install and was running into issues for any of the available PLIO versions (1.5.2–1.5.5). I install it, then open Python and do something simple like import plio and the following results:

  File "<stdin>", line 1, in <module>
  File "/opt/anaconda3/envs/plio/lib/python3.12/site-packages/plio/__init__.py", line 17, in <module>
    from . import io
  File "/opt/anaconda3/envs/plio/lib/python3.12/site-packages/plio/io/__init__.py", line 22, in <module>
    from . import io_controlnetwork
  File "/opt/anaconda3/envs/plio/lib/python3.12/site-packages/plio/io/io_controlnetwork.py", line 10, in <module>
    from plio.io import ControlNetFileV0002_pb2 as cnf
  File "/opt/anaconda3/envs/plio/lib/python3.12/site-packages/plio/io/ControlNetFileV0002_pb2.py", line 34, in <module>
    _descriptor.EnumValueDescriptor(
  File "/opt/anaconda3/envs/plio/lib/python3.12/site-packages/google/protobuf/descriptor.py", line 920, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

I exited Python, ran conda install protobuf=3.20, and now it seems to be working -- or, at least, doesn't throw errors when I just run import plio. The version that was installed originally was 5.27.5. I haven't tested the rest of my code so I'm not sure if this relatively simple work-around solved things, but it's something to look into. Note that this is on an M1-chip Mac -- I've been migrating to arm-64 architecture for all my conda installs except ISIS to try to avoid the Rosetta 2 translation hit.

@michaelaye
Copy link
Contributor

you could make it work with 3.20? I'm asking because I can't (on a M3 Mac):

  File "/Users/maye/Dropbox/src/nbcoreg/coreg/postprocessing.py", line 13, in <module>
    import plio.io.io_controlnetwork as cn
  File "/Users/maye/miniforge3/envs/py312/lib/python3.12/site-packages/plio/__init__.py", line 17, in <module>
    from . import io
  File "/Users/maye/miniforge3/envs/py312/lib/python3.12/site-packages/plio/io/__init__.py", line 22, in <module>
    from . import io_controlnetwork
  File "/Users/maye/miniforge3/envs/py312/lib/python3.12/site-packages/plio/io/io_controlnetwork.py", line 10, in <module>
    from plio.io import ControlNetFileV0002_pb2 as cnf
  File "/Users/maye/miniforge3/envs/py312/lib/python3.12/site-packages/plio/io/ControlNetFileV0002_pb2.py", line 34, in <module>
    _descriptor.EnumValueDescriptor(
  File "/Users/maye/miniforge3/envs/py312/lib/python3.12/site-packages/google/protobuf/descriptor.py", line 920, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

87-77-193-240 in Dropbox/src/nbcoreg on  master [!+?⇡] is 📦 v0.15.0 via 🅒 coreg took 6s 
❯ mamba list libprotobuf
# packages in environment at /Users/maye/miniforge3/envs/coreg:
#
# Name                    Version                   Build  Channel
libprotobuf               3.20.3               hb5ab8b9_0    conda-forge

87-77-193-240 in Dropbox/src/nbcoreg on  master [!+?⇡] is 📦 v0.15.0 via 🅒 coreg 
❯ mamba list protobuf
# packages in environment at /Users/maye/miniforge3/envs/coreg:
#
# Name                    Version                   Build  Channel
libprotobuf               3.20.3               hb5ab8b9_0    conda-forge
protobuf                  3.20.3          py311ha397e9f_1    conda-forge

87-77-193-240 in Dropbox/src/nbcoreg on  master [!+?⇡] is 📦 v0.15.0 via 🅒 coreg 
❯ mamba install protobuf=3.19 libprotobuf=3.19

Looking for: ['protobuf=3.19', 'libprotobuf=3.19']

conda-forge/osx-arm64                                       Using cache
conda-forge/noarch                                          Using cache

Pinned packages:
  - python 3.11.*
  - protobuf 3.20.*

and with 3.19 I'd would have to go down to Python 3.10 apparently? Is this all just a conda hickup? Maybe I should try pypi... On Python 3.12 this doesn't work at all as well. It looks like we "just" need to rerun the *_pb2.py files that are being delivered with the package? But I don't know how they are being created, or how hard it would be.

@michaelaye
Copy link
Contributor

Thanks for working on this! I'm not fully understanding why, but this seems to create near-catastrophic dependency issues for me ATM, maybe some older versions were removed from conda-forge? So, for the workaround I currently need to go down to numpy=1, protobuf=3.20, pandas=2.1, and pyarrow=9.0.0 which dask already warns me won't work for much longer. welcome to dependency hell... ;) guess i really should start freezing working environments...

@astrostu
Copy link
Author

astrostu commented Nov 7, 2024

Yes, I got it to work on Linux and Mac by just adding protobuf=3.20 install. I did have to ensure the Python environment was no higher than 3.12.x, though; default install on two of my computers went to Python 3.13 so I had to kill the environment and re-make it explicitly saying python=3.12.

@michaelaye
Copy link
Contributor

bizarre, doesn't work here at all with 3.12 on osx-arm:

❯ conda create -n test_plio python=3.12 protobuf=3.20
Channels:
 - conda-forge
Platform: osx-arm64
Collecting package metadata (repodata.json): done
Solving environment: failed

LibMambaUnsatisfiableError: Encountered problems while solving:
  - nothing provides _python_rc needed by python-3.12.0rc3-rc3_h47c9636_1_cpython

Could not solve for environment specs
The following packages are incompatible
├─ protobuf 3.20**  is installable with the potential options
│  ├─ protobuf [3.20.0|3.20.1|3.20.2|3.20.3] would require
│  │  └─ python_abi 3.10.* *_cp310, which can be installed;
│  ├─ protobuf [3.20.0|3.20.1|3.20.2|3.20.3] would require
│  │  └─ python_abi 3.8.* *_cp38, which can be installed;
│  ├─ protobuf [3.20.0|3.20.1|3.20.2|3.20.3] would require
│  │  └─ python_abi 3.9.* *_cp39, which can be installed;
│  └─ protobuf [3.20.2|3.20.3] would require
│     └─ python_abi 3.11.* *_cp311, which can be installed;
└─ python 3.12**  is not installable because there are no viable options
   ├─ python [3.12.0|3.12.1|...|3.12.7] would require
   │  └─ python_abi 3.12.* *_cp312, which conflicts with any installable versions previously reported;
   └─ python 3.12.0rc3 would require
      └─ _python_rc, which does not exist (perhaps a missing channel).

@astrostu
Copy link
Author

astrostu commented Nov 7, 2024

Not sure what to say. Not knowing you could specify protobuf in the initial call, here's what I do:

conda create -n plio python=3.12
conda activate plio
conda config --env --add channels conda-forge
conda install parallel #sometimes need -c conda-forge, usually Linux and not Apple
conda install psutil
conda install plio
conda install protobuf=3.20

Maybe deliberately installing protobuf=3.20 after everything else is what works? A case where my ignorance of more complex conda stuff came in handy?

@michaelaye
Copy link
Contributor

Note that simultaneous provision of all required library leads to more reliable and importantly repeatable results (b/c things are then not order-dependent in case the dependency solver is buggy). This is also noted in the official docs:
https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-with-commands

I really would like to understand this situation, if you have the time, also because you might have a broken enviroment without knowing it.

For linux-64, there does not exist a protobuf version 3.20.* on the conda-forge channel that satisfies the Python=3.12 requirement:

❯ mamba search 'protobuf[channel=conda-forge, subdir=linux-64, version=3.20.*]'|grep py312     (plio) 

maye in 🌐 apollo in ~ via 🅒 plio took 15s 

or, without the grep:

❯ mamba search 'protobuf[channel=conda-forge, subdir=linux-64, version=3.20.*]'                (plio) 
Loading channels: done
# Name                       Version           Build  Channel             
protobuf                      3.20.0 py310hd8f1fbe_0  conda-forge         
protobuf                      3.20.0 py310hd8f1fbe_1  conda-forge         
protobuf                      3.20.0 py310hd8f1fbe_2  conda-forge         
protobuf                      3.20.0 py310hd8f1fbe_3  conda-forge         
protobuf                      3.20.0 py310hd8f1fbe_4  conda-forge         
protobuf                      3.20.0 py310hd8f1fbe_5  conda-forge         
protobuf                      3.20.0  py37h2ca511a_0  conda-forge         
protobuf                      3.20.0  py37hd23a5d3_0  conda-forge         
protobuf                      3.20.0  py37hd23a5d3_1  conda-forge         
protobuf                      3.20.0  py37hd23a5d3_2  conda-forge         
protobuf                      3.20.0  py37hd23a5d3_3  conda-forge         
protobuf                      3.20.0  py37hd23a5d3_4  conda-forge         
protobuf                      3.20.0  py37hd23a5d3_5  conda-forge         
protobuf                      3.20.0  py38ha8be2cd_1  conda-forge         
protobuf                      3.20.0  py38ha8be2cd_2  conda-forge         
protobuf                      3.20.0  py38ha8be2cd_3  conda-forge         
protobuf                      3.20.0  py38ha8be2cd_4  conda-forge         
protobuf                      3.20.0  py38ha8be2cd_5  conda-forge         
protobuf                      3.20.0  py38hfa26641_0  conda-forge         
protobuf                      3.20.0  py38hfa26641_1  conda-forge         
protobuf                      3.20.0  py38hfa26641_2  conda-forge         
protobuf                      3.20.0  py38hfa26641_3  conda-forge         
protobuf                      3.20.0  py38hfa26641_4  conda-forge         
protobuf                      3.20.0  py38hfa26641_5  conda-forge         
protobuf                      3.20.0  py39h0e26352_1  conda-forge         
protobuf                      3.20.0  py39h0e26352_2  conda-forge         
protobuf                      3.20.0  py39h0e26352_3  conda-forge         
protobuf                      3.20.0  py39h0e26352_4  conda-forge         
protobuf                      3.20.0  py39h0e26352_5  conda-forge         
protobuf                      3.20.0  py39h5a03fae_0  conda-forge         
protobuf                      3.20.0  py39h5a03fae_1  conda-forge         
protobuf                      3.20.0  py39h5a03fae_2  conda-forge         
protobuf                      3.20.0  py39h5a03fae_3  conda-forge         
protobuf                      3.20.0  py39h5a03fae_4  conda-forge         
protobuf                      3.20.0  py39h5a03fae_5  conda-forge         
protobuf                      3.20.1 py310hd8f1fbe_0  conda-forge         
protobuf                      3.20.1  py37hd23a5d3_0  conda-forge         
protobuf                      3.20.1  py38ha8be2cd_0  conda-forge         
protobuf                      3.20.1  py38hfa26641_0  conda-forge         
protobuf                      3.20.1  py39h0e26352_0  conda-forge         
protobuf                      3.20.1  py39h5a03fae_0  conda-forge         
protobuf                      3.20.2 py310hd8f1fbe_0  conda-forge         
protobuf                      3.20.2 py310hd8f1fbe_1  conda-forge         
protobuf                      3.20.2 py311ha362b79_1  conda-forge         
protobuf                      3.20.2  py38ha8be2cd_0  conda-forge         
protobuf                      3.20.2  py38ha8be2cd_1  conda-forge         
protobuf                      3.20.2  py38hfa26641_0  conda-forge         
protobuf                      3.20.2  py38hfa26641_1  conda-forge         
protobuf                      3.20.2  py39h0e26352_0  conda-forge         
protobuf                      3.20.2  py39h0e26352_1  conda-forge         
protobuf                      3.20.2  py39h5a03fae_0  conda-forge         
protobuf                      3.20.2  py39h5a03fae_1  conda-forge         
protobuf                      3.20.3 py310heca2aa9_1  conda-forge         
protobuf                      3.20.3 py311hcafe171_1  conda-forge         
protobuf                      3.20.3  py38h8dc9893_1  conda-forge         
protobuf                      3.20.3  py38ha7a3bcd_1  conda-forge         
protobuf                      3.20.3  py39h227be39_1  conda-forge         
protobuf                      3.20.3  py39he08593d_1  conda-forge         

But they are available on the commercial `defaults' channel:

maye in 🌐 apollo in ~ via 🅒 plio took 51s 
❯ mamba search 'protobuf[channel=defaults, subdir=osx-arm64, version=3.20.*]'|grep py312       (plio) 
protobuf                      3.20.3 py312h313beb8_0  pkgs/main           

maye in 🌐 apollo in ~ via 🅒 plio 
❯ mamba search 'protobuf[channel=defaults, subdir=linux-64, version=3.20.*]'|grep py312        (plio) 
protobuf                      3.20.3 py312h6a678d5_0  pkgs/main      

So, most likely, your default channel is defaults, and by adding the conda-forge channel after creating the new environment, your allowed channel sources look like this:

❯ conda config --show-sources                                                                  (plio) 
==> /home/maye/miniforge3/.condarc <==
channels:
  - defaults

==> /home/maye/.condarc <==
channels:
  - defaults

==> /home/maye/miniforge3/envs/plio/.condarc <==
channels:
  - conda-forge
  - defaults

or something similar. Importantly, your env init already took all required packages from the default channel, as you can confirm with conda list:

❯ conda list                                                                                   (plio) 
# packages in environment at /home/maye/miniforge3/envs/plio:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             5.1                       1_gnu  
bzip2                     1.0.8                h5eee18b_6  
ca-certificates           2024.9.24            h06a4308_0  
expat                     2.6.3                h6a678d5_0  
ld_impl_linux-64          2.40                 h12ee557_0  
libffi                    3.4.4                h6a678d5_1  
libgcc-ng                 11.2.0               h1234567_1  
libgomp                   11.2.0               h1234567_1  
libstdcxx-ng              11.2.0               h1234567_1  
libuuid                   1.41.5               h5eee18b_0  
ncurses                   6.4                  h6a678d5_0  
openssl                   3.0.15               h5eee18b_0  
pip                       24.2            py312h06a4308_0  
python                    3.12.7               h5148396_0  
readline                  8.2                  h5eee18b_0  
setuptools                75.1.0          py312h06a4308_0  
sqlite                    3.45.3               h5eee18b_0  
tk                        8.6.14               h39e8969_0  
tzdata                    2024b                h04d1e81_0  
wheel                     0.44.0          py312h06a4308_0  
xz                        5.4.6                h5eee18b_1  
zlib                      1.2.13               h5eee18b_1  

no entry in channel means, they come from the defaults channel, arguable a bad interface design i might say

When I now continue with installing your list of packages up to plio, I (and most likely you, too, otherwise your system should not be able to find a protobuf=3.20 for py312) end up with an environment that is heavily mixing packages from the default channel and the conda-forge channel:

libev                     4.33                 hd590300_2    conda-forge
libexpat                  2.6.3                h5888daf_0    conda-forge
libffi                    3.4.4                h6a678d5_1  
libgcc                    14.2.0               h77fa898_1    conda-forge
libgcc-ng                 14.2.0               h69a702a_1    conda-forge
libgfortran               14.2.0               h69a702a_1    conda-forge
libgfortran-ng            14.2.0               h69a702a_1    conda-forge
libgfortran5              14.2.0               hd5240d6_1    conda-forge
libgomp                   14.2.0               h77fa898_1    conda-forge
liblapack                 3.9.0           25_linux64_openblas    conda-forge
libnghttp2                1.57.0               h2d74bed_0  
libnsl                    2.0.1                hd590300_0    conda-forge
libopenblas               0.3.28          pthreads_h94d23a6_1    conda-forge
libsqlite                 3.46.0               hde9e2c9_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx                 14.2.0               hc0a3c3a_1    conda-forge
libstdcxx-ng              11.2.0               h1234567_1  
libtiff                   4.3.0                h542a066_3    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libwebp-base              1.4.0                hd590300_0    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libzlib                   1.2.13               h4ab18f5_6    conda-forge
lz4-c                     1.9.4                h6a678d5_1  
markupsafe                3.0.2           py312h178313f_0    conda-forge
ncurses                   6.4                  h6a678d5_0  
networkx                  3.4.2              pyhd8ed1ab_1    conda-forge
numpy                     2.1.3           py312h58c1407_0    conda-forge
openssl                   3.3.2                hb9d3cd8_0    conda-forge
pandas                    2.2.3           py312hf9745cd_1    conda-forge
parallel                  20240922             ha770c72_0    conda-forge
perl                      5.32.1          7_hd590300_perl5    conda-forge
pip                       24.2            py312h06a4308_0  
plio                      1.5.5              pyh8c1a49c_0    conda-forge
proj                      9.3.1                he5811b7_0  
protobuf                  5.28.2          py312h2ec8cdc_0    conda-forge
psutil                    6.1.0           py312h66e93f0_0    conda-forge
pvl                       1.3.2              pyhd8ed1ab_0    conda-forge
pyproj                    3.6.1           py312h38f1c37_5    conda-forge
python                    3.12.2          hab00c5b_0_cpython    conda-forge
python-dateutil           2.9.0              pyhd8ed1ab_0    conda-forge
python-tzdata             2024.2             pyhd8ed1ab_0    conda-forge
python_abi                3.12                    5_cp312    conda-forge
pytz                      2024.1             pyhd8ed1ab_0    conda-forge
pyyaml                    6.0.2           py312h66e93f0_1    conda-forge
readline                  8.2                  h5eee18b_0  
scipy                     1.14.1          py312h62794b6_1    conda-forge
setuptools                75.1.0          py312h06a4308_0  
six                       1.16.0             pyh6c4a22f_0    conda-forge
sqlalchemy                2.0.36          py312h66e93f0_0    conda-forge
sqlite                    3.45.3               h5eee18b_0  
tk                        8.6.14               h39e8969_0  
typing-extensions         4.12.2               hd8ed1ab_0    conda-forge
typing_extensions         4.12.2             pyha770c72_0    conda-forge
tzdata                    2024b                h04d1e81_0  
wheel                     0.44.0          py312h06a4308_0  
xz                        5.4.6                h5eee18b_1  
yaml                      0.2.5                h7f98852_2    conda-forge
zlib                      1.2.13               h4ab18f5_6    conda-forge
zstd                      1.5.6                hc292b87_0  

This is just calling for trouble. It can work in 99% of time, but inexplicably fail in the remaining 1%, and if you ever had the feeling that your env is less stable than others, this might be just the reason. But don't take my word for granted and search the web for 'conda "mix channels"' and see what others say about this.

You can check your installed status for an env by calling (when the right env is active):

  • conda list
  • conda config --show-sources to check where you draw packages from.

So, if you want to avoid all this, plus make your IT department happy by absolutely staying away from the commercial defaults channel and only use the open-source channel conda-forge that is freely available (because the conda/mamba tools themselves are also free to use, you can edit all files that appear when doing conda config --show-sources and replace any appearance of defaults with conda-forge and then you are good to go and never have to add the extra channel anymore after creating new environments. I am doing this for several years now without any major issue (apart from the occasional hickup relying on open source packages with a low update frequency like plio).
My standard env is about 500 packages all from conda-forge working together like a charm for our, most likely quite comparable, planetary geospatial stack requirements. Hope this helps. And I can sleep now, understanding one more conda conundrum.. ;) happy weekend!

@github-project-automation github-project-automation bot moved this from In Progress to Done in Astro Software Support Sprint Nov 8, 2024
@Kelvinrr
Copy link
Collaborator

Kelvinrr commented Nov 8, 2024

Merging the PR caused this to auto close, I don't think this can be closed until we have a release that doesn't have this issue.

@Kelvinrr Kelvinrr reopened this Nov 8, 2024
@michaelaye
Copy link
Contributor

I tried installing the PR into a fresh env with Python 3.12 and import plio works now, with some kind of warning text:

In [1]: import plio
/Users/maye/Dropbox/src/plio/plio/io/io_krc.py:155: SyntaxWarning: invalid escape sequence '\d'
  header = re.findall(b'\d+', fullheader.split(b'<<')[0])
/Users/maye/Dropbox/src/plio/plio/io/io_krc.py:216: SyntaxWarning: invalid escape sequence '\d'
  ver = re.findall(b'\d+', head)

These are my versions:

❯ mamba list protobu
# packages in environment at /Users/maye/miniforge3/envs/coreg:
#
# Name                    Version                   Build  Channel
libprotobuf               5.28.2               h8f0b736_0    conda-forge
protobuf                  5.28.2          py312hf02c72a_0    conda-forge

epimetheus in Dropbox/src/plio on  proto-regen via 🐍 v3.12.7 via 🅒 coreg 
❯ mamba list plio
# packages in environment at /Users/maye/miniforge3/envs/coreg:
#
# Name                    Version                   Build  Channel
plio                      1.5.5                     dev_0    <develop>

@michaelaye
Copy link
Contributor

hm, for a different reason I did another fresh install of a python 3.12 env with my full stack of packages PLUS some Amazon lib dependencies for the new hostess package, and then I didn't get above warning? Not sure what happened, but maybe disregard my previous post.

@michaelaye
Copy link
Contributor

i see it was released in 1.6.0 but not yet disseminated via pypi/conda.

@Kelvinrr
Copy link
Collaborator

Kelvinrr commented Nov 20, 2024

Conda forge recipe is failing, it's still on my radar to fix it this week so the cond\a release goes through. Just a lot of stuff going on at once.

@Kelvinrr
Copy link
Collaborator

Kelvinrr commented Nov 25, 2024

new plio is out and should no longer have the import issue. Just make sure to get 1.6

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
Projects
Development

Successfully merging a pull request may close this issue.

5 participants