Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

pip installation fails in fresh virtualenv (Linux) #27

Closed
marthinwurer opened this issue Feb 20, 2018 · 10 comments
Closed

pip installation fails in fresh virtualenv (Linux) #27

marthinwurer opened this issue Feb 20, 2018 · 10 comments
Labels

Comments

@marthinwurer
Copy link

I ran pip install georasters in a fresh 3.6.4 virtualenv on a fairly new Ubuntu 16.04 install, which generated the following output:

Collecting georasters
  Downloading georasters-0.5.9-py2.py3-none-any.whl
Collecting geopandas (from georasters)
  Downloading geopandas-0.3.0-py2.py3-none-any.whl (888kB)
    100% |████████████████████████████████| 890kB 269kB/s 
Collecting affine (from georasters)
  Downloading affine-2.1.0-py3-none-any.whl
Collecting pandas (from georasters)
  Downloading pandas-0.22.0-cp36-cp36m-manylinux1_x86_64.whl (26.2MB)
    100% |████████████████████████████████| 26.3MB 25kB/s 
Collecting pyproj (from georasters)
  Downloading pyproj-1.9.5.1.tar.gz (4.4MB)
    100% |████████████████████████████████| 4.4MB 132kB/s 
Collecting pysal (from georasters)
  Downloading PySAL-1.14.3.tar.gz (17.8MB)
    100% |████████████████████████████████| 17.8MB 36kB/s 
Collecting matplotlib (from georasters)
  Using cached matplotlib-2.1.2-cp36-cp36m-manylinux1_x86_64.whl
Collecting scikit-image (from georasters)
  Downloading scikit_image-0.13.1-cp36-cp36m-manylinux1_x86_64.whl (35.8MB)
    100% |████████████████████████████████| 35.8MB 16kB/s 
Collecting GDAL (from georasters)
  Downloading GDAL-2.2.3.tar.gz (475kB)
    100% |████████████████████████████████| 481kB 495kB/s 
    Complete output from command python setup.py egg_info:
    running egg_info
    creating pip-egg-info/GDAL.egg-info
    writing pip-egg-info/GDAL.egg-info/PKG-INFO
    writing dependency_links to pip-egg-info/GDAL.egg-info/dependency_links.txt
    writing top-level names to pip-egg-info/GDAL.egg-info/top_level.txt
    writing manifest file 'pip-egg-info/GDAL.egg-info/SOURCES.txt'
    Traceback (most recent call last):
      File "/tmp/pip-build-hul8gsn8/GDAL/setup.py", line 131, in fetch_config
        p = subprocess.Popen([command, args], stdout=subprocess.PIPE)
      File "/home/benjamin/.pyenv/versions/3.6.4/lib/python3.6/subprocess.py", line 709, in __init__
        restore_signals, start_new_session)
      File "/home/benjamin/.pyenv/versions/3.6.4/lib/python3.6/subprocess.py", line 1344, in _execute_child
        raise child_exception_type(errno_num, err_msg, err_filename)
    FileNotFoundError: [Errno 2] No such file or directory: '../../apps/gdal-config': '../../apps/gdal-config'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/tmp/pip-build-hul8gsn8/GDAL/setup.py", line 179, in get_gdal_config
        return fetch_config(option, gdal_config = self.gdal_config)
      File "/tmp/pip-build-hul8gsn8/GDAL/setup.py", line 135, in fetch_config
        raise gdal_config_error(e)
    __main__.gdal_config_error: [Errno 2] No such file or directory: '../../apps/gdal-config': '../../apps/gdal-config'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/tmp/pip-build-hul8gsn8/GDAL/setup.py", line 131, in fetch_config
        p = subprocess.Popen([command, args], stdout=subprocess.PIPE)
      File "/home/benjamin/.pyenv/versions/3.6.4/lib/python3.6/subprocess.py", line 709, in __init__
        restore_signals, start_new_session)
      File "/home/benjamin/.pyenv/versions/3.6.4/lib/python3.6/subprocess.py", line 1344, in _execute_child
        raise child_exception_type(errno_num, err_msg, err_filename)
    FileNotFoundError: [Errno 2] No such file or directory: 'gdal-config': 'gdal-config'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-hul8gsn8/GDAL/setup.py", line 339, in <module>
        **extra )
      File "/home/benjamin/.pyenv/versions/3.6.4/envs/georasters/lib/python3.6/site-packages/setuptools/__init__.py", line 129, in setup
        return distutils.core.setup(**attrs)
      File "/home/benjamin/.pyenv/versions/3.6.4/lib/python3.6/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/home/benjamin/.pyenv/versions/3.6.4/lib/python3.6/distutils/dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "/home/benjamin/.pyenv/versions/3.6.4/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/home/benjamin/.pyenv/versions/3.6.4/envs/georasters/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 278, in run
        self.find_sources()
      File "/home/benjamin/.pyenv/versions/3.6.4/envs/georasters/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 293, in find_sources
        mm.run()
      File "/home/benjamin/.pyenv/versions/3.6.4/envs/georasters/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 524, in run
        self.add_defaults()
      File "/home/benjamin/.pyenv/versions/3.6.4/envs/georasters/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 560, in add_defaults
        sdist.add_defaults(self)
      File "/home/benjamin/.pyenv/versions/3.6.4/envs/georasters/lib/python3.6/site-packages/setuptools/command/py36compat.py", line 36, in add_defaults
        self._add_defaults_ext()
      File "/home/benjamin/.pyenv/versions/3.6.4/envs/georasters/lib/python3.6/site-packages/setuptools/command/py36compat.py", line 119, in _add_defaults_ext
        build_ext = self.get_finalized_command('build_ext')
      File "/home/benjamin/.pyenv/versions/3.6.4/lib/python3.6/distutils/cmd.py", line 299, in get_finalized_command
        cmd_obj.ensure_finalized()
      File "/home/benjamin/.pyenv/versions/3.6.4/lib/python3.6/distutils/cmd.py", line 107, in ensure_finalized
        self.finalize_options()
      File "/tmp/pip-build-hul8gsn8/GDAL/setup.py", line 214, in finalize_options
        self.gdaldir = self.get_gdal_config('prefix')
      File "/tmp/pip-build-hul8gsn8/GDAL/setup.py", line 188, in get_gdal_config
        return fetch_config(option)
      File "/tmp/pip-build-hul8gsn8/GDAL/setup.py", line 135, in fetch_config
        raise gdal_config_error(e)
    __main__.gdal_config_error: [Errno 2] No such file or directory: 'gdal-config': 'gdal-config'
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-hul8gsn8/GDAL/

pip list after running that command:

pip (9.0.1)
setuptools (38.5.1)
@ozak
Copy link
Owner

ozak commented Feb 20, 2018

From the message the problem seems to be your gdal installation. Either it is not recognized by your python environment or you may not have it installed.

@marthinwurer
Copy link
Author

Ok, I didn't know that I needed to have it installed. You might want to update the installation section of the README to say that it's a dependency. This is what I have tried to do to get it installed in a virtualenv: install just gdal-bin using the instructions found here, then finding that I need to have libgdal-dev installed from here, but that post also says to use a different gdal package that will work with virtualenvs. This gets me to:

$ gdal-config --version
2.2.2

so gdal is now installed. Now I try to install georasters again. This gets me a new installation error:

$ pip install georasters
Collecting georasters
  Using cached georasters-0.5.9-py2.py3-none-any.whl
Collecting affine (from georasters)
  Using cached affine-2.1.0-py3-none-any.whl
Collecting rasterstats (from georasters)
  Using cached rasterstats-0.12.0-py2.py3-none-any.whl
Collecting pandas (from georasters)
  Using cached pandas-0.22.0-cp36-cp36m-manylinux1_x86_64.whl
Collecting numpy (from georasters)
  Using cached numpy-1.14.1-cp36-cp36m-manylinux1_x86_64.whl
Collecting docopt (from georasters)
  Using cached docopt-0.6.2.tar.gz
Collecting matplotlib (from georasters)
  Using cached matplotlib-2.1.2-cp36-cp36m-manylinux1_x86_64.whl
Collecting scikit-image (from georasters)
  Using cached scikit_image-0.13.1-cp36-cp36m-manylinux1_x86_64.whl
Collecting pyproj (from georasters)
  Using cached pyproj-1.9.5.1.tar.gz
Collecting fiona (from georasters)
  Using cached Fiona-1.7.11.post1-cp36-cp36m-manylinux1_x86_64.whl
Collecting GDAL (from georasters)
  Using cached GDAL-2.2.3.tar.gz
Collecting coverage (from georasters)
  Using cached coverage-4.5.1-cp36-cp36m-manylinux1_x86_64.whl
Collecting pysal (from georasters)
  Using cached PySAL-1.14.3.tar.gz
Collecting geopandas (from georasters)
  Using cached geopandas-0.3.0-py2.py3-none-any.whl
Collecting shapely (from rasterstats->georasters)
  Downloading Shapely-1.6.4.post1-cp36-cp36m-manylinux1_x86_64.whl (1.5MB)
    100% |████████████████████████████████| 1.5MB 928kB/s 
Collecting cligj>=0.4 (from rasterstats->georasters)
  Downloading cligj-0.4.0-py3-none-any.whl
Collecting simplejson (from rasterstats->georasters)
  Downloading simplejson-3.13.2.tar.gz (79kB)
    100% |████████████████████████████████| 81kB 9.1MB/s 
Collecting rasterio>=0.27 (from rasterstats->georasters)
  Downloading rasterio-0.36.0.tar.gz (1.2MB)
    100% |████████████████████████████████| 1.2MB 1.3MB/s 
    Complete output from command python setup.py egg_info:
    CRITICAL:root:Numpy and its headers are required to run setup(). Exiting.
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-w7syp_qj/rasterio/

Looking here, I think I might need to install the numpy headers. First I try the suggestion to just install numpy, and this gets me a bit further. However, this explodes after a while:

$ pip install georasters
Collecting georasters
  Using cached georasters-0.5.9-py2.py3-none-any.whl
Collecting scikit-image (from georasters)
  Using cached scikit_image-0.13.1-cp36-cp36m-manylinux1_x86_64.whl
Collecting matplotlib (from georasters)
  Using cached matplotlib-2.1.2-cp36-cp36m-manylinux1_x86_64.whl
Collecting pandas (from georasters)
  Using cached pandas-0.22.0-cp36-cp36m-manylinux1_x86_64.whl
Requirement already satisfied: numpy in /home/benjamin/.pyenv/versions/3.6.4/envs/georasters/lib/python3.6/site-packages (from georasters)
Collecting geopandas (from georasters)
  Using cached geopandas-0.3.0-py2.py3-none-any.whl
Collecting pysal (from georasters)
  Using cached PySAL-1.14.3.tar.gz
Collecting affine (from georasters)
  Using cached affine-2.1.0-py3-none-any.whl
Collecting pyproj (from georasters)
  Using cached pyproj-1.9.5.1.tar.gz
Collecting coverage (from georasters)
  Using cached coverage-4.5.1-cp36-cp36m-manylinux1_x86_64.whl
Collecting fiona (from georasters)
  Using cached Fiona-1.7.11.post1-cp36-cp36m-manylinux1_x86_64.whl
Collecting docopt (from georasters)
  Using cached docopt-0.6.2.tar.gz
Collecting GDAL (from georasters)
  Using cached GDAL-2.2.3.tar.gz
Collecting rasterstats (from georasters)
  Using cached rasterstats-0.12.0-py2.py3-none-any.whl
Collecting networkx>=1.8 (from scikit-image->georasters)
Collecting six>=1.7.3 (from scikit-image->georasters)
  Using cached six-1.11.0-py2.py3-none-any.whl
Collecting scipy>=0.17.0 (from scikit-image->georasters)
  Using cached scipy-1.0.0-cp36-cp36m-manylinux1_x86_64.whl
Collecting PyWavelets>=0.4.0 (from scikit-image->georasters)
  Using cached PyWavelets-0.5.2-cp36-cp36m-manylinux1_x86_64.whl
Collecting pillow>=2.1.0 (from scikit-image->georasters)
  Using cached Pillow-5.0.0-cp36-cp36m-manylinux1_x86_64.whl
Collecting pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 (from matplotlib->georasters)
  Using cached pyparsing-2.2.0-py2.py3-none-any.whl
Collecting pytz (from matplotlib->georasters)
  Using cached pytz-2018.3-py2.py3-none-any.whl
Collecting cycler>=0.10 (from matplotlib->georasters)
  Using cached cycler-0.10.0-py2.py3-none-any.whl
Collecting python-dateutil>=2.1 (from matplotlib->georasters)
  Using cached python_dateutil-2.6.1-py2.py3-none-any.whl
Collecting shapely (from geopandas->georasters)
  Using cached Shapely-1.6.4.post1-cp36-cp36m-manylinux1_x86_64.whl
Collecting descartes (from geopandas->georasters)
  Downloading descartes-1.1.0-py3-none-any.whl
Collecting cligj (from fiona->georasters)
  Using cached cligj-0.4.0-py3-none-any.whl
Collecting click-plugins (from fiona->georasters)
  Downloading click-plugins-1.0.3.tar.gz
Collecting munch (from fiona->georasters)
  Downloading munch-2.2.0.tar.gz
Collecting rasterio>=0.27 (from rasterstats->georasters)
  Using cached rasterio-0.36.0.tar.gz
Collecting simplejson (from rasterstats->georasters)
  Using cached simplejson-3.13.2.tar.gz
Collecting decorator>=4.1.0 (from networkx>=1.8->scikit-image->georasters)
  Using cached decorator-4.2.1-py2.py3-none-any.whl
Collecting click>=4.0 (from cligj->fiona->georasters)
  Downloading click-6.7-py2.py3-none-any.whl (71kB)
    100% |████████████████████████████████| 71kB 4.5MB/s 
Collecting snuggs (from rasterio>=0.27->rasterstats->georasters)
  Downloading snuggs-1.4.1-py3-none-any.whl
Installing collected packages: decorator, networkx, six, scipy, PyWavelets, pyparsing, pytz, cycler, python-dateutil, matplotlib, pillow, scikit-image, pandas, shapely, pyproj, click, cligj, click-plugins, munch, fiona, descartes, geopandas, pysal, affine, coverage, docopt, GDAL, snuggs, rasterio, simplejson, rasterstats, georasters
  Running setup.py install for pyproj ... done
  Running setup.py install for click-plugins ... done
  Running setup.py install for munch ... done
  Running setup.py install for pysal ... done
  Running setup.py install for docopt ... done
  Running setup.py install for GDAL ... error
    Complete output from command /home/benjamin/.pyenv/versions/3.6.4/envs/georasters/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-fcajdje6/GDAL/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-bd12hsev-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/benjamin/.pyenv/versions/3.6.4/envs/georasters/include/site/python3.6/GDAL:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.6
    copying gdal.py -> build/lib.linux-x86_64-3.6
    copying ogr.py -> build/lib.linux-x86_64-3.6
    copying osr.py -> build/lib.linux-x86_64-3.6
    copying gdalconst.py -> build/lib.linux-x86_64-3.6
    copying gnm.py -> build/lib.linux-x86_64-3.6
    copying gdalnumeric.py -> build/lib.linux-x86_64-3.6
    creating build/lib.linux-x86_64-3.6/osgeo
    copying osgeo/osr.py -> build/lib.linux-x86_64-3.6/osgeo
    copying osgeo/gdalnumeric.py -> build/lib.linux-x86_64-3.6/osgeo
    copying osgeo/__init__.py -> build/lib.linux-x86_64-3.6/osgeo
    copying osgeo/gdalconst.py -> build/lib.linux-x86_64-3.6/osgeo
    copying osgeo/ogr.py -> build/lib.linux-x86_64-3.6/osgeo
    copying osgeo/gnm.py -> build/lib.linux-x86_64-3.6/osgeo
    copying osgeo/gdal_array.py -> build/lib.linux-x86_64-3.6/osgeo
    copying osgeo/gdal.py -> build/lib.linux-x86_64-3.6/osgeo
    Fixing build/lib.linux-x86_64-3.6/gdal.py build/lib.linux-x86_64-3.6/ogr.py build/lib.linux-x86_64-3.6/osr.py build/lib.linux-x86_64-3.6/gdalconst.py build/lib.linux-x86_64-3.6/gnm.py build/lib.linux-x86_64-3.6/gdalnumeric.py build/lib.linux-x86_64-3.6/osgeo/osr.py build/lib.linux-x86_64-3.6/osgeo/gdalnumeric.py build/lib.linux-x86_64-3.6/osgeo/__init__.py build/lib.linux-x86_64-3.6/osgeo/gdalconst.py build/lib.linux-x86_64-3.6/osgeo/ogr.py build/lib.linux-x86_64-3.6/osgeo/gnm.py build/lib.linux-x86_64-3.6/osgeo/gdal_array.py build/lib.linux-x86_64-3.6/osgeo/gdal.py
    Skipping optional fixer: ws_comma
    Fixing build/lib.linux-x86_64-3.6/gdal.py build/lib.linux-x86_64-3.6/ogr.py build/lib.linux-x86_64-3.6/osr.py build/lib.linux-x86_64-3.6/gdalconst.py build/lib.linux-x86_64-3.6/gnm.py build/lib.linux-x86_64-3.6/gdalnumeric.py build/lib.linux-x86_64-3.6/osgeo/osr.py build/lib.linux-x86_64-3.6/osgeo/gdalnumeric.py build/lib.linux-x86_64-3.6/osgeo/__init__.py build/lib.linux-x86_64-3.6/osgeo/gdalconst.py build/lib.linux-x86_64-3.6/osgeo/ogr.py build/lib.linux-x86_64-3.6/osgeo/gnm.py build/lib.linux-x86_64-3.6/osgeo/gdal_array.py build/lib.linux-x86_64-3.6/osgeo/gdal.py
    Skipping optional fixer: ws_comma
    running build_ext
    building 'osgeo._gdal' extension
    creating build/temp.linux-x86_64-3.6
    creating build/temp.linux-x86_64-3.6/extensions
    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I../../port -I../../gcore -I../../alg -I../../ogr/ -I../../ogr/ogrsf_frmts -I../../gnm -I../../apps -I/home/benjamin/.pyenv/versions/3.6.4/envs/georasters/include -I/home/benjamin/.pyenv/versions/3.6.4/include/python3.6m -I/home/benjamin/.pyenv/versions/3.6.4/envs/georasters/lib/python3.6/site-packages/numpy/core/include -I/usr/include -c extensions/gdal_wrap.cpp -o build/temp.linux-x86_64-3.6/extensions/gdal_wrap.o
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
    extensions/gdal_wrap.cpp:3168:22: fatal error: cpl_port.h: No such file or directory
    compilation terminated.
    error: command 'gcc' failed with exit status 1
    
    ----------------------------------------
Command "/home/benjamin/.pyenv/versions/3.6.4/envs/georasters/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-fcajdje6/GDAL/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-bd12hsev-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/benjamin/.pyenv/versions/3.6.4/envs/georasters/include/site/python3.6/GDAL" failed with error code 1 in /tmp/pip-build-fcajdje6/GDAL/

That's the error from my second link. I used the command found here, and that installed gdal-2.2.2 for me.
Next, I try to install georasters again. It worked!

Successfully installed georasters-0.5.9 rasterio-0.36.0 rasterstats-0.12.0 simplejson-3.13.2 snuggs-1.4.1

Ok, I'm basically going to post this here for posterity, and for me the next time I need to install this.

Suggestions: on the install line, list that georasters requires gdal. See if you can add numpy as a dependency. See if you can set a global include path for the compilation on Linux.

@ozak
Copy link
Owner

ozak commented Feb 21, 2018

I've added the GDAL requirement in the info. numpy is a requirement in the installation. Not sure why it is not being fulfilled.

Can you explain what you mean with "set global include path"? Not sure what you faced since I test using linux+conda and had not faced issues there. Although it may be related to something with gdal2 which has made the testing not work ok, but I have not figure out what it is.

@marthinwurer
Copy link
Author

marthinwurer commented Feb 22, 2018

Is there a way to force numpy to install first? If so, I have a feeling that might solve the issue.

As for global include path, when installing GDAL, compilation fails because it can't find the headers. The workaround found here involves telling pip to set the global option --global-option="-I/usr/include/gdal", and that gets it to compile. That won't work for all systems, but if GDAL is installed that way, it works on Ubuntu 16.04.

The full list of commands that I used to install georasters on another fresh machine are as follows:

sudo add-apt-repository -y ppa:ubuntugis/ubuntugis-unstable
sudo apt update
sudo apt upgrade
sudo apt install gdal-bin libgdal-dev
pyenv virtualenv 3.6.4 georasters
pyenv local georasters
pip install --upgrade setuptools
gdal-config --version
pip install numpy
pip install --global-option=build_ext --global-option="-I/usr/include/gdal" GDAL==`gdal-config --version`
pip install georasters

@ozak
Copy link
Owner

ozak commented Feb 22, 2018

@marthinwurer Thanks! I will link these instructions in the readme file. That way linux users who are not using conda can install it.

I am still perplexed as to why it does not install numpy, since it is listed as a requirement. I will have to do some research on that.

@ozak ozak changed the title pip installation fails in fresh virtualenv pip installation fails in fresh virtualenv (Linux) Feb 22, 2018
@ozak ozak added bug help wanted Installation Linux Linux related installation labels Feb 22, 2018
@ozak
Copy link
Owner

ozak commented Feb 22, 2018

From the information provided here it would seem the requirements should have been installed when you do the pip install command. Not sure why this fails in your case using virtualenv.

@marthinwurer
Copy link
Author

That makes me feel like it's either an install order issue or one of the other dependencies has numpy as a non-listed dependency.

@ozak
Copy link
Owner

ozak commented Feb 23, 2018

Strange since numpy is first in the requirements.txt file, so should be installed before anything else.

@ozak
Copy link
Owner

ozak commented Feb 26, 2018

I've added a recipe for conda-forge which should make installation easier... If you're using conda.

@marthinwurer
Copy link
Author

As of today, I also had to install the libspatialindex-dev package with apt. This was on pop!_os 19.04.

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants