pytype does not pin dependency versions, leading to build errors in --require-hashes mode #731

ToddG · 2020-11-16T17:24:17Z

SUMMARY

I'm following many of the prescribed suggestions on the hypermodern-python blog post. I have created a seed repo so that I can quickly and easily generate new projects with many of these great suggestions baked in. However, there is one flaw in all of this, and that is w/respect to building against python3.8 and including pytype as a dependency.

ERROR

ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    typed-ast from https://files.pythonhosted.org/packages/77/49/51308e8c529e14bb2399ff6d22998583aa9ae189ec191e6f7cbb4679f7d5/typed_ast-1.4.1-cp38-cp38-manylinux1_x86_64.whl#sha256=498b0f36cc7054c1fead3d7fc59d2150f4d5c6c56ba7fb150c013fbc683a8d2d (from pytype==2020.11.12->-c /tmp/tmp9zvk3gjy (line 211))

Note that this error only occurs when building against python3.8, python3.7 passes fine.

I've included the full output in the REPRO below, but it appears that the various nox sessions will fail with Command pip install --constraint=... because pytype is not explicitly pinning the versions of it's dependencies.

DETAILS

require-hashes mode

This snippet from the generated noxfile shows how to require the nox sessions to all use explicit dependencies:

def install_with_constraints(session: Session, *args: str, **kwargs: Any) -> None:
    """Install application dependencies using constraints."""
    with tempfile.NamedTemporaryFile() as requirements:
        session.run(
            "poetry",
            "export",
            "--dev",
            "--format=requirements.txt",
            f"--output={requirements.name}",
            external=True,
        )
        session.install(f"--constraint={requirements.name}", *args, **kwargs)

pytype dependencies

None of the dependencies are explicitly specified in the pytype requirements.txt:

# Python dependencies for building and testing the pytype source code.
# Make sure you also install the non-Python dependencies described in
# https://github.com/google/pytype/blob/master/CONTRIBUTING.md#pytype-dependencies.
attrs
importlab
ninja
pybind11>=2.6.0rc3
pylint
pyyaml
six
typed_ast

REPRO

generate a new project using cookiecutter per instructions in README and then lint/build/test:

$ cookiecutter https://github.com/ToddG/hypermodern-python-seed
You've downloaded /home/todd/.cookiecutters/hypermodern-python-seed before. Is it okay to delete and re-download it? [yes]: yes
directory_name [directory_name]: test01
project_name [project_name]: 
author_name [author_name]: 
author_email [author_email]: 
description [description]: 
license [license]: 
readme [README.md]: 
homepage [homepage]: 
repository [repository]: 
keywords ["comma","separated","list"]: 
~/temp $ cd test01/
~/temp/test01 $ h
---------------------------------------------
/home/todd/temp/test01
---------------------------------------------
total 32
59638804 drwxrwxr-x 2 todd 4096 Nov 16 09:11 docs
59638814 -rw-rw-r-- 1 todd 1062 Nov 16 09:11 LICENSE
59638811 -rw-rw-r-- 1 todd   58 Nov 16 09:11 mypy.ini
59638816 -rw-rw-r-- 1 todd 2896 Nov 16 09:11 noxfile.py
59638810 -rw-rw-r-- 1 todd 1165 Nov 16 09:11 pyproject.toml
59638815 -rw-rw-r-- 1 todd  479 Nov 16 09:11 README.md
59773452 drwxrwxr-x 3 todd 4096 Nov 16 09:11 src
59638809 drwxrwxr-x 2 todd 4096 Nov 16 09:11 tests
~/temp/test01 $ nox
nox > Running session lint-3.7
nox > Creating virtual environment (virtualenv) using python3.7 in .nox/lint-3-7
nox > poetry export --dev --format=requirements.txt --output=/tmp/tmp9uzpl4fp
The lock file does not exist. Locking.
Updating dependencies
Resolving dependencies... (5.5s)

Writing lock file
nox > pip install --constraint=/tmp/tmp9uzpl4fp flake8 flake8-annotations flake8-black flake8-isort flake8-docstrings darglint
nox > flake8 src tests noxfile.py docs/conf.py
nox > Session lint-3.7 was successful.
nox > Running session mypy-3.7
nox > Creating virtual environment (virtualenv) using python3.7 in .nox/mypy-3-7
nox > poetry export --dev --format=requirements.txt --output=/tmp/tmpimlqsr6e
nox > pip install --constraint=/tmp/tmpimlqsr6e mypy
nox > mypy src tests noxfile.py docs/conf.py
Success: no issues found in 6 source files
nox > Session mypy-3.7 was successful.
nox > Running session pytype-3.7
nox > Creating virtual environment (virtualenv) using python3.7 in .nox/pytype-3-7
nox > poetry export --dev --format=requirements.txt --output=/tmp/tmp9br_1bs8
nox > pip install --constraint=/tmp/tmp9br_1bs8 pytype
nox > pytype --disable=import-error src tests noxfile.py docs/conf.py
Computing dependencies
Analyzing 6 sources with 0 local dependencies
ninja: Entering directory `/home/todd/temp/test01/.pytype'
[6/6] check noxfile
Leaving directory '/home/todd/temp/test01/.pytype'
Success: no errors found
nox > Session pytype-3.7 was successful.
nox > Running session tests-3.7
nox > Creating virtual environment (virtualenv) using python3.7 in .nox/tests-3-7
nox > poetry install --no-dev
Installing dependencies from lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing click (7.1.2)

Installing the current project: project_name (0.1.0)
nox > poetry export --dev --format=requirements.txt --output=/tmp/tmpp90woq1r
nox > pip install --constraint=/tmp/tmpp90woq1r coverage[toml] pytest pytest-cov xdoctest
nox > pytest --cov --xdoctest
============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.7.9, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /home/todd/temp/test01
plugins: xdoctest-0.15.0, cov-2.10.1
collected 1 item                                                                                                                                                                                                 

tests/test_console.py .                                                                                                                                                                                    [100%]

----------- coverage: platform linux, python 3.7.9-final-0 -----------
Name                           Stmts   Miss Branch BrPart  Cover   Missing
--------------------------------------------------------------------------
src/project_name/__init__.py       1      0      0      0   100%
src/project_name/console.py        6      0      0      0   100%
--------------------------------------------------------------------------
TOTAL                              7      0      0      0   100%

Required test coverage of 80.0% reached. Total coverage: 100.00%

=============================================================================================== 1 passed in 0.15s ================================================================================================
nox > Session tests-3.7 was successful.
nox > Ran multiple sessions:
nox > * lint-3.7: success
nox > * mypy-3.7: success
nox > * pytype-3.7: success
nox > * tests-3.7: success

That worked fine. Now update the noxfile to use python3.8 instead of python3.7:

noxfile

_versions = ["3.8"]

Re-run the build

~/temp/test01 $ nox
nox > Running session lint-3.8
nox > Creating virtual environment (virtualenv) using python3.8 in .nox/lint-3-8
nox > poetry export --dev --format=requirements.txt --output=/tmp/tmpr4c7u0xa
nox > pip install --constraint=/tmp/tmpr4c7u0xa flake8 flake8-annotations flake8-black flake8-isort flake8-docstrings darglint
nox > Command pip install --constraint=/tmp/tmpr4c7u0xa flake8 flake8-annotations flake8-black flake8-isort flake8-docstrings darglint failed with exit code 1:
Ignoring atomicwrites: markers 'python_version >= "3.5" and python_full_version < "3.0.0" and sys_platform == "win32" and (python_version >= "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.5") or sys_platform == "win32" and python_version >= "3.5" and python_full_version >= "3.4.0" and (python_version >= "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.5")' don't match your environment
Ignoring colorama: markers 'sys_platform == "win32" and python_version >= "3.5" and python_full_version >= "3.5.2" and (python_version >= "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.5")' don't match your environment
Ignoring importlib-metadata: markers 'python_version >= "3.5" and python_full_version < "3.0.0" and python_version < "3.8" and (python_version >= "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.5") or python_version < "3.8" and python_version >= "3.5" and python_full_version >= "3.5.0" and (python_version >= "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.5")' don't match your environment
Ignoring typed-ast: markers 'python_version >= "3.7" and python_version < "3.8" and python_full_version >= "3.6.1" and python_full_version < "4.0.0"' don't match your environment
Ignoring zipp: markers 'python_version >= "3.6" and python_full_version < "3.0.0" and python_version < "3.8" or python_version < "3.8" and python_version >= "3.6" and python_full_version >= "3.5.0"' don't match your environment
Collecting darglint==1.5.5
  Using cached darglint-1.5.5-py3-none-any.whl (113 kB)
Collecting flake8-annotations==2.4.1
  Using cached flake8_annotations-2.4.1-py3-none-any.whl (18 kB)
Collecting flake8-black==0.2.1
  Using cached flake8-black-0.2.1.tar.gz (9.1 kB)
Collecting flake8-docstrings==1.5.0
  Using cached flake8_docstrings-1.5.0-py2.py3-none-any.whl (5.5 kB)
Collecting flake8-isort==4.0.0
  Using cached flake8_isort-4.0.0-py2.py3-none-any.whl (14 kB)
Collecting flake8==3.8.4
  Using cached flake8-3.8.4-py2.py3-none-any.whl (72 kB)
Collecting isort==5.6.4
  Using cached isort-5.6.4-py3-none-any.whl (98 kB)
Collecting mccabe==0.6.1
  Using cached mccabe-0.6.1-py2.py3-none-any.whl (8.6 kB)
Collecting pycodestyle==2.6.0
  Using cached pycodestyle-2.6.0-py2.py3-none-any.whl (41 kB)
Collecting pydocstyle==5.1.1
  Using cached pydocstyle-5.1.1-py3-none-any.whl (35 kB)
Collecting pyflakes==2.2.0
  Using cached pyflakes-2.2.0-py2.py3-none-any.whl (66 kB)
Collecting snowballstemmer==2.0.0
  Using cached snowballstemmer-2.0.0-py2.py3-none-any.whl (97 kB)
Collecting testfixtures==6.15.0
  Using cached testfixtures-6.15.0-py2.py3-none-any.whl (90 kB)
Collecting black==20.8b1
  Using cached black-20.8b1.tar.gz (1.1 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Collecting regex==2020.11.13
  Using cached regex-2020.11.13-cp38-cp38-manylinux2014_x86_64.whl (738 kB)
Collecting typed-ast>=1.4.0
Collecting typing-extensions==3.7.4.3
  Using cached typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
Collecting click==7.1.2
  Using cached click-7.1.2-py2.py3-none-any.whl (82 kB)
Collecting mypy-extensions==0.4.3
  Using cached mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB)
Collecting toml==0.10.2
  Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting pathspec==0.8.1
  Using cached pathspec-0.8.1-py2.py3-none-any.whl (28 kB)
Collecting appdirs==1.4.4
  Using cached appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    typed-ast>=1.4.0 from https://files.pythonhosted.org/packages/77/49/51308e8c529e14bb2399ff6d22998583aa9ae189ec191e6f7cbb4679f7d5/typed_ast-1.4.1-cp38-cp38-manylinux1_x86_64.whl#sha256=498b0f36cc7054c1fead3d7fc59d2150f4d5c6c56ba7fb150c013fbc683a8d2d (from black==20.8b1->-c /tmp/tmpr4c7u0xa (line 16))
nox > Session lint-3.8 failed.
nox > Running session mypy-3.8
nox > Creating virtual environment (virtualenv) using python3.8 in .nox/mypy-3-8
nox > poetry export --dev --format=requirements.txt --output=/tmp/tmp3qvkqlar
nox > pip install --constraint=/tmp/tmp3qvkqlar mypy
nox > Command pip install --constraint=/tmp/tmp3qvkqlar mypy failed with exit code 1:
Ignoring atomicwrites: markers 'python_version >= "3.5" and python_full_version < "3.0.0" and sys_platform == "win32" and (python_version >= "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.5") or sys_platform == "win32" and python_version >= "3.5" and python_full_version >= "3.4.0" and (python_version >= "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.5")' don't match your environment
Ignoring colorama: markers 'sys_platform == "win32" and python_version >= "3.5" and python_full_version >= "3.5.2" and (python_version >= "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.5")' don't match your environment
Ignoring importlib-metadata: markers 'python_version >= "3.5" and python_full_version < "3.0.0" and python_version < "3.8" and (python_version >= "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.5") or python_version < "3.8" and python_version >= "3.5" and python_full_version >= "3.5.0" and (python_version >= "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.5")' don't match your environment
Ignoring typed-ast: markers 'python_version >= "3.7" and python_version < "3.8" and python_full_version >= "3.6.1" and python_full_version < "4.0.0"' don't match your environment
Ignoring zipp: markers 'python_version >= "3.6" and python_full_version < "3.0.0" and python_version < "3.8" or python_version < "3.8" and python_version >= "3.6" and python_full_version >= "3.5.0"' don't match your environment
Collecting mypy==0.790
  Using cached mypy-0.790-cp38-cp38-manylinux1_x86_64.whl (22.0 MB)
Collecting typing-extensions==3.7.4.3
  Using cached typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
Collecting typed-ast<1.5.0,>=1.4.0
Collecting mypy-extensions==0.4.3
  Using cached mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB)
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    typed-ast<1.5.0,>=1.4.0 from https://files.pythonhosted.org/packages/77/49/51308e8c529e14bb2399ff6d22998583aa9ae189ec191e6f7cbb4679f7d5/typed_ast-1.4.1-cp38-cp38-manylinux1_x86_64.whl#sha256=498b0f36cc7054c1fead3d7fc59d2150f4d5c6c56ba7fb150c013fbc683a8d2d (from mypy==0.790->-c /tmp/tmp3qvkqlar (line 148))
nox > Session mypy-3.8 failed.
nox > Running session pytype-3.8
nox > Creating virtual environment (virtualenv) using python3.8 in .nox/pytype-3-8
nox > poetry export --dev --format=requirements.txt --output=/tmp/tmpw6v4m732
nox > pip install --constraint=/tmp/tmpw6v4m732 pytype
nox > Command pip install --constraint=/tmp/tmpw6v4m732 pytype failed with exit code 1:
Ignoring atomicwrites: markers 'python_version >= "3.5" and python_full_version < "3.0.0" and sys_platform == "win32" and (python_version >= "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.5") or sys_platform == "win32" and python_version >= "3.5" and python_full_version >= "3.4.0" and (python_version >= "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.5")' don't match your environment
Ignoring colorama: markers 'sys_platform == "win32" and python_version >= "3.5" and python_full_version >= "3.5.2" and (python_version >= "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.5")' don't match your environment
Ignoring importlib-metadata: markers 'python_version >= "3.5" and python_full_version < "3.0.0" and python_version < "3.8" and (python_version >= "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.5") or python_version < "3.8" and python_version >= "3.5" and python_full_version >= "3.5.0" and (python_version >= "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.5")' don't match your environment
Ignoring typed-ast: markers 'python_version >= "3.7" and python_version < "3.8" and python_full_version >= "3.6.1" and python_full_version < "4.0.0"' don't match your environment
Ignoring zipp: markers 'python_version >= "3.6" and python_full_version < "3.0.0" and python_version < "3.8" or python_version < "3.8" and python_version >= "3.6" and python_full_version >= "3.5.0"' don't match your environment
Collecting pytype==2020.11.12
  Using cached pytype-2020.11.12-cp38-cp38-manylinux2014_x86_64.whl (2.0 MB)
Collecting pyyaml==5.3.1
  Using cached PyYAML-5.3.1.tar.gz (269 kB)
Collecting six==1.15.0
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting ninja==1.10.0.post2
  Using cached ninja-1.10.0.post2-py3-none-manylinux1_x86_64.whl (107 kB)
Collecting attrs==20.3.0
  Using cached attrs-20.3.0-py2.py3-none-any.whl (49 kB)
Collecting typed-ast
Collecting importlab==0.5.1
  Using cached importlab-0.5.1.tar.gz (19 kB)
Collecting networkx==2.5
  Using cached networkx-2.5-py3-none-any.whl (1.6 MB)
Collecting decorator==4.4.2
  Using cached decorator-4.4.2-py2.py3-none-any.whl (9.2 kB)
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    typed-ast from https://files.pythonhosted.org/packages/77/49/51308e8c529e14bb2399ff6d22998583aa9ae189ec191e6f7cbb4679f7d5/typed_ast-1.4.1-cp38-cp38-manylinux1_x86_64.whl#sha256=498b0f36cc7054c1fead3d7fc59d2150f4d5c6c56ba7fb150c013fbc683a8d2d (from pytype==2020.11.12->-c /tmp/tmpw6v4m732 (line 211))
nox > Session pytype-3.8 failed.
nox > Running session tests-3.8
nox > Creating virtual environment (virtualenv) using python3.8 in .nox/tests-3-8
nox > poetry install --no-dev
Installing dependencies from lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing click (7.1.2)

Installing the current project: project_name (0.1.0)
nox > poetry export --dev --format=requirements.txt --output=/tmp/tmp323ni19r
nox > pip install --constraint=/tmp/tmp323ni19r coverage[toml] pytest pytest-cov xdoctest
nox > pytest --cov --xdoctest
============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.8.6, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /home/todd/temp/test01
plugins: xdoctest-0.15.0, cov-2.10.1
collected 1 item                                                                                                                                                                                                 

tests/test_console.py .                                                                                                                                                                                    [100%]

----------- coverage: platform linux, python 3.8.6-final-0 -----------
Name                           Stmts   Miss Branch BrPart  Cover   Missing
--------------------------------------------------------------------------
src/project_name/__init__.py       1      0      0      0   100%
src/project_name/console.py        6      0      0      0   100%
--------------------------------------------------------------------------
TOTAL                              7      0      0      0   100%

Required test coverage of 80.0% reached. Total coverage: 100.00%

=============================================================================================== 1 passed in 0.14s ================================================================================================
nox > Session tests-3.8 was successful.
nox > Ran multiple sessions:
nox > * lint-3.8: failed
nox > * mypy-3.8: failed
nox > * pytype-3.8: failed
nox > * tests-3.8: success

WRAP UP

My current work-around is to either use python3.7, or to not explicit versioning for all the nox sessions, essentially not use the install_with_constraints method.

The text was updated successfully, but these errors were encountered:

Due to issues with pytype not pinning dependency versions, I'm removing the explicit version checking semantics. See also: * google/pytype#731 * cjolowicz/hypermodern-python#174

rchen152 · 2020-11-20T04:59:19Z

Sorry for the slow response! I thought about this a bit, and I don't think we want to pin to exact dependency versions in requirements.txt, simply for reasons of practicality - our dependency management is entirely manual, and I don't want to be stuck on out-of-date versions of our dependencies in a couple years because we forgot about updating requirements.txt.

(This issue did remind me that I should at least copy over the version requirements from setup.cfg, so I'm going to do that.)

ToddG mentioned this issue Nov 16, 2020

Dependency errors when using python 3.9 and 3.8 cjolowicz/hypermodern-python#174

Open

rchen152 closed this as completed Nov 20, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pytype does not pin dependency versions, leading to build errors in --require-hashes mode #731

pytype does not pin dependency versions, leading to build errors in --require-hashes mode #731

ToddG commented Nov 16, 2020 •

edited

Loading

rchen152 commented Nov 20, 2020

pytype does not pin dependency versions, leading to build errors in --require-hashes mode #731

pytype does not pin dependency versions, leading to build errors in --require-hashes mode #731

Comments

ToddG commented Nov 16, 2020 • edited Loading

SUMMARY

ERROR

DETAILS

require-hashes mode

pytype dependencies

REPRO

WRAP UP

rchen152 commented Nov 20, 2020

ToddG commented Nov 16, 2020 •

edited

Loading