Solve feast conflict dependencies for [gcp]

[MattDelac]

Expected Behavior

pip-compile should run without error and result in a nice lock files of the libraries to install

Current Behavior

pip-compile will find conflict in the current feast[gcp] dependencies. Did not try for [aws] or [redis]

Steps to reproduce

echo "-e  sdk/python[gcp]" > sdk/python/requirements.txt
pip-compile --dry-run sdk/python/requirements.txt

Specifications

• Version:
• Platform:
• Subsystem:

Possible Solution

[ysk24ok]

Although #1955 is merged, this problem still remains in v0.14.1. Could any maintainer reopen this issue?

$ cat Pipfile
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
feast = {extras = ["gcp"], version = "0.14.1"}

[dev-packages]

[requires]
python_version = "3.8"
$ pipenv install
...
ERROR: Could not find a version that matches google-cloud-core<3.0.0dev,<3.0dev,==1.4.*,>=1.4.0,>=1.4.1,>=1.6.0 (from feast[gcp]==0.14.1->-r /tmp/pipenveozxhuo9requirements/pipenv-ai_14wht-constraints.txt (line 2))
Tried: 0.20.0, 0.20.0, 0.21.0, 0.21.0, 0.22.0, 0.22.0, 0.22.1, 0.22.1, 0.23.0, 0.23.0, 0.23.1, 0.23.1, 0.24.0, 0.24.0, 0.24.1, 0.24.1, 0.25.0, 0.25.0, 0.26.0, 0.26.0, 0.27.0, 0.27.0, 0.27.1, 0.27.1, 0.28.0, 0.28.0, 0.28.1, 0.28.1, 0.29.0, 0.29.0, 0.29.1, 0.29.1, 1.0.0, 1.0.0, 1.0.1, 1.0.1, 1.0.2, 1.0.2, 1.0.3, 1.0.3, 1.1.0, 1.1.0, 1.2.0, 1.2.0, 1.3.0, 1.3.0, 1.4.0, 1.4.0, 1.4.1, 1.4.1, 1.4.2, 1.4.2, 1.4.3, 1.4.3, 1.4.4, 1.4.4, 1.5.0, 1.5.0, 1.6.0, 1.6.0, 1.7.0, 1.7.0, 1.7.1, 1.7.1, 1.7.2, 1.7.2, 2.0.0, 2.0.0, 2.1.0, 2.1.0
Skipped pre-versions: 1.4.2rc1, 1.4.2rc1, 1.4.2rc2, 1.4.2rc2, 2.0.0b1, 2.0.0b1
There are incompatible versions in the resolved dependencies:
  google-cloud-core<3.0.0dev,>=1.4.0 (from google-cloud-datastore==2.3.0->feast[gcp]==0.14.1->-r /tmp/pipenveozxhuo9requirements/pipenv-ai_14wht-constraints.txt (line 2))
  google-cloud-core<3.0.0dev,>=1.4.1 (from google-cloud-bigquery==2.29.0->feast[gcp]==0.14.1->-r /tmp/pipenveozxhuo9requirements/pipenv-ai_14wht-constraints.txt (line 2))
  google-cloud-core<3.0dev,>=1.6.0 (from google-cloud-storage==1.42.3->feast[gcp]==0.14.1->-r /tmp/pipenveozxhuo9requirements/pipenv-ai_14wht-constraints.txt (line 2))
  google-cloud-core==1.4.* (from feast[gcp]==0.14.1->-r /tmp/pipenveozxhuo9requirements/pipenv-ai_14wht-constraints.txt (line 2))

It seems google-cloud-storage==1.42.3 is tried to be installed. That means <1.41 version specification doesn't work.

[MattDelac]

Indeed I also have some issues with protobuf

[ysk24ok]

#1992 will solve the problem written in #1912 (comment).

I've found out the reason why the problem written in #1912 (comment) still happens.
In the v0.14.1 tarball uploaded to PyPI , setup.py still shows google-cloud-storage>=1.34.*" which doesn't include <1.41 specification added by #1955. That means #1955 is not included in v1.14.1 ...

[ysk24ok]

Users who use pipenv or pip-compile and suffer from installing feast[gcp] can use poetry.

$ cat pyproject.toml
[tool.poetry]
name = "tmp"
version = "0.1.0"
description = ""
authors = ["Yusuke Nishioka <yusuke.nishioka.0713@gmail.com>"]

[tool.poetry.dependencies]
python = ">=3.8,<3.10"
feast = {extras = ["gcp"], version = "^0.14.1"}

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
$ poetry install
Installing dependencies from lock file
Warning: The lock file is not up to date with the latest changes in pyproject.toml. You may be getting outdated dependencies. Run update to update them.

Package operations: 56 installs, 2 updates, 1 removal

  • Removing platformdirs (2.4.0)
  • Installing pyasn1 (0.4.8)
  • Installing cachetools (4.2.4)
  • Installing protobuf (3.17.3)
  • Installing pyasn1-modules (0.2.8)
  • Updating pyparsing (3.0.4 -> 2.4.7)
  • Installing rsa (4.7.2)
  • Installing google-auth (1.35.0)
  • Installing googleapis-common-protos (1.52.0)
  • Installing mypy-extensions (0.4.3)
  • Updating packaging (20.9 -> 21.2)
  • Installing pytz (2021.3)
  • Installing sniffio (1.2.0)
  • Installing typing-extensions (3.10.0.2)
  • Installing anyio (3.3.4)
  • Installing google-api-core (1.31.3)
  • Installing google-crc32c (1.3.0)
  • Installing grpcio (1.41.1)
  • Installing numpy (1.21.3)
  • Installing python-dateutil (2.8.2)
  • Installing pyyaml (6.0)
  • Installing typing-inspect (0.7.1)
  • Installing asgiref (3.4.1)
  • Installing attrs (21.2.0)
  • Installing click (7.1.2)
  • Installing fastavro (1.4.7)
  • Installing google-cloud-core (1.4.4)
  • Installing google-resumable-media (1.3.3)
  • Installing h11 (0.12.0)
  • Installing httptools (0.2.0)
  • Installing libcst (0.3.21)
  • Installing markupsafe (2.0.1)
  • Installing pandas (1.3.4)
  • Installing proto-plus (1.19.6)
  • Installing pydantic (1.8.2)
  • Installing pyrsistent (0.18.0)
  • Installing python-dotenv (0.19.1)
  • Installing starlette (0.16.0)
  • Installing uvloop (0.16.0)
  • Installing watchgod (0.7)
  • Installing websockets (10.0)
  • Installing colorama (0.4.4)
  • Installing dill (0.3.4)
  • Installing fastapi (0.70.0)
  • Installing google-cloud-bigquery (2.29.0)
  • Installing google-cloud-bigquery-storage (2.9.1)
  • Installing google-cloud-datastore (2.3.0)
  • Installing google-cloud-storage (1.40.0)
  • Installing jinja2 (3.0.2)
  • Installing jsonschema (4.1.2)
  • Installing mmh3 (3.0.0)
  • Installing pandavro (1.5.2)
  • Installing pyarrow (6.0.0)
  • Installing tabulate (0.8.9)
  • Installing tenacity (8.0.1)
  • Installing toml (0.10.2)
  • Installing tqdm (4.62.3)
  • Installing uvicorn (0.15.0)
  • Installing feast (0.14.1)

google-cloud-storage==1.40.0 and proto-plus==1.19.6 are installed, which #1955 and #1992 aimed respectively.
Dependency resolver of poetry is quite smart.

[ysk24ok]

Now we can install feast[gcp]==0.15.0 via pipenv (not confirmed with pip-compile).
Is it a good time to close this issue?

[MattDelac]

Now we can install feast[gcp]==0.15.0 via pipenv

But does that mean that there is still no conflict dependencies when trying things like pip-compile --dry-run ?

[adchia]

Quick followup on whether this is temporarily resolved or not?

[ysk24ok]

But does that mean that there is still no conflict dependencies when trying things like pip-compile --dry-run ?

Yes, I guess so.

$ cat requirements.in
feast[gcp]
$ pip-compile --dry-run requirements.in
...
feast[gcp]==0.15.1
    # via -r requirements.in
...
Dry-run, so nothing updated.

Quick followup on whether this is temporarily resolved or not?

Yes, It seems solved temporarily.
But if a package which feast depends on updates its dependencies (e.g. using newer versions of the dependency) and it conflicts with that of feast, this problem happens again.
To fix this completely. we need a lock mechanism provided by e.g. pipenv, pip-compile or poetry. which is proposed in #1993.

[ysk24ok]

Now that #2093 is merged, requirements.txt created by pip-compile is used to install dependencies.
But I've noticed that this isn't enough because users who run pipenv or pip-compile to install Feast install dependencies based on setup.py, not requirements.txt.
If requirements.txt gets old, they might encounter the same problem.
To solve this, maintainers (or CI?) have to update requirements.txt regularly.