Provided requirements.txt does not work with Python 3.11

[eguiraud]

The current requirements.txt does not work for 3.11 because of the specified versions of llvmlite and numba. In general requiring specific versions of each package in the environment without fixing the python version will not work.

Possible solutions could be:

• only specifying the minimum set of packages that are directly required, with looser version requirements -- letting the dependency solver figure the rest out
• switching to conda environments, which let us fix the python version as well. In particular @matthewfeickert proposed using conda lock files for this in the discussion at Add a conda environment file that works for Python 3.11 #141 .

In case it can be useful, here is an environment.yml that works for me (Python 3.10). It could be a starting point for producing a conda lock file. However it has the issue that pip, in order to install the few packages that are not available from conda-forge, ends up uninstalling and reinstalling packages such as coffea and uproot, which is not nice. The resulting environment seems to be stable as far as I can tell though (EDIT: and it includes scikit-learn in order to fix #140 ) .

EDIT:
as of 12/10/2023, the environment file below does not work out of the box anymore: func_adl/servicex and coffea disagree on which awkward version they want, and since func_adl/servicex must be installed via pip while the rest is installed via conda, there is no one resolved that can work out the conflict.

name: agc-py310
channels:
  - conda-forge
dependencies:
  - python=3.10 # no numba package compatible with 3.11 in conda-forge at the moment
  - pip
  - pip:
    - fastjsonschema
    - func-adl
    - func-adl-servicex
    - ipython-genutils
      # these extensions are not available in conda-forge but I'm not sure it's a good idea to install them from pip
      #    - jupyter-console
      #    - jupyter-events
      #    - jupyterlab-pygments
      #    - jupyterlab-widgets
    - make-it-sync
    - mplhep-data
    - msgpack
    - notebook_shim
    - prometheus-client
    - pure-eval
    - qastle
    - servicex
    - stack-data
  - aiohttp
  - aiosignal
  - anyio
  - appnope
  - argon2-cffi
  - argon2-cffi-bindings
  - arrow
  - asttokens
  - async-timeout
  - attrs
  - awkward
  - awkward0
  - backcall
  - backoff
  - beautifulsoup4
  - bleach
  - bokeh
  - boost-histogram
  - cabinetry
  - cachetools
  - certifi
  - cffi
  - charset-normalizer
  - click
  - cloudpickle
  - coffea
  - comm
  - configparser
  - confuse
  - contourpy
  - correctionlib
  - cycler
  - dask
  - debugpy
  - decorator
  - defusedxml
  - distributed
  - executing
  - fonttools
  - fqdn
  - frozenlist
  - fsspec
  - google-auth
  - HeapDict
  - hist
  - histoprint
  - idna
  - iminuit
  - importlib-metadata
  - ipykernel
  - ipython
  - ipywidgets
  - isoduration
  - jedi
  - Jinja2
  - jsonpatch
  - jsonpointer
  - jsonschema
  - jupyter
  - jupyterlab
  - jupyter_client
  - jupyter_core
  - jupyter_server
  - jupyter_server_terminals
  - jupytext
  - kiwisolver
  - lark-parser
  - llvmlite
  - locket
  - lz4
  - markdown-it-py
  - MarkupSafe
  - matplotlib
  - matplotlib-inline
  - mdurl
  - minio
  - mistune
  - mplhep
  - multidict
  - nbclassic
  - nbclient
  - nbconvert
  - nbformat
  - nest-asyncio
  - notebook
  - numba
  - numpy<1.22 # for the coffea version that will be installed by pip
  - packaging
  - pandas
  - pandocfilters
  - parso
  - partd
  - particle
  - pexpect
  - pickleshare
  - Pillow
  - platformdirs
  - prompt-toolkit
  - psutil
  - ptyprocess
  - pyarrow
  - pyasn1
  - pyasn1-modules
  - pycparser
  - pydantic
  - Pygments
  - pyhf
  - pyparsing
  - pyrsistent
  - python-dateutil
  - python-json-logger
  - pytz
  - PyYAML
  - pyzmq
  - qtconsole
  - QtPy
  - rfc3339-validator
  - rfc3986-validator
  - rich
  - rsa
  - scipy
  - Send2Trash
  - six
  - sniffio
  - sortedcontainers
  - soupsieve
  - tabulate
  - tblib
  - terminado
  - tinycss2
  - toml
  - toolz
  - tornado
  - tqdm
  - traitlets
  - tritonclient
  - typing_extensions
  - uhi
  - uproot
  - uproot3
  - uproot3-methods
  - uri-template
  - urllib3
  - vector
  - wcwidth
  - webcolors
  - webencodings
  - websocket-client
  - widgetsnbextension
  - xgboost
  - yarl
  - zict
  - zipp
  - zstandard
  - scikit-learn

[agoose77]

If the goal is to build an "application" i.e. this will be the end consumer of these libraries, then any Python locking dependency manager would also suffice, e.g. pdm, pip-tools, or even poetry. PDM would be my recommendation, and its lockfile will (IIRC) solve across your Python versions supported in the python-version field. The caveat is that libraries shouldn't use upper caps here, but for applications I believe it's slightly better.

Of course, if you need conda-forge packages too, then the mixing problem arises.