First install of a large package takes HOURS.

[buriy]

So it's not usable for typical real projects.

Please speed it up.

Try yourself for "poetry new test; cd test; poetry add -vvv news-please".

Also, it feels like sometimes it does N^2 repetitions of the same kind of calculations (with each pair of colorama and awscli versions, for example ).

: 0: Creating possibility state for awscli (>= 1.11.117.0) (33 remaining)
: 65: Attempting to activate [awscli-1.11.149]
: 65: Activated awscli at [awscli-1.11.149]
: 0: Getting dependencies for awscli-1.11.149
: 65: Requiring nested dependencies (botocore (== 1.7.7.0), colorama (<= 0.3.7.0, >= 0.2.5.0), docutils (>= 0.10.0.0), rsa (<= 3.5.0.0, >= 3.1.2.0), s3transfer (< 0.2.0.0, >= 0.1.9.0), PyYAML (<= 3.12.0.0, >= 3.10.0.0), argparse (>= 1.1.0.0))
PyPI: 11 packages found for colorama <= 0.3.7.0, >= 0.2.5.0
: 0: Creating possibility state for colorama (<= 0.3.7.0, >= 0.2.5.0) (11 remaining)
: 66: Attempting to activate [colorama-0.3.7]
: 66: Found existing spec ([colorama-0.3.9])
: 66: Unsatisfied by existing spec ([colorama-0.3.9])
: 66: Unwinding for conflict: colorama (<= 0.3.7.0, >= 0.2.5.0) to 65
: 0: Creating possibility state for colorama (<= 0.3.7.0, >= 0.2.5.0) (10 remaining)
: 66: Attempting to activate [colorama-0.3.6]
: 66: Found existing spec ([colorama-0.3.9])
: 66: Unsatisfied by existing spec ([colorama-0.3.9])
: 66: Unwinding for conflict: colorama (<= 0.3.7.0, >= 0.2.5.0) to 65
: 0: Creating possibility state for colorama (<= 0.3.7.0, >= 0.2.5.0) (9 remaining)
: 66: Attempting to activate [colorama-0.3.5]
: 66: Found existing spec ([colorama-0.3.9])
: 66: Unsatisfied by existing spec ([colorama-0.3.9])
: 66: Unwinding for conflict: colorama (<= 0.3.7.0, >= 0.2.5.0) to 65
: 0: Creating possibility state for colorama (<= 0.3.7.0, >= 0.2.5.0) (8 remaining)
: 66: Attempting to activate [colorama-0.3.4]
: 66: Found existing spec ([colorama-0.3.9])
: 66: Unsatisfied by existing spec ([colorama-0.3.9])
: 66: Unwinding for conflict: colorama (<= 0.3.7.0, >= 0.2.5.0) to 65

We have a lot of versions for awscli :
...
: 0: Getting dependencies for awscli-1.11.117
: 0: Getting dependencies for awscli-1.11.118
: 0: Getting dependencies for awscli-1.11.119
: 0: Getting dependencies for awscli-1.11.120
: 0: Getting dependencies for awscli-1.11.121
: 0: Getting dependencies for awscli-1.11.122
: 0: Getting dependencies for awscli-1.11.123
: 0: Getting dependencies for awscli-1.11.124
: 0: Getting dependencies for awscli-1.11.125
: 0: Getting dependencies for awscli-1.11.126
: 0: Getting dependencies for awscli-1.11.127
: 0: Getting dependencies for awscli-1.11.128
: 0: Getting dependencies for awscli-1.11.129
: 0: Getting dependencies for awscli-1.11.130
: 0: Getting dependencies for awscli-1.11.131
: 0: Getting dependencies for awscli-1.11.132
: 0: Getting dependencies for awscli-1.11.133
: 0: Getting dependencies for awscli-1.11.134
: 0: Getting dependencies for awscli-1.11.135
: 0: Getting dependencies for awscli-1.11.136
: 0: Getting dependencies for awscli-1.11.137
: 0: Getting dependencies for awscli-1.11.138
: 0: Getting dependencies for awscli-1.11.139
: 0: Getting dependencies for awscli-1.11.140
: 0: Getting dependencies for awscli-1.11.141
: 0: Getting dependencies for awscli-1.11.142
: 0: Getting dependencies for awscli-1.11.143
: 0: Getting dependencies for awscli-1.11.144
: 0: Getting dependencies for awscli-1.11.145
: 0: Getting dependencies for awscli-1.11.146
: 0: Getting dependencies for awscli-1.11.147
: 0: Getting dependencies for awscli-1.11.148
: 0: Getting dependencies for awscli-1.11.149
: 0: Getting dependencies for awscli-1.11.150
: 0: Getting dependencies for awscli-1.11.151
: 0: Getting dependencies for awscli-1.11.152
: 0: Getting dependencies for awscli-1.11.153
: 0: Getting dependencies for awscli-1.11.154
: 0: Getting dependencies for awscli-1.11.155
: 0: Getting dependencies for awscli-1.11.156
: 0: Getting dependencies for awscli-1.11.157
: 0: Getting dependencies for awscli-1.11.158
: 0: Getting dependencies for awscli-1.11.159
: 0: Getting dependencies for awscli-1.11.160
: 0: Getting dependencies for awscli-1.11.161
: 0: Getting dependencies for awscli-1.11.162
: 0: Getting dependencies for awscli-1.11.163
: 0: Getting dependencies for awscli-1.11.164
: 0: Getting dependencies for awscli-1.11.165
: 0: Getting dependencies for awscli-1.11.166
: 0: Getting dependencies for awscli-1.11.167
: 0: Getting dependencies for awscli-1.11.168
: 0: Getting dependencies for awscli-1.11.169
: 0: Getting dependencies for awscli-1.11.170
: 0: Getting dependencies for awscli-1.11.171
: 0: Getting dependencies for awscli-1.11.172
: 0: Getting dependencies for awscli-1.11.173
: 0: Getting dependencies for awscli-1.11.174
: 0: Getting dependencies for awscli-1.11.175
: 0: Getting dependencies for awscli-1.11.176
: 0: Getting dependencies for awscli-1.11.177
: 0: Getting dependencies for awscli-1.11.178
: 0: Getting dependencies for awscli-1.11.179
: 0: Getting dependencies for awscli-1.11.180
: 0: Getting dependencies for awscli-1.11.181
: 0: Getting dependencies for awscli-1.11.182
: 0: Getting dependencies for awscli-1.11.183
: 0: Getting dependencies for awscli-1.11.184
: 0: Getting dependencies for awscli-1.11.185
: 0: Getting dependencies for awscli-1.11.186
: 0: Getting dependencies for awscli-1.11.187
: 0: Getting dependencies for awscli-1.11.188
: 0: Getting dependencies for awscli-1.11.189
: 0: Getting dependencies for awscli-1.11.190
: 0: Getting dependencies for awscli-1.12.0
: 0: Getting dependencies for awscli-1.12.1
: 0: Getting dependencies for awscli-1.12.2
...

[sdispater]

Please speed it up.

Please don't do that. This is neither helpful nor respectful. I know there isn't a code of conduct here but I ask you to be considerate for the time and effort I put into this project. Dependency resolution is hard and even more so in Python where the packaging is a mess.

If you are not happy with the time it takes, you can always consider help the project by contributing. It would be greatly appreciated.

Now, regarding the issue you are describing, poetry uses a dependency resolver with a backtracking algorithm. When there is a conflict, it needs to go back and flatten the dependency graph to be sure to not go back to a state that will cause a conflict. In you case there is a lot of conflict due to pinned dependencies. That's why it takes so much time.

I don't have a solution for now, unfortunately, but if you feel like tackling this issue I'd gladly review a PR.

[buriy]

Surely I can write code to fix that. I thought you'd never ask :)
Naive backtracking instead of a kind of approximate constraint solving doesn't look like a good way to do it efficiently.
Observing a tree with 100 dependencies shouldn't take hours.
But, even without that:
Specifically, there are two places that caught my attention and could be fixed fast:

1. Branch results caching would greatly speed it up (e.g. if we know that all versions of package A need package B of the versions from 1.2.1 to 2.5.3, then we can save this information after backtracking, as the new requirements for the package A).
2. It seems that tree state check takes about 1 second (!) and stores data to disk. Traversing 200 versions of package A * 15 versions of package B is what takes 3000 seconds then, that is one hour.

Could you maybe give me some hints on how to approach this in the code?

[sdispater]

@buriy Just so you know it is not currently possible to install news-please since there is a conflict that is not possible to resolve:

[SolverProblemError]
Unable to satisfy the following requirements:
- "beautifulsoup4 (== 4.3.2.0)" required by "newspaper-0.0.9.2"
- "beautifulsoup4 (>= 4.5.1.0)" required by "news-please-1.0.0.post1"

[sdispater]

Also, I am currently rewriting the resolver from scratch using a new algorithm to hopefully improve things.

[buriy]

@sdispater I believe this SolverProblemError is misleading again.
Have you tried it for python 2 or python 3?
newspaper-0.0.9.2 is not the latest version, and it's not needed for python3 .
For python3, one needs newspaper3k instead.
pip doesn't attempt to install "newspaper" package, it's optional dependency.
Compare with the versions installed by pip3 :
ago==0.0.92
asn1crypto==0.24.0
attrs==18.1.0
Automat==0.6.0
awscli==1.15.14
beautifulsoup4==4.6.0
botocore==1.10.14
CacheControl==0.12.4
cachy==0.1.1
certifi==2018.4.16
cffi==1.11.5
chardet==3.0.4
cleo==0.6.5
click==6.7
colorama==0.3.7
constantly==15.1.0
cryptography==2.2.2
cssselect==1.0.3
docutils==0.14
dotmap==1.2.20
elasticsearch==6.2.0
feedfinder2==0.0.4
feedparser==5.2.1
first==2.0.1
hjson==3.0.1
hurry.filesize==0.9
hyperlink==18.0.0
idna==2.6
incremental==17.5.0
jieba3k==0.35.1
jmespath==0.9.3
jsonschema==2.6.0
langdetect==1.0.7
lockfile==0.12.2
lxml==4.2.1
msgpack-python==0.5.6
news-please==1.2.36
newspaper3k==0.2.6
nltk==3.3
parsel==1.4.0
pastel==0.1.0
Pillow==5.1.0
pip-tools==2.0.2
pkginfo==1.4.2
plac==0.9.6
pyasn1==0.4.2
pyasn1-modules==0.2.1
pycparser==2.18
PyDispatcher==2.0.5
pylev==1.3.0
PyMySQL==0.8.1
pyOpenSSL==17.5.0
pyparsing==2.2.0
pyrsistent==0.14.2
python-dateutil==2.7.2
PyYAML==3.12
queuelib==1.5.0
readability-lxml==0.7
requests==2.18.4
requests-file==1.4.3
requests-toolbelt==0.8.0
rsa==3.4.2
s3transfer==0.1.13
Scrapy==1.5.0
service-identity==17.0.0
six==1.11.0
tldextract==2.2.0
toml==0.9.4
Twisted==18.4.0
urllib3==1.22
w3lib==1.19.0
warcio==1.5.1
zipfile36==0.1.3
zope.interface==4.5.0

[buriy]

@sdispater also see #96 for a detailed explanation of what's wrong with SolverProblemError.

[sdispater]

@buriy So, like I said I am working on a new resolver and I am happy to tell you that the successful resolution of news-please on a cold cache takes less than a minute and around 1 second on a warm cache.

[sdispater]

A new prerelease (0.10.0a0) has been published. It contains the new dependency resolver which should speed up things like I said above. So you can give it a try if you want :-)

[buriy]

@sdispater I'll definitely try, thanks a lot!

[buriy]

@sdispater 0.10.0a0 version resolution now works very well for this and another project.
It said "Version solving took 27.459 seconds." for news-please, which is great!

Now the project works great (when it does), but has a lot of different small quirks:

1. if a project from github has setup.py which depends on another project, "poetry install" will fail to resolve them ( source-compiled package installation order and dependency resolution #117 )
2. add/install inconsistency ( BUG: When poetry add failed, package is listed as added but is not installed. Also add proper handling for multiple packages install in "poetry add": when one package is already added, others are not getting added. #98 ). You run "poetry add", it fails at install, you run "poetry add" again and it tells that package is already added. You need to type "poetry install".
3. For this specific install, .venv is python3.6 , when installing hurry-filesize (0.9), gets a fatal error ( Installing hurry-filesize (0.9) fails with VenvCommandError: No matching distribution found #118 )
4. If you have no local .venv folder and system python3.5 is used, you get another fatal error ( Poetry doesn't work If you have no local .venv folder but have pygame==1.9.1release in system python folder #119 )

[blfpd]

@buriy Would you consider the issue “First install of a large package takes HOURS” as resolved?

[buriy]

Hi @batisteo,
Let's close it when alpha version solving it will meet production.

[sdispater]

Release 0.10.0 is out!

[mm-matthias]

Related to #5896.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.