Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Test Python 3.8/3.11 + backup multiprocessing to get geodata #178

Closed
wants to merge 70 commits into from
Closed

Test Python 3.8/3.11 + backup multiprocessing to get geodata #178

wants to merge 70 commits into from

Conversation

tfardet
Copy link
Collaborator

@tfardet tfardet commented Nov 19, 2023

Fixes #172 and #175

@hadrilec are these the main changes you had included in #162? If not, feel free to include whatever is missing.

tfardet and others added 30 commits July 18, 2023 13:13
@tfardet
Use configuration dictionary
b673ada 
- credentials are stored in the dict
- proxies are also taken from there

Removed all writing to environment except for the token.
Cleaned up bare excepts and imports.
@tfardet
Correct environment variables for credentials
4e86896
@tgrandje
Remove pynsee_print_url from env
49bd3a9
@tfardet
Fix get_location, add token logging + alt text
3cee35d
@hadrilec
bugfix
871df17
@tfardet
Move idbank and file list to settings, improved token management
c735cc4
@tfardet
Use configuration dictionary
f7c37f9 
- credentials are stored in the dict
- proxies are also taken from there

Removed all writing to environment except for the token.
Cleaned up bare excepts and imports.
@tfardet
Correct environment variables for credentials
73f6f84
@tgrandje @tfardet
Remove pynsee_print_url from env
15f0631
@tfardet
Move idbank and file list to settings, improved token management
9d455be
@tfardet
Merged remote
f01f66c
@tfardet
Merge branch 'master' into settings
6ed4f77
@tfardet
restore old imports
22d0f23
@tfardet
Merge remote-tracking branch 'refs/remotes/origin/settings' into sett…
9d44564 
…ings
@tfardet
Correct missing imports
a382201
@tfardet
Final correction of imports
3fe8df0
@hadrilec
bug fix geofrdf import
b1d3f5c
@hadrilec
bug fix - ci/cd - trigger pkg tests on pr head commit
a629c9a
@tfardet
Merge branch 'master' into settings
d1d0fb8
Fix test imports + update them
a26877f
Fix tests except number 2
f3c86e7
Fix logging
7a32ef2
cleanup
156b830
Move to platformdirs and add config file
aec3a9d
Merge master, fix test decorator
cb0fd7a
Fix tests
0f567dd
Insee key and secret must be passed simultaneously
9fbeae2
Restore order for token: env, then config, then request from insee cred
41cc4b9
Fix _get_credentials
3cf4a1b
Finalize credentials logic
2b9b436
hadrilec and others added 16 commits November 18, 2023 13:42
@hadrilec @tfardet
Update pynsee/utils/_clean_insee_folder.py
147d025 
Co-authored-by: tfardet <79037344+tfardet@users.noreply.github.com>
@hadrilec @tfardet
Update pynsee/utils/_create_insee_folder.py
e52ade6 
Co-authored-by: tfardet <79037344+tfardet@users.noreply.github.com>
@hadrilec @tfardet
Update pynsee/metadata/get_legal_entity.py
7b614d0 
Co-authored-by: tfardet <79037344+tfardet@users.noreply.github.com>
@hadrilec
cleaning tests
8117f61
@hadrilec
user agent used to try to avoid connection error
fb413cb
@hadrilec
Merge branch 'settings' of https://github.com/tfardet/pynsee into set…
269154a 
…tings
@hadrilec
use _request_insee in token config func
2366eb8
@hadrilec
catch error 429 and sleep
7763812
@hadrilec
init_conn - catch error 429
72e13ac
@hadrilec
share connections when testing the token
0afbd0f
Cleanup code, move to return early pattern
977b67d
@hadrilec
fix empl example
b830351
@hadrilec
multiprocessing backup solution if it fails
845af1e
Fix bare exception
3c2e92d
@hadrilec
tests only on py 3.8+3.11, geopandas version fix
35761cb
normalize tests
8cfb62e
@tfardet tfardet added the bugfix Fixes a bug label Nov 19, 2023
@tfardet tfardet requested a review from hadrilec November 19, 2023 17:27
@tfardet
Copy link
Collaborator Author

tfardet commented Nov 19, 2023

@hadrilec Well, it seems we now know where the 4h-long tests come from ^^"

@tfardet
Copy link
Collaborator Author

tfardet commented Nov 20, 2023

@hadrilec I won't have time to look at this for the moment, so if you can have a look, it would be worth checking what makes the multiprocessing fail.
I also wonder whether this is the best way to handle this and if we shouldn't rather move to something like this, might be just as fast (faster?) and more reliable.

@hadrilec
Copy link
Contributor

Thanks for the link, I will have a look. I agree async should be the way to go.

@hadrilec
Copy link
Contributor

I did a mistake by pushing commits from the other PR, my bad, I will clean this up

@tfardet
Copy link
Collaborator Author

tfardet commented Dec 19, 2023

@hadrilec do you know when you'd have time to finish this part?

@hadrilec
Copy link
Contributor

hi @tfardet, I plan to wrap this up before the 1st of January. In any case, a new release of the package must be created by next year because IGN and INSEE/SIREN APIs are changing.

@tfardet
Copy link
Collaborator Author

tfardet commented Dec 19, 2023 •
edited
Loading

Great, let me know if you want me to force push or create another branch with my local version of fix-tests in case cleaning up turns up to be complicated

EDIT: actually, just in case, https://github.com/tfardet/pynsee/tree/fix-test-backup

@hadrilec
Copy link
Contributor

nice, I will start from there

@tfardet tfardet closed this Dec 27, 2023
@tfardet tfardet deleted the fix-tests branch August 8, 2024 14:40
# for free to join this conversation on GitHub. Already have an account? # to comment
Reviewers

@hadrilec hadrilec Awaiting requested review from hadrilec

Assignees
No one assigned
Labels
bugfix Fixes a bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Test against relevant python versions
3 participants
@tfardet @hadrilec @tgrandje