Release Delphi Epidata 4.1.0 #1178

github-actions · 2023-05-24T15:10:30Z

Releasing Delphi Epidata 4.1.0.

API keys, local PR edition #1115

chore: sync main->dev

lol, good luck reading this list... * very basic admin interface * list all known roles * remove log statement * fixup tests * add record flag to user * basic record based on flag logging * rename column to tracking * add webhook endpoint for google forms and simplify table * tests for simplified api_user * prepare for sending mails with mailgun * disable security check for test setup * fix tests * add registered flag * request a simple api key form with recaptcha check * implement rate limiting * registered flag not saved * mark all tests no rate limit * add link to registration form within api key form * elevate sqlalchemy log level to ERROR for test purpose * add structlog to requirements * add structured logging module * implement api key scrubbing logs and getter functions for User class * minor commenting change * change logging to kward format and add a get function for API key * move mask_apikey function out of User class * minor change in variable name * Fixed test * added missing import * Fixed _is_public_route method. Added URL_PREFIX to list of routes * Added email column to User table * Fixed tests * Added Dockerfile for Redis * Made email optional * Added email address validaiton * Fixed API key creation via /create_key endpoint * Fixed /register endpoint after google script refactoring * Fixed registration * Reverted some changes in _config.py * Reverted changes in .env.example * Code refactoring * Removed unused variables * Code cleaning * Removed unused variables * Removed default redis configuration file (it is not needed for now) * Moved BASE_URL and auth to setUP, formatted code * Moved BASE_URL and auth to setUP(), formatted code * Revert "Moved BASE_URL and auth to setUP(), formatted code" This reverts commit b63faa3. * Formatted code & removed unused imports. Defined BASE_URL inside Epidata class and removed all BASE_URL variable definition in UnitTests. Removed version for `newrelic` in requirements.txt to use the latest one. * Created staticmethod _make_rqeuest() and reused it in test_caching() method. * Returned BASE_URL and auth back, because if we use default BASE_URL from Epidata class, tests will send requests to the wrong address and fail * Completely removed patch_flask_config() function as it is useless. There is no need to patch app.config is this case, because we are using that variables directly from _config.py * Removed duplicated _logger.py file (this is exact copy of src/acquisition/covidcast/logger.py). Changed _security.py to use that logger. * Fixed test * Removed unused method * Added missing changes in Dockerfiles for copying logger * Fixed misspelled filename * Added missing variable in CSVPrinter * Formatted code * Removed tests for admin endpoint for now * Moved config variables from _security to _config. Removed duplicated method in _security.py by creating parent class for DBUser and APIUser classes. * Added missing changes * Replaced db.execute delete and update with built-it sqlalchemy functions. * Removed email field. Edited google form link to prefill tracking consent option as Yes * Removed api_user ddl * Changed user roles to use m2m table instead of string in api_user table * Added new variable for api key expiration date * Updated admin endpoint. Removed email from user details and added creation_date, expiration_date and last_api_access_date fields * Updated integration tests * Updated endpoints to register user role on endpoint load * Removed user_role FK in api_user table. Consolidated DBUser and APIUser into 1 class -> APIUser. * Updated according to new structure * Removed API_KEY_EXPIRE_AFTER variable * Removed creation_date, expiration_date and last_api_access_date from admin UI as we don't have them anymore * Added roles registraion on endpoints load * Removed role registration statement * Fixed integration tests. Actually reverted latest changes as api_user table can be truncated again * Added new ddls * Fixed wrong import * Fixed issue after resolving conflicts * Moved admin endpoint to another file into endpoints folder * Removed from _security.py all unnecessary functions * Changed roles registration process. Now we have only necessary roles. Prev we had roles for all the endpoints where role.name == endpoint.name which added redundant roles to the database. Also changed register_user_role method to use raw sql to insert the role. * Added IF NOT EXISTS to table creation statement * Added missing import * Moved admin models to the separate file. Now we have only 1 User class which containts all necessary methods to work with User model. * Added declarative_base and session objects to the _db.py file. To have all db related stuff into one file * Updated tests to use statement in order to clean api_user table after each test run. This is done because of returning relationship into User model. * Reverted formatting * Removed copying of _logger.py * Fixed auth props * Added missing requirements * Added authentication params * Removed unused imports * Added new columns to api_user table * Added placeholder to handle api key last usage update * Added Redis host variable * Uncommented @after_reqeust function to update last API key usage in Redis. * Build image from api-keys branch * Updates TODO with issue * Added Redis to ci.yaml. Added several more env variables for delphi_web_epidata container * Added Redis to the Makefile * Changed ratelimit handling * Added default admin password, changed rate limit * Updated ci.yaml * Changed wrong redis host name * Added -d to redis container run command * Removed tailing colon * Fixed get_multiples_count method * Temporary changed port * Added check container status step + check delphi_web_epidata logs. Muted integration tests for now * Removed --rm flag from delphi_web_epidata run command * Added check db logs step to ci.yaml * Added delphi_redis to clean up step * Reverted test changes * Moved docker run delphi_web_epidata below wait command. * Added name to run delphi_web_epidata step * Fixed issue with parsing allowed singals, removed unnecessary if/else block * Added Redis auth options (set default password) * adding dashboard signals module, plus requirements cleanup * updated _security.py:check_signals_allowlist() to use new DashboardSignals * debugging w/ headers * better ip address logging and accounting * Added admin folder to setup.cfg * Upgraded Flask-Limiter version * Moved Flask-Limiter code to a separate file. Made limit to be host based(global) instead of endpoint based. * Little refactoring, replaced MissingAPIKey exception by Unauthorized * Fixed limits for allowed signals * Added necessary changes * Removed redundant user variable * Update dev/local/Makefile wrt redis log file Co-authored-by: Katie Mazaitis <krivard@cs.cmu.edu> * reset dev/docker/python/Dockerfile * remove unused import in src/server/main.py * alpha-order argument keys in get_multiples_count() * Removed API Key registration form * Removed recaptcha secrets from _config.py * Added 'stop container if running' and option to redis entry * Moved Flask-Limiter belov Flask to keep everyting sorted properly * Added missing newline * Added proper Epidata.auth handling for aiohttp ClientSession * Added missing request arg for require_all() * Merged api-keys related ddls into 1 file, added constraints * Removed tracking and registered fields * Removed usage of removed fields (tracking, registered). Added logging. * Adjusted flask-limiter with comms * Fixed tests * Added templates folder * Removed _db.py and moved all needed code to the _common.py * Exempted public routes from update_key_last_time_used function * Removed redundant tracking, registered mentions. Added email field to the admin interface * Imported engine from _common.py instead of _db.py which was removed * Removed leftovers of the old code * Add auth support to delphi-epidata.R Includes support for testing the R client: * R added to python image * Integration test file that does toy queries of all unrestricted endpoints * `r-test` target to run the integration test file Seems to work regardless of whether you run r-test before or after python tests, even though a database reset is not included in the integration test. r-test is not included in CI. * Bring auth in delphi-epidata.py in line with current policy Also adds some nicer error handling when json parsing fails * Embetter json decoding error message Co-authored-by: melange396 <george.haff@gmail.com> * Make user agent comply with spec; explain inactive R client endpoint tests * Removed api_analytics table * Added 'window' param handling in multiples. Removed hostname from limit string * Added TODO to remove api keys warning messages * Added issue handling when API Key is provided but doesn't exist in database * Fix * Changed limiter * made api key rollout phase determination more consistent * add comment about enable_admin() method * api_user table: UNSIGNED ids, removed description comments, renamed logger * move api_key argument logging, invalid api key check, and last key usage update from _security.py to _common.py (coalescing @app.before_request and @app.after_request events to one usage each) Co-authored-by: dmytrotsko <dmytrotsko@gmail.com> * missed import for _is_public_route * removed TESTING_MODE and fixed unused imports * refactored into _db.py to remove circular import dependencies * import for werkzeug Unauthorized exception * removed recaptcha references * moved missing import from _common to _db * distinguish config-sourced 'user roles' from db-based * logging additions, including experimental stuff * logging level change * user object debugging during the api request/response flow * fix logging case when there is no 'User' * Muted non-existing api key check * Formatting, added AUTH * Added 'old' endpoints auth logic * Added email/api key duplicate check * remove un-slashed app route * change url (route) prefix to be empty instead of a bare slash * API Keys: API Documentation (#1158) * Unmuted api_key validation * Removed AUTH, added TEMPORARY_API_KEY and REGISTRATION_FORM_LINK * Added requests_left() function to check user's remaining requests and in order to display warning message about limit exceed during Phase1/Phase2 * Removed old authentication, updated warning messages * Added limit exceed, multiple exceed, temporary key messages to the printers * Fixed fluview endpoint authentication * Fluview endpoint hotfix * Added ratelimit headers to the response * Adjusted limiter warning messages(Messages text taken from PR#1163). Changed default limit to 60/h. * Update _security.py -- checks for all 4 phases * solidified rate limit filter behavior Co-authored-by: dmytrotsko <dmytrotsko@gmail.com> * Fixed user update. Removed api_key check on phase 1 * Added handling for data_source&signal pairs * Fixed order of warnings * Fixed printer message * logging changes, including rows returned and api user db CrUD operations * Printer messages hotfix * Fixed printer warnings * Hotfix * _printer hotfix * Fix _printer * reduced usages of flask.g, improved usages of _get_current_user() / current_user * update api-keys config variables with now-announced values * took roles out of config (its in db now), simplified role checking in sensors.py * logging cleanup * force named arguments * update api-keys registration and removal links * added local redirects for gForms for registration and user deletion, plus reduced admin error message verbosity * fix typo in import statement * update api-key error messages to use local redirect to registration form * fix 'multiples' detection so it recognizes date ranges as well * fix 'multiples' detection, cant do instance checks w/ subscripted generics (as is ) * fix 'multiples' detection, check for list instead of Sequence) * Text check for rollout (#1171) * Fix temporary key and contact point msgs * Removed exceptions no longer in use * UnAuthenticatedException * MissingAPIKeyException * Synced rollout warnings to approved comms * Fixed per minute -> per hour, and added "free" where it was supposed to be * Increase longevity of rate limit warning Co-authored-by: melange396 <george.haff@gmail.com> * Fixed expired session by using new session for each User operation * Fix environment variable and 429 output method * Propagate API key rollout env variable namechange to .env template * updated docs to local forwards for google forms links * adding key request form script * fix user_id logging (logged user w/ email before, which was removed completely. now logs id) * fixed scope for db_user * orm/user debugging * grrr stupid typo * more user/role debugging * Update usage text in src/server/admin/api_key_form_script.js Co-authored-by: Katie Mazaitis <krivard@cs.cmu.edu> * cleaning up... * changing sqlalchemy logging to match levels in dev --------- Co-authored-by: Dmytro Trotsko <dmytrotsko@gmail.com> Co-authored-by: Brian Clark <clark.bg@gmail.com> Co-authored-by: george haff <george.haff@gmail.com>

melange396 · 2023-05-24T15:15:30Z

previous attempts to release v4.1.0 failed (see #1176 , #1177 ). this one passed by giving the version number as " --new-version 4.1.0 major" (note leading space) to fool the bump2version step in create-release.yml into receiving the proper arguments.