Feature/issue 229 - reapply db tables after fixing tags #264
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
torimcd
added a commit
that referenced
this pull request
Feb 4, 2025
* /version 1.5.0a0 * Fix after rebase (#241) * Update hydrocron-lambda.tf * Update pyproject.toml * update lock * fix lint * fix lint * /version 1.5.0a1 * Feature/issue 211 - Query track ingest table for granules with "to_ingest" status (#227) * /version 1.3.0a0 * Update build.yml * /version 1.3.0a1 * /version 1.3.0a2 * Feature/issue 175 - Update docs to point to OPS (#176) * changelog * update examples, remove load_data readme, info moved to wiki * Dependency update to fix snyk scan * issues/101: Support for HTTP Accept header (#172) * Reorganize timeseries code to prep for Accept header * Enable Accept header to return response of specific content-type * Fix whitespace and string continuation * Make error handling consistent and add an additional test where a reach can't be found * Update changelog with issue for unreleased version * Add 415 status code to API definition * Few minor cleanup items * Few minor cleanup items * Update to aiohttp@3.9.4 * Fix dependencies --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.3.0a3 * issues/102: Support compression of API response (#173) * Enable payload compression * Update changelog with issue --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.3.0a4 * Feature/issue 100 Add option to 'compact' GeoJSON result into single feature (#177) * Reorganize timeseries code to prep for Accept header * Enable Accept header to return response of specific content-type * Fix whitespace and string continuation * Make error handling consistent and add an additional test where a reach can't be found * Update changelog with issue for unreleased version * Add 415 status code to API definition * Few minor cleanup items * Few minor cleanup items * Update to aiohttp@3.9.4 * Fix dependencies * Update required query parameters based on current API functionality * Enable return of 'compact' GeoJSON response * Fix linting and add test data * Update documentation for API accept headers and compact GeoJSON response * Fix references to incorrect Accept header examples --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.3.0a5 * Feature/issue 183 (#185) * Provide introduction to timeseries endpoint * Remove _units in fields list * Fix typo * Update examples with Accept headers and compact query parameter * Add issue to changelog * Fix typo in timeseries documentation * Update pymysql * Update pymysql * Provide clarity on accept headers and request parameter fields * /version 1.3.0a6 * Feature/issue 186 Implement API keys (#188) * API Gateway Lambda authorizer to facilitate API keys and usage plans * Unit tests to test Lambda authorizer * Fix terraform file formatting * API Gateway Lambda Authorizer - Lambda function - API Keys and Authorizer definition in OpenAPI spec - API gateway API keys - API gateway usage plans - SSM parameters for API keys * Fix trailing whitespace * Set default region environment variable * Fix SNYK vulnerabilities * Add issue to changelog * Implement custom trusted partner header x-hydrocron-key * Update cryptography for SNYK vulnerability * Update documentation to include API key usage * Update quota and throttle settings for API Gateway * Update API keys documentation to indicate to be implemented * Move API key lookup to Lambda INIT * Remove API key authentication and update API key to x-hydrocron-key * /version 1.3.0a7 * Update changelog for 1.3.0 release * /version 1.4.0a0 * Feature/issue 198 (#207) * Update pylint to deal with errors and fix collection reference * Initial CMR and Hydrocron queries - Includes placeholders for other operations needed to track granule ingest. - GranuleUR query for Hydrocron tables. * Add and set up vcrpy for testing CMR API query * Test track ingest operations - Test CMR and hydrocron queries - Test granuleUR query - Update database to include granuleUR GSI * Update to use track_ingest naming consistently * Initial Lambda function and IAM role definition * Replace deprecated path function with as_file * Add SSM read IAM permissions * Add DynamoDB read permissions * Update track ingest lambda memory * Remove duplicate IAM permissions * Add in permissions to query index * Update changelog * Update changelog description * Use python_cmr for CMR API queries * /version 1.4.0a1 * Add doi to documentation pages (#216) * Update intro.md with DOI * Update overview.md with DOI * /version 1.4.0a2 * issue-193: Add Dynamo DB Table for SWOT Prior Lakes (#209) * add code to handle prior lakes shapefiles, add test prior lake data * update terraform to add prior lake table * fix tests, change to smaller test data file, changelog * linting * reconfigure main load_data method to make more readable and pass linting * lint * lint * fix string casting to lower storage req & update test responses to handle different rounding pattern in coords * update load benchmarking function for linting and add unit test * try parent collection for lakes * update version parsing for parent collection * fix case error * fix lake id reference * add logging to troubleshoot too large features * add item size logging and remove error raise for batch write * clean up logging statements & move numeric_columns assignment * update batch logging statement * Rename constant * Fix temp dir security risk https://rules.sonarsource.com/python/RSPEC-5443/ * Fix temp dir security risk https://rules.sonarsource.com/python/RSPEC-5443/ * fix code coverage calculation --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a3 * Feature/issue 201 Create a table for tracking granule ingest status (#214) * Define track ingest database and IAM permissions * Update changelog with issue * Modify table structure to support sparse status index * Updated to only apply PITR in ops --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a4 * Feature/issue 210 - Load large geometry polygons (#219) * add functions to handle null geometries and convert polygons to points * update doi in docs * fix fill null geometries * fix tests and update changelog * /version 1.4.0a5 * Feature/issue 222 - Add granule info to track ingest table on load (#223) * adjust lambdas to populate track ingest table on granule load * changelog * remove test cnm * lint * change error caught when handling checksum * update lambda role permissions to write to track ingest table * fix typo on lake table terraform * set default fill values for checksum and rev date in track status * fix checksum handling in bulk load data * lint * add logging to debug * /version 1.4.0a6 * Add SSM parameter read for last run time * Feature/issue-225: Create one track ingest table per feature type (#226) * add track ingest tables for each feature type and adjust load data to populate * changelog * /version 1.4.0a7 * Feature/issue 196 Add new feature type to query the API for lake data (#224) * Initial API queries for lake data * Unit tests for lake data * Updates after center point calculations - Removed temp code to calculate a point in API - Implemented unit test to test lake data retrieval - Updated fixtures to load in lake data for testing * Add read lake table permissions to lambda timeseries and track ingest roles * Update documenation to include lake data * Updated documentation to include info on lake centerpoints --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a8 * Feature/issue 205 - Add Confluence API key (#221) * Fix possible variable references before value is assigned * Define Confluence API key and trusted partner plan limits * Define a list of trusted partner keys and store under single parameter * Define API keys as encrypted envrionment variables for Lambda authorizer * Update authorizer and connection class to use KMS to retrieve API keys * Hack to force lambda deployment when ssm value changes (#218) * Add replace_triggered_by to hydrocron_lambda_authorizer * Introduce environment variable that contains random id which will change whenever an API key value changes. This will force lambda to publish new version of the function. * Remove unnecessary hash function * Update to SSM parameter API key storage and null_resource enviroment variable * Update Terraform and AWS provider * Update API key documentation * Set source_code_hash to force deployment of new image * Downgrade AWS provider to 4.0 to remove inline policy errors * Update docs/timeseries.md --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a9 * /version 1.4.0a10 * changelog for 1.4.0 release * /version 1.5.0a0 * Initial track ingest table query * Fix linting and code style * Implement feature count operations * Enable S3 permissions and set environment variable for track lambda * Fix trailing white spaces and code format * Update docstrings for class methods * Implement run time storage in SSM * Query track table unit tests * Update CHANGELOG with issue * Update SSM run time parameter * Fix trailing whitespace * Fix reference to IAM policy * Enable specification of temporal range to search revision date by * Fix SSM put parameter policy * Update IAM permissions for reading track ingest * Enable full temporal search on CMR granules * Add capability to download shapefile granule to count features * Update granule UR to include .zip * Count features via Hydrocron table query * Remove unnecessary s3 permissions * Remove whitespace from blank line * Update cryptography to 43.0.1 * update dependencies * upgrade geopandas * update dependencies --------- Co-authored-by: nikki-t <nikki-t@users.noreply.github.com> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: frankinspace <frankinspace@users.noreply.github.com> Co-authored-by: Victoria McDonald <49625194+torimcd@users.noreply.github.com> Co-authored-by: Cassie Nickles <44206543+cassienickles@users.noreply.github.com> Co-authored-by: cassienickles <cassienickles@users.noreply.github.com> Co-authored-by: podaac-cicd[bot] <podaac-cicd[bot]@users.noreply.github.com> Co-authored-by: Victoria McDonald <victoria.mcdonald@jpl.nasa.gov> Co-authored-by: torimcd <torimcd@users.noreply.github.com> * /version 1.5.0a2 * Feature/issue 212 - Update track ingest table with granule status (#228) * /version 1.3.0a0 * Update build.yml * /version 1.3.0a1 * /version 1.3.0a2 * Feature/issue 175 - Update docs to point to OPS (#176) * changelog * update examples, remove load_data readme, info moved to wiki * Dependency update to fix snyk scan * issues/101: Support for HTTP Accept header (#172) * Reorganize timeseries code to prep for Accept header * Enable Accept header to return response of specific content-type * Fix whitespace and string continuation * Make error handling consistent and add an additional test where a reach can't be found * Update changelog with issue for unreleased version * Add 415 status code to API definition * Few minor cleanup items * Few minor cleanup items * Update to aiohttp@3.9.4 * Fix dependencies --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.3.0a3 * issues/102: Support compression of API response (#173) * Enable payload compression * Update changelog with issue --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.3.0a4 * Feature/issue 100 Add option to 'compact' GeoJSON result into single feature (#177) * Reorganize timeseries code to prep for Accept header * Enable Accept header to return response of specific content-type * Fix whitespace and string continuation * Make error handling consistent and add an additional test where a reach can't be found * Update changelog with issue for unreleased version * Add 415 status code to API definition * Few minor cleanup items * Few minor cleanup items * Update to aiohttp@3.9.4 * Fix dependencies * Update required query parameters based on current API functionality * Enable return of 'compact' GeoJSON response * Fix linting and add test data * Update documentation for API accept headers and compact GeoJSON response * Fix references to incorrect Accept header examples --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.3.0a5 * Feature/issue 183 (#185) * Provide introduction to timeseries endpoint * Remove _units in fields list * Fix typo * Update examples with Accept headers and compact query parameter * Add issue to changelog * Fix typo in timeseries documentation * Update pymysql * Update pymysql * Provide clarity on accept headers and request parameter fields * /version 1.3.0a6 * Feature/issue 186 Implement API keys (#188) * API Gateway Lambda authorizer to facilitate API keys and usage plans * Unit tests to test Lambda authorizer * Fix terraform file formatting * API Gateway Lambda Authorizer - Lambda function - API Keys and Authorizer definition in OpenAPI spec - API gateway API keys - API gateway usage plans - SSM parameters for API keys * Fix trailing whitespace * Set default region environment variable * Fix SNYK vulnerabilities * Add issue to changelog * Implement custom trusted partner header x-hydrocron-key * Update cryptography for SNYK vulnerability * Update documentation to include API key usage * Update quota and throttle settings for API Gateway * Update API keys documentation to indicate to be implemented * Move API key lookup to Lambda INIT * Remove API key authentication and update API key to x-hydrocron-key * /version 1.3.0a7 * Update changelog for 1.3.0 release * /version 1.4.0a0 * Feature/issue 198 (#207) * Update pylint to deal with errors and fix collection reference * Initial CMR and Hydrocron queries - Includes placeholders for other operations needed to track granule ingest. - GranuleUR query for Hydrocron tables. * Add and set up vcrpy for testing CMR API query * Test track ingest operations - Test CMR and hydrocron queries - Test granuleUR query - Update database to include granuleUR GSI * Update to use track_ingest naming consistently * Initial Lambda function and IAM role definition * Replace deprecated path function with as_file * Add SSM read IAM permissions * Add DynamoDB read permissions * Update track ingest lambda memory * Remove duplicate IAM permissions * Add in permissions to query index * Update changelog * Update changelog description * Use python_cmr for CMR API queries * /version 1.4.0a1 * Add doi to documentation pages (#216) * Update intro.md with DOI * Update overview.md with DOI * /version 1.4.0a2 * issue-193: Add Dynamo DB Table for SWOT Prior Lakes (#209) * add code to handle prior lakes shapefiles, add test prior lake data * update terraform to add prior lake table * fix tests, change to smaller test data file, changelog * linting * reconfigure main load_data method to make more readable and pass linting * lint * lint * fix string casting to lower storage req & update test responses to handle different rounding pattern in coords * update load benchmarking function for linting and add unit test * try parent collection for lakes * update version parsing for parent collection * fix case error * fix lake id reference * add logging to troubleshoot too large features * add item size logging and remove error raise for batch write * clean up logging statements & move numeric_columns assignment * update batch logging statement * Rename constant * Fix temp dir security risk https://rules.sonarsource.com/python/RSPEC-5443/ * Fix temp dir security risk https://rules.sonarsource.com/python/RSPEC-5443/ * fix code coverage calculation --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a3 * Feature/issue 201 Create a table for tracking granule ingest status (#214) * Define track ingest database and IAM permissions * Update changelog with issue * Modify table structure to support sparse status index * Updated to only apply PITR in ops --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a4 * Feature/issue 210 - Load large geometry polygons (#219) * add functions to handle null geometries and convert polygons to points * update doi in docs * fix fill null geometries * fix tests and update changelog * /version 1.4.0a5 * Feature/issue 222 - Add granule info to track ingest table on load (#223) * adjust lambdas to populate track ingest table on granule load * changelog * remove test cnm * lint * change error caught when handling checksum * update lambda role permissions to write to track ingest table * fix typo on lake table terraform * set default fill values for checksum and rev date in track status * fix checksum handling in bulk load data * lint * add logging to debug * /version 1.4.0a6 * Add SSM parameter read for last run time * Feature/issue-225: Create one track ingest table per feature type (#226) * add track ingest tables for each feature type and adjust load data to populate * changelog * /version 1.4.0a7 * Feature/issue 196 Add new feature type to query the API for lake data (#224) * Initial API queries for lake data * Unit tests for lake data * Updates after center point calculations - Removed temp code to calculate a point in API - Implemented unit test to test lake data retrieval - Updated fixtures to load in lake data for testing * Add read lake table permissions to lambda timeseries and track ingest roles * Update documenation to include lake data * Updated documentation to include info on lake centerpoints --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a8 * Feature/issue 205 - Add Confluence API key (#221) * Fix possible variable references before value is assigned * Define Confluence API key and trusted partner plan limits * Define a list of trusted partner keys and store under single parameter * Define API keys as encrypted envrionment variables for Lambda authorizer * Update authorizer and connection class to use KMS to retrieve API keys * Hack to force lambda deployment when ssm value changes (#218) * Add replace_triggered_by to hydrocron_lambda_authorizer * Introduce environment variable that contains random id which will change whenever an API key value changes. This will force lambda to publish new version of the function. * Remove unnecessary hash function * Update to SSM parameter API key storage and null_resource enviroment variable * Update Terraform and AWS provider * Update API key documentation * Set source_code_hash to force deployment of new image * Downgrade AWS provider to 4.0 to remove inline policy errors * Update docs/timeseries.md --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a9 * /version 1.4.0a10 * changelog for 1.4.0 release * /version 1.5.0a0 * Initial track ingest table query * Fix linting and code style * Implement feature count operations * Enable S3 permissions and set environment variable for track lambda * Fix trailing white spaces and code format * Update docstrings for class methods * Implement run time storage in SSM * Query track table unit tests * Update CHANGELOG with issue * Update SSM run time parameter * Fix trailing whitespace * Fix reference to IAM policy * Enable specification of temporal range to search revision date by * Fix SSM put parameter policy * Update IAM permissions for reading track ingest * Enable full temporal search on CMR granules * Add capability to download shapefile granule to count features * Update granule UR to include .zip * Count features via Hydrocron table query * Remove unnecessary s3 permissions * Remove whitespace from blank line * Update cryptography to 43.0.1 * Update track ingest table operations * Update changelog with issue * update dependencies * upgrade geopandas * update dependencies --------- Co-authored-by: nikki-t <nikki-t@users.noreply.github.com> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: frankinspace <frankinspace@users.noreply.github.com> Co-authored-by: Victoria McDonald <49625194+torimcd@users.noreply.github.com> Co-authored-by: Cassie Nickles <44206543+cassienickles@users.noreply.github.com> Co-authored-by: cassienickles <cassienickles@users.noreply.github.com> Co-authored-by: podaac-cicd[bot] <podaac-cicd[bot]@users.noreply.github.com> Co-authored-by: Victoria McDonald <victoria.mcdonald@jpl.nasa.gov> Co-authored-by: torimcd <torimcd@users.noreply.github.com> * /version 1.5.0a3 * Feature/issue 203 - Construct CNM to trigger load data (#232) * /version 1.3.0a0 * Update build.yml * /version 1.3.0a1 * /version 1.3.0a2 * Feature/issue 175 - Update docs to point to OPS (#176) * changelog * update examples, remove load_data readme, info moved to wiki * Dependency update to fix snyk scan * issues/101: Support for HTTP Accept header (#172) * Reorganize timeseries code to prep for Accept header * Enable Accept header to return response of specific content-type * Fix whitespace and string continuation * Make error handling consistent and add an additional test where a reach can't be found * Update changelog with issue for unreleased version * Add 415 status code to API definition * Few minor cleanup items * Few minor cleanup items * Update to aiohttp@3.9.4 * Fix dependencies --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.3.0a3 * issues/102: Support compression of API response (#173) * Enable payload compression * Update changelog with issue --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.3.0a4 * Feature/issue 100 Add option to 'compact' GeoJSON result into single feature (#177) * Reorganize timeseries code to prep for Accept header * Enable Accept header to return response of specific content-type * Fix whitespace and string continuation * Make error handling consistent and add an additional test where a reach can't be found * Update changelog with issue for unreleased version * Add 415 status code to API definition * Few minor cleanup items * Few minor cleanup items * Update to aiohttp@3.9.4 * Fix dependencies * Update required query parameters based on current API functionality * Enable return of 'compact' GeoJSON response * Fix linting and add test data * Update documentation for API accept headers and compact GeoJSON response * Fix references to incorrect Accept header examples --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.3.0a5 * Feature/issue 183 (#185) * Provide introduction to timeseries endpoint * Remove _units in fields list * Fix typo * Update examples with Accept headers and compact query parameter * Add issue to changelog * Fix typo in timeseries documentation * Update pymysql * Update pymysql * Provide clarity on accept headers and request parameter fields * /version 1.3.0a6 * Feature/issue 186 Implement API keys (#188) * API Gateway Lambda authorizer to facilitate API keys and usage plans * Unit tests to test Lambda authorizer * Fix terraform file formatting * API Gateway Lambda Authorizer - Lambda function - API Keys and Authorizer definition in OpenAPI spec - API gateway API keys - API gateway usage plans - SSM parameters for API keys * Fix trailing whitespace * Set default region environment variable * Fix SNYK vulnerabilities * Add issue to changelog * Implement custom trusted partner header x-hydrocron-key * Update cryptography for SNYK vulnerability * Update documentation to include API key usage * Update quota and throttle settings for API Gateway * Update API keys documentation to indicate to be implemented * Move API key lookup to Lambda INIT * Remove API key authentication and update API key to x-hydrocron-key * /version 1.3.0a7 * Update changelog for 1.3.0 release * /version 1.4.0a0 * Feature/issue 198 (#207) * Update pylint to deal with errors and fix collection reference * Initial CMR and Hydrocron queries - Includes placeholders for other operations needed to track granule ingest. - GranuleUR query for Hydrocron tables. * Add and set up vcrpy for testing CMR API query * Test track ingest operations - Test CMR and hydrocron queries - Test granuleUR query - Update database to include granuleUR GSI * Update to use track_ingest naming consistently * Initial Lambda function and IAM role definition * Replace deprecated path function with as_file * Add SSM read IAM permissions * Add DynamoDB read permissions * Update track ingest lambda memory * Remove duplicate IAM permissions * Add in permissions to query index * Update changelog * Update changelog description * Use python_cmr for CMR API queries * /version 1.4.0a1 * Add doi to documentation pages (#216) * Update intro.md with DOI * Update overview.md with DOI * /version 1.4.0a2 * issue-193: Add Dynamo DB Table for SWOT Prior Lakes (#209) * add code to handle prior lakes shapefiles, add test prior lake data * update terraform to add prior lake table * fix tests, change to smaller test data file, changelog * linting * reconfigure main load_data method to make more readable and pass linting * lint * lint * fix string casting to lower storage req & update test responses to handle different rounding pattern in coords * update load benchmarking function for linting and add unit test * try parent collection for lakes * update version parsing for parent collection * fix case error * fix lake id reference * add logging to troubleshoot too large features * add item size logging and remove error raise for batch write * clean up logging statements & move numeric_columns assignment * update batch logging statement * Rename constant * Fix temp dir security risk https://rules.sonarsource.com/python/RSPEC-5443/ * Fix temp dir security risk https://rules.sonarsource.com/python/RSPEC-5443/ * fix code coverage calculation --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a3 * Feature/issue 201 Create a table for tracking granule ingest status (#214) * Define track ingest database and IAM permissions * Update changelog with issue * Modify table structure to support sparse status index * Updated to only apply PITR in ops --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a4 * Feature/issue 210 - Load large geometry polygons (#219) * add functions to handle null geometries and convert polygons to points * update doi in docs * fix fill null geometries * fix tests and update changelog * /version 1.4.0a5 * Feature/issue 222 - Add granule info to track ingest table on load (#223) * adjust lambdas to populate track ingest table on granule load * changelog * remove test cnm * lint * change error caught when handling checksum * update lambda role permissions to write to track ingest table * fix typo on lake table terraform * set default fill values for checksum and rev date in track status * fix checksum handling in bulk load data * lint * add logging to debug * /version 1.4.0a6 * Add SSM parameter read for last run time * Feature/issue-225: Create one track ingest table per feature type (#226) * add track ingest tables for each feature type and adjust load data to populate * changelog * /version 1.4.0a7 * Feature/issue 196 Add new feature type to query the API for lake data (#224) * Initial API queries for lake data * Unit tests for lake data * Updates after center point calculations - Removed temp code to calculate a point in API - Implemented unit test to test lake data retrieval - Updated fixtures to load in lake data for testing * Add read lake table permissions to lambda timeseries and track ingest roles * Update documenation to include lake data * Updated documentation to include info on lake centerpoints --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a8 * Feature/issue 205 - Add Confluence API key (#221) * Fix possible variable references before value is assigned * Define Confluence API key and trusted partner plan limits * Define a list of trusted partner keys and store under single parameter * Define API keys as encrypted envrionment variables for Lambda authorizer * Update authorizer and connection class to use KMS to retrieve API keys * Hack to force lambda deployment when ssm value changes (#218) * Add replace_triggered_by to hydrocron_lambda_authorizer * Introduce environment variable that contains random id which will change whenever an API key value changes. This will force lambda to publish new version of the function. * Remove unnecessary hash function * Update to SSM parameter API key storage and null_resource enviroment variable * Update Terraform and AWS provider * Update API key documentation * Set source_code_hash to force deployment of new image * Downgrade AWS provider to 4.0 to remove inline policy errors * Update docs/timeseries.md --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a9 * /version 1.4.0a10 * changelog for 1.4.0 release * /version 1.5.0a0 * Initial track ingest table query * Fix linting and code style * Implement feature count operations * Enable S3 permissions and set environment variable for track lambda * Fix trailing white spaces and code format * Update docstrings for class methods * Implement run time storage in SSM * Query track table unit tests * Update CHANGELOG with issue * Update SSM run time parameter * Fix trailing whitespace * Fix reference to IAM policy * Enable specification of temporal range to search revision date by * Fix SSM put parameter policy * Update IAM permissions for reading track ingest * Enable full temporal search on CMR granules * Add capability to download shapefile granule to count features * Update granule UR to include .zip * Count features via Hydrocron table query * Remove unnecessary s3 permissions * Remove whitespace from blank line * Update cryptography to 43.0.1 * Update track ingest table operations * Update changelog with issue * update dependencies * upgrade geopandas * update dependencies * Implement operations to publish CNM messages for granules requiring ingest * Implement unit test of publication operations * Fix linting * Add issue to changelog and fix linting * Add EventBridge schedules with appropriate Lambda permissions * Set initial schedule expressions and fix assume policy * Disable eventbridge schedules by default * Update schedule to run weekly * Define 1 hour latency to search by revision_date in CMR --------- Co-authored-by: nikki-t <nikki-t@users.noreply.github.com> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: frankinspace <frankinspace@users.noreply.github.com> Co-authored-by: Victoria McDonald <49625194+torimcd@users.noreply.github.com> Co-authored-by: Cassie Nickles <44206543+cassienickles@users.noreply.github.com> Co-authored-by: cassienickles <cassienickles@users.noreply.github.com> Co-authored-by: podaac-cicd[bot] <podaac-cicd[bot]@users.noreply.github.com> Co-authored-by: Victoria McDonald <victoria.mcdonald@jpl.nasa.gov> Co-authored-by: torimcd <torimcd@users.noreply.github.com> * Feature/issue 236: Update track ingest to allow UAT query of CMR (#240) * /version 1.3.0a0 * Update build.yml * /version 1.3.0a1 * /version 1.3.0a2 * Feature/issue 175 - Update docs to point to OPS (#176) * changelog * update examples, remove load_data readme, info moved to wiki * Dependency update to fix snyk scan * issues/101: Support for HTTP Accept header (#172) * Reorganize timeseries code to prep for Accept header * Enable Accept header to return response of specific content-type * Fix whitespace and string continuation * Make error handling consistent and add an additional test where a reach can't be found * Update changelog with issue for unreleased version * Add 415 status code to API definition * Few minor cleanup items * Few minor cleanup items * Update to aiohttp@3.9.4 * Fix dependencies --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.3.0a3 * issues/102: Support compression of API response (#173) * Enable payload compression * Update changelog with issue --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.3.0a4 * Feature/issue 100 Add option to 'compact' GeoJSON result into single feature (#177) * Reorganize timeseries code to prep for Accept header * Enable Accept header to return response of specific content-type * Fix whitespace and string continuation * Make error handling consistent and add an additional test where a reach can't be found * Update changelog with issue for unreleased version * Add 415 status code to API definition * Few minor cleanup items * Few minor cleanup items * Update to aiohttp@3.9.4 * Fix dependencies * Update required query parameters based on current API functionality * Enable return of 'compact' GeoJSON response * Fix linting and add test data * Update documentation for API accept headers and compact GeoJSON response * Fix references to incorrect Accept header examples --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.3.0a5 * Feature/issue 183 (#185) * Provide introduction to timeseries endpoint * Remove _units in fields list * Fix typo * Update examples with Accept headers and compact query parameter * Add issue to changelog * Fix typo in timeseries documentation * Update pymysql * Update pymysql * Provide clarity on accept headers and request parameter fields * /version 1.3.0a6 * Feature/issue 186 Implement API keys (#188) * API Gateway Lambda authorizer to facilitate API keys and usage plans * Unit tests to test Lambda authorizer * Fix terraform file formatting * API Gateway Lambda Authorizer - Lambda function - API Keys and Authorizer definition in OpenAPI spec - API gateway API keys - API gateway usage plans - SSM parameters for API keys * Fix trailing whitespace * Set default region environment variable * Fix SNYK vulnerabilities * Add issue to changelog * Implement custom trusted partner header x-hydrocron-key * Update cryptography for SNYK vulnerability * Update documentation to include API key usage * Update quota and throttle settings for API Gateway * Update API keys documentation to indicate to be implemented * Move API key lookup to Lambda INIT * Remove API key authentication and update API key to x-hydrocron-key * /version 1.3.0a7 * Update changelog for 1.3.0 release * /version 1.4.0a0 * Feature/issue 198 (#207) * Update pylint to deal with errors and fix collection reference * Initial CMR and Hydrocron queries - Includes placeholders for other operations needed to track granule ingest. - GranuleUR query for Hydrocron tables. * Add and set up vcrpy for testing CMR API query * Test track ingest operations - Test CMR and hydrocron queries - Test granuleUR query - Update database to include granuleUR GSI * Update to use track_ingest naming consistently * Initial Lambda function and IAM role definition * Replace deprecated path function with as_file * Add SSM read IAM permissions * Add DynamoDB read permissions * Update track ingest lambda memory * Remove duplicate IAM permissions * Add in permissions to query index * Update changelog * Update changelog description * Use python_cmr for CMR API queries * /version 1.4.0a1 * Add doi to documentation pages (#216) * Update intro.md with DOI * Update overview.md with DOI * /version 1.4.0a2 * issue-193: Add Dynamo DB Table for SWOT Prior Lakes (#209) * add code to handle prior lakes shapefiles, add test prior lake data * update terraform to add prior lake table * fix tests, change to smaller test data file, changelog * linting * reconfigure main load_data method to make more readable and pass linting * lint * lint * fix string casting to lower storage req & update test responses to handle different rounding pattern in coords * update load benchmarking function for linting and add unit test * try parent collection for lakes * update version parsing for parent collection * fix case error * fix lake id reference * add logging to troubleshoot too large features * add item size logging and remove error raise for batch write * clean up logging statements & move numeric_columns assignment * update batch logging statement * Rename constant * Fix temp dir security risk https://rules.sonarsource.com/python/RSPEC-5443/ * Fix temp dir security risk https://rules.sonarsource.com/python/RSPEC-5443/ * fix code coverage calculation --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a3 * Feature/issue 201 Create a table for tracking granule ingest status (#214) * Define track ingest database and IAM permissions * Update changelog with issue * Modify table structure to support sparse status index * Updated to only apply PITR in ops --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a4 * Feature/issue 210 - Load large geometry polygons (#219) * add functions to handle null geometries and convert polygons to points * update doi in docs * fix fill null geometries * fix tests and update changelog * /version 1.4.0a5 * Feature/issue 222 - Add granule info to track ingest table on load (#223) * adjust lambdas to populate track ingest table on granule load * changelog * remove test cnm * lint * change error caught when handling checksum * update lambda role permissions to write to track ingest table * fix typo on lake table terraform * set default fill values for checksum and rev date in track status * fix checksum handling in bulk load data * lint * add logging to debug * /version 1.4.0a6 * Add SSM parameter read for last run time * Feature/issue-225: Create one track ingest table per feature type (#226) * add track ingest tables for each feature type and adjust load data to populate * changelog * /version 1.4.0a7 * Feature/issue 196 Add new feature type to query the API for lake data (#224) * Initial API queries for lake data * Unit tests for lake data * Updates after center point calculations - Removed temp code to calculate a point in API - Implemented unit test to test lake data retrieval - Updated fixtures to load in lake data for testing * Add read lake table permissions to lambda timeseries and track ingest roles * Update documenation to include lake data * Updated documentation to include info on lake centerpoints --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a8 * Feature/issue 205 - Add Confluence API key (#221) * Fix possible variable references before value is assigned * Define Confluence API key and trusted partner plan limits * Define a list of trusted partner keys and store under single parameter * Define API keys as encrypted envrionment variables for Lambda authorizer * Update authorizer and connection class to use KMS to retrieve API keys * Hack to force lambda deployment when ssm value changes (#218) * Add replace_triggered_by to hydrocron_lambda_authorizer * Introduce environment variable that contains random id which will change whenever an API key value changes. This will force lambda to publish new version of the function. * Remove unnecessary hash function * Update to SSM parameter API key storage and null_resource enviroment variable * Update Terraform and AWS provider * Update API key documentation * Set source_code_hash to force deployment of new image * Downgrade AWS provider to 4.0 to remove inline policy errors * Update docs/timeseries.md --------- Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> * /version 1.4.0a9 * /version 1.4.0a10 * changelog for 1.4.0 release * update dependencies for 1.4.0 release * /version 1.5.0a0 * fix CMR query in UAT * /version 1.4.0rc1 * fix typo in load_data lambda * /version 1.4.0rc2 * Initial track ingest table query * Fix linting and code style * Implement feature count operations * Enable S3 permissions and set environment variable for track lambda * Fix trailing white spaces and code format * Update docstrings for class methods * Implement run time storage in SSM * Query track table unit tests * Update CHANGELOG with issue * Update SSM run time parameter * Fix trailing whitespace * Fix reference to IAM policy * Enable specification of temporal range to search revision date by * Fix SSM put parameter policy * Update IAM permissions for reading track ingest * Enable full temporal search on CMR granules * Add capability to download shapefile granule to count features * Update granule UR to include .zip * Count features via Hydrocron table query * Remove unnecessary s3 permissions * Remove whitespace from blank line * Update cryptography to 43.0.1 * Update track ingest table operations * Update changelog with issue * update dependencies * upgrade geopandas * update dependencies * fix index on rev date in load data lambda * update dependencies * lint readme * /version 1.4.0rc3 * /version 1.4.0rc4 * Implement operations to publish CNM messages for granules requiring ingest * Implement unit test of publication operations * Fix linting * Add issue to changelog and fix linting * Add EventBridge schedules with appropriate Lambda permissions * Set initial schedule expressions and fix assume policy * fix cmr env search by venue * /version 1.4.0rc5 * Disable eventbridge schedules by default * Update schedule to run weekly * Define 1 hour latency to search by revision_date in CMR * Allow CMR UAT query based on HYDROCRON_ENV environment variable * Update unit tests to accomodate UAT CMR query * Add earthdata login credentials to Lambda * Add issue to changelog * Fix linting white space --------- Co-authored-by: nikki-t <nikki-t@users.noreply.github.com> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: frankinspace <frankinspace@users.noreply.github.com> Co-authored-by: Victoria McDonald <49625194+torimcd@users.noreply.github.com> Co-authored-by: Cassie Nickles <44206543+cassienickles@users.noreply.github.com> Co-authored-by: cassienickles <cassienickles@users.noreply.github.com> Co-authored-by: podaac-cicd[bot] <podaac-cicd[bot]@users.noreply.github.com> Co-authored-by: Victoria McDonald <victoria.mcdonald@jpl.nasa.gov> Co-authored-by: torimcd <torimcd@users.noreply.github.com> * /version 1.5.0a4 * /version 1.6.0a0 * /version 1.5.0a1 * Feature/issue 235 - Track ingest table can be populated with granules that aren't loaded into Hydrocron (#245) * Raise an error if collection shortname does not match Hydrocron table names * Raise an error unsupported lake data in load granule operations * Remove trailing whitespace * Fix code formatting * Update CHANGELOG with issue * Feature/issue 248 - Track ingest operations need to query UAT for granule files (#249) * Query to return granule files should query UAT when running in SIT or UAT environments * SIT execution should return UAT files for load granule operations * Set venue environment variable before running test of query_cmr * Add issue to CHANGELOG * /version 1.5.0a2 * Update readme and changelog (#253) * /version 1.5.0a3 * Feature/issue 250 - Handle overlapping times with unique CRIDS (#257) * Handle overlapping times with unique CRIDs * Define unit test to test when reprocessed granule arrives * Add issue to CHANGELOG * Add reprocessed CRID to eventbridge schedule input * Handle cases with empty reprocessed_crid * /version 1.5.0a4 * Feature/issue 258 - Granules with large numbers of features cannot be loadedd (#259) * change assemble attrs function to avoid for loop * change how attributes are concatenated during shp unpack to avoid slow looping * remove unused import * Update API test data with less precise data coordinates * remove logging every item in batch writer * lint --------- Co-authored-by: Nikki <17799906+nikki-t@users.noreply.github.com> * /version 1.5.0a5 * 1.5.0 release and unreleased section * /version 1.6.0a0 * /version 1.5.0a0 * Feature/issue 229 - Create new database tables for version D collections (#260) * Add version d db tables, update load_data module to handle changing constants * linting * rework constants to sustain version changes, update tests * lint * fix changelog * /version 1.5.0a1 * update version * Feature/issue 229 - reapply db tables after fixing tags (#264) * Add version d db tables, update load_data module to handle changing constants * linting * rework constants to sustain version changes, update tests * lint * fix changelog * update readme to force rebuild (#266) * /version 1.6.0a1 * /version 1.6.0a2 * Feature/issue 233 - Create new API parameter for user to specify data version (#268) * Move track ingest constants and fix table info * Accept and validate collection_name query parameter * Determine table data to query on based off of collection and feature type * Log collection name and fix bug that interates over table info * Fix table_name reference in table info * Linting recommendations fix * Add api_feature_type to determine correct table info * Modified for new env vars and added additional test of collection name * Add issues to changelog * Fix reference to default collection version in Lambda environment variables * /version 1.6.0a3 * 267 - Modify track ingest operations to support new collection versions (#271) * Remove hydrocron tables in input parameters and duplicate overlap detection * Add version D track ingest tables, remove table input parameters, and fix formatting * Fix bug when raising table mismatch error for collection shortname * Add issue to changelog * /version 1.6.0a4 * Feature/issue 273 - Modify the logging of the timeseries Lambda response object to reduce the log statement size (#274) * Modify response logging to only include status, time, and hits * Update changelog with issue * Log errors with HTTP code and error message for all error types * Log CNM message with double quotes * Update changelog for new release candidate * Update depenedencies * Update vcrpy and cassettes to resolve missing version_string * /version 1.6.0rc1 * Remove no-update flag from poetry lock command * /version 1.6.0rc2 * Add version D SWOT and track ingest tables to IAM Dynamo policies (#276) * Add version D SWOT and track ingest tables to IAM Dynamo policies * /version 1.6.0rc3 --------- Co-authored-by: nikki-t <nikki-t@users.noreply.github.com> * /version 1.6.0rc4 * Update default and constant variables for version D collection * Fix tests so that any change to the collection version does not impact results * /version 1.6.0rc5 * Modify for version D data but default to 2.0 collection for API * /version 1.6.0rc6 * Update track ingest reprocessed CRID to PID0 * /version 1.6.0rc7 * Feature/issue-234 Reorganize docs and add versioning section (#269) * Reorganize docs and add versioning section * changelog * fix typos, minor phrasing in docs * /version 1.6.0rc8 * Update runtime SSM parameter name for collection * /version 1.6.0rc9 * Update to emptry reprocessed CRID * /version 1.6.0rc10 --------- Co-authored-by: nikki-t <nikki-t@users.noreply.github.com> Co-authored-by: Frank Greguska <89428916+frankinspace@users.noreply.github.com> Co-authored-by: frankinspace <frankinspace@users.noreply.github.com> Co-authored-by: Nikki Tebaldi <17799906+nikki-t@users.noreply.github.com> Co-authored-by: Cassie Nickles <44206543+cassienickles@users.noreply.github.com> Co-authored-by: cassienickles <cassienickles@users.noreply.github.com> Co-authored-by: podaac-cicd[bot] <podaac-cicd[bot]@users.noreply.github.com> Co-authored-by: torimcd <torimcd@users.noreply.github.com> Co-authored-by: James Wood <James.F.Wood@jpl.nasa.gov>
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Github Issue: #229 - reapply database tables after fixing tags
Description
See #260
Overview of work done
Overview of verification done
Overview of integration done
PR checklist:
See Pull Request Review Checklist for pointers on reviewing this pull request