Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Truncate staging tables after ingestion #778

Merged

Conversation

eiffel777
Copy link
Contributor

This change makes it so the cloud staging tables, openstack_staging_table and generic_cloud_staging_table, are truncated at the end of the jobs-cloud-extract-openstack pipeline. The reason this table was not truncated before is because the event_id generated on the staging table is used as part of a unique key on the event table to prevent duplicates. Instead of generating the event_id on the staging table it is now generated on the event table and uniqueness on the event table is now based on the resource id, instance id, event time, event type and host id.

There are also changes to the event_asset table to prevent duplicate rows from being added to it and changing the queries to account for the event_id being created on the event table instead of the staging table.

The action for populating the instance_data table also has to be moved to its own action instead of residing in the OpenStackEventIngestor and GenericCloudEventIngestor. This is because the instance_data action needs to run after the event table and the event_id is generated.

Tests performed

All component, integration and regression tests were run and passed. Manually tested in docker to make sure values were correct and consistent when using multiple resources. Also tested that duplicate data is not loaded into the event table.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project as found in the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

@eiffel777 eiffel777 merged commit 00c1791 into ubccr:xdmod8.1 Jan 17, 2019
@eiffel777 eiffel777 added this to the 8.1.0 milestone Mar 28, 2019
@eiffel777 eiffel777 self-assigned this Mar 28, 2019
@eiffel777 eiffel777 added enhancement Enhancement of the functionality of an existing feature Category:Cloud Cloud Realm labels Mar 28, 2019
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
Category:Cloud Cloud Realm enhancement Enhancement of the functionality of an existing feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants