Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Add Parallel Cosmos Export #3584

Merged
merged 133 commits into from
Jan 26, 2024
Merged

Conversation

mikaelweave
Copy link
Contributor

@mikaelweave mikaelweave commented Nov 3, 2023

Description

Adds parallel export to CosmosDB by resource type and feed range. This export does not work like SQL because CosmosDB doesn't work like SQL. It's costly to get date ranges in CosmosDB

Performance is >100K resources per second on a 100K RU CosmosDB wth 300 million documents. 20 node P2V2 App Service. Exporting history gets closer to 200K resources per second.

Related issues

Addresses AB112054.

Testing

LOTS of local testing and testing in Azure OSS. I've mainly been testing with 50 million docs in my CosmosDB.
Xoriant has also been testing with a 300 million doc db.

FHIR Team Checklist

  • Update the title of the PR to be succinct and less than 65 characters
  • Add a milestone to the PR for the sprint that it is merged (i.e. add S47)
  • Tag the PR with the type of update: Bug, Build, Dependencies, Enhancement, New-Feature or Documentation
  • Tag the PR with Open source, Azure API for FHIR (CosmosDB or common code) or Azure Healthcare APIs (SQL or common code) to specify where this change is intended to be released.
  • Tag the PR with Schema Version backward compatible or Schema Version backward incompatible or Schema Version unchanged if this adds or updates Sql script which is/is not backward compatible with the code.
  • CI is green before merge Build Status
  • Review squash-merge requirements

Semver Change (docs)

Patch|Skip|Feature|Breaking (reason)

mikaelweave and others added 30 commits June 20, 2023 21:44
@mikaelweave mikaelweave modified the milestones: S130, S132 Jan 9, 2024
@mikaelweave mikaelweave modified the milestones: S132, S133 Jan 22, 2024
@mikaelweave mikaelweave merged commit 7b59976 into main Jan 26, 2024
6 checks passed
@mikaelweave mikaelweave deleted the personal/mikaelw/parallel-cosmos-export branch January 26, 2024 22:51
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
Azure API for FHIR Label denotes that the issue or PR is relevant to the Azure API for FHIR Enhancement Enhancement on existing functionality.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants