Export with history and soft deletes

[mikaelweave]

Description

• Adds new query parameters (_includeHistory and _includeDeleted) to Export operation.
  • Note - not compatible with _typeFilter because there are no longer search parameters with history/soft deletes. Validation added to address this.
• Updated search options to include IncludeHistory and IncludeDeleted. This is passed down to the data store during searches.
• Cosmos search will not add filters for history or deleted if either of the corresponding parameters are present and set to true.
• Updated E2E ExportFixture to add test data. Should improve performance of LongRunningExportTests which have been updated to use this.
  • Note: Other export E2E tests are not using this yet.
• Updated LongRunningExportTests and added three new tests for historical/delete test cases.
• Added logic for history/delete export for SQL.

Testing

• Manual testing by exporting data, modifying resources, deleting resources, and testing again.
• Added additional E2E testing for export with history/soft deletes.

FHIR Team Checklist

• Update the title of the PR to be succinct and less than 65 characters
• Add a milestone to the PR for the sprint that it is merged (i.e. add S47)
• Tag the PR with the type of update: Bug, Build, Dependencies, Enhancement, New-Feature or Documentation
• Tag the PR with Open source, Azure API for FHIR (CosmosDB or common code) or Azure Healthcare APIs (SQL or common code) to specify where this change is intended to be released.
• CI is green before merge
• Review squash-merge requirements

Semver Change (docs)

Patch|Skip|Feature|Breaking (reason)

[mikaelweave]

@LTA-Thinking - ready for SQL implementation when you are ready.

I have a couple open questions for you as well:

• For exporting of soft-deleted resources, we have a requirement to add an extension flagging it as "deleted". I put the code in ResourceToNdjsonBytesSerializer as it's only ~3 lines. Looking for feedback here - is it worth creating a new generalized service for any resource modifications that need to occur in $export?
• Did I miss any area for testing?

[LTA-Thinking]

Can you also update the Operation Definition for export?

[LTA-Thinking]

@LTA-Thinking - ready for SQL implementation when you are ready.

I have a couple open questions for you as well:

• For exporting of soft-deleted resources, we have a requirement to add an extension flagging it as "deleted". I put the code in ResourceToNdjsonBytesSerializer as it's only ~3 lines. Looking for feedback here - is it worth creating a new generalized service for any resource modifications that need to occur in $export?
• Did I miss any area for testing?

I think adding it to the serializer is sufficient. I don't think a generic solution is needed right now.
The test coverage looks good, thanks for fixing up the export test files. Maybe add one to the SearchOptionsFactoryTests to cover the new query parameters?

[SergeyGaluzo]