-
Notifications
You must be signed in to change notification settings - Fork 38
Fix for export projection and null data excluded bugs #2583
New issue
Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? # to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR fixes two export bugs by ensuring that (1) when projections are not specified, all columns are exported and (2) null column values are correctly included in the output across JSON, JSONLines, and CSV formats.
- Added tests for partial projections with and without metadata.
- Updated producer tasks to correctly handle cases when no projections are provided and when a column value is null.
- Enhanced UnitTestUtils to support partial output data for more granular testing.
Reviewed Changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.
Show a summary per file
File | Description |
---|---|
data-loader/core/src/test/java/com/scalar/db/dataloader/core/dataexport/producer/JsonProducerTaskTest.java | Added tests covering both projection scenarios and metadata inclusion/exclusion for JSON export. |
data-loader/core/src/test/java/com/scalar/db/dataloader/core/dataexport/producer/JsonLineProducerTaskTest.java | Added tests for JSONLines export with various projection and metadata combinations. |
data-loader/core/src/test/java/com/scalar/db/dataloader/core/dataexport/producer/CsvProducerTaskTest.java | Added tests for CSV export with similar projection and metadata scenarios, though test method names reference JSONLine incorrectly. |
data-loader/core/src/test/java/com/scalar/db/dataloader/core/UnitTestUtils.java | Introduced helper methods to generate partial output data and projected column lists. |
data-loader/core/src/main/java/com/scalar/db/dataloader/core/dataexport/producer/JsonProducerTask.java | Updated column projection check and added null-handling logic using NullNode. |
data-loader/core/src/main/java/com/scalar/db/dataloader/core/dataexport/producer/JsonLineProducerTask.java | Made similar projection and null-handling updates as in JsonProducerTask. |
data-loader/core/src/main/java/com/scalar/db/dataloader/core/dataexport/producer/CsvProducerTask.java | Updated the projection check to ensure all columns are exported when no projection is specified. |
Comments suppressed due to low confidence (1)
data-loader/core/src/test/java/com/scalar/db/dataloader/core/dataexport/producer/CsvProducerTaskTest.java:64
- The test method name references 'JsonLineString' even though it belongs to a CSV producer test. Consider renaming it to '...ShouldReturnValidCsvString' for clarity.
@Test void process_withValidResultList_withPartialProjections_shouldReturnValidJsonLineString() {
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Thank you!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Thank you.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Thank you!
Description
This PR is to fix 2 bugs found during testing, The issues were raised and fixed by @thongdk8 (Thank you). I have added the fix to this branch along with more unit tests to check and confirm the fix
Related issues and/or PRs
NA
Changes made
The following issues are fixed
For example, if
projection
argument was not specified with required columns, the generated import file would be empty.For example, if a value of a column that is to be imported is null for a particular row of data, the column is not added for that data in the import data file. Let say table
sample
has columnsa
,b
,c
, considera
to be the partition key. Let say fora=1
, the respective value ofb
is null, hence the entry fora=1
, in the import fileb
will be excluded and onlya
andc
will be added to it.Checklist
Additional notes (optional)
This is a bug fix PR for data loader export part
Release notes
NA