added option for truncation in overwrite #509
Conversation
|
Alex Impert seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
Codecov Report
@@ Coverage Diff @@
## main #509 +/- ##
==========================================
- Coverage 87.52% 87.37% -0.16%
==========================================
Files 44 44
Lines 1988 1996 +8
Branches 110 124 +14
==========================================
+ Hits 1740 1744 +4
- Misses 248 252 +4
Flags with carried forward coverage won't be shown. Click here to find out more.
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
|
@ai-bq you'll need to sign the CLA (you should have gotten an email) before that check will pass. Also note there is a large file at |
| @@ -89,7 +89,8 @@ final case class DistributedFilesystemWriteConfig(jdbcConfig: JDBCConfig, | |||
| saveJobStatusTable: Boolean, | |||
| mergeKey: Option[ValidColumnList] = None, | |||
| timeOperations : Boolean = true, | |||
| arrayLength: Long = 0 | |||
| arrayLength: Long = 0, | |||
| truncate: Boolean | |||
There was a problem hiding this comment.
We should add this to the Javadoc above (and add the other missing params at the same time).
At some point, although not in this PR, we should consider just passing an options object rather than adding more and more parameters.
| @@ -462,6 +462,17 @@ object DSConfigSetupUtils { | |||
| config.get("target_table_sql").validNec | |||
| } | |||
|
|
|||
| def getTruncate(config: Map[String, String]): ValidationResult[Boolean] = { | |||
There was a problem hiding this comment.
We'll need to add a unit test for this in order to satisfy the coverage rules we have in place.
| @@ -260,6 +265,11 @@ class TableUtils(schemaTools: SchemaToolsInterface, jdbcLayer: JdbcLayerInterfac | |||
| .left.map(err => err.context("JDBC Error dropping table")) | |||
| } | |||
|
|
|||
| def truncateTable(tablename: TableName): ConnectorResult[Unit] = { | |||
There was a problem hiding this comment.
Not sure if we can add a unit test for this as well, we should follow whatever unit testing was done for drop table.
Summary
Some users require the option of truncating a table rather than dropping it in overwrite mode to retain table privileges.
Description
Truncate table was added to table utilities as well as the option of truncation through the dataframe writing options.
Related Issue
#507
Additional Reviewers
@alexey-temnikov
@jeremyp-bq
@jonathanl-bq
@alexr-bq