Optionally return null if element not exists map/array #4839
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
build |
amahussein
force-pushed
the
rapids-4668-fix
branch
from
3c520e5 to
0daee25
Compare
|
build |
Signed-off-by: Ahmed Hussein (amahussein) <a@ahussein.me>
revans2
approved these changes
Feb 22, 2022
| * This is to support ANSI mode: optionally return null result if element not exists | ||
| * in array/map. | ||
| */ | ||
| def shouldFailOnElementNotExists(): Boolean = false |
There was a problem hiding this comment.
So by default it does not fail, even if ANSI is enabled?
Is the following correct for what we want?
| version | throw exception if key not found... |
|---|---|
| 3.0.* | Never |
| 3.1.* | If ANSI is enabled |
| 3.2.* | If ANSI is enabled |
| 3.3.* | If ANSI is enabled and strictIndexOperator is enabled (SQLConf includes both together in the helper method) |
There was a problem hiding this comment.
The table is correct.
strictIndexOperator is true by default
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
fixes #4668
Signed-off-by: Ahmed Hussein (amahussein) a@ahussein.me
Spark 330 Added a new configuration
spark.sql.ansi.failOnElementNotExistswhich controls whether throwing exceptions or returning null results when element not exists in the[]operator in array/map type. Spark-37750The default value of the new configuration is true.
This PR adds support of the new flag for rapids.