You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Python/Pyspark do not really support dates/timestamps that are prior to the year 1. This can cause some issues in some cases as Spark will happily produce values that cannot be shipped back to python. Even before #9736 the current code restricted the range of values that could be generated to try and avoid this issue. But there is nothing that prevents a test from subtracting a month, or year, or whatever from a timestamp that would result in a value being out of range.
I think it would be better to have timestamp and data gen produce a full range of possible supported values. Especially for corner case tests with timezones/etc. We could also have the code automatically convert dates and timestamps to the raw number representation using date_diff and unix_micros. But there was also the suggestion to try and do both. But I am not 100% sure how to do both of them and coordinate things. So this is here to try and figure out if it is possible or if there is a better solution we have not thought of yet.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
Python/Pyspark do not really support dates/timestamps that are prior to the year 1. This can cause some issues in some cases as Spark will happily produce values that cannot be shipped back to python. Even before #9736 the current code restricted the range of values that could be generated to try and avoid this issue. But there is nothing that prevents a test from subtracting a month, or year, or whatever from a timestamp that would result in a value being out of range.
I think it would be better to have timestamp and data gen produce a full range of possible supported values. Especially for corner case tests with timezones/etc. We could also have the code automatically convert dates and timestamps to the raw number representation using
date_diff
andunix_micros
. But there was also the suggestion to try and do both. But I am not 100% sure how to do both of them and coordinate things. So this is here to try and figure out if it is possible or if there is a better solution we have not thought of yet.The text was updated successfully, but these errors were encountered: