fast_strptime() doesn't allow whitespace that base::strptime() allows

[kenahoo]

I wanted to switch some code at my company from base::strptime() to lubridate::fast_strptime(), because the %z in the latter understands ISO 8601 offsets like ±[hh]:[mm] and ±[hh][mm], whereas base::strptime()'s %z only understands the ±[hh][mm] format, without the colon.

However, this ended up causing problems, because base::strptime() allows varying whitespace in between the day and hour portion of the input, whereas lubridate::fast_strptime() does not.

lubridate::fast_strptime("8/1/2012  8:02:51.397000 AM", "%m/%d/%Y %I:%M:%OS %p")
#> [1] NA
lubridate::fast_strptime( "8/1/2012 8:02:51.397000 AM", "%m/%d/%Y %I:%M:%OS %p")
#> [1] "2012-08-01 08:02:51 UTC"
                strptime("8/1/2012  8:02:51.397000 AM", "%m/%d/%Y %I:%M:%OS %p")
#> [1] "2012-08-01 08:02:51 UTC"
                strptime( "8/1/2012 8:02:51.397000 AM", "%m/%d/%Y %I:%M:%OS %p")
#> [1] "2012-08-01 08:02:51 UTC"

^{Created on 2020-07-27 by the reprex package (v0.3.0)}

The colon-including format is the default (only?) format used by Python's pandas.Timeseries.isoformat(), and I wanted to consume some data emitted by pandas, so that led me to fast_strptime(). The stricter whitespace would likely break other stuff elsewhere in our code, though, so I'm not sure I can change.

It would be great if this small change in behavior between the two functions could be reconciled, or if not, at least noted in the docs as an intentional difference, for the wary.

[kenahoo]

Ooh, I get the privilege of submitting bug #911. ;-)

[vspinu]

I will have a look into fixing this. For now you can use parse_date_time2 which is almost equivalent to fast_strptime except it skips separators (not only spaces).

> lubridate::parse_date_time2("8/1/2012  8:02:51.397000 AM", "mdY IMOSp")
[1] "2012-08-01 08:02:51.39 UTC"

[kenahoo]

Thanks Vitalie, will do. I wasn't sure how that would handle a T separator between the date & time, but it looks like that works fine.

[kenahoo]

Hi Vitalie, any sense of when this might be released? I'm using a workaround in my code right now that does x <- gsub(" +", " ", x), but I'd rather put a condition on it like:

  if (packageVersion('lubridate') < '1.7.10')
    x <- gsub("  +", " ", x)  # See https://github.com/tidyverse/lubridate/issues/911

I decided against using parse_date_time2() because sometimes I want to explicitly require the T separator, and I can't see how to do that without using exact=TRUE, which takes away the flexible-whitespace behavior.

[kenahoo]

Hi @vspinu - any update on this?

[vspinu]

@kenahoo sorry for ditching this. Spaces are ignored in fast_strptime now. Thanks for bumping it up!

[kenahoo]

Hi @vspinu , no problem! I just tried the latest version from github and indeed it fixes the problems in my tests. Thanks for the fix.

[kenahoo]

I noticed that now whitespace is allowed between any tokens whatsoever, so the following happens - is that intentional?

lubridate::fast_strptime('2019 -    10-24   T    00:00   +1300', "%Y  -  %m  -  %dT%H:%M%z")
#> [1] "2019-10-23 11:00:00 UTC"
lubridate::fast_strptime('2019-10-24T00:00+1300', "%Y  -  %m  -  %dT%H:%M%z")
#> [1] "2019-10-23 11:00:00 UTC"
lubridate::fast_strptime('2019 -    10-24   T    00:00   +1300', "%Y-%m-%dT%H:%M%z")
#> [1] "2019-10-23 11:00:00 UTC"

strptime('2019 -    10-24   T    00:00   +1300', "%Y  -  %m  -  %dT%H:%M%z")
#> [1] NA
strptime('2019-10-24T00:00+1300', "%Y  -  %m  -  %dT%H:%M%z")
#> [1] "2019-10-23 11:00:00 UTC"
strptime('2019 -    10-24   T    00:00   +1300', "%Y-%m-%dT%H:%M%z")
#> [1] NA

^{Created on 2022-01-13 by the reprex package (v2.0.1)}

[vspinu]

yes, that's intentional. Why would middle space be special?

[kenahoo]

I'm just pointing out that whereas my original case was about varying nonzero amounts of whitespace, i.e. treating 1 or 2 space characters the same, the new code also treats 0 and 1 space characters the same. That seems like a bigger change and possibly contrary to many people's expectations.

As shown above, it also doesn't sync up very well with strptime's behavior.

[vspinu]

I see what you mean. space in format should match one or more spaces, no space in format should not match. This makes sense.

[vspinu]

Fixed.

[kenahoo]

Cool - looks more like what I expected:

lubridate::fast_strptime('2019 -    10-24   T    00:00   +1300', "%Y  -  %m  -  %dT%H:%M%z")
#> [1] NA
lubridate::fast_strptime('2019-10-24T00:00+1300', "%Y  -  %m  -  %dT%H:%M%z")
#> [1] "2019-10-23 11:00:00 UTC"
lubridate::fast_strptime('2019 -    10-24   T    00:00   +1300', "%Y-%m-%dT%H:%M%z")
#> [1] NA

strptime('2019 -    10-24   T    00:00   +1300', "%Y  -  %m  -  %dT%H:%M%z")
#> [1] NA
strptime('2019-10-24T00:00+1300', "%Y  -  %m  -  %dT%H:%M%z")
#> [1] "2019-10-23 11:00:00 UTC"
strptime('2019 -    10-24   T    00:00   +1300', "%Y-%m-%dT%H:%M%z")
#> [1] NA

^{Created on 2022-01-14 by the reprex package (v2.0.1)}