[C++] Add a strptime option to control the cutoff between 1900 and 2000 when %y

[asfimport]

When parsing to datetime a string with year in the short format ({}%y{}), it would be great if we could have control over the cutoff point between 1900 and 2000. Currently it is implicitly set to 68:

library(arrow, warn.conflicts = FALSE)

a <- Array$create(c("68-05-17", "69-05-17"))
call_function("strptime", a, options = list(format = "%y-%m-%d", unit = 0L))
#> Array
#> <timestamp[s]>
#> [
#>   2068-05-17 00:00:00,
#>   1969-05-17 00:00:00
#> ]

For example, lubridate named this argument cutoff_2000 argument (e.g. for {}fast_strptime){}. This works as follows:

library(lubridate, warn.conflicts = FALSE)

dates_vector <- c("68-05-17", "69-05-17", "55-05-17")
fast_strptime(dates_vector, format = "%y-%m-%d")
#> [1] "2068-05-17 UTC" "1969-05-17 UTC" "2055-05-17 UTC"
fast_strptime(dates_vector, format = "%y-%m-%d", cutoff_2000 = 50)
#> [1] "1968-05-17 UTC" "1969-05-17 UTC" "1955-05-17 UTC"
fast_strptime(dates_vector, format = "%y-%m-%d", cutoff_2000 = 70)
#> [1] "2068-05-17 UTC" "2069-05-17 UTC" "2055-05-17 UTC"

In the lubridate::fast_strptime() documentation it is described as follows:

cutoff_2000
integer. For y format, two-digit numbers smaller or equal to cutoff_2000 are parsed as though starting with 20, otherwise parsed as though starting with 19. {}Available only for functions relying on lubridates internal parser{}.

Reporter: Dragoș Moldovan-Grünfeld / @dragosmg

Related issues:

• [C++] Strptime issues umbrella (is a child of)

_{Note: This issue was originally created as ARROW-16596. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
"Available only for functions relying on lubridates internal parser" means we don't have to implement it, right?

[asfimport]

Dragoș Moldovan-Grünfeld / @dragosmg:
I think we would need to if we wanted to emulate the behaviour. I think it refers to the fact that other parsing functions in lubridate use the parsing functionality native to R (e.g. R's strptime as opposed to lubridate's {}fast_strptime){}.

I implemented this using the Arrow strptime kernel, but it would definitely be nice to have this option.
 
Having said that, it is not crucial, as the users can manipulate strings to add the first 2 digits and then parse with a "%Y-%m-%d" format.

[asfimport]

Dragoș Moldovan-Grünfeld / @dragosmg:
Maybe related. I find the following unexpected:

library(arrow, warn.conflicts = FALSE)

a <- Array$create("68-10-07 19:04:0")
call_function("strptime", a, options = list(format = "%Y-%m-%d %H:%M:%S", unit = 0L))
#> Array
#> <timestamp[s]>
#> [
#>   0068-10-07 19:04:00
#> ]
call_function("strptime", a, options = list(format = "%y-%m-%d %H:%M:%S", unit = 0L))
#> Array
#> <timestamp[s]>
#> [
#>   2068-10-07 19:04:00
#> ]

I would expect an error when there is a mismatch between the string and the format, i.e. string has a short year ({}%y{}) and we try to parse using a long format ({}%Y{}). I think it would be much better to error or return a null in this situation.

[asfimport]

Antoine Pitrou / @pitrou:
@dragosmg On which platform is this? On non-Windows platforms, we basically defer to the platform's strptime function, so there is not much we can do to prevent this.

[asfimport]

Antoine Pitrou / @pitrou:
Interestingly, Python's own strptime implementation catches this:

>>> time.strptime("68-10-07 19:04:00", "%Y-%m-%d %H:%M:%S")
Traceback (most recent call last):
  Input In [16] in <cell line: 1>
    time.strptime("68-10-07 19:04:00", "%Y-%m-%d %H:%M:%S")
  File ~/miniconda3/envs/pyarrow/lib/python3.9/_strptime.py:562 in _strptime_time
    tt = _strptime(data_string, format)[0]
  File ~/miniconda3/envs/pyarrow/lib/python3.9/_strptime.py:349 in _strptime
    raise ValueError("time data %r does not match format %r" %
ValueError: time data '68-10-07 19:04:00' does not match format '%Y-%m-%d %H:%M:%S'

>>> time.strptime("68-10-07 19:04:00", "%y-%m-%d %H:%M:%S")
time.struct_time(tm_year=2068, tm_mon=10, tm_mday=7, tm_hour=19, tm_min=4, tm_sec=0, tm_wday=6, tm_yday=281, tm_isdst=-1)
>>> time.strptime("1968-10-07 19:04:00", "%Y-%m-%d %H:%M:%S")
time.struct_time(tm_year=1968, tm_mon=10, tm_mday=7, tm_hour=19, tm_min=4, tm_sec=0, tm_wday=0, tm_yday=281, tm_isdst=-1)

[asfimport]

Dragoș Moldovan-Grünfeld / @dragosmg:
@pitrou  I'm on macOS. 

[asfimport]

Antoine Pitrou / @pitrou:
Also, by reading the code, the vendored implementation we use on Windows (which was copied from the musl libc) probably shows the same lenient behaviour.

[asfimport]

Dragoș Moldovan-Grünfeld / @dragosmg:
Interesting, but the base R implementation behaves the same. lubridate is a bit more robust though:

b <- "68-10-07 19:04:0"
strptime(b, format = "%Y-%m-%d %H:%M:%S")
#> [1] "0068-10-07 19:04:00 LMT"
strptime(b, format = "%y-%m-%d %H:%M:%S")
#> [1] "2068-10-07 19:04:00 BST"

lubridate::fast_strptime(b, format = "%Y-%m-%d %H:%M:%S")
#> [1] NA
lubridate::fast_strptime(b, format = "%y-%m-%d %H:%M:%S")
#> [1] "2068-10-07 19:04:00 UTC"

[github-actions]

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.