[Format] Add Timedelta type to describe time intervals

[asfimport]

xref #551 and #551 (comment)

this will allow round-tripping of pandas Timedelta and numpy timedelt64[ns] types. The will have a similar TimeUnit to TimestampType (s, us, ms, ns). Possible impl include making this pure 64-bit.

Reporter: Jeff Reback / @jreback
Assignee: Micah Kornfield / @emkornfield

Related issues:

• [JS] Add support for interval types in javascript (blocks)
• [Python] Serialize timedelta64 values from pandas to Arrow interval types (blocks)
• [Format] Interval(DAY_TIME) has no unit (relates to)

PRs and other links:

• GitHub Pull Request apache/arrow#3644

_{Note: This issue was originally created as ARROW-835. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
Marked for 0.6.0. We should discuss further on the mailing list

[asfimport]

Wes McKinney / @wesm:
Kicking this can a bit as not urgent for 0.6.0. We need to articulate what the Interval type at https://github.com/apache/arrow/blob/master/format/Schema.fbs#L177 is and how it is different from a Timedelta with unit. There is good precedent for supporting 64-bit timedelta from other projects

[asfimport]

Wes McKinney / @wesm:
Part of PR for ARROW-352 #920

[asfimport]

Jeff Reback / @jreback:
computationally using ns precesion causes overflow a fair amount, range is only 106k days (though it’s easily detected); might consider using an alternative storage (eg store days in an int32 and then ns in a int64), but this might be at odds with other systems

[asfimport]

Wes McKinney / @wesm:
Interesting. What are people doing that causes an overflow?

[asfimport]

Jeff Reback / @jreback:
pandas-dev/pandas#9442

an older issue, it’s mainly doing mean where you have to be careful

[asfimport]

Wes McKinney / @wesm:
I see. This seems like a numerical issue we should handle in our reduction implementations when we get to that

[asfimport]

Wes McKinney / @wesm:
This is blocked on ARROW-352. I just e-mailed the mailing list

[asfimport]

Wes McKinney / @wesm:
Moving to 0.14. We can make a push to get this into ship shape for that release cycle

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 3644
#3644