V3: Introduce timestamp_ns and timestamptz_ns#1632
Conversation
|
This is looking great @sungwy 🥳
Ideally, we want to use |
|
I think it would make sense to break this PR up into two separate items:
This PR will focus on item (1) |
kevinjqliu
left a comment
kevinjqliu
left a comment
There was a problem hiding this comment.
Thanks for adding this! Looks great, just have a few comments
|
Looks like there is some issue with the in I think this PR is ready to go. There are some parts we need to double-check, for example, the downcasting of micros to nanos, and that we don't have to fail for V3 when that isn't set. But since we don't yet support producing V3 metadata, I think that's okay. |
I removed this integration test because we can't write V3 metadata yet |
|
Thanks for working on this @sungwy, and thanks for the reviews @smaheshwar-pltr and @kevinjqliu 🙌 |
Fixes: apache#1552 - [x] Add TimestampNanoType and TimestampTzNanoType - [x] Add Readers and Writers - [x] Enhance Transforms - [x] Add String Expressions parsing for nanoseconds timestamps - [x] Add format-version compatibility check for each type - [x] Run compatibility check on TableMetadata creation - [x] Unit tests python native `datetime` module does not have support for nanoseconds. We'll need to update our internal date time representations to use a different library. numpy? arrow?
…ptz_to_nanos (#3505) # Rationale for this change `timestamptz_to_nanos` should raise a clear "Missing zone offset" error when it's given a nanosecond timestamp that has no timezone. Today it can't: the check meant to detect a zone-less timestamp accidentally tests the *same* pattern as the success check right above it, so it never matches. The input falls through to the vague "Invalid timestamp with zone" error instead. The fix is one line check the zone-less pattern (`ISO_TIMESTAMP_NANO`) instead, mirroring `timestamptz_to_micros`, which already does this correctly. (Introduced in #1632.) ## Are these changes tested? Yes added tests covering the error messages for both `timestamptz_to_nanos` and `timestamp_to_nanos`. ## Are there any user-facing changes? Only a clearer error message: a zone-less nanosecond timestamp now reports "Missing zone offset" instead of "Invalid timestamp with zone". No API changes.
4 participants
Fixes: #1552
python native
datetimemodule does not have support for nanoseconds. We'll need to update our internal date time representations to use a different library. numpy? arrow?