Skip to content

fix: cast to and from timestamp_ntz#4008

Open
parthchandra wants to merge 1 commit intoapache:mainfrom
parthchandra:cast_ntz
Open

fix: cast to and from timestamp_ntz#4008
parthchandra wants to merge 1 commit intoapache:mainfrom
parthchandra:cast_ntz

Conversation

@parthchandra
Copy link
Copy Markdown
Contributor

@parthchandra parthchandra commented Apr 20, 2026

Which issue does this PR close?

Part of #286
Part of #378

Rationale for this change

We currently fall back to Spark for timestamp_ntz casts

What changes are included in this PR?

Add native support for casting to and from TimestampNTZType (timestamp without timezone).

PR Description

Implemented cast directions:

  • TimestampNTZ -> String (timezone-independent)
  • TimestampNTZ -> Date (timezone-independent)
  • TimestampNTZ -> Timestamp (session-TZ dependent)
  • Date -> TimestampNTZ (timezone-independent)
  • Timestamp -> TimestampNTZ (session-TZ dependent)

Not yet implemented:

Key implementation details

  • Timezone-independent casts (NTZ↔Date, NTZ→String): Pure arithmetic on epoch microseconds; session timezone has no effect on results.
  • Timezone-dependent casts (NTZ↔Timestamp): Interprets/produces local datetimes in the session timezone. Uses resolve_local_datetime() helper to handle DST ambiguity (fall-back) and gaps (spring-forward) matching Spark's ZonedDateTime semantics.
  • **Note on array_with_timezone **: This PR changes the Timestamp(, Some()) -> Timestamp(, Some()) path in array_with_timezone (utils.rs) to pass through the array unchanged, deferring the actual NTZ-to-Timestamp conversion to the explicit timestamp_ntz_to_timestamp call in cast.rs. This only affects the Arrow timezone annotation on the intermediate array, not the stored microsecond values. Callers of array_with_timezone that pass a Timestamp(_, Some(tz)) target type should verify they are not relying on the old behavior of re-annotating the array with the session timezone.

How are these changes tested?

Cast-specific test coverage

Cast Test method Timezone coverage Notes
Date → NTZ cast DateType to TimestampNTZType 17 representative zones Includes half-hour (Kolkata +5:30), quarter-hour (Kathmandu +5:45, Chatham +12:45) offsets
Timestamp → NTZ cast TimestampType to TimestampNTZType 17 zones Exercises DST transitions (Sao Paulo, Sydney, New York)
NTZ → String cast TimestampNTZType to StringType N/A (TZ-independent)
NTZ → Date cast TimestampNTZType to DateType 17 zones
NTZ → Timestamp cast TimestampNTZType to TimestampType 17 zones
String → NTZ cast StringType to TimestampNTZType ignored; not yet implemented

SQL integration tests (cast_timestamp_ntz.sql)

  • NTZ → String, Date, Timestamp
  • Date → NTZ, Timestamp → NTZ
  • Literal casts (e.g. CAST(TIMESTAMP_NTZ'2020-01-01 12:34:56.789' AS string))

Test data

generateTimestampNTZ() reuses generateTimestampLiterals() which covers epoch, modern dates, DST-transition dates, and sub-second precision values.

Timezone diversity

The representativeTimezones list (17 zones) was chosen to cover:

  • Standard offsets: UTC, UTC+8 (Shanghai), UTC+9 (Tokyo), UTC-5/-4 (New York)
  • Half-hour offsets: Asia/Kolkata (UTC+5:30)
  • Quarter-hour offsets: Asia/Kathmandu (UTC+5:45), Pacific/Chatham (UTC+12:45)
  • DST-transitioning zones: New York, Sydney, London, Sao Paulo
  • Non-DST zones: Dubai, Cairo, Johannesburg

ANSI mode

Each castTimestampTest invocation tests both ANSI_ENABLED=false (null on invalid input) and ANSI_ENABLED=true (exception on invalid input), plus try_cast().

@parthchandra parthchandra marked this pull request as draft April 20, 2026 20:03
@parthchandra parthchandra marked this pull request as ready for review April 21, 2026 16:30
@parthchandra parthchandra requested a review from andygrove April 21, 2026 16:30
@parthchandra
Copy link
Copy Markdown
Contributor Author

@coderfender, fyi

@parthchandra parthchandra requested a review from comphead April 21, 2026 16:30
@andygrove
Copy link
Copy Markdown
Member

Seems like this PR overlaps with #3253?

I'm fine with closing mine and reviewing this one though.

@coderfender
Copy link
Copy Markdown
Contributor

Thanks @parthchandra

@parthchandra
Copy link
Copy Markdown
Contributor Author

Seems like this PR overlaps with #3253?

I'm fine with closing mine and reviewing this one though.

Oh, I didn't realize this was not merged. There is an overlap, but #3253 does have additional stuff. If we merge this PR first, I can update #3253 afterwards.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants