fix: truncate nanosecond-precision timestamps for BigQuery before casting#989
Conversation
…ting BigQuery TIMESTAMP only supports microsecond precision (6 fractional digits). Starting with dbt-bigquery >= 1.11, nanosecond-precision timestamps (9 fractional digits) may appear in string columns like execute_completed_at. This causes 'Invalid timestamp' errors when edr_cast_as_timestamp casts these values. Add a bigquery__edr_cast_as_timestamp dispatch that truncates fractional seconds beyond 6 digits before casting, matching the pattern used by the existing Dremio override. Co-Authored-By: mika@elementary-data.com <mika.kerman@gmail.com>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
|
👋 @devin-ai-integration[bot] |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughA BigQuery-specific macro override is added to Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Summary
Adds a
bigquery__edr_cast_as_timestampdispatch macro that truncates fractional seconds beyond microsecond precision (6 digits) before casting string values toTIMESTAMP.Problem: Starting with
dbt-bigquery >= 1.11, nanosecond-precision timestamps (9 fractional digits, e.g.2026-04-03T10:50:50.961498756Z) began appearing in string columns likeexecute_completed_atindbt_run_results. BigQuery'sTIMESTAMPtype only supports microsecond precision (6 digits), soCAST(... AS TIMESTAMP)fails withInvalid timestamperrors — completely blockingedr reportandedr send-report. Reported by multiple users in the community Slack.Fix: Uses
regexp_replaceto truncate to 6 fractional digits before casting, following the same pattern as the existing Dremio override. Timestamps with ≤6 fractional digits or no fractional seconds pass through unchanged. The existingbigquery__edr_cast_as_datemacro already callsedr_cast_as_timestampinternally, so it also benefits.Review & Testing Checklist for Human
r'(\.\d{6})\d+'with'\\1'backreference is valid BigQueryREGEXP_REPLACEusagedbt_run_results— CI likely does not cover BigQuery-specific behaviorNULLvalues should all cast correctly without errorsRecommended test plan: Run
edr reportoredr send-reportagainst a BigQuery project that has nanosecond-precision timestamps in thedbt_run_resultstable (or manually insert a row with e.g.2026-04-03T10:50:50.961498756Zinexecute_completed_at) and confirm the report generates successfully.Notes
cast(... as timestamp)pathLink to Devin session: https://app.devin.ai/sessions/8841375f9b774c8088e1bff6c8e44f52
Requested by: @MikaKerman
Summary by CodeRabbit