Support extracting struct fields as Variant using ExtensionType#9598
Support extracting struct fields as Variant using ExtensionType#9598codephage2020 wants to merge 3 commits intoapache:mainfrom
Conversation
scovich
left a comment
scovich
left a comment
There was a problem hiding this comment.
Thanks for looking into this. A bunch of questions.
| // Field was requested as Variant but data is all-null; | ||
| // preserve the data type from the child without extension metadata. |
There was a problem hiding this comment.
Is this intentional, to strip away e.g. JSON or other extensions that might be present?
Also, why is it correct to change the data type merely because the data are all-null?
Shouldn't it just come back as an all-null StructArray with variant annotations?
|
Thank you for the review. Looks like #9612 needs to be merged first. |
|
|
||
| // Row 0: data = {"score": 95} | ||
| let data0 = data_variant_array.value(0); | ||
| assert!(matches!(data0, Variant::Object(_))); |
There was a problem hiding this comment.
Do we need to assert the value of the variant
| } | ||
| // In safe mode, return NotShredded to let the caller | ||
| // handle it via value column. | ||
| return Ok(ShreddedPathStep::NotShredded); |
There was a problem hiding this comment.
Why does this isn't Missing? Correct me if I'm wrong: the code path requests a VariantPathElement::Field, which means it's in a struct, and the current field isn't a struct, so the requested name is missing?
3 participants
Which issue does this PR close?
Rationale for this change
variant_get cannot extract a struct where some fields remain as VariantArrays. Callers must call variant_get per leaf and assemble results manually.
What changes are included in this PR?
Support extracting struct fields as VariantArray via VariantType extension metadata in variant_get, and fix panics on absent fields and silent data loss on non-struct typed values.
Are these changes tested?
Yes — test_struct_extraction_with_variant_fields and test_struct_extraction_missing_variant_field_no_panic.
Are there any user-facing changes?
No breaking changes. Fields marked with VariantType extension metadata are extracted as VariantArrays in struct extraction.