-
Notifications
You must be signed in to change notification settings - Fork 0
Support compressed virtual chunk locations (needs fzstd dictionary support) #3
Copy link
Copy link
Open
Description
icechunk (earth-mover/icechunk#1776) introduced dictionary-compressed virtual chunk locations. The manifest FlatBuffer schema has location_dictionary, compression_algorithm, and compressed_location fields that we don't read yet.
Current state
- Our generated FlatBuffer classes already have the accessors
- Our dependency
fzstddoesn't support dictionary decompression but there's an open PR adding it: Support loading a dictionary for decompression 101arrowz/fzstd#18 - Tried swapping to
zstdifywhich has dictionary support but ran into issues I think related to fix(zstdify): correct sequence RLE decoding semantics bhouston/zstdify#3 - No upstream fixture data uses this feature yet
What's needed
- Wait for fzstd#18 to land and release
- Update
manifest-parser.tsto readcompressedLocationfrom ChunkRef - Update
manifest-parser.tsto readlocationDictionary/compressionAlgorithmfrom Manifest - Decompress locations at parse time (matching Rust impl), priority: compressed_location → location → chunk_id → inline
- Add tests (will need synthetic fixtures or wait for upstream data)
References
- Upstream schema:
manifest.fbs—ChunkRef.compressed_location,Manifest.location_dictionary,Manifest.compression_algorithm
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels