Skip to content

Support compressed virtual chunk locations (needs fzstd dictionary support) #3

@Shane98c

Description

@Shane98c

icechunk (earth-mover/icechunk#1776) introduced dictionary-compressed virtual chunk locations. The manifest FlatBuffer schema has location_dictionary, compression_algorithm, and compressed_location fields that we don't read yet.

Current state

What's needed

  • Wait for fzstd#18 to land and release
  • Update manifest-parser.ts to read compressedLocation from ChunkRef
  • Update manifest-parser.ts to read locationDictionary/compressionAlgorithm from Manifest
  • Decompress locations at parse time (matching Rust impl), priority: compressed_location → location → chunk_id → inline
  • Add tests (will need synthetic fixtures or wait for upstream data)

References

  • Upstream schema: manifest.fbsChunkRef.compressed_location, Manifest.location_dictionary, Manifest.compression_algorithm

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions