Skip to content

feat: Large File Support — proxy, operator, SDKs, and demos#128

Closed
kamir wants to merge 34 commits intoKafScale:mainfrom
kamir:lfs-core
Closed

feat: Large File Support — proxy, operator, SDKs, and demos#128
kamir wants to merge 34 commits intoKafScale:mainfrom
kamir:lfs-core

Conversation

@kamir
Copy link
Copy Markdown
Collaborator

@kamir kamir commented Mar 3, 2026

Summary

Complete LFS (Large File Support) subsystem for streaming large binary payloads through KafScale via S3-backed object storage with Kafka metadata references.

Core LFS Proxy (119 files)

  • LFS proxy server (cmd/lfs-proxy/) — HTTP upload/download, S3 backend, Kafka record publishing, Prometheus metrics, checksum verification, TLS/SASL support, OpenAPI spec
  • LFS package (pkg/lfs/) — shared types, producer, consumer, resolver, envelope, and S3 client logic
  • Operator integration — reconcile LFS proxy Deployment, Service, and ConfigMap as part of KafscaleCluster CR
  • Console LFS admin (internal/console/) — object/topic/orphan browsing handlers
  • IDoc exploder (cmd/idoc-explode/) — SAP IDoc payload processing with field aggregation into routed parent segments
  • Iceberg processor — LFS-aware decoding and config extensions
  • Helm charts — LFS proxy deployment, metrics, monitoring, and ServiceMonitor templates
  • E2E tests — LFS proxy broker, HTTP, etcd, SDK, and iceberg processor tests
  • CI pipeline — Docker build for LFS proxy image, stage-release workflow
  • Documentation — architecture, data-flow, security hardening, Helm deployment, SDK roadmap

Client SDKs & Demos (97 files)

  • Java SDK — retry/backoff, configurable timeouts, Maven build
  • Python SDK — retry/backoff, envelope codec, pip-installable
  • JavaScript SDK (Node.js) — streaming upload support
  • Browser SDK (JS) — SPA-based LFS file management
  • Demos — E60-E62 (industry designs), E70 (Java), E71 (Python), E72 (browser SPA)
  • Docker Compose stack for local LFS development

Fixes included

  • Fix missing time import in lfs_handlers.go
  • Fix Makefile script name mismatches (lfs-demo-medical.shmedical-lfs-demo.sh)
  • Add missing LFS demo scripts not included in initial branch split
  • Fix wrong LFS envelope field names in demo scripts (lfs_versionkfs_lfs)
  • Enrich IDoc exploder with Fields map for parent segment field aggregation
  • Enrich IDoc demo with full LFS pipeline (S3 upload → envelope → resolve → explode)
  • Add Apache 2.0 license headers to 12 files
  • Fix S3 blob verification in medical/video/industrial demos (*.metaobj-*)

Test plan

  • go vet ./... passes
  • go test -race ./... passes
  • go build ./cmd/lfs-proxy passes
  • License header check passes (hack/check_license_headers.py)
  • Helm lint passes (helm lint deploy/helm/kafscale)
  • Coverage gate passes (80% floor)
  • E2E tests with MinIO + broker (requires cluster infrastructure)
  • Helm chart deployment validation on kind cluster

🤖 Generated with Claude Code

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants