fix(security): block S3 bucket takeover via unsafe example defaults

[kamir]

Summary

A security researcher (Milan Katwal) registered the kafscale-lfs S3 bucket on AWS — the exact name used as an example in our OpenAPI specs. Any deployment using this default silently routes all LFS uploads to an attacker-controlled bucket.

Impact: data exfiltration, content tampering via presigned URLs, integrity bypass (no checksum on download).

This PR fixes the vulnerability with three defense layers:

• Remove unsafe examples from both OpenAPI specs and Go doc/test files
• Runtime blocklist that rejects known-compromised bucket names at startup
• Required field enforcement — S3_BUCKET and S3_REGION must now be explicitly set

Why a blocklist?

Removing the examples from the spec fixes future copy-paste. But the bucket name kafscale-lfs has been public in this repo since commit 1b5dbd8. Every deployment that was configured by copying from those docs — Helm values, docker-compose files, CI scripts, customer PoCs — may still have kafscale-lfs hardwired in config files.

The blocklist is a runtime safety net for existing deployments: even if someone upgrades the proxy binary but doesn't touch their config, the proxy refuses to start instead of silently routing data to an attacker-controlled bucket.

The bucket name is permanently compromised — it can never be safe again, regardless of who controls it now. The code must reject it forever. This is the same pattern used by credential scanners that block known-leaked API keys even after rotation.

Changes

File	Change
cmd/proxy/lfs.go	Add blockedBucketNames slice + validation in initLFSModule(). Require S3_BUCKET and S3_REGION to be non-empty.
cmd/proxy/openapi.yaml	Replace 6× kafscale-lfs → my-bucket
api/lfs-proxy/openapi.yaml	Replace 6× kafscale-lfs → my-bucket
pkg/lfs/doc.go	Replace 2× kafscale-lfs → my-bucket in doc examples
pkg/lfs/envelope_test.go	Replace test fixture bucket → test-bucket

Breaking change

Deployments that relied on an empty KAFSCALE_LFS_PROXY_S3_BUCKET will now fail at startup with a clear error message. This is intentional — an empty bucket was never valid, it just failed silently downstream.

Test plan

• go build ./cmd/proxy/ passes
• go vet ./pkg/lfs/ passes
• go test ./pkg/lfs/... passes
• Start proxy with KAFSCALE_LFS_PROXY_S3_BUCKET=kafscale-lfs → exits with blocklist error
• Start proxy with empty KAFSCALE_LFS_PROXY_S3_BUCKET → exits with "required" error
• Start proxy with valid bucket + region → starts normally
• Grep confirms no remaining kafscale-lfs bucket references (only in blocklist)

Related

• Reported via email to security@novatechflow.com on 2026-04-17
• Follows up on broader config hardening plan: scalytics-all-in-one/PLANS/plan-001.md

🤖 Generated with Claude Code

[kamir]

Design Notes (preserved from PR preparation)

Why a blocklist?

Removing the examples from the spec fixes future copy-paste. But the bucket name kafscale-lfs has been public in this repo since commit 1b5dbd8. Every deployment that was configured by copying from those docs — Helm values, docker-compose files, CI scripts, customer PoCs — may still have kafscale-lfs hardwired in config files.

The blocklist is a runtime safety net for existing deployments: even if someone upgrades the proxy binary but doesn't touch their config, the proxy refuses to start instead of silently routing data to an attacker-controlled bucket.

The bucket name is permanently compromised — it can never be safe again, regardless of who controls it now. The code must reject it forever. This is the same pattern used by credential scanners that block known-leaked API keys even after rotation.

Breaking change

Deployments that relied on an empty KAFSCALE_LFS_PROXY_S3_BUCKET will now fail at startup with a clear error message. This is intentional — an empty bucket was never valid, it just failed silently downstream.

Test plan

• go build ./cmd/proxy/ passes
• go vet ./pkg/lfs/ passes
• go test ./pkg/lfs/... passes
• Start proxy with KAFSCALE_LFS_PROXY_S3_BUCKET=kafscale-lfs → exits with blocklist error
• Start proxy with empty KAFSCALE_LFS_PROXY_S3_BUCKET → exits with "required" error
• Start proxy with valid bucket + region → starts normally
• Grep confirms no remaining kafscale-lfs bucket references (only in blocklist)

Context

This fix addresses a reported S3 bucket takeover vulnerability. A security researcher registered the kafscale-lfs bucket on AWS — the exact name used as examples in our OpenAPI specs. The broader config hardening plan (centralized config package, startup validation for all binaries, console auth fix) is tracked in scalytics-all-in-one/PLANS/plan-001.md.

[kamir]

Scope expansion — adding LFS download integrity fix

Per review feedback, consolidating follow-up fixes into this PR rather than opening a separate one.

Additional changes being added:

LFS download integrity verification — closes the tampering vector that complements the exfiltration fix already in this PR.

Current behavior (still vulnerable after just the blocklist fix):

• Any attacker/insider with S3 write access can tamper with LFS objects post-upload
• Neither stream nor presign mode verifies the envelope's sha256 checksum

Planned fix:

• Stream mode becomes default. Hashes bytes on the fly (io.TeeReader), compares to envelope checksum, aborts TCP connection on mismatch (no trailers — poor client support). Emits integrity_failure event to __lfs_ops_state tracker.
• Presign mode is disabled by default. Opt-in via KAFSCALE_LFS_PROXY_PRESIGN_ENABLED=true. When enabled, proxy logs a warning banner at startup and every 5 minutes. Response body includes integrity block with sha256/checksum_alg/size so SDK can verify client-side.
• Advisory response headers on stream mode: X-Kafscale-LFS-Checksum and X-Kafscale-LFS-Content-Length — let advanced clients hash independently and fail-fast without waiting for connection close.
• Updated OpenAPI spec reflects new default, presign_disabled error code, integrity block.

Trust model being enforced:

Kafka is the authority. S3 is untrusted storage.

The envelope lives in Kafka (attacker can't modify it). Downloaded bytes from S3 are verified against the Kafka-held checksum on every read.

Breaking change (intentional):

Default mode changes from presign → stream. Existing clients that hardcoded mode: presign must also have the operator set KAFSCALE_LFS_PROXY_PRESIGN_ENABLED=true. This is secure-by-default by design.

Full spec

Detailed API behavior, acceptance criteria, and out-of-scope items are in the forthcoming commit. The broader platform-wide integrity audit (broker segment CRC, RecordBatch CRC32C, S3 checksums) is tracked separately in scalytics-all-in-one/PLANS/plan-002.md and will be addressed in follow-up PRs.

Pushing commits shortly.

[novatechflow]

streamDownloadWithVerify writes the full body with io.Copy and sets Content-Length before comparing the digest. If the object is tampered but keeps the same size, the client can still receive a complete 200 OK response and treat it as success. Closing the connection afterward does not reliably turn that into a failed download.

The patch does not guarantee “truncated response on mismatch”, verification has to happen before sending the response, or the protocol needs an explicit post-body failure signal.