Skip to content

refactor(audit): migrate audit store from KV to JetStream stream#342

Merged
retr0h merged 16 commits intomainfrom
refactor/audit-stream-migration
Apr 2, 2026
Merged

refactor(audit): migrate audit store from KV to JetStream stream#342
retr0h merged 16 commits intomainfrom
refactor/audit-stream-migration

Conversation

@retr0h
Copy link
Copy Markdown
Collaborator

@retr0h retr0h commented Apr 2, 2026

Summary

  • Replace audit KV bucket with a JetStream stream for chronological ordering and efficient sequence-based pagination (no more fetch-all-keys bottleneck)
  • Add trace_id field to audit entries for OpenTelemetry correlation — populated from span context in the audit middleware
  • New StreamStore implementation using jetstream.Stream for reads (GetLastMsgForSubject, OrderedConsumer, Info) and a Publisher interface for writes
  • Config changes: nats.audit.bucketstream, ttlmax_age, new subject field
  • Fix pre-existing test convention violations in audit_list_public_test.go (validateFunc pattern, defer cleanup)

Test plan

  • 100% coverage on stream_store.go (Write, Get, List, ListAll)
  • Trace ID branch coverage in middleware, handler, and SDK
  • All audit handler tests pass
  • All SDK audit tests pass
  • Build clean, lint clean (just go::vet)
  • No stale KV references remain in production code
  • Test conventions verified against CLAUDE.md guidelines

🤖 Generated with Claude Code

retr0h and others added 14 commits April 1, 2026 19:37
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Replace NATSAudit.Bucket/TTL with Stream/Subject/MaxAge fields to
reflect the migration from KV bucket to JetStream stream storage.
Remove audit from AllKVBuckets() and update tests accordingly.

🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Replace BuildAuditKVConfig with BuildAuditStreamConfig to build a
jetstream.StreamConfig instead of KeyValueConfig. Update nats_setup.go
to create an audit stream instead of a KV bucket, and update tests.

🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Add TraceID to audit entries across all layers: domain type, OpenAPI
spec, API handler mapping, middleware extraction, and SDK types. The
field is optional and populated from the OTel span context when
tracing is enabled.

🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Replace KVStore with StreamStore backed by JetStream stream. Write
publishes to per-entry subjects, Get uses GetLastMsgForSubject, and
List/ListAll use ordered consumers with newest-first pagination.

Co-Authored-By: Claude <noreply@anthropic.com>
Replace KV-based audit store wiring with stream-based approach.
The controller now creates a JetStream stream, resolves it by name,
and passes it to NewStreamStore along with the NATS client as publisher.

🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Update configuration.md and audit-logging.md to reflect the migration
from NATS KV bucket to JetStream stream. Config fields bucket→stream,
ttl→max_age, new subject field. Add trace_id to audit entry table.

🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use validateFunc callback in TestWrite instead of inline wantErr/errMsg
pattern. Rename suite receiver from 'suite' to 's' in SDK audit types
test to match project convention.

🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Rename validate to validateFunc in stream store tests and replace
wantCode/wantContains with validateFunc callbacks in HTTP tests.
Remove defer ctrl.Finish() in favor of explicit call at end of closure.

🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
retr0h and others added 2 commits April 1, 2026 23:05
The integration test osapi.yaml still had the old KV bucket config
(bucket, ttl) instead of stream config (stream, subject, max_age),
causing the audit store to not initialize and routes to return 404.

🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Add Trace ID to audit get output and truncated TRACE ID column to
audit list table. Update CLI docs with example output.

🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
@codecov
Copy link
Copy Markdown

codecov bot commented Apr 2, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.

Impacted file tree graph

@@           Coverage Diff           @@
##             main     #342   +/-   ##
=======================================
  Coverage   99.91%   99.91%           
=======================================
  Files         438      438           
  Lines       21205    21239   +34     
=======================================
+ Hits        21186    21220   +34     
  Misses         11       11           
  Partials        8        8           
Files with missing lines Coverage Δ
internal/audit/stream_store.go 100.00% <100.00%> (ø)
internal/cli/nats.go 100.00% <100.00%> (ø)
internal/config/nats.go 100.00% <ø> (ø)
internal/controller/api/audit/audit_list.go 100.00% <100.00%> (ø)
internal/controller/api/middleware_audit.go 100.00% <100.00%> (ø)
pkg/sdk/client/audit_types.go 100.00% <100.00%> (ø)

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 118483a...c64915d. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@retr0h retr0h merged commit ed04edc into main Apr 2, 2026
11 checks passed
@retr0h retr0h deleted the refactor/audit-stream-migration branch April 2, 2026 06:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant