Skip to content

chore: modify sql settings for performance #50

Open
tac0turtle wants to merge 1 commit intomainfrom
marko/sql
Open

chore: modify sql settings for performance #50
tac0turtle wants to merge 1 commit intomainfrom
marko/sql

Conversation

@tac0turtle
Copy link
Contributor

@tac0turtle tac0turtle commented Mar 19, 2026

Overview

Summary by CodeRabbit

Release Notes

  • Bug Fixes

    • Enhanced blob data integrity with improved conflict detection during insertion operations.
  • Performance Improvements

    • Optimized database configuration with increased caching and memory-based temporary storage for faster query execution.

@coderabbitai
Copy link

coderabbitai bot commented Mar 19, 2026

📝 Walkthrough

Walkthrough

The changes introduce a unique index constraint on the blobs table via a new SQL migration and update SQLite configuration pragmas (synchronous, cache_size, temp_store). Additionally, blob insertion logic transitions from eager pre-insert validation to deferred conflict detection using INSERT OR IGNORE with post-insert verification.

Changes

Cohort / File(s) Summary
SQL Migration
pkg/store/migrations/003_blob_index_unique.sql
Adds unique index on blobs(namespace, height, blob_index) and removes prior non-unique index idx_blobs_ns_height (idempotently).
SQLite Configuration & Blob Insertion
pkg/store/sqlite.go
Adds SQLite pragmas for performance tuning (synchronous=NORMAL, cache_size=-65536, temp_store=MEMORY). Refactors blob insertion from eager validation via ensureSQLiteBlobInvariant to deferred conflict detection: INSERT OR IGNORE followed by RowsAffected check and conditional verifyBlobNotConflicting call.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🐰 A unique index blooms, so neat and tight,
SQLite pragmas tuned for speed and light!
Blobs now slip in with defer-then-check grace,
No eager guards—just conflicts we'll trace.
Database hops forward, optimized and bright! ✨

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'chore: modify sql settings for performance' is partially related to the changeset. It accurately describes one aspect of the changes (SQL pragmas for performance), but omits the significant migration change (new unique index on blobs) which is equally important to the blob insertion logic changes.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch marko/sql
📝 Coding Plan
  • Generate coding plan for human review comments

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
pkg/store/migrations/003_blob_index_unique.sql (1)

1-2: The unique index constraint is enforced at the application level, reducing migration risk.

While INSERT OR IGNORE only prevents duplicates on (height, namespace, commitment) rather than the new (namespace, height, blob_index) key, the PutBlobs method enforces this constraint at runtime via verifyBlobNotConflicting (sqlite.go:395-412), which errors on any conflicts before they accumulate. This application-level validation makes the CREATE UNIQUE INDEX unlikely to fail, and the migration is already deployed in production without reported failures.

For extra defensive programming, a pre-flight check could validate that the constraint holds before applying the migration, but the current implementation with IF NOT EXISTS is reasonably safe given the runtime enforcement.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@pkg/store/migrations/003_blob_index_unique.sql` around lines 1 - 2, Add a
pre-flight validation before applying the CREATE UNIQUE INDEX in
003_blob_index_unique.sql: run a query to detect duplicates grouped by
(namespace, height, blob_index) (e.g. SELECT namespace, height, blob_index,
COUNT(*) FROM blobs GROUP BY namespace, height, blob_index HAVING COUNT(*) > 1)
and if any rows are returned, abort the migration (or log/resolve the
duplicates) so the CREATE UNIQUE INDEX IF NOT EXISTS cannot fail; tie this check
to your migration runner (or the migration script) and note that runtime
enforcement exists in verifyBlobNotConflicting in sqlite.go so the pre-flight is
defensive only.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@pkg/store/sqlite.go`:
- Around line 400-401: The error message uses %s for b.Namespace (a byte slice)
which can produce garbled output; update the fmt.Errorf call inside the conflict
check (the branch with existing != nil && !sameBlob(existing, b)) to format the
namespace as hex by replacing %s with %x so the call becomes fmt.Errorf("blob
conflict at height %d namespace %x index %d", b.Height, b.Namespace, b.Index).

---

Nitpick comments:
In `@pkg/store/migrations/003_blob_index_unique.sql`:
- Around line 1-2: Add a pre-flight validation before applying the CREATE UNIQUE
INDEX in 003_blob_index_unique.sql: run a query to detect duplicates grouped by
(namespace, height, blob_index) (e.g. SELECT namespace, height, blob_index,
COUNT(*) FROM blobs GROUP BY namespace, height, blob_index HAVING COUNT(*) > 1)
and if any rows are returned, abort the migration (or log/resolve the
duplicates) so the CREATE UNIQUE INDEX IF NOT EXISTS cannot fail; tie this check
to your migration runner (or the migration script) and note that runtime
enforcement exists in verifyBlobNotConflicting in sqlite.go so the pre-flight is
defensive only.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 45e5b916-5389-40f6-9bb3-2236f4716f40

📥 Commits

Reviewing files that changed from the base of the PR and between 401b59b and c746213.

📒 Files selected for processing (2)
  • pkg/store/migrations/003_blob_index_unique.sql
  • pkg/store/sqlite.go

Comment on lines +400 to 401
if existing != nil && !sameBlob(existing, b) {
return fmt.Errorf("blob conflict at height %d namespace %s index %d", b.Height, b.Namespace, b.Index)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Use hex formatting for namespace in error message.

b.Namespace is a byte array. Using %s may produce non-printable or garbled output. Use %x for consistent hex representation.

Proposed fix
 	if existing != nil && !sameBlob(existing, b) {
-		return fmt.Errorf("blob conflict at height %d namespace %s index %d", b.Height, b.Namespace, b.Index)
+		return fmt.Errorf("blob conflict at height %d namespace %x index %d", b.Height, b.Namespace, b.Index)
 	}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if existing != nil && !sameBlob(existing, b) {
return fmt.Errorf("blob conflict at height %d namespace %s index %d", b.Height, b.Namespace, b.Index)
if existing != nil && !sameBlob(existing, b) {
return fmt.Errorf("blob conflict at height %d namespace %x index %d", b.Height, b.Namespace, b.Index)
}
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@pkg/store/sqlite.go` around lines 400 - 401, The error message uses %s for
b.Namespace (a byte slice) which can produce garbled output; update the
fmt.Errorf call inside the conflict check (the branch with existing != nil &&
!sameBlob(existing, b)) to format the namespace as hex by replacing %s with %x
so the call becomes fmt.Errorf("blob conflict at height %d namespace %x index
%d", b.Height, b.Namespace, b.Index).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant