fix: use jq instead of sed for JSON field stripping in gstack-telemetry-sync#749
Open
Gonzih wants to merge 1 commit intogarrytan:mainfrom
Open
fix: use jq instead of sed for JSON field stripping in gstack-telemetry-sync#749Gonzih wants to merge 1 commit intogarrytan:mainfrom
Gonzih wants to merge 1 commit intogarrytan:mainfrom
Conversation
The sed pattern [^"]* fails on JSON field values containing escaped quotes (\"...\") or other characters that break the regex match. This produces silently corrupt JSON that the Supabase edge function rejects without error — events are lost and the cursor doesn't advance. Fix: use jq del() for proper JSON-aware field removal. Fall back to the existing sed approach when jq is unavailable (older installs without the dev toolchain). jq is already a required dependency (the Dockerfile installs it, and gstack already uses it in other scripts). Closes garrytan#710
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Bug
`gstack-telemetry-sync` strips local-only fields before sending events to Supabase using sed:
```bash
CLEAN="$(echo "$LINE" | sed
-e 's/,"_repo_slug":"[^"]"//g'
-e 's/,"_branch":"[^"]"//g'
-e 's/,"repo":"[^"]*"//g')"
```
The `[^"]*` pattern breaks on JSON string values containing escaped quotes (`\"`) or regex metacharacters. This produces corrupt JSON. The Supabase edge function returns a non-2xx response (or 0 inserted), the cursor doesn't advance, and the same events get retried on next sync.
Issue #710.
Fix
Use `jq del()` for proper JSON-aware field removal. The sed code is kept as a fallback for older installs without `jq`. `jq` is already a required dependency (installed in `Dockerfile.ci` and used elsewhere in the codebase).
sent from mStack