vault: backport instrumentation and observation blob parallelization to 2.40.0 by prashantkumar1982 · Pull Request #21761 · smartcontractkit/chainlink

prashantkumar1982 · 2026-03-27T20:51:49Z

Summary

This backports the Vault instrumentation and Observation blob broadcast parallelization changes to release/2.40.0.

What changed

add KV store operation duration metrics to the Vault plugin
instrument the KV and blob broadcaster/fetcher wrappers
parallelize pending-queue blob broadcasts during Observation()
add debug logs for overall observation duration and blob broadcast duration
keep focused unit coverage for the parallel broadcast path and pending-queue validation
include small release-branch import adjustments needed for this backport

Testing

go test ./core/services/ocr2/plugins/vault -run 'TestPlugin_Observation_PendingQueueEnabled_(NoPendingQueueProvided|WithPendingQueueProvided|ItemBothInPendingQueueAndLocalQueue|BroadcastsPendingQueueBlobsInParallel|BroadcastBlobError)$|^TestPlugin_ValidateObservation_AcceptsFullPendingQueueObservation$' -count=1

Instrument all public KVStore methods with a histogram metric (platform_vault_plugin_kv_operation_duration_seconds) to help diagnose high P95 Observation latencies in production. Also adds a max-batch Observation test exercising 10 GetSecrets requests with 10 encryption keys each plus 20 pending queue items.

github-actions · 2026-03-27T20:52:33Z

✅ No conflicts with other open PRs targeting release/2.40.0

github-actions · 2026-03-27T20:52:51Z

I see you updated files related to core. Please run make gocs in the root directory to add a changeset as well as in the text include at least one of the following tags:

#added For any new functionality added.
#breaking_change For any functionality that requires manual action for the node to boot.
#bugfix For bug fixes.
#changed For any change to the existing functionality.
#db_update For any feature that introduces updates to database schema.
#deprecation_notice For any upcoming deprecation functionality.
#internal For changesets that need to be excluded from the final changelog.
#nops For any feature that is NOP facing and needs to be in the official Release Notes for the release.
#removed For any functionality/config that is removed.
#updated For any functionality that is updated.
#wip For any change that is not ready yet and external communication about it should be held off till it is feature complete.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 61526464b2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-03-27T20:55:52Z

core/services/ocr2/plugins/vault/plugin.go

-		batchSize.DefaultValue = int(configProto.BatchSize)
-	}
-	cfg.MaxBatchSize, err = limits.MakeUpperBoundLimiter(r.limitsFactory, batchSize)
+	cfg.MaxBatchSize, err = limits.MakeUpperBoundLimiter(r.limitsFactory, cresettings.Default.VaultPluginBatchSizeLimit)


Preserve offchain BatchSize override when creating limiter

NewReportingPlugin now always builds cfg.MaxBatchSize from cresettings.Default.VaultPluginBatchSizeLimit, even though it has already unmarshaled configProto and still honors other per-plugin overrides (like MaxSecretsPerOwner). This silently ignores ReportingPluginConfig.BatchSize, so deployments that set a non-default batch size will run with the default limit and can unexpectedly truncate pending queue observations (2*MaxBatchSize) or diverge from intended configuration.

Useful? React with 👍 / 👎.

trunk-io · 2026-03-27T21:07:41Z

_{View Full Report ↗︎ ⋅ Docs}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 54be050ca4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-03-30T18:25:39Z

core/services/ocr2/plugins/vault/plugin.go

-		batchSize.DefaultValue = int(configProto.BatchSize)
-	}
-	cfg.MaxBatchSize, err = limits.MakeUpperBoundLimiter(r.limitsFactory, batchSize)
+	cfg.MaxBatchSize, err = limits.MakeUpperBoundLimiter(r.limitsFactory, cresettings.Default.VaultPluginBatchSizeLimit)


Honor BatchSize from offchain config in limiter setup

NewReportingPlugin now always builds cfg.MaxBatchSize from cresettings.Default.VaultPluginBatchSizeLimit, so ReportingPluginConfig.BatchSize is ignored even when it is explicitly set in offchain config. This changes runtime behavior for deployed jobs that rely on non-default batch sizing, and can cause unexpected truncation/rejection paths tied to MaxBatchSize (for example, pending-queue observation limits at 2*MaxBatchSize).

Useful? React with 👍 / 👎.

cl-sonarqube-production · 2026-03-30T18:39:50Z

Quality Gate passed

Issues
1 New issue
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
1.4% Duplication on New Code

See analysis details on SonarQube

prashantkumar1982 and others added 14 commits March 27, 2026 13:47

vault: parallelize observation blob broadcasts

6b2c389

vault: fix test lint shadowing

7d34ef8

vault: log observation timing

ed0eaa3

Increase buckets to account for production usage

6b903b8

Instrument KV and BlobBroadcaster/Fetcher interfaces

e1a433a

Instrument KV and BlobBroadcaster/Fetcher interfaces

b35fa9e

Go fmt

918cbcc

Track KV durations as ms

8fadcef

Remove test

75ec862

Go fmt

3212114

vault: guard nil test metrics

e17c2d8

vault: restore plugin test import

7df2a96

vault: fix release 2.40 imports

6152646

prashantkumar1982 requested review from a team as code owners March 27, 2026 20:51

product-security-plaid-production bot requested review from Atrax1 and jinhoonbang March 27, 2026 20:52

chatgpt-codex-connector bot reviewed Mar 27, 2026

View reviewed changes

cedric-cordenier approved these changes Mar 27, 2026

View reviewed changes

bolekk approved these changes Mar 27, 2026

View reviewed changes

prashantkumar1982 enabled auto-merge (squash) March 27, 2026 21:20

justinkaseman approved these changes Mar 27, 2026

View reviewed changes

prashantkumar1982 added 2 commits March 27, 2026 20:11

chore: trigger CI

bc6f4fc

chore: trigger CI

54be050

auto-merge was automatically disabled March 30, 2026 18:25
Pull Request is not mergeable

chatgpt-codex-connector bot reviewed Mar 30, 2026

View reviewed changes

prashantkumar1982 enabled auto-merge (squash) March 30, 2026 18:27

prashantkumar1982 disabled auto-merge March 30, 2026 18:48

prashantkumar1982 enabled auto-merge (squash) March 30, 2026 18:48

prashantkumar1982 added the run-e2e-tests Use on your PR to run E2E tests before the merge queue label Mar 30, 2026

prashantkumar1982 temporarily deployed to integration March 30, 2026 19:11 — with GitHub Actions Inactive

prashantkumar1982 temporarily deployed to integration March 30, 2026 19:12 — with GitHub Actions Inactive

prashantkumar1982 temporarily deployed to integration March 30, 2026 19:18 — with GitHub Actions Inactive

prashantkumar1982 had a problem deploying to integration March 30, 2026 19:18 — with GitHub Actions Failure

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

vault: backport instrumentation and observation blob parallelization to 2.40.0#21761

vault: backport instrumentation and observation blob parallelization to 2.40.0#21761
prashantkumar1982 wants to merge 16 commits intorelease/2.40.0from
codex/vault-release-2.40.0

prashantkumar1982 commented Mar 27, 2026

Uh oh!

github-actions bot commented Mar 27, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Mar 27, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

chatgpt-codex-connector bot Mar 27, 2026

Uh oh!

trunk-io bot commented Mar 27, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

chatgpt-codex-connector bot Mar 30, 2026

Uh oh!

cl-sonarqube-production bot commented Mar 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

prashantkumar1982 commented Mar 27, 2026

Summary

What changed

Testing

Uh oh!

github-actions bot commented Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 27, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

trunk-io bot commented Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

cl-sonarqube-production bot commented Mar 30, 2026

Quality Gate passed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions bot commented Mar 27, 2026 •

edited

Loading

trunk-io bot commented Mar 27, 2026 •

edited

Loading