feat: traced pipeline outputs and persisted execution traces#557
Open
feat: traced pipeline outputs and persisted execution traces#557
Conversation
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughRemoved the internal Changes
Sequence Diagram(s)sequenceDiagram
participant Client as Client
participant Executor as Pipeline Executor
participant Source as Source Adapter
participant Route as Route Runtime
participant Outputs as Output Materializer
participant Traces as Trace Emitter
participant DB as Database
Client->>Executor: run(pipeline, options)
Executor->>Executor: resolveVersions(), buildRoutesByLayer()
Executor->>Traces: createTraceEmitter(onTrace)
Executor->>Source: createSourceAdapter(pipeline, priorResults)
Source->>DB: listFiles(version)
DB-->>Source: FileContext[]
loop per route layer
loop per file
Route->>Source: readFile(file)
Source-->>Route: content
Route->>Route: parse/filter/transform
Route->>Route: resolver(ctx, rows)
Route-->>Executor: produced values
Executor->>Outputs: materializeOutputs(values, outputsDefs)
Outputs->>Outputs: resolveOutputDestination()
Outputs->>Traces: emit(output.resolved)
Outputs->>Executor: writeOutputToSink(...)
Outputs->>Traces: emit(output.written)
end
end
Executor->>Traces: collect traces
Traces->>DB: insert execution_traces
Executor-->>Client: PipelineExecutionResult { traces, outputManifest, status }
sequenceDiagram
participant Browser as Browser Client
participant Server as Pipeline Server
participant DB as Database
participant Traces as Trace Query
Browser->>Server: GET /traces (executionId)
Server->>DB: SELECT execution WHERE id=...
DB-->>Server: execution
alt execution_traces table exists
Server->>DB: SELECT * FROM execution_traces WHERE executionId=...
DB-->>Server: trace rows[]
Server->>Traces: buildOutputManifestFromTraces(rows)
Traces-->>Server: outputManifest
Server-->>Browser: { traces, outputManifest, pagination }
else
Server-->>Browser: { traces: [], outputManifest: [], pagination }
end
Estimated code review effort🎯 5 (Critical) | ⏱️ ~120 minutes Possibly related PRs
Suggested labels
✨ Finishing Touches🧪 Generate unit tests (beta)
|
Contributor
🌏 Preview Deployments
Built from commit: 🤖 This comment will be updated automatically when you push new commits to this PR. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
luxass
force-pushed
the
codex/pipeline-outputs
branch
2 times, most recently
from
2b7bf47 to
295d73d
Compare
Shrink the executor root exports to the supported execution API and move trace projection access off the root entrypoint. Rename the internal trace emitter module so the package no longer has two different traces.ts files serving unrelated roles. Verification: - corepack pnpm exec tsc -p packages/pipelines/pipeline-executor/tsconfig.json --noEmit - corepack pnpm exec vitest run --project=pipeline-server packages/pipelines/pipeline-server/test/server/sources.traces.test.ts Note: packages/pipelines/pipeline-server/test/server/sources.logs.test.ts still has a pre-existing type error in the broader pipeline-server typecheck.
Remove the public memory sink and make runtime-only outputs the default when no sink is configured. Filesystem sinks remain the persistence mechanism, while runtime-only outputs continue to use internal memory:// locators in traces and manifests. Verification: - corepack pnpm exec vitest run --project=pipeline-core packages/pipelines/pipeline-core/test/output.test.ts - corepack pnpm exec vitest run --project=pipeline-executor packages/pipelines/pipeline-executor/test/traces.test.ts packages/pipelines/pipeline-executor/test/graph.test.ts - corepack pnpm exec tsc -p packages/pipelines/pipeline-executor/tsconfig.json --noEmit
Stop persisting execution graph payloads and derive runtime graph availability from execution traces instead. The graph route now rebuilds execution graphs from the stored trace stack, overview and execution listings compute graph availability from trace presence, and the database schema drops the duplicated executions.graph column. Verification: - corepack pnpm exec tsc -p packages/pipelines/pipeline-executor/tsconfig.json --noEmit - corepack pnpm exec tsc -p packages/pipelines/pipeline-server/tsconfig.json --noEmit - corepack pnpm exec vitest run --project=pipeline-server packages/pipelines/pipeline-server/test/server/sources.graph.test.ts packages/pipelines/pipeline-server/test/server/sources.traces.test.ts packages/pipelines/pipeline-server/test/server/overview.test.ts
Turn run-pipeline into a thin coordinator and move the execution hot path into focused helpers for version execution, output materialization, and final result projection. This keeps behavior intact while centralizing output tracing/writes and removing repeated manifest scans in the trace projection layer. Verification: - corepack pnpm exec tsc -p packages/pipelines/pipeline-executor/tsconfig.json --noEmit - corepack pnpm exec vitest run --project=pipeline-executor packages/pipelines/pipeline-executor/test/executor.test.ts packages/pipelines/pipeline-executor/test/traces.test.ts packages/pipelines/pipeline-executor/test/graph.test.ts
Add focused executor tests for output utilities, source adaptation, node runtime context/log capture, fallback output manifests, failed sink writes, and cache trace transitions. These tests harden the trace-driven output and runtime model introduced in the earlier stages while keeping the server-side trace-derived graph checks green. Verification: - corepack pnpm exec vitest run --project=pipeline-executor packages/pipelines/pipeline-executor/test/outputs.test.ts packages/pipelines/pipeline-executor/test/source-adapter.test.ts packages/pipelines/pipeline-executor/test/runtime.node.test.ts packages/pipelines/pipeline-executor/test/traces.test.ts packages/pipelines/pipeline-executor/test/graph.test.ts - corepack pnpm exec tsc -p packages/pipelines/pipeline-executor/tsconfig.json --noEmit - corepack pnpm exec vitest run --project=pipeline-server packages/pipelines/pipeline-server/test/server/sources.graph.test.ts packages/pipelines/pipeline-server/test/server/sources.traces.test.ts packages/pipelines/pipeline-server/test/server/overview.test.ts - corepack pnpm exec tsc -p packages/pipelines/pipeline-server/tsconfig.json --noEmit
Move executor internals out of the nested src/executor folder into top-level modules and rename the main flow file to run.ts. This stage is structural only: the execution behavior stays the same while the file layout becomes flatter and easier to navigate before the deeper cleanup. Verification: - corepack pnpm exec tsc -p packages/pipelines/pipeline-executor/tsconfig.json --noEmit - corepack pnpm exec vitest run --project=pipeline-executor packages/pipelines/pipeline-executor/test/executor.test.ts packages/pipelines/pipeline-executor/test/traces.test.ts packages/pipelines/pipeline-executor/test/graph.test.ts packages/pipelines/pipeline-executor/test/source-adapter.test.ts
Collapse the executor back around a single run entrypoint and remove the phase/materialization glue files. This keeps route parsing in route-runtime, source virtualization in source-files, and moves output and artifact details behind smaller helpers so the main flow is easier to follow without behavior changes. Verified with: - corepack pnpm exec tsc -p packages/pipelines/pipeline-executor/tsconfig.json --noEmit - corepack pnpm exec vitest run --project=pipeline-executor packages/pipelines/pipeline-executor/test/executor.test.ts packages/pipelines/pipeline-executor/test/traces.test.ts packages/pipelines/pipeline-executor/test/graph.test.ts packages/pipelines/pipeline-executor/test/source-adapter.test.ts
Move the tiny runtime event wrapper and fallback output default into the modules that already own those concepts, and trim a little more executor-only glue from run.ts. This keeps the main flow focused on orchestration while outputs and events own their small shared details. Runtime behavior stays the same. Verified with: - corepack pnpm exec tsc -p packages/pipelines/pipeline-executor/tsconfig.json --noEmit - corepack pnpm exec vitest run --project=pipeline-executor packages/pipelines/pipeline-executor/test/executor.test.ts packages/pipelines/pipeline-executor/test/traces.test.ts packages/pipelines/pipeline-executor/test/graph.test.ts packages/pipelines/pipeline-executor/test/source-adapter.test.ts
Add direct tests for output materialization and trace projections so executor behavior is asserted through structured trace facts and the manifest built from them. Verified with: - corepack pnpm exec vitest run --project=pipeline-executor packages/pipelines/pipeline-executor/test/executor.test.ts packages/pipelines/pipeline-executor/test/traces.test.ts packages/pipelines/pipeline-executor/test/graph.test.ts packages/pipelines/pipeline-executor/test/outputs.test.ts packages/pipelines/pipeline-executor/test/source-adapter.test.ts packages/pipelines/pipeline-executor/test/runtime.node.test.ts packages/pipelines/pipeline-executor/test/trace-projections.test.ts - corepack pnpm exec tsc -p packages/pipelines/pipeline-executor/tsconfig.json --noEmit - corepack pnpm exec vitest run --project=pipeline-server packages/pipelines/pipeline-server/test/server/sources.graph.test.ts packages/pipelines/pipeline-server/test/server/sources.traces.test.ts packages/pipelines/pipeline-server/test/server/overview.test.ts The pipeline-server typecheck command is still blocked by an existing inspect UI type error in packages/pipelines/pipeline-server/src/client/routes/s.///inspect/outputs.tsx.
Group the private run helper files under src/run so the package root keeps the public and top-level concepts visible while the orchestration internals sit together. This is a layout-only cleanup: run.ts stays the entrypoint, the public API stays the same, and behavior is unchanged. Verified with: - corepack pnpm exec tsc -p packages/pipelines/pipeline-executor/tsconfig.json --noEmit - corepack pnpm exec vitest run --project=pipeline-executor packages/pipelines/pipeline-executor/test/executor.test.ts packages/pipelines/pipeline-executor/test/traces.test.ts packages/pipelines/pipeline-executor/test/graph.test.ts packages/pipelines/pipeline-executor/test/outputs.test.ts packages/pipelines/pipeline-executor/test/source-adapter.test.ts packages/pipelines/pipeline-executor/test/runtime.node.test.ts packages/pipelines/pipeline-executor/test/trace-projections.test.ts
Move the internal observability and output support modules under src/internal so the package root shows the public layer and the two main implementation areas more clearly. The result is a cleaner top-level layout: public and exported files stay at the root, run orchestration lives in src/run, and private support code lives in src/internal. Verified with: - corepack pnpm exec tsc -p packages/pipelines/pipeline-executor/tsconfig.json --noEmit - corepack pnpm exec vitest run --project=pipeline-executor packages/pipelines/pipeline-executor/test/executor.test.ts packages/pipelines/pipeline-executor/test/traces.test.ts packages/pipelines/pipeline-executor/test/graph.test.ts packages/pipelines/pipeline-executor/test/outputs.test.ts packages/pipelines/pipeline-executor/test/source-adapter.test.ts packages/pipelines/pipeline-executor/test/runtime.node.test.ts packages/pipelines/pipeline-executor/test/trace-projections.test.ts
Replace the shared run state bag with an explicit RunContext split into config and mutable run data, use PipelineLogger directly, and collapse trace emission to a single emitTrace primitive. This also finishes the file layout cleanup by moving the private executor support modules into src/run so the package root stays focused on the public layer. Verified with: - corepack pnpm exec tsc -p packages/pipelines/pipeline-executor/tsconfig.json --noEmit - corepack pnpm exec vitest run --project=pipeline-executor packages/pipelines/pipeline-executor/test/executor.test.ts packages/pipelines/pipeline-executor/test/traces.test.ts packages/pipelines/pipeline-executor/test/graph.test.ts packages/pipelines/pipeline-executor/test/outputs.test.ts packages/pipelines/pipeline-executor/test/source-adapter.test.ts packages/pipelines/pipeline-executor/test/runtime.node.test.ts packages/pipelines/pipeline-executor/test/trace-projections.test.ts packages/pipelines/pipeline-executor/test/run.test.ts packages/pipelines/pipeline-executor/test/trace-emitter.test.ts
Add focused tests for the run setup helpers and for span-aware trace emission now that RunContext only exposes a single emitTrace primitive. Verified with: - corepack pnpm exec tsc -p packages/pipelines/pipeline-executor/tsconfig.json --noEmit - corepack pnpm exec vitest run --project=pipeline-executor packages/pipelines/pipeline-executor/test/executor.test.ts packages/pipelines/pipeline-executor/test/traces.test.ts packages/pipelines/pipeline-executor/test/graph.test.ts packages/pipelines/pipeline-executor/test/outputs.test.ts packages/pipelines/pipeline-executor/test/source-adapter.test.ts packages/pipelines/pipeline-executor/test/runtime.node.test.ts packages/pipelines/pipeline-executor/test/trace-projections.test.ts packages/pipelines/pipeline-executor/test/run.test.ts packages/pipelines/pipeline-executor/test/trace-emitter.test.ts
Move the support-only executor modules back under src/internal while keeping execution-domain code in src/run. This keeps events, logger, and trace emitter as plumbing details, while outputs, traces, graph projection, source handling, and route runtime remain part of the execution model. Verified with: - corepack pnpm exec tsc -p packages/pipelines/pipeline-executor/tsconfig.json --noEmit - corepack pnpm exec vitest run --project=pipeline-executor packages/pipelines/pipeline-executor/test/executor.test.ts packages/pipelines/pipeline-executor/test/traces.test.ts packages/pipelines/pipeline-executor/test/graph.test.ts packages/pipelines/pipeline-executor/test/outputs.test.ts packages/pipelines/pipeline-executor/test/source-adapter.test.ts packages/pipelines/pipeline-executor/test/runtime.node.test.ts packages/pipelines/pipeline-executor/test/trace-projections.test.ts packages/pipelines/pipeline-executor/test/run.test.ts packages/pipelines/pipeline-executor/test/trace-emitter.test.ts
- Merged `InspectOutputsPanel` and `InspectTransformsPanel` into a single `RouteComponent` for better maintainability. - Updated routing logic to streamline the inspection of outputs and transforms. - Removed redundant files and adjusted imports accordingly. - Enhanced the UI to provide a more cohesive experience when inspecting pipeline elements.
Point the executor build entries and workspace path aliases at the current graph and trace modules under src/run. Verified with: - corepack pnpm exec tsc -p packages/pipelines/pipeline-executor/tsconfig.build.json --noEmit - corepack pnpm --filter @ucdjs/pipelines-executor run build The tsdown bundle completed successfully; the remaining failure comes from the final pack step spawning pnpm without it on PATH in that subprocess.
Add isSourceFileContext type guard, split ArtifactTraceRecord and CacheTraceRecord into per-kind interfaces for distributive Omit, make trace emitter non-generic, and replace NodeWriteFn with an explicit WriteFunction interface. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Inline emitWithSpan into emitRuntimeEvent and remove the export. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extract shared executeParseResolve function handling the parse → filter → transform → resolve cycle. Both processRoute and processFallback are now thin wrappers. Replace inline resolve context in processFallback with createRouteResolveContext. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…raction Move node:fs/promises and node:path out of core src/ files into the Node runtime implementation. Add writeOutput and resolvePath as optional methods on PipelineExecutionRuntime, implemented in NodeExecutionRuntime. Replace node:path usage in source-files.ts with pure string utils in path-utils.ts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Replace vi.mock router/query with real renderFileRoute + mockFetch - Move tests to match source folder structure (home/ → overview/, routes/ → components/app/) - Apply beforeEach mockFetch pattern with parameterised :sourceId routes - Fix stale stream field removed from ExecutionLogItem schema - Fix stale artifact:produced → file:matched event type assertions - Add missing outputs.id/sink/format fields to route test fixtures - Activate pipeline-header and pipeline-sidebar todos as real tests
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
packages/pipelines/pipeline-server/test/browser/route-test-utils.tsx (1)
83-87: Avoid blanket suppression of router load failures in this shared helper.At Line 86, swallowing all load errors can mask broken route setup and produce false-positive tests. Prefer failing only when there is no route match, while still tolerating expected loader failures.
Proposed tighten-up
- await router.load().catch(() => {}); + try { + await router.load(); + } catch { + if (router.state.matches.length === 0) { + throw new Error(`No route match for initialLocation: ${initialLocation}`); + } + }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/pipelines/pipeline-server/test/browser/route-test-utils.tsx` around lines 83 - 87, The helper currently swallows all errors from router.load(), which can hide broken route setups; change the catch so it only suppresses expected loader failures but rethrows when the router has no matching route (i.e. when router.matches is empty or router.state indicates no match). Concretely, after calling router.load() (the existing router.load() call used to populate useParams), catch the error, inspect router.matches (or router.state.matches) and rethrow the error if there are zero matches (indicating a route setup problem), otherwise ignore the error as a loader failure.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.vscode/settings.json:
- Around line 50-51: Replace the non-standard keys "ts.tsdk" and
"ts.experimental.useTsgo" with the unified JS/TS namespace so VS Code recognizes
them: change "ts.tsdk" to "javascript.typescript.tsdk" and
"ts.experimental.useTsgo" to "javascript.typescript.experimental.useTsgo"
(update the settings values unchanged) to restore workspace TypeScript SDK
selection and tsgo toggling.
In
`@packages/pipelines/pipeline-server/test/browser/components/app/pipeline-sidebar-source-file-list.test.tsx`:
- Line 55: Replace the it.todo("renders nested files and lets the user expand a
file to reveal its pipelines") with an executable test that mounts the
PipelineSidebarSourceFileList using the same test harness (e.g.,
renderWithProviders or the existing render helper), provide a fixture that
includes nested files and pipelines, assert the nested pipelines are initially
not in the document, simulate a user click on the folder/expand control (use
screen.getByRole or getByLabelText for the expand toggle), assert the pipelines
for that file become visible, then simulate a second click to collapse and
assert the pipelines are hidden again; use existing helpers like
fireEvent/userEvent and screen to locate elements and match by text/role to keep
the test deterministic.
---
Nitpick comments:
In `@packages/pipelines/pipeline-server/test/browser/route-test-utils.tsx`:
- Around line 83-87: The helper currently swallows all errors from
router.load(), which can hide broken route setups; change the catch so it only
suppresses expected loader failures but rethrows when the router has no matching
route (i.e. when router.matches is empty or router.state indicates no match).
Concretely, after calling router.load() (the existing router.load() call used to
populate useParams), catch the error, inspect router.matches (or
router.state.matches) and rethrow the error if there are zero matches
(indicating a route setup problem), otherwise ignore the error as a loader
failure.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 43e7df5c-63a7-4382-9a14-debc384a5f4d
📒 Files selected for processing (19)
.vscode/settings.jsonpackages/pipelines/pipeline-server/test/browser/components/app/pipeline-command-palette.test.tsxpackages/pipelines/pipeline-server/test/browser/components/app/pipeline-sidebar-source-file-list.test.tsxpackages/pipelines/pipeline-server/test/browser/components/app/pipeline-sidebar.test.tsxpackages/pipelines/pipeline-server/test/browser/components/app/source-switcher.test.tsxpackages/pipelines/pipeline-server/test/browser/components/execution/execution-table.test.tsxpackages/pipelines/pipeline-server/test/browser/components/execution/logs/log-payload-panel.test.tsxpackages/pipelines/pipeline-server/test/browser/components/execution/logs/log-table.test.tsxpackages/pipelines/pipeline-server/test/browser/components/execution/span-drawer.test.tsxpackages/pipelines/pipeline-server/test/browser/components/execution/waterfall.test.tsxpackages/pipelines/pipeline-server/test/browser/components/graph/graph-details.test.tsxpackages/pipelines/pipeline-server/test/browser/components/overview/activity-chart.test.tsxpackages/pipelines/pipeline-server/test/browser/components/overview/status-overview.test.tsxpackages/pipelines/pipeline-server/test/browser/components/pipeline/pipeline-header.test.tsxpackages/pipelines/pipeline-server/test/browser/route-test-utils.tsxpackages/pipelines/pipeline-server/test/browser/routes/s.$sourceId.$sourceFileId.$pipelineId.executions.index.test.tsxpackages/pipelines/pipeline-server/test/browser/routes/s.$sourceId.$sourceFileId.$pipelineId.index.test.tsxpackages/pipelines/pipeline-server/test/browser/routes/s.$sourceId.$sourceFileId.$pipelineId.inspect.index.test.tsxpackages/pipelines/pipeline-server/test/browser/routes/s.$sourceId.$sourceFileId.$pipelineId.inspect.outputs.test.tsx
✅ Files skipped from review due to trivial changes (5)
- packages/pipelines/pipeline-server/test/browser/components/execution/waterfall.test.tsx
- packages/pipelines/pipeline-server/test/browser/components/overview/status-overview.test.tsx
- packages/pipelines/pipeline-server/test/browser/components/execution/span-drawer.test.tsx
- packages/pipelines/pipeline-server/test/browser/components/execution/logs/log-payload-panel.test.tsx
- packages/pipelines/pipeline-server/test/browser/routes/s.$sourceId.$sourceFileId.$pipelineId.index.test.tsx
🚧 Files skipped from review as they are similar to previous changes (3)
- packages/pipelines/pipeline-server/test/browser/routes/s.$sourceId.$sourceFileId.$pipelineId.inspect.index.test.tsx
- packages/pipelines/pipeline-server/test/browser/routes/s.$sourceId.$sourceFileId.$pipelineId.executions.index.test.tsx
- packages/pipelines/pipeline-server/test/browser/components/execution/logs/log-table.test.tsx
...lines/pipeline-server/test/browser/components/app/pipeline-sidebar-source-file-list.test.tsx
Show resolved
Hide resolved
- ReDoS: fix TEMPLATE_TOKEN_RE and KEBAB_TRIM_RE patterns - Path traversal: validate output paths stay within sink baseDir - Cache invalidation: hash upstream route outputs into cache key - Output manifest: prefix keys with pipelineId:version to avoid collisions - Graph reconciliation: use version:routeId:outputIndex keying - getRouteData: return readonly shallow copy - Configured sink no-op: throw when runtime.writeOutput is missing - parse:end timing: emit span after resolver consumes lazy iterables - Pipeline-output inputs: apply includes/excludes filters; add outputIndex to synthetic path - Trace IDs: add per-emitter UUID prefix for uniqueness - Negative pagination: clamp limit and offset to valid ranges - Trace ordering: deterministic sort by timestamp+id asc - Duplicate Mod+E hotkey: remove from command palette, keep in sidebar only
- sources.traces: narrow manifest query to output.resolved/written only - serialize: guard JSON.stringify returning undefined for non-serializable values - graph.test: add test for produced-only output nodes (no resolved sink) - sources.pipeline.test: assert workspace row is recreated, not just execute success - status-overview: add aria-label and aria-hidden for accessible status metrics
This commit removes the `ExecutionEvents` related code, including the API routes, database schema, and client-side queries. The focus has shifted to using `ExecutionTraces` instead, which provides a more streamlined approach to handling execution data.
Create new tracing module in pipeline-core/src that consolidates trace domain definitions. Move trace types (PipelineTraceKind, PipelineTraceRecord, all typed record interfaces) and trace-related types (PipelineError, PipelineGraph, phase helpers) into the tracing module. Structure: - tracing/types.ts: All trace record types and discriminated union - tracing/events.ts: Phase classification, error types, graph types - tracing/utils.ts: buildOutputManifestFromTraces utility - tracing/index.ts: Re-exports all tracing surface Update pipeline-core/src/index.ts to export tracing module. Add tracing/index.ts entry point to tsdown.config.ts.
…core Update pipeline-executor to import trace types and utilities from @ucdjs/pipelines-core instead of local traces.ts: - src/run.ts: Import trace types and buildOutputManifestFromTraces from core - src/types.ts: Import trace types from core - src/internal/trace-emitter.ts: Import trace types from core - src/run/graph.ts: Import PipelineTraceRecord from core - src/run/route-runtime.ts: Import trace types from core Keep trace emission mechanics (createTraceEmitter, context management) local to pipeline-executor as they are tightly coupled to runtime implementation.
Update all imports of trace types to use @ucdjs/pipelines-core instead of executor-internal traces: pipeline-executor tests: - trace-projections.test.ts - outputs.test.ts - events.test.ts - graph.test.ts - executor.test.ts - cache-executor.test.ts pipeline-server: - src/server/db/schema.ts: Import trace types from core - src/server/routes/sources.traces.ts: Import buildOutputManifestFromTraces from core This removes cross-layer dependencies where server was importing from executor internals.
…tion Update configuration files to reflect trace types now being in pipeline-core: - packages/pipelines/pipeline-executor/tsdown.config.ts: Remove 'traces' entry point (functionality moved to core) - tooling/tsconfig/base.json: Remove @ucdjs/pipelines-executor/traces path alias Add new @ucdjs/pipelines-core/tracing path alias for new tracing module
Remove pipeline-executor/src/run/traces.ts as all trace type definitions have been moved to the tracing module in pipeline-core. This file is no longer needed and references have been updated to import from core.
Update package.json files and related files that were affected by moving trace types to pipeline-core. These changes include dependency updates and any formatting adjustments from the refactoring.
- Introduced `createMockFilterContext` to streamline filter context creation in tests. - Removed redundant context creation functions and replaced them with the new mock context. - Updated filter tests to utilize the new mock context for better readability and maintainability.
…expansion, and normalization - Introduced `deduplicateRows` and `createDeduplicateTransform` with tests. - Added `expandRanges` and `createExpandRangesTransform` with corresponding tests. - Implemented `normalizeCodePoints` and `createNormalizeTransform` with tests. - Created sorting functionality with `sortByCodePoint` and `createSortTransform` tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
pipeline-executorpipeline-server, expose a traces route, and update inspect/graph UI to show output metadata and dynamic path functionsTesting
corepack pnpm exec vitest run --project=pipeline-core packages/pipelines/pipeline-core/test/output.test.tscorepack pnpm exec vitest run --project=pipeline-executor packages/pipelines/pipeline-executor/test/executor.test.ts packages/pipelines/pipeline-executor/test/traces.test.ts packages/pipelines/pipeline-executor/test/graph.test.tscorepack pnpm exec vitest run --project=pipeline-server packages/pipelines/pipeline-server/test/server/graph-utils.test.ts packages/pipelines/pipeline-server/test/server/sources.pipeline.test.ts packages/pipelines/pipeline-server/test/server/sources.traces.test.tscorepack pnpm exec tsc -p packages/pipelines/pipeline-executor/tsconfig.json --noEmitSummary by CodeRabbit
New Features
Breaking Changes
Improvements