Merge branch 'main' into docs/self-host-env-vars

Author: isshaddad

.changeset/fix-batch-queue-processing.md

@@ -0,0 +1,13 @@
+---
+paths:
+  - "internal-packages/database/**"
+---
+
+# Database Migration Safety
+
+- When adding indexes to **existing tables**, use `CREATE INDEX CONCURRENTLY IF NOT EXISTS` to avoid table locks. These must be in their own separate migration file (one index per file).
+- Indexes on **newly created tables** (same migration as `CREATE TABLE`) do not need CONCURRENTLY.
+- When indexing a **new column on an existing table**, split into two migrations: first `ADD COLUMN IF NOT EXISTS`, then `CREATE INDEX CONCURRENTLY IF NOT EXISTS` in a separate file.
+- After generating a migration with Prisma, remove extraneous lines for: `_BackgroundWorkerToBackgroundWorkerFile`, `_BackgroundWorkerToTaskQueue`, `_TaskRunToTaskRunTag`, `_WaitpointRunConnections`, `_completedWaitpoints`, `SecretStore_key_idx`, and unrelated TaskRun indexes.
+- Never drop columns or tables without explicit approval.
+- New code should target `RunEngineVersion.V2` only.

.changeset/metal-steaks-try.md

@@ -0,0 +1,14 @@
+---
+paths:
+  - "docs/**"
+---
+
+# Documentation Writing Rules
+
+- Use Mintlify MDX format. Frontmatter: `title`, `description`, `sidebarTitle` (optional).
+- After creating a new page, add it to `docs.json` navigation under the correct group.
+- Use Mintlify components: `<Note>`, `<Warning>`, `<Info>`, `<Tip>`, `<CodeGroup>`, `<Expandable>`, `<Steps>`/`<Step>`.
+- Code examples should be complete and runnable where possible.
+- Always import from `@trigger.dev/sdk`, never `@trigger.dev/sdk/v3`.
+- Keep paragraphs short. Use headers to break up content.
+- Link to related pages using relative paths (e.g., `[Tasks](/tasks/overview)`).

.changeset/modern-boxes-watch.md

@@ -0,0 +1,33 @@
+---
+paths:
+  - "apps/webapp/app/v3/**"
+---
+
+# Legacy V1 Engine Code in `app/v3/`
+
+The `v3/` directory name is misleading - most code here is actively used by the current V2 engine. Only the specific files below are legacy V1-only code.
+
+## V1-Only Files - Never Modify
+
+- `marqs/` directory (entire MarQS queue system: sharedQueueConsumer, devQueueConsumer, fairDequeuingStrategy, devPubSub)
+- `legacyRunEngineWorker.server.ts` (V1 background job worker)
+- `services/triggerTaskV1.server.ts` (deprecated V1 task triggering)
+- `services/cancelTaskRunV1.server.ts` (deprecated V1 cancellation)
+- `authenticatedSocketConnection.server.ts` (V1 dev WebSocket using DevQueueConsumer)
+- `sharedSocketConnection.ts` (V1 shared queue socket using SharedQueueConsumer)
+
+## V1/V2 Branching Pattern
+
+Some services act as routers that branch on `RunEngineVersion`:
+- `services/cancelTaskRun.server.ts` - calls V1 service or `engine.cancelRun()` for V2
+- `services/batchTriggerV3.server.ts` - uses marqs for V1 path, run-engine for V2
+
+When editing these shared services, only modify V2 code paths.
+
+## V2 Modern Stack
+
+- **Run lifecycle**: `@internal/run-engine` (internal-packages/run-engine)
+- **Background jobs**: `@trigger.dev/redis-worker` (not graphile-worker/zodworker)
+- **Queue operations**: RunQueue inside run-engine (not MarQS)
+- **V2 engine singleton**: `runEngine.server.ts`, `runEngineHandlers.server.ts`
+- **V2 workers**: `commonWorker.server.ts`, `alertsWorker.server.ts`, `batchTriggerWorker.server.ts`

.claude/rules/database-safety.md

@@ -0,0 +1,12 @@
+---
+paths:
+  - "packages/**"
+---
+
+# Public Package Rules
+
+- Changes to `packages/` are **customer-facing**. Always add a changeset: `pnpm run changeset:add`
+- Default to **patch**. Get maintainer approval for minor. Never select major without explicit approval.
+- `@trigger.dev/core`: **Never import the root**. Always use subpath imports (e.g., `@trigger.dev/core/v3`).
+- Do NOT update `rules/` or `.claude/skills/trigger-dev-tasks/` unless explicitly asked. These are maintained in separate dedicated passes.
+- Test changes using `references/hello-world` reference project.

.claude/rules/docs-writing.md

@@ -0,0 +1,23 @@
+---
+paths:
+  - "apps/**"
+---
+
+# Server App Changes
+
+When modifying server apps (webapp, supervisor, coordinator, etc.) with **no package changes**, add a `.server-changes/` file instead of a changeset:
+
+```bash
+cat > .server-changes/descriptive-name.md << 'EOF'
+---
+area: webapp
+type: fix
+---
+
+Brief description of what changed and why.
+EOF
+```
+
+- **area**: `webapp` | `supervisor` | `coordinator` | `kubernetes-provider` | `docker-provider`
+- **type**: `feature` | `fix` | `improvement` | `breaking`
+- If the PR also touches `packages/`, just the changeset is sufficient (no `.server-changes/` needed).

.claude/rules/legacy-v3-code.md

@@ -0,0 +1,78 @@
+---
+name: span-timeline-events
+description: Use when adding, modifying, or debugging OTel span timeline events in the trace view. Covers event structure, ClickHouse storage constraints, rendering in SpanTimeline component, admin visibility, and the step-by-step process for adding new events.
+allowed-tools: Read, Write, Edit, Glob, Grep, Bash
+---
+
+# Span Timeline Events
+
+The trace view's right panel shows a timeline of events for the selected span. These are OTel span events rendered by `app/utils/timelineSpanEvents.ts` and the `SpanTimeline` component.
+
+## How They Work
+
+1. **Span events** in OTel are attached to a parent span. In ClickHouse, they're stored as separate rows with `kind: "SPAN_EVENT"` sharing the parent span's `span_id`. The `#mergeRecordsIntoSpanDetail` method reassembles them into the span's `events` array at query time.
+2. The timeline only renders events whose `name` starts with `trigger.dev/` - all others are silently filtered out.
+3. The **display name** comes from `properties.event` (not the span event name), mapped through `getFriendlyNameForEvent()`.
+4. Events are shown on the **span they belong to** - events on one span don't appear in another span's timeline.
+
+## ClickHouse Storage Constraint
+
+When events are written to ClickHouse, `spanEventsToTaskEventV1Input()` filters out events whose `start_time` is not greater than the parent span's `startTime`. Events at or before the span start are silently dropped. This means span events must have timestamps strictly after the span's own `startTimeUnixNano`.
+
+## Timeline Rendering (SpanTimeline component)
+
+The `SpanTimeline` component in `app/components/run/RunTimeline.tsx` renders:
+
+1. **Events** (thin 1px line with hollow dots) - all events from `createTimelineSpanEventsFromSpanEvents()`
+2. **"Started"** marker (thick cap) - at the span's `startTime`
+3. **Duration bar** (thick 7px line) - from "Started" to "Finished"
+4. **"Finished"** marker (thick cap) - at `startTime + duration`
+
+The thin line before "Started" only appears when there are events with timestamps between the span start and the first child span. For the Attempt span this works well (Dequeued -> Pod scheduled -> Launched -> etc. all happen before execution starts). Events all get `lineVariant: "light"` (thin) while the execution bar gets `variant: "normal"` (thick).
+
+## Trace View Sort Order
+
+Sibling spans (same parent) are sorted by `start_time ASC` from the ClickHouse query. The `createTreeFromFlatItems` function preserves this order. Event timestamps don't affect sort order - only the span's own `start_time`.
+
+## Event Structure
+
+```typescript
+// OTel span event format
+{
+  name: "trigger.dev/run",        // Must start with "trigger.dev/" to render
+  timeUnixNano: "1711200000000000000",
+  attributes: [
+    { key: "event", value: { stringValue: "dequeue" } },  // The actual event type
+    { key: "duration", value: { intValue: 150 } },         // Optional: duration in ms
+  ]
+}
+```
+
+## Admin-Only Events
+
+`getAdminOnlyForEvent()` controls visibility. Events default to **admin-only** (`true`).
+
+| Event | Admin-only | Friendly name |
+|-------|-----------|---------------|
+| `dequeue` | No | Dequeued |
+| `fork` | No | Launched |
+| `import` | No (if no fork event) | Importing task file |
+| `create_attempt` | Yes | Attempt created |
+| `lazy_payload` | Yes | Lazy attempt initialized |
+| `pod_scheduled` | Yes | Pod scheduled |
+| (default) | Yes | (raw event name) |
+
+## Adding New Timeline Events
+
+1. Add OTLP span event with `name: "trigger.dev/<scope>"` and `properties.event: "<type>"`
+2. Event timestamp must be strictly after the parent span's `startTimeUnixNano` (ClickHouse drops earlier events)
+3. Add friendly name in `getFriendlyNameForEvent()` in `app/utils/timelineSpanEvents.ts`
+4. Set admin visibility in `getAdminOnlyForEvent()`
+5. Optionally add help text in `getHelpTextForEvent()`
+
+## Key Files
+
+- `app/utils/timelineSpanEvents.ts` - filtering, naming, admin logic
+- `app/components/run/RunTimeline.tsx` - `SpanTimeline` component (thin line + thick bar rendering)
+- `app/presenters/v3/SpanPresenter.server.ts` - loads span data including events
+- `app/v3/eventRepository/clickhouseEventRepository.server.ts` - `spanEventsToTaskEventV1Input()` (storage filter), `#mergeRecordsIntoSpanDetail` (reassembly)

.claude/rules/sdk-packages.md

@@ -14,7 +14,7 @@ alwaysApply: false
 
 ## Essential requirements when generating task code
 
-1. You MUST use `@trigger.dev/sdk/v3`
+1. You MUST import from `@trigger.dev/sdk` (NEVER `@trigger.dev/sdk/v3`)
 2. You MUST NEVER use `client.defineJob`
 3. YOU MUST `export` every task, including subtasks
 4. If you are able to generate an example payload for a task, do so.
@@ -53,7 +53,7 @@ Instead, you MUST ALWAYS generate ONLY this pattern:
 ```ts
 // ✅ ALWAYS GENERATE THIS EXACT PATTERN
 
-import { task } from "@trigger.dev/sdk/v3";
+import { task } from "@trigger.dev/sdk";
 
 //1. You need to export each task, even if it's a subtask
 export const helloWorld = task({
@@ -71,7 +71,7 @@ export const helloWorld = task({
 A task is a function that can run for a long time with resilience to failure:
 
 ```ts
-import { task } from "@trigger.dev/sdk/v3";
+import { task } from "@trigger.dev/sdk";
 
 export const helloWorld = task({
   id: "hello-world",
@@ -271,7 +271,7 @@ Global lifecycle hooks can also be defined in `trigger.config.ts` to apply to al
 ## Correct Schedules task (cron) implementations
 
 ```ts
-import { schedules } from "@trigger.dev/sdk/v3";
+import { schedules } from "@trigger.dev/sdk";
 
 export const firstScheduledTask = schedules.task({
   id: "first-scheduled-task",
@@ -312,7 +312,7 @@ export const firstScheduledTask = schedules.task({
 ### Attach a Declarative schedule
 
 ```ts
-import { schedules } from "@trigger.dev/sdk/v3";
+import { schedules } from "@trigger.dev/sdk";
 
 // Sepcify a cron pattern (UTC)
 export const firstScheduledTask = schedules.task({
@@ -326,7 +326,7 @@ export const firstScheduledTask = schedules.task({
 ```
 
 ```ts
-import { schedules } from "@trigger.dev/sdk/v3";
+import { schedules } from "@trigger.dev/sdk";
 
 // Specify a specific timezone like this:
 export const secondScheduledTask = schedules.task({
@@ -375,7 +375,7 @@ const createdSchedule = await schedules.create({
 Schema tasks validate payloads against a schema before execution:
 
 ```ts
-import { schemaTask } from "@trigger.dev/sdk/v3";
+import { schemaTask } from "@trigger.dev/sdk";
 import { z } from "zod";
 
 const myTask = schemaTask({
@@ -400,7 +400,7 @@ When you trigger a task from your backend code, you need to set the `TRIGGER_SEC
 Triggers a single run of a task with specified payload and options without importing the task. Use type-only imports for full type checking.
 
 ```ts
-import { tasks } from "@trigger.dev/sdk/v3";
+import { tasks } from "@trigger.dev/sdk";
 import type { emailSequence } from "~/trigger/emails";
 
 export async function POST(request: Request) {
@@ -418,7 +418,7 @@ export async function POST(request: Request) {
 Triggers multiple runs of a single task with different payloads without importing the task.
 
 ```ts
-import { tasks } from "@trigger.dev/sdk/v3";
+import { tasks } from "@trigger.dev/sdk";
 import type { emailSequence } from "~/trigger/emails";
 
 export async function POST(request: Request) {
@@ -436,7 +436,7 @@ export async function POST(request: Request) {
 Triggers multiple runs of different tasks at once, useful when you need to execute multiple tasks simultaneously.
 
 ```ts
-import { batch } from "@trigger.dev/sdk/v3";
+import { batch } from "@trigger.dev/sdk";
 import type { myTask1, myTask2 } from "~/trigger/myTasks";
 
 export async function POST(request: Request) {
@@ -621,7 +621,7 @@ const handle = await myTask.trigger(
 Access metadata inside a run:
 
 ```ts
-import { task, metadata } from "@trigger.dev/sdk/v3";
+import { task, metadata } from "@trigger.dev/sdk";
 
 export const myTask = task({
   id: "my-task",
@@ -713,7 +713,7 @@ Trigger.dev Realtime enables subscribing to runs for real-time updates on run st
 Subscribe to a run after triggering a task:
 
 ```ts
-import { runs, tasks } from "@trigger.dev/sdk/v3";
+import { runs, tasks } from "@trigger.dev/sdk";
 
 async function myBackend() {
   const handle = await tasks.trigger("my-task", { some: "data" });
@@ -735,7 +735,7 @@ async function myBackend() {
 You can infer types of run's payload and output by passing the task type:
 
 ```ts
-import { runs } from "@trigger.dev/sdk/v3";
+import { runs } from "@trigger.dev/sdk";
 import type { myTask } from "./trigger/my-task";
 
 for await (const run of runs.subscribeToRun<typeof myTask>(handle.id)) {
@@ -752,7 +752,7 @@ for await (const run of runs.subscribeToRun<typeof myTask>(handle.id)) {
 Stream data in realtime from inside your tasks using the metadata system:
 
 ```ts
-import { task, metadata } from "@trigger.dev/sdk/v3";
+import { task, metadata } from "@trigger.dev/sdk";
 import OpenAI from "openai";
 
 export type STREAMS = {
@@ -947,7 +947,7 @@ For most use cases, Realtime hooks are preferred over SWR hooks with polling due
 For client-side usage, generate a public access token with appropriate scopes:
 
 ```ts
-import { auth } from "@trigger.dev/sdk/v3";
+import { auth } from "@trigger.dev/sdk";
 
 const publicToken = await auth.createPublicToken({
   scopes: {
@@ -967,7 +967,7 @@ Idempotency ensures that an operation produces the same result when called multi
 Provide an `idempotencyKey` when triggering a task to ensure it runs only once with that key:
 
 ```ts
-import { idempotencyKeys, task } from "@trigger.dev/sdk/v3";
+import { idempotencyKeys, task } from "@trigger.dev/sdk";
 
 export const myTask = task({
   id: "my-task",
@@ -1058,7 +1058,7 @@ function hash(payload: any): string {
 
 ```ts
 // onFailure executes after all retries are exhausted; use for notifications, logging, or side effects on final failure:
-import { task, logger } from "@trigger.dev/sdk/v3";
+import { task, logger } from "@trigger.dev/sdk";
 
 export const loggingExample = task({
   id: "logging-example",
@@ -1078,7 +1078,7 @@ export const loggingExample = task({
 The `trigger.config.ts` file configures your Trigger.dev project, specifying task locations, retry settings, telemetry, and build options.
 
 ```ts
-import { defineConfig } from "@trigger.dev/sdk/v3";
+import { defineConfig } from "@trigger.dev/sdk";
 
 export default defineConfig({
   project: "<project ref>",
@@ -1226,7 +1226,7 @@ await myTask.trigger({ name: "Alice", age: 30 });
 
 Before generating any code, you MUST verify:
 
-1. Are you importing from `@trigger.dev/sdk/v3`? If not, STOP and FIX.
+1. Are you importing from `@trigger.dev/sdk` (NOT `@trigger.dev/sdk/v3`)? If not, STOP and FIX.
 2. Have you exported every task? If not, STOP and FIX.
 3. Have you generated any DEPRECATED code patterns? If yes, STOP and FIX.