Skip to content

chore(ci): migrate nightly benchmark baseline jobs to Seed runners#15140

Open
devin-ai-integration[bot] wants to merge 1 commit intomainfrom
devin/1776516002-benchmark-baseline-seed-runners
Open

chore(ci): migrate nightly benchmark baseline jobs to Seed runners#15140
devin-ai-integration[bot] wants to merge 1 commit intomainfrom
devin/1776516002-benchmark-baseline-seed-runners

Conversation

@devin-ai-integration
Copy link
Copy Markdown
Contributor

Description

The nightly benchmark-baseline.yml workflow was still running on ubuntu-latest (4-core GitHub-hosted runners) while the PR benchmark jobs in seed.yml were migrated to Seed runners in #15096/#15101. This caused an apples-to-oranges comparison — PR benchmarks appeared faster simply because they ran on better hardware, not due to actual code improvements.

Changes Made

  • Updated baseline job from runs-on: ubuntu-latest to runs-on: Seed
  • Updated baseline-e2e job from runs-on: ubuntu-latest to runs-on: Seed
  • Updated docs-baseline job from runs-on: ubuntu-latest to runs-on: Seed
  • Left fetch-specs and post-metrics on ubuntu-latest since they are lightweight jobs that don't run benchmarks

Testing

  • Manual testing completed — this is a CI-only change; will be validated on next nightly run

Link to Devin session: https://app.devin.ai/sessions/8d4785bd33564dc8a167df467ed4bc6f

Co-Authored-By: barry.zou <barry.zou@buildwithfern.com>
@devin-ai-integration
Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

Copy link
Copy Markdown

@claude claude bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.

Tip: disable this comment in your organization's Code Review settings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

0 participants