Skip to content

fix benchmark result posting#159

Open
hakkelt wants to merge 1 commit intoJuliaFirstOrder:masterfrom
hakkelt:maybe-fix-benchmarking
Open

fix benchmark result posting#159
hakkelt wants to merge 1 commit intoJuliaFirstOrder:masterfrom
hakkelt:maybe-fix-benchmarking

Conversation

@hakkelt
Copy link
Copy Markdown
Contributor

@hakkelt hakkelt commented Apr 15, 2026

In this PR, I try to fix this problem by dividing the benchmarking into two phases:

  1. Run the benchmark, save the results to a file, and upload it as an artifact. -> It does not need any special permissions.
  2. Download the artifact and post the content as a comment. -> This gets triggered by the first phase via "workflow_run", and my hope is that it is possible to selectively set write permission to this job only.

@hakkelt hakkelt force-pushed the maybe-fix-benchmarking branch from 517d2b2 to ab571a6 Compare April 15, 2026 22:08
@codecov
Copy link
Copy Markdown

codecov bot commented Apr 15, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.78%. Comparing base (83c73c9) to head (ab571a6).

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #159      +/-   ##
==========================================
- Coverage   94.79%   94.78%   -0.01%     
==========================================
  Files          79       79              
  Lines        2498     2494       -4     
==========================================
- Hits         2368     2364       -4     
  Misses        130      130              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@hakkelt
Copy link
Copy Markdown
Contributor Author

hakkelt commented Apr 15, 2026

To test this PR, post_benchmark_comment.yml needs to be on master already... So I'll play around a bit in some playground repositories, then I'll report if it works.

@hakkelt hakkelt marked this pull request as ready for review April 20, 2026 18:23
@hakkelt hakkelt marked this pull request as draft April 20, 2026 18:24
@hakkelt hakkelt force-pushed the maybe-fix-benchmarking branch from ab571a6 to 5f53814 Compare April 20, 2026 18:35
@hakkelt
Copy link
Copy Markdown
Contributor Author

hakkelt commented Apr 20, 2026

I opened a PR on AirspeedVelocity (MilesCranmer/AirspeedVelocity.jl#142) that adds this two-stage benchmarking workflow.

@lostella For now, there are three options, I think:

  • Merge the other benchmarking PR that saves the results into the job summary (Benchmarking results as job summary #158)
  • Merge this PR that uses my fork and posts comments safely
  • Wait until my PR on AirspeedVelocity gets merged, then update this PR to use the new version

Regarding the automatic detection of performance regressions, I considered it and concluded that making the job fail might not be the best solution. I opened another PR on AirspeedVelocity (MilesCranmer/AirspeedVelocity.jl#140) that adds emojis to its result table to mark significant speedups and slowdowns (actually, I just re-heated the original author's outdated PR). If it gets merged, I'm considering opening another one that adds a notice before the collapsible table in the posted comment, warning about the negative performance regressions.

@hakkelt hakkelt marked this pull request as ready for review April 20, 2026 18:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant