This PR uses pre-built dependencies for asics runner#291
This PR uses pre-built dependencies for asics runner#291jacobmerson wants to merge 3 commits intoSCOREC:developfrom
Conversation
The current CI workflow is very expensive because it rebuilds lots of dependencies from scratch. To help alleviate this process, this PR makes use of pre-built dependencies (see SCOREC#290). The risk of this change is if the dependencies do not get maintained on the SCOREC system, or if they become out of date with the rest of the CI infrastructure. However, since multiple folks use these dependencies on SCOREC, they will hopefully be maintained.
jacobmerson
force-pushed
the
cached-self-hosted-runner
branch
from
d0e96a5 to
4efa4ac
Compare
jacobmerson
force-pushed
the
cached-self-hosted-runner
branch
from
4efa4ac to
bfccee4
Compare
|
For testing this PR, I had to make |
jacobmerson
force-pushed
the
cached-self-hosted-runner
branch
from
fed3ca9 to
cf5af2d
Compare
|
With this change the self-hosted runner completes in 7 minutes instead of somewhere between 48 min - 1 hr. I think we should move forward with this approach as soon as I can move the prebuilt dependencies out of my home directory. |
|
Nice improvement! Thank you. IIRC, we have spack packages for these and could install them alongside the 'normal' user accessible modules instead of treating them as something 'special'. If we did that, we'd want to periodically install newer versions of the deps (ideally through cron etc.) to catch problems caused by upstream changes. |
|
In terms of how we use spack for this. We could create an environment with all the dependencies then do |
|
I prefer to just expose the modules (i.e., as instructed /opt/scorec/spack/rhel9/README) instead of having users run spack commands. |
|
Yes. That's what I meant. That command will give you a list of modules that replicate the spack environment. I didn't mean run spack in CI. Sichao and I can work on getting this in place when he's back from vacation. |
2 participants
The current CI workflow is very expensive because it rebuilds lots of dependencies from scratch. To help alleviate this process, this PR makes use of pre-built dependencies (see #290). The risk of this change is if the dependencies do not get maintained on the SCOREC system, or if they become out of date with the rest of the CI infrastructure. However, since multiple folks use these dependencies on SCOREC, they will hopefully be maintained.
@cwsmith @Sichao25 you may want to comment on the pitfalls of this approach.