chore(base-image): Migrate Konflux builds to UBI9/RHEL9 by davdhacs · Pull Request #2562 · stackrox/scanner

davdhacs · 2026-01-15T04:28:55Z

Description

Migrates scanner builds from UBI8/RHEL8 to UBI9/RHEL9 base images.

Key Changes

Konflux Base Images:

Builder: brew.registry.redhat.io/rh-osbs/openshift-golang-builder:rhel_9_1.24
Scanner runtime: registry.access.redhat.com/ubi9-minimal:latest
Scanner DB: registry.redhat.io/rhel9/postgresql-15:latest
Image names: rhacs-scanner-rhel9, rhacs-scanner-slim-rhel9, rhacs-scanner-db-rhel9, rhacs-scanner-db-slim-rhel9

Non-Konflux Base Images:

Scanner: ubi9-minimal
Scanner DB: ubi9 / ubi9-minimal
Vulnerabilities: ubi9-minimal

RPM/Repo Updates:

rpms.lock.yaml: xz updated from RHEL 8 (5.2.4) to RHEL 9 (5.2.5)
rpms.rhel.repo: repos updated from rhel8 to rhel9
PostgreSQL download script: pg_rhel_major=9

Tekton Labels:

CPE labels: el8 → el9

UBI9 Compatibility Fixes:

update-ca-trust extract -o /etc/pki/ca-trust/extracted for unprivileged containers (RHBZ#2241240)
cp --recursive --no-dereference --no-clobber in restore-all-dir-contents
microdnf install -y xz (explicit -y flag)

Checklist

Investigated and inspected CI test results

Testing Performed

TBD

openshift-ci · 2026-01-15T04:29:19Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

davdhacs · 2026-01-27T14:57:14Z

@tommartensen fyi this is the parallel scanner update for UBI9. In this, we also have to add the update-ca-trust workaround for UBI9 changed perms on second execution.

The arm build fails because tar hits a bug in a syscall in qemu; I still have some debug logging around that, and I expect the arm arch build to fail because of it.

davdhacs · 2026-02-18T05:39:24Z

/test all

Migrate all scanner base images from UBI8/RHEL8 to UBI9/RHEL9: Konflux base images: - Builder: openshift-golang-builder:rhel_9_golang_1.25 - Scanner runtime: ubi9-minimal - Scanner DB: rhel9/postgresql-15 Non-Konflux base images: - Scanner: ubi9-minimal - Scanner DB: ubi9 / ubi9-minimal - Vulnerabilities: ubi9-minimal Other updates: - Tekton CPE labels: el8 -> el9 - rpms.lock.yaml: xz updated to RHEL 9 version (5.2.5-8.el9_0) - rpms.rhel.repo: RHEL 8 repos -> RHEL 9 repos - PostgreSQL download script: pg_rhel_major=9 - Image name labels: rhel8 -> rhel9 - microdnf install -y xz (explicit -y for UBI9 compatibility) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

In UBI9, update-ca-trust fails when running as an unprivileged user (nobody:nobody) because it attempts to write to system-wide paths. Use the -o flag to specify a user-writable output directory. Also switch restore-all-dir-contents to use --no-clobber to avoid overwriting CA trust files that were already updated at runtime. See: https://bugzilla.redhat.com/show_bug.cgi?id=2241240 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

davdhacs · 2026-02-18T21:32:25Z

/retest

davdhacs · 2026-02-19T00:07:01Z

@tommartensen could you review this? (the ubi9 upgrade for scanner(v2); includes the update-ca-trust permissions workaround)

tommartensen

LGTM to me from a Konflux perspective. Have you deployed and smoke tested the resulting images (like we did for collector)?

davdhacs · 2026-02-19T14:44:43Z

LGTM to me from a Konflux perspective. Have you deployed and smoke tested the resulting images (like we did for collector)?

Yes, I tested with these changes (minus the master-merge [empty] commits). I'll re-do the smoke test with this and the latest master collector builds.

BradLugo · 2026-02-23T17:06:39Z

Note to self and @stackrox/scanner: we need to follow up on #2092 and update the image for s390x to avoid using a different version of psql.

mclasmeier · 2026-03-03T13:49:09Z

image/scanner/scripts/import-additional-cas

-update-ca-trust extract
+# The -o flag is required for running as an unprivileged user in containers.
+# See: https://bugzilla.redhat.com/show_bug.cgi?id=2241240
+update-ca-trust extract -o /etc/pki/ca-trust/extracted


I'd like to understand this better.
Is this path /etc/pki/ca-trust/extracted referenced somewhere? Is it defaulted to somewhere?

I think it's self-referential: https://src.fedoraproject.org/rpms/ca-certificates/blob/rawhide/f/update-ca-trust#_129-138

/etc/pki/ca-trust/extracted is the default. We're re-setting it to get the non-root user operation of setting a -o path.
Here you can see it hardcoded in the version of update-ca-trust in ubi9-minimal: https://pkgs.devel.redhat.com/cgit/rpms/ca-certificates/tree/update-ca-trust?h=rhel-9.4.0#n10
And this explains why we need to use the -o arg: https://pkgs.devel.redhat.com/cgit/rpms/ca-certificates/tree/update-ca-trust?h=rhel-9.4.0#n88
long-version:
In UBI8, the update-ca-trust trust extract execution did not change the permissions on the files it wrote. In the version in UBI9, the directory-hash output was added and write permissions are removed when it is written.
The -o(USER_DEST) arg was added to allow running as non-root because it does a chmod +/- around adding links into the directory-hash directory: https://src.fedoraproject.org/rpms/ca-certificates/pull-request/8#request_diff
Looking forward in https://src.fedoraproject.org/rpms/ca-certificates/c/7dc60cb it was reworked but changes it so the symlinks are only added when the USER_DIR is not specified (so we will still need the -o arg if the ca-certificates package updates are backported).

example usage: here this path is in the search paths for golang ssl: https://go.dev/src/crypto/x509/root_linux.go

more information in the man page, https://www.linux.org/docs/man8/update-ca-trust.html

Far future / TODO: The cert setup could be re-worked for these images so that it does not rely on update-ca-trust to add custom certs. Maybe that part isn't needed anymore, but I think that reaches outside of the scope of upgrading to UBI9. There was discussion about init container instead of the double-run that sensor and scanner do, and there are other options especially when we eventually switch to the golang built-in ssl.

Thank you @davdhacs

If you're okay with this, can I get a +1 and we'll merge this PR? We can also update the change in stackrox to follow update-ca-trust's -o arg use instead of needing the full chmod.

…migration

…migration # Conflicts: # image/db/rhel/konflux.Dockerfile

davdhacs · 2026-03-09T16:32:46Z

/test ?

davdhacs · 2026-03-10T04:25:22Z

The test failures are not-related to UBI9/changes in this PR.

davdhacs · 2026-03-10T04:25:34Z

#2888
test fix pr

…migration # Conflicts: # image/db/rhel/konflux.Dockerfile

davdhacs · 2026-03-10T20:51:26Z

/retest e2e-tests

davdhacs · 2026-03-10T23:33:48Z

/test e2e-tests

davdhacs · 2026-03-11T01:06:46Z

/test e2e-tests

davdhacs · 2026-03-11T03:47:19Z

/test e2e-tests

davdhacs · 2026-03-11T19:16:55Z

/test e2e-tests

davdhacs · 2026-03-11T21:09:17Z

@BradLugo can I get another +1? I merged changes from master to get the e2e test fix.

mclasmeier · 2026-03-16T11:29:15Z

I just did some testing with this scanner and I believe it suffers from the same problem as the one I have addressed with this commit.

See:

I am deploying ACS (some semi-recent master version, doesn't mater) on an OpenShift cluster, with scanner v2 enabled, scanner v4 disabled and an overlay which injects this PR's scanner version image into the scanner deployment:

❯ roxie deploy --resources=auto central \
  --set spec.scanner.scannerComponent=Enabled \
  --set spec.scannerV4.scannerComponent=Disabled \
  --set spec.overlays='[{"apiVersion":"apps/v1", "kind":"Deployment", "name":"scanner", "patches":[{"path":"spec.template.spec.containers[name:scanner].image", "value":"quay.io/rhacs-eng/scanner:2.39.x-86-gf65ee678a8"}]}]'
00:00 Using kubeconfig /Users/mclasmeier/.kube/infra/kubeconfig
00:00 Running with a controlling terminal.
00:01 🚀 ACS Deployer initialized
00:01 roxctl version: 4.9.1-0-g1bde1c73b2
00:01 
00:01 ┌────────────────────────────────────────────────────────────┐
00:01 │ Deployment Configuration                                   │
00:01 ├────────────────────────────────────────────────────────────┤
00:01 │              Component: Central                            │
00:01 │           Cluster Type: OpenShift4                         │
00:01 │               Main Tag:                                    │
00:01 │     Kubernetes Context: admin                              │
00:01 │      Deployment Method: Operator                           │
00:01 │               Exposure: loadbalancer                       │
00:01 └────────────────────────────────────────────────────────────┘
00:01 
00:01 Looking up main image tag
00:07 Using stackrox repository tag: 4.11.x-300-g280a4287f7
00:07 Auto-detected cluster type OpenShift4: using resource profile "medium"
00:07 Initiating deployment of Central
00:07 Deploying Central to namespace acs-central
00:07 Existing Central deployment found, tearing down...
00:07 🗑️  Tearing down acs-central
00:08 ⏳ Waiting for Central resources to be fully deleted...
00:08 Deleting Central resources
00:14 ✓ Central resources in namespace acs-central have been deleted
00:14 🚀 Deploying Central via Operator...
00:17 ✓ Operator already deployed with correct version
00:17 Preparing namespace acs-central
00:18 ✓ Admin password secret created
00:18 Using Central resource profile: medium
00:18 Applying Central custom resource
00:19 ✓ Central Custom Resource applied
00:19 ⏳ Waiting for Central to become ready...
[...]
01:00 ✓ Central is ready (1 replicas)
01:00 ⏳ Waiting for LoadBalancer central-loadbalancer to get external IP...
[...]
01:49 ✓ Central is ready and responding!
[...]

Then, I exec into the scanner container and show that the restore-all-dir-contents script cannot be invoked again (after the import-additional-cas script has run) -- something that I believe needs to be possible, because that would happen on container restarts.

❯ kubectl -n acs-central exec -it deploy/scanner -- /bin/sh
sh-5.1$ /restore-all-dir-contents 
cp: cannot create regular file '/etc/pki/ca-trust/extracted/pem/directory-hash/GlobalSign_ECC_Root_CA_-_R4.pem': Permission denied
cp: cannot create regular file '/etc/pki/ca-trust/extracted/pem/directory-hash/GlobalSign_ECC_Root_CA_-_R5.pem': Permission denied
cp: cannot create regular file '/etc/pki/ca-trust/extracted/pem/directory-hash/GlobalSign_Root_CA_-_R3.pem': Permission denied
cp: cannot create regular file '/etc/pki/ca-trust/extracted/pem/directory-hash/GlobalSign_Root_CA_-_R6.pem': Permission denied
cp: cannot create regular file '/etc/pki/ca-trust/extracted/pem/directory-hash/certSIGN_Root_CA_G2.pem': Permission denied
sh-5.1$

davdhacs added the konflux-build Run Konflux in PR. Push commit to trigger it. label Jan 15, 2026

openshift-ci bot added the do-not-merge/work-in-progress label Jan 15, 2026

davdhacs force-pushed the konflux-ubi9-rhel9-migration branch from d9cdc31 to d2f149d Compare February 5, 2026 16:58

davdhacs force-pushed the konflux-ubi9-rhel9-migration branch from 34c5d67 to 5a2e599 Compare February 18, 2026 05:37

davdhacs marked this pull request as ready for review February 18, 2026 05:39

davdhacs requested review from a team and rhacs-bot as code owners February 18, 2026 05:39

openshift-ci bot removed the do-not-merge/work-in-progress label Feb 18, 2026

rhacs-bot requested a review from a team February 18, 2026 05:48

davdhacs and others added 2 commits February 18, 2026 14:01

davdhacs force-pushed the konflux-ubi9-rhel9-migration branch from 5a2e599 to b3addf3 Compare February 18, 2026 21:07

empty commit

46d33f6

davdhacs requested a review from tommartensen February 19, 2026 00:06

tommartensen reviewed Feb 19, 2026

View reviewed changes

BradLugo approved these changes Feb 23, 2026

View reviewed changes

mclasmeier reviewed Mar 3, 2026

View reviewed changes

davdhacs added 2 commits March 5, 2026 13:22

Merge remote-tracking branch 'origin/master' into konflux-ubi9-rhel9-…

1ec1ed6

…migration

Merge remote-tracking branch 'origin/master' into konflux-ubi9-rhel9-…

64b335a

…migration # Conflicts: # image/db/rhel/konflux.Dockerfile

Merge remote-tracking branch 'origin/master' into konflux-ubi9-rhel9-…

f65ee67

…migration # Conflicts: # image/db/rhel/konflux.Dockerfile

davdhacs requested a review from BradLugo March 11, 2026 21:08

Conversation

davdhacs commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Key Changes

Checklist

Testing Performed

Uh oh!

openshift-ci bot commented Jan 15, 2026

Uh oh!

davdhacs commented Jan 27, 2026

Uh oh!

davdhacs commented Feb 18, 2026

Uh oh!

davdhacs commented Feb 18, 2026

Uh oh!

davdhacs commented Feb 19, 2026

Uh oh!

tommartensen left a comment

Choose a reason for hiding this comment

Uh oh!

davdhacs commented Feb 19, 2026

Uh oh!

BradLugo commented Feb 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davdhacs commented Mar 9, 2026

Uh oh!

davdhacs commented Mar 10, 2026

Uh oh!

davdhacs commented Mar 10, 2026

Uh oh!

davdhacs commented Mar 10, 2026

Uh oh!

davdhacs commented Mar 10, 2026

Uh oh!

davdhacs commented Mar 11, 2026

Uh oh!

davdhacs commented Mar 11, 2026

Uh oh!

davdhacs commented Mar 11, 2026

Uh oh!

davdhacs commented Mar 11, 2026

Uh oh!

mclasmeier commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

davdhacs commented Jan 15, 2026 •

edited

Loading

mclasmeier commented Mar 16, 2026 •

edited

Loading