Chcon command fails in nvidia-driver daemon - nvidia driver installation aborts

[aneeshhitachivantara]

1. Quick Debug Checklist

Are you running on an Ubuntu 18.04 node? RHEL 9.6

Are you running Kubernetes v1.13+? v.1.31.9 (k8s)

Are you running Docker (>= 18.06) or CRIO (>= 1.13+)? Containerd

Did you apply the CRD (kubectl describe clusterpolicies --all-namespaces) Yes

1. Issue or feature description

While installing GPU Operator(Version 25.3.1) on my K8S Setup with RHEL 9.6, gets below error with nvidia-driver-daemonset pod. Selinux is already disabled on my hosts.

Installing 'NVIDIA Accelerated Graphics Driver for Linux-x86_64' (570.148.08):: Installing

[##############################] 100%
Driver file installation is complete.
Running post-install sanity check:: Checking

[##############################] 100%
Post-install sanity check passed.

Installation of the kernel module for the NVIDIA Accelerated Graphics Driver for Linux-x86_64 (version: 570.148.08) is now complete.

• _load
• _load_driver
  Parsing kernel module parameters...
• echo 'Parsing kernel module parameters...'
• _get_module_params
• local base_path=/drivers
• '[' -f /drivers/nvidia.conf ']'
• '[' -f /drivers/nvidia-uvm.conf ']'
• '[' -f /drivers/nvidia-modeset.conf ']'
• '[' -f /drivers/nvidia-peermem.conf ']'
• local nv_fw_search_path=/run/nvidia/driver/lib/firmware
• local set_fw_path=true
• local fw_path_config_file=/sys/module/firmware_class/parameters/path
• [[ true == \t\r\u\e ]]
• echo 'Configuring the following firmware search path in '''/sys/module/firmware_class/parameters/path''': /run/nvidia/driver/lib/firmware'
  Configuring the following firmware search path in '/sys/module/firmware_class/parameters/path': /run/nvidia/driver/lib/firmware
  ++ grep '[^[:space:]]' /sys/module/firmware_class/parameters/path
• [[ ! -z '' ]]
• echo -n /run/nvidia/driver/lib/firmware
  Loading ipmi and i2c_core kernel modules...
• echo 'Loading ipmi and i2c_core kernel modules...'
• modprobe -a i2c_core ipmi_msghandler ipmi_devintf
• [[ 6 -ge 3 ]]
• echo 'Loading the video kernel module...'
• modprobe video
  Loading the video kernel module...
• echo 'Loading NVIDIA driver kernel modules...'
• set -o xtrace +o nounset
  Loading NVIDIA driver kernel modules...
• modprobe nvidia
• modprobe nvidia-uvm
• modprobe nvidia-modeset
• set +o xtrace -o nounset
  Starting NVIDIA persistence daemon...
  ls: cannot access '/proc/driver/nvidia-nvswitch/devices/*': No such file or directory
  Mounting NVIDIA driver rootfs...
  Check SELinux status
  SELinux is enabled
  Change device files security context for selinux compatibility
  chcon: can't apply partial context to unlabeled file 'nvidia-modeset'
  chcon: can't apply partial context to unlabeled file 'nvidia0'
  chcon: can't apply partial context to unlabeled file 'nvidiactl'
  chcon: can't apply partial context to unlabeled file 'core'
  chcon: can't apply partial context to unlabeled file 'stderr'
  chcon: can't apply partial context to unlabeled file 'stdout'
  chcon: can't apply partial context to unlabeled file 'stdin'
  chcon: can't apply partial context to unlabeled file 'fd'
  chcon: can't apply partial context to unlabeled file 'ptmx'
  chcon: can't apply partial context to unlabeled file 'zero'
  chcon: can't apply partial context to unlabeled file 'vhost-vsock'
  chcon: can't apply partial context to unlabeled file 'vhost-net'
  chcon: can't apply partial context to unlabeled file 'vhci'
  chcon: can't apply partial context to unlabeled file 'vga_arbiter'
  chcon: can't apply partial context to unlabeled file 'vfio'
  chcon: can't apply partial context to unlabeled file 'vfio'
  chcon: can't apply partial context to unlabeled file 'vcsu6'
  chcon: can't apply partial context to unlabeled file 'vcsu5'
  chcon: can't apply partial context to unlabeled file 'vcsu4'
  chcon: can't apply partial context to unlabeled file 'vcsu3'
  chcon: can't apply partial context to unlabeled file 'vcsu2'
  chcon: can't apply partial context to unlabeled file 'vcsu1'
  chcon: can't apply partial context to unlabeled file 'vcsu'
  chcon: can't apply partial context to unlabeled file 'vcsa6'
  chcon: can't apply partial context to unlabeled file 'vcsa5'
  chcon: can't apply partial context to unlabeled file 'vcsa4'
  chcon: can't apply partial context to unlabeled file 'vcsa3'
  chcon: can't apply partial context to unlabeled file 'vcsa2'
  chcon: can't apply partial context to unlabeled file 'vcsa1'
  chcon: can't apply partial context to unlabeled file 'vcsa'
  chcon: can't apply partial context to unlabeled file 'vcs6'
  chcon: can't apply partial context to unlabeled file 'vcs5'
  chcon: can't apply partial context to unlabeled file 'vcs4'
  chcon: can't apply partial context to unlabeled file 'vcs3'
  chcon: can't apply partial context to unlabeled file 'vcs2'
  chcon: can't apply partial context to unlabeled file 'vcs1'
  chcon: can't apply partial context to unlabeled file 'vcs'
  chcon: can't apply partial context to unlabeled file 'userfaultfd'
  chcon: can't apply partial context to unlabeled file 'usbmon2'
  chcon: can't apply partial context to unlabeled file 'usbmon1'
  chcon: can't apply partial context to unlabeled file 'usbmon0'
  chcon: can't apply partial context to unlabeled file 'urandom'
  chcon: can't apply partial context to unlabeled file 'uinput'
  chcon: can't apply partial context to unlabeled file 'uhid'
  chcon: can't apply partial context to unlabeled file 'udmabuf'
  chcon: can't apply partial context to unlabeled file 'ttyS3'
  chcon: can't apply partial context to unlabeled file 'ttyS2'
  chcon: can't apply partial context to unlabeled file 'ttyS1'
  chcon: can't apply partial context to unlabeled file 'ttyS0'
  chcon: can't apply partial context to unlabeled file 'tty9'
  chcon: can't apply partial context to unlabeled file 'tty8'
  chcon: can't apply partial context to unlabeled file 'tty7'
  chcon: can't apply partial context to unlabeled file 'tty63'
  chcon: can't apply partial context to unlabeled file 'tty62'
  chcon: can't apply partial context to unlabeled file 'tty61'
  chcon: can't apply partial context to unlabeled file 'tty60'
  chcon: can't apply partial context to unlabeled file 'tty6'
  chcon: can't apply partial context to unlabeled file 'tty59'
  chcon: can't apply partial context to unlabeled file 'tty58'
  chcon: can't apply partial context to unlabeled file 'tty57'
  chcon: can't apply partial context to unlabeled file 'tty56'
  chcon: can't apply partial context to unlabeled file 'tty55'
  chcon: can't apply partial context to unlabeled file 'tty54'
  chcon: can't apply partial context to unlabeled file 'tty53'
  chcon: can't apply partial context to unlabeled file 'tty52'
  chcon: can't apply partial context to unlabeled file 'tty51'
  chcon: can't apply partial context to unlabeled file 'tty50'
  chcon: can't apply partial context to unlabeled file 'tty5'
  chcon: can't apply partial context to unlabeled file 'tty49'
  chcon: can't apply partial context to unlabeled file 'tty48'
  chcon: can't apply partial context to unlabeled file 'tty47'
  chcon: can't apply partial context to unlabeled file 'tty46'
  chcon: can't apply partial context to unlabeled file 'tty45'
  chcon: can't apply partial context to unlabeled file 'tty44'
  chcon: can't apply partial context to unlabeled file 'tty43'
  chcon: can't apply partial context to unlabeled file 'tty42'
  chcon: can't apply partial context to unlabeled file 'tty41'
  chcon: can't apply partial context to unlabeled file 'tty40'
  chcon: can't apply partial context to unlabeled file 'tty4'
  chcon: can't apply partial context to unlabeled file 'tty39'
  chcon: can't apply partial context to unlabeled file 'tty38'
  chcon: can't apply partial context to unlabeled file 'tty37'
  chcon: can't apply partial context to unlabeled file 'tty36'
  chcon: can't apply partial context to unlabeled file 'tty35'
  chcon: can't apply partial context to unlabeled file 'tty34'
  chcon: can't apply partial context to unlabeled file 'tty33'
  chcon: can't apply partial context to unlabeled file 'tty32'
  chcon: can't apply partial context to unlabeled file 'tty31'
  chcon: can't apply partial context to unlabeled file 'tty30'
  chcon: can't apply partial context to unlabeled file 'tty3'
  chcon: can't apply partial context to unlabeled file 'tty29'
  chcon: can't apply partial context to unlabeled file 'tty28'
  chcon: can't apply partial context to unlabeled file 'tty27'
  chcon: can't apply partial context to unlabeled file 'tty26'
  chcon: can't apply partial context to unlabeled file 'tty25'
  chcon: can't apply partial context to unlabeled file 'tty24'
  chcon: can't apply partial context to unlabeled file 'tty23'
  chcon: can't apply partial context to unlabeled file 'tty22'
  chcon: can't apply partial context to unlabeled file 'tty21'
  chcon: can't apply partial context to unlabeled file 'tty20'
  chcon: can't apply partial context to unlabeled file 'tty2'
  chcon: can't apply partial context to unlabeled file 'tty19'
  chcon: can't apply partial context to unlabeled file 'tty18'
  chcon: can't apply partial context to unlabeled file 'tty17'
  chcon: can't apply partial context to unlabeled file 'tty16'
  chcon: can't apply partial context to unlabeled file 'tty15'
  chcon: can't apply partial context to unlabeled file 'tty14'
  chcon: can't apply partial context to unlabeled file 'tty13'
  chcon: can't apply partial context to unlabeled file 'tty12'
  chcon: can't apply partial context to unlabeled file 'tty11'
  chcon: can't apply partial context to unlabeled file 'tty10'
  chcon: can't apply partial context to unlabeled file 'tty1'
  chcon: can't apply partial context to unlabeled file 'tty0'
  chcon: can't apply partial context to unlabeled file 'tty'
  chcon: can't apply partial context to unlabeled file 'sr0'
  chcon: can't apply partial context to unlabeled file 'timer'
  chcon: can't apply partial context to unlabeled file 'seq'
  chcon: can't apply partial context to unlabeled file 'snd'
  chcon: can't apply partial context to unlabeled file 'snapshot'
  chcon: can't apply partial context to unlabeled file 'sg4'
  chcon: can't apply partial context to unlabeled file 'sg3'
  chcon: can't apply partial context to unlabeled file 'sg2'
  chcon: can't apply partial context to unlabeled file 'sg1'
  chcon: can't apply partial context to unlabeled file 'sg0'
  chcon: can't apply partial context to unlabeled file 'sdc'
  chcon: can't apply partial context to unlabeled file 'sdb'
  chcon: can't apply partial context to unlabeled file 'sda3'
  chcon: can't apply partial context to unlabeled file 'sda2'
  chcon: can't apply partial context to unlabeled file 'sda1'
  chcon: can't apply partial context to unlabeled file 'sda'
  chcon: can't apply partial context to unlabeled file 'rtc0'
  chcon: can't apply partial context to unlabeled file 'rfkill'
  chcon: can't apply partial context to unlabeled file 'random'
  chcon: can't apply partial context to unlabeled file 'ptp5'
  chcon: can't apply partial context to unlabeled file 'ptp4'
  chcon: can't apply partial context to unlabeled file 'ptp3'
  chcon: can't apply partial context to unlabeled file 'ptp2'
  chcon: can't apply partial context to unlabeled file 'ptp1'
  chcon: can't apply partial context to unlabeled file 'ptp0'
  chcon: can't apply partial context to unlabeled file 'ppp'
  chcon: can't apply partial context to unlabeled file 'port'
  chcon: can't apply partial context to unlabeled file 'nvram'
  chcon: can't apply partial context to unlabeled file 'nvme5n1p1'
  chcon: can't apply partial context to unlabeled file 'nvme5n1'
  chcon: can't apply partial context to unlabeled file 'nvme5'
  chcon: can't apply partial context to unlabeled file 'nvme4n1'
  chcon: can't apply partial context to unlabeled file 'nvme4'
  chcon: can't apply partial context to unlabeled file 'nvme3n1p4'
  chcon: can't apply partial context to unlabeled file 'nvme3n1p3'
  chcon: can't apply partial context to unlabeled file 'nvme3n1p2'
  chcon: can't apply partial context to unlabeled file 'nvme3n1p1'
  chcon: can't apply partial context to unlabeled file 'nvme3n1'
  chcon: can't apply partial context to unlabeled file 'nvme3'
  chcon: can't apply partial context to unlabeled file 'nvme2n1'
  chcon: can't apply partial context to unlabeled file 'nvme2'
  chcon: can't apply partial context to unlabeled file 'nvme1n1p5'
  chcon: can't apply partial context to unlabeled file 'nvme1n1p4'
  chcon: can't apply partial context to unlabeled file 'nvme1n1p3'
  chcon: can't apply partial context to unlabeled file 'nvme1n1p2'
  chcon: can't apply partial context to unlabeled file 'nvme1n1p1'
  chcon: can't apply partial context to unlabeled file 'nvme1n1'
  chcon: can't apply partial context to unlabeled file 'nvme1'
  chcon: can't apply partial context to unlabeled file 'nvme0n1'
  chcon: can't apply partial context to unlabeled file 'nvme0'
  chcon: can't apply partial context to unlabeled file 'null'
  chcon: can't apply partial context to unlabeled file 'ng5n1'
  chcon: can't apply partial context to unlabeled file 'ng4n1'
  chcon: can't apply partial context to unlabeled file 'ng3n1'
  chcon: can't apply partial context to unlabeled file 'ng2n1'
  chcon: can't apply partial context to unlabeled file 'ng1n1'
  chcon: can't apply partial context to unlabeled file 'ng0n1'
  chcon: can't apply partial context to unlabeled file 'tun'
  chcon: can't apply partial context to unlabeled file 'net'
  chcon: can't apply partial context to unlabeled file 'mem'
  chcon: can't apply partial context to unlabeled file 'megaraid_sas_ioctl_node'
  chcon: can't apply partial context to unlabeled file 'md127'
  chcon: can't apply partial context to unlabeled file 'md126p1'
  chcon: can't apply partial context to unlabeled file 'md126'
  chcon: can't apply partial context to unlabeled file 'mcelog'
  chcon: can't apply partial context to unlabeled file 'control'
  chcon: can't apply partial context to unlabeled file 'mapper'
  chcon: can't apply partial context to unlabeled file 'lp3'
  chcon: can't apply partial context to unlabeled file 'lp2'
  chcon: can't apply partial context to unlabeled file 'lp1'
  chcon: can't apply partial context to unlabeled file 'lp0'
  chcon: can't apply partial context to unlabeled file 'loop-control'
  chcon: can't apply partial context to unlabeled file 'kvm'
  chcon: can't apply partial context to unlabeled file 'kmsg'
  chcon: can't apply partial context to unlabeled file 'isst_interface'
  chcon: can't apply partial context to unlabeled file 'ipmi0'
  chcon: can't apply partial context to unlabeled file 'mouse0'
  chcon: can't apply partial context to unlabeled file 'mice'
  chcon: can't apply partial context to unlabeled file 'event3'
  chcon: can't apply partial context to unlabeled file 'event2'
  chcon: can't apply partial context to unlabeled file 'event1'
  chcon: can't apply partial context to unlabeled file 'event0'
  chcon: can't apply partial context to unlabeled file 'input'
  chcon: can't apply partial context to unlabeled file 'uverbs5'
  chcon: can't apply partial context to unlabeled file 'uverbs4'
  chcon: can't apply partial context to unlabeled file 'uverbs3'
  chcon: can't apply partial context to unlabeled file 'uverbs2'
  chcon: can't apply partial context to unlabeled file 'uverbs1'
  chcon: can't apply partial context to unlabeled file 'uverbs0'
  chcon: can't apply partial context to unlabeled file 'infiniband'
  chcon: can't apply partial context to unlabeled file 'hwrng'
  chcon: can't apply partial context to unlabeled file 'hpet'
  chcon: can't apply partial context to unlabeled file 'hidraw1'
  chcon: can't apply partial context to unlabeled file 'hidraw0'
  chcon: can't apply partial context to unlabeled file 'fuse'
  chcon: can't apply partial context to unlabeled file 'full'
  chcon: can't apply partial context to unlabeled file 'fb0'
  chcon: can't apply partial context to unlabeled file 'card0'
  chcon: can't apply partial context to unlabeled file 'dri'
  chcon: can't apply partial context to unlabeled file 'system'
  chcon: can't apply partial context to unlabeled file 'dma_heap'
  chcon: can't apply partial context to unlabeled file 'dm-2'
  chcon: can't apply partial context to unlabeled file 'dm-1'
  chcon: can't apply partial context to unlabeled file 'dm-0'
  chcon: can't apply partial context to unlabeled file 'cpu_dma_latency'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '9'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '8'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '7'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '63'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '62'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '61'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '60'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '6'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '59'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '58'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '57'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '56'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '55'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '54'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '53'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '52'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '51'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '50'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '5'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '49'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '48'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '47'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '46'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '45'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '44'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '43'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '42'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '41'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '40'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '4'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '39'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '38'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '37'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '36'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '35'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '34'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '33'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '32'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '31'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '30'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '3'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '29'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '28'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '27'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '26'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '25'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '24'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '23'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '22'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '21'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '20'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '2'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '19'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '18'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '17'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '16'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '15'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '14'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '13'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '12'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '11'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '10'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '1'
  chcon: can't apply partial context to unlabeled file 'msr'
  chcon: can't apply partial context to unlabeled file 'cpuid'
  chcon: can't apply partial context to unlabeled file '0'
  chcon: can't apply partial context to unlabeled file 'cpu'
  chcon: can't apply partial context to unlabeled file '001'
  chcon: can't apply partial context to unlabeled file '002'
  chcon: can't apply partial context to unlabeled file '006'
  chcon: can't apply partial context to unlabeled file '005'
  chcon: can't apply partial context to unlabeled file '004'
  chcon: can't apply partial context to unlabeled file '003'
  chcon: can't apply partial context to unlabeled file '002'
  chcon: can't apply partial context to unlabeled file '001'
  chcon: can't apply partial context to unlabeled file '001'
  chcon: can't apply partial context to unlabeled file 'usb'
  chcon: can't apply partial context to unlabeled file 'bus'
  chcon: can't apply partial context to unlabeled file '2:3:111:0'
  chcon: can't apply partial context to unlabeled file '2:1:123:0'
  chcon: can't apply partial context to unlabeled file '1:0:0:1'
  chcon: can't apply partial context to unlabeled file '1:0:0:0'
  chcon: can't apply partial context to unlabeled file '0:0:0:0'
  chcon: can't apply partial context to unlabeled file 'bsg'
  chcon: can't apply partial context to unlabeled file 'autofs'
  chcon: can't apply partial context to unlabeled file 'shm'
  chcon: can't apply partial context to unlabeled file 'termination-log'
  chcon: can't apply partial context to unlabeled file 'log'
  chcon: failed to get security context of 'mqueue': Operation not supported
  chcon: failed to get security context of 'ptmx': Operation not supported
  chcon: failed to get security context of 'pts': Operation not supported
  chcon: can't apply partial context to unlabeled file '/run/nvidia/driver/dev'
  Stopping NVIDIA persistence daemon...
  Unloading NVIDIA driver kernel modules...
  Unmounting NVIDIA driver rootfs...

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.

[shivamerla]

The chcon failures in the driver container are coming from how we detect SELinux. Right now we only check for the presence of /sys/fs/selinux and, if it exists, we assume SELinux is enabled and try to run chcon with container_file_t on all dev nodes created by the driver container.

On hosts where the kernel is built with SELinux and selinuxfs is mounted, /sys/fs/selinux can still be present even when SELinux is effectively disabled or set in a permissive mode. In that case our check passes, we call chcon, and it fails.

Instead of using the filesystem check, we should query SELinux directly (e.g. selinuxenabled, getenforce, or sestatus) and only run chcon when SELinux reports as enabled. If SELinux is disabled, or if the SELinux tools are not available in the container, we should skip the relabel step for the driver files.

[Ashutosh0x]

Hi @shivamerla ! I am interested in working on this issue.

Based on your guidance, I plan to:

1. Replace the filesystem check (/sys/fs/selinux) with a direct query to SELinux tools (e.g., selinuxenabled, getenforce, or sestatus).
2. Skip executing chcon if SELinux reports as disabled or if these tools are missing in the container.

Please let me know if this approach sounds good or if there are any other specific edge cases I should handle. Should I go ahead and open a PR with these changes?

[ssalagam]

This bug is resolved by NVIDIA/gpu-driver-container#539