From 4d4f27b8b3bb1beea5f57e0742f29c0ba6e24362 Mon Sep 17 00:00:00 2001
From: Komh <mail@guojing.io>
Date: Wed, 22 Apr 2026 15:05:11 +0800
Subject: [PATCH 1/2] [configure] Configure Kubelet Log Level Verbosity

---
 .../Configure_Kubelet_Log_Level_Verbosity.md  | 111 ++++++++++++++++++
 1 file changed, 111 insertions(+)
 create mode 100644 docs/en/solutions/Configure_Kubelet_Log_Level_Verbosity.md

diff --git a/docs/en/solutions/Configure_Kubelet_Log_Level_Verbosity.md b/docs/en/solutions/Configure_Kubelet_Log_Level_Verbosity.md
new file mode 100644
index 00000000..2b32ffef
--- /dev/null
+++ b/docs/en/solutions/Configure_Kubelet_Log_Level_Verbosity.md
@@ -0,0 +1,111 @@
+---
+kind:
+   - Troubleshooting
+products:
+   - Alauda Container Platform
+ProductsVersion:
+   - 4.1.0,4.2.x
+---
+## Issue
+
+When troubleshooting node-level problems, increasing the kubelet log verbosity helps identify the root cause. The default log level (`2`) may not provide enough detail for complex issues such as pod scheduling failures, volume mount errors, or container runtime communication problems.
+
+## Root Cause
+
+The kubelet supports configurable log verbosity levels ranging from `0` (least verbose) to `10` (most verbose). The default level is `2`, which provides basic operational information. Higher levels expose progressively more diagnostic data, but consume additional CPU, disk I/O, and memory on the node.
+
+## Resolution
+
+### Log Level Reference
+
+| Level Range | Purpose |
+|---|---|
+| 0 | Critical errors only |
+| 1–2 | Default operational output |
+| 3–4 | Debug-level information, suitable for most troubleshooting |
+| 5–8 | Trace-level output, verbose internal state dumps |
+| 9–10 | Maximum verbosity, rarely needed |
+
+### Persistent Configuration (Mutable Host OS)
+
+On mutable host OSes (standard Linux distributions with a writable `/etc`), set the kubelet log level persistently by adding or modifying the `--v` flag via a systemd drop-in file:
+
+```bash
+sudo mkdir -p /etc/systemd/system/kubelet.service.d/
+sudo tee /etc/systemd/system/kubelet.service.d/10-log-level.conf <<EOF
+[Service]
+Environment="KUBELET_LOG_LEVEL=4"
+ExecStart=
+ExecStart=/usr/bin/kubelet \$KUBELET_KUBECONFIG_ARGS \$KUBELET_CONFIG_ARGS \$KUBELET_LOG_LEVEL
+EOF
+sudo systemctl daemon-reload
+sudo systemctl restart kubelet
+```
+
+### Persistent Configuration (Immutable OS Nodes)
+
+On immutable-OS nodes — MicroOS, or any setup where `/etc` is backed by a read-mostly overlay that is reset on node upgrades or rollbacks — direct file edits under `/etc/systemd/system/kubelet.service.d/` **will not survive the next node update**. You may see the desired verbosity right after the change, then lose it silently when the node image is replaced.
+
+Persist the change through ACP's Immutable Infrastructure mechanism instead:
+
+- Define the drop-in file as part of the node configuration managed by ACP (under `configure/clusters/nodes`). The platform renders and re-applies it every time a node boots, so the override survives OS upgrades and rollbacks.
+- Trigger a rolling apply on the target node pool. ACP will cordon/drain, restart the kubelet with the new verbosity, and resume scheduling.
+- Revert the same way — update the node configuration to remove the override; do not `rm` the file directly on the node, because the mutation will be lost at the next reconcile.
+
+If the cluster spans both mutable and immutable nodes, scope the change to a node group / pool so that only the intended nodes carry the higher verbosity.
+
+### One-Time Change (Single Node)
+
+For temporary debugging on a single mutable-OS node without touching the persistent configuration, override the kubelet arguments directly on that node:
+
+```bash
+sudo systemctl edit kubelet
+```
+
+Add the following to raise verbosity to level 4:
+
+```ini
+[Service]
+Environment="KUBELET_EXTRA_ARGS=--v=4"
+```
+
+Then reload and restart:
+
+```bash
+sudo systemctl daemon-reload
+sudo systemctl restart kubelet
+```
+
+On immutable-OS nodes, prefer the Immutable Infrastructure flow above even for short investigations: running `systemctl edit` on a single node works until that node is re-imaged, at which point the change is gone without warning.
+
+> **Important:** Revert the log level back to the default (`2`) after collecting the necessary logs. Extended operation at high verbosity places significant load on node resources.
+
+## Diagnostic Steps
+
+Verify the current kubelet log level by inspecting the running process:
+
+```bash
+ps aux | grep kubelet | grep -o '\-\-v=[0-9]*'
+```
+
+Gather kubelet logs from a specific node:
+
+```bash
+kubectl get nodes
+kubectl debug node/<node-name> --image=busybox -- cat /host/var/log/kubelet.log
+```
+
+Alternatively, SSH into the node and use journalctl:
+
+```bash
+ssh <node-address>
+sudo journalctl -b -f -u kubelet.service
+```
+
+To collect logs from all nodes at once:
+
+```bash
+for n in $(kubectl get nodes --no-headers | awk '{print $1}'); do
+  ssh "$n" "sudo journalctl -u kubelet.service --since '1 hour ago'" > "${n}.kubelet.log"
+done
+```

From 7efbdfbd406ebfc5becd5d7b30de98ae0850f1a1 Mon Sep 17 00:00:00 2001
From: Komh <mail@guojing.io>
Date: Wed, 22 Apr 2026 19:01:35 +0800
Subject: [PATCH 2/2] [configure] Configure Kubelet Log Level Verbosity

---
 .../Configure_Kubelet_Log_Level_Verbosity.md  | 38 ++++++++++++++-----
 1 file changed, 28 insertions(+), 10 deletions(-)

diff --git a/docs/en/solutions/Configure_Kubelet_Log_Level_Verbosity.md b/docs/en/solutions/Configure_Kubelet_Log_Level_Verbosity.md
index 2b32ffef..68b627fb 100644
--- a/docs/en/solutions/Configure_Kubelet_Log_Level_Verbosity.md
+++ b/docs/en/solutions/Configure_Kubelet_Log_Level_Verbosity.md
@@ -26,22 +26,35 @@ The kubelet supports configurable log verbosity levels ranging from `0` (least v
 | 5–8 | Trace-level output, verbose internal state dumps |
 | 9–10 | Maximum verbosity, rarely needed |
 
-### Persistent Configuration (Mutable Host OS)
+### Persistent Configuration — KubeletConfiguration (preferred on kubeadm clusters)
 
-On mutable host OSes (standard Linux distributions with a writable `/etc`), set the kubelet log level persistently by adding or modifying the `--v` flag via a systemd drop-in file:
+On kubeadm-provisioned clusters the kubelet is configured through `/var/lib/kubelet/config.yaml`. Set the `verbosity` field there and restart kubelet:
+
+```bash
+# On the target node
+sudo sed -i 's/^\s*verbosity:.*/verbosity: 4/; t; $a\verbosity: 4' /var/lib/kubelet/config.yaml
+sudo systemctl restart kubelet
+```
+
+This works regardless of how the systemd unit passes arguments to kubelet, and kubeadm-based automation will preserve it across upgrades.
+
+### Persistent Configuration — systemd drop-in (fallback)
+
+If you cannot edit `config.yaml` (some operator-managed setups lock the file), override the kubelet `ExecStart` via a drop-in that **inlines the `--v` flag directly**. Setting a bare environment variable like `KUBELET_LOG_LEVEL=4` does **not** raise verbosity — the stock kubeadm systemd unit only expands the three specific variables `$KUBELET_KUBECONFIG_ARGS`, `$KUBELET_CONFIG_ARGS`, and `$KUBELET_KUBEADM_ARGS`; any other name (including `KUBELET_LOG_LEVEL` or `KUBELET_EXTRA_ARGS`) is silently ignored.
 
 ```bash
 sudo mkdir -p /etc/systemd/system/kubelet.service.d/
-sudo tee /etc/systemd/system/kubelet.service.d/10-log-level.conf <<EOF
+sudo tee /etc/systemd/system/kubelet.service.d/10-log-level.conf <<'EOF'
 [Service]
-Environment="KUBELET_LOG_LEVEL=4"
 ExecStart=
-ExecStart=/usr/bin/kubelet \$KUBELET_KUBECONFIG_ARGS \$KUBELET_CONFIG_ARGS \$KUBELET_LOG_LEVEL
+ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS --v=4
 EOF
 sudo systemctl daemon-reload
 sudo systemctl restart kubelet
 ```
 
+The `ExecStart=` (empty) line clears the parent unit's ExecStart; the second line rebuilds it with `--v=4` appended.
+
 ### Persistent Configuration (Immutable OS Nodes)
 
 On immutable-OS nodes — MicroOS, or any setup where `/etc` is backed by a read-mostly overlay that is reset on node upgrades or rollbacks — direct file edits under `/etc/systemd/system/kubelet.service.d/` **will not survive the next node update**. You may see the desired verbosity right after the change, then lose it silently when the node image is replaced.
@@ -56,17 +69,18 @@ If the cluster spans both mutable and immutable nodes, scope the change to a nod
 
 ### One-Time Change (Single Node)
 
-For temporary debugging on a single mutable-OS node without touching the persistent configuration, override the kubelet arguments directly on that node:
+For temporary debugging on a single mutable-OS node, use the same drop-in pattern shown above via `systemctl edit`:
 
 ```bash
 sudo systemctl edit kubelet
 ```
 
-Add the following to raise verbosity to level 4:
+Enter the override block — again, the flag must be inlined into `ExecStart`, not placed into a bare environment variable:
 
 ```ini
 [Service]
-Environment="KUBELET_EXTRA_ARGS=--v=4"
+ExecStart=
+ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS --v=4
 ```
 
 Then reload and restart:
@@ -82,10 +96,14 @@ On immutable-OS nodes, prefer the Immutable Infrastructure flow above even for s
 
 ## Diagnostic Steps
 
-Verify the current kubelet log level by inspecting the running process:
+Verify the current kubelet log level. On kubeadm-provisioned clusters kubelet typically does **not** carry `--v` on its command line — verbosity comes from `config.yaml` — so `ps` on its own reports nothing even when verbosity is set. Check both locations:
 
 ```bash
-ps aux | grep kubelet | grep -o '\-\-v=[0-9]*'
+# The KubeletConfiguration path (primary on kubeadm clusters)
+sudo grep -E '^\s*verbosity:' /var/lib/kubelet/config.yaml || echo "verbosity: (default 2)"
+
+# The command-line path (only populated if you added --v via a drop-in)
+ps aux | grep kubelet | grep -oE '\-\-v=[0-9]+' || echo "(no --v on cmdline)"
 ```
 
 Gather kubelet logs from a specific node: