Skip to content

userspace-resource-manager: Modify test runner for URM #352

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kartnema wants to merge 1 commit into
base: main
Choose a base branch
from
+64 −122
Open

userspace-resource-manager: Modify test runner for URM #352

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 23 additions & 42 deletions ...ites/Performance/resource-tuner/README.md → ...ance/userspace-resource-manager/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,45 +1,26 @@
# `resource-tuner` Test Runner (`run.sh`)
# `userspace-resource-manager` Test Runner (`run.sh`)

A pinned **whitelist** test runner for `resource-tuner` that produces per-suite logs and an overall gating result for CI.
A pinned **whitelist** test runner for `userspace-resource-manager` that produces per-suite logs and an overall gating result for CI.

---

## What this runs

Only these binaries are executed, in this order (anything else is ignored):
```
/usr/bin/ClientDataManagerTests
/usr/bin/ResourceProcessorTests
/usr/bin/MemoryPoolTests
/usr/bin/SignalConfigProcessorTests
/usr/bin/DeviceInfoTests
/usr/bin/ThreadPoolTests
/usr/bin/MiscTests
/usr/bin/SignalParsingTests
/usr/bin/SafeOpsTests
/usr/bin/ExtensionIntfTests
/usr/bin/RateLimiterTests
/usr/bin/SysConfigAPITests
/usr/bin/ExtFeaturesParsingTests
/usr/bin/RequestMapTests
/usr/bin/TargetConfigProcessorTests
/usr/bin/InitConfigParsingTests
/usr/bin/RequestQueueTests
/usr/bin/CocoTableTests
/usr/bin/ResourceParsingTests
/usr/bin/TimerTests
/usr/bin/resource_tuner_tests
/usr/bin/UrmComponentTests
/usr/bin/UrmIntegrationTests
```

---

## Gating policy

* **Service check (early gate):** If `resource-tuner.service` is **not active**, the test **SKIPs overall** and exits.
* **Service check (early gate):** If `urm.service` is **not active**, the test **SKIPs overall** and exits.
* **Per‑suite SKIP conditions (neutral):**
* Missing binary → **SKIP that suite**, continue.
* Missing base configs → **SKIP that suite**, continue.
* Missing test nodes for `resource_tuner_tests` → **SKIP that suite**, continue.
* Missing test nodes for `UrmIntegrationTests` → **SKIP that suite**, continue.
* **Final result:**
* If **any** suite **FAILS** → **overall FAIL**.
* Else if **≥1** suite **PASS** → **overall PASS**.
Expand All @@ -52,7 +33,7 @@ Only these binaries are executed, in this order (anything else is ignored):
## Pre‑checks

### 1) Service
The runner uses the repo helper `check_systemd_services()` to verify **`resource-tuner.service`** is active.
The runner uses the repo helper `check_systemd_services()` to verify **`urm.service`** is active.
- On failure: overall **SKIP** (ends early).
- Override service name: `SERVICE_NAME=your.service ./run.sh`

Expand All @@ -62,19 +43,19 @@ Suites that parse configs require **at least one** of these base config trees:
- `common/` (required files):
- `InitConfig.yaml`, `PropertiesConfig.yaml`, `ResourcesConfig.yaml`, `SignalsConfig.yaml`

- `custom/` (required files):
- `tests/configs/` (required files):
- `InitConfig.yaml`, `PropertiesConfig.yaml`, `ResourcesConfig.yaml`, `SignalsConfig.yaml`, `TargetConfig.yaml`, `ExtFeaturesConfig.yaml`

If **both** trees are missing required files/dirs, config‑parsing suites are **SKIP** only (neutral).

> Override required file lists without editing the script:
```bash
export RT_REQUIRE_COMMON_FILES="InitConfig.yaml PropertiesConfig.yaml ResourcesConfig.yaml SignalsConfig.yaml"
export RT_REQUIRE_CUSTOM_FILES="InitConfig.yaml PropertiesConfig.yaml ResourcesConfig.yaml SignalsConfig.yaml TargetConfig.yaml ExtFeaturesConfig.yaml"
export RT_REQUIRE_TEST_FILES="InitConfig.yaml PropertiesConfig.yaml ResourcesConfig.yaml SignalsConfig.yaml TargetConfig.yaml ExtFeaturesConfig.yaml Baseline.yaml"
```

### 3) Test ResourceSysFsNodes
`/etc/resource-tuner/tests/Configs/ResourceSysFsNodes` must exist and be non‑empty for **`/usr/bin/resource_tuner_tests`**. If missing/empty → **SKIP only that suite**.
### 3) Test test nodes
`/etc/urm/tests/nodes` must exist and be non‑empty for **`/usr/bin/UrmIntegrationTests`** and **`/usr/bin/UrmComponentTests`**. If missing/empty → **SKIP only that suite**.

### 4) Base tools
Requires: `awk`, `grep`, `date`, `printf`. If missing → **overall SKIP**.
Expand All @@ -93,21 +74,21 @@ Usage: ./run.sh [--all] [--bin <name|absolute>] [--list] [--timeout SECS]
- `--timeout SECS`: default per‑binary timeout **if** `run_with_timeout()` helper exists (else ignored).

Per‑suite default timeouts (if helper is present):
- `ThreadPoolTests`, `RateLimiterTests`: **1800s**
- `resource_tuner_tests`: **2400s**
- `UrmComponentTests`: **1800s**
- `UrmIntegrationTests`: **2400s**
- others: **1200s** (default)

---

## Output layout

- **Overall status file:** `./resource-tuner.res` → `PASS` / `FAIL` / `SKIP`
- **Logs directory:** `./logs/resource-tuner-YYYYMMDD-HHMMSS/`
- **Overall status file:** `./userspace-resource-manager.res` → `PASS` / `FAIL` / `SKIP`
- **Logs directory:** `./logs/userspace-resource-manager-YYYYMMDD-HHMMSS/`
- Per‑suite logs: `SUITE.log`
- Per‑suite result markers: `SUITE.res` (`PASS`/`FAIL`/`SKIP`)
- Coverage summaries: `coverage.txt`, `missing_bins.txt`, `coverage_counts.env`
- System snapshot: `dmesg_snapshot.log`
- **Symlink to latest:** `./logs/resource-tuner-latest`
- **Symlink to latest:** `./logs/userspace-resource-manager-latest`

**Parsing heuristics:** a suite is considered PASS if the binary exits 0 **or** its log contains
`Run Successful`, `executed successfully`, or `Ran Successfully`. Strings like `Assertion failed`, `Terminating Suite`, `Segmentation fault`, `Backtrace`, or `fail/failed` mark **FAIL**.
Expand All @@ -116,9 +97,9 @@ Per‑suite default timeouts (if helper is present):

## Environment overrides

- `SERVICE_NAME`: systemd unit to check (default: `resource-tuner.service`)
- `RT_CONFIG_DIR`: root of config tree (default: `/etc/resource-tuner`)
- `RT_REQUIRE_COMMON_FILES`, `RT_REQUIRE_CUSTOM_FILES`: *space‑separated* filenames that must exist in `common/` / `custom/` respectively to treat that tree as present.
- `SERVICE_NAME`: systemd unit to check (default: `urm.service`)
- `RT_CONFIG_DIR`: root of config tree (default: `/etc/urm`)
- `RT_REQUIRE_COMMON_FILES`, `RT_REQUIRE_CUSTOM_FILES`: *space‑separated* filenames that must exist in `common/` / `tests/` respectively to treat that tree as present.

---

Expand All @@ -131,7 +112,7 @@ Run all (normal CI mode):

Run a single suite by basename:
```bash
./run.sh --bin ResourceParsingTests
./run.sh --bin UrmComponentTests
```

List suites and presence coverage:
Expand All @@ -148,14 +129,14 @@ RT_CONFIG_DIR=/opt/rt/etc ./run.sh

## Exit status

The script writes the overall result to `resource-tuner.res`. The **process exit code is 0** in all cases in the current version (soft gating). If you want hard CI gating via non‑zero exit on FAIL, that can be added easily on request.
The script writes the overall result to `userspace-resource-manager.res`. The **process exit code is 0** in all cases in the current version (soft gating). If you want hard CI gating via non‑zero exit on FAIL, that can be added easily on request.

---

## Troubleshooting

- **Overall SKIP immediately** → service inactive. Check `systemctl status resource-tuner.service`.
- **Suite SKIP (config)** → confirm required files exist under `common/` or `custom/` (see lists above).
- **Overall SKIP immediately** → service inactive. Check `systemctl status urm.service`.
- **Suite SKIP (config)** → confirm required files exist under `common/` or `tests/` (see lists above).
- **Suite SKIP (missing bin)** → verify the binary is installed and executable under `/usr/bin`.
- **Suite FAIL** → inspect `logs/.../SUITE.log` for the first failure pattern or assertion.
- **Very long runs** → a `run_with_timeout` helper (if available in your repo toolchain) will be used automatically.
Expand Down
113 changes: 37 additions & 76 deletions .../suites/Performance/resource-tuner/run.sh → ...ormance/userspace-resource-manager/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/bin/sh
# Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries.
# SPDX-License-Identifier: BSD-3-Clause# resource-tuner test runner (pinned whitelist)
# SPDX-License-Identifier: BSD-3-Clause# userspace-resource-manager test runner (pinned whitelist)

# ---------- Repo env + helpers ----------
SCRIPT_DIR="$(
Expand Down Expand Up @@ -37,7 +37,7 @@ export PATH="/usr/sbin:/sbin:/usr/bin:/bin:${PATH}"
# shellcheck disable=SC3045
( ulimit -c unlimited ) >/dev/null 2>&1 || true

TESTNAME="resource-tuner"
TESTNAME="userspace-resource-manager"
test_path="$(find_test_case_by_name "$TESTNAME")"
cd "$test_path" || exit 1
RES_FILE="./${TESTNAME}.res"
Expand All @@ -62,42 +62,20 @@ fi

# ---------- Approved list (pinned whitelist) ----------
APPROVED_TESTS="
/usr/bin/ClientDataManagerTests
/usr/bin/ResourceProcessorTests
/usr/bin/MemoryPoolTests
/usr/bin/SignalConfigProcessorTests
/usr/bin/DeviceInfoTests
/usr/bin/ThreadPoolTests
/usr/bin/MiscTests
/usr/bin/SignalParsingTests
/usr/bin/SafeOpsTests
/usr/bin/ExtensionIntfTests
/usr/bin/RateLimiterTests
/usr/bin/SysConfigAPITests
/usr/bin/ExtFeaturesParsingTests
/usr/bin/RequestMapTests
/usr/bin/TargetConfigProcessorTests
/usr/bin/InitConfigParsingTests
/usr/bin/RequestQueueTests
/usr/bin/CocoTableTests
/usr/bin/ResourceParsingTests
/usr/bin/TimerTests
/usr/bin/resource_tuner_tests
/usr/bin/UrmComponentTests
/usr/bin/UrmIntegrationTests
"

# Suites that need base configs (accept either common/ OR custom/)
SUITES_REQUIRE_BASE_CFGS="ResourceProcessorTests SignalConfigProcessorTests SysConfigAPITests \
ExtFeaturesParsingTests TargetConfigProcessorTests InitConfigParsingTests \
ResourceParsingTests ExtensionIntfTests resource_tuner_tests"
# Suites that need base configs (accept either common/ OR tests/)
SUITES_REQUIRE_BASE_CFGS="UrmComponentTests UrmIntegrationTests"

# ---------- CLI ----------
print_usage() {
cat <<EOF
Usage: $0 [--all] [--bin <name|absolute>] [--list] [--timeout SECS]
Policy:
- Service INACTIVE => overall SKIP (end early)
- Base configs: suites require common/ OR custom/ (skip only if BOTH missing)
- resource_tuner_tests additionally needs tests/Configs/ResourceSysFsNodes
- Base configs: suites require common/ OR tests/ (skip only if BOTH missing)
- Any test FAIL => overall FAIL
- No FAIL & PASS>0 => overall PASS
- No FAIL & PASS=0 => overall SKIP (everything skipped)
Expand Down Expand Up @@ -180,10 +158,10 @@ parse_and_score_log() {
}
per_suite_timeout() {
case "$1" in
ThreadPoolTests|RateLimiterTests)
UrmComponentTests)
echo 1800
;;
resource_tuner_tests)
UrmIntegrationTests)
echo 2400
;;
*)
Expand Down Expand Up @@ -225,18 +203,18 @@ if command -v log_soc_info >/dev/null 2>&1; then
fi

# ---------- Service gate (use repo helper) ----------
SERVICE_NAME="${SERVICE_NAME:-resource-tuner.service}"
SERVICE_NAME="${SERVICE_NAME:-urm.service}"
log_info "[SERVICE] Checking $SERVICE_NAME via check_systemd_services()"
if check_systemd_services "$SERVICE_NAME"; then
log_pass "[SERVICE] $SERVICE_NAME is active"
else
log_warn "[SERVICE] $SERVICE_NAME not active — attempting enable/start"

if command -v systemctl >/dev/null 2>&1; then
systemctl enable resource-tuner >/dev/null 2>&1 || true
systemctl enable urm >/dev/null 2>&1 || true
systemctl daemon-reload >/dev/null 2>&1 || true
systemctl start resource-tuner >/dev/null 2>&1 || true
systemctl status resource-tuner --no-pager -l >/dev/null 2>&1 || true
systemctl start urm >/dev/null 2>&1 || true
systemctl status urm --no-pager -l >/dev/null 2>&1 || true
else
log_warn "[SERVICE] systemctl not available; cannot auto-start $SERVICE_NAME"
fi
Expand All @@ -250,64 +228,57 @@ else
fi
fi

# ---------- Config preflight (check both common/ and custom/) ----------
RT_CONFIG_DIR="${RT_CONFIG_DIR:-/etc/resource-tuner}"
COMMON_DIR="$RT_CONFIG_DIR/common"
CUSTOM_DIR="$RT_CONFIG_DIR/custom"
TEST_NODES_DIR="$RT_CONFIG_DIR/tests/Configs/ResourceSysFsNodes"
# ---------- Config preflight (check both common/ and tests/) ----------
URM_CONFIG_DIR="${URM_CONFIG_DIR:-/etc/urm}"
COMMON_DIR="$URM_CONFIG_DIR/common"
TEST_CONFIGS_DIR="$URM_CONFIG_DIR/tests/configs"
TEST_NODES_DIR="$URM_CONFIG_DIR/tests/nodes"

COMMON_OK=1
CUSTOM_OK=1
NODES_OK=1
COMMON_CONFIGS_OK=1
TEST_CONFIGS_OK=1
TEST_NODES_OK=1

REQ_COMMON_FILES="${RT_REQUIRE_COMMON_FILES:-InitConfig.yaml PropertiesConfig.yaml ResourcesConfig.yaml SignalsConfig.yaml}"
REQ_CUSTOM_FILES="${RT_REQUIRE_CUSTOM_FILES:-InitConfig.yaml PropertiesConfig.yaml ResourcesConfig.yaml SignalsConfig.yaml TargetConfig.yaml ExtFeaturesConfig.yaml}"
REQ_TEST_FILES="${RT_REQUIRE_TEST_FILES:-InitConfig.yaml PropertiesConfig.yaml ResourcesConfig.yaml SignalsConfig.yaml TargetConfig.yaml ExtFeaturesConfig.yaml Baseline.yaml}"

# common/
if [ ! -d "$COMMON_DIR" ]; then
log_warn "[CFG] Missing dir: $COMMON_DIR"
COMMON_OK=0
COMMON_CONFIGS_OK=0
else
for f in $REQ_COMMON_FILES; do
if [ ! -f "$COMMON_DIR/$f" ]; then
log_warn "[CFG] Missing file: $COMMON_DIR/$f"
COMMON_OK=0
COMMON_CONFIGS_OK=0
fi
done
fi

# custom/
if [ ! -d "$CUSTOM_DIR" ]; then
log_warn "[CFG] Missing dir: $CUSTOM_DIR"
CUSTOM_OK=0
# tests/configs
if [ ! -d "$TEST_CONFIGS_DIR" ]; then
log_warn "[CFG] Missing dir: $TEST_CONFIGS_DIR"
TEST_CONFIGS_OK=0
else
for f in $REQ_CUSTOM_FILES; do
if [ ! -f "$CUSTOM_DIR/$f" ]; then
log_warn "[CFG] Missing file: $CUSTOM_DIR/$f"
CUSTOM_OK=0
for f in $REQ_TEST_FILES; do
if [ ! -f "$TEST_CONFIGS_DIR/$f" ]; then
log_warn "[CFG] Missing file: $TEST_CONFIGS_DIR/$f"
TEST_CONFIGS_OK=0
fi
done
cn="$(
find "$CUSTOM_DIR/ResourceSysFsNodes" -mindepth 1 -maxdepth 1 -type f -print 2>/dev/null \
| wc -l | awk '{print $1}'
)"
if [ -n "$cn" ]; then
log_info "[CFG] custom/ResourceSysFsNodes entries: $cn"
fi
fi

# tests nodes (hard requirement for resource_tuner_tests)
# tests/nodes (hard requirement for UrmIntegrationTests and UrmComponentTests)
if [ ! -d "$TEST_NODES_DIR" ]; then
log_warn "[CFG] Missing dir: $TEST_NODES_DIR"
NODES_OK=0
TEST_NODES_OK=0
else
count_nodes="$(
find "$TEST_NODES_DIR" -mindepth 1 -maxdepth 1 -type f -print 2>/dev/null \
| wc -l | awk '{print $1}'
)"
if [ "${count_nodes:-0}" -le 0 ]; then
log_warn "[CFG] $TEST_NODES_DIR is empty"
NODES_OK=0
TEST_NODES_OK=0
fi
fi

Expand Down Expand Up @@ -399,26 +370,16 @@ run_one() {
return 2
fi

# base config requirement: accept common OR custom; skip only if BOTH missing
# base config requirement: accept common OR tests; skip only if BOTH missing
if suite_requires_base_cfgs "$name"; then
if [ $COMMON_OK -eq 0 ] && [ $CUSTOM_OK -eq 0 ]; then
if [ $COMMON_CONFIGS_OK -eq 0 ] && [ $TEST_CONFIGS_OK -eq 0 ] && [ $TEST_NODES_OK -eq 0 ]; then
Copy link
Contributor

@smuppand smuppand Mar 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this changes the gating semantics in a risky way. tests/nodes looks like a required input for the URM suites, but with this condition the suite will still run when configs are present and tests/nodes is missing. Should the nodes check remain a separate per-suite skip, instead of being folded into the “all three missing” condition?

Copy link
Author

@kartnema kartnema Mar 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the update test suites, both tests/nodes and tests/configs are required input, in addition to common configs.

The goal is since, running test case with tests/nodes present but tests/configs (or the other way around) does not make sense, hence a more exhaustive check needs to be added to see if all 3 requirements are satisfied. I'll make the code level checks, and change the comments as appropriate.

log_skip "[CFG] Base configs missing (common/ AND custom/) — skipping $name"
Copy link
Contributor

@smuppand smuppand Mar 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This still says custom/, but the new path is tests/configs. Please update the message so runtime logs match the new layout.

echo "SKIP" >"$tres"
echo "[SKIP] $name – base configs missing" >>"$LOGDIR/summary.txt"
return 2
fi
fi

# resource_tuner_tests also needs test nodes
if [ "$name" = "resource_tuner_tests" ]; then
if [ $NODES_OK -eq 0 ]; then
log_skip "[CFG] Test ResourceSysFsNodes missing/empty — skipping $name"
echo "SKIP" >"$tres"
echo "[SKIP] $name – test nodes missing" >>"$LOGDIR/summary.txt"
return 2
fi
fi

# resolve binary
if [ ! -x "$bin" ] && command -v "$bin" >/dev/null 2>&1; then
bin="$(command -v "$bin")"
Expand Down
8 changes: 4 additions & 4 deletions ...rmance/resource-tuner/resource-tuner.yaml → ...e-manager/userspace-resource-manager.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
metadata:
name: resource-tuner
name: userspace-resource-manager
format: "Lava-Test Test Definition 1.0"
description: "A pinned **whitelist** test runner for `resource-tuner` that produces per-suite logs and an overall gating result for CI."
description: "A pinned **whitelist** test runner for `userspace-resource-manager` that produces per-suite logs and an overall gating result for CI."
os:
- linux
scope:
Expand All @@ -14,8 +14,8 @@ params:
run:
steps:
- REPO_PATH=$PWD
- cd Runner/suites/Performance/resource-tuner
- cd Runner/suites/Performance/userspace-resource-manager
- RUN_OPTIONS="--all"
- if [ -n "${BIN_NAME}" ]; then RUN_OPTIONS="--bin ${BIN_NAME}"; fi
- ./run.sh --timeout "${TIMEOUT}" ${RUN_OPTIONS} || true
- $REPO_PATH/Runner/utils/send-to-lava.sh resource-tuner.res || true
- $REPO_PATH/Runner/utils/send-to-lava.sh userspace-resource-manager.res || true
Loading