Skip to content

[AMD/ROCm] qwen3.5 fp8 mi355x SGL performance update #995

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
seungrokj wants to merge 5 commits into
base: main
Choose a base branch
from
+35 −10
Draft

[AMD/ROCm] qwen3.5 fp8 mi355x SGL performance update #995

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
609d1f3
SGL qwen3.5 fp8 mi355x performance update
seungrokj Apr 2, 2026
7685bb0
SGL qwen3.5 fp8 mi355x performance update
seungrokj Apr 2, 2026
4f870c8
SGL qwen3.5 fp8 mi355x performance update
seungrokj Apr 2, 2026
cee748e
SGL qwen3.5 fp8 mi355x performance update
seungrokj Apr 2, 2026
4e42d21
SGL qwen3.5 fp8 mi355x performance update
seungrokj Apr 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions .github/configs/amd-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,7 @@ qwen3.5-fp8-mi325x-sglang:
- { tp: 8, conc-start: 4, conc-end: 64 }

qwen3.5-fp8-mi355x-sglang:
image: rocm/sgl-dev:v0.5.8.post1-rocm720-mi35x-20260218
image: lmsysorg/sglang-rocm:v0.5.10rc0-rocm720-mi35x-20260330
model: Qwen/Qwen3.5-397B-A17B-FP8
model-prefix: qwen3.5
runner: mi355x
Expand All @@ -197,11 +197,15 @@ qwen3.5-fp8-mi355x-sglang:
- isl: 1024
osl: 1024
search-space:
- { tp: 8, conc-start: 4, conc-end: 64 }
- { tp: 2, conc-start: 8, conc-end: 128 }
- { tp: 4, conc-start: 4, conc-end: 32 }
- { tp: 8, conc-start: 4, conc-end: 4 }
- isl: 8192
osl: 1024
search-space:
- { tp: 8, conc-start: 4, conc-end: 64 }
- { tp: 2, conc-start: 8, conc-end: 128 }
- { tp: 4, conc-start: 4, conc-end: 16 }
- { tp: 8, conc-start: 4, conc-end: 4 }

qwen3.5-fp8-mi300x-sglang:
image: lmsysorg/sglang:v0.5.9-rocm720-mi30x
Expand Down
28 changes: 21 additions & 7 deletions benchmarks/single_node/qwen3.5_fp8_mi355x.sh
100644 → 100755
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,29 +19,42 @@ hf download "$MODEL"

SERVER_LOG=/workspace/server.log
PORT=${PORT:-8888}
MEM_FRAC_STATIC=${MEM_FRAC_STATIC:-0.8}
CHUNK_SIZE=32768

EVAL_CONTEXT_ARGS=""
if [ "${EVAL_ONLY}" = "true" ]; then
setup_eval_context
EVAL_CONTEXT_ARGS="--context-length $EVAL_MAX_MODEL_LEN"
fi
#if [ "${EVAL_ONLY}" = "true" ]; then
# setup_eval_context
# EVAL_CONTEXT_ARGS="--context-length $EVAL_MAX_MODEL_LEN"
#fi
# Start GPU monitoring (power, temperature, clocks every second)
start_gpu_monitor

python3 -m sglang.launch_server \
set -x
sglang serve \
--attention-backend triton \
--model-path $MODEL \
--host=0.0.0.0 \
--port $PORT \
--tensor-parallel-size $TP \
--trust-remote-code \
--mem-fraction-static 0.8 $EVAL_CONTEXT_ARGS > $SERVER_LOG 2>&1 &
--mem-fraction-static $MEM_FRAC_STATIC \
--kv-cache-dtype fp8_e4m3 \
--cuda-graph-max-bs $CONC \
--max-running-requests $CONC \
--chunked-prefill-size $CHUNK_SIZE \
--max-prefill-tokens $CHUNK_SIZE \
--disable-radix-cache \
--num-continuous-decode-steps 2 \
$EVAL_CONTEXT_ARGS \
> $SERVER_LOG 2>&1 &

SERVER_PID=$!

# Wait for server to be ready
wait_for_server_ready --port "$PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID"

export PYTHONDONTWRITEBYTECODE=1
run_benchmark_serving \
--model "$MODEL" \
--port "$PORT" \
Expand All @@ -52,7 +65,8 @@ run_benchmark_serving \
--num-prompts "$((CONC * 10))" \
--max-concurrency "$CONC" \
--result-filename "$RESULT_FILENAME" \
--result-dir /workspace/
--result-dir /workspace/ \
--trust-remote-code

# After throughput, run evaluation only if RUN_EVAL is true
if [ "${RUN_EVAL}" = "true" ]; then
Expand Down
7 changes: 7 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1235,3 +1235,10 @@
- "New model support on ATOM framework"
- "Kimi-K2.5 FP4, and MiniMax-M2.5 FP8 configs added for MI355X ATOM"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/963

- config-keys:
- qwen3.5-fp8-mi355x-sglang
description:
- "Qwen3.5 fp8 mi355x performance update"
- "Relevant Issue: https://github.com/sgl-project/sglang/issues/19633"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/995
Copy link
Copy Markdown
Contributor

@claude claude bot Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The new changelog entry uses a placeholder PR link pull/9xx instead of the actual PR number. Since this PR is #995, the link should be https://github.com/SemiAnalysisAI/InferenceX/pull/995. Please update perf-changelog.yaml line 1244 to fix the broken link before merging.

Extended reasoning...

The changelog entry added by this PR (perf-changelog.yaml, last entry) sets pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/9xx. This is clearly a placeholder that was never replaced with the actual PR number.

The specific code path is straightforward: the diff shows the new entry was added at the bottom of perf-changelog.yaml with pull/9xx as the PR link. This PR is #995 based on the PR metadata, so the correct value would be pull/995.

Existing placeholder entries in the file consistently use the pattern pull/XXX (all uppercase letters) when the PR number is unknown at write time — for example, entries for kimik2.5-int4-mi300x-vllm, glm5-fp8-mi355x-sglang, minimaxm2.5-fp8-h200-vllm, qwen3.5-bf16-mi325x-sglang, and qwen3.5-fp8-mi325x-sglang all use pull/XXX. The pull/9xx pattern is a non-standard variant suggesting the author partially began filling in the number (starting with the known digit 9) but left it incomplete.

The impact is a broken/invalid hyperlink in the changelog. Anyone clicking the link to trace the history of the qwen3.5-fp8-mi355x-sglang configuration change would land on a 404 page instead of this PR.

The fix is a one-character change: replace pull/9xx with pull/995 on line 1244 of perf-changelog.yaml.

Step-by-step proof:

  1. PR metadata confirms this is PR [AMD/ROCm] qwen3.5 fp8 mi355x SGL performance update #995 (<pr number="995">)
  2. The diff shows the addition of pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/9xx
  3. Navigating to https://github.com/SemiAnalysisAI/InferenceX/pull/9xx would 404 since 9xx is not a valid PR number
  4. The correct URL https://github.com/SemiAnalysisAI/InferenceX/pull/995 would point to this very PR

Loading