Skip to content

Add rocprofv3 and att trace library#4797

Open
pfultz2 wants to merge 2 commits intodevelopfrom
rocprofv3
Open

Add rocprofv3 and att trace library#4797
pfultz2 wants to merge 2 commits intodevelopfrom
rocprofv3

Conversation

@pfultz2
Copy link
Copy Markdown
Collaborator

@pfultz2 pfultz2 commented Apr 17, 2026

Motivation

Technical Details

Changelog Category

Add a CHANGELOG.md entry for any option other than Not Applicable

    • Added: New functionality.
    • Changed: Changes to existing functionality.
    • Removed: Functionality or support that has been removed. (Compared to a previous release)
    • Optimized: Component performance that has been optimized or improved.
    • Resolved Issues: Known issues from a previous version that have been resolved.
    • Not Applicable: This PR is not to be included in the changelog.

@pfultz2 pfultz2 requested a review from causten as a code owner April 17, 2026 17:11
Copilot AI review requested due to automatic review settings April 17, 2026 17:11
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the project’s Ubuntu 22.04 development container to include ROCm profiling support (rocprofv3) and an ATT trace decoder library so profiling/trace workflows can run inside the container.

Changes:

  • Install rocprofiler-sdk via apt in the Docker image.
  • Download and place librocprof-trace-decoder.so into /opt/rocm/lib.
  • Minor Dockerfile formatting change (extra blank line).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread Dockerfile
apt-get clean && \
rm -rf /var/lib/apt/lists/*


Copy link

Copilot AI Apr 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There’s an extra blank line added here that doesn’t appear to separate logical sections. Consider removing it to keep the Dockerfile formatting consistent.

Suggested change

Copilot uses AI. Check for mistakes.
Comment thread Dockerfile Outdated
RUN ldconfig

# ATT library
RUN sudo wget -O /opt/rocm/lib/librocprof-trace-decoder.so https://github.com/ROCm/rocprof-trace-decoder/raw/refs/heads/amd-mainline/releases/linux_glibc_2_28_x86_64/librocprof-trace-decoder.so
Copy link

Copilot AI Apr 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sudo is used in this RUN step, but the Dockerfile never installs sudo and the build runs as root by default (no USER set). This will fail with sudo: not found; drop sudo (or explicitly install it / switch users if that’s intended).

Suggested change
RUN sudo wget -O /opt/rocm/lib/librocprof-trace-decoder.so https://github.com/ROCm/rocprof-trace-decoder/raw/refs/heads/amd-mainline/releases/linux_glibc_2_28_x86_64/librocprof-trace-decoder.so
RUN wget -O /opt/rocm/lib/librocprof-trace-decoder.so https://github.com/ROCm/rocprof-trace-decoder/raw/refs/heads/amd-mainline/releases/linux_glibc_2_28_x86_64/librocprof-trace-decoder.so

Copilot uses AI. Check for mistakes.
Comment thread Dockerfile Outdated
RUN ldconfig

# ATT library
RUN sudo wget -O /opt/rocm/lib/librocprof-trace-decoder.so https://github.com/ROCm/rocprof-trace-decoder/raw/refs/heads/amd-mainline/releases/linux_glibc_2_28_x86_64/librocprof-trace-decoder.so
Copy link

Copilot AI Apr 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This downloads a shared library from a moving branch URL (raw/refs/heads/...). That makes builds non-reproducible and increases supply-chain risk. Prefer pinning to an immutable release asset or commit SHA, and verify integrity (e.g., checksum/signature) as part of the build.

Copilot uses AI. Check for mistakes.
Comment thread Dockerfile Outdated
RUN ldconfig

# ATT library
RUN sudo wget -O /opt/rocm/lib/librocprof-trace-decoder.so https://github.com/ROCm/rocprof-trace-decoder/raw/refs/heads/amd-mainline/releases/linux_glibc_2_28_x86_64/librocprof-trace-decoder.so
Copy link

Copilot AI Apr 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ldconfig is run before this library is added to /opt/rocm/lib. Since /opt/rocm/lib is intended to be resolved via the dynamic loader cache, add an ldconfig after downloading the .so (or otherwise ensure the runtime can locate it) to avoid runtime/linker failures.

Suggested change
RUN sudo wget -O /opt/rocm/lib/librocprof-trace-decoder.so https://github.com/ROCm/rocprof-trace-decoder/raw/refs/heads/amd-mainline/releases/linux_glibc_2_28_x86_64/librocprof-trace-decoder.so
RUN sudo wget -O /opt/rocm/lib/librocprof-trace-decoder.so https://github.com/ROCm/rocprof-trace-decoder/raw/refs/heads/amd-mainline/releases/linux_glibc_2_28_x86_64/librocprof-trace-decoder.so && \
ldconfig

Copilot uses AI. Check for mistakes.
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 17, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.

Additional details and impacted files
@@           Coverage Diff            @@
##           develop    #4797   +/-   ##
========================================
  Coverage    92.49%   92.49%           
========================================
  Files          583      583           
  Lines        29561    29562    +1     
========================================
+ Hits         27342    27343    +1     
  Misses        2219     2219           

see 30 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@causten
Copy link
Copy Markdown
Collaborator

causten commented Apr 18, 2026

Test Batch New Rate (08f2d9) Old Rate (0681c6) Diff Status
torchvision-resnet50 64 866.83 3,166.53 -72.63% 🔴
torchvision-resnet50_fp16 64 732.05 6,820.89 -89.27% 🔴
torchvision-densenet121 32 2,277.00 2,422.10 -5.99% 🔴
torchvision-densenet121_fp16 32 4,244.68 4,086.15 3.88%
torchvision-inceptionv3 32 236.17 1,658.01 -85.76% 🔴
torchvision-inceptionv3_fp16 32 283.31 2,677.22 -89.42% 🔴
cadene-inceptionv4 16 247.49 789.08 -68.64% 🔴
cadene-resnext64x4 16 434.96 790.86 -45.00% 🔴
slim-mobilenet 64 3,620.44 8,521.12 -57.51% 🔴
slim-nasnetalarge 64 119.27 218.37 -45.38% 🔴
slim-resnet50v2 64 1,479.97 3,288.39 -54.99% 🔴
bert-mrpc-onnx 8 298.03 1,108.72 -73.12% 🔴
bert-mrpc-tf 1 78.80 456.31 -82.73% 🔴
pytorch-examples-wlang-gru 1 136.96 338.72 -59.56% 🔴
pytorch-examples-wlang-lstm 1 453.74 452.66 0.24%
torchvision-resnet50_1 1 636.63 748.69 -14.97% 🔴
cadene-dpn92_1 1 235.14 403.71 -41.76% 🔴
cadene-resnext101_1 1 366.58 323.10 13.46% 🔆
onnx-taau-downsample 1 201.59 391.86 -48.55% 🔴
dlrm-criteoterabyte 1 18.74 31.56 -40.62% 🔴
dlrm-criteoterabyte_fp16 1 51.86 50.03 3.67%
agentmodel 1 10,441.73 10,347.89 0.91%
unet_fp16 2 20.27 55.66 -63.58% 🔴
resnet50v1_fp16 1 201.27 994.78 -79.77% 🔴
resnet50v1_int8 1 549.99 943.47 -41.71% 🔴
bert_base_cased_fp16 64 1,050.01 1,086.36 -3.35%
bert_large_uncased_fp16 32 346.53 338.39 2.41%
bert_large_fp16 1 199.59 198.67 0.46%
distilgpt2_fp16 16 1,477.10 2,066.44 -28.52% 🔴
yolov5s 1 552.64 549.43 0.58%
tinyllama 1 45.38 43.80 3.62%
vicuna-fastchat 1 42.14 42.63 -1.15%
whisper-tiny-encoder 1 412.47 406.13 1.56%
whisper-tiny-decoder 1 408.41 395.21 3.34%
llama2_7b 1 20.20 19.10 5.79% 🔆
qwen1.5-7b 1 23.35 23.42 -0.29%
phi3-3.8b 1 26.64 26.63 0.05%
llama3-8b 1 21.62 21.68 -0.26%
whisper-large-encoder 1 10.23 10.12 1.04%
whisper-large-decoder 1 101.16 101.03 0.13%
mistral-7b 1 23.45 23.64 -0.78%
FLUX.1-schnell 1 748.26 737.25 1.49%

Regressions detected 🔴

@causten
Copy link
Copy Markdown
Collaborator

causten commented Apr 18, 2026

Test Status Result
bert-mrpc-onnx PASSED: MIGraphX meets tolerance
bert-mrpc-tf PASSED: MIGraphX meets tolerance
pytorch-examples-wlang-gru PASSED: MIGraphX meets tolerance
pytorch-examples-wlang-lstm PASSED: MIGraphX meets tolerance
dlrm-criteoterabyte PASSED: MIGraphX meets tolerance
agentmodel PASSED: MIGraphX meets tolerance
unet PASSED: MIGraphX meets tolerance
resnet50v1 PASSED: MIGraphX meets tolerance
bert_base_cased_fp16 PASSED: MIGraphX meets tolerance
bert_large_uncased_fp16 🔴 FAILED: MIGraphX is not within tolerance - check verbose output
bert_large PASSED: MIGraphX meets tolerance
yolov5s PASSED: MIGraphX meets tolerance
tinyllama PASSED: MIGraphX meets tolerance
vicuna-fastchat PASSED: MIGraphX meets tolerance
whisper-tiny-encoder PASSED: MIGraphX meets tolerance
whisper-tiny-decoder PASSED: MIGraphX meets tolerance
distilgpt2_fp16 🔴 FAILED: MIGraphX is not within tolerance - check verbose output
llama2_7b PASSED: MIGraphX meets tolerance
qwen1.5-7b PASSED: MIGraphX meets tolerance
phi3-3.8b PASSED: MIGraphX meets tolerance
llama3-8b PASSED: MIGraphX meets tolerance
whisper-large-encoder ERROR - check error output
traceback
Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 360, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 211, in main
model = migraphx.parse_onnx(model_name, default_dim_value=batch)
RuntimeError: /src/AMDMIGraphX/src/include/migraphx/op/convolution.hpp:103: normalize_compute_shape: CONVOLUTION: mismatched channel numbers
whisper-large-decoder PASSED: MIGraphX meets tolerance
mistral-7b PASSED: MIGraphX meets tolerance
FLUX.1-schnell PASSED: MIGraphX meets tolerance

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants