Skip to content

add triton backend sampler #7015

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
yuanlehome wants to merge 4 commits into
base: release/2.4
Choose a base branch
from
Open

add triton backend sampler #7015

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
5ae6da0
add triton backend sampler
yuanlehome Mar 25, 2026
9507743
update
yuanlehome Mar 25, 2026
fb2cb9a
fix no triton
yuanlehome Mar 26, 2026
81bd34f
fix perf
yuanlehome Mar 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .flake8
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@ max-line-length = 119
# E402: module level import not at top of file
per-file-ignores =
__init__.py:F401,F403,E402
fastdeploy/model_executor/layers/sample/ops/top_k_top_p_triton.py:E241,E121,E131,E266
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.flake8 对整个 top_k_top_p_triton.py 忽略 E121/E131 这类缩进/对齐规则会掩盖真实的缩进错误（不仅是表格对齐）。如果只是为了常量表的列对齐，建议改成在表格段落局部使用 # noqa: E241 等，或仅关闭必要的规则，避免对整文件放开缩进检查。

Suggested change
fastdeploy/model_executor/layers/sample/ops/top_k_top_p_triton.py:E241,E121,E131,E266
fastdeploy/model_executor/layers/sample/ops/top_k_top_p_triton.py:E241,E266

Copilot uses AI. Check for mistakes.
2 changes: 1 addition & 1 deletion fastdeploy/envs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@
# Set attention backend. "NATIVE_ATTN", "APPEND_ATTN"
# and "MLA_ATTN" can be set currently.
"FD_ATTENTION_BACKEND": lambda: os.getenv("FD_ATTENTION_BACKEND", "APPEND_ATTN"),
# Set sampling class. "base", "base_non_truncated", "air" and "rejection" can be set currently.
# Set sampling class. "base", "base_non_truncated", "air", "rejection" and "triton" can be set currently.
"FD_SAMPLING_CLASS": lambda: os.getenv("FD_SAMPLING_CLASS", "base"),
Comment on lines +53 to 54
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR 描述仍是模板占位内容，缺少至少“Motivation/Modifications/Usage/Accuracy Tests”中的必要信息；这会影响评审与回归验证。建议补充：为何引入 Triton sampler、如何启用（FD_SAMPLING_CLASS=triton）、以及基本正确性/性能/精度验证结果或未提供的原因。

Copilot uses AI. Check for mistakes.
Comment on lines +53 to 54
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR 标题目前不符合仓库约定的“[CLASS]Title”格式（例如 [Feature] Add triton backend sampler）。请按模板中的 tag 列表补充一个明确的前缀，并确保 Title 首字母/动词风格统一。

Copilot uses AI. Check for mistakes.
# Set moe backend."cutlass","marlin" and "triton" can be set currently.
"FD_MOE_BACKEND": lambda: os.getenv("FD_MOE_BACKEND", "cutlass"),
Expand Down
Loading
Loading