ClaudeBot/llama.cpp
0
0
Fork 1
mirror of https://github.com/ggml-org/llama.cpp.git synced 2026-09-17 20:31:47 +02:00
Code Issues Packages Projects Releases Wiki Activity

HIP: broaden MoE ncols_opt tile heuristic on RDNA3.5 architecture (#28935)

Browse Source 
It's found the MoE ncols_opt tile heuristic needs to be broadened
to include the RDNA3.5 architecture.

The code change is implemented in ggml/src/ggml-cuda/mmq.cu
and just change the GGML_CUDA_CC_IS_RDNA3_0 to
GGML_CUDA_CC_IS_RDNA3 in the condition.
The dense dispatch logic remains unchanged.
The Test machine configuration we used is
AMD Radeon 8060S, gfx1151 (RDNA3.5), 20 CU, wave32
+ AMD Ryzen AI MAX+ 388, 8C/16T, 23.79 GB RAM

we complete the Correctness verification and performance evaluation as follows:
  test-backend-ops test -b ROCm0 -o MUL_MAT    -p type_a=<q4_K|q5_K|q4_0|q5_0>
  test-backend-ops test -b ROCm0 -o MUL_MAT_ID -p type_a=<q4_K|q5_K|q4_0|q5_0>
  all pass: MUL_MAT 64/64, 29/29, 48/48, 14/14;
            MUL_MAT_ID 84/84, 3/3, 74/74, 3/3

Performance result on target machine:
  LFM2.5-8B-A1B-UD-Q4_K_M  (Q4_K MoE)   +16.198%  [+12.704, +19.799]   8/8
  Qwen1.5-MoE-A2.7B-Q2_K   (Q2_K MoE)    +6.189%  [ +5.245,  +7.141]   8/8
  pooled (16 pairs)                     +11.081%  [ +7.972, +14.279]  16/16

Token generation (tg128) is unchanged on the Q4_K MoE model and +2.188%
[+0.905, +3.488] on the Q2_K one.
This commit is contained in:
WenqiangJia2026
2026-09-16 09:55:02 +02:00
committed by GitHub
parent 0bec16e388
commit fccf7166fb
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ggml/src/ggml-cuda/mmq.cu
+1 -1
View File
@@ -247,7 +247,7 @@ void ggml_cuda_mul_mat_q(
// Each expert only sees ne12*n_expert_used/ne02 tokens on average.
// On RDNA3 and RDNA4 it is faster to pick the tile size against this value instead of ne12.
int64_t ncols_opt = ne12;
if (GGML_CUDA_CC_IS_RDNA3_0(cc) || GGML_CUDA_CC_IS_RDNA4(cc)) {
if (GGML_CUDA_CC_IS_RDNA3(cc) || GGML_CUDA_CC_IS_RDNA4(cc)) {
ncols_opt = (ne12*n_expert_used + ne02 - 1) / ne02;
}