* CUDA: runtime GGML_CUDA_MMVQ_MAX to tune the mvq->MMQ decode crossover
Add a runtime override of the mul_mat_vec_q -> MMQ batch crossover
(default MMVQ_MAX_BATCH_SIZE). Lowering it routes batches above the
threshold from the CUDA-core vector kernel to the int8 MMQ tensor-core
path, which is faster once quantized decode becomes compute-bound at
B>1 (measured +23-41% at B=8 on RTX 5090 for Q4_K dense, no low-batch loss).
The value is parsed once and clamped to [1, MMVQ_MAX_BATCH_SIZE], since
mul_mat_vec_q asserts ncols_dst <= that; invalid input warns and falls
back to the default. The override is applied consistently in both the
mul_mat_vec_q and MUL_MAT_ID dispatch paths. Default behavior unchanged.
* Added Blackwell specific switch point, to reduce dependence on runtime env var.
* Add per-HW switch point values for DGX Spark and removing runtime env var
* Adding switch points for Ada, tested on RTX 4090
* Modifying DGX Spark numbers based on latest run and adding some comments and small functional changes relating to MoE
* Reverting an unnecessary conditional
* Update ggml/src/ggml-cuda/mmvq.cu
---------
Co-authored-by: praneshgo <227579474+praneshgo@users.noreply.github.com>
Co-authored-by: Oliver Simons <osimons@nvidia.com>
* CUDA: dedup MoE gate/up activation quantization (fp4)
For MoE gate/up projections the src1 activation is broadcast across the
routed experts (ne11 == 1), so ids_src1 maps every one of a token's
n_expert_used slots to the same physical row. The MMQ path therefore
re-quantized each token's activation n_expert_used times.
For fp4 (NVFP4/MXFP4) src0, quantize each unique token row once instead of
once per expert. For NVFP4 a single quantize+scatter kernel
(quantize_scatter_mmq_nvfp4) quantizes each token once and writes the
resulting block_fp4_mmq straight to all n_expert_used slots, using an
inverse token->compact-row map (build_tok2c). MXFP4, and
GGML_CUDA_MOE_QUANT_GATHER=1, use a two-kernel variant: quantize unique
rows then gather into the expert-sorted layout (gather_mmq_fp4_blocks).
Both are bit-identical to the previous gather-then-quantize path (identical
source data, deterministic per-block quantization), verified by
test-backend-ops MUL_MAT_ID (type_a=nvfp4, broadcast b=1; 790/790 for the
default, gather, and per-expert paths) and by coherent end-to-end
generation. Set GGML_CUDA_NO_MOE_QUANT_DEDUP=1 to force the original
per-expert path.
Same-binary A/B on RTX 5090 (sm_120), Qwen3.6-35B-A3B-NVFP4 prefill @8192
(nsys, graphs-off; the unchanged mul_mat_q GEMM confirms stable clocks):
activation-quant GPU-busy drops 61% (78.2 -> 30.4 ms) with the fused
quantize+scatter, vs 33% (78.2 -> 52.8 ms) for the two-kernel gather. The
fused path avoids materializing and re-reading the 8x compact buffer,
writing the expert copies directly from registers.
* CUDA: bounds-check token ids in build_tok2c_kernel
Guard against malformed ids_src1: skip out-of-range token ids (t < 0 or
t >= n_tokens) and drop entries beyond n_expert_used per token instead of
writing past the token's tok2c region. No behavior change for valid MoE
routing data; test-backend-ops MUL_MAT_ID 790/790.
* Refactor the code based on review comments
- Removed previously added kernels that were not necessary anymore\
- Added an inverse mapping from (token, slot) to compact row. Each token is quantized once and scattered to its compact rows.
* Adding q8_1 support for dedup and addressing review comments
* Add pragma unrolls
* Remove redundant cudaMemsetAsync call
* Removing follow up redundancies
---------
Co-authored-by: praneshgo <227579474+praneshgo@users.noreply.github.com>
