* dflash : fuse the encoder into the KV injection decode
The encoder is a single fc + norm, but running it as a separate
llama_encode forced a device-to-host round trip of its output before the
injection decode could re-upload it, plus a second graph build per
round. Fold the encoder into the decoder's embd branch and feed the
target features directly to one llama_decode.
Assisted-by: Claude Fable
* nit
* Apply batched suggestions from code review
Co-authored-by: Ruixiang Wang <wangruixiang07@outlook.com>
* Fix missing references from renaming
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Ruixiang Wang <wangruixiang07@outlook.com>
* vulkan: RDNA3 static mat-vec rows above four columns
On RDNA3 above four columns a static 4 rows for all types benches faster than
the default.
* vulkan: RDNA3 static mat-vec-id rows
mul_mat_vec_id has no column dimension to switch on. On my Strix Halo machine,
a static 4 is faster here than the defaults across types and batch sizes.
* rpc: avoid serializing buffers from other servers
Only include remote buffer pointers when the buffer belongs to the RPC dispatcher receiving the graph. Add a two-server regression test for cross-server tensor serialization.
Assisted-by: Codex
* cont : add ref
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
for_each_token_in tested all LLAMA_MAX_SEQ sequences for every used cell,
while a cell almost always belongs to one. The scan now stops once the
cell's own sequences have been seen. Same visit order, same callback
arguments, so behaviour is unchanged.
get_prev_tokens is the only caller, so this affects the n-gram path.
RTX PRO 6000, Qwen3.8-Flash-Next UD-Q4_K_XL, fa on, warm runs:
55k context generation 56.3 -> 74.3 t/s
132k context generation 33.6 -> 50.9 t/s
Prompt processing is unchanged, the scan is amortised over the ubatch
there. The gain follows the number of used cells, so it grows with
context and is invisible on short prompts.
The optimized path grouped warp lanes by token and required
warp_size % n_expert_used == 0, with a single hardcoded exception
padding 6 up to 8. Every other count fell back to the generic path,
which walks the tokens one at a time with a warp reduction per token,
for each of the n_expert blocks.
The lane group only has to divide the warp, and the loop body already
guards the padded lanes with iex < n_expert_used, so the padding
generalizes to the next power of two. The 6 -> 8 case and every count
already dispatched keep the exact same padding as before.
n_expert_used = 10 now reaches the fast path. Measured on
Qwen3.8-Flash-Next (512 experts, 10 used) at 55k context on an
RTX PRO 6000, warm runs with the first one discarded:
prompt processing 2334 -> 2600 t/s
Token generation is unaffected, since a single token leaves nothing to
walk. Other expert counts reach the fast path by adding their case to
the dispatch.
- DFlash2 NVFP4 draft models produced almost no accepted speculative
tokens because the Q, K, V, and output projection scales were not
passed to the corresponding graph operations.
Rename the --tensor-read-lazy CLI argument to --lazy-mode, to match the
internal lazy_mode parameter, and add a -lzm shorthand. Sync the READMEs.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
some backends (Metal, SYCL, WebGPU) require additional memory for
fleeting data for certain ops, which is reflected in their
get_alloc_size implementations.
add ggml_backend_op_alloc_size_may_expand() to the backend utils,
listing these ops, and assert in ggml_backend_buft_get_alloc_size
that a backend expanding the alloc size of a compute op only does so
for ops listed in the helper.
use the helper in the RPC backend to decide whether to query the
remote server for the actual alloc size, instead of a hardcoded list.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* metal : fail closed on mul_mat shapes with missing F16 kernels
* metal : abort on nil pipeline in encoder_set_pipeline
* metal : address review comments
* metal : share mul_mat mm dispatch with supports_op
* opencl: default the Adreno xmem F16xF32 GEMM on for X2E
kernel_mul_mm_f16_f32_l4_lm is the slowest matmul this backend has on Adreno: on
the X2-90 it runs the gpt-oss-20b attention projections at roughly a quarter of
what the tuned dense q4_0 GEMM reaches on the same device. That matters for any
model whose non-expert weights stay f16 -- the stock gpt-oss-20b release is
exactly that, and its prefill spends 40.8% of GPU time in that one kernel. The
xmem route already existed but was left opt-in, so nobody hit it.
Worth about 25% prefill on gpt-oss-20b on an Adreno X2-90. Gated to X2E: the
Adreno 840 measures neutral. Decode is untouched -- the dispatch gate needs
N >= 16. It is worth nothing on the q8attn variant, whose attention weights
already take the dp4a dense GEMM.
The env var was presence-tested before, so =0 previously enabled it; it is now
atoi()'d. MUL_MAT 963 OK / 0 FAIL on both arms.
* opencl: bypass the tiled f32 GEMM on the Adreno A7X
The A7X (E031.41) compiler executes kernel_mul_mm_f32_f32_l4_lm at roughly a
tenth of what the same silicon reaches in its own f16 and q4_K kernels. It
allocates 488 B/WI of private memory against 304 for the same source on the
following generation, i.e. the older register allocator spills in the K-loop.
Models with per-layer F32 projection pairs kept F32 by quantization policy land
on this kernel twice per layer, and it dominates their prefill on that part.
Route batched f32xf32 (ne11 > 8) around the tiled path on the A7X and let it
fall through to the per-row f32 kernel, which that compiler handles fine; small
batches keep the tiled path. Weights stay GPU-resident, so decode placement is
untouched -- declining the op in supports_op instead was measured first and
rejected, because the per-layer CPU round-trips cost more decode than the
prefill it gained.
Worth about 9% prefill on gemma-3n-E4B on an Adreno 740, with MUL_MAT counts
identical on and off. No other generation is affected. Override with
GGML_OPENCL_A7X_F32_LM_BYPASS=0.
* opencl: enable xmem GEMM for adreno by default
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
improve the --fit algorithm to take into account the actual peak
required VRAM for a given context size on a SYCL backend.
This includes both properly accounting for how much VRAM is required
when the allocated context is fully used (which makes the reported
context drop below what it did before, but stop it OOMing) as well
as preventing some overly-conservative calculations which meant too much
VRAM was being reserved.
Tested on a Arc b70 with unsloth's qwen3.8 (Q4_K_XL), able to get 262144 context,
fully usable, with q8_0 KV and MTP and 4k ubatch size using --fit-target 1
The N padding is needed for mul_mat, but not mul_mat_id. For mul_mat_id,
we indirect the row index through a shared memory lookup table which avoids
any OOB row coordinate. But that callback doesn't bounds check K, so we
actually need K padding instead.
* port setup-build.ps1 to setup_sdk.py, to facilitate installation of Hexagon and OpenCL SDKs on Windows
* rename setup_sdk.py -> setup-sdk.py
* flake8 fix: print() -> logger.info()
---------
Co-authored-by: Kristopher Urquhart <kurquhar@qti.qualcom.com>
* vulkan: fix missing view-alias dependencies in ggml_vk_graph_optimize
is_src_of doesn't treat two views of one tensor as dependent, so the optimizer reorders nodes across aliased reads and writes.
Result: silently wrong tokens under greedy decoding, different output on every server start, and invalid speculative-decoding acceptance, with nothing logged.
Hits Qwen3.8's recurrent state (and any model with view-aliased state) on AMD and NVIDIA Vulkan. CUDA is clean.
Compare view_src bases on both sides.
Fixes#27805
* vulkan: don't treat view/no-op nodes as aliasing dependencies
Nodes whose op is NONE, RESHAPE, TRANSPOSE, VIEW or PERMUTE execute nothing, so aliasing through them is not a real dependency. The previous base comparison matched them anyway, which only costs the optimizer reordering freedom.
Co-authored-by: Jeff Bolz <jbolz@nvidia.com>
* vulkan: make the lambda parameter const and capture is_empty in is_src_of
Code will not compile without these changes.
is_src_of has an empty capture list, so is_empty was not visible inside it, and is_empty took a non-const pointer, while is_src_of receives const ones. Other call sites pass non-const pointers, which still convert as usual.
---------
Co-authored-by: Jeff Bolz <jbolz@nvidia.com>
* ggml : fix conv_transpose_2d for multiple batches
ggml_compute_forward_conv_transpose_2d_impl only computed the first
batch (ne[3] of the destination); every batch after the first was left
as zero. Both the src1 permutation and the main compute loop now iterate
over the batch dimension, and the work buffer size in ggml_graph_plan is
scaled by the src1 batch count so the extra permuted batches fit. A
multi-batch test case is added to test-backend-ops.
Fixesggml-org/ggml#1448
* metal : fix conv_transpose_2d for multiple batches
The kernel only computed batch 0 of the input (src1->ne[3]); every
output batch after the first was left as zero, so multi-batch
conv_transpose_2d results diverged from the CPU reference.
The grid now covers all batches (OW x OH x OC x N), the kernel decodes
the batch from the grid z coordinate and offsets both the input and
destination indices accordingly. nb3 is passed in the kernel args.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* vulkan: add hoisting support for row IDs and expert count in shaders
* use hoisted row ids in coopmat2
* vulkan: address review feedback on count_experts
- use vk_op_count_experts_push_constants instead of a raw uint vector
- apply the fastdiv trick to the ne00 div/mod in count_experts
- compute the per-expert offsets with subgroupExclusiveAdd when the
device supports it, keeping the serial path as fallback
- document the data_d layout and the hoisted_row_id_words bound
- drop a leftover debug print in ggml_vk_matmul_id
* vulkan: use init_pushconst_fastdiv for count_experts push constants
* vulkan: refine comments for row ID hoisting and data layout in count_experts shader
* Whitespace
---------
Co-authored-by: Jeff Bolz <jbolz@nvidia.com>
This adds fa_vec_tuned_table records for Apple M4 to ggml-metal-tuning.cpp.
Includes F16, Q4_0, Q4_1, Q5_0, Q5_1, and Q8_0. (M4, 10 GPU Cores)
Co-authored-by: Strongtut <8432058+Strongtut@users.noreply.github.com>
* OpenVINO Backend: Fuse IM2COL + MatMul convolution into OpenVINO convolution
* ci:ggml-ov: Skip recurrent state rollback tests
* ci:ggml-ov: Skip recurrent state rollback tests
* Update OPENVINO.md
* ggml-openvino : add env-var gated op support debugging
* Fix ggml_rope_set_offset case
* OpenVINO backend: Support Whisper.cpp
* Fix code style
* openvino : enable qwen35 on NPU
Static shapes:
- get_graph_input_shape() left the s_copy / s_copy-leaf inputs dynamic
([1,1,1,-1]) even in static mode, which propagated a dynamic slot dim through
GET_ROWS into the conv/GDN state, the state reshapes and the GDN output.
- With -np 1 the s_copy defrag remainder gathers zero rows; short-circuit that
CPY to the untouched cache instead of emitting a degenerate Slice/Concat, and
skip binding its zero-byte ggml tensor as an output (the dynamic path already
did the latter, the static path wrote the full cache over a 0-byte buffer).
Token-count independence:
- In static mode the compiled model's token count is the prefill chunk size or
1, not the captured cgraph's. Offsets derived from the captured count were
therefore wrong. Anchor the GDN state slice at the end of the packed
[attn | state] output and drop the rs_src_begin runtime inputs, and make
VIEWs over the GDN output / conv_input pass through so the consumer does the
slicing.
- CONT could not identify its token axis when the graph was captured with a
single token (every trailing dim has the same stride and size 1) and baked
the captured shape into the prefill model.
Chunked prefill:
- The last chunk is padded with fabricated tokens. Attention masks them, but
the recurrent path folded them into cache_r/cache_s permanently. Add a
chunk_valid_len runtime input, use it to zero g and beta for padded steps
(making the recurrence an exact identity) and to end the conv snapshot window
at the last valid token, and disable the recurrent-cache reset after the
first chunk so earlier chunks are not wiped.
- get_is_prefill() and the chunk loop bound read inp_pos->ne[0] directly, but
IMROPE stacks 4 position planes, so every decode step was run through the
padded prefill model and the loop ran extra out-of-bounds chunks.
cache_rs_reset_idx/len now stay runtime Parameters in static mode, since
can_reuse_statically() does not invalidate the cached model on ComputeParams
changes. Add GGML_OPENVINO_FORCE_STATIC to exercise the static path on CPU.
* Update to OpenVINO 2026.3.1
* ggml-openvino: forward NPU compilation mode parameters
Add GGML_OPENVINO_NPU_COMPILE_CONFIG to the backend's cached environment so callers can configure the NPU compiler without using the generic property escape hatch.
When the value is non-empty, pass it to OpenVINO as NPU_COMPILATION_MODE_PARAMS. This enables settings such as optimization-level=3 for NPU compilation while preserving the existing behavior when the variable is unset and leaving CPU and GPU configuration unchanged.
Document the variable, its NPU-only scope, and the optimization-level=3 example in the OpenVINO backend runtime configuration table.
* ggml-openvino : support RELU, POOL_2D, QUICK_GEGLU, and ROLL ops
* reorder op table
* exclude GPU/NPU failing POOL_2D case
* move op type detection to compute_op_case
* Relax rope supported cases
* Fix pool case
* Update openvino doc, gpu driver in ov docker
* openvino: remove unused static remote context branch
* openvino: parallelize static model build
* Apply editorconfig
---------
Co-authored-by: Mostafa Faheem <mostafaaafaheem@gmail.com>
Co-authored-by: Ravi Panchumarthy <ravi.panchumarthy@intel.com>
Co-authored-by: zhaixuejun1993 <xuejun.zhai@intel.com>
