* divide workload to 2D
This is to workaround FILL exceeding maxComputeWorkGroupCount for Intel GPUs on Qwen 3.8 flash next
* minor change
* Fixed comment
* ui : fix MCP image attachments not displayed in tool block (#25789)
Fixes regression from #25450 where ChatMessageAgenticContent passed
message.extra instead of section.toolResultExtras to tool blocks,
leaving tool images invisible. Also fixes TOOL_RESULT_JSON_OPEN_REGEX
which misclassified "[Attachment saved: ...]" as JSON.
Fixes#25789
Assisted-by: Muse Spark
* Addressed PR comments: 1.- Removed ·?? mesage?extra· as it has no case left to cover 2.- Added ·[\· to cover the case of ·[[1, 2], [3, 4]]· case suggested in the PR comment 3.- Added unit test for covering up this regex case
* ui : fix MCP image attachments not displayed in tool block (ggml-org#25789) - Addressed lint error on regex (redundant \)
Skip the nb[3] check when ne[3] == 1, the shader never reads it for a
single stream. Cache views carry the full-buffer stride there, so the old
check reduced to n_kv == kv_size and the path only engaged with the
cache full.
- DFlash2 NVFP4 draft models produced almost no accepted speculative
tokens because the Q, K, V, and output projection scales were not
passed to the corresponding graph operations.
* common : dedupe --n-cpu-moe / --spec-draft-n-cpu-moe override loops
* common : add --n-cpu-ffn to CPU-offload dense FFN weights of first N layers
* common : generalize llm_ffn_block_regex over the FFN regex, drop TODO
* vulkan : dequant q8_0 KV once in coopmat1
Assisted-by: Claude (Opus 4.8)
* vulkan : fall back instead of aborting when FA scratch exceeds maxStorageBufferRange
* vulkan : require KV-cache layout in FA dequant path
Assisted-by: Claude (Opus 4.8)
* vulkan : skip FA dequant path on coopmat2
Assisted-by: Claude (Opus 4.8)
* tests : add contiguously-allocated quant K/V FA tests
Assisted-by: Claude (Opus 4.8)
* vulkan : trim comments
* vulkan : tighten permutation checks for FA path
* vulkan : set prealloc_x_need_sync after the FA dispatch
* vulkan : exclude Intel Xe1 from FA dequant path
* CUDA: MMVQ nwarps=8 for bs=1 for dense models on DGX Spark
Signed-off-by: ynankani <ynankani@nvidia.com>
* skip moe experts and allow others based on k geometry (allow only small idle tail)
Signed-off-by: ynankani <ynankani@nvidia.com>
* rename MMVQ DGX Spark params to GB10 and fix MSVC constexpr lambda capture
Signed-off-by: ynankani <ynankani@nvidia.com>
---------
Signed-off-by: ynankani <ynankani@nvidia.com>
The log output does not append a newline, so the warning ran into the
next line printed on stdout, corrupting the benchmark table header.
Signed-off-by: Fathi Boudra <fathi.boudra@linaro.org>
Replace the deprecated --mmap, --no-mmap, --mlock, and --direct-io flags with
the unified --load-mode argument across scripts, examples, and documentation.
Internal warning message and env var docs updated accordingly.
Signed-off-by: Fathi Boudra <fathi.boudra@linaro.org>
* common: Add CLI > ENV > models-presets > INI precedence
1. CLI flags have the highest precedence
2. ENV vars have the second-highest precedence
3. System and User configs have the lowest precedence
- Linux/BSD/Mac
- /etc/llama.cpp/config.ini < ${XDG_CONFIG_HOME:-~/.config}/llama.cpp/config.ini
- Windows
- %PROGRAMDATA%\llama.cpp\config.ini < %APPDATA%\llama.cpp\config.ini
* fix UB
* use common_get_env
* ignore_unknown_keys
* nits
* add docs
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
- In MSL, declaring an array of matrix types like `threadgroup half4x4` causes
a 'no matching constructor' compilation error because MSL matrix types do not
have zero-argument default constructors and threadgroup variables cannot have
initializers.
- Fix this by declaring a POD `threadgroup half` array instead and casting
to `threadgroup half4x4 *` for matrix indexing.
Signed-off-by: JamePeng <jame_peng@sina.com>
* feat(silu_back): implemented silu_back op for f32
* fix(silu_back): removed redundant asserts in ggml-metal-ops.cpp function ggml_metal_op_silu_back.
* vulkan : add pool1d push constants and pipeline field
Declared data structures needed for POOL1D OP, which are the vk_op_pool1d_push_constants struct and pipeline_pool1d_f32 field.
* vulkan : add pool1d compute shader
Added pool1d.comp for Vulkan backend mirroring the existing pool2d shader.
* vulkan : add full GGML_OP_POOL_1D support
Added pipeline creation and op dispatch for 1D pooling in the Vulkan backend.
* vulkan : fix pool1d shader logic
Registered pool1d_f32 in vulkan-shaders-gen.cpp and fixed tensor dimension indices and avg pool scale.
* vulkan : fix pool1d end boundary crash and expand test coverage
Fixed an issue where the shader crashed when the end boundary was negative when k0 < p0. Also, added more test cases related to this fix.
* Removed crash guard for Intel
Crash fixed from driver 32.0.101.8860
* Added driver version check for windows
* Change to convert from driverVersion rather than string
* No need to use signed
* Refactor
* allow GPU other than Xe2+
* adjusted function body position
* support cuda virtual devices
* disable NCCL path when virtual devices are used
* label virtual devices in description; add GPUx2 server CI jobs
* code refactor
* ggml: uniformize im2col dst_type for all conv ops
* Update ggml/src/ggml.c
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* ggml : uniformize im2col casting logic across all conv ops
* fix : allow im2col_f16 to accept any kernel type
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* server: honour per-request reasoning_budget_tokens in chat completions
The reasoning-budget block in oaicompat_chat_params_parse read only the
server-level default (opt.reasoning_budget, typically -1) and the
Anthropic-style alias thinking_budget_tokens, but never the canonical
reasoning_budget_tokens field from the request body. Because the key
was then written into llama_params before the generic body-copy loop
ran, the copy loop found the key already present and silently skipped
the caller-supplied value. Any per-request override (e.g. 0 to
suppress thinking entirely) was therefore discarded.
Fix: read reasoning_budget_tokens from the request body first, so the
value that reaches the sampling layer is the one the caller intended.
Add a unit test in test-chat.cpp that exercises this path via
oaicompat_chat_params_parse with a Qwen3 template (which the autoparser
detects as a thinking-capable model) and asserts the returned
llama_params carries reasoning_budget_tokens == 0.
* server: honour per-request reasoning_budget_message in chat completions
The reasoning-budget block in oaicompat_chat_params_parse wrote
reasoning_budget_message into llama_params straight from the server-level
default (opt.reasoning_budget_message) and never read the canonical
reasoning_budget_message field from the request body. Because the key
was written before the generic body-copy loop ran, that loop found the
key already present and silently skipped the caller-supplied value. Any
per-request override of the message injected before the end tag when the
budget is exhausted was therefore discarded, even though server-task.cpp
already reads reasoning_budget_message from that data.
This mirrors the reasoning_budget_tokens bug fixed in the previous commit.
Fix: read reasoning_budget_message from the request body first, falling
back to the server default, so the value that reaches the sampling layer
is the one the caller intended.
While here, collapse the adjacent reasoning_budget_tokens override to a
single json_value() call; json_value already falls back to the default on
a missing/null/wrong-type key, so the explicit body.contains() guard was
redundant. No behavioral change.
Add a unit test in test-chat.cpp that exercises this path via
oaicompat_chat_params_parse with a Qwen3 template (which the autoparser
detects as a thinking-capable model) and asserts the returned
llama_params carries the per-request reasoning_budget_message rather than
the server default.
* cleanup
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* hexagon: add VISION RoPE support
* hexagon: support RoPE on strided half-dim views for all modes
* hex-rope: decouple src0 DMA copy size from row stride
* hex-rope: support non-contiguous dst for RoPE
* hex-rope: fix dst spad pitch for non-contiguous dst
* Update ggml-cuda.cu - Turing P2P access fix.
* Add original code as fallback behaviour when NCCL or P2P is not set/true.
* Update ggml/src/ggml-cuda/ggml-cuda.cu to add comment as per suggestion
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* server: add "X-Accel-Buffering": "no" header to streaming endpoints
This header tells Nginx (as a reverse proxy) to NOT buffer responses. (only affects streaming endpoints)
Without it, Nginx will break streaming with certain applications (notably the Pi coding harness).
* llama-graph : apply embedding scale when deepstack is not used
* nits: remove non-existant hunyuan-vl from the tests
* apply suggestion from @gabe-l-hart
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* ggml-hexagon: add PAD op HVX kernel
Implements GGML_OP_PAD on the Hexagon HTP backend using HVX vectorized
kernels. Supports zero-padding and circular padding across all 4 tensor
dimensions.
* hex-ggml: remove duplicate op cases (merge conflict)
* hex-pad: fix editorconfig checks and macro alignment
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* L2_NORM Updates
* Addressed PR Comments
* ggml-hexagon: add L2_NORM HVX kernel for Hexagon backend
* hex-unary: remove supported_unary_nc since the outer loop is the same for all unary ops
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* CUDA: batch out_prod inner loop with cublasSgemmStridedBatched
* CUDA: batch out_prod inner loop with cublasSgemmStridedBatched
* CUDA: add cublasSgemmStridedBatched mapping for HIP and MUSA backends
* Changed to leak logger singleton to prevent hanging on Windows
* Fix comment
* Stopped using static vector
Using std::vector will cause g_col to be released before the logger thread exits, causing the logger thread to touch freed memory causing a crash
* Change so all logs are output before exit
* Added debug logging
* added more logging
* Added logging
* Explicitly free logger to avoid hanging on Win
* Reverted to leak logger instance again
* Removed debug log and fixed comment
* Fixed comment
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* mtmd, llama : add HunyuanVL vision-language model support
- add LLM_ARCH_HUNYUAN_VL with M-RoPE (XD-RoPE) support
- add PROJECTOR_TYPE_HUNYUANVL with PatchMerger vision encoder
- add HunyuanVL-specific M-RoPE position encoding for image tokens
- add GGUF conversion for HunyuanVL vision and text models
- add smoke test in tools/mtmd/tests.sh
* fix: fix HunyuanVL XD-RoPE h/w section order
* fix: Remove redundant code
* convert : fix HunyuanOCR / HunyuanVL conversion
- Tested locally: both HunyuanOCR and HunyuanVL-4B convert to GGUF
- successfully and produce correct inference output on Metal (F16 / Q8_0).
* clip : fix -Werror=misleading-indentation in bilinear resize
* fix CI: convert_hf_to_gguf type check error
- convert_hf_to_gguf.py: give HunyuanVLTextModel.__init__ an explicit `dir_model: Path` parameter so ty can infer the type for load_hparams instead of reporting `Unknown | None`.
---------
Co-authored-by: wendadawen <wendadawen@tencent.com>
* optimize hmx_mat_mul functions by calculating row and column tiles upfront
* refactor core_dot_chunk_fp16 to use size_t for tile counts and improve readability
* wip
* set scale outside of loop
* wip
* refactor core_mma_chunk_fp16 and mat_mul_qk_0_d16a32 to use size_t for tile counts
* wip
* wip
* refactor transfer_output_chunk_fp16_to_fp32 to use size_t for dimensions
* refactor core_dot_chunk_fp16 to use size_t for tile row stride calculation
* wip
* refactor hmx_mat_mul functions to use hvx_vec_splat_f16 for column scales initialization
* refactor hmx_mat_mul_permuted_w16a32_batched to streamline scale setting and locking
* refactor core_dot_chunk_fp16 to improve tile stride calculations for output
* refactor hmx_mat_mul functions to use Q6_V_vsplat_R for column scales initialization
* fix compiling error
* wip
* optimize row and column tile indexing in core_mma_chunk_fp16 function
* wip
* Revert "wip"
This reverts commit cde679eff7.
* Add size limit check for HAP_mmap in htp_iface_mmap and drop_mmap functions
* wip
* fix NemotronH vocab loading by using trust_remote_code for unsupported config patterns
* fix NemotronH tokenizer loading by overriding set_vocab with trust_remote_code
* experimenting CI
* Experimenting CI fix for MinGW
* experimenting CI on Windows
* modified script for integration with VisualStudio
* added proxy handling
* adding python version for Windows execution
* fix iterator::end() dereference
* fixed proxy handling
* Fix errors occurring on Windows
* fixed ci script
* Reverted to master
* Stripping test items to simplify Windows test
* adjusting script for windows testing
* Changed shell
* Fixed shell
* Fixed shell
* Fix CI setting
* Fix CI setting
* Fix CI setting
* Experimenting ci fix
* Experimenting ci fix
* Experimenting ci fix
* Experimenting ci fix
* experimenting fix for unit test error
* Changed to use BUILD_LOW_PERF to skip python tests
* Fix CI
* Added option to specify Ninja generator
* Reverted proxy related changes
* flash attention support for head dimension 512 added
* FA D=512 - match 576 configs, limit ncols2, revert vec cap
* fix HIP tile kernel build for D=512
* fix HIP tile kernel occupancy for D=512 on AMD
* Apply suggestions from code review
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* fix tile FA compilation
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* Add missing features to WoS scripts to achieve parity with ADB scripts
* Fix line-ending in run-mtmd.ps1
Signed-off-by: Max Krasnyansky <maxk@qti.qualcomm.com>
---------
Signed-off-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* Remove make dependency
* Added option to specify Ninja generator
* use ninja-build as default for several CI
* Revert "use ninja-build as default for several CI"
This reverts commit f552c4559b.
* changed use plain string rather than arrays
* Enabled ninja build by default for experimentation
* ci: add run.sh to test conditions to trigger GitHub CI and self-hosted runners
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
* Enabled ninja build by default on self-hosted envs for experimentation
* ci: revert generator to ninja instead of ninja multi-config
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
* ci: install ninja-build for self-hosted workflows
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
* ci: revert ninja from self-hosted runners
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
* ci: missed one self-hosted step
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
* ci: fix windows ci errors from an errenous revert
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
* Added explicit build types for Ninja
Also reverted some needless change
* ci: use ninja multi-config for vulkan-x64 build
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
* added time command to measure build time
* Keeping some configs to use Ninja which show improvement
* minor fix based on review
Co-authored-by: Aaron Teo <taronaeo@gmail.com>
* ci: rm `time` from custom containers
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
---------
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
Co-authored-by: Aaron Teo <aaron.teo1@ibm.com>
Co-authored-by: Aaron Teo <taronaeo@gmail.com>
* convert : support is_causal hyperparameter
Check for the `is_causal` attribute in the Hugging Face model configuration and include it in the GGUF metadata.
* Update convert_hf_to_gguf.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
* style: fix F541 f-string is missing placeholders
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
* Fix errors occurring on Windows
* Reverted fix
#20365 will take care of CRLF isue
* Changed to write to directly to stdin
* Prevent fclose to happen twice
Add element-wise unary ops needed by Qwen 3.5's DeltaNet linear
attention layers. These ops follow the existing unary-ops pattern
with VTCM DMA double-buffering.
- neg: negate via scale by -1.0
- exp: uses existing hvx_exp_f32 HVX intrinsics
- sigmoid: uses existing hvx_sigmoid_f32_aa HVX intrinsics
- softplus: log(1 + exp(x)) scalar fallback
- CONT reuses the existing CPY infrastructure since making a tensor
contiguous is equivalent to a same-type copy.
- REPEAT implements tiled memory copy with multi-threaded execution via
the worker pool, supporting f32 and f16 types. The kernel parallelizes
across output rows and uses memcpy for each tile.
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Bernard Ladenthin