8786 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Berk IdemandGitHub	56666fa607	common: skip reasoning budget sampler when no budget is requested (#21870) ... * common: skip reasoning budget sampler when no budget is requested After I added thinking_start_tag / thinking_end_tag for gemma4 in #21697, the reasoning budget sampler gets unconditionally created even when no budget is configured (the default -1). The same applies to kimi_k2, lfm2, lfm2_5, and ministral_3 which also set these tags. The budget gets converted to INT_MAX, so the sampler never actually forces any tokens but still runs per-token checks (start tag matching in IDLE state, token-to-piece conversion + UTF-8 checks in COUNTING state). More importantly, the mere existence of the sampler (non-null rbudget) disables backend sampling. Backend sampling lets the GPU select tokens directly, avoiding a full logits transfer from GPU to CPU every token. This could explain the 30% speed regression reported in #21784 (98 t/s to 70 t/s on Vulkan). So I added a reasoning_budget_tokens >= 0 check to the sampler creation condition. When the budget is unlimited, the sampler is not created, backend sampling stays enabled, and no per-token overhead is added. When a budget is explicitly set (0, 128, 1024, etc.), the sampler is created and works as before. * common: preserve rbudget when grammar is lazy Following up on the review feedback on #21870: keep the reasoning budget sampler when grammar_lazy is true, so the thinking-block grammar suppression from #20970 still works when tools are in use. This way, we only skip the sampler when both no budget is set AND grammar is not lazy. b8786	2026-04-14 12:43:06 +02:00
Jeff BolzandGitHub	6a6780a232	vulkan: Support GGML_TYPE_NVFP4 (#21455) ... This adds nvfp4 support for get_rows, dequant, and mul_mat(_id). For mul_mat, it does not add support for the dp4/q8_1 path, it's all via fp16/fp32. b8785	2026-04-14 11:34:23 +02:00
Xuan-Son NguyenandGitHub	e489a5ca0e	server: support OAI /v1/audio/transcriptions API (#21863) ... * server: support OAI /v1/audio/transcriptions API * address autoreview comments * correct default response_format value b8784	2026-04-14 11:09:52 +02:00
Aldehir RojasandGitHub	e21cdc11a0	common/gemma4 : handle parsing edge cases (#21760) b8783	2026-04-13 18:18:18 -05:00
Xuan-Son NguyenandGitHub	e974923698	docs: listing qwen3-asr and qwen3-omni as supported (#21857) ... * docs: listing qwen3-asr and qwen3-omni as supported * nits	2026-04-13 22:28:17 +02:00
Piotr Wilkin (ilintar)andGitHub	1c0d9081fd	chat: dedicated DeepSeek v3.2 parser + "official" template (#21785) b8781	2026-04-13 22:23:53 +02:00
Christian KastnerandGitHub	a8bad3842e	ci: Also exempt 'security' tag from auto-close (#21844)	2026-04-14 01:18:44 +08:00
Ruben OrtlamandGitHub	75f3bc94e6	vulkan: Flash Attention DP4A shader for quantized KV cache (#20797) ... * use integer dot product for quantized KV flash attention * small improvements * fix SHMEM_STAGING indexing * add missing KV type quants * fixes * add supported quants to FA tests * readd fast paths for <8bit quants * fix mmq gate and shmem checks b8779	2026-04-13 14:21:31 +02:00
Adrien GallouëtandGitHub	aa00911d12	common : add download cancellation and temp file cleanup (#21813) ... Signed-off-by: Adrien Gallouët <angt@huggingface.co> b8778	2026-04-13 11:18:23 +02:00
Gaspard PetitandGitHub	ce8fd4b1a6	server: Expose build_info in router mode (#21835) b8777	2026-04-13 11:14:42 +02:00
Oliver SimonsandGitHub	9f5e1edb10	CUDA: Limit DeviceSegmentedSort to immediate mode (#21718) ... * CUDA: Limit DeviceSegmentedSort to immediate mode DeviceSegmentedSort is currently not capturable in a cuda graph. Hence, we have to go for the slower DeviceSegmentedRadixSort in that case. Perf numbers on RTX Pro 6000 Blackwell Max-Q: DeviceSegmentedRadixSort in graph mode (i.e. CUDA Graphs) ARGSORT(type=f32,ne=[2048,512,1,1],order=1): 12291 runs - 105.94 us/run - 8192 kB/run - 73.75 GB/s ARGSORT(type=f32,ne=[4096,512,1,1],order=1): 10245 runs - 115.08 us/run - 16384 kB/run - 135.77 GB/s ARGSORT(type=f32,ne=[8192,512,1,1],order=1): 5125 runs - 221.22 us/run - 32768 kB/run - 141.26 GB/s ARGSORT(type=f32,ne=[16384,512,1,1],order=1): 2565 runs - 430.98 us/run - 65536 kB/run - 145.02 GB/s ARGSORT(type=f32,ne=[32768,512,1,1],order=1): 1028 runs - 1185.83 us/run - 131072 kB/run - 105.41 GB/s ARGSORT(type=f32,ne=[65536,512,1,1],order=1): 387 runs - 2748.62 us/run - 262144 kB/run - 90.95 GB/s DeviceSegmentedSort in immediate mode ARGSORT(type=f32,ne=[2048,512,1,1],order=1): 16388 runs - 71.17 us/run - 8192 kB/run - 109.78 GB/s ARGSORT(type=f32,ne=[4096,512,1,1],order=1): 12294 runs - 81.38 us/run - 16384 kB/run - 192.00 GB/s ARGSORT(type=f32,ne=[8192,512,1,1],order=1): 5125 runs - 240.81 us/run - 32768 kB/run - 129.77 GB/s ARGSORT(type=f32,ne=[16384,512,1,1],order=1): 2565 runs - 406.60 us/run - 65536 kB/run - 153.71 GB/s ARGSORT(type=f32,ne=[32768,512,1,1],order=1): 1285 runs - 873.23 us/run - 131072 kB/run - 143.15 GB/s ARGSORT(type=f32,ne=[65536,512,1,1],order=1): 516 runs - 2288.46 us/run - 262144 kB/run - 109.24 GB/s * Add test case for dispatch to DeviceSegmentedRadixSort We currently lack a way to force graph mode in CUDA, patch callback to invoke ggml_backend_compare_graph_backend twice to enforce each test to run in graph mode b8776	2026-04-13 11:14:06 +02:00
Xuan-Son NguyenandGitHub	920b3e78cb	mtmd: use causal attn for gemma 4 audio (#21824) b8775	2026-04-13 09:47:55 +02:00
3 people	974c8c94cc	webui: add setting for first-line chat titles (#21797) ... * webui: add setting for first-line chat titles Add an opt-in setting (`titleGenerationUseFirstLine`) to use the first non-empty line of a prompt as the generated conversation title. Previously, the complete multi-line prompt was being used, which created long titles for complex queries. Coupled with "Ask for confirmation before changing conversation title", the dialog would overflow. * Update tools/server/webui/src/lib/utils/text.ts Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com> * Update tools/server/webui/src/lib/utils/text.ts Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com> * webui: Run build to update the bundle As requested in: https://github.com/ggml-org/llama.cpp/pull/21797#pullrequestreview-4094935065 * webui: Fix missing import for NEWLINE_SEPARATOR --------- Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>	2026-04-13 09:30:46 +02:00
3 people	227ed28e12	webui: MCP Diagnostics improvements (#21803) ... * Add MCP Connection diagnostics and CORS hint to web-ui * tidy up test * webui: Refactor and improve MCP diagnostic logging --------- Co-authored-by: evalstate <1936278+evalstate@users.noreply.github.com>	2026-04-13 07:58:38 +02:00
Masashi YoshimuraandGitHub	bafae27654	Remove extra conditional check on debug mode. (#21798) b8772	2026-04-12 20:13:04 -07:00
Akarshan BiswasandGitHub	873c825611	sycl: disable Q1_0 in backend and cleanup unused variables (#21807) b8771	2026-04-13 09:44:58 +08:00
SergiuandGitHub	82764d8f40	mtmd: fix crash when sending image under 2x2 pixels (#21711) b8770	2026-04-12 23:59:21 +02:00
3 people	21a4933042	mtmd: qwen3 audio support (qwen3-omni and qwen3-asr) (#19441) ... * add qwen3a * wip * vision ok * no more deepstack for audio * convert ASR model ok * qwen3 asr working * Apply suggestions from code review Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * nits * Apply suggestions from code review Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * fix bad merge * fix multi inheritance --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> b8769	2026-04-12 23:57:25 +02:00
Sigbjørn SkjæretandGitHub	1e9d771e2c	convert : force f16 or f32 on step3-vl conv weights (#21646)	2026-04-12 19:22:29 +02:00
Xuan-Son NguyenandGitHub	aa4695c5e5	mtmd: add gemma 4 test (vision + audio) [no ci] (#21806) ... * mtmd: add gemma 4 test (vision + audio) * add to docs	2026-04-12 16:29:03 +02:00
Stephen CoxandGitHub	547765a93e	mtmd: add Gemma 4 audio conformer encoder support (#21421) ... * mtmd: add Gemma 4 audio conformer encoder support Add audio processing for Gemma 4 E2B/E4B via a USM-style Conformer. Architecture: - 12-layer Conformer: FFN → Self-Attention → Causal Conv1D → FFN → Norm - Subsampling Conv Projection: 2x Conv2D(stride=2) with LayerNorm - Full self-attention with sinusoidal RPE and sliding window mask (24) - Logit softcapping at 50.0, ClippableLinear clamping - Output: 1024 → 1536 → RMSNorm → multimodal embedder Mel preprocessing (dedicated mtmd_audio_preprocessor_gemma4a): - HTK mel scale, 128 bins, magnitude STFT, mel_floor=1e-3 - Standard periodic Hann window (320 samples), zero-padded to FFT size - Semicausal left-padding (frame_length/2 samples) - Frame count matched to PyTorch (unfold formula) - No pre-emphasis, no Whisper-style normalization - Mel cosine similarity vs PyTorch: 0.9998 Key fixes: - Tensor loading dedup: prevent get_tensor() from creating duplicate entries in ctx_data. Fixed with std::set guard. - ClippableLinear clamp_info loading moved after per-layer tensors. - Sliding window mask (24 positions) matching PyTorch context_size. - Skip Whisper normalization for Gemma4 mel output. Tested on E2B and E4B with CPU and Vulkan backends. Transcribes: "Glad to see things are going well and business is starting to pick up" (matching ground truth). Ref: #21325 b8766	2026-04-12 14:15:26 +02:00
Aleksander GrygierandGitHub	9e209c5aee	fix: Proper messages rendering for "Show raw output" (#21672)	2026-04-12 13:08:11 +02:00
Xuan-Son NguyenandGitHub	6313acbef0	docs: add guide on how to add multimodal support (#21778) ... * docs: add guide on how to add multimodal support * nits	2026-04-12 13:02:38 +02:00
Johannes GäßlerandGitHub	ff5ef82786	CUDA: skip compilation of superfluous FA kernels (#21768) b8763	2026-04-11 18:52:11 +02:00
Sirui HeandGitHub	073bb2c20b	mtmd : add MERaLiON-2 multimodal audio support (#21756) ... * mtmd : add MERaLiON-2 multimodal audio support Adds support for A*STAR's MERaLiON-2 audio-language model (3B and 10B) to the multimodal framework. Architecture: - Whisper large-v2 encoder for audio feature extraction - Gated MLP adaptor: ln_speech -> frame stack (x15) -> Linear+SiLU -> GLU -> out_proj - Gemma2 3B / 27B decoder The mmproj GGUF is generated via convert_hf_to_gguf.py --mmproj on the full MERaLiON-2 model directory (architecture: MERaLiON2ForConditionalGeneration). The decoder is converted separately as a standard Gemma2 model after stripping the text_decoder. weight prefix. New projector type: PROJECTOR_TYPE_MERALION Supports tasks: speech transcription (EN/ZH/MS/TA), translation, spoken QA. Model: https://huggingface.co/MERaLiON/MERaLiON-2-3B https://huggingface.co/MERaLiON/MERaLiON-2-10B * simplify comments in meralion adaptor * meralion: use format_tensor_name, ascii arrows in comments b8762	2026-04-11 14:15:48 +02:00
shaofeiqiandGitHub	af1127d3c4	opencl: add basic support for q5_k (#21593) ... * opencl: add general q5_k mv * opencl: add flattened Q5_K mv and general Q5_K mm * opencl: fix Q5_K unit tests b8761	2026-04-11 01:46:19 -07:00
Johannes GäßlerandGitHub	865ff06b2f	TP: fix Qwen 3 Next data split (#21732) b8760	2026-04-11 09:23:42 +02:00
Sigbjørn SkjæretandGitHub	2b2cd57de6	ggml : fix a few instances of missing GGML_TYPE_Q1_0 cases (#21716) b8759	2026-04-11 09:45:00 +03:00
BartowskiandGitHub	660386f6f8	py : Bump typer to latest to fix huggingface_hub issue (#21701)	2026-04-11 09:44:15 +03:00
Aman GuptaandGitHub	a29e4c0b7b	CUDA: also store node->src ne/nb for graph equality (#21736) b8757	2026-04-11 10:30:30 +08:00
GalunidandGitHub	b136b62cf9	fix: Fix broken structured output when using $refs in json_schema (#21699) b8756	2026-04-10 18:26:36 -05:00
3 people	81069a808a	hexagon: add support for linux on snapdragon (#21707) ... * hexagon: add support for debian on ex2 * hexagon: add -fvectotize to c/c++ cmake flags * hexagon: remove trailing white space * update onboarding steps * hexagon: update linux setup documentation * hexagon: update intallation scripts * Hexagon: update docs * hexagon: update onboarding scripts --------- Co-authored-by: Zack Li <zackli@qti.qualcomm.com> b8755	2026-04-10 15:57:23 -07:00
4 people	9aa2807769	hexagon: improved Op queuing, buffer and cache management (#21705) ... * hexagon: introduce op request batching and rewrite buffer managment The host now prepares batches of requests and dispatches them via a single dspqueue message. Buffers are mapped explicitly by NPU while processing batches. * hex-dma: disable l2 bypass since to work around new issue due to no flushes between Ops * hex-utils: add explicit l2flush and l2clear helpers * hex-opreq: use fine-grain per tensor l2 management * hex-opreq: avoid redundant invalidates for tensors we already flushed * hex-opreq: update debug messages * htp-opreq: reuse ops_context * hex-opreq: do not flush or invalidate cache lines beyond buffer boundry * hex-opreq: fix errors in log message * Revert "hex-opreq: do not flush or invalidate cache lines beyond buffer boundry" This reverts commit 8b7f0a55a750a6430ce4eb1874c7feb3d720056d. * hexagon: limit l2 flushes to 1MB which covers l2 cache * hex-opreq: limit cache flush to 4MB Looks like 4MB cont. vitual space should cover the 1MB cache. * hexagon: drop cache flush size to 2MB * hex-opreq: start reworking opreq packing * hex-opreq: introduce new way of packing opbatch where tensors are stored separately * hex-opreq: add a simple fastrpc call to force unmap all buffers * hex-l2flush: somehow 2MB does not seem robust, also cleanup step size to use line-size * hex-opreq: bump opreq batch size to 256 * hex-mm: place src1 spad at the top of vtcm for easy reuse * hex-ops: introduce internal types and disable src1 reuse for now Nothing new just formalizing the repack / qyn.quant types we've been using. * htp-opreq: use tensor pointers instead of copies * hex-opreq: introduce more robust way for tracking vtcm/spad reuse This removes the SKIP_QUANTIZE flag that became fragile with the addition of HMX and other ops. * hex-cumsum: fix error post opreq merge * hex-opreq: move request batch handling into the session Prepping everything for using dspqueue buffers and doing that inside the session is much cleaner. * hex-mm: yet another fix for src1 reuse when we're mixing hmx/hvx * hex-bufs: introduce pinned mmapings and use non-pinned ones for model buffers * hex-buf: add support for allocating shared/pinned buffer for opreqs * hex-opbatch: make opbatches configurable * hex-naming: better name for ggml_hexagon_shared_buffer * hex-naming: add session->c_name() helper * hex-opbatch: start using shm but still copy for now * hex-opbatch: use shared buffer for packing opbatch * hex-opbatch: beter naming for opbatch related classes and code * hex-opbatch: reuse batched tensors with same data/dims/strides * hex-opbatch: update logging * hex-opbatch: add support for vmem limit for op batching * hex-opbatch: update htp side to properly support dynamic mmap/unmap * hex-opbatch: add OB and OQ params for run-completion script and fix the asserts in batch processing * hex-opbatch: fixed src1 handling in act ops * hex-act: fix empty src1 handling in swiglu and friends Simplify preamble macro while at it * hex-mm: minor fix vtcm and dma handling in matmul cleaning up some left-overs from merges * hex-opbatch: allocate extra 1KB for dspqueue overhead * hexagon: fix softmax for non-aligned tensors and cleanup vtcm alloc * hex-mm: properly handle hmx_disabled flag * hex-ops: update comments * hex-ops: add debug output for get/set-rows * hex-mmap: optimize un/mapping of buffers * hex-opreq: global cache flush and invalidate beyond 128KB threshold * hex-ops: add super simple opfilter regex for debugging If an Op matches the regex hex backend will reject it. * hex-opbatch: wireup newer ops missed in merge and update main switch to detect this in future * hexagon: improved vtcm acquision to remove inter-op overhead Fully compatible with QNN-HTP coex * hex-mm: fixed hvx fallback path * hex-mm: lower the vmem threshold a bit further to ~3GB * hexagon: update debug & error logs This also fixes an issue with newer llvm merging repack and non-repack functions. We use those pointer to distinguish between buffer types. * hexagon: move ops context into main context Just a cleanup. We don't need separate contexts at this point. * hex-opbatch: cleanup naming and headers for opbatch and related descriptors * hex-fa: it's now better to enable FA during TG to reduce graph splits * hexagon: remove GGML_HEXAGON_EXPERIMENTAL env var It's no longer useful. Please use more flexible GGML_HEXAGON_OPFILTER to disable Ops if needed for debugging or validation. * hexagon: fixed editorconfig check * Update ggml/src/ggml-hexagon/ggml-hexagon.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> --------- Co-authored-by: Trivikram Reddy <tamarnat@qti.qualcomm.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> b8754	2026-04-10 15:47:43 -07:00
Aldehir RojasandGitHub	3fc65063d9	common : better align to the updated official gemma4 template (#21704) b8753	2026-04-10 16:12:53 -05:00
Adrien GallouëtandGitHub	05b3caaa48	common : add callback interface for download progress (#21735) ... Signed-off-by: Adrien Gallouët <angt@huggingface.co> b8752	2026-04-10 22:17:00 +02:00
MoonRide303andGitHub	e62fa13c24	model : make Gemma 4 shared-KV tail attn_k tensors optional on load (#21739) b8751	2026-04-10 21:45:50 +02:00
Rithik SharmaandGitHub	bfd1f453cb	ggml-webgpu: support non-square subgroup matrix configs for Intel GPUs (#21669) b8750	2026-04-10 10:52:38 -07:00
3 people	e4fed9d08d	ggml-webgpu: address quantization precision and backend lifecycle managment (#21521) ... * ggml(webgpu): fix the busy-polls in Emscripten in the waitAny after #20618, and remove the busy webgpu log * Merge with upstream * Fix GET_ROWS packed integer NaN when using f16 as memory buffer in shader quants * Update Unary wgsl EXP and EXPM1 for f16 stability * Fix GET_ROWS IQ4_XS strcut for NaN f16 canonicalization * Fix numerical percision for unary sqrt when working with f16 * Fix NaN canonicalization for packed integers using f16 * Update err threshold for binary div ops when using f16 * backend: Keep one Dawn/WebGPU instance alive for the lifetime of the static backend * clean: uncomment existing code logs * clean: clean the unncessary debug info * Refactor and generalize dequant helpers * Remove deprecated quant structs * Refactor shader defines to reduce repetition * Remove error override for F16 type * fix: fix the accidential removal of the proper initialization of ctx * clean: clean legacy and format code * fix: did not modify tests ops --------- Co-authored-by: Jeremy J. Hartmann <jeremy@mtion.tv> b8749	2026-04-10 10:52:01 -07:00
Adrien GallouëtandGitHub	5dd102539b	server : ignore --alias when using --models-preset (#21380) ... I'm not sure what the purpose of keeping `--alias` was when using `--models-preset`, but the result is really weird, as shown in the following logs: $ build/bin/llama-server --models-preset preset.ini --alias "Gemma 4 E4B UD Q8_K_XL" ... init: using 31 threads for HTTP server srv load_models: Loaded 2 cached model presets srv load_models: Loaded 1 custom model presets from preset.ini main: failed to initialize router models: alias 'Gemma 4 E4B UD Q8_K_XL' for model 'angt/test-split-model-stories260K:F32' conflicts with existing model name So I propose to simply ignore `--alias` too in this case. With this commit, the server starts in routing mode correctly. Signed-off-by: Adrien Gallouët <angt@huggingface.co> b8748	2026-04-10 17:42:56 +02:00
Adrien GallouëtandGitHub	fb38d6f278	common : fix when loading a cached HF models with unavailable API (#21670) ... Signed-off-by: Adrien Gallouët <angt@huggingface.co> b8747	2026-04-10 16:37:46 +02:00
Johannes GäßlerandGitHub	0893f50f2d	common: mark --split-mode tensor as experimental (#21684) b8746	2026-04-10 12:27:27 +02:00
Aleksander GrygierandGitHub	f989a6e39e	webui: Static build output improvements (#21667) ... * refactor: Build improvements * chore: Formatting + package lock update	2026-04-10 11:49:47 +02:00
Berk IdemandGitHub	d7ff074c87	common : enable reasoning budget sampler for gemma4 (#21697) ... * fix: enable reasoning budget sampler for gemma4 Add thinking_start_tag and thinking_end_tag to common_chat_params_init_gemma4(). Without these, the reasoning budget sampler never activates for gemma4. Make the newline after "thought" optional in the PEG parser to handle budget=0 (sampler forces end tag before the newline). Add test case for empty thinking block. Fixes #21487 * use p.space() instead of p.optional(p.literal("\n")) in gemma4 thought parser b8744	2026-04-10 11:49:14 +02:00
Belem ZhangandGitHub	3f8752b559	docs : fix broken link to ggml-openvino in OPENVINO.md (#21709)	2026-04-10 09:50:08 +02:00
Jeff BolzandGitHub	7b69125331	vulkan: Support Q1_0 (#21539) ... * vulkan: Support Q1_0 * use get_dm b8742	2026-04-10 08:35:27 +02:00
Adrien GallouëtandGitHub	e095a482a0	common : add fluidity to the progress bar (#21671) ... Signed-off-by: Adrien Gallouët <angt@huggingface.co> b8741	2026-04-10 08:24:53 +02:00
Aman GuptaandGitHub	e34f042154	CUDA: fuse muls (#21665) b8740	2026-04-10 10:24:09 +08:00
3 people	d132f22fc9	HIP: add CDNA4 (gfx950) architecture support for MI350X/MI355X (#21570) ... Add AMD Instinct MI350X/MI355X (gfx950, CDNA4) support: - vendors/hip.h: Add CDNA4 preprocessor define for __gfx950__ - common.cuh: Add GGML_CUDA_CC_CDNA4 and GGML_CUDA_CC_IS_CDNA4 macros - mma.cuh: Route CDNA4 to compatible MFMA instructions: * f32 matmul: mfma_f32_16x16x4f32 (xf32 variant unavailable on gfx950) * bf16 matmul: mfma_f32_16x16x16bf16_1k (same as CDNA3) * int8 matmul: mfma_i32_16x16x32_i8/32x32x16 (same as CDNA3) - mmq.cuh: Include CDNA4 in stream-k kernel dispatch CDNA4 is largely compatible with CDNA3 except: - No xf32 MFMA (mfma_f32_16x16x8_xf32) — routes to f32 path - Different FP8 format (e4m3fn vs e4m3_fnuz) — not changed here Tested on AMD Instinct MI355X (gfx950), ROCm 7.0.1: - Build: compiles cleanly with -DAMDGPU_TARGETS=gfx950 - llama-bench (Qwen2.5-1.5B Q4_K_M, single GPU): * f16+FA: 40,013 tok/s prefill, 254 tok/s decode * q8_0+FA: functional - Flash attention: works correctly - MMQ: works correctly with stream-k dispatch Co-authored-by: Andy Luo <andyluo7@users.noreply.github.com> b8739	2026-04-09 21:13:32 +02:00
5 people	d6f3030047	ggml: backend-agnostic tensor parallelism (experimental) (#19378) ... * ggml: backend-agnostic tensor parallelism * support for GPT-OSS, Qwen 3 MoE * partial Vulkan fix * add support for 4/8 GPUs * unconditional peer access * re-use buffers + ggml contexts * fix output pattern * NCCL support * GGML: HIP: add RCCL support * Remove shfl and AllReduce from backend interface * move allocation workaround out of ggml-alloc.c * 2d tensor set/get support * Fix the seg fault without NCCL * Apply suggestion from JohannesGaessler * support for tensor dims % n_devs != 0 * fix view_offs scaling * arbitrary num. of GPUs/tensor split * fix compilation * better granularity estimate * Support device-specific host buffer types if all underlying backends expose the same type. This allows using pinned memory instead of pageable memory for CUDA. Fix compilation errors. * partial Qwen 3 Next support * Fix qwen3 30b (#8) * Fix crash with Qwen-30B-A3B Q4_0 Qwen-30B-A3B Q4_0 has an intermediate dimension of 768. Using a granularity of 256 forces an uneven split between GPUs, which is not supported by the current implementation. * Decide block size based on tensor quantization type * Fix crashes due to KV cache serialization (#9) KV cache serialization requires non-zero offsets on the tensor. Add support in the meta backend to set/get a tensor with a non-zero offset. * metal : fix build (#7) * static memory allocations, fix usage count * fix tensor granularity * more even memory distribution * use BF16 for allreduce * rebase fixup * better error message for unsupported architectures * Fix device mismatch during scatter of allReduce. (#11) There is a mismatch between the dst buffer device and the backend device, causing the use of sync copies * Enable the previous allreduce implementation. It is better in both perf and stability (#12) * delay AllReduce for Moe for less I/O * build : clean-up compile warnings * backend : move most of the meta backend API to ggml-backend-impl.h * cont : hide unused public API in the implementation * llama : use llama_device + remove ggml_backend_dev_is_meta() * ggml-backend : remove unused alloc include * minor : remove regex include * ggml : introduce ggml-ext.h for staging new APIs * rebase fixup * fix tests * llama : more robust logic for determining Meta devices (#16) * llama : more robust logic for determining Meta devices * cont : fix devs size check Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * cont : fix log type Co-authored-by: Johannes Gäßler <johannesg@5d6.de> --------- Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * disable roundtrip for meta backend * fix arch selection * Qwen 3.5 support * fix Gemma 4 MoE * fix OpenVino, SYCL * fix test-llama-archs for CPU-only builds * Fix Qwen 3.5 MoE * disable meta backend tests for WebGPU * tests : filter CPU-based devices from the Meta backend tests (#17) * meta : formatting, naming, indentation (#18) * formatting : llama-model.cpp * formatting : ggml-ext.h * formatting : ggml-backend-meta.cpp * meta : add TODO * add documentation * better error messages * fix GPT-OSS --------- Co-authored-by: Carl Philipp Klemm <carl@uvos.xyz> Co-authored-by: Gaurav Garg <gaugarg@nvidia.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> b8738	2026-04-09 16:42:19 +02:00
3 people	009a113326	ggml : check return value of CUB calls used in argsort and top-k (they all return cudaError_t) (#21676) ... Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> b8737	2026-04-09 21:17:11 +08:00