Xuan Son Nguyen
b786bb2e60
wip 3
2026-06-20 18:56:58 +02:00
Xuan Son Nguyen
2b2eed8fd7
wip 2
2026-06-20 18:41:56 +02:00
Xuan Son Nguyen
6c5c5a29d6
wip
2026-06-20 16:48:12 +02:00
Xuan Son Nguyen
d5037c508a
server: refactor batch construction
2026-06-20 16:35:57 +02:00
e27f308597
server: avoid forwarding auth headers in CORS proxy ( #24373 )
...
* server: avoid forwarding auth headers in CORS proxy
* format
* fix test
* fix e2e test
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co >
2026-06-20 15:34:47 +02:00
Aldehir Rojas and GitHub
67e9fd3b74
docker : prebuild web UI for s390x build [no release] ( #24829 )
2026-06-20 05:54:42 -05:00
davidrhodus and GitHub
796f41bedc
model : glm-dsa load DSA indexer tensors as optional ( #24770 )
...
GLM-5.2 ships the DSA "lightning indexer" on only a subset of layers (the
"full" layers; others omit it), but the GLM_DSA loader created the five
indexer tensors on every layer as required, so loading any GLM-5.2 GGUF
failed with e.g. `missing tensor 'blk.3.indexer.k_norm.weight'`.
GLM_DSA's graph is llama_model_deepseek2::graph (plain MLA) and does not use
the indexer tensors (indexer runtime not yet implemented), so they are
loaded-but-unused. Marking them TENSOR_NOT_REQUIRED lets layers without an
indexer load as nullptr and the model runs as full MLA attention.
DeepSeek-V3.2 (uniform indexer on all layers) is unaffected.
2026-06-20 13:48:24 +03:00
Adrien Gallouët and GitHub
37a77fb057
ggml : optimize AMX ( #24806 )
...
Flatten the partition over n_batch * M so every thread participates in
the quantization
| CPU | Model | Test | t/s OLD | t/s NEW | Speedup |
|:--------------------------------|:------------------------------|:-------|----------:|----------:|----------:|
| Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B IQ4_NL - 4.5 bpw | pp512 | 730.71 | 779.86 | 1.07 |
| Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B IQ4_NL - 4.5 bpw | tg128 | 87.88 | 86.79 | 0.99 |
| Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B IQ4_XS - 4.25 bpw | pp512 | 725.09 | 1023.31 | 1.41 |
| Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B IQ4_XS - 4.25 bpw | tg128 | 83.64 | 83.62 | 1.00 |
| Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_0 | pp512 | 820.51 | 924.05 | 1.13 |
| Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_0 | tg128 | 90.59 | 92.46 | 1.02 |
| Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_1 | pp512 | 776.88 | 872.79 | 1.12 |
| Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_1 | tg128 | 89.39 | 90.94 | 1.02 |
| Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_K_M | pp512 | 719.28 | 1009.27 | 1.40 |
| Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_K_M | tg128 | 80.62 | 80.86 | 1.00 |
| Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_K_S | pp512 | 732.29 | 1077.29 | 1.47 |
| Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_K_S | tg128 | 86.42 | 83.53 | 0.97 |
Signed-off-by: Adrien Gallouët <angt@huggingface.co >
2026-06-20 13:43:06 +03:00
Sigbjørn Skjæret and GitHub
f4043fec01
convert : more consistent handling of rope_parameters ( #24833 )
2026-06-20 13:42:36 +03:00
Masashi Yoshimura and GitHub
f449e05537
ggml-webgpu: add adapter toggles for F16 on Vulkan + NVIDIA
2026-06-20 08:12:32 +09:00
Xuan-Son Nguyen and GitHub
2b686a9120
server: refactor child --> router communication ( #24821 )
...
* server: refactor child --> router communication
* fix wakeup case
* add docs
* improve update_status()
* nits
2026-06-20 01:02:26 +02:00
Adrien Gallouët and GitHub
4b48a53b6c
server : optimize get_token_probabilities ( #24796 )
...
Use std::partial_sort to order only the requested top-n tokens instead
of the full vocabulary
logprobs sort: vocab=128000 n_top=0 iters=100
full sort: 8555.6 us/op
partial sort: 704.3 us/op
Signed-off-by: Adrien Gallouët <angt@huggingface.co >
2026-06-19 23:26:54 +02:00
Xuan-Son Nguyen and GitHub
e475fa2b5f
mtmd, arg: fix utf8 handling on windows ( #24779 )
...
* mtmd, arg: fix utf8 handling on windows
* also fix ggml_fopen
* fix build fail
* also fix CLI
2026-06-19 22:28:38 +02:00
Xuan-Son Nguyen and GitHub
175147e8f6
server: remove all internal mentions about "webui" ( #24817 )
2026-06-19 22:12:46 +02:00
Mikolaj Kucharski and GitHub
fabde3bf51
arg: Add comment line support to --api-key-file ( #23168 )
2026-06-19 17:33:54 +02:00
Alessandro de Oliveira Faria (A.K.A.CABELO) and GitHub
0d2d9ccbf6
vendor : update cpp-httplib to 0.48.0 ( #24787 )
2026-06-19 22:16:35 +08:00
Xuan-Son Nguyen and GitHub
8c2d6f6475
server: add --agent arg, remove redundant webui naming compat ( #24801 )
...
* server: add --agent arg, remove redundant webui naming compat
* corrent env
* fix the test
* llama-gen-docs
* nits: wordings
2026-06-19 16:06:13 +02:00
Aldehir Rojas and GitHub
38724ab593
docker : build the UI ( #24794 )
...
* docker : build the UI
* cont : use existing APP_VERSION
2026-06-19 15:32:31 +02:00
Xuan-Son Nguyen and GitHub
e2e7a9b2d0
mtmd: several bug fixes ( #24784 )
...
* mtmd: several bug fixes
* fix build
* fix gemma4ua
* add sanity check in get_u32()
* fix build (2)
* area() avoid overflow
2026-06-19 12:18:36 +02:00
b14e3fb90c
spec: support eagle3 for qwen3.5 & 3.6 ( #24593 )
...
* spec: support qwen3.5 & 3.6 eagle3 draft
* eagle3: Add deferred boundary checkpoints restore support for hybrid models
* apply suggestions
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
* spec: adapt to API change
* spec: fix naming
* cont : add TODO
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2026-06-19 13:08:50 +03:00
159d093a43
server: fix non-bound n_discard value (ctx shifting) ( #24786 )
...
* server: fix non-bound n_discard value
* Update tools/server/server-context.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2026-06-19 10:53:44 +02:00
Georgi Gerganov
5fd2dc2c41
sync : ggml
2026-06-19 10:19:14 +03:00
Georgi Gerganov
1868af13ac
ggml : bump version to 0.15.2 (ggml/1548)
2026-06-19 10:19:14 +03:00
Georgi Gerganov and GitHub
5bd21b8555
pi : remove docs from system prompt ( #24791 )
2026-06-19 09:34:00 +03:00
Georgi Gerganov and GitHub
80452d65b9
server : consolidate slot selection into get_available_slot ( #24755 )
...
Absorb get_slot_by_id logic into get_available_slot so slot selection
is handled by a single function call. When a specific slot id is
requested, the LCP similarity check still runs to enable proper
prompt cache updates.
Assisted-by: pi:llama.cpp/Qwen3.6-27B
2026-06-19 09:22:34 +03:00
8141e730f1
ggml-cpu: support K tails in power10 Q8/Q4 MMA matmul ( #24753 )
...
* ggml-cpu: support K tails in Power10 MMA Q8/Q4 matmul
This patch removes the requirement that K be divisible by kc in the tinyBlas_Q0_PPC tiled matmul path. Process the final K panel using its actual depth and pass the reduced panel size through packing and kernel execution. This allows more workloads to use the MMA kernel and reduces fallback to mnpack.
* Apply suggestion from @taronaeo
Co-authored-by: Aaron Teo <taronaeo@gmail.com >
---------
Co-authored-by: Aaron Teo <taronaeo@gmail.com >
2026-06-19 08:55:38 +03:00
Xuan-Son Nguyen and GitHub
db52540f73
mtmd: add batching support for internvl ( #24775 )
2026-06-19 01:16:16 +02:00
Pascal and GitHub
3a3edc9ac6
Ggml/cuda col2im 1d ( #24417 )
...
* cuda: add GGML_OP_COL2IM_1D, follow-up to the CPU op
* cuda: col2im_1d use fast_div_modulo for the index decomposition
* cuda: col2im_1d tighten supports_op, type match and contiguous dst
2026-06-18 22:23:01 +02:00
Reguna and GitHub
40f3aafc45
server: add "X-Accel-Buffering": "no" header to streaming endpoints ( #24774 )
...
* server: add "X-Accel-Buffering": "no" header to streaming endpoints
This header tells Nginx (as a reverse proxy) to NOT buffer responses. (only affects streaming endpoints)
Without it, Nginx will break streaming with certain applications (notably the Pi coding harness).
2026-06-18 22:01:24 +02:00
Xuan-Son Nguyen and GitHub
a6b3260a42
mtmd: add batching for mtmd-cli, add video tests ( #24778 )
2026-06-18 21:55:04 +02:00
o7si and GitHub
32eddaf2ea
cmake : fix ui build with read-only source ( #24752 )
2026-06-18 18:59:18 +02:00
Xuan-Son Nguyen and GitHub
060ce1bf72
mtmd: refactor llava-uhd overview image handling (always use ov_img_first) ( #24769 )
...
* add dedicated "overview" for mtmd_image_preproc_out
* corrections
* correct (again)
* nits
* nits (2)
2026-06-18 18:53:49 +02:00
Max Krasnyansky and GitHub
d2c67959b3
hexagon: support for op-trace (fine-grain tracing of HVX/HMX/DMA events) ( #24592 )
...
* hex-optrace: add support for optrace and instrument matmul and flash-atten code
* hex-trace: improve trace event and prefetto generator
* hex-trace: add new script dedicated to handling traces, specifically perfetto traces
* hex-trace: add --head/--tail options to profile and trace tools
* hex-trace: fix whitespaces
* hex-trace: fix flake8 warnings
* hex-trace: fix flake8 warnings
* hmx-fa: restore q_tiles clearing
* hex-profile: remove circular dep in includes
* hex-trace: simplify trace sizing check
* hex-profile: sort events in the summary by name
2026-06-18 08:35:02 -07:00
Kangjia Gao and GitHub
7b6c5a2aed
docs: fix export-lora --lora-scaled syntax [no release] ( #24703 )
...
Assisted-by: Codex
2026-06-18 16:46:17 +02:00
Xuan-Son Nguyen and GitHub
fe7c8b2414
server: (router) fix stopping_thread potentially hang ( #24728 )
...
* server: (router) fix stopping_thread potentially hang
* fix windows build
2026-06-18 15:41:09 +02:00
Xuan-Son Nguyen and GitHub
e1efd0991d
server: add "schema" and validation ( #24150 )
...
* wip
* working
* correct some limits
* add field name to error message
2026-06-18 15:40:58 +02:00
08023072ef
server : add last-5-seconds generation speed display ( #24291 )
...
* server : add last-5-seconds generation speed display
* cont : clean-up
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2026-06-18 14:02:20 +02:00
Amos Wong and GitHub
20832179e2
ui: provide touch accessible model selection UI ( #24604 )
...
* ui : add model selector storybook stories
Covers list, favorites, single-model, all status states
(loading/loaded/sleeping/failed/idle), and selection states.
* ui : improve model selector mobile UX with hover media queries
Use @media (hover:none) to show action buttons directly on touch
devices and color-code them by model status (amber=sleeping,
green=loaded, muted=idle). Status dots hidden on touch. Desktop
hover behavior unchanged.
2026-06-18 13:14:20 +02:00
Anuj Attri and GitHub
10786217e9
server : return HTTP 400 on invalid grammar ( #24144 ) ( #24154 )
...
Throw on grammar parse failure so the server returns HTTP 400
instead of silently dropping the constraint.
Add a regression test for the invalid-grammar response.
Fixes #24144
2026-06-18 12:49:14 +02:00
Xuan-Son Nguyen and GitHub
552258c535
server: (router) rework -hf preset repo ( #24739 )
...
* server: temporary remove HF remote preset
* rework remove preset.ini support
* rm unused get_remote_preset_whitelist()
* print warning
* add docs
* rm stray file
2026-06-18 12:45:23 +02:00
Xuan-Son Nguyen and GitHub
968c43891a
server: fix router args not being forwarded to child instances ( #24760 )
2026-06-18 12:15:46 +02:00
Xuan-Son Nguyen and GitHub
24bba7b98e
mtmd: refactor preprocessor, add mtmd_image_preproc_out ( #24736 )
...
* add mtmd_image_preproc_out
* add dev docs
* remove unused clip API
* rm unused clip_image_f32_batch::grid
* change preprocess() call signature
2026-06-18 12:04:39 +02:00
Neo Zhang and GitHub
9724f664e8
[SYCL] rename GGML_SYCL_SUPPORT_LEVEL_ZERO ( #24719 )
...
* rename GGML_SYCL_SUPPORT_LEVEL_ZERO to GGML_SYCL_SUPPORT_LEVEL_ZERO_API, and GGML_SYCL_ENABLE_LEVEL_ZERO to GGML_SYCL_USE_LEVEL_ZERO_API
* fix code format
* fix error when rebase
2026-06-18 11:18:26 +03:00
Neo Zhang and GitHub
dd69db2924
sycl : support MUL_MAT and OUT_PROD with Q1_0 ( #24721 )
2026-06-18 11:17:37 +03:00
Adrien Gallouët and GitHub
6ec59ddaea
app : enable self-update only when built with llama-install.sh ( #24754 )
...
Signed-off-by: Adrien Gallouët <angt@huggingface.co >
2026-06-18 09:57:59 +02:00
Sigbjørn Skjæret and GitHub
32e806b9c1
ci : fix check-release message parsing ( #24751 )
2026-06-18 09:32:56 +02:00
Neo Zhang and GitHub
6f1034b32a
[SYCL] support OPs: conv_2d, conv_2d_dw, conv2d_transpose ( #24600 )
...
* fix conflict
* fix format issue, rename
* rm debug code
* correct the file name
2026-06-18 09:40:03 +03:00
Aleksander Grygier and GitHub
0b73fc79fe
ui: Update code formatting command in pre-commit hook ( #24685 )
2026-06-18 08:33:50 +02:00
Ravi Panchumarthy and GitHub
4a79037b8b
ci : fix Windows x64 (OpenVINO) release link ( #24731 )
2026-06-18 08:30:08 +02:00
Georgi Gerganov and GitHub
cae0a3b0b0
metal : check for BF16 support in concat kernel ( #24747 )
2026-06-18 09:16:06 +03:00
Xuan Son Nguyen