wylab/llama.cpp
0
0
Fork 1
mirror of https://github.com/ggml-org/llama.cpp.git synced 2026-09-07 16:37:57 +02:00
Code Issues Packages Projects Releases Wiki Activity
10,841 Commits 778 Branches 7,351 Tags
0cc4m/vulkan-mul-mm-refactor
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Adrien GallouëtandGitHub c457e3bf7f ui : embed assets directly with CMake (#28445) 
Remove the build-time C++ helper and external gzip dependency,
simplifying cross-compilation. Keep the generated C++ in templates for
readability and preserve fully embedded UI assets.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-09-06 07:49:39 +02:00
Adrien GallouëtandGitHub 4cbe8b070b ggml : don't crash when backend search path can't be read (#28271) 
Use std::error_code overloads of fs::current_path() and
fs::directory_iterator in ggml_backend_load_best() so an
inaccessible search path (WebDAV mount, removed CWD) is
skipped instead of terminating the process with an uncaught
filesystem_error.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-09-04 10:24:06 +03:00
Adrien GallouëtandGitHub 24f5bf8a41 ggml : remove GGML_CUDA_PEER_MAX_BATCH_SIZE (#28177) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-09-04 10:22:01 +03:00
Adrien GallouëtandGitHub c390d0abbc common : make build info output stream configurable (#28322) 
Let llama_print_build_info write to a caller-provided FILE* instead of
hardcoding stderr. The parameter defaults to stderr so existing callers
keep their current behavior.

The version command in llama-app now passes stdout, so plain version
output goes to stdout where users expect it.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-09-04 09:13:20 +03:00
Adrien GallouëtandGitHub 3d3d7c8181 ggml-cuda : remove unused vars (#28235) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-09-02 18:54:11 +02:00
Adrien GallouëtandGitHub 0e4a036223 common : add common_print_available_devices() (#26170) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-07-27 18:19:59 +02:00
Adrien GallouëtandGitHub 720d7fa409 vendor : update cpp-httplib to 0.51.0 (#26067) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-07-25 18:16:29 +02:00
Adrien GallouëtandGitHub aff6eb6e75 tokenize : drop --stdin mutual-exclusion check (#25672) 
match cli and completion, which don't enforce it
 2026-07-15 18:41:51 +02:00
Adrien GallouëtandGitHub 8ff8c4299d tokenize : align usage by using common args (#25516) 
Migrate the tokenize tool to common_params_parse, replacing its
hand-rolled argv parsing, Windows UTF-8 handling and file reading
with the shared common helpers.

Expose the model-sourcing flags (-m, -mu, -dr, -hf, -hff, --offline,
HF_TOKEN) to LLAMA_EXAMPLE_TOKENIZE, and register --ids, --stdin,
--no-bos, --no-parse-special and --show-count as common args.
parse_special defaults to true for TOKENIZE to preserve the old
behavior. Errors now go through LOG_ERR instead of fprintf(stderr).

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-07-14 15:20:53 +02:00
Adrien GallouëtandGitHub 259f2e2a53 llama-bench : init params.offline (#25476) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-07-09 11:56:56 +02:00
Adrien GallouëtandGitHub 48719618e8 scripts : use HF_TOKEN when downloading UI assets (#25280) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-07-06 09:53:35 +02:00
Adrien GallouëtandGitHub fdb1db877c llama : add llama_model_ftype_name() (#25134) 
* llama : add llama_model_ftype_name()

Expose the model file type (quantization) name, e.g. "Q8_0" or
"Q4_K - Medium", through a new public C API. The returned pointer is
valid for the lifetime of the model and nullptr when the model is
invalid or the file type is unknown.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Export enum

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* s/llama_model_ftype_name/llama_ftype_name/

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Move "(guessed)" to the front in llama_ftype_name

Prepend the "(guessed)" label instead of appending it. This allows removing
the non-thread-safe static std::string, making the function allocation-free.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Add LLAMA_FTYPE_PREFIX

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Dont check for model

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-07-02 17:26:47 +02:00
Adrien GallouëtandGitHub a6647b1a32 common : use hf primary split as model path (#25194) 
Fixes #25181
 2026-07-01 18:33:00 +02:00
Adrien GallouëtandGitHub 6f4f53f2b7 common : dedup preset and cached model entries in /v1/models (#25131) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-06-29 17:37:23 +02:00
Adrien GallouëtandGitHub c1a1c8ee94 common : allow --offline in llama download (#25091) 
Expose the existing --offline flag to `llama download` so a script can
run it to check whether a model is already cached and ready to be served
without touching the network.

Also fix a latent use-after-free in the URL-task on_done callback:
first_path is block-scoped and was captured by reference, but invoked
after the block ends.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-06-28 12:34:11 +02:00
Adrien GallouëtandGitHub 050ee92d04 app : allow --version, --licenses & --help (#25054) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-06-26 23:18:11 +02:00
Adrien GallouëtandGitHub 5d8ccdf9d1 devops : add llama in all docker images (#25035) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-06-26 15:15:48 +02:00
Adrien GallouëtandGitHub 683b04cc4a app : add the llama download subcommand (#24982) 
* app : add the download command (with llama-download)

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Remove llama-download tool for now

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-06-25 13:36:36 +02:00
Adrien GallouëtandGitHub 84de01a1f1 llama : use LLM_KV for quantization_version & file_type (#24802) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-06-20 20:07:01 +02:00
Adrien GallouëtandGitHub 37a77fb057 ggml : optimize AMX (#24806) 
Flatten the partition over n_batch * M so every thread participates in
the quantization

    | CPU                             | Model                         | Test   |   t/s OLD |   t/s NEW |   Speedup |
    |:--------------------------------|:------------------------------|:-------|----------:|----------:|----------:|
    | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B IQ4_NL - 4.5 bpw  | pp512  |    730.71 |    779.86 |      1.07 |
    | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B IQ4_NL - 4.5 bpw  | tg128  |     87.88 |     86.79 |      0.99 |
    | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B IQ4_XS - 4.25 bpw | pp512  |    725.09 |   1023.31 |      1.41 |
    | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B IQ4_XS - 4.25 bpw | tg128  |     83.64 |     83.62 |      1.00 |
    | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_0              | pp512  |    820.51 |    924.05 |      1.13 |
    | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_0              | tg128  |     90.59 |     92.46 |      1.02 |
    | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_1              | pp512  |    776.88 |    872.79 |      1.12 |
    | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_1              | tg128  |     89.39 |     90.94 |      1.02 |
    | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_K_M            | pp512  |    719.28 |   1009.27 |      1.40 |
    | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_K_M            | tg128  |     80.62 |     80.86 |      1.00 |
    | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_K_S            | pp512  |    732.29 |   1077.29 |      1.47 |
    | Intel(R) Xeon(R) Platinum 8488C | qwen35 0.8B Q4_K_S            | tg128  |     86.42 |     83.53 |      0.97 |

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-06-20 13:43:06 +03:00
Adrien GallouëtandGitHub 4b48a53b6c server : optimize get_token_probabilities (#24796) 
Use std::partial_sort to order only the requested top-n tokens instead
of the full vocabulary

    logprobs sort: vocab=128000 n_top=0 iters=100
    full    sort:   8555.6 us/op
    partial sort:    704.3 us/op

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-06-19 23:26:54 +02:00
Adrien GallouëtandGitHub 6ec59ddaea app : enable self-update only when built with llama-install.sh (#24754) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-06-18 09:57:59 +02:00
Adrien GallouëtandGitHub e3a74b2990 bench : add --offline (#24511) 
* bench : add --offline

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Add default

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-06-16 08:26:05 +02:00
Adrien GallouëtandGitHub 70b54e140c vendor : update cpp-httplib to 0.47.0 (#24395) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-06-12 11:34:44 +02:00
Adrien GallouëtandGitHub ac4cddeb0d vendor : update LibreSSL to 4.3.2 (#24397) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-06-10 22:28:03 +02:00
Adrien GallouëtandGitHub 6b80c74f28 completion : remove useless statics (#24226) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-06-06 12:16:16 +02:00
Adrien GallouëtandGitHub 588f0dc2ce completion : fix format specifier in LOG_INF (#24213) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-06-06 11:24:27 +02:00
Adrien GallouëtandGitHub 335abed17d vendor : update cpp-httplib to 0.46.1 (#23980) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-06-01 19:40:10 +03:00
Adrien GallouëtandGitHub 98e480a32e app : move licences to llama-app (#23824) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-05-29 07:46:11 +02:00
Adrien GallouëtandGitHub 479a9a1b03 app : improve help output (#23805) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-05-28 16:45:06 +02:00
Adrien GallouëtandGitHub 48e7eae41c perplexity : fix format specifier in LOG_ERR (#23788) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-05-28 10:34:58 +03:00
Adrien GallouëtandGitHub 1d7ab2b947 app : add batched-bench, fit-params, quantize & perplexity (#23459) 
* app : add batched-bench, fit-params, quantize & perplexity

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Add missing main.cpp

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Add EOL

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-05-21 10:29:44 +03:00
Adrien GallouëtandGitHub ce02093fdd app : show version (#23426) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-05-21 06:21:13 +02:00
Adrien GallouëtandGitHub 29f1482221 app : introduce the llama unified executable (#23296) 
* app : introduce the llama unified executable

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Use serve for server

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Hide completion and bench, add help command

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Remove STATIC

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Use -impl targets instead of -lib

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Revert "Remove STATIC"

This reverts commit cc44caccb9.

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-05-20 13:22:22 +02:00
Adrien GallouëtandGitHub 45b455e66f common : remove hf cache migration (#23266) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-05-18 17:11:47 +02:00
Adrien GallouëtandGitHub 3980e04d5a llama : add missing call to ggml_backend_load_all() (#22752) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-05-07 08:24:47 +03:00
Adrien GallouëtandGitHub 2635ac76e8 common : fix missing-noreturn warnings when compiling with clang 21 (#22702) 
common/arg.cpp:3719:9: error: function 'operator()' could be declared with attribute 'noreturn' [-Werror,-Wmissing-noreturn]
     3719 |         [](common_params & /*params*/, int /*value*/) {
          |         ^
    common/arg.cpp:3726:9: error: function 'operator()' could be declared with attribute 'noreturn' [-Werror,-Wmissing-noreturn]
     3726 |         [](common_params & /*params*/, int /*value*/) {
          |         ^
    common/arg.cpp:3733:9: error: function 'operator()' could be declared with attribute 'noreturn' [-Werror,-Wmissing-noreturn]
     3733 |         [](common_params & /*params*/, int /*value*/) {
          |         ^
    common/arg.cpp:3740:9: error: function 'operator()' could be declared with attribute 'noreturn' [-Werror,-Wmissing-noreturn]
     3740 |         [](common_params & /*params*/, int /*value*/) {
          |         ^
    common/arg.cpp:3747:9: error: function 'operator()' could be declared with attribute 'noreturn' [-Werror,-Wmissing-noreturn]
     3747 |         [](common_params & /*params*/, int /*value*/) {
          |         ^

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-05-05 13:16:25 +03:00
Adrien GallouëtandGitHub bf76ac77be common : only load backends when required (#22290) 
* common : only load backends when required

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* llama : call ggml_backend_load_all() directly from llama_backend_init()

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Add ggml_backend_load_all() where llama_backend_init() is not used

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-05-05 09:23:50 +02:00
Adrien GallouëtandGitHub beb42fffa4 common : check for null getpwuid in hf-cache (#22550) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-04-30 21:32:41 +02:00
Adrien GallouëtandGitHub 5f0ab726f7 vendor : update cpp-httplib to 0.43.2 (#22548) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-04-30 15:04:39 +02:00
Adrien GallouëtandGitHub 50494a2800 ggml : skip already registered backends and devices (#22296) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-04-28 10:02:32 +03:00
Adrien GallouëtandGitHub d530d6e7a2 ggml : revert to -lm linking instead of find_library (#22355) 
* ggml : revert to -lm linking instead of find_library

`find_library(MATH_LIBRARY m)` was introduced recently, but it breaks
CUDA compilation with GGML_STATIC. I could not find any valid use case
where we would prefer `find_library` over the standard `-lm` approach.

This commit is also meant to start a discussion if there is a valid
reason to keep `find_library(MATH_LIBRARY m)`, we should clarify what
problem it was solving and find an alternative fix that does not break
CUDA with GGML_STATIC.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* ggml : use MATH_LIBRARY only if defined

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* ggml : fix initial broken condition

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* ggml : always respect MATH_LIBRARY when defined

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-04-28 09:56:02 +03:00
Adrien GallouëtandGitHub f84270ea10 ggml : use 64 bytes aligned tile buffers (#21058) 
| Model                            | Test   |   t/s OLD |   t/s NEW |   Speedup |
|:---------------------------------|:-------|----------:|----------:|----------:|
| qwen35 0.8B BF16                 | pp512  |    584.59 |    595.41 |      1.02 |
| qwen35 0.8B BF16                 | tg128  |     52.23 |     52.82 |      1.01 |
| qwen35 0.8B IQ2_M - 2.7 bpw      | pp512  |    260.64 |    261.70 |      1.00 |
| qwen35 0.8B IQ2_M - 2.7 bpw      | tg128  |     81.17 |     80.89 |      1.00 |
| qwen35 0.8B IQ2_XXS - 2.0625 bpw | pp512  |    302.36 |    302.56 |      1.00 |
| qwen35 0.8B IQ2_XXS - 2.0625 bpw | tg128  |     84.93 |     85.12 |      1.00 |
| qwen35 0.8B IQ3_XXS - 3.0625 bpw | pp512  |    263.22 |    260.01 |      0.99 |
| qwen35 0.8B IQ3_XXS - 3.0625 bpw | tg128  |     80.29 |     78.94 |      0.98 |
| qwen35 0.8B IQ4_NL - 4.5 bpw     | pp512  |    728.65 |    742.09 |      1.02 |
| qwen35 0.8B IQ4_NL - 4.5 bpw     | tg128  |     82.39 |     84.46 |      1.03 |
| qwen35 0.8B IQ4_XS - 4.25 bpw    | pp512  |    681.33 |    677.06 |      0.99 |
| qwen35 0.8B IQ4_XS - 4.25 bpw    | tg128  |     80.18 |     79.28 |      0.99 |
| qwen35 0.8B Q2_K_M               | pp512  |    413.28 |    415.94 |      1.01 |
| qwen35 0.8B Q2_K_M               | tg128  |     81.90 |     82.78 |      1.01 |
| qwen35 0.8B Q3_K_M               | pp512  |    493.17 |    495.08 |      1.00 |
| qwen35 0.8B Q3_K_M               | tg128  |     82.75 |     83.23 |      1.01 |
| qwen35 0.8B Q3_K_S               | pp512  |    429.35 |    427.64 |      1.00 |
| qwen35 0.8B Q3_K_S               | tg128  |     86.69 |     87.02 |      1.00 |
| qwen35 0.8B Q4_0                 | pp512  |    783.46 |    782.32 |      1.00 |
| qwen35 0.8B Q4_0                 | tg128  |     88.23 |     87.90 |      1.00 |
| qwen35 0.8B Q4_1                 | pp512  |    741.71 |    729.76 |      0.98 |
| qwen35 0.8B Q4_1                 | tg128  |     85.44 |     86.01 |      1.01 |
| qwen35 0.8B Q4_K_M               | pp512  |    676.24 |    681.31 |      1.01 |
| qwen35 0.8B Q4_K_M               | tg128  |     76.59 |     77.06 |      1.01 |
| qwen35 0.8B Q4_K_S               | pp512  |    683.12 |    688.81 |      1.01 |
| qwen35 0.8B Q4_K_S               | tg128  |     80.50 |     81.19 |      1.01 |
| qwen35 0.8B Q5_K_M               | pp512  |    635.33 |    642.11 |      1.01 |
| qwen35 0.8B Q5_K_M               | tg128  |     72.07 |     72.49 |      1.01 |
| qwen35 0.8B Q5_K_S               | pp512  |    660.95 |    658.18 |      1.00 |
| qwen35 0.8B Q5_K_S               | tg128  |     72.19 |     72.95 |      1.01 |
| qwen35 0.8B Q6_K                 | pp512  |    647.97 |    638.84 |      0.99 |
| qwen35 0.8B Q6_K                 | tg128  |     72.83 |     72.49 |      1.00 |
| qwen35 0.8B Q8_0                 | pp512  |    805.01 |    785.49 |      0.98 |
| qwen35 0.8B Q8_0                 | tg128  |     70.10 |     70.13 |      1.00 |

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-04-27 09:30:55 +03:00
Adrien GallouëtandGitHub dc80c5252a common : fix jinja warnings with clang 21 (#22313) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-04-24 12:36:02 +02:00
Adrien GallouëtandGitHub 12568ca8c8 vendor : update LibreSSL to 4.3.1 (#22285) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-04-23 17:45:56 +02:00
Adrien GallouëtandGitHub be76dd0bb2 vendor : update BoringSSL to 0.20260413.0 (#21881) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-04-14 14:25:09 +03:00
Adrien GallouëtandGitHub aa00911d12 common : add download cancellation and temp file cleanup (#21813)
CI (android) / android-ndk (arm64-cpu, -D ANDROID_ABI=arm64-v8a -D ANDROID_PLATFORM=android-31 -D CMAKE_TOOLCHAIN_FILE=${ANDROID_NDK_ROOT}/build/cmake/android.toolchain.cmake -D GGML_NATIVE=OFF -DGGML_CPU_ARM_ARCH=armv8.5-a+fp16+i8mm -G Ninja -D LLAMA_OPENSSL=OFF -D … (push) Failing after 1m49s
CI (android) / android-ndk (arm64-snapdragon, --preset arm64-android-snapdragon-release) (push) Failing after 8s
CI (sanitize) / ubuntu-latest-sanitizer (Debug, ADDRESS) (push) Failing after 13s
CI (sanitize) / ubuntu-latest-sanitizer (Debug, THREAD) (push) Failing after 13s
CI (sanitize) / ubuntu-latest-sanitizer (Debug, UNDEFINED) (push) Failing after 15s
CI (android) / android (push) Failing after 5m1s
CI / ubuntu-latest-rpc (push) Failing after 6m52s
CI / ubuntu-latest-cuda (push) Failing after 2m1s
CI / build-cmake-pkg (push) Successful in 28m2s
Build Actions Cache / ubuntu-24-vulkan-cache (push) Has been cancelled
Build Actions Cache / ubuntu-24-openvino-cache (push) Has been cancelled
Build Actions Cache / windows-2022-rocm-cache (push) Has been cancelled
Server (sanitize) / server (RelWithDebInfo, UNDEFINED) (push) Successful in 1h6m25s
Server (sanitize) / server (RelWithDebInfo, ADDRESS) (push) Failing after 1h28m57s
Server / server (default) (push) Successful in 43m24s
Server / server (backend-sampling) (push) Successful in 40m40s
CI (3rd-party) / ubuntu-24-llguidance (push) Has been cancelled
CI (apple) / macOS-latest-ios (push) Has been cancelled
CI (apple) / macos-latest-ios-xcode (push) Has been cancelled
CI (apple) / macOS-latest-tvos (push) Has been cancelled
CI (apple) / macOS-latest-visionos (push) Has been cancelled
CI (apple) / macOS-latest-swift (generic/platform=iOS) (push) Has been cancelled
CI (apple) / macOS-latest-swift (generic/platform=macOS) (push) Has been cancelled
CI (apple) / macOS-latest-swift (generic/platform=tvOS) (push) Has been cancelled
CI (cann) / openEuler-latest-cann (aarch64, Release, 310p, off) (push) Has been cancelled
CI (cann) / openEuler-latest-cann (aarch64, Release, 910b, off) (push) Has been cancelled
CI (cann) / openEuler-latest-cann (aarch64, Release, 910b, on) (push) Has been cancelled
CI (cann) / openEuler-latest-cann (x86, Release, 310p, off) (push) Has been cancelled
CI (cann) / openEuler-latest-cann (x86, Release, 910b, off) (push) Has been cancelled
CI (cann) / openEuler-latest-cann (x86, Release, 910b, on) (push) Has been cancelled
CI (riscv) / ubuntu-riscv64-native-sanitizer (Debug, ADDRESS) (push) Has been cancelled
CI (riscv) / ubuntu-riscv64-native-sanitizer (Debug, THREAD) (push) Has been cancelled
CI (riscv) / ubuntu-riscv64-native-sanitizer (Debug, UNDEFINED) (push) Has been cancelled
CI (self-hosted) / ggml-ci-nvidia-cuda (push) Has been cancelled
CI (self-hosted) / ggml-ci-nvidia-vulkan-cm (push) Has been cancelled
CI (self-hosted) / ggml-ci-nvidia-vulkan-cm2 (push) Has been cancelled
CI (self-hosted) / ggml-ci-linux-intel-vulkan (push) Has been cancelled
CI (self-hosted) / ggml-ci-win-intel-vulkan (push) Has been cancelled
CI (self-hosted) / ggml-ci-intel-openvino-gpu-low-perf (push) Has been cancelled
CI (vulkan) / ubuntu-24-vulkan-llvmpipe (push) Has been cancelled
CI / macOS-latest-arm64 (push) Has been cancelled
CI / macOS-latest-x64 (push) Has been cancelled
CI / macOS-latest-arm64-webgpu (push) Has been cancelled
CI / ubuntu-cpu (arm64, ubuntu-24.04-arm) (push) Has been cancelled
CI / ubuntu-cpu (ppc64le, ubuntu-24.04-ppc64le) (push) Has been cancelled
CI / ubuntu-cpu (s390x, ubuntu-24.04-s390x) (push) Has been cancelled
CI / ubuntu-cpu (x64, ubuntu-22.04) (push) Has been cancelled
CI / ubuntu-24-vulkan (arm64, ubuntu-24.04-arm) (push) Has been cancelled
CI / ubuntu-24-vulkan (x64, ubuntu-24.04) (push) Has been cancelled
CI / ubuntu-24-webgpu (push) Has been cancelled
CI / ubuntu-24-webgpu-wasm (push) Has been cancelled
CI / ubuntu-22-hip (push) Has been cancelled
CI / ubuntu-22-musa (push) Has been cancelled
CI / ubuntu-22-sycl (push) Has been cancelled
CI / ubuntu-22-sycl-fp16 (push) Has been cancelled
CI / ubuntu-24-openvino-CPU (push) Has been cancelled
CI / ubuntu-24-openvino-GPU (push) Has been cancelled
CI / windows-latest (arm64, llvm-arm64, -G "Ninja Multi-Config" -D CMAKE_TOOLCHAIN_FILE=cmake/arm64-windows-llvm.cmake -DGGML_NATIVE=OFF -DLLAMA_BUILD_SERVER=ON) (push) Has been cancelled
CI / windows-latest (arm64, llvm-arm64-opencl-adreno, -G "Ninja Multi-Config" -D CMAKE_TOOLCHAIN_FILE=cmake/arm64-windows-llvm.cmake -DCMAKE_PREFIX_PATH="$env:RUNNER_TEMP/opencl-arm64-release" -DGGML_OPENCL=ON -DGGML_OPENCL_USE_ADRENO_KERNELS=ON) (push) Has been cancelled
CI / windows-latest (x64, cpu-x64 (static), -G "Ninja Multi-Config" -D CMAKE_TOOLCHAIN_FILE=cmake/x64-windows-llvm.cmake -DGGML_NATIVE=OFF -DLLAMA_BUILD_SERVER=ON -DGGML_RPC=ON -DBUILD_SHARED_LIBS=OFF) (push) Has been cancelled
CI / windows-latest (x64, openblas-x64, -G "Ninja Multi-Config" -D CMAKE_TOOLCHAIN_FILE=cmake/x64-windows-llvm.cmake -DGGML_NATIVE=OFF -DLLAMA_BUILD_SERVER=ON -DGGML_RPC=ON -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON -DGGML_OPENMP=OFF -DGGML_BLAS=ON -DG… (push) Has been cancelled
CI / windows-latest (x64, vulkan-x64, -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=OFF -DLLAMA_BUILD_SERVER=ON -DGGML_RPC=ON -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON -DGGML_VULKAN=ON) (push) Has been cancelled
CI / windows-2022-cuda (12.4) (push) Has been cancelled
CI / windows-latest-sycl (push) Has been cancelled
CI / windows-latest-hip (push) Has been cancelled
CI / ubuntu-cpu-riscv64-native (push) Has been cancelled
CI / ggml-ci-x64-cpu-low-perf (push) Has been cancelled
CI / ggml-ci-arm64-cpu-low-perf (push) Has been cancelled
CI / ggml-ci-x64-cpu-high-perf (push) Has been cancelled
CI / ggml-ci-arm64-cpu-high-perf (push) Has been cancelled
CI / ggml-ci-arm64-cpu-high-perf-sve (push) Has been cancelled
CI / ggml-ci-arm64-cpu-kleidiai (push) Has been cancelled
CI / ggml-ci-arm64-cpu-kleidiai-graviton4 (push) Has been cancelled
EditorConfig Checker / editorconfig (push) Has been cancelled
Release / macOS-cpu (arm64, arm64, -DGGML_METAL_USE_BF16=ON -DGGML_METAL_EMBED_LIBRARY=ON, macos-14) (push) Has been cancelled
Release / macOS-cpu (arm64, arm64-kleidiai, -DGGML_METAL_USE_BF16=ON -DGGML_METAL_EMBED_LIBRARY=ON -DGGML_CPU_KLEIDIAI=ON, macos-14) (push) Has been cancelled
Release / macOS-cpu (x64, x64, -DGGML_METAL=OFF -DCMAKE_OSX_DEPLOYMENT_TARGET=13.3, macos-15-intel) (push) Has been cancelled
Release / ubuntu-cpu (arm64, ubuntu-24.04-arm) (push) Has been cancelled
Release / ubuntu-cpu (s390x, ubuntu-24.04-s390x) (push) Has been cancelled
Release / ubuntu-cpu (x64, ubuntu-22.04) (push) Has been cancelled
Release / ubuntu-vulkan (arm64, ubuntu-24.04-arm) (push) Has been cancelled
Release / ubuntu-vulkan (x64, ubuntu-22.04) (push) Has been cancelled
Release / ubuntu-24-openvino (push) Has been cancelled
Release / windows-cpu (arm64) (push) Has been cancelled
Release / windows-cpu (x64) (push) Has been cancelled
Release / windows (arm64, opencl-adreno, -G "Ninja Multi-Config" -D CMAKE_TOOLCHAIN_FILE=cmake/arm64-windows-llvm.cmake -DCMAKE_PREFIX_PATH="$env:RUNNER_TEMP/opencl-arm64-release" -DGGML_OPENCL=ON -DGGML_OPENCL_USE_ADRENO_KERNELS=ON, ggml-opencl) (push) Has been cancelled
Release / windows (x64, vulkan, -DGGML_VULKAN=ON, ggml-vulkan) (push) Has been cancelled
Release / windows-cuda (12.4) (push) Has been cancelled
Release / windows-cuda (13.1) (push) Has been cancelled
Release / windows-sycl (push) Has been cancelled
Release / ubuntu-22-rocm (7.2.1, x64, gfx908;gfx90a;gfx942;gfx1030;gfx1100;gfx1101;gfx1102;gfx1151;gfx1150;gfx1200;gfx1201) (push) Has been cancelled
Release / windows-hip (gfx1150;gfx1151;gfx1200;gfx1201;gfx1100;gfx1101;gfx1102;gfx1030;gfx1031;gfx1032, radeon) (push) Has been cancelled
Release / ios-xcode-build (push) Has been cancelled
Release / openEuler-cann (aarch64, Release, 310p, off) (push) Has been cancelled
Release / openEuler-cann (aarch64, Release, 910b, on) (push) Has been cancelled
Release / openEuler-cann (x86, Release, 310p, off) (push) Has been cancelled
Release / openEuler-cann (x86, Release, 910b, on) (push) Has been cancelled
Release / release (push) Has been cancelled
Server (self-hosted) / server-metal (GPUx2, backend-sampling) (push) Has been cancelled
Server (self-hosted) / server-metal (GPUx2) (push) Has been cancelled
Server (self-hosted) / server-metal (GPUx1) (push) Has been cancelled
Server (self-hosted) / server-metal (GPUx1, backend-sampling) (push) Has been cancelled
Server (self-hosted) / server-cuda (GPUx1) (push) Has been cancelled
Server (self-hosted) / server-cuda (GPUx1, backend-sampling) (push) Has been cancelled
Server / server-windows (push) Has been cancelled
HIP quality check / ubuntu-22-hip-quality-check (push) Has been cancelled
flake8 Lint / Lint (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Server WebUI / WebUI Checks (push) Has been cancelled 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-04-13 11:18:23 +02:00
Adrien GallouëtandGitHub 05b3caaa48 common : add callback interface for download progress (#21735) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-04-10 22:17:00 +02:00
Adrien GallouëtandGitHub 5dd102539b server : ignore --alias when using --models-preset (#21380) 
I'm not sure what the purpose of keeping `--alias` was when using
`--models-preset`, but the result is really weird, as shown in the
following logs:

    $ build/bin/llama-server --models-preset preset.ini --alias "Gemma 4 E4B UD Q8_K_XL"
    ...
    init: using 31 threads for HTTP server
    srv   load_models: Loaded 2 cached model presets
    srv   load_models: Loaded 1 custom model presets from preset.ini
    main: failed to initialize router models: alias 'Gemma 4 E4B UD Q8_K_XL' for model 'angt/test-split-model-stories260K:F32' conflicts with existing model name

So I propose to simply ignore `--alias` too in this case. With this
commit, the server starts in routing mode correctly.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-04-10 17:42:56 +02:00
Adrien GallouëtandGitHub fb38d6f278 common : fix when loading a cached HF models with unavailable API (#21670) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-04-10 16:37:46 +02:00
Adrien GallouëtandGitHub e095a482a0 common : add fluidity to the progress bar (#21671) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-04-10 08:24:53 +02:00
Adrien GallouëtandGitHub 4293919068 common : skip non-primary GGUF split files when selecting model (#21633) 
We should not assume files are listed in order.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-04-09 07:28:06 +02:00
Adrien GallouëtandGitHub d01f6274c0 common : respect specified tag, only fallback when tag is empty (#21413) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-04-04 15:08:03 +02:00
Adrien GallouëtandGitHub 6307ec07d3 common : cleanup logs and modernize the progress bar (#21215) 
```
$ build/bin/llama-server -hf unsloth/Qwen3.5-0.8B-GGUF
common_download_file_single_online: HEAD failed, status: 404
no remote preset found, skipping
Downloading mmproj-BF16.gguf ——————————————————————————————————————— 100%
Downloading Qwen3.5-0.8B-Q4_K_M.gguf ——————————————————————————————— 100%
...
```

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-03-31 16:18:00 +02:00
Adrien GallouëtandGitHub 41361c8599 common : move up common_init() and fix Windows UTF-8 logs (#21176) 
The build info is now only for debug, so we avoid the duplicate
with `--version`.

The UTF-8 setup at the beginning is needed to avoid logging
garbage on Windows.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-03-31 12:53:41 +02:00
Adrien GallouëtandGitHub 26dac845cc vendor : update BoringSSL to 0.20260327.0 (#21211) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-03-31 09:21:54 +02:00
Adrien GallouëtandGitHub 3d66da1809 ci : gracefully shut down the server (#21110) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-03-28 14:49:57 +01:00
Adrien GallouëtandGitHub b0f0dd3e51 vendor : update cpp-httplib to 0.40.0 (#21100) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-03-28 08:59:44 +01:00
Adrien GallouëtandGitHub 5c1a7b8355 server : add custom socket options to disable SO_REUSEPORT (#21056) 
* server : add custom socket options to disable SO_REUSEPORT

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Add --reuse-port

    $ strace -e trace=setsockopt,bind build/bin/llama-server -lv 2 --reuse-port
    setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
    setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
    setsockopt(3, SOL_SOCKET, SO_REUSEPORT, [1], 4) = 0
    bind(3, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("127.0.0.1")}, 16) = 0

    $ strace -e trace=setsockopt,bind build/bin/llama-server -lv 2
    setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
    setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
    bind(3, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("127.0.0.1")}, 16) = 0

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Update tools/server/README.md (llama-gen-docs)

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Fix windows

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-03-28 01:12:43 +01:00
Adrien GallouëtandGitHub 287b5b1eab common : add getpwuid fallback for HF cache when HOME is not set (#21035) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-03-26 20:34:23 +01:00
Adrien GallouëtandGitHub 9900b29c3a common : filter out imatrix when finding models (#21023) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-03-26 15:37:18 +01:00
Adrien GallouëtandGitHub 93dfbc1291 common : make LLAMA_CACHE the one cache for everything (#21009) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-03-26 12:04:57 +01:00
Adrien GallouëtandGitHub 3cba8bba18 common : fix split model migration (#21019) 
Sadly the manifest does not list all required files, i honestly thought
it was the case

Without the files listed we don't have the sha256, so if the first file
is valid, and all others have the correct size, then we can assume we
are good and do the migration...

Here my test:

    $ find /home/angt/.cache/llama.cpp
    /home/angt/.cache/llama.cpp
    /home/angt/.cache/llama.cpp/angt_test-split-model-stories260K_stories260K-f32-00002-of-00002.gguf
    /home/angt/.cache/llama.cpp/angt_test-split-model-stories260K_stories260K-f32-00001-of-00002.gguf
    /home/angt/.cache/llama.cpp/angt_test-split-model-stories260K_stories260K-f32-00001-of-00002.gguf.etag
    /home/angt/.cache/llama.cpp/angt_test-split-model-stories260K_stories260K-f32-00002-of-00002.gguf.etag
    /home/angt/.cache/llama.cpp/manifest=angt=test-split-model-stories260K=latest.json

    $ build/bin/llama-server
    ================================================================================
    WARNING: Migrating cache to HuggingFace cache directory
      Old cache: /home/angt/.cache/llama.cpp/
      New cache: /home/angt/.cache/huggingface/hub
    This one-time migration moves models previously downloaded with -hf
    from the legacy llama.cpp cache to the standard HuggingFace cache.
    Models downloaded with --model-url are not affected.
    ================================================================================
    migrate_file: migrated angt_test-split-model-stories260K_stories260K-f32-00001-of-00002.gguf -> /home/angt/.cache/huggingface/hub/models--angt--test-split-model-stories260K/snapshots/68c3ea2061e8c7688455fab07597dde0f4d7f0db/stories260K-f32-00001-of-00002.gguf
    migrate_file: migrated angt_test-split-model-stories260K_stories260K-f32-00002-of-00002.gguf -> /home/angt/.cache/huggingface/hub/models--angt--test-split-model-stories260K/snapshots/68c3ea2061e8c7688455fab07597dde0f4d7f0db/stories260K-f32-00002-of-00002.gguf
    migrate_old_cache_to_hf_cache: migration complete, deleting manifest: /home/angt/.cache/llama.cpp/manifest=angt=test-split-model-stories260K=latest.json

    $ find /home/angt/.cache/llama.cpp /home/angt/.cache/huggingface
    /home/angt/.cache/llama.cpp
    /home/angt/.cache/huggingface
    /home/angt/.cache/huggingface/hub
    /home/angt/.cache/huggingface/hub/models--angt--test-split-model-stories260K
    /home/angt/.cache/huggingface/hub/models--angt--test-split-model-stories260K/blobs
    /home/angt/.cache/huggingface/hub/models--angt--test-split-model-stories260K/blobs/50d019817c2626eb9e8a41f361ff5bfa538757e6f708a3076cd3356354a75694
    /home/angt/.cache/huggingface/hub/models--angt--test-split-model-stories260K/blobs/7b273e1dbfab11dc67dce479deb5923fef27c39cbf56a20b3a928a47b77dab3c
    /home/angt/.cache/huggingface/hub/models--angt--test-split-model-stories260K/refs
    /home/angt/.cache/huggingface/hub/models--angt--test-split-model-stories260K/refs/main
    /home/angt/.cache/huggingface/hub/models--angt--test-split-model-stories260K/snapshots
    /home/angt/.cache/huggingface/hub/models--angt--test-split-model-stories260K/snapshots/68c3ea2061e8c7688455fab07597dde0f4d7f0db
    /home/angt/.cache/huggingface/hub/models--angt--test-split-model-stories260K/snapshots/68c3ea2061e8c7688455fab07597dde0f4d7f0db/stories260K-f32-00002-of-00002.gguf
    /home/angt/.cache/huggingface/hub/models--angt--test-split-model-stories260K/snapshots/68c3ea2061e8c7688455fab07597dde0f4d7f0db/stories260K-f32-00001-of-00002.gguf

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-03-26 12:04:37 +01:00
Adrien GallouëtandGitHub c0159f9c1f common : do not delete old files from the old cache when updating (#21000) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-03-25 22:28:04 +01:00
Adrien GallouëtandGitHub 056b50c319 common : fix verbosity setup (#20989) 
The verbosity threshold was set at the end of common_params_parse_ex(),
after doing many things (like downloading files..)

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-03-25 19:41:01 +01:00
Adrien GallouëtandGitHub f2c72b8f1f common : fix gguf selection in common_list_cached_models (#20996) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-03-25 19:18:06 +01:00
Adrien GallouëtandGitHub 42ebce3beb common : fix get_gguf_split_info (#20946) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-03-24 13:33:14 +01:00
Adrien GallouëtandGitHub c9dc43333f readme : clarify MODEL_ENDPOINT usage (#20941) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-03-24 10:35:07 +01:00
Adrien GallouëtandGitHub 2d2d9c2062 common : add a WARNING for HF cache migration (#20935) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-03-24 09:24:39 +01:00
Adrien GallouëtandGitHub 8c7957ca33 common : add standard Hugging Face cache support (#20775) 
* common : add standard Hugging Face cache support

- Use HF API to find all files
- Migrate all manifests to hugging face cache at startup

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Check with the quant tag

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Cleanup

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Improve error handling and report API errors

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Restore common_cached_model_info and align mmproj filtering

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Prefer main when getting cached ref

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Use cached files when HF API fails

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Use final_path..

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Check all inputs

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-03-24 07:30:33 +01:00
Adrien GallouëtandGitHub 07c6a59b4f vendor : update cpp-httplib to 0.38.0 (#20578) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-03-15 17:30:06 +01:00
Adrien GallouëtandGitHub 0685848bc6 scripts : remove get-wikitext-103.sh (#20543) 
It doesn't work and no one seems to use it.

    $ wget https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-103-raw-v1.zip
    HTTP request sent, awaiting response... 301 Moved Permanently
    Location: unspecified
    ERROR: Redirection (301) without location.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-03-14 11:22:04 +01:00
Adrien GallouëtandGitHub 0024a69b70 scripts : update get-hellaswag.sh and get-winogrande.sh (#20542) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-03-14 11:21:50 +01:00
Adrien GallouëtandGitHub d0b79aaa2f ggml : add native AVX512-FP16 support for F16 operations (#20529) 
The overall benchmark speed remains almost the same because the CPU is
now calculating faster than the RAM can deliver the data. (See perf stat
results below showing 2.7 billion fewer instructions).

Also note that this path will be only enabled for native build or with
custom flags.

now:
```
 Performance counter stats for 'build/bin/llama-bench -m Qwen3-0.6B-f16.gguf -p 512 -n 128':

        189,073.52 msec task-clock                       #   14.658 CPUs utilized
               404      context-switches                 #    2.137 /sec
                19      cpu-migrations                   #    0.100 /sec
           372,390      page-faults                      #    1.970 K/sec
   310,877,195,595      instructions                     #    0.54  insn per cycle
   581,071,530,602      cycles                           #    3.073 GHz
    19,352,107,994      branches                         #  102.352 M/sec
        48,304,438      branch-misses                    #    0.25% of all branches
    84,998,431,152      L1-dcache-loads                  #  449.552 M/sec
    12,186,410,279      L1-dcache-load-misses            #   14.34% of all L1-dcache accesses

      12.899358742 seconds time elapsed

     187.823044000 seconds user
       1.253416000 seconds sys
```

before:
```
 Performance counter stats for 'build/bin/llama-bench -m Qwen3-0.6B-f16.gguf -p 512 -n 128':

        190,594.56 msec task-clock                       #   14.652 CPUs utilized
               436      context-switches                 #    2.288 /sec
                22      cpu-migrations                   #    0.115 /sec
           372,782      page-faults                      #    1.956 K/sec
   313,574,921,966      instructions                     #    0.54  insn per cycle
   586,064,970,425      cycles                           #    3.075 GHz
    19,585,778,563      branches                         #  102.761 M/sec
        48,437,488      branch-misses                    #    0.25% of all branches
    86,219,336,628      L1-dcache-loads                  #  452.370 M/sec
    12,232,085,771      L1-dcache-load-misses            #   14.19% of all L1-dcache accesses

      13.007923164 seconds time elapsed

     189.395316000 seconds user
       1.202612000 seconds sys
```

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-03-14 10:06:14 +01:00
Adrien GallouëtandGitHub 77e20cc107 vendor : update cpp-httplib to 0.37.2 (#20484) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-03-14 06:51:02 +01:00
Adrien GallouëtandGitHub 463b6a963c tools : enable kvu in perplexity for hellaswag, winogrande, multiple-choice (#19954) 
llama-perplexity -hf unsloth/Qwen3-0.6B-GGUF:Q4_K_M -f winogrande-debiased-eval.csv --winogrande

    winogrande_score : tokenizing selected tasks
    winogrande_score : calculating winogrande score over selected tasks.
    split_equal: sequential split is not supported when there are coupled sequences in the input batch (you may need to use the -kvu flag)
    decode: failed to find a memory slot for batch of size 46
    failed to decode the batch, n_batch = 2048, ret = 1
    winogrande_score: llama_decode() failed

same for hellaswag:

    split_equal: sequential split is not supported when there are coupled sequences in the input batch (you may need to use the -kvu flag)
    decode: failed to find a memory slot for batch of size 99
    failed to decode the batch, n_batch = 2048, ret = 1
    hellaswag_score: llama_decode() failed

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-03-13 21:25:57 +01:00
Adrien GallouëtandGitHub b5e1212063 ggml : fix typo gmml (#20512) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-03-13 14:36:13 +01:00
Adrien GallouëtandGitHub 66199c9f03 ggml : use a simple std::thread in AMX without OpenMP (#20074) 
Disabling OpenMP generally provides better inference performance (at
least in my testing) but the loading becomes slightly slower.

Benchmark results for `convert_B_packed_format()`:

Before this commit:

         N      K |  No OpenMP     OpenMP |    Diff |  Speedup
    ------------------------------------------------------------
       512   2880 |    640.9us    263.5us |  -58.9% |    0.41x
      2880   4096 |     2.55ms    261.7us |  -89.8% |    0.10x
    201088   2880 |   256.44ms    21.61ms |  -91.6% |    0.08x
    ------------------------------------------------------------

    Total: 325.43ms vs 31.05ms

After:

         N      K |  No OpenMP     OpenMP |    Diff |  Speedup
    ------------------------------------------------------------
       512   2880 |     1.49ms    263.5us |  -82.3% |    0.18x
      2880   4096 |     1.55ms    261.7us |  -83.1% |    0.17x
    201088   2880 |    24.03ms    21.61ms |  -10.1% |    0.90x
    ------------------------------------------------------------

    Total: 78.97ms vs 31.05ms

Tested with unsloth/gpt-oss-20b-GGUF:Q4_K_M.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-03-04 11:57:09 +01:00
Adrien GallouëtandGitHub ec88c3ceea scripts : improve get-wikitext-2.sh (#19952) 
* scripts : improve get-wikitext-2.sh

Switch to sh, add curl fallback, and avoid redundant downloads

Signed-off-by: Adrien Gallouët <adrien@gallouet.fr>

* fix indent

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <adrien@gallouet.fr>
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-03-02 15:40:49 +01:00
Adrien GallouëtandGitHub 4720819d45 vendor : update cpp-httplib to 0.35.0 (#19969) 
Signed-off-by: Adrien Gallouët <adrien@gallouet.fr>
 2026-02-28 13:53:56 +01:00
Adrien GallouëtandGitHub 4e76d24f28 ggml : fix AMX and add batched support (#19925) 
llama-perplexity -hf ggml-org/Qwen3-0.6B-GGUF:Q4_0 -f wikitext-2-raw/wiki.test.raw -c 2048 -b 2048 --chunks 2

before this commit:

```
perplexity: calculating perplexity over 2 chunks, n_ctx=2048, batch_size=2048, n_seq=1
perplexity: 2.31 seconds per pass - ETA 0.07 minutes
[1]17.3868,[2]22.2199,
Final estimate: PPL = 22.2199 +/- 1.59692

llama_perf_context_print:        load time =     878.56 ms
llama_perf_context_print: prompt eval time =    2037.82 ms /  4096 tokens (    0.50 ms per token,  2009.99 tokens per second)
llama_perf_context_print:        eval time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_perf_context_print:       total time =    6403.17 ms /  4097 tokens
llama_perf_context_print:    graphs reused =          0
llama_memory_breakdown_print: | memory breakdown [MiB] | total   free    self   model   context   compute    unaccounted |
llama_memory_breakdown_print: |   - Host               |                  845 =   318 +     224 +     302                |
llama_memory_breakdown_print: |   - CPU_REPACK         |                  288 =   288 +       0 +       0                |
llama_memory_breakdown_print: |   - AMX                |                   31 =    31 +       0 +       0                |
```

after this commit:

```
perplexity: calculating perplexity over 2 chunks, n_ctx=2048, batch_size=2048, n_seq=1
perplexity: 1.98 seconds per pass - ETA 0.05 minutes
[1]17.2005,[2]21.8220,
Final estimate: PPL = 21.8220 +/- 1.56485

llama_perf_context_print:        load time =     719.23 ms
llama_perf_context_print: prompt eval time =    1676.23 ms /  4096 tokens (    0.41 ms per token,  2443.58 tokens per second)
llama_perf_context_print:        eval time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_perf_context_print:       total time =    4258.74 ms /  4097 tokens
llama_perf_context_print:    graphs reused =          0
llama_memory_breakdown_print: | memory breakdown [MiB] | total   free    self   model   context   compute    unaccounted |
llama_memory_breakdown_print: |   - Host               |                  845 =   318 +     224 +     302                |
llama_memory_breakdown_print: |   - AMX                |                  319 =   319 +       0 +       0                |
```
(no more CPU_REPACK)

after this commit, disabling amx:

```
perplexity: calculating perplexity over 2 chunks, n_ctx=2048, batch_size=2048, n_seq=1
perplexity: 2.34 seconds per pass - ETA 0.07 minutes
[1]17.2005,[2]21.8220,
Final estimate: PPL = 21.8220 +/- 1.56485

llama_perf_context_print:        load time =     841.91 ms
llama_perf_context_print: prompt eval time =    2057.28 ms /  4096 tokens (    0.50 ms per token,  1990.98 tokens per second)
llama_perf_context_print:        eval time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_perf_context_print:       total time =    6454.51 ms /  4097 tokens
llama_perf_context_print:    graphs reused =          0
llama_memory_breakdown_print: | memory breakdown [MiB] | total   free    self   model   context   compute    unaccounted |
llama_memory_breakdown_print: |   - Host               |                  845 =   318 +     224 +     302                |
llama_memory_breakdown_print: |   - CPU_REPACK         |                  319 =   319 +       0 +       0                |
```
=> same perplexity.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-02-26 21:39:11 +01:00
Adrien GallouëtandGitHub b68a83e641 vendor : update cpp-httplib to 0.34.0 (#19830) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-02-23 21:05:48 +01:00
Adrien GallouëtandGitHub 99156f3a5f vendor : update cpp-httplib to 0.33.1 (#19778) 
Signed-off-by: Adrien Gallouët <adrien@gallouet.fr>
 2026-02-21 19:12:31 +01:00
Adrien GallouëtandGitHub a569bda445 common : make small string helpers as inline functions (#19693) 
Also use string_view when it make sense and fix some corner cases.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-02-18 08:03:01 +01:00
Adrien GallouëtandGitHub ae46a61e41 build : link ws2_32 as PUBLIC on Windows (#19666) 
Signed-off-by: Adrien Gallouët <adrien@gallouet.fr>
 2026-02-17 08:37:07 +01:00
Adrien GallouëtandGitHub 65cede7c70 build : cleanup library linking logic (#19665) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-02-17 08:36:45 +01:00
Adrien GallouëtandGitHub 4408494144 build : rework llama_option_depr to handle LLAMA_CURL (#19658) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-02-16 16:06:48 +01:00
Adrien GallouëtandGitHub 9e118b97c4 build : remove LLAMA_HTTPLIB option (#19623) 
This option was introduced as a workaround because cpp-httplib could not
build on visionOS. Since it has been fixed and now compiles on all platforms,
we can remove it and simplify many things.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-02-15 15:38:50 +01:00
Adrien GallouëtandGitHub b7742cf321 ggml : fix GGML_DEBUG with OpenMP (#19599) 
last_graph is only available without OpenMP, but
ggml_graph_compute_thread() is called in both cases.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-02-14 11:22:57 +01:00
Adrien GallouëtandGitHub c7db95f106 scripts : use official split.py for cpp-httplib (#19588) 
* scripts : use official split.py for cpp-httplib

Using the official script is safer and ensures the generated code aligns
with the library's standards.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Catch generic errors

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Allow print()

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Ensure robust cleanup

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-02-14 08:41:16 +01:00
Adrien GallouëtandGitHub 91ea5d67f2 build : fix libtool call in build-xcframework.sh (#19605) 
Run libtool via xcrun like strip and dsymutil, to have proper tool resolution.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-02-14 06:48:37 +01:00
Adrien GallouëtandGitHub b48e80f677 common : update download code (#19573) 
* common : remove legacy .json to .etag migration code

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* common : simplify common_download_file_single_online

This commit also force a redownload if the file exists
but has no .etag file.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-02-13 15:10:46 +01:00
Adrien GallouëtandGitHub 25224c8021 llama : remove deprecated codecvt (#19565) 
Using the same conversion function ensures a consistent matching between
the regex pattern and the text.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-02-13 06:43:53 +01:00
Adrien GallouëtandGitHub 2f5d8f8edc vendor : update BoringSSL to 0.20260211.0 (#19562) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-02-13 06:43:26 +01:00
Adrien GallouëtandGitHub 4b385bfcf8 vendor : update cpp-httplib (#19537) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-02-12 16:11:22 +01:00
Adrien GallouëtandGitHub 4ae1b7517a common : replace deprecated codecvt using parse_utf8_codepoint (#19517) 
Signed-off-by: Adrien Gallouët <adrien@gallouet.fr>
 2026-02-12 07:27:52 +01:00
Adrien GallouëtandGitHub 0c1f39a9ae common : improve download error reporting (#19491) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-02-11 09:27:55 +01:00
Adrien GallouëtandGitHub 5fa1c190d9 rpc : update from common.cpp (#19400) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-02-08 09:06:45 +01:00
Adrien GallouëtandGitHub a4ea7a188f vendor : update BoringSSL to 0.20260204.0 (#19333) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-02-05 09:53:35 +01:00
Adrien GallouëtandGitHub 14be5a39b1 common : improve error message when HTTPS is missing but required (#18987) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-01-21 17:58:38 +01:00