mirror of
https://github.com/ggml-org/llama.cpp.git
synced 2026-09-15 18:13:29 +02:00
b10917
10
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Georgi GerganovandGitHub
|
a2878d30df |
metal : single-source fusion table + fusion debug rework (#28164)
* metal : rework fusion patterns into a single table All fusable op patterns for the Metal backend are now declared once in a fusion table (ggml-metal-fuse.cpp) and consumed by both the graph optimizer (ggml_metal_fuse_max, packing) and the op encoders (ggml_metal_fuse_next, compute). The two phases share the same pattern table plus ggml_can_fuse_subgraph_ext for the structural checks, and differ only in the mode used for the pattern check (STRUCTURAL at optimize time, since tensors are not allocated yet, and FULL at compute time, including Metal buffer placement). This also protects the snake activation (MUL + SIN + SQR + MUL + ADD) from being reordered during graph optimization, which was previously unprotected. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * metal : fix absolute output indices in fusion patterns ggml_can_fuse_subgraph_ext expects the outputs array to contain absolute graph node indices (it indexes cgraph->nodes[outputs[i]]), but the fusion table query was passing a relative index (n_ops - 1). As a result the last node of every pattern was not recognized as an output and was subjected to the elidable use-count check, which failed for essentially all fusions. This silently disabled the norm/MUL fusion and caused a ~5% token-generation regression. Pass the absolute graph index of the last node instead. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * metal : fuse gated_delta_net with cache cpy Add GGML_METAL_FUSE_GDN_CACHE to the fusion table: when the gated_delta_net kernel is followed by a cpy that scatters its recurrent state snapshots into the KV cache, the kernel writes the snapshots straight into the cache buffer and the trailing cpy is elided. The gdn output has other consumers (the attn scores view), so unlike the elision-chain patterns this is not a simple chain: a 'raw' flag on the fusion pattern skips the generic chain/shape and ggml_can_fuse_subgraph_ext checks, making the pattern-specific check callback the sole validator. Packing (ggml_metal_fuse_max) now matches on the same view-transparent node sequence that the compute phase uses, so the gdn + cache cpy group is packed along with any intermediate views and stays adjacent through the reorder. The fused cpy is a view consumer of the gdn (it writes the cache directly), so its mem-range is skipped in the encoder; the skip is restricted to CPY nodes consuming the previous fused node through a view so other fusions are unaffected. Add test_gated_delta_net_cache_fusion and register 5 cases. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * metal : drop is_view_consumer mem-range skip The is_view_consumer skip was carried over from the upstream gated_delta_net cache-fusion draft, but it is not needed: keeping the elided cpy's mem-range in the concurrency tracker only ever adds a (conservative) memory barrier at the fusion point. It can never remove a barrier, so it cannot introduce a race. The worst case is one spurious barrier per gdn+cache-cpy fusion, which is within run-to-run noise on Qwen3.5-0.8B Q8_0. Dropping the check keeps the mem-range loop uniform for all fused groups. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * metal : rename gated_delta_net fused state output args Rename the fused cache-write kernel argument to match the rest of the kargs: state_out_stride -> nb_out (and widen it to uint64_t), and the local buffer id bid_state_out -> bid_out. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * metal : rename raw fusion flag to unsafe raw did not convey that the flag opts a fusion pattern out of the generic elision-chain safety net (ggml_can_fuse_subgraph_ext + chain/shape checks). rename it to 'unsafe' to make explicit that the pattern's check callback is the sole validator and must re-establish the safety guarantees itself. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * metal : tidy fusion pattern checks and table - const-correct ggml_metal_fuse_outputs buffer - annotate unused check-callback parameters - drop a redundant size_t cast - align the ops/table initializers and add blank-line separation Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * metal : add generic fusion stats via ad-hoc proc-address API Add a device-owned fusion context that lets a test tool count how many times each fusion pattern fires and toggle fusion. It is exposed through the ad-hoc ggml_backend_reg_get_proc_address mechanism with generic names so the testing tool is backend-agnostic: - ggml_backend_fusion_stats_init: start collecting fusion stats; when a context is created afterwards it registers the labels/counters and encodes single-threaded (n_cb == 0) so the counters are race-free - ggml_backend_fusion_stats_reset / _get_stats / _set_enabled The context lives on the metal device (not on the last backend context), so counters accumulate across contexts and reads are always consistent. The enable/disable toggle is initialized from GGML_METAL_FUSION_DISABLE and can be overridden by the test through set_enabled. Labels are synthesized from the fuse table via ggml_metal_fuse_label (e.g. "GATED_DELTA_NET+CPY"). Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : add fusion count regression test with per-backend baseline test-fusion runs every dummy model generated by test-llama-archs on a single backend (single-threaded encoding, n_cb == 0) with fusion enabled and disabled, and for each mode (prefill / decode) reports the per-fusion counters and the NMSE between the fused and unfused logits, plus the NMSE against a CPU reference. A fusion pattern that silently stops matching (or fires when it should not) is caught as a regression by comparing the counters against a committed per-backend TSV baseline: - --record writes the golden baseline, --check (default) validates it - the unfused run doubles as a control: its counters must be all-zero - NMSE is skipped when it is NaN or the arch is already broken on the device (e.g. plamo2 on Metal), so the count check is the hard gate - baseline counts depend only on graph structure, not weights (verified stable across weight seeds) - the fusion stats API is resolved through the ad-hoc get_proc_address mechanism with generic names; a backend that does not export it makes the test fail with an error The committed MTL0.tsv baseline covers 110 dummy archs (298 rows). Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : rename fusion api helpers to match stats_init signature Align the test with the ad-hoc fusion stats API: fusion_stats_init no longer takes an enable bool (stats are turned on by calling it), so the proc-address wrappers and typedefs are renamed to the api_* convention. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : rename backend to device in fusion test CLI The fusion test operates on a compute device (e.g. MTL0), not a backend, so rename the --backend argument to --device and the backend_name variable to device_name. Keep "backend" where it refers to the ggml backend interface (the ad-hoc proc-address mechanism). Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : add --model and --help to fusion test --model FILE runs the fusion regression test over a single model file instead of enumerating a --models DIR. --models and --model are mutually exclusive. Also add a --help/-h option that prints the usage. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : use backend base name for fusion baseline output The fusion test is invoked with a specific device name (e.g. MTL0), but its output - the recorded baseline and the header it writes - should be named after the backend base name (e.g. MTL, via ggml_backend_reg_name), since the counters depend on the backend, not on the specific device index. Rename the committed baseline to MTL.tsv. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : run fusion test from ci instead of ctest The fusion test needs Metal and generates a lot of dummy models, so it does not belong in the generic ctest suite. Move it to ci/run.sh as gg_run_test_fusion, gated on GG_BUILD_METAL like gg_run_test_llama_archs_tensor_split: it generates the dummy models with test-llama-archs -o and then validates the fusion counts against the committed baseline. test-fusion.cpp is still built (llama_build) but no longer registered as a ctest. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : align fusion baseline TSV columns Pad the TSV fields to fixed widths so the columns line up regardless of the variable arch and fusion-label lengths, and trim each field on parse so the padded file is still accepted. Regenerate the committed MTL.tsv baseline in the padded format (data unchanged, verified identical modulo padding). Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : widen label column and align fusion TSV header Give the label column more room (28 chars) and fix the column header widths so they match the data rows (moe/mode/label), keeping the header aligned with the values. Regenerate the MTL.tsv baseline in the new format (data unchanged). Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : switch fusion baseline from TSV to CSV Use comma-separated values like the rest of the project, keeping the padded, aligned columns. Split on ',' and trim on parse. Rename the committed baseline to MTL.csv (data unchanged, verified identical modulo padding/separator). Update the ci/run.sh check path accordingly. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * cont : rebase + update MTL stats * tests : avoid graph reallocations for some archs * metal : tidy fusion debugging context and op init - simplify the shared fusion debugging context comments - shorten the ggml_metal_fusion struct comment - align the ggml_metal_fuse struct fields and comments - move the fusion parameter of ggml_metal_op_init right after dev Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : dedup fusion baseline into any mode prefill and decode always produce the same per-graph fusion count, so store a single row per label with mode = "any" and the per-graph count instead of two rows. this halves the baseline size and keeps the check stable. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * ci : move fusion model generation to a separate step the dummy models generated by test-llama-archs are reused by other tests, so generate them once in their own step instead of inside test_fusion. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : bump nmse thold * models : fix plamo2 graph * tests : remove "skip" logic from test-fusion * tests : set qwen3tts dummy vocab to codec head size the dummy qwen3tts model used a vocab of 4096 while the codec head is 3072, so the graph padded the output with -inf which made the NMSE in test-fusion produce NaN. use the exact codec head size instead so the padding is not generated at all. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : regen fusion baseline reflect the plamo2 graph fix, which changed its fusion pattern split (RMS_NORM+MUL 11->10, RMS_NORM+MUL+ADD 3->4; same total). Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * ci : skip dummy model generation on OpenVINO test-llama-archs does not build on the OpenVINO platform, so do not try to generate the dummy models there. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * cont : minor * tests : enable test-llama-archs on windows * cont : disable on windows + workaround * metal : naming nits * test-fusion : add instructions to update baseline * context : fix Kimi-K3 graph reserve * fusion : update MTL * cont : fix naming * metal : rework fusion info storage Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : align fusion info API Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : use opaque fusion handle in ad-hoc API Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * ci : move fusion test to dedicated workflow Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * cont : run only on ggml changes * cont : simplify * fusion : remove multi-output stuff for now * ci : fix typo |
||
|
Daniel HanandGitHub
|
36b1015438 |
qwen4exp: fix seq_cp, block position keying, mtmd input, cuda abort, add tests (#27941)
* qwen4exp: follow up fixes * -kvu NaN collapse fix Assisted-by: Claude * indexer cache ext.x/ext.y restore fix Assisted-by: Claude * kv-cells: rename seq_set to seq_get_all seq_get is already taken by the single-id getter, so the suggested name cannot be overloaded on return type alone. Assisted-by: Claude * memory-hybrid-idx: implement set_input_qsa on the memory class The context held the whole implementation, where the pattern elsewhere is a thin context forwarding to the memory class, as llama_kv_cache_context does for set_input_kq_mask. The body reads no context state, so it moves unchanged and the context keeps a forwarder. Also shortens the seq_get_all comment as suggested. * tests: check that a sequence state survives a save/restore round-trip Saves seq 0, erases it, restores the blob and saves again, requiring the two blobs to match. Compares blobs rather than generated text, which cannot see a field dropped on the way back in. Note this passes on master for qwen4exp, so it does not demonstrate the ext.x/ext.y drop this PR fixes; reaching that needs 2D mrope content. * tests: give the synthetic qwen4exp a PLE so the state test bites has_cell_ext() is n_pos_per_embd() > 1 || ple_n_heads > 0, and the indexer cache sets rope_type = NONE, so without a PLE it serializes no cell ext at all and the round-trip test cannot see a dropped ext.x/ext.y. With one, removing the ext_set restore in state_read_meta fails the test: 198 of 335692 bytes differ, first at offset 282092. Loading such a model needed two fixes: - the row count of per_layer_token_embd came from require_weight(), which a model synthesised from metadata alone has no file to answer. Derive it from the head ranges and prefer the file's padded count where there is one. - the PLE conv history is a row of the recurrent cache, so a PLE on a full attention layer dereferenced a null p_l. Reject it at load time instead. The meta mirror is skipped for qwen4exp. It returned NaN logits before this fixture carried a PLE, which the nmse check passes since a NaN comparison is false, and aborts with one. -sm tensor on real devices works. Assisted-by: Claude * llama: disable -sm tensor for qwen4exp test-llama-archs skipped the tensor split for this arch from inside the test, so the arch still advertised support it does not have. Declare it in llm_arch_supports_sm_tensor instead and drop the test-side exception; the existing llm_arch_supports_sm_tensor branch then does the skipping. Assisted-by: Claude |
||
|
itsnotogerandGitHub
|
2d8d612e4c |
kv-cache : optimize restoring non-contiguous cells (#27991)
* kv cache : batch state restore scatter reads per contiguous run When restoring state into non-contiguous destination cells (e.g. a prompt-cache snapshot into a fragmented ring), state_read_data issued one small copy per KV cell - ~1.4M copies of a few KiB each for a 40k+ token restore, taking 25-63 s on the CUDA backend. The snapshot stores cell rows in cell order, so a maximal run of consecutive destination indices maps to one contiguous block and can be restored with a single copy. Precompute the runs once and use them in all three scatter loops (K, V, transposed V). Byte-identical. The on-device reader copies with a byte cursor when the read and write chunking differs, so the batched reads are safe for it as well. Batching makes equal tensor counts with a different split reachable (save ranges [2,1] vs restore runs [1,2]); the next commit teaches the reader's 1:1 path to fall back to the byte cursor in that case. Verified in a production setup: 1,363,616 copies / 25-63 s -> 224 copies / 221-424 ms for the same restores (42,603 cells, 4 runs). Assisted-by: Claude Code (unsloth/qwen3.8-27b) * context : fall back to the byte cursor when read and write chunking differ the on-device reader copies saved state back with a 1:1 copy by tensor index whenever the write and read sides recorded the same number of tensors, guarded by a per-tensor size assert. equal tensor counts do not imply equal chunking: a state restore may batch its reads per contiguous run of destination cells while the save used per-range reads, so both sides can record two tensors that split the same data differently, and the assert aborts in all builds. compare the per-tensor sizes and only take the 1:1 path when the chunking actually matches, otherwise fall through to the existing byte-cursor copy. both sides enumerate the same logical data in the same order, so the cursor copy is well-defined across tensor boundaries. Assisted-by: Claude Code (unsloth/qwen3.8-27b) * tests : cover state restore scatter reads on host and on-device paths decode the same prefix on two sequences, interleaving the seq 0 cells between the seq 1 cells, so the seq 1 cells are isolated from each other in the kv cache (three cells, two saved ranges). save the seq 1 state, free the interleaved seq 0 cells, and restore: the destination is then non-contiguous (two runs), and the restore-side chunking has the same tensor count as the save-side with a different split, so the scatter path is batched per contiguous run and the on-device reader's byte-cursor fallback is exercised. the restored state is saved again on the host and compared byte for byte with the first save: the blob is serialized in sequence cell order, so the two saves are identical if and only if the scatter restore wrote exactly the same KV content. this documents the byte-identical guarantee of the run-batched scatter reads. one test per io backend: the host (CPU) path and the on-device path. Assisted-by: Claude Code (unsloth/qwen3.8-27b) |
||
|
Georgi GerganovandGitHub
|
4e97ac86eb |
tests : run test-save-load-state across all architectures (#27755)
* tests : run test-save-load-state across all architectures test-save-load-state previously only ran in ctest against a single downloaded model (tinyllamas/stories15M), i.e. only the llama arch. Add a --models DIR mode to test-save-load-state that runs the full save/load suite over every *.gguf in a directory, reporting a per-model PASS/FAIL and exiting non-zero if any model fails, and wire a ctest to run it over all architectures using the existing generate-models fixture (test-llama-archs). The single-model -m mode is preserved (still used by ci/run.sh). Also bump the dummy-model training context in test-llama-archs from 128 to 256 so that the per-sequence context (which is padded up to a multiple of 256) no longer exceeds n_ctx_train and emits the "possible training context overflow" warning. The test is expected to fail until the affected arches are fixed: deepseek4 (host seq-copy), gemma2/gpt-oss/lfm2 (device seq-copy), minimax-01 (state load). It aborts at the first arch that crashes. Assisted-by: pi:llama.cpp/Qwen3.8-27B * tests : match dummy DSA indexer to fused Lightning Indexer kernel The dummy DSA indexer (deepseek32, glm-dsa, ...) used key_length=64 and head_count=1, so the fused Lightning Indexer op's q tensor was shaped [64, 1, ...]. The Metal fused kernel is fixed to DK=128, NH=64, so it rejected the op and the scheduler fell back to CPU, emitting a 'layer assigned to MTL but Lightning Indexer on CPU' warning. Bump key_length to 128 and the DSA head_count to 64 so the fused op runs on the GPU. Assisted-by: pi:llama.cpp/Qwen3.8-27B * tests : add --help and document -o in test-llama-archs Add a --help/-h flag to test-llama-archs and list the existing -o/--out option in the usage text, which was previously missing. Assisted-by: pi:llama.cpp/Qwen3.8-27B * tests : use 64 indexer heads for deepseek4 deepseek4's indexer head count was set to n_head (8), which does not match the fused Lightning Indexer kernel's fixed NH=64, so the fused op fell back to the CPU backend and emitted a device-mismatch warning. Give it the same fixed 64 as the other indexer archs by dropping it from the n_head ternary (only minimax-m3 keeps n_head, since it does not use the fused Lightning Indexer op). Assisted-by: pi:llama.cpp/Qwen3.8-27B * tests : fix dsv4 save-load n_stream mismatch The dsv4 KV cache keeps per-sequence KV/state streams even in unified mode, so its n_stream equals n_seq_max. The test saved the state in the baseline with n_seq_max=1 but loaded it in the seq-copy tests with n_seq_max=2, so state_read threw an n_stream mismatch. Use n_seq_max=2 in the baseline and state-load tests so the save and load agree. Assisted-by: pi:llama.cpp/Qwen3.8-27B * context : relax on-device seq-copy chunk alignment The on-device state seq copy (llama_state_seq_set_data with LLAMA_STATE_SEQ_FLAGS_ON_DEVICE) copied the write-side cpy tensors to the read-side targets 1:1 by index, requiring the writer and reader to emit the same number of chunks in the same order with the same per-chunk sizes. state_write_data chunks per cell-range while state_read_data chunks contiguous-or-per-cell, so the counts diverged for non-contiguous sources (dsv4, SWA) and the copy aborted with "memory buffer mismatch". All state writers and readers enumerate the same logical data in the same order, differing only in chunking. Copy the flat write-side data into the read-side targets with a byte cursor that walks both tensor lists across their boundaries, so the chunking no longer needs to match. Keep the total-size guard; drop the n_tensors equality check. Assisted-by: pi:llama.cpp/Qwen3.8-27B * model : fix dangling hparams ref in minimax-01 LA graph input llm_graph_input_la stored const llama_hparams & hparams, bound to the llm_graph_params temporary in llama_context::process_ubatch. The input object outlives that temporary (it is kept in llm_graph_result::inputs for graph reuse), so set_input() read destroyed stack memory on every graph reuse - test-save-load-state crashed for minimax-01 when the stack region was overwritten (n_layer_all read as 0, abort in llama_hparams::n_head). Store a copy like every other graph input class. Assisted-by: pi:llama.cpp/Qwen3.8-27B * context : handle "worst case" graph and add TODO |
||
|
Georgi GerganovandGitHub
|
7ef790f90a | tests : remove unnecessary sync in test-save-load-state (#26166) | ||
|
helanfxzandGitHub
|
d67c0b4107 | tests: synchronize save-load-state generation (#26056) | ||
|
Aman GuptaandGitHub
|
13f2b28b09 | DeepseekV4: clear cache only for seq rather than full (#25521) | ||
|
Georgi GerganovandGitHub
|
65ef50a0a4 |
tests : refactor test-save-load-state to accept token input (#24073)
* tests : refactor test-save-load-state to accept token input - Default prompt is now empty; when not provided, generate n_batch random tokens (useful for models without a tokenizer) - Tokenization happens once upfront; pass token vector to test functions - generate_tokens prints token IDs instead of decoded pieces - Use llama_model_get_vocab / llama_vocab_n_tokens API - Upgrade log level from LOG_TRC to LOG_INF for visibility Assisted-by: llama.cpp:local pi * cont : use llama_tokens alias |
||
|
|
0b7154066e |
common : fix state save in common_prompt_batch_decode (#23468)
* common : fix state save in common_prompt_batch_decode This commit addresses a bug in common_prompt_batch_decode that affects the session state store/restore in completion.cpp and save-load-state.cpp. The motivation for this is that currently the code is saving n-1 tokens in both the session_tokens and in the KV cache. Then when loading the session tokens, and if the prompt matches, it would replay the last saved token (n-1) into the next position, effectively replaying the same token in the wrong position. The fix is to store all n tokens in session_tokens, while the memory state only reflects n-1 processed tokens as the saving happens before the last token is decoded in common_prompt_batch_decode. I ran both completion.cpp and save-load-state.cpp with a transformer, a recurrent, and a hybrid model. Resolves: https://github.com/ggml-org/llama.cpp/issues/23400 Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com> |
||
|
Georgi GerganovandGitHub
|
40d5358d3c |
tests : move save-load-state from examples to tests (#23336)
* tests : move save-load-state from examples to tests - Move examples/save-load-state/ to tests/test-save-load-state.cpp - Remove subdirectory reference from examples/CMakeLists.txt - Add test to tests/CMakeLists.txt as a model test - Remove CODEOWNERS entry for removed example directory Assisted-by: llama.cpp:local pi * cont : update ci |