mirror of
https://github.com/ggml-org/llama.cpp.git
synced 2026-09-07 16:37:57 +02:00
0cc4m/vulkan-mul-mm-refactor
136
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
49c0dc82b8 |
model : add Tencent Hy 4 (hy_v4) preview architecture support (#28127)
* model: add Tencent Hy 4 (hy_v4) preview architecture support Adds support for the Tencent Hy 4 model (Hugging Face architecture HYV4ForCausalLM, GGUF arch hy_v4): Add HF -> GGUF conversion script (conversion/hy_v4.py) and wire it into the conversion registry Register hy_v4 GGUF constants, arch enum, and writer support Implement the hy-v4 model graph, hparams, vocab and context changes Register the new arch in llama-arch and models registry Extend arch tests to cover hy_v4 Assisted by Claude Opus 5 * Update convert_hf_to_gguf_update.py Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com> * Update conversion/base.py Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com> * convert : move hy_v4 entry to the same place as in convert_hf_to_gguf_update.py * model : apply changes related to n_ff_exp becoming per-layer in Hy4-preview * n_layer_all --------- Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com> Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> |
||
|
Yaniss AmazouzandGitHub
|
c61b98b875 |
model: add NVIDIA Nemotron-3-Puzzle-75B-A9B (NemotronHPuzzle) support (#25444)
* hparams: add per-layer n_ff_exp/n_expert_used arrays with scalar-or-array loading
G1/G2 infrastructure for variable-per-layer expert FFN size and top-k routing
(required for Puzzle-75B which has 5 distinct n_ff_exp values and 7 top-k values
across its 40 MoE layers).
Design: rename scalar members to _impl suffix (following existing convention),
add LLAMA_MAX_LAYERS arrays, add n_ff_exp(il)/n_expert_used(il) accessors with
scalar fallback. No new GGUF keys: reuses existing expert_feed_forward_length and
expert_used_count keys via get_key_or_arr (scalar -> broadcast, array -> per-layer).
- llama-hparams.h: n_ff_exp -> n_ff_exp_impl, n_expert_used -> n_expert_used_impl;
add n_ff_exp_arr / n_expert_used_arr arrays; add per-layer accessor declarations.
- llama-hparams.cpp: implement n_ff_exp(il) and n_expert_used(il); out-of-range
il returns impl safely (shared code, no abort).
- llama-model.cpp: central n_expert_used load changed to get_key_or_arr; derive
impl as max-of-array for validations and backward compat; zero both new arrays;
HunyuanVL override also zeroes n_expert_used_arr.
- llama-graph.cpp: aggregation loop in build_moe_ffn uses hparams.n_expert_used(il)
so per-layer top-k bounds the ggml_view loop correctly.
- All other files: mechanical rename hparams.n_{ff_exp,expert_used} -> *_impl.
Scalar arches are unaffected (broadcast fills all array slots with the single value).
(cherry picked from commit
|
||
|
Xuan-Son NguyenandGitHub
|
b19cbe925b | convert: prevent ndarray conversion in LazyChunkedTensor (#27869) | ||
|
|
ca3d5a3e10 |
model: add DSpark support for Nemotron3.5 (#27804)
* model: add DSpark support for Nemotron3.5 * Update src/models/dflash.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> Co-authored-by: Xuan Son Nguyen <son@huggingface.co> |
||
|
|
6c84c7d5d8 |
model: add Qwen3.8-Flash-Next (qwen4exp) (#27742)
* gguf: add qwen4exp (Qwen3.8-Flash-Next) arch and converter
Adds the GGUF-side plumbing for HF model_type qwen4_exp:
- MODEL_ARCH.QWEN4EXP plus tensors for the low-rank hyper-connection
variant (hc_*_norm/down/up/inject) and the PLE n-gram hash embeddings.
The DeepSeek-V4 hc_*_fn/base/scale tensors are a different
parameterisation, so these are separate entries rather than reuse.
- Reuses the existing indexer, per_layer_token_embd, SSM and
compress_ratios keys unchanged.
- conversion/qwen4exp.py inherits the Qwen3.5 linear-attention V-head
reorder and interleaved mrope, concatenates the 128 PLE embedding
shards, and splits index_qk_proj into separate indexer q/k tensors.
The PLE hash multipliers reach ~2.4e13. prepare_tensors() casts every
non-float dtype to float32 before modify_tensors() runs, and GGUF array
writes infer INT32 from Python ints, so both paths are bypassed: the
constants are read from the pre-cast lazy tensors and written as
explicit UINT64 arrays.
Additive only; no existing arch changes behaviour.
* llama: load qwen4exp (Qwen3.8-Flash-Next) hparams and tensors
Adds LLM_ARCH_QWEN4EXP with its hparams and tensor loading. The graph
comes in the next commit; this makes the model load and report correct
metadata.
- hyper-connections set n_embd_out_impl = hc_count * n_embd, so the
residual stream is 4x wide and there is no output_norm: the final
mixer's hc_norm is the last norm in the model.
- registered as hybrid and given the same recurrent/attention memory
filters as Qwen3-Next and Qwen3.5.
- reuses the existing indexer, per_layer_token_embd, SSM and
compress_ratios keys as-is.
- the PLE table row count is read back from the file rather than
recomputing the vocab padding rule.
llama-model-loader gains UINT64 array support. That branch previously
threw, so no existing caller changes behaviour; it is needed because the
PLE hash multipliers do not fit in int32.
* qwen4exp: shorten comments
* llama: qwen4exp text graph with hyper-connections, GDN and MoE
Implements the decode graph for Qwen3.8-Flash-Next: the hyper-connection
residual stream, gated delta net layers, the MoE block with its gated shared
expert, and dense full attention. The QSA indexer and the PLE n-gram embedding
are not wired up yet and land in later commits.
Hyper-connections are implemented here rather than shared with deepseek4.cpp.
The two formulations agree on the [n_embd, hc, n_tokens] layout and little
else: DeepSeek-V4 mixes with a full-rank projection and Sinkhorn-normalises
it, whereas this model uses a low-rank down/silu/up sigmoid gate and collapses
by a plain mean. Only the ~10 line stream mean is genuinely common, so sharing
would mean touching DSV4's hot path and its three fused CUDA ops to reuse very
little. What is reused is the substantive part: the LLM_KV_HYPER_CONNECTION_*
keys, the n_embd_out_impl wide-residual support already in the loader, and the
layout convention.
Also allows a checkpoint to carry no PLE layers at all, which makes it
possible to bring the graph up and validate it in stages.
Validated against vLLM, the only working reference implementation. On a
scaled-down model with an init scale large enough to give non-uniform logits,
agreement with vLLM sits at the numerical noise floor: llama.cpp f32 against
its own bf16 gives 84.3% top-1 agreement over 255 positions, and this graph
against vLLM gives 85.1%. The comparison was calibrated by seeding three
deliberate bugs (silu instead of sigmoid on the delta net gate, dropping the
1/hc scale in the mix, dropping the 2x in the combine); each drops top-1 to
between 0% and 11%, an order of magnitude below the floor.
* llama: qwen4exp PLE n-gram hash embedding
Adds the per-layer embedding: a custom I32 graph input hashes each token with
its ngram_size-1 predecessors host-side and the result is a plain row gather
over the shared table, the same shape gemma3n's per-layer embedding uses. The
hash has to run on the host because the splitmix64-derived multipliers reach
2^45, so the products need 64-bit integers and an xor, neither of which ggml
has.
Predecessors that fall outside the ubatch come from a small per-sequence
history on the model, mirroring the per-request ngram_context the reference
carries. It is only trusted when contiguous with the incoming position, so a
fresh prompt or a rewound cache falls back to EOS padding rather than hashing
against stale tokens.
The depthwise conv is written out as a sum of shifted, per-channel-scaled
copies rather than through ggml_conv_1d_dw, which carries a correctness
warning upstream.
Verified two ways. The row indices match a transcription of the reference's
tensor formulation exactly, 1024 of 1024 rows, including sequences with EOS
tokens sprinkled through them to exercise the segment reset. Separately, with
PLE placed on layer 0 so its input is just the token embedding, ple_embd and
ple_gated_value match a PyTorch computation from the same checkpoint to every
printed digit.
End to end over 1023 scored positions the port sits the same distance from
vLLM with PLE as without it, 6.3 points of top-1 against 6.0, so PLE costs no
accuracy relative to the rest of the model. That common offset is vLLM's bf16
activations, which cannot be removed: its QSA kernel refuses float32.
Two bugs found along the way, both caught by the row-index check. The history
was read and updated in the same pass, so a token early in a ubatch could pick
up an earlier token of that same ubatch as prior context; it is now snapshotted
first. And an EOS token was cutting its own context, where the reference takes
the last EOS strictly before the position, so a boundary only hides tokens from
the positions after it.
Known gap: the conv carries no state across ubatches, so it is exact only for a
prefill that starts at position 0. Chunked prefill and decode need the conv
state wired into the recurrent memory, and the conv branch itself is still
numerically unverified because the fixture zeroes its weights.
* llama: carry the qwen4exp PLE conv state across ubatches
The PLE depthwise conv was zero-padding on the left, which is only right for a
prefill that starts at position 0. Decode and chunked prefill saw a truncated
history for the first (kernel-1)*ngram_size positions of every ubatch.
The PLE module sits on a layer that is also a delta-net layer, so both need a
conv history in the same recurrent row. Rather than plumb a per-layer state
size through build_rs and build_conv_state, the row is widened once and each
convolution addresses its own slice through a local helper. n_embd_r() gains
the extra span, which is zero for every other architecture because it is
derived from ple_n_heads.
Verified by feeding the same 1024 token sequence in chunks instead of one
shot: at 64 tokens per decode the logits are bit-identical to the single-shot
run, 1023 of 1023 top-1 and a maximum logprob deviation of exactly zero. At
one token per decode they differ slightly, but the no-PLE model differs more
under the same test (94.6% against 97.1%), so that is the usual gemv-versus-
gemm accumulation difference and not the state.
The conv branch is also no longer unverified. With non-zero conv weights the
port sits 6.3 points of top-1 below the numerical floor, the same distance as
with the weights zeroed and as the model with no PLE at all, so the branch
adds no error of its own.
test-llama-archs passes every existing architecture at 0.00e+00, including the
delta-net models that share this code path.
* llama: fix the qwen4exp PLE conv state and unblock test-llama-archs
build_rs writes into the state tensor in place, zeroing one row and copying the
carried-over states, so calling it twice for the same layer let the second call
clobber the first write-back. The PLE layer is also a delta-net layer, so that
is exactly what happened: both convolutions gathered the same row. They now
share a single gather per layer.
The earlier claim that the conv state was carried correctly was tested on a
fixture whose conv weights are zero, where the branch contributes nothing and
chunking matches trivially. Re-running with non-zero conv weights showed the
divergence, growing with the number of ubatch boundaries: 97.1% top-1 at one
boundary down to 90.2% at seven. With the shared gather it is bit-identical to
the single-shot run at every chunk size tried, 512, 128 and 64, with a maximum
logprob deviation of exactly zero over 1023 positions. The delta-net-only model
stays bit-identical too, so nothing regressed there.
Also derive the delta-net conv channel count the way load_arch_tensors sizes
wqkv instead of from ssm_d_inner. The two agree for this model, but n_embd_r()
only bounds the row and the convolution has to match the tensor feeding it.
test-llama-archs previously aborted on this architecture and took every later
architecture with it. qwen4exp is marked MoE-only, given the hyper-connection
keys and an ssm_d_inner consistent with its tensor derivation, and skipped for
now: the hyper-connection keys written by get_gguf_ctx are not reaching the
synthesised file, which needs a separate look. The suite completes again, 124
architectures at 0.00e+00.
* llama: optional indexer key cache in llama_memory_hybrid
Groundwork for qwen4exp's QSA sparse attention. Its indexer needs a per-token
key history for the full-attention layers, but a hybrid model cannot use
llama_kv_cache_dsa: that class derives from llama_memory_i rather than
llama_kv_cache, and llama_memory_hybrid constructs its attention cache
directly. No existing architecture pairs recurrent state with a sparse
indexer, so there was nothing to reuse wholesale.
llama_memory_hybrid therefore gains a third, optional cache, shaped the same
way llama_kv_cache_dsa shapes its lightning-indexer cache: a copy of hparams
with n_head_kv forced to 1 and n_embd_head_k_full set to indexer_head_size.
It is built only when a filter_idx callback is passed, which defaults to
nullptr, so every existing architecture gets exactly what it got before. The
per-sequence operations and the batch preparation forward to it under a null
check, matching how the DSA cache prepares its two caches over the same
ubatches.
test-llama-archs passes all 124 architectures at 0.00e+00, including the 12 in
the hybrid family that share this code. The qwen4exp fixtures are unchanged:
same logits against vLLM, and chunked evaluation still bit-identical to
single-shot.
* llama: QSA sparse attention for qwen4exp
The full-attention layers of this model do not attend to everything. An
indexer scores one mean-pooled key per block of compress_ratio tokens and
keeps a budget of the best blocks, plus the tail of tokens that do not yet
form a complete block. Below indexer_top_k + compress_ratio - 1 cached
tokens every block fits in the budget, so the result is exactly dense.
What is reused rather than rebuilt:
- the mask machinery. build_attn's DSA overload already turns a list of
token indices into a KQ mask via ggml_set_rows, so that block is lifted
out verbatim into build_attn_mask_top_k and shared with a new overload
on llm_graph_input_attn_kv. DSA's node sequence is unchanged; the new
overload exists because llama_kv_cache_dsa assumes MLA and cannot be
dropped into a hybrid model.
- the indexer key cache, which is the optional third cache added to
llama_memory_hybrid in the previous commit. It holds raw keys, because
pooling happens before the norm and the rotation.
The graph expands block scores rather than block indices: giving every
token of a block its block's score needs only a gather, where expanding
indices would need an integer multiply-add that ggml has no op for. Since
the budget is a whole number of blocks and a block's members tie exactly,
the cut still lands on a block boundary.
Everything that depends on cache layout is computed host-side in
set_input_qsa. Blocks are cuts of the position line rather than of the cell
array, so nothing assumes the cache is contiguous.
Measured on the tiny fixture against vLLM, comparing the selected token
indices directly rather than the logits:
below the budget selection identical, and 1024-token logits are
bit-identical to the pre-QSA dense path
above the budget mean jaccard 0.975
The direct index comparison is what made this correct. The reference
rectifies each head's dot product before summing over heads, which an
earlier reading of it had missed; on logits alone the resulting port looked
fine, because on a randomly initialised fixture the known-correct dense
path already disagrees with vLLM by more than the bug did. Comparing the
indices showed 0.794, and fixing the ReLU moved it to 0.975.
* llama: give the qwen4exp indexer cache the attention cache's slots
The indexer cache found its own slots, independently of the attention
cache. Both are the same size and see the same ubatches, so in a
straight-through prefill they agree, which is why every fixture and every
single-shot parity run passed. They drift once the context is being
rewritten between turns, and then the QSA top-k indices, which are applied
against the attention mask, point at the wrong cells.
The seven-turn chat test caught it on the third turn: llama-server aborted
on the assertion that the two caches report the same n_kv.
The cache is a side buffer addressed by the attention cache's cells, so it
now takes that cache's slot layout instead of computing one. Applying that
layout also marks its cells identically, so the two agree cell for cell by
construction rather than by coincidence, and the assertion can no longer
fire.
Inert where the caches already agreed: test-llama-archs green at 126 archs
and 0.00e+00, and the 4096-token tiny fixture is unchanged at max logit
delta 0.0.
* tests: record what the qwen4exp arch-test skip actually observes
The old note guessed that the hyper-connection keys never reach the file.
They do: dumping the gguf_context handed to llama_model_init_from_user
shows both among its 67 KVs, and the loader still reports one missing.
* tests: cover qwen4exp in test-llama-archs
The arch was skipped with a note guessing that the hyper-connection keys
never reached the synthesised file. They did. The suite builds a model, then
saves and reloads it, and llama_model_saver did not re-emit those keys, so
the failure was in the roundtrip leg rather than the first load. Three gaps,
all in shared code and all additive:
- add_kv_from_model wrote no hyper-connection, compress-ratio or PLE keys.
The PLE group only means anything whole, so it is written or omitted
together; the rest follow the file's existing style of writing every key
unconditionally, since an architecture that does not read one is
unaffected by a zero.
- the saver had no uint64 path at all, which the PLE hash constants need.
- add_tensors_from_model enumerates model-level tensors by hand and was
missing per_layer_tok_embd and the three final-mixer tensors.
Two smaller fixes on the qwen4exp side, both found by running the test:
- build_qsa_top_k divided by the compression ratio before asserting it was
non-zero, so a file without the key crashed instead of reporting.
- a layer with no compression ratio now falls back to dense attention,
which is what the model computes below the budget anyway. The test then
has to write a ratio to reach QSA at all, and an indexer key length no
narrower than n_rot, since the indexer ropes with the main attention's
rotary width.
Full suite: 126 archs, qwen4exp at 0.00e+00 with roundtrip OK. The tiny
fixture is unchanged, max logit delta 0.0 against the pre-QSA dense run.
* convert: stream the qwen4exp PLE table instead of concatenating it
The n-gram table arrives as 128 shards that were held in a dict and then
torch.cat-ed, so the peak was the shards plus the concatenation: around
300 GB of RSS on the real checkpoint, which rules out machines that could
otherwise convert this model.
Each shard is now written straight into a memory-mapped file at its final
row offset and dropped, so the resident set is one shard and the rest is
the page cache's problem. The temporary file sits beside the output and is
removed once the write finishes, including on failure.
Shards other than the last must be uniform for direct placement, which is
asserted rather than assumed, and a shard arriving before the stride is
known is held instead of misplaced.
Verified on the tiny fixture: the resulting GGUF is byte-identical to the
one the concatenating path produced (md5 2d274efac91ad1e9a6007efb0687e597).
* quantize: fall back to F16 for 32-block types with an odd ncols
tensor_type_fallback demotes a tensor whose ncols is not a multiple of the
target's block size, but its switch only enumerates the 256-block types. A
target that is already a 32-block type (iq4_nl, q4_0, q5_0, q8_0, ...) falls
into default: and throws, even though the function already knows how to answer
that case: the ncols check right below the switch resolves an unrepresentable
shape to F16.
Route those types into that check instead of throwing. Only paths that abort
today change, so no quantization that currently succeeds is affected.
Found on a 4-wide depthwise conv kernel. llama-quantize reported nothing but
"failed to quantize model from ...", with no tensor name and no exception text,
which made a quant recipe that had simply not pinned the tensor look like a
corrupt model. It now names the tensor and continues.
* quantize: let --tensor-type name per_layer_token_embd
per_layer_token_embd shares the TOKEN_EMBD category with token_embd.weight, so
--token-embedding-type is returned for it before any --tensor-type pattern is
consulted, and there is no way to give it a tier of its own.
That grouping is fine as a default and stays the default. It is a poor fit for
the size, though: on qwen4exp the table is 97.7 GiB of a 337.6 GiB BF16 file and
about 46% of a 4-bit one, roughly eighty times token_embd.weight, and it is
read by ggml_get_rows rather than a matmul so no imatrix ever covers it.
Allow an explicit --tensor-type pattern to name it, and only it. Nothing
changes unless such a pattern is passed, and token_embd.weight keeps the old
precedence in either case.
Measured on Qwen3.8-Flash-Next, Q4_K_M with an imatrix: the table lands at q8_0
(51.9 GiB, 113.5 GiB total) by following --token-embedding-type, and pinning it
q4_1 gives 30.5 GiB for 92.1 GiB total, 19% off the file.
* quantize: size the output buffer exactly instead of nelements * 4
The per-tensor output buffer was sized `nelements * 4`, described as an upper
bound. It is a very loose one: the output is at most 2 bytes per element
(f16/bf16) and usually well under 1.1 (q8_0 and below), so between 2x and 4x of
it is never touched. The exact size is already known here, since it is what the
quantization loop writes, what new_size sums to, and what the GGUF metadata is
asserted against a few lines later.
On a model whose largest tensor is a few GB none of this matters. On
Qwen3.8-Flash-Next it does: per_layer_token_embd is 51.2 G elements, so the
buffer was 205 GB where 54 GB is needed at q8_0 and 32 GB at q4_1.
Measured on that model, VmHWM of a live llama-quantize was 485 GB per process.
Three of them fit in 2 TB and five did not, which is what an OOM-killed quant
ladder looks like. This removes about 150 GB of that.
Byte-identical output, verified against the same binary built at the parent
commit: q4_K, q8_0, q5_K, q6_K and IQ4_XS, over BF16 and F32 sources, with and
without a PLE table present. Six cases, six matching md5s.
* qwen4exp: hash the image placeholder for multimodal batches
The PLE row indices are computed host-side from ubatch->token, and set_input
returned early when that was null. A multimodal ubatch is exactly that case:
the mtmd layer consumes the image placeholder ids and hands llama_decode
embeddings instead. The early return left the I32 index tensor uninitialised,
so ggml_get_rows indexed a 320 M row table with whatever the buffer happened to
contain, and aborted:
GGML_ASSERT(i01 >= 0 && i01 < ne01) failed
ggml_compute_forward_get_rows
mtmd_helper_decode_image_chunk -> llama_decode
Every image request crashed. Nothing caught it because the vision work had only
ever been verified by converting an mmproj, never by running one.
The reference computes the hash over input_ids, where those positions still
hold the image placeholder, so carry that id through as qwen4exp.ple.image_token_id
and hash it. The key is optional: a file converted before it existed falls back
to the PLE EOS token, which is defined and treats the image as a segment
boundary rather than crashing.
Verified end to end with llama-mtmd-cli, a Q4_K_M base and the F16 mmproj, on a
generated image with known content. The model names the red circle, the blue
square, the inverted green triangle and reads "UNSLOTH 42", each with the right
position.
* qwen4exp: support a non-unified KV cache in QSA
set_input_qsa asserted n_stream == 1, so llama-server could not serve this
model with more than one slot unless -kvu was passed. With a non-unified
cache each sequence owns its own cells, and a cell index means a different
token in each stream, so a single shared mapping is wrong.
- cell_blk, blk_cells and bias gain a stream dimension. At n_stream == 1
these collapse to the shapes they had, so the unified path is unchanged.
- Scoring is now batched over streams. ggml_mul_mat matches ne[2] on both
operands, so stream s's queries only ever meet stream s's blocks; without
this sequences would score against each other's context.
- set_input_qsa loops per stream and resolves cells through
v_cells[seq_to_stream[seq_id]], following set_input_kq_mask_impl, instead
of hardcoding v_cells[0].
- llama_kv_cache_context::get_n_stream() is added, mirroring the ns that
get_k and get_v already derive from the slot info.
build_attn_mask_top_k needed no change: it already expects
[n_top_k, n_batch, 1, n_stream], so the top-k result is reshaped to meet it.
set_input_qsa has exactly one caller, so the blast radius is qwen4exp only.
Validation, UD-Q4_K_XL on one B200:
- unified cache unchanged within noise: 1802.9/68.85 -> 1807.2/69.11 t/s at
batch 1, 2262.5/192.43 -> 2270.1/193.75 at batch 4.
- non-unified now runs at npl 1, 4, 16 where it previously aborted, and is
22% faster than the -kvu workaround at batch 16 (1205 vs 984 t/s total),
since per-stream cells avoid the cross-sequence masking a unified cache
pays for.
- no cross-stream contamination: four concurrent sequences each carrying a
distinct secret all recall their own and no other, on both cache modes.
- test-llama-archs green on qwen4exp, deepseek2, gemma3n, qwen3next, llama.
Note on testing: comparing concurrent output against solo output exactly is
not a valid check. It failed 0/4 with no bug present, and the unified-cache
control failed the same way, because batch composition changes the
floating-point reduction order and near-tied tokens flip. The contamination
test above is what the exit code gates on.
* llama: keep the qwen4exp top-k attention mask arch-local
The QSA graph needed a build_attn that attends only to the cells named by a
top_k tensor, and the first version got it by adding a llm_graph_input_attn_kv
overload to llm_graph_context and factoring the mask construction out of the
existing MLA sparse path into a shared build_attn_mask_top_k.
That put a new arch on the shared attention path and made the deepseek32 and
glm-dsa attention build depend on a helper introduced for qwen4exp. Build the
mask in src/models/qwen4exp.cpp instead and leave llama-graph.{h,cpp} exactly as
they were: the MLA path keeps its own copy of the same node sequence.
The nodes emitted are unchanged, so this is bit-identical.
* llama: hold the qwen4exp indexer cache in a new llama_memory_hybrid_idx
The indexer key cache was added by extending llama_memory_hybrid with an
optional third cache, and the host-side cell/block mapping that drives QSA was
added as set_input_qsa on llama_kv_cache. Both are shared classes that every
hybrid and every attention model goes through.
Move both into a new memory type, llama_memory_hybrid_idx, following
llama_kv_cache_msa: the indexer cache and the pos<->cell translation live with
the sparse-attention memory rather than in the classes that serve every other
architecture. llama-kv-cache.{h,cpp} and llama-memory-hybrid.{h,cpp} are
restored to their unmodified state.
init_batch is repeated from llama_memory_hybrid because the indexer cache has to
be handed the attention cache's slot infos, and those are not reachable through
the context the base returns. Allocating them separately lets the two caches
drift, which is what pointed QSA's top-k at the wrong cells before.
The context derives from llama_memory_hybrid_context so build_inp_mem_hybrid
keeps working unchanged, and get_n_stream is computed from the slot infos
exactly as llama_kv_cache_context did.
Behaviour is unchanged: logits over an 8192-token sequence are bit-identical to
the previous implementation, sparse and dense alike.
* llama: save and restore the qwen4exp indexer KV cache
llama_memory_hybrid_idx forwarded clear, seq_rm, seq_cp, seq_keep, seq_add and
seq_div to the indexer cache but not state_write / state_read, so a saved
session dropped the indexer keys and a restored one selected QSA top-k against
an empty cache. The effect is invisible until the context passes
indexer_top_k + compress_ratio - 1 cells, because QSA is exactly dense below
that and the indexer contents cannot change the result.
The indexer section is written last rather than next to the attention cache it
mirrors. As a suffix, a reader that does not expect it stops early and the
trailing bytes are caught by the size check in state_load_file; placed between
the attention and recurrent sections it would instead be parsed as recurrent
state, which can succeed and restore silent garbage. It follows the same
LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY gate as the attention cache, since a partial
checkpoint deliberately skips the token-level attention caches.
The indexer restores its own cells instead of taking the attention cache's
restored slots. The two caches share size, padding and every sequence
operation, and init_batch hands the indexer the attention cache's slot infos,
so both state_read_meta calls run find_slot over identical occupancy and land
on identical cells.
The overrides live on llama_memory_hybrid_idx, the only memory type that owns
an indexer cache, so llama_memory_hybrid and every architecture that uses it
write and read exactly the bytes they did before.
The session and sequence state versions are bumped because the qwen4exp state
layout changed. The session path already rejects a short read via its size
check, but llama_state_seq_load_file accepts one silently, so only the version
check stops a pre-fix blob from being half-restored by a fixed build.
(cherry picked from commit 2721542354f8e158c3217625f4e2e7b83e51e3fe)
* llama: make the qwen4exp PLE n-gram history per context and serialise it
The PLE hash of a token mixes in the ple_ngram_size - 1 tokens before it, which
a decode ubatch does not carry, so they were remembered in a map on
llama_model_qwen4exp. That is the wrong owner twice over.
A llama_model is shared by every context that loads it, and the map was keyed
only by llama_seq_id, so two contexts running the same sequence id - two server
instances on one model, or a draft/target pair - overwrote each other's window.
The next_pos guard turned that into EOS padding instead of a crash, so it
degraded quality silently.
The map was also in no state blob: grep found ple_hist in neither
llama-kv-cache.cpp nor llama-memory-*.cpp nor llama-context.cpp. A restored
context therefore failed the next_pos check on its first ubatch and hashed the
first tokens after the restore against EOS padding. This is why a session blob
round-tripped byte for byte while the restored context computed different
logits: the state was never in the bytes.
It moves to llama_memory_hybrid_idx, which is per context, is the memory type
qwen4exp always builds, and already does the per-sequence bookkeeping this
needs. Every sequence operation now carries the window with it:
seq_rm a rewind (p1 < 0) truncates the window to the surviving prefix and
moves next_pos to p0, so a rollback keeps exact context; a hole
punched in the middle leaves the window non-contiguous, so it is
dropped
seq_cp the destination inherits the source's window, truncated to the
copied position range - a copied sequence continues with the same
n-grams the source would have used
seq_keep every other sequence's window is dropped, like its cells
seq_add a shift that moves the whole window keeps it and moves next_pos with
it, which is the context-shift case; one that cuts through it drops
it
seq_div positions stop being consecutive, so an overlapping window is
dropped
clear everything is dropped
Dropping means next_pos = -1, which set_input turns into full EOS padding: the
same thing a fresh sequence gets, and the same thing this code did before it
followed the sequence operations at all, so no case is worse than before.
The state payload is a self-delimiting list, u32 count then per entry
{ i32 seq_id, i32 next_pos, u32 n_toks, i32 toks[n_toks] }, so a whole-context
save and a single-sequence save share one format and a single-sequence restore
can retarget the window at its destination seq_id. It is written after the
indexer section, last, for the same reason that one is: as a pure suffix an
older reader stops early instead of parsing these bytes as something else.
Unlike the indexer section it is not under LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY.
The window is recurrent state - it is the input the PLE convolution's own
recurrent state is derived from - and the recurrent cache beside it is written
for partial checkpoints too. Gating it would leave the server's speculative
decoding checkpoints restoring the conv state without the window that produced
it.
No further version bump: LLAMA_SESSION_VERSION 10 and LLAMA_STATE_SEQ_VERSION 3
were introduced for the indexer section in the same unreleased series, and both
changes are qwen4exp-only additions to the same blob layout.
Also fixes the padding of a short window. set_input pads a window shorter than
ngram_size - 1 up to that length, but prev() indexes the snapshot with the most
recent token last, and resize() pads at the back, so the filler EOS landed where
the immediately preceding token belongs. It now pads at the front. A window is
short at a sequence start after a one-token prefill, and after a seq_rm rewind,
which the new bookkeeping makes common.
Every architecture other than qwen4exp builds llama_memory_hybrid rather than
llama_memory_hybrid_idx, has no PLE table and never asks for a history, so
nothing about its graph, its sequence operations or its state bytes changes.
(cherry picked from commit de170364c052c68fcf63285cc0028095edb9f23c)
* qwen4exp: tidy comments and simplify image token read
Rewrite the comments this series adds to the AGENTS.md rules: one or two lines,
no prose hard-wrapped mid-sentence, no narrative or history, and no comment that
only restates the code. Net 146 fewer comment lines, no code change.
Correct the PLE image comment: mtmd does not consume the placeholder ids. An
image is decoded as an embeddings-only batch, so ubatch->token is null and the
per-position ids never exist here. gemma3n and gemma4 hit the same case and
stand in row 0 of per_layer_token_embd; qwen4exp stands in the configured image
token id instead.
Read image_token_id straight from self.hparams in the converter. base.py merges
text_config into the root of hparams, and the key sits at the root of
config.json, so the config.json re-read was redundant.
(cherry picked from commit 205840c12169057da3e8d2f65ec4ceec3e18b980)
* qwen4exp: support a quantized KV cache in the QSA attention path
(cherry picked from commit 4c30574f81dc1115d08078c47b6cf8c789c0a842)
* llama: give qwen4exp a large-graph node budget
(cherry picked from commit 37c8c194e6a30e4c46ac29bee3fb264f091596ef)
* qwen4exp: drop an unused variable that breaks -Werror builds
(cherry picked from commit 528d032b51fa3cf935ed3ef6e0fb1c7401df53b5)
* quantize: dequantize and quantize large tensors in row bands
f32_conv_buf held the whole dequantized tensor, which is 204.8 GB for
per_layer_token_embd alone and dies with std::bad_alloc long before the
work buffer is reached. Dequantize and quantize in bands of whole rows
instead, capping the f32 staging at 1 GiB per band.
Rows are independent and the imatrix is indexed by column, so band
boundaries cannot change any output byte. Bands nest inside the existing
per-expert loop so each expert slice keeps its own imatrix, and a band is
kept to at least one quantization chunk per worker thread so the existing
multithreading still has work. F32 sources still stage nothing and are
banded by pointer arithmetic into the tensor.
llama_tensor_dequantize_impl now takes a first element offset; the single
caller is updated.
(cherry picked from commit 658c22549613555dbce57a772be4de8509eba3ee)
* llama: segment the qwen4exp fused QKV for tensor split
qwen4exp was missing from the gated delta net branch of get_split_segments,
so its attn_qkv.weight, shaped {n_embd, 2*key_dim + value_dim}, fell through
to the generic fused QKV rule and tripped
GGML_ASSERT(tensor->ne[axis] == n_embd + 2*n_embd_gqa) while loading with
--split-mode tensor. --split-mode layer was unaffected.
qwen4exp broadcasts K to the V heads by tiling, k_conv is grown with a plain
ggml_repeat_4d over the head axis so that v head j pairs with k head
j % n_k_heads. That is the Qwen 3.5 pattern, not the repeat interleave that
Qwen 3 Next builds explicitly, so qwen4exp takes the else branch and its V is
segmented on the scale of K.
Reported by benklop.
(cherry picked from commit 353d753f595dc81634ae6130188b31f06018f5ae)
* llama: fix the qwen4exp PLE history seq_rm(-1) iterator invalidation and the fatal-warning build
ple_hist_rm recursed over ple_hist with a range-based for and the recursive call
erases the entry it is iterating when the whole sequence is removed (p0 <= 0,
p1 < 0), so the loop then increments an invalidated iterator. It is unreachable
today only because llama_memory_recurrent::seq_rm rejects seq_id < 0 before
llama_memory_hybrid_idx::seq_rm reaches the history, which is a guard in another
class. Advance past the entry before recursing.
Two smaller things in the same area:
- the n_toks sanity bound in ple_hist_state_read was the literal 64, which is
the value of LLAMA_MAX_PLE_HEADS, not of the quantity being checked. The
window is at most ple_ngram_size - 1 tokens, so the bound is
LLAMA_MAX_PLE_NGRAM - 1, eight times tighter.
- build_conv_state_at left mem_size unused, so -DLLAMA_FATAL_WARNINGS=ON does
not compile. Predates this series; drop the line.
(cherry picked from commit 6eba44a89d5f328eb4859b844e1d28fb564cbe3e)
* qwen4exp: include llama-impl.h explicitly for llama_mul_mat_hadamard
(cherry picked from commit b634fd4d250d181ef82bf78bd00c1ae3b96a7af6)
* convert: fix the qwen4exp lint and type-check failures
flake8 flagged an unused MmprojModel import, and ty flagged seven errors in
the PLE streaming path: eos_token_id can be absent, and _ple_map, _ple_path,
_ple_row_dim and _ple_rows_per_shard are all Optional at the declaration but
were dereferenced without narrowing.
The map is opened and the stride fixed before the first shard is written, and
_finish_ple_table only runs once every shard has landed, so the invariants
hold. Assert them so the checker can see it. A missing eos_token_id now raises
with the reason instead of a TypeError from int(None).
* llama: give the qwen4exp indexer cache its own tensor names
The indexer KV cache and the attention KV cache both named their tensors
cache_k_l%d, so the Meta backend matched the indexer cache against the
attention split pattern and aborted in handle_set_rows. Tag the names
instead, and mirror the indexer cache: it has one key head and its
projections are mirrored.
(cherry picked from commit a1cdc8181134659766763a17762545a1f0e5db7b)
* qwen4exp: double the Q split granularity for tensor parallelism
qwen4exp fuses the attention gate into attn_q.weight the same way qwen3next
and qwen 3.5 do, so a device boundary must fall on a whole q+gate pair or the
Q heads stop lining up with the K/V heads and attn_output rows.
(cherry picked from commit 6c9a592f0a425a459ab6efae3b897cf68460e244)
* qwen4exp: keep the indexer cache in step across server slots
The QSA indexer keeps a side cache addressed by the cells of the attention
cache, so cell j has to hold the same token in both: the top-k indices it
produces are applied to the attention KQ mask. init_batch already hands the
indexer the attention cache's slot layout rather than letting it look for its
own, but the restore path did not. state_read called llama_kv_cache::state_read
on the two caches in turn and each ran its own find_slot over its own occupancy.
That agrees only for as long as nothing has already pushed the two caches apart,
which is the property a restore is supposed to re-establish rather than one it
can lean on.
The failure path was the worse half, and it is reachable from the public API
with nothing more than a short buffer. Truncating a good blob at 35 offsets and
feeding it to llama_state_seq_set_data left the two caches disagreeing at 5 of
them, and every one of 23 truncations of a whole-context blob did. Four of those
five land inside the attention section, so the attention cache drops the
sequence and the indexer keeps it; only the cut that lands in the indexer
section gives the opposite direction. llama_kv_cache::state_read cleans up its
own cache and rethrows, so whichever way it falls, nothing is left to bring the
two back together. The server papers over this by clearing the slot when a
prompt cache load fails; a caller of llama_state_seq_set_data that does not is
left with an indexer addressing cells that no longer mean what it thinks.
llama_kv_cache::state_read_sinfo reports the cells a restore landed in, or takes
a copy of them, and state_read_meta uses a supplied layout in place of find_slot
once it has checked that those cells are free here too. The indexer now adopts
the attention cache's restored layout by construction instead of reproducing it
by coincidence, and a layout that does not fit fails the read rather than being
applied over cells that already drifted. The hybrid restore is wrapped so that
any failure drops the sequence, or for a whole-context restore the context, from
all three caches at once, which is a state they do agree on.
* kv-cache: clear the cache once when restoring a whole context
state_read walks the streams of the cache in turn, and for a whole-context restore
each stream went through state_read_meta, which starts by calling clear(). clear()
resets every stream at once, so each stream after the first threw away the streams
already restored, and the K/V buffers with them. A non-unified cache holds one
stream per sequence, so a context saved with N sequences in it came back with only
the sequence in the last stream that carried any cells - the highest sequence id.
A unified cache has one stream and never showed it.
The cache is now emptied once, before the loop, which is what a whole-context
restore means. A blob whose streams are all empty now empties the cache as well,
where before it left the old contents in place.
* kv-cache: check the mirrored slot layout on a whole-context restore too
state_read_meta only looked at the layout it was given on the single-sequence path.
A whole-context restore lays the cells out from 0 in both caches, so they agree as
long as they restore the same number of cells, but nothing checked that they did: an
indexer section belonging to some other context was read over cells the attention
cache had filled from a different one, which is the state the indexer must never be
left in.
* qwen4exp: give the PLE conv history its own mirrored recurrent row
n_embd_r() reserved n_conv + ple_conv_state() so that one cache_r_l row could
carry both the delta-net conv state and the PLE dilated conv history, but the
QWEN4EXP arm of get_split_segments only described n_conv. Under -sm tensor the
segment sum came up short by ple_conv_state() and llama_memory_recurrent
construction aborted in ggml_backend_meta_alloc_ctx_tensors_from_buft.
Widening the segment list is not the fix. The Meta backend propagates a view's
split descriptor from its parent unchanged, so a view of one sub-range of a
split axis is sized as the whole row on every device; declaring the PLE tail as
a second segment merely moves the abort to "shape mismatch for VIEW" at graph
allocation. The two histories also want opposite policies: the delta-net state
is split by head to match wqkv and ssm_conv1d, while per_layer_tok_embd,
ple_conv1d and ple_norm_conv are all mirrored, so every device computes the
whole dilated conv and needs the whole history. One tensor cannot be both, and
the split state has no per-segment mirroring.
Move the PLE history into its own cache_ple_r_l%d row, mark it MIRRORED, and
return n_embd_r() to n_conv. The row is allocated only on layers where is_ple
holds, so mirroring one 92160-element row per device replaces a 92160-element
tail on all 36 recurrent rows: the recurrent R footprint drops rather than
grows. build_conv_state_at now takes its width from the tensor it was handed
and keys its gather on that tensor, which also drops a cont of a strided view.
* no more ple_hist (use master version)
* llama: give the qwen4exp full memory context its indexer cache
graph_reserve() walks a full memory context, and qwen4exp builds its
sparse attention only when the context exposes an indexer cache. the
full-context constructor left ctx_idx null, so the reserved worst case
was the dense fallback: a smaller graph than the one decode executes.
ggml-alloc then had to grow the compute buffer on the first decode,
past the size reported at load.
with -np 4 -c 32768 -fa on -ctk q8_0 -ctv q8_0 on an IQ1_S qwen4exp,
the reserved CUDA0 buffer was 217.00 MiB against 275.71 MiB actually
used, and CUDA_Host 42.31 MiB against 191.14 MiB. reserving the sparse
graph makes both match exactly, in unified and non-unified cache mode.
Co-authored-by: Pascal <admin@serveurperso.com>
Assisted-by: Claude
* qwen4exp: shrink the PLE hparams storage
llama_hparams is held by value inside llm_graph_params and every llm_graph_input_*,
and llm_graph_params is a stack local in graph_reserve and process_ubatch, so its
width is paid on every worker thread stack.
is_ple_impl spent 2048 bytes carrying 512 bits. It is the one per-layer flag that is
not moved through the loader's uint32 array templates, so a bitset costs nothing in
call sites and also removes the uninitialized read that non-qwen4exp archs had, since
nothing filled the array for them.
The PLE head offsets and vocab sizes are token-space indices; the gather that consumes
them already truncates to int32, so 64-bit storage was never reachable. The gguf arrays
stay uint64 for file compatibility and are narrowed on load.
sizeof(llama_hparams) 34440 -> 31944, sizeof(llm_graph_params) 34872 -> 32376.
* llama: opt-in random-access mmap advice for host-resident gather tables
qwen4exp keeps per_layer_token_embd on the host: 26.8 GiB at IQ4_NL, read
by ggml_get_rows as 16 gathers of ~90-170 bytes per token, spread across
16 head regions ~20M rows apart. Measured over 4.75M gathers, no two
consecutive gathers land on the same 4 KiB page, so the readahead the
loader asks for buys nothing here and the whole table ends up cached to
serve about 4% of itself.
llama_mmap applies POSIX_FADV_SEQUENTIAL, MAP_POPULATE and a whole-file
POSIX_MADV_WILLNEED unconditionally. Those are right for streaming the
file once into buffers and wrong for whatever stays mapped afterwards.
Under LLAMA_MMAP_RANDOM the eager pull-in is skipped and the mapping is
advised random once every tensor has been read, so the load itself keeps
its sequential readahead. That alone drops the table to 4.4% resident but
serializes one NVMe latency per gather.
The second half is what pays for it: the PLE input already computes every
row index for the ubatch before the graph runs, so the pages those rows
fall on are handed to the kernel in one batch and the reads overlap.
POSIX_MADV_WILLNEED on POSIX, PrefetchVirtualMemory on Windows, which
takes the discontiguous ranges in a single call.
Off by default and off for every other model: the batched prefetch keys
off "this mapping was advised random", which nothing sets unless the user
opts in.
-c 512 --chunks 60, cold, IQ1_S, mean of 3:
default 35.3 s 26.82 GiB resident (100%)
advice only 104.5 s 1.19 GiB resident (4.4%)
advice + prefetch 34.2 s 1.19 GiB resident (4.4%)
PPL 4.2346 +/- 0.07862 in all three. IQ1_S KLD is unchanged in every
field, including Mean KLD 0.396070 +/- 0.001931 and Same top p 77.325%.
* llama: narrow the random-access mmap advice to the gather table
The advice was applied per mapping: every mapping the model kept got
POSIX_MADV_RANDOM plus a whole-file POSIX_FADV_RANDOM, and the eager
pull-in was skipped for every file. On qwen4exp that also hit
token_embd.weight, which sits 0.33 GiB past the PLE table in the same
shard and is read densely, not by sparse gathers. Measured over
-c 512 --chunks 60 on IQ1_S it fell to 8.45% resident, against 100% with
the feature off.
A model now nominates its gather tables (qwen4exp: per_layer_tok_embd)
and only those byte ranges are advised. The range is rounded out to
whole pages, which on this model takes in 832 bytes before and 192
after. token_embd goes back to 86.55% resident and the PLE table still
drops to 4.44%; smaps shows one VM_RAND_READ VMA of exactly the table
instead of one over all 27.16 GiB that stays mapped.
posix_fadvise is dropped from the narrowed path. POSIX_FADV_RANDOM
ignores its offset and length and marks the whole open file, and the
FMODE_RANDOM it sets is only read by page_cache_sync_ra() on the read()
path, which a fault on a MADV_RANDOM vma never reaches. POSIX_FADV_
DONTNEED does take a range, so the drop mode keeps it.
The eager pull-in is now skipped only for the files holding a nominated
table, and re-issued as WILLNEED over the rest of such a file, so other
shards load exactly as before.
prefetch_rows() keys off the tensor being nominated rather than off a
mapping-level flag, so the batched readahead lands only where the advice
did.
-c 512 --chunks 60, cold, IQ1_S, mean of 3, total wall:
default 32.50 s
whole mapping 30.05 s
narrowed 30.35 s
PPL 4.2061 in all three. IQ1_S KLD is bit-identical with the feature on
and off, including Mean KLD 0.396070 +/- 0.001931 and Same top p
77.325%. tg128 73.65 +/- 0.33 narrowed against 73.49 +/- 0.34 whole.
Assisted-by: Claude
* llama: fold the random-access prefetch into its own feature flag
LLAMA_MMAP_RANDOM_PREFETCH existed to measure the two halves of the feature
apart, and the measurement is done: on a cold cache over the same wikitext
run, MADV_RANDOM without the batched readahead takes 94.4 s against 36.7 s
for an untouched mapping, while the pair together take 34.1 s. Suppressing
the kernel's readahead only pays if we replace it, so the split let a user
select a 2.6x regression through a documented switch.
Keep the accessor, since the call site reads better than a mode comparison,
but derive it from the mode alone.
* FACP (Fewer Acronym Classes Please)
* qwen4exp: bias the QSA selection per block, not per cell
The QSA bias is a graph input, so it is pinned on the host and uploaded every
decode, and at -c 32768 -np 4 its twelve copies were 768 of the 815 MiB of
reserved host compute buffer.
Only one half of it needs a cell: whether the cell sits in the always-visible
tail, and whether its block was pooled. Both are properties of the block. The
other half - empty, other sequence, or in the future - is the plain visible/not
test the attention mask already carries over the same cells, so add that mask
instead of repeating it. The bias then holds one value per block.
A block sits wholly inside or wholly outside the tail because the tail starts on
a block boundary, so one value per block is exact. Cells no block covers keep
their -inf from the mask.
The mask is F16 and the bias F32, and a mixed ggml_add reinterprets the F16
buffer as float rather than converting it, so the cast is required.
reserved host compute buffer at -c 32768 -np 4:
--kv-unified 814.86 -> 238.86 MiB, CUDA0 721.07 -> 421.07 MiB
--no-kv-unified 214.86 -> 70.86 MiB, CUDA0 317.07 -> 265.07 MiB
Selection is unchanged: over 8192 tokens, four times the budget, every QSA
layer returns identical top-k indices and the logprobs are bitwise equal.
Two things a reviewer should know. A cell whose position divides past the last
block is guarded by an assert rather than handled, because no run reached it.
And the mask's same-position M-RoPE rule cannot fire for text and was never
exercised for images, so the 2D case is unverified.
* clean up code comments
* clean up new comments
* revert LLAMA_MMAP_RANDOM
* nits
* replace some changes with #27795
* improve the m-rope image for get_prev_tokens
* LazyChunkedTensor
* fix lint
* add some validations
* reduce input nodes
* trim output tokens
* nits
* some more sanity checks
* fix llm_graph_input_ple reuse
* exclude from webgpu test
---------
Co-authored-by: danielhanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: danielhanchen <unslothshared@gmail.com>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pascal <admin@serveurperso.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
|
||
|
|
b10f9ca58c |
spec : add DFlash2 support (local convolution + candidate selector) (#27342) (#27816)
* spec : add DFlash2 support (local convolution + candidate selector) (#27342) * support DFlash2 * Add p_min in DFlash2 Assisted-by: Claude Opus 5 * Revert unnecessary changes Assisted-by: Claude Opus 5 * Revert draft sampling in rejection sampling Assisted-by: Claude Opus 5 * Refactor code structure Assisted-by: Claude Opus 5 * Delete embedding scaling Assisted-by: Claude Opus 5 * Gate output transforms on DFlash2 Assisted-by: Claude Opus 5 * Optimize Dflash 2 cost Assisted-by: Claude Opus 5 * Avoid using atoi Assisted-by: Claude Opus 5 * Modify comments Assisted-by: Claude Opus 5 * Move llama_model_dflash_selector_top_k to llama-ext.h Assisted-by: Claude Opus 5 * Formatting Assisted-by: Claude Opus 5 * Apply patch to fix the mrope bug Assisted-by: Claude Opus 5 * fix ci Assisted-by: Claude Opus 5 * Fix graph number calculation Assisted-by: Claude Opus 5 * rename hid and unary Assisted-by: Claude Opus 5 --------- Co-authored-by: Jian Chen <jianchen0311@gmail.com> Co-authored-by: Xuan-Son Nguyen <son@huggingface.co> * revert top-k.cu changes --------- Co-authored-by: Zihan Zhang <tiancaizhangdaxian@sjtu.edu.cn> Co-authored-by: Jian Chen <jianchen0311@gmail.com> |
||
|
|
54ee5ee643 |
mtmd: support dots3-note vision+audio (#27524)
* text: conversion * init impl * mtmd: conversion * impl mtmd cpp * Update gguf-py/gguf/tensor_mapping.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> |
||
|
Xuan-Son NguyenandGitHub
|
5a32f7b66e |
model: add dots3-note (#27060)
* text: conversion * init impl * address review comments * fix rope * move to a new llama_kv_cache_dsa_iswa |
||
|
|
7221e24f57 |
model : GraniteSWAForCausalLM / GraniteMoeSWAForCausalLM (#25505)
* feat(convert): Add conversion for GraniteSWAForCausalLM Branch: GraniteSWAForCausalLM AI-usage: full (Bob, OpenCode + Qwen3.6-35b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat(llama): Add granite_swa support Branch: GraniteSWAForCausalLM AI-usage: full (Bob, OpenCode + Qwen3.6-35b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat(conversion): Add conversion infra for rope_pattern array NOTE: There is other work also targeting this, so this may be removed depending on merge order. Branch: GraniteSWAForCausalLM AI-usage: full (Bob) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix(conversion): Fix SWA pattern logic and support for non-rope layers Branch: GraniteSWAForCausalLM AI-usage: full (Bob) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat(conversion): Add support for GraniteMoeSWA Branch: GraniteSWAForCausalLM AI-usage: full (Bob) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add llama_hparams::has_rope and arch constants NOTE: This shadows the work done for Granite Speech https://github.com/ggml-org/llama.cpp/pull/25107 Branch: GraniteSWAForCausalLM AI-usage: full (Bob) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add support for per-layer rope determination Branch: GraniteSWAForCausalLM AI-usage: full (Bob) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * style: Fix failing flake8 for extra newlines Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * test: Write out SLIDING_WINDOW_PATTERN in llama-model-saver Branch: GraniteSWAForCausalLM AI-usage: full (OpenCode + Qwen3.6-35b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix(convert): Fix missing registration for GraniteMoeSWAForCausalLM Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Load MoE params as optional Branch: GraniteSWAForCausalLM AI-usage: draft (OpenCode + Qwen3.6-35b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Handle MoE params in conversion branch: GraniteSWAForCausalLM AI-usage: full (OpenCode + Qwen3.6-35b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * style: Remove unnecessary newline AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Remove unnecessary tensor additions to GRANITE architecture Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Correctly handle naming for ffn gate inp Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Always default hparams.rope_pattern to 1s This isn't strictly necessary, but it will allow other models to rely on hparams.has_rope(il) without needting to prepopulate. Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Move to has_rope for all granite model architectures Now that we have a proper hparam for this, it's better to use it and not require a hacky fallback in the hparam method itself. Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: No hacky rope_finetuned fallback in has_rope Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Fully remove rope hparam filling in granitemoe There are no granitemoe models that use NoPE (it's not actually used in the layer building below), so this was just dead code. Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Save out rope_pattern in model-saver Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Set hparams.rope_finetuned for round trip Since the value is _read_ from rope_finetuned, we need to persist it when the model is saved with the saver. Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Code review cleanup Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * refactor: Keep gate/up fused for MoE path Branch: GraniteSWAForCausalLM AI-usage: full (Claude + Sonnet 5) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Skip GRANITE_SWA in model saver https://github.com/ggml-org/llama.cpp/pull/25505#discussion_r3773175651 Keeping is_swa_impl in the saver can break other models. Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * add sliding window pattern for model in test * style: Fix indentation Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Fix \r\n Thanks Claude! Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Keep shared expert fused Branch: GraniteSWAForCausalLM AI-usage: full (Claude + Sonnet 5) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * style: More indentation fixes Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> --------- Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> |
||
|
|
9cd719af21 |
model: support speculators-format checkpoints for DSpark (#26275)
* dspark: support speculators-format checkpoints (SpecForge exports) Speculators-format DSpark drafts (e.g. SpecForge exports for the Gemma-4-26B-A4B target) differ from the dense DeepSpec checkpoints in three ways: - the config nests the backbone hparams under transformer_layer_config and gives the extract layers as aux_hidden_state_layer_ids - the block is the DFlash 1+N fill-in layout: the anchor slot is a bonus token, not a prediction slot. Written as dflash.bonus_anchor; such drafts build the block and read the mask positions exactly like DFlash (n_max drafts from a 1+n_max block), only the Markov/confidence sampling comes from DSpark - the draft output vocab may be reduced (draft_vocab_size < vocab_size) with a d2t remap table. The converter expands lm_head/markov_w2 back to the full vocab and synthesizes an lm_head bias of -1e9 on the rows the draft cannot produce, so the runtime needs no d2t remapping. Such drafts ship their own (now optional) token_embd/output tensors instead of sharing the target's Verified against gemma4-26b-a4b-dspark: greedy outputs are byte-identical with and without the draft; acceptance 0.46, mean draft len 3.7 (n_max 6). Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Fable 5 * dspark: fold the speculators draft class into DSparkModel One class now covers every DSpark variant. What used to pick the class is a single flag, because the arch name turns out to be the only thing that separates the two families: SpecForge also exports a flat schema that carries no speculators_* fields yet still uses the 1+N bonus-anchor block, so keying on those fields would silently mis-read its drafts. Also rename i0 to i_first_pred in the draft read loop and the Markov head, and give the head a real bonus_anchor bool instead of testing i0 > 0. Converting the Qwen3-8B DeepSpec draft and both gemma-4 speculators drafts produces byte-identical GGUFs. The one behaviour change is that the markov_head_type check now also covers the DeepSpec checkpoints, which previously skipped it. Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Opus 5 * dspark: address review comments - rename bonus_anchor to sample_from_anchor (GGUF key and code), matching the checkpoint config field; absent key still means anchor-first - rework the reduced draft vocab to match EAGLE3: d2t is written as I64 absolute target ids and the logits are scattered at runtime, instead of expanding lm_head/markov_w2 and synthesizing an output bias at conversion - move the t2d skip to modify_tensors, like EAGLE3 - drop _is_specforge: the arch name only picks the sample_from_anchor default, embed/lm_head sharing is decided by the draft vocab size - deduplicate the tok_embd create_tensor left behind by the rebase Verified with the RedHat gemma-4-31b speculator draft: greedy output is byte-identical with and without the draft; acceptance 0.26 (n_max 7). Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Fable 5 * dspark: fold the sample_from_anchor read into the block_size block * dspark: fix flake8 continuation indent * clean up * dspark: key the sample_from_anchor default off the export format Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Fable * dspark: drop t2d in filter_tensors Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Fable * dspark: map model.lm_head instead of bypassing the dflash prefix Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Fable --------- Co-authored-by: desovo7 <942845546@qq.com> Co-authored-by: ruixiang63 <wangruixiang07@outlook.com> |
||
|
|
3733366720 |
model : BailingMoE3 Support (#26608)
* Adding support for bailingmoe3
* Adds speculative decoding support
* Make BailingMoE3 safe gate metadata optional
* bailingmoe3: apply trained SwiGLU clamps
* common: fix Bailing V3 tool argument parsing
* llama-model-saver, instantiate float vector metadata writer
* bailingmoe3: support Q-LoRA (Ling-3.0-tiny)
Ling-3.0-flash sets q_lora_rank: None and projects Q directly, so the current
implementation loads a single ATTN_Q tensor. Ling-3.0-tiny sets q_lora_rank: 256
and routes Q through a LoRA bottleneck instead:
q_a_proj -> q_a_layernorm -> q_b_proj
Conversion therefore failed with:
ValueError: Can not map tensor 'model.layers.3.attention.q_a_layernorm.weight'
Add the missing path, mirroring the existing deepseek2 MLA implementation:
* constants.py - add ATTN_Q_A / ATTN_Q_B / ATTN_Q_A_NORM to BAILINGMOE3
* tensor_mapping.py - map model.layers.{bid}.attention.q_{a,b}_proj and
q_a_layernorm
* conversion - emit attention.q_lora_rank when the config has it
* bailingmoe3.cpp - read n_lora_q; create the Q-LoRA tensors and build Q
through the bottleneck when q_lora_rank > 0
Everything is gated on q_lora_rank > 0. Ling-3.0-flash's config has no
q_lora_rank, the converter only emits the key when present, hparams.n_lora_q
defaults to 0, and get_key(..., required=false) leaves the target untouched when
the key is absent - so flash keeps taking the existing direct-Q branch.
The LoRA path produces the same shape as the direct projection, so the
nope/rope split, RoPE application and wk_b absorption downstream are unchanged.
* small mtp change
* bailingmoe3: support separate MTP GGUF and Q-LoRA MTP
* gguf: remove duplicate add_kda_gate_lower_bound definition
---------
Co-authored-by: bloomer <bloomer@booper.brushtail.me>
Co-authored-by: Dyluhn <dylanranejohnston1@gmail.com>
|
||
|
|
ad1de39e07 |
model: add Kimi-K3 text model (#26185)
* model: add Kimi-K3 text model Hybrid KDA (linear) + MLA (full) attention as in Kimi-Linear-48B, plus five things that architecture does not have: 1. cross-layer residual attention (attn_res_block_size) 2. latent MoE (routed experts run at n_expert_latent) 3. situ activation (replaces SwiGLU everywhere) 4. MLA output gate (sigmoid gate before o_proj) 5. full-rank KDA gate (single ssm_g instead of ssm_g_a/ssm_g_b) K3's text_config reports KimiLinearForCausalLM - the older 48B architecture - so get_model_architecture routes on the top-level name instead. The KDA decay gate has two forms, selected by linear_attn_config's gate_lower_bound. It is not a clamp: when set it swaps the activation entirely (fla/ops/kda/gate.py), from -exp(A_log)*softplus(x) to lower_bound*sigmoid(exp(A_log)*x). K3 sets it to -5.0; kimi-linear leaves it unset, so that path is unchanged. Cross-layer residuals reuse ggml_dsv4_hc_pre for the weighted sum. That op is CPU + CUDA only, so Metal/Vulkan will fall back per-node until those kernels exist. The routed experts ship as compressed-tensors "mxfp4-pack-quantized". That is bit-compatible with ggml's MXFP4 - same E2M1 code assignment, same E8M0 scale byte, only the nibble positions within a block differ - so they are repacked rather than dequantized, losslessly and without a ~5.5 TB bf16 round-trip. The repack is built lazily because gguf_writer holds every added tensor until the final write. DeepSeek-V4 was already doing the identical bit-shuffling, so it now shares the helper. Verified against Moonshot's own code path (transformers + fla's Triton KDA kernels) on a tiny model exercising every K3-specific feature. Final-position logits vs the fp32 reference: 6.7e-05 rel / corr 1.00000000 for both the chunked and the recurrent delta-net path. MXFP4 blocks dequantize to the source weights with 0.0e+00 error. Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * model: fix ty errors in the Kimi-K3 converter - `_res_parts` buffers (kind, tensor) pairs, not bare tensors - `get_tensors` must return an Iterator, matching ModelBase - LazyBase's `func` takes one argument, so pass the expert loaders through `args` instead of the closure - borrowing KimiLinearModel.set_vocab from an unrelated TextModel is deliberate and safe, but not expressible in the signature No behaviour change: the MXFP4 repack still dequantizes to the source weights with 0.0e+00 error and end-to-end logits are unchanged (8.386e-03 rel, corr 0.99996630). Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Update conversion/kimi_k3.py Co-authored-by: Boris Dvorkin <b_dvorkin@niuitmo.ru> * Increase LLAMA_MAX_EXPERTS from 512 to 1024 * tests : support for Kimi K3 in archs test * chat : add Kimi K3 chat format (reasoning, content, typed tool calls) K3's assistant output is an XTML-ish tagged format built by the template's open_tag/close_tag macros. Two properties break generic parsing: 1. The generation prompt ends with open_tag('think'), so the completion starts inside the think section with no opening marker in the output (thinking_forced_open). 2. Only <|open|>/<|close|>/<|sep|>/<|end_of_msg|> are special tokens; tag names ("think", "response", "message") are ordinary text tokens. Adds common_chat_params_init_kimi_k3 (PEG_NATIVE) with detection on the marker trio, reasoning extraction, response unwrapping, and tool-call parsing of the tools/call/argument tag structure with argument types taken from the tool schema. Includes the K3 chat template fixture and 9 test-chat cases derived from real generations of the full 2.8T model. Verified end-to-end against Kimi-K3-Q2_K (GrEarl/Kimi-K3-GGUF) on 8x B200: content, reasoning_content, streaming deltas, and tool_calls all correct; finish_reason stop/tool_calls as appropriate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chat : add message_delimiters for Kimi K3 Per-role message-start markers for token-level span splitting. User and assistant messages carry only the role attribute, so their full opener (through <|sep|>) is used; system and tool messages continue with more attributes (type=/tool=/index=), so those delimiters stop after the role's closing quote. Verified against the K3 tiktoken vocabulary that the closing quote is always a standalone token across all attribute variants, so the token-level prefix match stays exact. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: apply nits from @ngxson and text fixes from @danielhanchen * tests : added missing hyperparameters and tensors for Kimi K3 in test-llama-archs * chore : move overly verbose header file comments to Kimi K3 source file * tests : re-enabled KIMI_K3 in test-llama-archs for WebGPU backend * model-saver : emit kda_gate_lower_bound for Kimi K3 Quick fix. The Kimi K3 loader reads kda_gate_lower_bound and gates a graph branch on it (it scales the KDA gate when the bound is above -INFINITY), but the model saver never wrote the key, so a save->load roundtrip silently dropped it back to the -INFINITY default and changed the model's output. The real K3 config sets gate_lower_bound = -5.0. I propose to emit it from the saver, and set it to -5.0 in the test-llama-archs K3 case so the roundtrip check exercises it (the roundtrip fails without the saver line). * Refactor conditional for model architecture check * tests : re-enabled (again) KIMI_K3 and MINIMAX_M3 in test-llama-archs for WebGPU backend * fix code comments * add template on conversion * move repack_mxfp4_blocks to model base * nits * add_value_length * optimize res_stack construction * nits --------- Co-authored-by: Boris Dvorkin <b_dvorkin@niuitmo.ru> Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> Co-authored-by: Deepankar Singh <singh.deepankar39@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Caleb DeLeeuw <caleb.deleeuw@gmail.com> Co-authored-by: Xuan Son Nguyen <son@huggingface.co> |
||
|
|
6e62ba5384 |
mtmd: support pocket-tts (#26871)
* adapt the api * text model ok * working impl, need verify and clean up * mtmd: build the pocket-tts transposed convolutions as GEMM + col2im ggml_conv_transpose_1d has no grouped mode, so the depthwise upsample was built as one convolution and one concat per channel, which floods the graph with small nodes and makes kernel launches dominate the decoder. Fold both cases into the column form the seanet decoder already needs: the general case reshapes the kernel to [IC, K * OC] and matmuls it with the input, the depthwise case batches a matmul over the channels so a step scales its own kernel. A single col2im_1d then scatter-adds the columns back to the signal, with the same shape as before, so the overlap-add tail, the streaming state and the bias are untouched. Generation time per frame drops by 80% on CUDA and by 50% on CPU. The output matches the previous implementation sample for sample, with a correlation of 0.999994 and identical frame counts. * flow_temp + frames_after_eos * chunking * mtmd: carry the remaining pocket-tts per-pack settings The language packs also tune the end-of-speech padding and the padding of short prompts, next to the temperature already carried in the mmproj: french_24l asks for 8 tail frames instead of the guessed 3, english_2026-01 asks for short prompts to be padded with spaces. Write both in the mmproj as clip.gen.audio.frames_after_eos and clip.gen.audio.pad_short_text, keyed on the pack in the conversion script like the temperature. The loader keeps them optional, so a mmproj without them behaves as before. Map semicolons to commas for every pack instead, the reference only asks for it on three of them and it costs nothing elsewhere. Existing mmproj files must be converted again to carry the two keys. On a long french text the port now lands within 2% of the reference: 22.96s against 23.44s, with the same peak level and the same amount of silence. * clip.gen.audio.model_variant * clean up code comments * nit: drop the dead flow_temp hparam, the pack table holds the default * update docs * address security problems * less invasive base.py * lint * add mtmd_gen_inp_default * add docs * rm gen_flow_temp --------- Co-authored-by: Pascal <admin@serveurperso.com> |
||
|
Bar HaimandGitHub
|
157b81fe6d |
model : Granite-Switch Architecture (#25107)
* granite-switch: add llama.cpp backend (POC, CPU)
New "granite-switch" architecture: a dense, all-attention Granite-4.1
model with N embedded LoRA adapters selected per-token by control tokens.
- gguf-py schema (arch, KV keys, stacked LoRA tensor names) + writer helpers
- conversion/granite.py: GraniteSwitchModel converter (stacks N adapters +
zero base slot into per-projection A/B tensors; emits switch metadata)
- C++ arch registration (llama-arch.{h,cpp}, llama-model.{h,cpp})
- src/models/granite_switch.cpp: load + per-token switched-LoRA graph via
ggml_mul_mat_id over stacked tensors; sticky per-token index + control-token
substitution in llm_graph_input_switch::set_input
- llm_graph_input_switch in src/models/models.h
Runs end-to-end on CPU: convert 3b checkpoint (842 tensors, stacked dim 13)
and generate on both base and control-token paths. Sticky switch state is
single-sequence (POC); full multi-sequence machinery is a follow-up.
* granite-switch: add Mac (Metal) build + mid-sequence switch demo script
Self-contained script to build llama.cpp on Apple Silicon (Metal),
convert the composed 3b checkpoint, and run the crisp mid-sequence
adapter-switch demos verified on Vela:
- answerability: <|answerability|> mid-seq -> "unanswerable"
- query_rewrite: <|query_rewrite|> mid-seq -> {"rewritten_question": ...}
Each demo runs the same prompt twice, differing only by a control token
placed before the assistant turn, so the per-token switch is visible.
* granite-switch mac demo: add -no-cnv so each run is one-shot
The composed model ships a chat template, so llama-completion auto-enables
interactive conversation mode and halts at a `>` prompt after generating,
stalling the script. -no-cnv disables conversation mode: generate once from
the raw prompt and exit (also prints special tokens, making the switch visible).
* granite-switch: replace global sticky index with in-graph router attention
The POC computed the per-token adapter index on the CPU and carried it
across ubatches in ONE global `mutable int32_t poc_sticky_index`, reset
only when a ubatch contained sequence position 0. That global had two
problems:
1. Concurrency: with multiple sequences in a batch it was last-writer-
wins — one sequence's adapter leaked into the others.
2. Multi-turn: an interactive `ollama run` chat continues one KV cache,
so turn 2 never saw position 0 and the index never reset — the
adapter stayed stuck on across turns.
Port the vLLM/HF backend mechanism faithfully: a single-head causal
"router" attention recovers the adapter index in-graph. Per token, only
dim 0 carries signal — Q[0]=1, K[0]=+gain for a control token / -gain
otherwise, V[0]=adapter slot / 0 — and the causal softmax over the single
visible control token recovers that adapter's slot (readback =
clamp(round(V[0]), 0, n_adapters)). gain=15 matches config.py and is
F16-safe (no F32 cache).
The router's K/V live in the model KV cache at an extra layer
R == hparams.router_layer (== n_layer). We bump n_layer_all to n_real+1
so the cache allocator gives the router its own per-sequence slot, and
set n_layer_nextn=1 so n_layer() stays n_real — the decoder loop and
tensor loading are untouched and never reference layer R. The router K is
exempted from the k-shift RoPE loop (its dim-0 value is a literal
magnitude, not a rotation).
Because the selection now lives in the per-sequence KV cache, CONCURRENT
requests are isolated for free (problem 1 fixed; verified by
scratch/concurrent_switch_test.cpp). set_input becomes stateless pure
per-token maps; the global is gone.
Single-switch contract / known limitation, identical to vLLM & HF: the
gain is flat (no recency), so within one sequence there is no mechanism to
revert to base mid-sequence — once an adapter fires it stays on until that
sequence ends (problem 2 is therefore NOT fixed by a faithful copy; vLLM/HF
avoid it only because each served request is a fresh sequence). A client
continuing one KV cache across turns must start a fresh sequence per turn,
or opt into a recency-biased router (a deliberate divergence, not done
here). Documented in granite_switch.cpp and asserted by
scratch/multiturn_leak_test.cpp.
Verified (CPU): both demos unchanged (answerability -> "unanswerable",
query_rewrite -> rewritten query); concurrent two-sequence isolation
passes; multi-turn carry-over matches the vLLM/HF contract.
* granite-switch: drop scratch tests and mac demo for upstream PR
Remove the local-only development artifacts that should not ship in the
upstream PR:
- granite-switch-mac-demo.sh (local Metal build + demo driver)
- scratch/concurrent_switch_test.cpp
- scratch/multiturn_leak_test.cpp
Also drop the now-dangling reference to the scratch tests from the
granite_switch.cpp header comment. Leaves only the core architecture
support (conversion, gguf constants, llama-arch/model/kv-cache, and the
granite_switch graph).
* granite-switch: trim comments to match native llama.cpp style
* granite-switch: trim conversion comments to match native style
* granite-switch: drop unused adapter_ranks metadata
* granite-switch: rename arch to graniteswitch and drop obid alias
* granite-switch: fix non-ASCII comments and document router gain assumption
* granite-switch: drop section comments from constants.py to match native style
* granite-switch: add functional tensor block comments matching Granite4 Vision style
* granite-switch: clarify n_expert_used comment
State the actual constraint: mul_mat_id needs n_expert_used == 1, and
since the GGUF carries expert_count = 0 the generic loader's
n_expert == 0 => n_expert_used == 0 assertion has already passed by the
time load_arch_hparams runs, so it is forced to 1 here.
* granite-switch: note n_layer_nextn reuse has no MTP
The router carving reuses n_layer_nextn, normally the MTP/next-token
count. Clarify in the comment that it is borrowed here purely as the
trailing-layers lever and that there is no MTP head, to spare readers
the double-take.
* granite-switch: rename source file and apply review nits
* granite-switch: don't force LoRA tensors to F16, follow --outtype instead
* granite-switch: drop redundant _permute_qk wrapper, call LlamaModel.permute directly
* granite-switch: read router gain from GGUF (control_token_gain) instead of hardcoding 15.0
* granite-switch: derive n_slots()
* granite-switch: move llm_graph_input_switch into granite-switch.cpp
* granite-switch: cut AI-style narration comments
* granite-switch: collapse multi-line comments
* granite-switch: rename control_token_* maps to adapter_token_*
* granite-switch: cut noise comments
* granite-switch: rename embedded LoRA tensors to <base>.lora_a/lora_b
* granite-switch: GGML_ASSERT token input to avoid UB on embeddings
* granite-switch: TODO for raw embedding input support
* granite-switch: collapse LoRA tensor constants to .lora_a/.lora_b suffix
* granite-switch: drop n_expert_used hack, guard mul_mat_id buft probe
* granite-switch: stop forcing dense expert counts, read from config
* granite-switch: renamed control_token_gain metadata key to router_gain
* granite-switch: trim header comments to match native style
* granite-switch: collapse LoRA tensors to base name + suffix
* granite-switch: inline suffix checks in tensor op resolution
* granite-switch: drop switch-lora struct comment
* granite-switch: guard router layer index and inline n_slots
* granite-switch: group adapter metadata under {arch}.adapters.* namespace
* granite-switch: add hparams.has_rope(il) for KV-shift rope skipping
* granite-switch: skip arch in test-llama-archs (adapter fixture missing, TODO)
* granite-switch: Keys.Adapters namespace + simplify n_slots
* granite-switch: validate substitute token ids against n_vocab
* granite-switch: bound adapter count and lora rank from GGUF
* granite-switch: reject MTP context type when router_layer is set
* granite-switch: throw on bad adapter metadata instead of GGML_ASSERT
* granite-switch: use ASCII +/- in router K signal comment
* granite-switch: document n_layer_nextn repurpose and its leak points
* granite-switch: gate lora_a/lora_b op mapping on router_layer
* granite-switch: label all three preview model sizes
|
||
|
|
0713275082 |
mtmd: support Qwen3-TTS (note: breaking change to llama-tts binary) (#26254)
* convert text model * main model load ok * convert encoder ok * speaker encoder loading ok * speaker enc graph * adapt vocab for backbone (with some tricks) * add suppress_tokens * poc new mtmd gen api * convert code_predictor to gguf * load gen_code model ok * add clip_encode * wire up * code gen cgraph init version Co-authored-by: Pascal <admin@serveurperso.com> * code2wav convert to gguf * code2wav graph ok * wire up in/out * (wip) subgraph * wire up * wip, correct code2wav * demo (to be removed) * code2wav preserve kv between calls * demo voice clone * llama: add llama_model_get_tok_embd * mtmd_helper_gen_audio API * fix clamp cold prefix Co-authored-by: Pascal <admin@serveurperso.com> * fuse snake op Co-authored-by: Pascal <admin@serveurperso.com> * demo: use proper sampling * update dev docs * polymorphism helper * revamp llama-tts binary * update docs * fix compile * fix lint * nits * add guide + docs * more timings info * clean up code comments * security fixes * update docs * use ggml_build_forward_select, clean up comments * fix ci * use ISO 639-1 language code * rename CODE2WAV --> GEN_WAV, update docs * clean up * clean up tts.cpp * add seq_id * add step_prompt() * mtmd_helper_model_can_chat * clean up comments --------- Co-authored-by: Pascal <admin@serveurperso.com> |
||
|
|
db7d8b24b5 |
mtmd: add n_embd_head (#26342)
Co-authored-by: Daniel Han <unslothai@gmail.com> |
||
|
|
7e1e28cae3 |
mtmd : add Nemotron 3 Nano Omni support (parakeet) (#22520)
* mtmd : add Nemotron 3 Nano Omni support (parakeet) This commit adds support for the subsampling and encoder part of Nemotron Nemo 3 omni model. The Parakeet subsampling/encoder were taken from parakeet.cpp which is currently a pull request against whisper.cpp. I've tried to copy the code a close as possible to hopefully enable easy patching between the these two project later. Refs: https://github.com/ggml-org/whisper.cpp/pull/3735 * mtmd : generate rel pos tensor in graph instead of in conversion [no ci] This commit removes the generation of the relative positional tensor in the model conversion script and instead computes it in the encoder graph. This is only done for the window of positions required for the current audio sample. * mtmd : add clip_get_model to clip API [no ci] This commit adds a function to get access to the clip_model. It also removes the two functions clip_get_mel_filter_tensor, and clip_get_window_tensor(const struct clip_ctx * ctx) which can now use clip_get_model to access the model tensors that it needs. * mtmd : read mel_filters and window into hparams * mtmd : use set_input_f32 lambda [no ci] * mtmd : add better asserts for mel_filters and hann window [no ci] * mtmd : add missing size_t cast * mtmd : change type of pad to size_t * mtmd : zero initialize samples_padded * mtmd : remove unsued ctx member from parakeet preprocessor * mtmd : make log_mel_spectrogram_parakeet_worker_thread private static * mtmd : sync/update parakeeet impl with latest whisper.cpp This commit updates the parakeet code in mtmd to reflect the latest updates to parakeet.cpp in whisper.cpp. A follow up commit will address the currently hardcoded dw_pad and see if we can add n_conv_kernel as a model metadata field. * mtmd : add audio_conv_kernel_size to model conversion This commit updates the model conversion to read the conv_kernel_size field from the sound_config section of the models config.json file. It then uses this field instead of the hardcoded values in parakeet.cpp. * mtmd : cleanup [no ci] * conversion : call super().filter_tensors [no ci] * do not discard result of super filter_tensors * mtmd : use build_mm instead of ggml_mul_mat * mtmd : use build_ffn * mtmd : move and reuse get_vector lambda * mtmd : use build_inp_raw for parakeet * mtmd : throw exception in get_scalar instead of assert * mtmd : fix std::min call * mtmt : use .c_str in throw clause in get_vector * mtmd : check for F32 type and non-empty tensor in get_vector The get_vector lambda is used by get_scalar but also standalone to read in the mel_filters and the window data. Therefor we are not checking for 1D tensors but allowing multiple dimensions. We do have a check in get_scalar to verify the size of the vector. * mtmd : replace hardcoded 1101 for n_tokens_real * mtmd : assert subsampling_factor is 8 This commit adds an assert of the parakeet subsampling factor to check that it is 8. The motivation for this is that this model currently has three convolutions with a stride of 2. If the underlying model updates the subsampling factor these convolution operations will need to be updated and this will produce and error if this occurs. * mtmd : remove unused ggml_tensors attn_pos_w and mm_norm_w * mtmd : remove single thread path This commit removes the single thread path which was a left over from the original parakeet.cpp where n_threads is configurable. * fix some security issues --------- Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> Co-authored-by: Xuan Son Nguyen <son@huggingface.co> |
||
|
Ruixiang WangandGitHub
|
f87067841b | spec: add eagle3-v3 support for gpt-oss model (#25794) | ||
|
Xuan-Son NguyenandGitHub
|
1cbfd19883 |
mtmd: support MiMo-V2.5 audio input (RVQ-based model) (#26190)
* gguf converter for mimo audio * fix conv * cpp impl * nits * nits 2 |
||
|
|
b77d646751 |
model: Add support for Nanbeige4.2 (#25994)
* support nanbeige4.2 model * fix * fix flake8 Lint check * fix loop bound check and drop redundant head_dim --------- Co-authored-by: root <lizongqiang@kanzhun.com> |
||
|
|
b1d4c65524 |
model: Add MiniMax-M3 (MSA: MiniMax Sparse Attention) support (#24908)
* Add preliminary MiniMax-M3 support Text-only port that re-uses existing components: MiniMax-M2 style GQA with per-head QK-norm and partial rotary, DeepSeek-V3 style leading-dense and routed/shared experts, and swigluoai activation. Sparse attention is not yet supported (dense fallback); vision tower and MTP heads are dropped. * MiniMax-M3 vision tower (mmproj + clip graph) * Delete m3_vision_ref.py * Update clip.cpp * MSA * Update constants.py * Update minimax.py * Cache creation. Working withotu flash attention * Added flash attention for sparse layers * Decomposed slow cpu OP into GPU + CPU ops. Massive speedup over long ctx * Rewrote indexer op to be cuda native. Modified flash attention to match per group block picking * Implement sparse attention calc out of stock ops. * Fix a cache allocation and cont issue * Fixed -fa auto crash, flagged debug spots * Delete vocab.json * Delete model.safetensors.index.json * Delete generation_config.json * Delete Minimax directory * Handled multi stream case to fall back on Dense Attention * Development scaffolding cleanup. No functional change to the decode or 4-way paths. Full debug harness remains at <8136a9c68ed7a5eb009aa67bba3fda8062f4648f> for reproducing the selection-parity validation. * Remove redundant comment from minimax-m3.cpp * Changed 3 Gelu Ops for vision into Gelu_erf ops * Assert that n_kv is multiple of 128 * Rename MSA index tensors to indexer convention Note: All GGUFs generated before this change will need to be regenerated. * Fix incorrect Assert * Review driven changes (#3) * Remove comment from conversion minimax.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Remove whitespaces from constants.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Tighten comment in minimax.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * inherit MiniMax-M3 from MiniMax-M2 * drop dead text_config fallbacks * Add indexer writer methods * Reuse LLM_FFN_SWIGLU_OAI_MOE * Remove duplicate indexer setters, add only block_size/local_blocks, follow value naming convention * Fix conversion error /gguf_writer.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Update gguf-py/gguf/gguf_writer.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Update gguf-py/gguf/tensor_mapping.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Update conversion/minimax.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Update conversion/minimax.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Remove whitespace in src/llama-kv-cache.cpp Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Remove Whitespace in Update src/llama-model.h Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Remove whitespace in src/llama-hparams.h Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * remove multimodal code upon maintainer request. Will be made as a separate PR * Whitespace clean in tensor_mapping.py * Log cache size on launch, block ctx shift, support prompt caching Log indexer cache size on launch Disallow ctx shift Support prompt caching * Update minimax-m3.cpp * Optimize implementation, add multi stream support. Fully rewrote minimax-m3.cpp for speed and buffer size gains: Unified the 4-way + decode, 1 FA call per layer instead of 4, with the groups mapped onto ne[3] Custom CPU op now emits block-level mask, expanded on GPU, which causes CPU to GPU transfer to shrinks at prefill Decode: ~25 nodes/layer vs ~50, no per-group concats/conts Unified selection semantics, so both regimes rank bs + local bias (position-anchored local force), which means prefill/decode can no longer disagree on selection can_reuse on the MSA bias input. Graph reuse at decode restored (was rebuilding the full graph every token) In-place mask adds, shrinking compute buffer ~6.8 to ~4.2 GiB at ub2048/62k Multi-stream: MSA now runs with -np N when kv_unified=false. Decode stays batched across streams (still 1 FA call), prefill loops per stream. dense fallback only for --kv-unified + multi-seq Measured effect on expert offload bound setup: decode 6.2(4WAY)–7.15(MSA_decode) -> 7.7~7.8 t/s, flat from 5k to 60k+. prefill around 10% faster. buffer about 20% smaller, multi-user support. * set default cache type to F32 * Fix potential DSA double indexer cache allocation bug, only allocate in-cache k_idx for archs that opt in * remove F16 downcasts in MSA attention, force F32 indexer score accum * Add Minimax eos to llama vocab * Guard edge case where idx cache can become stale after a tail trim * Update llama-kv-cache.h * Update llama-kv-cache.cpp * Update llama-kv-cache.cpp * Update llama-kv-cache.h * Update llama-kv-cache.cpp * Review driven changes * style fix * indexer hparams are required * fix tests * fix lint --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> Co-authored-by: Xuan Son Nguyen <son@huggingface.co> |
||
|
|
88bfee1429 |
model: add GLM 5.2 Indexer support (#25407)
* Start building graph - reuse deepseek32 * Enable kv cache and rotation for glm_dsa architecture Just follow Deepseek 3.2 for now. * Reuse prev_top_k for "shared" indexer layers * GLM 5.2 uses LLAMA_ROPE_TYPE_NORM for the indexer. This is transformers' `apply_rotary_pos_emb_interleave` * Default indexer types to GLM pattern Previous converted GGUFs like https://huggingface.co/unsloth/GLM-5.2-GGUF write indexer weights to _all_ layers, even if they are only required for "full" types. This PR relies on a new key "%s.attention.indexer.types"; if absent, it will use the default GLM 5.2 schedule as defined in https://huggingface.co/zai-org/GLM-5.2/blob/main/config.json#L26. Note that conversion is not saving this key yet. * Save indexer types to gguf, restore on load * Use ggml_lightning_indexer when cparams.fused_lid Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com> * GLM 5 and 5.1 use full indexers Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com> * Fix indentation * Ensure array is zero-filled * Prefer explicit std::fill * Assert prev_top_k exists for shared indexer --------- Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com> |
||
|
|
8c146a8366 |
DeepSeek V4 (#24162)
* convert: add dsv4 conversion * add basic setup * add llm_graph_input_dsv4 * add save-load state * add sinkhorn eps - correction by @fairydreaming * add rope fix * cleanup dead code * fix bugs * support pro model: added by @fairydreaming * remove redundant V cache * Chat template * remove debugging leftovers * Add mechanism for inlining templates based on architecture * s/deepseek-v4-flash/deepseek4/g * s/deepseek-v4-flash/deepseek4/g continued * enable graph reuse * enable FA * fix test llama archs * rename * compatibility with antirez ds4 GGUFs * simplified set_gguf_parameters() by calling super class method, replaced moe.score_func with expert_gating_func. * reserve worst-case kv-cache * revert max split inputs * address review comments * add padding to enable FA * pad only the final value of plan.n_kv to 256 * remove built-in cpp chat template * cont: remove cpp built-in template * rm outdated test * replace ggml_view_3d() with ggml_reshape_3d() Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * only support n_seq=1 for now * remove unused var * cont: remove unused var * use scale bias * use correct ptr for can_reuse * remove gen-chat-inline-templates.py * simplify graph reuse * cont: cleanup * remove unused inputs * enable partial checkpointing * add correct shape for kq_mask + set llama_model_n_swa to 0 for dsv4 * precompute source_idx + add comment about dummy write * support multi-seq * remove restored_trim_pos * use split_equal when possible * fix indent * address review comments * use LLM_KV * fix ci --------- Co-authored-by: Piotr Wilkin <piotr.wilkin@syndatis.com> Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> Co-authored-by: Xuan Son Nguyen <son@huggingface.co> Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> |
||
|
Ruixiang WangandGitHub
|
fa72bc6826 |
dflash: refactor draft model conversion (#25110)
* dflash: refactor draft model conversion * apply fix for eagle3 convert |
||
|
|
a3900a6694 |
model: Granite Speech Plus (#24818)
* feat: Add conversion support for Granite Speech Plus Branch: GraniteSpeechPlus AI-usage: full (Bob, OpenCode + Qwen3.6-35b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Extend granite_speech to support plus multi-layer concatenation Branch: GraniteSpeechPlus AI-usage: draft (Bob, OpenCode + Qwen3.6-35b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix(conversion): Fix plural naming for feature_layers for audio Branch: GraniteSpeechPlus AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix(mtmd): Align feature_layer usage and naming everywhere Branch: GraniteSpeechPlus AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * style: Use fstring for log Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com> --------- Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com> |
||
|
|
68f30663cf |
vocab : refactor normalizer flags into options struct, add strip_accents (#24371)
* vocab : refactor normalizer flags into options struct, add strip_accents * Update src/llama-vocab.h Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/llama-vocab.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> |
||
|
Gabe GoodhartandGitHub
|
64086f2b2f |
model, mtmd: Granite4 Vision (#23545)
* feat(convert): Get language model conversion working for 4.1 vision Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat(convert): Skip multimodal tensors for GraniteMoeHybrid (vision 4.0) Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Disable vocab padding for non-hybrid models that use GraniteMoeHybrid Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Plumb python-side vision projector names and mappings There are several awkward things here: 1. Most of these are essentially identical to the audio qformer tensors. On the c++ side, that's mapped using the prefix, so the rest of the GGUF name needs to align, but on the python side there's no prefix notion, so they all get duplicated. 2. There are a couple of net-new tensors for vision, in particular PROJ_NORM. In both speech and vision, the QF_PROJ_NORM is qualified as belonging to the qformer portion, but the GGUF name is simply proj_norm which conflicts with the ideal name for this new PROJ_NORM that is not qualified as part of the qformer. To get around this, I used "proj_layernorm" as the GGUF name. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add python side architecture name Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add python-side plumbing for setting FEATURE_LAYERS hparam Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add c++ side tensor naming defines NOTE: Usage of these hasn't been updated to include prefix yet Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat(mtmd): Convert vision_feature_layer to an ordered vector We need to preserve the ordering of these feature index values so that they can be mapped to the sub-tensors within the stacked projectors. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat(mtmd): Add architecture label plumbing Branch: Granite4Vision AI-usage: full (OpenCode + qwen3.5:122b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat(wip): Add partial conversion for mmproj This handles stacking the projector tensors and setting the new harams Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add gguf_writer and constant support for new hparams and deepstack layer arr Branch: Granite4Vision AI-usage: draft (OpenCode + qwen3.5:122b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Full conversion for mmproj w/ tensor mappings Branch: Granite4Vision AI-usage: full (OpenCode + qwen3.5:122b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Add lm_head skip for mmproj for 4.0 Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: De-alias text_config architecture in convert_lora_to_gguf.py Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add --trust-remote-code arg to convert_lora_to_gguf.py This defaults to False, but allows a user to enable it programmaticly instead of using the interactive prompt. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: De-alias model.language_model. -> model. for lora adapters Branch: Granite4Vision AI-usage: full (OpenCode + qwen3.5:122b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Extend language model tensor dealiasing in adapters Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Remove unnecessary registration for GraniteSpeech in language model Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Plumb through mm prefix formatting for qformer tensors Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor: Refactor vision projector tensors to use predictor ID as the block This is cleaner than stacking them. The modeling file hard-codes single-layer qformers, so we can punt on the multiipule multi-layer projectors problem. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add spatial offests array hparam conversion Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add stub plumbing for granite vision in mtmd Branch: Granite4Vision AI-usage: draft (OpenCode + qwen3.5:122b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add new hparam and tensor naming in clip-impl.h New hparams: - KEY_PROJ_SAMPLE_QUERY_SIDE - KEY_PROJ_SAMPLE_WINDOW_SIDE - KEY_PROJ_SPATIAL_OFFSETS New tensors: - TN_MULTI_PROJ_IMG_POS - TN_MULTI_PROJ_QUERY - TN_MULTI_PROJ_LAYERNORM - TN_MULTI_PROJ_LINEAR - TN_MULTI_PROJ_NORM Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Move deepstack_layer_arr to llm hparam instead of mmproj Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Remove IS_DEEPSTACK_LAYERS This appears to have been added during Qwen3 VL (https://github.com/ggml-org/llama.cpp/pull/16780), but it was never actually used. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor: n_deepstack_layers -> deepstack_layer_arr The old logic hard coded a correspondence between the first N layers of the LLM and the 1->N entries in the input embeddings. Now, that relationship is maintained at loading time if the GGUF value is single-valued. If it is multi-valued, it loads directly allowing for deepstack layers to be spaced out throughout the model. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Use try/catch for single/multi valued deepstack info The alternative would be to use get_key_or_arr, but then the single value would be populated through the entire array and we'd need to detect that and update it with the right correspondence. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add deepstack injection point for granite LLM The use of ggml_add here assumes that the elements of inp_embd will be pre- arranged to be the full embedding length with only the vision-mask'ed portions non-zero from the projector. This matches how Qwen3VL does it. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: add missing vision attn layernorm eps Branch: Granite4Vision AI-usage: full (OpenCode + Qwen 3.6-35B) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor: Hoist qformer tensors into qf_block and hold a vector for multi-proj Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Fix missing prefix template for TN_QF_PROJ_LINEAR It's not strictly necessary since vision uses the blockwise version, but it makes the loading consistent. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Add embedding scale and image grid pinpoints hparams in conversion Also remove dead parsing for self._deepstack_layer_arr Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add mtmd KEY_ section for hparams shared with the LLM In this case, we need the EMBEDDING_SCALE so we can unscale the image embeddings to compensate for applying embedding scale to the input embeddings Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Implement c++ hparam parsing Branch: Granite4Vision AI-usage: draft (Claude Code) Co-authored-by: Eli Schwartz <eliyahu.schwartz@ibm.com> Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Flatten pinpoints in conversion Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Add missing break Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: No reason to have modality prefix for img_pos Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add tensor loading Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix(convert): Fix confusion between proj.norm and proj.qformer.layernorm Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Use the right portion of speech for tensor loading! Also plumb through the layernorm -> post_norm naming change Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add logging of deepstack_layers_arr if set I also changed the print_f output type to int32_t to avoid printing overflow values for -1. This could cause overflows on the other side, but I can't imagine a value for any of the current array hparams that would trigger that. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Make sure input embeddings are cont before f_embedding_scale Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add init and mmproj_embd cases for g4v The n_mmproj_embd is 1+ to make space for the text embedding and all 8 projectors Branch: Granite4Vision AI-usage: draft (Bob) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Invert (h, w) -> (w, h) pinpoints Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Reorder projectors based on llm index and skip the first injection The multi-projector stack has a strange asymmetry based on how it's currently implemented for qwen3vl: on the mmproj side, it's all N projectors, but the output of the "first" (by inp_embd index) projector is automatically consumed as if it were a standard single-projector mmproj, so the deepstack portion needs to only contain the 1-N entries. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> Co-authored-by: Eli Schwartz <eliyahu.schwartz@ibm.com> * fix: Fix mmproj hparams in conversion Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> Co-authored-by: Eli Schwartz <eliyahu.schwartz@ibm.com> * fix: Fix ordering/logic for deepstack injection in granite Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> Co-authored-by: Eli Schwartz <eliyahu.schwartz@ibm.com> * fix: Fix preprocessing config to match what the model needs Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> Co-authored-by: Eli Schwartz <eliyahu.schwartz@ibm.com> * wip: Partial port of Eli's implementation This is still pretty broken, but it's getting closer. It now happily generates tokens, but the values are quite incorrect still. I suspect it's caused by the mapping of projectors from safetensors to their respective orders here. Also, this implementation breaks encapsulation pretty badly in mtmd_encode. This will need a big refactor to put the G4V-specific encoding logic somewhere more appropriate. Branch: Granite4Vision AI-usage: draft (Claude Code, Bob) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> Co-authored-by: Eli Schwartz <eliyahu.schwartz@ibm.com> * fix: Fix the pre-scaling on the input embeddings to correctly invert the scale We've got tokens! They still don't line up quite right, so something's a little off, but we're getting much closer now. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: invert embedding multiplier -> base_scale at load Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Fix setting image_resize_pad after new enum introduced Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Add G4V to mmproj mapping in conversion Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Re-add padding disable for non-hybrid hybrid models Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor: Simplify G4V n_tokens computation This is slightly more efficient and flexible for when we implement the unpad cropping. IMO, it's also clearer that it is adding the number of image_newline tokens (embeddings) to the grid, rather than recomputing the entire count. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add new clip APIs for post-tile-encoding assembly Granite 4 Vision uses llava-next style pack-and-unpad which requires injecting the learned newline after each row of the tile grid. A row here is a single row of the grid which is composed of (grid_x * cols_per_tile) * (grid_y * rows_per_tile), so the result is newlines injected in between individual tile rows, thus not something that can be handled with the standard llava-uhd block-wise endcoding. Branch: Granite4Vision AI-usage: draft (Claude Code + Opus 4.7) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add model interfaces for granite 4 vision assembler I'm on the fence about the best organization of this. These free functions allow the per-architecture logic in clip.cpp to access the model-specific graph building, but they still require a fair bit of model-specific logic in clip.cpp which is not ideal. I think a better approach may be to replicate what is done with the graph builders themselves (and possibly even make the assembler part of the model's existing graph builder). Branch: Granite4Vision AI-usage: full (Claude Code + Opus 4.7) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor: Remove all g4v-specific branching from mtmd.cpp in favor of clip assembler Branch: Granite4Vision AI-usage: full (Claude Code + Opus 4.7) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor(mtmd): Consolidate assembler logic into clip_assembler class family Just like `clip_graph` is the base class for building the model-specific encoder graphs, `clip_assembler` will be the base class for building the model-specific assembler graphs. This allows the assembly pattern to follow how the encoder pattern is implemented where the model-specific logic lives in a subclass co-located with the encoder graph builder that gets constructed by a simple factory method. Branch: Granite4Vision AI-usage: full (Claude Code + Opus 4.7) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * style: Comment improvement Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor: granite_vision -> granite4_vision Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Remove dead codepath for Qwen3VL add_vision_is_deepstack These pieces were never used on the c++ side (removed there in an earlier commit), so this is just cleanup that I missed before. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Oops! I did not mean to commit one of my prompt files But now it's too far back in history to effectively rebase out, even with interactive and --rebase-merges :( Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Add missing <algorithm> include for std::find It seems that this was already pulled in on some platforms, but not on others Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Fix Flake8 warnings in granite conversion module Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor: Remove clip_assembler in favor of clip_image_f32.append_token Per conversation in the PR, the clip_assembler pattern was too invasive. This is a compromise that limits model-specific blocks to add_media where each preprocessed tile is annotated with an injection type, after which all the token counting logic is generic and the newline injection itself is handled in the graph based on the value for the given tile image. Branch: Granite4Vision AI-usage: draft (Bob, OpenCode + Qwen 3.6 35b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor(convert): Split n_deepstack_layers and deepstack_layers (array) Branch: Granite4Vision AI-usage: full (Bob, OpenCode + Qwen3.6-35b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor(src): Handle n_deepstack_layers and deepstack_layers GGUF keys Branch: Granite4Vision AI-usage: draft (Bob, OpenCode + Qwen3.6-35b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Fix GGUF key for deepstack_layers_arr Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor: Remove pre-scaling embeddings and skip scaling for raw embd inputs This follows how gemma3 and gemma4 handle embedding scaling by skipping the multiplier for raw input embeddings. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor: deepstack_layers(_arr) -> deepstack_mapping(_arr) Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor: Fully revert changes to n_deepstack_layers and qwen3vl* Since we're going to keep the GGUF KVs separate, it makes sense to just keep the hparams separate too to limit the scope of this branch. The down side is that n_deepstack_layers and deepstack_mapping_arr are potentially conflicting. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Revert removal of "is_deepstack_layers" GGUF KV This KV is not used at all on the c++ side, so it's fully dead, but there's also no need to conflate this cleanup with the addition of G4V. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Remove unnecessary ggml_cont and build_forward_expand in cbx Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * style: Clean up comments Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Tighter and more flexible code for g4v_build_block This could be refactored to look a lot more like granite-speech, but the overall block constructs before/after the qformer are pretty different, so for now I'm going to leave it as is and just tighten a bit. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Remove unnecessary `unordered_set` include Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Add architecture guard on deepstack_mapping_arr printout Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Remove unnecessary AI-gen comment Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Always initialize deepstack_mapping_arr with -1 values This was causing `test-llama-archs` to fail, likely due to trying to save the uninitialized values, then re-loading them. It's safer to always initialize so that other models don't forget and end up with undefined behavior. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * style: Remove TODO about block/vs non-block tensor mapping Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor: Move is_vision_feature_layer logic into clip_hparams Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor: Use a bool for append_token Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * style: Remove unnecessary comment Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Remove unused get_model api yikes! Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor: Rearrange helpers for g4v to be private members and use build_attn Branch: Granite4Vision AI-usage: full (Bob, OpenCode + Qwen3.6-35b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Fix off-by-one in vision layer index This was inherited from the Claude Code implementation that pushed the negative index inversion down into the model file. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Fix norm/post_norm mixup in conversion face. palm. :( Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * style: More descriptive tensor names Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Apply PR cleanup for new conversion changes AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * fix(convert): Remove duplicate V_ENC_EMBD_IMGNL Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor: append_token -> add_newline Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * style: Comment cleanup Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Cleaner error handling/checking NOTE: format_string is not available in granite.cpp (and including clip-impl.h to get it doesn't compile, so I think it violates the intended encapsulation), so std::stringstream is the simplest answer. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> --------- Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> |
||
|
Xuan-Son NguyenandGitHub
|
a731805ced |
mtmd, model: allow skip build_vit() (#24077)
* add model * nits |
||
|
Hans FlorianandGitHub
|
bfb4308b05 |
model : support granite multilingual embeddings R2 (ibm-granite/granite-embedding-{97,311}m-multilingual-r2) (#22716)
* Add support for the ibm-granite/granite-embedding-{97m,311m}-multilingual-r2 embedding models:
* Added a version of the gpt4o tokenizer that has a fixed regex (better handling of marks), and different token merging setting for the 97m model
* Reused gemma4 tokenizer for the 311m model
* granite-embedding-*-multilingual-r2 : add support SwiGLU FFN for Granite Embedding Multilingual R2
* added new GGUF key <arch>.hidden_activation (LLM_KV_HIDDEN_ACT) + writer
* added a forward declaration of llm_ffn_op_type to llama-hparams.h
* added llm_ffn_op in hparams
* added LLM_FFN_NONE = 0 sentinel to llm_ffn_op_type (value-initialization), modern-bert: explicitly assigns LLM_FFN_GEGLU before reading GGUF (unchanged).
* centralized hidden_act mapping in llama-model.cpp, added llm_ffn_op_type_from_string() helper, mirroring rope_scaling_type/llama_rope_scaling_type_from_string()
* modern-bert reads the GGUF key (when present) and uses the resulting op in its FFN graph
* Added granite-embedding-{97m,311m}-multilingual-r2 to the converter code
* Added the hashes for the granite embedding multilingual R2 models
* Set the hidden_activation in the GGUF if the field is present in config.json (such as for the granite embedding models)
|
||
|
|
d4c8e2c29c |
vocab : add tokenizer support for jina-embeddings-v2-base-zh (#18756)
* vocab : add jina-embeddings-v2-base-zh (whitespace tokenizer) * lowercase defaults to true * type fix --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> |
||
|
AesSedaiandGitHub
|
4178259130 |
mtmd: add MiMo v2.5 vision (#22883)
* mimo-v2.5: vision support * mimo-v2.5: use fused qkv for vision * mimi-v2.5: fix f16 vision overflow * mimo-v2.5: comment cleanups * mimo-v2.5: Flash doesn't have mmproj more cleanup remember to use filter_tensors * mimo-v2.5: fix trailing whitespace |
||
|
|
8e52631d55 |
model: Add Mimo v2.5 model support (#22493)
* add mimo-v2.5 support * mimo-v2.5: fix modify_tensors row split * mimi-v2.5: forgot `add_attn_value_scale` plumbing * mimi-v2.5: fix tp dequant to detect tp rows * mimo-v2.5: fix TP iteration to be descending * mimo-v2.5: fix comment * mimo-v2.5: retain fused qkv * mimo-v2.5: missed the attn_value scale during merge * mimo-v2.5: fused QKV needs contiguous for scaling attention value * mimo-v2.5: move `speech_embeddings.` to TextModel filter_tensors * Update src/llama-hparams.h Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/models/mimo2.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/models/mimo2.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update convert_hf_to_gguf.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update convert_hf_to_gguf.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/models/mimo2.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * mimo-v2.5: include MTP weights in gguf --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> |
||
|
Yakine TahtahandGitHub
|
a00e47e422 |
mtmd: add granite-speech support (ibm-granite/granite-4.0-1b-speech) (#22101)
* mtmd: add granite-speech support (ibm-granite/granite-4.0-1b-speech) Conformer encoder with Shaw relative position encoding, QFormer projector, log-mel spectrogram with frame stacking. Encoder uses GLU gating, folded batch norm, and SSM depthwise conv. QFormer compresses encoder output via windowed cross-attention (window=15, queries=3) into the LLM embedding space. Audio preprocessing: reflect-padded STFT, 80-bin mel filterbank, dynamic range compression, 2x frame stacking (80->160 mel). GGUF converter handles batch norm folding at export time, fused K/V split, and Conv1d weight reshaping. Tested against HF transformers reference: token-for-token match on 30s/60s audio clips with greedy decoding. * mtmd: rename gs_ prefixed tensors to generic/architecture names * mtmd: use tensor_mapping.py for all granite_speech tensors * convert: fold GraniteSpeechTextModel into GraniteModel * mtmd: replace n_layer hack with explicit has_standard_layers flag * mtmd: replace hardcoded magic numbers with GGUF hparams for granite speech * mtmd: align KEY_A_ define spacing * convert: register GraniteModel for GraniteSpeechForConditionalGeneration * convert: fix ty type-check for GraniteSpeechMmprojModel registration * mtmd: align TN_ define spacing * mtmd: use generic layer loop for granite speech tensor loading * mtmd: merge qformer_proj_layer into clip_layer * mtmd: granite_speech remove redundant ggml_build_forward_expand on inputs * mtmd: granite_speech add comment explaining why build_attn is not used * mtmd: granite_speech hard-code eps in cpp, remove from GGUF metadata * gguf: add spacing between granite_speech tensor mapping blocks * mtmd: make generic audio layer_norm_eps read optional * mtmd: granite_speech keep encoder eps in GGUF, only hard-code projector eps * mtmd: align defines and struct fields in clip-impl.h and clip-model.h * mtmd: fix alignment and ordering issues across granite speech files * convert: granite_speech use filter_tensors instead of modify_tensors for skipping |
||
|
|
7bfe60fdf9 |
mtmd, llama : Update HunyuanVL vision-language model support (#22037)
* mtmd, llama : add HunyuanVL vision-language model support - add LLM_ARCH_HUNYUAN_VL with M-RoPE (XD-RoPE) support - add PROJECTOR_TYPE_HUNYUANVL with PatchMerger vision encoder - add HunyuanVL-specific M-RoPE position encoding for image tokens - add GGUF conversion for HunyuanVL vision and text models - add smoke test in tools/mtmd/tests.sh * fix: fix HunyuanVL XD-RoPE h/w section order * fix: Remove redundant code * convert : fix HunyuanOCR / HunyuanVL conversion - Tested locally: both HunyuanOCR and HunyuanVL-4B convert to GGUF - successfully and produce correct inference output on Metal (F16 / Q8_0). * clip : fix -Werror=misleading-indentation in bilinear resize * fix CI: convert_hf_to_gguf type check error - convert_hf_to_gguf.py: give HunyuanVLTextModel.__init__ an explicit `dir_model: Path` parameter so ty can infer the type for load_hparams instead of reporting `Unknown | None`. --------- Co-authored-by: wendadawen <wendadawen@tencent.com> |
||
|
Xuan-Son NguyenandGitHub
|
63f8fe0ef4 |
model, mtmd: fix gguf conversion for audio/vision mmproj (#21309)
* fix gguf conversion for audio/vision mmproj * fix test |
||
|
|
a970515bdb |
mtmd: Add DeepSeekOCR Support (#17400)
* mtmd: llama.cpp DeepSeekOCR support init commit * loading sam tensors * mtmd: fix vision model processing * deepseek-ocr clip-vit model impl * mtmd: add DeepSeek-OCR LM support with standard attention * mtmd: successfully runs DeepSeek-OCR LM in llama-cli * mtmd: Fix RoPE type for DeepSeek-OCR LM. * loading LM testing Vision model loading * sam warmup working * sam erroneous return corrected * clip-vit: corrected cls_embd concat * clip-vit: model convert qkv_proj split * corrected combining of image encoders' results * fix: update callback for ffn_moe_weighted and add callback for attn_out in deepseek2 model * concat image_newline and image_seperator tokens * visual_model warmup (technically) works * window partitioning using standard ggml ops * sam implementation without using CPU only ops * clip: fixed warnings * Merge branch 'sf/deepseek-ocr' of github.com:sfallah/llama.cpp into sf/deepseek-ocr * mtmd: fix get_rel_pos * mtmd: fixed the wrong scaler for get_rel_pos * image encoding technically works but the output can't be checked singe image decoding fails * mtmd: minor changed * mtmd: add native resolution support * - image encoding debugged - issues fixed mainly related wrong config like n_patches etc. - configs need to be corrected in the converter * mtmd: correct token order * - dynamic resizing - changes are concerning PR https://github.com/sfallah/llama.cpp/pull/4 * mtmd: quick fix token order * mtmd: fix danling pointer * mtmd: SAM numerically works * mtmd: debug CLIP-L (vit_pre_ln) * mtmd: debug CLIP-L & first working DeepSeek-OCR model * mtmd : add --dsocr-mode CLI argument for DeepSeek-OCR resolution control & all native resolution modes work * mtmd: simplify SAM patch embedding * mtmd: adapt Pillow image resizing function * mtmd: simplify DeepSeek-OCR dynamic resolution preprocessing * mtmd: remove --dsocr-mode argument * mtmd: refactor code & remove unused helper functions * mtmd: fix tensor names for image newlines and view separator * clean up * reverting automatically removed spaces * reverting automatically removed spaces * mtmd: fixed bad ocr check in Deepseek2 (LM) * mtmd: support combined QKV projection in buid_vit * using common build_attn in sam * corrected code-branch when flash-attn disabled enabling usage of --flash-attn option * mtmd: minor fix * minor formatting and style * fixed flake8 lint issues * minor editorconfig-check fixes * minor editorconfig-check fixes * mtmd: simplify get_rel_pos * mtmd: make sam hparams configurable * mtmd: add detailed comments for resize_bicubic_pillow * mtmd: fixed wrong input setting * mtmd: convert model in FP16 * mtmd: minor fix * mtmd: remove tweak to llama-mtmd-cli & deepseek-ocr template * fix: test-1.jpg ORC issue with small (640) resolution setting min-resolution base (1024) max large (1280) for dynamic-resolution * minor: editconfig-check fix * merge with changes from https://github.com/ggml-org/llama.cpp/pull/17909 added new opt to tests.sh to disable flash-attn * minor: editconfig-check fix * testing deepseek-ocr quick and dirty test script comparing results of Qwen2.5-VL vs DeepSeek-OCR * quick and (potential) dirty merge with https://github.com/ggml-org/llama.cpp/pull/17909 * refactoring, one single builder function and static helpers * added deepseek-ocr test to tests.sh * minor formatting fixes * check with fixed expected resutls * minor formatting * editorconfig-check fix * merge with changes from https://github.com/ggml-org/llama.cpp/pull/18042 * minor - added GLM-4.6V to big tests - added missing deps for python test * convert: minor fix * mtmd: format code * convert: quick fix * convert: quick fix * minor python formatting * fixed merge build issue * merge resolved - fixed issues in convert - tested several deepseek models * minor fix * minor * Update convert_hf_to_gguf.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * - removed clip_is_deepseekocr - removed redundant RESIZE_ALGO_BICUBIC_PILLOW resize-algo - simplified image-preprocessing - removed/simplified debug functions * - cleaning commented out code * fixing instabilities issues reintroducing resize_bicubic_pillow * - use f16 model for deepseek-ocr test - ignore llama-arch test for deepseek-ocr * rename fc_w --> mm_fc_w * add links to OCR discussion * cleaner loading code * add missing .weight to some tensors * add default jinja template (to be used by server) * move test model to ggml-org * rolling back upscale change * Update convert_hf_to_gguf.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> --------- Co-authored-by: bluebread <hotbread70127@gmail.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> Co-authored-by: Xuan Son Nguyen <son@huggingface.co> Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com> |
||
|
|
ec2b787ebe |
mtmd: Add dynamic high-resolution image preprocessing for InternVL model (#20847)
* added support for internvl's dynamic high-resolution (Qianfan-OCR needed) * add min/max dynamic patch to gguf meta * clean up * simplified handling min/max dynamic patch * reuse llava_uhd logic for slice images * provide default values for older models * flake8 * prevent writing 0 value to gguf * remove duplicated resolution candidates with a better algorithm * fix indentation * format * add protection from divide by zero * change to 0 to be safe --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co> |
||
|
Sigbjørn SkjæretandGitHub
|
29b28a9824 |
ci : switch from pyright to ty (#20826)
* type fixes * switch to ty * tweak rules * tweak more rules * more tweaks * final tweak * use common import-not-found rule |
||
|
Sigbjørn SkjæretandGitHub
|
d13d60af1d | gguf-py : cleaner way to get the first key (#20727) | ||
|
|
5eae9cb1d9 |
ggml : add NVFP4 quantization type support (#19769)
* WIP: add NVFP4 quantization support * tests * improve NVFP4 dot product implementation performance and fix bad super call * typo * Use nvfp4 kvalues * vulkan : fix NVFP4 shader compilation by including kvalues_mxfp4 lookup table * vulcal and perf fixes * wip * Fix metal * fix vulcan * Rename threshold & fix wrong scale * Fix MOE * Shelf backend implementations (CUDA, Metal, Vulkan, arch-specific SIMD) Remove NVFP4 support from GPU backends and architecture-specific optimized dot products. These should be added in separate PRs so backend specialists can review them independently. Reverted files: - ggml-cuda: common.cuh, convert.cu, mmq.cu/cuh, mmvq.cu, vecdotq.cuh, quantize.cu/cuh, mma.cuh, ggml-cuda.cu, fattn-tile.cuh - ggml-metal: ggml-metal.metal, ggml-metal-device.cpp, ggml-metal-impl.h, ggml-metal-ops.cpp - ggml-vulkan: ggml-vulkan.cpp, all vulkan-shaders/* - ggml-cpu arch: arm/quants.c, x86/quants.c, powerpc/quants.c, s390/quants.c Core NVFP4 support (type definition, CPU fallback dot product, quantization, dequantization, conversion) is retained. * Fix arch-fallback.h: add NVFP4 generic fallback for all platforms After shelving backend-specific SIMD implementations, the generic CPU dot product needs to be aliased on ARM, x86, PowerPC, and s390 platforms that previously relied on arch-specific versions. * quantize: add NVFP4 as a quantization type option * Fix ggml_fp32_to_ue4m3: handle subnormal values Previously, values with ue4m3_exp <= 0 were clamped to 0, causing all small scales to underflow. This made NVFP4 quantization via llama-quantize produce garbage (PPL = 5.8M) since typical transformer weights have amax/6.0 in the range 0.001-0.01, which falls in the UE4M3 subnormal range. Now subnormals are properly encoded as man * 2^-9 (exp=0, man=1..7), matching the decode path in ggml_ue4m3_to_fp32. Result: NVFP4 requantization now produces PPL = 15.25 (vs F16 = 14.33), comparable to Q4_1 (PPL = 15.81) at slightly lower BPW (4.70 vs 5.15). * Restore ARM NEON NVFP4 dot product implementation Restores the optimized ggml_vec_dot_nvfp4_q8_0 for ARM NEON using vqtbl1q_s8 lookup and ggml_vdotq_s32 dot products. tg128 performance: 4.37 t/s (generic) -> 13.66 t/s (NEON) = 3.1x speedup * Optimize ARM NEON NVFP4 dot product: LUT + vpaddq + vfmaq - Add ue4m3_scale_lut[128] to ggml-common.h replacing branch-heavy ggml_ue4m3_to_fp32() in the hot loop - Use vpaddq_s32 for pairwise int32 reduction instead of vaddvq_s32 - Accumulate with vfmaq_f32 into float32x4_t vector accumulators tg128: 8.1 -> 31.0 t/s (3.8x speedup, 77% of Q4_1 speed) * ARM NEON NVFP4: rearrange q8 to match nibble layout Alternative approach: rearrange q8 data to match the NVFP4 lo/hi nibble layout instead of rearranging the looked-up NVFP4 values. Eliminates vcombine_s8(vget_low, vget_low) shuffles. Performance is equivalent (~18.5 t/s) - the bottleneck is the 2x block overhead from QK=16 vs QK=32, not the shuffle instructions. * CPU only backend 64 super-block layout * cleanup * Remove unused LUT * int * exclude NVFP4 from unsupported ops in metal build * remove quantization for now * store scales as native UE4M3, preserve original model bits when possible * Update convert_hf_to_gguf.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * correct comment * format * reduce duplication and cleanup * Address comments * move detection to prepare_tensors * Use math instead of const * Move * fix comment * Shelf quantize tests * Rebase and move check * cleanup * lint * Update gguf-py/gguf/scripts/gguf_convert_endian.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Use fallback quant config * Simplify Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * organize * Refactor * Update convert_hf_to_gguf.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update convert_hf_to_gguf.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update convert_hf_to_gguf.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * add quantize_nvfp4 (required for test_quants.py) * add quantize_nvfp4 (required for test_quants.py) * add quantize_nvfp4 (required for test_quants.py) * fix return type --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> |
||
|
|
eaf1d7930c |
llama : add support for Nemotron 3 Super (#20411)
* llama : add support for Nemotron 3 Super This commit adds support for the Nemotron 3 Super model (120B.A12B) enabling this model to be converted to GGUF format and run in llama.cpp. Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> Co-authored-by: Matt Clayton <156335168+mattjcly@users.noreply.github.com> |
||
|
Xuan-Son NguyenandGitHub
|
59db9a357d |
llama: dynamic head_dim and n_rot for SWA (#20301)
* llama: dynamic head_dim and n_rot for SWA * also add gguf_writer wrappers * fix build * build_rope_shift arg reorder |
||
|
|
418dea39ce |
ggml/gguf : prevent integer overflows (#19856)
* gguf : prevent integer overflow for ggml_context mem size * ggml : fix int overflows in ggml_new_object() * gguf : prevent string exhaustion * gguf : prevent array elements exhaustion * ggml : fix negative tensor type oob * py : assert that alignment is non-zero power of 2 * ggml : check int overflow in ggml_new_tensor_impl and ggml_new_object * gguf-py : error on duplicate keys when reading * py : restore tensor_fields * enforce proper alignment in add_custom_alignment * gguf : better name * gguf : fix ctx size for no_alloc == true * gguf : minor print fix * ggml : print values when overflow * ggml : remove deprecated ggml_type_sizef() * ggml : relax ggml_type asserts to debug-only * gguf : add mem_size overflow test * gguf : add file size check for arrays * ggml : relax asseerts for ggml_get_type_traits() * flake8 fix --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> |
||
|
|
752584d5f5 |
model: support GLM MoE DSA arch (NOTE: indexer is not yet supported) (#19460)
* model: support GLM MoE DSA arch * working version * pyright * keep indexer tensors * add indexer gguf params * loaded now * Apply suggestions from code review Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * update * Update src/llama-model.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * minor fix and cleanup --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> |
||
|
JJJYmmmandGitHub
|
fc0fe40049 |
models : support qwen3.5 series (#19468)
* support qwen3.5 series * remove deepstack for now, and some code clean * code clean * add FULL_ATTENTION_INTERVAL metadata * code clean * reorder v heads for linear attention to avoid expensive interleaved repeat |
||
|
|
b83111815e |
model : support Step3.5-Flash (#19283)
* Support Step3.5-Flash * fix: norm.weight + 1 (HF zero_centered=true) * step35: simplify GGUF conversion + drop redundant rope KVs * Address review feedback * rename limits -> clamp * Apply suggestions from code review Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Apply suggestion from @CISC Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * rename swiglu limits -> swiglu clamp in LLM_KV * avoid CI fail * Apply suggestions from code review * Apply suggestions from code review * disabled KV shifting for LLM_ARCH_STEP35 * Apply suggestions from code review * mistakenly removed cmath * add model size && apply missed suggestion * assert partial_rotary_factors * fix CI errors: * load freq_base_swa --------- Co-authored-by: lvyichen <lvyichen@stepfun.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> |
||
|
|
3688c4f504 |
Kimi-Linear support (backend agnostic + MLA KV cache) (#18755)
* kimi linear model implementation * kimi linear convert_hf_to_gguf * kimi linear constants.py tensor_mapping.py * Kimi Linear ggml.h * kimi linear ggml-cpu * Kimi Linear ggml-cuda * Kimi Linear ggml.c * kimi linear src/llama * remove "const int64_t n_seq_tokens = q->ne[2];" to get rid of unused variable warning * remove type mismatch warning * read MoE params * removed some hard coded code * removed all hard code * use DeepseekV2 tokenizer * removed unnecessary internal methods called by the old set_vocab of KimiLinear * rewrite get_vocab for KimiLinear. Removed all kda_scan code * removed all traces of kda_scan * reduce OP count by 1 due to removal of kda_scan * Move KIMI_LINEAR to llm_arch_is_hybrid to enable KV cache * set n_embd_head_k/v to ensure kv cache works * don't quantize conv1d of Kimi Linear * Kimi Linear backend agnostic * removed LOG_INFO * naive chunking form implemented * fixed some comments * add Kimi-K2 specific tokens to be recognized as EOG * build_kda_autoregressive is implemented to replace build_kda_recurrent for faster inference. sync'd to b7682 * replaced Akk and Aqk with mul_mat and clamp * no clamp version * Moved Aqk computation out of the loop * fixed typo and split wkv_b into wk_b and wv_b * MLA KV cache support * fix trailing spaces * moved const llama_model & model; around to follow qwen3next format and see if it cna pass the -Wunused-private-field error * fix trailing whitespace * removed traling whitespaces in empty line + make sure indentation is multiple of 4 * try to make lint happy * remove blank lines to make lint happy * removed at least blank line containing white space * fixed flake8 complaints locally * return ggml_tensor * pair in kda_autoregressive and kda_chunking as in ngxson's Qwen3Next improvement * removed Kimi-Linear specific change that causes failure at server-windows * removed private: from kimi_linear to make build checks happy * removed unnecessary ggml_cont before ggml_reshape * created static function causal_conv1d to abtract similar code for q/k/v * merged dt_bias to SSM_DT. Do -exp(log_A) in convert_hf_to_gguf.py. * reverted to original * fixed find_hparam calls. Fixed e_score_correction_bias to use bias instead of weight. Removed all ssm_conv bias terms. * remove DT_B from constants.py. remove one comment line in llama-model.cpp * new class llm_graph_input_mem_hybrid_k to get around the new MLA change. switch the concat order of ggml_concat calls in kimi-linear.cpp to accommodate MLA changes. Removed support for exp_probs_b.weight * remove ssm_o_norm_b * remove ssm_o_norm_b * changed hparams.kda_head_dim to hparams.n_embd_head_kda. added TODO comment for class llama_graph_mem_hybrid_k * removed all ggml_cont b4 ggml_reshape_4d * Whitespace * replaced all hparams.get with find_hparams * added new names for n_experts, n_experts_used and score_func in TextModel and removed their code in KimiLinear in convert_hf_to_gguf.py. Removed unnecessary ggml_cont and GGML_ASSERT in kimi-linear.cpp * use is_mla to switch between different mem_hybrid types * fixed logical errors in convert_hf_to_gguf.py pointed out by CISC * removed if else for required parameters kv_lora_rank and qk_rope_head_dim * add back ggml_cont for Vcur * minor changes * removed extra line in llama-vocab.cpp. Added back the comment in llama-graph.cpp * f16 gguf cannot run without context length * made a mistake of adding back n_ctx parsing --------- Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> |
||
|
Xuan-Son NguyenandGitHub
|
07a7412a3b | mtmd: add min/max pixels gguf metadata (#19273) | ||
|
|
a61c8bc3bf |
mtmd: Add Gemma3n multimodal support with MobileNetV5 vision encoder (#18256)
* Add Gemma3nVisionModel - MobileNetV5 vision encoder convertor to convert_hf_to_gguf.py. Add gemma3n to vision projectors in gguf-py/gguf/constants.py. * Add mobilenetv5 impl * Fix comments, remove unused vars * Fix permute and remove transpose of projection weights * Fix comments, remove debugging prints from hf_to_gguf * 1. Hard-code image_mean = 0 and image_std = 1 2. Use available tensor mapping logic 3. Remove redundant chat template replacement of soft tokens placeholder with media placeholder * 1. Move mobilenetv5 helpers declarations to `clip_graph_mobilenetv5` struct and definitions to mobilenetv5.cpp 2.Remove unused `clip_is_gemma3n` func declarations and definitions 3. Remove redundant `rescale_image_u8_to_f32` func and use `normalize_image_u8_to_f32` with zero mean and unit std 4. Calculate n_patches using image_size / patch_size * Remove obsolete comments * - convert_hf_to_gguf.py & constants.py & tensor_mapping.py: Use explicit mapping: Custom map for double indexed blocks and tensor_mapping.py for rest - convert_hf_to_gguf.py: Unsqueeze Stem Bias and Layer scale tensors to correct shape while converting to gguf - mobilenetv5.cpp: Remove explicit reshaping of Stem Bias and Layer scale which are now handled while converting to gguf, replace fprintf with LOG_* - clip.cpp: Remove unused embedding and hard_emb_norm tensor loading * - Rename tensors to v.conv..., v.blk..., v.msfa... to better align with already existing terminology * Fix stem conv bias name * Remove explicit handling of bias term for stem conv * - Change order of addition in "project_per_layer_inputs" to support broadcasting of vision inp_per_layer - Simplify the vision embeddings path of "get_per_layer_inputs" to output [n_embd_altup, n_layer, 1], broadcastable * clean up conversion script * fix code style * also preserve audio tensors * trailing space * split arch A and V * rm unused gemma3 func * fix alignment --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co> |
||
|
Tarek DakhranandGitHub
|
73d284a250 |
model : add LFM2-ColBert-350M (#18607)
* model : add LFM2-ColBert-350M * llama_model_n_embd_out() - returns `hparams.n_embd_out` if set and fallbacks to `hparams.n_embd` |