Compare commits

...
Author SHA1 Message Date
Pascal f990aa7065 server: drop the now unused ctime include
strftime() and gmtime_r() were the only users, both went away with the
get_datetime tool. Also make the renderer's catch inert: the browser
executor always emits JSON, so a non-JSON result is no longer a date to
display.
2026-08-17 14:27:18 +02:00
Xuan Son Nguyen 480fedca78 Merge remote-tracking branch 'origin/master' into xsn/ui_get_datetime 2026-08-17 13:19:40 +02:00
7077abbe14 ui: add browser get_info tool (#27251)
* ui: add browser get_info tool

* format

* nits

* Update tools/ui/src/lib/stores/tools.svelte.ts

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>

---------

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
2026-08-17 13:14:33 +02:00
Xuan Son Nguyen 5a61d48d08 clarify docs 2026-08-17 12:57:02 +02:00
Xuan Son Nguyen 35293d0bb0 ui: move get_datetime tool to frontend 2026-08-17 12:52:12 +02:00
Georgi GerganovandGitHub d83f72d463 ci : restore release.yml check during make-release.yml (#27247)
* ci : restore release.yml check during make-release.yml

* cont : bump version to v0.1.1
2026-08-17 13:10:56 +03:00
Xuan-Son NguyenandGitHub 9f0d017efb mtmd: harden preprocessor_granite (#27235)
* mtmd: harden preprocessor_granite

* simply cap to 1024
2026-08-17 11:30:15 +02:00
Georgi GerganovandGitHub 7c35571e5d ci : allow make-release to target a specific commit (#27234)
* ci : allow make-release to target a specific commit

The make-release workflow now accepts an optional 'commit' input. When
set, that commit is checked out and the release checks verify that it
belongs to the branch selected in the "Run workflow" dialog and is not
older than 3 days from the branch tip. The check is part of
make-release-checks.sh (driven by the RELEASE_BRANCH env), so it follows
the same dry-run semantics as the other checks.

Assisted-by: pi:llama.cpp/Qwen3.8-27B

* cont : scan latest 100 relase workflow runs

* cont : do not check manually for release.yml success
2026-08-17 11:52:46 +03:00
Georgi GerganovandGitHub f9779dda86 ci : make release workflows use a deploy key (#27229)
* ci : make release workflows use a deply key

* cont : update docker.yml
2026-08-17 11:31:20 +03:00
Xuan-Son NguyenandGitHub fa88ae9368 convert: add @ModelBase.example (#27208)
* convert: add @ModelBase.example

* add docs

* add more variants

* BailingMoeV3ForCausalLM

* rm pocket-tts
2026-08-17 10:15:11 +02:00
3733366720 model : BailingMoE3 Support (#26608)
* Adding support for bailingmoe3

* Adds speculative decoding support

* Make BailingMoE3 safe gate metadata optional

* bailingmoe3: apply trained SwiGLU clamps

* common: fix Bailing V3 tool argument parsing

* llama-model-saver, instantiate float vector metadata writer

* bailingmoe3: support Q-LoRA (Ling-3.0-tiny)

Ling-3.0-flash sets q_lora_rank: None and projects Q directly, so the current
implementation loads a single ATTN_Q tensor. Ling-3.0-tiny sets q_lora_rank: 256
and routes Q through a LoRA bottleneck instead:

    q_a_proj -> q_a_layernorm -> q_b_proj

Conversion therefore failed with:

    ValueError: Can not map tensor 'model.layers.3.attention.q_a_layernorm.weight'

Add the missing path, mirroring the existing deepseek2 MLA implementation:

  * constants.py       - add ATTN_Q_A / ATTN_Q_B / ATTN_Q_A_NORM to BAILINGMOE3
  * tensor_mapping.py  - map model.layers.{bid}.attention.q_{a,b}_proj and
                         q_a_layernorm
  * conversion         - emit attention.q_lora_rank when the config has it
  * bailingmoe3.cpp    - read n_lora_q; create the Q-LoRA tensors and build Q
                         through the bottleneck when q_lora_rank > 0

Everything is gated on q_lora_rank > 0. Ling-3.0-flash's config has no
q_lora_rank, the converter only emits the key when present, hparams.n_lora_q
defaults to 0, and get_key(..., required=false) leaves the target untouched when
the key is absent - so flash keeps taking the existing direct-Q branch.

The LoRA path produces the same shape as the direct projection, so the
nope/rope split, RoPE application and wk_b absorption downstream are unchanged.

* small mtp change

* bailingmoe3: support separate MTP GGUF and Q-LoRA MTP

* gguf: remove duplicate add_kda_gate_lower_bound definition

---------

Co-authored-by: bloomer <bloomer@booper.brushtail.me>
Co-authored-by: Dyluhn <dylanranejohnston1@gmail.com>
2026-08-17 09:49:49 +02:00
Georgi Gerganov 4197155add sync : ggml 2026-08-17 09:50:06 +03:00
Georgi Gerganov cea66f4c5a ggml : bump version to 0.20.1 (ggml/1587) 2026-08-17 09:50:06 +03:00
Fathi BoudraandGitHub 4695f001fe llama-bench: fix deprecation warnings missing trailing newline (#27179)
The log output does not append a newline, so the warning ran into the
next line printed on stdout, corrupting the benchmark table header.

Signed-off-by: Fathi Boudra <fathi.boudra@linaro.org>
2026-08-17 07:33:24 +03:00
TitaniumtownandGitHub f275595dd1 sycl: fix thread/block count in quantized cpy kernel launches (#27160)
Adjusts the thread/block count to be proportional to the size
of the quant, reducing under/over subscription.

Largest perf improvement is the q4_0 -> f32 path, with, on
a Arc 70, throughput goes from 20.21 GB/s to 158.19 GB/s

The rest of the quants are flat in performance uplift.
2026-08-17 07:32:16 +03:00
37a215c9e9 [SYCL] support OP OPT_STEP_ADAMW, OPT_STEP_SGD (#25268)
* fix conflict

* fix conflict of ops.md

* fix conflict of ops.md

* update the ops.md

---------

Co-authored-by: Neo Zhang Jianyu <jianyu.zhang@intel.com>
2026-08-17 07:31:29 +03:00
Daniel BeveniusandGitHub 4df29be4f4 ci : fix dry-run reporting in make-release job [no ci] (#27167)
This commit fixes the reporting in the make-release CI job when
--dry-run is used. It will currently incorrectly report that all checks
pass even if there are steps that fail.

Refs: https://github.com/ggml-org/llama.cpp/pull/26839#issuecomment-5306189828
2026-08-16 14:53:13 +02:00
3cb7ffb1a1 model : remove some ggml_concat (#27176)
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-08-16 14:12:55 +02:00
Xuan-Son NguyenandGitHub b94041a98e chat: refactor handling supports_string_content / supports_typed_content (#27130)
* better supports_string_content cap detect

* test: add "skip"

* messages_inp_normalizer
2026-08-16 12:45:33 +02:00
10bf611e53 llama : check LoRA tensor data is within file bounds (#27056)
* llama : check LoRA tensor data is within file bounds

* Update src/llama-adapter.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-08-16 09:38:01 +03:00
141 changed files with 2518 additions and 20293 deletions
+1
View File
@@ -44,6 +44,7 @@ jobs:
uses: actions/checkout@v6
with:
fetch-depth: 0
ssh-key: ${{ secrets.DEPLOY_KEY_RELEASE }}
- name: Determine source tag name
id: srctag
+17 -2
View File
@@ -3,6 +3,11 @@ name: Make Release
on:
workflow_dispatch:
inputs:
commit:
description: 'Commit SHA to release (empty = branch HEAD)'
required: false
default: ''
type: string
dry_run:
description: 'Dry run - validate without creating the tag'
required: true
@@ -22,12 +27,17 @@ jobs:
steps:
- name: Checkout
uses: actions/checkout@v6
with:
ssh-key: ${{ secrets.DEPLOY_KEY_RELEASE }}
ref: ${{ inputs.commit != '' && inputs.commit || github.ref_name }}
fetch-depth: 0
- name: Run release checks
id: checks
run: bash scripts/make-release-checks.sh ${{ github.event.inputs.dry_run == 'true' && '--dry-run' || '' }}
env:
GITHUB_REPOSITORY: ${{ github.repository }}
RELEASE_BRANCH: ${{ github.ref_name }}
- name: Create release tag
if: ${{ github.event.inputs.dry_run == 'false' }}
@@ -42,5 +52,10 @@ jobs:
- name: Dry run summary
if: ${{ github.event.inputs.dry_run == 'true' }}
run: |
echo "Dry run complete - all checks passed."
echo "Would have created tag: ${{ steps.checks.outputs.version }}"
if [[ "${{ steps.checks.outputs.checks_passed }}" == "true" ]]; then
echo "Dry run complete - all checks passed."
echo "Would have created tag: ${{ steps.checks.outputs.version }}"
else
echo "::error::Dry run found release check failures. A release tag would not be created."
exit 1
fi
+1
View File
@@ -1598,6 +1598,7 @@ jobs:
uses: actions/checkout@v6
with:
fetch-depth: 0
ssh-key: ${{ secrets.DEPLOY_KEY_RELEASE }}
- name: Determine tag name
id: tag
+1 -1
View File
@@ -5,7 +5,7 @@ include(CheckIncludeFileCXX)
### llama.cpp version
set(LLAMA_VERSION_MAJOR 0)
set(LLAMA_VERSION_MINOR 1)
set(LLAMA_VERSION_PATCH 0)
set(LLAMA_VERSION_PATCH 1)
set(LLAMA_VERSION_BASE "${LLAMA_VERSION_MAJOR}.${LLAMA_VERSION_MINOR}.${LLAMA_VERSION_PATCH}")
# whether this is a development/nightly build
+1 -1
View File
@@ -3362,7 +3362,7 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
{"--tools"}, "TOOL1,TOOL2,...",
"experimental: whether to enable built-in tools for AI agents - do not enable in untrusted environments (default: no tools)\n"
"specify \"all\" to enable all tools\n"
"available tools: read_file, file_glob_search, grep_search, exec_shell_command, write_file, edit_file, get_datetime, get_info\n"
"available tools: read_file, file_glob_search, grep_search, exec_shell_command, write_file, edit_file, get_info\n"
"note: for security reasons, this will limit --cors-origins to localhost by default",
[](common_params & params, const std::string & value) {
params.server_tools = parse_csv_row(value);
+8
View File
@@ -193,6 +193,14 @@ static std::vector<std::function<void(const common_chat_template & tmpl, autopar
LOG_DBG(ANSI_ORANGE "[Patch: Laguna]\n" ANSI_RESET);
}
},
// Bailing V3
[](const common_chat_template & tmpl, autoparser & analysis) -> void {
if (tmpl.src.find("Bailing V3 chat template") != std::string::npos) {
analysis.tools.arguments.value_suffix = trim_whitespace(analysis.tools.arguments.value_suffix);
analysis.tools.arguments.tolerate_intertag_whitespace = true;
LOG_DBG(ANSI_ORANGE "[Patch: Bailing V3]\n" ANSI_RESET);
}
},
});
+72 -27
View File
@@ -470,36 +470,80 @@ std::vector<common_chat_msg> common_chat_msgs_parse_oaicompat(const json & messa
return msgs;
}
struct messages_inp_normalizer {
const jinja::caps & caps;
messages_inp_normalizer(const jinja::caps & c) : caps(c) {}
// handle supports_string_content / supports_typed_content
// if string=true and array=false, convert array to string
// if string=false and array=true, convert string to array
// if both are true, do nothing
json normalize(const json & messages) {
bool only_string = caps.supports_string_content && !caps.supports_typed_content;
bool only_typed = !caps.supports_string_content && caps.supports_typed_content;
if ((!only_string && !only_typed) || !messages.is_array()) {
return messages;
}
json normalized = json::array();
for (const auto & msg : messages) {
json copy = msg;
auto it = copy.find("content");
if (it != copy.end()) {
if (only_typed && it->is_string()) {
*it = json::array({
json{
{"type", "text"},
{"text", it->get<std::string>()},
}
});
} else if (only_string && it->is_array()) {
*it = concat_content_parts(*it);
}
}
normalized.push_back(std::move(copy));
}
return normalized;
}
// join parts with newline, do not add newline before or after media markers
static std::string concat_content_parts(const json & parts) {
std::string text;
bool last_was_media_marker = false;
for (const auto & part : parts) {
std::string type = part.value("type", "");
bool add_new_line = true;
if (type == "text") {
add_new_line = !last_was_media_marker && !text.empty();
last_was_media_marker = false;
} else if (type == "media_marker") {
add_new_line = false;
last_was_media_marker = true;
} else {
LOG_WRN("Ignoring content part type: %s\n", type.c_str());
continue;
}
if (add_new_line) {
text += '\n';
}
text += part.value("text", "");
}
return text;
}
};
static json render_message_to_json(const std::vector<common_chat_msg> & msgs, const jinja::caps & c) {
if (!c.supports_string_content && !c.supports_typed_content) {
LOG_WRN("%s: Neither string content nor typed content is supported by the template. This is unexpected and may lead to issues.\n", __func__);
}
bool only_string_accepted = c.supports_string_content && !c.supports_typed_content;
bool only_typed_accepted = !c.supports_string_content && c.supports_typed_content;
json messages = json::array();
for (const auto & msg : msgs) {
if (only_string_accepted) {
json jmsg = msg.to_json_oaicompat(/* concat_typed_text= */ true);
messages.push_back(jmsg);
} else if (only_typed_accepted) {
json jmsg = msg.to_json_oaicompat(/* concat_typed_text= */ false);
if (jmsg.at("content").is_string()) {
jmsg["content"] = json::array({
json{
{"type", "text"},
{"text", jmsg.at("content").get<std::string>()},
}
});
}
messages.push_back(jmsg);
} else {
json jmsg = msg.to_json_oaicompat(/* concat_typed_text= */ false);
messages.push_back(jmsg);
}
messages.push_back(msg.to_json_oaicompat(/* concat_typed_text= */ false));
}
return messages;
return messages_inp_normalizer(c).normalize(messages);
}
// DEPRECATED: only used in tests
@@ -892,8 +936,11 @@ static std::string common_chat_template_direct_apply_impl(
const std::optional<json> & additional_context = std::nullopt) {
jinja::context ctx(tmpl.source());
// messages_override is already built for this template, do not touch its content parts
nlohmann::ordered_json inp = nlohmann::ordered_json{
{"messages", messages_override.has_value() ? *messages_override : inputs.messages},
{"messages", messages_override.has_value()
? *messages_override
: messages_inp_normalizer(tmpl.original_caps()).normalize(inputs.messages)},
{"bos_token", tmpl.bos_token()},
{"eos_token", tmpl.eos_token()},
{"enable_thinking", inputs.enable_thinking},
@@ -957,14 +1004,12 @@ static std::string common_chat_template_generation_prompt_impl(
const std::optional<json> & tools_override = std::nullopt,
const std::optional<json> & additional_context = std::nullopt) {
auto adjusted_messages = messages_override ? *messages_override : inputs.messages;
autoparser::generation_params params = inputs;
params.add_generation_prompt = false;
params.continue_final_message = COMMON_CHAT_CONTINUATION_NONE;
std::string no_gen_prompt = common_chat_template_direct_apply_impl(tmpl, params, adjusted_messages, tools_override, additional_context);
std::string no_gen_prompt = common_chat_template_direct_apply_impl(tmpl, params, messages_override, tools_override, additional_context);
params.add_generation_prompt = true;
std::string gen_prompt = common_chat_template_direct_apply_impl(tmpl, params, adjusted_messages, tools_override, additional_context);
std::string gen_prompt = common_chat_template_direct_apply_impl(tmpl, params, messages_override, tools_override, additional_context);
size_t prefix_len = 0;
size_t min_size = std::min(no_gen_prompt.size(), gen_prompt.size());
+10 -4
View File
@@ -23,7 +23,7 @@ void caps_apply_preserve_reasoning(jinja::context & ctx, bool enabled) {
ctx.set_val("preserve_thinking", mk_val<value_bool>(enabled));
ctx.set_val("clear_thinking", mk_val<value_bool>(!enabled));
ctx.set_val("truncate_history_thinking", mk_val<value_bool>(!enabled));
ctx.set_val("drop_thinking", mk_val<value_bool>(!enabled));
ctx.set_val("drop_thinking", mk_val<value_bool>(!enabled));
}
void caps_apply_reasoning_effort(jinja::context & ctx, const std::string & effort) {
@@ -117,6 +117,8 @@ caps caps_get(jinja::program & prog) {
JJ_DEBUG("%s\n", ">>> Running capability check: typed content");
static const std::string content_marker = "STRING_MARKER";
// case: typed content support
caps_try_execute(
prog,
@@ -125,22 +127,26 @@ caps caps_get(jinja::program & prog) {
return json::array({
{
{"role", "user"},
{"content", "content"}
{"content", content_marker}
}
});
},
nullptr, // ctx_fn
nullptr, // tools_fn
[&](context &, bool success, value & messages, value &, const std::string &) {
[&](context &, bool success, value & messages, value &, const std::string & rendered) {
auto & content = messages->at(0)->at("content");
caps_print_stats(content, "messages[0].content");
if (has_op(content, "selectattr") || has_op(content, "array_access")) {
bool used_as_array = has_op(content, "selectattr") || has_op(content, "array_access");
if (used_as_array) {
// accessed as an array
result.supports_typed_content = true;
}
if (!success) {
// failed to execute with content as string
result.supports_string_content = false;
} else if (used_as_array && rendered.find(content_marker) == std::string::npos) {
// edge case: string may be accessed for checking, but does not appear in the output
result.supports_string_content = false;
}
}
);
+1
View File
@@ -27,6 +27,7 @@ TEXT_MODEL_MAP: dict[str, str] = {
"BaichuanForCausalLM": "baichuan",
"BailingMoeForCausalLM": "bailingmoe",
"BailingMoeV2ForCausalLM": "bailingmoe",
"BailingMoeV3ForCausalLM": "bailingmoe3",
"BambaForCausalLM": "granite",
"BertForMaskedLM": "bert",
"BertForSequenceClassification": "bert",
+1
View File
@@ -13,6 +13,7 @@ from .llama import LlamaModel
@ModelBase.register("AfmoeForCausalLM")
@ModelBase.example("arcee-ai/Trinity-Large-Thinking")
class AfmoeModel(LlamaModel):
model_arch = gguf.MODEL_ARCH.AFMOE
+1
View File
@@ -16,6 +16,7 @@ from .llama import LlamaModel
@ModelBase.register("ArcticForCausalLM")
@ModelBase.example("Snowflake/snowflake-arctic-instruct")
class ArcticModel(TextModel):
model_arch = gguf.MODEL_ARCH.ARCTIC
+1
View File
@@ -9,6 +9,7 @@ from .base import ModelBase, TextModel, gguf, logger
@ModelBase.register("BaichuanForCausalLM", "BaiChuanForCausalLM")
@ModelBase.example("baichuan-inc/Baichuan2-7B-Chat", "baichuan-inc/Baichuan-7B")
class BaichuanModel(TextModel):
model_arch = gguf.MODEL_ARCH.BAICHUAN
+3
View File
@@ -11,6 +11,7 @@ from .base import ModelBase, TextModel, gguf
@ModelBase.register("BailingMoeForCausalLM")
@ModelBase.example("inclusionAI/Ling-lite")
class BailingMoeModel(TextModel):
model_arch = gguf.MODEL_ARCH.BAILINGMOE
@@ -108,6 +109,7 @@ class BailingMoeModel(TextModel):
@ModelBase.register("BailingMoeV2ForCausalLM")
@ModelBase.example("inclusionAI/Ling-mini-2.0")
class BailingMoeV2Model(TextModel):
model_arch = gguf.MODEL_ARCH.BAILINGMOE2
@@ -189,6 +191,7 @@ class BailingMoeV2Model(TextModel):
@ModelBase.register("SarvamMoEForCausalLM", "modeling_sarvam_moe.SarvamMoEForCausalLM")
@ModelBase.example("sarvamai/sarvam-30b")
class SarvamMoEModel(BailingMoeV2Model):
model_arch = gguf.MODEL_ARCH.BAILINGMOE2
# Sarvam-MoE shares the BailingMoeV2 architecture; only differences:
+193
View File
@@ -0,0 +1,193 @@
from __future__ import annotations
import re
from typing import Callable, Iterable, TYPE_CHECKING
import torch
if TYPE_CHECKING:
from torch import Tensor
from .base import ModelBase, TextModel, gguf
@ModelBase.register("BailingMoeV3ForCausalLM")
@ModelBase.example("inclusionAI/Ling-3.0-tiny", "inclusionAI/Ling-3.0-flash")
class BailingMoeV3Model(TextModel):
model_arch = gguf.MODEL_ARCH.BAILINGMOE3
supports_mtp_export = True
_experts: list[dict[str, Tensor]] | None = None
_main_layers: int | None = None
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
nextn_layers = self.hparams.get("num_nextn_predict_layers", 0) or 0
if self.no_mtp:
nextn_layers = 0
self.block_count = self.hparams["num_hidden_layers"] + nextn_layers
self.tensor_map = gguf.get_tensor_name_map(self.model_arch, self.block_count)
def index_tensors(self, remote_hf_model_id: str | None = None):
type(self)._main_layers = self.hparams["num_hidden_layers"]
return super().index_tensors(remote_hf_model_id=remote_hf_model_id)
def set_vocab(self):
self._set_vocab_gpt2()
def is_full_attention(self, bid: int) -> bool:
n_layer = self.hparams["num_hidden_layers"]
layer_group_size = self.hparams["layer_group_size"]
return bid >= n_layer or (bid + 1) % layer_group_size == 0 or bid >= n_layer // layer_group_size * layer_group_size
def set_gguf_parameters(self):
if not self.hparams.get("no_kda_lora", False):
raise ValueError("BailingMoeV3 KDA LoRA projections are not supported")
if not self.hparams.get("kda_safe_gate", False):
raise ValueError("BailingMoeV3 non-safe KDA gates are not supported")
if self.hparams.get("gated_attention_proj_granularity_type") != "head_wise":
raise ValueError("BailingMoeV3 requires head-wise attention gates")
self.hparams["num_key_value_heads"] = 1
super().set_gguf_parameters()
n_head_kv = [1 if self.is_full_attention(il) else 0 for il in range(self.block_count)]
self.gguf_writer.add_head_count_kv(n_head_kv)
self.gguf_writer.add_vocab_size(self.hparams["vocab_size"])
self.gguf_writer.add_ssm_conv_kernel(self.hparams["short_conv_kernel_size"])
self.gguf_writer.add_kda_head_dim(self.hparams["head_dim"])
self.gguf_writer.add_kda_safe_gate(self.hparams["kda_safe_gate"])
self.gguf_writer.add_kda_gate_lower_bound(self.hparams["kda_lower_bound"])
kv_lora_rank = self.hparams["kv_lora_rank"]
qk_nope_head_dim = self.hparams["qk_nope_head_dim"]
qk_rope_head_dim = self.hparams["qk_rope_head_dim"]
if (q_lora_rank := self.hparams.get("q_lora_rank")) is not None:
self.gguf_writer.add_q_lora_rank(q_lora_rank)
self.gguf_writer.add_kv_lora_rank(kv_lora_rank)
self.gguf_writer.add_rope_dimension_count(qk_rope_head_dim)
self.gguf_writer.add_key_length(kv_lora_rank + qk_rope_head_dim)
self.gguf_writer.add_key_length_mla(qk_nope_head_dim + qk_rope_head_dim)
self.gguf_writer.add_value_length_mla(self.hparams["v_head_dim"])
self.gguf_writer.add_expert_feed_forward_length(self.hparams["moe_intermediate_size"])
self.gguf_writer.add_expert_shared_feed_forward_length(self.hparams["moe_shared_expert_intermediate_size"])
self.gguf_writer.add_expert_shared_count(self.hparams["num_shared_experts"])
self.gguf_writer.add_leading_dense_block_count(self.hparams["first_k_dense_replace"])
self.gguf_writer.add_expert_weights_scale(self.hparams["routed_scaling_factor"])
self.gguf_writer.add_expert_weights_norm(self.hparams["norm_topk_prob"])
def clamp_limits(key: str) -> list[float] | None:
values = self.hparams.get(key)
if values is None:
return None
values = [0.0 if value is None else float(value) for value in values[:self.block_count]]
return values + [0.0] * (self.block_count - len(values))
if (values := clamp_limits("expert_swiglu_limit_list")) is not None:
self.gguf_writer.add_swiglu_clamp_exp(values)
if (values := clamp_limits("share_expert_swiglu_limit_list")) is not None:
self.gguf_writer.add_swiglu_clamp_shexp(values)
if not self.no_mtp and (nextn_layers := self.hparams.get("num_nextn_predict_layers", 0)):
self.gguf_writer.add_nextn_predict_layers(nextn_layers)
def prepare_metadata(self, vocab_only: bool):
from_dir = self.fname_out.is_dir()
super().prepare_metadata(vocab_only=vocab_only)
if not self.mtp_only or not from_dir:
return
output_type: str = self.ftype.name.partition("_")[2]
fname_default: str = gguf.naming_convention(
self.metadata.name, self.metadata.basename, self.metadata.finetune,
self.metadata.version, size_label=None, output_type=output_type, model_type=None)
self.fname_out = self.fname_out.parent / f"mtp-{fname_default}.gguf"
@classmethod
def filter_tensors(cls, item: tuple[str, Callable[[], Tensor]]) -> tuple[str, Callable[[], Tensor]] | None:
name, gen = item
if name.endswith(".expert_bias"):
name += ".bias"
if cls._main_layers is None:
return super().filter_tensors((name, gen))
m = re.match(r"model\.layers\.(\d+)\.", name)
is_mtp = m is not None and int(m.group(1)) >= cls._main_layers
if is_mtp and cls.no_mtp:
return None
if cls.mtp_only and not is_mtp and name not in (
"model.word_embeddings.weight", "model.norm.weight", "lm_head.weight",
):
return None
return super().filter_tensors((name, gen))
def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None) -> Iterable[tuple[str, Tensor]]:
if name.endswith((".q_conv1d.weight", ".k_conv1d.weight", ".v_conv1d.weight")) and data_torch.ndim in (2, 3):
d_inner = data_torch.shape[0]
d_conv = data_torch.shape[-1]
data_torch = data_torch.reshape(1, d_inner, 1, d_conv)
if name.endswith(".A_log"):
data_torch = torch.exp(data_torch).reshape(-1, 1)
if name.endswith(".dt_bias"):
name = name.rpartition(".dt_bias")[0] + ".dt_proj.bias"
if name.endswith(".attention.f_proj.weight"):
assert bid is not None
if self.is_full_attention(bid):
raise ValueError(f"unexpected f_proj on full-attention layer {bid}")
name = self.format_tensor_name(gguf.MODEL_TENSOR.SSM_F_A, bid)
if name.endswith(".attention.g_proj.weight"):
assert bid is not None
tensor = gguf.MODEL_TENSOR.ATTN_GATE if self.is_full_attention(bid) else gguf.MODEL_TENSOR.SSM_G_A
name = self.format_tensor_name(tensor, bid)
if ".mlp.experts." in name:
n_experts = self.hparams["num_experts"]
assert bid is not None
if self._experts is None:
self._experts = [{} for _ in range(self.block_count)]
self._experts[bid][name] = data_torch
if len(self._experts[bid]) >= n_experts * 3:
for weight_name in ("down_proj", "gate_proj", "up_proj"):
tensors = []
for expert_id in range(n_experts):
expert_name = f"model.layers.{bid}.mlp.experts.{expert_id}.{weight_name}.weight"
tensors.append(self._experts[bid].pop(expert_name))
merged_name = f"model.layers.{bid}.mlp.experts.{weight_name}.weight"
yield from super().modify_tensors(torch.stack(tensors, dim=0), merged_name, bid)
return
if name.endswith(".attention.kv_b_proj.weight"):
assert bid is not None
n_head = self.hparams["num_attention_heads"]
v_head_dim = self.hparams["v_head_dim"]
qk_nope_head_dim = self.hparams["qk_nope_head_dim"]
assert data_torch.shape[0] == n_head * (v_head_dim + qk_nope_head_dim)
kv_b = data_torch.view(n_head, v_head_dim + qk_nope_head_dim, data_torch.shape[-1])
k_b, v_b = torch.split(kv_b, [qk_nope_head_dim, v_head_dim], dim=1)
name_k = self.format_tensor_name(gguf.MODEL_TENSOR.ATTN_K_B, bid)
name_v = self.format_tensor_name(gguf.MODEL_TENSOR.ATTN_V_B, bid)
yield from super().modify_tensors(k_b.transpose(1, 2), name_k, bid)
yield from super().modify_tensors(v_b, name_v, bid)
return
yield from super().modify_tensors(data_torch, name, bid)
def prepare_tensors(self):
super().prepare_tensors()
if self._experts is not None:
experts = [name for layer in self._experts for name in layer]
if experts:
raise ValueError(f"Unprocessed experts: {experts}")
+8
View File
@@ -1149,6 +1149,14 @@ class ModelBase:
return modelcls
return func
@classmethod
def example(cls, *hf_repos: str) -> Callable[[AnyModel], AnyModel]:
del hf_repos # unused
def func(modelcls: AnyModel) -> AnyModel:
return modelcls
return func
@classmethod
def print_registered_models(cls):
for model_type, model_classes in cls._model_classes.items():
+9
View File
@@ -15,6 +15,7 @@ from .base import ModelBase, SentencePieceTokenTypes, TextModel, gguf, logger
@ModelBase.register("BertModel", "BertForMaskedLM", "CamembertModel", "BertForSequenceClassification")
@ModelBase.example("BAAI/bge-small-en-v1.5", "dangvantuan/sentence-camembert-base")
class BertModel(TextModel):
model_arch = gguf.MODEL_ARCH.BERT
@@ -240,6 +241,7 @@ class BertModel(TextModel):
@ModelBase.register("DistilBertModel", "DistilBertForMaskedLM", "DistilBertForSequenceClassification")
@ModelBase.example("distilbert/distilbert-base-uncased")
class DistilBertModel(BertModel):
model_arch = gguf.MODEL_ARCH.BERT
@@ -263,6 +265,7 @@ class DistilBertModel(BertModel):
@ModelBase.register("RobertaModel", "RobertaForSequenceClassification")
@ModelBase.example("sentence-transformers/stsb-roberta-base")
class RobertaModel(BertModel):
model_arch = gguf.MODEL_ARCH.BERT
@@ -312,6 +315,7 @@ class RobertaModel(BertModel):
@ModelBase.register("NomicBertModel")
@ModelBase.example("nomic-ai/nomic-embed-text-v1.5")
class NomicBertModel(BertModel):
model_arch = gguf.MODEL_ARCH.BERT
@@ -400,6 +404,7 @@ class NomicBertModel(BertModel):
@ModelBase.register("NeoBERT", "NeoBERTLMHead", "NeoBERTForSequenceClassification")
@ModelBase.example("chandar-lab/NeoBERT")
class NeoBert(BertModel):
model_arch = gguf.MODEL_ARCH.NEO_BERT
@@ -431,6 +436,7 @@ class NeoBert(BertModel):
@ModelBase.register("EuroBertModel", "JinaEmbeddingsV5Model")
@ModelBase.example("hf-tiny-v2/tiny-random-EuroBertModel", "jinaai/jina-embeddings-v5-text-nano")
class EuroBertModel(TextModel):
model_arch = gguf.MODEL_ARCH.EUROBERT
@@ -459,6 +465,7 @@ class EuroBertModel(TextModel):
@ModelBase.register("XLMRobertaModel", "XLMRobertaForSequenceClassification")
@ModelBase.example("BAAI/bge-m3")
class XLMRobertaModel(BertModel):
model_arch = gguf.MODEL_ARCH.BERT
_lora_files = {}
@@ -561,6 +568,7 @@ class XLMRobertaModel(BertModel):
@ModelBase.register("JinaBertModel", "JinaBertForMaskedLM")
@ModelBase.example("jinaai/jina-embeddings-v2-base-en")
class JinaBertV2Model(BertModel):
model_arch = gguf.MODEL_ARCH.JINA_BERT_V2
@@ -588,6 +596,7 @@ class JinaBertV2Model(BertModel):
@ModelBase.register("ModernBertModel", "ModernBertForMaskedLM", "ModernBertForSequenceClassification")
@ModelBase.example("answerdotai/ModernBERT-base")
class ModernBertModel(BertModel):
model_arch = gguf.MODEL_ARCH.MODERN_BERT
+1
View File
@@ -9,6 +9,7 @@ from .base import ModelBase, TextModel, gguf
@ModelBase.register("BitnetForCausalLM", "BitNetForCausalLM")
@ModelBase.example("microsoft/bitnet-b1.58-2B-4T")
class BitnetModel(TextModel):
model_arch = gguf.MODEL_ARCH.BITNET
+1
View File
@@ -13,6 +13,7 @@ from .base import ModelBase, TextModel, gguf, logger
@ModelBase.register("BloomForCausalLM", "BloomModel")
@ModelBase.example("bigscience/bloom-560m")
class BloomModel(TextModel):
model_arch = gguf.MODEL_ARCH.BLOOM
+2
View File
@@ -12,6 +12,8 @@ from .llama import LlamaModel
@ModelBase.register("ChameleonForConditionalGeneration")
@ModelBase.register("ChameleonForCausalLM") # obsolete
# [TAG_HF_EXAMPLE_GATED] facebook/chameleon-7b is gated
# [TAG_HF_EXAMPLE_MISSING]
class ChameleonModel(TextModel):
model_arch = gguf.MODEL_ARCH.CHAMELEON
+1
View File
@@ -9,6 +9,7 @@ from .base import ModelBase, SentencePieceTokenTypes, TextModel, gguf
@ModelBase.register("GlmForCausalLM", "ChatGLMModel", "ChatGLMForConditionalGeneration")
@ModelBase.example("THUDM/chatglm3-6b", "zai-org/glm-4-9b-chat-hf")
class ChatGLMModel(TextModel):
model_arch = gguf.MODEL_ARCH.CHATGLM
+1
View File
@@ -4,6 +4,7 @@ from .base import ModelBase, TextModel, gguf
@ModelBase.register("CodeShellForCausalLM")
@ModelBase.example("WisdomShell/CodeShell-7B")
class CodeShellModel(TextModel):
model_arch = gguf.MODEL_ARCH.CODESHELL
+2
View File
@@ -11,6 +11,7 @@ from .llama import LlamaModel
@ModelBase.register("CogVLMForCausalLM")
@ModelBase.example("THUDM/cogvlm2-llama3-chat-19B", "THUDM/cogvlm-chat-hf")
class CogVLMVisionModel(MmprojModel):
def set_gguf_parameters(self):
@@ -29,5 +30,6 @@ class CogVLMVisionModel(MmprojModel):
@ModelBase.register("CogVLMForCausalLM")
@ModelBase.example("THUDM/cogvlm2-llama3-chat-19B", "THUDM/cogvlm-chat-hf")
class CogVLMModel(LlamaModel):
model_arch = gguf.MODEL_ARCH.COGVLM
+5
View File
@@ -12,6 +12,8 @@ from .base import ModelBase, TextModel, gguf, logger
@ModelBase.register("CohereForCausalLM")
# [TAG_HF_EXAMPLE_GATED] CohereLabs/c4ai-command-r-v01 is gated
# [TAG_HF_EXAMPLE_MISSING]
class CommandR2Model(TextModel):
model_arch = gguf.MODEL_ARCH.COMMAND_R
@@ -30,6 +32,8 @@ class CommandR2Model(TextModel):
@ModelBase.register("Cohere2ForCausalLM")
# [TAG_HF_EXAMPLE_GATED] CohereLabs/c4ai-command-r7b-12-2024 is gated
@ModelBase.example("hf-tiny-v2/tiny-random-Cohere2ForCausalLM")
class Cohere2Model(TextModel):
model_arch = gguf.MODEL_ARCH.COHERE2
@@ -59,6 +63,7 @@ class Cohere2Model(TextModel):
@ModelBase.register("Cohere2MoeForCausalLM")
@ModelBase.example("CohereLabs/North-Mini-Code-1.0")
class Cohere2MoeModel(TextModel):
model_arch = gguf.MODEL_ARCH.COHERE2MOE
_n_main_layers: int | None = None
+1
View File
@@ -9,6 +9,7 @@ from .base import ModelBase, TextModel, gguf, logger
@ModelBase.register("DbrxForCausalLM")
@ModelBase.example("alpindale/dbrx-instruct")
class DbrxModel(TextModel):
model_arch = gguf.MODEL_ARCH.DBRX
+1
View File
@@ -13,6 +13,7 @@ from .base import ModelBase, TextModel, gguf
@ModelBase.register("DeciLMForCausalLM")
@ModelBase.example("nvidia/Llama-3_1-Nemotron-51B-Instruct", "Deci/DeciLM-7B")
class DeciModel(TextModel):
model_arch = gguf.MODEL_ARCH.DECI
+8
View File
@@ -18,6 +18,7 @@ from .qwen import QwenModel
@ModelBase.register("DeepseekOCRForCausalLM")
@ModelBase.example("deepseek-ai/DeepSeek-OCR")
class DeepseekOCRVisionModel(MmprojModel):
# HF dynamic_preprocess() max_num, which differs per model
preproc_max_tiles = 9
@@ -100,11 +101,13 @@ class DeepseekOCRVisionModel(MmprojModel):
@ModelBase.register("UnlimitedOCRForCausalLM")
@ModelBase.example("baidu/Unlimited-OCR")
class UnlimitedOCRVisionModel(DeepseekOCRVisionModel):
preproc_max_tiles = 32
@ModelBase.register("DeepseekOCR2ForCausalLM")
@ModelBase.example("deepseek-ai/DeepSeek-OCR-2")
class DeepseekOCR2VisionModel(DeepseekOCRVisionModel):
preproc_max_tiles = 6
@@ -134,6 +137,7 @@ class DeepseekOCR2VisionModel(DeepseekOCRVisionModel):
@ModelBase.register("DeepseekForCausalLM")
@ModelBase.example("deepseek-ai/deepseek-moe-16b-chat")
class DeepseekModel(TextModel):
model_arch = gguf.MODEL_ARCH.DEEPSEEK
@@ -228,6 +232,7 @@ class DeepseekModel(TextModel):
"YoutuForCausalLM",
"YoutuVLForConditionalGeneration",
)
@ModelBase.example("deepseek-ai/DeepSeek-V2-Lite", "deepseek-ai/DeepSeek-V3")
class DeepseekV2Model(TextModel):
model_arch = gguf.MODEL_ARCH.DEEPSEEK2
@@ -457,6 +462,7 @@ class DeepseekV2Model(TextModel):
@ModelBase.register("DeepseekV32ForCausalLM")
@ModelBase.example("deepseek-ai/DeepSeek-V3.2-Exp")
class DeepseekV32Model(DeepseekV2Model):
model_arch = gguf.MODEL_ARCH.DEEPSEEK32
skip_mtp = False
@@ -517,6 +523,7 @@ class DeepseekV32Model(DeepseekV2Model):
@ModelBase.register("DeepseekV4ForCausalLM")
@ModelBase.example("deepseek-ai/DeepSeek-V4-Flash-Base")
class DeepseekV4Model(TextModel):
model_arch = gguf.MODEL_ARCH.DEEPSEEK4
supports_mtp_export = True
@@ -911,6 +918,7 @@ class DeepseekV4Model(TextModel):
@ModelBase.register("DeepseekV4DSparkModel")
@ModelBase.example("deepseek-ai/DeepSeek-V4-Flash-DSpark")
class DeepseekV4DSparkModel(DeepseekV4Model):
model_arch = gguf.MODEL_ARCH.DFLASH
+1
View File
@@ -11,6 +11,7 @@ from .qwen import Qwen2MoeModel
@ModelBase.register("Dots1ForCausalLM")
@ModelBase.example("rednote-hilab/dots.llm1.inst")
class Dots1Model(Qwen2MoeModel):
model_arch = gguf.MODEL_ARCH.DOTS1
+1
View File
@@ -9,6 +9,7 @@ from .base import MmprojModel, ModelBase, gguf
@ModelBase.register("DotsOCRForCausalLM")
@ModelBase.example("rednote-hilab/dots.ocr")
class DotsOCRVisionModel(MmprojModel):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
+1
View File
@@ -9,6 +9,7 @@ from .base import ModelBase, TextModel, gguf
@ModelBase.register("DreamModel")
@ModelBase.example("Dream-org/Dream-v0-Instruct-7B")
class DreamModel(TextModel):
model_arch = gguf.MODEL_ARCH.DREAM
+4
View File
@@ -15,6 +15,7 @@ from .base import MmprojModel, ModelBase, TextModel, gguf
@ModelBase.register("Ernie4_5_ForCausalLM", "Ernie4_5ForCausalLM")
@ModelBase.example("baidu/ERNIE-4.5-0.3B-PT")
class Ernie4_5Model(TextModel):
model_arch = gguf.MODEL_ARCH.ERNIE4_5
@@ -73,6 +74,7 @@ class Ernie4_5Model(TextModel):
@ModelBase.register("Ernie4_5_MoeForCausalLM")
@ModelBase.example("baidu/ERNIE-4.5-21B-A3B-PT")
class Ernie4_5MoeModel(Ernie4_5Model):
model_arch = gguf.MODEL_ARCH.ERNIE4_5_MOE
_experts: list[dict[str, Tensor]] | None = None
@@ -156,11 +158,13 @@ class Ernie4_5MoeModel(Ernie4_5Model):
@ModelBase.register("PaddleOCRVLForConditionalGeneration")
@ModelBase.example("PaddlePaddle/PaddleOCR-VL")
class PaddleOCRModel(Ernie4_5Model):
model_arch = gguf.MODEL_ARCH.PADDLEOCR
@ModelBase.register("PaddleOCRVisionModel")
@ModelBase.example("PaddlePaddle/PaddleOCR-VL")
class PaddleOCRVisionModel(MmprojModel):
# PaddleOCR-VL uses a modified version of Siglip
min_pixels: int = 0
+5
View File
@@ -15,6 +15,7 @@ from .qwenvl import Qwen2VLVisionModel
@ModelBase.register("ExaoneForCausalLM")
@ModelBase.example("LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct")
class ExaoneModel(TextModel):
model_arch = gguf.MODEL_ARCH.EXAONE
@@ -60,6 +61,7 @@ class ExaoneModel(TextModel):
@ModelBase.register("Exaone4ForCausalLM")
@ModelBase.example("LGAI-EXAONE/EXAONE-4.0-32B")
class Exaone4Model(TextModel):
model_arch = gguf.MODEL_ARCH.EXAONE4
@@ -126,6 +128,7 @@ class Exaone4Model(TextModel):
# note: transformers >= 5.1 renamed the class to "ExaoneMoeForCausalLM" (lowercase 'e'),
# so accept both spellings - LG AI have updated the configs of already-released models
@ModelBase.register("ExaoneMoEForCausalLM", "ExaoneMoeForCausalLM")
@ModelBase.example("LGAI-EXAONE/K-EXAONE-236B-A23B")
class ExaoneMoEModel(Exaone4Model):
model_arch = gguf.MODEL_ARCH.EXAONE_MOE
@@ -214,6 +217,7 @@ class ExaoneMoEModel(Exaone4Model):
@ModelBase.register("Exaone4_5_ForConditionalGeneration")
@ModelBase.example("LGAI-EXAONE/EXAONE-4.5-33B")
class Exaone4_5_TextModel(Exaone4Model):
"""Text tower of EXAONE 4.5; Tensors match EXAONE4"""
@@ -267,6 +271,7 @@ class Exaone4_5_TextModel(Exaone4Model):
@ModelBase.register("Exaone4_5_ForConditionalGeneration")
@ModelBase.example("LGAI-EXAONE/EXAONE-4.5-33B")
class Exaone4_5VisionModel(Qwen2VLVisionModel):
"""Vision tower for EXAONE 4.5; Qwen2-VL-style ViT (GQA) + patch merger"""
+1
View File
@@ -11,6 +11,7 @@ from .base import ModelBase, TextModel, gguf
@ModelBase.register("FalconForCausalLM", "RWForCausalLM")
@ModelBase.example("tiiuae/falcon-7b")
class FalconModel(TextModel):
model_arch = gguf.MODEL_ARCH.FALCON
+1
View File
@@ -12,6 +12,7 @@ from .mamba import Mamba2Model
@ModelBase.register("FalconH1ForCausalLM")
@ModelBase.example("tiiuae/Falcon-H1-0.5B-Base")
class FalconH1Model(Mamba2Model):
model_arch = gguf.MODEL_ARCH.FALCON_H1
+19
View File
@@ -14,6 +14,8 @@ from .base import MmprojModel, ModelBase, TextModel, gguf, logger
@ModelBase.register("GemmaForCausalLM")
# [TAG_HF_EXAMPLE_GATED] google/gemma-2b is gated
@ModelBase.example("trl-internal-testing/tiny-GemmaForCausalLM")
class GemmaModel(TextModel):
model_arch = gguf.MODEL_ARCH.GEMMA
@@ -68,6 +70,8 @@ class GemmaModel(TextModel):
@ModelBase.register("Gemma2ForCausalLM")
# [TAG_HF_EXAMPLE_GATED] google/gemma-2-9b-it is gated
@ModelBase.example("trl-internal-testing/tiny-Gemma2ForCausalLM")
class Gemma2Model(TextModel):
model_arch = gguf.MODEL_ARCH.GEMMA2
@@ -118,6 +122,8 @@ class Gemma2Model(TextModel):
@ModelBase.register("Gemma3ForCausalLM", "Gemma3ForConditionalGeneration")
# [TAG_HF_EXAMPLE_GATED] google/gemma-3-4b-it is gated
@ModelBase.example("trl-internal-testing/tiny-Gemma3ForConditionalGeneration", "hf-tiny-v2/tiny-random-Gemma3ForCausalLM")
class Gemma3Model(TextModel):
model_arch = gguf.MODEL_ARCH.GEMMA3
@@ -174,6 +180,8 @@ class Gemma3Model(TextModel):
@ModelBase.register("Gemma3TextModel")
# [TAG_HF_EXAMPLE_GATED] google/embeddinggemma-300m is gated
@ModelBase.example("hf-tiny-v2/tiny-random-Gemma3TextModel")
class EmbeddingGemma(Gemma3Model):
model_arch = gguf.MODEL_ARCH.GEMMA_EMBEDDING
module_paths = []
@@ -248,6 +256,8 @@ class EmbeddingGemma(Gemma3Model):
@ModelBase.register("Gemma3ForConditionalGeneration")
# [TAG_HF_EXAMPLE_GATED] google/gemma-3-4b-it is gated
@ModelBase.example("trl-internal-testing/tiny-Gemma3ForConditionalGeneration")
class Gemma3VisionModel(MmprojModel):
def set_gguf_parameters(self):
super().set_gguf_parameters()
@@ -352,6 +362,8 @@ class ConformerAudioModel(MmprojModel):
@ModelBase.register("Gemma3nForConditionalGeneration")
# [TAG_HF_EXAMPLE_GATED] google/gemma-3n-E2B-it is gated
@ModelBase.example("hf-tiny-v2/tiny-random-Gemma3nForConditionalGeneration")
class Gemma3nVisionAudioModel(ConformerAudioModel):
has_audio_encoder = True
has_vision_encoder = True
@@ -471,6 +483,8 @@ class Gemma3nVisionAudioModel(ConformerAudioModel):
@ModelBase.register("Gemma3nForCausalLM", "Gemma3nForConditionalGeneration")
# [TAG_HF_EXAMPLE_GATED] google/gemma-3n-E2B-it is gated
@ModelBase.example("hf-tiny-v2/tiny-random-Gemma3nForConditionalGeneration")
class Gemma3NModel(Gemma3Model):
model_arch = gguf.MODEL_ARCH.GEMMA3N
@@ -615,6 +629,7 @@ class Gemma3NModel(Gemma3Model):
@ModelBase.register("Gemma4ForConditionalGeneration", "Gemma4ForCausalLM")
@ModelBase.example("google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-E2B-it")
class Gemma4Model(Gemma3Model):
model_arch = gguf.MODEL_ARCH.GEMMA4
@@ -795,6 +810,7 @@ class Gemma4Model(Gemma3Model):
@ModelBase.register("Gemma4UnifiedForConditionalGeneration")
@ModelBase.example("hf-tiny-v2/tiny-random-Gemma4UnifiedForConditionalGeneration")
class Gemma4UnifiedModel(Gemma4Model):
model_arch = gguf.MODEL_ARCH.GEMMA4
@@ -815,6 +831,7 @@ class Gemma4UnifiedModel(Gemma4Model):
@ModelBase.register("Gemma4AssistantForCausalLM", "Gemma4UnifiedAssistantForCausalLM")
@ModelBase.example("google/gemma-4-31B-it-assistant", "google/gemma-4-26B-A4B-it-assistant", "google/gemma-4-E2B-it-assistant")
class Gemma4AssistantModel(Gemma4Model):
model_arch = gguf.MODEL_ARCH.GEMMA4_ASSISTANT
@@ -835,6 +852,7 @@ class Gemma4AssistantModel(Gemma4Model):
@ModelBase.register("Gemma4ForConditionalGeneration")
@ModelBase.example("google/gemma-4-31B-it", "google/gemma-4-26B-A4B-it", "google/gemma-4-E2B-it")
class Gemma4VisionAudioModel(MmprojModel):
has_audio_encoder = True
has_vision_encoder = True
@@ -913,6 +931,7 @@ class Gemma4VisionAudioModel(MmprojModel):
@ModelBase.register("Gemma4UnifiedForConditionalGeneration")
@ModelBase.example("hf-tiny-v2/tiny-random-Gemma4UnifiedForConditionalGeneration")
class Gemma4UnifiedVisionAudioModel(Gemma4VisionAudioModel):
has_audio_encoder = True
has_vision_encoder = True
+6
View File
@@ -15,6 +15,7 @@ from .deepseek import DeepseekV2Model
@ModelBase.register("Glm4ForCausalLM", "Glm4vForConditionalGeneration")
@ModelBase.example("zai-org/GLM-4-9B-0414")
class Glm4Model(TextModel):
model_arch = gguf.MODEL_ARCH.GLM4
use_mrope = False
@@ -86,6 +87,7 @@ class Glm4Model(TextModel):
@ModelBase.register("GlmOcrForConditionalGeneration")
@ModelBase.example("zai-org/GLM-OCR")
class GlmOCRModel(Glm4Model):
model_arch = gguf.MODEL_ARCH.GLM4
use_mrope = False
@@ -107,6 +109,7 @@ class GlmOCRModel(Glm4Model):
@ModelBase.register("Glm4MoeForCausalLM", "Glm4vMoeForConditionalGeneration")
@ModelBase.example("zai-org/GLM-4.5-Air")
class Glm4MoeModel(TextModel):
model_arch = gguf.MODEL_ARCH.GLM4_MOE
@@ -204,6 +207,7 @@ class Glm4MoeModel(TextModel):
@ModelBase.register("Glm4MoeLiteForCausalLM")
@ModelBase.example("zai-org/GLM-4.7-Flash")
class Glm4MoeLiteModel(DeepseekV2Model):
model_arch = gguf.MODEL_ARCH.DEEPSEEK2
skip_mtp = False
@@ -272,6 +276,7 @@ class Glm4MoeLiteModel(DeepseekV2Model):
@ModelBase.register("GlmMoeDsaForCausalLM")
@ModelBase.example("zai-org/GLM-5.2")
class GlmMoeDsaModel(DeepseekV2Model):
model_arch = gguf.MODEL_ARCH.GLM_DSA
skip_mtp = False
@@ -340,6 +345,7 @@ class GlmMoeDsaModel(DeepseekV2Model):
@ModelBase.register("SolarOpenForCausalLM")
@ModelBase.example("upstage/Solar-Open-100B")
class SolarOpenModel(Glm4MoeModel):
model_arch = gguf.MODEL_ARCH.GLM4_MOE
+2
View File
@@ -11,6 +11,7 @@ from .base import ModelBase, TextModel, gguf, logger
@ModelBase.register("GPT2LMHeadModel")
@ModelBase.example("openai-community/gpt2")
class GPT2Model(TextModel):
model_arch = gguf.MODEL_ARCH.GPT2
@@ -38,6 +39,7 @@ class GPT2Model(TextModel):
@ModelBase.register("RuGPT3XLForCausalLM")
@ModelBase.example("evilfreelancer/ruGPT3XL")
class RuGPT3XLModel(TextModel):
model_arch = gguf.MODEL_ARCH.GPT2
+1
View File
@@ -11,6 +11,7 @@ from .base import ModelBase, TextModel, gguf, logger
@ModelBase.register("GptOssForCausalLM")
@ModelBase.example("openai/gpt-oss-20b")
class GptOssModel(TextModel):
model_arch = gguf.MODEL_ARCH.GPT_OSS
+1
View File
@@ -13,6 +13,7 @@ from .base import ModelBase, TextModel, gguf, logger
@ModelBase.register("GPTNeoXForCausalLM")
@ModelBase.example("EleutherAI/pythia-70m")
class GPTNeoXModel(TextModel):
model_arch = gguf.MODEL_ARCH.GPTNEOX
+7
View File
@@ -15,6 +15,7 @@ from .mamba import Mamba2Model
@ModelBase.register("GraniteForCausalLM")
@ModelBase.example("ibm-granite/granite-3.3-2b-instruct")
class GraniteModel(LlamaModel):
"""Conversion for IBM's GraniteForCausalLM"""
model_arch = gguf.MODEL_ARCH.GRANITE
@@ -74,6 +75,7 @@ class GraniteModel(LlamaModel):
@ModelBase.register("GraniteMoeForCausalLM", "GraniteMoeSharedForCausalLM")
@ModelBase.example("ibm-granite/granite-3.1-3b-a800m-instruct")
class GraniteMoeModel(GraniteModel):
"""Conversion for IBM's GraniteMoeForCausalLM"""
model_arch = gguf.MODEL_ARCH.GRANITE_MOE
@@ -124,6 +126,7 @@ class GraniteMoeModel(GraniteModel):
@ModelBase.register("GraniteSwitchForCausalLM")
@ModelBase.example("ibm-granite/granite-switch-4.1-3b-preview")
class GraniteSwitchModel(GraniteMoeModel):
"""Dense, all-attention Granite with N per-token embedded LoRA adapters, stacked
over the adapter dim with a zero adapter at slot 0 (N = num_adapters + 1)."""
@@ -284,6 +287,7 @@ class GraniteSwitchModel(GraniteMoeModel):
@ModelBase.register("GraniteMoeHybridForCausalLM", "BambaForCausalLM")
@ModelBase.example("ibm-granite/granite-4.0-h-tiny", "ibm-ai-platform/Bamba-9B-v2")
class GraniteHybridModel(Mamba2Model, GraniteMoeModel):
"""GraniteHybrid is a hybrid SSM + Attention model that uses Mamba2 SSM
layers and optionally uses MoE w/ a shared expert"""
@@ -426,6 +430,7 @@ class GraniteHybridModel(Mamba2Model, GraniteMoeModel):
@ModelBase.register("GraniteSpeechForConditionalGeneration")
@ModelBase.example("ibm-granite/granite-speech-3.3-2b", "ibm-granite/granite-4.0-1b-speech")
class GraniteSpeechMmprojModel(MmprojModel):
has_vision_encoder = False
has_audio_encoder = True
@@ -509,6 +514,7 @@ class GraniteSpeechMmprojModel(MmprojModel):
@ModelBase.register("GraniteSpeechPlusForConditionalGeneration")
@ModelBase.example("ibm-granite/granite-speech-4.1-2b-plus")
class GraniteSpeechPlusMmprojModel(GraniteSpeechMmprojModel):
"""Conversion for GraniteSpeechPlus - extends GraniteSpeech with feature layer concatenation"""
has_vision_encoder = False
@@ -537,6 +543,7 @@ class GraniteSpeechPlusMmprojModel(GraniteSpeechMmprojModel):
@ModelBase.register("Granite4VisionForConditionalGeneration")
@ModelBase.example("ibm-granite/granite-4.0-3b-vision")
class Granite4VisionMmprojModel(MmprojModel):
has_vision_encoder = True
has_audio_encoder = False
+1
View File
@@ -13,6 +13,7 @@ from .base import ModelBase, TextModel, gguf, logger
@ModelBase.register("GrokForCausalLM", "Grok1ForCausalLM")
@ModelBase.example("keyfan/grok-1-hf")
class GrokModel(TextModel):
model_arch = gguf.MODEL_ARCH.GROK
+1
View File
@@ -11,6 +11,7 @@ from .base import ModelBase, TextModel, gguf, logger
@ModelBase.register("GroveMoeForCausalLM", "modeling_grove_moe.GroveMoeForCausalLM")
@ModelBase.example("inclusionAI/GroveMoE-Inst")
class GroveMoeModel(TextModel):
model_arch = gguf.MODEL_ARCH.GROVEMOE
+5
View File
@@ -17,6 +17,7 @@ from .qwen import QwenModel
@ModelBase.register("HunYuanMoEV1ForCausalLM")
@ModelBase.example("tencent/Hunyuan-A13B-Instruct")
class HunYuanMoEModel(TextModel):
model_arch = gguf.MODEL_ARCH.HUNYUAN_MOE
@@ -154,6 +155,7 @@ class HunYuanMoEModel(TextModel):
@ModelBase.register("HunYuanDenseV1ForCausalLM")
@ModelBase.example("tencent/Hunyuan-4B-Instruct")
class HunYuanModel(TextModel):
model_arch = gguf.MODEL_ARCH.HUNYUAN_DENSE
@@ -290,6 +292,7 @@ class HunYuanModel(TextModel):
@ModelBase.register("HunYuanVLForConditionalGeneration")
@ModelBase.example("tencent/HunyuanOCR")
class HunyuanVLVisionModel(MmprojModel):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
@@ -333,6 +336,7 @@ class HunyuanVLVisionModel(MmprojModel):
@ModelBase.register("HunYuanVLForConditionalGeneration")
@ModelBase.example("tencent/HunyuanOCR")
class HunyuanVLTextModel(HunYuanModel):
model_arch = gguf.MODEL_ARCH.HUNYUAN_VL
@@ -365,6 +369,7 @@ class HunyuanVLTextModel(HunYuanModel):
@ModelBase.register("HYV3ForCausalLM")
@ModelBase.example("tencent/Hy3")
class HYV3Model(TextModel):
model_arch = gguf.MODEL_ARCH.HY_V3
supports_mtp_export = True
+2
View File
@@ -14,6 +14,7 @@ from .llama import LlamaModel
@ModelBase.register("InternLM2ForCausalLM")
@ModelBase.example("internlm/internlm2-chat-7b")
class InternLM2Model(TextModel):
model_arch = gguf.MODEL_ARCH.INTERNLM2
@@ -170,6 +171,7 @@ class InternLM2Model(TextModel):
@ModelBase.register("InternLM3ForCausalLM")
@ModelBase.example("internlm/internlm3-8b-instruct")
class InternLM3Model(TextModel):
model_arch = gguf.MODEL_ARCH.LLAMA
+1
View File
@@ -9,6 +9,7 @@ from .base import MmprojModel, ModelBase, gguf
@ModelBase.register("InternVisionModel")
@ModelBase.example("OpenGVLab/InternVL3-2B", "OpenGVLab/InternVL2_5-1B")
class InternVisionModel(MmprojModel):
min_dynamic_tiles: int = 0
+3
View File
@@ -11,6 +11,8 @@ from .base import ModelBase, TextModel, gguf
@ModelBase.register("Jais2ForCausalLM")
# [TAG_HF_EXAMPLE_GATED] inceptionai/Jais-2-8B-Chat is gated
# [TAG_HF_EXAMPLE_MISSING]
class Jais2Model(TextModel):
model_arch = gguf.MODEL_ARCH.JAIS2
@@ -22,6 +24,7 @@ class Jais2Model(TextModel):
@ModelBase.register("JAISLMHeadModel")
@ModelBase.example("inceptionai/jais-family-590m")
class JaisModel(TextModel):
model_arch = gguf.MODEL_ARCH.JAIS
+1
View File
@@ -11,6 +11,7 @@ from .base import ModelBase, TextModel, gguf, logger
@ModelBase.register("JambaForCausalLM")
@ModelBase.example("ai21labs/Jamba-v0.1")
class JambaModel(TextModel):
model_arch = gguf.MODEL_ARCH.JAMBA
+2
View File
@@ -11,6 +11,7 @@ from .llama import LlamaModel
@ModelBase.register("JanusForConditionalGeneration")
@ModelBase.example("deepseek-community/Janus-Pro-1B")
class JanusProModel(LlamaModel):
model_arch = gguf.MODEL_ARCH.LLAMA # reuse Llama arch
@@ -34,6 +35,7 @@ class JanusProModel(LlamaModel):
@ModelBase.register("JanusForConditionalGeneration")
@ModelBase.example("deepseek-community/Janus-Pro-1B")
class JanusProVisionModel(MmprojModel):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
+1
View File
@@ -16,6 +16,7 @@ from .kimi_linear import KimiLinearModel
@ModelBase.register("KimiK3ForConditionalGeneration")
@ModelBase.example("moonshotai/Kimi-K3")
class KimiK3Model(TextModel):
"""
Kimi-K3 text model (KimiLinearForCausalLM under a `language_model.` prefix).
+1
View File
@@ -13,6 +13,7 @@ from .qwen import QwenModel
@ModelBase.register("KimiLinearModel", "KimiLinearForCausalLM")
@ModelBase.example("moonshotai/Kimi-Linear-48B-A3B-Instruct")
class KimiLinearModel(TextModel):
"""Kimi-Linear model with hybrid MLA+KDA architecture"""
model_arch = gguf.MODEL_ARCH.KIMI_LINEAR
+3
View File
@@ -11,6 +11,7 @@ from .base import MmprojModel, ModelBase, gguf
@ModelBase.register("KimiVLForConditionalGeneration")
@ModelBase.example("moonshotai/Kimi-VL-A3B-Instruct")
class KimiVLModel(MmprojModel):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
@@ -52,6 +53,7 @@ class KimiVLModel(MmprojModel):
@ModelBase.register("KimiK25ForConditionalGeneration")
@ModelBase.example("moonshotai/Kimi-K2.5")
class KimiK25Model(MmprojModel):
"""Kimi-K2.5 with MoonViT3d vision encoder"""
@@ -155,6 +157,7 @@ class KimiK25Model(MmprojModel):
@ModelBase.register("Glm5vForConditionalGeneration")
# [TAG_HF_EXAMPLE_MISSING]
class Glm5vModel(KimiK25Model):
"""GLM-5.2-Vision MoonViT3d encoder and projector
+1
View File
@@ -13,6 +13,7 @@ from .base import ModelBase, TextModel, gguf, logger
@ModelBase.register("LagunaForCausalLM")
@ModelBase.example("poolside/Laguna-XS.2", "poolside/Laguna-S-2.1")
class LagunaModel(TextModel):
model_arch = gguf.MODEL_ARCH.LAGUNA
_experts: list[dict] | None = None
+6
View File
@@ -13,6 +13,7 @@ from .gemma import ConformerAudioModel
@ModelBase.register("Lfm2ForCausalLM", "LFM2ForCausalLM")
@ModelBase.example("LiquidAI/LFM2-1.2B", "LiquidAI/LFM2.5-350M")
class LFM2Model(TextModel):
model_arch = gguf.MODEL_ARCH.LFM2
@@ -65,6 +66,7 @@ class LFM2Model(TextModel):
@ModelBase.register("Lfm2Model", "Lfm2BidirectionalModel")
@ModelBase.example("LiquidAI/LFM2.5-ColBERT-350M", "LiquidAI/LFM2.5-Embedding-350M")
class LFM2ColBertModel(LFM2Model):
model_arch = gguf.MODEL_ARCH.LFM2
dense_tensor_name = "dense_2"
@@ -93,6 +95,7 @@ class LFM2ColBertModel(LFM2Model):
@ModelBase.register("Lfm2MoeForCausalLM")
@ModelBase.example("LiquidAI/LFM2-8B-A1B")
class LFM2MoeModel(TextModel):
model_arch = gguf.MODEL_ARCH.LFM2MOE
@@ -166,6 +169,7 @@ class LFM2MoeModel(TextModel):
@ModelBase.register("Lfm2VlForConditionalGeneration")
@ModelBase.example("LiquidAI/LFM2-VL-450M")
class LFM2VLModel(MmprojModel):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
@@ -200,6 +204,7 @@ class LFM2VLModel(MmprojModel):
@ModelBase.register("Lfm2AudioForConditionalGeneration")
@ModelBase.example("LiquidAI/LFM2.5-Audio-1.5B", "LiquidAI/LFM2-Audio-1.5B")
class LFM2AudioModel(ConformerAudioModel):
has_vision_encoder = False
has_audio_encoder = True
@@ -238,6 +243,7 @@ class LFM2AudioModel(ConformerAudioModel):
@ModelBase.register("Lfm25AudioTokenizer")
@ModelBase.example("LiquidAI/LFM2.5-Audio-1.5B")
class LFM25AudioTokenizer(LFM2Model):
model_arch = gguf.MODEL_ARCH.LFM2
+1
View File
@@ -11,6 +11,7 @@ from .llava import LlavaVisionModel
@ModelBase.register("LightOnOCRForConditionalGeneration")
@ModelBase.example("lightonai/LightOnOCR-1B-1025")
class LightOnOCRVisionModel(LlavaVisionModel):
is_mistral_format = False
use_break_tok = False
+2
View File
@@ -11,6 +11,7 @@ from .base import ModelBase, TextModel, gguf
@ModelBase.register("LLaDAModelLM")
@ModelBase.example("GSAI-ML/LLaDA-8B-Instruct")
class LLaDAModel(TextModel):
model_arch = gguf.MODEL_ARCH.LLADA
undo_permute = True
@@ -114,6 +115,7 @@ class LLaDAModel(TextModel):
@ModelBase.register("LLaDAMoEModel", "LLaDAMoEModelLM")
@ModelBase.example("inclusionAI/LLaDA-MoE-7B-A1B-Instruct")
class LLaDAMoEModel(TextModel):
model_arch = gguf.MODEL_ARCH.LLADA_MOE
+8
View File
@@ -28,6 +28,8 @@ from .base import ModelBase, TextModel, gguf, logger
"Eagle3DraftModel",
"IQuestCoderForCausalLM",
"LlamaModel")
# [TAG_HF_EXAMPLE_GATED] meta-llama/Llama-3.2-1B-Instruct is gated
@ModelBase.example("unsloth/Llama-3.2-1B-Instruct", "mistralai/Mistral-7B-Instruct-v0.3", "mistralai/Mixtral-8x7B-Instruct-v0.1")
class LlamaModel(TextModel):
model_arch = gguf.MODEL_ARCH.LLAMA
undo_permute = True
@@ -359,6 +361,7 @@ class LlamaModel(TextModel):
@ModelBase.register("ArceeForCausalLM")
@ModelBase.example("arcee-ai/AFM-4.5B")
class ArceeModel(LlamaModel):
model_arch = gguf.MODEL_ARCH.ARCEE
@@ -371,6 +374,8 @@ class ArceeModel(LlamaModel):
"Llama4ForConditionalGeneration",
"Llama4ForCausalLM",
)
# [TAG_HF_EXAMPLE_GATED] meta-llama/Llama-4-Scout-17B-16E-Instruct is gated
@ModelBase.example("unsloth/Llama-4-Scout-17B-16E-Instruct")
class Llama4Model(LlamaModel):
model_arch = gguf.MODEL_ARCH.LLAMA4
undo_permute = False
@@ -412,16 +417,19 @@ class Llama4Model(LlamaModel):
@ModelBase.register("LlamaBidirectionalModel")
@ModelBase.example("nvidia/llama-embed-nemotron-8b")
class LlamaEmbedNemotronModel(LlamaModel):
model_arch = gguf.MODEL_ARCH.LLAMA_EMBED
@ModelBase.register("SmolLM3ForCausalLM")
@ModelBase.example("HuggingFaceTB/SmolLM3-3B")
class SmolLM3Model(LlamaModel):
model_arch = gguf.MODEL_ARCH.SMOLLM3
@ModelBase.register("ApertusForCausalLM")
@ModelBase.example("swiss-ai/Apertus-8B-Instruct-2509")
class ApertusModel(LlamaModel):
model_arch = gguf.MODEL_ARCH.APERTUS
undo_permute = False
+2
View File
@@ -9,6 +9,8 @@ from .base import MmprojModel, ModelBase, gguf
@ModelBase.register("Llama4ForConditionalGeneration")
# [TAG_HF_EXAMPLE_GATED] meta-llama/Llama-4-Scout-17B-16E-Instruct is gated
@ModelBase.example("unsloth/Llama-4-Scout-17B-16E-Instruct")
class Llama4VisionModel(MmprojModel):
def set_gguf_parameters(self):
super().set_gguf_parameters()
+1
View File
@@ -16,6 +16,7 @@ from .llama import LlamaModel
"LlavaForConditionalGeneration", # pixtral
"Mistral3ForConditionalGeneration", # mistral small 3.1
)
@ModelBase.example("mistral-community/pixtral-12b", "mistralai/Mistral-Small-3.1-24B-Instruct-2503")
class LlavaVisionModel(MmprojModel):
img_break_tok_id = -1
use_break_tok = True
+1
View File
@@ -4,6 +4,7 @@ from .base import ModelBase, TextModel, gguf
@ModelBase.register("MaincoderForCausalLM")
@ModelBase.example("Maincode/Maincoder-1B")
class MaincoderModel(TextModel):
model_arch = gguf.MODEL_ARCH.MAINCODER
+2
View File
@@ -14,6 +14,7 @@ from .base import ModelBase, TextModel, gguf, logger
@ModelBase.register("MambaForCausalLM", "MambaLMHeadModel", "FalconMambaForCausalLM")
@ModelBase.example("state-spaces/mamba-130m-hf", "tiiuae/falcon-mamba-7b")
class MambaModel(TextModel):
model_arch = gguf.MODEL_ARCH.MAMBA
@@ -100,6 +101,7 @@ class MambaModel(TextModel):
@ModelBase.register("Mamba2ForCausalLM")
@ModelBase.example("mistralai/Mamba-Codestral-7B-v0.1")
class Mamba2Model(TextModel):
model_arch = gguf.MODEL_ARCH.MAMBA2
+1
View File
@@ -11,6 +11,7 @@ from .base import ModelBase, TextModel, gguf, logger
@ModelBase.register("MellumForCausalLM")
@ModelBase.example("JetBrains/Mellum2-12B-A2.5B-Base")
class MellumModel(TextModel):
model_arch = gguf.MODEL_ARCH.MELLUM
+2
View File
@@ -14,6 +14,7 @@ from .base import MmprojModel, ModelBase, TextModel, gguf
@ModelBase.register("MiMoV2FlashForCausalLM", "MiMoV2ForCausalLM")
@ModelBase.example("XiaomiMiMo/MiMo-V2.5")
class MimoV2Model(TextModel):
model_arch = gguf.MODEL_ARCH.MIMO2
@@ -230,6 +231,7 @@ class MimoV2Model(TextModel):
@ModelBase.register("MiMoV2ForCausalLM")
@ModelBase.example("XiaomiMiMo/MiMo-V2.5")
class MiMoV2VisionAudioModel(MmprojModel):
has_audio_encoder = True
+4
View File
@@ -14,6 +14,7 @@ from .qwen import Qwen3_5TextModel
@ModelBase.register("MiniCPMForCausalLM")
@ModelBase.example("openbmb/MiniCPM-2B-sft-bf16")
class MiniCPMModel(TextModel):
model_arch = gguf.MODEL_ARCH.MINICPM
@@ -61,6 +62,7 @@ class MiniCPMModel(TextModel):
@ModelBase.register("MiniCPM3ForCausalLM")
@ModelBase.example("openbmb/MiniCPM3-4B")
class MiniCPM3Model(TextModel):
model_arch = gguf.MODEL_ARCH.MINICPM3
@@ -117,6 +119,7 @@ class MiniCPM3Model(TextModel):
# the LM (text mode) and once as the mmproj (vision mode), mirroring the Qwen3-VL setup.
@ModelBase.register("MiniCPMV4_6ForConditionalGeneration")
@ModelBase.example("openbmb/MiniCPM-V-4_6")
class MiniCPMV4_6TextModel(Qwen3_5TextModel):
model_arch = gguf.MODEL_ARCH.QWEN35
@@ -134,6 +137,7 @@ class MiniCPMV4_6TextModel(Qwen3_5TextModel):
@ModelBase.register("MiniCPMV4_6ForConditionalGeneration")
@ModelBase.example("openbmb/MiniCPM-V-4_6")
class MiniCPMV4_6VisionModel(MmprojModel):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
+4
View File
@@ -12,6 +12,7 @@ from .base import ModelBase, TextModel, MmprojModel, gguf, logger
@ModelBase.register("MiniMaxText01ForCausalLM")
@ModelBase.register("MiniMaxM1ForCausalLM")
@ModelBase.example("MiniMaxAI/MiniMax-Text-01", "MiniMaxAI/MiniMax-M1-40k")
class MiniMaxText01Model(TextModel):
model_arch = gguf.MODEL_ARCH.MINIMAX01
@@ -119,6 +120,7 @@ class MiniMaxText01Model(TextModel):
@ModelBase.register("MiniMaxM2ForCausalLM")
@ModelBase.example("MiniMaxAI/MiniMax-M2")
class MiniMaxM2Model(TextModel):
model_arch = gguf.MODEL_ARCH.MINIMAXM2
_experts_cache: dict[int, dict[str, Tensor]] = {}
@@ -163,6 +165,7 @@ class MiniMaxM2Model(TextModel):
@ModelBase.register("MiniMaxM3SparseForCausalLM", "MiniMaxM3SparseForConditionalGeneration")
@ModelBase.example("MiniMaxAI/MiniMax-M3")
class MiniMaxM3Model(MiniMaxM2Model):
model_arch = gguf.MODEL_ARCH.MINIMAXM3
@@ -203,6 +206,7 @@ class MiniMaxM3Model(MiniMaxM2Model):
@ModelBase.register("MiniMaxM3SparseForConditionalGeneration", "MiniMaxM3VLForConditionalGeneration")
@ModelBase.example("MiniMaxAI/MiniMax-M3")
class MiniMaxM3VisionModel(MmprojModel):
@classmethod
def filter_tensors(cls, item):
+1
View File
@@ -15,6 +15,7 @@ from .llama import LlamaModel
"Mistral3ForConditionalGeneration",
"Ministral3ForCausalLM",
)
@ModelBase.example("mistralai/Mistral-Small-3.1-24B-Instruct-2503", "hf-tiny-v2/tiny-random-Ministral3ForCausalLM")
class Mistral3Model(TextModel):
class Ministral3Model(LlamaModel):
model_arch = gguf.MODEL_ARCH.MISTRAL3
+1
View File
@@ -9,6 +9,7 @@ from .base import ModelBase, TextModel, gguf
@ModelBase.register("MPTForCausalLM")
@ModelBase.example("anas-awadalla/mpt-7b")
class MPTModel(TextModel):
model_arch = gguf.MODEL_ARCH.MPT
+3
View File
@@ -24,6 +24,7 @@ def _unpermute_for_rope(tensor: "Tensor", n_heads: int) -> "Tensor":
@ModelBase.register("MuseGlimmerForConditionalGeneration")
@ModelBase.example("meta-models/Muse-Glimmer-30B")
class MuseGlimmerModel(TextModel):
model_arch = gguf.MODEL_ARCH.MUSE_GLIMMER
@@ -78,6 +79,7 @@ class MuseGlimmerModel(TextModel):
@ModelBase.register("MuseGlimmerForConditionalGeneration")
@ModelBase.example("meta-models/Muse-Glimmer-30B")
class MuseGlimmerVisionModel(MmprojModel):
def get_vision_config(self) -> dict[str, Any] | None:
c = self.global_config.get("vision_config")
@@ -131,6 +133,7 @@ class MuseGlimmerVisionModel(MmprojModel):
@ModelBase.register("MuseGlimmerAssistantModel")
@ModelBase.example("meta-models/Muse-Glimmer-30B-assistant")
class MuseGlimmerAssistantModel(TextModel):
model_arch = gguf.MODEL_ARCH.DFLASH
+1
View File
@@ -5,6 +5,7 @@ from .llama import LlamaModel
@ModelBase.register("NanbeigeForCausalLM")
@ModelBase.example("Nanbeige/Nanbeige4.2-3B")
class NanbeigeModel(LlamaModel):
model_arch = gguf.MODEL_ARCH.NANBEIGE
undo_permute = True
+3
View File
@@ -16,6 +16,7 @@ from .granite import GraniteHybridModel
"NemotronH_Nano_VL_V2",
"RADIOModel",
)
@ModelBase.example("nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16")
class NemotronNanoV2VLModel(MmprojModel):
# ViT-Huge architecture parameters for RADIO v2.5-h
_vit_hidden_size = 1280
@@ -151,6 +152,7 @@ class NemotronNanoV2VLModel(MmprojModel):
@ModelBase.register("NemotronForCausalLM")
@ModelBase.example("nvidia/Minitron-4B-Base")
class NemotronModel(TextModel):
model_arch = gguf.MODEL_ARCH.NEMOTRON
@@ -193,6 +195,7 @@ class NemotronModel(TextModel):
@ModelBase.register("NemotronHForCausalLM")
@ModelBase.example("nvidia/Nemotron-H-8B-Base-8K")
class NemotronHModel(GraniteHybridModel):
"""Hybrid mamba2/attention model from NVIDIA"""
model_arch = gguf.MODEL_ARCH.NEMOTRON_H
+4
View File
@@ -14,6 +14,7 @@ from .llama import LlamaModel
@ModelBase.register("OlmoForCausalLM")
@ModelBase.register("OLMoForCausalLM")
@ModelBase.example("allenai/OLMo-1.7-7B-hf")
class OlmoModel(TextModel):
model_arch = gguf.MODEL_ARCH.OLMO
@@ -39,12 +40,14 @@ class OlmoModel(TextModel):
@ModelBase.register("SeedOssForCausalLM")
@ModelBase.example("ByteDance-Seed/Seed-OSS-36B-Instruct")
class SeedOssModel(TextModel):
model_arch = gguf.MODEL_ARCH.SEED_OSS
@ModelBase.register("Olmo2ForCausalLM")
@ModelBase.register("Olmo3ForCausalLM")
@ModelBase.example("allenai/OLMo-2-1124-7B-Instruct", "allenai/Olmo-3-7B-Instruct")
class Olmo2Model(TextModel):
model_arch = gguf.MODEL_ARCH.OLMO2
@@ -67,6 +70,7 @@ class Olmo2Model(TextModel):
@ModelBase.register("OlmoeForCausalLM")
@ModelBase.example("allenai/OLMoE-1B-7B-0924")
class OlmoeModel(TextModel):
model_arch = gguf.MODEL_ARCH.OLMOE
+1
View File
@@ -9,6 +9,7 @@ from .base import ModelBase, TextModel, gguf
@ModelBase.register("OpenELMForCausalLM")
@ModelBase.example("apple/OpenELM-270M")
class OpenELMModel(TextModel):
model_arch = gguf.MODEL_ARCH.OPENELM
+1
View File
@@ -4,6 +4,7 @@ from .base import ModelBase, TextModel, gguf
@ModelBase.register("OrionForCausalLM")
@ModelBase.example("OrionStarAI/Orion-14B-Base")
class OrionModel(TextModel):
model_arch = gguf.MODEL_ARCH.ORION
+1
View File
@@ -11,6 +11,7 @@ from .base import ModelBase, TextModel, gguf, logger
@ModelBase.register("PanguEmbeddedForCausalLM")
@ModelBase.example("FreedomIntelligence/openPangu-Embedded-7B-V1.1")
class PanguEmbeddedModel(TextModel):
model_arch = gguf.MODEL_ARCH.PANGU_EMBED
+4
View File
@@ -14,6 +14,7 @@ from .base import MmprojModel, ModelBase, SentencePieceTokenTypes, TextModel, gg
@ModelBase.register("PhiForCausalLM")
@ModelBase.example("microsoft/phi-2")
class Phi2Model(TextModel):
model_arch = gguf.MODEL_ARCH.PHI2
@@ -36,6 +37,7 @@ class Phi2Model(TextModel):
@ModelBase.register("Phi3ForCausalLM", "Phi4ForCausalLMV")
@ModelBase.example("microsoft/Phi-3-mini-4k-instruct")
class Phi3MiniModel(TextModel):
model_arch = gguf.MODEL_ARCH.PHI3
@@ -210,6 +212,7 @@ class Phi3MiniModel(TextModel):
@ModelBase.register("Phi4ForCausalLMV")
# [TAG_HF_EXAMPLE_MISSING]
class Phi4VisionMmprojModel(MmprojModel):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
@@ -336,6 +339,7 @@ class Phi4VisionMmprojModel(MmprojModel):
@ModelBase.register("PhiMoEForCausalLM")
@ModelBase.example("microsoft/Phi-3.5-MoE-instruct")
class PhiMoeModel(Phi3MiniModel):
model_arch = gguf.MODEL_ARCH.PHIMOE
+4
View File
@@ -13,6 +13,7 @@ from .base import ModelBase, TextModel, gguf
@ModelBase.register("PlamoForCausalLM")
@ModelBase.example("pfnet/plamo-13b")
class PlamoModel(TextModel):
model_arch = gguf.MODEL_ARCH.PLAMO
@@ -58,6 +59,7 @@ class PlamoModel(TextModel):
@ModelBase.register("Plamo2ForCausalLM", "PLaMo2ForCausalLM")
@ModelBase.example("pfnet/plamo-2-1b")
class Plamo2Model(TextModel):
model_arch = gguf.MODEL_ARCH.PLAMO2
@@ -147,6 +149,8 @@ class Plamo2Model(TextModel):
@ModelBase.register("Plamo3ForCausalLM", "PLaMo3ForCausalLM")
# [TAG_HF_EXAMPLE_GATED] pfnet/plamo-3-nict-2b-base is gated
@ModelBase.example("midorin-Linux/plamo-3-12b-self-merged-base")
class Plamo3Model(TextModel):
model_arch = gguf.MODEL_ARCH.PLAMO3
+1
View File
@@ -4,6 +4,7 @@ from .base import ModelBase, TextModel, gguf
@ModelBase.register("PLMForCausalLM")
@ModelBase.example("PLM-Team/PLM-1.8B-Instruct")
class PLMModel(TextModel):
model_arch = gguf.MODEL_ARCH.PLM
+2
View File
@@ -77,6 +77,7 @@ def _load_hparams(dir_model: Path) -> dict[str, Any]:
@ModelBase.register("PocketTTSModel")
# [TAG_HF_EXAMPLE_MISSING] model is gated, and the checkpoint requires cd to subdir, not supported here
class PocketTTSModel(TextModel):
model_arch = gguf.MODEL_ARCH.POCKETTTS
@@ -174,6 +175,7 @@ class PocketTTSModel(TextModel):
@ModelBase.register("PocketTTSModel")
# [TAG_HF_EXAMPLE_MISSING] model is gated, and the checkpoint requires cd to subdir, not supported here
class PocketTTSMmprojModel(MmprojModel):
has_audio_encoder = True
has_vision_encoder = False
+11
View File
@@ -13,6 +13,7 @@ from .base import ModelBase, TextModel, gguf, logger
@ModelBase.register("QWenLMHeadModel")
@ModelBase.example("Qwen/Qwen-7B")
class QwenModel(TextModel):
model_arch = gguf.MODEL_ARCH.QWEN
@@ -51,6 +52,7 @@ class QwenModel(TextModel):
"AudioFlamingo3ForConditionalGeneration",
"DotsOCRForCausalLM",
)
@ModelBase.example("Qwen/Qwen2.5-7B-Instruct")
class Qwen2Model(TextModel):
model_arch = gguf.MODEL_ARCH.QWEN2
@@ -71,6 +73,7 @@ class Qwen2Model(TextModel):
@ModelBase.register("Qwen2MoeForCausalLM")
@ModelBase.example("Qwen/Qwen1.5-MoE-A2.7B")
class Qwen2MoeModel(TextModel):
model_arch = gguf.MODEL_ARCH.QWEN2MOE
@@ -153,6 +156,7 @@ class Qwen2MoeModel(TextModel):
@ModelBase.register("Qwen3ForCausalLM", "Qwen3Model")
@ModelBase.example("Qwen/Qwen3-8B")
class Qwen3Model(Qwen2Model):
model_arch = gguf.MODEL_ARCH.QWEN3
@@ -251,6 +255,7 @@ class Qwen3Model(Qwen2Model):
@ModelBase.register("Qwen3MoeForCausalLM")
@ModelBase.example("Qwen/Qwen3-30B-A3B")
class Qwen3MoeModel(Qwen2MoeModel):
model_arch = gguf.MODEL_ARCH.QWEN3MOE
@@ -362,6 +367,7 @@ class _QwenMtpMixin:
@ModelBase.register("Qwen3NextForCausalLM")
@ModelBase.example("Qwen/Qwen3-Next-80B-A3B-Instruct")
class Qwen3NextModel(_QwenMtpMixin, Qwen2MoeModel):
model_arch = gguf.MODEL_ARCH.QWEN3NEXT
@@ -421,6 +427,7 @@ class Qwen3NextModel(_QwenMtpMixin, Qwen2MoeModel):
@ModelBase.register("RND1")
@ModelBase.example("radicalnumerics/RND1-Base-0910")
class RND1Model(Qwen2MoeModel):
model_arch = gguf.MODEL_ARCH.RND1
@@ -620,16 +627,19 @@ class _Qwen35MRopeMixin:
@ModelBase.register("Qwen3_5ForConditionalGeneration", "Qwen3_5ForCausalLM")
@ModelBase.example("Qwen/Qwen3.5-9B")
class Qwen3_5TextModel(_Qwen35MRopeMixin, _LinearAttentionVReorderBase):
model_arch = gguf.MODEL_ARCH.QWEN35
@ModelBase.register("Qwen3_5MoeForConditionalGeneration", "Qwen3_5MoeForCausalLM")
@ModelBase.example("Qwen/Qwen3.5-35B-A3B")
class Qwen3_5MoeTextModel(_Qwen35MRopeMixin, _LinearAttentionVReorderBase):
model_arch = gguf.MODEL_ARCH.QWEN35MOE
@ModelBase.register("DFlashDraftModel")
@ModelBase.example("z-lab/Qwen3.5-9B-DFlash")
class DFlashModel(Qwen3Model):
model_arch = gguf.MODEL_ARCH.DFLASH
@@ -699,6 +709,7 @@ class DFlashModel(Qwen3Model):
@ModelBase.register("Qwen3DSparkModel")
@ModelBase.example("satgeze/Qwen3.6-27B-DSpark")
class DSparkModel(DFlashModel):
# DSpark = DFlash + a semi-autoregressive Markov head
model_arch = gguf.MODEL_ARCH.DFLASH
+2
View File
@@ -37,6 +37,7 @@ _ACT2FN = {
@ModelBase.register("Qwen3TTSForConditionalGeneration")
@ModelBase.example("Qwen/Qwen3-TTS-12Hz-1.7B-Base")
class Qwen3TTSTalkerModel(TextModel):
model_arch = gguf.MODEL_ARCH.QWEN3TTS
@@ -185,6 +186,7 @@ class Qwen3TTSTalkerModel(TextModel):
@ModelBase.register("Qwen3TTSForConditionalGeneration")
@ModelBase.example("Qwen/Qwen3-TTS-12Hz-1.7B-Base")
class Qwen3TTSSpeakerEncoderModel(MmprojModel):
has_vision_encoder = False
has_audio_encoder = True
+8
View File
@@ -14,6 +14,7 @@ from .qwenvl import Qwen25AudioModel
@ModelBase.register("Qwen3VLForConditionalGeneration", "Qwen3VLMoeForConditionalGeneration", "Qwen3_5ForConditionalGeneration", "Qwen3_5MoeForConditionalGeneration")
@ModelBase.example("Qwen/Qwen3-VL-4B-Instruct", "Qwen/Qwen3-VL-30B-A3B-Instruct", "Qwen/Qwen3.5-9B", "Qwen/Qwen3.5-35B-A3B")
class Qwen3VLVisionModel(MmprojModel):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
@@ -144,6 +145,7 @@ class Qwen3VLVisionModel(MmprojModel):
@ModelBase.register("Qwen3OmniMoeForConditionalGeneration")
@ModelBase.example("Qwen/Qwen3-Omni-30B-A3B-Instruct")
class Qwen3OmniMmprojModel(Qwen3VLVisionModel, Qwen25AudioModel):
has_audio_encoder = True
has_vision_encoder = True
@@ -217,12 +219,14 @@ class Qwen3OmniMmprojModel(Qwen3VLVisionModel, Qwen25AudioModel):
@ModelBase.register("Qwen3ASRForConditionalGeneration")
@ModelBase.example("Qwen/Qwen3-ASR-0.6B-hf")
class Qwen3ASRMmprojModel(Qwen3OmniMmprojModel):
has_audio_encoder = True
has_vision_encoder = False
@ModelBase.register("Glm4vForConditionalGeneration", "Glm4vMoeForConditionalGeneration", "GlmOcrForConditionalGeneration")
@ModelBase.example("zai-org/GLM-4.1V-9B-Thinking", "zai-org/GLM-4.5V")
class Glm4VVisionModel(Qwen3VLVisionModel):
def set_gguf_parameters(self):
MmprojModel.set_gguf_parameters(self) # skip Qwen3VLVisionModel parameters
@@ -246,6 +250,7 @@ class Glm4VVisionModel(Qwen3VLVisionModel):
@ModelBase.register("Qwen3VLForConditionalGeneration")
@ModelBase.example("Qwen/Qwen3-VL-4B-Instruct")
class Qwen3VLTextModel(Qwen3Model):
model_arch = gguf.MODEL_ARCH.QWEN3VL
@@ -268,6 +273,7 @@ class Qwen3VLTextModel(Qwen3Model):
@ModelBase.register("Qwen3VLMoeForConditionalGeneration")
@ModelBase.example("Qwen/Qwen3-VL-30B-A3B-Instruct")
class Qwen3VLMoeTextModel(Qwen3MoeModel):
model_arch = gguf.MODEL_ARCH.QWEN3VLMOE
@@ -317,6 +323,7 @@ class Qwen3VLMoeTextModel(Qwen3MoeModel):
@ModelBase.register("Qwen3OmniMoeForConditionalGeneration")
@ModelBase.example("Qwen/Qwen3-Omni-30B-A3B-Instruct")
class Qwen3OmniMoeTextModel(Qwen3VLMoeTextModel):
model_arch = gguf.MODEL_ARCH.QWEN3VLMOE
@@ -338,6 +345,7 @@ class Qwen3OmniMoeTextModel(Qwen3VLMoeTextModel):
@ModelBase.register("Qwen3ASRForConditionalGeneration")
@ModelBase.example("Qwen/Qwen3-ASR-0.6B-hf")
class Qwen3ASRTextModel(Qwen3VLTextModel):
model_arch = gguf.MODEL_ARCH.QWEN3VL
+3
View File
@@ -17,6 +17,7 @@ from .base import MmprojModel, ModelBase, TextModel, gguf
"Qwen2_5_VLForConditionalGeneration",
"Qwen2_5OmniModel",
)
@ModelBase.example("Qwen/Qwen2-VL-2B-Instruct", "Qwen/Qwen2.5-VL-3B-Instruct")
class Qwen2VLModel(TextModel):
model_arch = gguf.MODEL_ARCH.QWEN2VL
@@ -40,6 +41,7 @@ class Qwen2VLModel(TextModel):
@ModelBase.register("Qwen2VLModel", "Qwen2VLForConditionalGeneration", "Qwen2_5_VLForConditionalGeneration")
@ModelBase.example("Qwen/Qwen2-VL-2B-Instruct", "Qwen/Qwen2.5-VL-3B-Instruct")
class Qwen2VLVisionModel(MmprojModel):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
@@ -161,6 +163,7 @@ class Qwen25AudioModel(MmprojModel):
@ModelBase.register("Qwen2_5OmniModel")
@ModelBase.example("Qwen/Qwen2.5-Omni-3B")
class Qwen25OmniModel(Qwen2VLVisionModel, Qwen25AudioModel):
has_audio_encoder = True
has_vision_encoder = True
+1
View File
@@ -9,6 +9,7 @@ from .base import ModelBase, TextModel, gguf
@ModelBase.register("GPTRefactForCausalLM")
@ModelBase.example("smallcloudai/Refact-1_6-base")
class RefactModel(TextModel):
model_arch = gguf.MODEL_ARCH.REFACT
+4
View File
@@ -11,6 +11,7 @@ from .base import ModelBase, TextModel, gguf
@ModelBase.register("Rwkv6ForCausalLM")
@ModelBase.example("RWKV/v6-Finch-1B6-HF")
class Rwkv6Model(TextModel):
model_arch = gguf.MODEL_ARCH.RWKV6
@@ -83,6 +84,7 @@ class Rwkv6Model(TextModel):
@ModelBase.register("RWKV6Qwen2ForCausalLM")
@ModelBase.example("recursal/QRWKV6-32B-Instruct-Preview-v0.1")
class RWKV6Qwen2Model(Rwkv6Model):
model_arch = gguf.MODEL_ARCH.RWKV6QWEN2
@@ -136,6 +138,7 @@ class RWKV6Qwen2Model(Rwkv6Model):
@ModelBase.register("Rwkv7ForCausalLM", "RWKV7ForCausalLM")
@ModelBase.example("fla-hub/rwkv7-1.5B-world")
class Rwkv7Model(TextModel):
model_arch = gguf.MODEL_ARCH.RWKV7
@@ -261,6 +264,7 @@ class Rwkv7Model(TextModel):
@ModelBase.register("RwkvHybridForCausalLM")
@ModelBase.example("RWKV-Red-Team/ARWKV-7B-Preview-0.1")
class ARwkv7Model(Rwkv7Model):
model_arch = gguf.MODEL_ARCH.ARWKV7
+2
View File
@@ -12,6 +12,7 @@ from .qwenvl import Qwen2VLVisionModel
@ModelBase.register("Sarashina2VisionForCausalLM")
@ModelBase.example("sbintuitions/sarashina2.2-vision-3b")
class Sarashina2VLTextModel(LlamaModel):
model_arch = gguf.MODEL_ARCH.LLAMA
@@ -26,6 +27,7 @@ class Sarashina2VLTextModel(LlamaModel):
@ModelBase.register("Sarashina2VisionForCausalLM")
@ModelBase.example("sbintuitions/sarashina2.2-vision-3b")
class Sarashina2VLVisionModel(Qwen2VLVisionModel):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
+1
View File
@@ -11,6 +11,7 @@ from .base import ModelBase, TextModel, gguf, logger
@ModelBase.register("SmallThinkerForCausalLM")
@ModelBase.example("PowerInfer/SmallThinker-4BA0.6B-Instruct")
class SmallThinkerModel(TextModel):
model_arch = gguf.MODEL_ARCH.SMALLTHINKER
+1
View File
@@ -9,6 +9,7 @@ from .base import MmprojModel, ModelBase, gguf
@ModelBase.register("Idefics3ForConditionalGeneration", "SmolVLMForConditionalGeneration")
@ModelBase.example("HuggingFaceTB/SmolVLM-Instruct", "HuggingFaceM4/Idefics3-8B-Llama3")
class SmolVLMModel(MmprojModel):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
+1
View File
@@ -11,6 +11,7 @@ from .base import ModelBase, TextModel, gguf
@ModelBase.register("StableLmForCausalLM", "StableLMEpochForCausalLM", "LlavaStableLMEpochForCausalLM")
@ModelBase.example("stabilityai/stablelm-2-1_6b")
class StableLMModel(TextModel):
model_arch = gguf.MODEL_ARCH.STABLELM
+2
View File
@@ -4,6 +4,7 @@ from .base import ModelBase, TextModel, gguf
@ModelBase.register("GPTBigCodeForCausalLM")
@ModelBase.example("bigcode/gpt_bigcode-santacoder")
class StarCoderModel(TextModel):
model_arch = gguf.MODEL_ARCH.STARCODER
@@ -19,5 +20,6 @@ class StarCoderModel(TextModel):
@ModelBase.register("Starcoder2ForCausalLM")
@ModelBase.example("bigcode/starcoder2-3b")
class StarCoder2Model(TextModel):
model_arch = gguf.MODEL_ARCH.STARCODER2
+3
View File
@@ -16,6 +16,7 @@ from .qwen import Qwen3Model
@ModelBase.register("StepVLForConditionalGeneration", "Step3p7ForConditionalGeneration")
@ModelBase.example("stepfun-ai/Step3-VL-10B", "stepfun-ai/Step-3.7-Flash")
class Step3VLVisionModel(MmprojModel):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
@@ -91,11 +92,13 @@ class Step3VLVisionModel(MmprojModel):
@ModelBase.register("StepVLForConditionalGeneration")
@ModelBase.example("stepfun-ai/Step3-VL-10B")
class Step3VLTextModel(Qwen3Model):
model_arch = gguf.MODEL_ARCH.QWEN3
@ModelBase.register("Step3p5ForCausalLM", "Step3p7ForConditionalGeneration")
@ModelBase.example("stepfun-ai/Step-3.7-Flash")
class Step35Model(TextModel):
model_arch = gguf.MODEL_ARCH.STEP35
supports_mtp_export = True
+2
View File
@@ -16,6 +16,7 @@ from .base import ModelBase, SentencePieceTokenTypes, TextModel, gguf, logger
@ModelBase.register("MT5ForConditionalGeneration")
@ModelBase.register("UMT5ForConditionalGeneration")
@ModelBase.register("UMT5Model")
@ModelBase.example("google-t5/t5-small", "google/flan-t5-small", "google/umt5-small")
class T5Model(TextModel):
model_arch = gguf.MODEL_ARCH.T5
@@ -153,6 +154,7 @@ class T5Model(TextModel):
@ModelBase.register("T5EncoderModel")
@ModelBase.example("sentence-transformers/sentence-t5-base")
class T5EncoderModel(TextModel):
model_arch = gguf.MODEL_ARCH.T5ENCODER
+1
View File
@@ -11,6 +11,7 @@ from .base import LazyTorchTensor, ModelBase, TextModel, gguf
@ModelBase.register("TalkieForCausalLM")
@ModelBase.example("lewtun/talkie-1930-13b-it-hf")
class TalkieModel(TextModel):
model_arch = gguf.MODEL_ARCH.TALKIE
+7
View File
@@ -9,6 +9,7 @@ from .base import MmprojModel, ModelBase, TextModel, gguf
@ModelBase.register("UltravoxModel")
@ModelBase.example("fixie-ai/ultravox-v0_5-llama-3_2-1b")
class UltravoxModel(TextModel):
model_arch = gguf.MODEL_ARCH.LLAMA # dummy
@@ -18,6 +19,7 @@ class UltravoxModel(TextModel):
@ModelBase.register("GlmasrModel")
@ModelBase.example("zai-org/GLM-ASR-Nano-2512")
class GlmASRWhisperEncoderModel(MmprojModel):
has_vision_encoder = False
has_audio_encoder = True
@@ -82,6 +84,7 @@ class GlmASRWhisperEncoderModel(MmprojModel):
@ModelBase.register("Qwen2AudioForConditionalGeneration")
@ModelBase.example("Qwen/Qwen2-Audio-7B-Instruct")
class WhisperEncoderModel(MmprojModel):
has_vision_encoder = False # no vision encoder
has_audio_encoder = True
@@ -123,6 +126,7 @@ class WhisperEncoderModel(MmprojModel):
@ModelBase.register("UltravoxModel")
@ModelBase.example("fixie-ai/ultravox-v0_5-llama-3_2-1b")
class UltravoxWhisperEncoderModel(WhisperEncoderModel):
has_vision_encoder = False # no vision encoder
has_audio_encoder = True
@@ -134,6 +138,7 @@ class UltravoxWhisperEncoderModel(WhisperEncoderModel):
@ModelBase.register("MERaLiON2ForConditionalGeneration")
@ModelBase.example("MERaLiON/MERaLiON-2-3B")
class MERaLiONWhisperEncoderModel(WhisperEncoderModel):
has_vision_encoder = False
has_audio_encoder = True
@@ -180,6 +185,7 @@ class MERaLiONWhisperEncoderModel(WhisperEncoderModel):
@ModelBase.register("VoxtralForConditionalGeneration")
@ModelBase.example("mistralai/Voxtral-Mini-3B-2507")
class VoxtralWhisperEncoderModel(WhisperEncoderModel):
has_vision_encoder = False # no vision encoder
has_audio_encoder = True
@@ -191,6 +197,7 @@ class VoxtralWhisperEncoderModel(WhisperEncoderModel):
@ModelBase.register("AudioFlamingo3ForConditionalGeneration")
@ModelBase.example("nvidia/audio-flamingo-3-hf")
class AudioFlamingo3WhisperEncoderModel(WhisperEncoderModel):
def set_gguf_parameters(self):
super().set_gguf_parameters()
+1
View File
@@ -9,6 +9,7 @@ from .base import ModelBase, TextModel, gguf, logger
@ModelBase.register("WavTokenizerDec")
@ModelBase.example("novateur/WavTokenizer-large-speech-75token")
class WavTokenizerDecModel(TextModel):
model_arch = gguf.MODEL_ARCH.WAVTOKENIZER_DEC
+1
View File
@@ -11,6 +11,7 @@ from .base import ModelBase, TextModel, gguf
@ModelBase.register("XverseForCausalLM")
@ModelBase.example("xverse/XVERSE-7B")
class XverseModel(TextModel):
model_arch = gguf.MODEL_ARCH.XVERSE
+1
View File
@@ -9,6 +9,7 @@ from .base import MmprojModel, ModelBase, gguf, logger
@ModelBase.register("YoutuVLForConditionalGeneration")
@ModelBase.example("tencent/Youtu-VL-4B-Instruct")
class YoutuVLVisionModel(MmprojModel):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
+4
View File
@@ -29,6 +29,7 @@ The required steps to implement for an HF model are:
```python
@ModelBase.register("MyModelForCausalLM")
@ModelBase.example("user/model")
class MyModel(TextModel):
model_arch = gguf.MODEL_ARCH.MYMODEL
```
@@ -37,10 +38,13 @@ or
```python
@ModelBase.register("MyModelForConditionalGeneration")
@ModelBase.example("user/model")
class MyModel(MmprojModel):
model_arch = gguf.MODEL_ARCH.MYMODEL
```
The `example` should point to a valid Hugging Face model that will be used for testing. You can add multiple models if necessary. Prefer a non-gated model, or tiny random weights if no such model exists.
2. Define the layout of the GGUF tensors in [constants.py](/gguf-py/gguf/constants.py)
Add an enum entry in `MODEL_ARCH`, the model human friendly name in `MODEL_ARCH_NAMES` and the GGUF tensor names in `MODEL_TENSORS`.
+3 -3
View File
@@ -77,8 +77,8 @@ Legend:
| MUL_MAT_ID | ❌ | 🟡 | ✅ | ✅ | 🟡 | 🟡 | 🟡 | ✅ | ✅ | 🟡 | 🟡 | ❌ |
| NEG | ❌ | ✅ | ✅ | 🟡 | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| NORM | ❌ | ✅ | ✅ | ✅ | 🟡 | ✅ | ✅ | ✅ | ✅ | 🟡 | ❌ | ❌ |
| OPT_STEP_ADAMW | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | | ✅ | ❌ | ❌ | ❌ |
| OPT_STEP_SGD | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | | ✅ | ❌ | ❌ | ❌ |
| OPT_STEP_ADAMW | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | | ✅ | ❌ | ❌ | ❌ |
| OPT_STEP_SGD | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | | ✅ | ❌ | ❌ | ❌ |
| OUT_PROD | 🟡 | 🟡 | 🟡 | 🟡 | ❌ | ❌ | ❌ | 🟡 | 🟡 | ❌ | ❌ | 🟡 |
| PAD | ❌ | 🟡 | ✅ | 🟡 | ❌ | 🟡 | 🟡 | 🟡 | ✅ | ✅ | ❌ | ❌ |
| PAD_REFLECT_1D | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
@@ -98,7 +98,7 @@ Legend:
| RWKV_WKV7 | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| SCALE | ❌ | 🟡 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| SET | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | 🟡 | ✅ | ✅ | ❌ | ❌ |
| SET_ROWS | ❌ | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | ❌ | ❌ |
| SET_ROWS | ❌ | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | | 🟡 | 🟡 | ❌ | ❌ |
| SGN | ❌ | ✅ | ✅ | 🟡 | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| SIGMOID | ❌ | ✅ | ✅ | 🟡 | 🟡 | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
| SILU | ❌ | ✅ | ✅ | 🟡 | 🟡 | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
+640 -20006
View File
File diff suppressed because it is too large Load Diff
+6
View File
@@ -29,6 +29,12 @@ identify which PRs require a version bump before cutting a release._
Releases are created by running the [make-release](.github/workflows/make-release.yml)
which is a manual workflow.
The workflow runs against the branch selected in the "Run workflow" dialog
(default `master`) and takes an optional `commit` SHA. When a commit is given,
the workflow validates that the commit belongs to the branch and is not older
than 3 days from the branch HEAD, then releases that commit instead of the
branch HEAD.
The workflow creates an annotated git tag (e.g. `v0.1.0`) and pushes it to the
remote. No GitHub Release object is created, the tag is the release artifact.
+1 -1
View File
@@ -4,6 +4,6 @@
# Copyright (C) 2026 Intel Corporation
# SPDX-License-Identifier: MIT
./build/bin/test-backend-ops support --output csv > docs/ops/SYCL.csv
./build/bin/test-backend-ops -b SYCL0 support --output csv > docs/ops/SYCL.csv
./scripts/create_ops_docs.py
+1 -1
View File
@@ -5,7 +5,7 @@ project("ggml" C CXX ASM)
### GGML Version
set(GGML_VERSION_MAJOR 0)
set(GGML_VERSION_MINOR 20)
set(GGML_VERSION_PATCH 0)
set(GGML_VERSION_PATCH 1)
set(GGML_VERSION_BASE "${GGML_VERSION_MAJOR}.${GGML_VERSION_MINOR}.${GGML_VERSION_PATCH}")
list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake/")

Some files were not shown because too many files have changed in this diff Show More