* llama : support for MiniMax-Text-01 model
* chore : renames to match the other MiniMax models
* model : add logits mask as MiniMax-Text-01 embeddings tensor has zero-valued embeddings for tokens >= 200032 that produce zero logits disrupting the token sampling process
* llama : replace hardcoded conditions with hparams.is_recr()
* model : used build_rs() for recurrent state management
* chore : code cleanup
* model : optimized MiniMax-Text-01 by removing the state tranpose operations
* chore : removed unnecessary ggml_cont() in MiniMax-Text-01 implementation
* llama : add generic logits mask graph input
* model : permuted diag_decay dimensions to avoid doing it inside MiniMax-Text-01 graph
* chore : code cleanup
* chore : code cleanup
* model : use token positions when calculating MiniMax-Text-01 decay tensors
* convert : add support for MiniMaxM1ForCausalLM as it seems to be the same as MiniMaxText01ForCausalLM
* chat : add jinja template for MiniMax-M1
Co-authored-by: QscQ <qscqesze@gmail.com>
* chore : code cleanup
* tests : MINIMAX_01-related fixes
* chore : silence Python lint errors
* vocab : remove unnecessary vocab type
* convert : update MiniMaxText01Model conversion to use yield when modifying tensors
* convert : suppress tokens with zero-valued embeddings during MiniMax-Text-01 conversion
* llama : removed logits mask - no longer necessary as token suppression is used instead
* model : use common functions to make MiniMax-Text-01 implementation more concise
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* model : use common functions to make MiniMax-Text-01 implementation more concise
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* convert : override non-working built-in chat template during conversion
* tests : skip arch MINIMAX_01 tests for WebGPU backend (it breaks again)
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Co-authored-by: QscQ <qscqesze@gmail.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* llama : MTP support for DeepSeek V3.2
* model : no need to include MTP layers during DeepSeek V3.2 model type discovery
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* llama : load MTP tensors only if they are really used
* llama : skip loading MTP (if not used) in remaining models that support MTP
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* llama : enforce the same K and V cache types for DeepSeek V4; enable FA if V cache is quantized
* llama : enforce the same K and V cache types for MLA models
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* cuda : CUDA GGML_OP_LIGHTNING_INDEXER implementation (generic vector kernel + wmma kernel)
* chore : remove indentation of #pragma unroll
* cuda : remove unnecessary kernel template declarations
* cuda : add WARPS_PER_BLOCK and K_VECS_PER_BLOCK template parameters in lightning indexer kernels to avoid duplication of constants.
* cuda : relax MMA architecture requirements to Turing in lightning indexer implementation
* chore : renamed variables
* chore : rename ggml_cuda_op_lightning_indexer() to ggml_cuda_lightning_indexer()
* chore : TODO for AMD rocWMMA
* chore : whitespace formatting
* chore : another variable rename to fix problems caused by shadowing
* chore : yet another rename, this time uppercased all constants
* cuda : added alignment checks for Q and K tensors in lightning indexer implementation
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* llama : make all KQ masks (except the lightning indexer one) f16 if FA is used and remove zero attention bias in DeepSeek V4
* llama : remove dead code that repeats unified raw_k cache for each stream in DeepSeek V4 - no longer needed as raw_k is always non-unified.
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* ggml : process data in smaller chunks in CUDA ggml_top_k() implementation to reduce temporary buffers memory usage
* ggml : allocate tmp_dst only only once before the loop
* chore : whitespaces
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* ggml : use chunked processing in both CUDA CUB top-k and argsort implementations
* chore : separate argsort_f32_i32_cuda_bitonic() call from return statement
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* chore : replace ternary operators with min/max
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* ggml : add support for CPU f16->f16 GGML_OP_SET_ROWS
* ggml : add missing type checks in f16 GGML_OP_SET_ROWS
* ggml : merge ggml_compute_forward_set_rows_f32() and ggml_compute_forward_set_rows_f16() into ggml_compute_forward_set_rows_impl()
* chore : replace assert() with GGML_ASSERT()
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* vulkan : check src0 type in GGML_OP_SET_ROWS to avoid failures due to unimplemented f16 support
* chore : get rid of else
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* cuda : concat implementation for quantized types
* chore : apply am17an clever suggestion to shorten the code
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* llama : introduce llama_set_warmup() API call that controls warmup mode; use all MoE experts during warmup
* common : use new API to enable warmup mode during model warmup
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* convert : extend DEEPSEEK2 model architecture to support DeepseekV3ForCausalLM by adding EXPERT_WEIGHTS_NORM and EXPERT_GATING_FUNC model parameters and FFN_EXP_PROBS_B tensor type
* vocab : add DeepSeek V3 pre-tokenizer regexes
* unicode : handle ACCENT_MARK and SYMBOL categories in regex
* llama : add DeepSeek V3 chat template, handle new model parameters and tensor types
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* convert : use GPT2 vocab for Phi-4 model
* convert : use null value of sliding_window to distinguish Phi-4 from other PHI3-based models
* llama : do not use sliding window attention mask for Phi-4 model
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* gguf-py : add T5ENCODER model architecture
* common : call llama_decode() during warmup only if the model has decoder
* convert-hf : add T5EncoderModel
* llama : add llama_model_has_decoder() API function
* llama : split build_t5() into build_t5_encoder() and build_t5_decoder()
* llama : add support for LLM_ARCH_T5ENCODER
* llama-embedding : add support for LLAMA_POOLING_TYPE_NONE
* llama-embedding : add support for encoder-only models
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* gguf-py, llama : add constants and methods related to Llama-3.1 <|eom_id|> token
* llama : find Llama-3.1 <|eom_id|> token id during vocab loading
* llama-vocab : add Llama-3.1 <|eom_id|> token to the set of tokens stopping the generation
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* llama : add inference support and model types for T5 and FLAN-T5 model families
* llama : add new API functions to support encoder-decoder models: llama_encode(), llama_model_has_encoder(), llama_model_decoder_start_token()
* common, llama-cli, llama-batched : add support for encoder-decoder models
* convert-hf : handle shared token embeddings tensors in T5Model
* convert-hf : add support for SentencePiece BPE tokenizer in T5Model (for Pile-T5 models)
* convert-hf : add MT5ForConditionalGeneration and UMT5ForConditionalGeneration to architectures supported by T5Model
* convert : add t5 tokenizer tests, use "slow" HF tokenizer for t5
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* llama : add T5 model architecture, tensors and model header parameters
* llama : add implementation of Unigram tokenizer with SentencePiece-like text normalization using precompiled charsmap
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* gguf-py : add T5 model architecture
* gguf-py : add separate tensors for encoder and decoder
* gguf-py : add new model header parameters: decoder_start_token_id, attention.relative_buckets_count, tokenizer.ggml.remove_extra_whitespaces, tokenizer.ggml.precompiled_charsmap
* convert-hf : add model conversion support for T5ForConditionalGeneration and T5WithLMHeadModel
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* common : increase max number of experts to 160
* common : add tensors ATTN_Q_A, ATTN_Q_A_NORM, ATTN_Q_B, ATTN_KV_A_MQA, ATTN_KV_A_NORM, ATTN_KV_B needed by DeepSeek-V2 MLA (multi-head latent attention) architecture
* common : add model header parameters: leading_dense_block_count, expert_feed_forward_length, expert_shared_count, expert_weights_scale, attention.q_lora_rank, attention.kv_lora_rank, rope.scaling.yarn_log_multiplier
* convert-hf : add model conversion support for DeepseekV2ForCausalLM
* llama : add model types for DeepSeek-V2 and DeepSeek-V2-Lite models
* llama : add two new llm_build_moe_ffn() arguments: scale_w (whether to scale weights of selected MoE experts) and w_scale (numerical value of the scaling factor)
* llama : add inference support for LLM_ARCH_DEEPSEEK2
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* common : increase max number of experts to 128
* common : add tensor LLM_TENSOR_FFN_NORM_EXPS for normalization before MoE that runs in parallel to attention + ffn
* gguf-py : add architecture-specific block mappings that override selected general block mappings
* convert-hf : add model conversion support for ArcticForCausalLM
* convert-hf : use added_tokens_decoder from tokenizer_config.json to redefine tokens from SentencePiece model (only for ArcticForCausalLM)
* llama : add inference support for LLM_ARCH_ARCTIC
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* convert-hf : add conversion of bloom-style qkv tensor to gpt-style qkv (code borrowed from BloomModel)
* llama : add inference support for LLM_ARCH_GPTNEOX
* llama : add model types for every Pythia variant and GPT-NeoX
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* llama : use n_embd_head_v instead of n_embd_head_k when reshaping kqv
* llama : use n_embd_v_gqa and n_embd_head_v instead of n_embd_k_gqa and n_embd_head_k when making a view of cached value vectors.
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
