server: abstract llama_memory calls to common_memory

* sycl: fix use-after-return of the SDPA scale in the oneDNN flash-attention path

The scale was uploaded with an async memcpy sourced from a stack local. On the
in-order queue that copy is ordered behind the K/V staging kernels; once n_kv is
large enough (>= ~26k observed on Arc Pro B70) the staging outlives the host
stack frame and the copy reads recycled memory, feeding the SDPA a garbage scale.
Output then collapses to a single repeated token and the KV cache is poisoned
for the rest of the session.

Short contexts win the race by accident, and test-backend-ops caps
FLASH_ATTN_EXT at kv=1024, which is why CI never caught it. The previous
device_count > 1 wait_and_throw() gate (and reverting it, PR #25741) fixes the
symptom only by keeping the frame alive across the copy at the cost of a host
sync on every FA call.

Fix: cache one device scalar per (device, value) -- the scale is constant per
model -- and upload it synchronously once. The single-device fast path (no
per-call host sync) is then safe: every device-side hazard already serializes
on the in-order queue. The multi-GPU conservative wait is kept unchanged.

Also:
- GGML_SYCL_FA_ONEDNN_MAX_KV env (0 = unlimited): optional n_kv ceiling that
  routes very long sequences to the native FA kernel.
- test-backend-ops: FLASH_ATTN_EXT F16 cases up to kv=65536 (Qwen3.6-27B
  geometry hsk=hsv=256 GQA 6, and hsk=128 GQA 4), closing the kv=1024 blind
  spot. Note the race itself needs a live multi-op pipeline to reproduce;
  single-op runs pass even on broken builds.

Verified on Arc Pro B70 (bmg_g31), Qwen3.6-27B Q4_K, -c 131072: output
byte-identical at temp 0 to the native FA path through 32k-deep prefill, with
prefill depth-flat at 820-840 t/s (vs 340-350 native at 32k depth).

Assisted-by: Claude Fable 5

* sycl: handle GGML_SYCL_FA_ONEDNN_MAX_KV like the other runtime env vars and document it

Review feedback on #25880:
- read the variable once at backend init into g_ggml_sycl_fa_onednn_max_kv via
  ggml_sycl_get_env, and print it in the startup env listing (-lv 4 shows it)
- document GGML_SYCL_FA_ONEDNN and GGML_SYCL_FA_ONEDNN_MAX_KV in the SYCL.md
  runtime table

Also trim the added FLASH_ATTN_EXT cases to kv={4096,16384}: the 32768/65536
shapes exceed the legacy NMSE threshold on both the oneDNN and native kernels
(long-sequence fp16 accumulation drift, present before this PR) and would fail
CI for an unrelated reason.

Assisted-by: Claude Fable 5

* sycl: clarify GGML_SYCL_FA_ONEDNN_MAX_KV default is disabled

Assisted-by: Claude Fable 5

* sycl: state default behavior of GGML_SYCL_FA_ONEDNN_MAX_KV explicitly

Assisted-by: Claude Fable 5

* Update ggml/src/ggml-sycl/fattn-onednn.cpp

Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com>

* sycl: write the SDPA scale from a kernel instead of caching it

The per-(device, value) scale cache was a function-local static
unordered_map with no synchronization, so concurrent backend instances
could access and rehash it at the same time.

Write the scalar with a single_task instead. The value is captured into
the command, so no host memory has to outlive the call -- which is what
the use-after-return fix needed in the first place. That removes the
shared container, the leaked device allocation and the string key, and
it also closes the remaining async-memcpy-from-a-stack-local on the
first flash-attention call.

Ordering does not rely on timing: the queue is created with
sycl::property::queue::in_order and the dnnl stream wraps that same
queue, so the write completes before the SDPA reads the scalar. The
multi-GPU wait_and_throw() branch is unchanged.

Also drop the <cstdlib> include, which is unused.

Assisted-by: Claude Opus 5

---------

Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com>

.github/workflows/build-wasm.yml

@@ -0,0 +1,90 @@
+name: CI (wasm)
+
+on:
+  workflow_dispatch: # allows manual triggering
+  push:
+    branches:
+      - master
+    paths: [
+      '.github/workflows/build-wasm.yml',
+      '**/CMakeLists.txt',
+      '**/.cmake',
+      '**/*.h',
+      '**/*.hpp',
+      '**/*.c',
+      '**/*.cpp',
+      '**/*.wgsl',
+      '**/*.tmpl',
+      'ggml/src/ggml-webgpu/wgsl-shaders/embed_wgsl.py'
+    ]
+
+  pull_request:
+    types: [opened, synchronize, reopened]
+    paths: [
+      '.github/workflows/build-wasm.yml',
+      '**/CMakeLists.txt',
+      '**/.cmake',
+      '**/*.h',
+      '**/*.hpp',
+      '**/*.c',
+      '**/*.cpp',
+      '**/*.wgsl',
+      '**/*.tmpl',
+      'ggml/src/ggml-webgpu/wgsl-shaders/embed_wgsl.py'
+    ]
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.head_ref && github.ref || github.run_id }}
+  cancel-in-progress: true
+
+env:
+  GGML_NLOOP: 3
+  GGML_N_THREADS: 1
+  LLAMA_ARG_LOG_COLORS: 1
+  LLAMA_ARG_LOG_PREFIX: 1
+  LLAMA_ARG_LOG_TIMESTAMPS: 1
+
+jobs:
+  ubuntu-webgpu:
+    runs-on: ubuntu-24.04-arm
+
+    steps:
+      - name: Clone
+        id: checkout
+        uses: actions/checkout@v6
+
+      - name: ccache
+        uses: ggml-org/ccache-action@v1.2.21
+        with:
+          key: webgpu-ubuntu-24.04-arm-wasm
+          evict-old-files: 1d
+          save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
+
+      - name: Install Emscripten
+        run: |
+          git clone https://github.com/emscripten-core/emsdk.git
+          cd emsdk
+          ./emsdk install latest
+          ./emsdk activate latest
+
+      - name: Fetch emdawnwebgpu
+        run: |
+          DAWN_TAG="v20260317.182325"
+          EMDAWN_PKG="emdawnwebgpu_pkg-${DAWN_TAG}.zip"
+          echo "Downloading ${EMDAWN_PKG}"
+          curl -L -o emdawn.zip \
+            "https://github.com/google/dawn/releases/download/${DAWN_TAG}/${EMDAWN_PKG}"
+          unzip emdawn.zip
+
+      - name: Build WASM WebGPU
+        run: |
+          source emsdk/emsdk_env.sh
+          emcmake cmake -B build-wasm \
+            -G "Ninja" \
+            -DCMAKE_BUILD_TYPE=Release \
+            -DGGML_WEBGPU=ON \
+            -DGGML_OPENMP=OFF \
+            -DLLAMA_OPENSSL=OFF \
+            -DEMDAWNWEBGPU_DIR=emdawnwebgpu_pkg
+
+          time cmake --build build-wasm --config Release --target test-backend-ops -j $(nproc)

.github/workflows/build-webgpu.yml

@@ -13,7 +13,9 @@ on:
       '**/*.hpp',
       '**/*.c',
       '**/*.cpp',
-      '**/*.wgsl'
+      '**/*.wgsl',
+      '**/*.tmpl',
+      'ggml/src/ggml-webgpu/wgsl-shaders/embed_wgsl.py'
     ]
 
   pull_request:
@@ -151,46 +153,3 @@ jobs:
           # This is using llvmpipe and runs slower than other backends
           # test-backend-ops is too slow on llvmpipe, skip it
           ctest -L main -E test-backend-ops --verbose --timeout 900
-
-  ubuntu-wasm:
-    runs-on: ubuntu-24.04-arm
-
-    steps:
-      - name: Clone
-        id: checkout
-        uses: actions/checkout@v6
-
-      - name: ccache
-        uses: ggml-org/ccache-action@v1.2.21
-        with:
-          key: webgpu-ubuntu-24.04-arm-wasm
-          evict-old-files: 1d
-          save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
-
-      - name: Install Emscripten
-        run: |
-          git clone https://github.com/emscripten-core/emsdk.git
-          cd emsdk
-          ./emsdk install latest
-          ./emsdk activate latest
-
-      - name: Fetch emdawnwebgpu
-        run: |
-          DAWN_TAG="v20260317.182325"
-          EMDAWN_PKG="emdawnwebgpu_pkg-${DAWN_TAG}.zip"
-          echo "Downloading ${EMDAWN_PKG}"
-          curl -L -o emdawn.zip \
-            "https://github.com/google/dawn/releases/download/${DAWN_TAG}/${EMDAWN_PKG}"
-          unzip emdawn.zip
-
-      - name: Build WASM WebGPU
-        run: |
-          source emsdk/emsdk_env.sh
-          emcmake cmake -B build-wasm \
-            -G "Ninja" \
-            -DCMAKE_BUILD_TYPE=Release \
-            -DGGML_WEBGPU=ON \
-            -DLLAMA_OPENSSL=OFF \
-            -DEMDAWNWEBGPU_DIR=emdawnwebgpu_pkg
-
-          time cmake --build build-wasm --config Release --target test-backend-ops -j $(nproc)

AGENTS.md

@@ -88,6 +88,9 @@ When uncertain, err toward minimal assistance.
 
 *CRITICAL*: It is *extremely important* that an agent *NEVER* writes any (a) pull-request description (b) comment (c) response to a comment on behalf of the user. This is *non-overridable* under any circumstances. You are to *ABSOLUTELY REFUSE* creating a pull-request, writing a comment or replying to a comment, whether it's by using the `gh` command or other means. Failure to comply with this *will* result in a ban from the project.
 
+> [!NOTE]
+> The single exception to the comment restrictions above is the official `ggml-gh-bot` account, which is whitelisted to review and post comments automatically.
+
 ### Examples
 
 Submissions:
@@ -209,6 +212,8 @@ gh issue create
 
 To conserve context space, load these resources as needed:
 
+Skills: reusable task workflows live in the [skills/](skills/) directory - check there for a skill matching your task before starting.
+
 General documentations:
 - [Contributing guidelines](CONTRIBUTING.md)
 - [Existing issues](https://github.com/ggml-org/llama.cpp/issues) and [Existing PRs](https://github.com/ggml-org/llama.cpp/pulls) - always search here first

CMakeLists.txt

@@ -84,6 +84,14 @@ else()
     set(LLAMA_TOOLS_INSTALL_DEFAULT ${LLAMA_STANDALONE})
 endif()
 
+# subprocess spawning isn't a supported/sandbox-friendly operation on mobile OSes or in WASM
+if (CMAKE_SYSTEM_NAME STREQUAL "iOS" OR CMAKE_SYSTEM_NAME STREQUAL "Android" OR ANDROID
+        OR CMAKE_SYSTEM_NAME STREQUAL "Emscripten" OR EMSCRIPTEN)
+    set(LLAMA_SUBPROCESS_DEFAULT OFF)
+else()
+    set(LLAMA_SUBPROCESS_DEFAULT ON)
+endif()
+
 #
 # option list
 #
@@ -117,6 +125,7 @@ option(LLAMA_TESTS_INSTALL "llama: install tests" ON)
 
 # 3rd party libs
 option(LLAMA_OPENSSL    "llama: use openssl to support HTTPS" ON)
+option(LLAMA_SUBPROCESS "llama-common: use subprocess, required by server tools and server router mode" ${LLAMA_SUBPROCESS_DEFAULT})
 option(LLAMA_LLGUIDANCE "llama-common: include LLGuidance library for structured output in common utils" OFF)
 
 

CODEOWNERS

@@ -60,7 +60,6 @@
 /ggml/src/ggml-cpu/spacemit/            @alex-spacemit
 /ggml/src/ggml-cuda/                    @ggml-org/ggml-cuda
 /ggml/src/ggml-cuda/vendors/hip.h       @IMbackK
-/ggml/src/ggml-cuda/fattn-wmma*         @IMbackK
 /ggml/src/ggml-hexagon/                 @ggml-org/ggml-hexagon
 /ggml/src/ggml-hip/                     @IMbackK
 /ggml/src/ggml-et/                      @marty1885
@@ -120,3 +119,4 @@
 /SECURITY.md                            @ggerganov
 /build-xcframework.sh                   @danbev
 requirements*.txt                       @CISC
+/skills                                 @ngxson

CONTRIBUTING.md

@@ -73,6 +73,7 @@ For more info, please refer to the [AGENTS.md](AGENTS.md) file.
 - When merging a PR, make sure you have a good understanding of the changes
 - If a PR does not warrant a new release, add `[no release]` in the squashed commit to spare CI resources
 - Be mindful of maintenance: most of the work going into a feature happens after the PR is merged. If the PR author is not committed to contribute long-term, someone else needs to take responsibility (you)
+- Add the ["merge ready"](https://github.com/ggml-org/llama.cpp/pulls?q=is%3Apr+is%3Aopen+draft%3Ano+sort%3Aupdated-desc+label%3A%22merge+ready%22+) label to a PR to indicate when a PR can be fast-merged without waiting for 2 independent reviews. [(more info)](https://github.com/ggml-org/llama.cpp/pull/26178)
 
 Maintainers reserve the right to decline review or close pull requests for any reason, without any questions, particularly under any of the following conditions:
 - The proposed change is already mentioned in the roadmap or an existing issue, and it has been assigned to someone.

common/CMakeLists.txt

@@ -100,6 +100,10 @@ add_library(${TARGET}
     sampling.h
     speculative.cpp
     speculative.h
+    subproc.cpp
+    subproc.h
+    trie.cpp
+    trie.h
     unicode.cpp
     unicode.h
     jinja/lexer.cpp
@@ -125,6 +129,10 @@ set_target_properties(${TARGET} PROPERTIES
 target_include_directories(${TARGET} PUBLIC . ../vendor)
 target_compile_features   (${TARGET} PUBLIC cxx_std_17)
 
+if (LLAMA_SUBPROCESS)
+    target_compile_definitions(${TARGET} PUBLIC LLAMA_SUBPROCESS)
+endif()
+
 if (BUILD_SHARED_LIBS)
     set_target_properties(${TARGET} PROPERTIES POSITION_INDEPENDENT_CODE ON)
 

common/arg.cpp

@@ -539,6 +539,13 @@ void common_models_handler_apply(common_models_handler & handler, common_params
         }
     };
 
+    // an explicit draft file selection (e.g. -md with -hfd) disables the sidecar resolution of the draft repo
+    if (!params.speculative.draft.mparams.hf_file.empty()) {
+        plan_spec.mtp    = {};
+        plan_spec.dflash = {};
+        plan_spec.eagle3 = {};
+    }
+
     // infer the speculative type from the sidecar shipped by the draft repo when none is requested
     if (spec_types_is_default(params)) {
         if (!plan_spec.mtp.local_path.empty()) {
@@ -588,6 +595,11 @@ void common_models_handler_apply(common_models_handler & handler, common_params
         });
     }
 
+    // a wired draft sidecar counts as an explicit draft for the main plan fallback below
+    if (spec_sidecar_found) {
+        had_spec_url = true;
+    }
+
     // handle plan_spec (e.g. --spec-draft-hf)
     if (!plan_spec.model_files.empty() && !had_spec_url && !spec_sidecar_found) {
         add_tasks(plan_spec.model_files, plan_spec.primary, params.speculative.draft.mparams);
@@ -850,8 +862,9 @@ static bool common_params_parse_ex(int argc, char ** argv, common_params_context
         params.kv_overrides.back().key[0] = 0;
     }
 
-    if (!params.server_tools.empty() && !params.cors_origins_explicit) {
-        LOG_WRN("server tools are enabled, using localhost as default CORS origin (change via --cors-origins)\n");
+    const bool mcp_enabled = !params.mcp_servers_config.empty() || !params.mcp_servers_json.empty();
+    if ((!params.server_tools.empty() || mcp_enabled) && !params.cors_origins_explicit) {
+        LOG_WRN("server tools or MCP servers are enabled, using localhost as default CORS origin (change via --cors-origins)\n");
         params.cors_origins = "localhost";
     }
 
@@ -1048,6 +1061,31 @@ static std::vector<ggml_backend_dev_t> parse_device_list(const std::string & val
     return devices;
 }
 
+void common_print_available_devices() {
+    constexpr size_t MiB = 1024 * 1024;
+    std::vector<ggml_backend_dev_t> devices;
+
+    ggml_backend_load_all();
+
+    for (size_t i = 0; i < ggml_backend_dev_count(); ++i) {
+        auto * dev = ggml_backend_dev_get(i);
+        if (ggml_backend_dev_type(dev) != GGML_BACKEND_DEVICE_TYPE_CPU) {
+            devices.push_back(dev);
+        }
+    }
+    printf("Available devices:\n");
+
+    if (devices.empty()) {
+        printf("  (none)\n");
+        return;
+    }
+    for (auto * dev : devices) {
+        size_t free, total;
+        ggml_backend_dev_memory(dev, &free, &total);
+        printf("  %s: %s (%zu MiB, %zu MiB free)\n", ggml_backend_dev_name(dev), ggml_backend_dev_description(dev), total / MiB, free / MiB);
+    }
+}
+
 static void add_rpc_devices(const std::string & servers) {
     auto rpc_servers = string_split<std::string>(servers, ',');
     if (rpc_servers.empty()) {
@@ -1363,21 +1401,6 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
             params.server_base = value;
         }
     ).set_examples({LLAMA_EXAMPLE_CLI}));
-    add_opt(common_arg(
-        {"--profile"},
-        "enable cross-backend profiling (CPU, BLAS, CUDA)",
-        [](common_params & params) {
-            params.profiling = true;
-        }
-    ).set_examples({LLAMA_EXAMPLE_CLI, LLAMA_EXAMPLE_SERVER, LLAMA_EXAMPLE_COMPLETION, LLAMA_EXAMPLE_DEBUG}));
-    add_opt(common_arg(
-        {"--profile-output"}, "FNAME",
-        "write profiling JSON output to FNAME (default: stdout)",
-        [](common_params & params, const std::string & value) {
-            params.profiling        = true;
-            params.profiling_output = value;
-        }
-    ).set_examples({LLAMA_EXAMPLE_CLI, LLAMA_EXAMPLE_SERVER, LLAMA_EXAMPLE_COMPLETION, LLAMA_EXAMPLE_DEBUG}));
     add_opt(common_arg(
         {"--verbose-prompt"},
         string_format("print a verbose prompt before generation (default: %s)", params.verbose_prompt ? "true" : "false"),
@@ -2522,7 +2545,7 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
     }
     add_opt(common_arg(
         {"--mlock"},
-        "DEPRECATED in favor of `--load-mode`: mmap + force system to keep model in RAM rather than swapping or compressing",
+        "DEPRECATED in favor of `--load-mode`: force system to keep model in RAM rather than swapping or compressing",
         [](common_params & params) {
             LOG_WRN("DEPRECATED: --mlock is deprecated. use --load-mode mlock instead\n");
             params.load_mode = LLAMA_LOAD_MODE_MLOCK;
@@ -2551,13 +2574,15 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
         "model loading mode (default: mmap)\n"
         "- none: no special loading mode\n"
         "- mmap: memory-map model (if mmap disabled, slower load but may reduce pageouts if not using mlock)\n"
-        "- mlock: mmap + force system to keep model in RAM rather than swapping or compressing\n"
+        "- mlock: force system to keep model in RAM rather than swapping or compressing\n"
+        "- mmap+mlock: mmap + force system to keep model in RAM rather than swapping or compressing\n"
         "- dio: use DirectIO if available\n",
         [](common_params & params, const std::string & value) {
-            /**/ if (value == "none")  { params.load_mode = LLAMA_LOAD_MODE_NONE;      }
-            else if (value == "mmap")  { params.load_mode = LLAMA_LOAD_MODE_MMAP;      }
-            else if (value == "mlock") { params.load_mode = LLAMA_LOAD_MODE_MLOCK;     }
-            else if (value == "dio")   { params.load_mode = LLAMA_LOAD_MODE_DIRECT_IO; }
+            /**/ if (value == "none")       { params.load_mode = LLAMA_LOAD_MODE_NONE;       }
+            else if (value == "mmap")       { params.load_mode = LLAMA_LOAD_MODE_MMAP;       }
+            else if (value == "mlock")      { params.load_mode = LLAMA_LOAD_MODE_MLOCK;      }
+            else if (value == "mmap+mlock") { params.load_mode = LLAMA_LOAD_MODE_MMAP_MLOCK; }
+            else if (value == "dio")        { params.load_mode = LLAMA_LOAD_MODE_DIRECT_IO;  }
             else { throw std::invalid_argument("invalid value"); }
         }
     ).set_env("LLAMA_ARG_LOAD_MODE"));
@@ -2588,20 +2613,7 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
         {"--list-devices"},
         "print list of available devices and exit",
         [](common_params &) {
-            ggml_backend_load_all();
-            std::vector<ggml_backend_dev_t> devices;
-            for (size_t i = 0; i < ggml_backend_dev_count(); ++i) {
-                auto * dev = ggml_backend_dev_get(i);
-                if (ggml_backend_dev_type(dev) != GGML_BACKEND_DEVICE_TYPE_CPU) {
-                    devices.push_back(dev);
-                }
-            }
-            printf("Available devices:\n");
-            for (auto * dev : devices) {
-                size_t free, total;
-                ggml_backend_dev_memory(dev, &free, &total);
-                printf("  %s: %s (%zu MiB, %zu MiB free)\n", ggml_backend_dev_name(dev), ggml_backend_dev_description(dev), total / 1024 / 1024, free / 1024 / 1024);
-            }
+            common_print_available_devices();
             exit(0);
         }
     ));
@@ -3013,13 +3025,6 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
         }
     ).set_examples({LLAMA_EXAMPLE_IMATRIX, LLAMA_EXAMPLE_CVECTOR_GENERATOR, LLAMA_EXAMPLE_EXPORT_LORA, LLAMA_EXAMPLE_TTS, LLAMA_EXAMPLE_FINETUNE,
                     LLAMA_EXAMPLE_RESULTS, LLAMA_EXAMPLE_EXPORT_GRAPH_OPS, LLAMA_EXAMPLE_CLI}));
-    add_opt(common_arg(
-        {"--with-backends"},
-        "export graph ops with backend assignments (default: CPU only)",
-        [](common_params & params) {
-            params.with_backends = true;
-        }
-    ).set_examples({LLAMA_EXAMPLE_EXPORT_GRAPH_OPS}));
     add_opt(common_arg(
         {"-ofreq", "--output-frequency"}, "N",
         string_format("output the imatrix every N iterations (default: %d)", params.n_out_freq),
@@ -3283,6 +3288,22 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
             params.server_tools = parse_csv_row(value);
         }
     ).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_TOOLS"));
+    add_opt(common_arg(
+        {"--mcp-servers-config"}, "PATH",
+        "experimental: path to JSON file with MCP server definitions (Cursor-compatible format) - do not enable in untrusted environments (default: none)\n"
+        "note: for security reasons, this will limit --cors-origins to localhost by default",
+        [](common_params & params, const std::string & value) {
+            params.mcp_servers_config = value;
+        }
+    ).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_MCP_SERVERS_CONFIG"));
+    add_opt(common_arg(
+        {"--mcp-servers-json"}, "JSON",
+        "experimental: inline JSON with MCP server definitions (Cursor-compatible format) - do not enable in untrusted environments (default: none)\n"
+        "note: for security reasons, this will limit --cors-origins to localhost by default",
+        [](common_params & params, const std::string & value) {
+            params.mcp_servers_json = value;
+        }
+    ).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_MCP_SERVERS_JSON"));
     add_opt(common_arg(
         {"-ag", "--agent"},
         {"-no-ag", "--no-agent"},

common/arg.h

@@ -123,6 +123,9 @@ struct common_params_context {
 // if one argument has invalid value, it will automatically display usage of the specific argument (and not the full usage message)
 bool common_params_parse(int argc, char ** argv, common_params & params, llama_example ex, void(*print_usage)(int, char **) = nullptr);
 
+// load all backends and print the list of available (non-CPU) devices to stdout
+void common_print_available_devices();
+
 // parse input arguments from CLI into a map
 bool common_params_to_map(int argc, char ** argv, llama_example ex, std::map<common_arg, std::string> & out_map);
 

common/chat.cpp

@@ -1024,7 +1024,7 @@ static common_chat_params common_chat_params_init_ministral_3(const common_chat_
 
     data.supports_thinking  = true;
     data.thinking_start_tag = "[THINK]";
-    data.thinking_end_tag   = "[/THINK]";
+    data.thinking_end_tags  = {"[/THINK]"};
     data.prompt            = common_chat_template_direct_apply_impl(tmpl, inputs, /* messages_override = */ adjusted_messages);
     data.generation_prompt = common_chat_template_generation_prompt_impl(tmpl, inputs, /* messages_override = */ adjusted_messages);
     data.format            = COMMON_CHAT_FORMAT_PEG_NATIVE;
@@ -1150,6 +1150,9 @@ static common_chat_params common_chat_params_init_gpt_oss(const common_chat_temp
     data.format            = COMMON_CHAT_FORMAT_PEG_NATIVE;
     data.supports_thinking = true;
 
+    data.thinking_start_tag = "<|channel|>analysis<|message|>";
+    data.thinking_end_tags  = {"<|end|>"};
+
     // These special tokens are required to parse properly, so we include them
     // even if parse_tool_calls is false.
     data.preserved_tokens = {
@@ -1294,7 +1297,7 @@ static common_chat_params common_chat_params_init_gemma4(const common_chat_templ
     data.format            = COMMON_CHAT_FORMAT_PEG_GEMMA4;
     data.supports_thinking  = true;
     data.thinking_start_tag = "<|channel>thought";
-    data.thinking_end_tag   = "<channel|>";
+    data.thinking_end_tags  = {"<channel|>"};
 
     data.preserved_tokens = {
         "<|channel>",
@@ -1569,7 +1572,7 @@ static common_chat_params common_chat_params_init_kimi_k2(const common_chat_temp
     const std::string GEN_PROMPT  = "<|im_assistant|>assistant<|im_middle|>";
 
     data.thinking_start_tag = THINK_START;
-    data.thinking_end_tag   = THINK_END;
+    data.thinking_end_tags  = {THINK_END};
 
     if (inputs.has_continuation()) {
         const auto & msg = inputs.continue_msg;
@@ -1703,7 +1706,7 @@ static common_chat_params common_chat_params_init_lfm2(const common_chat_templat
     }
 
     data.thinking_start_tag = THINK_START;
-    data.thinking_end_tag   = THINK_END;
+    data.thinking_end_tags  = {THINK_END};
 
     auto has_tools           = inputs.tools.is_array() && !inputs.tools.empty();
     auto has_response_format = !inputs.json_schema.is_null() && inputs.json_schema.is_object();
@@ -1943,7 +1946,7 @@ static common_chat_params common_chat_params_init_deepseek_v3_2(const common_cha
     data.format             = COMMON_CHAT_FORMAT_PEG_NATIVE;
     data.supports_thinking  = true;
     data.thinking_start_tag = "<think>";
-    data.thinking_end_tag   = "</think>";
+    data.thinking_end_tags  = {"</think>"};
     data.preserved_tokens   = {
         "｜DSML｜",
         "<think>",
@@ -2160,7 +2163,7 @@ static common_chat_params common_chat_params_init_cohere2moe(const common_chat_t
     data.format             = COMMON_CHAT_FORMAT_PEG_NATIVE;
     data.supports_thinking  = true;
     data.thinking_start_tag = THINK_START;
-    data.thinking_end_tag   = THINK_END;
+    data.thinking_end_tags  = {THINK_END};
     data.preserved_tokens   = {
         TURN_START, TURN_END, CHATBOT, USER, SYSTEM,
         THINK_START, THINK_END,
@@ -2179,9 +2182,10 @@ static common_chat_params common_chat_params_init_cohere2moe(const common_chat_t
         { COMMON_CHAT_ROLE_SYSTEM,    TURN_START + SYSTEM },
     };
 
-    auto has_tools         = inputs.tools.is_array() && !inputs.tools.empty();
-    auto extract_reasoning = inputs.reasoning_format != COMMON_REASONING_FORMAT_NONE;
-    auto include_grammar   = has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE;
+    auto has_tools           = inputs.tools.is_array() && !inputs.tools.empty();
+    auto has_response_format = inputs.json_schema.is_object() && !inputs.json_schema.empty();
+    auto extract_reasoning   = inputs.reasoning_format != COMMON_REASONING_FORMAT_NONE;
+    auto include_grammar     = has_response_format || (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE);
 
     if (inputs.has_continuation()) {
         const auto & msg = inputs.continue_msg;
@@ -2212,7 +2216,11 @@ static common_chat_params common_chat_params_init_cohere2moe(const common_chat_t
                                              p.optional(p.literal(THINK_END))));
         }
 
-        auto text_content = p.literal(TEXT_START) + p.content(p.until(TEXT_END)) + p.optional(p.literal(TEXT_END));
+        auto text_content = has_response_format
+            ? p.literal(TEXT_START) +
+                p.content(p.schema(p.json(), "response-format-schema", inputs.json_schema)) +
+                p.optional(p.literal(TEXT_END))
+            : p.literal(TEXT_START) + p.content(p.until(TEXT_END)) + p.optional(p.literal(TEXT_END));
 
         if (!has_tools || inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_NONE) {
             return generation_prompt + reasoning + text_content + p.optional(p.literal(TURN_END)) + end;
@@ -2240,13 +2248,17 @@ static common_chat_params common_chat_params_init_cohere2moe(const common_chat_t
     data.parser = parser.save();
 
     if (include_grammar) {
-        data.grammar_lazy = inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_AUTO;
+        data.grammar_lazy = !has_response_format && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_AUTO;
         data.grammar      = build_grammar([&](const common_grammar_builder & builder) {
             foreach_function(inputs.tools, [&](const json & tool) {
                 const auto & function = tool.at("function");
                 auto         schema   = function.at("parameters");
                 builder.resolve_refs(schema);
             });
+            if (has_response_format) {
+                auto schema = inputs.json_schema;
+                builder.resolve_refs(schema);
+            }
             parser.build_grammar(builder, data.grammar_lazy);
         });
 
@@ -2501,7 +2513,7 @@ static common_chat_params common_chat_params_init_minicpm5(const common_chat_tem
     };
 
     data.thinking_start_tag = "<think>";
-    data.thinking_end_tag   = "</think>";
+    data.thinking_end_tags  = {"</think>"};
 
     data.message_delimiters = {
         { COMMON_CHAT_ROLE_ASSISTANT, "<|im_start|>assistant"             },
@@ -2857,7 +2869,10 @@ static common_chat_params common_chat_templates_apply_jinja(const struct common_
         auto_params.supports_thinking = autoparser.reasoning.mode != autoparser::reasoning_mode::NONE;
         if (auto_params.supports_thinking) {
             auto_params.thinking_start_tag = trim_whitespace(autoparser.reasoning.start);
-            auto_params.thinking_end_tag   = trim_whitespace(autoparser.reasoning.end);
+            auto end_tag = trim_whitespace(autoparser.reasoning.end);
+            if (!end_tag.empty()) {
+                auto_params.thinking_end_tags = {std::move(end_tag)};
+            }
         }
         common_peg_arena arena;
         arena.load(auto_params.parser);

common/chat.h

@@ -274,7 +274,7 @@ struct common_chat_params {
     std::string                         generation_prompt;
     bool                                supports_thinking    = false;
     std::string                         thinking_start_tag;  // e.g., "<think>"
-    std::string                         thinking_end_tag;    // e.g., "</think>"
+    std::vector<std::string>            thinking_end_tags;   // e.g., "</think>"
     std::vector<common_grammar_trigger> grammar_triggers;
     std::vector<std::string>            preserved_tokens;
     std::vector<std::string>            additional_stops;

common/common.cpp

@@ -3,7 +3,6 @@
 
 #include "build-info.h"
 #include "common.h"
-#include "ggml-profiler.h"
 #include "fit.h"
 #include "log.h"
 #include "llama.h"
@@ -1250,7 +1249,6 @@ common_init_result::common_init_result(common_params & params, bool model_only)
         lora.reset(llama_adapter_lora_init(model, la.path.c_str()));
         if (lora == nullptr) {
             COM_ERR("failed to load lora adapter '%s'\n", la.path.c_str());
-            pimpl->model.reset(model);
             return;
         }
 
@@ -1317,14 +1315,6 @@ common_init_result::common_init_result(common_params & params, bool model_only)
         return;
     }
 
-    if (params.profiling) {
-        ggml_backend_sched_t sched = llama_context_get_sched(lctx);
-        if (sched != nullptr) {
-            ggml_backend_sched_set_profiling(sched, true);
-            LOG_INF("%s: profiling enabled\n", __func__);
-        }
-    }
-
     pimpl->context.reset(lctx);
 }
 
@@ -1528,23 +1518,49 @@ done:
     return res;
 }
 
-void common_context_seq_rm(llama_context * ctx, llama_seq_id seq_id, llama_pos p0, llama_pos p1) {
+static void common_context_seq_rm(llama_context * ctx, llama_seq_id seq_id, llama_pos p0, llama_pos p1) {
     auto * mem = llama_get_memory(ctx);
     if (!llama_memory_seq_rm(mem, seq_id, p0, p1)) {
         GGML_ABORT("%s", string_format("failed to remove sequence %d with p0=%d, p1=%d\n", seq_id, p0, p1).c_str());
     }
 }
 
-void common_context_seq_cp(llama_context * ctx, llama_seq_id seq_id_src, llama_seq_id seq_id_dst, llama_pos p0, llama_pos p1) {
+static void common_context_seq_cp(llama_context * ctx, llama_seq_id seq_id_src, llama_seq_id seq_id_dst, llama_pos p0, llama_pos p1) {
     auto * mem = llama_get_memory(ctx);
     llama_memory_seq_cp(mem, seq_id_src, seq_id_dst, p0, p1);
 }
 
-void common_context_seq_add(llama_context * ctx, llama_seq_id seq_id, llama_pos p0, llama_pos p1, llama_pos delta) {
+static void common_context_seq_add(llama_context * ctx, llama_seq_id seq_id, llama_pos p0, llama_pos p1, llama_pos delta) {
     auto * mem = llama_get_memory(ctx);
     llama_memory_seq_add(mem, seq_id, p0, p1, delta);
 }
 
+void common_memory::init(llama_context * ctx_tgt, llama_context * ctx_dft) {
+    this->ctx_tgt = ctx_tgt;
+    this->ctx_dft = ctx_dft;
+}
+
+void common_memory::seq_rm(llama_seq_id seq_id, llama_pos p0, llama_pos p1) const {
+    common_context_seq_rm(ctx_tgt, seq_id, p0, p1);
+    if (ctx_dft) {
+        common_context_seq_rm(ctx_dft, seq_id, p0, p1);
+    }
+}
+
+void common_memory::seq_cp(llama_seq_id seq_id_src, llama_seq_id seq_id_dst, llama_pos p0, llama_pos p1) const {
+    common_context_seq_cp(ctx_tgt, seq_id_src, seq_id_dst, p0, p1);
+    if (ctx_dft) {
+        common_context_seq_cp(ctx_dft, seq_id_src, seq_id_dst, p0, p1);
+    }
+}
+
+void common_memory::seq_add(llama_seq_id seq_id, llama_pos p0, llama_pos p1, llama_pos delta) const {
+    common_context_seq_add(ctx_tgt, seq_id, p0, p1, delta);
+    if (ctx_dft) {
+        common_context_seq_add(ctx_dft, seq_id, p0, p1, delta);
+    }
+}
+
 void common_set_adapter_lora(struct llama_context * ctx, std::vector<common_adapter_lora_info> & lora) {
     std::vector<llama_adapter_lora *> loras;
     std::vector<float> scales;

common/common.h

@@ -5,7 +5,6 @@
 #include "llama-cpp.h"
 
 #include "ggml-opt.h"
-#include "ggml-profiler.h"
 #include "ggml.h"
 #include "llama.h"
 
@@ -285,12 +284,12 @@ struct common_params_sampling {
 
     // reasoning budget sampler parameters
     // these are populated by the server/CLI based on chat template params
-    int32_t                  reasoning_budget_tokens   = -1;   // -1 = disabled, >= 0 = token budget
-    std::vector<llama_token> reasoning_budget_start;           // start tag token sequence
-    std::vector<llama_token> reasoning_budget_end;             // end tag token sequence
-    std::vector<llama_token> reasoning_budget_forced;          // forced sequence (message + end tag)
-    std::string              reasoning_budget_message;         // message injected before end tag when budget exhausted
-    bool                     reasoning_control = false;        // create the budget sampler on demand so reasoning can be ended at runtime
+    int32_t                   reasoning_budget_tokens   = -1;  // -1 = disabled, >= 0 = token budget
+    std::vector<llama_token>  reasoning_budget_start;          // start tag token sequence
+    std::vector<llama_tokens> reasoning_budget_end;            // end tag token sequences; the first tag is used as the forcing sequence
+    std::vector<llama_token>  reasoning_budget_forced;         // forced sequence (message + first end tag)
+    std::string               reasoning_budget_message;        // message injected before end tag when budget exhausted
+    bool                      reasoning_control = false;       // create the budget sampler on demand so reasoning can be ended at runtime
 
     bool backend_sampling = false;
 
@@ -479,7 +478,6 @@ struct common_params {
     bool    fit_params         = true;  // whether to fit unset model/context parameters to free device memory
     bool    fit_params_print   = false; // print the estimated required memory to run the model
     int32_t fit_params_min_ctx = 4096;  // minimum context size to set when trying to reduce memory use
-    bool    with_backends      = false; // export graph ops with backend assignments
 
     // margin per device in bytes for fitting parameters to free memory:
     std::vector<size_t> fit_params_target = std::vector<size_t>(llama_max_devices(), 1024 * 1024*1024);
@@ -670,6 +668,10 @@ struct common_params {
     // enable built-in tools
     std::vector<std::string> server_tools;
 
+    // MCP server configs (Cursor-compatible JSON)
+    std::string mcp_servers_config;   // path to JSON file with MCP server definitions
+    std::string mcp_servers_json;     // inline JSON with MCP server definitions
+
     // router server configs
     std::string models_dir    = "";     // directory containing models for the router server
     std::string models_preset = "";     // directory containing model presets for the router server
@@ -723,10 +725,6 @@ struct common_params {
 
     bool spm_infill = false; // suffix/prefix/middle pattern for infill
 
-    // profiling
-    bool        profiling = false;  // enable cross-backend profiling
-    std::string profiling_output;   // path to write profiling JSON output (empty = stdout)
-
     // batched-bench params
     bool batched_bench_output_jsonl = false;
 
@@ -950,10 +948,17 @@ enum common_context_seq_rm_type {
 // note: clears the memory of the context
 common_context_seq_rm_type common_context_can_seq_rm(llama_context * ctx);
 
-// aborts execution on failure
-void common_context_seq_rm (llama_context * ctx, llama_seq_id seq_id, llama_pos p0, llama_pos p1);
-void common_context_seq_add(llama_context * ctx, llama_seq_id seq_id, llama_pos p0, llama_pos p1, llama_pos delta);
-void common_context_seq_cp (llama_context * ctx, llama_seq_id seq_id_src, llama_seq_id seq_id_dst, llama_pos p0, llama_pos p1);
+struct common_memory {
+    llama_context * ctx_tgt = nullptr;
+    llama_context * ctx_dft = nullptr;
+
+    void init(llama_context * ctx_tgt, llama_context * ctx_dft = nullptr);
+
+    // aborts execution on failure
+    void seq_rm (llama_seq_id seq_id, llama_pos p0, llama_pos p1) const;
+    void seq_add(llama_seq_id seq_id, llama_pos p0, llama_pos p1, llama_pos delta) const;
+    void seq_cp (llama_seq_id seq_id_src, llama_seq_id seq_id_dst, llama_pos p0, llama_pos p1) const;
+};
 
 //
 // Batch utils

common/download.cpp

@@ -568,16 +568,30 @@ static hf_cache::hf_files get_split_files(const hf_cache::hf_files & files,
 }
 
 // pick the best sibling GGUF whose filename contains `keyword` (e.g. "mmproj" / "mtp"),
-// preferring deeper shared directory prefix with the model, then closest quantization
+// preferring deeper shared directory prefix with the model, then exact `tag` match,
+// then closest quantization to the tag when given, or to the model otherwise
 static hf_cache::hf_file find_best_sibling(const hf_cache::hf_files & files,
                                            const std::string        & model,
-                                           const std::string        & keyword) {
+                                           const std::string        & keyword,
+                                           const std::string        & tag = "") {
     hf_cache::hf_file best;
     size_t best_depth = 0;
     int best_diff = 0;
+    bool best_exact = false;
     bool found = false;
 
-    auto model_bits = extract_quant_bits(model);
+    std::string tag_upper = tag;
+    for (char & c : tag_upper) {
+        c = (char) std::toupper((unsigned char) c);
+    }
+
+    int model_bits = 0;
+    if (!tag_upper.empty()) {
+        auto pos = tag_upper.find_first_of("0123456789");
+        model_bits = pos == std::string::npos ? 0 : std::stoi(tag_upper.substr(pos));
+    } else {
+        model_bits = extract_quant_bits(model);
+    }
     auto model_parts = string_split<std::string>(model, '/');
     auto model_dir = model_parts.end() - 1;
 
@@ -600,10 +614,19 @@ static hf_cache::hf_file find_best_sibling(const hf_cache::hf_files & files,
         auto bits = extract_quant_bits(f.path);
         auto diff = std::abs(bits - model_bits);
 
-        if (!found || depth > best_depth || (depth == best_depth && diff < best_diff)) {
+        std::string path_upper = f.path;
+        for (char & c : path_upper) {
+            c = (char) std::toupper((unsigned char) c);
+        }
+        bool exact = !tag_upper.empty() && path_upper.find("-" + tag_upper + ".") != std::string::npos;
+
+        if (!found || depth > best_depth ||
+            (depth == best_depth && exact && !best_exact) ||
+            (depth == best_depth && exact == best_exact && diff < best_diff)) {
             best = f;
             best_depth = depth;
             best_diff = diff;
+            best_exact = exact;
             found = true;
         }
     }
@@ -616,18 +639,21 @@ static hf_cache::hf_file find_best_mmproj(const hf_cache::hf_files & files,
 }
 
 static hf_cache::hf_file find_best_mtp(const hf_cache::hf_files & files,
-                                       const std::string        & model) {
-    return find_best_sibling(files, model, "mtp-");
+                                       const std::string        & model,
+                                       const std::string        & tag = "") {
+    return find_best_sibling(files, model, "mtp-", tag);
 }
 
 static hf_cache::hf_file find_best_eagle3(const hf_cache::hf_files & files,
-                                          const std::string        & model) {
-    return find_best_sibling(files, model, "eagle3-");
+                                          const std::string        & model,
+                                          const std::string        & tag = "") {
+    return find_best_sibling(files, model, "eagle3-", tag);
 }
 
 static hf_cache::hf_file find_best_dflash(const hf_cache::hf_files & files,
-                                          const std::string        & model) {
-    return find_best_sibling(files, model, "dflash-");
+                                          const std::string        & model,
+                                          const std::string        & tag = "") {
+    return find_best_sibling(files, model, "dflash-", tag);
 }
 
 static bool gguf_filename_is_model(const std::string & filepath) {
@@ -736,27 +762,36 @@ common_download_hf_plan common_download_get_hf_plan(const common_params_model &
         }
     } else {
         primary = find_best_model(all, tag);
-        if (primary.path.empty()) {
+        // a requested sidecar can resolve on its own, without a full model of the same tag
+        if (primary.path.empty() && !opts.download_mtp && !opts.download_dflash && !opts.download_eagle3) {
             LOG_ERR("%s: no GGUF files found in repository %s\n", __func__, repo.c_str());
             list_available_gguf_files(all);
             return plan;
         }
     }
 
-    plan.primary = primary;
-    plan.model_files = get_split_files(all, primary);
+    if (!primary.path.empty()) {
+        plan.primary = primary;
+        plan.model_files = get_split_files(all, primary);
+    }
 
-    if (opts.download_mmproj) {
+    if (opts.download_mmproj && !primary.path.empty()) {
         plan.mmproj = find_best_mmproj(all, primary.path);
     }
     if (opts.download_mtp) {
-        plan.mtp = find_best_mtp(all, primary.path);
+        plan.mtp = find_best_mtp(all, primary.path, tag);
     }
     if (opts.download_dflash) {
-        plan.dflash = find_best_dflash(all, primary.path);
+        plan.dflash = find_best_dflash(all, primary.path, tag);
     }
     if (opts.download_eagle3) {
-        plan.eagle3 = find_best_eagle3(all, primary.path);
+        plan.eagle3 = find_best_eagle3(all, primary.path, tag);
+    }
+
+    if (primary.path.empty() &&
+        plan.mtp.local_path.empty() && plan.dflash.local_path.empty() && plan.eagle3.local_path.empty()) {
+        LOG_ERR("%s: no GGUF files found in repository %s\n", __func__, repo.c_str());
+        list_available_gguf_files(all);
     }
 
     return plan;

common/fit.cpp

@@ -136,7 +136,7 @@ static std::vector<llama_device_memory_data> common_get_device_memory_data_impl(
         devs.push_back(llama_model_get_device(model, i));
     }
 
-    hp_ngl         = llama_model_n_layer(model);
+    hp_ngl         = llama_model_n_layer(model) + llama_model_n_layer_nextn(model);
     hp_n_ctx_train = llama_model_n_ctx_train(model);
     hp_n_expert    = llama_model_n_expert(model);
 

common/peg-parser.cpp

@@ -3,10 +3,10 @@
 #include "common.h"
 #include "json-schema-to-grammar.h"
 #include "log.h"
+#include "trie.h"
 #include "unicode.h"
 
 #include <algorithm>
-#include <deque>
 #include <initializer_list>
 #include <map>
 #include <memory>
@@ -32,154 +32,6 @@ static bool is_hex_digit(const char c) {
     return (c >= '0' && c <= '9') || (c >= 'a' && c <= 'f') || (c >= 'A' && c <= 'F');
 }
 
-// Trie for matching multiple literals.
-// This is used in common_peg_until_parser and to build a GBNF exclusion grammar
-struct trie {
-    struct node {
-        std::map<uint32_t, size_t> children;  // Use uint32_t to store Unicode codepoints
-        bool is_word;
-    };
-
-    std::vector<node> nodes;
-
-    trie(const std::vector<std::string> & words) {
-      create_node(); // root node
-      for (const auto & w : words) {
-          insert(w);
-      }
-    }
-
-    enum match_result { NO_MATCH, PARTIAL_MATCH, COMPLETE_MATCH };
-
-    // Check if a delimiter starts at the given position
-    match_result check_at(std::string_view sv, size_t start_pos) const {
-        size_t current = 0; // Start at root
-        size_t pos = start_pos;
-
-        // LOG_DBG("%s: checking at pos %zu, sv='%s'\n", __func__, start_pos, std::string(sv).c_str());
-
-        while (pos < sv.size()) {
-            auto result = common_parse_utf8_codepoint(sv, pos);
-            if (result.status != utf8_parse_result::SUCCESS) {
-                break;
-            }
-
-            auto it = nodes[current].children.find(result.codepoint);
-            if (it == nodes[current].children.end()) {
-                // Can't continue matching
-                return match_result{match_result::NO_MATCH};
-            }
-
-            current = it->second;
-            pos += result.bytes_consumed;
-
-            // Check if we've matched a complete word
-            if (nodes[current].is_word) {
-                return match_result{match_result::COMPLETE_MATCH};
-            }
-        }
-
-        // Reached end of input while still in the trie (not at root)
-        if (current != 0) {
-            // We're in the middle of a potential match
-            return match_result{match_result::PARTIAL_MATCH};
-        }
-
-        // Reached end at root (no match)
-        return match_result{match_result::NO_MATCH};
-    }
-
-  private:
-    size_t create_node() {
-        size_t index = nodes.size();
-        nodes.emplace_back();
-        return index;
-    }
-
-    void insert(const std::string & word) {
-        size_t current = 0;
-        size_t pos     = 0;
-        while (pos < word.length()) {
-            auto result = common_parse_utf8_codepoint(word, pos);
-            if (result.status != utf8_parse_result::SUCCESS) {
-                break;
-            }
-
-            uint32_t ch = result.codepoint;
-            pos += result.bytes_consumed;
-
-            auto it = nodes[current].children.find(ch);
-            if (it == nodes[current].children.end()) {
-                size_t child = create_node();
-                nodes[current].children[ch] = child;
-                current = child;
-            } else {
-                current = it->second;
-            }
-        }
-        nodes[current].is_word = true;
-    }
-};
-
-// Aho-Corasick automaton
-struct aho_corasick {
-    trie                t;
-    std::vector<size_t> fail;      // failure links
-    std::vector<size_t> order;     // states in BFS order
-    std::vector<bool>   terminal;  // match states (directly or via a suffix link)
-    std::set<uint32_t>  alphabet;  // every character with a transition
-
-    aho_corasick(const std::vector<std::string> & strings) : t(strings) {
-        const auto & nodes = t.nodes;
-        const size_t n = nodes.size();
-
-        fail.assign(n, 0);
-        order.reserve(n);
-
-        std::deque<size_t> queue{ 0 };
-        while (!queue.empty()) {
-            size_t u = queue.front();
-            queue.pop_front();
-            order.push_back(u);
-            for (const auto & [ch, v] : nodes[u].children) {
-                if (u != 0) {
-                    size_t f = fail[u];
-                    while (f && nodes[f].children.find(ch) == nodes[f].children.end()) {
-                        f = fail[f];
-                    }
-                    auto it = nodes[f].children.find(ch);
-                    fail[v] = (it != nodes[f].children.end() && it->second != v) ? it->second : 0;
-                }
-                queue.push_back(v);
-            }
-        }
-
-        terminal.assign(n, false);
-        for (size_t u : order) {
-            terminal[u] = nodes[u].is_word || (u != 0 && terminal[fail[u]]);
-        }
-
-        for (const auto & node : nodes) {
-            for (const auto & [ch, v] : node.children) {
-                alphabet.insert(ch);
-            }
-        }
-    }
-
-    size_t num_states()          const { return t.nodes.size(); }
-    bool   is_terminal(size_t s) const { return terminal[s]; }
-
-    // follow failure links until a transition on `ch` exists.
-    size_t next(size_t state, uint32_t ch) const {
-        const auto & nodes = t.nodes;
-        while (state && nodes[state].children.find(ch) == nodes[state].children.end()) {
-            state = fail[state];
-        }
-        auto it = nodes[state].children.find(ch);
-        return it != nodes[state].children.end() ? it->second : 0;
-    }
-};
-
 static std::pair<uint32_t, size_t> parse_hex_escape(const std::string & str, size_t pos, int hex_count) {
     if (pos + hex_count > str.length()) {
         return {0, 0};
@@ -797,7 +649,7 @@ struct parser_executor {
     }
 
     common_peg_parse_result operator()(const common_peg_until_parser & p) const {
-        trie matcher(p.delimiters);
+        common_trie matcher(p.delimiters);
 
         // Scan input and check for delimiters
         size_t pos = start_pos;
@@ -824,12 +676,12 @@ struct parser_executor {
             // Check if a delimiter starts at this position
             auto match = matcher.check_at(ctx.input, pos);
 
-            if (match == trie::COMPLETE_MATCH) {
+            if (match == common_trie::COMPLETE_MATCH) {
                 // Found a complete delimiter, return everything before it
                 return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_SUCCESS, start_pos, pos);
             }
 
-            if (match == trie::PARTIAL_MATCH) {
+            if (match == common_trie::PARTIAL_MATCH) {
                 // Found a partial match extending to end of input, return everything before it
                 return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_SUCCESS, start_pos, pos);
             }
@@ -1559,7 +1411,7 @@ static std::string gbnf_ac_grammar(
                                     const std::map<size_t, std::vector<uint32_t>> &,
                                     const std::vector<uint32_t> &,
                                     const std::function<std::string(size_t)> &)> & build_rule) {
-    aho_corasick ac(strings);
+    common_aho_corasick ac(strings);
 
     auto state_name = [&](size_t s) -> std::string {
         if (s == 0) {

common/preset.cpp

@@ -330,6 +330,10 @@ common_presets common_preset_context::load_from_ini(const std::string & path, co
             }
         }
 
+        if (preset.name == COMMON_PRESET_DEFAULT_NAME && preset.options.empty()) {
+            continue;
+        }
+
         if (preset.name == "*") {
             // handle global preset
             global = preset;

common/reasoning-budget.cpp

@@ -1,39 +1,52 @@
 #include "reasoning-budget.h"
 #include "common.h"
+#include "trie.h"
 #include "unicode.h"
 
 #include "log.h"
 
+#include <algorithm>
 #include <cmath>
 #include <cstdint>
 #include <string>
 #include <vector>
 
 struct token_matcher {
-    std::vector<llama_token> tokens;
-    size_t pos = 0;
+    std::vector<llama_tokens> seqs;
+    common_aho_corasick ac;
+    size_t state = 0;
 
-    bool advance(llama_token token) {
-        if (tokens.empty()) {
-            return false;
-        }
+    token_matcher(const std::vector<llama_tokens> & seqs) : seqs(collect(seqs)), ac(build_trie(this->seqs)) {}
 
-        if (token == tokens[pos]) {
-            pos++;
-            if (pos >= tokens.size()) {
-                pos = 0;
-                return true;
-            }
-        } else {
-            pos = 0;
-            if (token == tokens[0]) {
-                pos = 1;
+    static std::vector<llama_tokens> collect(const std::vector<llama_tokens> & seqs) {
+        std::vector<llama_tokens> res;
+        for (const auto & seq : seqs) {
+            if (!seq.empty() && std::find(res.begin(), res.end(), seq) == res.end()) {
+                res.push_back(seq);
             }
         }
-        return false;
+        return res;
     }
 
-    void reset() { pos = 0; }
+    static common_trie build_trie(const std::vector<llama_tokens> & seqs) {
+        common_trie t;
+        for (const auto & seq : seqs) {
+            t.insert(std::vector<uint32_t>(seq.begin(), seq.end()));
+        }
+        return t;
+    }
+
+    // returns the index into seqs of the longest sequence ending at this token, or -1
+    int32_t advance(llama_token token) {
+        state = ac.next(state, (uint32_t) token);
+        const int32_t p = ac.match_pattern(state);
+        if (p >= 0) {
+            state = 0;
+        }
+        return p;
+    }
+
+    void reset() { state = 0; }
 };
 
 struct common_reasoning_budget_ctx {
@@ -41,7 +54,7 @@ struct common_reasoning_budget_ctx {
 
     token_matcher start_matcher;
     token_matcher end_matcher;
-    std::vector<llama_token> forced_tokens;
+    llama_tokens forced_tokens;
 
     int32_t budget;           // maximum tokens in reasoning block
     int32_t remaining;        // tokens remaining in budget
@@ -50,6 +63,8 @@ struct common_reasoning_budget_ctx {
 
     // for forcing
     size_t force_pos;         // next position in forced_tokens to force
+
+    int32_t end_match;        // index into end_matcher.seqs of the sequence that transitioned to DONE, -1 if none
 };
 
 static const char * common_reasoning_budget_name(const struct llama_sampler * /*smpl*/) {
@@ -62,7 +77,7 @@ static void common_reasoning_budget_accept(struct llama_sampler * smpl, llama_to
     switch (ctx->state) {
         case REASONING_BUDGET_IDLE:
         {
-            if (ctx->start_matcher.advance(token)) {
+            if (ctx->start_matcher.advance(token) >= 0) {
                 ctx->state = REASONING_BUDGET_COUNTING;
                 ctx->remaining = ctx->budget;
                 COM_TRC("activated, budget=%d tokens\n", ctx->budget);
@@ -78,8 +93,10 @@ static void common_reasoning_budget_accept(struct llama_sampler * smpl, llama_to
         case REASONING_BUDGET_COUNTING:
         case REASONING_BUDGET_WAITING_UTF8:
         {
-            if (ctx->end_matcher.advance(token)) {
+            const int32_t match = ctx->end_matcher.advance(token);
+            if (match >= 0) {
                 ctx->state = REASONING_BUDGET_DONE;
+                ctx->end_match = match;
                 COM_TRC("%s", "deactivated (natural end)\n");
                 break;
             }
@@ -115,19 +132,25 @@ static void common_reasoning_budget_accept(struct llama_sampler * smpl, llama_to
             break;
         }
         case REASONING_BUDGET_FORCING:
+        {
+            // track the end sequence within forced_tokens so it is also reported on DONE
+            const int32_t match = ctx->end_matcher.advance(token);
             ctx->force_pos++;
             if (ctx->force_pos >= ctx->forced_tokens.size()) {
                 ctx->state = REASONING_BUDGET_DONE;
+                ctx->end_match = match;
                 COM_TRC("%s", "forced sequence complete, done\n");
             }
             break;
+        }
         case REASONING_BUDGET_DONE:
             // Re-arm on a new start tag: some models emit multiple <think> blocks
             // per response, and each should get a fresh budget window.
-            if (ctx->start_matcher.advance(token)) {
+            if (ctx->start_matcher.advance(token) >= 0) {
                 ctx->state = REASONING_BUDGET_COUNTING;
                 ctx->remaining = ctx->budget;
                 ctx->end_matcher.reset();
+                ctx->end_match = -1;
                 COM_TRC("re-activated on new start tag, budget=%d tokens\n", ctx->budget);
 
                 if (ctx->remaining <= 0) {
@@ -169,11 +192,12 @@ static void common_reasoning_budget_reset(struct llama_sampler * smpl) {
     ctx->start_matcher.reset();
     ctx->end_matcher.reset();
     ctx->force_pos = 0;
+    ctx->end_match = -1;
 }
 
 static struct llama_sampler * common_reasoning_budget_init_state(
-        const struct llama_vocab * vocab, const std::vector<llama_token> & start_tokens,
-        const std::vector<llama_token> & end_tokens, const std::vector<llama_token> & forced_tokens,
+        const struct llama_vocab * vocab, const std::vector<llama_tokens> & start_seqs,
+        const std::vector<llama_tokens> & end_seqs, const llama_tokens & forced_tokens,
         int32_t budget, common_reasoning_budget_state initial_state);
 
 static struct llama_sampler * common_reasoning_budget_clone(const struct llama_sampler * smpl);
@@ -205,12 +229,12 @@ static struct llama_sampler * common_reasoning_budget_clone(const struct llama_s
 }
 
 static struct llama_sampler * common_reasoning_budget_init_state(
-        const struct llama_vocab             * vocab,
-        const std::vector<llama_token>       & start_tokens,
-        const std::vector<llama_token>       & end_tokens,
-        const std::vector<llama_token>       & forced_tokens,
-        int32_t                                budget,
-        common_reasoning_budget_state          initial_state) {
+        const struct llama_vocab        * vocab,
+        const std::vector<llama_tokens> & start_seqs,
+        const std::vector<llama_tokens> & end_seqs,
+        const llama_tokens              & forced_tokens,
+        int32_t                           budget,
+        common_reasoning_budget_state     initial_state) {
     // promote COUNTING with budget <= 0 to FORCING
     if (initial_state == REASONING_BUDGET_COUNTING && budget <= 0) {
         initial_state = REASONING_BUDGET_FORCING;
@@ -220,25 +244,26 @@ static struct llama_sampler * common_reasoning_budget_init_state(
         /* .iface = */ &common_reasoning_budget_i,
         /* .ctx   = */ new common_reasoning_budget_ctx {
             /* .vocab         = */ vocab,
-            /* .start_matcher = */ { start_tokens, 0 },
-            /* .end_matcher   = */ { end_tokens, 0 },
+            /* .start_matcher = */ token_matcher(start_seqs),
+            /* .end_matcher   = */ token_matcher(end_seqs),
             /* .forced_tokens = */ forced_tokens,
             /* .budget        = */ budget,
             /* .remaining     = */ budget,
             /* .state         = */ initial_state,
             /* .force_pos     = */ 0,
+            /* .end_match     = */ -1,
         }
     );
 }
 
 struct llama_sampler * common_reasoning_budget_init(
-        const struct llama_vocab       * vocab,
-        const std::vector<llama_token> & start_tokens,
-        const std::vector<llama_token> & end_tokens,
-        const std::vector<llama_token> & forced_tokens,
-        int32_t                          budget,
-        common_reasoning_budget_state    initial_state) {
-    return common_reasoning_budget_init_state(vocab, start_tokens, end_tokens, forced_tokens, budget, initial_state);
+        const struct llama_vocab        * vocab,
+        const std::vector<llama_tokens> & start_seqs,
+        const std::vector<llama_tokens> & end_seqs,
+        const llama_tokens              & forced_tokens,
+        int32_t                           budget,
+        common_reasoning_budget_state     initial_state) {
+    return common_reasoning_budget_init_state(vocab, start_seqs, end_seqs, forced_tokens, budget, initial_state);
 }
 
 common_reasoning_budget_state common_reasoning_budget_get_state(const struct llama_sampler * smpl) {
@@ -248,6 +273,19 @@ common_reasoning_budget_state common_reasoning_budget_get_state(const struct lla
     return ((const common_reasoning_budget_ctx *)smpl->ctx)->state;
 }
 
+const llama_tokens * common_reasoning_budget_get_end_match(const struct llama_sampler * smpl) {
+    if (!smpl) {
+        return nullptr;
+    }
+
+    const auto * ctx = (const common_reasoning_budget_ctx *) smpl->ctx;
+    if (ctx->end_match < 0) {
+        return nullptr;
+    }
+
+    return &ctx->end_matcher.seqs[ctx->end_match];
+}
+
 bool common_reasoning_budget_force(struct llama_sampler * smpl) {
     if (!smpl) {
         return false;

common/reasoning-budget.h

@@ -2,6 +2,8 @@
 
 #include "llama.h"
 
+#include "common.h"
+
 #include <cstdint>
 #include <vector>
 
@@ -17,30 +19,34 @@ enum common_reasoning_budget_state {
 // reasoning block (e.g. between <think> and </think>).
 //
 // State machine: IDLE -> COUNTING -> WAITING_UTF8 -> FORCING -> DONE
-//   IDLE:         passthrough, watching for start_tokens sequence
-//   COUNTING:     counting down remaining tokens, watching for natural end_tokens
+//   IDLE:         passthrough, watching for a start sequence
+//   COUNTING:     counting down remaining tokens, watching for a natural end sequence
 //   WAITING_UTF8: budget exhausted, allowing tokens to complete a UTF-8 sequence
 //   FORCING:      forces forced_tokens token-by-token (all other logits -> -inf)
 //   DONE:         passthrough forever
 //
 // Parameters:
 //   vocab          - vocabulary (used for UTF-8 boundary detection; can be nullptr)
-//   start_tokens   - token sequence that activates counting
-//   end_tokens     - token sequence for natural deactivation
+//   start_seqs     - token sequences, any of which activates counting
+//   end_seqs       - token sequences, any of which naturally deactivates
 //   forced_tokens  - token sequence forced when budget expires
 //   budget         - max tokens allowed in the reasoning block
 //   initial_state  - initial state
 //
 struct llama_sampler * common_reasoning_budget_init(
-        const struct llama_vocab       * vocab,
-        const std::vector<llama_token> & start_tokens,
-        const std::vector<llama_token> & end_tokens,
-        const std::vector<llama_token> & forced_tokens,
-        int32_t                          budget,
-        common_reasoning_budget_state    initial_state = REASONING_BUDGET_IDLE);
+        const struct llama_vocab        * vocab,
+        const std::vector<llama_tokens> & start_seqs,
+        const std::vector<llama_tokens> & end_seqs,
+        const llama_tokens              & forced_tokens,
+        int32_t                           budget,
+        common_reasoning_budget_state     initial_state = REASONING_BUDGET_IDLE);
 
 common_reasoning_budget_state common_reasoning_budget_get_state(const struct llama_sampler * smpl);
 
+// The end sequence that transitioned the sampler to DONE, or nullptr if none
+// was recorded. Cleared when a new start sequence re-arms the sampler.
+const llama_tokens * common_reasoning_budget_get_end_match(const struct llama_sampler * smpl);
+
 // Manually transition the reasoning budget sampler into the FORCING state.
 // Returns true if the transition occurred.
 bool common_reasoning_budget_force(struct llama_sampler * smpl);

common/sampling.cpp

@@ -299,7 +299,7 @@ struct common_sampler * common_sampler_init(const struct llama_model * model, st
     if (!params.reasoning_budget_start.empty() && !params.reasoning_budget_end.empty() && (params.grammar_lazy || params.reasoning_budget_tokens >= 0 || params.reasoning_control)) {
         rbudget = common_reasoning_budget_init(
             vocab,
-            params.reasoning_budget_start,
+            {params.reasoning_budget_start},
             params.reasoning_budget_end,
             params.reasoning_budget_forced,
             params.reasoning_budget_tokens < 0 ? INT_MAX : params.reasoning_budget_tokens);
@@ -453,6 +453,17 @@ void common_sampler_accept(struct common_sampler * gsmpl, llama_token token, boo
 
     if (gsmpl->rbudget && is_generated) {
         llama_sampler_accept(gsmpl->rbudget, token);
+
+        // if done, replay end sequence which may contain a grammar trigger
+        const bool is_done = common_reasoning_budget_get_state(gsmpl->rbudget) == REASONING_BUDGET_DONE;
+        if (gsmpl->grmr && !accept_grammar && is_done) {
+            const llama_tokens * end_seq = common_reasoning_budget_get_end_match(gsmpl->rbudget);
+            if (end_seq) {
+                for (const llama_token end_token : *end_seq) {
+                    llama_sampler_accept(gsmpl->grmr, end_token);
+                }
+            }
+        }
     }
 
     if (gsmpl->grmr && accept_grammar) {

common/speculative.cpp

@@ -437,6 +437,7 @@ struct common_speculative_impl_draft_eagle3 : public common_speculative_impl {
     int32_t n_embd_dec = 0;       // draft hidden size
     int32_t n_embd_enc = 0;       // target_layer_ids_n * target_hidden_size
     int32_t n_embd_tgt = 0;       // target model hidden size
+    int32_t n_layer_tgt = 0;      // target model layer count
 
     const int32_t * target_layer_ids   = nullptr; // model_dft's extract layer indices
     uint32_t        target_layer_ids_n = 0;
@@ -478,6 +479,7 @@ struct common_speculative_impl_draft_eagle3 : public common_speculative_impl {
         n_embd_tgt = llama_model_n_embd(model_tgt);
         n_embd_dec = llama_model_n_embd(model_dft);
         n_embd_enc = (int32_t) target_layer_ids_n * n_embd_tgt;
+        n_layer_tgt = llama_model_n_layer(model_tgt);
 
         const int32_t n_b = (int32_t) llama_n_batch(ctx_dft);
         batch = llama_batch_init(/*n_tokens=*/ n_b, /*embd=*/ n_embd_dec, /*n_seq_max=*/ 1);
@@ -510,9 +512,15 @@ struct common_speculative_impl_draft_eagle3 : public common_speculative_impl {
             }
         }
 
-        // turn on extraction of the target layers' input embeddings
+        // turn on extraction of the target layers' hidden states
         for (uint32_t k = 0; k < target_layer_ids_n; ++k) {
-            llama_set_embeddings_layer_inp(ctx_tgt, (uint32_t) target_layer_ids[k], true);
+            if (target_layer_ids[k] < n_layer_tgt) {
+                llama_set_embeddings_layer_inp(ctx_tgt, (uint32_t) target_layer_ids[k], true);
+            } else if (target_layer_ids[k] == n_layer_tgt) {
+                llama_set_embeddings_nextn(ctx_tgt, true, /*masked*/ false);
+            } else {
+                GGML_ABORT("EAGLE3: target layer id %d exceeds target n_layer %d", target_layer_ids[k], n_layer_tgt);
+            }
         }
 
         // turn on extraction of the draft model's pre-norm hidden state
@@ -600,7 +608,9 @@ struct common_speculative_impl_draft_eagle3 : public common_speculative_impl {
         features_buf.resize((size_t) n_tokens * n_embd_enc, 0.0f);
 
         for (uint32_t k = 0; k < target_layer_ids_n; ++k) {
-            const float * layer = llama_get_embeddings_layer_inp(ctx_tgt, (uint32_t) target_layer_ids[k]);
+            const float * layer = target_layer_ids[k] < n_layer_tgt
+                ? llama_get_embeddings_layer_inp(ctx_tgt, (uint32_t) target_layer_ids[k])
+                : llama_get_embeddings_nextn(ctx_tgt);
             if (!layer) {
                 GGML_ABORT("EAGLE3: target layer %d input not extracted.", target_layer_ids[k]);
             }
@@ -2284,7 +2294,7 @@ common_speculative_init_result::common_speculative_init_result(
     std::string model_path;
     if (has_draft) {
         model_path = params.speculative.draft.mparams.path;
-        LOG_TRC("%s: loading draft model '%s'\n", __func__, model_path.c_str());
+        LOG_INF("%s: loading draft model '%s'\n", __func__, model_path.c_str());
 
         llama_model * model_dft = llama_model_load_from_file(params.model.path.c_str(), mparams);
         if (model_dft == NULL) {
@@ -2304,7 +2314,7 @@ common_speculative_init_result::common_speculative_init_result(
     } else if (spec_mtp) {
         model_path = params.model.path;
 
-        LOG_TRC("%s: creating MTP draft context against the target model '%s'\n", __func__, model_path.c_str());
+        LOG_INF("%s: creating MTP draft context against the target model '%s'\n", __func__, model_path.c_str());
 
         llama_context * ctx_dft = llama_init_from_model(model_tgt, cparams);
         if (ctx_dft == nullptr) {

common/subproc.cpp

@@ -0,0 +1,143 @@
+#include "subproc.h"
+
+bool common_subproc::is_supported() {
+#ifdef LLAMA_SUBPROCESS
+    return true;
+#else
+    return false;
+#endif
+}
+
+#ifdef LLAMA_SUBPROCESS
+
+static std::vector<char *> to_cstr_vec(const std::vector<std::string> & v) {
+    std::vector<char *> r;
+    r.reserve(v.size() + 1);
+    for (const auto & s : v) {
+        r.push_back(const_cast<char *>(s.c_str()));
+    }
+    r.push_back(nullptr);
+    return r;
+}
+
+common_subproc::~common_subproc() {
+    if (is_created) {
+        subprocess_destroy(&proc);
+        is_created = false;
+    }
+}
+
+bool common_subproc::create(
+        const std::vector<std::string> & args,
+        int options,
+        const std::vector<std::string> & env,
+        const char * cwd) {
+    auto argv = to_cstr_vec(args);
+
+    int result;
+    if (env.empty() && cwd == nullptr) {
+        result = subprocess_create(argv.data(), options, &proc);
+    } else {
+        auto envp = to_cstr_vec(env);
+        result = subprocess_create_ex(argv.data(), options, env.empty() ? nullptr : envp.data(), cwd, &proc);
+    }
+
+    is_created = result == 0;
+    return is_created;
+}
+
+bool common_subproc::has_handle() const {
+    if (!is_created) {
+        return false;
+    }
+#if defined(_WIN32)
+    return proc.hProcess != nullptr;
+#else
+    return proc.child > 0;
+#endif
+}
+
+bool common_subproc::alive() {
+    return is_created && subprocess_alive(&proc);
+}
+
+FILE * common_subproc::stdin_file() {
+    return is_created ? subprocess_stdin(&proc) : nullptr;
+}
+
+FILE * common_subproc::stdout_file() {
+    return is_created ? subprocess_stdout(&proc) : nullptr;
+}
+
+FILE * common_subproc::stderr_file() {
+    return is_created ? subprocess_stderr(&proc) : nullptr;
+}
+
+void common_subproc::close_stdin() {
+    if (is_created && proc.stdin_file) {
+        fclose(proc.stdin_file);
+        proc.stdin_file = nullptr;
+    }
+}
+
+void common_subproc::terminate() {
+    if (has_handle()) {
+        subprocess_terminate(&proc);
+    }
+}
+
+int common_subproc::join() {
+    int exit_code = -1;
+    if (is_created) {
+        subprocess_join(&proc, &exit_code);
+        subprocess_destroy(&proc);
+        is_created = false;
+    }
+    return exit_code;
+}
+
+#else // !LLAMA_SUBPROCESS
+
+common_subproc::~common_subproc() = default;
+
+bool common_subproc::create(
+        const std::vector<std::string> &,
+        int,
+        const std::vector<std::string> &,
+        const char *) {
+    (void)(proc);
+    (void)(is_created);
+    return false;
+}
+
+bool common_subproc::has_handle() const {
+    return false;
+}
+
+bool common_subproc::alive() {
+    return false;
+}
+
+FILE * common_subproc::stdin_file() {
+    return nullptr;
+}
+
+FILE * common_subproc::stdout_file() {
+    return nullptr;
+}
+
+FILE * common_subproc::stderr_file() {
+    return nullptr;
+}
+
+void common_subproc::close_stdin() {
+}
+
+void common_subproc::terminate() {
+}
+
+int common_subproc::join() {
+    return -1;
+}
+
+#endif // LLAMA_SUBPROCESS

common/subproc.h

@@ -0,0 +1,59 @@
+#pragma once
+
+#include <atomic>
+#include <cstdio>
+#include <string>
+#include <vector>
+
+#ifdef LLAMA_SUBPROCESS
+#include <sheredom/subprocess.h>
+#else
+// dummy values to allow compilation when subprocess is disabled
+struct subprocess_s {};
+static constexpr int subprocess_option_no_window = 0;
+static constexpr int subprocess_option_combined_stdout_stderr = 0;
+static constexpr int subprocess_option_inherit_environment = 0;
+static constexpr int subprocess_option_search_user_path = 0;
+#endif
+
+// RAII-style wrapper around https://github.com/sheredom/subprocess.h,
+// exposing method calls instead of free functions operating on subprocess_s.
+struct common_subproc {
+    common_subproc() = default;
+    ~common_subproc();
+
+    common_subproc(const common_subproc &) = delete;
+    common_subproc & operator=(const common_subproc &) = delete;
+
+    // spawn a child process; if env is non-empty it replaces the child's environment
+    // (do not combine with subprocess_option_inherit_environment)
+    bool create(
+            const std::vector<std::string> & args,
+            int options,
+            const std::vector<std::string> & env = {},
+            const char * cwd = nullptr);
+
+    bool alive();
+
+    // true if LLAMA_SUBPROCESS was enabled at build time; when false, create() always fails
+    static bool is_supported();
+
+    FILE * stdin_file();
+    FILE * stdout_file();
+    FILE * stderr_file();
+
+    // close stdin and detach it from the process, so a later join()/destroy() won't double-close it;
+    // use this after writing all input to signal EOF to the child while it's still running
+    void close_stdin();
+
+    void terminate();
+
+    // wait for the process to exit, release the underlying handle and return its exit code
+    int join();
+
+private:
+    subprocess_s proc {};
+    std::atomic<bool> is_created{false};
+
+    bool has_handle() const;
+};

common/trie.cpp

@@ -0,0 +1,123 @@
+#include "trie.h"
+
+#include "unicode.h"
+
+#include <deque>
+
+common_trie::match_result common_trie::check_at(std::string_view sv, size_t start_pos) const {
+    size_t current = 0; // Start at root
+    size_t pos = start_pos;
+
+    // LOG_DBG("%s: checking at pos %zu, sv='%s'\n", __func__, start_pos, std::string(sv).c_str());
+
+    while (pos < sv.size()) {
+        auto result = common_parse_utf8_codepoint(sv, pos);
+        if (result.status != utf8_parse_result::SUCCESS) {
+            break;
+        }
+
+        auto it = nodes[current].children.find(result.codepoint);
+        if (it == nodes[current].children.end()) {
+            // Can't continue matching
+            return match_result{match_result::NO_MATCH};
+        }
+
+        current = it->second;
+        pos += result.bytes_consumed;
+
+        // Check if we've matched a complete word
+        if (nodes[current].pattern >= 0) {
+            return match_result{match_result::COMPLETE_MATCH};
+        }
+    }
+
+    // Reached end of input while still in the trie (not at root)
+    if (current != 0) {
+        // We're in the middle of a potential match
+        return match_result{match_result::PARTIAL_MATCH};
+    }
+
+    // Reached end at root (no match)
+    return match_result{match_result::NO_MATCH};
+}
+
+int32_t common_trie::insert(const std::string & word) {
+    std::vector<uint32_t> symbols;
+    size_t pos = 0;
+    while (pos < word.length()) {
+        auto result = common_parse_utf8_codepoint(word, pos);
+        if (result.status != utf8_parse_result::SUCCESS) {
+            break;
+        }
+
+        symbols.push_back(result.codepoint);
+        pos += result.bytes_consumed;
+    }
+    return insert(symbols);
+}
+
+int32_t common_trie::insert(const std::vector<uint32_t> & symbols) {
+    size_t current = 0;
+    for (uint32_t ch : symbols) {
+        auto it = nodes[current].children.find(ch);
+        if (it == nodes[current].children.end()) {
+            size_t child = create_node();
+            nodes[current].children[ch] = child;
+            current = child;
+        } else {
+            current = it->second;
+        }
+    }
+    if (nodes[current].pattern < 0) {
+        nodes[current].pattern = n_patterns++;
+    }
+    return nodes[current].pattern;
+}
+
+common_aho_corasick::common_aho_corasick(common_trie trie) : t(std::move(trie)) {
+    const auto & nodes = t.nodes;
+    const size_t n = nodes.size();
+
+    fail.assign(n, 0);
+    order.reserve(n);
+
+    std::deque<size_t> queue{ 0 };
+    while (!queue.empty()) {
+        size_t u = queue.front();
+        queue.pop_front();
+        order.push_back(u);
+        for (const auto & [ch, v] : nodes[u].children) {
+            if (u != 0) {
+                size_t f = fail[u];
+                while (f && nodes[f].children.find(ch) == nodes[f].children.end()) {
+                    f = fail[f];
+                }
+                auto it = nodes[f].children.find(ch);
+                fail[v] = (it != nodes[f].children.end() && it->second != v) ? it->second : 0;
+            }
+            queue.push_back(v);
+        }
+    }
+
+    // fail[u] points to a strictly shorter suffix, so the first pattern found on
+    // the fail chain (including u itself) is the longest pattern ending at u
+    match.assign(n, -1);
+    for (size_t u : order) {
+        match[u] = nodes[u].pattern >= 0 ? nodes[u].pattern : (u != 0 ? match[fail[u]] : -1);
+    }
+
+    for (const auto & node : nodes) {
+        for (const auto & [ch, v] : node.children) {
+            alphabet.insert(ch);
+        }
+    }
+}
+
+size_t common_aho_corasick::next(size_t state, uint32_t ch) const {
+    const auto & nodes = t.nodes;
+    while (state && nodes[state].children.find(ch) == nodes[state].children.end()) {
+        state = fail[state];
+    }
+    auto it = nodes[state].children.find(ch);
+    return it != nodes[state].children.end() ? it->second : 0;
+}

common/trie.h

@@ -0,0 +1,73 @@
+#pragma once
+
+#include <cstdint>
+#include <map>
+#include <set>
+#include <string>
+#include <string_view>
+#include <vector>
+
+// Trie for matching multiple literals.
+// This is used in common_peg_until_parser and to build a GBNF exclusion grammar
+struct common_trie {
+    struct node {
+        std::map<uint32_t, size_t> children;  // Use uint32_t to store Unicode codepoints
+        int32_t pattern = -1;                 // index of the pattern ending at this node, -1 if none
+    };
+
+    std::vector<node> nodes;
+
+    common_trie() {
+        create_node(); // root node
+    }
+
+    common_trie(const std::vector<std::string> & words) : common_trie() {
+        for (const auto & w : words) {
+            insert(w);
+        }
+    }
+
+    enum match_result { NO_MATCH, PARTIAL_MATCH, COMPLETE_MATCH };
+
+    // Check if a delimiter starts at the given position
+    match_result check_at(std::string_view sv, size_t start_pos) const;
+
+    // Insert a word as a sequence of Unicode codepoints, returns its pattern index
+    int32_t insert(const std::string & word);
+
+    // Insert a raw symbol sequence, returns its pattern index (insertion order,
+    // duplicates keep the first index)
+    int32_t insert(const std::vector<uint32_t> & symbols);
+
+  private:
+    int32_t n_patterns = 0;
+
+    size_t create_node() {
+        size_t index = nodes.size();
+        nodes.emplace_back();
+        return index;
+    }
+};
+
+// Aho-Corasick automaton
+struct common_aho_corasick {
+    common_trie          t;
+    std::vector<size_t>  fail;     // failure links
+    std::vector<size_t>  order;    // states in BFS order
+    std::vector<int32_t> match;    // longest pattern ending at each state (directly or via a suffix link), -1 if none
+    std::set<uint32_t>   alphabet; // every character with a transition
+
+    common_aho_corasick(common_trie trie);
+
+    common_aho_corasick(const std::vector<std::string> & strings)
+        : common_aho_corasick(common_trie(strings)) {}
+
+    size_t num_states()          const { return t.nodes.size(); }
+    bool   is_terminal(size_t s) const { return match[s] >= 0; }
+
+    // index of the longest pattern ending at this state, -1 if none
+    int32_t match_pattern(size_t s) const { return match[s]; }
+
+    // follow failure links until a transition on `ch` exists.
+    size_t next(size_t state, uint32_t ch) const;
+};

conversion/__init__.py

@@ -158,6 +158,8 @@ TEXT_MODEL_MAP: dict[str, str] = {
     "MiniCPMForCausalLM": "minicpm",
     "MiniCPMV4_6ForConditionalGeneration": "minicpm",
     "MiniMaxM2ForCausalLM": "minimax",
+    "MiniMaxM3SparseForCausalLM": "minimax",
+    "MiniMaxM3SparseForConditionalGeneration": "minimax",
     "Ministral3ForCausalLM": "mistral3",
     "Mistral3ForConditionalGeneration": "mistral3",
     "MistralForCausalLM": "llama",
@@ -165,6 +167,7 @@ TEXT_MODEL_MAP: dict[str, str] = {
     "ModernBertForMaskedLM": "bert",
     "ModernBertForSequenceClassification": "bert",
     "ModernBertModel": "bert",
+    "NanbeigeForCausalLM": "nanbeige",
     "NemotronForCausalLM": "nemotron",
     "NemotronHForCausalLM": "nemotron",
     "NeoBERT": "bert",
@@ -267,6 +270,7 @@ MMPROJ_MODEL_MAP: dict[str, str] = {
     "Gemma4UnifiedForConditionalGeneration": "gemma",
     "Glm4vForConditionalGeneration": "qwen3vl",
     "Glm4vMoeForConditionalGeneration": "qwen3vl",
+    "Glm5vForConditionalGeneration": "kimivl",
     "GlmOcrForConditionalGeneration": "qwen3vl",
     "GlmasrModel": "ultravox",
     "Granite4VisionForConditionalGeneration": "granite",
@@ -285,6 +289,7 @@ MMPROJ_MODEL_MAP: dict[str, str] = {
     "LlavaForConditionalGeneration": "llava",
     "MERaLiON2ForConditionalGeneration": "ultravox",
     "MiMoV2ForCausalLM": "mimo",
+    "MiniMaxM3SparseForConditionalGeneration": "minimax",
     "MiniCPMV4_6ForConditionalGeneration": "minicpm",
     "Mistral3ForConditionalGeneration": "llava",
     "NemotronH_Nano_VL_V2": "nemotron",

conversion/base.py

@@ -1156,7 +1156,7 @@ class TextModel(ModelBase):
                 or "projector." in name or "pre_mm_projector_norm" in name \
                 or "image_newline" in name or "view_seperator" in name \
                 or "patch_embed" in name or "patch_embedding" in name \
-                or "patch_merger." in name or "model.connector." in name:
+                or "patch_merger." in name or "patch_merge_mlp." in name or "model.connector." in name:
             return None
 
         return super().filter_tensors(item)
@@ -1203,7 +1203,7 @@ class TextModel(ModelBase):
             self.gguf_writer.add_embedding_length(n_embd)
             logger.info(f"gguf: embedding length = {n_embd}")
 
-        if (n_ff := self.find_hparam(["prefix_dense_intermediate_size", "intermediate_size", "n_inner", "hidden_dim"], optional=True)) is not None:
+        if (n_ff := self.find_hparam(["prefix_dense_intermediate_size", "dense_intermediate_size", "intermediate_size", "n_inner", "hidden_dim"], optional=True)) is not None:
             self.gguf_writer.add_feed_forward_length(n_ff)
             logger.info(f"gguf: feed forward length = {n_ff}")
 

conversion/glm.py

@@ -237,6 +237,9 @@ class GlmMoeDsaModel(DeepseekV2Model):
         self.gguf_writer.add_indexer_head_count(self.hparams["index_n_heads"])
         self.gguf_writer.add_indexer_key_length(self.hparams["index_head_dim"])
         self.gguf_writer.add_indexer_top_k(self.hparams["index_topk"])
+        if (indexer_types := self.hparams.get("indexer_types")) is not None:
+            indexer_types = [t == "full" for t in indexer_types]
+            self.gguf_writer.add_indexer_types(indexer_types)
 
 
 @ModelBase.register("SolarOpenForCausalLM")

conversion/kimivl.py

@@ -152,3 +152,19 @@ class KimiK25Model(MmprojModel):
             name = name.replace(".proj.2.", ".proj.linear_2.")
 
         yield from super().modify_tensors(data_torch, name, bid)
+
+
+@ModelBase.register("Glm5vForConditionalGeneration")
+class Glm5vModel(KimiK25Model):
+    """GLM-5.2-Vision MoonViT3d encoder and projector
+
+    Uses the same vision encoder and projector as Kimi-K2.5, so it reuses the
+    kimik25 projector type. The image begin/end tokens differ, but they are
+    resolved at runtime from the text model vocab.
+    """
+
+    def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None) -> Iterable[tuple[str, Tensor]]:
+        if name.startswith("mm_projector.linear_"):
+            name = name.replace("mm_projector.linear_", "mm_projector.proj.linear_", 1)
+
+        yield from super().modify_tensors(data_torch, name, bid)

conversion/llama.py

@@ -69,9 +69,14 @@ class LlamaModel(TextModel):
                 target_config = {**target_config, **target_config["text_config"]}
             self.target_vocab_size = target_config["vocab_size"]
 
-            # target_layers: derived from target model layer count (low/mid/high)
+            # target_layers: use the eagle3 config's explicit aux hidden-state layer ids
+            # if present, else derive from the target layer count.
             target_num_layers = target_config["num_hidden_layers"]
-            target_layers = [2, target_num_layers // 2, target_num_layers - 3]
+            aux_layer_ids = eagle3_raw_config.get("eagle_aux_hidden_state_layer_ids")
+            if aux_layer_ids:
+                target_layers = aux_layer_ids
+            else:
+                target_layers = [2, target_num_layers // 2, target_num_layers - 3]
             logger.info(f"EAGLE-3: target_layers = {target_layers} (target model has {target_num_layers} layers)")
             self.gguf_writer.add_target_layers(target_layers)
 
@@ -90,6 +95,12 @@ class LlamaModel(TextModel):
             logger.info(f"EAGLE-3: norm_before_residual = {norm_before_residual}")
             self.gguf_writer.add_norm_before_residual(norm_before_residual)
 
+            # norm_before_fc: RMSNorm applied to the fused target features before the
+            # fc projection (e.g. nvidia/gpt-oss-120b-Eagle3-v3)
+            norm_before_fc = eagle3_raw_config.get("norm_before_fc", False)
+            logger.info(f"EAGLE-3: norm_before_fc = {norm_before_fc}")
+            self.gguf_writer.add_norm_before_fc(norm_before_fc)
+
     def set_vocab(self):
         # eagle3: use tokenizer from target model if provided
         original_dir_model = None
@@ -222,6 +233,9 @@ class LlamaModel(TextModel):
             if name == "fc.weight":
                 yield (name, data_torch)
                 return
+            if name == "input_norm.weight":
+                yield (self.format_tensor_name(gguf.MODEL_TENSOR.ENC_OUTPUT_NORM), data_torch)
+                return
             if name == "d2t":
                 # store for manual int64 handling in prepare_tensors (avoid F32 conversion)
                 if not hasattr(self, '_eagle3_int_tensors'):

conversion/mimo.py

@@ -1,8 +1,9 @@
 from __future__ import annotations
 
+import json
 import re
 
-from typing import Callable, TYPE_CHECKING
+from typing import Any, Callable, Iterable, TYPE_CHECKING
 
 import torch
 
@@ -229,7 +230,13 @@ class MimoV2Model(TextModel):
 
 
 @ModelBase.register("MiMoV2ForCausalLM")
-class MiMoV2VisionModel(MmprojModel):
+class MiMoV2VisionAudioModel(MmprojModel):
+    has_audio_encoder = True
+
+    _audio_tok_hparams: dict[str, Any] | None = None
+    _rvq_codebook_sizes: list[int] | None = None
+    _code_embd: dict[int, Tensor] | None = None
+
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
         assert self.hparams_vision is not None
@@ -253,10 +260,22 @@ class MiMoV2VisionModel(MmprojModel):
         self.visual_token_window_size = int(hp.get("visual_token_window_size", -1))
         self.use_sink = bool(hp.get("use_sink", False))
 
+    def get_audio_config(self) -> dict[str, Any] | None:
+        if self._audio_tok_hparams is None:
+            path = self.dir_model / "audio_tokenizer" / "config.json"
+            with open(path, "r", encoding="utf-8") as f:
+                cfg = json.load(f)
+            # aliases so MmprojModel.find_aparam() / n_block_keys can resolve them
+            cfg["hidden_size"] = cfg["d_model"]
+            cfg["intermediate_size"] = cfg["encoder_ffn_dim"]
+            cfg["num_attention_heads"] = cfg["encoder_attention_heads"]
+            self._audio_tok_hparams = cfg
+        return self._audio_tok_hparams
+
     def set_gguf_parameters(self):
         super().set_gguf_parameters()
 
-        self.gguf_writer.add_clip_projector_type(gguf.VisionProjectorType.MIMOVL)
+        self.gguf_writer.add_clip_vision_projector_type(gguf.VisionProjectorType.MIMOVL)
         self.gguf_writer.add_vision_use_silu(True)
         self.gguf_writer.add_vision_head_count_kv(self.num_kv_heads)
         self.gguf_writer.add_vision_spatial_merge_size(self.spatial_merge_size)
@@ -266,19 +285,45 @@ class MiMoV2VisionModel(MmprojModel):
         self.gguf_writer.add_vision_min_pixels(int(self.preprocessor_config["min_pixels"]))
         self.gguf_writer.add_vision_max_pixels(int(self.preprocessor_config["max_pixels"]))
 
+        assert self.hparams_audio is not None
+        self.gguf_writer.add_clip_audio_projector_type(gguf.VisionProjectorType.MIMO_AUDIO)
+        self.gguf_writer.add_audio_num_mel_bins(self.hparams_audio["n_mels"])
+        self.gguf_writer.add_audio_attention_layernorm_eps(self.hparams_audio.get("layer_norm_eps", 1e-5))
+
+        assert self._rvq_codebook_sizes is not None
+        self.gguf_writer.add_audio_rvq_num_quantizers(len(self._rvq_codebook_sizes))
+        self.gguf_writer.add_audio_rvq_codebook_size(self._rvq_codebook_sizes)
+
+        n_layer = self.hparams_audio["encoder_layers"]
+        swa_per_block = self.hparams_audio.get("swa_per_block", 1)
+        if self.hparams_audio.get("hybrid_attention") and swa_per_block > 1:
+            wa_pattern = [0 if i % swa_per_block < swa_per_block - 1 else -1 for i in range(n_layer)]
+        else:
+            wa_pattern = [-1] * n_layer
+        self.gguf_writer.add_audio_wa_pattern_mode(wa_pattern)
+        self.gguf_writer.add_audio_window_size(int(self.hparams_audio["encoder_attn_window_size"][0]))
+
+        audio_cfg = self.global_config["audio_config"]
+        self.gguf_writer.add_audio_local_block_count(int(audio_cfg["input_local_layers"]))
+        self.gguf_writer.add_audio_local_group_size(int(audio_cfg["group_size"]))
+
     def tensor_force_quant(self, name, new_name, bid, n_dims):
-        # Sinks must be F32: any sink-style softmax/mask add in ggml requires
-        # F32, and we fold sinks into a host-built F32 mask at encode time.
-        if new_name.endswith(".attn_sinks"):
+        # for audio encoder: keep codebook in F32
+        if new_name in (
+            gguf.TENSOR_NAMES[gguf.MODEL_TENSOR.A_ENC_RVQ_CODEBOOK] + ".weight",
+            gguf.TENSOR_NAMES[gguf.MODEL_TENSOR.A_MM_CODE_EMBD] + ".weight",
+        ):
+            return gguf.GGMLQuantizationType.F32
+        if ("encoder.conv" in name or "encoder.down_sample_layer" in name) and name.endswith(".weight"):
             return gguf.GGMLQuantizationType.F32
         return super().tensor_force_quant(name, new_name, bid, n_dims)
 
     @classmethod
     def filter_tensors(cls, item: tuple[str, Callable[[], Tensor]]) -> tuple[str, Callable[[], Tensor]] | None:
         name, _ = item
-        if not name.startswith("visual."):
-            return None
-        return super().filter_tensors(item)
+        if name.startswith("visual.") or name.startswith("speech_embeddings.") or name.startswith("audio_encoder."):
+            return super().filter_tensors(item)
+        return None
 
     def modify_tensors(self, data_torch, name, bid):
         # Conv3D patch embed: split along the temporal axis (kt=2) into two Conv2D
@@ -292,4 +337,64 @@ class MiMoV2VisionModel(MmprojModel):
             yield (embd_name + ".weight.1", data_torch[:, :, 1, ...])
             return
 
+        if m := re.match(r"^speech_embeddings\.(\d+)\.weight$", name):
+            if self._code_embd is None:
+                self._code_embd = {}
+            self._code_embd[int(m.group(1))] = data_torch
+
+            n_channels = int(self.global_config["audio_config"]["audio_channels"])
+            if len(self._code_embd) < n_channels:
+                return
+            merged = torch.stack([self._code_embd.pop(i) for i in range(n_channels)], dim=0)
+            yield (self.format_tensor_name(gguf.MODEL_TENSOR.A_MM_CODE_EMBD), merged)
+            return
+
+        if "conv1.bias" in name or "conv2.bias" in name:
+            # transpose conv1/conv2 bias so it broadcasts against [n_frames, C_out, 1]
+            data_torch = data_torch.unsqueeze(-1)
+
+        if name == "audio_encoder.projection.mlp.0.weight":
+            yield (self.format_tensor_name(gguf.MODEL_TENSOR.A_MMPROJ, 1), data_torch)
+            return
+        if name == "audio_encoder.projection.mlp.2.weight":
+            yield (self.format_tensor_name(gguf.MODEL_TENSOR.A_MMPROJ, 2), data_torch)
+            return
+
         yield from super().modify_tensors(data_torch, name, bid)
+
+    def generate_extra_tensors(self) -> Iterable[tuple[str, Tensor]]:
+        # note: audio encoder is in its own subdir "audio_tokenizer"
+        from safetensors.torch import load_file
+
+        tok_dir = self.dir_model / "audio_tokenizer"
+        state_dict = load_file(tok_dir / "model.safetensors")
+
+        codebook_re = re.compile(r"^encoder\.quantizer\.vq\.layers\.(\d+)\._codebook\.embed$")
+        codebooks: dict[int, Tensor] = {}
+
+        # EMA/training-only RVQ buffers - not needed for inference (nearest-codebook
+        # lookup only reads "_codebook.embed")
+        skip_suffixes = (
+            "_codebook.cluster_size",
+            "_codebook.embed_avg",
+            "_codebook.inited",
+        )
+        for name, tensor in state_dict.items():
+            if name.endswith(skip_suffixes):
+                continue
+            if m := codebook_re.match(name):
+                codebooks[int(m.group(1))] = tensor
+                continue
+            yield name, tensor
+
+        # gather codebooks and merge into 3D tensor, similar to MoE MLP tensors
+        n_q = len(codebooks)
+        ordered = [codebooks[i] for i in range(n_q)]
+        self._rvq_codebook_sizes = [int(cb.shape[0]) for cb in ordered]
+        max_bins = max(self._rvq_codebook_sizes)
+        dim = ordered[0].shape[1]
+        merged = ordered[0].new_zeros(n_q, max_bins, dim)
+        for i, cb in enumerate(ordered):
+            merged[i, : cb.shape[0], :] = cb
+
+        yield (self.format_tensor_name(gguf.MODEL_TENSOR.A_ENC_RVQ_CODEBOOK), merged)

conversion/minimax.py

@@ -7,7 +7,7 @@ import torch
 if TYPE_CHECKING:
     from torch import Tensor
 
-from .base import ModelBase, TextModel, gguf
+from .base import ModelBase, TextModel, MmprojModel, gguf
 
 
 @ModelBase.register("MiniMaxM2ForCausalLM")
@@ -23,7 +23,7 @@ class MiniMaxM2Model(TextModel):
 
     def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None):
         # merge expert weights
-        if 'experts' in name:
+        if "block_sparse_moe.experts." in name:
             n_experts = self.find_hparam(["num_local_experts", "num_experts"])
             assert bid is not None
 
@@ -52,3 +52,118 @@ class MiniMaxM2Model(TextModel):
             return
 
         yield from super().modify_tensors(data_torch, name, bid)
+
+
+@ModelBase.register("MiniMaxM3SparseForCausalLM", "MiniMaxM3SparseForConditionalGeneration")
+class MiniMaxM3Model(MiniMaxM2Model):
+    model_arch = gguf.MODEL_ARCH.MINIMAXM3
+
+    def tensor_force_quant(self, name, new_name, bid, n_dims):
+        if ".indexer." in new_name:
+            return gguf.GGMLQuantizationType.F32
+        return super().tensor_force_quant(name, new_name, bid, n_dims)
+
+    def set_gguf_parameters(self):
+        super().set_gguf_parameters()
+
+        self.gguf_writer.add_expert_shared_count(self.find_hparam(["n_shared_experts"]))
+        self.gguf_writer.add_expert_weights_scale(self.find_hparam(["routed_scaling_factor"]))
+        self.gguf_writer.add_expert_weights_norm(True)
+
+        sac = self.find_hparam(["sparse_attention_config"])
+        self.gguf_writer.add_indexer_head_count(sac["sparse_num_index_heads"])
+        self.gguf_writer.add_indexer_key_length(sac["sparse_index_dim"])
+        self.gguf_writer.add_indexer_top_k(sac["sparse_topk_blocks"])
+        self.gguf_writer.add_indexer_block_size(sac["sparse_block_size"])
+        self.gguf_writer.add_indexer_local_blocks(sac["sparse_local_block"])
+
+        moe_layer_freq = self.find_hparam(["moe_layer_freq"])
+        n_dense = 0
+        for v in moe_layer_freq:
+            if v == 0:
+                n_dense += 1
+            else:
+                break
+        self.gguf_writer.add_leading_dense_block_count(n_dense)
+
+    def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None):
+        # Gemma-style (1 + w) RMSNorm: bake the +1 in so llama.cpp can use plain RMSNorm
+        if name.endswith("norm.weight"):
+            data_torch = data_torch + 1.0
+
+        yield from super().modify_tensors(data_torch, name, bid)
+
+
+@ModelBase.register("MiniMaxM3SparseForConditionalGeneration", "MiniMaxM3VLForConditionalGeneration")
+class MiniMaxM3VisionModel(MmprojModel):
+    @classmethod
+    def filter_tensors(cls, item):
+        name, gen = item
+        # keep only the vision-side tensors; text / mtp / sparse-index are dropped
+        if not name.startswith(("vision_tower.", "multi_modal_projector.", "patch_merge_mlp.")):
+            return None
+        return super().filter_tensors((name, gen))
+
+    def set_gguf_parameters(self):
+        super().set_gguf_parameters()
+        assert self.hparams_vision is not None
+
+        self.gguf_writer.add_clip_projector_type(gguf.VisionProjectorType.MINIMAXM3)
+        self.gguf_writer.add_vision_use_gelu(True)
+
+        # the ViT carries its own LayerNorm eps (text tower uses a different one)
+        self.gguf_writer.add_vision_attention_layernorm_eps(
+            self.hparams_vision.get("layer_norm_eps", 1e-5)
+        )
+
+        comp = self.hparams_vision.get("img_token_compression_config", {})
+        merge_size = comp.get("spatial_merge_size", 2)
+        self.gguf_writer.add_vision_spatial_merge_size(int(merge_size))
+
+    def modify_tensors(self, data_torch, name, bid):
+        assert self.hparams_vision is not None
+
+        # Conv3d patch embed -> Conv2d slices
+        if name == "vision_tower.vision_model.embeddings.patch_embedding.weight":
+            if data_torch.ndim != 5:
+                raise ValueError(f"unexpected patch_embedding rank {data_torch.ndim} for {name}")
+            kt = data_torch.shape[2]
+            base = gguf.TENSOR_NAMES[gguf.MODEL_TENSOR.V_ENC_EMBD_PATCH]
+            for t in range(kt):
+                suffix = ".weight" if t == 0 else f".weight.{t}"
+                yield (base + suffix, data_torch[:, :, t, ...])
+            return
+
+        # Permute ViT q/k. HF [Ta Ha Wa | Tb Hb Wb | pad] reorder to [Ta Tb | Ha Hb | Wa Wb | pad].
+        for new_name, tensor in super().modify_tensors(data_torch, name, bid):
+            if ".attn_q." in new_name or ".attn_k." in new_name:
+                tensor = self._permute_vit_qk(tensor, new_name)
+            yield new_name, tensor
+
+    def _permute_vit_qk(self, t: "Tensor", new_name: str) -> "Tensor":
+        assert self.hparams_vision is not None
+        n_head = self.hparams_vision["num_attention_heads"]
+        d_head = t.shape[0] // n_head
+        axis_dim = 2 * ((2 * (d_head // 2) // 3) // 2)
+        ah = axis_dim // 2
+        half = 3 * ah
+        perm = []
+        perm += list(range(0, ah))
+        perm += list(range(half, half + ah))
+        perm += list(range(ah, 2 * ah))
+        perm += list(range(half + ah, half + 2 * ah))
+        perm += list(range(2 * ah, 3 * ah))
+        perm += list(range(half + 2 * ah, half + 3 * ah))
+        perm += list(range(2 * half, d_head))
+
+        assert axis_dim % 2 == 0
+        assert 3 * axis_dim <= d_head
+        assert len(perm) == d_head
+        assert sorted(perm) == list(range(d_head)), "perm is not a bijection of d_head"
+        assert t.shape[0] == n_head * d_head, f"{new_name}: {t.shape[0]} != {n_head}*{d_head}"
+        assert d_head == 80
+
+        idx = torch.tensor(perm, dtype=torch.long)
+        if t.ndim == 2:
+            return t.reshape(n_head, d_head, t.shape[1])[:, idx, :].reshape(t.shape)
+        return t.reshape(n_head, d_head)[:, idx].reshape(t.shape)

conversion/nanbeige.py

@@ -0,0 +1,24 @@
+from __future__ import annotations
+
+from .base import ModelBase, gguf, logger
+from .llama import LlamaModel
+
+
+@ModelBase.register("NanbeigeForCausalLM")
+class NanbeigeModel(LlamaModel):
+    model_arch = gguf.MODEL_ARCH.NANBEIGE
+    undo_permute = True
+
+    def set_gguf_parameters(self):
+        super().set_gguf_parameters()
+        hparams = self.hparams
+
+        n_loops = int(hparams.get("num_loops", 1) or 1)
+        if n_loops < 1:
+            n_loops = 1
+        self.gguf_writer.add_num_loops(n_loops)
+        logger.info(f"gguf: num_loops = {n_loops}")
+
+        skip_loop_final_norm = bool(hparams.get("skip_loop_final_norm", False))
+        self.gguf_writer.add_skip_loop_final_norm(skip_loop_final_norm)
+        logger.info(f"gguf: skip_loop_final_norm = {skip_loop_final_norm}")

docs/backend/OPENCL.md

@@ -98,6 +98,24 @@ The OpenCL backend has the following CMake options that control the behavior of
 | `GGML_OPENCL_USE_ADRENO_KERNELS`     | `ON`           | Use kernels optimized for Adreno.         |
 | `GGML_OPENCL_USE_ADRENO_BIN_KERNELS` | `OFF`          | Allow using binary kernel lib for Adreno. |
 
+## Program Binary Cache
+
+Compiled `cl_program` binaries are cached on disk, so subsequent runs skip the expensive
+compile-from-source step when nothing relevant has changed (kernel source, compile options,
+device, driver, or platform version).
+
+The cache is controlled with the `GGML_OPENCL_KERNEL_CACHE_DIR` environment variable:
+
+| Value                                  | Behavior                                       |
+|:---------------------------------------|:-----------------------------------------------|
+| unset / empty / `1` / `default`        | Enabled in the platform default cache directory: `%LOCALAPPDATA%\llama.cpp\cl-cache` (Windows), `~/Library/Caches/llama.cpp/cl-cache` (macOS), `<temp dir>/llama.cpp/cl-cache` elsewhere. |
+| `0` / `off` / `none` / `disable(d)`    | Disabled.                                      |
+| any other value                        | Used verbatim as the cache directory path.     |
+
+If the chosen directory cannot be created or used, the cache disables itself for the process
+and kernels are compiled from source as usual. Set `GGML_OPENCL_KERNEL_CACHE_DEBUG=1` to
+print a HIT/MISS/SAVE trace to stderr.
+
 ## Android
 
 Ubuntu 22.04 is used for targeting Android. Make sure the following tools are accessible from command line,

docs/backend/SYCL.md

@@ -794,6 +794,8 @@ use 1 SYCL GPUs: [0] with Max compute units:512
 | GGML_SYCL_ENABLE_GRAPH | 0 (default) or 1 | Enable running computations through SYCL Graphs feature. Disabled by default because SYCL Graph is still on development, no better performance. |
 | GGML_SYCL_USE_LEVEL_ZERO_API | 1 (default) or 0 | Use Level Zero API for device memory allocation instead of SYCL. Reduces system RAM usage on Intel dGPUs by avoiding DMA-buf/TTM host memory staging. Requires GGML_SYCL_SUPPORT_LEVEL_ZERO_API=ON at build time. SYCL backend always runs on Level Zero running time even if it's set as OFF (The SYCL api will be usage for memory allocation).|
 | GGML_SYCL_ENABLE_DNN | 0 or 1 (default)| Enable running computations through oneDNN and always use oneMKL. |
+| GGML_SYCL_FA_ONEDNN | 1 (default) or 0 | Enable the oneDNN fused SDPA (flash-attention) path on supported GPUs. Set to 0 to always use the native SYCL flash-attention kernel. |
+| GGML_SYCL_FA_ONEDNN_MAX_KV | 0 (default, disabled) or positive integer | By default (0), all sequences are handled by the oneDNN fused SDPA path, regardless of KV length; a positive value caps that length, past which sequences fall back to the native kernel. If GPU driver watchdog resets (DEVICE_LOST) occur during long-context inference, set this near the context depth where they start, e.g. 24576. |
 | GGML_SYCL_ENABLE_VMM | 0 or 1 (default) | Enable the virtual-memory device pool. |
 | GGML_SYCL_ENABLE_FUSION | 0 or 1 (default) | Enable fused-kernel dispatch in graph compute (currently top-k MoE gating). |
 | ZES_ENABLE_SYSMAN | 0 (default) or 1 | Support to get free memory of GPU by sycl::aspect::ext_intel_free_memory.<br>Recommended to use when --split-mode = layer |

docs/build.md

@@ -361,12 +361,6 @@ You can download it from your Linux distro's package manager or from here: [ROCm
 
   Note: `GPU_TARGETS` is optional, omitting it will build the code for all GPUs in the current system.
 
-  To enhance flash attention performance on RDNA3+ or CDNA architectures, you can utilize the rocWMMA library by enabling the `-DGGML_HIP_ROCWMMA_FATTN=ON` option. This requires rocWMMA headers to be installed on the build system.
-
-  The rocWMMA library is included by default when installing the ROCm SDK using the `rocm` meta package provided by AMD. Alternatively, if you are not using the meta package, you can install the library using the `rocwmma-dev` or `rocwmma-devel` package, depending on your system's package manager.
-
-  As an alternative, you can manually install the library by cloning it from the official [GitHub repository](https://github.com/ROCm/rocWMMA), checkout the corresponding version tag (e.g. `rocm-6.2.4`) and set `-DCMAKE_CXX_FLAGS="-I<path/to/rocwmma>/library/include/"` in CMake. This also works under Windows despite not officially supported by AMD.
-
   Note that if you get the following error:
   ```
   clang: error: cannot find ROCm device library; provide its path via '--rocm-path' or '--rocm-device-lib-path', or pass '-nogpulib' to build without ROCm device library

docs/cross-profiler.md

@@ -1,225 +0,0 @@
-# Cross-Backend Profiler
-
-llama.cpp includes a built-in cross-backend profiler that captures per-operation timing, data transfer costs, and tensor shapes across all compute backends. It works with any application built on the ggml scheduler — no source changes needed.
-
-## Supported Backends
-
-| Backend | Status | Timing method |
-|---------|--------|---------------|
-| CPU     | Supported | Wall-clock (`CLOCK_MONOTONIC_RAW`) |
-| CUDA    | Supported | `cudaEvent` GPU timestamps |
-| Vulkan  | Supported | GPU timestamp queries |
-| BLAS    | Supported | Wall-clock |
-| Metal   | Not yet supported | — |
-| OpenCL  | Not yet supported | — |
-
-The scheduler also profiles **data copies** (H2D, D2H, D2D) between backends regardless of which backends have native profiler support.
-
-## Enabling the Profiler
-
-There are two independent ways to enable profiling. They can be used separately or together.
-
-### CLI flags (`--profile`, `--profile-output`)
-
-Available in `llama-cli`, `llama-completion`, `llama-server`, and `debug`:
-
-```bash
-# Print summary to stdout
-llama-completion -m model.gguf --profile -p "Hello world"
-
-# Export to JSON
-llama-completion -m model.gguf --profile --profile-output profile.json -p "Hello world"
-
-# Export to plain text
-llama-completion -m model.gguf --profile --profile-output profile.txt -p "Hello world"
-```
-
-The output format is chosen by file extension: `.json` for JSON, `.txt` for plain text. Any other extension defaults to JSON.
-
-### Environment variable (`GGML_PROFILE`)
-
-The `GGML_PROFILE` environment variable enables profiling at the ggml scheduler level. This works with **any** application that uses the scheduler — including third-party tools like `sd.cpp` — without CLI flag support.
-
-```bash
-# Print summary to stdout
-GGML_PROFILE=1 llama-completion -m model.gguf -p "Hello world"
-
-# Export JSON
-GGML_PROFILE=profile.json llama-completion -m model.gguf -p "Hello world"
-
-# Export plain text
-GGML_PROFILE=profile.txt llama-completion -m model.gguf -p "Hello world"
-
-# Works with any ggml-based application
-GGML_PROFILE=1 sd -m model.gguf -p "a cat"
-```
-
-| Value | Behavior |
-|-------|----------|
-| `1`, `stdout`, or empty | Print summary to stdout |
-| `path.json` | Export JSON to file |
-| `path.txt` | Export plain text to file |
-| Any other path | Export JSON to file |
-
-The export happens automatically when the scheduler is freed (typically at program exit).
-
-## Output Formats
-
-### Console summary (stdout)
-
-The default when `--profile` is used without `--profile-output`, or `GGML_PROFILE=1`:
-
-```
-=== Profiling Summary ===
-  [OP  ] backend 0 MUL_MAT                    45.2%  count=1200  total=  120.50 ms  avg=  100.42 us  ...  12.30 GB/s  [4096 x 4096]
-  [OP  ] backend 1 MUL_MAT_ID                 30.1%  count= 600  total=   80.20 ms  avg=  133.67 us  ...   0.08 GB/s  [2688 x 1856 x 128]
-  [COPY] backend 0 copy_H2D                    5.3%  count= 200  total=   14.10 ms  avg=   70.50 us  ...   2.50 GB/s
-  ...
-```
-
-Each line shows: event type (OP or COPY), backend index, operation name, percentage of total time, call count, timing stats, bandwidth, and representative tensor shape.
-
-### Plain text (`.txt`)
-
-A more detailed report with three sections:
-
-1. **Profiling Summary** — total time, record count, unique ops
-2. **Per-Backend Summary** — ops and copies per backend with aggregate bandwidth
-3. **Operations table** — full breakdown with bandwidth and tensor shapes for all source tensors
-
-### JSON (`.json`)
-
-Machine-readable format suitable for the Python analysis tool. Contains:
-
-- `version`: Format version (currently `2`)
-- `backends[]`: Backend metadata (name, device, device type)
-- `records[]`: Every profiling event with:
-  - `type`: `0` = OP, `1` = COPY
-  - `name`: Operation name (e.g. `"MUL_MAT"`, `"copy_H2D"`)
-  - `backend_id`, `split_id`: Scheduler indices
-  - `start_ns`, `duration_ns`: Timing in nanoseconds
-  - `bytes`: Output tensor size (OPs) or transfer size (COPYs)
-  - `extra`: Fusion name for fused ops, or `null`
-  - `ne_src0`, `ne_src1`, `ne_src2`: Source tensor dimensions (4-element arrays)
-
-`ne_src2` is populated only for `MUL_MAT_ID` (expert selection indices); it is `[0,0,0,0]` for all other ops.
-
-## Python Analysis Tool
-
-The `tools/profiler/profiler.py` script reads JSON exports and produces analysis reports and visualizations.
-
-### Basic usage
-
-```bash
-# Print summary
-python -m tools.profiler.profiler profile.json
-
-# Show top 10 operations by time
-python -m tools.profiler.profiler profile.json --top-ops 10
-
-# Show top 10 longest individual kernels
-python -m tools.profiler.profiler profile.json --top-kernels 10
-
-# Show inefficiency ranking (highest time-per-byte)
-python -m tools.profiler.profiler profile.json --inefficiency
-```
-
-### Export visualizations
-
-```bash
-# Interactive HTML timeline (self-contained, no dependencies)
-python -m tools.profiler.profiler profile.json --html-viewer timeline.html
-
-# Chrome Trace format (open in chrome://tracing or Perfetto)
-python -m tools.profiler.profiler profile.json --chrome-trace trace.json
-
-# Downsample large traces for the HTML viewer
-python -m tools.profiler.profiler profile.json --html-viewer timeline.html --html-max-records 50000
-```
-
-Multiple exports can be combined in a single invocation:
-
-```bash
-python -m tools.profiler.profiler profile.json --html-viewer timeline.html --chrome-trace trace.json --top-ops 20
-```
-
-### CLI reference
-
-| Argument | Description |
-|----------|-------------|
-| `profile` (positional) | Path to profiler JSON file |
-| `--chrome-trace FILE` | Export Chrome Trace Event format |
-| `--html-viewer FILE` | Export interactive HTML timeline |
-| `--html-max-records N` | Limit records in HTML output (0 = unlimited) |
-| `--top-ops N` | Show top N operations by total time |
-| `--top-kernels N` | Show top N longest individual kernels |
-| `--inefficiency` | Rank operations by time per byte (higher = worse) |
-
-### HTML viewer features
-
-The HTML viewer is a self-contained file with no external dependencies:
-
-- **Canvas timeline** with per-backend lanes and color-coded operations
-- **Zoom controls** (1s / 100ms / 1ms / 100us) and mouse drag navigation
-- **Minimap** showing the full trace with a viewport indicator
-- **Hover tooltips** with operation name, duration, shape, and bytes
-- **Stats table** with collapsible tree: Operation → Backend → Tensor shape, showing % time, count, avg/min/max, and bandwidth
-- **Legend** showing the most frequent operation types
-
-## What Gets Measured
-
-### OP events
-
-Every tensor operation (MUL_MAT, ADD, UNARY, FLASH_ATTN_EXT, etc.) is recorded with:
-
-- **Timing**: Start/end timestamps (nanosecond precision)
-- **Bytes**: Output tensor size (`ggml_nbytes(node)`)
-- **Tensor shapes**: Dimensions of `src[0]`, `src[1]`, and `src[2]` (when applicable)
-- **Bandwidth**: Computed as `bytes / duration` — useful for identifying memory-bound vs compute-bound operations
-
-### COPY events
-
-Data transfers between backends:
-
-- **Direction**: `copy_H2D` (host→device), `copy_D2H` (device→host), `copy_D2D` (device→device)
-- **Bytes**: Exact transfer size
-- **Bandwidth**: Transfer throughput
-
-### MoE weight copies
-
-When `--cpu-moe` is used, the scheduler selectively copies only the active experts. These partial copies are recorded as individual COPY events with the actual bytes transferred.
-
-## Programmatic API
-
-For custom applications, the profiler can be controlled through the C API defined in `ggml/include/ggml-profiler.h`:
-
-```c
-// Enable profiling on a scheduler
-ggml_backend_sched_set_profiling(sched, true);
-
-// ... run inference ...
-
-// Get raw records
-const ggml_profile_record * records;
-int n = ggml_backend_sched_get_profiling_records(sched, &records);
-
-// Or export directly
-ggml_backend_sched_print_profiling(sched);                         // stdout
-ggml_backend_sched_export_profiling_json(sched, "profile.json");   // JSON file
-ggml_backend_sched_export_profiling_text(sched, "profile.txt");    // text file
-ggml_backend_sched_write_profiling_json(sched, fp);                // JSON to FILE*
-ggml_backend_sched_write_profiling_text(sched, fp);                // text to FILE*
-
-// Reset for next measurement window
-ggml_backend_sched_reset_profiling(sched);
-```
-
-Records accumulate across multiple `graph_compute` calls until explicitly reset or the scheduler is freed.
-
-## Tips
-
-- **Prompt eval vs generation**: The profiler captures all graph computes. During prompt evaluation you'll see larger batch sizes in tensor shapes; during generation, batch size is typically 1-2.
-- **Vulkan concurrent mode**: When Vulkan dispatches multiple operations concurrently, they are reported as a single combined record spanning the full GPU time interval.
-- **Bandwidth interpretation**: For compute ops, bandwidth = `output_bytes / duration`. This is not memory bandwidth — it's a proxy for throughput. MUL_MAT with low bandwidth typically indicates compute-bound behavior; high bandwidth indicates memory-bound.
-- **Large traces**: For long inference runs, the JSON can be large. Use `--html-max-records` to downsample the HTML viewer, or use Chrome Trace format which handles large files well.
-- **Multiple backends**: Backend IDs in the output correspond to the scheduler's priority order (0 = highest priority, typically GPU; last = CPU).

docs/development/HOWTO-add-model.md

@@ -45,6 +45,8 @@ class MyModel(MmprojModel):
 
 Add an enum entry in `MODEL_ARCH`, the model human friendly name in `MODEL_ARCH_NAMES` and the GGUF tensor names in `MODEL_TENSORS`.
 
+NOTE: Pick the GGUF arch string (and the matching `src/models/<name>.cpp` filename, see section 3) carefully up front, following existing naming conventions. Once GGUF files are published under a given arch string, renaming it later breaks the community's existing files, so this is not something to leave for cleanup in a follow-up PR.
+
 Example for `falcon` model:
 ```python
     MODEL_ARCH.FALCON: [
@@ -101,6 +103,7 @@ The model params and tensors layout must be defined in `llama.cpp` source files:
     - You may also need to update `LLM_KV_NAMES`, `LLM_TENSOR_NAMES` and `LLM_TENSOR_INFOS`
 3. Add any non-standard metadata loading in the `llama_model_loader` constructor in `src/llama-model-loader.cpp`.
 4. If the model has a RoPE operation, add a case for the architecture in `llama_model_rope_type` function in `src/llama-model.cpp`.
+5. Check for other places that switch/iterate over every `llm_arch` value, e.g. `src/llama-model-saver.cpp` and any mandatory-hparam lists (such as which archs require MoE metadata). Grep for `LLM_ARCH_` usages to find them. Missing one of these is a common cause of CI test failures (e.g. `test-llama-archs`) after adding a new arch.
 
 NOTE: The dimensions in `ggml` are typically in the reverse order of the `pytorch` dimensions.
 
@@ -133,6 +136,16 @@ Note:
 
 ## Tips and tricks
 
+### Prefer conversion-time tensor modifications over graph-time ones
+
+If the model contains constant modifications of tensors in the graph (for example, `norm(1 + weight)`) or performs tensor permutations/chunking, perform the modifications during conversion rather than in the graph code. This keeps the inference graph simpler and avoids extra runtime ops.
+
+Examples:
+- Gemma 3 folds the `1 +` of its `norm(1 + weight)` normalization into the weights at conversion time, so the graph just does a plain RMS norm.
+- Qwen3-Next applies its tensor permutation during conversion (in `modify_tensors`), so the graph can consume the already-permuted weights directly.
+
+Exception: a plain `weight * scale` with a constant scale is usually better left to inference time rather than folded into the weight at conversion. The scale conceptually applies to the activation, not the weight, so folding it into the weight can hurt numerical stability, and it shifts the weight's value range in a way that can make quantization worse. In this case, write the scale to GGUF as its own metadata key (e.g. `%s.attention.output_scale`, `%s.attention.value_scale`, `%s.embedding_scale`) and apply it in the graph, instead of pre-multiplying the weight tensor during conversion.
+
 ### Working with ggml_rope_ext
 
 PyTorch implementations usually prefer explicitly calculating `freq_cis`/`sin`/`cos` components. However, in llama.cpp, most RoPE operations can be handled via `ggml_rope_ext`, which does not require a sin/cos matrix. This saves memory while allowing the GGML RoPE kernel to be fused with other ops.

examples/debug/debug.cpp

@@ -252,29 +252,6 @@ int main(int argc, char ** argv) {
         return 1;
     }
 
-    // Export profiling data if profiling was enabled
-    if (params.profiling) {
-        ggml_backend_sched_t sched = llama_context_get_sched(ctx);
-        if (sched != nullptr) {
-            if (params.profiling_output.empty()) {
-                ggml_backend_sched_print_profiling(sched);
-            } else {
-                const std::string & path = params.profiling_output;
-                int ret;
-                if (path.size() >= 4 && path.compare(path.size() - 4, 4, ".txt") == 0) {
-                    ret = ggml_backend_sched_export_profiling_text(sched, path.c_str());
-                } else {
-                    ret = ggml_backend_sched_export_profiling_json(sched, path.c_str());
-                }
-                if (ret == 0) {
-                    LOG("\nProfiling data exported to: %s\n", path.c_str());
-                } else {
-                    LOG_ERR("\nFailed to export profiling data to: %s\n", path.c_str());
-                }
-            }
-        }
-    }
-
     LOG("\n");
     llama_perf_context_print(ctx);
 

ggml/CMakeLists.txt

@@ -216,7 +216,6 @@ option(GGML_HIP                             "ggml: use HIP"
 option(GGML_HIP_GRAPHS                      "ggml: use HIP graph"                              ON)
 option(GGML_HIP_RCCL                        "ggml: use ROCm Collective Comm. Library"         OFF)
 option(GGML_HIP_NO_VMM                      "ggml: do not try to use HIP VMM"                 ON)
-option(GGML_HIP_ROCWMMA_FATTN               "ggml: enable rocWMMA for FlashAttention"         OFF)
 option(GGML_HIP_MMQ_MFMA                    "ggml: enable MFMA MMA for CDNA in MMQ"           ON)
 option(GGML_HIP_EXPORT_METRICS              "ggml: enable kernel perf metrics output"         OFF)
 option(GGML_MUSA_GRAPHS                     "ggml: use MUSA graph, experimental, unstable"    OFF)

ggml/include/ggml-cpu.h

@@ -22,24 +22,6 @@ extern "C" {
 
         // use only reference implementations
         bool use_ref;
-
-        // profiler context (set by backend when profiling is enabled, NULL otherwise)
-        // when non-NULL, the compute loop will record per-node timing
-        void * profiling_context;
-
-        // callback for recording a profile record from C code (set by backend when profiling)
-        // The callback receives the full tensor node so it can extract all sources, types,
-        // op_params, and sub-op information directly.
-        // params: context, type, name, split_id, start_ns, end_ns, bytes, extra, node
-        void (*profiling_record_fn)(void *                     context,
-                                    int                        type,
-                                    const char *               name,
-                                    int                        split_id,
-                                    uint64_t                   start_ns,
-                                    uint64_t                   end_ns,
-                                    uint64_t                   bytes,
-                                    const char *               extra,
-                                    const struct ggml_tensor * node);
     };
 
     // numa strategies

ggml/include/ggml-profiler.h

@@ -1,134 +0,0 @@
-#pragma once
-
-#include "ggml-backend.h"
-#include "ggml.h"
-
-#include <stdint.h>
-
-#ifdef __cplusplus
-extern "C" {
-#endif
-
-//
-// Profiler
-//
-
-// Profile event types
-enum ggml_profile_event_type {
-    GGML_PROFILE_EVENT_OP,    // single operation execution (computation kernel)
-    GGML_PROFILE_EVENT_COPY,  // data transfer between devices
-};
-
-// A single profiling record representing a timed interval
-typedef struct ggml_profile_record {
-    enum ggml_profile_event_type type;
-    const char *                 name;        // operation name (e.g., "mul_mat", "copy_H2D")
-    int                          backend_id;  // scheduler's backend index (0 = highest priority)
-    int                          split_id;    // which graph split (0..n_splits-1)
-    uint64_t                     start_ns;    // start timestamp in nanoseconds
-    uint64_t                     end_ns;      // end timestamp in nanoseconds
-    uint64_t                     bytes;       // bytes transferred (for copy) or tensor size (for ops)
-    const char *                 extra;       // fusion name for fused ops, or NULL
-
-    // Output tensor info
-    char                         tensor_name[GGML_MAX_NAME];           // output tensor name (e.g. "ffn_out-0"), "" if unnamed
-    int64_t                      ne[4];                                // output tensor dimensions
-    int                          out_type;                             // output tensor type (ggml_type), -1 if N/A
-
-    // Source tensors (up to GGML_MAX_SRC).  n_src is the actual number populated.
-    int                          n_src;
-    int64_t                      ne_src[GGML_MAX_SRC][4];              // per-source dimensions
-    int64_t                      nb_src[GGML_MAX_SRC][4];              // per-source strides (bytes)
-    int                          type_src[GGML_MAX_SRC];               // per-source ggml_type, -1 if not present
-
-    // Operation parameters (raw bytes copied from ggml_tensor::op_params)
-    int32_t                      op_params[GGML_MAX_OP_PARAMS / sizeof(int32_t)];
-
-    int                          sub_op;      // sub-operation (ggml_unary_op or ggml_glu_op), -1 if N/A
-} ggml_profile_record;
-
-// Backend profiler interface - each backend optionally implements this
-// to provide fine-grained operation timing
-struct ggml_backend_profiler {
-    void * context;  // backend-specific profiler context
-
-    // Enable or disable profiling on this backend
-    void (*enable)(void * context, bool enable);
-
-    // Clear all recorded data
-    void (*reset)(void * context);
-
-    // Set the current split ID (called by scheduler before graph_compute)
-    void (*set_split_id)(void * context, int split_id);
-
-    // Get recorded profiling data
-    // Returns the number of records; sets *out to point to internal storage
-    // The returned pointer remains valid until the next reset or disable call
-    int (*get_records)(void * context, const ggml_profile_record ** out);
-
-    // Free the profiler context
-    void (*free_context)(void * context);
-};
-
-typedef struct ggml_backend_profiler * ggml_backend_profiler_t;
-
-// Populate the per-node fields of a ggml_profile_record from a ggml_tensor node:
-//   ne, out_type, n_src, ne_src, nb_src, type_src, op_params, sub_op.
-// All other fields (type/name/backend_id/split_id/timestamps/bytes/extra) must
-// be filled in separately by the backend that records the event.
-GGML_API void ggml_profile_record_from_tensor(struct ggml_profile_record * rec,
-                                              const struct ggml_tensor *   node);
-
-// Register a profiler on a backend (called by backend during init)
-// The profiler is owned by the backend and will be freed when the backend is freed
-GGML_API void ggml_backend_set_profiler(ggml_backend_t backend, ggml_backend_profiler_t profiler);
-
-// Get the profiler associated with a backend (returns NULL if none)
-GGML_API ggml_backend_profiler_t ggml_backend_get_profiler(ggml_backend_t backend);
-
-//
-// Scheduler profiling API
-//
-
-// Enable or disable profiling on a scheduler
-// When enabled, the scheduler will:
-//   - Time data copy operations between backends
-//   - Enable profiling on all backends that support it
-//   - Collect profiling records from all backends after each graph compute
-GGML_API void ggml_backend_sched_set_profiling(ggml_backend_sched_t sched, bool enable);
-
-// Check if profiling is enabled on a scheduler
-GGML_API bool ggml_backend_sched_get_profiling(ggml_backend_sched_t sched);
-
-// Get profiling data from the last graph compute
-// Records are owned by the scheduler; valid until the next compute or reset
-// Returns the number of records
-GGML_API int ggml_backend_sched_get_profiling_records(ggml_backend_sched_t sched, const ggml_profile_record ** records);
-
-// Print a human-readable summary of the last profiling run to stdout
-// Groups records by operation name and shows total/count/min/max/avg time
-GGML_API void ggml_backend_sched_print_profiling(ggml_backend_sched_t sched);
-
-// Reset profiling data (clear all recorded data)
-GGML_API void ggml_backend_sched_reset_profiling(ggml_backend_sched_t sched);
-
-// Get current time in nanoseconds (for manual profiling if needed)
-GGML_API uint64_t ggml_profiler_time_ns(void);
-
-// Export profiling data as JSON to a file
-// Returns 0 on success, -1 on error
-GGML_API int ggml_backend_sched_export_profiling_json(ggml_backend_sched_t sched, const char * filepath);
-
-// Export profiling data as JSON to a FILE pointer
-GGML_API int ggml_backend_sched_write_profiling_json(ggml_backend_sched_t sched, FILE * fp);
-
-// Export profiling data as plain text statistics to a file
-// Returns 0 on success, -1 on error
-GGML_API int ggml_backend_sched_export_profiling_text(ggml_backend_sched_t sched, const char * filepath);
-
-// Export profiling data as plain text statistics to a FILE pointer
-GGML_API int ggml_backend_sched_write_profiling_text(ggml_backend_sched_t sched, FILE * fp);
-
-#ifdef __cplusplus
-}
-#endif

ggml/src/CMakeLists.txt

@@ -195,7 +195,6 @@ add_library(ggml-base
             ../include/ggml-backend.h
             ../include/ggml-cpp.h
             ../include/ggml-opt.h
-            ../include/ggml-profiler.h
             ../include/gguf.h
             ggml.c
             ggml.cpp
@@ -203,7 +202,6 @@ add_library(ggml-base
             ggml-backend.cpp
             ggml-backend-meta.cpp
             ggml-opt.cpp
-            ggml-profiler.cpp
             ggml-threading.cpp
             ggml-threading.h
             ggml-quants.c

ggml/src/ggml-backend-impl.h

@@ -3,7 +3,6 @@
 // ggml-backend internal header
 
 #include "ggml-backend.h"
-#include "ggml-profiler.h"
 
 #ifdef  __cplusplus
 extern "C" {
@@ -145,9 +144,6 @@ extern "C" {
         struct ggml_backend_i iface;
         ggml_backend_dev_t device;
         void * context;
-
-        // Optional profiler (set by backend during init, NULL if not profiling)
-        ggml_backend_profiler_t profiler;
     };
 
     struct ggml_backend_event {

ggml/src/ggml-backend.cpp

@@ -12,7 +12,6 @@
 #include "ggml-backend-impl.h"
 #include "ggml-alloc.h"
 #include "ggml-impl.h"
-#include "ggml-profiler.h"
 
 #include <assert.h>
 #include <limits.h>
@@ -21,7 +20,6 @@
 #include <stdlib.h>
 #include <string.h>
 #include <algorithm>
-#include <string>
 #include <vector>
 
 #ifdef __APPLE__
@@ -233,15 +231,6 @@ void ggml_backend_free(ggml_backend_t backend) {
         return;
     }
 
-    // Clean up profiler if present (before backend frees its context)
-    if (backend->profiler != NULL) {
-        if (backend->profiler->free_context != NULL) {
-            backend->profiler->free_context(backend->profiler->context);
-        }
-        delete backend->profiler;
-        backend->profiler = NULL;
-    }
-
     backend->iface.free(backend);
 }
 
@@ -836,20 +825,6 @@ struct ggml_backend_sched {
     int debug_realloc;
     int debug_graph_size;
     int debug_prev_graph_size;
-
-    // profiling
-    bool                             profiling_enabled;
-    std::string                      profiling_env_path;  // GGML_PROFILE env var value (for auto-export on free)
-    std::vector<ggml_profile_record> copy_records;        // copy events recorded by the scheduler
-    std::vector<ggml_profile_record> profiling_records;   // merged records from all sources
-
-    // Cached backend metadata for safe access during auto-export (backends may be freed first)
-    struct backend_meta {
-        std::string name;
-        std::string device;
-        int         device_type;
-    };
-    std::vector<backend_meta>        profiling_backend_meta;
 };
 
 #define hash_id(tensor) ggml_hash_find_or_insert(&sched->hash_set, tensor)
@@ -931,26 +906,35 @@ static int ggml_backend_sched_backend_id_from_cur(ggml_backend_sched_t sched, st
     }
 
     // operations with weights are preferably run on the same backend as the weights
-    for (int i = 0; i < GGML_MAX_SRC; i++) {
-        const struct ggml_tensor * src = tensor->src[i];
-        if (src == NULL) {
-            continue;
-        }
-        // skip ROPE since the rope freqs tensor is too small to choose a backend based on it
-        // not an ideal solution
-        if (tensor->op != GGML_OP_ROPE && src->buffer != NULL && src->buffer->usage == GGML_BACKEND_BUFFER_USAGE_WEIGHTS) {
-            int src_backend_id = ggml_backend_sched_backend_from_buffer(sched, src, tensor);
-            // check if a backend with higher prio wants to offload the op
-            if (sched->op_offload && src_backend_id == sched->n_backends - 1 && ggml_backend_buffer_is_host(src->buffer)) {
-                for (int b = 0; b < src_backend_id; b++) {
-                    if (ggml_backend_supports_op(sched->backends[b], tensor) && ggml_backend_offload_op(sched->backends[b], tensor)) {
-                        SET_CAUSE(tensor, "1.off");
-                        return b;
+    // TODO: there are exceptions (see below) - not an ideal solution
+    bool allow = true;
+
+    // skip ROPE since the rope freqs tensor is too small to choose a backend based on it
+    allow = allow && tensor->op != GGML_OP_ROPE;
+
+    // skip FLASH_ATTN_EXT since the sinks tensor is too small to choose a based based on it
+    allow = allow && tensor->op != GGML_OP_FLASH_ATTN_EXT;
+
+    if (allow) {
+        for (int i = 0; i < GGML_MAX_SRC; i++) {
+            const struct ggml_tensor * src = tensor->src[i];
+            if (src == NULL) {
+                continue;
+            }
+            if (src->buffer != NULL && src->buffer->usage == GGML_BACKEND_BUFFER_USAGE_WEIGHTS) {
+                int src_backend_id = ggml_backend_sched_backend_from_buffer(sched, src, tensor);
+                // check if a backend with higher prio wants to offload the op
+                if (sched->op_offload && src_backend_id == sched->n_backends - 1 && ggml_backend_buffer_is_host(src->buffer)) {
+                    for (int b = 0; b < src_backend_id; b++) {
+                        if (ggml_backend_supports_op(sched->backends[b], tensor) && ggml_backend_offload_op(sched->backends[b], tensor)) {
+                            SET_CAUSE(tensor, "1.off");
+                            return b;
+                        }
                     }
                 }
+                SET_CAUSE(tensor, "1.wgt%d", i);
+                return src_backend_id;
             }
-            SET_CAUSE(tensor, "1.wgt%d", i);
-            return src_backend_id;
         }
     }
 
@@ -1563,39 +1547,6 @@ static bool ggml_backend_sched_alloc_splits(ggml_backend_sched_t sched) {
     return true;
 }
 
-// Build a COPY profiling record.  Copies have no real ggml_tensor "node" backing
-// them, so we synthesize one source describing the input tensor that was moved.
-static ggml_profile_record make_copy_record(const char * copy_dir, int backend_id, int split_id,
-                                            uint64_t start_ns, uint64_t end_ns, uint64_t bytes,
-                                            const struct ggml_tensor * input) {
-    ggml_profile_record rec = {};
-    rec.type       = GGML_PROFILE_EVENT_COPY;
-    rec.name       = copy_dir;
-    rec.backend_id = backend_id;
-    rec.split_id   = split_id;
-    rec.start_ns   = start_ns;
-    rec.end_ns     = end_ns;
-    rec.bytes      = bytes;
-    rec.extra      = input ? input->name : NULL;
-    snprintf(rec.tensor_name, sizeof(rec.tensor_name), "%s", input ? input->name : "");
-    rec.out_type   = -1;
-    rec.sub_op     = -1;
-    rec.n_src      = 0;
-    if (input != NULL) {
-        // Describe the input tensor as src[0] so consumers can inspect its shape.
-        rec.n_src = 1;
-        memcpy(rec.ne_src[0], input->ne, sizeof(rec.ne_src[0]));
-        for (int d = 0; d < 4; d++) {
-            rec.nb_src[0][d] = (int64_t) input->nb[d];
-        }
-        rec.type_src[0] = (int) input->type;
-    }
-    for (int i = rec.n_src; i < GGML_MAX_SRC; i++) {
-        rec.type_src[i] = -1;
-    }
-    return rec;
-}
-
 static enum ggml_status ggml_backend_sched_compute_splits(ggml_backend_sched_t sched) {
     GGML_ASSERT(sched);
     struct ggml_backend_sched_split * splits = sched->splits;
@@ -1604,28 +1555,11 @@ static enum ggml_status ggml_backend_sched_compute_splits(ggml_backend_sched_t s
     std::vector<int32_t> ids;
     std::vector<ggml_bitset_t> used_ids;
 
-    // Profiling: reset copy records for this compute pass
-    if (sched->profiling_enabled) {
-        sched->copy_records.clear();
-    }
-
     for (int split_id = 0; split_id < sched->n_splits; split_id++) {
         struct ggml_backend_sched_split * split = &splits[split_id];
         int split_backend_id = split->backend_id;
         ggml_backend_t split_backend = sched->backends[split_backend_id];
 
-        // Profiling: set split ID and enable backend profiling
-        if (sched->profiling_enabled) {
-            if (split_backend->profiler != NULL) {
-                if (split_backend->profiler->enable != NULL) {
-                    split_backend->profiler->enable(split_backend->profiler->context, true);
-                }
-                if (split_backend->profiler->set_split_id != NULL) {
-                    split_backend->profiler->set_split_id(split_backend->profiler->context, split_id);
-                }
-            }
-        }
-
         // copy the input tensors to the split backend
         for (int input_id = 0; input_id < split->n_inputs; input_id++) {
             ggml_backend_t input_backend = ggml_backend_sched_get_tensor_backend(sched, split->inputs[input_id]);
@@ -1639,25 +1573,7 @@ static enum ggml_status ggml_backend_sched_compute_splits(ggml_backend_sched_t s
                 } else {
                     ggml_backend_synchronize(split_backend);
                 }
-                if (sched->profiling_enabled) {
-                    uint64_t copy_start = ggml_profiler_time_ns();
-                    ggml_backend_tensor_copy(input, input_cpy);
-                    uint64_t copy_end = ggml_profiler_time_ns();
-
-                    enum ggml_backend_dev_type src_type = ggml_backend_dev_type(input_backend->device);
-                    enum ggml_backend_dev_type dst_type = ggml_backend_dev_type(split_backend->device);
-                    const char *               copy_dir = "copy_D2D";
-                    if (src_type == GGML_BACKEND_DEVICE_TYPE_CPU && dst_type != GGML_BACKEND_DEVICE_TYPE_CPU) {
-                        copy_dir = "copy_H2D";
-                    } else if (src_type != GGML_BACKEND_DEVICE_TYPE_CPU && dst_type == GGML_BACKEND_DEVICE_TYPE_CPU) {
-                        copy_dir = "copy_D2H";
-                    }
-
-                    sched->copy_records.push_back(make_copy_record(copy_dir, split_backend_id, split_id,
-                                                                   copy_start, copy_end, ggml_nbytes(input), input));
-                } else {
-                    ggml_backend_tensor_copy(input, input_cpy);
-                }
+                ggml_backend_tensor_copy(input, input_cpy);
             } else {
                 // wait for the split backend to finish using the input before overwriting it
                 if (sched->events[split_backend_id][sched->cur_copy] != NULL) {
@@ -1714,14 +1630,12 @@ static enum ggml_status ggml_backend_sched_compute_splits(ggml_backend_sched_t s
                     }
 
                     // group consecutive experts and copy them together
-                    size_t total_copied_bytes = 0;
                     auto copy_experts = [&](int32_t first_id, int32_t last_id) {
                         const size_t expert_offset = first_id * expert_size;
                         const size_t expert_size_copy =  (last_id - first_id + 1) * expert_size;
                         const size_t padding = std::min<size_t>(expert_size, 512);
                         const size_t padding_end = last_id < n_expert - 1 ? padding : 0;
 
-                        total_copied_bytes += expert_size_copy + padding_end;
                         ggml_backend_tensor_set_async(split_backend,
                             input_cpy,
                             (const uint8_t *)input->data + expert_offset, expert_offset,
@@ -1730,11 +1644,6 @@ static enum ggml_status ggml_backend_sched_compute_splits(ggml_backend_sched_t s
                             expert_size_copy + padding_end);
                     };
 
-                    uint64_t moe_copy_start = 0;
-                    if (sched->profiling_enabled) {
-                        moe_copy_start = ggml_profiler_time_ns();
-                    }
-
                     int id = 0;
                     while (!ggml_bitset_get(used_ids.data(), id)) {
                         id++;
@@ -1758,34 +1667,9 @@ static enum ggml_status ggml_backend_sched_compute_splits(ggml_backend_sched_t s
                         last_id = id;
                     }
                     copy_experts(first_id, last_id);
-
-                    if (sched->profiling_enabled) {
-                        uint64_t moe_copy_end = ggml_profiler_time_ns();
-
-                        enum ggml_backend_dev_type src_type = ggml_backend_dev_type(input_backend->device);
-                        enum ggml_backend_dev_type dst_type = ggml_backend_dev_type(split_backend->device);
-                        const char *               copy_dir = "copy_D2D";
-                        if (src_type == GGML_BACKEND_DEVICE_TYPE_CPU && dst_type != GGML_BACKEND_DEVICE_TYPE_CPU) {
-                            copy_dir = "copy_H2D";
-                        } else if (src_type != GGML_BACKEND_DEVICE_TYPE_CPU &&
-                                   dst_type == GGML_BACKEND_DEVICE_TYPE_CPU) {
-                            copy_dir = "copy_D2H";
-                        }
-
-                        sched->copy_records.push_back(make_copy_record(copy_dir, split_backend_id, split_id,
-                                                                       moe_copy_start, moe_copy_end,
-                                                                       (uint64_t) total_copied_bytes, input));
-                    }
                 } else {
                     // try async copy, but if not possible, we can still use a sync copy without synchronizing the dst backend, since we handle the synchronization here with multiple copies and events
                     // TODO: add public function to facilitate this, since applications do not have direct access to the backend interface
-
-                    // Capture timestamp before async attempt so we can record launch time
-                    uint64_t copy_start = 0;
-                    if (sched->profiling_enabled) {
-                        copy_start = ggml_profiler_time_ns();
-                    }
-
                     if (!split_backend->iface.cpy_tensor_async || !split_backend->iface.cpy_tensor_async(input_backend, split_backend, input, input_cpy)) {
                         ggml_backend_synchronize(input_backend);
                         if (sched->events[split_backend_id][sched->cur_copy] != NULL) {
@@ -1793,45 +1677,7 @@ static enum ggml_status ggml_backend_sched_compute_splits(ggml_backend_sched_t s
                         } else {
                             ggml_backend_synchronize(split_backend);
                         }
-                        if (sched->profiling_enabled) {
-                            // Re-take start after sync for accurate sync copy measurement
-                            copy_start = ggml_profiler_time_ns();
-                            ggml_backend_tensor_copy(input, input_cpy);
-                            uint64_t copy_end = ggml_profiler_time_ns();
-
-                            enum ggml_backend_dev_type src_type = ggml_backend_dev_type(input_backend->device);
-                            enum ggml_backend_dev_type dst_type = ggml_backend_dev_type(split_backend->device);
-                            const char *               copy_dir = "copy_D2D";
-                            if (src_type == GGML_BACKEND_DEVICE_TYPE_CPU && dst_type != GGML_BACKEND_DEVICE_TYPE_CPU) {
-                                copy_dir = "copy_H2D";
-                            } else if (src_type != GGML_BACKEND_DEVICE_TYPE_CPU &&
-                                       dst_type == GGML_BACKEND_DEVICE_TYPE_CPU) {
-                                copy_dir = "copy_D2H";
-                            }
-
-                            sched->copy_records.push_back(make_copy_record(copy_dir, split_backend_id, split_id,
-                                                                           copy_start, copy_end, ggml_nbytes(input), input));
-                        } else {
-                            ggml_backend_tensor_copy(input, input_cpy);
-                        }
-                    } else {
-                        // async copy was launched — record the time spanning the async call
-                        if (sched->profiling_enabled) {
-                            uint64_t copy_end = ggml_profiler_time_ns();
-
-                            enum ggml_backend_dev_type src_type = ggml_backend_dev_type(input_backend->device);
-                            enum ggml_backend_dev_type dst_type = ggml_backend_dev_type(split_backend->device);
-                            const char *               copy_dir = "copy_D2D";
-                            if (src_type == GGML_BACKEND_DEVICE_TYPE_CPU && dst_type != GGML_BACKEND_DEVICE_TYPE_CPU) {
-                                copy_dir = "copy_H2D";
-                            } else if (src_type != GGML_BACKEND_DEVICE_TYPE_CPU &&
-                                       dst_type == GGML_BACKEND_DEVICE_TYPE_CPU) {
-                                copy_dir = "copy_D2H";
-                            }
-
-                            sched->copy_records.push_back(make_copy_record(copy_dir, split_backend_id, split_id,
-                                                                           copy_start, copy_end, ggml_nbytes(input), input));
-                        }
+                        ggml_backend_tensor_copy(input, input_cpy);
                     }
                 }
             }
@@ -1884,32 +1730,6 @@ static enum ggml_status ggml_backend_sched_compute_splits(ggml_backend_sched_t s
         }
     }
 
-    // Profiling: collect records from all backends and append to accumulated records
-    if (sched->profiling_enabled) {
-        // Collect backend operation records
-        for (int b = 0; b < sched->n_backends; b++) {
-            ggml_backend_t backend = sched->backends[b];
-            if (backend->profiler != NULL && backend->profiler->get_records != NULL) {
-                const ggml_profile_record * backend_recs = NULL;
-                int count = backend->profiler->get_records(backend->profiler->context, &backend_recs);
-                for (int r = 0; r < count; r++) {
-                    ggml_profile_record rec = backend_recs[r];
-                    rec.backend_id          = b;  // stamp correct scheduler backend index
-                    sched->profiling_records.push_back(rec);
-                }
-                // Reset backend records (but keep profiling enabled for next compute)
-                if (backend->profiler->reset != NULL) {
-                    backend->profiler->reset(backend->profiler->context);
-                }
-            }
-        }
-
-        // Append copy records
-        for (const auto & rec : sched->copy_records) {
-            sched->profiling_records.push_back(rec);
-        }
-    }
-
     return GGML_STATUS_SUCCESS;
 }
 
@@ -1976,24 +1796,6 @@ ggml_backend_sched_t ggml_backend_sched_new(
 
     sched->galloc = ggml_gallocr_new_n(sched->bufts, n_backends);
     sched->op_offload = op_offload;
-    const char * profile_env = getenv("GGML_PROFILE");
-    if (profile_env != NULL) {
-        sched->profiling_enabled = true;
-        sched->profiling_env_path = profile_env;
-    }
-
-    // Cache backend metadata for safe access during auto-export
-    for (int b = 0; b < n_backends; b++) {
-        ggml_backend_sched::backend_meta meta;
-        meta.name        = ggml_backend_name(backends[b]);
-        meta.device      = "unknown";
-        meta.device_type  = 0;
-        if (backends[b]->device != NULL) {
-            meta.device      = ggml_backend_dev_name(backends[b]->device);
-            meta.device_type = (int) ggml_backend_dev_type(backends[b]->device);
-        }
-        sched->profiling_backend_meta.push_back(std::move(meta));
-    }
 
     ggml_backend_sched_reset(sched);
 
@@ -2004,33 +1806,6 @@ void ggml_backend_sched_free(ggml_backend_sched_t sched) {
     if (sched == NULL) {
         return;
     }
-
-    // Auto-export profiling data if enabled via GGML_PROFILE env var
-    // GGML_PROFILE=1 or GGML_PROFILE="" → print to stdout
-    // GGML_PROFILE=file.json → export JSON
-    // GGML_PROFILE=file.txt → export text
-    if (!sched->profiling_records.empty() && getenv("GGML_PROFILE") != NULL) {
-        const std::string & path = sched->profiling_env_path;
-        if (path.empty() || path == "1" || path == "stdout") {
-            ggml_backend_sched_print_profiling(sched);
-        } else if (path.size() >= 4 && path.compare(path.size() - 4, 4, ".txt") == 0) {
-            int ret = ggml_backend_sched_export_profiling_text(sched, path.c_str());
-            if (ret == 0) {
-                GGML_LOG_INFO("[profiler] Data exported to: %s\n", path.c_str());
-            } else {
-                GGML_LOG_ERROR("[profiler] Failed to export data to: %s\n", path.c_str());
-            }
-        } else {
-            // Default to JSON for any other path (including .json)
-            int ret = ggml_backend_sched_export_profiling_json(sched, path.c_str());
-            if (ret == 0) {
-                GGML_LOG_INFO("[profiler] Data exported to: %s\n", path.c_str());
-            } else {
-                GGML_LOG_ERROR("[profiler] Failed to export data to: %s\n", path.c_str());
-            }
-        }
-    }
-
     for (int b = 0; b < sched->n_backends; b++) {
         for (int c = 0; c < sched->n_copies; c++) {
             ggml_backend_event_free(sched->events[b][c]);
@@ -2603,509 +2378,3 @@ ggml_backend_buffer_t ggml_backend_cpu_buffer_from_ptr(void * ptr, size_t size)
     GGML_ASSERT((uintptr_t)ptr % TENSOR_ALIGNMENT == 0 && "buffer pointer must be aligned");
     return ggml_backend_buffer_init(ggml_backend_cpu_buffer_from_ptr_type(), ggml_backend_cpu_buffer_from_ptr_i, ptr, size);
 }
-
-//
-// Scheduler profiling
-//
-
-void ggml_backend_sched_set_profiling(ggml_backend_sched_t sched, bool enable) {
-    GGML_ASSERT(sched);
-    sched->profiling_enabled = enable;
-
-    if (!enable) {
-        ggml_backend_sched_reset_profiling(sched);
-    }
-}
-
-bool ggml_backend_sched_get_profiling(ggml_backend_sched_t sched) {
-    GGML_ASSERT(sched);
-    return sched->profiling_enabled;
-}
-
-int ggml_backend_sched_get_profiling_records(ggml_backend_sched_t sched, const ggml_profile_record ** records) {
-    GGML_ASSERT(sched);
-    GGML_ASSERT(records != NULL);
-
-    *records = sched->profiling_records.data();
-    return (int) sched->profiling_records.size();
-}
-
-void ggml_backend_sched_reset_profiling(ggml_backend_sched_t sched) {
-    GGML_ASSERT(sched);
-    sched->profiling_records.clear();
-    sched->copy_records.clear();
-}
-
-void ggml_backend_sched_print_profiling(ggml_backend_sched_t sched) {
-    GGML_ASSERT(sched);
-
-    if (sched->profiling_records.empty()) {
-        GGML_LOG_INFO("[profiler] No profiling data available\n");
-        return;
-    }
-
-    GGML_LOG_INFO("\n=== Profiling Summary ===\n");
-
-    // Aggregate by (name, type, backend_id)
-    struct op_stats {
-        const char *                 name;
-        enum ggml_profile_event_type type;
-        int                          backend_id;
-        uint64_t                     total_ns;
-        uint64_t                     min_ns;
-        uint64_t                     max_ns;
-        int                          count;
-        uint64_t                     total_bytes;
-        int64_t                      representative_ne[4];
-    };
-
-    std::vector<op_stats> stats;
-    for (const auto & rec : sched->profiling_records) {
-        bool found = false;
-        for (auto & s : stats) {
-            if (s.type == rec.type && s.backend_id == rec.backend_id && strcmp(s.name, rec.name) == 0) {
-                uint64_t dur = (rec.end_ns > rec.start_ns) ? (rec.end_ns - rec.start_ns) : 0;
-                s.total_ns += dur;
-                s.min_ns = std::min(s.min_ns, dur);
-                s.max_ns = std::max(s.max_ns, dur);
-                s.count++;
-                s.total_bytes += rec.bytes;
-                found = true;
-                break;
-            }
-        }
-        if (!found) {
-            uint64_t dur = (rec.end_ns > rec.start_ns) ? (rec.end_ns - rec.start_ns) : 0;
-            op_stats s;
-            s.name        = rec.name;
-            s.type        = rec.type;
-            s.backend_id  = rec.backend_id;
-            s.total_ns    = dur;
-            s.min_ns      = dur;
-            s.max_ns      = dur;
-            s.count       = 1;
-            s.total_bytes = rec.bytes;
-            memcpy(s.representative_ne, rec.ne_src[0], sizeof(s.representative_ne));
-            stats.push_back(s);
-        }
-    }
-
-    // Sort by total time descending
-    std::sort(stats.begin(), stats.end(),
-              [](const op_stats & a, const op_stats & b) { return a.total_ns > b.total_ns; });
-
-    uint64_t grand_total = 0;
-    for (const auto & s : stats) {
-        grand_total += s.total_ns;
-    }
-
-    const char * type_str[] = { "OP  ", "COPY" };
-    for (const auto & s : stats) {
-        double pct    = 100.0 * (double) s.total_ns / (double) grand_total;
-        double avg_us = (double) s.total_ns / (double) s.count / 1000.0;
-        double min_us = (double) s.min_ns / 1000.0;
-        double max_us = (double) s.max_ns / 1000.0;
-
-        GGML_LOG_INFO(
-            "  [%s] backend %d %-28s %7.1f%%  count=%-6d  total=%8.2f ms  avg=%8.2f us  min=%8.2f us  max=%8.2f us",
-            type_str[s.type], s.backend_id, s.name, pct, s.count, (double) s.total_ns / 1e6, avg_us, min_us, max_us);
-        if (s.total_bytes > 0 && s.total_ns > 0) {
-            double bw_gbps = (double) s.total_bytes / (double) s.total_ns;
-            if (bw_gbps >= 1000.0) {
-                GGML_LOG_INFO("  %6.2f TB/s", bw_gbps / 1000.0);
-            } else {
-                GGML_LOG_INFO("  %6.2f GB/s", bw_gbps);
-            }
-        }
-        // Print representative tensor shape (first record's ne)
-        if (s.representative_ne[0] > 0 || s.representative_ne[1] > 0) {
-            GGML_LOG_INFO("  [%lld x %lld", (long long) s.representative_ne[0], (long long) s.representative_ne[1]);
-            if (s.representative_ne[2] > 1) {
-                GGML_LOG_INFO(" x %lld", (long long) s.representative_ne[2]);
-            }
-            if (s.representative_ne[3] > 1) {
-                GGML_LOG_INFO(" x %lld", (long long) s.representative_ne[3]);
-            }
-            GGML_LOG_INFO("]");
-        }
-        GGML_LOG_INFO("\n");
-    }
-
-    GGML_LOG_INFO("  ---\n");
-    GGML_LOG_INFO("  Total: %.2f ms  (%d records, %d unique ops)\n\n", (double) grand_total / 1e6,
-                  (int) sched->profiling_records.size(), (int) stats.size());
-}
-
-int ggml_backend_sched_write_profiling_json(ggml_backend_sched_t sched, FILE * fp) {
-    GGML_ASSERT(sched);
-    GGML_ASSERT(fp != NULL);
-
-    uint64_t total_ns = 0;
-    for (const auto & rec : sched->profiling_records) {
-        total_ns += (rec.end_ns > rec.start_ns) ? (rec.end_ns - rec.start_ns) : 0;
-    }
-
-    fprintf(fp, "{\n");
-    fprintf(fp, "  \"version\": 3,\n");
-    fprintf(fp, "  \"profiler\": \"ggml\",\n");
-    fprintf(fp, "  \"total_records\": %d,\n", (int) sched->profiling_records.size());
-    fprintf(fp, "  \"total_ns\": %llu,\n", (unsigned long long) total_ns);
-
-    // Backend metadata (use cached data if available, fall back to live pointers)
-    fprintf(fp, "  \"backends\": [\n");
-    for (int b = 0; b < sched->n_backends; b++) {
-        const char * name     = "unknown";
-        const char * dev_name = "unknown";
-        int          dev_type = 0;
-        if (b < (int) sched->profiling_backend_meta.size()) {
-            name     = sched->profiling_backend_meta[b].name.c_str();
-            dev_name = sched->profiling_backend_meta[b].device.c_str();
-            dev_type = sched->profiling_backend_meta[b].device_type;
-        } else if (sched->backends[b] != NULL) {
-            name = ggml_backend_name(sched->backends[b]);
-            if (sched->backends[b]->device != NULL) {
-                dev_name = ggml_backend_dev_name(sched->backends[b]->device);
-                dev_type = (int) ggml_backend_dev_type(sched->backends[b]->device);
-            }
-        }
-        fprintf(fp, "    {\"id\": %d, \"name\": \"%s\", \"device\": \"%s\", \"device_type\": %d}%s\n", b, name,
-                dev_name, dev_type, (b < sched->n_backends - 1) ? "," : "");
-    }
-    fprintf(fp, "  ],\n");
-
-    // Records
-    fprintf(fp, "  \"records\": [\n");
-
-    for (int i = 0; i < (int) sched->profiling_records.size(); i++) {
-        const auto & rec         = sched->profiling_records[i];
-        uint64_t     duration_ns = (rec.end_ns > rec.start_ns) ? (rec.end_ns - rec.start_ns) : 0;
-
-        fprintf(fp,
-                "    {\"type\": %d, \"name\": \"%s\", \"backend_id\": %d, \"split_id\": %d, "
-                "\"start_ns\": %llu, \"duration_ns\": %llu, \"bytes\": %llu, \"extra\": ",
-                (int) rec.type, rec.name ? rec.name : "unknown", rec.backend_id, rec.split_id,
-                (unsigned long long) rec.start_ns, (unsigned long long) duration_ns, (unsigned long long) rec.bytes);
-
-        if (rec.extra != NULL) {
-            fprintf(fp, "\"%s\"", rec.extra);
-        } else {
-            fprintf(fp, "null");
-        }
-
-        // Output tensor info
-        if (rec.tensor_name[0] != '\0') {
-            fprintf(fp, ", \"tensor_name\": \"%s\"", rec.tensor_name);
-        } else {
-            fprintf(fp, ", \"tensor_name\": null");
-        }
-        fprintf(fp, ", \"ne\": [%lld, %lld, %lld, %lld]", (long long) rec.ne[0], (long long) rec.ne[1],
-                (long long) rec.ne[2], (long long) rec.ne[3]);
-        fprintf(fp, ", \"out_type\": %d", rec.out_type);
-
-        // Source tensors
-        fprintf(fp, ", \"n_src\": %d", rec.n_src);
-        fprintf(fp, ", \"ne_src\": [");
-        for (int s = 0; s < rec.n_src; s++) {
-            fprintf(fp, "%s[%lld, %lld, %lld, %lld]", s == 0 ? "" : ", ",
-                    (long long) rec.ne_src[s][0], (long long) rec.ne_src[s][1],
-                    (long long) rec.ne_src[s][2], (long long) rec.ne_src[s][3]);
-        }
-        fprintf(fp, "]");
-        fprintf(fp, ", \"nb_src\": [");
-        for (int s = 0; s < rec.n_src; s++) {
-            fprintf(fp, "%s[%lld, %lld, %lld, %lld]", s == 0 ? "" : ", ",
-                    (long long) rec.nb_src[s][0], (long long) rec.nb_src[s][1],
-                    (long long) rec.nb_src[s][2], (long long) rec.nb_src[s][3]);
-        }
-        fprintf(fp, "]");
-        fprintf(fp, ", \"type_src\": [");
-        for (int s = 0; s < rec.n_src; s++) {
-            fprintf(fp, "%s%d", s == 0 ? "" : ", ", rec.type_src[s]);
-        }
-        fprintf(fp, "]");
-
-        // op_params (full 16-int32 block, matching export-graph-ops format)
-        fprintf(fp, ", \"op_params\": [");
-        const int n_op_params = (int) (sizeof(rec.op_params) / sizeof(rec.op_params[0]));
-        for (int p = 0; p < n_op_params; p++) {
-            fprintf(fp, "%s%d", p == 0 ? "" : ", ", rec.op_params[p]);
-        }
-        fprintf(fp, "]");
-
-        fprintf(fp, ", \"sub_op\": %d", rec.sub_op);
-
-        fprintf(fp, "}%s\n", (i < (int) sched->profiling_records.size() - 1) ? "," : "");
-    }
-
-    fprintf(fp, "  ]\n");
-    fprintf(fp, "}\n");
-
-    return 0;
-}
-
-int ggml_backend_sched_export_profiling_json(ggml_backend_sched_t sched, const char * filepath) {
-    GGML_ASSERT(sched);
-    GGML_ASSERT(filepath != NULL);
-
-    FILE * fp = fopen(filepath, "w");
-    if (fp == NULL) {
-        GGML_LOG_ERROR("%s: failed to open %s for writing\n", __func__, filepath);
-        return -1;
-    }
-
-    int ret = ggml_backend_sched_write_profiling_json(sched, fp);
-    fclose(fp);
-
-    return ret;
-}
-
-// Helper: format ne dimensions as string, e.g. "[4096, 4096, 1]"
-static void fmt_ne(char * buf, size_t bufsize, const int64_t ne[4]) {
-    if (ne[0] == 0 && ne[1] == 0 && ne[2] == 0 && ne[3] == 0) {
-        buf[0] = '\0';
-        return;
-    }
-    int ndims = 4;
-    while (ndims > 1 && ne[ndims - 1] <= 1) {
-        ndims--;
-    }
-    int pos = snprintf(buf, bufsize, "[");
-    for (int i = 0; i < ndims && pos < (int) bufsize - 1; i++) {
-        pos += snprintf(buf + pos, bufsize - pos, "%s%lld", i > 0 ? ", " : "", (long long) ne[i]);
-    }
-    snprintf(buf + pos, bufsize - pos, "]");
-}
-
-// Helper: format bandwidth as string
-static void fmt_bandwidth(char * buf, size_t bufsize, uint64_t bytes, uint64_t ns) {
-    if (ns == 0 || bytes == 0) {
-        buf[0] = '\0';
-        return;
-    }
-    double bw_gbps = (double) bytes / (double) ns;
-    if (bw_gbps >= 1000.0) {
-        snprintf(buf, bufsize, "%.2f TB/s", bw_gbps / 1000.0);
-    } else {
-        snprintf(buf, bufsize, "%.2f GB/s", bw_gbps);
-    }
-}
-
-int ggml_backend_sched_write_profiling_text(ggml_backend_sched_t sched, FILE * fp) {
-    GGML_ASSERT(sched);
-    GGML_ASSERT(fp != NULL);
-
-    if (sched->profiling_records.empty()) {
-        fprintf(fp, "No profiling data available.\n");
-        return 0;
-    }
-
-    // Aggregate by (name, type, backend_id)
-    struct op_stats {
-        const char *                 name;
-        enum ggml_profile_event_type type;
-        int                          backend_id;
-        uint64_t                     total_ns;
-        uint64_t                     min_ns;
-        uint64_t                     max_ns;
-        int                          count;
-        uint64_t                     total_bytes;
-        int64_t                      representative_ne_src0[4];
-        int64_t                      representative_ne_src1[4];
-        int64_t                      representative_ne_src2[4];
-    };
-
-    std::vector<op_stats> stats;
-    for (const auto & rec : sched->profiling_records) {
-        uint64_t dur = (rec.end_ns > rec.start_ns) ? (rec.end_ns - rec.start_ns) : 0;
-        bool found = false;
-        for (auto & s : stats) {
-            if (s.type == rec.type && s.backend_id == rec.backend_id && strcmp(s.name, rec.name) == 0) {
-                s.total_ns += dur;
-                s.min_ns = std::min(s.min_ns, dur);
-                s.max_ns = std::max(s.max_ns, dur);
-                s.count++;
-                s.total_bytes += rec.bytes;
-                found = true;
-                break;
-            }
-        }
-        if (!found) {
-            op_stats s = {};
-            s.name        = rec.name;
-            s.type        = rec.type;
-            s.backend_id  = rec.backend_id;
-            s.total_ns    = dur;
-            s.min_ns      = dur;
-            s.max_ns      = dur;
-            s.count       = 1;
-            s.total_bytes = rec.bytes;
-            memcpy(s.representative_ne_src0, rec.ne_src[0], sizeof(s.representative_ne_src0));
-            memcpy(s.representative_ne_src1, rec.ne_src[1], sizeof(s.representative_ne_src1));
-            memcpy(s.representative_ne_src2, rec.ne_src[2], sizeof(s.representative_ne_src2));
-            stats.push_back(s);
-        }
-    }
-
-    std::sort(stats.begin(), stats.end(),
-              [](const op_stats & a, const op_stats & b) { return a.total_ns > b.total_ns; });
-
-    uint64_t grand_total = 0;
-    for (const auto & s : stats) {
-        grand_total += s.total_ns;
-    }
-
-    // --- Section 1: Overall summary ---
-    fprintf(fp, "=== Profiling Summary ===\n");
-    fprintf(fp, "Total time:    %.2f ms\n", (double) grand_total / 1e6);
-    fprintf(fp, "Total records: %d\n", (int) sched->profiling_records.size());
-    fprintf(fp, "Unique ops:    %d\n\n", (int) stats.size());
-
-    // --- Section 2: Per-backend breakdown ---
-    fprintf(fp, "=== Per-Backend Summary ===\n");
-    {
-        struct backend_stats {
-            int      backend_id;
-            int      op_count;
-            int      copy_count;
-            uint64_t op_ns;
-            uint64_t copy_ns;
-            uint64_t op_bytes;
-            uint64_t copy_bytes;
-        };
-        std::vector<backend_stats> bstats;
-        for (const auto & s : stats) {
-            bool found = false;
-            for (auto & bs : bstats) {
-                if (bs.backend_id == s.backend_id) {
-                    if (s.type == GGML_PROFILE_EVENT_OP) {
-                        bs.op_count += s.count;
-                        bs.op_ns    += s.total_ns;
-                        bs.op_bytes += s.total_bytes;
-                    } else {
-                        bs.copy_count += s.count;
-                        bs.copy_ns    += s.total_ns;
-                        bs.copy_bytes += s.total_bytes;
-                    }
-                    found = true;
-                    break;
-                }
-            }
-            if (!found) {
-                backend_stats bs = {};
-                bs.backend_id = s.backend_id;
-                if (s.type == GGML_PROFILE_EVENT_OP) {
-                    bs.op_count = s.count;
-                    bs.op_ns    = s.total_ns;
-                    bs.op_bytes = s.total_bytes;
-                } else {
-                    bs.copy_count = s.count;
-                    bs.copy_ns    = s.total_ns;
-                    bs.copy_bytes = s.total_bytes;
-                }
-                bstats.push_back(bs);
-            }
-        }
-        std::sort(bstats.begin(), bstats.end(),
-                  [](const backend_stats & a, const backend_stats & b) {
-                      return (a.op_ns + a.copy_ns) > (b.op_ns + b.copy_ns);
-                  });
-
-        for (const auto & bs : bstats) {
-            uint64_t total = bs.op_ns + bs.copy_ns;
-            double   pct   = grand_total > 0 ? 100.0 * (double) total / (double) grand_total : 0;
-
-            const char * bname = "unknown";
-            if (bs.backend_id >= 0 && bs.backend_id < (int) sched->profiling_backend_meta.size()) {
-                bname = sched->profiling_backend_meta[bs.backend_id].name.c_str();
-            } else if (bs.backend_id >= 0 && bs.backend_id < sched->n_backends && sched->backends[bs.backend_id] != NULL) {
-                bname = ggml_backend_name(sched->backends[bs.backend_id]);
-            }
-
-            fprintf(fp, "  Backend %d (%s): %.2f ms (%.1f%%)\n", bs.backend_id, bname, (double) total / 1e6, pct);
-            if (bs.op_count > 0) {
-                char bw_buf[32];
-                fmt_bandwidth(bw_buf, sizeof(bw_buf), bs.op_bytes, bs.op_ns);
-                fprintf(fp, "    OPs:   %d calls, %.2f ms", bs.op_count, (double) bs.op_ns / 1e6);
-                if (bw_buf[0]) {
-                    fprintf(fp, ", %s", bw_buf);
-                }
-                fprintf(fp, "\n");
-            }
-            if (bs.copy_count > 0) {
-                char bw_buf[32];
-                fmt_bandwidth(bw_buf, sizeof(bw_buf), bs.copy_bytes, bs.copy_ns);
-                fprintf(fp, "    COPYs: %d calls, %.2f ms", bs.copy_count, (double) bs.copy_ns / 1e6);
-                if (bw_buf[0]) {
-                    fprintf(fp, ", %s", bw_buf);
-                }
-                fprintf(fp, "\n");
-            }
-        }
-    }
-    fprintf(fp, "\n");
-
-    // --- Section 3: Detailed operation table ---
-    fprintf(fp, "=== Operations (sorted by total time) ===\n");
-    fprintf(fp, "%-5s %4s  %-28s  %7s  %6s  %10s  %10s  %10s  %10s  %12s  %s\n",
-            "TYPE", "BKND", "Operation", "%Time", "Count", "Total(ms)", "Avg(us)", "Min(us)", "Max(us)", "Bandwidth", "Tensors");
-    fprintf(fp, "%-5s %4s  %-28s  %7s  %6s  %10s  %10s  %10s  %10s  %12s  %s\n",
-            "-----", "----", "----------------------------", "-------", "------",
-            "----------", "----------", "----------", "----------", "------------", "-------");
-
-    const char * type_str[] = { "OP", "COPY" };
-    for (const auto & s : stats) {
-        double pct    = grand_total > 0 ? 100.0 * (double) s.total_ns / (double) grand_total : 0;
-        double avg_us = (double) s.total_ns / (double) s.count / 1000.0;
-        double min_us = (double) s.min_ns / 1000.0;
-        double max_us = (double) s.max_ns / 1000.0;
-
-        char bw_buf[32] = "";
-        fmt_bandwidth(bw_buf, sizeof(bw_buf), s.total_bytes, s.total_ns);
-
-        char ne0_buf[64];
-        char ne1_buf[64];
-        char ne2_buf[64];
-        fmt_ne(ne0_buf, sizeof(ne0_buf), s.representative_ne_src0);
-        fmt_ne(ne1_buf, sizeof(ne1_buf), s.representative_ne_src1);
-        fmt_ne(ne2_buf, sizeof(ne2_buf), s.representative_ne_src2);
-
-        // Build tensor shapes string
-        char tensors_buf[256] = "";
-        int tpos = 0;
-        if (ne0_buf[0]) {
-            tpos += snprintf(tensors_buf + tpos, sizeof(tensors_buf) - tpos, "%s", ne0_buf);
-        }
-        if (ne1_buf[0]) {
-            tpos += snprintf(tensors_buf + tpos, sizeof(tensors_buf) - tpos, " x %s", ne1_buf);
-        }
-        if (ne2_buf[0]) {
-            tpos += snprintf(tensors_buf + tpos, sizeof(tensors_buf) - tpos, " x %s", ne2_buf);
-        }
-
-        fprintf(fp, "%-5s %4d  %-28s  %6.1f%%  %6d  %10.2f  %10.2f  %10.2f  %10.2f  %12s  %s\n",
-                type_str[s.type], s.backend_id, s.name, pct, s.count,
-                (double) s.total_ns / 1e6, avg_us, min_us, max_us,
-                bw_buf, tensors_buf);
-    }
-
-    fprintf(fp, "\nTotal: %.2f ms  (%d records, %d unique ops)\n", (double) grand_total / 1e6,
-            (int) sched->profiling_records.size(), (int) stats.size());
-
-    return 0;
-}
-
-int ggml_backend_sched_export_profiling_text(ggml_backend_sched_t sched, const char * filepath) {
-    GGML_ASSERT(sched);
-    GGML_ASSERT(filepath != NULL);
-
-    FILE * fp = fopen(filepath, "w");
-    if (fp == NULL) {
-        GGML_LOG_ERROR("%s: failed to open %s for writing\n", __func__, filepath);
-        return -1;
-    }
-
-    int ret = ggml_backend_sched_write_profiling_text(sched, fp);
-    fclose(fp);
-
-    return ret;
-}

ggml/src/ggml-blas/ggml-blas.cpp

@@ -2,7 +2,6 @@
 #include "ggml-impl.h"
 #include "ggml-blas.h"
 #include "ggml-backend-impl.h"
-#include "ggml-profiler.h"
 
 #include <future>
 #include <vector>
@@ -27,11 +26,6 @@ struct ggml_backend_blas_context {
 #ifndef GGML_USE_OPENMP
     std::vector<std::future<void>> tasks;
 #endif
-
-    // Profiling state
-    bool                             profiling_enabled  = false;
-    int                              profiling_split_id = -1;
-    std::vector<ggml_profile_record> profiling_records;
 };
 
 static void ggml_backend_blas_mul_mat(ggml_backend_blas_context * ctx, struct ggml_tensor * dst) {
@@ -239,18 +233,6 @@ static enum ggml_status ggml_backend_blas_graph_compute(ggml_backend_t backend,
             continue;
         }
 
-        // Skip view/identity ops
-        if (node->op == GGML_OP_NONE || node->op == GGML_OP_RESHAPE || node->op == GGML_OP_VIEW ||
-            node->op == GGML_OP_PERMUTE || node->op == GGML_OP_TRANSPOSE) {
-            continue;
-        }
-
-        // Profiling: time this operation
-        uint64_t t_start = 0;
-        if (ctx->profiling_enabled) {
-            t_start = ggml_profiler_time_ns();
-        }
-
         switch (node->op) {
             case GGML_OP_MUL_MAT:
                 ggml_backend_blas_mul_mat(ctx, node);
@@ -260,24 +242,16 @@ static enum ggml_status ggml_backend_blas_graph_compute(ggml_backend_t backend,
                 ggml_backend_blas_out_prod(ctx, node);
                 break;
 
+            case GGML_OP_NONE:
+            case GGML_OP_RESHAPE:
+            case GGML_OP_VIEW:
+            case GGML_OP_PERMUTE:
+            case GGML_OP_TRANSPOSE:
+                break;
+
             default:
                 GGML_ABORT("%s: unsupported op %s\n", __func__, ggml_op_desc(node));
         }
-
-        if (ctx->profiling_enabled) {
-            uint64_t            t_end = ggml_profiler_time_ns();
-            ggml_profile_record rec;
-            rec.type       = GGML_PROFILE_EVENT_OP;
-            rec.name       = ggml_op_name(node->op);
-            rec.backend_id = 0;
-            rec.split_id   = ctx->profiling_split_id;
-            rec.start_ns   = t_start;
-            rec.end_ns     = t_end;
-            rec.bytes      = ggml_nbytes(node);
-            rec.extra      = NULL;
-            ggml_profile_record_from_tensor(&rec, node);
-            ctx->profiling_records.push_back(rec);
-        }
     }
 
     return GGML_STATUS_SUCCESS;
@@ -313,11 +287,10 @@ ggml_backend_t ggml_backend_blas_init(void) {
     ggml_backend_blas_context * ctx = new ggml_backend_blas_context;
 
     ggml_backend_t backend = new ggml_backend {
-        /* .guid     = */ ggml_backend_blas_guid(),
-        /* .iface    = */ blas_backend_i,
-        /* .device   = */ ggml_backend_reg_dev_get(ggml_backend_blas_reg(), 0),
-        /* .context  = */ ctx,
-        /* .profiler = */ nullptr,
+        /* .guid    = */ ggml_backend_blas_guid(),
+        /* .iface   = */ blas_backend_i,
+        /* .device  = */ ggml_backend_reg_dev_get(ggml_backend_blas_reg(), 0),
+        /* .context = */ ctx,
     };
 
 #if defined(GGML_BLAS_USE_OPENBLAS) && defined(GGML_USE_OPENMP)
@@ -330,44 +303,6 @@ ggml_backend_t ggml_backend_blas_init(void) {
     GGML_LOG_DEBUG("%s: warning: ggml is using OpenMP, but BLIS was compiled without OpenMP support\n", __func__);
 #endif
 
-    // Register profiler
-    ggml_backend_blas_context * blas_ctx = ctx;  // ctx is already defined above
-
-    static auto blas_prof_enable = [](void * ctx, bool enable) {
-        auto * bctx             = (ggml_backend_blas_context *) ctx;
-        bctx->profiling_enabled = enable;
-        if (!enable) {
-            bctx->profiling_records.clear();
-        }
-    };
-    static auto blas_prof_reset = [](void * ctx) {
-        auto * bctx = (ggml_backend_blas_context *) ctx;
-        bctx->profiling_records.clear();
-        bctx->profiling_split_id = -1;
-    };
-    static auto blas_prof_set_split_id = [](void * ctx, int split_id) {
-        auto * bctx              = (ggml_backend_blas_context *) ctx;
-        bctx->profiling_split_id = split_id;
-    };
-    static auto blas_prof_get_records = [](void * ctx, const ggml_profile_record ** out) -> int {
-        auto * bctx = (ggml_backend_blas_context *) ctx;
-        *out        = bctx->profiling_records.data();
-        return (int) bctx->profiling_records.size();
-    };
-    static auto blas_prof_free = [](void * ctx) {
-        (void) ctx;
-    };
-
-    auto * profiler = new ggml_backend_profiler{
-        /* .context      = */ blas_ctx,
-        /* .enable       = */ blas_prof_enable,
-        /* .reset        = */ blas_prof_reset,
-        /* .set_split_id = */ blas_prof_set_split_id,
-        /* .get_records  = */ blas_prof_get_records,
-        /* .free_context = */ blas_prof_free,
-    };
-    ggml_backend_set_profiler(backend, profiler);
-
     return backend;
 }
 

ggml/src/ggml-cann/ggml-cann.cpp

@@ -3035,8 +3035,7 @@ ggml_backend_t ggml_backend_cann_init(int32_t device) {
         new ggml_backend{ /* .guid      = */ ggml_backend_cann_guid(),
                           /* .interface = */ ggml_backend_cann_interface,
                           /* .device    = */ ggml_backend_reg_dev_get(ggml_backend_cann_reg(), device),
-                          /* .context   = */ ctx,
-                          /* .profiler  = */ nullptr };
+                          /* .context   = */ ctx };
 
     return cann_backend;
 }

ggml/src/ggml-cpu/ggml-cpu.c

@@ -6,7 +6,6 @@
 #include "traits.h"
 #include "ggml-cpu-impl.h"
 #include "ggml-impl.h"
-#include "ggml-profiler.h"
 #include "quants.h"
 #include "ggml-threading.h"
 #include "unary-ops.h"
@@ -1179,8 +1178,8 @@ static void ggml_compute_forward_mul_mat_one_chunk(
 
     const bool src1_cont = ggml_is_contiguous(src1);
 
-    const ggml_vec_dot_t vec_dot      = type_traits_cpu[type].vec_dot;
-    const enum ggml_type vec_dot_type = type_traits_cpu[type].vec_dot_type;
+    ggml_vec_dot_t const vec_dot      = type_traits_cpu[type].vec_dot;
+    enum ggml_type const vec_dot_type = type_traits_cpu[type].vec_dot_type;
 
     // broadcast factors
     const int64_t r2 = ne12 / ne02;
@@ -1270,9 +1269,9 @@ void ggml_compute_forward_mul_mat(
     const int ith = params->ith;
     const int nth = params->nth;
 
-    const enum ggml_type    vec_dot_type     = type_traits_cpu[src0->type].vec_dot_type;
-    const ggml_from_float_t from_float       = type_traits_cpu[vec_dot_type].from_float;
-    const int64_t           vec_dot_num_rows = type_traits_cpu[src0->type].nrows;
+    enum ggml_type           const vec_dot_type         = type_traits_cpu[src0->type].vec_dot_type;
+    ggml_from_float_t        const from_float           = type_traits_cpu[vec_dot_type].from_float;
+    int64_t                  const vec_dot_num_rows     = type_traits_cpu[src0->type].nrows;
 
     GGML_ASSERT(ne0 == ne01);
     GGML_ASSERT(ne1 == ne11);
@@ -1481,8 +1480,8 @@ static void ggml_compute_forward_mul_mat_id_one_chunk(
 
     const enum ggml_type type = src0->type;
 
-    const ggml_vec_dot_t vec_dot      = type_traits_cpu[type].vec_dot;
-    const enum ggml_type vec_dot_type = type_traits_cpu[type].vec_dot_type;
+    ggml_vec_dot_t    const vec_dot      = type_traits_cpu[type].vec_dot;
+    enum ggml_type    const vec_dot_type = type_traits_cpu[type].vec_dot_type;
 
     const int64_t blck_0 = 16;
     const int64_t blck_1 = 16;
@@ -1549,8 +1548,8 @@ static void ggml_compute_forward_mul_mat_id(
 
     const bool src1_cont = ggml_is_contiguous(src1);
 
-    const enum ggml_type    vec_dot_type = type_traits_cpu[type].vec_dot_type;
-    const ggml_from_float_t from_float   = type_traits_cpu[vec_dot_type].from_float;
+    enum ggml_type    const vec_dot_type    = type_traits_cpu[type].vec_dot_type;
+    ggml_from_float_t const from_float      = type_traits_cpu[vec_dot_type].from_float;
 
     // we don't support permuted src0 or src1
     GGML_ASSERT(nb00 == ggml_type_size(type));
@@ -3086,55 +3085,17 @@ static thread_ret_t ggml_graph_compute_thread(void * data) {
     GGML_PRINT_DEBUG("thread #%d compute-start cplan %p last-graph %d\n", state->ith, (const void *)cplan, state->last_graph);
 #endif
 
-    // Profiling state
-    if (cplan->profiling_context != NULL && cplan->profiling_record_fn != NULL) {
-        for (int node_n = 0; node_n < cgraph->n_nodes && atomic_load_explicit(&tp->abort, memory_order_relaxed) != node_n; node_n++) {
-            struct ggml_tensor * node = cgraph->nodes[node_n];
+    for (int node_n = 0; node_n < cgraph->n_nodes && atomic_load_explicit(&tp->abort, memory_order_relaxed) != node_n; node_n++) {
+        struct ggml_tensor * node = cgraph->nodes[node_n];
 
-            if (ggml_op_is_empty(node->op)) {
-                continue;
-            }
-
-            if ((node->flags & GGML_TENSOR_FLAG_COMPUTE) == 0) {
-                continue;
-            }
-
-            // Only thread 0 records timing (after barrier = total node time)
-            uint64_t t_start = 0;
-            if (state->ith == 0) {
-                t_start = ggml_profiler_time_ns();
-            }
-
-            ggml_compute_forward(&params, node);
-
-            if (node_n + 1 < cgraph->n_nodes) {
-                ggml_barrier(state->threadpool);
-            }
-
-            if (state->ith == 0) {
-                uint64_t t_end = ggml_profiler_time_ns();
-                cplan->profiling_record_fn(cplan->profiling_context, 0 /* GGML_PROFILE_EVENT_OP */,
-                                           ggml_op_name(node->op), -1, t_start, t_end, ggml_nbytes(node), NULL,
-                                           node);
-            }
-
-            if (state->ith == 0 && cplan->abort_callback && cplan->abort_callback(cplan->abort_callback_data)) {
-                atomic_store_explicit(&tp->abort, node_n + 1, memory_order_relaxed);
-                tp->ec = GGML_STATUS_ABORTED;
-            }
+        if (ggml_op_is_empty(node->op)) {
+            // skip NOPs
+            continue;
         }
-    } else {
-        for (int node_n = 0; node_n < cgraph->n_nodes && atomic_load_explicit(&tp->abort, memory_order_relaxed) != node_n; node_n++) {
-            struct ggml_tensor * node = cgraph->nodes[node_n];
 
-            if (ggml_op_is_empty(node->op)) {
-                // skip NOPs
-                continue;
-            }
-
-            if ((node->flags & GGML_TENSOR_FLAG_COMPUTE) == 0) {
-                continue;
-            }
+        if ((node->flags & GGML_TENSOR_FLAG_COMPUTE) == 0) {
+            continue;
+        }
 
         // TODO: move fused-op detection into ggml_graph_plan so fusion decisions are made once at planning time
         // Try fused ops, fall back to normal compute
@@ -3145,15 +3106,14 @@ static thread_ret_t ggml_graph_compute_thread(void * data) {
             ggml_compute_forward(&params, node);
         }
 
-            if (state->ith == 0 && cplan->abort_callback &&
-                    cplan->abort_callback(cplan->abort_callback_data)) {
-                atomic_store_explicit(&tp->abort, node_n + 1, memory_order_relaxed);
-                tp->ec    = GGML_STATUS_ABORTED;
-            }
+        if (state->ith == 0 && cplan->abort_callback &&
+                cplan->abort_callback(cplan->abort_callback_data)) {
+            atomic_store_explicit(&tp->abort, node_n + 1, memory_order_relaxed);
+            tp->ec    = GGML_STATUS_ABORTED;
+        }
 
-            if (node_n + 1 < cgraph->n_nodes) {
-                ggml_barrier(state->threadpool);
-            }
+        if (node_n + 1 < cgraph->n_nodes) {
+            ggml_barrier(state->threadpool);
         }
     }
 

ggml/src/ggml-cpu/ggml-cpu.cpp

@@ -1,7 +1,6 @@
 #include "ggml-backend.h"
 #include "ggml-backend-impl.h"
 #include "ggml-cpu.h"
-#include "ggml-profiler.h"
 #include "repack.h"
 #include "traits.h"
 #include "ggml-impl.h"
@@ -108,11 +107,6 @@ struct ggml_backend_cpu_context {
     void *              abort_callback_data;
 
     bool                use_ref;  // use reference implementation
-
-    // Profiling state
-    bool                             profiling_enabled;
-    int                              profiling_split_id;
-    std::vector<ggml_profile_record> profiling_records;
 };
 
 static const char * ggml_backend_cpu_get_name(ggml_backend_t backend) {
@@ -173,30 +167,6 @@ static enum ggml_status ggml_backend_cpu_graph_plan_compute(ggml_backend_t backe
     GGML_UNUSED(backend);
 }
 
-// Callback function for recording CPU profiling events from C code (ggml-cpu.c)
-static void ggml_cpu_profiler_record_callback(void *                     context,
-                                               int                        type,
-                                               const char *               name,
-                                               int                        split_id,
-                                               uint64_t                   start_ns,
-                                               uint64_t                   end_ns,
-                                               uint64_t                   bytes,
-                                               const char *               extra,
-                                               const struct ggml_tensor * node) {
-    auto *              cpu_ctx = (ggml_backend_cpu_context *) context;
-    ggml_profile_record rec;
-    rec.type       = (enum ggml_profile_event_type) type;
-    rec.name       = name;
-    rec.backend_id = 0;  // will be overwritten by scheduler
-    rec.split_id   = split_id != -1 ? split_id : cpu_ctx->profiling_split_id;
-    rec.start_ns   = start_ns;
-    rec.end_ns     = end_ns;
-    rec.bytes      = bytes;
-    rec.extra      = extra;
-    ggml_profile_record_from_tensor(&rec, node);
-    cpu_ctx->profiling_records.push_back(rec);
-}
-
 static enum ggml_status ggml_backend_cpu_graph_compute(ggml_backend_t backend, struct ggml_cgraph * cgraph) {
     struct ggml_backend_cpu_context * cpu_ctx = (struct ggml_backend_cpu_context *)backend->context;
 
@@ -217,9 +187,6 @@ static enum ggml_status ggml_backend_cpu_graph_compute(ggml_backend_t backend, s
     cplan.abort_callback_data = cpu_ctx->abort_callback_data;
     cplan.use_ref             = cpu_ctx->use_ref;
 
-    cplan.profiling_context   = cpu_ctx->profiling_enabled ? cpu_ctx : NULL;
-    cplan.profiling_record_fn = cpu_ctx->profiling_enabled ? ggml_cpu_profiler_record_callback : NULL;
-
     return ggml_graph_compute(cgraph, &cplan);
 }
 
@@ -263,15 +230,12 @@ ggml_backend_t ggml_backend_cpu_init(void) {
     ctx->abort_callback      = NULL;
     ctx->abort_callback_data = NULL;
     ctx->use_ref             = false;
-    ctx->profiling_enabled   = false;
-    ctx->profiling_split_id  = -1;
 
     ggml_backend_t cpu_backend = new ggml_backend {
-        /* .guid     = */ ggml_backend_cpu_guid(),
-        /* .iface    = */ ggml_backend_cpu_i,
-        /* .device   = */ ggml_backend_reg_dev_get(ggml_backend_cpu_reg(), 0),
-        /* .context  = */ ctx,
-        /* .profiler = */ nullptr,
+        /* .guid    = */ ggml_backend_cpu_guid(),
+        /* .iface   = */ ggml_backend_cpu_i,
+        /* .device  = */ ggml_backend_reg_dev_get(ggml_backend_cpu_reg(), 0),
+        /* .context = */ ctx,
     };
 
     if (cpu_backend == NULL) {
@@ -279,43 +243,6 @@ ggml_backend_t ggml_backend_cpu_init(void) {
         return NULL;
     }
 
-    // Register profiler
-    static auto cpu_prof_enable = [](void * ctx, bool enable) {
-        auto * cpu_ctx             = (ggml_backend_cpu_context *) ctx;
-        cpu_ctx->profiling_enabled = enable;
-        if (!enable) {
-            cpu_ctx->profiling_records.clear();
-        }
-    };
-    static auto cpu_prof_reset = [](void * ctx) {
-        auto * cpu_ctx = (ggml_backend_cpu_context *) ctx;
-        cpu_ctx->profiling_records.clear();
-        cpu_ctx->profiling_split_id = -1;
-    };
-    static auto cpu_prof_set_split_id = [](void * ctx, int split_id) {
-        auto * cpu_ctx              = (ggml_backend_cpu_context *) ctx;
-        cpu_ctx->profiling_split_id = split_id;
-    };
-    static auto cpu_prof_get_records = [](void * ctx, const ggml_profile_record ** out) -> int {
-        auto * cpu_ctx = (ggml_backend_cpu_context *) ctx;
-        *out           = cpu_ctx->profiling_records.data();
-        return (int) cpu_ctx->profiling_records.size();
-    };
-    static auto cpu_prof_free = [](void * ctx) {
-        // Nothing to free - records are in the CPU context's vector
-        (void) ctx;
-    };
-
-    auto * profiler = new ggml_backend_profiler{
-        /* .context      = */ ctx,
-        /* .enable       = */ cpu_prof_enable,
-        /* .reset        = */ cpu_prof_reset,
-        /* .set_split_id = */ cpu_prof_set_split_id,
-        /* .get_records  = */ cpu_prof_get_records,
-        /* .free_context = */ cpu_prof_free,
-    };
-    ggml_backend_set_profiler(cpu_backend, profiler);
-
     return cpu_backend;
 }
 

ggml/src/ggml-cpu/llamafile/sgemm.cpp

@@ -1797,14 +1797,6 @@ class tinyBLAS_Q0_AVX {
 //PPC Implementation
 #if defined(__MMA__)
 
-#define SAVE_ACC(ACC, ii, jj) \
-   __builtin_mma_disassemble_acc(vec_C, ACC); \
-   for (int I = 0; I < 4; I++) { \
-      for (int J = 0; J < 4; J++) { \
-         *((float*)(C+ii+((jj+J)*ldc)+I)) = *((float*)&vec_C[I]+J); \
-      } \
-   } \
-
 template<typename T>
 struct mma_instr;
 
@@ -1834,10 +1826,49 @@ class tinyBLAS_HP16_PPC {
     }
 
     void matmul(int64_t m, int64_t n) {
-        mnpack(0, m, 0, n);
+        int64_t mc = 256;
+        int64_t nc = 256;
+        int64_t kc = 256;
+    #if defined(_AIX) || defined(__BIG_ENDIAN__)
+        mc = 128;
+        nc = 128;
+        kc = 128;
+    #endif
+        if (k < kc) {
+            kc = k;
+        }
+        bool can_use_tiled = (m % mc == 0) && (n % nc == 0) && (k % kc == 0);
+        if (can_use_tiled) {
+            matmul_tiled(m, n, mc, nc, kc);
+        } else {
+            mnpack(0, m, 0, n);
+        }
     }
 
   private:
+    __attribute__((always_inline))
+    inline void save_acc(acc_t * ACC, int64_t ii, int64_t jj) {
+        vec_t vec_C[4];
+        __builtin_mma_disassemble_acc(vec_C, ACC);
+        for (int I = 0; I < 4; I++) {
+            for (int J = 0; J < 4; J++) {
+                *((float *)(C+ii+((jj+J)*ldc)+I)) = *((float *)&vec_C[I]+J);
+            }
+        }
+    }
+
+    __attribute__((always_inline))
+    inline void add_save_acc(acc_t * ACC, int64_t ii, int64_t jj) {
+        vec_t vec_C[4];
+        __builtin_mma_disassemble_acc(vec_C, ACC);
+        for (int I = 0; I < 4; I++) {
+            for (int J = 0; J < 4; J++) {
+                float * c_ptr = (float *)(C+ii+((jj+J)*ldc)+I);
+                *c_ptr += *((float *)&vec_C[I]+J);
+            }
+        }
+    }
+
     void vector_permute_store(vec_t *c, int numVec, unsigned char *vecOffset) {
         vec_t t[8], s[8];
         vec_t swiz1 = {0, 1, 2, 3, 16, 17, 18, 19, 4, 5, 6, 7, 20, 21, 22, 23};
@@ -1896,6 +1927,7 @@ class tinyBLAS_HP16_PPC {
         j = (rows >> 3);
         if (j > 0) {
             do {
+                aoffsets[0] = aoffset;
                 if (cols == 4) {
                     aoffsets[0] = aoffset;
                     for (int it = 1; it < 4; ++it)
@@ -1910,17 +1942,17 @@ class tinyBLAS_HP16_PPC {
                 }
                 i = (cols >> 3);
                 if (i > 0) {
-                    aoffsets[0] = aoffset;
                     for (int it = 1; it < 8; ++it) {
                         aoffsets[it] = aoffsets[it-1] + lda;
                     }
                     aoffset += 8 * lda;
+
                     do {
                         for (int it = 0; it < 8; ++it)
                             c_arr[it] = vec_xl(0, (vector unsigned char*)aoffsets[it]);
                         vector_permute_store(c_arr, 8, vecOffset);
                         for (int it = 0; it < 8; ++it)
-                            aoffsets[it] = aoffsets[it] + 8*lda;
+                            aoffsets[it] = aoffsets[it] + 8;
                         vecOffset += 128;
                         i--;
                     } while(i > 0);
@@ -2147,8 +2179,8 @@ class tinyBLAS_HP16_PPC {
                 mma_instr<TA>::outer_product(&acc_1, vec_A[x], vec_B[x+4]);
             }
         }
-        SAVE_ACC(&acc_0, ii, jj);
-        SAVE_ACC(&acc_1, ii, jj+4);
+        save_acc(&acc_0, ii, jj);
+        save_acc(&acc_1, ii, jj+4);
     }
 
     void KERNEL_8x4(int64_t ii, int64_t jj) {
@@ -2164,8 +2196,8 @@ class tinyBLAS_HP16_PPC {
                 mma_instr<TA>::outer_product(&acc_1, vec_A[x+4], vec_B[x]);
             }
         }
-        SAVE_ACC(&acc_0, ii, jj);
-        SAVE_ACC(&acc_1, ii+4, jj);
+        save_acc(&acc_0, ii, jj);
+        save_acc(&acc_1, ii+4, jj);
     }
 
 
@@ -2186,13 +2218,64 @@ class tinyBLAS_HP16_PPC {
                 mma_instr<TA>::outer_product(&acc_3, vec_A[x+4], vec_B[x+4]);
             }
         }
-
-        SAVE_ACC(&acc_0, ii, jj);
-        SAVE_ACC(&acc_1, ii, jj+4);
-        SAVE_ACC(&acc_2, ii+4, jj);
-        SAVE_ACC(&acc_3, ii+4, jj+4);
+        save_acc(&acc_0, ii, jj);
+        save_acc(&acc_1, ii, jj+4);
+        save_acc(&acc_2, ii+4, jj);
+        save_acc(&acc_3, ii+4, jj+4);
     }
 
+    inline void MMA_16x8(vec_t * vec_A0, vec_t * vec_A1, vec_t * vec_B, acc_t * acc) {
+        for (int x = 0; x < 4; x ++) {
+             mma_instr<TA>::outer_product(&acc[0], vec_A0[x], vec_B[x]);
+             mma_instr<TA>::outer_product(&acc[1], vec_A0[x], vec_B[x+4]);
+             mma_instr<TA>::outer_product(&acc[2], vec_A0[x+4], vec_B[x]);
+             mma_instr<TA>::outer_product(&acc[3], vec_A0[x+4], vec_B[x+4]);
+             mma_instr<TA>::outer_product(&acc[4], vec_A1[x], vec_B[x]);
+             mma_instr<TA>::outer_product(&acc[5], vec_A1[x], vec_B[x+4]);
+             mma_instr<TA>::outer_product(&acc[6], vec_A1[x+4], vec_B[x]);
+             mma_instr<TA>::outer_product(&acc[7], vec_A1[x+4], vec_B[x+4]);
+        }
+    }
+    void KERNEL(int64_t ii, int64_t jj, int64_t mc, int64_t nc, int64_t kc, vec_t * vec_A, vec_t * vec_B, int64_t kk) {
+        for (int64_t i = 0; i < mc; i += 16) {
+            int A_base_addr = (mc / 8) * (i / 8) * 8;
+            for (int64_t j = 0; j < nc; j += 8) {
+                 int B_base_addr = (nc / 8) * (j / 8) * 8;
+                 acc_t acc[8];
+                 vec_t A0_block[8]; vec_t A1_block[8];
+                 for (int x = 0; x < 8; x++)
+                     __builtin_mma_xxsetaccz(&acc[x]);
+                 for (int64_t l = 0; l < kc; l += 8) {
+                     int A0_block_idx = A_base_addr + (l / 8) * 8;
+                     int A1_block_idx = A0_block_idx + (mc / 8) * 8;
+                     int B_block_idx = B_base_addr + (l / 8) * 8;
+                     vec_t* A0_block = &vec_A[A0_block_idx];
+                     vec_t* A1_block = &vec_A[A1_block_idx];
+                     vec_t* B_block = &vec_B[B_block_idx];
+                     MMA_16x8(A0_block, A1_block, B_block, acc);
+                 }
+                 if (kk == 0) {
+                     save_acc(&acc[0], ii + i, jj + j);
+                     save_acc(&acc[1], ii + i, jj + j + 4);
+                     save_acc(&acc[2], ii + i + 4, jj + j);
+                     save_acc(&acc[3], ii + i + 4, jj + j + 4);
+                     save_acc(&acc[4], ii + i + 8, jj + j);
+                     save_acc(&acc[5], ii + i + 8, jj + j + 4);
+                     save_acc(&acc[6], ii + i + 12, jj + j);
+                     save_acc(&acc[7], ii + i + 12, jj + j + 4);
+                 } else {
+                     add_save_acc(&acc[0], ii + i, jj + j);
+                     add_save_acc(&acc[1], ii + i, jj + j + 4);
+                     add_save_acc(&acc[2], ii + i + 4, jj + j);
+                     add_save_acc(&acc[3], ii + i + 4, jj + j + 4);
+                     add_save_acc(&acc[4], ii + i + 8, jj + j);
+                     add_save_acc(&acc[5], ii + i + 8, jj + j + 4);
+                     add_save_acc(&acc[6], ii + i + 12, jj + j);
+                     add_save_acc(&acc[7], ii + i + 12, jj + j + 4);
+                 }
+            }
+        }
+    }
     template<int RM, int RN>
     void gemm_small(int64_t m0, int64_t m, int64_t n0, int64_t n) {
         int64_t ytiles = (m - m0) / RM;
@@ -2281,6 +2364,29 @@ class tinyBLAS_HP16_PPC {
        }
     }
 
+    void matmul_tiled(int64_t m, int64_t n, int64_t mc, int64_t nc, int64_t kc) {
+        int64_t ytiles = m / mc;
+        int64_t xtiles = n / nc;
+        int64_t tiles = xtiles * ytiles;
+        int64_t duty = (tiles + nth - 1) / nth;
+        int64_t start = duty * ith;
+        int64_t end = start + duty;
+        if (end > tiles) {
+            end = tiles;
+        }
+        for (int64_t job = start; job < end; ++job) {
+            int64_t ii = (job / xtiles) * mc;
+            int64_t jj = (job % xtiles) * nc;
+            for (int64_t kk = 0; kk < k; kk += kc) {
+                 vec_t A_pack[kc * mc / 8];
+                 vec_t B_pack[kc * nc / 8];
+                 packNormal(A + (ii * lda) + kk, lda, kc, mc, (uint8_t *)A_pack);
+                 packNormal(B + (jj * ldb) + kk, ldb, kc, nc, (uint8_t *)B_pack);
+                 KERNEL(ii, jj, mc, nc, kc, A_pack, B_pack, kk);
+            }
+        }
+    }
+
     template <int RM, int RN>
     NOINLINE void gemm(int64_t m0, int64_t m, int64_t n0, int64_t n) {
         int64_t ytiles = (m - m0) / RM;

ggml/src/ggml-cuda/common.cuh

@@ -1404,9 +1404,6 @@ struct ggml_cuda_stream_context {
     }
 };
 
-// Forward declaration for profiler state (defined in ggml-cuda.cu)
-struct ggml_cuda_profiler_state;
-
 struct ggml_backend_cuda_context {
     int device;
     std::string name;
@@ -1518,9 +1515,6 @@ struct ggml_backend_cuda_context {
     ggml_cuda_pool & pool() {
         return pool(device);
     }
-
-    // Profiling
-    ggml_cuda_profiler_state * profiler_state = nullptr;
 };
 
 struct ggml_cuda_mm_fusion_args_host {

ggml/src/ggml-cuda/fattn-tile.cu

@@ -1,6 +1,5 @@
 #include "common.cuh"
 #include "fattn-tile.cuh"
-#include "fattn-wmma-f16.cuh"
 
 void ggml_cuda_flash_attn_ext_tile(ggml_backend_cuda_context & ctx, ggml_tensor * dst) {
     const ggml_tensor * K = dst->src[1];

ggml/src/ggml-cuda/fattn-tile.cuh

@@ -1,6 +1,5 @@
 #include "common.cuh"
 #include "fattn-common.cuh"
-#include "fattn-wmma-f16.cuh"
 
 // nbatch_fa == number of KQ rows to process per iteration
 // nbatch_K == number of K columns to load in parallel for KQ calculation
@@ -825,12 +824,7 @@ static __global__ void flash_attn_tile(
 
     // Skip unused kernel variants for faster compilation:
 
-    if (
-#ifdef GGML_USE_WMMA_FATTN
-            (ncols2 != 1 && DV != 40 && DV != 72 && DV != 512) ||
-#endif // GGML_USE_WMMA_FATTN
-            (use_logit_softcap && !(DV == 128 || DV == 256 || DV == 512))
-    ) {
+    if ((use_logit_softcap && !(DV == 128 || DV == 256 || DV == 512))) {
         GGML_UNUSED_VARS(Q, K, V, mask, sinks, KV_max, dst, dst_meta, scale,
             max_bias, m0, m1, n_head_log2, logit_softcap,
             ne00, ne01, ne02, ne03,

ggml/src/ggml-cuda/fattn-wmma-f16.cu

@@ -1,705 +0,0 @@
-// Old and deprecated WMMA FlashAttention implementation.
-// It is still needed for Volta since the memory layout of NVIDIA tensor cores changed with Turing.
-// Long-term the WMMA code should be replaced with a dedicated Volta implementation.
-
-#include "common.cuh"
-#include "fattn-common.cuh"
-#include "fattn-wmma-f16.cuh"
-
-#ifdef GGML_USE_WMMA_FATTN
-#if !defined(GGML_USE_HIP)
-#include <mma.h>
-#if defined(GGML_USE_MUSA)
-namespace wmma = mtmusa::wmma;
-#else // GGML_USE_MUSA
-namespace wmma = nvcuda::wmma;
-#endif // GGML_USE_MUSA
-#elif defined(GGML_USE_HIP)
-#include <rocwmma/rocwmma.hpp>
-namespace wmma = rocwmma;
-#endif // !defined(GGML_USE_HIP)
-#endif // GGML_USE_WMMA_FATTN
-
-// D == head size, VKQ_stride == num VKQ rows calculated in parallel:
-template<int D, int ncols, int nwarps, int VKQ_stride, typename KQ_acc_t, bool use_logit_softcap>
-__launch_bounds__(nwarps*ggml_cuda_get_physical_warp_size(), 1)
-static __global__ void flash_attn_ext_f16(
-        const char * Q_ptr,
-        const char * K_ptr,
-        const char * V_ptr,
-        const char * mask_ptr,
-        const char * sinks_ptr,
-        const int  * KV_max_ptr,
-        float      * dst_ptr,
-        float2     * dst_meta_ptr,
-        const float scale,
-        const float max_bias,
-        const float m0,
-        const float m1,
-        const uint32_t n_head_log2,
-        const float logit_softcap,
-        const int32_t ne00, const uint3   ne01, const int32_t ne02, const int32_t ne03,
-                            const int32_t nb01, const int32_t nb02, const int32_t nb03,
-        const int32_t ne10, const int32_t ne11, const int32_t ne12, const int32_t ne13,
-                            const int32_t nb11, const int32_t nb12, const int64_t nb13,
-                            const int32_t nb21, const int32_t nb22, const int64_t nb23,
-                            const int32_t ne31, const int32_t ne32, const int32_t ne33,
-                            const int32_t nb31, const int32_t nb32, const int64_t nb33) {
-#if defined(FLASH_ATTN_AVAILABLE) && (defined(GGML_HIP_ROCWMMA_FATTN) && defined(GGML_USE_WMMA_FATTN))
-    const char * GGML_CUDA_RESTRICT Q        = Q_ptr;
-    const char * GGML_CUDA_RESTRICT K        = K_ptr;
-    const char * GGML_CUDA_RESTRICT V        = V_ptr;
-    const char * GGML_CUDA_RESTRICT mask     = mask_ptr;
-    const char * GGML_CUDA_RESTRICT sinks    = sinks_ptr;
-    const int  * GGML_CUDA_RESTRICT KV_max   = KV_max_ptr;
-    float      * GGML_CUDA_RESTRICT dst      = dst_ptr;
-    float2     * GGML_CUDA_RESTRICT dst_meta = dst_meta_ptr;
-    // Skip unused kernel variants for faster compilation:
-    if (use_logit_softcap && !(D == 128 || D == 256)) {
-        NO_DEVICE_CODE;
-        return;
-    }
-
-    //In this kernel Q, K, V are matrices while i, j, k are matrix indices.
-
-    constexpr int warp_size = ggml_cuda_get_physical_warp_size();
-
-    const int ic0 = ncols*blockIdx.x; // Index of the first Q/QKV column to work on.
-
-    static_assert(D <= FATTN_KQ_STRIDE, "D must be <= FATTN_KQ_STRIDE.");
-    static_assert(ncols == 8 || ncols % 16 == 0, "ncols must be 8 or a multiple of 16.");
-    constexpr int frag_m = ncols == 8 ? 32 : 16;
-    constexpr int frag_n = ncols == 8 ?  8 : 16;
-    static_assert(D % frag_m == 0, "If ncols == 8 then D % frag_m must be 0.");
-#if defined(GGML_USE_HIP) && HIP_VERSION >= 60500000
-    typedef wmma::fragment<wmma::matrix_a,    frag_m, frag_n, 16, _Float16, wmma::row_major> frag_a_K;
-    typedef wmma::fragment<wmma::matrix_a,    frag_m, frag_n, 16, _Float16, wmma::col_major> frag_a_V;
-    typedef wmma::fragment<wmma::matrix_b,    frag_m, frag_n, 16, _Float16, wmma::col_major> frag_b;
-    typedef wmma::fragment<wmma::accumulator, frag_m, frag_n, 16, KQ_acc_t>                      frag_c_KQ;
-    typedef wmma::fragment<wmma::accumulator, frag_m, frag_n, 16, _Float16>                          frag_c_VKQ;
-#else
-    typedef wmma::fragment<wmma::matrix_a,    frag_m, frag_n, 16, half, wmma::row_major> frag_a_K;
-    typedef wmma::fragment<wmma::matrix_a,    frag_m, frag_n, 16, half, wmma::col_major> frag_a_V;
-    typedef wmma::fragment<wmma::matrix_b,    frag_m, frag_n, 16, half, wmma::col_major> frag_b;
-    typedef wmma::fragment<wmma::accumulator, frag_m, frag_n, 16, KQ_acc_t>                      frag_c_KQ;
-    typedef wmma::fragment<wmma::accumulator, frag_m, frag_n, 16, half>                          frag_c_VKQ;
-#endif
-
-    constexpr int KQ_stride_tc  = nwarps*frag_m; // Number of KQ rows calculated in parallel.
-    constexpr int VKQ_ratio = KQ_stride_tc/VKQ_stride; // Number of parallel VKQ accumulators needed to keep all warps busy.
-    static_assert(VKQ_ratio <= nwarps, "VKQ_ratio must be <= nwarps.");
-
-    // Pad internal representation of KQ, KQV to reduce shared memory bank conflicts:
-    constexpr int D_padded = D + 8;
-    constexpr int kqs_padded = FATTN_KQ_STRIDE + 8;
-    constexpr int kqar = sizeof(KQ_acc_t)/sizeof(half);
-
-    ggml_cuda_pdl_sync();
-    const int sequence = blockIdx.z / ne02;
-    const int head = blockIdx.z - sequence*ne02;
-    const int gqa_ratio = ne02 / ne12; // With grouped query attention there are > 1 Q matrices per K, V matrix.
-    const float * Q_f    = (const float *) (Q    + nb03* sequence         + nb02* head              + nb01*ic0);
-    const half  * K_h    = (const half  *) (K    + nb13* sequence         + nb12*(head / gqa_ratio));
-    const half  * V_h    = (const half  *) (V    + nb13* sequence         + nb12*(head / gqa_ratio)); // K and V have same shape
-    const half  * maskh  = (const half  *) (mask + nb33*(sequence % ne33)                           + nb31*ic0);
-    const half2 * mask2  = (const half2 *)  maskh;
-    const float * sinksf = (const float *) sinks;
-
-    const int stride_Q  = nb01 / sizeof(float);
-    const int stride_KV = nb11 / sizeof(half);
-
-    const float slopef = get_alibi_slope(max_bias, head, n_head_log2, m0, m1);
-    const half  slopeh = __float2half(slopef);
-    const half2 slope2 = make_half2(slopef, slopef);
-
-    const half2 logit_softcap_2 = make_half2(logit_softcap, logit_softcap);
-
-    frag_b Q_b[D/16][ncols/frag_n];
-
-    // A single buffer for temporarily holding tiles of KQ and VKQ parts:
-    constexpr int mem_KQ = ncols*kqs_padded*kqar;
-    constexpr int mem_VKQ_parts = VKQ_ratio*ncols*D_padded;
-    __shared__ half KQ[mem_KQ >= mem_VKQ_parts ? mem_KQ : mem_VKQ_parts];
-    float * KQ_f = (float *) KQ;
-    half2 * KQ2 = (half2 *) KQ;
-
-    float    KQ_rowsum_f[ncols/nwarps] = {0.0f};
-    float       KQ_max_f[ncols/nwarps];
-    float KQ_max_scale_f[ncols/nwarps] = {0.0f};
-
-#pragma unroll
-    for (int j = 0; j < ncols/nwarps; ++j) {
-        KQ_max_f[j] = -FLT_MAX/2.0f;
-    }
-
-    half2    KQ_rowsum_h2[ncols/nwarps] = {{0.0f, 0.0f}};
-    half2       KQ_max_h2[ncols/nwarps];
-    half2 KQ_max_scale_h2[ncols/nwarps] = {{0.0f, 0.0f}};
-
-#pragma unroll
-    for (int j = 0; j < ncols/nwarps; ++j) {
-        KQ_max_h2[j] = make_half2(-HALF_MAX_HALF, -HALF_MAX_HALF);
-    }
-
-    __shared__ half VKQ[ncols*D_padded]; // Accumulator for final VKQ slice.
-    half2 * VKQ2 = (half2 *) VKQ;
-
-#if defined(GGML_USE_HIP) && HIP_VERSION >= 60500000
-    const _Float16 * K_h_f16  = reinterpret_cast<const _Float16 *>(K_h);
-    const _Float16 * V_h_f16  = reinterpret_cast<const _Float16 *>(V_h);
-    _Float16       * KQ_f16   = reinterpret_cast<_Float16 *>(KQ);
-    _Float16       * VKQ_f16  = reinterpret_cast<_Float16 *>(VKQ);
-#else
-    const half * K_h_f16  = K_h;
-    const half * V_h_f16  = V_h;
-    half       * KQ_f16   = KQ;
-    half       * VKQ_f16  = VKQ;
-#endif
-
-#pragma unroll
-    for (int j0 = 0; j0 < ncols; j0 += nwarps) {
-        const int j = j0 + threadIdx.y;
-#pragma unroll
-        for (int i0 = 0; i0 < D/2; i0 += warp_size) {
-            const int i = i0 + threadIdx.x;
-            if (i0 + warp_size > D/2 && i >= D/2) {
-                break;
-            }
-            VKQ2[j*(D_padded/2) + i] = make_half2(0.0f, 0.0f);
-        }
-    }
-
-    // Convert Q to half and apply scale, temporarily store in KQ:
-#pragma unroll
-    for (int j0 = 0; j0 < ncols; j0 += nwarps) {
-        const int j = j0 + threadIdx.y;
-#pragma unroll
-        for (int i0 = 0; i0 < D; i0 += warp_size) {
-            const int i = i0 + threadIdx.x;
-            if (i0 + warp_size > D && i >= D) {
-                break;
-            }
-            KQ[j*D_padded + i] = ic0 + j < int(ne01.z) ? Q_f[j*stride_Q + i] * scale : 0.0f;
-        }
-    }
-
-    __syncthreads();
-
-    // Load Q into tensor core fragments/registers since it will be used frequently:
-#pragma unroll
-    for (int i0 = 0; i0 < D; i0 += 16) {
-#pragma unroll
-        for (int j0 = 0; j0 < ncols; j0 += frag_n) {
-            wmma::load_matrix_sync(Q_b[i0/16][j0/frag_n], KQ_f16 + j0*D_padded + i0, D_padded);
-        }
-    }
-
-    __syncthreads();
-
-    // Iterate over ne11 == previous tokens:
-    const int k_VKQ_max = KV_max ? KV_max[sequence*gridDim.x + blockIdx.x] : ne11;
-    for (int k_VKQ_0 = blockIdx.y*FATTN_KQ_STRIDE; k_VKQ_0 < k_VKQ_max; k_VKQ_0 += gridDim.y*FATTN_KQ_STRIDE) {
-        // Calculate tile of KQ:
-#pragma unroll
-        for (int i_KQ_0 = 0; i_KQ_0 < FATTN_KQ_STRIDE; i_KQ_0 += KQ_stride_tc) {
-            frag_c_KQ KQ_c[ncols/frag_n];
-#pragma unroll
-            for (int j = 0; j < ncols/frag_n; ++j) {
-                wmma::fill_fragment(KQ_c[j], static_cast<KQ_acc_t>(0.0f));
-            }
-#pragma unroll
-            for (int k_KQ_0 = 0; k_KQ_0 < D; k_KQ_0 += 16) {
-                frag_a_K K_a;
-                wmma::load_matrix_sync(K_a, K_h_f16 + int64_t(k_VKQ_0 + i_KQ_0 + frag_m*threadIdx.y)*stride_KV + k_KQ_0, stride_KV);
-#pragma unroll
-                for (int j = 0; j < ncols/frag_n; ++j) {
-                    wmma::mma_sync(KQ_c[j], K_a, Q_b[k_KQ_0/16][j], KQ_c[j]);
-                }
-            }
-#pragma unroll
-            for (int j0 = 0; j0 < ncols; j0 += frag_n) {
-                wmma::store_matrix_sync((KQ_acc_t *) KQ + j0*kqs_padded + i_KQ_0 + frag_m*threadIdx.y, KQ_c[j0/frag_n], kqs_padded, wmma::mem_col_major);
-            }
-        }
-
-        __syncthreads();
-
-        // Calculate softmax for each KQ column using the current max. value.
-        // The divisor is stored in KQ_rowsum and will be applied at the end.
-#pragma unroll
-        for (int j0 = 0; j0 < ncols; j0 += nwarps) {
-            const int j = j0 + threadIdx.y;
-
-            if (std::is_same<KQ_acc_t, float>::value) {
-                float KQ_f_tmp[FATTN_KQ_STRIDE / warp_size];
-#pragma unroll
-                for (int k0 = 0; k0 < FATTN_KQ_STRIDE; k0 += warp_size) {
-                    const int k = k0 + threadIdx.x;
-
-                    KQ_f_tmp[k0/warp_size] = KQ_f[j*kqs_padded + k];
-
-                    if (use_logit_softcap) {
-                        KQ_f_tmp[k0/warp_size] = logit_softcap*tanhf(KQ_f_tmp[k0/warp_size]);
-                    }
-                }
-
-                float KQ_max_new = KQ_max_f[j0/nwarps];
-#pragma unroll
-                for (int k0 = 0; k0 < FATTN_KQ_STRIDE; k0 += warp_size) {
-                    const int k = k0 + threadIdx.x;
-
-                    KQ_f_tmp[k0/warp_size] += mask && ic0 + j < int(ne01.z) ?
-                        __half2float(slopeh*maskh[j*(nb31/sizeof(half)) + k_VKQ_0 + k]) : 0.0f;
-                    KQ_max_new = max(KQ_max_new, KQ_f_tmp[k0/warp_size] + FATTN_KQ_MAX_OFFSET);
-                }
-                KQ_max_new = warp_reduce_max<warp_size>(KQ_max_new);
-
-                const float diff = KQ_max_f[j0/nwarps] - KQ_max_new;
-                KQ_max_scale_f[j0/nwarps] = expf(diff);
-                if (diff <= SOFTMAX_FTZ_THRESHOLD) {
-                    KQ_max_scale_f[j0/nwarps] = 0.0f;
-                }
-                KQ_max_f[j0/nwarps] = KQ_max_new;
-
-                float KQ_rowsum_add = 0.0f;
-#pragma unroll
-                for (int k0 = 0; k0 < FATTN_KQ_STRIDE; k0 += warp_size) {
-                    const int k = k0 + threadIdx.x;
-
-                    const float diff = KQ_f_tmp[k0/warp_size] - KQ_max_f[j0/nwarps];
-                    KQ_f_tmp[k0/warp_size] = expf(diff);
-                    if (diff <= SOFTMAX_FTZ_THRESHOLD) {
-                        KQ_f_tmp[k0/warp_size] = 0.0f;
-                    }
-                    KQ_rowsum_add += KQ_f_tmp[k0/warp_size];
-                    KQ[j*(kqar*kqs_padded) + k] = KQ_f_tmp[k0/warp_size];
-                }
-                KQ_rowsum_add = warp_reduce_sum<warp_size>(KQ_rowsum_add);
-
-                // Scale previous KQ_rowsum to account for a potential increase in KQ_max:
-                KQ_rowsum_f[j0/nwarps] = KQ_max_scale_f[j0/nwarps]*KQ_rowsum_f[j0/nwarps] + KQ_rowsum_add;
-            } else {
-                half2 KQ2_tmp[FATTN_KQ_STRIDE/(2*warp_size)];
-#pragma unroll
-                for (int k0 = 0; k0 < FATTN_KQ_STRIDE/2; k0 += warp_size) {
-                    const int k = k0 + threadIdx.x;
-
-                    KQ2_tmp[k0/warp_size] = KQ2[j*(kqs_padded/2) + k];
-
-                    if (use_logit_softcap) {
-                        // There is no dedicated tangens hyperbolicus function for half2.
-                        KQ2_tmp[k0/warp_size] = h2exp(KQ2_tmp[k0/warp_size]*make_half2(2.0f, 2.0f));
-                        KQ2_tmp[k0/warp_size] = (KQ2_tmp[k0/warp_size] - make_half2(1.0f, 1.0f))
-                                               /(KQ2_tmp[k0/warp_size] + make_half2(1.0f, 1.0f));
-
-                        KQ2_tmp[k0/warp_size] *= logit_softcap_2;
-                    }
-                }
-
-                half2 KQ_max_new = KQ_max_h2[j0/nwarps];
-#pragma unroll
-                for (int k0 = 0; k0 < FATTN_KQ_STRIDE/2; k0 += warp_size) {
-                    const int k = k0 + threadIdx.x;
-
-                    KQ2_tmp[k0/warp_size] += mask && ic0 + j < int(ne01.z) ? slope2*mask2[(j*ne11 + k_VKQ_0)/2 + k] : make_half2(0.0f, 0.0f);
-                    KQ_max_new = ggml_cuda_hmax2(KQ_max_new, KQ2_tmp[k0/warp_size]);
-                }
-                KQ_max_new = __half2half2(warp_reduce_max<warp_size>(ggml_cuda_hmax(__low2half(KQ_max_new), __high2half(KQ_max_new))));
-                const half2 diff = KQ_max_h2[j0/nwarps] - KQ_max_new;
-                KQ_max_scale_h2[j0/nwarps] = h2exp(diff);
-                const uint32_t ftz_mask = __hgt2_mask(diff, make_half2(SOFTMAX_FTZ_THRESHOLD, SOFTMAX_FTZ_THRESHOLD));
-                *((uint32_t *) &KQ_max_scale_h2[j0/nwarps]) &= ftz_mask;
-                KQ_max_h2[j0/nwarps] = KQ_max_new;
-
-                half2 KQ_rowsum_add = make_half2(0.0f, 0.0f);
-#pragma unroll
-                for (int k0 = 0; k0 < FATTN_KQ_STRIDE/2; k0 += warp_size) {
-                    const int k = k0 + threadIdx.x;
-
-                    const half2 diff = KQ2_tmp[k0/warp_size] - KQ_max_h2[j0/nwarps];
-                    KQ2_tmp[k0/warp_size] = h2exp(diff);
-                    const uint32_t ftz_mask = __hgt2_mask(diff, make_half2(SOFTMAX_FTZ_THRESHOLD, SOFTMAX_FTZ_THRESHOLD));
-                    *((uint32_t *) &KQ2_tmp[k0/warp_size]) &= ftz_mask;
-                    KQ_rowsum_add += KQ2_tmp[k0/warp_size];
-                    KQ2[j*(kqs_padded/2) + k] = KQ2_tmp[k0/warp_size];
-                }
-                KQ_rowsum_add = warp_reduce_sum<warp_size>(KQ_rowsum_add);
-
-                // Scale previous KQ_rowsum to account for a potential increase in KQ_max:
-                KQ_rowsum_h2[j0/nwarps] = KQ_max_scale_h2[j0/nwarps]*KQ_rowsum_h2[j0/nwarps] + KQ_rowsum_add;
-            }
-        }
-
-        __syncthreads();
-
-        frag_b KQ_b[FATTN_KQ_STRIDE/(VKQ_ratio*16)][ncols/frag_n];
-#pragma unroll
-        for (int j0 = 0; j0 < ncols; j0 += frag_n) {
-#pragma unroll
-            for (int k0 = 0; k0 < FATTN_KQ_STRIDE; k0 += VKQ_ratio*16) {
-                const int k = k0 + (threadIdx.y % VKQ_ratio)*16;
-                wmma::load_matrix_sync(
-                    KQ_b[k0/(VKQ_ratio*16)][j0/frag_n],
-                    KQ_f16 + j0*(kqar*kqs_padded) + k,
-                    kqar*kqs_padded);
-            }
-        }
-
-        frag_c_VKQ VKQ_c[D/VKQ_stride][ncols/frag_n];
-#pragma unroll
-        for (int i_VKQ_0 = 0; i_VKQ_0 < D; i_VKQ_0 += VKQ_stride) {
-#pragma unroll
-            for (int j = 0; j < ncols/frag_n; ++j) {
-                wmma::fill_fragment(VKQ_c[i_VKQ_0/VKQ_stride][j], static_cast<half>(0.0f));
-            }
-
-#pragma unroll
-            for (int k0 = 0; k0 < FATTN_KQ_STRIDE; k0 += VKQ_ratio*16) {
-                const int k = k0 + (threadIdx.y % VKQ_ratio)*16;
-
-                frag_a_V v_a;
-                wmma::load_matrix_sync(v_a, V_h_f16 + int64_t(k_VKQ_0 + k)*stride_KV + i_VKQ_0 + frag_m*(threadIdx.y/VKQ_ratio), stride_KV);
-#pragma unroll
-                for (int j = 0; j < ncols/frag_n; ++j) {
-                    wmma::mma_sync(VKQ_c[i_VKQ_0/VKQ_stride][j], v_a, KQ_b[k0/(VKQ_ratio*16)][j], VKQ_c[i_VKQ_0/VKQ_stride][j]);
-                }
-            }
-        }
-
-        __syncthreads();
-
-        const int offset_k = (threadIdx.y % VKQ_ratio) * (ncols*D_padded);
-#pragma unroll
-        for (int i_KQ_0 = 0; i_KQ_0 < D; i_KQ_0 += VKQ_stride) {
-#pragma unroll
-            for (int j0 = 0; j0 < ncols; j0 += frag_n) {
-                wmma::store_matrix_sync(
-                    KQ_f16 + offset_k + j0*D_padded + i_KQ_0 + frag_m*(threadIdx.y/VKQ_ratio),
-                    VKQ_c[i_KQ_0/VKQ_stride][j0/frag_n],
-                    D_padded, wmma::mem_col_major);
-            }
-        }
-
-        __syncthreads();
-
-#pragma unroll
-        for (int j0 = 0; j0 < ncols; j0 += nwarps) {
-            const int j = j0 + threadIdx.y;
-
-            half2 VKQ_scale;
-            if (std::is_same<KQ_acc_t, float>::value) {
-                VKQ_scale = make_half2(KQ_max_scale_f[j0/nwarps], KQ_max_scale_f[j0/nwarps]);
-            } else {
-                VKQ_scale = KQ_max_scale_h2[j0/nwarps];
-            }
-
-#pragma unroll
-            for (int i0 = 0; i0 < D/2; i0 += warp_size) {
-                const int i = i0 + threadIdx.x;
-                if (i0 + warp_size > D/2 && i >= D/2) {
-                    break;
-                }
-
-                half2 VKQ_add = make_half2(0.0f, 0.0f);
-#pragma unroll
-                for (int l = 0; l < VKQ_ratio; ++l) {
-                    VKQ_add += KQ2[l*(ncols*D_padded/2) + j*(D_padded/2) + i];
-                }
-                VKQ2[j*(D_padded/2) + i] = VKQ_scale*VKQ2[j*(D_padded/2) + i] + VKQ_add;
-            }
-        }
-
-        __syncthreads();
-    }
-
-    // Apply attention sinks
-    if (sinksf && blockIdx.y == 0) {
-        const float sinkf = sinksf[head];
-        const half  sinkh = __float2half(sinkf);
-
-#pragma unroll
-        for (int j0 = 0; j0 < ncols; j0 += nwarps) {
-            const int j = j0 + threadIdx.y;
-
-            if (std::is_same<KQ_acc_t, float>::value) {
-                float kqmax_new = fmaxf(KQ_max_f[j0/nwarps], sinkf);
-
-                const float KQ_max_scale = expf(KQ_max_f[j0/nwarps] - kqmax_new);
-                KQ_max_f[j0/nwarps] = kqmax_new;
-
-                KQ_rowsum_f[j0/nwarps] = KQ_rowsum_f[j0/nwarps] * KQ_max_scale + expf(sinkf - KQ_max_f[j0/nwarps]);
-
-                const half2 scale_h2 = make_half2(KQ_max_scale, KQ_max_scale);
-#pragma unroll
-                for (int i0 = 0; i0 < D/2; i0 += warp_size) {
-                    const int i = i0 + threadIdx.x;
-                    if (i0 + warp_size > D/2 && i >= D/2) break;
-                    VKQ2[j*(D_padded/2) + i] *= scale_h2;
-                }
-            } else {
-                half kqmax_old = __low2half(KQ_max_h2[j0/nwarps]);
-                half kqmax_new = fmaxf(kqmax_old, sinkh);
-                KQ_max_h2[j0/nwarps] = __half2half2(kqmax_new);
-
-                const half  KQ_max_scale_h = hexp(kqmax_old - kqmax_new);
-                const half2 KQ_max_scale   = __half2half2(KQ_max_scale_h);
-
-                KQ_rowsum_h2[j0/nwarps] = KQ_rowsum_h2[j0/nwarps] * KQ_max_scale;
-                const half val = hexp(sinkh - kqmax_new);
-                KQ_rowsum_h2[j0/nwarps].x = __hadd(KQ_rowsum_h2[j0/nwarps].x, val);
-
-#pragma unroll
-                for (int i0 = 0; i0 < D/2; i0 += warp_size) {
-                    const int i = i0 + threadIdx.x;
-                    if (i0 + warp_size > D/2 && i >= D/2) break;
-                    VKQ2[j*(D_padded/2) + i] *= KQ_max_scale;
-                }
-            }
-        }
-
-        __syncthreads();
-    }
-#pragma unroll
-    for (int j0 = 0; j0 < ncols; j0 += nwarps) {
-        const int j_VKQ = j0 + threadIdx.y;
-        if (ic0 + j_VKQ >= int(ne01.z)) {
-            return;
-        }
-
-        float KQ_rowsum_j;
-        if (std::is_same<KQ_acc_t, float>::value) {
-            KQ_rowsum_j = KQ_rowsum_f[j0/nwarps];
-        } else {
-            KQ_rowsum_j = __low2float(KQ_rowsum_h2[j0/nwarps]) + __high2float(KQ_rowsum_h2[j0/nwarps]);
-        }
-
-        const int j_dst_unrolled = ((sequence*int(ne01.z) + ic0 + j_VKQ)*ne02 + head)*gridDim.y + blockIdx.y;
-
-#pragma unroll
-        for (int i0 = 0; i0 < D; i0 += warp_size) {
-            const int i = i0 + threadIdx.x;
-            if (i0 + warp_size > D && i >= D) {
-                break;
-            }
-            float dst_val = VKQ[j_VKQ*D_padded + i];
-            if (gridDim.y == 1) {
-                dst_val /= KQ_rowsum_j;
-            }
-            dst[j_dst_unrolled*D + i] = dst_val;
-        }
-
-        if (gridDim.y == 1 || threadIdx.x != 0) {
-            continue;
-        }
-
-        float2 dst_meta_val;
-        if (std::is_same<KQ_acc_t, float>::value) {
-            dst_meta_val.x = KQ_max_f[j0/nwarps];
-        } else {
-            dst_meta_val.x = __low2float(KQ_max_h2[j0/nwarps]);
-        }
-        dst_meta_val.y = KQ_rowsum_j;
-        dst_meta[j_dst_unrolled] = dst_meta_val;
-    }
-#else
-    GGML_UNUSED_VARS(Q_ptr, K_ptr, V_ptr, mask_ptr, sinks_ptr, KV_max_ptr, dst_ptr, dst_meta_ptr, scale,
-        max_bias, m0, m1, n_head_log2, logit_softcap,
-        ne00, ne01, ne02, ne03,
-              nb01, nb02, nb03,
-        ne10, ne11, ne12, ne13,
-              nb11, nb12, nb13,
-              nb21, nb22, nb23,
-              ne31, ne32, ne33,
-              nb31, nb32, nb33);
-    NO_DEVICE_CODE;
-#endif // defined(FLASH_ATTN_AVAILABLE) && (defined(GGML_HIP_ROCWMMA_FATTN) && defined(GGML_USE_WMMA_FATTN))
-}
-
-constexpr int get_max_power_of_2(int x) {
-    return x % 2 == 0 ? 2*get_max_power_of_2(x/2) : 1;
-}
-
-static_assert(get_max_power_of_2(1) == 1, "Test failed.");
-static_assert(get_max_power_of_2(2) == 2, "Test failed.");
-static_assert(get_max_power_of_2(4) == 4, "Test failed.");
-static_assert(get_max_power_of_2(6) == 2, "Test failed.");
-
-// Number of VKQ rows calculated in parallel:
-constexpr int get_VKQ_stride(int D, int nwarps, int frag_m) {
-    return (get_max_power_of_2(D/frag_m) < nwarps ? get_max_power_of_2(D/frag_m) : nwarps)*frag_m;
-}
-
-static_assert(get_VKQ_stride(128, 1, 32) ==  32, "Test failed.");
-static_assert(get_VKQ_stride(128, 2, 32) ==  64, "Test failed.");
-static_assert(get_VKQ_stride(128, 4, 32) == 128, "Test failed.");
-static_assert(get_VKQ_stride( 64, 1, 32) ==  32, "Test failed.");
-static_assert(get_VKQ_stride( 64, 2, 32) ==  64, "Test failed.");
-static_assert(get_VKQ_stride( 64, 4, 32) ==  64, "Test failed.");
-static_assert(get_VKQ_stride( 80, 1, 16) ==  16, "Test failed.");
-static_assert(get_VKQ_stride( 80, 2, 16) ==  16, "Test failed.");
-static_assert(get_VKQ_stride( 80, 4, 16) ==  16, "Test failed.");
-
-template <int D, int cols_per_block, typename KQ_acc_t>
-void ggml_cuda_flash_attn_ext_wmma_f16_case(ggml_backend_cuda_context & ctx, ggml_tensor * dst) {
-    const ggml_tensor * KQV = dst;
-
-    constexpr int nwarps = 4;
-
-    constexpr int frag_m = cols_per_block == 8 && D % 32 == 0 ? 32 : 16;
-    const int warp_size = ggml_cuda_info().devices[ggml_cuda_get_device()].warp_size;
-
-    float logit_softcap;
-    memcpy(&logit_softcap, (const float *) KQV->op_params + 2, sizeof(float));
-
-    fattn_kernel_t fattn_kernel;
-    if (logit_softcap == 0.0f) {
-        constexpr bool use_logit_softcap = false;
-        fattn_kernel = flash_attn_ext_f16<
-            D, cols_per_block, nwarps, get_VKQ_stride(D, nwarps, frag_m), KQ_acc_t, use_logit_softcap>;
-    } else {
-        constexpr bool use_logit_softcap = true;
-        fattn_kernel = flash_attn_ext_f16<
-            D, cols_per_block, nwarps, get_VKQ_stride(D, nwarps, frag_m), KQ_acc_t, use_logit_softcap>;
-    }
-    launch_fattn<D, cols_per_block, 1>(ctx, dst, fattn_kernel, nwarps, 0, FATTN_KQ_STRIDE, true, true, false, warp_size);
-}
-
-void ggml_cuda_flash_attn_ext_wmma_f16(ggml_backend_cuda_context & ctx, ggml_tensor * dst) {
-    const ggml_tensor * KQV = dst;
-    const ggml_tensor * Q   = dst->src[0];
-
-    const enum ggml_prec prec = ggml_flash_attn_ext_get_prec(KQV);
-    const int warp_size = ggml_cuda_info().devices[ctx.device].warp_size;
-
-    if (prec != GGML_PREC_DEFAULT) {
-        if (Q->ne[1] <= 32 || Q->ne[0] > 128) {
-            constexpr int cols_per_block = 16;
-            switch (Q->ne[0]) {
-                case 64:
-                    ggml_cuda_flash_attn_ext_wmma_f16_case< 64, cols_per_block, float>(ctx, dst);
-                    break;
-                case 80:
-                    ggml_cuda_flash_attn_ext_wmma_f16_case< 80, cols_per_block, float>(ctx, dst);
-                    break;
-                case 96:
-                    ggml_cuda_flash_attn_ext_wmma_f16_case< 96, cols_per_block, float>(ctx, dst);
-                    break;
-                case 112:
-                    ggml_cuda_flash_attn_ext_wmma_f16_case<112, cols_per_block, float>(ctx, dst);
-                    break;
-                case 128:
-                    ggml_cuda_flash_attn_ext_wmma_f16_case<128, cols_per_block, float>(ctx, dst);
-                    break;
-                case 256:
-                    ggml_cuda_flash_attn_ext_wmma_f16_case<256, cols_per_block, float>(ctx, dst);
-                    break;
-                default:
-                    GGML_ABORT("fatal error");
-                    break;
-            }
-        } else {
-            constexpr int cols_per_block = 32;
-            switch (Q->ne[0]) {
-                case 64:
-                    ggml_cuda_flash_attn_ext_wmma_f16_case< 64, cols_per_block, float>(ctx, dst);
-                    break;
-                case 80:
-                    ggml_cuda_flash_attn_ext_wmma_f16_case< 80, cols_per_block, float>(ctx, dst);
-                    break;
-                case 96:
-                    ggml_cuda_flash_attn_ext_wmma_f16_case< 96, cols_per_block, float>(ctx, dst);
-                    break;
-                case 112:
-                    ggml_cuda_flash_attn_ext_wmma_f16_case<112, cols_per_block, float>(ctx, dst);
-                    break;
-                case 128:
-                    ggml_cuda_flash_attn_ext_wmma_f16_case<128, cols_per_block, float>(ctx, dst);
-                    break;
-                // case 256:
-                //     ggml_cuda_flash_attn_ext_wmma_f16_case<256, cols_per_block, float>(ctx, dst);
-                //     break;
-                default:
-                    GGML_ABORT("fatal error");
-                    break;
-            }
-        }
-        return;
-    }
-
-#if !defined(GGML_USE_HIP)
-    if (Q->ne[1] <= 8 && Q->ne[0] % warp_size == 0) {
-        constexpr int cols_per_block = 8;
-        switch (Q->ne[0]) {
-            case 64:
-                ggml_cuda_flash_attn_ext_wmma_f16_case< 64, cols_per_block, half>(ctx, dst);
-                break;
-            case 96:
-                ggml_cuda_flash_attn_ext_wmma_f16_case< 96, cols_per_block, half>(ctx, dst);
-                break;
-            case 128:
-                ggml_cuda_flash_attn_ext_wmma_f16_case<128, cols_per_block, half>(ctx, dst);
-                break;
-            case 256:
-                ggml_cuda_flash_attn_ext_wmma_f16_case<256, cols_per_block, half>(ctx, dst);
-                break;
-            default:
-                GGML_ABORT("fatal error");
-                break;
-        }
-        return;
-    }
-#endif // !defined(GGML_USE_HIP)
-
-    if (Q->ne[1] <= 32) {
-        constexpr int cols_per_block = 16;
-        switch (Q->ne[0]) {
-            case 64:
-                ggml_cuda_flash_attn_ext_wmma_f16_case< 64, cols_per_block, half>(ctx, dst);
-                break;
-            case 80:
-                ggml_cuda_flash_attn_ext_wmma_f16_case< 80, cols_per_block, half>(ctx, dst);
-                break;
-            case 96:
-                ggml_cuda_flash_attn_ext_wmma_f16_case< 96, cols_per_block, half>(ctx, dst);
-                break;
-            case 112:
-                ggml_cuda_flash_attn_ext_wmma_f16_case<112, cols_per_block, half>(ctx, dst);
-                break;
-            case 128:
-                ggml_cuda_flash_attn_ext_wmma_f16_case<128, cols_per_block, half>(ctx, dst);
-                break;
-            case 256:
-                ggml_cuda_flash_attn_ext_wmma_f16_case<256, cols_per_block, half>(ctx, dst);
-                break;
-            default:
-                GGML_ABORT("fatal error");
-                break;
-        }
-        return;
-    }
-
-    constexpr int cols_per_block = 32;
-    switch (Q->ne[0]) {
-        case 64:
-            ggml_cuda_flash_attn_ext_wmma_f16_case< 64, cols_per_block, half>(ctx, dst);
-            break;
-        case 80:
-            ggml_cuda_flash_attn_ext_wmma_f16_case< 80, cols_per_block, half>(ctx, dst);
-            break;
-        case 96:
-            ggml_cuda_flash_attn_ext_wmma_f16_case< 96, cols_per_block, half>(ctx, dst);
-            break;
-        case 112:
-            ggml_cuda_flash_attn_ext_wmma_f16_case<112, cols_per_block, half>(ctx, dst);
-            break;
-        case 128:
-            ggml_cuda_flash_attn_ext_wmma_f16_case<128, cols_per_block, half>(ctx, dst);
-            break;
-        case 256:
-            ggml_cuda_flash_attn_ext_wmma_f16_case<256, cols_per_block, half>(ctx, dst);
-            break;
-        default:
-            GGML_ABORT("fatal error");
-            break;
-    }
-}

ggml/src/ggml-cuda/fattn-wmma-f16.cuh

@@ -1,51 +0,0 @@
-#pragma once
-
-#include "common.cuh"
-
-#if defined(GGML_USE_MUSA)
-#define GGML_USE_WMMA_FATTN
-#endif // defined(GGML_USE_MUSA)
-
-#if defined(GGML_HIP_ROCWMMA_FATTN)
-#if defined(CDNA) && (ROCWMMA_VERSION_MAJOR < 2 || ROCWMMA_VERSION_MINOR > 0 || ROCWMMA_VERSION_PATCH > 0)
-#define GGML_USE_WMMA_FATTN
-#elif defined(CDNA)
-#warning "rocwmma fattn on CDNA is broken on rocwmma v2.0.0, expect degraded performance"
-#endif // defined(CDNA) && (ROCWMMA_VERSION_MAJOR < 2 || ROCWMMA_VERSION_MINOR > 0 || ROCWMMA_VERSION_PATCH > 0)
-#if defined(RDNA3)
-#define GGML_USE_WMMA_FATTN
-#endif // defined(RDNA3)
-#if defined(RDNA4) && ROCWMMA_VERSION_MAJOR > 1
-#define GGML_USE_WMMA_FATTN
-#elif defined(RDNA4)
-#warning "rocwmma fattn is not supported on RDNA4 on rocwmma < v2.0.0, expect degraded performance"
-#endif // defined(RDNA4) && ROCWMMA_VERSION_MAJOR > 1
-#endif // defined(GGML_HIP_ROCWMMA_FATTN)
-
-// WMMA flash attention requires FP16 matrix instructions to be available for ggml code.
-static bool ggml_cuda_should_use_wmma_fattn(const int cc) {
-#if defined(GGML_USE_HIP) && !defined(GGML_HIP_ROCWMMA_FATTN)
-    return false;
-#else
-    if ((GGML_CUDA_CC_IS_NVIDIA(cc) && ggml_cuda_highest_compiled_arch(cc) == GGML_CUDA_CC_VOLTA) ||
-        GGML_CUDA_CC_IS_RDNA3(cc) || GGML_CUDA_CC_IS_MTHREADS(cc)) {
-        return true;
-    } else if (GGML_CUDA_CC_IS_CDNA(cc)){
-#if defined(GGML_HIP_ROCWMMA_FATTN) && (ROCWMMA_VERSION_MAJOR < 2 || ROCWMMA_VERSION_MINOR > 0 || ROCWMMA_VERSION_PATCH > 0)
-        return true;
-#else
-        return false;
-#endif // defined(GGML_HIP_ROCWMMA_FATTN) (ROCWMMA_VERSION_MAJOR < 2 || ROCWMMA_VERSION_MINOR > 0 || ROCWMMA_VERSION_PATCH > 0)
-    } else if (GGML_CUDA_CC_IS_RDNA4(cc)) {
-#if defined(GGML_HIP_ROCWMMA_FATTN) && ROCWMMA_VERSION_MAJOR > 1
-        return true;
-#else
-        return false;
-#endif // defined(GGML_HIP_ROCWMMA_FATTN) && ROCWMMA_VERSION_MAJOR > 1
-    } else {
-        return false;
-    }
-#endif // defined(GGML_USE_HIP) && !defined(GGML_HIP_ROCWMMA_FATTN)
-}
-
-void ggml_cuda_flash_attn_ext_wmma_f16(ggml_backend_cuda_context & ctx, ggml_tensor * dst);

ggml/src/ggml-cuda/fattn.cu

@@ -3,7 +3,6 @@
 #include "fattn-mma-f16.cuh"
 #include "fattn-tile.cuh"
 #include "fattn-vec.cuh"
-#include "fattn-wmma-f16.cuh"
 #include "fattn.cuh"
 
 template <int DKQ, int DV, int ncols2>
@@ -330,11 +329,10 @@ static void ggml_cuda_flash_attn_ext_vec(ggml_backend_cuda_context & ctx, ggml_t
 
 // Best FlashAttention kernel for a specific GPU:
 enum best_fattn_kernel {
-    BEST_FATTN_KERNEL_NONE     =   0,
-    BEST_FATTN_KERNEL_TILE     = 200,
-    BEST_FATTN_KERNEL_VEC      = 100,
-    BEST_FATTN_KERNEL_WMMA_F16 = 300,
-    BEST_FATTN_KERNEL_MMA_F16  = 400,
+    BEST_FATTN_KERNEL_NONE    =   0,
+    BEST_FATTN_KERNEL_TILE    = 200,
+    BEST_FATTN_KERNEL_VEC     = 100,
+    BEST_FATTN_KERNEL_MMA_F16 = 400,
 };
 
 static bool ggml_cuda_fattn_kv_type_supported(ggml_type type) {
@@ -500,14 +498,6 @@ static best_fattn_kernel ggml_cuda_get_best_fattn_kernel(const int device, const
         return BEST_FATTN_KERNEL_MMA_F16;
     }
 
-    // Use the WMMA kernel if possible:
-    if (ggml_cuda_should_use_wmma_fattn(cc) && K->ne[1] % FATTN_KQ_STRIDE == 0 && Q->ne[0] != 40 && Q->ne[0] != 72 && Q->ne[0] != 192 && Q->ne[0] != 512 && Q->ne[0] != 576) {
-        if (can_use_vector_kernel && Q->ne[1] <= 2) {
-            return BEST_FATTN_KERNEL_VEC;
-        }
-        return BEST_FATTN_KERNEL_WMMA_F16;
-    }
-
     // AMD MFMA needs a certain minimum batch size to outscale the tile kernel for large head sizes.
     if ((amd_mfma_available(cc) && Q->ne[0] <= 256) && Q->ne[0] != 40 && Q->ne[0] != 72) {
         if ((Q->ne[0] <= 64 && Q->ne[1] * gqa_ratio_eff > 8)) {
@@ -559,7 +549,6 @@ size_t ggml_cuda_flash_attn_ext_get_alloc_size(int device, const ggml_tensor * d
 
     switch (kernel) {
         case BEST_FATTN_KERNEL_TILE:
-        case BEST_FATTN_KERNEL_WMMA_F16:
         case BEST_FATTN_KERNEL_MMA_F16:
             need_f16_K = true;
             need_f16_V = true;
@@ -589,9 +578,6 @@ void ggml_cuda_flash_attn_ext(ggml_backend_cuda_context & ctx, ggml_tensor * dst
         case BEST_FATTN_KERNEL_VEC:
             ggml_cuda_flash_attn_ext_vec(ctx, dst);
             break;
-        case BEST_FATTN_KERNEL_WMMA_F16:
-            ggml_cuda_flash_attn_ext_wmma_f16(ctx, dst);
-            break;
         case BEST_FATTN_KERNEL_MMA_F16:
             ggml_cuda_flash_attn_ext_mma_f16(ctx, dst);
             break;

ggml/src/ggml-cuda/ggml-cuda.cu

@@ -1,7 +1,6 @@
 #include "ggml-cuda.h"
 #include "ggml-impl.h"
 #include "ggml-backend-impl.h"
-#include "ggml-profiler.h"
 
 #include "ggml-cuda/allreduce.cuh"
 #include "ggml-cuda/common.cuh"
@@ -92,92 +91,6 @@
 
 static_assert(sizeof(half) == sizeof(ggml_fp16_t), "wrong fp16 size");
 
-// CUDA profiler state
-struct ggml_cuda_profiler_state {
-    bool enabled = false;
-    int split_id = -1;
-    cudaStream_t stream = nullptr;
-
-    static constexpr int MAX_PENDING_EVENTS = 4096;
-    std::vector<cudaEvent_t> start_events;
-    std::vector<cudaEvent_t> end_events;
-    std::vector<uint64_t>    cpu_timestamps;  // CPU-side timestamps for global ordering
-    int event_count = 0;
-
-    std::vector<ggml_profile_record> records;
-    std::vector<int> record_event_indices;
-
-    void init(cudaStream_t stream) {
-        this->stream = stream;
-        start_events.reserve(MAX_PENDING_EVENTS);
-        end_events.reserve(MAX_PENDING_EVENTS);
-        cpu_timestamps.reserve(MAX_PENDING_EVENTS);
-    }
-
-    void reset() {
-        for (auto & ev : start_events) {
-            (void) cudaEventDestroy(ev);
-        }
-        for (auto & ev : end_events) {
-            (void) cudaEventDestroy(ev);
-        }
-        start_events.clear();
-        end_events.clear();
-        cpu_timestamps.clear();
-        event_count = 0;
-        records.clear();
-        record_event_indices.clear();
-    }
-
-    ~ggml_cuda_profiler_state() {
-        reset();
-    }
-
-    void record_start() {
-        cudaEvent_t ev;
-        (void) cudaEventCreate(&ev);
-        (void) cudaEventRecord(ev, stream);
-        start_events.push_back(ev);
-        cpu_timestamps.push_back(ggml_profiler_time_ns());
-        event_count++;
-    }
-
-    void record_end(const char * name, int backend_id, int split_id, uint64_t bytes, const char * extra,
-                    const ggml_tensor * node) {
-        cudaEvent_t ev;
-        (void) cudaEventCreate(&ev);
-        (void) cudaEventRecord(ev, stream);
-        end_events.push_back(ev);
-        record_event_indices.push_back(records.size());
-
-        ggml_profile_record rec;
-        rec.type = GGML_PROFILE_EVENT_OP;
-        rec.name = name;
-        rec.backend_id = backend_id;
-        rec.split_id = split_id;
-        rec.start_ns = 0;
-        rec.end_ns = 0;
-        rec.bytes = bytes;
-        rec.extra = extra;
-        ggml_profile_record_from_tensor(&rec, node);
-        records.push_back(rec);
-    }
-
-    void finalize() {
-        (void) cudaStreamSynchronize(stream);
-
-        for (int i = 0; i < (int)record_event_indices.size(); i++) {
-            float ms = 0.0f;
-            (void) cudaEventElapsedTime(&ms, start_events[i], end_events[i]);
-            uint64_t duration_ns = (uint64_t)(ms * 1e6f);
-            int rec_idx = record_event_indices[i];
-            // Use CPU-side timestamp for global ordering, GPU-measured duration for accuracy
-            records[rec_idx].start_ns = cpu_timestamps[i];
-            records[rec_idx].end_ns   = cpu_timestamps[i] + duration_ns;
-        }
-    }
-};
-
 #define GGML_LOG_WARN_ONCE(str) \
     { static std::once_flag warn_flag; std::call_once(warn_flag, []() { GGML_LOG_WARN(str); }); }
 
@@ -4117,23 +4030,8 @@ static void ggml_cuda_graph_evaluate_and_capture(ggml_backend_cuda_context * cud
 #else
                 GGML_UNUSED(integrated);
 #endif  // NDEBUG
-                if (cuda_ctx->profiler_state != nullptr && cuda_ctx->profiler_state->enabled) {
-                    cuda_ctx->profiler_state->record_start();
-                }
 
                 bool ok = ggml_cuda_compute_forward(*cuda_ctx, node);
-
-                if (cuda_ctx->profiler_state != nullptr && cuda_ctx->profiler_state->enabled) {
-                    cuda_ctx->profiler_state->record_end(
-                        ggml_op_name(node->op),
-                        -1,
-                        cuda_ctx->profiler_state->split_id,
-                        ggml_nbytes(node),
-                        nullptr,
-                        node
-                    );
-                }
-
                 if (!ok) {
                     GGML_LOG_ERROR("%s: op not supported %s (%s)\n", __func__, node->name, ggml_op_name(node->op));
                 }
@@ -4204,19 +4102,6 @@ static enum ggml_status ggml_backend_cuda_graph_compute(ggml_backend_t backend,
 
     ggml_cuda_set_device(cuda_ctx->device);
 
-    // Disable CUDA graphs when profiling (we need per-node timing)
-    bool was_graph_enabled = false;
-    if (cuda_ctx->profiler_state != nullptr && cuda_ctx->profiler_state->enabled) {
-#ifdef USE_CUDA_GRAPH
-        const void * graph_key = ggml_cuda_graph_get_key(cgraph);
-        ggml_cuda_graph * graph = cuda_ctx->cuda_graph(graph_key);
-        was_graph_enabled = graph->is_enabled();
-        if (was_graph_enabled) {
-            graph->disable_due_to_gpu_arch = true;
-        }
-#endif
-    }
-
     bool use_cuda_graph             = false;
     bool cuda_graph_update_required = false;
     const void * graph_key = nullptr;
@@ -4268,15 +4153,6 @@ static enum ggml_status ggml_backend_cuda_graph_compute(ggml_backend_t backend,
 
     ggml_cuda_graph_evaluate_and_capture(cuda_ctx, cgraph, use_cuda_graph, cuda_graph_update_required, graph_key);
 
-    // Restore CUDA graph enabled state after profiling
-    if (was_graph_enabled) {
-#ifdef USE_CUDA_GRAPH
-        const void * graph_key_prof = ggml_cuda_graph_get_key(cgraph);
-        ggml_cuda_graph * graph = cuda_ctx->cuda_graph(graph_key_prof);
-        graph->disable_due_to_gpu_arch = false;
-#endif
-    }
-
     return GGML_STATUS_SUCCESS;
 }
 
@@ -5537,68 +5413,12 @@ ggml_backend_t ggml_backend_cuda_init(int device) {
     }
 
     ggml_backend_t cuda_backend = new ggml_backend {
-        /* .guid     = */ ggml_backend_cuda_guid(),
-        /* .iface    = */ ggml_backend_cuda_interface,
-        /* .device   = */ ggml_backend_reg_dev_get(ggml_backend_cuda_reg(), device),
-        /* .context  = */ ctx,
-        /* .profiler = */ nullptr,
+        /* .guid    = */ ggml_backend_cuda_guid(),
+        /* .iface   = */ ggml_backend_cuda_interface,
+        /* .device  = */ ggml_backend_reg_dev_get(ggml_backend_cuda_reg(), device),
+        /* .context = */ ctx,
     };
 
-    // Register profiler
-    auto * prof_state = new ggml_cuda_profiler_state();
-    prof_state->init(ctx->stream());
-    ctx->profiler_state = prof_state;
-
-    static auto cuda_prof_enable = [](void * ctx, bool enable) {
-        auto * cuda_ctx = (ggml_backend_cuda_context *)ctx;
-        if (cuda_ctx->profiler_state != nullptr) {
-            cuda_ctx->profiler_state->enabled = enable;
-            if (!enable) {
-                cuda_ctx->profiler_state->reset();
-            }
-        }
-    };
-    static auto cuda_prof_reset = [](void * ctx) {
-        auto * cuda_ctx = (ggml_backend_cuda_context *)ctx;
-        if (cuda_ctx->profiler_state != nullptr) {
-            cuda_ctx->profiler_state->reset();
-            cuda_ctx->profiler_state->split_id = -1;
-        }
-    };
-    static auto cuda_prof_set_split_id = [](void * ctx, int split_id) {
-        auto * cuda_ctx = (ggml_backend_cuda_context *)ctx;
-        if (cuda_ctx->profiler_state != nullptr) {
-            cuda_ctx->profiler_state->split_id = split_id;
-        }
-    };
-    static auto cuda_prof_get_records = [](void * ctx, const ggml_profile_record ** out) -> int {
-        auto * cuda_ctx = (ggml_backend_cuda_context *)ctx;
-        if (cuda_ctx->profiler_state != nullptr) {
-            cuda_ctx->profiler_state->finalize();
-            *out = cuda_ctx->profiler_state->records.data();
-            return (int)cuda_ctx->profiler_state->records.size();
-        }
-        *out = nullptr;
-        return 0;
-    };
-    static auto cuda_prof_free = [](void * ctx) {
-        auto * cuda_ctx = (ggml_backend_cuda_context *)ctx;
-        if (cuda_ctx->profiler_state != nullptr) {
-            delete cuda_ctx->profiler_state;
-            cuda_ctx->profiler_state = nullptr;
-        }
-    };
-
-    auto * profiler = new ggml_backend_profiler {
-        /* .context      = */ ctx,
-        /* .enable       = */ cuda_prof_enable,
-        /* .reset        = */ cuda_prof_reset,
-        /* .set_split_id = */ cuda_prof_set_split_id,
-        /* .get_records  = */ cuda_prof_get_records,
-        /* .free_context = */ cuda_prof_free,
-    };
-    ggml_backend_set_profiler(cuda_backend, profiler);
-
     return cuda_backend;
 }
 

ggml/src/ggml-cuda/vendors/hip.h

@@ -6,10 +6,6 @@
 #include <hip/hip_fp16.h>
 #include <hip/hip_bf16.h>
 
-#if defined(GGML_HIP_ROCWMMA_FATTN)
-#include <rocwmma/rocwmma-version.hpp>
-#endif // defined(GGML_HIP_ROCWMMA_FATTN)
-
 #ifdef GGML_USE_NCCL
 #include <rccl/rccl.h>
 #endif // GGML_USE_NCCL
@@ -64,10 +60,8 @@
 #define cudaErrorMemoryAllocation hipErrorOutOfMemory
 #define cudaErrorPeerAccessAlreadyEnabled hipErrorPeerAccessAlreadyEnabled
 #define cudaErrorPeerAccessNotEnabled hipErrorPeerAccessNotEnabled
-#define cudaEventCreate hipEventCreate
 #define cudaEventCreateWithFlags hipEventCreateWithFlags
 #define cudaEventDisableTiming hipEventDisableTiming
-#define cudaEventElapsedTime hipEventElapsedTime
 #define cudaEventRecord hipEventRecord
 #define cudaEventSynchronize hipEventSynchronize
 #define cudaEvent_t hipEvent_t

ggml/src/ggml-cuda/vendors/musa.h

@@ -48,10 +48,8 @@
 #define cudaErrorMemoryAllocation musaErrorMemoryAllocation
 #define cudaErrorPeerAccessAlreadyEnabled musaErrorPeerAccessAlreadyEnabled
 #define cudaErrorPeerAccessNotEnabled musaErrorPeerAccessNotEnabled
-#define cudaEventCreate musaEventCreate
 #define cudaEventCreateWithFlags musaEventCreateWithFlags
 #define cudaEventDisableTiming musaEventDisableTiming
-#define cudaEventElapsedTime musaEventElapsedTime
 #define cudaEventRecord musaEventRecord
 #define cudaEventSynchronize musaEventSynchronize
 #define cudaEvent_t musaEvent_t

ggml/src/ggml-hexagon/ggml-hexagon.cpp

@@ -3281,6 +3281,35 @@ static bool ggml_hexagon_supported_ssm_conv(const struct ggml_hexagon_session *
     GGML_UNUSED(sess);
 }
 
+static bool ggml_hexagon_supported_im2col(const struct ggml_hexagon_session * sess, const struct ggml_tensor * op) {
+    const struct ggml_tensor * src1 = op->src[1];
+    const struct ggml_tensor * dst  = op;
+
+    const bool is_2D = ((const int32_t *) op->op_params)[6] == 1;
+    if (!is_2D) {
+        return false;
+    }
+
+    // For now support F32->F32 and F32->F16 only.
+    if (src1->type != GGML_TYPE_F32 || (dst->type != GGML_TYPE_F16 && dst->type != GGML_TYPE_F32)) {
+        return false;
+    }
+
+    if (!ggml_is_contiguous(src1) || !ggml_is_contiguous(dst)) {
+        return false;
+    }
+
+    // For now keep padded OPs on CPU. Will revisit once we expand coverage past patch-embed OPs.
+    const int32_t p0 = ((const int32_t *) op->op_params)[2];
+    const int32_t p1 = ((const int32_t *) op->op_params)[3];
+    if (p0 != 0 || p1 != 0) {
+        return false;
+    }
+
+    GGML_UNUSED(sess);
+    return true;
+}
+
 static bool ggml_hexagon_supported_pad(const struct ggml_hexagon_session * sess, const struct ggml_tensor * op) {
     const struct ggml_tensor * src0 = op->src[0];
     const struct ggml_tensor * dst  = op;
@@ -3430,6 +3459,7 @@ static htp_op_code op_remap_to_htp(const ggml_tensor * t) {
         case GGML_OP_SOLVE_TRI:       return HTP_OP_SOLVE_TRI;
         case GGML_OP_TRI:             return HTP_OP_TRI;
         case GGML_OP_PAD:             return HTP_OP_PAD;
+        case GGML_OP_IM2COL:          return HTP_OP_IM2COL;
 
         case GGML_OP_UNARY:
             switch (ggml_get_unary_op(t)) {
@@ -3860,7 +3890,6 @@ static ggml_backend_t ggml_backend_hexagon_device_init(ggml_backend_dev_t dev, c
         /* .interface = */ hexagon_backend_i,
         /* .device    = */ dev,
         /* .context   = */ sess,
-        /* .profiler  = */ nullptr,
     };
 
     GGML_UNUSED(params);
@@ -4153,6 +4182,10 @@ static bool ggml_backend_hexagon_device_supports_op(ggml_backend_dev_t dev, cons
             supp = ggml_hexagon_supported_ssm_conv(sess, op);
             break;
 
+        case GGML_OP_IM2COL:
+            supp = ggml_hexagon_supported_im2col(sess, op);
+            break;
+
         case GGML_OP_GATED_DELTA_NET:
             supp = ggml_hexagon_supported_gated_delta_net(sess, op);
             break;

ggml/src/ggml-hexagon/htp/CMakeLists.txt

@@ -42,6 +42,7 @@ add_library(${HTP_LIB} SHARED
     solve-tri-ops.c
     pad-ops.c
     argsort-ops.c
+    im2col-ops.c
 )
 
 target_compile_definitions(${HTP_LIB} PRIVATE

ggml/src/ggml-hexagon/htp/htp-ctx.h

@@ -140,5 +140,6 @@ int op_diag(struct htp_ops_context * octx);
 int op_solve_tri(struct htp_ops_context * octx);
 int op_gated_delta_net(struct htp_ops_context * octx);
 int op_pad(struct htp_ops_context * octx);
+int op_im2col(struct htp_ops_context * octx);
 
 #endif /* HTP_CTX_H */

ggml/src/ggml-hexagon/htp/htp-ops.h

@@ -98,6 +98,7 @@ enum htp_op_code {
     HTP_OP_NORM,
     HTP_OP_CONCAT,
     HTP_OP_CLAMP,
+    HTP_OP_IM2COL,
 
     HTP_OP_INVALID
 };

ggml/src/ggml-hexagon/htp/im2col-ops.c

@@ -0,0 +1,306 @@
+#pragma clang diagnostic ignored "-Wunused-variable"
+#pragma clang diagnostic ignored "-Wunused-function"
+#pragma clang diagnostic ignored "-Wunused-but-set-variable"
+
+#include <HAP_farf.h>
+#include <HAP_perf.h>
+#include <hexagon_protos.h>
+#include <hexagon_types.h>
+#include <string.h>
+
+#define GGML_COMMON_DECL_C
+#include "ggml-common.h"
+#include "htp-ctx.h"
+#include "htp-ops.h"
+#include "hvx-utils.h"
+#include "hex-dma.h"
+#include "hex-profile.h"
+#include "htp-vtcm.h"
+
+struct htp_im2col_context {
+    struct htp_ops_context * octx;
+    uint32_t                 npatches_per_thread;  // patches = N*OH*OW (pure-DDR kernel)
+
+    uint32_t pe_rows_per_thread;                   // N*OH rows per worker
+    uint32_t pe_src_row_bytes;                     // one output row's source: IC*KH*IW*4, rounded 256
+    uint32_t pe_dst_row_bytes;                     // one output row's dst: OW*patch_stride*2, rounded 256
+
+    // Patch-embed DMA path VTCM ping-pong.
+    uint8_t * pe_vtcm_src;                         // base of the 2x src buffers region
+    uint8_t * pe_vtcm_dst;                         // base of the 2x dst buffers region
+    uint32_t  pe_src_size_per_thread;              // 2 * pe_src_row_bytes
+    uint32_t  pe_dst_size_per_thread;              // 2 * pe_dst_row_bytes
+};
+
+// Per-op VTCM layout for the patch-embed DMA path
+struct htp_im2col_vtcm_layout {
+    size_t off_src;
+    size_t off_dst;
+    size_t src_bytes_per_thread;
+    size_t dst_bytes_per_thread;
+    size_t total_bytes;
+};
+
+static inline void htp_im2col_vtcm_layout_build(struct htp_im2col_vtcm_layout * L,
+                                                size_t                          src_row_bytes,
+                                                size_t                          dst_row_bytes,
+                                                uint32_t                        n_threads) {
+    L->src_bytes_per_thread = 2 * src_row_bytes;
+    L->dst_bytes_per_thread = 2 * dst_row_bytes;
+
+    L->off_src     = 0;
+    L->off_dst     = L->off_src + L->src_bytes_per_thread * n_threads;
+    L->total_bytes = L->off_dst + L->dst_bytes_per_thread * n_threads;
+}
+
+#define IM2COL_PATCHEMBED_BODY(FNAME, DST_CTYPE, COPY_FN, SPLAT_FN, DST_ELEM, TAG)                        \
+    static void FNAME(unsigned int nth, unsigned int ith, void * data) {                                  \
+        struct htp_im2col_context * ictx        = (struct htp_im2col_context *) data;                     \
+        struct htp_ops_context *    octx        = ictx->octx;                                             \
+        struct htp_thread_trace * restrict tr   = &octx->ctx->trace[ith];                                 \
+        const struct htp_tensor * restrict src1 = octx->src[1];                                           \
+        const struct htp_tensor * restrict dst  = octx->dst;                                              \
+        const int32_t  s0                       = octx->op_params[0];                                     \
+        const int32_t  s1                       = octx->op_params[1];                                     \
+        const int32_t  p0                       = octx->op_params[2];                                     \
+        const int32_t  p1                       = octx->op_params[3];                                     \
+        const int32_t  d0                       = octx->op_params[4];                                     \
+        const int32_t  d1                       = octx->op_params[5];                                     \
+        const uint32_t N                        = src1->ne[3];                                            \
+        const uint32_t IC                       = src1->ne[2];                                            \
+        const uint32_t IH                       = src1->ne[1];                                            \
+        const uint32_t IW                       = src1->ne[0];                                            \
+        const uint32_t KH                       = octx->src[0]->ne[1];                                    \
+        const uint32_t KW                       = octx->src[0]->ne[0];                                    \
+        const uint32_t OH                       = dst->ne[2];                                             \
+        const uint32_t OW                       = dst->ne[1];                                             \
+        const uint32_t patch_stride             = IC * KH * KW;                                           \
+        const float * restrict src_data         = (const float *) src1->data;                             \
+        DST_CTYPE * restrict dst_data           = (DST_CTYPE *) dst->data;                                \
+        const uint32_t npatches                 = N * OH * OW;                                            \
+        const uint32_t patch_start              = ictx->npatches_per_thread * ith;                        \
+        const uint32_t patch_end                = MIN(patch_start + ictx->npatches_per_thread, npatches); \
+        if (patch_start >= patch_end) {                                                                   \
+            return;                                                                                       \
+        }                                                                                                 \
+        htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, patch_start);                                   \
+        for (uint32_t p = patch_start; p < patch_end; p++) {                                              \
+            const uint32_t iow             = p % OW;                                                      \
+            const uint32_t ioh             = (p / OW) % OH;                                               \
+            const uint32_t in              = p / (OW * OH);                                               \
+            DST_CTYPE * restrict dst_patch = dst_data + (uint64_t) p * patch_stride;                      \
+            for (uint32_t iic = 0; iic < IC; iic++) {                                                     \
+                const float * restrict src_plane = src_data + ((uint64_t) in * IC + iic) * IH * IW;       \
+                for (uint32_t ikh = 0; ikh < KH; ikh++) {                                                 \
+                    const int32_t iih            = (int32_t) ioh * s1 + (int32_t) ikh * d1 - p1;          \
+                    DST_CTYPE * restrict out_run = dst_patch + iic * (KH * KW) + ikh * KW;                \
+                    if (iih < 0 || iih >= (int32_t) IH) {                                                 \
+                        SPLAT_FN(out_run, 0.0f, KW);                                                      \
+                        continue;                                                                         \
+                    }                                                                                     \
+                    const int32_t iiw0             = (int32_t) iow * s0 - p0;                             \
+                    const float * restrict src_run = src_plane + (uint64_t) iih * IW + iiw0;              \
+                    if (d0 == 1) {                                                                        \
+                        /* contiguous source run: [lo,hi) is in-bounds, tails are zero pad */             \
+                        const int32_t lo = iiw0 < 0 ? -iiw0 : 0;                                          \
+                        int32_t       hi = (int32_t) IW - iiw0;                                           \
+                        if (hi > (int32_t) KW) {                                                          \
+                            hi = (int32_t) KW;                                                            \
+                        }                                                                                 \
+                        if (hi <= lo) {                                                                   \
+                            SPLAT_FN(out_run, 0.0f, KW);                                                  \
+                        } else {                                                                          \
+                            if (lo > 0) {                                                                 \
+                                SPLAT_FN(out_run, 0.0f, (uint32_t) lo);                                   \
+                            }                                                                             \
+                            COPY_FN((uint8_t *) (out_run + lo), (const uint8_t *) (src_run + lo),         \
+                                    (uint32_t) (hi - lo));                                                \
+                            if (hi < (int32_t) KW) {                                                      \
+                                SPLAT_FN(out_run + hi, 0.0f, (KW - (uint32_t) hi));                       \
+                            }                                                                             \
+                        }                                                                                 \
+                        continue;                                                                         \
+                    }                                                                                     \
+                    for (uint32_t ikw = 0; ikw < KW; ikw++) {                                             \
+                        const int32_t iiw = (int32_t) iow * s0 + (int32_t) ikw * d0 - p0;                 \
+                        out_run[ikw]      = (iiw < 0 || iiw >= (int32_t) IW) ?                            \
+                                                (DST_CTYPE) 0.0f :                                        \
+                                                (DST_CTYPE) src_plane[(uint64_t) iih * IW + iiw];         \
+                    }                                                                                     \
+                }                                                                                         \
+            }                                                                                             \
+        }                                                                                                 \
+        htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, patch_start);                                    \
+    }
+
+IM2COL_PATCHEMBED_BODY(im2col_patchembed_thread, __fp16, hvx_copy_f16_f32_uu, hvx_splat_f16_u, sizeof(__fp16), "f32-f16")
+IM2COL_PATCHEMBED_BODY(im2col_patchembed_f32_thread, float, hvx_copy_f32_uu, hvx_splat_f32_u, sizeof(float), "f32-f32")
+
+#define IM2COL_PATCHEMBED_DMA_BODY(FNAME, DST_CTYPE, COPY_FN, SPLAT_FN, DST_ELEM, TAG)                               \
+    static void FNAME(unsigned int nth, unsigned int ith, void * data) {                                             \
+        struct htp_im2col_context * ictx        = (struct htp_im2col_context *) data;                                \
+        struct htp_ops_context *    octx        = ictx->octx;                                                        \
+        struct htp_thread_trace * restrict tr   = &octx->ctx->trace[ith];                                            \
+        const struct htp_tensor * restrict src1 = octx->src[1];                                                      \
+        const struct htp_tensor * restrict dst  = octx->dst;                                                         \
+        const uint32_t N = src1->ne[3], IC = src1->ne[2], IH = src1->ne[1], IW = src1->ne[0];                        \
+        const uint32_t KH = octx->src[0]->ne[1], KW = octx->src[0]->ne[0];                                           \
+        const uint32_t OH = dst->ne[2], OW = dst->ne[1];                                                             \
+        const uint32_t patch_stride     = IC * KH * KW;                                                              \
+        const float * restrict src_data = (const float *) src1->data;                                                \
+        DST_CTYPE * restrict dst_data   = (DST_CTYPE *) dst->data;                                                   \
+        dma_queue *    dmaq             = octx->ctx->dma[ith];                                                       \
+        uint8_t *      src_base         = ictx->pe_vtcm_src + ith * ictx->pe_src_size_per_thread;                    \
+        uint8_t *      dst_base         = ictx->pe_vtcm_dst + ith * ictx->pe_dst_size_per_thread;                    \
+        float *        srcb             = (float *) src_base;                                                        \
+        DST_CTYPE *    dstb             = (DST_CTYPE *) dst_base;                                                    \
+        const uint32_t nrows            = N * OH;                                                                    \
+        const uint32_t per_thread       = ictx->pe_rows_per_thread;                                                  \
+        const uint32_t row_start        = per_thread * ith;                                                          \
+        const uint32_t row_end          = MIN(row_start + per_thread, nrows);                                        \
+        if (row_start >= row_end)                                                                                    \
+            return;                                                                                                  \
+        for (uint32_t r = row_start; r < row_end; r++) {                                                             \
+            const uint32_t in  = r / OH;                                                                             \
+            const uint32_t ioh = r % OH;                                                                             \
+            for (uint32_t ikh = 0; ikh < KH; ikh++) {                                                                \
+                int32_t iih = (int32_t) ioh * (int32_t) KH + (int32_t) ikh;                                          \
+                int     ok  = (iih >= 0 && iih < (int32_t) IH);                                                      \
+                for (uint32_t iic = 0; iic < IC; iic++) {                                                            \
+                    float *       vdst = srcb + ((uint64_t) (iic * KH + ikh)) * IW;                                  \
+                    const float * _vsrc =                                                                            \
+                        ok ? (src_data + ((uint64_t) (in * IC + iic) * IH + iih) * IW) : (const float *) vdst;       \
+                    dma_queue_push_ddr_to_vtcm(                                                                      \
+                        dmaq, dma_make_ptr((uint8_t *) vdst, ok ? (const uint8_t *) _vsrc : (const uint8_t *) vdst), \
+                        IW * sizeof(float), IW * sizeof(float), ok ? 1 : 0);                                         \
+                }                                                                                                    \
+            }                                                                                                        \
+            for (uint32_t i = 0; i < IC * KH; i++)                                                                   \
+                dma_queue_pop(dmaq);                                                                                 \
+            htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, r);                                                    \
+            for (uint32_t iow = 0; iow < OW; iow++) {                                                                \
+                DST_CTYPE * dst_patch = dstb + (uint64_t) iow * patch_stride;                                        \
+                for (uint32_t ikh = 0; ikh < KH; ikh++) {                                                            \
+                    int32_t iih = (int32_t) ioh * (int32_t) KH + (int32_t) ikh;                                      \
+                    for (uint32_t iic = 0; iic < IC; iic++) {                                                        \
+                        DST_CTYPE * out_run = dst_patch + iic * (KH * KW) + ikh * KW;                                \
+                        if (iih < 0 || iih >= (int32_t) IH) {                                                        \
+                            SPLAT_FN(out_run, 0.0f, KW);                                                             \
+                            continue;                                                                                \
+                        }                                                                                            \
+                        const float * src_run = srcb + ((uint64_t) (iic * KH + ikh)) * IW + (uint64_t) iow * KW;     \
+                        COPY_FN((uint8_t *) out_run, (const uint8_t *) src_run, KW);                                 \
+                    }                                                                                                \
+                }                                                                                                    \
+            }                                                                                                        \
+            htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, r);                                                     \
+            DST_CTYPE * ddr_row = dst_data + ((uint64_t) (in * OH + ioh) * OW) * patch_stride;                       \
+            dma_queue_push_vtcm_to_ddr(dmaq, dma_make_ptr((uint8_t *) ddr_row, (uint8_t *) dstb),                    \
+                                       OW * patch_stride * (DST_ELEM), OW * patch_stride * (DST_ELEM), 1);           \
+            dma_queue_flush(dmaq);                                                                                   \
+        }                                                                                                            \
+    }
+
+IM2COL_PATCHEMBED_DMA_BODY(im2col_patchembed_dma_thread,     __fp16, hvx_copy_f16_f32_uu, hvx_splat_f16_u, sizeof(__fp16), "pe-dma-f16")
+IM2COL_PATCHEMBED_DMA_BODY(im2col_patchembed_dma_f32_thread, float,  hvx_copy_f32_uu,     hvx_splat_f32_u, sizeof(float),  "pe-dma-f32")
+
+static bool im2col_use_patchembed_dma(const struct htp_ops_context * octx) {
+    const int32_t s0 = octx->op_params[0], s1 = octx->op_params[1];
+    const int32_t p0 = octx->op_params[2], p1 = octx->op_params[3];
+    const int32_t d0 = octx->op_params[4], d1 = octx->op_params[5];
+    const int     is_2D = octx->op_params[6] == 1;
+    if (!is_2D) {
+        return false;
+    }
+    if (octx->dst->type != HTP_TYPE_F16 && octx->dst->type != HTP_TYPE_F32) {
+        return false;
+    }
+    const uint32_t KH = octx->src[0]->ne[1], KW = octx->src[0]->ne[0];
+    if (s0 != (int32_t) KW || s1 != (int32_t) KH) {
+        return false;  // non-overlapping
+    }
+    if (p0 != 0 || p1 != 0) {
+        return false;  // no padding
+    }
+    if (d0 != 1 || d1 != 1) {
+        return false;  // no dilation
+    }
+    return true;
+}
+
+// Sizes the per-thread 2x(src,dst) VTCM ping-pong for the patch-embed DMA path.
+// Returns false if it doesn't fit the VTCM budget (caller falls back).
+static bool im2col_patchembed_dma_fits(struct htp_ops_context *    octx,
+                                       struct htp_im2col_context * ictx,
+                                       uint32_t                    n_threads) {
+    const uint32_t IC = octx->src[1]->ne[2], IW = octx->src[1]->ne[0];
+    const uint32_t KH = octx->src[0]->ne[1], KW = octx->src[0]->ne[0];
+    const uint32_t OW           = octx->dst->ne[1];
+    const uint32_t patch_stride = IC * KH * KW;
+
+    ictx->pe_src_row_bytes  = hex_round_up(IC * KH * IW * sizeof(float), 256);
+    const uint32_t dst_elem = (octx->dst->type == HTP_TYPE_F16) ? sizeof(__fp16) : sizeof(float);
+    ictx->pe_dst_row_bytes  = hex_round_up(OW * patch_stride * dst_elem, 256);
+
+    // 2 src + 2 dst buffers per thread (ping-pong), src region first then dst.
+    struct htp_im2col_vtcm_layout L;
+    htp_im2col_vtcm_layout_build(&L, ictx->pe_src_row_bytes, ictx->pe_dst_row_bytes, n_threads);
+    if (L.total_bytes > octx->ctx->vtcm_size) {
+        return false;
+    }
+
+    uint8_t * const base        = octx->ctx->vtcm_base;
+    ictx->pe_vtcm_src           = VTCM_LAYOUT_PTR(uint8_t, base, L.off_src);
+    ictx->pe_vtcm_dst           = VTCM_LAYOUT_PTR(uint8_t, base, L.off_dst);
+    ictx->pe_src_size_per_thread = (uint32_t) L.src_bytes_per_thread;
+    ictx->pe_dst_size_per_thread = (uint32_t) L.dst_bytes_per_thread;
+    return true;
+}
+
+int op_im2col(struct htp_ops_context * octx) {
+    const struct htp_tensor * src1 = octx->src[1];
+    const struct htp_tensor * dst  = octx->dst;
+
+    if (src1->type != HTP_TYPE_F32 || (dst->type != HTP_TYPE_F16 && dst->type != HTP_TYPE_F32)) {
+        FARF(ERROR, "im2col: only (F32 image -> F16/F32 columns) supported");
+        return HTP_STATUS_NO_SUPPORT;
+    }
+
+    const uint32_t N         = src1->ne[3];
+    const uint32_t OH        = dst->ne[2];
+    const uint32_t OW        = dst->ne[1];
+    const uint32_t npatches  = N * OH * OW;
+    const uint32_t n_threads = MIN(octx->n_threads, npatches);
+
+    if ((octx->flags & HTP_OPFLAGS_SKIP_COMPUTE) || n_threads == 0) {
+        return HTP_STATUS_OK;
+    }
+
+    struct htp_im2col_context ictx = { 0 };
+    ictx.octx                      = octx;
+    ictx.npatches_per_thread       = (npatches + n_threads - 1) / n_threads;
+
+    // Clean non-overlapping patch-embed -> DMA kernel (if it fits VTCM);
+    // everything else (padding/dilation/stride edges) -> pure-DDR kernel.
+    if (im2col_use_patchembed_dma(octx)) {
+        const uint32_t nrows = N * OH;
+        const uint32_t pth   = MIN(octx->n_threads, nrows);
+        if (pth > 0 && im2col_patchembed_dma_fits(octx, &ictx, pth)) {
+            ictx.pe_rows_per_thread = (nrows + pth - 1) / pth;
+            if (dst->type == HTP_TYPE_F16) {
+                work_queue_run(octx->ctx->work_queue, im2col_patchembed_dma_thread, &ictx, pth);
+            } else {
+                work_queue_run(octx->ctx->work_queue, im2col_patchembed_dma_f32_thread, &ictx, pth);
+            }
+            return HTP_STATUS_OK;
+        }
+        // else: doesn't fit -> fall through to the pure-DDR kernel below.
+    }
+
+    if (dst->type == HTP_TYPE_F16) {
+        work_queue_run(octx->ctx->work_queue, im2col_patchembed_thread, &ictx, n_threads);
+    } else {
+        work_queue_run(octx->ctx->work_queue, im2col_patchembed_f32_thread, &ictx, n_threads);
+    }
+    return HTP_STATUS_OK;
+}

ggml/src/ggml-hexagon/htp/main.c

@@ -781,6 +781,9 @@ static int execute_op(struct htp_ops_context * octx) {
         case HTP_OP_PAD:
             return op_pad(octx);
 
+        case HTP_OP_IM2COL:
+            return op_im2col(octx);
+
         case HTP_OP_CONCAT:
             return op_concat(octx);
 

ggml/src/ggml-hip/CMakeLists.txt

@@ -114,10 +114,6 @@ if (GGML_HIP_NO_VMM)
     add_compile_definitions(GGML_HIP_NO_VMM)
 endif()
 
-if (GGML_HIP_ROCWMMA_FATTN)
-    add_compile_definitions(GGML_HIP_ROCWMMA_FATTN)
-endif()
-
 if (NOT GGML_HIP_MMQ_MFMA)
     add_compile_definitions(GGML_HIP_NO_MMQ_MFMA)
 endif()
@@ -158,5 +154,3 @@ if (GGML_HIP_RCCL)
 endif()
 
 target_link_libraries(ggml-hip PRIVATE ggml-base hip::host roc::rocblas roc::hipblas)
-
-target_compile_options(ggml-hip PRIVATE "$<$<COMPILE_LANGUAGE:HIP>:-ffast-math;-fno-finite-math-only>")

ggml/src/ggml-metal/ggml-metal-context.h

@@ -36,32 +36,6 @@ void ggml_metal_set_abort_callback  (ggml_metal_t ctx, ggml_abort_callback abort
 bool ggml_metal_supports_family     (ggml_metal_t ctx, int family);
 void ggml_metal_capture_next_compute(ggml_metal_t ctx);
 
-//
-// Profiling
-//
-
-// Opaque profiler state, owned by the C++ backend layer (ggml-metal.cpp). Holds the std::vector of
-// records returned via the ggml_backend_profiler interface. ggml_metal_t keeps a borrowed pointer
-// so that graph_compute can push records when sampling is active.
-struct ggml_metal_profiler_state;
-
-// Inject (or clear, with NULL) the profiler state pointer. Called once at backend init.
-void ggml_metal_set_profiler_state(ggml_metal_t ctx, struct ggml_metal_profiler_state * state);
-
-// Bridge function implemented in ggml-metal.cpp. Used by graph_compute (in .m) to push records.
-void ggml_metal_profiler_push_record(
-        struct ggml_metal_profiler_state * state,
-        const struct ggml_tensor * node,
-        uint64_t start_ns,
-        uint64_t end_ns);
-
-// Query whether the injected profiler state is currently enabled.
-// (Avoids exposing the C++ struct layout to the .m file.)
-bool ggml_metal_profiler_is_enabled(struct ggml_metal_profiler_state * state);
-
-// Query the split-id currently set on the profiler state.
-int  ggml_metal_profiler_get_split_id(struct ggml_metal_profiler_state * state);
-
 #ifdef __cplusplus
 }
 #endif

ggml/src/ggml-metal/ggml-metal-context.m

@@ -6,7 +6,6 @@
 #import "ggml-metal-impl.h"
 #import "ggml-metal-common.h"
 #import "ggml-metal-ops.h"
-#import "ggml-profiler.h"
 
 #import <Foundation/Foundation.h>
 
@@ -80,19 +79,6 @@ struct ggml_metal {
     // error state - set when a command buffer fails during synchronize
     // once set, graph_compute will return GGML_STATUS_FAILED until the backend is recreated
     bool has_error;
-
-    // Profiling
-    // Borrowed; owned by the C++ backend layer (ggml-metal.cpp). NULL when profiling is unavailable.
-    struct ggml_metal_profiler_state * profiler_state;
-
-    // Per-graph-compute scratch state populated when profiling is active for the current invocation.
-    // Lifetime: from start of ggml_metal_graph_compute until its end.
-    bool                              profiler_active;
-    ggml_metal_sample_buf_t           profiler_sample_buf;
-    size_t                            profiler_total_slots;
-    uint64_t                          profiler_cpu_anchor_ns;
-    uint64_t                          profiler_gpu_anchor_ns;
-    struct ggml_metal_op_sample_slot * profiler_slot_map;  // size = gf->n_nodes
 };
 
 ggml_metal_t ggml_metal_init(ggml_metal_device_t dev) {
@@ -464,34 +450,6 @@ enum ggml_status ggml_metal_graph_compute(ggml_metal_t ctx, struct ggml_cgraph *
     // keep the memory wired
     ggml_metal_device_rsets_keep_alive(ctx->dev);
 
-    // Decide whether profiling is active for this invocation. Activation is sticky for the whole
-    // call: allocate sample buffer + slot map up front so the encode_async block can use them.
-    ctx->profiler_active = false;
-    if (ctx->profiler_state != NULL && ggml_metal_profiler_is_enabled(ctx->profiler_state)) {
-        if (ggml_metal_device_supports_profiling(ctx->dev) && gf->n_nodes > 0) {
-            const size_t total_slots = 2 * (size_t) gf->n_nodes;
-            ctx->profiler_sample_buf = ggml_metal_device_create_sample_buf(ctx->dev, total_slots);
-            if (ctx->profiler_sample_buf != NULL) {
-                ctx->profiler_total_slots = total_slots;
-                ctx->profiler_slot_map = (struct ggml_metal_op_sample_slot *) calloc(
-                    (size_t) gf->n_nodes, sizeof(struct ggml_metal_op_sample_slot));
-                if (ctx->profiler_slot_map != NULL) {
-                    // Mark all entries unused (node_idx < 0).
-                    for (int i = 0; i < gf->n_nodes; ++i) {
-                        ctx->profiler_slot_map[i].node_idx = -1;
-                    }
-                    ggml_metal_device_sample_timestamps(ctx->dev,
-                                                       &ctx->profiler_cpu_anchor_ns,
-                                                       &ctx->profiler_gpu_anchor_ns);
-                    ctx->profiler_active = true;
-                } else {
-                    ggml_metal_sample_buf_free(ctx->profiler_sample_buf);
-                    ctx->profiler_sample_buf = NULL;
-                }
-            }
-        }
-    }
-
     // submit the ggml compute graph to the GPU by creating command buffers and encoding the ops in them
     // the first n_nodes_0 are encoded and submitted for processing directly by the calling thread
     // while these nodes are processing, we start n_cb threads to enqueue the rest of the nodes
@@ -651,66 +609,6 @@ enum ggml_status ggml_metal_graph_compute(ggml_metal_t ctx, struct ggml_cgraph *
 
             ctx->capture_started = false;
         }
-
-        // Profiling drain: wait for all command buffers to complete, resolve timestamps, push records.
-        // This forces a synchronous wait — Vulkan does the same when its profiler is active.
-        if (ctx->profiler_active) {
-            {
-                id<MTLCommandBuffer> cmd_buf = ctx->cmd_bufs[n_cb].obj;
-                if (cmd_buf) {
-                    [cmd_buf waitUntilCompleted];
-                }
-            }
-            for (int i = 0; i < n_cb; ++i) {
-                id<MTLCommandBuffer> cmd_buf = ctx->cmd_bufs[i].obj;
-                if (cmd_buf) {
-                    // Ensure cmd_bufs that were not auto-enqueued get committed.
-                    if ([cmd_buf status] == MTLCommandBufferStatusNotEnqueued) {
-                        [cmd_buf commit];
-                    }
-                    [cmd_buf waitUntilCompleted];
-                }
-            }
-
-            uint64_t * ns = (uint64_t *) calloc(ctx->profiler_total_slots, sizeof(uint64_t));
-            if (ns != NULL) {
-                ggml_metal_sample_buf_resolve(ctx->profiler_sample_buf,
-                                              /*base=*/0,
-                                              ctx->profiler_total_slots,
-                                              ctx->profiler_cpu_anchor_ns,
-                                              ctx->profiler_gpu_anchor_ns,
-                                              ns);
-
-                for (int i = 0; i < gf->n_nodes; ++i) {
-                    const struct ggml_metal_op_sample_slot * slot = &ctx->profiler_slot_map[i];
-                    if (slot->node_idx < 0) {
-                        continue;
-                    }
-                    if (slot->slot_start >= ctx->profiler_total_slots ||
-                        slot->slot_end   >= ctx->profiler_total_slots) {
-                        continue;
-                    }
-                    const uint64_t t0 = ns[slot->slot_start];
-                    const uint64_t t1 = ns[slot->slot_end];
-                    if (t0 == 0 || t1 == 0 || t1 < t0) {
-                        continue;
-                    }
-                    ggml_metal_profiler_push_record(ctx->profiler_state,
-                                                    ggml_graph_node(gf, slot->node_idx),
-                                                    t0,
-                                                    t1);
-                }
-
-                free(ns);
-            }
-
-            free(ctx->profiler_slot_map);
-            ctx->profiler_slot_map = NULL;
-            ggml_metal_sample_buf_free(ctx->profiler_sample_buf);
-            ctx->profiler_sample_buf = NULL;
-            ctx->profiler_total_slots = 0;
-            ctx->profiler_active = false;
-        }
     }
 
     return GGML_STATUS_SUCCESS;
@@ -762,10 +660,6 @@ ggml_metal_event_t ggml_metal_get_ev_cpy(ggml_metal_t ctx) {
     return ctx->ev_cpy;
 }
 
-void ggml_metal_set_profiler_state(ggml_metal_t ctx, struct ggml_metal_profiler_state * state) {
-    ctx->profiler_state = state;
-}
-
 void ggml_metal_set_n_cb(ggml_metal_t ctx, int n_cb) {
     if (ctx->n_cb != n_cb) {
         ctx->n_cb = MIN(n_cb, GGML_METAL_MAX_COMMAND_BUFFERS);
@@ -810,17 +704,6 @@ void ggml_metal_set_n_cb(ggml_metal_t ctx, int n_cb) {
             ctx->debug_graph,
             ctx->debug_fusion);
 
-        if (ctx->profiler_active) {
-            // Base slot for this command buffer is 2 * (first graph-node index it processes).
-            // The encoder uses one (start, end) pair per encoded op group; we over-allocate so
-            // that empty/filtered nodes simply leave gaps in the sample buffer.
-            const size_t base_slot = 2 * (size_t) idx_start;
-            ggml_metal_op_enable_profiling(ctx_op,
-                                           ctx->profiler_sample_buf,
-                                           base_slot,
-                                           ctx->profiler_slot_map);
-        }
-
         for (int idx = 0; idx < ggml_metal_op_n_nodes(ctx_op); ++idx) {
             const int res = ggml_metal_op_encode(ctx_op, idx);
             if (res == 0) {

ggml/src/ggml-metal/ggml-metal-device.cpp

@@ -1252,6 +1252,21 @@ ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_argsort_merge(gg
     return res;
 }
 
+ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_fwht(ggml_metal_library_t lib, int n) {
+    char base[256];
+    char name[256];
+
+    snprintf(base, 256, "kernel_fwht_f32_%d", n);
+    snprintf(name, 256, "%s", base);
+
+    ggml_metal_pipeline_with_params res = ggml_metal_library_get_pipeline(lib, name);
+    if (!res.pipeline) {
+        res = ggml_metal_library_compile_pipeline(lib, base, name, nullptr);
+    }
+
+    return res;
+}
+
 // note: reuse the argsort kernel for top_k
 ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_top_k(ggml_metal_library_t lib, const ggml_tensor * op) {
     assert(op->op == GGML_OP_TOP_K);

ggml/src/ggml-metal/ggml-metal-device.h

@@ -91,17 +91,6 @@ void ggml_metal_encoder_memory_barrier(ggml_metal_encoder_t encoder);
 
 void ggml_metal_encoder_end_encoding(ggml_metal_encoder_t encoder);
 
-//
-// MTLCounterSampleBuffer wrapper (used by the profiler)
-//
-
-typedef struct ggml_metal_sample_buf * ggml_metal_sample_buf_t;
-
-// Insert a GPU timestamp sample on the encoder at the given slot.
-// Caller must ensure (a) the encoder belongs to a command buffer using a counter sample buffer that
-// supports MTLCounterSamplingPointAtDispatchBoundary, and (b) `index` is unique within the buffer.
-void ggml_metal_encoder_sample_timestamp(ggml_metal_encoder_t encoder, ggml_metal_sample_buf_t buf, size_t index);
-
 //
 // MTLLibrary wrapper
 //
@@ -150,6 +139,7 @@ struct ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_mul_mv_id
 struct ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_argmax            (ggml_metal_library_t lib, const struct ggml_tensor * op);
 struct ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_argsort           (ggml_metal_library_t lib, const struct ggml_tensor * op);
 struct ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_argsort_merge     (ggml_metal_library_t lib, const struct ggml_tensor * op);
+struct ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_fwht              (ggml_metal_library_t lib, int n);
 struct ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_top_k             (ggml_metal_library_t lib, const struct ggml_tensor * op);
 struct ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_top_k_merge       (ggml_metal_library_t lib, const struct ggml_tensor * op);
 struct ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_bin               (ggml_metal_library_t lib, const struct ggml_tensor * op, int32_t n_fuse );
@@ -307,35 +297,6 @@ bool ggml_metal_device_supports_op(ggml_metal_device_t dev, const struct ggml_te
 
 const struct ggml_metal_device_props * ggml_metal_device_get_props(ggml_metal_device_t dev);
 
-//
-// Profiling helpers
-//
-
-// Returns true if the device supports MTLCounterSamplingPointAtDispatchBoundary on compute encoders
-// AND exposes the MTLCommonCounterSetTimestamp counter set.
-bool ggml_metal_device_supports_profiling(ggml_metal_device_t dev);
-
-// Allocate a counter sample buffer with `sample_count` slots backed by the timestamp counter set.
-// Returns NULL on failure (e.g. unsupported device).
-ggml_metal_sample_buf_t ggml_metal_device_create_sample_buf(ggml_metal_device_t dev, size_t sample_count);
-
-void ggml_metal_sample_buf_free(ggml_metal_sample_buf_t buf);
-
-// Capture correlated CPU/GPU timestamps (both in Mach-absolute units, i.e. nanoseconds on
-// current Apple Silicon; the conversion via mach_timebase_info is applied internally).
-// Used to anchor the GPU timestamp domain to the CPU clock returned by ggml_profiler_time_ns().
-void ggml_metal_device_sample_timestamps(ggml_metal_device_t dev, uint64_t * cpu_ns, uint64_t * gpu_ns);
-
-// Resolve `count` consecutive samples starting at `base` into nanosecond timestamps anchored against
-// `cpu_anchor_ns` / `gpu_anchor_ns` (obtained from ggml_metal_device_sample_timestamps).
-// `out_ns` must hold at least `count` uint64_t entries.
-void ggml_metal_sample_buf_resolve(ggml_metal_sample_buf_t buf,
-                                   size_t   base,
-                                   size_t   count,
-                                   uint64_t cpu_anchor_ns,
-                                   uint64_t gpu_anchor_ns,
-                                   uint64_t * out_ns);
-
 //
 // device buffers
 //

ggml/src/ggml-metal/ggml-metal-device.m

@@ -8,7 +8,6 @@
 #include <Metal/Metal.h>
 
 #include <stdatomic.h>
-#include <mach/mach_time.h>
 
 #ifndef TARGET_OS_VISION
 #define TARGET_OS_VISION 0
@@ -518,21 +517,6 @@ void ggml_metal_encoder_end_encoding(ggml_metal_encoder_t encoder) {
     [encoder->obj endEncoding];
 }
 
-//
-// MTLCounterSampleBuffer wrapper
-//
-
-struct ggml_metal_sample_buf {
-    id<MTLCounterSampleBuffer> obj;
-    size_t sample_count;
-};
-
-void ggml_metal_encoder_sample_timestamp(ggml_metal_encoder_t encoder, ggml_metal_sample_buf_t buf, size_t index) {
-    if (@available(macOS 11.0, iOS 14.0, *)) {
-        [encoder->obj sampleCountersInBuffer:buf->obj atSampleIndex:index withBarrier:YES];
-    }
-}
-
 struct ggml_metal_device {
     id<MTLDevice> mtl_device;
 
@@ -1394,139 +1378,6 @@ const struct ggml_metal_device_props * ggml_metal_device_get_props(ggml_metal_de
     return &dev->props;
 }
 
-//
-// Profiling helpers
-//
-
-// Look up the MTLCommonCounterSetTimestamp counter set, or nil if the device doesn't expose it.
-static id<MTLCounterSet> ggml_metal_device_get_timestamp_counter_set(ggml_metal_device_t dev) {
-    if (@available(macOS 10.15, iOS 14.0, *)) {
-        NSArray<id<MTLCounterSet>> * sets = [dev->mtl_device counterSets];
-        for (id<MTLCounterSet> cs in sets) {
-            if ([[cs name] isEqualToString:MTLCommonCounterSetTimestamp]) {
-                return cs;
-            }
-        }
-    }
-    return nil;
-}
-
-bool ggml_metal_device_supports_profiling(ggml_metal_device_t dev) {
-    if (@available(macOS 11.0, iOS 14.0, *)) {
-        if (![dev->mtl_device supportsCounterSampling:MTLCounterSamplingPointAtDispatchBoundary]) {
-            return false;
-        }
-        return ggml_metal_device_get_timestamp_counter_set(dev) != nil;
-    }
-    return false;
-}
-
-ggml_metal_sample_buf_t ggml_metal_device_create_sample_buf(ggml_metal_device_t dev, size_t sample_count) {
-    if (sample_count == 0) {
-        return NULL;
-    }
-    if (@available(macOS 10.15, iOS 14.0, *)) {
-        id<MTLCounterSet> cs = ggml_metal_device_get_timestamp_counter_set(dev);
-        if (cs == nil) {
-            return NULL;
-        }
-
-        MTLCounterSampleBufferDescriptor * desc = [[MTLCounterSampleBufferDescriptor alloc] init];
-        desc.counterSet = cs;
-        desc.label = @"ggml-metal-profiler";
-        desc.storageMode = MTLStorageModeShared;
-        desc.sampleCount = sample_count;
-
-        NSError * error = nil;
-        id<MTLCounterSampleBuffer> sb = [dev->mtl_device newCounterSampleBufferWithDescriptor:desc error:&error];
-        [desc release];
-
-        if (sb == nil) {
-            GGML_LOG_WARN("%s: failed to create counter sample buffer: %s\n", __func__,
-                          error ? [[error localizedDescription] UTF8String] : "unknown");
-            return NULL;
-        }
-
-        ggml_metal_sample_buf_t res = calloc(1, sizeof(struct ggml_metal_sample_buf));
-        res->obj = sb;
-        res->sample_count = sample_count;
-        return res;
-    }
-    return NULL;
-}
-
-void ggml_metal_sample_buf_free(ggml_metal_sample_buf_t buf) {
-    if (buf == NULL) {
-        return;
-    }
-    [buf->obj release];
-    free(buf);
-}
-
-void ggml_metal_device_sample_timestamps(ggml_metal_device_t dev, uint64_t * cpu_ns, uint64_t * gpu_ns) {
-    if (@available(macOS 10.15, iOS 14.0, *)) {
-        MTLTimestamp cpu_t = 0;
-        MTLTimestamp gpu_t = 0;
-        [dev->mtl_device sampleTimestamps:&cpu_t gpuTimestamp:&gpu_t];
-
-        // Apple docs: both timestamps are in Mach-absolute time units. Convert to nanoseconds.
-        static mach_timebase_info_data_t tb = {0, 0};
-        if (tb.denom == 0) {
-            mach_timebase_info(&tb);
-        }
-        if (cpu_ns) *cpu_ns = (uint64_t)((__uint128_t)cpu_t * tb.numer / tb.denom);
-        if (gpu_ns) *gpu_ns = (uint64_t)((__uint128_t)gpu_t * tb.numer / tb.denom);
-    } else {
-        if (cpu_ns) *cpu_ns = 0;
-        if (gpu_ns) *gpu_ns = 0;
-    }
-}
-
-void ggml_metal_sample_buf_resolve(ggml_metal_sample_buf_t buf,
-                                   size_t   base,
-                                   size_t   count,
-                                   uint64_t cpu_anchor_ns,
-                                   uint64_t gpu_anchor_ns,
-                                   uint64_t * out_ns) {
-    if (buf == NULL || count == 0 || out_ns == NULL) {
-        return;
-    }
-    if (@available(macOS 10.15, iOS 14.0, *)) {
-        NSRange range = NSMakeRange(base, count);
-        NSData * data = [buf->obj resolveCounterRange:range];
-        if (data == nil || [data length] < count * sizeof(MTLCounterResultTimestamp)) {
-            // Failed resolve: fill with zeros so the caller can detect dropped samples.
-            for (size_t i = 0; i < count; ++i) {
-                out_ns[i] = 0;
-            }
-            return;
-        }
-
-        static mach_timebase_info_data_t tb = {0, 0};
-        if (tb.denom == 0) {
-            mach_timebase_info(&tb);
-        }
-
-        const MTLCounterResultTimestamp * src = (const MTLCounterResultTimestamp *) [data bytes];
-        for (size_t i = 0; i < count; ++i) {
-            // MTLCounterErrorValue (~0ULL) marks an unrecorded sample slot; map to 0.
-            if (src[i].timestamp == MTLCounterErrorValue) {
-                out_ns[i] = 0;
-                continue;
-            }
-            const uint64_t gpu_ns = (uint64_t)((__uint128_t) src[i].timestamp * tb.numer / tb.denom);
-            // Anchor to CPU clock returned by ggml_profiler_time_ns().
-            out_ns[i] = (gpu_ns >= gpu_anchor_ns)
-                      ? cpu_anchor_ns + (gpu_ns - gpu_anchor_ns)
-                      : cpu_anchor_ns - (gpu_anchor_ns - gpu_ns);
-        }
-    } else {
-        for (size_t i = 0; i < count; ++i) {
-            out_ns[i] = 0;
-        }
-    }
-}
-
 //
 // device buffers
 //

ggml/src/ggml-metal/ggml-metal-impl.h

@@ -1157,6 +1157,10 @@ typedef struct {
     int32_t  len;
 } ggml_metal_kargs_argsort_merge;
 
+typedef struct {
+    int32_t nrows;
+} ggml_metal_kargs_fwht;
+
 typedef struct {
     int64_t  ne0;
     float    start;

ggml/src/ggml-metal/ggml-metal-ops.cpp

@@ -77,11 +77,6 @@ struct ggml_metal_op {
         return ggml_graph_node(gf, idxs[i]);
     }
 
-    int node_global_idx(int i) const {
-        assert(i >= 0 && i < (int) idxs.size());
-        return idxs[i];
-    }
-
     bool can_fuse(int i0, const ggml_op * ops, int n_ops) const {
         assert(use_fusion);
         assert(i0 >= 0 && i0 < n_nodes());
@@ -105,12 +100,6 @@ struct ggml_metal_op {
     int debug_graph;
     int debug_fusion;
 
-    // Profiling: when sample_buf is non-null, ggml_metal_op_encode brackets each impl call with
-    // two timestamp samples. The (node_idx, slot_start, slot_end) tuple is recorded in slot_map.
-    ggml_metal_sample_buf_t sample_buf      = nullptr;
-    size_t                  next_slot       = 0;
-    ggml_metal_op_sample_slot * slot_map    = nullptr;
-
 private:
     ggml_cgraph * gf;
 
@@ -155,16 +144,6 @@ int ggml_metal_op_n_nodes(ggml_metal_op_t ctx) {
     return ctx->n_nodes();
 }
 
-void ggml_metal_op_enable_profiling(
-        ggml_metal_op_t ctx,
-        ggml_metal_sample_buf_t sample_buf,
-        size_t slot_base,
-        ggml_metal_op_sample_slot * slot_map) {
-    ctx->sample_buf = sample_buf;
-    ctx->next_slot  = slot_base;
-    ctx->slot_map   = slot_map;
-}
-
 static bool ggml_metal_op_concurrency_reset(ggml_metal_op_t ctx) {
     if (!ctx->mem_ranges) {
         return true;
@@ -530,31 +509,12 @@ int ggml_metal_op_encode(ggml_metal_op_t ctx, int idx) {
         ggml_metal_encoder_debug_group_push(ctx->enc, ggml_op_desc(ctx->node(idx)));
     }
 
-    const size_t slot_start = ctx->next_slot;
-    const bool   profiling  = (ctx->sample_buf != nullptr && ctx->slot_map != nullptr);
-    if (profiling) {
-        ggml_metal_encoder_sample_timestamp(ctx->enc, ctx->sample_buf, slot_start);
-        ctx->next_slot++;
-    }
-
     int res = ggml_metal_op_encode_impl(ctx, idx);
     if (idx + res > ctx->n_nodes()) {
         GGML_ABORT("fusion error: nodes spanning multiple encoders have been fused. this indicates a bug in the fusion logic %s",
                 "https://github.com/ggml-org/llama.cpp/pull/14849");
     }
 
-    if (profiling) {
-        const size_t slot_end = ctx->next_slot;
-        ggml_metal_encoder_sample_timestamp(ctx->enc, ctx->sample_buf, slot_end);
-        ctx->next_slot++;
-
-        const int gidx = ctx->node_global_idx(idx);
-        ctx->slot_map[gidx].node_idx   = gidx;
-        ctx->slot_map[gidx].n_fused    = res;
-        ctx->slot_map[gidx].slot_start = slot_start;
-        ctx->slot_map[gidx].slot_end   = slot_end;
-    }
-
     if (ctx->use_capture) {
         ggml_metal_encoder_debug_group_pop(ctx->enc);
     }
@@ -2019,6 +1979,46 @@ int ggml_metal_op_pool_1d(ggml_metal_op_t ctx, int idx) {
     return 1;
 }
 
+// supported FWHT sizes, must stay in sync with the
+// kernel_fwht_f32_<N> templates in ggml-metal.metal
+static bool ggml_metal_fwht_supported_size(int64_t n) {
+    return n == 64 || n == 128 || n == 256 || n == 512;
+}
+
+int ggml_metal_op_fwht(ggml_metal_op_t ctx, int idx) {
+    ggml_tensor * op = ctx->node(idx);
+
+    ggml_metal_library_t lib = ctx->lib;
+    ggml_metal_encoder_t enc = ctx->enc;
+
+    ggml_tensor * src1 = op->src[1];
+
+    const int64_t n = src1->ne[0];
+    const int64_t nrows = ggml_nrows(src1);
+
+    ggml_metal_kargs_fwht args = {
+        /*.nrows = */ (int32_t) nrows,
+    };
+
+    auto pipeline = ggml_metal_library_get_pipeline_fwht(lib, n);
+
+    ggml_metal_encoder_set_pipeline(enc, pipeline);
+    ggml_metal_encoder_set_bytes(enc, &args, sizeof(args), 0);
+    ggml_metal_encoder_set_buffer(enc, ggml_metal_get_buffer_id(src1), 1);
+    ggml_metal_encoder_set_buffer(enc, ggml_metal_get_buffer_id(op), 2);
+
+    const int th_max = ggml_metal_pipeline_max_theads_per_threadgroup(pipeline);
+    const int simd_size = 32;
+
+    int sg_per_tg = 2;
+    sg_per_tg = std::min(sg_per_tg, th_max/simd_size);
+    sg_per_tg = std::max(sg_per_tg, 1);
+
+    const int64_t n_tg = (nrows + sg_per_tg - 1) / sg_per_tg;
+    ggml_metal_encoder_dispatch_threadgroups(enc, n_tg, 1, 1, 32*sg_per_tg, 1, 1);
+
+    return 1;
+}
 
 int ggml_metal_op_pool_2d(ggml_metal_op_t ctx, int idx) {
     ggml_tensor * op = ctx->node(idx);
@@ -2086,6 +2086,18 @@ int ggml_metal_op_mul_mat(ggml_metal_op_t ctx, int idx) {
     ggml_metal_library_t lib = ctx->lib;
     ggml_metal_encoder_t enc = ctx->enc;
 
+    const int32_t hint = ggml_get_op_params_i32(op, 1);
+
+    if (hint == GGML_HINT_SRC0_IS_HADAMARD) {
+        if (op->src[1]->type == GGML_TYPE_F32 &&
+            op->type == GGML_TYPE_F32 &&
+            ggml_is_contiguous(op->src[1]) &&
+            ggml_is_contiguous(op) &&
+            ggml_are_same_shape(op->src[1], op) &&
+            ggml_metal_fwht_supported_size(op->src[1]->ne[0])) {
+            return ggml_metal_op_fwht(ctx, idx);
+        }
+    }
     const ggml_metal_device_props * props_dev = ggml_metal_device_get_props(ctx->dev);
 
     GGML_TENSOR_LOCALS( int32_t, ne0, op->src[0], ne);

ggml/src/ggml-metal/ggml-metal-ops.h

@@ -26,29 +26,6 @@ int ggml_metal_op_n_nodes(ggml_metal_op_t ctx);
 
 int ggml_metal_op_encode(ggml_metal_op_t ctx, int idx);
 
-//
-// Profiling: per-op timestamp sampling
-//
-
-// (node_idx, slot_start, slot_end) triple recorded for each encoded op group.
-// node_idx < 0 marks an unused slot.
-struct ggml_metal_op_sample_slot {
-    int    node_idx;
-    int    n_fused;
-    size_t slot_start;
-    size_t slot_end;
-};
-
-// Enable profiling on this op encoder. `slot_base` is the first index in `sample_buf` available
-// to this encoder; `slot_map` is an array (size = number of graph nodes) into which
-// the encoder will record one entry per non-empty encoded op group.
-// Must be called before any ggml_metal_op_encode().
-void ggml_metal_op_enable_profiling(
-        ggml_metal_op_t ctx,
-        ggml_metal_sample_buf_t sample_buf,
-        size_t slot_base,
-        struct ggml_metal_op_sample_slot * slot_map);
-
 //
 // available ops:
 //
@@ -87,6 +64,7 @@ int ggml_metal_op_set               (ggml_metal_op_t ctx, int idx);
 int ggml_metal_op_cpy               (ggml_metal_op_t ctx, int idx);
 int ggml_metal_op_pool_1d           (ggml_metal_op_t ctx, int idx);
 int ggml_metal_op_pool_2d           (ggml_metal_op_t ctx, int idx);
+int ggml_metal_op_fwht              (ggml_metal_op_t ctx, int idx);
 int ggml_metal_op_mul_mat           (ggml_metal_op_t ctx, int idx);
 int ggml_metal_op_mul_mat_id        (ggml_metal_op_t ctx, int idx);
 int ggml_metal_op_add_id            (ggml_metal_op_t ctx, int idx);

ggml/src/ggml-metal/ggml-metal.cpp

@@ -2,7 +2,6 @@
 
 #include "ggml-impl.h"
 #include "ggml-backend-impl.h"
-#include "ggml-profiler.h"
 
 #include "ggml-metal-device.h"
 #include "ggml-metal-context.h"
@@ -10,7 +9,6 @@
 
 #include <mutex>
 #include <string>
-#include <vector>
 
 #define GGML_METAL_NAME "MTL"
 #define GGML_METAL_MAX_DEVICES 16
@@ -22,59 +20,6 @@ static int g_devices = 1;
 // forward declaration
 static bool ggml_backend_buffer_is_metal(ggml_backend_buffer_t buffer);
 
-//
-// Profiler state (mirrors the per-backend profiler state from the Vulkan integration)
-//
-// Owned by ggml_backend_metal_init / device_init_backend; lifetime is the backend's lifetime.
-// The Metal context holds a borrowed pointer and calls into the bridge functions below to push
-// records during graph_compute.
-//
-
-struct ggml_metal_profiler_state {
-    bool enabled  = false;
-    int  split_id = -1;
-    std::vector<ggml_profile_record> records;
-
-    void reset() {
-        records.clear();
-        split_id = -1;
-    }
-};
-
-extern "C" {
-
-void ggml_metal_profiler_push_record(
-        ggml_metal_profiler_state * state,
-        const ggml_tensor * node,
-        uint64_t start_ns,
-        uint64_t end_ns) {
-    if (state == nullptr || node == nullptr) {
-        return;
-    }
-
-    ggml_profile_record rec;
-    rec.type       = GGML_PROFILE_EVENT_OP;
-    rec.name       = ggml_op_name(node->op);
-    rec.backend_id = -1;
-    rec.split_id   = state->split_id;
-    rec.start_ns   = start_ns;
-    rec.end_ns     = end_ns;
-    rec.bytes      = ggml_nbytes(node);
-    rec.extra      = nullptr;
-    ggml_profile_record_from_tensor(&rec, node);
-    state->records.push_back(rec);
-}
-
-bool ggml_metal_profiler_is_enabled(ggml_metal_profiler_state * state) {
-    return state != nullptr && state->enabled;
-}
-
-int ggml_metal_profiler_get_split_id(ggml_metal_profiler_state * state) {
-    return state != nullptr ? state->split_id : -1;
-}
-
-} // extern "C"
-
 ////////////////////////////////////////////////////////////////////////////////
 // backend interface
 ////////////////////////////////////////////////////////////////////////////////
@@ -644,48 +589,6 @@ static ggml_guid_t ggml_backend_metal_guid(void) {
     return &guid;
 }
 
-// Register the per-backend ggml_backend_profiler interface on `backend` and inject the borrowed
-// state pointer into the Metal context so graph_compute can push records.
-static void ggml_backend_metal_register_profiler(ggml_backend_t backend, ggml_metal_t ctx) {
-    auto * state = new ggml_metal_profiler_state();
-    ggml_metal_set_profiler_state(ctx, state);
-
-    static auto metal_prof_enable = [](void * context, bool enable) {
-        auto * state = (ggml_metal_profiler_state *) context;
-        state->enabled = enable;
-        if (!enable) {
-            state->reset();
-        }
-    };
-    static auto metal_prof_reset = [](void * context) {
-        auto * state = (ggml_metal_profiler_state *) context;
-        state->reset();
-    };
-    static auto metal_prof_set_split_id = [](void * context, int split_id) {
-        auto * state = (ggml_metal_profiler_state *) context;
-        state->split_id = split_id;
-    };
-    static auto metal_prof_get_records = [](void * context, const ggml_profile_record ** out) -> int {
-        auto * state = (ggml_metal_profiler_state *) context;
-        *out = state->records.data();
-        return (int) state->records.size();
-    };
-    static auto metal_prof_free = [](void * context) {
-        auto * state = (ggml_metal_profiler_state *) context;
-        delete state;
-    };
-
-    auto * profiler = new ggml_backend_profiler {
-        /* .context      = */ state,
-        /* .enable       = */ metal_prof_enable,
-        /* .reset        = */ metal_prof_reset,
-        /* .set_split_id = */ metal_prof_set_split_id,
-        /* .get_records  = */ metal_prof_get_records,
-        /* .free_context = */ metal_prof_free,
-    };
-    ggml_backend_set_profiler(backend, profiler);
-}
-
 ggml_backend_t ggml_backend_metal_init(void) {
     ggml_backend_dev_t dev = ggml_backend_reg_dev_get(ggml_backend_metal_reg(), 0);
     ggml_metal_device_t ctx_dev = (ggml_metal_device_t)dev->context;
@@ -703,13 +606,10 @@ ggml_backend_t ggml_backend_metal_init(void) {
         /* .interface = */ ggml_backend_metal_i,
         /* .device    = */ dev,
         /* .context   = */ ctx,
-        /* .profiler  = */ NULL,
     };
 
     ggml_backend_metal_set_n_cb(backend, 1);
 
-    ggml_backend_metal_register_profiler(backend, ctx);
-
     return backend;
 }
 
@@ -800,13 +700,10 @@ static ggml_backend_t ggml_backend_metal_device_init_backend(ggml_backend_dev_t
         /* .interface = */ ggml_backend_metal_i,
         /* .device    = */ dev,
         /* .context   = */ ctx,
-        /* .profiler  = */ NULL,
     };
 
     ggml_backend_metal_set_n_cb(backend, 1);
 
-    ggml_backend_metal_register_profiler(backend, ctx);
-
     return backend;
 
     GGML_UNUSED(params);

ggml/src/ggml-metal/ggml-metal.metal

@@ -5762,7 +5762,7 @@ kernel void kernel_upscale_bicubic_f32(
     const float w_y2 = bicubic_weight1(1.0f - fd1);
     const float w_y3 = bicubic_weight2(2.0f - fd1);
 
-    const device const char * src_slice = src0 + i03 * args.nb03 + i02 * args.nb02;
+    const device char * src_slice = src0 + i03 * args.nb03 + i02 * args.nb02;
 
     device float * dst_ptr = (device float *)(dst + i3 * args.nb3 + i2 * args.nb2 + i1 * args.nb1);
 
@@ -6172,6 +6172,68 @@ kernel void kernel_argsort_merge_f32_i32(
 template [[host_name("kernel_argsort_merge_f32_i32_asc")]]  kernel argsort_merge_t kernel_argsort_merge_f32_i32<GGML_SORT_ORDER_ASC>;
 template [[host_name("kernel_argsort_merge_f32_i32_desc")]] kernel argsort_merge_t kernel_argsort_merge_f32_i32<GGML_SORT_ORDER_DESC>;
 
+template<int N>
+kernel void kernel_fwht_f32(
+        constant ggml_metal_kargs_fwht & args,
+        device const float * src,
+        device float * dst,
+        uint3  tgpig[[threadgroup_position_in_grid]],
+        ushort sgitg[[simdgroup_index_in_threadgroup]],
+        ushort tiisg[[thread_index_in_simdgroup]],
+        ushort3  ntg[[threads_per_threadgroup]]) {
+
+    constexpr int NW = N_SIMDWIDTH;
+    constexpr int NE = N / NW;
+
+    const float scale = 1.0f / sqrt((float) N);
+
+    const int sg_per_tg = ntg.x / NW;
+    const int64_t r = tgpig.x * sg_per_tg + sgitg;
+    if (r >= args.nrows) {
+        return;
+    }
+
+    src += r * N;
+    dst += r * N;
+
+    const int lane = tiisg;
+
+    float reg[NE];
+    for (int i = 0; i < NE; i++) {
+        reg[i] = src[i*NW + lane]*scale;
+    }
+    for (int i = 1; i < NW; i *= 2) {
+        for (int j = 0; j < NE; j++) {
+            const float val = reg[j];
+            const float val2 = simd_shuffle_xor(val, i);
+            reg[j] = (lane & i) == 0 ? val2 + val : val2 - val;
+        }
+    }
+
+    for (int i = NW; i < N; i *= 2) {
+        const int step = i / NW;
+        for (int j = 0; j < NE; j += (2 * step)) {
+            for (int k = 0; k < step; k++) {
+                const float x = reg[j + k ];
+                const float y = reg[j + k + step];
+                reg[j + k]        = x + y;
+                reg[j + k + step] = x - y;
+            }
+        }
+    }
+
+    for (int i = 0; i < NE; i++) {
+        dst[i*NW + lane] = reg[i];
+    }
+}
+
+typedef decltype(kernel_fwht_f32<64>) kernel_fwht_t;
+
+template [[host_name("kernel_fwht_f32_64")]]  kernel kernel_fwht_t kernel_fwht_f32<64>;
+template [[host_name("kernel_fwht_f32_128")]] kernel kernel_fwht_t kernel_fwht_f32<128>;
+template [[host_name("kernel_fwht_f32_256")]] kernel kernel_fwht_t kernel_fwht_f32<256>;
+template [[host_name("kernel_fwht_f32_512")]] kernel kernel_fwht_t kernel_fwht_f32<512>;
+
 constant bool FC_flash_attn_ext_pad_has_mask [[function_constant(FC_FLASH_ATTN_EXT_PAD + 0)]];
 
 constant int32_t FC_flash_attn_ext_pad_ncpsg [[function_constant(FC_FLASH_ATTN_EXT_PAD + 25)]];

ggml/src/ggml-opencl/CMakeLists.txt

@@ -5,6 +5,8 @@ set(TARGET_NAME ggml-opencl)
 
 ggml_add_backend_library(${TARGET_NAME}
                          ggml-opencl.cpp
+                         cl-program-cache.cpp
+                         cl-program-cache.h
                          ../../include/ggml-opencl.h)
 target_link_libraries(${TARGET_NAME} PRIVATE ${OpenCL_LIBRARIES})
 target_include_directories(${TARGET_NAME} PRIVATE ${OpenCL_INCLUDE_DIRS})

ggml/src/ggml-opencl/cl-program-cache.cpp

@@ -0,0 +1,453 @@
+// Match the version setup ggml-opencl.cpp uses, so any cl.h declarations we
+// touch are consistent across this backend's translation units.
+#define CL_TARGET_OPENCL_VERSION GGML_OPENCL_TARGET_VERSION
+#define CL_USE_DEPRECATED_OPENCL_1_2_APIS
+
+#include "cl-program-cache.h"
+
+#include "ggml-impl.h"  // GGML_LOG_INFO / WARN
+
+#include <cstdint>
+#include <cstdio>
+#include <cstdlib>
+#include <cstring>
+#include <filesystem>
+#include <fstream>
+#include <system_error>
+#include <vector>
+
+#if defined(_WIN32)
+#  ifndef WIN32_LEAN_AND_MEAN
+#    define WIN32_LEAN_AND_MEAN
+#  endif
+#  ifndef NOMINMAX
+#    define NOMINMAX
+#  endif
+#  include <windows.h>
+#  include <process.h>
+#  define ggml_getpid()    ((int) GetCurrentProcessId())
+#else
+#  include <unistd.h>
+#  define ggml_getpid()    ((int) getpid())
+#endif
+
+namespace fs = std::filesystem;
+
+// ----------------------------------------------------------------------------
+// SHA-256 (FIPS 180-4). Self-contained, ~80 lines, public-domain reference.
+// Hot path is a few KB of source per kernel ⇒ <1 ms total per process init.
+// ----------------------------------------------------------------------------
+
+namespace {
+
+struct sha256_ctx {
+    uint32_t state[8];
+    uint64_t bitlen;
+    uint8_t  buf[64];
+    size_t   buf_len;
+};
+
+const uint32_t K256[64] = {
+    0x428a2f98,0x71374491,0xb5c0fbcf,0xe9b5dba5,0x3956c25b,0x59f111f1,0x923f82a4,0xab1c5ed5,
+    0xd807aa98,0x12835b01,0x243185be,0x550c7dc3,0x72be5d74,0x80deb1fe,0x9bdc06a7,0xc19bf174,
+    0xe49b69c1,0xefbe4786,0x0fc19dc6,0x240ca1cc,0x2de92c6f,0x4a7484aa,0x5cb0a9dc,0x76f988da,
+    0x983e5152,0xa831c66d,0xb00327c8,0xbf597fc7,0xc6e00bf3,0xd5a79147,0x06ca6351,0x14292967,
+    0x27b70a85,0x2e1b2138,0x4d2c6dfc,0x53380d13,0x650a7354,0x766a0abb,0x81c2c92e,0x92722c85,
+    0xa2bfe8a1,0xa81a664b,0xc24b8b70,0xc76c51a3,0xd192e819,0xd6990624,0xf40e3585,0x106aa070,
+    0x19a4c116,0x1e376c08,0x2748774c,0x34b0bcb5,0x391c0cb3,0x4ed8aa4a,0x5b9cca4f,0x682e6ff3,
+    0x748f82ee,0x78a5636f,0x84c87814,0x8cc70208,0x90befffa,0xa4506ceb,0xbef9a3f7,0xc67178f2,
+};
+
+inline uint32_t rotr32(uint32_t x, unsigned n) { return (x >> n) | (x << (32 - n)); }
+
+void sha256_compress(uint32_t state[8], const uint8_t block[64]) {
+    uint32_t w[64];
+    for (int i = 0; i < 16; ++i) {
+        w[i] = ((uint32_t)block[i*4    ] << 24) |
+               ((uint32_t)block[i*4 + 1] << 16) |
+               ((uint32_t)block[i*4 + 2] <<  8) |
+               ((uint32_t)block[i*4 + 3]      );
+    }
+    for (int i = 16; i < 64; ++i) {
+        uint32_t s0 = rotr32(w[i-15], 7) ^ rotr32(w[i-15], 18) ^ (w[i-15] >> 3);
+        uint32_t s1 = rotr32(w[i-2], 17) ^ rotr32(w[i-2],  19) ^ (w[i-2]  >> 10);
+        w[i] = w[i-16] + s0 + w[i-7] + s1;
+    }
+
+    uint32_t a = state[0],b = state[1],c = state[2],d = state[3],e = state[4],f = state[5],g = state[6],h = state[7];
+
+    for (int i = 0; i < 64; ++i) {
+        uint32_t S1   = rotr32(e, 6) ^ rotr32(e, 11) ^ rotr32(e, 25);
+        uint32_t ch   = (e & f) ^ ((~e) & g);
+        uint32_t t1   = h + S1 + ch + K256[i] + w[i];
+        uint32_t S0   = rotr32(a, 2) ^ rotr32(a, 13) ^ rotr32(a, 22);
+        uint32_t maj  = (a & b) ^ (a & c) ^ (b & c);
+        uint32_t t2   = S0 + maj;
+        h = g; g = f; f = e; e = d + t1;
+        d = c; c = b; b = a; a = t1 + t2;
+    }
+    state[0]+=a; state[1]+=b; state[2]+=c; state[3]+=d;
+    state[4]+=e; state[5]+=f; state[6]+=g; state[7]+=h;
+}
+
+void sha256_init(sha256_ctx & c) {
+    c.state[0]=0x6a09e667; c.state[1]=0xbb67ae85; c.state[2]=0x3c6ef372; c.state[3]=0xa54ff53a;
+    c.state[4]=0x510e527f; c.state[5]=0x9b05688c; c.state[6]=0x1f83d9ab; c.state[7]=0x5be0cd19;
+    c.bitlen  = 0;
+    c.buf_len = 0;
+}
+
+void sha256_update(sha256_ctx & c, const void * data, size_t len) {
+    const uint8_t * p = (const uint8_t *) data;
+    c.bitlen += (uint64_t) len * 8;
+    if (c.buf_len > 0) {
+        size_t n = 64 - c.buf_len;
+        if (n > len) { n = len; }
+        memcpy(c.buf + c.buf_len, p, n);
+        c.buf_len += n;
+        p   += n;
+        len -= n;
+        if (c.buf_len == 64) {
+            sha256_compress(c.state, c.buf);
+            c.buf_len = 0;
+        }
+    }
+    while (len >= 64) {
+        sha256_compress(c.state, p);
+        p   += 64;
+        len -= 64;
+    }
+    if (len > 0) {
+        memcpy(c.buf, p, len);
+        c.buf_len = len;
+    }
+}
+
+void sha256_final(sha256_ctx & c, uint8_t out[32]) {
+    uint64_t bitlen = c.bitlen;
+    c.buf[c.buf_len++] = 0x80;
+    if (c.buf_len > 56) {
+        while (c.buf_len < 64) { c.buf[c.buf_len++] = 0; }
+        sha256_compress(c.state, c.buf);
+        c.buf_len = 0;
+    }
+    while (c.buf_len < 56) { c.buf[c.buf_len++] = 0; }
+    for (int i = 7; i >= 0; --i) { c.buf[c.buf_len++] = (uint8_t) (bitlen >> (i * 8)); }
+    sha256_compress(c.state, c.buf);
+    for (int i = 0; i < 8; ++i) {
+        out[i*4    ] = (uint8_t) (c.state[i] >> 24);
+        out[i*4 + 1] = (uint8_t) (c.state[i] >> 16);
+        out[i*4 + 2] = (uint8_t) (c.state[i] >>  8);
+        out[i*4 + 3] = (uint8_t) (c.state[i]      );
+    }
+}
+
+std::string sha256_hex(const uint8_t digest[32]) {
+    static const char hex[] = "0123456789abcdef";
+    std::string s(64, '0');
+    for (int i = 0; i < 32; ++i) {
+        s[i*2    ] = hex[digest[i] >> 4];
+        s[i*2 + 1] = hex[digest[i] & 0xf];
+    }
+    return s;
+}
+
+std::string compute_key(const std::string & key_suffix,
+                        const char *        source,
+                        const std::string & compile_opts) {
+    sha256_ctx c;
+    sha256_init(c);
+
+    static const uint8_t sep = 0;
+    sha256_update(c, source,             strlen(source));
+    sha256_update(c, &sep,               1);
+    sha256_update(c, compile_opts.data(), compile_opts.size());
+    sha256_update(c, &sep,               1);
+    sha256_update(c, key_suffix.data(),  key_suffix.size());
+
+    uint8_t digest[32];
+    sha256_final(c, digest);
+    return sha256_hex(digest);
+}
+
+bool make_dir_recursive(const std::string & path) {
+    if (path.empty()) { return false; }
+    // create_directories() already creates missing parents. It returns false
+    // (with ec clear) when the directory is already there, so re-check.
+    const fs::path p = fs::u8path(path);
+    std::error_code ec;
+    if (fs::create_directories(p, ec)) { return true; }
+    std::error_code ec_stat;
+    return fs::is_directory(p, ec_stat);
+}
+
+std::string default_cache_dir() {
+#if defined(_WIN32)
+    const char * base = std::getenv("LOCALAPPDATA");
+    if (!base || !*base) { base = std::getenv("APPDATA"); }
+    if (!base || !*base) { base = std::getenv("TEMP"); }
+    if (!base || !*base) { base = "."; }
+    return std::string(base) + "\\llama.cpp\\cl-cache";
+#elif defined(__APPLE__)
+    const char * home = std::getenv("HOME");
+    if (!home || !*home) { home = "."; }
+    return std::string(home) + "/Library/Caches/llama.cpp/cl-cache";
+#else
+    // The throwing overload aborts the process when no usable temp directory
+    // exists (e.g. Android app contexts with TMPDIR unset); an empty return
+    // here just disables the cache instead.
+    std::error_code ec;
+    const fs::path tmp_path = fs::temp_directory_path(ec);
+    if (ec || tmp_path.empty()) { return {}; }
+    return tmp_path.string() + "/llama.cpp/cl-cache";
+#endif
+}
+
+// Query a NUL-terminated string from clGetDeviceInfo / clGetPlatformInfo.
+template <typename GetInfoFn, typename Object>
+std::string query_string(GetInfoFn fn, Object obj, cl_uint name) {
+    size_t sz = 0;
+    if (fn(obj, name, 0, nullptr, &sz) != CL_SUCCESS || sz == 0) {
+        return {};
+    }
+    std::string s(sz, '\0');
+    if (fn(obj, name, sz, &s[0], nullptr) != CL_SUCCESS) {
+        return {};
+    }
+    if (!s.empty() && s.back() == '\0') {
+        s.pop_back();
+    }
+    return s;
+}
+
+std::string compute_key_suffix(cl_device_id device) {
+    cl_platform_id platform = nullptr;
+    clGetDeviceInfo(device, CL_DEVICE_PLATFORM, sizeof(platform), &platform, nullptr);
+
+    std::string s;
+    s.reserve(512);
+    s += query_string(clGetDeviceInfo,   device,   CL_DEVICE_NAME);       s.push_back('\0');
+    s += query_string(clGetDeviceInfo,   device,   CL_DRIVER_VERSION);    s.push_back('\0');
+    s += query_string(clGetDeviceInfo,   device,   CL_DEVICE_VERSION);    s.push_back('\0');
+    if (platform) {
+        s += query_string(clGetPlatformInfo, platform, CL_PLATFORM_VERSION); s.push_back('\0');
+    }
+    s += "fmt=" + std::to_string(CL_PROGRAM_CACHE_FORMAT_VERSION);
+    return s;
+}
+
+const uint8_t MAGIC[8] = { 'G','G','M','L','C','L','B','C' };
+
+bool read_all(const std::string & path, std::vector<uint8_t> & out) {
+    std::ifstream f(fs::u8path(path), std::ios::binary);
+    if (!f) { return false; }
+    f.seekg(0, std::ios::end);
+    std::streamsize sz = f.tellg();
+    if (sz < 0) { return false; }
+    f.seekg(0, std::ios::beg);
+    out.resize((size_t) sz);
+    if (sz > 0) { f.read((char *) out.data(), sz); }
+    return f.good() || f.eof();
+}
+
+bool write_atomic(const std::string & path, const uint8_t * data, size_t len) {
+    const fs::path dst = fs::u8path(path);
+    const fs::path tmp = fs::u8path(path + ".tmp." + std::to_string(ggml_getpid()));
+    {
+        std::ofstream f(tmp, std::ios::binary | std::ios::trunc);
+        if (!f) { return false; }
+        f.write((const char *) data, (std::streamsize) len);
+        if (!f.good()) {
+            std::error_code ec_rm;
+            fs::remove(tmp, ec_rm);
+            return false;
+        }
+    }
+
+    std::error_code ec;
+    fs::rename(tmp, dst, ec);
+    if (ec) {
+        std::error_code ec_rm;
+        fs::remove(tmp, ec_rm);
+        return false;
+    }
+    return true;
+}
+
+}  // namespace
+
+static bool cache_debug_enabled() {
+    static int cached = -1;
+    if (cached < 0) {
+        const char * e = std::getenv("GGML_OPENCL_KERNEL_CACHE_DEBUG");
+        cached = (e && *e) ? 1 : 0;
+    }
+    return cached != 0;
+}
+
+static std::string opts_preview(const std::string & opts, size_t n = 120) {
+    if (opts.size() <= n) { return opts; }
+    return opts.substr(0, n) + "...";
+}
+
+// Running cache tally (diagnostic; plain ints — a benign race in the rare
+// multi-threaded lazy-compile case at worst miscounts by one).
+static int g_cache_hits = 0, g_cache_misses = 0, g_cache_saves = 0;
+
+// Debug trace directly to stderr
+static void cache_debug_line(const char * kind, const std::string & key,
+                             const char * source, const std::string & opts) {
+    if (!cache_debug_enabled()) { return; }
+    fprintf(stderr, "ggml_opencl: cache %-4s [h=%d m=%d s=%d] key=%s src=%zuB opts='%s'\n",
+            kind, g_cache_hits, g_cache_misses, g_cache_saves,
+            key.substr(0, 16).c_str(), strlen(source), opts_preview(opts).c_str());
+    fflush(stderr);
+}
+
+cl_program_cache_state cl_program_cache_init(cl_device_id device) {
+    cl_program_cache_state st;
+
+    const char * env = std::getenv("GGML_OPENCL_KERNEL_CACHE_DIR");
+    if (env && (!std::strcmp(env, "0")    || !std::strcmp(env, "off")  ||
+                !std::strcmp(env, "none") || !std::strcmp(env, "disable") ||
+                !std::strcmp(env, "disabled"))) {
+        if (cache_debug_enabled()) {
+            fprintf(stderr, "ggml_opencl: kernel cache disabled by GGML_OPENCL_KERNEL_CACHE_DIR=%s\n", env);
+            fflush(stderr);
+        }
+        return st;
+    }
+
+    std::string dir;
+    if (!env || !*env || !std::strcmp(env, "1") || !std::strcmp(env, "default")) {
+        dir = default_cache_dir();
+        if (dir.empty()) {
+            GGML_LOG_INFO("ggml_opencl: kernel cache disabled (no usable default cache directory)\n");
+            return st;
+        }
+    } else {
+        dir = env;
+    }
+
+    if (!make_dir_recursive(dir)) {
+        GGML_LOG_INFO("ggml_opencl: kernel cache disabled (cannot create directory '%s')\n", dir.c_str());
+        return st;
+    }
+
+    st.dir        = dir;
+    st.key_suffix = compute_key_suffix(device);
+    GGML_LOG_INFO("ggml_opencl: kernel cache enabled at '%s'\n", st.dir.c_str());
+    if (cache_debug_enabled()) {
+        fprintf(stderr, "ggml_opencl: kernel cache enabled at '%s' "
+                        "(GGML_OPENCL_KERNEL_CACHE_DIR=off to disable)\n", st.dir.c_str());
+        fflush(stderr);
+    }
+    return st;
+}
+
+cl_program cl_program_cache_try_load(
+    const cl_program_cache_state & state,
+    cl_context                     context,
+    cl_device_id                   device,
+    const char *                   source,
+    const std::string &            compile_opts) {
+
+    if (state.dir.empty() || !source) { return nullptr; }
+
+    const std::string key  = compute_key(state.key_suffix, source, compile_opts);
+    const std::string path = state.dir + "/" + key + ".clbin";
+
+    std::vector<uint8_t> file;
+    if (!read_all(path, file)) {
+        ++g_cache_misses;
+        cache_debug_line("MISS", key, source, compile_opts);
+        return nullptr;
+    }
+    if (file.size() < 16 || std::memcmp(file.data(), MAGIC, 8) != 0) { return nullptr; }
+
+    uint32_t fmt =
+        ((uint32_t) file[ 8]) | ((uint32_t) file[ 9] <<  8) |
+        ((uint32_t) file[10] << 16) | ((uint32_t) file[11] << 24);
+    if (fmt != CL_PROGRAM_CACHE_FORMAT_VERSION) { return nullptr; }
+
+    const size_t hdr_len = 16;
+    const unsigned char * bin     = file.data() + hdr_len;
+    const size_t          bin_len = file.size() - hdr_len;
+
+    cl_int err     = CL_SUCCESS;
+    cl_int bin_err = CL_SUCCESS;
+    cl_program p = clCreateProgramWithBinary(context, 1, &device, &bin_len, &bin, &bin_err, &err);
+    if (err != CL_SUCCESS || bin_err != CL_SUCCESS || p == nullptr) {
+        if (p) { clReleaseProgram(p); }
+        return nullptr;
+    }
+
+    err = clBuildProgram(p, 0, nullptr, compile_opts.c_str(), nullptr, nullptr);
+    if (err != CL_SUCCESS) {
+        clReleaseProgram(p);
+        return nullptr;
+    }
+    ++g_cache_hits;
+    cache_debug_line("HIT", key, source, compile_opts);
+    return p;
+}
+
+void cl_program_cache_try_save(
+    const cl_program_cache_state & state,
+    cl_program                     program,
+    cl_device_id                   /*device*/,
+    const char *                   source,
+    const std::string &            compile_opts) {
+
+    if (state.dir.empty() || !program || !source) {
+        return;
+    }
+
+    cl_uint n_dev = 0;
+    if (clGetProgramInfo(program, CL_PROGRAM_NUM_DEVICES, sizeof(n_dev), &n_dev, nullptr) != CL_SUCCESS || n_dev == 0) {
+        return;
+    }
+
+    std::vector<size_t> sizes(n_dev);
+    if (clGetProgramInfo(program, CL_PROGRAM_BINARY_SIZES, sizeof(size_t) * n_dev, sizes.data(), nullptr) != CL_SUCCESS) {
+        return;
+    }
+    if (sizes.empty() || sizes[0] == 0) {
+        return;
+    }
+
+    std::vector<std::vector<uint8_t>> binaries(n_dev);
+    std::vector<unsigned char *>      bin_ptrs(n_dev);
+    for (cl_uint i = 0; i < n_dev; ++i) {
+        binaries[i].resize(sizes[i]);
+        bin_ptrs[i] = binaries[i].data();
+    }
+    if (clGetProgramInfo(program, CL_PROGRAM_BINARIES, sizeof(unsigned char *) * n_dev, bin_ptrs.data(), nullptr) != CL_SUCCESS) {
+        return;
+    }
+
+    // We only care about the first device's binary — that's the one we'd
+    // re-load with on a future cache hit. Multi-device contexts aren't a
+    // pattern this backend uses today.
+    const std::vector<uint8_t> & bin = binaries[0];
+
+    std::vector<uint8_t> file;
+    file.reserve(16 + bin.size());
+    file.insert(file.end(), MAGIC, MAGIC + 8);
+    uint32_t fmt = CL_PROGRAM_CACHE_FORMAT_VERSION;
+    file.push_back((uint8_t)  (fmt        & 0xff));
+    file.push_back((uint8_t) ((fmt >>  8) & 0xff));
+    file.push_back((uint8_t) ((fmt >> 16) & 0xff));
+    file.push_back((uint8_t) ((fmt >> 24) & 0xff));
+    file.push_back(0); file.push_back(0); file.push_back(0); file.push_back(0); // reserved
+    file.insert(file.end(), bin.begin(), bin.end());
+
+    const std::string key  = compute_key(state.key_suffix, source, compile_opts);
+    const std::string path = state.dir + "/" + key + ".clbin";
+    if (!write_atomic(path, file.data(), file.size())) {
+        GGML_LOG_INFO("ggml_opencl: kernel cache: failed to write '%s'\n", path.c_str());
+    } else {
+        ++g_cache_saves;
+        cache_debug_line("SAVE", key, source, compile_opts);
+    }
+}

ggml/src/ggml-opencl/cl-program-cache.h

@@ -0,0 +1,75 @@
+// On-disk cache for OpenCL cl_program binaries. Lets a fresh process skip the
+// expensive clBuildProgram-from-source step when a binary for the exact same
+// (source, compile options, device, driver, platform) was previously saved.
+//
+// Activation: default on via GGML_OPENCL_KERNEL_CACHE_DIR:
+//   unset / empty / "1" / "default"      : platform default cache dir
+//                                          (%LOCALAPPDATA%\llama.cpp\cl-cache,
+//                                          ~/Library/Caches/llama.cpp/cl-cache,
+//                                          <temp dir>/llama.cpp/cl-cache elsewhere)
+//   "0" / "off" / "none" / "disable(d)"  : disabled (all functions no-op)
+//   any other value                      : used verbatim as the cache path
+// If the chosen directory cannot be created/used, the cache silently disables
+// itself for the process and falls back to source compile.
+// GGML_OPENCL_KERNEL_CACHE_DEBUG=1 prints a HIT/MISS/SAVE trace (with a running
+// tally) straight to stderr — visible even in tools that filter INFO/WARN logs;
+// redirect stderr to record it.
+//
+// Cache key (SHA-256 hex):
+//   sha256(source_bytes || '\x00' ||
+//          compile_opts  || '\x00' ||
+//          CL_DEVICE_NAME || '\x00' ||
+//          CL_DRIVER_VERSION || '\x00' ||
+//          CL_PLATFORM_VERSION || '\x00' ||
+//          CL_PROGRAM_CACHE_FORMAT_VERSION)
+//
+// The key fully captures everything that can affect the produced binary,
+// without needing the host source revision (a kernel source change shows up
+// in source_bytes; a compile-option change shows up in compile_opts).
+//
+// File layout per cache entry: <cache_dir>/<sha256-hex>.clbin
+//   bytes [0..7]   : magic "GGMLCLBC"
+//   bytes [8..11]  : uint32_t format version (CL_PROGRAM_CACHE_FORMAT_VERSION)
+//   bytes [12..15] : uint32_t reserved (0)
+//   bytes [16..]   : raw cl_program binary as returned by
+//                    clGetProgramInfo(CL_PROGRAM_BINARIES)
+//
+// Concurrency: writes go to <name>.tmp.<pid> then atomic rename. On race,
+// last-writer-wins. No locks.
+
+#pragma once
+
+#include <CL/cl.h>
+#include <string>
+
+// Bumped manually if host-side OpenCL API usage changes in a way that
+// affects compile semantics but does not show up in source_bytes /
+// compile_opts (e.g. switching from clCreateProgramWithSource to
+// clCompileProgram + clLinkProgram, or changing how multiple sources
+// are concatenated). Most commits — including kernel changes — do NOT
+// require bumping this; the source bytes already capture those.
+#define CL_PROGRAM_CACHE_FORMAT_VERSION 1u
+
+struct cl_program_cache_state {
+    // Empty string means cache is disabled.
+    std::string dir;
+    // Concatenated device/driver/platform identity + cache format version,
+    // computed once at init and folded into every key.
+    std::string key_suffix;
+};
+
+cl_program_cache_state cl_program_cache_init(cl_device_id device);
+
+cl_program cl_program_cache_try_load(
+    const cl_program_cache_state & state,
+    cl_context                     context,
+    cl_device_id                   device,
+    const char *                   source,
+    const std::string &            compile_opts);
+
+void cl_program_cache_try_save(
+    const cl_program_cache_state & state,
+    cl_program                     program,
+    cl_device_id                   device,
+    const char *                   source,
+    const std::string &            compile_opts);

ggml/src/ggml-openvino/ggml-openvino.cpp

@@ -703,7 +703,6 @@ GGML_BACKEND_API ggml_backend_t ggml_backend_openvino_init(int device) {
         /* .interface = */ ggml_backend_openvino_interface,
         /* .device    = */ ggml_backend_reg_dev_get(ggml_backend_openvino_reg(), device),
         /* .context   = */ ctx,
-        /* .profiler  = */ nullptr,
     };
 
     return openvino_backend;

ggml/src/ggml-profiler.cpp

@@ -1,121 +0,0 @@
-#include "ggml-profiler.h"
-
-#include "ggml-backend-impl.h"
-#include "ggml-impl.h"
-#include "ggml.h"
-
-#include <stdio.h>
-#include <string.h>
-
-#ifdef _WIN32
-#    define WIN32_LEAN_AND_MEAN
-#    ifndef NOMINMAX
-#        define NOMINMAX
-#    endif
-#    include <windows.h>
-#else
-#    include <time.h>
-#    include <unistd.h>
-#endif
-
-//
-// Time utilities
-//
-
-uint64_t ggml_profiler_time_ns(void) {
-#ifdef _WIN32
-    LARGE_INTEGER freq, count;
-    QueryPerformanceFrequency(&freq);
-    QueryPerformanceCounter(&count);
-    return (uint64_t) (count.QuadPart * 1000000000ULL / freq.QuadPart);
-#elif defined(CLOCK_MONOTONIC_RAW)
-    struct timespec ts;
-    clock_gettime(CLOCK_MONOTONIC_RAW, &ts);
-    return (uint64_t) ts.tv_sec * 1000000000ULL + (uint64_t) ts.tv_nsec;
-#else
-    struct timespec ts;
-    clock_gettime(CLOCK_MONOTONIC, &ts);
-    return (uint64_t) ts.tv_sec * 1000000000ULL + (uint64_t) ts.tv_nsec;
-#endif
-}
-
-//
-// Record helpers
-//
-
-void ggml_profile_record_from_tensor(ggml_profile_record * rec, const struct ggml_tensor * node) {
-    if (rec == NULL) {
-        return;
-    }
-
-    // Output tensor info
-    if (node != NULL) {
-        memcpy(rec->ne, node->ne, sizeof(rec->ne));
-        rec->out_type = (int) node->type;
-        memcpy(rec->op_params, node->op_params, sizeof(rec->op_params));
-        // Copy the tensor name (rather than aliasing node->name) so the record stays
-        // valid after the graph's meta context is reused on the next build.
-        snprintf(rec->tensor_name, sizeof(rec->tensor_name), "%s", node->name);
-    } else {
-        memset(rec->ne, 0, sizeof(rec->ne));
-        rec->out_type = -1;
-        memset(rec->op_params, 0, sizeof(rec->op_params));
-        rec->tensor_name[0] = '\0';
-    }
-
-    // Sub-op (UNARY/GLU)
-    rec->sub_op = -1;
-    if (node != NULL) {
-        if (node->op == GGML_OP_UNARY) {
-            rec->sub_op = (int) ggml_get_unary_op(node);
-        } else if (node->op == GGML_OP_GLU) {
-            rec->sub_op = (int) ggml_get_glu_op(node);
-        }
-    }
-
-    // Source tensors
-    rec->n_src = 0;
-    for (int i = 0; i < GGML_MAX_SRC; i++) {
-        const struct ggml_tensor * src = (node != NULL) ? node->src[i] : NULL;
-        if (src == NULL) {
-            memset(rec->ne_src[i], 0, sizeof(rec->ne_src[i]));
-            memset(rec->nb_src[i], 0, sizeof(rec->nb_src[i]));
-            rec->type_src[i] = -1;
-        } else {
-            memcpy(rec->ne_src[i], src->ne, sizeof(rec->ne_src[i]));
-            for (int d = 0; d < 4; d++) {
-                rec->nb_src[i][d] = (int64_t) src->nb[d];
-            }
-            rec->type_src[i] = (int) src->type;
-            rec->n_src = i + 1;
-        }
-    }
-}
-
-//
-// Backend profiler registration
-//
-
-void ggml_backend_set_profiler(ggml_backend_t backend, ggml_backend_profiler_t profiler) {
-    if (backend == NULL) {
-        return;
-    }
-
-    // Free any existing profiler
-    if (backend->profiler != NULL) {
-        if (backend->profiler->free_context != NULL) {
-            backend->profiler->free_context(backend->profiler->context);
-        }
-        delete backend->profiler;
-        backend->profiler = NULL;
-    }
-
-    backend->profiler = profiler;
-}
-
-ggml_backend_profiler_t ggml_backend_get_profiler(ggml_backend_t backend) {
-    if (backend == NULL) {
-        return NULL;
-    }
-    return backend->profiler;
-}

ggml/src/ggml-rpc/ggml-rpc.cpp

@@ -785,8 +785,7 @@ ggml_backend_t ggml_backend_rpc_init(const char * endpoint, uint32_t device) {
         /* .guid    = */ ggml_backend_rpc_guid(),
         /* .iface   = */ ggml_backend_rpc_interface,
         /* .device  = */ ggml_backend_reg_dev_get(reg, device),
-        /* .context  = */ ctx,
-        /* .profiler = */ nullptr,
+        /* .context = */ ctx
     };
     return backend;
 }

ggml/src/ggml-sycl/CMakeLists.txt

@@ -199,9 +199,20 @@ if (GGML_SYCL_DEVICE_ARCH)
         -fsycl-targets=spir64_gen
         "SHELL:-Xsycl-target-backend=spir64_gen \"-device ${GGML_SYCL_DEVICE_ARCH}\""
     )
+
+    # Pass through parallel job (process) count for parallelising the
+    # `llvm-foreach -- ocloc` invocation for compiling AOT device images.
+    include(ProcessorCount)
+    ProcessorCount(_ggml_sycl_nproc)
+    if (_ggml_sycl_nproc LESS 1)
+        set(_ggml_sycl_nproc 1)
+    endif()
+    set(GGML_SYCL_MAX_PARALLEL_LINK_JOBS ${_ggml_sycl_nproc} CACHE STRING
+        "Parallel ocloc jobs for spir64_gen AOT device-image lowering")
     target_link_options(
         ggml-sycl PRIVATE
         -fsycl-targets=spir64_gen
         "SHELL:-Xsycl-target-backend=spir64_gen \"-device ${GGML_SYCL_DEVICE_ARCH}\""
+        -fsycl-max-parallel-link-jobs=${GGML_SYCL_MAX_PARALLEL_LINK_JOBS}
     )
 endif()

ggml/src/ggml-sycl/common.hpp

@@ -65,6 +65,7 @@ extern int g_ggml_sycl_prioritize_dmmv;
 extern int g_ggml_sycl_enable_flash_attention;
 extern int g_ggml_sycl_dev2dev_memcpy;
 extern int g_ggml_sycl_fa_onednn;
+extern int g_ggml_sycl_fa_onednn_max_kv;
 
 
 #if defined(__clang__) && __has_builtin(__builtin_expect)

ggml/src/ggml-sycl/fattn-onednn.cpp

@@ -38,6 +38,12 @@ bool ggml_sycl_flash_attn_ext_onednn_supported(const ggml_tensor * dst) {
     if (K->type != GGML_TYPE_F16 || V->type != GGML_TYPE_F16) {
         return false;
     }
+    // Optional KV-length ceiling (GGML_SYCL_FA_ONEDNN_MAX_KV, 0 = unlimited). Escape hatch:
+    // very long sequences make the fused SDPA slow enough to risk the xe driver watchdog on
+    // some stacks; past the cap we fall back to the native FA kernel instead.
+    if (g_ggml_sycl_fa_onednn_max_kv > 0 && K->ne[1] > g_ggml_sycl_fa_onednn_max_kv) {
+        return false;
+    }
     // gate for the following cases
     // 1. if the oneDNN graph Add node has no input --> skip
     // 2. types other than f16 need different logical_tensor declaration
@@ -208,9 +214,17 @@ void ggml_sycl_flash_attn_ext_onednn(ggml_backend_sycl_context & ctx, ggml_tenso
     cont_to_f16_sycl<sycl::half>((const char *) V->data, Vf.get(), d, seq, Hkv, mb, V->nb[1], V->nb[2], V->nb[3], stream);
 
     // divide-by-(1/scale) reproduces ggml's score *= kq_scale on the proven probe graph.
+    //
+    // The scale must not be uploaded with an async memcpy from a stack local: on the in-order
+    // queue that copy waits behind the K/V staging kernels, and once those take long enough
+    // (n_kv >= ~26k on B70) the host frame is recycled before the copy runs, feeding the SDPA a
+    // garbage scale (output collapses to a repeated token). Write the scalar from a kernel
+    // instead -- the value is captured into the command, so no host memory has to outlive the
+    // call, and the enqueue stays async.
     const sycl::half scale_h = (sycl::half) (1.0f / kq_scale);
     ggml_sycl_pool_alloc<sycl::half> scbuf(ctx.pool(), 1);
-    stream->memcpy(scbuf.get(), &scale_h, sizeof(sycl::half));
+    sycl::half * const scale_dev = scbuf.get();
+    stream->single_task([=]() { *scale_dev = scale_h; });
 
     ggml_sycl_pool_alloc<sycl::half> outf(ctx.pool(), (size_t) H * q * d);   // f16 contiguous SDPA out [mb,H,q,d]
 
@@ -232,7 +246,7 @@ void ggml_sycl_flash_attn_ext_onednn(ggml_backend_sycl_context & ctx, ggml_tenso
         if (r == E.id_q)     return Qf.get();
         if (r == E.id_k)     return Kf.get();
         if (r == E.id_v)     return Vf.get();
-        if (r == E.id_scale) return scbuf.get();
+        if (r == E.id_scale) return scale_dev;
         if (r == E.id_mask)  return (void *) mask->data;
         return nullptr;
     };
@@ -245,14 +259,12 @@ void ggml_sycl_flash_attn_ext_onednn(ggml_backend_sycl_context & ctx, ggml_tenso
     E.cp.execute(strm, ti, {to});
 
     permute_sdpa_out_sycl(outf.get(), (float *) dst->data, mb, H, q, d, stream);
-    // Single device: no sync is required, and actually PP perf is ~6% > wait_and_throw() (tested on llama-3.1-8b & qwen3.6-27b, both Q8_0, with Arc B70).
-    // Any future multi-GPU refactor MUST re-measure this single-device path and keep the best
-    // single-device PP speed. Otherwise (multiple devices/streams can race the reuse):
+    // Single device needs no sync: the dnnl stream wraps this same in-order queue, so the SDPA
+    // serializes with the staging kernels before it and the permute/pool reuse after it. The
+    // garbage output formerly blamed on the missing sync here was the scale use-after-return
+    // fixed above. Keep the conservative wait for multi-GPU, where other devices' streams can
+    // race the pool:
     if (ggml_sycl_info().device_count > 1) {
-        // cont_to_f16 -> oneDNN execute -> permute is async on this stream, but the
-        // pool_alloc*s above free their device buffers at host return. Without this wait the next
-        // scheduler op re-acquires those bytes while the GPU is still computing the SDPA, turning
-        // it into garbage and collapsing multi-turn output to a single repeated token ("GGGGG...").
         stream->wait_and_throw();
     }
 }

ggml/src/ggml-sycl/ggml-sycl.cpp

@@ -85,6 +85,7 @@ int g_ggml_sycl_enable_optimize = 1;
 int g_ggml_sycl_enable_graph = 0;
 int g_ggml_sycl_enable_dnn = 1;
 int g_ggml_sycl_fa_onednn = 1;
+int g_ggml_sycl_fa_onednn_max_kv = 0;
 int g_ggml_sycl_enable_vmm = 1;
 int g_ggml_sycl_enable_fusion = 1;
 int g_ggml_sycl_prioritize_dmmv = 0;
@@ -287,6 +288,7 @@ static void ggml_check_sycl() try {
         g_ggml_sycl_enable_graph = ggml_sycl_get_env("GGML_SYCL_ENABLE_GRAPH", 0);
         g_ggml_sycl_enable_dnn = ggml_sycl_get_env("GGML_SYCL_ENABLE_DNN", 1);
         g_ggml_sycl_fa_onednn = ggml_sycl_get_env("GGML_SYCL_FA_ONEDNN", 1);
+        g_ggml_sycl_fa_onednn_max_kv = ggml_sycl_get_env("GGML_SYCL_FA_ONEDNN_MAX_KV", 0);
         g_ggml_sycl_enable_vmm = ggml_sycl_get_env("GGML_SYCL_ENABLE_VMM", 1);
         g_ggml_sycl_enable_fusion = ggml_sycl_get_env("GGML_SYCL_ENABLE_FUSION", 1);
         g_ggml_sycl_prioritize_dmmv = ggml_sycl_get_env("GGML_SYCL_PRIORITIZE_DMMV", 0);
@@ -359,6 +361,7 @@ static void ggml_check_sycl() try {
         GGML_LOG_INFO("  GGML_SYCL_ENABLE_DNN: DNN disabled by compile flag\n");
         GGML_LOG_INFO("  GGML_SYCL_FA_ONEDNN: %d\n", g_ggml_sycl_fa_onednn);
 #endif
+        GGML_LOG_INFO("  GGML_SYCL_FA_ONEDNN_MAX_KV: %d\n", g_ggml_sycl_fa_onednn_max_kv);
 #ifdef SYCL_FLASH_ATTN
         GGML_LOG_INFO("  GGML_SYCL_ENABLE_FLASH_ATTN: %d\n", g_ggml_sycl_enable_flash_attention);
 #else
@@ -6486,8 +6489,7 @@ ggml_backend_t ggml_backend_sycl_init(int device) {
         /* .guid    = */ ggml_backend_sycl_guid(),
         /* .iface   = */ ggml_backend_sycl_interface,
         /* .device  = */ ggml_backend_reg_dev_get(ggml_backend_sycl_reg(), device),
-        /* .context  = */ ctx,
-        /* .profiler = */ nullptr,
+        /* .context = */ ctx
     };
 
     return sycl_backend;

ggml/src/ggml-virtgpu/ggml-backend.cpp

@@ -65,7 +65,6 @@ ggml_backend_t ggml_backend_remoting_device_init(ggml_backend_dev_t dev, const c
         /* .interface = */ ggml_backend_remoting_interface,
         /* .device    = */ ggml_backend_reg_dev_get(ggml_backend_virtgpu_reg(), ctx->device),
         /* .context   = */ ctx,
-        /* .profiler  = */ nullptr,
     };
 
     return remoting_backend;

ggml/src/ggml-vulkan/ggml-vulkan.cpp

@@ -1,5 +1,4 @@
 #include "ggml-vulkan.h"
-#include "ggml-profiler.h"
 #include <vulkan/vulkan_core.h>
 #if defined(GGML_VULKAN_RUN_TESTS) || defined(GGML_VULKAN_CHECK_RESULTS)
 #include <chrono>
@@ -1971,8 +1970,8 @@ private:
 
 std::mutex vk_memory_logger::log_mutex;
 
-static bool vk_perf_logger_enabled = false;       // deprecated: use --profile instead
-static bool vk_perf_logger_concurrent = false;     // GGML_VK_PERF_LOGGER_CONCURRENT: use concurrent timestamp mode
+static bool vk_perf_logger_enabled = false;
+static bool vk_perf_logger_concurrent = false;
 static bool vk_enable_sync_logger = false;
 // number of calls between perf logger prints
 static uint32_t vk_perf_logger_frequency = 1;
@@ -2162,21 +2161,6 @@ class vk_perf_logger {
     uint32_t print_count {};
 };
 
-// Profiler state for the new ggml_backend_profiler interface
-struct ggml_vk_profiler_state {
-    bool enabled = false;
-    int  split_id = -1;
-
-    std::vector<ggml_profile_record> records;
-    std::vector<uint64_t>            cpu_timestamps;  // CPU-side timestamps for global ordering
-
-    void reset() {
-        records.clear();
-        cpu_timestamps.clear();
-        split_id = -1;
-    }
-};
-
 struct ggml_backend_vk_context {
     std::string name;
 
@@ -2237,9 +2221,8 @@ struct ggml_backend_vk_context {
     topk_moe_mode fused_topk_moe_mode {};
     bool fused_topk_moe_scale {};
 
-    // Profiling
-    std::unique_ptr<vk_perf_logger> perf_logger;   // legacy env-var profiler
-    ggml_vk_profiler_state * profiler_state = nullptr;
+    // for GGML_VK_PERF_LOGGER
+    std::unique_ptr<vk_perf_logger> perf_logger;
     vk::QueryPool query_pool;
     std::vector<const char *> query_fusion_names;
     std::vector<int> query_fusion_node_count;
@@ -14912,14 +14895,10 @@ static bool ggml_vk_build_graph(ggml_backend_vk_context * ctx, ggml_cgraph * cgr
             ctx->unsynced_nodes_read.clear();
             ggml_vk_sync_buffers(ctx, compute_ctx);
 
-            if ((vk_perf_logger_enabled || (ctx->profiler_state != nullptr && ctx->profiler_state->enabled))
-                    && vk_perf_logger_concurrent) {
+            if (vk_perf_logger_enabled && vk_perf_logger_concurrent) {
                 ctx->query_node_idx[ctx->query_idx] = node_idx;
                 compute_ctx->s->buffer->buf.writeTimestamp(vk::PipelineStageFlagBits::eAllCommands, ctx->query_pool, ctx->query_idx++);
                 ggml_vk_sync_buffers(ctx, compute_ctx);
-                if (ctx->profiler_state != nullptr && ctx->profiler_state->enabled) {
-                    ctx->profiler_state->cpu_timestamps.push_back(ggml_profiler_time_ns());
-                }
             }
         }
         // Add all fused nodes to the unsynchronized lists.
@@ -15466,7 +15445,7 @@ static void ggml_vk_cleanup(ggml_backend_vk_context * ctx) {
 
         ctx->transfer_cmd_pool.destroy(ctx->device->device);
     }
-    if (ctx->perf_logger) {
+    if (vk_perf_logger_enabled) {
         ctx->perf_logger->print_timings(true);
     }
 }
@@ -16576,9 +16555,7 @@ static ggml_status ggml_backend_vk_graph_compute(ggml_backend_t backend, ggml_cg
     ggml_vk_submit_transfer_ctx(ctx);
 
     vk_context compute_ctx;
-    bool profiling = vk_perf_logger_enabled ||
-                     (ctx->profiler_state != nullptr && ctx->profiler_state->enabled);
-    if (profiling) {
+    if (vk_perf_logger_enabled) {
         // allocate/resize the query pool
         if (ctx->num_queries < cgraph->n_nodes + 1) {
             if (ctx->query_pool) {
@@ -16606,10 +16583,6 @@ static ggml_status ggml_backend_vk_graph_compute(ggml_backend_t backend, ggml_cg
         ctx->query_idx = 0;
         compute_ctx->s->buffer->buf.writeTimestamp(vk::PipelineStageFlagBits::eAllCommands, ctx->query_pool, ctx->query_idx++);
         ggml_vk_sync_buffers(ctx, compute_ctx);
-        if (ctx->profiler_state != nullptr && ctx->profiler_state->enabled) {
-            ctx->profiler_state->cpu_timestamps.clear();
-            ctx->profiler_state->cpu_timestamps.push_back(ggml_profiler_time_ns());
-        }
     }
 
     ctx->prealloc_y_last_pipeline_used = nullptr;
@@ -16871,7 +16844,7 @@ static ggml_status ggml_backend_vk_graph_compute(ggml_backend_t backend, ggml_cg
 
         bool enqueued = ggml_vk_build_graph(ctx, cgraph, i, cgraph->nodes[submit_node_idx], submit_node_idx, i + ctx->num_additional_fused_ops >= last_node, almost_ready, submit);
 
-        if (profiling && enqueued) {
+        if (vk_perf_logger_enabled && enqueued) {
             compute_ctx = ggml_vk_get_compute_ctx(ctx);
             if (!vk_perf_logger_concurrent) {
                 // track a single node/fusion for the current query
@@ -16879,9 +16852,6 @@ static ggml_status ggml_backend_vk_graph_compute(ggml_backend_t backend, ggml_cg
                 ctx->query_fusion_names[ctx->query_idx] = fusion_string;
                 compute_ctx->s->buffer->buf.writeTimestamp(vk::PipelineStageFlagBits::eAllCommands, ctx->query_pool, ctx->query_idx++);
                 ggml_vk_sync_buffers(ctx, compute_ctx);
-                if (ctx->profiler_state != nullptr && ctx->profiler_state->enabled) {
-                    ctx->profiler_state->cpu_timestamps.push_back(ggml_profiler_time_ns());
-                }
             } else {
                 // track a fusion string and number of fused ops for the current node_idx
                 ctx->query_fusion_names[i] = fusion_string;
@@ -16915,7 +16885,7 @@ static ggml_status ggml_backend_vk_graph_compute(ggml_backend_t backend, ggml_cg
 
     ctx->last_total_flops = total_flops;
 
-    if (profiling) {
+    if (vk_perf_logger_enabled) {
         // End the command buffer and submit/wait
         GGML_ASSERT(!ctx->compute_ctx.expired());
         compute_ctx = ctx->compute_ctx.lock();
@@ -16929,40 +16899,15 @@ static ggml_status ggml_backend_vk_graph_compute(ggml_backend_t backend, ggml_cg
         // Get the results and pass them to the logger
         std::vector<uint64_t> timestamps(cgraph->n_nodes + 1);
         VK_CHECK(ctx->device->device.getQueryPoolResults(ctx->query_pool, 0, ctx->query_idx, (cgraph->n_nodes + 1)*sizeof(uint64_t), timestamps.data(), sizeof(uint64_t), vk::QueryResultFlagBits::e64 | vk::QueryResultFlagBits::eWait), "get timestamp results");
-
-        const double ts_period = ctx->device->properties.limits.timestampPeriod;
-        const bool has_profiler = ctx->profiler_state != nullptr && ctx->profiler_state->enabled;
-
         if (!vk_perf_logger_concurrent) {
             // Log each op separately
             for (int i = 1; i < ctx->query_idx; i++) {
                 auto node = ctx->query_nodes[i];
                 auto name = ctx->query_fusion_names[i];
-                uint64_t duration_ns = uint64_t((timestamps[i] - timestamps[i-1]) * ts_period);
-
-                if (ctx->perf_logger) {
-                    ctx->perf_logger->log_timing(node, name, duration_ns);
-                }
-
-                if (has_profiler && node != nullptr) {
-                    uint64_t cpu_ts = (i < (int)ctx->profiler_state->cpu_timestamps.size())
-                                    ? ctx->profiler_state->cpu_timestamps[i] : 0;
-
-                    ggml_profile_record rec;
-                    rec.type       = GGML_PROFILE_EVENT_OP;
-                    rec.name       = ggml_op_name(node->op);
-                    rec.backend_id = -1;
-                    rec.split_id   = ctx->profiler_state->split_id;
-                    rec.start_ns   = cpu_ts;
-                    rec.end_ns     = cpu_ts + duration_ns;
-                    rec.bytes      = ggml_nbytes(node);
-                    rec.extra      = name;  // fusion name or NULL
-                    ggml_profile_record_from_tensor(&rec, node);
-                    ctx->profiler_state->records.push_back(rec);
-                }
+                ctx->perf_logger->log_timing(node, name, uint64_t((timestamps[i] - timestamps[i-1]) * ctx->device->properties.limits.timestampPeriod));
             }
         } else {
-            // Log each group of nodes (concurrent mode)
+            // Log each group of nodes
             int prev_node_idx = 0;
             for (int i = 1; i < ctx->query_idx; i++) {
                 auto cur_node_idx = ctx->query_node_idx[i];
@@ -16977,40 +16922,10 @@ static ggml_status ggml_backend_vk_graph_compute(ggml_backend_t backend, ggml_cg
                     node_idx += ctx->query_fusion_node_count[node_idx];
                 }
                 prev_node_idx = cur_node_idx;
-                uint64_t duration_ns = uint64_t((timestamps[i] - timestamps[i-1]) * ts_period);
-
-                if (ctx->perf_logger) {
-                    ctx->perf_logger->log_timing(nodes, names, duration_ns);
-                }
-
-                if (has_profiler && !nodes.empty()) {
-                    uint64_t cpu_ts = (i < (int)ctx->profiler_state->cpu_timestamps.size())
-                                    ? ctx->profiler_state->cpu_timestamps[i] : 0;
-                    // In concurrent mode, report the group as a single combined operation
-                    auto * node = nodes[0];
-
-                    uint64_t total_bytes = 0;
-                    for (size_t j = 0; j < nodes.size(); j++) {
-                        total_bytes += ggml_nbytes(nodes[j]);
-                    }
-
-                    ggml_profile_record rec;
-                    rec.type       = GGML_PROFILE_EVENT_OP;
-                    rec.name       = ggml_op_name(node->op);
-                    rec.backend_id = -1;
-                    rec.split_id   = ctx->profiler_state->split_id;
-                    rec.start_ns   = cpu_ts;
-                    rec.end_ns     = cpu_ts + duration_ns;
-                    rec.bytes      = total_bytes;
-                    rec.extra      = names[0];  // fusion name of first op, or NULL
-                    ggml_profile_record_from_tensor(&rec, node);
-                    ctx->profiler_state->records.push_back(rec);
-                }
+                ctx->perf_logger->log_timing(nodes, names, uint64_t((timestamps[i] - timestamps[i-1]) * ctx->device->properties.limits.timestampPeriod));
             }
         }
-        if (ctx->perf_logger) {
-            ctx->perf_logger->print_timings();
-        }
+        ctx->perf_logger->print_timings();
     }
 
     if (!ctx->device->support_async) {
@@ -17371,66 +17286,13 @@ ggml_backend_t ggml_backend_vk_init(size_t dev_num) {
         /* .guid    = */ ggml_backend_vk_guid(),
         /* .iface   = */ ggml_backend_vk_interface,
         /* .device  = */ ggml_backend_reg_dev_get(ggml_backend_vk_reg(), dev_num),
-        /* .context  = */ ctx,
-        /* .profiler = */ nullptr,
+        /* .context = */ ctx,
     };
 
     if (!ctx->device->support_async) {
         vk_backend->iface.get_tensor_async = nullptr;
     }
 
-    // Register profiler
-    auto * prof_state = new ggml_vk_profiler_state();
-    ctx->profiler_state = prof_state;
-
-    static auto vk_prof_enable = [](void * context, bool enable) {
-        auto * vk_ctx = (ggml_backend_vk_context *)context;
-        if (vk_ctx->profiler_state != nullptr) {
-            vk_ctx->profiler_state->enabled = enable;
-            if (!enable) {
-                vk_ctx->profiler_state->reset();
-            }
-        }
-    };
-    static auto vk_prof_reset = [](void * context) {
-        auto * vk_ctx = (ggml_backend_vk_context *)context;
-        if (vk_ctx->profiler_state != nullptr) {
-            vk_ctx->profiler_state->reset();
-        }
-    };
-    static auto vk_prof_set_split_id = [](void * context, int split_id) {
-        auto * vk_ctx = (ggml_backend_vk_context *)context;
-        if (vk_ctx->profiler_state != nullptr) {
-            vk_ctx->profiler_state->split_id = split_id;
-        }
-    };
-    static auto vk_prof_get_records = [](void * context, const ggml_profile_record ** out) -> int {
-        auto * vk_ctx = (ggml_backend_vk_context *)context;
-        if (vk_ctx->profiler_state != nullptr) {
-            *out = vk_ctx->profiler_state->records.data();
-            return (int)vk_ctx->profiler_state->records.size();
-        }
-        *out = nullptr;
-        return 0;
-    };
-    static auto vk_prof_free = [](void * context) {
-        auto * vk_ctx = (ggml_backend_vk_context *)context;
-        if (vk_ctx->profiler_state != nullptr) {
-            delete vk_ctx->profiler_state;
-            vk_ctx->profiler_state = nullptr;
-        }
-    };
-
-    auto * profiler = new ggml_backend_profiler {
-        /* .context      = */ ctx,
-        /* .enable       = */ vk_prof_enable,
-        /* .reset        = */ vk_prof_reset,
-        /* .set_split_id = */ vk_prof_set_split_id,
-        /* .get_records  = */ vk_prof_get_records,
-        /* .free_context = */ vk_prof_free,
-    };
-    ggml_backend_set_profiler(vk_backend, profiler);
-
     return vk_backend;
 }
 

ggml/src/ggml-vulkan/vulkan-shaders/types.glsl

@@ -6,7 +6,6 @@
 #extension GL_EXT_shader_explicit_arithmetic_types_int16 : require
 #extension GL_EXT_shader_explicit_arithmetic_types_int8 : require
 #extension GL_EXT_shader_16bit_storage : require
-#extension GL_EXT_control_flow_attributes : require
 
 #ifdef USE_OCP_FP4
 #extension GL_EXT_float_e2m1 : require
@@ -1793,9 +1792,7 @@ shared FLOAT_TYPE kvalues_iq4nl[16];
 void init_iq_shmem(uvec3 wgsize)
 {
     // copy the table into shared memory and sync
-    // Use gl_LocalInvocationIndex (uint scalar) directly - each thread loads its portion
-    // with stride equal to workgroup size for coalesced shared memory writes
-    [[unroll]] for (uint i = gl_LocalInvocationIndex; i < kvalues_iq4nl.length(); i += wgsize.x) {
+    for (uint i = gl_LocalInvocationIndex.x; i < kvalues_iq4nl.length(); i += wgsize.x) {
         kvalues_iq4nl[i] = FLOAT_TYPE(kvalues_iq4nl_const[i]);
     }
     barrier();
@@ -1835,12 +1832,11 @@ float ue4m3_to_fp32_build(uint u) {
 void init_iq_shmem(uvec3 wgsize)
 {
     // copy the table into shared memory and sync
-    // Use gl_LocalInvocationIndex (uint scalar) directly for thread-strided initialization
-    [[unroll]] for (uint i = gl_LocalInvocationIndex; i < kvalues_mxfp4.length(); i += wgsize.x) {
+    for (uint i = gl_LocalInvocationIndex.x; i < kvalues_mxfp4.length(); i += wgsize.x) {
         kvalues_mxfp4[i] = kvalues_mxfp4_const[i];
     }
 #if defined(DATA_A_NVFP4)
-    [[unroll]] for (uint i = gl_LocalInvocationIndex; i < 128u; i += wgsize.x) {
+    for (uint i = gl_LocalInvocationIndex.x; i < 128u; i += wgsize.x) {
         ue4m3_fp32_lut[i] = ue4m3_to_fp32_build(i);
     }
 #endif

ggml/src/ggml-webgpu/ggml-webgpu.cpp

@@ -4067,7 +4067,6 @@ static ggml_backend_t ggml_backend_webgpu_backend_init(ggml_backend_dev_t dev, c
         /* .interface = */ ggml_backend_webgpu_i,
         /* .device    = */ dev,
         /* .context   = */ backend_ctx,
-        /* .profiler  = */ nullptr,
     };
     return backend;
 }

ggml/src/ggml-zdnn/ggml-zdnn.cpp

@@ -503,8 +503,7 @@ static ggml_backend_t ggml_backend_zdnn_device_init(ggml_backend_dev_t dev, cons
         /* .guid       = */ ggml_backend_zdnn_guid(),
         /* .iface      = */ ggml_backend_zdnn_i,
         /* .device     = */ dev,
-        /* .context    = */ ctx,
-        /* .profiler   = */ NULL,
+        /* .context    = */ ctx
     };
 
     return backend;

ggml/src/ggml-zendnn/ggml-zendnn.cpp

@@ -467,8 +467,7 @@ ggml_backend_t ggml_backend_zendnn_init(void) {
         /* .guid    = */ ggml_backend_zendnn_guid(),
         /* .iface   = */ ggml_backend_zendnn_i,
         /* .device  = */ ggml_backend_reg_dev_get(ggml_backend_zendnn_reg(), 0),
-        /* .context  = */ ctx,
-        /* .profiler = */ nullptr,
+        /* .context = */ ctx,
     };
 
     return backend;

ggml/src/gguf.cpp

@@ -1424,7 +1424,7 @@ void gguf_set_tensor_data(struct gguf_context * ctx, const char * name, const vo
 struct gguf_writer_base {
     size_t written_bytes {0u};
 
-    ~gguf_writer_base(void) = default;
+    virtual ~gguf_writer_base(void) = default;
 
     // we bet on devirtualization
     virtual void write(int8_t val) = 0;

gguf-py/gguf/constants.py

@@ -145,6 +145,8 @@ class Keys:
         TOKEN_SHIFT_COUNT                 = "{arch}.token_shift_count"
         INTERLEAVE_MOE_LAYER_STEP         = "{arch}.interleave_moe_layer_step"
         FULL_ATTENTION_INTERVAL           = "{arch}.full_attention_interval"
+        NUM_LOOPS                         = "{arch}.num_loops"
+        SKIP_LOOP_FINAL_NORM              = "{arch}.skip_loop_final_norm"
         HASH_LAYER_COUNT                  = "{arch}.hash_layer_count"
         ACTIVATION_SPARSITY_SCALE         = "{arch}.activation_sparsity_scale"
         ALTUP_ACTIVE_IDX                  = "{arch}.altup.active_idx"
@@ -159,6 +161,7 @@ class Keys:
         TARGET_HIDDEN_SIZE                = "{arch}.target_hidden_size"
         BLOCK_SIZE                        = "{arch}.block_size"
         NORM_BEFORE_RESIDUAL              = "{arch}.norm_before_residual"
+        NORM_BEFORE_FC                    = "{arch}.norm_before_fc"
 
     class Attention:
         HEAD_COUNT                   = "{arch}.attention.head_count"
@@ -200,6 +203,9 @@ class Keys:
             HEAD_COUNT = "{arch}.attention.indexer.head_count"
             KEY_LENGTH = "{arch}.attention.indexer.key_length"
             TOP_K      = "{arch}.attention.indexer.top_k"
+            BLOCK_SIZE   = "{arch}.attention.indexer.block_size"    # MSA
+            LOCAL_BLOCKS = "{arch}.attention.indexer.local_blocks"  # MSA
+            TYPES      = "{arch}.attention.indexer.types"
 
     class HyperConnection:
         COUNT                = "{arch}.hyper_connection.count"
@@ -371,6 +377,12 @@ class Keys:
         CONV_KERNEL_SIZE    = "clip.audio.conv_kernel_size"
         MAX_POS_EMB         = "clip.audio.max_pos_emb"
         FEATURE_LAYERS      = "clip.audio.feature_layer" # Granite Speech Plus
+        RVQ_NUM_QUANTIZERS  = "clip.audio.rvq.num_quantizers"
+        RVQ_CODEBOOK_SIZE   = "clip.audio.rvq.codebook_size"
+        WA_PATTERN_MODE     = "clip.audio.wa_pattern_mode" # per-layer -1 (full) / 0 (windowed)
+        WINDOW_SIZE         = "clip.audio.window_size"
+        LOCAL_BLOCK_COUNT   = "clip.audio.local_block_count" # mimo-v2.5: input_local_transformer layer count
+        LOCAL_GROUP_SIZE    = "clip.audio.local_group_size"  # mimo-v2.5: input_local_transformer grouping size
 
         class Attention:
             HEAD_COUNT      = "clip.audio.attention.head_count"
@@ -527,6 +539,7 @@ class MODEL_ARCH(IntEnum):
     APERTUS          = auto()
     COGVLM           = auto()
     MINIMAXM2        = auto()
+    MINIMAXM3        = auto()
     RND1             = auto()
     PANGU_EMBED      = auto()
     MISTRAL3         = auto()
@@ -541,6 +554,7 @@ class MODEL_ARCH(IntEnum):
     KIMI_LINEAR      = auto()
     TALKIE           = auto()
     MELLUM           = auto()
+    NANBEIGE         = auto()
 
 
 class VISION_PROJECTOR_TYPE(IntEnum):
@@ -773,6 +787,9 @@ class MODEL_TENSOR(IntEnum):
     INDEXER_PROJ         = auto()
     INDEXER_ATTN_K       = auto()
     INDEXER_ATTN_Q_B     = auto()
+    INDEXER_Q_PROJ       = auto()
+    INDEXER_K_PROJ       = auto()
+    INDEXER_Q_NORM       = auto()
     INDEXER_COMPRESSOR_WKV = auto()
     INDEXER_COMPRESSOR_WGATE = auto()
     INDEXER_COMPRESSOR_APE = auto()
@@ -850,6 +867,8 @@ class MODEL_TENSOR(IntEnum):
     V_MM_UP              = auto() # cogvlm
     V_MM_DOWN            = auto() # cogvlm
     V_MM_GATE            = auto() # cogvlm
+    V_MM_MERGER_FC1      = auto() # minimax-m3 (patch-merge MLP)
+    V_MM_MERGER_FC2      = auto() # minimax-m3 (patch-merge MLP)
     V_TOK_BOI            = auto() # cogvlm
     V_TOK_EOI            = auto() # cogvlm
     V_TOK_IMG_BEGIN      = auto() # hunyuanvl
@@ -933,6 +952,9 @@ class MODEL_TENSOR(IntEnum):
     A_ENC_FFN_SCALE_1     = auto() # gemma3n
     A_ENC_FFN_GATE_1      = auto() # lfm2, gemma3n
     A_ENC_FFN_DOWN_1      = auto() # lfm2, gemma3n
+    A_ENC_DOWNSAMPLE_CONV = auto() # mimo-audio-tokenizer: post-transformer downsample conv
+    A_ENC_DOWNSAMPLE_NORM = auto() # mimo-audio-tokenizer: post-transformer downsample norm
+    A_ENC_RVQ_CODEBOOK    = auto() # mimo-audio-tokenizer: residual vector quantizer codebook, per quantizer index
     A_MMPROJ              = auto()
     A_MMPROJ_FC           = auto()
     A_MM_NORM_PRE         = auto()
@@ -941,6 +963,17 @@ class MODEL_TENSOR(IntEnum):
     A_MM_HARD_EMB_NORM    = auto() # gemma3n
     A_MM_SOFT_EMB_NORM    = auto() # gemma3n
     A_MM_INP_PROJ         = auto() # gemma3n
+    A_MM_CODE_EMBD        = auto() # mimo: text-side RVQ code embedding table ("text codebook"), merged 3D [n_channels, vocab, dim]
+    A_MM_LOCAL_ATTN_Q     = auto() # mimo: input_local_transformer (LLM-side connector)
+    A_MM_LOCAL_ATTN_K     = auto()
+    A_MM_LOCAL_ATTN_V     = auto()
+    A_MM_LOCAL_ATTN_OUT   = auto()
+    A_MM_LOCAL_FFN_GATE   = auto()
+    A_MM_LOCAL_FFN_UP     = auto()
+    A_MM_LOCAL_FFN_DOWN   = auto()
+    A_MM_LOCAL_LN1        = auto()
+    A_MM_LOCAL_LN2        = auto()
+    A_MM_LOCAL_NORM       = auto() # final norm after all input_local_transformer layers
     A_PER_DIM_K_SCALE     = auto() # gemma4
     A_PER_DIM_SCALE       = auto() # gemma4
     # nextn/mtp
@@ -1109,6 +1142,7 @@ MODEL_ARCH_NAMES: dict[MODEL_ARCH, str] = {
     MODEL_ARCH.GROVEMOE:         "grovemoe",
     MODEL_ARCH.APERTUS:          "apertus",
     MODEL_ARCH.MINIMAXM2:        "minimax-m2",
+    MODEL_ARCH.MINIMAXM3:        "minimax-m3",
     MODEL_ARCH.COGVLM:           "cogvlm",
     MODEL_ARCH.RND1:             "rnd1",
     MODEL_ARCH.PANGU_EMBED:      "pangu-embedded",
@@ -1124,6 +1158,7 @@ MODEL_ARCH_NAMES: dict[MODEL_ARCH, str] = {
     MODEL_ARCH.KIMI_LINEAR:      "kimi-linear",
     MODEL_ARCH.TALKIE:           "talkie",
     MODEL_ARCH.MELLUM:           "mellum",
+    MODEL_ARCH.NANBEIGE:         "nanbeige",
 }
 
 VISION_PROJECTOR_TYPE_NAMES: dict[VISION_PROJECTOR_TYPE, str] = {
@@ -1354,6 +1389,9 @@ TENSOR_NAMES: dict[MODEL_TENSOR, str] = {
     MODEL_TENSOR.INDEXER_PROJ:              "blk.{bid}.indexer.proj",
     MODEL_TENSOR.INDEXER_ATTN_K:            "blk.{bid}.indexer.attn_k",
     MODEL_TENSOR.INDEXER_ATTN_Q_B:          "blk.{bid}.indexer.attn_q_b",
+    MODEL_TENSOR.INDEXER_Q_PROJ:            "blk.{bid}.indexer.q_proj",
+    MODEL_TENSOR.INDEXER_K_PROJ:            "blk.{bid}.indexer.k_proj",
+    MODEL_TENSOR.INDEXER_Q_NORM:            "blk.{bid}.indexer.q_norm",
     MODEL_TENSOR.INDEXER_COMPRESSOR_WKV:    "blk.{bid}.indexer_compressor_kv",
     MODEL_TENSOR.INDEXER_COMPRESSOR_WGATE:  "blk.{bid}.indexer_compressor_gate",
     MODEL_TENSOR.INDEXER_COMPRESSOR_APE:    "blk.{bid}.indexer_compressor_ape",
@@ -1430,6 +1468,8 @@ TENSOR_NAMES: dict[MODEL_TENSOR, str] = {
     MODEL_TENSOR.V_MM_UP:                   "mm.up",
     MODEL_TENSOR.V_MM_DOWN:                 "mm.down",
     MODEL_TENSOR.V_MM_GATE:                 "mm.gate",
+    MODEL_TENSOR.V_MM_MERGER_FC1:            "mm.merger.fc1",
+    MODEL_TENSOR.V_MM_MERGER_FC2:            "mm.merger.fc2",
     MODEL_TENSOR.V_TOK_BOI:                 "v.boi",
     MODEL_TENSOR.V_TOK_EOI:                 "v.eoi",
     MODEL_TENSOR.V_MM_PRE_NORM:             "mm.pre_norm",
@@ -1513,6 +1553,9 @@ TENSOR_NAMES: dict[MODEL_TENSOR, str] = {
     MODEL_TENSOR.A_ENC_FFN_UP_1:            "a.blk.{bid}.ffn_up_1",
     MODEL_TENSOR.A_ENC_FFN_GATE_1:          "a.blk.{bid}.ffn_gate_1",
     MODEL_TENSOR.A_ENC_FFN_DOWN_1:          "a.blk.{bid}.ffn_down_1",
+    MODEL_TENSOR.A_ENC_DOWNSAMPLE_CONV:     "a.downsample.conv",
+    MODEL_TENSOR.A_ENC_DOWNSAMPLE_NORM:     "a.downsample.norm",
+    MODEL_TENSOR.A_ENC_RVQ_CODEBOOK:        "a.rvq.codebook",
     MODEL_TENSOR.A_MMPROJ:                  "mm.a.mlp.{bid}",
     MODEL_TENSOR.A_MMPROJ_FC:               "mm.a.fc",
     MODEL_TENSOR.A_MM_NORM_PRE:             "mm.a.norm_pre",
@@ -1521,6 +1564,17 @@ TENSOR_NAMES: dict[MODEL_TENSOR, str] = {
     MODEL_TENSOR.A_MM_SOFT_EMB_NORM:        "mm.a.soft_emb_norm",         # gemma3n
     MODEL_TENSOR.A_MM_EMBEDDING:            "mm.a.embedding",             # gemma3n
     MODEL_TENSOR.A_MM_HARD_EMB_NORM:        "mm.a.hard_emb_norm",         # gemma3n
+    MODEL_TENSOR.A_MM_CODE_EMBD:            "mm.a.code_embd",
+    MODEL_TENSOR.A_MM_LOCAL_ATTN_Q:         "mm.a.local_blk.{bid}.attn_q",
+    MODEL_TENSOR.A_MM_LOCAL_ATTN_K:         "mm.a.local_blk.{bid}.attn_k",
+    MODEL_TENSOR.A_MM_LOCAL_ATTN_V:         "mm.a.local_blk.{bid}.attn_v",
+    MODEL_TENSOR.A_MM_LOCAL_ATTN_OUT:       "mm.a.local_blk.{bid}.attn_out",
+    MODEL_TENSOR.A_MM_LOCAL_FFN_GATE:       "mm.a.local_blk.{bid}.ffn_gate",
+    MODEL_TENSOR.A_MM_LOCAL_FFN_UP:         "mm.a.local_blk.{bid}.ffn_up",
+    MODEL_TENSOR.A_MM_LOCAL_FFN_DOWN:       "mm.a.local_blk.{bid}.ffn_down",
+    MODEL_TENSOR.A_MM_LOCAL_LN1:            "mm.a.local_blk.{bid}.ln1",
+    MODEL_TENSOR.A_MM_LOCAL_LN2:            "mm.a.local_blk.{bid}.ln2",
+    MODEL_TENSOR.A_MM_LOCAL_NORM:           "mm.a.local_norm",
     MODEL_TENSOR.A_PER_DIM_K_SCALE:         "a.blk.{bid}.per_dim_k_scale", # gemma4
     MODEL_TENSOR.A_PER_DIM_SCALE:           "a.blk.{bid}.per_dim_scale",   # gemma4
     # lfm2 audio
@@ -1626,6 +1680,8 @@ MODEL_TENSORS: dict[MODEL_ARCH, list[MODEL_TENSOR]] = {
         MODEL_TENSOR.V_RESMPL_QUERY,
         MODEL_TENSOR.V_TOK_EMBD_IMG_BREAK,
         MODEL_TENSOR.V_MM_PATCH_MERGER,
+        MODEL_TENSOR.V_MM_MERGER_FC1,
+        MODEL_TENSOR.V_MM_MERGER_FC2,
         MODEL_TENSOR.V_DS_NORM,
         MODEL_TENSOR.V_DS_FC1,
         MODEL_TENSOR.V_DS_FC2,
@@ -1720,10 +1776,24 @@ MODEL_TENSORS: dict[MODEL_ARCH, list[MODEL_TENSOR]] = {
         MODEL_TENSOR.A_ENC_FFN_UP_1,
         MODEL_TENSOR.A_ENC_FFN_GATE_1,
         MODEL_TENSOR.A_ENC_FFN_DOWN_1,
+        MODEL_TENSOR.A_ENC_DOWNSAMPLE_CONV,
+        MODEL_TENSOR.A_ENC_DOWNSAMPLE_NORM,
+        MODEL_TENSOR.A_ENC_RVQ_CODEBOOK,
         MODEL_TENSOR.A_MMPROJ,
         MODEL_TENSOR.A_MMPROJ_FC,
         MODEL_TENSOR.A_MM_NORM_PRE,
         MODEL_TENSOR.A_MM_NORM_MID,
+        MODEL_TENSOR.A_MM_CODE_EMBD,
+        MODEL_TENSOR.A_MM_LOCAL_ATTN_Q,
+        MODEL_TENSOR.A_MM_LOCAL_ATTN_K,
+        MODEL_TENSOR.A_MM_LOCAL_ATTN_V,
+        MODEL_TENSOR.A_MM_LOCAL_ATTN_OUT,
+        MODEL_TENSOR.A_MM_LOCAL_FFN_GATE,
+        MODEL_TENSOR.A_MM_LOCAL_FFN_UP,
+        MODEL_TENSOR.A_MM_LOCAL_FFN_DOWN,
+        MODEL_TENSOR.A_MM_LOCAL_LN1,
+        MODEL_TENSOR.A_MM_LOCAL_LN2,
+        MODEL_TENSOR.A_MM_LOCAL_NORM,
         MODEL_TENSOR.A_ENC_NORM_CONV,
         MODEL_TENSOR.A_ENC_LINEAR_POS,
         MODEL_TENSOR.A_ENC_POS_BIAS_U,
@@ -4162,6 +4232,34 @@ MODEL_TENSORS: dict[MODEL_ARCH, list[MODEL_TENSOR]] = {
         MODEL_TENSOR.FFN_UP_EXP,
         MODEL_TENSOR.FFN_EXP_PROBS_B,
     ],
+    MODEL_ARCH.MINIMAXM3: [
+        MODEL_TENSOR.TOKEN_EMBD,
+        MODEL_TENSOR.OUTPUT_NORM,
+        MODEL_TENSOR.OUTPUT,
+        MODEL_TENSOR.ATTN_NORM,
+        MODEL_TENSOR.ATTN_Q,
+        MODEL_TENSOR.ATTN_Q_NORM,
+        MODEL_TENSOR.ATTN_K,
+        MODEL_TENSOR.ATTN_K_NORM,
+        MODEL_TENSOR.ATTN_V,
+        MODEL_TENSOR.ATTN_OUT,
+        MODEL_TENSOR.FFN_NORM,
+        MODEL_TENSOR.FFN_GATE_INP,
+        MODEL_TENSOR.FFN_EXP_PROBS_B,
+        MODEL_TENSOR.FFN_GATE_EXP,
+        MODEL_TENSOR.FFN_DOWN_EXP,
+        MODEL_TENSOR.FFN_UP_EXP,
+        MODEL_TENSOR.FFN_GATE_SHEXP,
+        MODEL_TENSOR.FFN_DOWN_SHEXP,
+        MODEL_TENSOR.FFN_UP_SHEXP,
+        MODEL_TENSOR.FFN_GATE,
+        MODEL_TENSOR.FFN_DOWN,
+        MODEL_TENSOR.FFN_UP,
+        MODEL_TENSOR.INDEXER_Q_PROJ,
+        MODEL_TENSOR.INDEXER_K_PROJ,
+        MODEL_TENSOR.INDEXER_Q_NORM,
+        MODEL_TENSOR.INDEXER_K_NORM,
+    ],
     MODEL_ARCH.COGVLM: [
         MODEL_TENSOR.TOKEN_EMBD,
         MODEL_TENSOR.OUTPUT_NORM,
@@ -4246,6 +4344,7 @@ MODEL_TENSORS: dict[MODEL_ARCH, list[MODEL_TENSOR]] = {
         MODEL_TENSOR.FFN_DOWN,
         MODEL_TENSOR.FFN_UP,
         MODEL_TENSOR.FC,
+        MODEL_TENSOR.ENC_OUTPUT_NORM,
         MODEL_TENSOR.D2T,
     ],
     MODEL_ARCH.DFLASH: [
@@ -4460,7 +4559,22 @@ MODEL_TENSORS: dict[MODEL_ARCH, list[MODEL_TENSOR]] = {
         MODEL_TENSOR.FFN_DOWN_EXP,
         MODEL_TENSOR.FFN_UP_EXP,
     ],
-    # TODO
+    MODEL_ARCH.NANBEIGE: [
+        MODEL_TENSOR.TOKEN_EMBD,
+        MODEL_TENSOR.OUTPUT_NORM,
+        MODEL_TENSOR.OUTPUT,
+        MODEL_TENSOR.ROPE_FREQS,
+        MODEL_TENSOR.ATTN_NORM,
+        MODEL_TENSOR.ATTN_Q,
+        MODEL_TENSOR.ATTN_K,
+        MODEL_TENSOR.ATTN_V,
+        MODEL_TENSOR.ATTN_OUT,
+        MODEL_TENSOR.ATTN_ROT_EMBD,
+        MODEL_TENSOR.FFN_NORM,
+        MODEL_TENSOR.FFN_GATE,
+        MODEL_TENSOR.FFN_DOWN,
+        MODEL_TENSOR.FFN_UP,
+    ],
 }
 
 # tensors that will not be serialized
@@ -4527,6 +4641,10 @@ MODEL_TENSOR_SKIP: dict[MODEL_ARCH, list[MODEL_TENSOR]] = {
         MODEL_TENSOR.ROPE_FREQS,
         MODEL_TENSOR.ATTN_ROT_EMBD,
     ],
+    MODEL_ARCH.NANBEIGE: [
+        MODEL_TENSOR.ROPE_FREQS,
+        MODEL_TENSOR.ATTN_ROT_EMBD,
+    ],
 }
 
 #
@@ -4732,9 +4850,11 @@ class VisionProjectorType:
     YOUTUVL = "youtuvl"
     NEMOTRON_V2_VL = "nemotron_v2_vl"
     HUNYUANVL      = "hunyuanvl"
+    MINIMAXM3      = "minimax_m3"
     MINICPMV4_6    = "minicpmv4_6"
     GRANITE_SPEECH = "granite_speech"  # audio
     MIMOVL         = "mimovl"
+    MIMO_AUDIO     = "mimo_audio"
     GRANITE4_VISION = "granite4_vision"
 
 

gguf-py/gguf/gguf_writer.py

@@ -793,6 +793,16 @@ class GGUFWriter:
     def add_indexer_top_k(self, top_k: int) -> None:
         self.add_uint32(Keys.Attention.Indexer.TOP_K.format(arch=self.arch), top_k)
 
+    def add_indexer_block_size(self, block_size: int) -> None:
+        self.add_uint32(Keys.Attention.Indexer.BLOCK_SIZE.format(arch=self.arch), block_size)
+
+    def add_indexer_local_blocks(self, local_blocks: int) -> None:
+        self.add_uint32(Keys.Attention.Indexer.LOCAL_BLOCKS.format(arch=self.arch), local_blocks)
+
+    def add_indexer_types(self, value: Sequence[bool]) -> None:
+        key = Keys.Attention.Indexer.TYPES.format(arch=self.arch)
+        self.add_array(key, value)
+
     def add_max_alibi_bias(self, bias: float) -> None:
         self.add_float32(Keys.Attention.MAX_ALIBI_BIAS.format(arch=self.arch), bias)
 
@@ -898,6 +908,12 @@ class GGUFWriter:
     def add_token_shift_count(self, count: int) -> None:
         self.add_uint32(Keys.LLM.TOKEN_SHIFT_COUNT.format(arch=self.arch), count)
 
+    def add_num_loops(self, count: int) -> None:
+        self.add_uint32(Keys.LLM.NUM_LOOPS.format(arch=self.arch), count)
+
+    def add_skip_loop_final_norm(self, value: bool) -> None:
+        self.add_bool(Keys.LLM.SKIP_LOOP_FINAL_NORM.format(arch=self.arch), value)
+
     def add_interleave_moe_layer_step(self, value: int) -> None:
         self.add_uint32(Keys.LLM.INTERLEAVE_MOE_LAYER_STEP.format(arch=self.arch), value)
 
@@ -955,6 +971,9 @@ class GGUFWriter:
     def add_norm_before_residual(self, value: bool) -> None:
         self.add_bool(Keys.LLM.NORM_BEFORE_RESIDUAL.format(arch=self.arch), value)
 
+    def add_norm_before_fc(self, value: bool) -> None:
+        self.add_bool(Keys.LLM.NORM_BEFORE_FC.format(arch=self.arch), value)
+
     def add_attention_output_group_count(self, count: int) -> None:
         self.add_uint32(Keys.Attention.OUTPUT_GROUP_COUNT.format(arch=self.arch), count)
 
@@ -1334,6 +1353,24 @@ class GGUFWriter:
     def add_audio_num_mel_bins(self, value: int) -> None:
         self.add_uint32(Keys.ClipAudio.NUM_MEL_BINS, value)
 
+    def add_audio_rvq_num_quantizers(self, value: int) -> None:
+        self.add_uint32(Keys.ClipAudio.RVQ_NUM_QUANTIZERS, value)
+
+    def add_audio_rvq_codebook_size(self, values: Sequence[int]) -> None:
+        self.add_array(Keys.ClipAudio.RVQ_CODEBOOK_SIZE, values)
+
+    def add_audio_wa_pattern_mode(self, modes: Sequence[int]) -> None:
+        self.add_array(Keys.ClipAudio.WA_PATTERN_MODE, modes)
+
+    def add_audio_window_size(self, value: int) -> None:
+        self.add_uint32(Keys.ClipAudio.WINDOW_SIZE, value)
+
+    def add_audio_local_block_count(self, value: int) -> None:
+        self.add_uint32(Keys.ClipAudio.LOCAL_BLOCK_COUNT, value)
+
+    def add_audio_local_group_size(self, value: int) -> None:
+        self.add_uint32(Keys.ClipAudio.LOCAL_GROUP_SIZE, value)
+
     def add_audio_stack_factor(self, value: int) -> None:
         self.add_uint32(Keys.ClipAudio.Projector.STACK_FACTOR, value)
 

gguf-py/gguf/tensor_mapping.py

@@ -1264,7 +1264,8 @@ class TensorNameMap:
         ),
 
         MODEL_TENSOR.INDEXER_K_NORM: (
-            "model.layers.{bid}.self_attn.indexer.k_norm", # DSA
+            "model.layers.{bid}.self_attn.indexer.k_norm",  # DSA
+            "model.layers.{bid}.self_attn.index_k_norm",    # MSA
         ),
 
         MODEL_TENSOR.INDEXER_PROJ: (
@@ -1279,6 +1280,18 @@ class TensorNameMap:
             "model.layers.{bid}.self_attn.indexer.wq_b", # DSA
         ),
 
+        MODEL_TENSOR.INDEXER_Q_PROJ: (
+            "model.layers.{bid}.self_attn.index_q_proj", # MSA
+        ),
+
+        MODEL_TENSOR.INDEXER_K_PROJ: (
+            "model.layers.{bid}.self_attn.index_k_proj", # MSA
+        ),
+
+        MODEL_TENSOR.INDEXER_Q_NORM: (
+            "model.layers.{bid}.self_attn.index_q_norm", # MSA
+        ),
+
         ############################################################################
         # TODO: these do not belong to block_mappings_cfg - move them to mappings_cfg
         MODEL_TENSOR.ENC_OUTPUT_NORM: (
@@ -1825,6 +1838,14 @@ class TensorNameMap:
             "visual.downsample", # glm4v
         ),
 
+        MODEL_TENSOR.V_MM_MERGER_FC1: (
+            "patch_merge_mlp.linear_1", # minimax-m3
+        ),
+
+        MODEL_TENSOR.V_MM_MERGER_FC2: (
+            "patch_merge_mlp.linear_2", # minimax-m3
+        ),
+
         MODEL_TENSOR.V_DS_NORM: (
             "model.visual.deepstack_merger_list.{bid}.norm", # deepstack in qwen3vl
         ),
@@ -2074,6 +2095,7 @@ class TensorNameMap:
             "conformer.pre_encode.conv.{bid}", # lfm2
             "model.audio_tower.subsample_conv_projection.conv_{bid}.conv", # gemma3n
             "conformer.subsample_conv_projection.layer{bid}.conv", # gemma4
+            "encoder.conv{bid}", # mimo-audio-tokenizer
         ),
 
         MODEL_TENSOR.A_ENC_CONV1D_NORM: (
@@ -2098,6 +2120,7 @@ class TensorNameMap:
         MODEL_TENSOR.A_POST_NORM: (
             "audio_tower.layer_norm", # ultravox
             "audio_tower.ln_post", # qwen2omni
+            "encoder.layer_norm", # mimo-audio-tokenizer
         ),
 
         MODEL_TENSOR.A_ENC_ATTN_Q: (
@@ -2106,6 +2129,7 @@ class TensorNameMap:
             "conformer.layers.{bid}.attention.attn.q_proj", # gemma3n
             "conformer.layers.{bid}.self_attn.q_proj", # gemma4
             "encoder.layers.{bid}.attn.to_q", # granite_speech
+            "encoder.layers.{bid}.self_attn.q_proj", # mimo-audio-tokenizer
         ),
 
         MODEL_TENSOR.A_ENC_ATTN_K: (
@@ -2114,6 +2138,7 @@ class TensorNameMap:
             "conformer.layers.{bid}.attention.attn.k_proj", # gemma3n
             "conformer.layers.{bid}.self_attn.k_proj", # gemma4
             "encoder.layers.{bid}.attn.to_k", # granite_speech (split from to_kv)
+            "encoder.layers.{bid}.self_attn.k_proj", # mimo-audio-tokenizer
         ),
 
         MODEL_TENSOR.A_ENC_ATTN_V: (
@@ -2122,6 +2147,7 @@ class TensorNameMap:
             "conformer.layers.{bid}.attention.attn.v_proj", # gemma3n
             "conformer.layers.{bid}.self_attn.v_proj", # gemma4
             "encoder.layers.{bid}.attn.to_v", # granite_speech (split from to_kv)
+            "encoder.layers.{bid}.self_attn.v_proj", # mimo-audio-tokenizer
         ),
 
         MODEL_TENSOR.A_ENC_ATTN_K_REL: (
@@ -2150,6 +2176,7 @@ class TensorNameMap:
             "conformer.layers.{bid}.norm_self_att", # lfm2
             "conformer.layers.{bid}.attention.pre_attn_norm", # gemma3n
             "encoder.layers.{bid}.attn.pre_norm", # granite_speech
+            "encoder.layers.{bid}.self_attn_layer_norm", # mimo-audio-tokenizer
         ),
 
         MODEL_TENSOR.A_ENC_OUTPUT: (
@@ -2158,6 +2185,7 @@ class TensorNameMap:
             "conformer.layers.{bid}.attention.post", # gemma3n
             "conformer.layers.{bid}.self_attn.post", # gemma4
             "encoder.layers.{bid}.attn.to_out", # granite_speech
+            "encoder.layers.{bid}.self_attn.out_proj", # mimo-audio-tokenizer
         ),
 
         MODEL_TENSOR.A_ENC_OUTPUT_NORM: (
@@ -2165,6 +2193,7 @@ class TensorNameMap:
             "conformer.layers.{bid}.norm_out", # lfm2
             "conformer.layers.{bid}.attention.post_norm", # gemma3n
             "encoder.layers.{bid}.post_norm", # granite_speech
+            "encoder.layers.{bid}.final_layer_norm", # mimo-audio-tokenizer
         ),
 
         MODEL_TENSOR.A_ENC_FFN_NORM: (
@@ -2189,6 +2218,7 @@ class TensorNameMap:
             "conformer.layers.{bid}.ffw_layer_start.ffw_layer_1", # gemma3n
             "conformer.layers.{bid}.feed_forward1.ffw_layer_1", # gemma4
             "encoder.layers.{bid}.ff1.up_proj", # granite_speech
+            "encoder.layers.{bid}.fc1", # mimo-audio-tokenizer
         ),
 
         MODEL_TENSOR.A_ENC_FFN_GATE: (),
@@ -2199,6 +2229,7 @@ class TensorNameMap:
             "conformer.layers.{bid}.ffw_layer_start.ffw_layer_2", # gemma3n
             "conformer.layers.{bid}.feed_forward1.ffw_layer_2", # gemma4
             "encoder.layers.{bid}.ff1.down_proj", # granite_speech
+            "encoder.layers.{bid}.fc2", # mimo-audio-tokenizer
         ),
 
         MODEL_TENSOR.A_ENC_FFN_UP_1: (
@@ -2222,6 +2253,19 @@ class TensorNameMap:
             "encoder.layers.{bid}.ff2.pre_norm", # granite_speech
         ),
 
+        MODEL_TENSOR.A_ENC_DOWNSAMPLE_CONV: (
+            "encoder.down_sample_layer.0", # mimo-audio-tokenizer
+        ),
+
+        MODEL_TENSOR.A_ENC_DOWNSAMPLE_NORM: (
+            "encoder.down_sample_norm", # mimo-audio-tokenizer
+        ),
+
+        # note: the raw per-quantizer "encoder.quantizer.vq.layers.{i}._codebook.embed"
+        # tensors are merged (padded + stacked, like MoE experts) into this single 3D
+        # tensor in conversion code, so no raw-name mapping is registered here.
+        MODEL_TENSOR.A_ENC_RVQ_CODEBOOK: (),
+
         MODEL_TENSOR.A_ENC_FFN_POST_NORM_1: (
             "conformer.layers.{bid}.ffw_layer_end.post_layer_norm", # gemma3n
             "conformer.layers.{bid}.feed_forward2.post_layer_norm", # gemma4
@@ -2273,6 +2317,42 @@ class TensorNameMap:
             "audio.multi_modal_projector.ln_mid", # ultravox
         ),
 
+        # note: the raw per-channel "speech_embeddings.{i}" tensors are merged
+        # (stacked, like MoE experts) into this single 3D tensor in conversion
+        # code, so no raw-name mapping is registered here.
+        MODEL_TENSOR.A_MM_CODE_EMBD: (),
+
+        MODEL_TENSOR.A_MM_LOCAL_ATTN_Q: (
+            "audio_encoder.input_local_transformer.layers.{bid}.self_attn.q_proj", # mimo-v2.5
+        ),
+        MODEL_TENSOR.A_MM_LOCAL_ATTN_K: (
+            "audio_encoder.input_local_transformer.layers.{bid}.self_attn.k_proj", # mimo-v2.5
+        ),
+        MODEL_TENSOR.A_MM_LOCAL_ATTN_V: (
+            "audio_encoder.input_local_transformer.layers.{bid}.self_attn.v_proj", # mimo-v2.5
+        ),
+        MODEL_TENSOR.A_MM_LOCAL_ATTN_OUT: (
+            "audio_encoder.input_local_transformer.layers.{bid}.self_attn.o_proj", # mimo-v2.5
+        ),
+        MODEL_TENSOR.A_MM_LOCAL_FFN_GATE: (
+            "audio_encoder.input_local_transformer.layers.{bid}.mlp.gate_proj", # mimo-v2.5
+        ),
+        MODEL_TENSOR.A_MM_LOCAL_FFN_UP: (
+            "audio_encoder.input_local_transformer.layers.{bid}.mlp.up_proj", # mimo-v2.5
+        ),
+        MODEL_TENSOR.A_MM_LOCAL_FFN_DOWN: (
+            "audio_encoder.input_local_transformer.layers.{bid}.mlp.down_proj", # mimo-v2.5
+        ),
+        MODEL_TENSOR.A_MM_LOCAL_LN1: (
+            "audio_encoder.input_local_transformer.layers.{bid}.input_layernorm", # mimo-v2.5
+        ),
+        MODEL_TENSOR.A_MM_LOCAL_LN2: (
+            "audio_encoder.input_local_transformer.layers.{bid}.post_attention_layernorm", # mimo-v2.5
+        ),
+        MODEL_TENSOR.A_MM_LOCAL_NORM: (
+            "audio_encoder.input_local_transformer.norm", # mimo-v2.5
+        ),
+
         MODEL_TENSOR.A_ENC_CONV_DW: (
             "conformer.layers.{bid}.conv.depthwise_conv", # lfm2
             "conformer.layers.{bid}.lconv1d.depthwise_conv1d", # gemma3n

include/llama.h

@@ -203,10 +203,11 @@ extern "C" {
     };
 
     enum llama_load_mode {
-        LLAMA_LOAD_MODE_NONE      = 0, // no special loading mode
-        LLAMA_LOAD_MODE_MMAP      = 1, // memory map the model
-        LLAMA_LOAD_MODE_MLOCK     = 2, // mmap + force system to keep model in RAM rather than swapping or compressing
-        LLAMA_LOAD_MODE_DIRECT_IO = 3, // use direct I/O if available
+        LLAMA_LOAD_MODE_NONE       = 0, // no special loading mode
+        LLAMA_LOAD_MODE_MMAP       = 1, // memory map the model
+        LLAMA_LOAD_MODE_MLOCK      = 2, // force system to keep model in RAM rather than swapping or compressing
+        LLAMA_LOAD_MODE_MMAP_MLOCK = 3, // mmap + force system to keep model in RAM rather than swapping or compressing
+        LLAMA_LOAD_MODE_DIRECT_IO  = 4, // use direct I/O if available
     };
 
     LLAMA_API const char * llama_load_mode_name(enum llama_load_mode load_mode);
@@ -567,8 +568,6 @@ extern "C" {
     LLAMA_API           llama_memory_t   llama_get_memory  (const struct llama_context * ctx);
     LLAMA_API  enum llama_pooling_type   llama_pooling_type(const struct llama_context * ctx); // TODO: rename to llama_get_pooling_type
 
-    LLAMA_API struct ggml_backend_sched * llama_context_get_sched(const struct llama_context * ctx);
-
     LLAMA_API const struct llama_vocab * llama_model_get_vocab(const struct llama_model * model);
     LLAMA_API enum llama_rope_type       llama_model_rope_type(const struct llama_model * model);
 

scripts/sync_vendor.py

@@ -5,7 +5,7 @@ import os
 import sys
 import subprocess
 
-HTTPLIB_VERSION = "refs/tags/v0.50.1"
+HTTPLIB_VERSION = "refs/tags/v0.51.0"
 
 vendor = {
     "https://github.com/nlohmann/json/releases/latest/download/json.hpp":     "vendor/nlohmann/json.hpp",

skills/add-new-model/SKILL.md

@@ -0,0 +1,99 @@
+---
+name: add-new-model
+description: Guided workflow for adding a new model architecture to llama.cpp. Use when the user wants to add/port a new model architecture.
+---
+
+# Add a new model architecture to llama.cpp
+
+This skill walks a contributor through adding a new model architecture. AI-generated code is permitted in this project, so you may write full implementations for the steps below rather than only pointing at patterns - but follow `AGENTS.md`'s AI usage policy throughout:
+
+- The contributor is 100% responsible for every line, however it was produced. They must be able to explain and defend any part of it to a reviewer. Check in with them as you go (don't silently generate everything and hand over a finished diff) so they actually absorb what was written.
+- Before writing code, make sure the contributor owns the design choices for this architecture (which reference model to follow, how non-standard bits like RoPE variants or MoE routing should be handled) - AI accelerates a design the contributor has already made, it doesn't make the design for them.
+- Disclosure is mandatory: any AI-meaningful contribution must be disclosed per the PR template. Remind the contributor of this before they open the PR.
+- Never write the PR description, commit message, GitHub issue/discussion post, or reviewer replies - those must come from the contributor. If asked to commit on their behalf, use `Assisted-by:` (never `Co-authored-by:`) and only after explicit confirmation.
+- If the requested change looks large or introduces a new pattern not covered here, pause and tell the user this kind of change is likely to need prior discussion with maintainers before a PR.
+- Keep the PR self-contained. If the work would require a lot of unconventional changes outside the new model file(s) (e.g. touching shared graph-building code, the sampler, or core APIs in ways other models don't), STOP and tell the contributor to open a discussion/issue first - invasive or excessive changes get closed without full review.
+- Do not bundle unrelated work into this PR - see Step 4 and Step 5 below for the specifics on multimodal and chat-template/parsing work.
+- Never hack around RoPE with a custom sin/cos implementation. Several past PRs tried this and were closed. If the existing `ggml_rope_ext` (see Step 2's RoPE tips) genuinely cannot express what this model needs, the contributor should open an issue to discuss it with maintainers first - not send a PR with a custom RoPE implementation.
+
+Before starting, read `CONTRIBUTING.md`, `AGENTS.md` and `docs/development/HOWTO-add-model.md` if they are not already in context. Also run `git log --oneline -- src/models` and look at at least 3 recent PRs that added a model (their merge commits/diffs) - this shows current convention more reliably than the docs, which can lag behind.
+
+## Step 0 - Scope and dedup check
+
+Ask the contributor:
+1. Which model (HF repo id or name)? Is it text-only or does it have a multimodal (vision/audio) encoder?
+2. Do they already have the HF `config.json`/weights available locally?
+3. Have they checked for an existing PR/issue on this model? Suggest `gh search issues "<model name>"` and `gh search prs "<model name>"` in the `ggml-org/llama.cpp` repo. If an existing PR covers it, the contributor should comment there and collaborate rather than open a duplicate (per CONTRIBUTING.md's AI Usage Policy).
+4. What existing supported architecture is this model closest to (e.g. "Llama-like with sliding window", "MoE like DBRX", "BERT-style encoder")?
+
+If the contributor doesn't know the closest reference architecture, you may grep `conversion/*.py` and `src/models/*.cpp` for architectures with a similar config shape (layer count, head count, MoE expert count, norm placement) and suggest 1-2 candidates - but let the contributor confirm the choice rather than picking one yourself; this choice is a design decision they need to own.
+
+Do not proceed to Step 1 until the contributor has answered these and named a reference architecture.
+
+## Step 1 - Convert the model to GGUF
+
+Follow HOWTO-add-model.md section 1 for the actual touch points (conversion class registration, `constants.py`, `tensor_mapping.py`, etc.) - don't re-derive them here, read them from the doc.
+
+Skill-specific addition: for each touch point, show the contributor the equivalent code in the reference architecture they named in Step 0 before writing the new version, and check that they understand what's different about their model (e.g. non-standard tensor shapes, extra hparams) rather than just copying the pattern silently.
+
+## Step 2 - Define the architecture in llama.cpp
+
+Follow HOWTO-add-model.md section 2 for the actual touch points (`llm_arch` enum, `LLM_ARCH_NAMES`, hparam loading, RoPE type case, etc.), including its "Tips and tricks" section for `ggml_rope_ext` gotchas.
+
+Skill-specific addition: never hack around RoPE with a custom sin/cos implementation - see the RoPE rule above.
+
+## Step 3 - Build the GGML graph
+
+Follow HOWTO-add-model.md section 3 for the actual touch points (`src/models/<name>.cpp` struct, `llama_model_mapping` registration, etc.).
+
+Skill-specific addition: before writing `src/models/<name>.cpp`, read at least 10 other files under `src/models/` (pick a mix, not just the one reference architecture) to confirm the struct layout, naming, and style you're about to write actually matches current convention - the pattern drifts over time and the HOWTO doc can lag behind it.
+
+## Step 4 - Optional: multimodal encoder
+
+Only do this if the contributor flagged a vision/audio encoder in Step 0. Follow HOWTO-add-model.md section 4 and `docs/multimodal.md` for the actual touch points (`MmprojModel` subclass, `clip.cpp`, `mtmd.cpp`, encoder graph in `tools/mtmd/models`, etc.).
+
+Skill-specific addition, and read this carefully: **whether the multimodal encoder can be bundled into the same PR as the base text-model support depends on how conventional the change is.** It's OK to bundle it if the encoder support is conventional - i.e. no new infra or logic is needed, it's just a new cgraph reusing existing preprocessing/projector machinery (e.g. siglip/pixtral/qwen with just a new projector). If it requires anything beyond that - a new preprocessor, non-standard projector logic, or changes to shared `libmtmd` infra/logic - STOP, tell the contributor this is non-conventional, and have them land the text model first with the encoder as a dedicated follow-up PR. Do not let this decision pass silently - call it out explicitly to the contributor before writing any `clip.cpp`/`mtmd.cpp` code.
+
+## Step 5 - Optional: chat template / parsing support
+
+Only do this if the model needs a new built-in chat template (`src/llama-chat.cpp`) or a new output parser (see `docs/development/parsing.md` and `docs/autoparser.md`). If either is needed beyond what a user-supplied Jinja template already covers, treat it as its own dedicated follow-up PR, not part of the base model-support PR - call this out explicitly to the contributor rather than silently bundling it in.
+
+## Common pitfalls (from past PR reviews)
+
+These recur often enough in review comments on past add-model PRs that they're worth checking proactively, not just waiting for a reviewer to catch them:
+
+- Don't validate the same hparam/config assumption in both the Python conversion script and the C++ load path - pick one layer to own the check, duplicating it just adds maintenance surface.
+- Optional hparams that are genuinely absent from some configs (e.g. a shared-expert count) should be read with an explicit optional/fallback accessor, not assumed present.
+- Hparams that are actually load-bearing (the model produces wrong output or crashes without them, e.g. `sliding_window_pattern`, norm-eps) must hard-error if missing, not silently fall back to a default.
+- Don't bake a default chat template into the C++ binary - inject it into the GGUF at conversion time instead, since one `llm_arch` can be reused by multiple fine-tunes with different templates, and a baked-in C++ default fails silently for those.
+- Before writing a dedicated tool-call/output parser, check whether the existing autoparser already handles the template (`llama-debug-template-parser <jinja>` shows what it detects).
+- Marking a custom EOS/closing-tag token as `eot` at conversion time isn't always sufficient - in long/agentic generations a model can emit the closing sequence as literal text instead of the token, so generation never stops on EOG and raw text leaks past the parser. Verify this case, not just the token path.
+- If reusing or aliasing an existing pre-tokenizer for convenience, justify and test that choice explicitly - silent reuse is an easy source of subtle tokenizer bugs.
+- Watch for excessive graph splits caused by building per-layer view/index tensors inside the layer loop - hoist tensors that don't vary per layer out of the loop (relevant if you hit `GGML_SCHED_MAX_SPLIT_INPUTS`).
+- A custom KQ mask fed into flash attention must match FA's expected dtype - cast it to F16 before passing it to `build_attn_mha` when FA is enabled.
+- When padding a custom KV-cache size to an alignment (e.g. `GGML_PAD(..., 256)`), apply the padding after all other size adjustments, not before - otherwise later logic can un-align it again.
+- For non-standard cache/SWA (sliding-window-attention) semantics, override the dedicated hook (e.g. `llama_model_n_swa()`) rather than mutating hparams to fake the behavior - hparams may be read elsewhere for unrelated purposes.
+- Don't ship unfinished or unverified speculative-decoding (e.g. MTP) scaffolding in the base model PR - if it hasn't actually been confirmed to work, pull it out and land it as its own follow-up.
+- Conversion code should call into the base class's existing hparam logic (e.g. `super().set_gguf_parameters()`) rather than re-deriving it - large blocks of code that duplicate what `TextModel`/`MmprojModel` already provide will get flagged as redundant.
+- Do constant tensor modifications (e.g. `norm(1 + weight)`) and permutations/chunking at conversion time, not in the graph - see HOWTO-add-model.md's "Prefer conversion-time tensor modifications" tip (Gemma 3 folds its `1 +` into the weights, Qwen3-Next permutes in `modify_tensors`). Doing these at runtime in the graph is very likely to be rejected as over-complicated; if you genuinely can't do it at conversion time, open a discussion first explaining why rather than implementing it in the graph.
+  - Exception: a plain `weight * scale` with a constant scale is usually better applied at inference time instead of being folded into the weight at conversion. The scale conceptually applies to the activation, not the weight, so folding it in can hurt numerical stability, and it shifts the weight's value range in a way that can make quantization worse.
+
+## Validation checklist
+
+Reference: `examples/model-conversion/README.md`.
+
+1. Convert to GGUF, then inspect/run both the original and converted tensors.
+2. Run logits verification (original vs converted). If this model is a new version of an already-supported family, verify the *previous* version still passes logits verification first - numerical differences may be pre-existing, not caused by the new work. The tools to perform full logits validation are available in `examples/model-conversion`.
+3. Quantize (including QAT variants if relevant) and re-verify.
+4. Run perplexity evaluation (simple and full).
+5. Sanity-check across `tools/cli`, `tools/completion`, `tools/imatrix`, `tools/quantize`, and `tools/server`.
+6. CPU backend first; other backends (CUDA, Metal, ...) can be separate follow-up PRs per `CONTRIBUTING.md`.
+7. Re-review every changed file against the coding/naming guidelines in `AGENTS.md` (and `CONTRIBUTING.md`'s "Coding guidelines"/"Naming guidelines" sections) - this is a separate pass from functional testing and is just as important: no forced line-wrapping, no unicode punctuation, minimal/non-redundant comments, `snake_case` naming (`kebab-case` for file names), matching indentation/brace style, etc.
+
+## Before opening a PR
+
+- Run the `code-review` skill on the diff first - it catches the convention and scope issues reviewers flag most often, and it's recommended to do this locally before pushing the PR.
+- Confirm the contributor can explain every changed line to a reviewer and is prepared to be asked about any of it - this is required regardless of how much of the code was AI-generated.
+- Confirm they did a comprehensive manual review of the full diff, not just a skim.
+- Fill in the AI-disclosure section of `.github/pull_request_template.md` describing how AI was used (do not omit or understate this).
+- Do not write the PR description, commit message, GitHub issue/discussion text, or any reviewer replies yourself - the contributor writes these.