Compare commits

...
26 Commits
Author SHA1 Message Date
d7fa69b7de ci : run ccache-clear as the last step of release jobs (#27503)
* ci : run ccache-clear as the last step of release jobs

Assisted-by: pi:llama.cpp/Qwen3.8-27B

* update disabled job too to force rebase

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-08-21 18:56:04 +03:00
Georgi GerganovandGitHub bb4caa7540 llama.cpp : bump version to 0.2.0 (#27498) 2026-08-21 15:01:24 +03:00
Georgi GerganovandGitHub c4b0225d85 scripts : add release.sh for release preparation (#27497)
Similar to ggml/scripts/release.sh: validates repo state, creates a
release candidate branch (llama-rc-vX.Y.Z), bumps LLAMA_VERSION_* in
CMakeLists.txt and commits the version bump.

Usage: ./scripts/release.sh [major|minor|patch] [--dry-run]

Assisted-by: pi:llama.cpp/Qwen3.8-27B
2026-08-21 14:51:26 +03:00
Georgi Gerganov 5de25a7487 sync : ggml 2026-08-21 14:36:36 +03:00
Georgi Gerganov 01ff204fbd ggml : bump version to 0.21.0 (ggml/1597) 2026-08-21 14:36:36 +03:00
Georgi GerganovandGitHub 353b32d8b9 ci : remove duplicate flag (#27488) 2026-08-21 14:10:06 +03:00
Georgi GerganovandGitHub 7a0e42fd01 Revert "sycl : add Q2_K reordered MMVQ and ESIMD kernels (#26336)" (#27486)
This reverts commit ff14356e0c.
2026-08-21 14:02:03 +03:00
Aleksander GrygierandGitHub 5b6ddc9675 ui: Settings navigation cleanup (#27241)
* ui : rework the settings registry into ordered raw-data sections

SETTINGS_REGISTRY becomes an ordered SettingsSectionEntry[] array; the
array order is the sidebar display order. Section titles, color mode
options and title radio options are declared inline in their section or
entry. Entries gain showInUi; MCP servers, the system-message toggle and
the title LLM flag become hidden entries of their own section.

Derived values (config defaults, help info, chat sections, numeric field
lists, syncable parameters) are still derived here; they move to their
actual consumers in follow-up commits.

* ui : extract settings localStorage persistence into SettingsService

Stateless load/save of the settings config and user-override keys, plus the
legacy theme key migration. Business logic (default merging, mobile
sendOnEnter default, applying the migrated theme) stays in the store.

* ui : move the settings exit route into ROUTES

SETTINGS_FALLBACK_EXIT_ROUTE is just a route, so it lives with the other
routes as ROUTES.SETTINGS_EXIT.

* ui : derive the syncable parameter list in the parameter sync service

The syncable parameter mapping is only consumed by the sync service, so
derive it there from the registry instead of exporting it from the
constants file.

* ui : restore isPrivate for API key masking

* ui : clean up settings registry and router fetch guard

Drop the per-entry section field (duplicates the parent slug and is
never read) and guard the router model fetch on fields?.length so the
Tools/Import-Export pages with empty fields are excluded again.

Assisted-by: pi

* ui : merge sampling and penalties settings into one section

Assisted-by: pi
2026-08-21 12:30:03 +02:00
Georgi GerganovandGitHub e467c2ff61 ci : add nightly-tag.txt to make-release (#27485)
As agreed in ggml discussion #1579, the official semver releases now
include a nightly-tag.txt asset containing the tag of the corresponding
nightly release (e.g. b10485). The Web UI assets are published to the
HF bucket under the nightly tag, so this makes them discoverable for
each official release.

- make-release-desc.sh: expose the resolved nightly tag as a
  nightly_tag output
- make-release.yml: create nightly-tag.txt from that tag, upload it
  as a release asset (skipped on dry-run), mention it in the release
  body and in the dry-run summary

Assisted-by: pi:llama.cpp/Qwen3.8-27B
2026-08-21 13:20:44 +03:00
Georgi GerganovandGitHub 1719747451 ci : release clean-up (#27477) 2026-08-21 11:33:40 +03:00
Charles XuandGitHub 62b2269060 kleidiai : add SME2 F32 GEMV kernel support (#26891) 2026-08-21 11:33:30 +03:00
Todd MalsbaryandGitHub ff14356e0c sycl : add Q2_K reordered MMVQ and ESIMD kernels (#26336)
* Add DMMV Q4_K and Q6_K ESIMD kernels

Configure cmake build with -DGGML_SYCL_ESIMD=ON to enable.

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* Refactor ESIMD kernels to share common code

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* Move control of ESIMD from compile to runtime

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* Use ESIMD by default when available

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* Fix possible error when using ESIMD by default

While not an issue in the current version, this will become an
issue when additional QK ESIMD kernels are added (such as Q2_K).

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* Add explicit unroll to ESIMD kernels

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* Tidy up ESIMD kernels a bit

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* Add a reordered Q2_K MMVQ kernel

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* Add DMMV Q2_K ESIMD kernel

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

---------

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
2026-08-21 11:01:40 +03:00
Georgi GerganovandGitHub 5fff128451 test : make the FA V-is-view-of-K case a test case parameter (#27394)
Resolve the TODO in test_flash_attn_ext: the branch that creates V as a
sub-view of K (MLA-based models) was hardcoded for the 576/512 head shapes.
Add a v_is_view_of_k test case parameter (default false) and select the
sub-view branch on it; the existing 576/512 (DeepSeek MLA) cases now pass it
explicitly, so the test coverage is unchanged.

Also add more V-is-sub-view-of-K cases: the 320/256 (Mistral4 MLA) and
192/128 head shapes, and full views with equal head sizes (128/128 F16,
64/64 q8_0).

Assisted-by: pi:llama.cpp/Qwen3.8-27B
2026-08-21 10:29:17 +03:00
Todd MalsbaryandGitHub 9e89a196b8 sycl : Add Q5_K ESIMD kernel (#26376)
* Add DMMV Q4_K and Q6_K ESIMD kernels

Configure cmake build with -DGGML_SYCL_ESIMD=ON to enable.

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* Refactor ESIMD kernels to share common code

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* Move control of ESIMD from compile to runtime

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* Use ESIMD by default when available

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* Fix possible error when using ESIMD by default

While not an issue in the current version, this will become an
issue when additional QK ESIMD kernels are added (such as Q2_K).

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* Add explicit unroll to ESIMD kernels

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* Tidy up ESIMD kernels a bit

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* Add DMMV Q5_K ESIMD kernel

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* Remove redundant copyright notice

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

---------

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
2026-08-21 10:23:02 +03:00
cd26896c19 opencl: keep the vocab-scale K-quant lm_head on the CPU for Adreno A7X (compiler issue workaround) (#26440)
* opencl: keep the vocab-scale K-quant lm_head on the CPU on the Adreno A7X

* opencl: revise comments

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
2026-08-20 22:30:17 -07:00
1cb3f5eb41 sycl: Update gate logic for Alchemist GPUs regarding OneDNN features. (#26635)
* feat: updated gating logic of fattn-onednn.cpp

* verified device types

* Update ggml/src/ggml-sycl/fattn-onednn.cpp

Accepted recommendations to add bmg_g31 arch.

Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com>

* Improved SPDA gate, added documentation.

* Added arch var to reworked gate, fixing build errors.

* Fix trailing whitespaces.

---------

Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com>
2026-08-21 08:16:29 +03:00
Ian FaustandGitHub 6602dd3389 sycl: fix multiple warnings in compiling sycl backend (#26713)
* Update norm.cpp

* Update helper.hpp

* Update im2col.cpp

* Update fattn-mkl.cpp

* Update element_wise.cpp

* Update fattn-mkl.cpp

* Update set_rows.cpp

* Update element_wise.cpp

* Update ggml-sycl.cpp

* Update ggml-sycl.cpp

* Update ggml-sycl.cpp

* Update ggml-sycl.cpp

* Update ggml-sycl.cpp

* Update norm.cpp

* Update CMakeLists.txt

* Update CMakeLists.txt

* Update CMakeLists.txt

* Update ggml-sycl.cpp
2026-08-21 08:15:40 +03:00
Neo ZhangandGitHub 9e96cf77ff sycl : fix load model with mlock issue (#27250) 2026-08-21 08:14:54 +03:00
Chris DanisandGitHub b2e5e9b28b TP: enable tensor split for LFM2/LFM2MOE (#26993)
Assisted-by: deepseek-v4-flash
2026-08-21 08:13:58 +03:00
vkandGitHub a298422da7 docs: fix typos in ET.md (#27457) 2026-08-21 12:36:59 +08:00
Xuan-Son NguyenandGitHub 749f688fca ggml: support ggml_rope_set_offset on opencl, sycl, wgpu, hexagon (#27345)
* ggml: support ggml_rope_set_offset on opencl, sycl, wgpu, hexagon

* rm inplace optimization
2026-08-21 00:36:57 +02:00
EveandGitHub 0e1d9185c5 ci: use shell script to check cmake pkg (#27414)
* use regular script to build cmake pkg

* use old grep without perl
2026-08-20 20:01:32 +00:00
Georgi GerganovandGitHub a30273376e metal : clamp K extent in tensor API mat-mat kernel for K not a multiple of 32 (#27450)
The Tensor API mat-mat path of kernel_mul_mm (GGML_METAL_HAS_TENSOR) fed a
static K=32 tile to the matmul2d op on every iteration. On the last, partial
K tile (ne00 % 32 != 0) the src1 slice extends past the K extent of the
tensor, and the op reads those out-of-bounds elements (undefined behavior per
the MSL specification, section 2.22.2). Depending on stale memory contents,
this corrupted the result or produced NaN.

Make the matmul2d op use dynamic_extent for K, and clamp the K extent of both
operand tensor views to the remaining valid K range (min(32, K - loop_k)) per
iteration, so the op reads exactly the valid K range on every iteration
(mirroring the tail handling of the MPP matmul2d examples). On K-aligned
inputs the clamp degenerates to the full 32-wide tile: the only difference
from the static-K op is that the dynamic-K op derives K from the operand
extents and edge-checks the tile against the tensor extents (a handful of
integer ops per iteration).

Add test-backend-ops MUL_MAT cases with K not a multiple of 32 to exercise
the unaligned K path.

Assisted-by: pi:llama.cpp/Qwen3.8-27B
2026-08-20 21:31:29 +03:00
Hongqiang WangandGitHub 6503355df0 opencl: fix q6_K flat mul_mat for Adreno A6x/A7x GPUs with older E031 compilers (#26476)
* opencl: decline KV-convert flash_attn variants on Adreno A7X (compiler SIGSEGV)

The Adreno 740 (A7X) compiler E031.41 crashes inside clBuildProgram when
building the flash_attn programs whose KV path is mixed-type or dequantized:
flash_attn_f32_f16, flash_attn_f32_q8_0, flash_attn_f32_q4_0. It is a driver
crash rather than a compile-error return, so build_program_from_source_ex()
cannot catch it. The uniform f32 and f16 programs build correctly.

Decline the three KV-convert variants on the A7X in supports_op so they never
lazy-compile; those attention layers run on the CPU backend instead. Same
idiom as the existing Intel DK=512 and X1E carve-outs.

test-backend-ops FLASH_ATTN_EXT on the 740: 226 OK / 0 FAIL, previously exit
139. Other parts are unaffected - the gate is dead code there.

* opencl: fix q6_K flat mul_mat on older Adreno E031 compilers, gated

kernel_mul_mv_q6_K_f32_flat produces ~10x-wrong output on the older Adreno
E031 compilers while q4_K and q5_K are correct. Four codegen defects, each
confirmed on-device against the CPU reference:

  1. 64-bit ulong arithmetic is miscompiled, so every weight and scale read
     hit the wrong address - the primary cause, and why q5_K (int offsets)
     was unaffected. The block index is computed in int and widened only
     inside the pointer expression.
  2. The vectorized dequant (int4/float4 bit-ops, convert_*4, dot()) is
     miscompiled; the 6-bit weights are reconstructed and the dot done
     scalar.
  3. vload4 of the f32 activations is miscompiled; replaced by a
     scalar-indexed load.
  4. The accumulation is miscompiled unless a side effect forces the partial
     sums to materialize. A printf under a guard the compiler cannot prove
     false acts as a zero-cost optimizer barrier; its placement is
     load-bearing.

The defect tracks the compiler, not the GPU generation: it reproduces on
E031.38 (Adreno 642L) and E031.41 (Adreno 740) and is fixed by E031.45
(Adreno 619), so the workarounds are gated on the compiler version. Where
they are not needed they cost real throughput - 42.4 -> 35.1 GFLOPS on an
Adreno 840 q6_K GEMV. The explicit compiler-type check is required, not
redundant: newer_than_or_same() is false for every non-E031 compiler, so
negating it alone would enable the workarounds on E17 and DX.

test-backend-ops MUL_MAT is 919/919 on the Adreno 740, 642L, 619, 840 and
850; the 740 and 642L were 909/919 before. The 642L additionally needs the
A6X per-kernel-program support to reach these tests at all.
2026-08-20 10:58:35 -07:00
lhezandGitHub 6b4fa88a6c opencl: fix local size for norm (#27339) 2026-08-20 10:52:07 -07:00
Aleksander GrygierandGitHub 521a64cd01 ui: Stores split refactor (#27240)
* ui: Extract server stream lifecycle from chatStore into ChatStreamManager

Discovery, attach/replay, resume retry and the remote-running snapshot
formed a cohesive cluster inside chatStore. It now lives in
chat-streams.svelte.ts as ChatStreamManager, owned by chatStore, which
keeps the public entry points as delegates so components are
unchanged. chatStore: 2877 -> 2418 lines.

* ui: Extract user interaction gates from agenticStore into AgenticGates

Tool permission requests, turn-limit continue prompts and queued
steering messages are the state the loop waits on between turns. They
had no coupling to session state, so they now live in
agentic-gates.svelte.ts; agenticStore keeps delegates so components
are unchanged. agenticStore: 1196 -> 1073 lines.

* ui: Compose MCP resources under mcpStore.resources

Resource state was a second import scope next to mcpStore. Consumers
now go through mcpStore.resources, so the MCP surface is one store;
mcp-resources.svelte.ts stays a separate file owned by mcpStore.

* ui: Reorganize stores into domain namespaces

* fix: Update stale doc comments

* ui: Consolidate conv running-state into a chat activity ledger

Running-state was split across chatStore.chatLoadingStates (local
pipes), ChatStreamManager.remoteRunningConvs (backend sessions) and
attachingConvs (attach lifecycle), unioned by hand in
getAllLoadingChats and cross-cleaned by setChatLoading calling
streams.clearRemoteRunning - the 'spinner ghosts until tab toggle'
workaround.

chatActivityStore now owns both sets with one transition per event:
markLocal / localEnded (local pipe end also drops the stale remote
hint, no cross-owner call) / applyRemoteSnapshot (diffed). The
sidebar reads chatStore.activity.loadingConvs through the unchanged
getAllLoadingChats entry point.

Consequences:
- isStreamingActive and its five manual writers are gone; isStreaming()
  now reports whether the active conversation has a live streaming
  pipe, which is what all four consumers (assistant row, stop action,
  context gauge, chat screen) actually check
- isLoading/isReasoning become derived from the per-conv maps plus
  the active conversation, dropping the manual resync in
  syncLoadingStateForChat and clearUIState
- attachingConvs and the last-attach coordination disappear from
  ChatStreamManager
- getAllStreamingChats (no consumers) is removed

* ui: Give store collaborators narrow host interfaces

Collaborators took 'host: typeof <store>', i.e. the store's entire
public surface, which is how chatStore's streamChatCompletion,
createAssistantMessage, getApiOptions and setStreamingActive got
widened to public. Replace with per-collaborator interfaces carrying
only the members each one drives:

- ChatStreamHost (chat/streams) - activity, processing, streaming
  states, abort controller, loading/streaming setters
- ChatFlowsHost (chat/flows) - streaming core, message creation,
  per-conv state setters
- McpHealthHost (mcp/health) - connection registry + reconnection
- ModelPropsHost / ModelStatusHost (models) - model rows, feed
  updates; the managers write modalities/status back onto the host's
  rows, so those members stay writable
- ConversationsPreferencesHost (conversations) - the active row and
  the conversation list

The store classes now declare 'implements <Host>' so the contract is
visible at the class level, and the 'import type { <store> }' back
references in the collaborators disappear entirely - the host
contract is local to each collaborator file, and collaborators can
no longer reach around their slice. Members stay public (structural
typing), but the collaborator side is now compiler-enforced.

* test: Chat Activity store test

* refactor: Cleanup

* chore: Remove legacy architecture docs

* ui: Memoize findMessageIndex for the streaming hot path

Streaming looks up the same message index on every chunk, a linear
scan of activeMessages each time. Cache the last lookup and reuse it
after validating the id still sits at the same position (O(1)); any
structural change to the array fails validation and falls back to a
full scan.

* ui: Throttle per-chunk stream state writes to localStorage

saveStreamState ran JSON.stringify + a synchronous localStorage.setItem
on every decoded chunk of the stream. The read loop now goes through a
new saveStreamStateThrottled (one write per conversation per 500ms,
latest value held pending); the public saveStreamState keeps its
immediate-write contract for stream start and pre-fetch, and also
resets the throttle window.

A pending offset is force-flushed at resume boundaries (resumeStream
reads the offset back from localStorage), on visibilitychange->hidden
and on pagehide, so a reload always finds a usable offset. The resume
offset only needs to be roughly current since the server retransmits
from a line boundary and the client discards its partial line.

Adds unit tests for the throttled/flush/clear interplay.

* ui: Compute context gauge timing stats in one pass

currentRead/Fresh/Cache/Output were separate deriveds, each running a
full reverse scan of activeMessages for the last assistant timings,
and cumulative ran its own forward scan plus an agentic filter - 4-5
O(n) passes per chunk while streaming. Replace with a single
summarizeAssistantTimings() pass (last assistant timings, last
agentic llm totals and the cumulative sums) feeding a shared derived
snapshot. Semantics unchanged, including the live-stats overrides and
the agentic llm-totals branch.

* agentic : clear session state when a conversation is deleted

Every conversation that ran an agentic flow left an AgenticSession in the
store forever; clearSession was never called. conversationsStore now
notifies deletion listeners and agenticStore drops the matching sessions,
avoiding a circular import back into conversationsStore.

* chat : extract ChatService.normalizeMessagesForApi

The DB->API message normalization (convert + drop empty system messages)
was duplicated in sendMessage, preEncode and the agentic flow. Extract it
into one shared method and call it from all three.

* sse : share record splitting and data extraction

splitSseRecords and extractSseDataPayload centralize the record-boundary
splitting and data: line extraction used by parseSseJsonStream and the
models status feed. chat.service keeps its own line-based parser for
resume support.

* api : delegate apiFetchWithParams to apiFetch

apiFetchWithParams duplicated apiFetch's headers/fetch/error handling
body-for-body; it only differs in URL construction. Build the URL and
delegate.

* chat flows : dedupe title, timings and cleanup handling

- conversationsStore.applyTitleFromContent centralizes the title-from-first-
  message logic duplicated in 5 places
- ChatProcessingStore.applyStreamTimings centralizes the onTimings handler
  shared by the chat and continue flows
- host.cleanupStreaming centralizes the loading/streaming/processing reset
  repeated across the continue flow's exit paths

* conversations : centralize conversation update mirroring

rename, pin, mcp override, reasoning effort and cwd all repeated the same
write-DB-then-mirror-into-list-and-active dance. A single
applyConversationUpdate(id, updates) on the host collapses all five and
removes the forgot-to-mirror-one-field bug class. Drops the redundant
array reassignment in setCwd (deep  field assignment is reactive).

* mcp : dedupe tool execution, server parsing and tool indexing

- executeTool delegates to executeToolByName (only diff was argument parsing)
- drop the private #parseServerSettings copy; use parseMcpServerSettings
- cache getServers() keyed on the raw config value (hot path)
- indexServerTools() unifies the three identical toolsIndex rebuild loops

Assisted-by: Claude

* mcp : share cursor pagination and tool indexing

- MCPService.paginate() collapses the identical do-while loops in
  listAllResources and listAllResourceTemplates
- promoteHealthCheckToConnection now uses indexServerTools like the other
  connect paths

Assisted-by: Claude

* database : share message parent-child bookkeeping

- addChildToParent() dedups the append-to-children update in createMessageBranch
  and createSystemMessage
- removeChildFromParent() dedups the remove-from-children cleanup in deleteMessage
  and deleteMessageCascading
- bulkAdd the cloned messages when forking a conversation instead of one add
  per message

Assisted-by: Claude

* chore: Lint/format

* fix: `pagehide` event from `window`

* refactor: Api Fetch util

* docs : rewrite architecture sections in README

Update the high-level diagram, routes, hooks, stores, services and data
flow tables to match the current UI structure (mcp/settings/search
routes, agentic/tools/mcp stores, MCPService/ToolsService/SandboxService,
/tools API). Fix stale architectural patterns for per-conversation state
and modality validation.

* chore : add ESLint rule for blank lines between accessors

Enforce a blank line between consecutive class accessors. The core
padding-line-between-statements rule does not cover class members, so a
local rule is needed.

* refactor : reorder store members and unify naming

Order store class members as public fields, private fields, constructor,
getters, public methods, then private methods. Normalize private naming
to the `private` keyword (drop `#` and the `_` prefix where there is no
matching public getter). Rename conversationsStore.init() to
initialize() to match the other stores.

* refactor : prefix lookup methods with get in agentic and chat stores

Unify bare-name lookup methods with the get* prefix used across the
other stores (mcp, models, tools, settings). Renames currentTurn,
totalToolCalls, lastError, streamingToolCall, executingToolCallId,
pendingPermissionRequest, pendingContinueRequest,
pendingSteeringMessageContent, pendingSteeringMessageExtras in the
agentic store and pendingMessageContent, pendingMessageExtras in the
chat store. Updates the two consuming components and a doc comment.

* refactor: Clean up comments in stores' and services' code

* chore : add ESLint rule for class member ordering

Enforce structural order (public fields -> private fields -> constructor ->
getters -> setters -> public methods -> private methods) with alphabetical
sorting within each group via perfectionist/sort-classes. Dependency
detection keeps Svelte $derived fields in a valid dependency order instead
of alphabetizing them, since Svelte rejects forward references.

Assisted-by: Claude

* refactor : reorder class members to match new ESLint rule

Apply the sort-classes rule across stores, services, hooks and utils.
Pure reordering - verified no logic changes by comparing sorted line
multisets before/after. All tests and svelte-check pass.
2026-08-20 19:02:04 +02:00
163 changed files with 12692 additions and 13668 deletions
+1
View File
@@ -1,3 +1,4 @@
# note: place this as the last step of the job, so the new cache is saved by "Post ccache" right after the old one is cleared
name: "ccache-clear"
description: "Delete all GitHub Actions caches matching a key prefix"
inputs:
+16 -20
View File
@@ -27,30 +27,26 @@ jobs:
cmake --install build --prefix "$PREFIX" --config Release
export LLAMA_CONFIG="$PREFIX"/lib/cmake/llama/llama-config.cmake
tclsh <<'EOF'
set build(commit) [string trim [exec git rev-parse --short HEAD]]
set build(number) [string trim [exec git rev-list --count HEAD]]
build_commit=$(git rev-parse --short HEAD | xargs)
build_number=$(git rev-list --count HEAD | xargs)
set cmakelists [read [open "CMakeLists.txt" r]]
regexp {set\(LLAMA_VERSION_MAJOR\s+(\d+)\)} $cmakelists -> major
regexp {set\(LLAMA_VERSION_MINOR\s+(\d+)\)} $cmakelists -> minor
regexp {set\(LLAMA_VERSION_PATCH\s+(\d+)\)} $cmakelists -> patch
set build(version) "$major.$minor.$patch"
major=$(grep -oE "set\(LLAMA_VERSION_MAJOR[[:space:]]+[0-9]+" CMakeLists.txt | grep -oE "[0-9]+$")
minor=$(grep -oE "set\(LLAMA_VERSION_MINOR[[:space:]]+[0-9]+" CMakeLists.txt | grep -oE "[0-9]+$")
patch=$(grep -oE "set\(LLAMA_VERSION_PATCH[[:space:]]+[0-9]+" CMakeLists.txt | grep -oE "[0-9]+$")
build_version="$major.$minor.$patch"
set llamaconfig [read [open "$env(LLAMA_CONFIG)" r]]
set checks [list "set\\(LLAMA_VERSION \\s+$build(version)\\)" \
"set\\(LLAMA_BUILD_COMMIT\\s+$build(commit)\\)" \
"set\\(LLAMA_BUILD_NUMBER\\s+$build(number)\\)"]
checks=("set\(LLAMA_VERSION[[:space:]]+$build_version\)"
"set\(LLAMA_BUILD_COMMIT[[:space:]]+$build_commit\)"
"set\(LLAMA_BUILD_NUMBER[[:space:]]+$build_number\)")
puts -nonewline "Checking llama-config.cmake version... "
foreach check $checks {
if {![regexp -expanded -- $check $llamaconfig]} {
puts "\"$check\" failed!"
for check in "${checks[@]}"; do
if ! grep -qE "$check" "$LLAMA_CONFIG"; then
echo "Checking llama-config.cmake version... \"$check\" failed!"
exit 1
}
}
puts "success."
EOF
fi
done
echo "Checking llama-config.cmake version... success."
cd examples/simple-cmake-pkg
cmake -S . -B build -DCMAKE_PREFIX_PATH="$PREFIX"/lib/cmake
+1 -2
View File
@@ -97,8 +97,7 @@ jobs:
cmake -B build \
-DGGML_NATIVE=OFF \
-DLLAMA_FATAL_WARNINGS=ON \
-DGGML_RPC=ON \
-DGGML_NATIVE=OFF
-DGGML_RPC=ON
time cmake --build build --config Release -j $(nproc)
- name: Test
+43 -6
View File
@@ -55,6 +55,20 @@ jobs:
env:
GITHUB_REPOSITORY: ${{ github.repository }}
- name: Create nightly-tag.txt
id: nightly_tag_file
run: |
NIGHTLY_TAG="${{ steps.desc.outputs.nightly_tag }}"
if [[ -z "${NIGHTLY_TAG}" ]]; then
echo "Warning: no nightly tag found for the release commit - nightly-tag.txt will not be created"
echo "create=false" >> "$GITHUB_OUTPUT"
exit 0
fi
echo "${NIGHTLY_TAG}" > nightly-tag.txt
echo "create=true" >> "$GITHUB_OUTPUT"
echo "nightly-tag.txt:"
cat nightly-tag.txt
- name: Create release
if: ${{ github.event.inputs.dry_run == 'false' }}
uses: ggml-org/action-create-release@v1
@@ -62,26 +76,49 @@ jobs:
GITHUB_TOKEN: ${{ github.token }}
with:
tag_name: ${{ steps.checks.outputs.version }}
# TODO: remove the prerelease flag once the semantic versioning workflow is ready
# ref: https://github.com/ggml-org/ggml/discussions/1579
prerelease: true
prerelease: false
# TODO: enrich the body of the release with more information
body: |
> [!NOTE]
> Semantic versioning is still work in progress.
> More info can be found in https://github.com/ggml-org/ggml/discussions/1579
## Overview
New version has been released.
${{ steps.desc.outputs.nightly }}
**Web UI:** the `nightly-tag.txt` asset contains the tag of the corresponding nightly release
**More info:** [dist : releases and versioning of ggml-org projects](https://github.com/ggml-org/ggml/discussions/1579)
## ${{ steps.desc.outputs.changelog_title }}
${{ steps.desc.outputs.changelog }}
- name: Upload nightly-tag.txt
if: ${{ github.event.inputs.dry_run == 'false' && steps.nightly_tag_file.outputs.create == 'true' }}
uses: actions/github-script@v8
with:
github-token: ${{secrets.GITHUB_TOKEN}}
script: |
const fs = require('fs');
const release_id = '${{ steps.create_release.outputs.id }}';
console.log('uploadReleaseAsset', 'nightly-tag.txt');
await github.rest.repos.uploadReleaseAsset({
owner: context.repo.owner,
repo: context.repo.repo,
release_id: release_id,
name: 'nightly-tag.txt',
data: await fs.readFileSync('./nightly-tag.txt')
});
- name: Dry run summary
if: ${{ github.event.inputs.dry_run == 'true' }}
run: |
if [[ "${{ steps.checks.outputs.checks_passed }}" == "true" ]]; then
echo "Dry run complete - all checks passed."
echo "Would have created tag: ${{ steps.checks.outputs.version }}"
if [[ -n "${{ steps.desc.outputs.nightly_tag }}" ]]; then
echo "Would have uploaded nightly-tag.txt: ${{ steps.desc.outputs.nightly_tag }}"
fi
else
echo "::error::Dry run found release check failures. A release tag would not be created."
exit 1
+57 -56
View File
@@ -145,11 +145,6 @@ jobs:
${{ env.CMAKE_ARGS }}
cmake --build build --config Release -j $(sysctl -n hw.logicalcpu)
- name: ccache-clear
uses: ./.github/actions/ccache-clear
with:
key: release-${{ matrix.os }}-${{ matrix.arch }}
- name: Determine tag name
id: tag
uses: ./.github/actions/get-tag-name
@@ -166,6 +161,11 @@ jobs:
path: llama-${{ steps.tag.outputs.name }}-bin-macos-${{ matrix.build }}.tar.gz
name: llama-bin-macos-${{ matrix.build }}.tar.gz
- name: ccache-clear
uses: ./.github/actions/ccache-clear
with:
key: release-${{ matrix.os }}-${{ matrix.arch }}
ubuntu-cpu:
needs: [check-release, get-version]
if: ${{ needs.check-release.outputs.should_release == 'true' }}
@@ -231,12 +231,6 @@ jobs:
${{ env.CMAKE_ARGS }}
cmake --build build --config Release -j $(nproc)
- name: ccache-clear
if: ${{ matrix.build != 's390x' }}
uses: ./.github/actions/ccache-clear
with:
key: release-${{ matrix.os }}-cpu
- name: Determine tag name
id: tag
uses: ./.github/actions/get-tag-name
@@ -253,6 +247,12 @@ jobs:
path: llama-${{ steps.tag.outputs.name }}-bin-ubuntu-${{ matrix.build }}.tar.gz
name: llama-bin-ubuntu-${{ matrix.build }}.tar.gz
- name: ccache-clear
if: ${{ matrix.build != 's390x' }}
uses: ./.github/actions/ccache-clear
with:
key: release-${{ matrix.os }}-cpu
ubuntu-vulkan:
needs: [check-release, get-version]
if: ${{ needs.check-release.outputs.should_release == 'true' }}
@@ -318,11 +318,6 @@ jobs:
${{ env.CMAKE_ARGS }}
cmake --build build --config Release -j $(nproc)
- name: ccache-clear
uses: ./.github/actions/ccache-clear
with:
key: release-${{ matrix.os }}-vulkan
- name: Determine tag name
id: tag
uses: ./.github/actions/get-tag-name
@@ -339,6 +334,11 @@ jobs:
path: llama-${{ steps.tag.outputs.name }}-bin-ubuntu-vulkan-${{ matrix.build }}.tar.gz
name: llama-bin-ubuntu-vulkan-${{ matrix.build }}.tar.gz
- name: ccache-clear
uses: ./.github/actions/ccache-clear
with:
key: release-${{ matrix.os }}-vulkan
android-arm64:
needs: [check-release, get-version]
if: ${{ needs.check-release.outputs.should_release == 'true' }}
@@ -512,11 +512,6 @@ jobs:
${{ env.CMAKE_ARGS }}
cmake --build build/ReleaseOV --config Release --parallel
- name: ccache-clear
uses: ./.github/actions/ccache-clear
with:
key: release-ubuntu-24.04-openvino-release-no-preset-v1
- name: Determine tag name
id: tag
uses: ./.github/actions/get-tag-name
@@ -551,6 +546,11 @@ jobs:
path: llama-${{ steps.tag.outputs.name }}-bin-ubuntu-openvino-${{ env.OPENVINO_VERSION_MAJOR }}-x64.tar.gz
name: llama-bin-ubuntu-openvino-${{ env.OPENVINO_VERSION_MAJOR }}-x64.tar.gz
- name: ccache-clear
uses: ./.github/actions/ccache-clear
with:
key: release-ubuntu-24.04-openvino-release-no-preset-v1
windows-openvino:
needs: [check-release]
if: ${{ needs.check-release.outputs.should_release == 'true' }}
@@ -637,11 +637,6 @@ jobs:
cmake --build build\ReleaseOV --config Release -- /m
- name: ccache-clear
uses: ./.github/actions/ccache-clear
with:
key: release-windows-2022-openvino
- name: Determine tag name
id: tag
uses: ./.github/actions/get-tag-name
@@ -680,6 +675,11 @@ jobs:
path: llama-${{ steps.tag.outputs.name }}-bin-win-openvino-${{ env.OPENVINO_VERSION_MAJOR }}-x64.zip
name: llama-bin-win-openvino-${{ env.OPENVINO_VERSION_MAJOR }}-x64.zip
- name: ccache-clear
uses: ./.github/actions/ccache-clear
with:
key: release-windows-2022-openvino
windows-cpu:
name: windows-cpu / ${{ matrix.arch }}
needs: [check-release]
@@ -733,11 +733,6 @@ jobs:
${{ env.CMAKE_ARGS }}
cmake --build build --config Release
- name: ccache-clear
uses: ./.github/actions/ccache-clear
with:
key: release-windows-2025-vs2026-${{ matrix.arch }}-cpu
- name: Pack artifacts
id: pack_artifacts
run: |
@@ -749,6 +744,11 @@ jobs:
path: llama-bin-win-cpu-${{ matrix.arch }}.zip
name: llama-bin-win-cpu-${{ matrix.arch }}.zip
- name: ccache-clear
uses: ./.github/actions/ccache-clear
with:
key: release-windows-2025-vs2026-${{ matrix.arch }}-cpu
windows-rocm:
needs: [check-release]
if: ${{ needs.check-release.outputs.should_release == 'true' }}
@@ -841,11 +841,6 @@ jobs:
-DAMDGPU_TARGETS="${{ matrix.gpu_targets }}"
cmake --build . --config Release --parallel ${env:NUMBER_OF_PROCESSORS}
- name: ccache-clear
uses: ./.github/actions/ccache-clear
with:
key: windows-rocm-${{ matrix.ROCM_VERSION }}-${{ matrix.build }}
- name: Verify HIP backend was built
run: |
$hipDll = Get-ChildItem -Path build\bin -Filter "ggml-hip*.dll" -ErrorAction SilentlyContinue
@@ -878,6 +873,11 @@ jobs:
path: llama-bin-win-rocm-${{ env.ROCM_VERSION_SHORT }}-${{ matrix.build }}.zip
name: llama-bin-win-rocm-${{ env.ROCM_VERSION_SHORT }}-${{ matrix.build }}.zip
- name: ccache-clear
uses: ./.github/actions/ccache-clear
with:
key: windows-rocm-${{ matrix.ROCM_VERSION }}-${{ matrix.build }}
windows:
needs: [check-release]
if: ${{ needs.check-release.outputs.should_release == 'true' }}
@@ -1043,11 +1043,6 @@ jobs:
set /A NINJA_JOBS=%NUMBER_OF_PROCESSORS%-1
cmake --build build --config Release -j %NINJA_JOBS% --target ggml-cuda
- name: ccache-clear
uses: ./.github/actions/ccache-clear
with:
key: release-windows-2022-${{ matrix.arch }}-cuda-${{ matrix.cuda }}
- name: Pack artifacts
id: pack_artifacts
run: |
@@ -1083,6 +1078,11 @@ jobs:
path: cudart-llama-bin-win-cuda-${{ matrix.cuda }}-${{ matrix.arch }}.zip
name: cudart-llama-bin-win-cuda-${{ matrix.cuda }}-${{ matrix.arch }}.zip
- name: ccache-clear
uses: ./.github/actions/ccache-clear
with:
key: release-windows-2022-${{ matrix.arch }}-cuda-${{ matrix.cuda }}
windows-sycl:
needs: [check-release]
if: ${{ needs.check-release.outputs.should_release == 'true' }}
@@ -1142,11 +1142,6 @@ jobs:
-DLLAMA_BUILD_BORINGSSL=ON
cmake --build build --target ggml-sycl -j %NUMBER_OF_PROCESSORS%
- name: ccache-clear
uses: ./.github/actions/ccache-clear
with:
key: release-windows-2022-x64-sycl
- name: Build the release package
id: pack_artifacts
run: |
@@ -1193,6 +1188,11 @@ jobs:
path: llama-bin-win-sycl-x64.zip
name: llama-bin-win-sycl-x64.zip
- name: ccache-clear
uses: ./.github/actions/ccache-clear
with:
key: release-windows-2022-x64-sycl
ubuntu-24-sycl:
needs: [check-release]
if: ${{ needs.check-release.outputs.should_release == 'true' }}
@@ -1265,11 +1265,6 @@ jobs:
-DGGML_SYCL_F16=${{ matrix.fp16 }}
time cmake --build build --config Release -j $(nproc)
- name: ccache-clear
uses: ./.github/actions/ccache-clear
with:
key: release-ubuntu-24.04-sycl-${{ matrix.build }}
- name: Determine tag name
id: tag
uses: ./.github/actions/get-tag-name
@@ -1286,6 +1281,11 @@ jobs:
path: llama-${{ steps.tag.outputs.name }}-bin-ubuntu-sycl-${{ matrix.build }}-x64.tar.gz
name: llama-bin-ubuntu-sycl-${{ matrix.build }}-x64.tar.gz
- name: ccache-clear
uses: ./.github/actions/ccache-clear
with:
key: release-ubuntu-24.04-sycl-${{ matrix.build }}
# ubuntu-22-rocm:
# needs: [check-release, get-version]
# if: ${{ needs.check-release.outputs.should_release == 'true' }}
@@ -1380,11 +1380,6 @@ jobs:
# ${{ env.CMAKE_ARGS }}
# cmake --build build --config Release -j $(nproc)
# # - name: ccache-clear
# # uses: ./.github/actions/ccache-clear
# # with:
# # key: release-ubuntu-22.04-rocm-${{ matrix.ROCM_VERSION }}
# - name: Determine tag name
# id: tag
# uses: ./.github/actions/get-tag-name
@@ -1404,6 +1399,11 @@ jobs:
# path: llama-${{ steps.tag.outputs.name }}-bin-ubuntu-rocm-${{ env.ROCM_VERSION_SHORT }}-${{ matrix.build }}.tar.gz
# name: llama-bin-ubuntu-rocm-${{ env.ROCM_VERSION_SHORT }}-${{ matrix.build }}.tar.gz
# # - name: ccache-clear
# # uses: ./.github/actions/ccache-clear
# # with:
# # key: release-ubuntu-22.04-rocm-${{ matrix.ROCM_VERSION }}
ios-xcode:
needs: [check-release, get-version]
if: ${{ needs.check-release.outputs.should_release == 'true' }}
@@ -1688,6 +1688,7 @@ jobs:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
tag_name: ${{ steps.tag.outputs.name }}
prerelease: true
body: |
<details open>
+2 -2
View File
@@ -4,8 +4,8 @@ include(CheckIncludeFileCXX)
### llama.cpp version
set(LLAMA_VERSION_MAJOR 0)
set(LLAMA_VERSION_MINOR 1)
set(LLAMA_VERSION_PATCH 2)
set(LLAMA_VERSION_MINOR 2)
set(LLAMA_VERSION_PATCH 0)
set(LLAMA_VERSION_BASE "${LLAMA_VERSION_MAJOR}.${LLAMA_VERSION_MINOR}.${LLAMA_VERSION_PATCH}")
# whether this is a development/nightly build
+4 -4
View File
@@ -116,7 +116,7 @@ in inline assembler.
Most kernels are very naive with lots of low hanging fruits left:
> [!IMPORTANT]
> Several assembly instructions emmited by the compiler are not implemented
> Several assembly instructions emitted by the compiler are not implemented
> in hardware and software emulation in firmware is not ready yet.
> Eventually firmware will transparently trap unimplemented instructions
> and will emulate them inside exception handler. Until then, kernel
@@ -138,12 +138,12 @@ Most kernels are very naive with lots of low hanging fruits left:
> kernel build process. Feel free to take ideas/code from there or try linking
> it in.
Before commiting any changes to operations and/or kernels, don't forget
Before committing any changes to operations and/or kernels, don't forget
to update supported ops reports (instructions at `docs/ops.md`).
When logging is enabled (e.g. by setting `--log-file` cli param),
each compute kernel run outputs a line with
pipe-delimited key-value pairs containing kernel level performance infomation.
pipe-delimited key-value pairs containing kernel level performance information.
Line is prefixed with `ET_PERF`:
```
@@ -160,7 +160,7 @@ to `GGML_ET_PROFILE/et_runtime_trace.json` and `GGML_ET_PROFILE/kernel_map` on e
### Uberkernel
The in-knernel implementaiton of device dispatch/kernel fusion. The ET SDK has a non-trivial op-to-op gap. `Uberkernel` (name taken from the original Esperanto AI's compiler)
The in-kernel implementation of device dispatch/kernel fusion. The ET SDK has a non-trivial op-to-op gap. `Uberkernel` (name taken from the original Esperanto AI's compiler)
dispatches multiple already existing kernel implementations with device side synchronization. Due to the processor's design, there is no natural memory visibility
horizon between sub-kernel invocations. This makes uberkernel much more difficult to develop and debug. Currently Uberkerel is hidden begind the
`GGML_ET_UBERKERNEL` environment variable and is disabled by default. Setting it to 1 enables it and provides significant performance improvements but is only
+2 -2
View File
@@ -4,8 +4,8 @@ project("ggml" C CXX ASM)
### GGML Version
set(GGML_VERSION_MAJOR 0)
set(GGML_VERSION_MINOR 20)
set(GGML_VERSION_PATCH 2)
set(GGML_VERSION_MINOR 21)
set(GGML_VERSION_PATCH 0)
set(GGML_VERSION_BASE "${GGML_VERSION_MAJOR}.${GGML_VERSION_MINOR}.${GGML_VERSION_PATCH}")
list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake/")
+8 -4
View File
@@ -639,6 +639,7 @@ function(ggml_add_cpu_backend_variant_impl tag_name)
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_fp32_bf16p_bf16p/
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_f16p_qsi4c32p/
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_f32p_f32p/
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_f32_f32p/
${KLEIDIAI_SRC}/kai/ukernels/matmul/pack/)
set(ARCH_FLAGS_TEMP "${ARCH_FLAGS}")
@@ -701,6 +702,8 @@ function(ggml_add_cpu_backend_variant_impl tag_name)
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_f16p_qsi4c32p/kai_matmul_clamp_f32_f16p1vlx2_qsi4c32p4vlx2_1vlx4vl_sme2_mopa_asm.S
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_f32p_f32p/kai_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa.c
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_f32p_f32p/kai_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa_asm.S
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_f32_f32p/kai_matmul_clamp_f32_f32_f32p2vlx1b_1x16vl_sme2_mla.c
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_f32_f32p/kai_matmul_clamp_f32_f32_f32p2vlx1b_1x16vl_sme2_mla_asm.S
${KLEIDIAI_SRC}/kai/ukernels/matmul/pack/kai_lhs_pack_bf16p2vlx2_f32_sme.c
${KLEIDIAI_SRC}/kai/ukernels/matmul/pack/kai_rhs_pack_kxn_bf16p2vlx2b_f32_x32_sme.c
${KLEIDIAI_SRC}/kai/ukernels/matmul/pack/kai_lhs_pack_f16pmrx2_f32_neon.c
@@ -737,8 +740,9 @@ function(ggml_add_cpu_backend_variant_impl tag_name)
set_target_properties(${GGML_CPU_NAME} PROPERTIES COMPILE_FLAGS "-msimd128")
endif()
if (CMAKE_CXX_COMPILER_ID STREQUAL "IntelLLVM")
# The compiler automatically enables "-ffast-math" which can cause NaNs in tests due to "-fassociative-math"
target_compile_options(${GGML_CPU_NAME} PRIVATE "-fno-associative-math")
endif()
if (CMAKE_C_COMPILER_ID STREQUAL "IntelLLVM" OR CMAKE_CXX_COMPILER_ID STREQUAL "IntelLLVM")
# The compiler automatically enables "-ffast-math" which can cause NaNs in tests due to "-fassociative-math"
target_compile_options(${GGML_CPU_NAME} PRIVATE "$<$<OR:$<COMPILE_LANG_AND_ID:C,IntelLLVM>,$<COMPILE_LANG_AND_ID:CXX,IntelLLVM>>:$<$<BOOL:${WIN32}>:/clang:>-fno-associative-math>")
endif()
endfunction()
+32 -16
View File
@@ -23,6 +23,7 @@
#include "kai_matmul_clamp_f32_qsi8d32p1x8_qsi4c32p8x8_1x8_sve_dotprod.h"
#include "kai_matmul_clamp_f32_f16p1vlx2_qsi4c32p4vlx2_1vlx4vl_sme2_mopa.h"
#include "kai_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa.h"
#include "kai_matmul_clamp_f32_f32_f32p2vlx1b_1x16vl_sme2_mla.h"
#include "kai_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa.h"
#include "kai_lhs_pack_bf16p2vlx2_f32_sme.h"
@@ -76,6 +77,21 @@ static inline void kernel_run_fn10(size_t m, size_t n, size_t k, size_t /*bl*/,
Fn(m, n, k, lhs, rhs, dst, dst_stride_row, dst_stride_col, clamp_min, clamp_max);
}
template <void (*Fn)(size_t, size_t, size_t, const void *, size_t, const void *, void *, size_t, size_t, float, float)>
static inline void kernel_run_lhs_stride_fn10(size_t m,
size_t n,
size_t k,
size_t lhs_stride,
const void * lhs,
const void * rhs,
void * dst,
size_t dst_stride_row,
size_t dst_stride_col,
float clamp_min,
float clamp_max) {
Fn(m, n, k, lhs, lhs_stride, rhs, dst, dst_stride_row, dst_stride_col, clamp_min, clamp_max);
}
template<void(*Fn)(size_t,size_t,size_t,const void*,const void*,float*,size_t,size_t,float,float)>
static inline void kernel_run_float_fn10(size_t m, size_t n, size_t k, size_t /*bl*/,
const void* lhs, const void* rhs, void* dst,
@@ -947,25 +963,25 @@ static ggml_kleidiai_kernels ggml_kleidiai_kernels_f32[] = {
/* .packed_size_ex = */ &lhs_ps_fn5<kai_get_lhs_packed_size_lhs_pack_f32p2vlx1_f32_sme>,
/* .pack_func_ex = */ &lhs_pack_void_fn9<kai_run_lhs_pack_f32p2vlx1_f32_sme>,
},
/* SME GEMV */
/* SME2 GEMV */
{
/* .get_m_step = */ kai_get_m_step_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
/* .get_n_step = */ kai_get_n_step_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
/* .get_mr = */ kai_get_mr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
/* .get_nr = */ kai_get_nr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
/* .get_kr = */ kai_get_kr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
/* .get_sr = */ kai_get_sr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
/* .get_dst_offset = */ kai_get_dst_offset_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
/* .get_dst_size = */ kai_get_dst_size_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
/* .get_lhs_offset_ex = */ nullptr,
/* .get_rhs_packed_offset_ex = */ nullptr,
/* .run_kernel_ex = */ nullptr,
/* .get_m_step = */ kai_get_m_step_matmul_clamp_f32_f32_f32p2vlx1b_1x16vl_sme2_mla,
/* .get_n_step = */ kai_get_n_step_matmul_clamp_f32_f32_f32p2vlx1b_1x16vl_sme2_mla,
/* .get_mr = */ kai_get_m_step_matmul_clamp_f32_f32_f32p2vlx1b_1x16vl_sme2_mla,
/* .get_nr = */ kai_get_nr_matmul_clamp_f32_f32_f32p2vlx1b_1x16vl_sme2_mla,
/* .get_kr = */ kai_get_kr_matmul_clamp_f32_f32_f32p2vlx1b_1x16vl_sme2_mla,
/* .get_sr = */ kai_get_sr_matmul_clamp_f32_f32_f32p2vlx1b_1x16vl_sme2_mla,
/* .get_dst_offset = */ kai_get_dst_offset_matmul_clamp_f32_f32_f32p2vlx1b_1x16vl_sme2_mla,
/* .get_dst_size = */ kai_get_dst_size_matmul_clamp_f32_f32_f32p2vlx1b_1x16vl_sme2_mla,
/* .get_lhs_offset_ex = */ &kernel_offs_fn2<kai_get_lhs_offset_matmul_clamp_f32_f32_f32p2vlx1b_1x16vl_sme2_mla>,
/* .get_rhs_packed_offset_ex = */ &kernel_offs_fn2<kai_get_rhs_packed_offset_matmul_clamp_f32_f32_f32p2vlx1b_1x16vl_sme2_mla>,
/* .run_kernel_ex = */ &kernel_run_lhs_stride_fn10<kai_run_matmul_clamp_f32_f32_f32p2vlx1b_1x16vl_sme2_mla>,
},
/* .gemv_lhs_info = */ {
/* .get_offset = */ kai_get_lhs_offset_lhs_pack_f32p2vlx1_f32_sme,
/* .get_packed_offset_ex = */ &lhs_offs_fn5<kai_get_lhs_packed_offset_lhs_pack_f32p2vlx1_f32_sme>,
/* .packed_size_ex = */ &lhs_ps_fn5<kai_get_lhs_packed_size_lhs_pack_f32p2vlx1_f32_sme>,
/* .pack_func_ex = */ &lhs_pack_void_fn9<kai_run_lhs_pack_f32p2vlx1_f32_sme>,
/* .get_offset = */ nullptr,
/* .get_packed_offset_ex = */ nullptr,
/* .packed_size_ex = */ nullptr,
/* .pack_func_ex = */ nullptr,
},
/* .rhs_info = */ {
/* .packed_stride = */ nullptr,
+35 -13
View File
@@ -696,6 +696,15 @@ class tensor_traits : public ggml::cpu::tensor_traits {
}
if (op->src[0]->type == GGML_TYPE_F32) {
ggml_kleidiai_kernels * primary = kernel_chain[0];
kernel_info * gemv_kernel = primary ? &primary->gemv : nullptr;
if (is_gemv && op->src[1]->nb[0] == (int64_t) sizeof(float) && gemv_kernel &&
gemv_kernel->get_lhs_offset_ex && gemv_kernel->get_rhs_packed_offset_ex &&
gemv_kernel->run_kernel_ex && gemv_kernel->get_dst_offset) {
size = 0;
return true;
}
size_t cursor = 0;
bool any_slot = false;
@@ -811,15 +820,28 @@ class tensor_traits : public ggml::cpu::tensor_traits {
return false;
}
kernel_info * kernel = &kernels->gemm;
const size_t k = ne00;
const size_t m = ne11;
const size_t n = ne01;
const bool use_gemv = m == 1 && src1->nb[0] == (int64_t) sizeof(float) &&
kernels->gemv.get_lhs_offset_ex &&
kernels->gemv.get_rhs_packed_offset_ex &&
kernels->gemv.run_kernel_ex &&
kernels->gemv.get_dst_offset;
kernel_info * kernel = use_gemv ? &kernels->gemv : &kernels->gemm;
lhs_packing_info * lhs_info = &kernels->gemm_lhs_info;
if (!kernel || !lhs_info || !lhs_info->get_offset || !lhs_info->get_packed_offset_ex ||
!lhs_info->packed_size_ex || !lhs_info->pack_func_ex ||
if (!kernel || !kernel->get_lhs_offset_ex ||
!kernel->get_rhs_packed_offset_ex || !kernel->run_kernel_ex || !kernel->get_dst_offset) {
return false;
}
if (!use_gemv && (!lhs_info || !lhs_info->get_offset || !lhs_info->get_packed_offset_ex ||
!lhs_info->packed_size_ex || !lhs_info->pack_func_ex)) {
return false;
}
const kleidiai_weight_header * header = kleidiai_weight_header_from_ptr(src0->data);
const bool has_header = kleidiai_is_weight_header_valid(header);
@@ -832,16 +854,14 @@ class tensor_traits : public ggml::cpu::tensor_traits {
const int nth = params->nth > 0 ? params->nth : 1;
const int ith = params->ith;
const size_t k = ne00;
const size_t m = ne11;
const size_t n = ne01;
const size_t mr = kernel->get_mr();
const size_t kr = kernel->get_kr();
const size_t sr = kernel->get_sr();
const size_t lhs_packed_size = lhs_info->packed_size_ex(m, k, 0, mr, kr, sr);
GGML_ASSERT(lhs_packed_size <= params->wsize);
const size_t lhs_packed_size = use_gemv ? 0 : lhs_info->packed_size_ex(m, k, 0, mr, kr, sr);
if (!use_gemv) {
GGML_ASSERT(lhs_packed_size <= params->wsize);
}
uint8_t * lhs_packed = static_cast<uint8_t *>(params->wdata);
const size_t dst_stride = dst->nb[1];
@@ -853,7 +873,7 @@ class tensor_traits : public ggml::cpu::tensor_traits {
const uint8_t * lhs_batch_base = static_cast<const uint8_t *>(src1->data) + batch_idx * src1->nb[2];
uint8_t * dst_batch_base = static_cast<uint8_t *>(dst->data) + batch_idx * dst->nb[2];
{
if (!use_gemv) {
const int64_t m_roundup_mr = kai_roundup((int64_t)m, (int64_t)mr);
int64_t max_threads = mr ? (m_roundup_mr / (int64_t)mr) : nth;
max_threads = std::max<int64_t>(1, max_threads);
@@ -903,15 +923,17 @@ class tensor_traits : public ggml::cpu::tensor_traits {
const size_t n_to_process = std::min(chunk_cols, n - n_start);
if (n_to_process > 0) {
const size_t lhs_packed_offset = lhs_info->get_packed_offset_ex(0, k, 0, mr, kr, sr);
const size_t lhs_offset = use_gemv ? kernel->get_lhs_offset_ex(0, k, 0)
: lhs_info->get_packed_offset_ex(0, k, 0, mr, kr, sr);
const size_t rhs_packed_offset = kernel->get_rhs_packed_offset_ex(n_start, k, 0);
const size_t dst_offset = kernel->get_dst_offset(0, n_start, dst_stride);
const void * lhs_ptr = lhs_packed + lhs_packed_offset;
const void * lhs_ptr = use_gemv ? lhs_batch_base + lhs_offset
: lhs_packed + lhs_offset;
const void * rhs_ptr = rhs_base + rhs_packed_offset;
float * dst_ptr = reinterpret_cast<float *>(dst_batch_base + dst_offset);
kernel->run_kernel_ex(m, n_to_process, k, 0,
kernel->run_kernel_ex(m, n_to_process, k, use_gemv ? src1->nb[1] : 0,
lhs_ptr,
rhs_ptr,
dst_ptr,
+3 -2
View File
@@ -3180,8 +3180,9 @@ static bool ggml_hexagon_supported_argsort(const struct ggml_hexagon_session * s
static bool ggml_hexagon_supported_rope(const struct ggml_hexagon_session * sess, const struct ggml_tensor * op) {
const int32_t * op_params = &op->op_params[0];
if (op_params[15] != 0) {
return false; // FIXME: support ggml_rope_set_offset
// ggml_rope_set_offset: HVX kernels need a VLEN-aligned window start (32 f32 elems)
if (op_params[15] % 32 != 0) {
return false;
}
int mode = op_params[2];
+16 -6
View File
@@ -53,6 +53,7 @@
struct htp_rope_context {
int32_t n_dims;
int32_t n_offs;
int32_t mode;
int32_t n_ctx_orig;
int32_t sections[4];
@@ -405,32 +406,40 @@ static inline void hvx_rope_f32_aa(float * restrict dst, const float * restrict
static void inline rope_basic_f32(struct htp_rope_context * rctx, uint8_t * restrict dst, uint8_t * restrict src,
uint32_t nr, uint32_t ne0, const float * restrict theta_cache) {
const uint32_t n_offs = rctx->n_offs; // VLEN-aligned (enforced by supports_op)
#pragma unroll(4)
for (uint32_t i = 0; i < nr; i++) {
float * d = (float *) (dst + i * rctx->dst_row_size_aligned);
float * s = (float *) (src + i * rctx->src0_row_size_aligned);
hvx_rope_f32_aa(d, s, rctx->n_dims, theta_cache);
hvx_rope_f32_aa(d + n_offs, s + n_offs, rctx->n_dims, theta_cache);
// fill the remain channels with data from src tensor
if (rctx->n_dims < ne0) {
hvx_copy_f32_uu((uint8_t *)(d + rctx->n_dims), (uint8_t *)(s + rctx->n_dims), ne0 - rctx->n_dims);
if (n_offs > 0) {
hvx_copy_f32_uu((uint8_t *) d, (uint8_t *) s, n_offs);
}
if (n_offs + rctx->n_dims < ne0) {
hvx_copy_f32_uu((uint8_t *)(d + n_offs + rctx->n_dims), (uint8_t *)(s + n_offs + rctx->n_dims), ne0 - n_offs - rctx->n_dims);
}
}
}
static void inline rope_neox_f32(struct htp_rope_context * rctx, uint8_t * restrict dst, uint8_t * restrict src,
uint32_t nr, uint32_t ne0, const float * restrict theta_cache) {
const uint32_t n_offs = rctx->n_offs; // VLEN-aligned (enforced by supports_op)
#pragma unroll(4)
for (uint32_t i = 0; i < nr; i++) {
float * d = (float *) (dst + i * rctx->dst_row_size_aligned);
float * s = (float *) (src + i * rctx->src0_row_size_aligned);
hvx_rope_neox_f32_aa(d, s, rctx->n_dims, theta_cache);
hvx_rope_neox_f32_aa(d + n_offs, s + n_offs, rctx->n_dims, theta_cache);
// fill the remain channels with data from src tensor
if (rctx->n_dims < ne0) {
hvx_copy_f32_uu((uint8_t *)(d + rctx->n_dims), (uint8_t *)(s + rctx->n_dims), ne0 - rctx->n_dims);
if (n_offs > 0) {
hvx_copy_f32_uu((uint8_t *) d, (uint8_t *) s, n_offs);
}
if (n_offs + rctx->n_dims < ne0) {
hvx_copy_f32_uu((uint8_t *)(d + n_offs + rctx->n_dims), (uint8_t *)(s + n_offs + rctx->n_dims), ne0 - n_offs - rctx->n_dims);
}
}
}
@@ -673,6 +682,7 @@ static int execute_op_rope_f32(struct htp_ops_context * octx) {
rctx.n_dims = ((const int32_t *) op_params)[1];
rctx.mode = ((const int32_t *) op_params)[2];
rctx.n_ctx_orig = ((const int32_t *) op_params)[4];
rctx.n_offs = ((const int32_t *) op_params)[15];
memcpy(&rctx.freq_base, (int32_t *) op_params + 5, sizeof(float));
memcpy(&rctx.freq_scale, (int32_t *) op_params + 6, sizeof(float));
+11 -4
View File
@@ -10365,9 +10365,12 @@ kernel void kernel_mul_mm(
auto tB = tensor(ptrB, dextents<int32_t, 2>(K, N), array<int, 2>({1, strideB}));
// Configure matmul operation
// note: K is dynamic_extent (clamped to the valid range in PHASE 2), since a static
// N_MM_NK_TOTAL K tile would read src1 out of bounds when K % N_MM_NK_TOTAL != 0
// ref: https://github.com/ggml-org/llama.cpp/pull/27064
mpp::tensor_ops::matmul2d<
mpp::tensor_ops::matmul2d_descriptor(
NRB, NRA, N_MM_NK_TOTAL, false, true, true,
NRB, NRA, static_cast<int>(dynamic_extent), false, true, true,
mpp::tensor_ops::matmul2d_descriptor::mode::multiply_accumulate),
execution_simdgroups<N_MM_SIMD_GROUP_X * N_MM_SIMD_GROUP_Y>> mm;
@@ -10419,10 +10422,14 @@ kernel void kernel_mul_mm(
threadgroup_barrier(mem_flags::mem_threadgroup);
// === PHASE 2: Tensor matmul ===
auto mA = tA.slice(0, 0);
auto mB = tB.slice(loop_k, rb);
// Clamp the K extent of both operand tensors to the remaining valid K range so
// the dynamic-K op never reads past the K extent of src1 (or the staged A tile).
const int kExt = min(N_MM_NK_TOTAL, K - loop_k);
mm.run(mB, mA, cT);
auto tAv = tensor(sa, dextents<int32_t, 2>(kExt, NRA), array<int, 2>({1, N_MM_NK_TOTAL}));
auto tBv = tensor(ptrB + loop_k + rb * strideB, dextents<int32_t, 2>(kExt, N - rb), array<int, 2>({1, strideB}));
mm.run(tBv, tAv, cT);
threadgroup_barrier(mem_flags::mem_threadgroup);
}
+66 -5
View File
@@ -582,6 +582,8 @@ struct ggml_backend_opencl_context {
bool adreno_use_bin_kernels;
get_adreno_bin_kernel_func_t get_adreno_bin_kernel_func = nullptr;
ggml_cl_compiler_version adreno_cl_compiler_version;
// The q6_K flat mul_mat codegen workarounds are needed by old E031 compilers only.
bool q6_k_flat_old_compiler;
std::string kernel_compile_opts; // cached for lazy-compiled kernels.
@@ -1931,8 +1933,14 @@ static void load_cl_kernels(ggml_backend_opencl_context *backend_ctx) {
#else
const std::string kernel_src = read_file("mul_mv_q6_k_f32_flat.cl");
#endif
// The codegen workarounds in this kernel are a measured 13-20% loss on
// compilers that do not need them, so only the affected ones build them;
// everyone else gets the original source.
const std::string q6k_opts = backend_ctx->q6_k_flat_old_compiler
? compile_opts + " -DADRENO_OLD_COMPILER=1"
: compile_opts;
cl_program prog =
build_program_from_source(backend_ctx, kernel_src.c_str(), compile_opts);
build_program_from_source(backend_ctx, kernel_src.c_str(), q6k_opts);
CL_CHECK((backend_ctx->kernel_mul_mv_q6_K_f32_flat = clCreateKernel(prog, "kernel_mul_mv_q6_K_f32_flat", &err), err));
CL_CHECK(clReleaseProgram(prog));
@@ -5917,6 +5925,16 @@ static ggml_backend_opencl_context * ggml_cl_init(ggml_backend_dev_t dev) {
(backend_ctx->adreno_cl_compiler_version.type == E031 && backend_ctx->adreno_cl_compiler_version.major >= 47) ||
(backend_ctx->adreno_cl_compiler_version.type == DX && backend_ctx->adreno_cl_compiler_version.major >= 17);
// The q6_K flat mul_mat miscompile is a defect of the older E031 compilers, not a
// property of any GPU generation: it reproduces on E031.38 (Adreno 642L) and E031.41
// (Adreno 740) and is fixed by E031.45 (Adreno 619). Gate on the compiler so parts
// that do not need the workarounds do not pay for them. The explicit type check is
// required: newer_than_or_same() is false for every non-E031 compiler, so negating it
// alone would enable the workarounds on E17/DX.
backend_ctx->q6_k_flat_old_compiler =
backend_ctx->adreno_cl_compiler_version.type == E031 &&
!backend_ctx->adreno_cl_compiler_version.newer_than_or_same(E031, 45, 0, 0);
size_t ext_str_size;
clGetDeviceInfo(device, CL_DEVICE_EXTENSIONS, 0, NULL, &ext_str_size);
char *ext_buffer = (char *)alloca(ext_str_size + 1);
@@ -7375,6 +7393,19 @@ static bool ggml_opencl_supports_op(ggml_backend_dev_t dev, const struct ggml_te
op->src[0]->type == GGML_TYPE_Q4_K ||
op->src[0]->type == GGML_TYPE_Q5_K ||
op->src[0]->type == GGML_TYPE_Q6_K) {
// The E031.41 compiler (usually with A7x) miscompiles the flat K-quant
// GEMV kernels (kernel_mul_mv_q*_K_f32_flat) and makes lm_head run much
// slower than it should. So, make it fallback to CPU to preserve performance
// for this compiler series.
static const char * a7x_lmhead_env = getenv("GGML_OPENCL_A7X_LMHEAD_CPU");
static const bool a7x_lmhead_cpu = (a7x_lmhead_env == nullptr || a7x_lmhead_env[0] != '0');
if (a7x_lmhead_cpu &&
backend_ctx->adreno_gen == ADRENO_GPU_GEN::A7X &&
(op->src[0]->type == GGML_TYPE_Q4_K || op->src[0]->type == GGML_TYPE_Q5_K ||
op->src[0]->type == GGML_TYPE_Q6_K) &&
op->src[0]->ne[1] >= 32768) { // vocab-scale weight; no FFN/attn weight is this tall
return false;
}
return op->src[1]->type == GGML_TYPE_F32 && ggml_is_contiguous(op->src[0]) && ggml_is_contiguous(op->src[1]);
} else if (op->src[0]->type == GGML_TYPE_Q8_0) {
return op->src[1]->type == GGML_TYPE_F32;
@@ -7416,9 +7447,6 @@ static bool ggml_opencl_supports_op(ggml_backend_dev_t dev, const struct ggml_te
case GGML_OP_DIAG_MASK_INF:
return op->ne[3] == 1;
case GGML_OP_ROPE: {
if (((const int32_t *) op->op_params)[15] != 0) {
return false; // FIXME: support ggml_rope_set_offset
}
const int mode = ((const int32_t *) op->op_params)[2];
const bool is_mrope = mode & GGML_ROPE_TYPE_MROPE;
const bool is_vision = mode == GGML_ROPE_TYPE_VISION;
@@ -7496,6 +7524,7 @@ static bool ggml_opencl_supports_op(ggml_backend_dev_t dev, const struct ggml_te
v->type == GGML_TYPE_F16 && op->type == GGML_TYPE_F16;
const bool is_f32_f16 = q->type == GGML_TYPE_F32 && k->type == GGML_TYPE_F16 &&
v->type == GGML_TYPE_F16 && op->type == GGML_TYPE_F32;
const bool is_f32_q8_0 = q->type == GGML_TYPE_F32 && k->type == GGML_TYPE_Q8_0 &&
v->type == GGML_TYPE_Q8_0 && op->type == GGML_TYPE_F32 &&
dk % 32 == 0 && dv % 32 == 0;
@@ -7503,6 +7532,21 @@ static bool ggml_opencl_supports_op(ggml_backend_dev_t dev, const struct ggml_te
v->type == GGML_TYPE_Q4_0 && op->type == GGML_TYPE_F32 &&
dk % 32 == 0 && dv % 32 == 0;
// A7X (Adreno 740, compiler E031.41) SIGSEGVs inside clBuildProgram
// building the flash_attn programs whose KV path is mixed-type or
// dequantized — f32_f16, q8_0, q4_0 (reproduced at DK=40 and DK=64; it
// is DK-independent). It is a driver crash, not codegen-wrong-output, so
// it cannot be caught in-process (fatal=false only handles clean compile
// errors). The uniform f16_f16 / f32_f32 programs compile fine on this
// compiler, so decline only the KV-convert variants; ggml then runs
// those (f16-KV / quant-KV) attention layers on the CPU backend.
// Negative compiler carve-out, same idiom as the Intel DK=512 decline
// below and the X1E driver-quirk guards.
if (backend_ctx && backend_ctx->adreno_gen == ADRENO_GPU_GEN::A7X &&
(is_f32_f16 || is_f32_q8_0 || is_f32_q4_0)) {
return false;
}
// Asymmetric KV: host-dequants both sides to F32, uses f32 kernel.
auto is_kv_type_ok = [](ggml_type t) {
return t == GGML_TYPE_F16 || t == GGML_TYPE_F32 ||
@@ -12830,7 +12874,10 @@ static void ggml_cl_norm(ggml_backend_t backend, const ggml_tensor * src0, const
GGML_TENSOR_LOCALS(int, ne0, src0, ne);
GGML_TENSOR_LOCALS(cl_ulong, nb0, src0, nb);
const int nth = MIN(64, ne00);
int nth = 1;
while (nth < ne00 && nth < 64) {
nth *= 2;
}
cl_kernel kernel = backend_ctx->kernel_norm;
@@ -20580,6 +20627,12 @@ static void ggml_cl_mul_mat(ggml_backend_t backend, const ggml_tensor * src0, co
CL_CHECK(clSetKernelArg(kernel, 14, sizeof(int), &ne1));
CL_CHECK(clSetKernelArg(kernel, 15, sizeof(int), &r2));
CL_CHECK(clSetKernelArg(kernel, 16, sizeof(int), &r3));
// The optimizer-barrier arg exists only in the ADRENO_OLD_COMPILER build of
// this kernel; conformant compilers get the original 17-arg signature.
if (backend_ctx->q6_k_flat_old_compiler) {
cl_uchar q6k_mask = 0xFF; // never 0xFE in prod; see the kernel note
CL_CHECK(clSetKernelArg(kernel, 17, sizeof(cl_uchar), &q6k_mask));
}
#else
kernel = backend_ctx->kernel_mul_mv_q6_K_f32;
@@ -23867,6 +23920,7 @@ static void ggml_cl_rope(ggml_backend_t backend, const ggml_tensor * src0, const
const int n_dims = ((int *) dst->op_params)[1];
const int mode = ((int *) dst->op_params)[2];
const int n_ctx_orig = ((int32_t *) dst->op_params)[4];
const int n_offs = ((int32_t *) dst->op_params)[15];
float freq_base;
float freq_scale;
@@ -23895,6 +23949,7 @@ static void ggml_cl_rope(ggml_backend_t backend, const ggml_tensor * src0, const
if (is_vision) {
GGML_ASSERT(n_dims == ne00/2);
GGML_ASSERT(n_offs == 0); // offset not supported for vision, as the rotated pairs span the whole row
}
cl_kernel kernel;
@@ -23986,6 +24041,12 @@ static void ggml_cl_rope(ggml_backend_t backend, const ggml_tensor * src0, const
if (is_mrope && !is_vision) {
CL_CHECK(clSetKernelArg(kernel, 34, sizeof(int), &is_imrope));
}
// norm and neox have n_offs after beta_slow, mrope has it after is_imrope
if (!is_mrope && !is_vision) {
CL_CHECK(clSetKernelArg(kernel, 33, sizeof(int), &n_offs));
} else if (is_mrope && !is_vision) {
CL_CHECK(clSetKernelArg(kernel, 35, sizeof(int), &n_offs));
}
size_t global_work_size[] = {(size_t)ne01*nth, (size_t)ne02, (size_t)ne03};
size_t local_work_size[] = {(size_t)nth, 1, 1};
@@ -28,6 +28,13 @@
#define QK_K 256
// ADRENO_OLD_COMPILER is defined by the host (-D) only for the Adreno E031
// compilers older than E031.45, which miscompile several constructs this kernel
// used (confirmed on E031.38 and E031.41; E031.45 is clean). Every other
// compiler -- newer E031, E17, DX, Intel, and every non-Adreno device that
// builds this program -- takes the #else branches, which are the original
// source: the workarounds below cost ~13% on the q6_K flat n=1 GEMV where they
// are not needed.
inline float block_q_6_K_dot_y_flat(
global uchar * blk_ql,
global uchar * blk_qh,
@@ -37,6 +44,9 @@ inline float block_q_6_K_dot_y_flat(
int ip,
int is,
int l0,
#if defined(ADRENO_OLD_COMPILER)
int dbg,
#endif
float4 y0,
float4 y1,
float4 y2,
@@ -48,10 +58,40 @@ inline float block_q_6_K_dot_y_flat(
global uchar * q1 = blk_ql + ib*128 + q_offset_l;
global uchar * q2 = q1 + QK_K/8;
global uchar * qh = blk_qh + ib*64 + q_offset_h;
global char * sc = blk_scales + ib*16 + is;
float dall = blk_d[ib];
#if defined(ADRENO_OLD_COMPILER)
// The vectorized dequant (int4/float4 bit-ops, convert_*4, dot()) and vload4
// are miscompiled here -> garbage weights. Reconstruct the 6-bit weights and
// take the dot product scalar. q4_K/q5_K flat already use scalar paths, which
// is why q6_K was the only flat GEMV that failed.
// Scales are SIGNED int8; read as uchar and sign-extend arithmetically so the
// result does not depend on whether the compiler treats `char` as signed.
global uchar * sc = (global uchar *)(blk_scales + ib*16 + is);
int s0 = (int)sc[0] - 256*(sc[0] >> 7);
int s2 = (int)sc[2] - 256*(sc[2] >> 7);
int s4 = (int)sc[4] - 256*(sc[4] >> 7);
int s6 = (int)sc[6] - 256*(sc[6] >> 7);
// one 6-bit weight: low/high nibble of a ql byte OR'd with a 2-bit qh plane
// (plane p in {0,1,2,3} selects qh bits 2p..2p+1) placed at bits 4-5, minus 32.
#define Q6W(qb, sh, hb, p) ((float)((((int)(qb) >> (sh)) & 15) | ((((int)(hb) >> (2*(p))) & 3) << 4)) - 32.f)
float d0 = y0.s0*Q6W(q1[0],0,qh[0],0) + y0.s1*Q6W(q1[1],0,qh[1],0) + y0.s2*Q6W(q1[2],0,qh[2],0) + y0.s3*Q6W(q1[3],0,qh[3],0);
float d1 = y1.s0*Q6W(q2[0],0,qh[0],1) + y1.s1*Q6W(q2[1],0,qh[1],1) + y1.s2*Q6W(q2[2],0,qh[2],1) + y1.s3*Q6W(q2[3],0,qh[3],1);
float d2 = y2.s0*Q6W(q1[0],4,qh[0],2) + y2.s1*Q6W(q1[1],4,qh[1],2) + y2.s2*Q6W(q1[2],4,qh[2],2) + y2.s3*Q6W(q1[3],4,qh[3],2);
float d3 = y3.s0*Q6W(q2[0],4,qh[0],3) + y3.s1*Q6W(q2[1],4,qh[1],3) + y3.s2*Q6W(q2[2],4,qh[2],3) + y3.s3*Q6W(q2[3],4,qh[3],3);
#undef Q6W
if (dbg) printf("HELPER dall=%f s=[%d %d %d %d] d=[%f %f %f %f] ql0=%d qh0=%d y00=%f\n",
dall, s0, s2, s4, s6, d0, d1, d2, d3, (int)q1[0], (int)qh[0], y0.s0);
return dall * (d0 * s0 + d1 * s2 + d2 * s4 + d3 * s6);
#else
global char * sc = blk_scales + ib*16 + is;
// Vectorized loads: 3 uchar4 weight loads instead of 12 scalar byte reads.
// q_offset_l/h are 4-aligned, so these are aligned vector loads.
uchar4 q1v = vload4(0, q1);
@@ -72,6 +112,7 @@ inline float block_q_6_K_dot_y_flat(
return dall * (dot(y0, w0) * sc[0] + dot(y1, w1) * sc[2] +
dot(y2, w2) * sc[4] + dot(y3, w3) * sc[6]);
#endif
}
#undef N_DST
@@ -113,6 +154,11 @@ kernel void kernel_mul_mv_q6_K_f32_flat(
int ne1,
int r2,
int r3
#if defined(ADRENO_OLD_COMPILER)
,
uchar q6k_mask // runtime 0xFF; the host passes it so the compiler cannot
// constant-fold the printf guards below into nothing
#endif
) {
src1 = (global float*)((global char*)src1 + offset1);
dst = (global float*)((global char*)dst + offsetd);
@@ -128,6 +174,22 @@ kernel void kernel_mul_mv_q6_K_f32_flat(
int first_row = (N_SIMDGROUP * r0 + get_sub_group_id()) * N_DST;
#if defined(ADRENO_OLD_COMPILER)
// 64-bit `ulong` integer arithmetic is miscompiled here -> the base-pointer byte
// offsets came out wrong, so EVERY weight/scale read hit the wrong address. This
// was the primary cause of the q6_K flat failure (q5_K uses int offsets and is
// unaffected). Compute the block index in `int` and widen to `ulong` only inside
// the pointer expression: the byte offset stays 64-bit, but there is no ulong
// arithmetic chain to miscompile. The int index would overflow past ~2^31 blocks,
// which no realistic weight reaches -- but that is a narrowing, so keep it off the
// conformant path, which retains full ulong arithmetic.
int offset_src0 = first_row*nb + (i12/r2)*(nb*ne01) + (i13/r3)*(nb*ne01*ne02);
global uchar * blk_ql = (global uchar *) src0_ql + (ulong)offset_src0 * 128;
global uchar * blk_qh = (global uchar *) src0_qh + (ulong)offset_src0 * 64;
global char * blk_scales = (global char *) src0_s + (ulong)offset_src0 * 16;
global half * blk_d = (global half *) src0_d + offset_src0;
#else
ulong offset_src0 = first_row*nb + (i12/r2)*(nb*ne01) + (i13/r3)*(nb*ne01*ne02);
ulong offset_src0_ql = offset_src0 * 128;
ulong offset_src0_qh = offset_src0 * 64;
@@ -138,6 +200,7 @@ kernel void kernel_mul_mv_q6_K_f32_flat(
global uchar * blk_qh = (global uchar *) src0_qh + offset_src0_qh;
global char * blk_scales = (global char *) src0_s + offset_src0_s;
global half * blk_d = (global half *) src0_d + offset_src0_d;
#endif
global float * yy = (global float *) src1 + r1*ne10 + im*ne00*ne1;
int tid = get_sub_group_local_id()%(N_SIMDWIDTH/BLOCK_STRIDE); // within-super-block part, 0..15
@@ -155,24 +218,55 @@ kernel void kernel_mul_mv_q6_K_f32_flat(
for (int ib = ix; ib < nb; ib += BLOCK_STRIDE) {
global float * y = yy + ib * QK_K + 128*ip + l0;
#if defined(ADRENO_OLD_COMPILER)
// vload4 of f32 is miscompiled here; index the lanes scalar instead.
float4 y0 = (float4)(y[ 0], y[ 1], y[ 2], y[ 3]);
float4 y1 = (float4)(y[32], y[33], y[34], y[35]);
float4 y2 = (float4)(y[64], y[65], y[66], y[67]);
float4 y3 = (float4)(y[96], y[97], y[98], y[99]);
#else
float4 y0 = vload4(0, y + 0);
float4 y1 = vload4(0, y + 32);
float4 y2 = vload4(0, y + 64);
float4 y3 = vload4(0, y + 96);
#endif
for (int row = 0; row < N_DST; row++) {
if (first_row + row < ne01) {
#if defined(ADRENO_OLD_COMPILER)
int dbg = (q6k_mask==0xFE && r0==0 && r1==0 && im==0 && row==0 && ib==0 &&
ne00==256 && ne01==16 && get_sub_group_local_id()==0) ? 1 : 0;
sumf[row] += block_q_6_K_dot_y_flat(
blk_ql + row*nb*128, blk_qh + row*nb*64, blk_scales + row*nb*16, blk_d + row*nb,
ib, ip, is, l0, dbg, y0, y1, y2, y3);
#else
sumf[row] += block_q_6_K_dot_y_flat(
blk_ql + row*nb*128, blk_qh + row*nb*64, blk_scales + row*nb*16, blk_d + row*nb,
ib, ip, is, l0, y0, y1, y2, y3);
#endif
}
}
}
#if defined(ADRENO_OLD_COMPILER)
// Optimizer barrier. This compiler drops the sumf partials unless a side effect
// forces them to materialize. q6k_mask is a kernel arg the compiler cannot prove
// is never 0xFE (the host always passes 0xFF), so the printf survives compilation
// but never executes. FRAGILE: the exact set and placement of these guarded
// printfs is load-bearing on E031.41 -- removing any one re-breaks q6_K.
if (q6k_mask==0xFE && r0==0 && r1==0 && im==0 && ne00==256 && ne01==16 && get_sub_group_local_id()<16) {
printf("Q6KLANE lane=%d ip=%d il=%d is=%d l0=%d sumf0=%f\n",
get_sub_group_local_id(), ip, il, is, l0, sumf[0]);
}
#endif
for (int row = 0; row < N_DST; row++) {
float tot = sub_group_reduce_add(sumf[row]);
if (get_sub_group_local_id() == 0 && first_row + row < ne01) {
dst[r1*ne0 + im*ne0*ne1 + first_row + row] = tot;
#if defined(ADRENO_OLD_COMPILER)
if (q6k_mask==0xFE && r0==0 && r1==0 && im==0 && row==0 && ne00==256 && ne01==16)
printf("Q6KTOT tot=%f\n", tot);
#endif
}
}
}
+52 -40
View File
@@ -75,7 +75,8 @@ kernel void kernel_rope_norm_f32(
float ext_factor,
float attn_factor,
float beta_fast,
float beta_slow
float beta_slow,
int n_offs
) {
src0 = (global void*)((global char*)src0 + offset0);
src1 = (global int*)((global char*)src1 + offset1);
@@ -94,14 +95,15 @@ kernel void kernel_rope_norm_f32(
float inv_ndims = -1.f/n_dims;
for (int i0 = 2*get_local_id(0); i0 < ne0; i0 += 2*get_local_size(0)) {
if (i0 < n_dims) {
int ic = i0/2;
if (i0 >= n_offs && i0 < n_offs + n_dims) {
int iw = i0 - n_offs; // relative idx
int ic = iw/2;
float theta = theta_base * pow(freq_base, inv_ndims*i0);
float theta = theta_base * pow(freq_base, inv_ndims*iw);
float freq_factor = src2 != src0 ? src2[ic] : 1.0f;
float2 cos_sin_theta = rope_yarn(theta/freq_factor, freq_scale, corr_dims, i0, ext_factor, attn_factor);
float2 cos_sin_theta = rope_yarn(theta/freq_factor, freq_scale, corr_dims, iw, ext_factor, attn_factor);
global float * src = (global float *)((global char *) src0 + i3*nb03 + i2*nb02 + i1*nb01 + i0*nb00);
global float * dst_data = (global float *)((global char *) dst + i3*nb3 + i2*nb2 + i1*nb1 + i0*nb0);
@@ -154,7 +156,8 @@ kernel void kernel_rope_norm_f16(
float ext_factor,
float attn_factor,
float beta_fast,
float beta_slow
float beta_slow,
int n_offs
) {
src0 = (global void*)((global char*)src0 + offset0);
src1 = (global int*)((global char*)src1 + offset1);
@@ -173,14 +176,15 @@ kernel void kernel_rope_norm_f16(
float inv_ndims = -1.f/n_dims;
for (int i0 = 2*get_local_id(0); i0 < ne0; i0 += 2*get_local_size(0)) {
if (i0 < n_dims) {
int ic = i0/2;
if (i0 >= n_offs && i0 < n_offs + n_dims) {
int iw = i0 - n_offs; // relative idx
int ic = iw/2;
float theta = theta_base * pow(freq_base, inv_ndims*i0);
float theta = theta_base * pow(freq_base, inv_ndims*iw);
float freq_factor = src2 != src0 ? src2[ic] : 1.0f;
float2 cos_sin_theta = rope_yarn(theta/freq_factor, freq_scale, corr_dims, i0, ext_factor, attn_factor);
float2 cos_sin_theta = rope_yarn(theta/freq_factor, freq_scale, corr_dims, iw, ext_factor, attn_factor);
global half * src = (global half *)((global char *) src0 + i3*nb03 + i2*nb02 + i1*nb01 + i0*nb00);
global half * dst_data = (global half *)((global char *) dst + i3*nb3 + i2*nb2 + i1*nb1 + i0*nb0);
@@ -233,7 +237,8 @@ kernel void kernel_rope_neox_f32(
float ext_factor,
float attn_factor,
float beta_fast,
float beta_slow
float beta_slow,
int n_offs
) {
src0 = (global void*)((global char*)src0 + offset0);
src1 = (global int*)((global char*)src1 + offset1);
@@ -252,17 +257,18 @@ kernel void kernel_rope_neox_f32(
float inv_ndims = -1.f/n_dims;
for (int i0 = 2*get_local_id(0); i0 < ne0; i0 += 2*get_local_size(0)) {
if (i0 < n_dims) {
int ic = i0/2;
if (i0 >= n_offs && i0 < n_offs + n_dims) {
int iw = i0 - n_offs; // relative idx
int ic = iw/2;
const float theta = theta_base * pow(freq_base, inv_ndims*i0);
const float theta = theta_base * pow(freq_base, inv_ndims*iw);
const float freq_factor = src2 != src0 ? src2[ic] : 1.0f;
float2 cos_sin_theta = rope_yarn(theta/freq_factor, freq_scale, corr_dims, i0, ext_factor, attn_factor);
float2 cos_sin_theta = rope_yarn(theta/freq_factor, freq_scale, corr_dims, iw, ext_factor, attn_factor);
global float * src = (global float *)((global char *) src0 + i3*nb03 + i2*nb02 + i1*nb01 + ic*nb00);
global float * dst_data = (global float *)((global char *) dst + i3*nb3 + i2*nb2 + i1*nb1 + ic*nb0);
global float * src = (global float *)((global char *) src0 + i3*nb03 + i2*nb02 + i1*nb01 + (n_offs + ic)*nb00);
global float * dst_data = (global float *)((global char *) dst + i3*nb3 + i2*nb2 + i1*nb1 + (n_offs + ic)*nb0);
const float x0 = src[0];
const float x1 = src[n_dims/2];
@@ -312,7 +318,8 @@ kernel void kernel_rope_neox_f16(
float ext_factor,
float attn_factor,
float beta_fast,
float beta_slow
float beta_slow,
int n_offs
) {
src0 = (global void*)((global char*)src0 + offset0);
src1 = (global int*)((global char*)src1 + offset1);
@@ -331,17 +338,18 @@ kernel void kernel_rope_neox_f16(
float inv_ndims = -1.f/n_dims;
for (int i0 = 2*get_local_id(0); i0 < ne0; i0 += 2*get_local_size(0)) {
if (i0 < n_dims) {
int ic = i0/2;
if (i0 >= n_offs && i0 < n_offs + n_dims) {
int iw = i0 - n_offs; // relative idx
int ic = iw/2;
const float theta = theta_base * pow(freq_base, inv_ndims*i0);
const float theta = theta_base * pow(freq_base, inv_ndims*iw);
const float freq_factor = src2 != src0 ? src2[ic] : 1.0f;
float2 cos_sin_theta = rope_yarn(theta/freq_factor, freq_scale, corr_dims, i0, ext_factor, attn_factor);
float2 cos_sin_theta = rope_yarn(theta/freq_factor, freq_scale, corr_dims, iw, ext_factor, attn_factor);
global half * src = (global half *)((global char *) src0 + i3*nb03 + i2*nb02 + i1*nb01 + ic*nb00);
global half * dst_data = (global half *)((global char *) dst + i3*nb3 + i2*nb2 + i1*nb1 + ic*nb0);
global half * src = (global half *)((global char *) src0 + i3*nb03 + i2*nb02 + i1*nb01 + (n_offs + ic)*nb00);
global half * dst_data = (global half *)((global char *) dst + i3*nb3 + i2*nb2 + i1*nb1 + (n_offs + ic)*nb0);
const float x0 = src[0];
const float x1 = src[n_dims/2];
@@ -393,7 +401,8 @@ kernel void kernel_rope_multi_f32(
float beta_fast,
float beta_slow,
int4 sections,
int is_imrope
int is_imrope,
int n_offs
) {
src0 = (global void*)((global char*)src0 + offset0);
src1 = (global int*)((global char*)src1 + offset1);
@@ -414,10 +423,11 @@ kernel void kernel_rope_multi_f32(
float inv_ndims = -1.f/n_dims;
for (int i0 = 2*get_local_id(0); i0 < ne0; i0 += 2*get_local_size(0)) {
if (i0 < n_dims) {
int ic = i0/2;
if (i0 >= n_offs && i0 < n_offs + n_dims) {
int iw = i0 - n_offs; // relative idx
int ic = iw/2;
const int sector = (i0 / 2) % sect_dims;
const int sector = ic % sect_dims;
float theta_base = 0.0f;
if (is_imrope) {
@@ -445,14 +455,14 @@ kernel void kernel_rope_multi_f32(
}
}
const float theta = theta_base * pow(freq_base, inv_ndims*i0);
const float theta = theta_base * pow(freq_base, inv_ndims*iw);
const float freq_factor = src2 != src0 ? src2[ic] : 1.0f;
float2 cos_sin_theta = rope_yarn(theta/freq_factor, freq_scale, corr_dims, i0, ext_factor, attn_factor);
float2 cos_sin_theta = rope_yarn(theta/freq_factor, freq_scale, corr_dims, iw, ext_factor, attn_factor);
global float * src = (global float *)((global char *) src0 + i3*nb03 + i2*nb02 + i1*nb01 + ic*nb00);
global float * dst_data = (global float *)((global char *) dst + i3*nb3 + i2*nb2 + i1*nb1 + ic*nb0);
global float * src = (global float *)((global char *) src0 + i3*nb03 + i2*nb02 + i1*nb01 + (n_offs + ic)*nb00);
global float * dst_data = (global float *)((global char *) dst + i3*nb3 + i2*nb2 + i1*nb1 + (n_offs + ic)*nb0);
const float x0 = src[0];
const float x1 = src[n_dims/2];
@@ -504,7 +514,8 @@ kernel void kernel_rope_multi_f16(
float beta_fast,
float beta_slow,
int4 sections,
int is_imrope
int is_imrope,
int n_offs
) {
src0 = (global void*)((global char*)src0 + offset0);
src1 = (global int*)((global char*)src1 + offset1);
@@ -525,10 +536,11 @@ kernel void kernel_rope_multi_f16(
float inv_ndims = -1.f/n_dims;
for (int i0 = 2*get_local_id(0); i0 < ne0; i0 += 2*get_local_size(0)) {
if (i0 < n_dims) {
int ic = i0/2;
if (i0 >= n_offs && i0 < n_offs + n_dims) {
int iw = i0 - n_offs; // relative idx
int ic = iw/2;
const int sector = (i0 / 2) % sect_dims;
const int sector = ic % sect_dims;
float theta_base = 0.0f;
if (is_imrope) {
@@ -556,14 +568,14 @@ kernel void kernel_rope_multi_f16(
}
}
const float theta = theta_base * pow(freq_base, inv_ndims*i0);
const float theta = theta_base * pow(freq_base, inv_ndims*iw);
const float freq_factor = src2 != src0 ? src2[ic] : 1.0f;
float2 cos_sin_theta = rope_yarn(theta/freq_factor, freq_scale, corr_dims, i0, ext_factor, attn_factor);
float2 cos_sin_theta = rope_yarn(theta/freq_factor, freq_scale, corr_dims, iw, ext_factor, attn_factor);
global half * src = (global half *)((global char *) src0 + i3*nb03 + i2*nb02 + i1*nb01 + ic*nb00);
global half * dst_data = (global half *)((global char *) dst + i3*nb3 + i2*nb2 + i1*nb1 + ic*nb0);
global half * src = (global half *)((global char *) src0 + i3*nb03 + i2*nb02 + i1*nb01 + (n_offs + ic)*nb00);
global half * dst_data = (global half *)((global char *) dst + i3*nb3 + i2*nb2 + i1*nb1 + (n_offs + ic)*nb0);
const float x0 = src[0];
const float x1 = src[n_dims/2];
+26 -1
View File
@@ -1955,6 +1955,23 @@ static void dequantize_mul_mat_vec_q4_K_sycl_reorder_esimd(const void *vx, const
});
}
static void dequantize_mul_mat_vec_q5_K_sycl_reorder_esimd(const void *vx, const float *y,
float *dst, const int ncols,
const int nrows,
dpct::queue_ptr stream) {
GGML_ASSERT(ncols % QK_K == 0);
const int workgroups = (nrows + 1) / 2;
stream->submit([&](sycl::handler &h) {
sycl::local_accessor<float, 1> lmem(sycl::range<1>(GGML_SYCL_DMMV_ESIMD_WG_SIZE * 2), h);
h.parallel_for(
sycl::nd_range<1>(sycl::range<1>((size_t)workgroups * GGML_SYCL_DMMV_ESIMD_WG_SIZE), sycl::range<1>(GGML_SYCL_DMMV_ESIMD_WG_SIZE)),
[=](sycl::nd_item<1> it) [[intel::sycl_explicit_simd]] {
dequantize_mul_mat_vec_reorder_esimd<GGML_TYPE_Q5_K>(
vx, y, dst, ncols, nrows, lmem, it);
});
});
}
static void dequantize_mul_mat_vec_q6_K_sycl_reorder_esimd(const void *vx, const float *y,
float *dst, const int ncols,
const int nrows,
@@ -2134,7 +2151,15 @@ void ggml_sycl_op_dequantize_mul_mat_vec(
case GGML_TYPE_Q5_K:
if ((ggml_tensor_extra_gpu *) dst->src[0]->extra &&
((ggml_tensor_extra_gpu *) dst->src[0]->extra)->optimized_feature.reorder) {
dequantize_mul_mat_vec_q5_K_sycl_reorder(src0_dd_i, src1_ddf_i, dst_dd_i, ne00, row_diff, stream);
#ifdef GGML_SYCL_DMMV_HAS_ESIMD
if (g_ggml_sycl_enable_esimd) {
dequantize_mul_mat_vec_q5_K_sycl_reorder_esimd(src0_dd_i, src1_ddf_i, dst_dd_i, ne00, row_diff, stream);
}
else
#endif
{
dequantize_mul_mat_vec_q5_K_sycl_reorder(src0_dd_i, src1_ddf_i, dst_dd_i, ne00, row_diff, stream);
}
} else {
dequantize_mul_mat_vec_q5_K_sycl(src0_dd_i, src1_ddf_i, dst_dd_i, ne00, row_diff, stream);
}
+1 -1
View File
@@ -62,7 +62,7 @@
#define DPCT_UNUSED(x) (void)(x)
inline void _abort(const char * str) {
[[noreturn]] inline void _abort(const char * str) {
std::cerr << str << std::endl;
std::abort();
}
+4 -4
View File
@@ -10,7 +10,7 @@
(ITEM.get_local_range(IDX) * ITEM.get_group(IDX) + ITEM.get_local_id(IDX))
static void acc_f32(const char * x, const char * y, float * dst, const int64_t ne,
const int64_t ne0, const int64_t ne1, const int64_t ne2, const int64_t ne3,
const int64_t ne0, const int64_t ne1, const int64_t ne2,
const int64_t nb00, const int64_t nb01, const int64_t nb02, const int64_t nb03,
const int64_t ne10, const int64_t ne11, const int64_t ne12, const int64_t ne13,
const int64_t nb10, const int64_t nb11, const int64_t nb12, const int64_t nb13,
@@ -455,7 +455,7 @@ static void unary_mul_sycl(const T * x, const T * g, T * dst, const int64_t k, c
namespace ggml_sycl_detail {
static void acc_f32_sycl(const char *x, const char *y, float *dst,
const int64_t n_elements,
const int64_t ne0, const int64_t ne1, const int64_t ne2, const int64_t ne3,
const int64_t ne0, const int64_t ne1, const int64_t ne2,
const int64_t nb00, const int64_t nb01, const int64_t nb02, const int64_t nb03,
const int64_t ne10, const int64_t ne11, const int64_t ne12, const int64_t ne13,
const int64_t nb10, const int64_t nb11, const int64_t nb12, const int64_t nb13,
@@ -466,7 +466,7 @@ static void acc_f32_sycl(const char *x, const char *y, float *dst,
sycl::range<3>(1, 1, SYCL_ACC_BLOCK_SIZE)),
[=](sycl::nd_item<3> /*item_ct1*/) [[sycl::reqd_sub_group_size(WARP_SIZE)]] {
acc_f32(x, y, dst, n_elements,
ne0, ne1, ne2, ne3,
ne0, ne1, ne2,
nb00, nb01, nb02, nb03,
ne10, ne11, ne12, ne13,
nb10, nb11, nb12, nb13,
@@ -970,7 +970,7 @@ static inline void ggml_sycl_op_acc(ggml_backend_sycl_context & ctx, ggml_tensor
const int64_t offset = (int64_t) ((const int32_t *) dst->op_params)[3] / (int64_t) sizeof(float);
ggml_sycl_detail::acc_f32_sycl(src0_d, src1_d, dst_d, ggml_nelements(dst),
dst->ne[0], dst->ne[1], dst->ne[2], dst->ne[3],
dst->ne[0], dst->ne[1], dst->ne[2],
src0->nb[0], src0->nb[1], src0->nb[2], src0->nb[3],
src1->ne[0], src1->ne[1], src1->ne[2], src1->ne[3],
src1->nb[0], src1->nb[1], src1->nb[2], src1->nb[3],
+122 -12
View File
@@ -1,15 +1,3 @@
//
// MIT license
// Copyright (C) 2026 Intel Corporation
// SPDX-License-Identifier: MIT
//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
#ifndef GGML_SYCL_ESIMD_HPP
#define GGML_SYCL_ESIMD_HPP
@@ -287,6 +275,128 @@ template <> struct esimd_reorder_q_traits<GGML_TYPE_Q4_K> {
}
};
// ---------------------------------------------------------------------------
// Q5_K, SOA reorder layout produced by reorder_qw_q5_k:
// [qs: nb*(QK_K/2)] [qh: nb*(QK_K/8)] [scales: nb*K_SCALE_SIZE] [dm: nb*sizeof(half2)]
// with nb = nrows*num_blocks_per_row.
//
// Identical to Q4_K except each 4-bit quant gains a 5th (high) bit from qh:
// output chunk c (0..7) adds 16 when bit c of qh[l] is set, where qh[l] indexes
// the same 32 bytes for every chunk (matches dequantize_row_q5_K).
// ---------------------------------------------------------------------------
template <> struct esimd_reorder_q_traits<GGML_TYPE_Q5_K> {
struct ptrs {
const uint8_t * qs;
const uint8_t * qh;
const uint8_t * scales;
const sycl::half * dm;
};
static ESIMD_INLINE ptrs make_ptrs(const void * vx, size_t nb) {
const uint8_t * qs = (const uint8_t *) vx;
const uint8_t * qh = qs + nb * (QK_K / 2);
const uint8_t * scales = qh + nb * (QK_K / 8);
const sycl::half * dm = (const sycl::half *) (scales + nb * K_SCALE_SIZE);
return { qs, qh, scales, dm };
}
// extract bit `bit` (0..7) of each lane and move it to bit position 4,
// e.g. for the 4-bit base quant's 5th (high) bit. `bit` is always a
// compile-time-known unrolled loop constant at call sites, so this folds
// to a single mask (bit==4), mask+left-shift (bit<4), or mask+right-shift
// (bit>4) instead of the shift+mask+shift a naive `(qh>>bit & 1) << 4` emits.
static ESIMD_INLINE sycl::ext::intel::esimd::simd<uint16_t, 32> extract_bit_to_pos4(
sycl::ext::intel::esimd::simd<uint8_t, 32> qh, int bit) {
using namespace sycl::ext::intel::esimd;
simd<uint16_t, 32> masked = convert<uint16_t>(qh & simd<uint8_t, 32>((uint8_t) (1u << bit)));
if (bit < 4) {
return masked << simd<uint16_t, 32>((uint16_t) (4 - bit));
} else if (bit > 4) {
return masked >> simd<uint16_t, 32>((uint16_t) (bit - 4));
}
return masked;
}
static ESIMD_INLINE void mac_pair(
const ptrs & pa, size_t bia,
const ptrs & pb, size_t bib, bool has_b,
sycl::ext::intel::esimd::simd<float, 256> & y_vec,
sycl::ext::intel::esimd::simd<float, 32> & acc_a,
sycl::ext::intel::esimd::simd<float, 32> & acc_b) {
using namespace sycl::ext::intel::esimd;
simd<uint8_t, 128> qs_a = block_load<uint8_t, 128>(pa.qs + bia * (QK_K / 2));
simd<uint8_t, 128> qs_b = 0;
simd<uint8_t, 32> qh_a = block_load<uint8_t, 32>(pa.qh + bia * (QK_K / 8));
simd<uint8_t, 32> qh_b = 0;
simd<uint8_t, 12> scales_a = block_load<uint8_t, 12>(pa.scales + bia * K_SCALE_SIZE);
simd<uint8_t, 12> scales_b = 0;
const float dall_a = (float) pa.dm[bia * 2 + 0];
const float dmin_a = (float) pa.dm[bia * 2 + 1];
float dall_b = 0.0f;
float dmin_b = 0.0f;
if (has_b) {
qs_b = block_load<uint8_t, 128>(pb.qs + bib * (QK_K / 2));
qh_b = block_load<uint8_t, 32>(pb.qh + bib * (QK_K / 8));
scales_b = block_load<uint8_t, 12>(pb.scales + bib * K_SCALE_SIZE);
dall_b = (float) pb.dm[bib * 2 + 0];
dmin_b = (float) pb.dm[bib * 2 + 1];
}
simd<float, 8> scale_f_a, min_f_a, scale_f_b, min_f_b;
unpack_scale_min_k4(scales_a, dall_a, dmin_a, scale_f_a, min_f_a);
unpack_scale_min_k4(scales_b, dall_b, dmin_b, scale_f_b, min_f_b);
simd<uint8_t, 128> qs_lo_a = qs_a & simd<uint8_t, 128>(0x0F);
simd<uint8_t, 128> qs_hi_a = qs_a >> simd<uint8_t, 128>(4);
simd<uint8_t, 128> qs_lo_b = qs_b & simd<uint8_t, 128>(0x0F);
simd<uint8_t, 128> qs_hi_b = qs_b >> simd<uint8_t, 128>(4);
#pragma unroll
for (int sb = 0; sb < 8; sb += 2) {
const int q_offset = sb * 16;
simd<float, 32> y_lo = y_vec.select<32, 1>(sb * 32);
simd<float, 32> y_hi = y_vec.select<32, 1>((sb + 1) * 32);
const float scale_a_lo = scale_f_a[sb];
const float scale_a_hi = scale_f_a[sb + 1];
const float min_a_lo = min_f_a[sb];
const float min_a_hi = min_f_a[sb + 1];
const float scale_b_lo = scale_f_b[sb];
const float scale_b_hi = scale_f_b[sb + 1];
const float min_b_lo = min_f_b[sb];
const float min_b_hi = min_f_b[sb + 1];
simd<uint8_t, 32> qa_lo_u8 = qs_lo_a.select<32, 1>(q_offset);
simd<uint8_t, 32> qa_hi_u8 = qs_hi_a.select<32, 1>(q_offset);
simd<uint8_t, 32> qb_lo_u8 = qs_lo_b.select<32, 1>(q_offset);
simd<uint8_t, 32> qb_hi_u8 = qs_hi_b.select<32, 1>(q_offset);
simd<uint16_t, 32> qa_lo = convert<uint16_t>(qa_lo_u8);
simd<uint16_t, 32> qa_hi = convert<uint16_t>(qa_hi_u8);
simd<uint16_t, 32> qb_lo = convert<uint16_t>(qb_lo_u8);
simd<uint16_t, 32> qb_hi = convert<uint16_t>(qb_hi_u8);
// add the 5th bit: chunk sb uses qh bit sb, chunk sb+1 uses qh bit sb+1;
// qh always indexes the same 32 bytes regardless of chunk
qa_lo += extract_bit_to_pos4(qh_a, sb);
qa_hi += extract_bit_to_pos4(qh_a, sb + 1);
qb_lo += extract_bit_to_pos4(qh_b, sb);
qb_hi += extract_bit_to_pos4(qh_b, sb + 1);
simd<float, 32> deq_a_lo = convert<float>(qa_lo) * scale_a_lo + min_a_lo;
simd<float, 32> deq_a_hi = convert<float>(qa_hi) * scale_a_hi + min_a_hi;
simd<float, 32> deq_b_lo = convert<float>(qb_lo) * scale_b_lo + min_b_lo;
simd<float, 32> deq_b_hi = convert<float>(qb_hi) * scale_b_hi + min_b_hi;
acc_a += y_lo * deq_a_lo;
acc_b += y_lo * deq_b_lo;
acc_a += y_hi * deq_a_hi;
acc_b += y_hi * deq_b_hi;
}
}
};
// ---------------------------------------------------------------------------
// Q6_K, SOA reorder layout:
// [ql: nb*(QK_K/2)] [qh: nb*(QK_K/4)] [scales(int8): nb*(QK_K/16)] [d: nb*half]
+4 -5
View File
@@ -43,7 +43,7 @@ static void mkl_fa_pack_q_fp16(
dpct::queue_ptr stream,
sycl::half * __restrict dst,
const float * __restrict q_src,
int n_queries, int n_query_rows, int DKQ,
int n_queries, int DKQ,
int gqa_ratio, int kvh_base_head,
float q_scale, int64_t q_row_stride, int64_t q_head_stride,
int64_t wg_size) {
@@ -121,7 +121,7 @@ static void mkl_fa_online_softmax_chunk(
float * __restrict VKQ_accum,
int q0, int q_rows, int n_queries, int DV,
int chunk_size, int chunk_start,
int kvh_head, int gqa_ratio,
int kvh_head,
const sycl::half * mask_data, int64_t mask_head_stride,
int64_t mask_row_stride, int mask_n_heads,
float logit_softcap, int64_t wg_size) {
@@ -473,7 +473,6 @@ void ggml_sycl_flash_attn_ext_mkl(ggml_backend_sycl_context & ctx, ggml_tensor *
MKL_ACCUM(dequant_time_us, t_deq);
// --- Resolve mask pointers ---
const sycl::half * mask_data = nullptr;
int64_t mask_head_stride = 0;
int64_t mask_row_stride = 0;
int mask_n_heads = 0;
@@ -547,7 +546,7 @@ void ggml_sycl_flash_attn_ext_mkl(ggml_backend_sycl_context & ctx, ggml_tensor *
// 1. Pack all GQA Q heads into fp16 (full n_query_rows)
mkl_fa_pack_q_fp16(stream,
Q_head_f16_ptr, Q_batch,
n_queries, n_query_rows, DKQ,
n_queries, DKQ,
gqa_ratio, kvh_base_head,
q_scale, q_row_stride, q_head_stride, wg_size);
@@ -605,7 +604,7 @@ void ggml_sycl_flash_attn_ext_mkl(ggml_backend_sycl_context & ctx, ggml_tensor *
KQ_max_ptr, KQ_sum_ptr, VKQ_accum_ptr,
q0, q_rows, n_queries, DV,
this_chunk, chunk_start,
kvh_base_head, gqa_ratio,
kvh_base_head,
mask_batch, mask_head_stride,
mask_row_stride, mask_n_heads,
logit_softcap, wg_size);
+11 -8
View File
@@ -21,14 +21,6 @@ bool ggml_sycl_flash_attn_ext_onednn_supported(const ggml_tensor * dst) {
if (!g_ggml_sycl_fa_onednn) {
return false;
}
// Battlemage (Xe2) only, for now. On other Intel archs oneDNN's fused SDPA returns wrong results
// for some shapes (e.g. head_dim=64 on Arc / xe_hpg) -- an oneDNN bug tracked upstream at
// https://github.com/uxlfoundation/oneDNN/issues/5510. Remove this hardware limitation once that
// is fixed; until then non-BMG archs fall back to the existing FA kernel.
const gpu_arch arch = ggml_sycl_info().devices[ggml_sycl_get_device()].hw_info.arch;
if (arch != gpu_arch::intel_gpu_bmg_g21 && arch != gpu_arch::intel_gpu_bmg_g31) {
return false;
}
const ggml_tensor * Q = dst->src[0];
const ggml_tensor * K = dst->src[1];
const ggml_tensor * V = dst->src[2];
@@ -60,6 +52,17 @@ bool ggml_sycl_flash_attn_ext_onednn_supported(const ggml_tensor * dst) {
}
}
}
// This is the improved SPDA gate. Rather than gating Alchemist GPUs from all SPDA features, we instead target only the failing shapes.
// If the GPU being assessed isn't in the grouping below, it has full access to all SPDA shapes. Otherwise, if it's an Alchemist GPU, we block only the shapes with head sizes that fail.
// It is much easier to compare the device to a small list of failing cases than to define all the passing ones.
const gpu_arch arch = ggml_sycl_info().devices[ggml_sycl_get_device()].hw_info.arch;
bool support_spda = !(arch == gpu_arch::intel_gpu_dg2_g10 ||
arch == gpu_arch::intel_gpu_dg2_g11 ||
arch == gpu_arch::intel_gpu_dg2_g12);
if (!support_spda && K->ne[0] == 64) {
return false;
}
// Optional KV-length ceiling (GGML_SYCL_FA_ONEDNN_MAX_KV, 0 = unlimited). Escape hatch:
// very long sequences make the fused SDPA slow enough to risk the xe driver watchdog on
// some stacks; past the cap we fall back to the native FA kernel instead.
+19 -13
View File
@@ -921,16 +921,16 @@ ggml_backend_sycl_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft,
void * dev_ptr;
if (use_usm_system) {
GGML_SYCL_DEBUG("[SYCL] allocating %lu Bytes with USM system\n", size);
GGML_SYCL_DEBUG("[SYCL] allocating %zu Bytes with USM system\n", size);
dev_ptr = (void *)aligned_malloc_host(alignment, aligned_size);
if (!dev_ptr) {
GGML_LOG_ERROR("%s: can't allocate %lu Bytes of memory on host\n", __func__, size);
GGML_LOG_ERROR("%s: can't allocate %zu Bytes of memory on host\n", __func__, size);
return nullptr;
}
} else {
SYCL_CHECK(CHECK_TRY_ERROR(dev_ptr = (void *)ggml_sycl_malloc_device(size, *stream)));
if (!dev_ptr) {
GGML_LOG_ERROR("%s: can't allocate %lu Bytes of memory on device\n", __func__, size);
GGML_LOG_ERROR("%s: can't allocate %zu Bytes of memory on device\n", __func__, size);
return nullptr;
}
}
@@ -1177,7 +1177,7 @@ ggml_backend_sycl_split_buffer_init_tensor(ggml_backend_buffer_t buffer,
SYCL_CHECK(CHECK_TRY_ERROR(buf = (char *)ggml_sycl_malloc_device(size, *stream)));
if (!buf) {
char err_buf[1024];
snprintf(err_buf, 1023, "%s: can't allocate %lu Bytes of memory on device\n", __func__, size);
snprintf(err_buf, 1023, "%s: can't allocate %zu Bytes of memory on device\n", __func__, size);
throw std::runtime_error(err_buf);
}
// set padding to 0 to avoid possible NaN values
@@ -1517,8 +1517,13 @@ static ggml_backend_buffer_t ggml_backend_sycl_host_buffer_type_alloc_buffer(ggm
}
static size_t ggml_backend_sycl_host_buffer_type_get_max_size(ggml_backend_buffer_type_t buft) {
ggml_backend_sycl_device_context * dev_ctx = (ggml_backend_sycl_device_context *) buft->device->context;
return dpct::dev_mgr::instance().get_device(dev_ctx->device).get_max_mem_alloc_size();
if (g_ggml_sycl_enable_host_pinned_mem) {
ggml_backend_sycl_device_context * dev_ctx = (ggml_backend_sycl_device_context *) buft->device->context;
return dpct::dev_mgr::instance().get_device(dev_ctx->device).get_max_mem_alloc_size();
} else {
return SIZE_MAX;
}
}
ggml_backend_buffer_type_t ggml_backend_sycl_host_buffer_type() {
@@ -1646,7 +1651,7 @@ struct ggml_sycl_pool_leg : public ggml_sycl_pool {
SYCL_CHECK(CHECK_TRY_ERROR(ptr = (void *)ggml_sycl_malloc_device(look_ahead_size, *qptr)));
if (!ptr) {
GGML_LOG_ERROR("%s: can't allocate %lu Bytes of memory on device/GPU\n", __func__, look_ahead_size);
GGML_LOG_ERROR("%s: can't allocate %zu Bytes of memory on device/GPU\n", __func__, look_ahead_size);
return nullptr;
}
@@ -1658,7 +1663,7 @@ struct ggml_sycl_pool_leg : public ggml_sycl_pool {
(uint32_t)(max_size/1024/1024), (uint32_t)(g_sycl_pool_size[id]/1024/1024), (uint32_t)(size/1024/1024));
#endif
// GGML_SYCL_DEBUG("ggml_sycl_pool_malloc_leg look_ahead_size=%lu, return %p\n", look_ahead_size, ptr);
// GGML_SYCL_DEBUG("ggml_sycl_pool_malloc_leg look_ahead_size=%zu, return %p\n", look_ahead_size, ptr);
return ptr;
}
@@ -1838,7 +1843,7 @@ struct ggml_sycl_pool_host : public ggml_sycl_pool {
SYCL_CHECK(CHECK_TRY_ERROR(ptr = (void *) sycl::malloc_host(size, *qptr)));
if (!ptr) {
GGML_LOG_ERROR("%s: can't allocate %lu Bytes of memory on host\n", __func__, size);
GGML_LOG_ERROR("%s: can't allocate %zu Bytes of memory on host\n", __func__, size);
return nullptr;
}
pool_size += size;
@@ -2774,9 +2779,9 @@ inline void ggml_sycl_op_mul_mat_sycl(
const float * src1_ddf1_i = src1->type == GGML_TYPE_F32 ? (const float *) src1_ddf_i : src1_ddq_as_f32.get();
{
#if GGML_SYCL_DNNL
const int64_t gemm_flops = (int64_t)row_diff * src1_ncols * ne10;
const bool use_mkl_direct = gemm_flops < 256 * 256 * 256;
#if GGML_SYCL_DNNL
if (g_ggml_sycl_enable_dnn && !use_mkl_direct) {
DnnlGemmWrapper::row_gemm(ctx, row_diff, src1_ncols, ne10, src0_ddf_i,
DnnlGemmWrapper::to_dt<float>(), src1_ddf1_i, DnnlGemmWrapper::to_dt<float>(),
@@ -3513,7 +3518,9 @@ static void ggml_sycl_mul_mat_batched_sycl(ggml_backend_sycl_context & ctx, cons
float * dst_ddf = static_cast<float *>(dst->data);
const sycl::half * src1_f16 = static_cast<const sycl::half *>(src1->data);
#if GGML_SYCL_DNNL
const size_t type_size_src0 = ggml_type_size(src0->type);
#endif
const size_t type_size_src1 = ggml_type_size(src1->type);
bool is_src0_cont_2 = ggml_is_contiguous_2(src0);
@@ -3530,6 +3537,7 @@ static void ggml_sycl_mul_mat_batched_sycl(ggml_backend_sycl_context & ctx, cons
scope_op_debug_print scope_dbg_print(__func__, "/to_fp16_nc_sycl", dst, /*num_src=*/2,
" : converting src1 to fp16");
#if GGML_SYCL_DNNL
// iterate tensor dims and find the slowest moving dim and stride
int last_dim=0;
int last_str=0;
@@ -3549,7 +3557,6 @@ static void ggml_sycl_mul_mat_batched_sycl(ggml_backend_sycl_context & ctx, cons
}
}
#if GGML_SYCL_DNNL
// oneDNN handles strided data and does not need overhead of ggml_get_to_fp16_nc_sycl
const int64_t ne_src1 = src1->nb[last_str] * src1->ne[last_dim] / type_size_src1;
src1_f16_alloc.alloc(ne_src1);
@@ -3804,6 +3811,7 @@ static bool ggml_sycl_supports_reorder_esimd(enum ggml_type type) {
switch (type) {
case GGML_TYPE_Q3_K:
case GGML_TYPE_Q4_K:
case GGML_TYPE_Q5_K:
case GGML_TYPE_Q6_K:
return true;
default:
@@ -6242,8 +6250,6 @@ static bool do_ggml_backend_sycl_device_supports_op(ggml_backend_dev_t dev, cons
}
case GGML_OP_ROPE:
case GGML_OP_ROPE_BACK:
// FIXME: support ggml_rope_set_offset
return ((const int32_t *) op->op_params)[15] == 0;
case GGML_OP_IM2COL:
case GGML_OP_IM2COL_3D:
case GGML_OP_UPSCALE:
+2 -2
View File
@@ -85,7 +85,7 @@ static void im2col_sycl(const float * x,
*/
stream->parallel_for(sycl::nd_range<3>(block_nums * sycl::range<3>(1, 1, MIN(IC_KH_KW, SYCL_IM2COL_BLOCK_SIZE)),
sycl::range<3>(1, 1, MIN(IC_KH_KW, SYCL_IM2COL_BLOCK_SIZE))),
[=](sycl::nd_item<3> item_ct1) {
[=](sycl::nd_item<3>) {
im2col_kernel(x, dst, IC, IW, IH, OH, OW, KW, KH, IC_IH_IW, IH_IW, N_OH, KH_KW, IC_KH_KW,
s0, s1, p0, p1, d0, d1);
});
@@ -271,7 +271,7 @@ static void im2col_3d_sycl(const float * src,
*/
stream->parallel_for(sycl::nd_range<3>(block_nums * sycl::range<3>(1, 1, MIN(IC_KD_KH_KW, SYCL_IM2COL_BLOCK_SIZE)),
sycl::range<3>(1, 1, MIN(IC_KD_KH_KW, SYCL_IM2COL_BLOCK_SIZE))),
[=](sycl::nd_item<3> item_ct1) {
[=](sycl::nd_item<3>) {
im2col_3d_kernel(src, dst, N, IC, ID, IH, IW, OC, KD, KH, KW, OD, OH, OW, OH_OW, KD_KH_KW,
ID_IH_IW, KH_KW, IH_IW, IC_ID_IH_IW, IC_KD_KH_KW, OW_KD_KH_KW,
OD_OH_OW_IC_KD_KH_KW, OH_OW_IC_KD_KH_KW, OW_IC_KD_KH_KW, N_OD_OH, OD_OH,
-8
View File
@@ -7,9 +7,6 @@ static void norm_f32(const float* x, float* dst, const int ncols,
const int64_t dst_stride_col, const int64_t dst_stride_row, const int64_t dst_stride_channel, const int64_t dst_stride_sample,
const float eps, const sycl::nd_item<3>& item_ct1, sycl::float2* s_sum, int block_size) {
const int nrows = item_ct1.get_group_range(2);
const int nchannels = item_ct1.get_group_range(1);
const int nthreads = item_ct1.get_local_range(2);
const int sample = item_ct1.get_group(0);
const int channel = item_ct1.get_group(1);
@@ -155,9 +152,6 @@ static void rms_norm_f32(const float* x, float* dst, const int ncols,
const float* mul = nullptr, const int64_t mul_stride_row = 0, const int64_t mul_stride_channel = 0,
const int64_t mul_stride_sample = 0, const int mul_nrows = 0, const int mul_nchannels = 0, const int mul_nsamples = 0) {
const int nrows = item_ct1.get_group_range(2);
const int nchannels = item_ct1.get_group_range(1);
const int sample = item_ct1.get_group(0);
const int channel = item_ct1.get_group(1);
const int row = item_ct1.get_group(2);
@@ -225,8 +219,6 @@ static void l2_norm_f32(const float * x, float * dst, const int ncols,
const int64_t src_stride_sample, const int64_t dst_stride_col, const int64_t dst_stride_row,
const int64_t dst_stride_channel, const int64_t dst_stride_sample, const float eps,
const sycl::nd_item<3>& item_ct1, float* s_sum, const int block_size) {
const int nrows = item_ct1.get_group_range(2);
const int nchannels = item_ct1.get_group_range(1);
const int row = item_ct1.get_group(2);
const int channel = item_ct1.get_group(1);
+58 -48
View File
@@ -41,7 +41,7 @@ template <bool forward, bool has_ff, typename T, typename D>
static void rope_norm(const T *x, D *dst, const int ne00, const int ne01,
const int ne02, const int s01, const int s02,
const int s03, const int s1, const int s2, const int s3,
const int n_dims, const int32_t *pos,
const int n_dims, const int n_offs, const int32_t *pos,
const float freq_scale, const float ext_factor,
const float attn_factor, const rope_corr_dims corr_dims,
const float theta_scale, const float *freq_factors,
@@ -78,19 +78,21 @@ static void rope_norm(const T *x, D *dst, const int ne00, const int ne01,
ggml_sycl_memcpy_1<4>(dst + idst, &v);
}
};
if (i0 >= n_dims) {
if (i0 < n_offs || i0 >= n_offs + n_dims) {
store_coaelsced(x[ix + 0], x[ix + 1]);
return;
}
const float theta_base = pos[i2] * dpct::pow(theta_scale, i0 / 2.0f);
const int iw = i0 - n_offs; // relative idx
const float freq_factor = has_ff ? freq_factors[i0 / 2] : 1.0f;
const float theta_base = pos[i2] * dpct::pow(theta_scale, iw / 2.0f);
const float freq_factor = has_ff ? freq_factors[iw / 2] : 1.0f;
float cos_theta;
float sin_theta;
rope_yarn<forward>(theta_base / freq_factor, freq_scale, corr_dims, i0,
rope_yarn<forward>(theta_base / freq_factor, freq_scale, corr_dims, iw,
ext_factor, attn_factor, cos_theta, sin_theta);
const float x0 = x[ix + 0];
@@ -104,7 +106,7 @@ template <bool forward, bool has_ff, typename T, typename D>
static void rope_neox(const T *x, D *dst, const int ne00, const int ne01,
const int ne02, const int s01, const int s02,
const int s03, const int s1, const int s2, const int s3,
const int n_dims, const int32_t *pos,
const int n_dims, const int n_offs, const int32_t *pos,
const float freq_scale, const float ext_factor,
const float attn_factor, const rope_corr_dims corr_dims,
const float theta_scale, const float *freq_factors,
@@ -132,35 +134,38 @@ static void rope_neox(const T *x, D *dst, const int ne00, const int ne01,
idst += row_indices[i2] * set_rows_stride;
}
if (i0 >= n_dims) {
if (i0 < n_offs || i0 >= n_offs + n_dims) {
dst[idst + i0 / 2 + 0] = ggml_sycl_cast<D>(x[ix + i0 / 2 + 0]);
dst[idst + i0 / 2 + 1] = ggml_sycl_cast<D>(x[ix + i0 / 2 + 1]);
return;
}
const float theta_base = pos[i2] * dpct::pow(theta_scale, i0 / 2.0f);
const int iw = i0 - n_offs; // relative idx
const float freq_factor = has_ff ? freq_factors[i0 / 2] : 1.0f;
const float theta_base = pos[i2] * dpct::pow(theta_scale, iw / 2.0f);
const float freq_factor = has_ff ? freq_factors[iw / 2] : 1.0f;
float cos_theta;
float sin_theta;
rope_yarn<forward>(theta_base / freq_factor, freq_scale, corr_dims, i0,
rope_yarn<forward>(theta_base / freq_factor, freq_scale, corr_dims, iw,
ext_factor, attn_factor, cos_theta, sin_theta);
const float x0 = x[ix + 0];
const float x1 = x[ix + n_dims / 2];
// idst/ix point at channel i0/2; the first channel of the rotated pair is n_offs + iw/2 = i0/2 + n_offs/2
const float x0 = x[ix + n_offs / 2 + 0];
const float x1 = x[ix + n_offs / 2 + n_dims / 2];
dst[idst + 0] = ggml_sycl_cast<D>(x0 * cos_theta - x1 * sin_theta);
dst[idst + n_dims / 2] = ggml_sycl_cast<D>(x0 * sin_theta + x1 * cos_theta);
dst[idst + n_offs / 2 + 0] = ggml_sycl_cast<D>(x0 * cos_theta - x1 * sin_theta);
dst[idst + n_offs / 2 + n_dims / 2] = ggml_sycl_cast<D>(x0 * sin_theta + x1 * cos_theta);
}
template <bool forward, bool has_ff, typename T>
static void rope_multi(const T *x, T *dst, const int ne00, const int ne01,
const int ne02, const int s01, const int s02,
const int s03, const int s1, const int s2, const int s3,
const int n_dims, const int32_t *pos,
const int n_dims, const int n_offs, const int32_t *pos,
const float freq_scale, const float ext_factor,
const float attn_factor, const rope_corr_dims corr_dims,
const float theta_scale, const float *freq_factors,
@@ -183,54 +188,57 @@ static void rope_multi(const T *x, T *dst, const int ne00, const int ne01,
int idst = i0 / 2 + i1 * s1 + i2 * s2 + i3 * s3;
const int ix = i0 / 2 + i1 * s01 + i2 * s02 + i3 * s03;
if (i0 >= n_dims) {
if (i0 < n_offs || i0 >= n_offs + n_dims) {
dst[idst + i0 / 2 + 0] = x[ix + i0 / 2 + 0];
dst[idst + i0 / 2 + 1] = x[ix + i0 / 2 + 1];
return;
}
const int iw = i0 - n_offs; // relative idx
const int sect_dims =
sections.v[0] + sections.v[1] + sections.v[2] + sections.v[3];
const int sec_w = sections.v[1] + sections.v[0];
const int sector = (i0 / 2) % sect_dims;
const int sector = (iw / 2) % sect_dims;
float theta_base = 0.0;
if (is_imrope) {
if (sector % 3 == 1 && sector < 3 * sections.v[1]) { // h
theta_base = pos[i2 + ne02 * 1] * dpct::pow(theta_scale, i0 / 2.0f);
theta_base = pos[i2 + ne02 * 1] * dpct::pow(theta_scale, iw / 2.0f);
} else if (sector % 3 == 2 && sector < 3 * sections.v[2]) { // w
theta_base = pos[i2 + ne02 * 2] * dpct::pow(theta_scale, i0 / 2.0f);
theta_base = pos[i2 + ne02 * 2] * dpct::pow(theta_scale, iw / 2.0f);
} else if (sector % 3 == 0 && sector < 3 * sections.v[0]) { // t
theta_base = pos[i2] * dpct::pow(theta_scale, i0 / 2.0f);
theta_base = pos[i2] * dpct::pow(theta_scale, iw / 2.0f);
} else {
theta_base = pos[i2 + ne02 * 3] * dpct::pow(theta_scale, i0 / 2.0f);
theta_base = pos[i2 + ne02 * 3] * dpct::pow(theta_scale, iw / 2.0f);
}
} else {
if (sector < sections.v[0]) {
theta_base = pos[i2] * dpct::pow(theta_scale, i0 / 2.0f);
theta_base = pos[i2] * dpct::pow(theta_scale, iw / 2.0f);
} else if (sector >= sections.v[0] && sector < sec_w) {
theta_base = pos[i2 + ne02 * 1] * dpct::pow(theta_scale, i0 / 2.0f);
theta_base = pos[i2 + ne02 * 1] * dpct::pow(theta_scale, iw / 2.0f);
} else if (sector >= sec_w && sector < sec_w + sections.v[2]) {
theta_base = pos[i2 + ne02 * 2] * dpct::pow(theta_scale, i0 / 2.0f);
theta_base = pos[i2 + ne02 * 2] * dpct::pow(theta_scale, iw / 2.0f);
} else if (sector >= sec_w + sections.v[2]) {
theta_base = pos[i2 + ne02 * 3] * dpct::pow(theta_scale, i0 / 2.0f);
theta_base = pos[i2 + ne02 * 3] * dpct::pow(theta_scale, iw / 2.0f);
}
}
const float freq_factor = has_ff ? freq_factors[i0 / 2] : 1.0f;
const float freq_factor = has_ff ? freq_factors[iw / 2] : 1.0f;
float cos_theta;
float sin_theta;
rope_yarn<forward>(theta_base / freq_factor, freq_scale, corr_dims, i0,
rope_yarn<forward>(theta_base / freq_factor, freq_scale, corr_dims, iw,
ext_factor, attn_factor, cos_theta, sin_theta);
const float x0 = x[ix + 0];
const float x1 = x[ix + n_dims / 2];
// idst/ix point at channel i0/2; the first channel of the rotated pair is n_offs + iw/2 = i0/2 + n_offs/2
const float x0 = x[ix + n_offs / 2 + 0];
const float x1 = x[ix + n_offs / 2 + n_dims / 2];
dst[idst + 0] = x0 * cos_theta - x1 * sin_theta;
dst[idst + n_dims / 2] = x0 * sin_theta + x1 * cos_theta;
dst[idst + n_offs / 2 + 0] = x0 * cos_theta - x1 * sin_theta;
dst[idst + n_offs / 2 + n_dims / 2] = x0 * sin_theta + x1 * cos_theta;
}
template <bool forward, bool has_ff, typename T>
@@ -293,7 +301,7 @@ static void
rope_norm_sycl(const T *x, D *dst, const int ne00, const int ne01,
const int ne02, const int s01, const int s02, const int s03,
const int s1, const int s2, const int s3, const int n_dims,
const int nr, const int32_t *pos, const float freq_scale,
const int n_offs, const int nr, const int32_t *pos, const float freq_scale,
const float freq_base, const float ext_factor,
const float attn_factor, const rope_corr_dims corr_dims,
const float *freq_factors, const int64_t *row_indices,
@@ -313,7 +321,7 @@ rope_norm_sycl(const T *x, D *dst, const int ne00, const int ne01,
GGML_UNUSED(item_ct1);
rope_norm<forward, false>(
x, dst, ne00, ne01, ne02, s01, s02, s03, s1, s2, s3, n_dims,
pos, freq_scale, ext_factor, attn_factor, corr_dims,
n_offs, pos, freq_scale, ext_factor, attn_factor, corr_dims,
theta_scale, freq_factors, row_indices, set_rows_stride);
});
} else {
@@ -323,7 +331,7 @@ rope_norm_sycl(const T *x, D *dst, const int ne00, const int ne01,
GGML_UNUSED(item_ct1);
rope_norm<forward, true>(
x, dst, ne00, ne01, ne02, s01, s02, s03, s1, s2, s3, n_dims,
pos, freq_scale, ext_factor, attn_factor, corr_dims,
n_offs, pos, freq_scale, ext_factor, attn_factor, corr_dims,
theta_scale, freq_factors, row_indices, set_rows_stride);
});
}
@@ -334,7 +342,7 @@ static void
rope_neox_sycl(const T *x, D *dst, const int ne00, const int ne01,
const int ne02, const int s01, const int s02, const int s03,
const int s1, const int s2, const int s3, const int n_dims,
const int nr, const int32_t *pos, const float freq_scale,
const int n_offs, const int nr, const int32_t *pos, const float freq_scale,
const float freq_base, const float ext_factor,
const float attn_factor, const rope_corr_dims corr_dims,
const float *freq_factors, const int64_t *row_indices,
@@ -354,7 +362,7 @@ rope_neox_sycl(const T *x, D *dst, const int ne00, const int ne01,
GGML_UNUSED(item_ct1);
rope_neox<forward, false>(
x, dst, ne00, ne01, ne02, s01, s02, s03, s1, s2, s3, n_dims,
pos, freq_scale, ext_factor, attn_factor, corr_dims,
n_offs, pos, freq_scale, ext_factor, attn_factor, corr_dims,
theta_scale, freq_factors, row_indices, set_rows_stride);
});
} else {
@@ -364,7 +372,7 @@ rope_neox_sycl(const T *x, D *dst, const int ne00, const int ne01,
GGML_UNUSED(item_ct1);
rope_neox<forward, true>(
x, dst, ne00, ne01, ne02, s01, s02, s03, s1, s2, s3, n_dims,
pos, freq_scale, ext_factor, attn_factor, corr_dims,
n_offs, pos, freq_scale, ext_factor, attn_factor, corr_dims,
theta_scale, freq_factors, row_indices, set_rows_stride);
});
}
@@ -375,7 +383,7 @@ static void
rope_multi_sycl(const T *x, T *dst, const int ne00, const int ne01,
const int ne02, const int s01, const int s02, const int s03,
const int s1, const int s2, const int s3, const int n_dims,
const int nr, const int32_t *pos, const float freq_scale,
const int n_offs, const int nr, const int32_t *pos, const float freq_scale,
const float freq_base, const float ext_factor,
const float attn_factor, const rope_corr_dims corr_dims,
const float *freq_factors, const mrope_sections sections,
@@ -395,7 +403,7 @@ rope_multi_sycl(const T *x, T *dst, const int ne00, const int ne01,
GGML_UNUSED(item_ct1);
rope_multi<forward, false, T>(
x, dst, ne00, ne01, ne02, s01, s02, s03, s1, s2, s3, n_dims,
pos, freq_scale, ext_factor, attn_factor, corr_dims,
n_offs, pos, freq_scale, ext_factor, attn_factor, corr_dims,
theta_scale, freq_factors, sections, is_imrope);
});
} else {
@@ -405,7 +413,7 @@ rope_multi_sycl(const T *x, T *dst, const int ne00, const int ne01,
GGML_UNUSED(item_ct1);
rope_multi<forward, true, T>(
x, dst, ne00, ne01, ne02, s01, s02, s03, s1, s2, s3, n_dims,
pos, freq_scale, ext_factor, attn_factor, corr_dims,
n_offs, pos, freq_scale, ext_factor, attn_factor, corr_dims,
theta_scale, freq_factors, sections, is_imrope);
});
}
@@ -497,6 +505,7 @@ void ggml_sycl_op_rope_impl(ggml_backend_sycl_context &ctx, ggml_tensor *dst,
const int n_dims = ((int32_t *)dst->op_params)[1];
const int mode = ((int32_t *)dst->op_params)[2];
const int n_ctx_orig = ((int32_t *)dst->op_params)[4];
const int n_offs = ((int32_t *)dst->op_params)[15];
mrope_sections sections;
float freq_base;
@@ -526,6 +535,7 @@ void ggml_sycl_op_rope_impl(ggml_backend_sycl_context &ctx, ggml_tensor *dst,
if (is_vision) {
GGML_ASSERT(n_dims == ne00 / 2);
GGML_ASSERT(n_offs == 0); // offset not supported for vision, as the rotated pairs span the whole row
}
const int32_t *pos = (const int32_t *)src1_d;
@@ -545,19 +555,19 @@ void ggml_sycl_op_rope_impl(ggml_backend_sycl_context &ctx, ggml_tensor *dst,
if (src0->type == GGML_TYPE_F32 && dst_type == GGML_TYPE_F32) {
rope_neox_sycl<forward, float, float>(
(const float *)src0_d, (float *)dst_d, ne00, ne01, ne02, s01,
s02, s03, s1, s2, s3, n_dims, nr, pos, freq_scale, freq_base,
s02, s03, s1, s2, s3, n_dims, n_offs, nr, pos, freq_scale, freq_base,
ext_factor, attn_factor, corr_dims, freq_factors, row_indices,
set_rows_stride, stream);
} else if (src0->type == GGML_TYPE_F32 && dst_type == GGML_TYPE_F16) {
rope_neox_sycl<forward, float, sycl::half>(
(const float *)src0_d, (sycl::half *)dst_d, ne00, ne01, ne02,
s01, s02, s03, s1, s2, s3, n_dims, nr, pos, freq_scale,
s01, s02, s03, s1, s2, s3, n_dims, n_offs, nr, pos, freq_scale,
freq_base, ext_factor, attn_factor, corr_dims, freq_factors,
row_indices, set_rows_stride, stream);
} else if (src0->type == GGML_TYPE_F16 && dst_type == GGML_TYPE_F16) {
rope_neox_sycl<forward, sycl::half, sycl::half>(
(const sycl::half *)src0_d, (sycl::half *)dst_d, ne00, ne01,
ne02, s01, s02, s03, s1, s2, s3, n_dims, nr, pos, freq_scale,
ne02, s01, s02, s03, s1, s2, s3, n_dims, n_offs, nr, pos, freq_scale,
freq_base, ext_factor, attn_factor, corr_dims, freq_factors,
row_indices, set_rows_stride, stream);
} else {
@@ -568,13 +578,13 @@ void ggml_sycl_op_rope_impl(ggml_backend_sycl_context &ctx, ggml_tensor *dst,
if (src0->type == GGML_TYPE_F32) {
rope_multi_sycl<forward>((const float *)src0_d, (float *)dst_d,
ne00, ne01, ne02, s01, s02, s03, s1, s2,
s3, n_dims, nr, pos, freq_scale, freq_base,
s3, n_dims, n_offs, nr, pos, freq_scale, freq_base,
ext_factor, attn_factor, corr_dims,
freq_factors, sections, is_imrope, stream);
} else if (src0->type == GGML_TYPE_F16) {
rope_multi_sycl<forward>(
(const sycl::half *)src0_d, (sycl::half *)dst_d, ne00, ne01,
ne02, s01, s02, s03, s1, s2, s3, n_dims, nr, pos, freq_scale,
ne02, s01, s02, s03, s1, s2, s3, n_dims, n_offs, nr, pos, freq_scale,
freq_base, ext_factor, attn_factor, corr_dims, freq_factors,
sections, is_imrope, stream);
} else {
@@ -602,19 +612,19 @@ void ggml_sycl_op_rope_impl(ggml_backend_sycl_context &ctx, ggml_tensor *dst,
if (src0->type == GGML_TYPE_F32 && dst_type == GGML_TYPE_F32) {
rope_norm_sycl<forward, float, float>(
(const float *)src0_d, (float *)dst_d, ne00, ne01, ne02, s01,
s02, s03, s1, s2, s3, n_dims, nr, pos, freq_scale, freq_base,
s02, s03, s1, s2, s3, n_dims, n_offs, nr, pos, freq_scale, freq_base,
ext_factor, attn_factor, corr_dims, freq_factors, row_indices,
set_rows_stride, stream);
} else if (src0->type == GGML_TYPE_F32 && dst_type == GGML_TYPE_F16) {
rope_norm_sycl<forward, float, sycl::half>(
(const float *)src0_d, (sycl::half *)dst_d, ne00, ne01, ne02,
s01, s02, s03, s1, s2, s3, n_dims, nr, pos, freq_scale,
s01, s02, s03, s1, s2, s3, n_dims, n_offs, nr, pos, freq_scale,
freq_base, ext_factor, attn_factor, corr_dims, freq_factors,
row_indices, set_rows_stride, stream);
} else if (src0->type == GGML_TYPE_F16 && dst_type == GGML_TYPE_F16) {
rope_norm_sycl<forward, sycl::half, sycl::half>(
(const sycl::half *)src0_d, (sycl::half *)dst_d, ne00, ne01,
ne02, s01, s02, s03, s1, s2, s3, n_dims, nr, pos, freq_scale,
ne02, s01, s02, s03, s1, s2, s3, n_dims, n_offs, nr, pos, freq_scale,
freq_base, ext_factor, attn_factor, corr_dims, freq_factors,
row_indices, set_rows_stride, stream);
} else {
+1 -1
View File
@@ -291,7 +291,7 @@ static void set_rows_sycl(
stream->parallel_for(
sycl::nd_range<1>(grid_size * block_size, block_size),
[=](sycl::nd_item<1> item_ct1) [[intel::reqd_sub_group_size(WARP_SIZE)]] {
[=](sycl::nd_item<1> item_ct1) [[sycl::reqd_sub_group_size(WARP_SIZE)]] {
k_set_rows<TIn, TIdx, TOut>(
src0_d, src1_d, dst_d,
ne00, ne01, ne02,
+4 -4
View File
@@ -2714,6 +2714,7 @@ static webgpu_encoded_op ggml_webgpu_rope(webgpu_context & ctx,
const int n_dims = ((int32_t *) dst->op_params)[1];
const int mode = ((int32_t *) dst->op_params)[2];
const int n_ctx_orig = ((int32_t *) dst->op_params)[4];
const int n_offs = ((int32_t *) dst->op_params)[15];
float freq_base;
float freq_scale;
@@ -2762,7 +2763,8 @@ static webgpu_encoded_op ggml_webgpu_rope(webgpu_context & ctx,
(uint32_t) sections[0],
(uint32_t) sections[1],
(uint32_t) sections[2],
(uint32_t) sections[3]
(uint32_t) sections[3],
(uint32_t) n_offs
};
std::vector<wgpu::BindGroupEntry> entries = { ggml_webgpu_make_tensor_bind_group_entry(ctx, 0, src0),
@@ -4472,9 +4474,7 @@ static bool ggml_backend_webgpu_device_supports_op(ggml_backend_dev_t dev, const
supports_op = (op->type == GGML_TYPE_F32 && src0->type == GGML_TYPE_F32) && ggml_is_contiguous_rows(src0);
break;
case GGML_OP_ROPE:
// FIXME: support ggml_rope_set_offset
supports_op =
(op->type == GGML_TYPE_F32 || op->type == GGML_TYPE_F16) && ((const int32_t *) op->op_params)[15] == 0;
supports_op = op->type == GGML_TYPE_F32 || op->type == GGML_TYPE_F16;
break;
case GGML_OP_GLU:
switch (ggml_get_glu_op(op)) {
+11 -7
View File
@@ -38,7 +38,8 @@ struct Params {
sections0: u32,
sections1: u32,
sections2: u32,
sections3: u32
sections3: u32,
n_offs: u32
};
@group(0) @binding(0)
@@ -126,7 +127,8 @@ fn rope_yarn(theta_extrap: f32, i: u32) -> vec2<f32> {
fn pair_base(i0: u32, div_2: bool) -> u32 {
if (div_2) {
return i0 / 2;
// first channel of the rotated pair: n_offs + (i0 - n_offs)/2
return i0 / 2 + params.n_offs / 2;
} else {
return i0;
}
@@ -165,20 +167,22 @@ fn main(@builtin(global_invocation_id) gid: vec3<u32>) {
let i_src_row = params.offset_src0 + i3 * params.stride_src03 + i2 * params.stride_src02 + i1 * params.stride_src01;
let i_dst_row = params.offset_dst + i3 * params.stride_dst3 + i2 * params.stride_dst2 + i1 * params.stride_dst1;
if (i0 >= params.n_dims && !is_vision) {
if ((i0 < params.n_offs || i0 >= params.n_offs + params.n_dims) && !is_vision) {
let i_src = i_src_row + i0;
let i_dst = i_dst_row + i0;
rotate(i_dst, i_dst + 1, f32(src0[i_src]), f32(src0[i_src + 1]));
return;
}
let iw = i0 - params.n_offs; // relative idx
var theta_base_mult: u32 = 0;
var theta_scale_pwr: u32 = i0 / 2;
var theta_scale_pwr: u32 = iw / 2;
if (is_mrope) {
let sect_dims = params.sections0 + params.sections1 + params.sections2 + params.sections3;
let sec_w = params.sections1 + params.sections0;
let sec_e = params.sections2 + sec_w;
let sector = (i0 / 2) % sect_dims;
let sector = (iw / 2) % sect_dims;
if (is_imrope) {
if (sector % 3 == 1 && sector < 3 * params.sections1) {
theta_base_mult = 1;
@@ -203,7 +207,7 @@ fn main(@builtin(global_invocation_id) gid: vec3<u32>) {
} else if (sector >= sec_e) {
if (is_vision) {
theta_scale_pwr = sector - sec_e;
theta_scale_pwr = (i0 / 2) % sec_e;
theta_scale_pwr = (iw / 2) % sec_e;
}
theta_base_mult = 3;
} else if (is_vision) {
@@ -212,7 +216,7 @@ fn main(@builtin(global_invocation_id) gid: vec3<u32>) {
}
}
let theta_base = f32(src1[params.offset_src1 + i2 + params.ne2 * theta_base_mult]) * pow(params.theta_scale, f32(theta_scale_pwr));
let thetas = rope_yarn(theta_base/freq_factor(i0), i0);
let thetas = rope_yarn(theta_base/freq_factor(iw), iw);
let i_src = i_src_row + pair_base(i0, is_neox || is_mrope || is_vision);
let i_dst = i_dst_row + pair_base(i0, is_neox || is_mrope || is_vision);
+5 -3
View File
@@ -15,7 +15,8 @@
# tag exists.
#
# Env (when running in GitHub Actions):
# GITHUB_OUTPUT: previous_tag, changelog_title, changelog and nightly are written here
# GITHUB_OUTPUT: previous_tag, changelog_title, changelog, nightly and nightly_tag
# are written here
# GITHUB_REPOSITORY: owner/repo, used to build the nightly release URL (skipped when unset)
set -euo pipefail
@@ -52,10 +53,10 @@ PREV="$( { git tag --list; echo "${VERSION}"; } \
if [[ -n "${PREV}" ]]; then
CHANGELOG="$(git log --oneline "${PREV}..${RELEASE_COMMIT}")"
CHANGELOG_TITLE="Change log since ${PREV}"
CHANGELOG_TITLE="Changelog since ${PREV}"
else
CHANGELOG="(no previous release tag found)"
CHANGELOG_TITLE="Change log"
CHANGELOG_TITLE="Changelog"
fi
# Nightly release: the b* tag pointing at the release commit (|| true: no match is not an error)
@@ -80,6 +81,7 @@ if [[ -n "${GITHUB_OUTPUT:-}" ]]; then
echo "previous_tag=${PREV}"
echo "changelog_title=${CHANGELOG_TITLE}"
echo "nightly=${NIGHTLY}"
echo "nightly_tag=${NIGHTLY_TAG}"
echo "changelog<<CHANGELOG_EOF"
echo "${CHANGELOG}"
echo "CHANGELOG_EOF"
+204
View File
@@ -0,0 +1,204 @@
#!/bin/bash
#
# Release preparation script for llama.cpp.
#
# Bumps the version in CMakeLists.txt on a release candidate branch.
# The branch should then be pushed and a PR created, reviewed, and
# merged. After the PR is merged and the build-cpu workflow has
# completed successfully, the release is finalized by the make-release
# workflow (.github/workflows/make-release.yml), which creates the tag.
#
# Usage:
# ./scripts/release.sh [major|minor|patch] [--dry-run]
#
# Example:
# $ ./scripts/release.sh minor
#
# The script:
# 1. Creates a release candidate branch (llama-rc-v<major>.<minor>.<patch>)
# 2. Bumps the version in CMakeLists.txt
# 3. Commits the version bump
#
set -e
if [ ! -f "CMakeLists.txt" ] || [ ! -d "scripts" ]; then
echo "Error: Must be run from llama.cpp root directory"
exit 1
fi
# Parse command line arguments
VERSION_TYPE=""
DRY_RUN=false
for arg in "$@"; do
case $arg in
--dry-run)
DRY_RUN=true
;;
major|minor|patch)
VERSION_TYPE="$arg"
;;
*)
echo "Error: Unknown argument '$arg'"
echo "Usage: $0 [major|minor|patch] [--dry-run]"
exit 1
;;
esac
done
# Default to patch if no version type specified
VERSION_TYPE="${VERSION_TYPE:-patch}"
# Common validation functions
check_git_status() {
# Check for uncommitted changes (skip in dry-run)
if [ "$DRY_RUN" = false ] && ! git diff-index --quiet HEAD --; then
echo "Error: You have uncommitted changes. Please commit or stash them first."
exit 1
fi
}
check_master_branch() {
# Ensure we're on master branch
CURRENT_BRANCH=$(git branch --show-current)
if [ "$CURRENT_BRANCH" != "master" ]; then
if [ "$DRY_RUN" = true ]; then
echo "[dry run] Warning: Not on master branch (currently on: $CURRENT_BRANCH). Continuing with dry-run..."
echo ""
else
echo "Error: Must be on master branch. Currently on: $CURRENT_BRANCH"
exit 1
fi
fi
}
check_master_up_to_date() {
# Check if we have the latest from master (skip in dry-run)
if [ "$DRY_RUN" = false ]; then
echo "Checking if local master is up-to-date with remote..."
git fetch origin master
LOCAL=$(git rev-parse HEAD)
REMOTE=$(git rev-parse origin/master)
if [ "$LOCAL" != "$REMOTE" ]; then
echo "Error: Your local master branch is not up-to-date with origin/master."
echo "Please run 'git pull origin master' first."
exit 1
fi
echo "✓ Local master is up-to-date with remote"
echo ""
elif [ "$(git branch --show-current)" = "master" ]; then
echo "[dry run] Warning: Dry-run mode - not checking if master is up-to-date with remote"
echo ""
fi
}
# In-place sed that works on both GNU (Linux) and BSD (macOS) sed
sed_inplace() {
if sed --version >/dev/null 2>&1; then
sed -i "$@"
else
sed -i '' "$@"
fi
}
prepare_release() {
if [ "$DRY_RUN" = true ]; then
echo "[dry-run] Preparing release (no changes will be made)"
else
echo "Starting release preparation..."
fi
echo ""
check_git_status
check_master_branch
check_master_up_to_date
# Extract current version from CMakeLists.txt
echo "Step 1: Reading current version..."
MAJOR=$(grep "set(LLAMA_VERSION_MAJOR" CMakeLists.txt | sed 's/.*MAJOR \([0-9]*\).*/\1/')
MINOR=$(grep "set(LLAMA_VERSION_MINOR" CMakeLists.txt | sed 's/.*MINOR \([0-9]*\).*/\1/')
PATCH=$(grep "set(LLAMA_VERSION_PATCH" CMakeLists.txt | sed 's/.*PATCH \([0-9]*\).*/\1/')
echo "Current version: $MAJOR.$MINOR.$PATCH"
# Calculate new version
case $VERSION_TYPE in
major)
NEW_MAJOR=$((MAJOR + 1))
NEW_MINOR=0
NEW_PATCH=0
;;
minor)
NEW_MAJOR=$MAJOR
NEW_MINOR=$((MINOR + 1))
NEW_PATCH=0
;;
patch)
NEW_MAJOR=$MAJOR
NEW_MINOR=$MINOR
NEW_PATCH=$((PATCH + 1))
;;
esac
NEW_VERSION="$NEW_MAJOR.$NEW_MINOR.$NEW_PATCH"
RC_BRANCH="llama-rc-v$NEW_VERSION"
echo "New release version: $NEW_VERSION"
echo "Release candidate branch: $RC_BRANCH"
echo ""
# Create release candidate branch
echo "Step 2: Creating release candidate branch..."
if [ "$DRY_RUN" = true ]; then
echo " [dry-run] Would create branch: $RC_BRANCH"
else
git checkout -b "$RC_BRANCH"
echo "✓ Created and switched to branch: $RC_BRANCH"
fi
echo ""
# Update CMakeLists.txt for release
echo "Step 3: Updating version in CMakeLists.txt..."
if [ "$DRY_RUN" = true ]; then
echo " [dry-run] Would update LLAMA_VERSION_MAJOR to $NEW_MAJOR"
echo " [dry-run] Would update LLAMA_VERSION_MINOR to $NEW_MINOR"
echo " [dry-run] Would update LLAMA_VERSION_PATCH to $NEW_PATCH"
else
sed_inplace -e "s/set(LLAMA_VERSION_MAJOR [0-9]*)/set(LLAMA_VERSION_MAJOR $NEW_MAJOR)/" CMakeLists.txt
sed_inplace -e "s/set(LLAMA_VERSION_MINOR [0-9]*)/set(LLAMA_VERSION_MINOR $NEW_MINOR)/" CMakeLists.txt
sed_inplace -e "s/set(LLAMA_VERSION_PATCH [0-9]*)/set(LLAMA_VERSION_PATCH $NEW_PATCH)/" CMakeLists.txt
fi
echo ""
# Commit version bump
echo "Step 4: Committing version bump..."
if [ "$DRY_RUN" = true ]; then
echo " [dry-run] Would commit: 'llama.cpp : bump version to $NEW_VERSION'"
else
git add CMakeLists.txt
git commit -m "llama.cpp : bump version to $NEW_VERSION"
fi
echo ""
echo ""
if [ "$DRY_RUN" = true ]; then
echo "[dry-run] Summary (no changes were made):"
echo " • Would have created branch: $RC_BRANCH"
echo " • Would have updated version to: $NEW_VERSION"
else
echo "Release preparation completed!"
echo "Summary:"
echo " • Created branch: $RC_BRANCH"
echo " • Updated version to: $NEW_VERSION"
echo ""
echo "Next steps:"
echo " • Push branch to remote: git push origin $RC_BRANCH"
echo " • Create a Pull Request from $RC_BRANCH to master"
echo " • After the PR is merged and the build-cpu workflow has passed,"
echo " create the release with the make-release workflow"
echo " (.github/workflows/make-release.yml)"
fi
}
prepare_release
+1 -1
View File
@@ -1 +1 @@
8c63e70982c95ceb862e3a1073a2c1beef75d60a
8599e0ea3756c4bac4ef813af2241cb1a8bbfb0b
-2
View File
@@ -1062,8 +1062,6 @@ bool llm_arch_supports_sm_tensor(const llm_arch & arch) {
case LLM_ARCH_NEMOTRON_H:
case LLM_ARCH_NEMOTRON_H_MOE:
case LLM_ARCH_GRANITE_HYBRID:
case LLM_ARCH_LFM2:
case LLM_ARCH_LFM2MOE:
case LLM_ARCH_MINIMAX_01:
case LLM_ARCH_MINIMAX_M2:
case LLM_ARCH_MINIMAX_M3:
+4
View File
@@ -487,6 +487,10 @@ struct ggml_backend_meta_split_state llama_meta_device_get_split_state(const str
return get_tensor_config_impl(GGML_BACKEND_SPLIT_AXIS_1, "ssm_out.weight");
}
if (std::regex_match(tensor_name, pattern_r_cache) || std::regex_match(tensor_name, pattern_s_cache)) {
if (ud->model->arch == LLM_ARCH_LFM2 || ud->model->arch == LLM_ARCH_LFM2MOE) {
// the LFM2 shortconv block runs fully mirrored, so its conv state must be mirrored too
return get_tensor_config_impl(GGML_BACKEND_SPLIT_AXIS_MIRRORED, "");
}
return get_tensor_config_impl(GGML_BACKEND_SPLIT_AXIS_0, "ssm_out.weight");
}
if (std::regex_match(tensor_name, pattern_ssm_conv1d)) {
+30 -14
View File
@@ -7085,9 +7085,10 @@ struct test_flash_attn_ext : public test_case {
const ggml_type type_V;
std::array<int32_t, 4> permute;
const bool kv_view; // create K/V as views of a larger buffer (like a KV cache)
const bool v_is_view_of_k;
std::string vars() override {
return VARS_TO_STR15(hsk, hsv, nh, nr23, kv, nb, mask, sinks, max_bias, logit_softcap, prec, type_K, type_V, permute, kv_view);
return VARS_TO_STR16(hsk, hsv, nh, nr23, kv, nb, mask, sinks, max_bias, logit_softcap, prec, type_K, type_V, permute, kv_view, v_is_view_of_k);
}
double max_nmse_err() override {
@@ -7104,9 +7105,9 @@ struct test_flash_attn_ext : public test_case {
test_flash_attn_ext(int64_t hsk = 128, int64_t hsv = 128, int64_t nh = 32, std::array<int64_t, 2> nr23 = {1, 1}, int64_t kv = 96, int64_t nb = 8,
bool mask = true, bool sinks = false, float max_bias = 0.0f, float logit_softcap = 0.0f, ggml_prec prec = GGML_PREC_F32,
ggml_type type_K = GGML_TYPE_F16, ggml_type type_V = GGML_TYPE_F16, std::array<int32_t, 4> permute = {0, 1, 2, 3},
bool kv_view = true)
bool kv_view = true, bool v_is_view_of_k = false)
: hsk(hsk), hsv(hsv), nh(nh), nr23(nr23), kv(kv), nb(nb), mask(mask), sinks(sinks), max_bias(max_bias), logit_softcap(logit_softcap), prec(prec),
type_K(type_K), type_V(type_V), permute(permute), kv_view(kv_view) {}
type_K(type_K), type_V(type_V), permute(permute), kv_view(kv_view), v_is_view_of_k(v_is_view_of_k) {}
ggml_tensor * build_graph(ggml_context * ctx) override {
const int64_t hsk_padded = GGML_PAD(hsk, ggml_blck_size(type_K));
@@ -7138,14 +7139,14 @@ struct test_flash_attn_ext : public test_case {
ggml_set_name(k, "k");
ggml_tensor * v = nullptr;
if (type_K == type_V && hsk_padded == 576 && hsv_padded == 512) {
// TODO: this branch should become a separate test case parameter instead of hardcoding this for these head shapes
// in this branch, the V cache is sub-view of the K cache. this is used by some MLA-based models
if (v_is_view_of_k) {
// the V cache is a sub-view of the K cache. this is used by some MLA-based models
// for more info:
// - https://github.com/ggml-org/llama.cpp/pull/13435
// - https://github.com/ggml-org/llama.cpp/pull/18953#issuecomment-3774948392
// - https://github.com/ggml-org/llama.cpp/pull/18986
GGML_ASSERT(type_K == type_V && hsv_padded <= hsk_padded);
v = ggml_view_4d(ctx, k, hsv_padded, kv, nh, nr23[1], k->nb[1], k->nb[2], k->nb[3], 0);
} else {
v = create_permuted(type_V, hsv_padded, kv, nh, nr23[1], kv_view); // the V tensor is usually a view of the V cache
@@ -9298,6 +9299,14 @@ static std::vector<std::unique_ptr<test_case>> make_test_cases_eval() {
test_cases.emplace_back(new test_mul_mat(GGML_TYPE_Q8_0, GGML_TYPE_F32, 6, 4096, 5120, {1, 1}, {1, 1}));
// K not a multiple of 32
test_cases.emplace_back(new test_mul_mat(GGML_TYPE_F16, GGML_TYPE_F16, 64, 32, 65, {1, 1}, {1, 1}));
test_cases.emplace_back(new test_mul_mat(GGML_TYPE_F16, GGML_TYPE_F16, 64, 32, 80, {1, 1}, {1, 1}));
test_cases.emplace_back(new test_mul_mat(GGML_TYPE_F16, GGML_TYPE_F32, 64, 32, 80, {1, 1}, {1, 1}));
test_cases.emplace_back(new test_mul_mat(GGML_TYPE_F32, GGML_TYPE_F32, 64, 32, 80, {1, 1}, {1, 1}));
test_cases.emplace_back(new test_mul_mat(GGML_TYPE_F16, GGML_TYPE_F16, 64, 32, 588, {1, 1}, {1, 1})); // 14*14*3, e.g. conv_2d im2col
test_cases.emplace_back(new test_mul_mat(GGML_TYPE_F16, GGML_TYPE_F16, 64, 32, 80, {4, 1}, {1, 1}));
#if 0
// test the mat-mat path for Metal
for (int k = 1; k < 512; ++k) {
@@ -9898,12 +9907,14 @@ static std::vector<std::unique_ptr<test_case>> make_test_cases_eval() {
if (hsk != 128 && prec == GGML_PREC_DEFAULT) continue;
for (ggml_type type_KV : {GGML_TYPE_F32, GGML_TYPE_F16, GGML_TYPE_BF16, GGML_TYPE_Q8_0, GGML_TYPE_Q5_1, GGML_TYPE_Q5_0, GGML_TYPE_Q4_1, GGML_TYPE_Q4_0, GGML_TYPE_IQ4_NL}) {
if (type_KV != GGML_TYPE_F16 && hsk != 64 && hsk != 72) continue;
// DeepSeek MLA: the V cache is a sub-view of the K cache
const bool v_is_view_of_k = hsk == 576;
test_cases.emplace_back(new test_flash_attn_ext(
hsk, hsv, nh, {nr2, nr3}, kv, nb, mask, sinks, max_bias, logit_softcap, prec, type_KV, type_KV));
hsk, hsv, nh, {nr2, nr3}, kv, nb, mask, sinks, max_bias, logit_softcap, prec, type_KV, type_KV, {0, 1, 2, 3}, true, v_is_view_of_k));
// run fewer test cases permuted
if (mask == true && max_bias == 0.0f && logit_softcap == 0 && kv == 512) {
test_cases.emplace_back(new test_flash_attn_ext(
hsk, hsv, nh, {nr2, nr3}, kv, nb, mask, sinks, max_bias, logit_softcap, prec, type_KV, type_KV, {0, 2, 1, 3}));
hsk, hsv, nh, {nr2, nr3}, kv, nb, mask, sinks, max_bias, logit_softcap, prec, type_KV, type_KV, {0, 2, 1, 3}, true, v_is_view_of_k));
}
}
}
@@ -9942,11 +9953,16 @@ static std::vector<std::unique_ptr<test_case>> make_test_cases_eval() {
test_cases.emplace_back(new test_flash_attn_ext(256, 256, 2, {16, 1}, 1025, 64, true, false, 0, 0, GGML_PREC_F32, GGML_TYPE_Q8_0, GGML_TYPE_Q8_0, {0, 2, 1, 3}));
test_cases.emplace_back(new test_flash_attn_ext(256, 256, 2, {16, 1}, 16384, 1, true, false, 0, 0, GGML_PREC_F32, GGML_TYPE_Q8_0, GGML_TYPE_Q8_0));
// MLA shape (V is a view of K) with quantized KV
// (the test harness builds V as a view of K for this shape; see build_graph)
test_cases.emplace_back(new test_flash_attn_ext(576, 512, 1, {20, 1}, 113, 1, true, false, 0, 0, GGML_PREC_F32, GGML_TYPE_Q8_0, GGML_TYPE_Q8_0));
test_cases.emplace_back(new test_flash_attn_ext(576, 512, 1, {20, 1}, 1024, 1, true, false, 0, 0, GGML_PREC_F32, GGML_TYPE_Q8_0, GGML_TYPE_Q8_0));
test_cases.emplace_back(new test_flash_attn_ext(576, 512, 1, {20, 1}, 1024, 64, true, false, 0, 0, GGML_PREC_F32, GGML_TYPE_Q8_0, GGML_TYPE_Q8_0));
// MLA shape: the V cache is a sub-view of the K cache, with quantized KV
test_cases.emplace_back(new test_flash_attn_ext(576, 512, 1, {20, 1}, 113, 1, true, false, 0, 0, GGML_PREC_F32, GGML_TYPE_Q8_0, GGML_TYPE_Q8_0, {0, 1, 2, 3}, true, true));
test_cases.emplace_back(new test_flash_attn_ext(576, 512, 1, {20, 1}, 1024, 1, true, false, 0, 0, GGML_PREC_F32, GGML_TYPE_Q8_0, GGML_TYPE_Q8_0, {0, 1, 2, 3}, true, true));
test_cases.emplace_back(new test_flash_attn_ext(576, 512, 1, {20, 1}, 1024, 64, true, false, 0, 0, GGML_PREC_F32, GGML_TYPE_Q8_0, GGML_TYPE_Q8_0, {0, 1, 2, 3}, true, true));
// more V-is-sub-view-of-K cases: other head shapes, and full views with equal head sizes
test_cases.emplace_back(new test_flash_attn_ext(320, 256, 1, {32, 1}, 512, 1, true, false, 0, 0, GGML_PREC_F32, GGML_TYPE_F16, GGML_TYPE_F16, {0, 1, 2, 3}, true, true));
test_cases.emplace_back(new test_flash_attn_ext(192, 128, 4, {8, 1}, 512, 1, true, false, 0, 0, GGML_PREC_F32, GGML_TYPE_F16, GGML_TYPE_F16, {0, 1, 2, 3}, true, true));
test_cases.emplace_back(new test_flash_attn_ext(128, 128, 8, {4, 1}, 512, 8, true, false, 0, 0, GGML_PREC_F32, GGML_TYPE_F16, GGML_TYPE_F16, {0, 1, 2, 3}, true, true));
test_cases.emplace_back(new test_flash_attn_ext(64, 64, 4, {1, 1}, 512, 1, true, false, 0, 0, GGML_PREC_F32, GGML_TYPE_Q8_0, GGML_TYPE_Q8_0, {0, 1, 2, 3}, true, true));
// large-KV F16 cases (Qwen3.6-27B geometry and a llama-class control): the upstream matrix
// stops at kv=1024, blind to long-context FA bugs (e.g. the oneDNN SDPA ordering race on BMG).
+106 -60
View File
@@ -239,31 +239,44 @@ Routes → Components → Hooks → Stores → Services → Storage/API
### High-Level Architecture
See: [`docs/architecture/high-level-architecture-simplified.md`](docs/architecture/high-level-architecture-simplified.md)
```mermaid
flowchart TB
subgraph Routes["📍 Routes"]
R1["/ (Welcome)"]
R2["/chat/[id]"]
R3["/mcp-servers"]
R4["/search"]
R5["/settings"]
RL["+layout.svelte"]
end
subgraph Components["🧩 Components"]
C_Sidebar["ChatSidebar"]
C_Screen["ChatScreen"]
C_Form["ChatForm"]
C_Messages["ChatMessages"]
C_ModelsSelector["ModelsSelector"]
C_Sidebar["ChatSidebar"]
C_Models["ModelsSelector"]
C_Settings["ChatSettings"]
C_Mcp["McpServers"]
end
subgraph Hooks["🔌 Hooks"]
H1["use-chat-screen-active-model"]
H2["use-processing-state"]
H3["use-context-gauge"]
H4["use-models-selector"]
H5["use-tools-panel"]
end
subgraph Stores["🗄️ Stores"]
S1["chatStore"]
S2["conversationsStore"]
S3["modelsStore"]
S4["serverStore"]
S5["settingsStore"]
S4["mcpStore"]
S5["agenticStore"]
S6["serverStore"]
S7["settingsStore"]
S8["toolsStore"]
end
subgraph Services["⚙️ Services"]
@@ -271,6 +284,9 @@ flowchart TB
SV2["ModelsService"]
SV3["PropsService"]
SV4["DatabaseService"]
SV5["MCPService"]
SV6["ToolsService"]
SV7["SandboxService"]
end
subgraph Storage["💾 Storage"]
@@ -282,19 +298,28 @@ flowchart TB
API1["/v1/chat/completions"]
API2["/props"]
API3["/models/*"]
API4["/tools"]
end
R1 & R2 --> C_Screen
RL --> C_Sidebar
C_Screen --> C_Form & C_Messages & C_Settings
C_Screen --> S1 & S2
C_ModelsSelector --> S3 & S4
C_Screen --> H1 & H2 & H3
C_Models --> H4
C_Mcp --> S4
C_Screen --> S1 & S2 & S3
C_Models --> S3
H1 --> S3
S1 --> SV1 & SV4
S2 --> SV4
S3 --> SV2 & SV3
S4 --> SV5
S5 --> SV1 & SV5 & SV6 & SV7
SV4 --> ST1
SV1 --> API1
SV2 --> API3
SV3 --> API2
SV6 --> API4
```
### Layer Breakdown
@@ -303,6 +328,9 @@ flowchart TB
- **`/`** - Welcome screen, creates new conversation
- **`/chat/[id]`** - Active chat interface
- **`/mcp-servers`** - MCP server management
- **`/search`** - Conversation search
- **`/settings`** - Settings (optional `[[section]]`)
- **`+layout.svelte`** - Sidebar, navigation, global initialization
#### Components (`src/lib/components/`)
@@ -348,28 +376,68 @@ Components are organized in `app/` (application-specific) and `ui/` (shadcn-svel
#### Hooks (`src/lib/hooks/`)
- **`useModelChangeValidation`** - Validates model switch against conversation modalities
- **`useProcessingState`** - Tracks streaming progress and token generation
Hooks are the thin view-layer between components and stores: they own UI concerns (scroll, drag-and-drop, keyboard shortcuts, pickers, selection) and translate store state into view state.
| Hook | Responsibility |
| ------------------------------- | -------------------------------------------------------------- |
| `use-chat-screen-active-model` | Active model resolution + modality capability detection |
| `use-processing-state` | View over `chatStore.processing` for streaming progress/tokens |
| `use-context-gauge` | View over `contextStatsStore` for the context usage gauge |
| `use-models-selector` | Model selector dropdown state (loaded/available groups) |
| `use-tools-panel` | Tools panel state |
| `use-reasoning-menu` | Reasoning-effort menu state |
| `use-attachment-menu` | Attachment menu + modality flags |
| `use-draft-messages` | Per-chat draft message/files persistence |
| `use-chat-form-pickers` | Chat form pickers (commands, mentions) |
| `use-debounced-search` | Shared debounced async search for pickers |
| `use-picker-navigation` | Picker keyboard navigation |
| `use-chat-message-edit-context` | Message edit context (content + extras) |
| `use-chat-screen-drag-and-drop` | Drag-and-drop state machine |
| `use-chat-screen-file-upload` | File upload queue + capability validation |
| `use-chat-screen-scroll` | Scroll container binding + navigation guard |
| `use-auto-scroll` | Auto-scroll controller for streaming |
| `use-marquee-selection` | Shift+click / marquee range selection |
| `use-keyboard-shortcuts` | Global keyboard shortcuts |
| `use-settings-navigation` | Settings section navigation |
| `use-pwa` | PWA install/update + version mismatch detection |
#### Stores (`src/lib/stores/`)
| Store | Responsibility |
| -------------------- | --------------------------------------------------------- |
| `chatStore` | Message sending, streaming, abort control, error handling |
| `conversationsStore` | CRUD for conversations, message branching, navigation |
| `modelsStore` | Model list, selection, loading/unloading (ROUTER) |
| `serverStore` | Server properties, role detection, modalities |
| `settingsStore` | User preferences, parameter sync with server defaults |
Stores own reactive application state as Svelte 5 runes. Larger stores are split into directories and compose focused sub-stores behind a narrow host interface (see Architectural Patterns).
| Store | Responsibility |
| -------------------- | --------------------------------------------------------------------------------------------------------------- |
| `chatStore` | Chat lifecycle, streaming, abort control, error handling; composes `processing`, `activity`, `streams`, `flows` |
| `conversationsStore` | Conversation CRUD, message branching, navigation, import/export; composes `preferences` |
| `modelsStore` | Model list, selection, loading/unloading (ROUTER); composes `props`, `status` |
| `mcpStore` | MCP host role: multi-server lifecycle, tool routing; composes `health`, `resources` |
| `agenticStore` | Multi-turn agentic loop orchestration, tool execution; composes `gates` |
| `serverStore` | Server connection state, `/props`, role detection, modalities |
| `settingsStore` | User preferences, theme, parameter sync with server defaults |
| `toolsStore` | Tool registry: server + MCP tools, enabled set for the LLM |
| `permissionsStore` | Persisted tool permission grants |
| `contextStatsStore` | Context window usage for the active conversation |
| `draftMessagesStore` | Per-chat draft message/files |
| `deviceStore` | Browser environment signals (mobile, OS, theme) |
| `versionStore` | Build version information |
#### Services (`src/lib/services/`)
| Service | Responsibility |
| ---------------------- | ----------------------------------------------- |
| `ChatService` | API calls to`/v1/chat/completions`, SSE parsing |
| `ModelsService` | `/models`, `/models/load`, `/models/unload` |
| `PropsService` | `/props`, `/props?model=` |
| `DatabaseService` | IndexedDB operations via Dexie |
| `ParameterSyncService` | Syncs settings with server defaults |
Services are a stateless protocol layer: static methods, pure I/O, no reactive state. Stores consume them for all API and storage access.
| Service | Responsibility |
| ----------------------------- | ------------------------------------------------------------------------- |
| `ChatService` | `/v1/chat/completions` streaming + SSE parsing, message format conversion |
| `ModelsService` | `/models`, `/models/load`, `/models/unload` |
| `PropsService` | `/props`, `/props?model=` |
| `DatabaseService` | IndexedDB operations via Dexie |
| `MCPService` | MCP protocol: transports, connect, list/execute tools, prompts, resources |
| `ToolsService` | Server tool list/execute/stream (`/tools`) |
| `SandboxService` | Browser JS execution in a sandboxed worker |
| `ParameterSyncService` | Syncs settings with server defaults |
| `ConversationTransferService` | Conversation import/export JSONL + ZIP format |
| `MigrationService` | Non-destructive localStorage/IndexedDB migrations |
| `RouterService` | Dynamic route URL construction |
---
@@ -377,8 +445,6 @@ Components are organized in `app/` (application-specific) and `ui/` (shadcn-svel
### MODEL Mode (Single Model)
See: [`docs/flows/data-flow-simplified-model-mode.md`](docs/flows/data-flow-simplified-model-mode.md)
```mermaid
sequenceDiagram
participant User
@@ -388,8 +454,9 @@ sequenceDiagram
participant API as llama-server
Note over User,API: Initialization
UI->>Stores: initialize()
Stores->>DB: load conversations
UI->>Stores: initStores() (awaited by route loads)
Stores->>Stores: run migrations
Stores->>DB: load conversations (background)
Stores->>API: GET /props
API-->>Stores: server config
Stores->>API: GET /v1/models
@@ -408,8 +475,6 @@ sequenceDiagram
### ROUTER Mode (Multi-Model)
See: [`docs/flows/data-flow-simplified-router-mode.md`](docs/flows/data-flow-simplified-router-mode.md)
```mermaid
sequenceDiagram
participant User
@@ -441,17 +506,6 @@ sequenceDiagram
end
```
### Detailed Flow Diagrams
| Flow | Description | File |
| ------------- | ------------------------------------------ | ----------------------------------------------------------- |
| Chat | Message lifecycle, streaming, regeneration | [`chat-flow.md`](docs/flows/chat-flow.md) |
| Models | Loading, unloading, modality caching | [`models-flow.md`](docs/flows/models-flow.md) |
| Server | Props fetching, role detection | [`server-flow.md`](docs/flows/server-flow.md) |
| Conversations | CRUD, branching, import/export | [`conversations-flow.md`](docs/flows/conversations-flow.md) |
| Database | IndexedDB schema, operations | [`database-flow.md`](docs/flows/database-flow.md) |
| Settings | Parameter sync, user overrides | [`settings-flow.md`](docs/flows/settings-flow.md) |
---
## Architectural Patterns
@@ -505,13 +559,14 @@ Components dispatch actions to stores, stores coordinate with services for I/O,
### 3. Per-Conversation State
Enables concurrent streaming across multiple conversations:
Enables concurrent streaming across multiple conversations. Loading is tracked
per conversation by the activity ledger (`chatStore.activity`), while streaming
state and abort controllers live in per-conversation maps:
```typescript
class ChatStore {
chatLoadingStates = new Map<string, boolean>();
chatStreamingStates = new Map<string, { response: string; messageId: string }>();
abortControllers = new Map<string, AbortController>();
chatStreamingStates = new SvelteMap<string, { response: string; messageId: string }>();
abortControllers = new SvelteMap<string, AbortController>();
}
```
@@ -567,20 +622,14 @@ get isRouterMode() {
### 7. Modality Validation
Prevents sending attachments to incompatible models:
Prevents sending attachments to incompatible models. The
`use-chat-screen-active-model` hook derives the active model's capabilities
from `modelsStore.props`:
```typescript
// useModelChangeValidation hook
const validate = (modelId: string) => {
const modelModalities = modelsStore.getModelModalities(modelId);
const conversationModalities = conversationsStore.usedModalities;
// Check if model supports all used modalities
if (conversationModalities.hasImages && !modelModalities.vision) {
return { valid: false, reason: 'Model does not support images' };
}
// ...
};
// use-chat-screen-active-model hook
const hasVisionModality = $derived.by(() => modelsStore.props.modelSupportsVision(activeModelId));
const hasAudioModality = $derived.by(() => modelsStore.props.modelSupportsAudio(activeModelId));
```
### 8. Persistent Storage Strategy
@@ -673,9 +722,6 @@ tools/ui/
│ └── styles/ # Global styles
├── static/ # Static assets
├── tests/ # Test files
├── docs/ # Architecture diagrams
│ ├── architecture/ # High-level architecture
│ └── flows/ # Feature-specific flows
└── .storybook/ # Storybook configuration
```
@@ -1,145 +0,0 @@
```mermaid
flowchart TB
subgraph Routes["📍 Routes"]
R1["/ (Welcome)"]
R2["/chat/[id]"]
RL["+layout.svelte"]
end
subgraph Components["🧩 Components"]
C_Sidebar["ChatSidebar"]
C_Screen["ChatScreen"]
C_Form["ChatForm"]
C_Messages["ChatMessages"]
C_Message["ChatMessage"]
C_ChatMessageAgenticContent["ChatMessageAgenticContent"]
C_MessageEditForm["ChatMessageEditForm"]
C_ModelsSelector["ModelsSelector"]
C_Settings["ChatSettings"]
C_McpSettings["McpServersSettings"]
C_McpResourceBrowser["McpResourceBrowser"]
C_McpServersSelector["McpServersSelector"]
end
subgraph Hooks["🪝 Hooks"]
H1["useModelChangeValidation"]
H2["useProcessingState"]
end
subgraph Stores["🗄️ Stores"]
S1["chatStore<br/><i>Chat interactions & streaming</i>"]
SA["agenticStore<br/><i>Multi-turn agentic loop orchestration</i>"]
S2["conversationsStore<br/><i>Conversation data, messages & MCP overrides</i>"]
S3["modelsStore<br/><i>Model selection & loading</i>"]
S4["serverStore<br/><i>Server props & role detection</i>"]
S5["settingsStore<br/><i>User configuration incl. MCP</i>"]
S6["mcpStore<br/><i>MCP servers, tools, prompts</i>"]
S7["mcpResourceStore<br/><i>MCP resources & attachments</i>"]
end
subgraph Services["⚙️ Services"]
SV1["ChatService"]
SV2["ModelsService"]
SV3["PropsService"]
SV4["DatabaseService"]
SV5["ParameterSyncService"]
SV6["MCPService<br/><i>protocol operations</i>"]
end
subgraph Storage["💾 Storage"]
ST1["IndexedDB<br/><i>conversations, messages</i>"]
ST2["LocalStorage<br/><i>config, userOverrides, mcpServers</i>"]
end
subgraph APIs["🌐 llama-server API"]
API1["/v1/chat/completions"]
API2["/props"]
API3["/models/*"]
API4["/v1/models"]
end
subgraph ExternalMCP["🔌 External MCP Servers"]
EXT1["MCP Server 1<br/><i>WebSocket/HTTP/SSE</i>"]
EXT2["MCP Server N"]
end
%% Routes → Components
R1 & R2 --> C_Screen
RL --> C_Sidebar
%% Layout runs MCP health checks
RL --> S6
%% Component hierarchy
C_Screen --> C_Form & C_Messages & C_Settings
C_Messages --> C_Message
C_Message --> C_ChatMessageAgenticContent
C_Message --> C_MessageEditForm
C_Form & C_MessageEditForm --> C_ModelsSelector
C_Form --> C_McpServersSelector
C_Settings --> C_McpSettings
C_McpSettings --> C_McpResourceBrowser
%% Components → Hooks → Stores
C_Form & C_Messages --> H1 & H2
H1 --> S3 & S4
H2 --> S1 & S5
%% Components → Stores
C_Screen --> S1 & S2
C_Sidebar --> S2
C_ModelsSelector --> S3 & S4
C_Settings --> S5
C_McpSettings --> S6
C_McpResourceBrowser --> S6 & S7
C_McpServersSelector --> S6
C_Form --> S6
%% chatStore → agenticStore → mcpStore (agentic loop)
S1 --> SA
SA --> SV1
SA --> S6
%% Stores → Services
S1 --> SV1 & SV4
S2 --> SV4
S3 --> SV2 & SV3
S4 --> SV3
S5 --> SV5
S6 --> SV6
S7 --> SV6
%% Services → Storage
SV4 --> ST1
SV5 --> ST2
%% Services → APIs
SV1 --> API1
SV2 --> API3 & API4
SV3 --> API2
%% MCP → External Servers
SV6 --> EXT1 & EXT2
%% Styling
classDef routeStyle fill:#e1f5fe,stroke:#01579b,stroke-width:2px
classDef componentStyle fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
classDef hookStyle fill:#fff8e1,stroke:#ff8f00,stroke-width:2px
classDef storeStyle fill:#fff3e0,stroke:#e65100,stroke-width:2px
classDef serviceStyle fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
classDef storageStyle fill:#fce4ec,stroke:#c2185b,stroke-width:2px
classDef apiStyle fill:#e3f2fd,stroke:#1565c0,stroke-width:2px
classDef mcpStyle fill:#e0f2f1,stroke:#00695c,stroke-width:2px
classDef agenticStyle fill:#e8eaf6,stroke:#283593,stroke-width:2px
classDef externalStyle fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px,stroke-dasharray: 5 5
class R1,R2,RL routeStyle
class C_Sidebar,C_Screen,C_Form,C_Messages,C_Message,C_ChatMessageAgenticContent,C_MessageEditForm,C_ModelsSelector,C_Settings componentStyle
class C_McpSettings,C_McpResourceBrowser,C_McpServersSelector componentStyle
class H1,H2 hookStyle
class S1,S2,S3,S4,S5,SA,S6,S7 storeStyle
class SV1,SV2,SV3,SV4,SV5,SV6 serviceStyle
class ST1,ST2 storageStyle
class API1,API2,API3,API4 apiStyle
class EXT1,EXT2 externalStyle
```
@@ -1,373 +0,0 @@
```mermaid
flowchart TB
subgraph Routes["📍 Routes"]
R1["/ (+page.svelte)"]
R2["/chat/[id]"]
RL["+layout.svelte"]
end
subgraph Components["🧩 Components"]
direction TB
subgraph LayoutComponents["Layout"]
C_Sidebar["ChatSidebar"]
C_Screen["ChatScreen"]
end
subgraph ChatUIComponents["Chat UI"]
C_Form["ChatForm"]
C_Messages["ChatMessages"]
C_Message["ChatMessage"]
C_MessageUser["ChatMessageUser"]
C_MessageEditForm["ChatMessageEditForm"]
C_Attach["ChatAttachments"]
C_ModelsSelector["ModelsSelector"]
C_Settings["ChatSettings"]
end
subgraph MCPComponents["MCP UI"]
C_McpSettings["McpServersSettings"]
C_McpServerCard["McpServerCard"]
C_McpResourceBrowser["McpResourceBrowser"]
C_McpResourcePreview["McpResourcePreview"]
C_McpServersSelector["McpServersSelector"]
end
end
subgraph Hooks["🪝 Hooks"]
H1["useModelChangeValidation"]
H2["useProcessingState"]
H3["isMobile"]
end
subgraph Stores["🗄️ Stores"]
direction TB
subgraph S1["chatStore"]
S1State["<b>State:</b><br/>isLoading, currentResponse<br/>errorDialogState<br/>activeProcessingState<br/>chatLoadingStates<br/>chatStreamingStates<br/>abortControllers<br/>processingStates<br/>activeConversationId<br/>isStreamingActive"]
S1LoadState["<b>Loading State:</b><br/>setChatLoading()<br/>isChatLoading()<br/>syncLoadingStateForChat()<br/>clearUIState()<br/>isChatLoadingPublic()<br/>getAllLoadingChats()<br/>getAllStreamingChats()"]
S1ProcState["<b>Processing State:</b><br/>setActiveProcessingConversation()<br/>getProcessingState()<br/>clearProcessingState()<br/>getActiveProcessingState()<br/>updateProcessingStateFromTimings()<br/>getCurrentProcessingStateSync()<br/>restoreProcessingStateFromMessages()"]
S1Stream["<b>Streaming:</b><br/>streamChatCompletion()<br/>startStreaming()<br/>stopStreaming()<br/>stopGeneration()<br/>isStreaming()"]
S1Error["<b>Error Handling:</b><br/>showErrorDialog()<br/>dismissErrorDialog()<br/>isAbortError()"]
S1Msg["<b>Message Operations:</b><br/>addMessage()<br/>sendMessage()<br/>updateMessage()<br/>deleteMessage()<br/>getDeletionInfo()"]
S1Regen["<b>Regeneration:</b><br/>regenerateMessage()<br/>regenerateMessageWithBranching()<br/>continueAssistantMessage()"]
S1Edit["<b>Editing:</b><br/>editAssistantMessage()<br/>editUserMessagePreserveResponses()<br/>editMessageWithBranching()<br/>clearEditMode()<br/>isEditModeActive()<br/>getAddFilesHandler()<br/>setEditModeActive()"]
S1Utils["<b>Utilities:</b><br/>getApiOptions()<br/>parseTimingData()<br/>getOrCreateAbortController()<br/>getConversationModel()"]
end
subgraph SA["agenticStore"]
SAState["<b>State:</b><br/>sessions (Map)<br/>isAnyRunning"]
SASession["<b>Session Management:</b><br/>getSession()<br/>updateSession()<br/>clearSession()<br/>getActiveSessions()<br/>isRunning()<br/>currentTurn()<br/>totalToolCalls()<br/>lastError()<br/>streamingToolCall()"]
SAConfig["<b>Configuration:</b><br/>getConfig()<br/>maxTurns, maxToolPreviewLines"]
SAFlow["<b>Agentic Loop:</b><br/>runAgenticFlow()<br/>executeAgenticLoop()<br/>normalizeToolCalls()<br/>emitToolCallResult()<br/>extractBase64Attachments()"]
end
subgraph S2["conversationsStore"]
S2State["<b>State:</b><br/>conversations<br/>activeConversation<br/>activeMessages<br/>isInitialized<br/>pendingMcpServerOverrides<br/>titleUpdateConfirmationCallback"]
S2Lifecycle["<b>Lifecycle:</b><br/>initialize()<br/>loadConversations()<br/>clearActiveConversation()"]
S2ConvCRUD["<b>Conversation CRUD:</b><br/>createConversation()<br/>loadConversation()<br/>deleteConversation()<br/>deleteAll()<br/>updateConversationName()<br/>updateConversationTitleWithConfirmation()"]
S2MsgMgmt["<b>Message Management:</b><br/>refreshActiveMessages()<br/>addMessageToActive()<br/>updateMessageAtIndex()<br/>findMessageIndex()<br/>sliceActiveMessages()<br/>removeMessageAtIndex()<br/>getConversationMessages()"]
S2Nav["<b>Navigation:</b><br/>navigateToSibling()<br/>updateCurrentNode()<br/>updateConversationTimestamp()"]
S2McpOverrides["<b>MCP Per-Chat Overrides:</b><br/>getMcpServerOverride()<br/>getAllMcpServerOverrides()<br/>setMcpServerOverride()<br/>toggleMcpServerForChat()<br/>removeMcpServerOverride()<br/>isMcpServerEnabledForChat()<br/>clearPendingMcpServerOverrides()"]
S2Export["<b>Import/Export:</b><br/>downloadConversation()<br/>exportAllConversations()<br/>importConversations()<br/>importConversationsData()<br/>triggerDownload()"]
S2Utils["<b>Utilities:</b><br/>setTitleUpdateConfirmationCallback()"]
end
subgraph S3["modelsStore"]
S3State["<b>State:</b><br/>models, routerModels<br/>selectedModelId<br/>selectedModelName<br/>loading, updating, error<br/>modelLoadingStates<br/>modelPropsCache<br/>modelPropsFetching<br/>propsCacheVersion"]
S3Getters["<b>Computed Getters:</b><br/>selectedModel<br/>loadedModelIds<br/>loadingModelIds<br/>singleModelName"]
S3Modal["<b>Modalities:</b><br/>getModelModalities()<br/>modelSupportsVision()<br/>modelSupportsAudio()<br/>getModelModalitiesArray()<br/>getModelProps()<br/>updateModelModalities()"]
S3Status["<b>Status Queries:</b><br/>isModelLoaded()<br/>isModelOperationInProgress()<br/>getModelStatus()<br/>isModelPropsFetching()"]
S3Fetch["<b>Data Fetching:</b><br/>fetch()<br/>fetchRouterModels()<br/>fetchModelProps()<br/>fetchModalitiesForLoadedModels()"]
S3Select["<b>Model Selection:</b><br/>selectModelById()<br/>selectModelByName()<br/>clearSelection()<br/>findModelByName()<br/>findModelById()<br/>hasModel()"]
S3LoadUnload["<b>Loading/Unloading Models:</b><br/>loadModel()<br/>unloadModel()<br/>ensureModelLoaded()<br/>waitForModelStatus()<br/>pollForModelStatus()"]
S3Utils["<b>Utilities:</b><br/>toDisplayName()<br/>clear()"]
end
subgraph S4["serverStore"]
S4State["<b>State:</b><br/>props<br/>loading, error<br/>role<br/>fetchPromise"]
S4Getters["<b>Getters:</b><br/>defaultParams<br/>contextSize<br/>isRouterMode<br/>isModelMode"]
S4Data["<b>Data Handling:</b><br/>fetch()<br/>getErrorMessage()<br/>clear()"]
S4Utils["<b>Utilities:</b><br/>detectRole()"]
end
subgraph S5["settingsStore"]
S5State["<b>State:</b><br/>config<br/>theme<br/>isInitialized<br/>userOverrides"]
S5Lifecycle["<b>Lifecycle:</b><br/>initialize()<br/>loadConfig()<br/>saveConfig()<br/>loadTheme()<br/>saveTheme()"]
S5Update["<b>Config Updates:</b><br/>updateConfig()<br/>updateMultipleConfig()<br/>updateTheme()"]
S5Reset["<b>Reset:</b><br/>resetConfig()<br/>resetTheme()<br/>resetAll()<br/>resetParameterToServerDefault()"]
S5Sync["<b>Server Sync:</b><br/>syncWithServerDefaults()<br/>forceSyncWithServerDefaults()"]
S5Utils["<b>Utilities:</b><br/>getConfig()<br/>getAllConfig()<br/>getParameterInfo()<br/>getParameterDiff()<br/>getServerDefaults()<br/>clearAllUserOverrides()"]
end
subgraph S6["mcpStore"]
S6State["<b>State:</b><br/>isInitializing, error<br/>toolCount, connectedServers<br/>healthChecks (Map)<br/>connections (Map)<br/>toolsIndex (Map)"]
S6Lifecycle["<b>Lifecycle:</b><br/>ensureInitialized()<br/>initialize()<br/>shutdown()<br/>acquireConnection()<br/>releaseConnection()"]
S6Health["<b>Health Checks:</b><br/>runHealthCheck()<br/>runHealthChecksForServers()<br/>updateHealthCheck()<br/>getHealthCheckState()<br/>clearHealthCheck()"]
S6Servers["<b>Server Management:</b><br/>getServers()<br/>addServer()<br/>updateServer()<br/>removeServer()<br/>getServerById()<br/>getServerDisplayName()"]
S6Tools["<b>Tool Operations:</b><br/>getToolDefinitionsForLLM()<br/>getToolNames()<br/>hasTool()<br/>getToolServer()<br/>executeTool()<br/>executeToolByName()"]
S6Prompts["<b>Prompt Operations:</b><br/>getAllPrompts()<br/>getPrompt()<br/>hasPromptsCapability()<br/>getPromptCompletions()"]
end
subgraph S7["mcpResourceStore"]
S7State["<b>State:</b><br/>serverResources (Map)<br/>cachedResources (Map)<br/>subscriptions (Map)<br/>attachments[]<br/>isLoading"]
S7Resources["<b>Resource Discovery:</b><br/>setServerResources()<br/>getServerResources()<br/>getAllResourceInfos()<br/>getAllTemplateInfos()<br/>clearServerResources()"]
S7Cache["<b>Caching:</b><br/>cacheResourceContent()<br/>getCachedContent()<br/>invalidateCache()<br/>clearCache()"]
S7Subs["<b>Subscriptions:</b><br/>addSubscription()<br/>removeSubscription()<br/>isSubscribed()<br/>handleResourceUpdate()"]
S7Attach["<b>Attachments:</b><br/>addAttachment()<br/>updateAttachmentContent()<br/>removeAttachment()<br/>clearAttachments()<br/>toMessageExtras()"]
end
subgraph ReactiveExports["⚡ Reactive Exports"]
direction LR
subgraph ChatExports["chatStore"]
RE1["isLoading()"]
RE2["currentResponse()"]
RE3["errorDialog()"]
RE4["activeProcessingState()"]
RE5["isChatStreaming()"]
RE6["isChatLoading()"]
RE7["getChatStreaming()"]
RE8["getAllLoadingChats()"]
RE9["getAllStreamingChats()"]
RE9a["isEditModeActive()"]
RE9b["getAddFilesHandler()"]
RE9c["setEditModeActive()"]
RE9d["clearEditMode()"]
end
subgraph AgenticExports["agenticStore"]
REA1["agenticIsRunning()"]
REA2["agenticCurrentTurn()"]
REA3["agenticTotalToolCalls()"]
REA4["agenticLastError()"]
REA5["agenticStreamingToolCall()"]
REA6["agenticIsAnyRunning()"]
end
subgraph ConvExports["conversationsStore"]
RE10["conversations()"]
RE11["activeConversation()"]
RE12["activeMessages()"]
RE13["isConversationsInitialized()"]
end
subgraph ModelsExports["modelsStore"]
RE15["modelOptions()"]
RE16["routerModels()"]
RE17["modelsLoading()"]
RE18["modelsUpdating()"]
RE19["modelsError()"]
RE20["selectedModelId()"]
RE21["selectedModelName()"]
RE22["selectedModelOption()"]
RE23["loadedModelIds()"]
RE24["loadingModelIds()"]
RE25["propsCacheVersion()"]
RE26["singleModelName()"]
end
subgraph ServerExports["serverStore"]
RE27["serverProps()"]
RE28["serverLoading()"]
RE29["serverError()"]
RE30["serverRole()"]
RE31["defaultParams()"]
RE32["contextSize()"]
RE33["isRouterMode()"]
RE34["isModelMode()"]
end
subgraph SettingsExports["settingsStore"]
RE35["config()"]
RE36["theme()"]
RE37["isInitialized()"]
end
subgraph MCPExports["mcpStore / mcpResourceStore"]
RE38["mcpResources()"]
RE39["mcpResourceAttachments()"]
RE40["mcpHasResourceAttachments()"]
RE41["mcpTotalResourceCount()"]
RE42["mcpResourcesLoading()"]
end
end
end
subgraph Services["⚙️ Services"]
direction TB
subgraph SV1["ChatService"]
SV1Msg["<b>Messaging:</b><br/>sendMessage()"]
SV1Stream["<b>Streaming:</b><br/>handleStreamResponse()<br/>handleNonStreamResponse()"]
SV1Convert["<b>Conversion:</b><br/>convertDbMessageToApiChatMessageData()<br/>mergeToolCallDeltas()"]
SV1Utils["<b>Utilities:</b><br/>stripReasoningContent()<br/>extractModelName()<br/>parseErrorResponse()"]
end
subgraph SV2["ModelsService"]
SV2List["<b>Listing:</b><br/>list()<br/>listRouter()"]
SV2LoadUnload["<b>Load/Unload:</b><br/>load()<br/>unload()"]
SV2Status["<b>Status:</b><br/>isModelLoaded()<br/>isModelLoading()"]
end
subgraph SV3["PropsService"]
SV3Fetch["<b>Fetching:</b><br/>fetch()<br/>fetchForModel()"]
end
subgraph SV4["DatabaseService"]
SV4Conv["<b>Conversations:</b><br/>createConversation()<br/>getConversation()<br/>getAllConversations()<br/>updateConversation()<br/>deleteConversation()"]
SV4Msg["<b>Messages:</b><br/>createMessageBranch()<br/>createRootMessage()<br/>createSystemMessage()<br/>getConversationMessages()<br/>updateMessage()<br/>deleteMessage()<br/>deleteMessageCascading()"]
SV4Node["<b>Navigation:</b><br/>updateCurrentNode()"]
SV4Import["<b>Import:</b><br/>importConversations()"]
end
subgraph SV5["ParameterSyncService"]
SV5Extract["<b>Extraction:</b><br/>extractServerDefaults()"]
SV5Merge["<b>Merging:</b><br/>mergeWithServerDefaults()"]
SV5Info["<b>Info:</b><br/>getParameterInfo()<br/>canSyncParameter()<br/>getSyncableParameterKeys()<br/>validateServerParameter()"]
SV5Diff["<b>Diff:</b><br/>createParameterDiff()"]
end
subgraph SV6["MCPService"]
SV6Transport["<b>Transport:</b><br/>createTransport()<br/>WebSocket / StreamableHTTP / SSE"]
SV6Conn["<b>Connection:</b><br/>connect()<br/>disconnect()"]
SV6Tools["<b>Tools:</b><br/>listTools()<br/>callTool()"]
SV6Prompts["<b>Prompts:</b><br/>listPrompts()<br/>getPrompt()"]
SV6Resources["<b>Resources:</b><br/>listResources()<br/>listResourceTemplates()<br/>readResource()<br/>subscribeResource()<br/>unsubscribeResource()"]
SV6Complete["<b>Completions:</b><br/>complete()"]
end
end
subgraph ExternalMCP["🔌 External MCP Servers"]
EXT1["MCP Server 1<br/>(WebSocket/StreamableHTTP/SSE)"]
EXT2["MCP Server N"]
end
subgraph Storage["💾 Storage"]
ST1["IndexedDB"]
ST2["conversations"]
ST3["messages"]
ST5["LocalStorage"]
ST6["config"]
ST7["userOverrides"]
ST8["mcpServers"]
end
subgraph APIs["🌐 llama-server API"]
API1["/v1/chat/completions"]
API2["/props<br/>/props?model="]
API3["/models<br/>/models/load<br/>/models/unload"]
API4["/v1/models"]
end
%% Routes render Components
R1 --> C_Screen
R2 --> C_Screen
RL --> C_Sidebar
%% Layout runs MCP health checks on startup
RL --> S6
%% Component hierarchy
C_Screen --> C_Form & C_Messages & C_Settings
C_Messages --> C_Message
C_Message --> C_MessageUser
C_MessageUser --> C_MessageEditForm
C_MessageEditForm --> C_ModelsSelector
C_MessageEditForm --> C_Attach
C_Form --> C_ModelsSelector
C_Form --> C_Attach
C_Form --> C_McpServersSelector
C_Message --> C_Attach
%% MCP Components hierarchy
C_Settings --> C_McpSettings
C_McpSettings --> C_McpServerCard
C_McpServerCard --> C_McpResourceBrowser
C_McpResourceBrowser --> C_McpResourcePreview
%% Components use Hooks
C_Form --> H1
C_Message --> H1 & H2
C_MessageEditForm --> H1
C_Screen --> H2
%% Hooks use Stores
H1 --> S3 & S4
H2 --> S1 & S5
%% Components use Stores
C_Screen --> S1 & S2
C_Messages --> S2
C_Message --> S1 & S2 & S3
C_Form --> S1 & S3 & S6
C_Sidebar --> S2
C_ModelsSelector --> S3 & S4
C_Settings --> S5
C_McpSettings --> S6
C_McpServerCard --> S6
C_McpResourceBrowser --> S6 & S7
C_McpServersSelector --> S6
%% Stores export Reactive State
S1 -. exports .-> ChatExports
SA -. exports .-> AgenticExports
S2 -. exports .-> ConvExports
S3 -. exports .-> ModelsExports
S4 -. exports .-> ServerExports
S5 -. exports .-> SettingsExports
S6 -. exports .-> MCPExports
S7 -. exports .-> MCPExports
%% chatStore → agenticStore (agentic loop orchestration)
S1 --> SA
SA --> SV1
SA --> S6
%% Stores use Services
S1 --> SV1 & SV4
S2 --> SV4
S3 --> SV2 & SV3
S4 --> SV3
S5 --> SV5
S6 --> SV6
S7 --> SV6
%% Services to Storage
SV4 --> ST1
ST1 --> ST2 & ST3
SV5 --> ST5
ST5 --> ST6 & ST7 & ST8
%% Services to APIs
SV1 --> API1
SV2 --> API3 & API4
SV3 --> API2
%% MCP → External Servers
SV6 --> EXT1 & EXT2
%% Styling
classDef routeStyle fill:#e1f5fe,stroke:#01579b,stroke-width:2px
classDef componentStyle fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
classDef componentGroupStyle fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
classDef hookStyle fill:#fff8e1,stroke:#ff8f00,stroke-width:2px
classDef storeStyle fill:#fff3e0,stroke:#e65100,stroke-width:2px
classDef stateStyle fill:#ffe0b2,stroke:#e65100,stroke-width:1px
classDef methodStyle fill:#ffecb3,stroke:#e65100,stroke-width:1px
classDef reactiveStyle fill:#fffde7,stroke:#f9a825,stroke-width:1px
classDef serviceStyle fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
classDef serviceMStyle fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
classDef externalStyle fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px,stroke-dasharray: 5 5
classDef storageStyle fill:#fce4ec,stroke:#c2185b,stroke-width:2px
classDef apiStyle fill:#e3f2fd,stroke:#1565c0,stroke-width:2px
class R1,R2,RL routeStyle
class C_Sidebar,C_Screen,C_Form,C_Messages,C_Message,C_MessageUser,C_MessageEditForm componentStyle
class C_ModelsSelector,C_Settings componentStyle
class C_Attach componentStyle
class C_McpSettings,C_McpServerCard,C_McpResourceBrowser,C_McpResourcePreview,C_McpServersSelector componentStyle
class H1,H2,H3 hookStyle
class LayoutComponents,ChatUIComponents,MCPComponents componentGroupStyle
class Hooks hookStyle
classDef agenticStyle fill:#e8eaf6,stroke:#283593,stroke-width:2px
classDef agenticMethodStyle fill:#c5cae9,stroke:#283593,stroke-width:1px
class S1,S2,S3,S4,S5,SA,S6,S7 storeStyle
class S1State,S2State,S3State,S4State,S5State,SAState,S6State,S7State stateStyle
class S1Msg,S1Regen,S1Edit,S1Stream,S1LoadState,S1ProcState,S1Error,S1Utils methodStyle
class SASession,SAConfig,SAFlow methodStyle
class S2Lifecycle,S2ConvCRUD,S2MsgMgmt,S2Nav,S2McpOverrides,S2Export,S2Utils methodStyle
class S3Getters,S3Modal,S3Status,S3Fetch,S3Select,S3LoadUnload,S3Utils methodStyle
class S4Getters,S4Data,S4Utils methodStyle
class S5Lifecycle,S5Update,S5Reset,S5Sync,S5Utils methodStyle
class S6Lifecycle,S6Health,S6Servers,S6Tools,S6Prompts methodStyle
class S7Resources,S7Cache,S7Subs,S7Attach methodStyle
class ChatExports,AgenticExports,ConvExports,ModelsExports,ServerExports,SettingsExports,MCPExports reactiveStyle
class SV1,SV2,SV3,SV4,SV5,SV6 serviceStyle
class SV6Transport,SV6Conn,SV6Tools,SV6Prompts,SV6Resources,SV6Complete serviceMStyle
class EXT1,EXT2 externalStyle
class SV1Msg,SV1Stream,SV1Convert,SV1Utils serviceMStyle
class SV2List,SV2LoadUnload,SV2Status serviceMStyle
class SV3Fetch serviceMStyle
class SV4Conv,SV4Msg,SV4Node,SV4Import serviceMStyle
class SV5Extract,SV5Merge,SV5Info,SV5Diff serviceMStyle
class ST1,ST2,ST3,ST5,ST6,ST7,ST8 storageStyle
class API1,API2,API3,API4 apiStyle
```
-228
View File
@@ -1,228 +0,0 @@
```mermaid
sequenceDiagram
participant UI as 🧩 ChatForm / ChatMessage
participant chatStore as 🗄️ chatStore
participant agenticStore as 🗄️ agenticStore
participant convStore as 🗄️ conversationsStore
participant settingsStore as 🗄️ settingsStore
participant mcpStore as 🗄️ mcpStore
participant ChatSvc as ⚙️ ChatService
participant DbSvc as ⚙️ DatabaseService
participant API as 🌐 /v1/chat/completions
Note over chatStore: State:<br/>isLoading, currentResponse<br/>errorDialogState, activeProcessingState<br/>chatLoadingStates (Map)<br/>chatStreamingStates (Map)<br/>abortControllers (Map)<br/>processingStates (Map)
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,API: 💬 SEND MESSAGE
%% ═══════════════════════════════════════════════════════════════════════════
UI->>chatStore: sendMessage(content, extras)
activate chatStore
chatStore->>chatStore: setChatLoading(convId, true)
chatStore->>chatStore: clearChatStreaming(convId)
alt no active conversation
chatStore->>convStore: createConversation()
Note over convStore: → see conversations-flow.mmd
end
chatStore->>mcpStore: consumeResourceAttachmentsAsExtras()
Note right of mcpStore: Converts pending MCP resource<br/>attachments into message extras
chatStore->>chatStore: addMessage("user", content, extras)
chatStore->>DbSvc: createMessageBranch(userMsg, parentId)
chatStore->>convStore: addMessageToActive(userMsg)
chatStore->>convStore: updateCurrentNode(userMsg.id)
chatStore->>chatStore: createAssistantMessage(userMsg.id)
chatStore->>DbSvc: createMessageBranch(assistantMsg, userMsg.id)
chatStore->>convStore: addMessageToActive(assistantMsg)
chatStore->>chatStore: streamChatCompletion(messages, assistantMsg)
deactivate chatStore
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,API: 🌊 STREAMING (with agentic flow detection)
%% ═══════════════════════════════════════════════════════════════════════════
activate chatStore
chatStore->>chatStore: startStreaming()
Note right of chatStore: isStreamingActive = true
chatStore->>chatStore: setActiveProcessingConversation(convId)
chatStore->>chatStore: getOrCreateAbortController(convId)
Note right of chatStore: abortControllers.set(convId, new AbortController())
chatStore->>chatStore: getApiOptions()
Note right of chatStore: Merge from settingsStore.config:<br/>temperature, max_tokens, top_p, etc.
alt agenticConfig.enabled && mcpStore has connected servers
chatStore->>agenticStore: runAgenticFlow(convId, messages, assistantMsg, options, signal)
Note over agenticStore: Multi-turn agentic loop:<br/>1. Call ChatService.sendMessage()<br/>2. If response has tool_calls → execute via mcpStore<br/>3. Append tool results as messages<br/>4. Loop until no more tool_calls or maxTurns<br/>→ see agentic flow details below
agenticStore-->>chatStore: final response with timings
else standard (non-agentic) flow
chatStore->>ChatSvc: sendMessage(messages, options, signal)
end
activate ChatSvc
ChatSvc->>ChatSvc: convertDbMessageToApiChatMessageData(messages)
Note right of ChatSvc: DatabaseMessage[] → ApiChatMessageData[]<br/>Process attachments (images, PDFs, audio)
ChatSvc->>API: POST /v1/chat/completions
Note right of API: {messages, model?, stream: true, ...params}
loop SSE chunks
API-->>ChatSvc: data: {"choices":[{"delta":{...}}]}
ChatSvc->>ChatSvc: handleStreamResponse(response)
alt content chunk
ChatSvc-->>chatStore: onChunk(content)
chatStore->>chatStore: setChatStreaming(convId, response, msgId)
Note right of chatStore: currentResponse = $state(accumulated)
chatStore->>convStore: updateMessageAtIndex(idx, {content})
end
alt reasoning chunk
ChatSvc-->>chatStore: onReasoningChunk(reasoning)
chatStore->>convStore: updateMessageAtIndex(idx, {thinking})
end
alt tool_calls chunk
ChatSvc-->>chatStore: onToolCallChunk(toolCalls)
chatStore->>convStore: updateMessageAtIndex(idx, {toolCalls})
end
alt model info
ChatSvc-->>chatStore: onModel(modelName)
chatStore->>chatStore: recordModel(modelName)
chatStore->>DbSvc: updateMessage(msgId, {model})
end
alt timings (during stream)
ChatSvc-->>chatStore: onTimings(timings, promptProgress)
chatStore->>chatStore: updateProcessingStateFromTimings()
end
chatStore-->>UI: reactive $state update
end
API-->>ChatSvc: data: [DONE]
ChatSvc-->>chatStore: onComplete(content, reasoning, timings, toolCalls)
deactivate ChatSvc
chatStore->>chatStore: stopStreaming()
chatStore->>DbSvc: updateMessage(msgId, {content, timings, model})
chatStore->>convStore: updateCurrentNode(msgId)
chatStore->>chatStore: setChatLoading(convId, false)
chatStore->>chatStore: clearChatStreaming(convId)
chatStore->>chatStore: clearProcessingState(convId)
deactivate chatStore
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,API: ⏹️ STOP GENERATION
%% ═══════════════════════════════════════════════════════════════════════════
UI->>chatStore: stopGeneration()
activate chatStore
chatStore->>chatStore: savePartialResponseIfNeeded(convId)
Note right of chatStore: Save currentResponse to DB if non-empty
chatStore->>chatStore: abortControllers.get(convId).abort()
Note right of chatStore: fetch throws AbortError → caught by isAbortError()
chatStore->>chatStore: stopStreaming()
chatStore->>chatStore: setChatLoading(convId, false)
chatStore->>chatStore: clearChatStreaming(convId)
chatStore->>chatStore: clearProcessingState(convId)
deactivate chatStore
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,API: 🔁 REGENERATE
%% ═══════════════════════════════════════════════════════════════════════════
UI->>chatStore: regenerateMessageWithBranching(msgId, model?)
activate chatStore
chatStore->>convStore: findMessageIndex(msgId)
chatStore->>chatStore: Get parent of target message
chatStore->>chatStore: createAssistantMessage(parentId)
chatStore->>DbSvc: createMessageBranch(newAssistantMsg, parentId)
chatStore->>convStore: refreshActiveMessages()
Note right of chatStore: Same streaming flow
chatStore->>chatStore: streamChatCompletion(...)
deactivate chatStore
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,API: ➡️ CONTINUE
%% ═══════════════════════════════════════════════════════════════════════════
UI->>chatStore: continueAssistantMessage(msgId)
activate chatStore
chatStore->>chatStore: Get existing content from message
chatStore->>chatStore: streamChatCompletion(..., existingContent)
Note right of chatStore: Appends to existing message content
deactivate chatStore
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,API: ✏️ EDIT USER MESSAGE
%% ═══════════════════════════════════════════════════════════════════════════
UI->>chatStore: editMessageWithBranching(msgId, newContent, extras)
activate chatStore
chatStore->>chatStore: Get parent of target message
chatStore->>DbSvc: createMessageBranch(editedMsg, parentId)
chatStore->>convStore: refreshActiveMessages()
Note right of chatStore: Creates new branch, original preserved
chatStore->>chatStore: createAssistantMessage(editedMsg.id)
chatStore->>chatStore: streamChatCompletion(...)
Note right of chatStore: Automatically regenerates response
deactivate chatStore
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,API: ❌ ERROR HANDLING
%% ═══════════════════════════════════════════════════════════════════════════
Note over chatStore: On stream error (non-abort):
chatStore->>chatStore: showErrorDialog(type, message)
Note right of chatStore: errorDialogState = {type: 'timeout'|'server', message}
chatStore->>convStore: removeMessageAtIndex(failedMsgIdx)
chatStore->>DbSvc: deleteMessage(failedMsgId)
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,API: 🤖 AGENTIC LOOP (when agenticConfig.enabled)
%% ═══════════════════════════════════════════════════════════════════════════
Note over agenticStore: agenticStore.runAgenticFlow(convId, messages, assistantMsg, options, signal)
activate agenticStore
agenticStore->>agenticStore: getSession(convId) or create new
agenticStore->>agenticStore: updateSession(turn: 0, running: true)
loop executeAgenticLoop (until no tool_calls or maxTurns)
agenticStore->>agenticStore: turn++
agenticStore->>ChatSvc: sendMessage(messages, options, signal)
ChatSvc->>API: POST /v1/chat/completions
API-->>ChatSvc: response with potential tool_calls
ChatSvc-->>agenticStore: onComplete(content, reasoning, timings, toolCalls)
alt response has tool_calls
agenticStore->>agenticStore: normalizeToolCalls(toolCalls)
loop for each tool_call
agenticStore->>agenticStore: updateSession(streamingToolCall)
agenticStore->>mcpStore: executeTool(mcpCall, signal)
mcpStore-->>agenticStore: tool result
agenticStore->>agenticStore: extractBase64Attachments(result)
agenticStore->>agenticStore: emitToolCallResult(convId, ...)
agenticStore->>convStore: addMessageToActive(toolResultMsg)
agenticStore->>DbSvc: createMessageBranch(toolResultMsg)
end
agenticStore->>agenticStore: Create new assistantMsg for next turn
Note right of agenticStore: Continue loop with updated messages
else no tool_calls (final response)
agenticStore->>agenticStore: buildFinalTimings(allTurns)
Note right of agenticStore: Break loop, return final response
end
end
agenticStore->>agenticStore: updateSession(running: false)
agenticStore-->>chatStore: final content, timings, model
deactivate agenticStore
```
-183
View File
@@ -1,183 +0,0 @@
```mermaid
sequenceDiagram
participant UI as 🧩 ChatSidebar / ChatScreen
participant convStore as 🗄️ conversationsStore
participant chatStore as 🗄️ chatStore
participant DbSvc as ⚙️ DatabaseService
participant IDB as 💾 IndexedDB
Note over convStore: State:<br/>conversations: DatabaseConversation[]<br/>activeConversation: DatabaseConversation | null<br/>activeMessages: DatabaseMessage[]<br/>isInitialized: boolean<br/>pendingMcpServerOverrides: Map&lt;string, McpServerOverride&gt;
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,IDB: 🚀 INITIALIZATION
%% ═══════════════════════════════════════════════════════════════════════════
Note over convStore: Auto-initialized in constructor (browser only)
convStore->>convStore: initialize()
activate convStore
convStore->>convStore: loadConversations()
convStore->>DbSvc: getAllConversations()
DbSvc->>IDB: SELECT * FROM conversations ORDER BY lastModified DESC
IDB-->>DbSvc: Conversation[]
DbSvc-->>convStore: conversations
convStore->>convStore: conversations = $state(data)
convStore->>convStore: isInitialized = true
deactivate convStore
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,IDB: CREATE CONVERSATION
%% ═══════════════════════════════════════════════════════════════════════════
UI->>convStore: createConversation(name?)
activate convStore
convStore->>DbSvc: createConversation(name || "New Chat")
DbSvc->>IDB: INSERT INTO conversations
IDB-->>DbSvc: conversation {id, name, lastModified, currNode: ""}
DbSvc-->>convStore: conversation
convStore->>convStore: conversations.unshift(conversation)
convStore->>convStore: activeConversation = $state(conversation)
convStore->>convStore: activeMessages = $state([])
alt pendingMcpServerOverrides has entries
loop each pending override
convStore->>DbSvc: Store MCP server override for new conversation
end
convStore->>convStore: clearPendingMcpServerOverrides()
end
deactivate convStore
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,IDB: 📂 LOAD CONVERSATION
%% ═══════════════════════════════════════════════════════════════════════════
UI->>convStore: loadConversation(convId)
activate convStore
convStore->>DbSvc: getConversation(convId)
DbSvc->>IDB: SELECT * FROM conversations WHERE id = ?
IDB-->>DbSvc: conversation
convStore->>convStore: activeConversation = $state(conversation)
convStore->>convStore: refreshActiveMessages()
convStore->>DbSvc: getConversationMessages(convId)
DbSvc->>IDB: SELECT * FROM messages WHERE convId = ?
IDB-->>DbSvc: allMessages[]
convStore->>convStore: filterByLeafNodeId(allMessages, currNode)
Note right of convStore: Filter to show only current branch path
convStore->>convStore: activeMessages = $state(filtered)
Note right of convStore: Route (+page.svelte) then calls:<br/>chatStore.syncLoadingStateForChat(convId)
deactivate convStore
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,IDB: 🌳 MESSAGE BRANCHING MODEL
%% ═══════════════════════════════════════════════════════════════════════════
Note over IDB: Message Tree Structure:<br/>- Each message has parent (null for root)<br/>- Each message has children[] array<br/>- Conversation.currNode points to active leaf<br/>- filterByLeafNodeId() traverses from root to currNode
rect rgb(240, 240, 255)
Note over convStore: Example Branch Structure:
Note over convStore: root → user1 → assistant1 → user2 → assistant2a (currNode)<br/> ↘ assistant2b (alt branch)
end
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,IDB: ↔️ BRANCH NAVIGATION
%% ═══════════════════════════════════════════════════════════════════════════
UI->>convStore: navigateToSibling(msgId, direction)
activate convStore
convStore->>convStore: Find message in activeMessages
convStore->>convStore: Get parent message
convStore->>convStore: Find sibling in parent.children[]
convStore->>convStore: findLeafNode(siblingId, allMessages)
Note right of convStore: Navigate to leaf of sibling branch
convStore->>convStore: updateCurrentNode(leafId)
convStore->>DbSvc: updateCurrentNode(convId, leafId)
DbSvc->>IDB: UPDATE conversations SET currNode = ?
convStore->>convStore: refreshActiveMessages()
deactivate convStore
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,IDB: 📝 UPDATE CONVERSATION
%% ═══════════════════════════════════════════════════════════════════════════
UI->>convStore: updateConversationName(convId, newName)
activate convStore
convStore->>DbSvc: updateConversation(convId, {name: newName})
DbSvc->>IDB: UPDATE conversations SET name = ?
convStore->>convStore: Update in conversations array
deactivate convStore
Note over convStore: Auto-title update (after first response):
convStore->>convStore: updateConversationTitleWithConfirmation()
convStore->>convStore: titleUpdateConfirmationCallback?()
Note right of convStore: Shows dialog if title would change
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,IDB: 🗑️ DELETE CONVERSATION
%% ═══════════════════════════════════════════════════════════════════════════
UI->>convStore: deleteConversation(convId)
activate convStore
convStore->>DbSvc: deleteConversation(convId)
DbSvc->>IDB: DELETE FROM conversations WHERE id = ?
DbSvc->>IDB: DELETE FROM messages WHERE convId = ?
convStore->>convStore: conversations.filter(c => c.id !== convId)
alt deleted active conversation
convStore->>convStore: clearActiveConversation()
end
deactivate convStore
UI->>convStore: deleteAll()
activate convStore
convStore->>DbSvc: Delete all conversations and messages
convStore->>convStore: conversations = []
convStore->>convStore: clearActiveConversation()
deactivate convStore
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,IDB: MCP SERVER PER-CHAT OVERRIDES
%% ═══════════════════════════════════════════════════════════════════════════
Note over convStore: Conversations can override which MCP servers are enabled.
Note over convStore: Uses pendingMcpServerOverrides before conversation<br/>is created, then persists to conversation metadata.
UI->>convStore: setMcpServerOverride(convId, serverName, override)
Note right of convStore: override = {enabled: boolean}
UI->>convStore: toggleMcpServerForChat(convId, serverName, enabled)
activate convStore
convStore->>convStore: setMcpServerOverride(convId, serverName, {enabled})
deactivate convStore
UI->>convStore: isMcpServerEnabledForChat(convId, serverName)
Note right of convStore: Check override → fall back to global MCP config
UI->>convStore: getAllMcpServerOverrides(convId)
Note right of convStore: Returns all overrides for a conversation
UI->>convStore: removeMcpServerOverride(convId, serverName)
UI->>convStore: getMcpServerOverride(convId, serverName)
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,IDB: 📤 EXPORT / 📥 IMPORT
%% ═══════════════════════════════════════════════════════════════════════════
UI->>convStore: exportAllConversations()
activate convStore
convStore->>DbSvc: getAllConversations()
loop each conversation
convStore->>DbSvc: getConversationMessages(convId)
end
convStore->>convStore: triggerDownload(JSON blob)
deactivate convStore
UI->>convStore: importConversations(file)
activate convStore
convStore->>convStore: Parse JSON file
convStore->>convStore: importConversationsData(parsed)
convStore->>DbSvc: importConversations(parsed)
Note right of DbSvc: Skips duplicate conversations<br/>(checks existing by ID)
DbSvc->>IDB: INSERT conversations + messages (skip existing)
convStore->>convStore: loadConversations()
deactivate convStore
```
@@ -1,45 +0,0 @@
```mermaid
%% MODEL Mode Data Flow (single model)
%% Detailed flows: ./flows/server-flow.mmd, ./flows/models-flow.mmd, ./flows/chat-flow.mmd
sequenceDiagram
participant User as 👤 User
participant UI as 🧩 UI
participant Stores as 🗄️ Stores
participant DB as 💾 IndexedDB
participant API as 🌐 llama-server
Note over User,API: 🚀 Initialization (see: server-flow.mmd, models-flow.mmd)
UI->>Stores: initialize()
Stores->>DB: load conversations
Stores->>API: GET /props
API-->>Stores: server config + modalities
Stores->>API: GET /v1/models
API-->>Stores: single model (auto-selected)
Note over User,API: 💬 Chat Flow (see: chat-flow.mmd)
User->>UI: send message
UI->>Stores: sendMessage()
Stores->>DB: save user message
Stores->>API: POST /v1/chat/completions (stream)
loop streaming
API-->>Stores: SSE chunks
Stores-->>UI: reactive update
end
API-->>Stores: done + timings
Stores->>DB: save assistant message
Note over User,API: 🔁 Regenerate
User->>UI: regenerate
Stores->>DB: create message branch
Note right of Stores: same streaming flow
Note over User,API: ⏹️ Stop
User->>UI: stop
Stores->>Stores: abort stream
Stores->>DB: save partial response
```
@@ -1,77 +0,0 @@
```mermaid
%% ROUTER Mode Data Flow (multi-model)
%% Detailed flows: ./flows/server-flow.mmd, ./flows/models-flow.mmd, ./flows/chat-flow.mmd
sequenceDiagram
participant User as 👤 User
participant UI as 🧩 UI
participant Stores as 🗄️ Stores
participant DB as 💾 IndexedDB
participant API as 🌐 llama-server
Note over User,API: 🚀 Initialization (see: server-flow.mmd, models-flow.mmd)
UI->>Stores: initialize()
Stores->>DB: load conversations
Stores->>API: GET /props
API-->>Stores: {role: "router"}
Stores->>API: GET /v1/models
API-->>Stores: models[] with status (loaded/available)
loop each loaded model
Stores->>API: GET /props?model=X
API-->>Stores: modalities (vision/audio)
end
Note over User,API: 🔄 Model Selection (see: models-flow.mmd)
User->>UI: select model
alt model not loaded
Stores->>API: POST /models/load
loop poll status
Stores->>API: GET /v1/models
API-->>Stores: check if loaded
end
Stores->>API: GET /props?model=X
API-->>Stores: cache modalities
end
Stores->>Stores: validate modalities vs conversation
alt valid
Stores->>Stores: select model
else invalid
Stores->>API: POST /models/unload
UI->>User: show error toast
end
Note over User,API: 💬 Chat Flow (see: chat-flow.mmd)
User->>UI: send message
UI->>Stores: sendMessage()
Stores->>DB: save user message
Stores->>API: POST /v1/chat/completions {model: X}
Note right of API: router forwards to model
loop streaming
API-->>Stores: SSE chunks + model info
Stores-->>UI: reactive update
end
API-->>Stores: done + timings
Stores->>DB: save assistant message + model used
Note over User,API: 🔁 Regenerate (optional: different model)
User->>UI: regenerate
Stores->>Stores: validate modalities up to this message
Stores->>DB: create message branch
Note right of Stores: same streaming flow
Note over User,API: ⏹️ Stop
User->>UI: stop
Stores->>Stores: abort stream
Stores->>DB: save partial response
Note over User,API: 🗑️ LRU Unloading
Note right of API: Server auto-unloads LRU models<br/>when cache full
User->>UI: select unloaded model
Note right of Stores: triggers load flow again
```
-174
View File
@@ -1,174 +0,0 @@
```mermaid
sequenceDiagram
participant Store as 🗄️ Stores
participant DbSvc as ⚙️ DatabaseService
participant Dexie as 📦 Dexie ORM
participant IDB as 💾 IndexedDB
Note over DbSvc: Stateless service - all methods static<br/>Database: "LlamacppWebui"
%% ═══════════════════════════════════════════════════════════════════════════
Note over Store,IDB: 📊 SCHEMA
%% ═══════════════════════════════════════════════════════════════════════════
rect rgb(240, 248, 255)
Note over IDB: conversations table:<br/>id (PK), lastModified, currNode, name
end
rect rgb(255, 248, 240)
Note over IDB: messages table:<br/>id (PK), convId (FK), type, role, timestamp,<br/>parent, children[], content, thinking,<br/>toolCalls, extra[], model, timings
end
%% ═══════════════════════════════════════════════════════════════════════════
Note over Store,IDB: 💬 CONVERSATIONS CRUD
%% ═══════════════════════════════════════════════════════════════════════════
Store->>DbSvc: createConversation(name)
activate DbSvc
DbSvc->>DbSvc: Generate UUID
DbSvc->>Dexie: db.conversations.add({id, name, lastModified, currNode: ""})
Dexie->>IDB: INSERT
IDB-->>Dexie: success
DbSvc-->>Store: DatabaseConversation
deactivate DbSvc
Store->>DbSvc: getConversation(convId)
DbSvc->>Dexie: db.conversations.get(convId)
Dexie->>IDB: SELECT WHERE id = ?
IDB-->>DbSvc: DatabaseConversation
Store->>DbSvc: getAllConversations()
DbSvc->>Dexie: db.conversations.orderBy('lastModified').reverse().toArray()
Dexie->>IDB: SELECT ORDER BY lastModified DESC
IDB-->>DbSvc: DatabaseConversation[]
Store->>DbSvc: updateConversation(convId, updates)
DbSvc->>Dexie: db.conversations.update(convId, {...updates, lastModified})
Dexie->>IDB: UPDATE
Store->>DbSvc: deleteConversation(convId)
activate DbSvc
DbSvc->>Dexie: db.conversations.delete(convId)
Dexie->>IDB: DELETE FROM conversations
DbSvc->>Dexie: db.messages.where('convId').equals(convId).delete()
Dexie->>IDB: DELETE FROM messages WHERE convId = ?
deactivate DbSvc
%% ═══════════════════════════════════════════════════════════════════════════
Note over Store,IDB: 📝 MESSAGES CRUD
%% ═══════════════════════════════════════════════════════════════════════════
Store->>DbSvc: createRootMessage(convId)
activate DbSvc
DbSvc->>DbSvc: Create root message {type: "root", parent: null}
DbSvc->>Dexie: db.messages.add(rootMsg)
Dexie->>IDB: INSERT
DbSvc-->>Store: rootMessageId
deactivate DbSvc
Store->>DbSvc: createSystemMessage(convId, content, parentId)
activate DbSvc
DbSvc->>DbSvc: Create message {role: "system", parent: parentId}
DbSvc->>Dexie: db.messages.add(systemMsg)
Dexie->>IDB: INSERT
DbSvc-->>Store: DatabaseMessage
deactivate DbSvc
Store->>DbSvc: createMessageBranch(message, parentId)
activate DbSvc
DbSvc->>DbSvc: Generate UUID for new message
DbSvc->>Dexie: db.messages.add({...message, id, parent: parentId})
Dexie->>IDB: INSERT message
alt parentId exists
DbSvc->>Dexie: db.messages.get(parentId)
Dexie->>IDB: SELECT parent
DbSvc->>DbSvc: parent.children.push(newId)
DbSvc->>Dexie: db.messages.update(parentId, {children})
Dexie->>IDB: UPDATE parent.children
end
DbSvc->>Dexie: db.conversations.update(convId, {currNode: newId})
Dexie->>IDB: UPDATE conversation.currNode
DbSvc-->>Store: DatabaseMessage
deactivate DbSvc
Store->>DbSvc: getConversationMessages(convId)
DbSvc->>Dexie: db.messages.where('convId').equals(convId).toArray()
Dexie->>IDB: SELECT WHERE convId = ?
IDB-->>DbSvc: DatabaseMessage[]
Store->>DbSvc: updateMessage(msgId, updates)
DbSvc->>Dexie: db.messages.update(msgId, updates)
Dexie->>IDB: UPDATE
Store->>DbSvc: deleteMessage(msgId)
DbSvc->>Dexie: db.messages.delete(msgId)
Dexie->>IDB: DELETE
%% ═══════════════════════════════════════════════════════════════════════════
Note over Store,IDB: 🌳 BRANCHING OPERATIONS
%% ═══════════════════════════════════════════════════════════════════════════
Store->>DbSvc: updateCurrentNode(convId, nodeId)
DbSvc->>Dexie: db.conversations.update(convId, {currNode: nodeId, lastModified})
Dexie->>IDB: UPDATE
Store->>DbSvc: deleteMessageCascading(msgId)
activate DbSvc
DbSvc->>DbSvc: findDescendantMessages(msgId, allMessages)
Note right of DbSvc: Recursively find all children
loop each descendant
DbSvc->>Dexie: db.messages.delete(descendantId)
Dexie->>IDB: DELETE
end
DbSvc->>Dexie: db.messages.delete(msgId)
Dexie->>IDB: DELETE target message
alt target message has a parent
DbSvc->>Dexie: db.messages.get(parentId)
DbSvc->>DbSvc: parent.children.filter(id !== msgId)
DbSvc->>Dexie: db.messages.update(parentId, {children})
Note right of DbSvc: Remove deleted message from parent's children[]
end
deactivate DbSvc
%% ═══════════════════════════════════════════════════════════════════════════
Note over Store,IDB: 📥 IMPORT
%% ═══════════════════════════════════════════════════════════════════════════
Store->>DbSvc: importConversations(data)
activate DbSvc
loop each conversation in data
DbSvc->>Dexie: db.conversations.get(conv.id)
alt conversation already exists
Note right of DbSvc: Skip duplicate (keep existing)
else conversation is new
DbSvc->>Dexie: db.conversations.add(conversation)
Dexie->>IDB: INSERT conversation
loop each message
DbSvc->>Dexie: db.messages.add(message)
Dexie->>IDB: INSERT message
end
end
end
deactivate DbSvc
%% ═══════════════════════════════════════════════════════════════════════════
Note over Store,IDB: 🔗 MESSAGE TREE UTILITIES
%% ═══════════════════════════════════════════════════════════════════════════
Note over DbSvc: Used by stores (imported from utils):
rect rgb(240, 255, 240)
Note over DbSvc: filterByLeafNodeId(messages, leafId)<br/>→ Returns path from root to leaf<br/>→ Used to display current branch
end
rect rgb(240, 255, 240)
Note over DbSvc: findLeafNode(startId, messages)<br/>→ Traverse to deepest child<br/>→ Used for branch navigation
end
rect rgb(240, 255, 240)
Note over DbSvc: findDescendantMessages(msgId, messages)<br/>→ Find all children recursively<br/>→ Used for cascading deletes
end
```
-226
View File
@@ -1,226 +0,0 @@
```mermaid
sequenceDiagram
participant UI as 🧩 McpServersSettings / ChatForm
participant chatStore as 🗄️ chatStore
participant mcpStore as 🗄️ mcpStore
participant mcpResStore as 🗄️ mcpResourceStore
participant convStore as 🗄️ conversationsStore
participant MCPSvc as ⚙️ MCPService
participant LS as 💾 LocalStorage
participant ExtMCP as 🔌 External MCP Server
Note over mcpStore: State:<br/>isInitializing, error<br/>toolCount, connectedServers<br/>healthChecks (Map)<br/>connections (Map)<br/>toolsIndex (Map)<br/>serverConfigs (Map)
Note over mcpResStore: State:<br/>serverResources (Map)<br/>cachedResources (Map)<br/>subscriptions (Map)<br/>attachments[]
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,ExtMCP: 🚀 INITIALIZATION (App Startup)
%% ═══════════════════════════════════════════════════════════════════════════
UI->>mcpStore: ensureInitialized()
activate mcpStore
mcpStore->>LS: get(MCP_SERVERS_LOCALSTORAGE_KEY)
LS-->>mcpStore: MCPServerSettingsEntry[]
mcpStore->>mcpStore: parseServerSettings(servers)
Note right of mcpStore: Filter enabled servers<br/>Build MCPServerConfig objects<br/>Per-chat overrides checked via convStore
loop For each enabled server
mcpStore->>mcpStore: runHealthCheck(serverId)
mcpStore->>mcpStore: updateHealthCheck(id, CONNECTING)
mcpStore->>MCPSvc: connect(serverName, config, clientInfo, capabilities, onPhase)
activate MCPSvc
MCPSvc->>MCPSvc: createTransport(config)
Note right of MCPSvc: WebSocket / StreamableHTTP / SSE<br/>with optional CORS proxy
MCPSvc->>ExtMCP: Transport handshake
ExtMCP-->>MCPSvc: Connection established
MCPSvc->>ExtMCP: Initialize request
Note right of ExtMCP: Exchange capabilities<br/>Server info, protocol version
ExtMCP-->>MCPSvc: InitializeResult (serverInfo, capabilities)
MCPSvc->>ExtMCP: listTools()
ExtMCP-->>MCPSvc: Tool[]
MCPSvc-->>mcpStore: MCPConnection
deactivate MCPSvc
mcpStore->>mcpStore: connections.set(serverName, connection)
mcpStore->>mcpStore: indexTools(connection.tools, serverName)
Note right of mcpStore: toolsIndex.set(toolName, serverName)<br/>Handle name conflicts with prefixes
mcpStore->>mcpStore: updateHealthCheck(id, SUCCESS)
mcpStore->>mcpStore: _connectedServers.push(serverName)
alt Server supports resources
mcpStore->>MCPSvc: listAllResources(connection)
MCPSvc->>ExtMCP: listResources()
ExtMCP-->>MCPSvc: MCPResource[]
MCPSvc-->>mcpStore: resources
mcpStore->>MCPSvc: listAllResourceTemplates(connection)
MCPSvc->>ExtMCP: listResourceTemplates()
ExtMCP-->>MCPSvc: MCPResourceTemplate[]
MCPSvc-->>mcpStore: templates
mcpStore->>mcpResStore: setServerResources(serverName, resources, templates)
end
end
mcpStore->>mcpStore: _isInitializing = false
deactivate mcpStore
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,ExtMCP: 🔧 TOOL EXECUTION (Chat with Tools)
%% ═══════════════════════════════════════════════════════════════════════════
UI->>mcpStore: executeTool(mcpCall: MCPToolCall, signal?)
activate mcpStore
mcpStore->>mcpStore: toolsIndex.get(mcpCall.function.name)
Note right of mcpStore: Resolve serverName from toolsIndex<br/>MCPToolCall = {id, type, function: {name, arguments}}
mcpStore->>mcpStore: acquireConnection()
Note right of mcpStore: activeFlowCount++<br/>Prevent shutdown during execution
mcpStore->>mcpStore: connection = connections.get(serverName)
mcpStore->>MCPSvc: callTool(connection, {name, arguments}, signal)
activate MCPSvc
MCPSvc->>MCPSvc: throwIfAborted(signal)
MCPSvc->>ExtMCP: callTool(name, arguments)
alt Tool execution success
ExtMCP-->>MCPSvc: ToolCallResult (content, isError)
MCPSvc->>MCPSvc: formatToolResult(result)
Note right of MCPSvc: Handle text, image (base64),<br/>embedded resource content
MCPSvc-->>mcpStore: ToolExecutionResult
else Tool execution error
ExtMCP-->>MCPSvc: Error
MCPSvc-->>mcpStore: throw Error
else Aborted
MCPSvc-->>mcpStore: throw AbortError
end
deactivate MCPSvc
mcpStore->>mcpStore: releaseConnection()
Note right of mcpStore: activeFlowCount--
mcpStore-->>UI: ToolExecutionResult
deactivate mcpStore
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,ExtMCP: RESOURCE ATTACHMENT CONSUMPTION
%% ═══════════════════════════════════════════════════════════════════════════
chatStore->>mcpStore: consumeResourceAttachmentsAsExtras()
activate mcpStore
mcpStore->>mcpResStore: getAttachments()
mcpResStore-->>mcpStore: MCPResourceAttachment[]
mcpStore->>mcpStore: Convert attachments to message extras
mcpStore->>mcpResStore: clearAttachments()
mcpStore-->>chatStore: MessageExtra[] (for user message)
deactivate mcpStore
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,ExtMCP: 📝 PROMPT OPERATIONS
%% ═══════════════════════════════════════════════════════════════════════════
UI->>mcpStore: getAllPrompts()
activate mcpStore
loop For each connected server with prompts capability
mcpStore->>MCPSvc: listPrompts(connection)
MCPSvc->>ExtMCP: listPrompts()
ExtMCP-->>MCPSvc: Prompt[]
MCPSvc-->>mcpStore: prompts
end
mcpStore-->>UI: MCPPromptInfo[] (with serverName)
deactivate mcpStore
UI->>mcpStore: getPrompt(serverName, promptName, args?)
activate mcpStore
mcpStore->>MCPSvc: getPrompt(connection, name, args)
MCPSvc->>ExtMCP: getPrompt({name, arguments})
ExtMCP-->>MCPSvc: GetPromptResult (messages)
MCPSvc-->>mcpStore: GetPromptResult
mcpStore-->>UI: GetPromptResult
deactivate mcpStore
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,ExtMCP: 📁 RESOURCE OPERATIONS
%% ═══════════════════════════════════════════════════════════════════════════
UI->>mcpResStore: addAttachment(resourceInfo)
activate mcpResStore
mcpResStore->>mcpResStore: Create MCPResourceAttachment (loading: true)
mcpResStore-->>UI: attachment
UI->>mcpStore: readResource(serverName, uri)
activate mcpStore
mcpStore->>MCPSvc: readResource(connection, uri)
MCPSvc->>ExtMCP: readResource({uri})
ExtMCP-->>MCPSvc: MCPReadResourceResult (contents)
MCPSvc-->>mcpStore: contents
mcpStore-->>UI: MCPResourceContent[]
deactivate mcpStore
UI->>mcpResStore: updateAttachmentContent(attachmentId, content)
mcpResStore->>mcpResStore: cacheResourceContent(resource, content)
deactivate mcpResStore
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,ExtMCP: 🔄 AUTO-RECONNECTION
%% ═══════════════════════════════════════════════════════════════════════════
Note over mcpStore: On WebSocket close or connection error:
mcpStore->>mcpStore: autoReconnect(serverName, attempt)
activate mcpStore
mcpStore->>mcpStore: Calculate backoff delay
Note right of mcpStore: delay = min(30s, 1s * 2^attempt)
mcpStore->>mcpStore: Wait for delay
mcpStore->>mcpStore: reconnectServer(serverName)
alt Reconnection success
mcpStore->>mcpStore: updateHealthCheck(id, SUCCESS)
else Max attempts reached
mcpStore->>mcpStore: updateHealthCheck(id, ERROR)
end
deactivate mcpStore
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,ExtMCP: 🛑 SHUTDOWN
%% ═══════════════════════════════════════════════════════════════════════════
UI->>mcpStore: shutdown()
activate mcpStore
mcpStore->>mcpStore: Wait for activeFlowCount == 0
loop For each connection
mcpStore->>MCPSvc: disconnect(connection)
MCPSvc->>MCPSvc: transport.onclose = undefined
MCPSvc->>ExtMCP: close()
end
mcpStore->>mcpStore: connections.clear()
mcpStore->>mcpStore: toolsIndex.clear()
mcpStore->>mcpStore: _connectedServers = []
mcpStore->>mcpResStore: clear()
deactivate mcpStore
```
-181
View File
@@ -1,181 +0,0 @@
```mermaid
sequenceDiagram
participant UI as 🧩 ModelsSelector
participant Hooks as 🪝 useModelChangeValidation
participant modelsStore as 🗄️ modelsStore
participant serverStore as 🗄️ serverStore
participant convStore as 🗄️ conversationsStore
participant ModelsSvc as ⚙️ ModelsService
participant PropsSvc as ⚙️ PropsService
participant API as 🌐 llama-server
Note over modelsStore: State:<br/>models: ModelOption[]<br/>routerModels: ApiModelDataEntry[]<br/>selectedModelId, selectedModelName<br/>loading, updating, error<br/>modelLoadingStates (Map)<br/>modelPropsCache (Map)<br/>propsCacheVersion
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,API: 🚀 INITIALIZATION (MODEL mode)
%% ═══════════════════════════════════════════════════════════════════════════
UI->>modelsStore: fetch()
activate modelsStore
modelsStore->>modelsStore: loading = true
alt serverStore.props not loaded
modelsStore->>serverStore: fetch()
Note over serverStore: → see server-flow.mmd
end
modelsStore->>ModelsSvc: list()
ModelsSvc->>API: GET /v1/models
API-->>ModelsSvc: ApiModelListResponse {data: [model]}
modelsStore->>modelsStore: models = $state(mapped)
Note right of modelsStore: Map to ModelOption[]:<br/>{id, name, model, description, capabilities}
Note over modelsStore: MODEL mode: Get modalities from serverStore.props
modelsStore->>modelsStore: modelPropsCache.set(model.id, serverStore.props)
modelsStore->>modelsStore: models[0].modalities = props.modalities
modelsStore->>modelsStore: Auto-select single model
Note right of modelsStore: selectedModelId = models[0].id
modelsStore->>modelsStore: loading = false
deactivate modelsStore
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,API: 🚀 INITIALIZATION (ROUTER mode)
%% ═══════════════════════════════════════════════════════════════════════════
UI->>modelsStore: fetch()
activate modelsStore
modelsStore->>ModelsSvc: list()
ModelsSvc->>API: GET /v1/models
API-->>ModelsSvc: ApiModelListResponse
modelsStore->>modelsStore: models = $state(mapped)
deactivate modelsStore
Note over UI: After models loaded, layout triggers:
UI->>modelsStore: fetchRouterModels()
activate modelsStore
modelsStore->>ModelsSvc: listRouter()
ModelsSvc->>API: GET /v1/models
API-->>ModelsSvc: ApiRouterModelsListResponse
Note right of API: {data: [{id, status, path, in_cache}]}
modelsStore->>modelsStore: routerModels = $state(data)
modelsStore->>modelsStore: fetchModalitiesForLoadedModels()
loop each model where status === "loaded"
modelsStore->>PropsSvc: fetchForModel(modelId)
PropsSvc->>API: GET /props?model={modelId}
API-->>PropsSvc: ApiLlamaCppServerProps
modelsStore->>modelsStore: modelPropsCache.set(modelId, props)
end
modelsStore->>modelsStore: propsCacheVersion++
deactivate modelsStore
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,API: 🔄 MODEL SELECTION (ROUTER mode)
%% ═══════════════════════════════════════════════════════════════════════════
UI->>Hooks: useModelChangeValidation({getRequiredModalities, onSuccess?, onValidationFailure?})
Note over Hooks: Hook configured per-component:<br/>ChatForm: getRequiredModalities = usedModalities<br/>ChatMessage: getRequiredModalities = getModalitiesUpToMessage(msgId)
UI->>Hooks: handleModelChange(modelId, modelName)
activate Hooks
Hooks->>Hooks: previousSelectedModelId = modelsStore.selectedModelId
Hooks->>modelsStore: isModelLoaded(modelName)?
alt model NOT loaded
Hooks->>modelsStore: loadModel(modelName)
Note over modelsStore: → see LOAD MODEL section below
end
Note over Hooks: Always fetch props (from cache or API)
Hooks->>modelsStore: fetchModelProps(modelName)
modelsStore-->>Hooks: props
Hooks->>convStore: getRequiredModalities()
convStore-->>Hooks: {vision, audio}
Hooks->>Hooks: Validate: model.modalities ⊇ required?
alt validation PASSED
Hooks->>modelsStore: selectModelById(modelId)
Hooks-->>UI: return true
else validation FAILED
Hooks->>UI: toast.error("Model doesn't support required modalities")
alt model was just loaded
Hooks->>modelsStore: unloadModel(modelName)
end
alt onValidationFailure provided
Hooks->>modelsStore: selectModelById(previousSelectedModelId)
end
Hooks-->>UI: return false
end
deactivate Hooks
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,API: ⬆️ LOAD MODEL (ROUTER mode)
%% ═══════════════════════════════════════════════════════════════════════════
modelsStore->>modelsStore: loadModel(modelId)
activate modelsStore
alt already loaded
modelsStore-->>modelsStore: return (no-op)
end
modelsStore->>modelsStore: modelLoadingStates.set(modelId, true)
modelsStore->>ModelsSvc: load(modelId)
ModelsSvc->>API: POST /models/load {model: modelId}
API-->>ModelsSvc: {status: "loading"}
modelsStore->>modelsStore: pollForModelStatus(modelId, LOADED)
loop poll every 500ms (max 60 attempts)
modelsStore->>modelsStore: fetchRouterModels()
modelsStore->>ModelsSvc: listRouter()
ModelsSvc->>API: GET /v1/models
API-->>ModelsSvc: models[]
modelsStore->>modelsStore: getModelStatus(modelId)
alt status === LOADED
Note right of modelsStore: break loop
else status === LOADING
Note right of modelsStore: wait 500ms, continue
end
end
modelsStore->>modelsStore: updateModelModalities(modelId)
modelsStore->>PropsSvc: fetchForModel(modelId)
PropsSvc->>API: GET /props?model={modelId}
API-->>PropsSvc: props with modalities
modelsStore->>modelsStore: modelPropsCache.set(modelId, props)
modelsStore->>modelsStore: propsCacheVersion++
modelsStore->>modelsStore: modelLoadingStates.set(modelId, false)
deactivate modelsStore
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,API: ⬇️ UNLOAD MODEL (ROUTER mode)
%% ═══════════════════════════════════════════════════════════════════════════
modelsStore->>modelsStore: unloadModel(modelId)
activate modelsStore
modelsStore->>modelsStore: modelLoadingStates.set(modelId, true)
modelsStore->>ModelsSvc: unload(modelId)
ModelsSvc->>API: POST /models/unload {model: modelId}
modelsStore->>modelsStore: pollForModelStatus(modelId, UNLOADED)
loop poll until unloaded
modelsStore->>ModelsSvc: listRouter()
ModelsSvc->>API: GET /v1/models
end
modelsStore->>modelsStore: modelLoadingStates.set(modelId, false)
deactivate modelsStore
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,API: 📊 COMPUTED GETTERS
%% ═══════════════════════════════════════════════════════════════════════════
Note over modelsStore: Getters:<br/>- selectedModel: ModelOption | null<br/>- loadedModelIds: string[] (from routerModels)<br/>- loadingModelIds: string[] (from modelLoadingStates)<br/>- singleModelName: string | null (MODEL mode only)
Note over modelsStore: Modality helpers:<br/>- getModelModalities(modelId): {vision, audio}<br/>- modelSupportsVision(modelId): boolean<br/>- modelSupportsAudio(modelId): boolean
```
-76
View File
@@ -1,76 +0,0 @@
```mermaid
sequenceDiagram
participant UI as 🧩 +layout.svelte
participant serverStore as 🗄️ serverStore
participant PropsSvc as ⚙️ PropsService
participant API as 🌐 llama-server
Note over serverStore: State:<br/>props: ApiLlamaCppServerProps | null<br/>loading, error<br/>role: ServerRole | null (MODEL | ROUTER)<br/>fetchPromise (deduplication)
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,API: 🚀 INITIALIZATION
%% ═══════════════════════════════════════════════════════════════════════════
UI->>serverStore: fetch()
activate serverStore
alt fetchPromise exists (already fetching)
serverStore-->>UI: return fetchPromise
Note right of serverStore: Deduplicate concurrent calls
end
serverStore->>serverStore: loading = true
serverStore->>serverStore: fetchPromise = new Promise()
serverStore->>PropsSvc: fetch()
PropsSvc->>API: GET /props
API-->>PropsSvc: ApiLlamaCppServerProps
Note right of API: {role, model_path, model_alias,<br/>modalities, default_generation_settings, ...}
PropsSvc-->>serverStore: props
serverStore->>serverStore: props = $state(data)
serverStore->>serverStore: detectRole(props)
Note right of serverStore: role = props.role === "router"<br/> ? ServerRole.ROUTER<br/> : ServerRole.MODEL
serverStore->>serverStore: loading = false
serverStore->>serverStore: fetchPromise = null
deactivate serverStore
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,API: 📊 COMPUTED GETTERS
%% ═══════════════════════════════════════════════════════════════════════════
Note over serverStore: Getters from props:
rect rgb(240, 255, 240)
Note over serverStore: defaultParams<br/>→ props.default_generation_settings.params<br/>(temperature, top_p, top_k, etc.)
end
rect rgb(240, 255, 240)
Note over serverStore: contextSize<br/>→ props.default_generation_settings.n_ctx
end
rect rgb(255, 240, 240)
Note over serverStore: isRouterMode<br/>→ role === ServerRole.ROUTER
end
rect rgb(255, 240, 240)
Note over serverStore: isModelMode<br/>→ role === ServerRole.MODEL
end
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,API: 🔗 RELATIONSHIPS
%% ═══════════════════════════════════════════════════════════════════════════
Note over serverStore: Used by:
Note right of serverStore: - modelsStore: role detection, MODEL mode modalities<br/>- settingsStore: syncWithServerDefaults (defaultParams)<br/>- chatStore: contextSize for processing state<br/>- UI components: isRouterMode for conditional rendering
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,API: ❌ ERROR HANDLING
%% ═══════════════════════════════════════════════════════════════════════════
Note over serverStore: getErrorMessage(): string | null<br/>Returns formatted error for UI display
Note over serverStore: clear(): void<br/>Resets all state (props, error, loading, role)
```
-156
View File
@@ -1,156 +0,0 @@
```mermaid
sequenceDiagram
participant UI as 🧩 ChatSettings
participant settingsStore as 🗄️ settingsStore
participant serverStore as 🗄️ serverStore
participant ParamSvc as ⚙️ ParameterSyncService
participant LS as 💾 LocalStorage
Note over settingsStore: State:<br/>config: SettingsConfigType<br/>theme: string ("auto" | "light" | "dark")<br/>isInitialized: boolean<br/>userOverrides: Set&lt;string&gt;
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,LS: 🚀 INITIALIZATION
%% ═══════════════════════════════════════════════════════════════════════════
Note over settingsStore: Auto-initialized in constructor (browser only)
settingsStore->>settingsStore: initialize()
activate settingsStore
settingsStore->>settingsStore: loadConfig()
settingsStore->>LS: get("llama-config")
LS-->>settingsStore: StoredConfig | null
alt config exists
settingsStore->>settingsStore: Merge with SETTING_CONFIG_DEFAULT
Note right of settingsStore: Fill missing keys with defaults
else no config
settingsStore->>settingsStore: config = SETTING_CONFIG_DEFAULT
end
settingsStore->>LS: get("llama-userOverrides")
LS-->>settingsStore: string[] | null
settingsStore->>settingsStore: userOverrides = new Set(data)
settingsStore->>settingsStore: loadTheme()
settingsStore->>LS: get("llama-theme")
LS-->>settingsStore: theme | "auto"
settingsStore->>settingsStore: isInitialized = true
deactivate settingsStore
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,LS: 🔄 SYNC WITH SERVER DEFAULTS
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI: Triggered from +layout.svelte when serverStore.props loaded
UI->>settingsStore: syncWithServerDefaults()
activate settingsStore
settingsStore->>serverStore: defaultParams
serverStore-->>settingsStore: {temperature, top_p, top_k, ...}
loop each SYNCABLE_PARAMETER
alt key NOT in userOverrides
settingsStore->>settingsStore: config[key] = serverDefault[key]
Note right of settingsStore: Non-overridden params adopt server default
else key in userOverrides
Note right of settingsStore: Keep user value, skip server default
end
end
alt serverStore.props has uiSettings
settingsStore->>settingsStore: Apply uiSettings from server
Note right of settingsStore: Server-provided UI settings<br/>(e.g. showRawOutputSwitch)
end
settingsStore->>settingsStore: saveConfig()
deactivate settingsStore
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,LS: ⚙️ UPDATE CONFIG
%% ═══════════════════════════════════════════════════════════════════════════
UI->>settingsStore: updateConfig(key, value)
activate settingsStore
settingsStore->>settingsStore: config[key] = value
alt value matches server default for key
settingsStore->>settingsStore: userOverrides.delete(key)
Note right of settingsStore: Matches server default, remove override
else value differs from server default
settingsStore->>settingsStore: userOverrides.add(key)
Note right of settingsStore: Mark as user-modified (won't be overwritten)
end
settingsStore->>settingsStore: saveConfig()
settingsStore->>LS: set(CONFIG_LOCALSTORAGE_KEY, config)
settingsStore->>LS: set(USER_OVERRIDES_LOCALSTORAGE_KEY, [...userOverrides])
deactivate settingsStore
UI->>settingsStore: updateMultipleConfig({key1: val1, key2: val2})
activate settingsStore
Note right of settingsStore: Batch update, single save
settingsStore->>settingsStore: For each key: config[key] = value
settingsStore->>settingsStore: For each key: userOverrides.add(key)
settingsStore->>settingsStore: saveConfig()
deactivate settingsStore
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,LS: 🔄 RESET
%% ═══════════════════════════════════════════════════════════════════════════
UI->>settingsStore: resetConfig()
activate settingsStore
settingsStore->>settingsStore: config = {...SETTING_CONFIG_DEFAULT}
settingsStore->>settingsStore: userOverrides.clear()
Note right of settingsStore: All params reset to defaults<br/>Next syncWithServerDefaults will adopt server values
settingsStore->>settingsStore: saveConfig()
deactivate settingsStore
UI->>settingsStore: resetParameterToServerDefault(key)
activate settingsStore
settingsStore->>settingsStore: userOverrides.delete(key)
settingsStore->>serverStore: defaultParams[key]
settingsStore->>settingsStore: config[key] = serverDefault
settingsStore->>settingsStore: saveConfig()
deactivate settingsStore
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,LS: 🎨 THEME
%% ═══════════════════════════════════════════════════════════════════════════
UI->>settingsStore: updateTheme(newTheme)
activate settingsStore
settingsStore->>settingsStore: theme = newTheme
settingsStore->>settingsStore: saveTheme()
settingsStore->>LS: set("llama-theme", theme)
deactivate settingsStore
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,LS: 📊 PARAMETER INFO
%% ═══════════════════════════════════════════════════════════════════════════
UI->>settingsStore: getParameterInfo(key)
settingsStore->>ParamSvc: getParameterInfo(key, config, serverDefaults, userOverrides)
ParamSvc-->>settingsStore: ParameterInfo
Note right of ParamSvc: {<br/> currentValue,<br/> serverDefault,<br/> isUserOverride: boolean,<br/> canSync: boolean,<br/> isDifferentFromServer: boolean<br/>}
UI->>settingsStore: getParameterDiff()
settingsStore->>ParamSvc: createParameterDiff(config, serverDefaults, userOverrides)
ParamSvc-->>settingsStore: ParameterDiff[]
Note right of ParamSvc: Array of parameters where user != server
%% ═══════════════════════════════════════════════════════════════════════════
Note over UI,LS: 📋 CONFIG CATEGORIES
%% ═══════════════════════════════════════════════════════════════════════════
Note over settingsStore: Syncable with server (from /props):
rect rgb(240, 255, 240)
Note over settingsStore: temperature, top_p, top_k, min_p<br/>repeat_penalty, presence_penalty, frequency_penalty<br/>dynatemp_range, dynatemp_exponent<br/>typ_p, xtc_probability, xtc_threshold<br/>dry_multiplier, dry_base, dry_allowed_length, dry_penalty_last_n
end
Note over settingsStore: UI-only (not synced):
rect rgb(255, 240, 240)
Note over settingsStore: systemMessage, custom (JSON)<br/>showStatistics, enableContinueGeneration<br/>autoMicOnEmpty, disableAutoScroll<br/>apiKey, pdfAsImage, disableReasoningParsing, showRawOutputSwitch
end
```
+83 -1
View File
@@ -12,6 +12,49 @@ import { fileURLToPath } from 'node:url';
import ts from 'typescript-eslint';
const gitignorePath = fileURLToPath(new URL('./.gitignore', import.meta.url));
// Require a blank line between consecutive class accessors (get/set). The core
// `padding-line-between-statements` rule only handles statements, not class
// members, so this is enforced with a small custom rule.
const blankLineBetweenAccessors = {
create(context) {
return {
MethodDefinition(node) {
if (node.kind !== 'get' && node.kind !== 'set') return;
const body = node.parent;
if (!body || body.type !== 'ClassBody') return;
const index = body.body.indexOf(node);
if (index <= 0) return;
const prev = body.body[index - 1];
if (prev.type !== 'MethodDefinition' || (prev.kind !== 'get' && prev.kind !== 'set'))
return;
if (node.loc.start.line - prev.loc.end.line <= 1) {
context.report({
fix(fixer) {
// Insert after the previous accessor's closing brace so the blank
// line keeps the current accessor's indentation.
return fixer.insertTextAfter(prev, '\n');
},
message: 'Expected a blank line between class accessors (get/set).',
node
});
}
}
};
},
meta: {
docs: { description: 'Require a blank line between consecutive class accessors (get/set).' },
fixable: 'whitespace',
schema: [],
type: 'layout'
}
};
export default ts.config(
includeIgnoreFile(gitignorePath),
@@ -22,7 +65,11 @@ export default ts.config(
...svelte.configs.prettier,
{
languageOptions: { globals: { ...globals.browser, ...globals.node } },
plugins: { perfectionist, 'simple-import-sort': simpleImportSort },
plugins: {
local: { rules: { 'blank-line-between-accessors': blankLineBetweenAccessors } },
perfectionist,
'simple-import-sort': simpleImportSort
},
rules: {
// Snippet bodies often ignore one or more of the parent's params
// (e.g. `{#snippet children(_meta, ctx)}` when only ctx is read).
@@ -30,8 +77,11 @@ export default ts.config(
'error',
{ argsIgnorePattern: '^_', varsIgnorePattern: '^_' }
],
// Enforce empty line at end of file
'eol-last': 'error',
// Enforce a blank line between consecutive get/set accessors
'local/blank-line-between-accessors': 'error',
// typescript-eslint strongly recommend that you do not use the no-undef lint rule on TypeScript projects.
// see: https://typescript-eslint.io/troubleshooting/faqs/eslint/#i-get-errors-from-the-no-undef-rule-about-global-variables-not-being-defined-even-though-there-are-no-typescript-errors
'no-undef': 'off',
@@ -61,6 +111,38 @@ export default ts.config(
{ blankLine: 'always', next: ['return', 'throw', 'break', 'continue'], prev: '*' }
],
// Class member order: public fields -> private fields -> constructor -> getters
// -> setters -> public methods -> private methods, alphabetical within each.
// Svelte $derived fields must stay in dependency order (forward references are
// rejected), so the two stores that rely on that are exempted below.
'perfectionist/sort-classes': [
'error',
{
customGroups: [
{ groupName: 'public-field', modifiers: ['public'], selector: 'property' },
{ groupName: 'private-field', modifiers: ['private'], selector: 'property' },
{ groupName: 'get-method', selector: 'get-method' },
{ groupName: 'set-method', selector: 'set-method' },
{ groupName: 'public-method', modifiers: ['public'], selector: 'method' },
{ groupName: 'private-method', modifiers: ['private'], selector: 'method' }
],
groups: [
'public-field',
'private-field',
'constructor',
'get-method',
'set-method',
'public-method',
'private-method',
'unknown'
],
type: 'natural',
// Keep members in dependency order (Svelte rejects forward references in
// $derived fields), while still sorting the rest alphabetically.
useExperimentalDependencyDetection: true
}
],
// Alphabetical order for enum members
'perfectionist/sort-enums': ['error', { type: 'natural' }],
@@ -139,7 +139,7 @@
let fileSize = $derived(currentItem?.size ? formatFileSize(currentItem.size) : '');
let hasVisionModality = $derived(
currentItem && activeModelId ? modelsStore.modelSupportsVision(activeModelId) : false
currentItem && activeModelId ? modelsStore.props.modelSupportsVision(activeModelId) : false
);
let audioSrc = $derived(
@@ -28,7 +28,6 @@
import {
chatStore,
conversationsStore,
mcpResourceStore,
mcpStore,
modelsStore,
serverStore,
@@ -140,7 +139,9 @@
// float above the box.
let mentionAnchor: HTMLDivElement | null = $state(null);
let cwd = $derived(conversationsStore.activeConversation?.cwd ?? conversationsStore.pendingCwd);
let cwd = $derived(
conversationsStore.activeConversation?.cwd ?? conversationsStore.preferences.pendingCwd
);
const pickers = useChatFormPickers({
focusInput: refocusInput,
@@ -151,7 +152,8 @@
getShowModelSelector: () => showModelSelector,
getValue: () => value,
hasCwdTools: () => toolsStore.hasEnabledCwdTools,
hasPrompts: () => mcpStore.hasPromptsCapability(conversationsStore.getAllMcpServerOverrides()),
hasPrompts: () =>
mcpStore.hasPromptsCapability(conversationsStore.preferences.getAllMcpServerOverrides()),
openModelSelector: () => chatFormActionsRef?.openModelSelector(),
setCaretOffset: (offset) => inputRef?.setCaretOffset(offset),
setValue: (v) => {
@@ -170,7 +172,7 @@
onValueChange?.('');
}
await conversationsStore.setCwd(newDir);
await conversationsStore.preferences.setCwd(newDir);
if (conversationsStore.activeConversation) {
await chatStore.recordCwdChange(newDir?.trim() || null);
@@ -595,7 +597,7 @@
{useRichInput}
/>
{#if mcpResourceStore.hasAttachments}
{#if mcpStore.resources.hasAttachments}
<ChatFormMcpResourcesList
class="mb-3"
onResourceClick={(uri) => {
@@ -38,11 +38,11 @@
}
function isServerEnabledForChat(serverId: string): boolean {
return conversationsStore.isMcpServerEnabledForChat(serverId);
return conversationsStore.preferences.isMcpServerEnabledForChat(serverId);
}
async function toggleServerForChat(serverId: string) {
await conversationsStore.toggleMcpServerForChat(serverId);
await conversationsStore.preferences.toggleMcpServerForChat(serverId);
}
function handleMcpSubMenuOpen(open: boolean) {
@@ -218,12 +218,15 @@
{@const hasError = healthState.status === HealthCheckStatus.ERROR}
{@const displayName = mcpStore.getServerLabel(server)}
{@const faviconUrl = mcpStore.getServerFavicon(server.id)}
{@const isEnabled = conversationsStore.isMcpServerEnabledForChat(server.id)}
{@const isEnabled = conversationsStore.preferences.isMcpServerEnabledForChat(
server.id
)}
<button
type="button"
class={sheetItemRowClass}
onclick={() => !hasError && conversationsStore.toggleMcpServerForChat(server.id)}
onclick={() =>
!hasError && conversationsStore.preferences.toggleMcpServerForChat(server.id)}
disabled={hasError}
>
<div class="flex min-w-0 flex-1 items-center gap-2">
@@ -250,7 +253,8 @@
{:else}
<Switch
checked={isEnabled}
onCheckedChange={() => conversationsStore.toggleMcpServerForChat(server.id)}
onCheckedChange={() =>
conversationsStore.preferences.toggleMcpServerForChat(server.id)}
/>
{/if}
</button>
@@ -81,10 +81,10 @@
$effect(() => {
if (activeModelId) {
const cached = modelsStore.getModelProps(activeModelId);
const cached = modelsStore.props.getModelProps(activeModelId);
if (!cached) {
modelsStore.fetchModelProps(activeModelId).then(() => {
modelsStore.props.fetchModelProps(activeModelId).then(() => {
modelPropsVersion++;
});
}
@@ -94,19 +94,21 @@
$effect(() => {
void modelPropsVersion;
hasAudioModality = activeModelId ? modelsStore.modelSupportsAudio(activeModelId) : false;
hasAudioModality = activeModelId ? modelsStore.props.modelSupportsAudio(activeModelId) : false;
});
$effect(() => {
void modelPropsVersion;
hasVideoModality = activeModelId ? modelsStore.modelSupportsVideo(activeModelId) : false;
hasVideoModality = activeModelId ? modelsStore.props.modelSupportsVideo(activeModelId) : false;
});
$effect(() => {
void modelPropsVersion;
hasVisionModality = activeModelId ? modelsStore.modelSupportsVision(activeModelId) : false;
hasVisionModality = activeModelId
? modelsStore.props.modelSupportsVision(activeModelId)
: false;
});
$effect(() => {
@@ -58,13 +58,13 @@
let currentConfig = $derived(settingsStore.config);
let hasMcpPromptsSupport = $derived.by(() => {
const perChatOverrides = conversationsStore.getAllMcpServerOverrides();
const perChatOverrides = conversationsStore.preferences.getAllMcpServerOverrides();
return mcpStore.hasPromptsCapability(perChatOverrides);
});
let hasMcpResourcesSupport = $derived.by(() => {
const perChatOverrides = conversationsStore.getAllMcpServerOverrides();
const perChatOverrides = conversationsStore.preferences.getAllMcpServerOverrides();
return mcpStore.hasResourcesCapability(perChatOverrides);
});
@@ -121,7 +121,7 @@
if (!chatStore.isLoading && !chatStore.isStreaming()) return false;
const processingState = chatStore.activeProcessingState;
const processingState = chatStore.processing.activeState;
if (!processingState) return false;
@@ -16,7 +16,7 @@
$effect(() => {
const conv = conversationsStore.activeConversation;
untrack(() => chatStore.setActiveProcessingConversation(conv?.id ?? null));
untrack(() => chatStore.processing.setActiveConversation(conv?.id ?? null));
});
$effect(() => {
@@ -28,12 +28,12 @@
if (chatStore.isLoading || chatStore.isStreaming()) return;
if (messages.length === 0) {
untrack(() => chatStore.clearProcessingState(conv.id));
untrack(() => chatStore.processing.setState(conv.id, null));
return;
}
untrack(() => chatStore.restoreProcessingStateFromMessages(messages, conv.id));
untrack(() => chatStore.processing.restoreFromMessages(messages, conv.id));
});
$effect(() => {
@@ -3,7 +3,7 @@
ChatAttachmentsListItemMcpResource,
HorizontalScrollCarousel
} from '$lib/components/app';
import { mcpResourceStore, mcpStore } from '$lib/stores';
import { mcpStore } from '$lib/stores';
interface Props {
class?: string;
@@ -12,8 +12,8 @@
let { class: className, onResourceClick }: Props = $props();
const attachments = $derived(mcpResourceStore.attachments);
const hasAttachments = $derived(mcpResourceStore.hasAttachments);
const attachments = $derived(mcpStore.resources.attachments);
const hasAttachments = $derived(mcpStore.resources.hasAttachments);
function handleRemove(attachmentId: string) {
mcpStore.removeResourceAttachment(attachmentId);
@@ -87,7 +87,7 @@
isLoading = true;
try {
const perChatOverrides = conversationsStore.getAllMcpServerOverrides();
const perChatOverrides = conversationsStore.preferences.getAllMcpServerOverrides();
const initialized = await mcpStore.ensureInitialized(perChatOverrides);
if (!initialized) {
@@ -59,7 +59,7 @@
message.model ?? chatStore.getResumeModel(message.convId) ?? modelsStore.selectedModelName
);
let modelLoadProgress = $derived(
isRouter && loadTargetModel ? modelsStore.getLoadProgress(loadTargetModel) : null
isRouter && loadTargetModel ? modelsStore.status.getLoadProgress(loadTargetModel) : null
);
let modelLoadingText = $derived(modelLoadProgressText(modelLoadProgress));
@@ -31,7 +31,7 @@
pendingModel = modelId;
try {
await modelsStore.loadModel(modelId);
await modelsStore.status.load(modelId);
} finally {
pendingModel = null;
}
@@ -43,14 +43,14 @@
);
const hasReasoningError = $derived(
isLastAssistantMessage ? !!agenticStore.lastError(message.convId) : false
isLastAssistantMessage ? !!agenticStore.getLastError(message.convId) : false
);
let permissionDismissed = $state(false);
const pendingPermission = $derived(
isStreaming && isLastAssistantMessage
? agenticStore.pendingPermissionRequest(message.convId)
? agenticStore.getPendingPermissionRequest(message.convId)
: null
);
@@ -74,7 +74,7 @@
const pendingContinue = $derived(
isStreaming && isLastAssistantMessage
? agenticStore.pendingContinueRequest(message.convId)
? agenticStore.getPendingContinueRequest(message.convId)
: false
);
@@ -97,7 +97,7 @@
const sections = $derived(deriveAgenticSections(message, toolMessages, [], isStreaming));
const currentlyExecutingToolCallId = $derived(
isStreaming ? agenticStore.executingToolCallId(message.convId) : null
isStreaming ? agenticStore.getExecutingToolCallId(message.convId) : null
);
type TurnGroup = {
@@ -238,30 +238,30 @@
/>
{/each}
{#if conversationsStore.activeConversation && agenticStore.pendingSteeringMessageContent(conversationsStore.activeConversation!.id)}
{#if conversationsStore.activeConversation && agenticStore.getPendingSteeringMessageContent(conversationsStore.activeConversation!.id)}
{@const convId = conversationsStore.activeConversation!.id}
{@const pendingContent = agenticStore.pendingSteeringMessageContent(convId)}
{@const pendingContent = agenticStore.getPendingSteeringMessageContent(convId)}
{#if pendingContent}
<ChatMessageUserPending
class="mx-auto mt-12 w-full max-w-[48rem]"
content={pendingContent}
extras={agenticStore.pendingSteeringMessageExtras(convId)}
extras={agenticStore.getPendingSteeringMessageExtras(convId)}
onSendImmediately={() => chatStore.abortCurrentFlow(convId)}
onEdit={(newContent, extras) =>
agenticStore.injectSteeringMessage(convId, newContent, extras)}
onDelete={() => agenticStore.clearSteeringMessage(convId)}
/>
{/if}
{:else if conversationsStore.activeConversation && chatStore.pendingMessageContent(conversationsStore.activeConversation!.id)}
{:else if conversationsStore.activeConversation && chatStore.getPendingMessageContent(conversationsStore.activeConversation!.id)}
{@const convId = conversationsStore.activeConversation!.id}
{@const pendingContent = chatStore.pendingMessageContent(convId)}
{@const pendingContent = chatStore.getPendingMessageContent(convId)}
{#if pendingContent}
<ChatMessageUserPending
class="mx-auto mt-12 w-full max-w-[48rem]"
content={pendingContent}
extras={chatStore.pendingMessageExtras(convId)}
extras={chatStore.getPendingMessageExtras(convId)}
onSendImmediately={() => chatStore.abortCurrentFlow(convId)}
onEdit={(newContent, extras) => chatStore.injectPendingMessage(convId, newContent, extras)}
onDelete={() => chatStore.clearPendingMessage(convId)}
@@ -8,7 +8,7 @@
import { Button } from '$lib/components/ui/button';
import * as Dialog from '$lib/components/ui/dialog';
import { ICON_CLASS_DEFAULT } from '$lib/constants';
import { conversationsStore, mcpResourceStore, mcpStore } from '$lib/stores';
import { conversationsStore, mcpStore } from '$lib/stores';
import type { MCPResourceContent, MCPResourceInfo, MCPResourceTemplateInfo } from '$lib/types';
import { getResourceDisplayName } from '$lib/utils';
import { SvelteSet } from 'svelte/reactivity';
@@ -33,7 +33,7 @@
let templatePreviewLoading = $state(false);
let templatePreviewError = $state<string | null>(null);
const totalCount = $derived(mcpResourceStore.totalResourceCount);
const totalCount = $derived(mcpStore.resources.totalResourceCount);
$effect(() => {
if (open) {
@@ -48,7 +48,7 @@
});
async function loadResources() {
const perChatOverrides = conversationsStore.getAllMcpServerOverrides();
const perChatOverrides = conversationsStore.preferences.getAllMcpServerOverrides();
const initialized = await mcpStore.ensureInitialized(perChatOverrides);
if (initialized) {
@@ -126,16 +126,16 @@
isAttaching = true;
try {
const knownResource = mcpResourceStore.findResourceByUri(templatePreviewUri);
const knownResource = mcpStore.resources.findResourceByUri(templatePreviewUri);
if (knownResource) {
if (!mcpResourceStore.isAttached(knownResource.uri)) {
if (!mcpStore.resources.isAttached(knownResource.uri)) {
await mcpStore.attachResource(knownResource.uri);
}
toast.success(`Resource attached: ${knownResource.title || knownResource.name}`);
} else {
if (mcpResourceStore.isAttached(templatePreviewUri)) {
if (mcpStore.resources.isAttached(templatePreviewUri)) {
toast.info('Resource already attached');
handleOpenChange(false);
@@ -147,9 +147,9 @@
serverName: selectedTemplate.serverName,
uri: templatePreviewUri
};
const attachment = mcpResourceStore.addAttachment(resourceInfo);
const attachment = mcpStore.resources.addAttachment(resourceInfo);
mcpResourceStore.updateAttachmentContent(attachment.id, templatePreviewContent);
mcpStore.resources.updateAttachmentContent(attachment.id, templatePreviewContent);
toast.success(`Resource attached: ${resourceInfo.name}`);
}
@@ -199,7 +199,7 @@
function getAllResourcesFlatInTreeOrder(): MCPResourceInfo[] {
const allResources: MCPResourceInfo[] = [];
const resourcesMap = mcpResourceStore.serverResources;
const resourcesMap = mcpStore.resources.serverResources;
for (const [serverName, serverRes] of resourcesMap.entries()) {
for (const resource of serverRes.resources) {
@@ -234,7 +234,7 @@
useProxy: newServerUseProxy
});
conversationsStore.setMcpServerOverride(newServerId, true);
conversationsStore.preferences.setMcpServerOverride(newServerId, true);
handleOpenChange(false);
}
@@ -42,7 +42,7 @@
let modalities = $derived.by(() => {
if (!firstModel?.id) return [];
return modelsStore.getModelModalitiesArray(firstModel.id);
return modelsStore.props.getModelModalitiesArray(firstModel.id);
});
// Ensure models are fetched when dialog opens
@@ -56,7 +56,7 @@
$effect(() => {
if (open && isRouter && modelId) {
isLoadingRouterProps = true;
modelsStore
modelsStore.props
.fetchModelProps(modelId)
.then((props) => {
routerModelProps = props;
@@ -14,7 +14,9 @@
let mcpServers = $derived(mcpStore.getServers().filter((s) => s.enabled));
let enabledMcpServersForChat = $derived(
mcpServers.filter((s) => conversationsStore.isMcpServerEnabledForChat(s.id) && s.url.trim())
mcpServers.filter(
(s) => conversationsStore.preferences.isMcpServerEnabledForChat(s.id) && s.url.trim()
)
);
let healthyEnabledMcpServers = $derived(
enabledMcpServersForChat.filter((s) => {
@@ -2,7 +2,7 @@
import McpResourcesBrowserEmptyState from './McpResourcesBrowserEmptyState.svelte';
import McpResourcesBrowserHeader from './McpResourcesBrowserHeader.svelte';
import McpResourcesBrowserServerItem from './McpResourcesBrowserServerItem.svelte';
import { mcpResourceStore, mcpStore } from '$lib/stores';
import { mcpStore } from '$lib/stores';
import type { MCPResourceInfo, MCPResourceTemplateInfo, MCPServerResources } from '$lib/types';
import { parseResourcePath } from '$lib/utils';
import { SvelteMap, SvelteSet } from 'svelte/reactivity';
@@ -31,8 +31,8 @@
let expandedFolders = new SvelteSet<string>();
let searchQuery = $state('');
const resources = $derived(mcpResourceStore.serverResources);
const isLoading = $derived(mcpResourceStore.isLoading);
const resources = $derived(mcpStore.resources.serverResources);
const isLoading = $derived(mcpStore.resources.isLoading);
const filteredResources = $derived.by(() => {
if (!searchQuery.trim()) {
@@ -116,7 +116,7 @@
if (status === ServerModelStatus.LOADING) return;
await modelsStore.unloadModel(modelId);
await modelsStore.status.unload(modelId);
}
export function open() {
@@ -174,9 +174,9 @@
{@const triggerLoading =
!!triggerModel &&
(triggerStatus === ServerModelStatus.LOADING ||
modelsStore.isModelOperationInProgress(triggerModel))}
modelsStore.status.isOperationInProgress(triggerModel))}
{@const triggerLoadPercent = triggerLoading
? Math.round(modelLoadFraction(modelsStore.getLoadProgress(triggerModel)) * 100)
? Math.round(modelLoadFraction(modelsStore.status.getLoadProgress(triggerModel)) * 100)
: 0}
{#if ms.isRouter}
@@ -47,7 +47,7 @@
return (model?.status?.value as ServerModelStatus) ?? null;
});
let isOperationInProgress = $derived(modelsStore.isModelOperationInProgress(option.model));
let isOperationInProgress = $derived(modelsStore.status.isOperationInProgress(option.model));
let isFailed = $derived(serverStatus === ServerModelStatus.FAILED);
let isSleeping = $derived(serverStatus === ServerModelStatus.SLEEPING);
let isLoaded = $derived(
@@ -55,7 +55,7 @@
);
let isLoading = $derived(serverStatus === ServerModelStatus.LOADING || isOperationInProgress);
let loadProgress = $derived(isLoading ? modelsStore.getLoadProgress(option.model) : null);
let loadProgress = $derived(isLoading ? modelsStore.status.getLoadProgress(option.model) : null);
let loadPercent = $derived(Math.round(modelLoadFraction(loadProgress) * 100));
let loadTitle = $derived(modelLoadProgressText(loadProgress));
</script>
@@ -138,7 +138,7 @@
icon={RotateCw}
tooltip="Retry loading model"
class="h-3 w-3 text-red-500 hover:text-foreground"
onclick={() => modelsStore.loadModel(option.model)}
onclick={() => modelsStore.status.load(option.model)}
stopPropagationOnClick
/>
</div>
@@ -157,7 +157,7 @@
class="h-3 w-3 text-red-500 hover:text-red-600 [@media(pointer:coarse)]:text-amber-500 [@media(pointer:coarse)]:hover:text-amber-600"
onclick={(e) => {
e?.stopPropagation();
modelsStore.unloadModel(option.model);
modelsStore.status.unload(option.model);
}}
/>
</div>
@@ -174,7 +174,7 @@
icon={PowerOff}
tooltip="Unload model"
class="h-3 w-3 text-red-500 hover:text-red-600 [@media(pointer:coarse)]:text-green-500 [@media(pointer:coarse)]:hover:text-green-600"
onclick={() => modelsStore.unloadModel(option.model)}
onclick={() => modelsStore.status.unload(option.model)}
stopPropagationOnClick
/>
</div>
@@ -191,7 +191,7 @@
icon={Power}
tooltip="Load model"
class="h-3 w-3 [@media(pointer:coarse)]:text-muted-foreground"
onclick={() => modelsStore.loadModel(option.model)}
onclick={() => modelsStore.status.load(option.model)}
stopPropagationOnClick
/>
</div>
@@ -72,9 +72,9 @@
{@const triggerLoading =
!!triggerModel &&
(triggerStatus === ServerModelStatus.LOADING ||
modelsStore.isModelOperationInProgress(triggerModel))}
modelsStore.status.isOperationInProgress(triggerModel))}
{@const triggerLoadPercent = triggerLoading
? Math.round(modelLoadFraction(modelsStore.getLoadProgress(triggerModel)) * 100)
? Math.round(modelLoadFraction(modelsStore.status.getLoadProgress(triggerModel)) * 100)
: 0}
{#if ms.isRouter}
@@ -15,7 +15,7 @@
NUMERIC_FIELDS,
POSITIVE_INTEGER_FIELDS,
SETTINGS_CHAT_SECTIONS,
SETTINGS_SECTION_TITLES
SETTINGS_SECTION_SLUGS
} from '$lib/constants';
import { ColorMode } from '$lib/enums/ui.enums';
import { RouterService } from '$lib/services/router.service';
@@ -46,13 +46,13 @@
let fetchInitiated = false;
$effect(() => {
if (serverStore.isRouterMode && currentSection.fields && !fetchInitiated) {
if (serverStore.isRouterMode && currentSection.fields?.length && !fetchInitiated) {
fetchInitiated = true;
void modelsStore
.fetch()
.then(() => modelsStore.fetchRouterModels())
.then(() => modelsStore.fetchModalitiesForLoadedModels())
.then(() => modelsStore.props.fetchModalitiesForLoadedModels())
.then(() => modelsStore.ensureFirstModelSelected());
}
});
@@ -148,9 +148,9 @@
<h3 class="text-lg font-semibold">{currentSection.title}</h3>
</div>
{#if currentSection.title === SETTINGS_SECTION_TITLES.TOOLS}
{#if currentSection.slug === SETTINGS_SECTION_SLUGS.TOOLS}
<SettingsChatToolsTab />
{:else if currentSection.title === SETTINGS_SECTION_TITLES.IMPORT_EXPORT}
{:else if currentSection.slug === SETTINGS_SECTION_SLUGS.IMPORT_EXPORT}
<SettingsChatImportExportTab />
{:else if currentSection.fields}
<div class="space-y-6">
@@ -161,7 +161,7 @@
onThemeChange={handleThemeChange}
/>
{#if currentSection.title === SETTINGS_SECTION_TITLES.GENERAL}
{#if currentSection.slug === SETTINGS_SECTION_SLUGS.GENERAL}
<div class="flex justify-end">
<Button variant="outline" onclick={() => window.location.reload()}>
<RefreshCw class="h-3 w-3" />
@@ -23,13 +23,13 @@
let { fields, localConfig, onConfigChange, onThemeChange }: Props = $props();
let currentModelParams = $derived.by(() => {
void modelsStore.propsCacheVersion;
void modelsStore.props.cacheVersion;
if (serverStore.isRouterMode) {
const currentModelName = modelsStore.selectedModelName;
if (currentModelName) {
const currentModelProps = modelsStore.getModelProps(currentModelName);
const currentModelProps = modelsStore.props.getModelProps(currentModelName);
return (currentModelProps?.default_generation_settings?.params ?? {}) as Record<
string,
@@ -121,11 +121,13 @@
{:else}
<McpServerCard
{server}
enabled={conversationsStore.isMcpServerEnabledForChat(server.id)}
enabled={conversationsStore.preferences.isMcpServerEnabledForChat(server.id)}
onToggle={async () => {
const wasEnabled = conversationsStore.isMcpServerEnabledForChat(server.id);
const wasEnabled = conversationsStore.preferences.isMcpServerEnabledForChat(
server.id
);
await conversationsStore.toggleMcpServerForChat(server.id);
await conversationsStore.preferences.toggleMcpServerForChat(server.id);
if (!wasEnabled) {
// Promote the connection so tools/prompts/resources become
@@ -74,7 +74,7 @@ export const ATTACHMENT_PROMPT_ITEMS: AttachmentMenuItem[] = [
enabledWhen: AttachmentItemEnabledWhen.ALWAYS,
icon: Zap,
id: AttachmentMenuItemId.MCP_PROMPT,
label: 'MCP Prompt',
label: 'MCP Prompts',
visibleWhen: AttachmentItemVisibleWhen.HAS_MCP_PROMPTS_SUPPORT
}
];
@@ -32,13 +32,3 @@ export const MCP_RESOURCE_CACHE = {
/** TTL for MCP resource cache entries in milliseconds (5 minutes) */
TTL_MS: 5 * 60 * 1000
} as const;
/**
* Limits for pruning inactive conversation states held in memory.
*/
export const INACTIVE_CONVERSATION = {
/** Maximum age (in ms) for inactive conversation states before cleanup (30 minutes) */
MAX_AGE_MS: 30 * 60 * 1000,
/** Maximum number of inactive conversation states to keep in memory */
MAX_STATES: 10
} as const;
+1 -1
View File
@@ -48,7 +48,7 @@ export * from './pwa.constants';
export * from './routes.constants';
export * from './sandbox.constants';
export * from './settings-keys.constants';
export * from './settings-registry.constants';
export * from './settings.constants';
export * from './special-characters.constants';
export * from './stream.constants';
export * from './supported-file-types.constants';
+2 -12
View File
@@ -10,18 +10,6 @@ export const URL_PARAMS = {
QUERY: 'q'
} as const;
/** Settings section slugs — used for routes and navigation. */
export const SETTINGS_SECTION_SLUGS = {
AGENTIC: 'agentic',
DEVELOPER: 'developer',
DISPLAY: 'display',
GENERAL: 'general',
IMPORT_EXPORT: 'import-export',
PENALTIES: 'penalties',
SAMPLING: 'sampling',
TOOLS: 'tools'
} as const;
export const ROUTES = {
/** Chat base — for dynamic chat URLs use RouterService. */
CHAT: '#/chat',
@@ -33,6 +21,8 @@ export const ROUTES = {
SEARCH: '#/search',
/** Settings base — for dynamic settings URLs use RouterService. */
SETTINGS: '#/settings',
/** Exit destination for the settings view (fallback when no referrer). */
SETTINGS_EXIT: '#/',
/** Root — start of the app. */
START: '#/'
} as const;
@@ -1,3 +1,5 @@
import { UrlProtocol } from '$lib/enums';
const STD = ['com', 'net', 'org', 'gov', 'edu'] as const;
const STD_MIL = [...STD, 'mil'] as const;
const ccTLD_PREFIXES: Record<string, readonly string[]> = {
@@ -184,3 +186,7 @@ export const WILDCARD_PUBLIC_SUFFIXES = buildSuffixSet(WILDCARD_BASES);
// Matches one or more trailing "/" characters at the end of a URL/path.
export const TRAILING_SLASHES_REGEX = /\/+$/;
// Protocols that apiFetch treats as absolute and passes through untouched.
// Add a protocol here when a caller needs to fetch an absolute URL with it.
export const API_ABSOLUTE_URL_PROTOCOLS = [UrlProtocol.HTTP, UrlProtocol.HTTPS] as const;
@@ -14,18 +14,14 @@ export interface AutoScrollOptions {
*/
export class AutoScrollController {
private _autoScrollEnabled = $state(true);
private _userScrolledUp = $state(false);
private _lastScrollTop = $state(0);
private _scrollInterval: ReturnType<typeof setInterval> | undefined;
private _container: HTMLElement | undefined;
private _disabled: boolean;
private _lastScrollTop = $state(0);
private _mutationObserver: MutationObserver | null = null;
private _rafPending = false;
private _observerEnabled = false;
constructor(options: AutoScrollOptions = {}) {
this._disabled = options.disabled ?? false;
}
private _rafPending = false;
private _scrollInterval: ReturnType<typeof setInterval> | undefined;
private _userScrolledUp = $state(false);
get autoScrollEnabled(): boolean {
return this._autoScrollEnabled;
}
@@ -34,6 +30,71 @@ export class AutoScrollController {
return this._userScrolledUp;
}
constructor(options: AutoScrollOptions = {}) {
this._disabled = options.disabled ?? false;
}
/**
* Cleans up resources. Call this in onDestroy or when the component unmounts.
*/
destroy(): void {
this.stopInterval();
this._doStopObserving();
}
/**
* Enables auto-scroll (e.g., when user sends a message).
*/
enable(): void {
if (this._disabled) return;
this._userScrolledUp = false;
this._autoScrollEnabled = true;
}
/**
* Handles scroll events to detect user scroll direction and toggle auto-scroll.
*/
handleScroll(): void {
if (this._disabled || !this._container) return;
const { clientHeight, scrollHeight, scrollTop } = this._container;
const distanceFromBottom = scrollHeight - clientHeight - scrollTop;
const isScrollingUp = scrollTop < this._lastScrollTop;
const isAtBottom = distanceFromBottom < AUTO_SCROLL_AT_BOTTOM_THRESHOLD;
if (isScrollingUp && !isAtBottom) {
this._userScrolledUp = true;
this._autoScrollEnabled = false;
} else if (isAtBottom && this._userScrolledUp) {
this._userScrolledUp = false;
this._autoScrollEnabled = true;
}
this._lastScrollTop = scrollTop;
}
/**
* Resets scroll state when switching conversations.
*/
resetScrollState(): void {
this._userScrolledUp = false;
this._autoScrollEnabled = !this._disabled;
if (this._container) {
this._lastScrollTop = this._container.scrollTop;
}
}
/**
* Scrolls the container to the bottom instantly.
*/
scrollToBottom(): void {
if (this._disabled || !this._container) return;
this._container.scrollTop = this._container.scrollHeight;
}
/**
* Binds the controller to a scrollable container element.
*/
@@ -63,59 +124,6 @@ export class AutoScrollController {
}
}
/**
* Handles scroll events to detect user scroll direction and toggle auto-scroll.
*/
handleScroll(): void {
if (this._disabled || !this._container) return;
const { clientHeight, scrollHeight, scrollTop } = this._container;
const distanceFromBottom = scrollHeight - clientHeight - scrollTop;
const isScrollingUp = scrollTop < this._lastScrollTop;
const isAtBottom = distanceFromBottom < AUTO_SCROLL_AT_BOTTOM_THRESHOLD;
if (isScrollingUp && !isAtBottom) {
this._userScrolledUp = true;
this._autoScrollEnabled = false;
} else if (isAtBottom && this._userScrolledUp) {
this._userScrolledUp = false;
this._autoScrollEnabled = true;
}
this._lastScrollTop = scrollTop;
}
/**
* Scrolls the container to the bottom instantly.
*/
scrollToBottom(): void {
if (this._disabled || !this._container) return;
this._container.scrollTop = this._container.scrollHeight;
}
/**
* Enables auto-scroll (e.g., when user sends a message).
*/
enable(): void {
if (this._disabled) return;
this._userScrolledUp = false;
this._autoScrollEnabled = true;
}
/**
* Resets scroll state when switching conversations.
*/
resetScrollState(): void {
this._userScrolledUp = false;
this._autoScrollEnabled = !this._disabled;
if (this._container) {
this._lastScrollTop = this._container.scrollTop;
}
}
/**
* Starts the auto-scroll interval for continuous scrolling during streaming.
*/
@@ -127,6 +135,18 @@ export class AutoScrollController {
}, AUTO_SCROLL_INTERVAL);
}
/**
* Starts a MutationObserver on the container that auto-scrolls to bottom
* on content changes. More responsive than interval-based polling.
*/
startObserving(): void {
this._observerEnabled = true;
if (this._container && !this._disabled && !this._mutationObserver) {
this._doStartObserving();
}
}
/**
* Stops the auto-scroll interval.
*/
@@ -137,6 +157,14 @@ export class AutoScrollController {
}
}
/**
* Stops the MutationObserver.
*/
stopObserving(): void {
this._observerEnabled = false;
this._doStopObserving();
}
/**
* Updates the auto-scroll interval based on streaming state.
* Call this in a $effect to automatically manage the interval.
@@ -157,34 +185,6 @@ export class AutoScrollController {
}
}
/**
* Cleans up resources. Call this in onDestroy or when the component unmounts.
*/
destroy(): void {
this.stopInterval();
this._doStopObserving();
}
/**
* Starts a MutationObserver on the container that auto-scrolls to bottom
* on content changes. More responsive than interval-based polling.
*/
startObserving(): void {
this._observerEnabled = true;
if (this._container && !this._disabled && !this._mutationObserver) {
this._doStartObserving();
}
}
/**
* Stops the MutationObserver.
*/
stopObserving(): void {
this._observerEnabled = false;
this._doStopObserving();
}
private _doStartObserving(): void {
if (!this._container || this._mutationObserver) return;
@@ -22,10 +22,10 @@ export function useChatScreenActiveModel() {
$effect(() => {
if (activeModelId) {
const cached = modelsStore.getModelProps(activeModelId);
const cached = modelsStore.props.getModelProps(activeModelId);
if (!cached) {
modelsStore.fetchModelProps(activeModelId).then(() => {
modelsStore.props.fetchModelProps(activeModelId).then(() => {
modelPropsVersion++;
});
}
@@ -36,7 +36,7 @@ export function useChatScreenActiveModel() {
if (activeModelId) {
void modelPropsVersion;
return modelsStore.modelSupportsAudio(activeModelId);
return modelsStore.props.modelSupportsAudio(activeModelId);
}
return false;
@@ -45,7 +45,7 @@ export function useChatScreenActiveModel() {
if (activeModelId) {
void modelPropsVersion;
return modelsStore.modelSupportsVideo(activeModelId);
return modelsStore.props.modelSupportsVideo(activeModelId);
}
return false;
@@ -54,7 +54,7 @@ export function useChatScreenActiveModel() {
if (activeModelId) {
void modelPropsVersion;
return modelsStore.modelSupportsVision(activeModelId);
return modelsStore.props.modelSupportsVision(activeModelId);
}
return false;
@@ -54,10 +54,10 @@ export function useContextGauge(): UseContextGaugeReturn {
const modelId = contextStatsStore.activeModelId;
if (modelId && contextStatsStore.isActiveModelLoaded) {
const cached = modelsStore.getModelProps(modelId);
const cached = modelsStore.props.getModelProps(modelId);
if (!cached) {
void modelsStore.fetchModelProps(modelId);
void modelsStore.props.fetchModelProps(modelId);
}
}
});
@@ -80,9 +80,9 @@ export function useContextGauge(): UseContextGaugeReturn {
if (!modelId || contextStatsStore.isActiveModelLoading) return;
try {
await modelsStore.loadModel(modelId);
await modelsStore.status.load(modelId);
} catch {
// toast already surfaced by modelsStore.loadModel
// toast already surfaced by modelsStore.status.load
}
}
@@ -47,7 +47,7 @@ export interface UseModelsSelectorReturn {
export function useModelsSelector(opts: UseModelsSelectorOptions): UseModelsSelectorReturn {
const options = $derived(
modelsStore.models.filter((option) => {
const modelProps = modelsStore.getModelProps(option.model);
const modelProps = modelsStore.props.getModelProps(option.model);
return modelProps?.ui !== false;
})
@@ -103,7 +103,7 @@ export function useModelsSelector(opts: UseModelsSelectorOptions): UseModelsSele
if (open) {
modelsStore.fetchRouterModels().then(() => {
modelsStore.fetchModalitiesForLoadedModels();
modelsStore.props.fetchModalitiesForLoadedModels();
});
}
@@ -143,8 +143,8 @@ export function useModelsSelector(opts: UseModelsSelectorOptions): UseModelsSele
if (!onModelChange && isRouter && !modelsStore.isModelLoaded(option.model)) {
isLoadingModel = true;
modelsStore
.loadModel(option.model)
modelsStore.status
.load(option.model)
.catch((error) => console.error('Failed to load model:', error))
.finally(() => (isLoadingModel = false));
}
@@ -43,7 +43,7 @@ export function useProcessingState(): UseProcessingStateReturn {
}
// Read directly from the reactive state
return chatStore.activeProcessingState;
return chatStore.processing.activeState;
});
$effect(() => {
@@ -42,19 +42,20 @@ export function useReasoningMenu(): UseReasoningMenuReturn {
});
const modelSupportsThinking = $derived.by(() => {
void modelsStore.loadedModelIds;
void modelsStore.propsCacheVersion;
void modelsStore.props.cacheVersion;
if (serverStore.isRouterMode) {
const modelId = modelsStore.selectedModelName || conversationModel;
return (
modelsStore.checkModelSupportsThinking(modelId ?? '') || modelSupportsThinkingFromMessages
modelsStore.props.checkModelSupportsThinking(modelId ?? '') ||
modelSupportsThinkingFromMessages
);
}
return modelsStore.supportsThinking || modelSupportsThinkingFromMessages;
return modelsStore.props.supportsThinking || modelSupportsThinkingFromMessages;
});
const currentEffort = $derived(conversationsStore.getReasoningEffort());
const currentEffort = $derived(conversationsStore.preferences.getReasoningEffort());
const thinkingEnabled = $derived(
currentEffort !== ReasoningEffort.OFF && currentEffort !== ReasoningEffort.DEFAULT
);
@@ -76,7 +77,7 @@ export function useReasoningMenu(): UseReasoningMenuReturn {
return modelSupportsThinking;
},
select(level: ReasoningEffortLevel): void {
conversationsStore.setReasoningEffort(level.value as ReasoningEffort);
conversationsStore.preferences.setReasoningEffort(level.value as ReasoningEffort);
},
get thinkingEnabled() {
return thinkingEnabled;
@@ -35,7 +35,7 @@ export function useToolsPanel(): UseToolsPanelReturn {
(g) =>
g.source !== ToolSource.MCP ||
!g.serverId ||
conversationsStore.isMcpServerEnabledForChat(g.serverId)
conversationsStore.preferences.isMcpServerEnabledForChat(g.serverId)
)
);
const totalToolCount = $derived(activeGroups.reduce((n, g) => n + g.tools.length, 0));
@@ -73,7 +73,7 @@ export function useToolsPanel(): UseToolsPanelReturn {
return (
group.source === ToolSource.MCP &&
!!group.serverId &&
!conversationsStore.isMcpServerEnabledForChat(group.serverId)
!conversationsStore.preferences.isMcpServerEnabledForChat(group.serverId)
);
}
File diff suppressed because it is too large Load Diff
@@ -16,187 +16,6 @@ import {
import { strFromU8, strToU8, unzipSync, zipSync } from 'fflate';
export class ConversationTransferService {
/**
*
*
* JSONL Session Format
*
*
*/
/**
* Serializes a session (a conversation with its messages) as JSONL.
* The first line is the session header (a `SessionRecordType.SESSION` record
* carrying the conversation properties); each subsequent line is a single message.
* @param data - The exported conversation payload
* @returns The JSONL string (one record per line)
*/
static serializeSessionToJsonl(data: ExportedConversation): string {
const { conv, messages } = data;
const sessionLine = JSON.stringify({
harness: EXPORT_CONV.HARNESS,
type: SessionRecordType.SESSION,
...conv
});
const messageLines = messages.map((message: DatabaseMessage) => {
// `toolCalls` is stored as a JSON string; drop it when empty, otherwise parse it.
const { toolCalls, ...rest } = message;
const normalized = toolCalls ? { ...rest, toolCalls: JSON.parse(toolCalls) } : rest;
return JSON.stringify({ message: normalized, type: SessionRecordType.MESSAGE });
});
return [sessionLine, ...messageLines].join(NEWLINE);
}
/**
* Parses the JSONL session format produced by {@link serializeSessionToJsonl}.
* A `SessionRecordType.SESSION` line starts a new session; following
* `SessionRecordType.MESSAGE` lines are appended to it. Supports multiple
* sessions in a single file.
* @param text - The JSONL file contents
* @returns The parsed conversations with their messages
*/
static parseSessionsJsonl(text: string): ExportedConversation[] {
const sessions: ExportedConversation[] = [];
let current: ExportedConversation | null = null;
for (const line of text.split(NEWLINE)) {
const trimmed = line.trim();
if (!trimmed) continue;
const record = JSON.parse(trimmed);
if (record.type === SessionRecordType.SESSION) {
// Drop the discriminator and harness marker; the rest is the conversation.
const conv = { ...record };
delete conv.type;
delete conv.harness;
current = { conv: conv as DatabaseConversation, messages: [] };
sessions.push(current);
} else if (record.type === SessionRecordType.MESSAGE) {
if (!current) {
throw new Error('Invalid JSONL: message record before any session record');
}
const message = record.message as DatabaseMessage;
// `toolCalls` is parsed to an array on export; the DB stores it as a string.
if (message.toolCalls !== undefined && typeof message.toolCalls !== 'string') {
message.toolCalls = JSON.stringify(message.toolCalls);
}
current.messages.push(message);
}
// Ignore unknown record types for forward compatibility.
}
return sessions;
}
/**
* Reports whether the text is the JSONL session format, whose first non-empty
* line is a `SessionRecordType.SESSION` record. A legacy JSON export starts
* with an array or an object that has no such discriminator.
* @param text - The file contents
*/
private static isSessionsJsonl(text: string): boolean {
const trimmed = text.trimStart();
const lineEnd = trimmed.indexOf(NEWLINE);
const firstLine = lineEnd === -1 ? trimmed : trimmed.slice(0, lineEnd);
try {
return JSON.parse(firstLine).type === SessionRecordType.SESSION;
} catch {
// Not a standalone JSON record, so not the JSONL format.
return false;
}
}
/**
* Parses an import file into conversations, accepting the current JSONL and
* ZIP formats as well as the legacy JSON format. The format comes from the
* contents, so an import works whatever the file is named.
* @param file - The user-selected file
* @returns The parsed conversations with their messages
*/
static async parseImportFile(file: File): Promise<ExportedConversation[]> {
const bytes = new Uint8Array(await file.arrayBuffer());
if (ZIP_MAGIC.every((byte, index) => bytes[index] === byte)) {
const entries = unzipSync(bytes);
const sessions: ExportedConversation[] = [];
for (const [entryName, entryBytes] of Object.entries(entries)) {
if (!entryName.toLowerCase().endsWith(FileExtensionText.JSONL)) continue;
sessions.push(...ConversationTransferService.parseSessionsJsonl(strFromU8(entryBytes)));
}
return sessions;
}
const text = strFromU8(bytes);
if (ConversationTransferService.isSessionsJsonl(text)) {
return ConversationTransferService.parseSessionsJsonl(text);
}
// Legacy JSON format: an array of conversations or a single conversation object.
const parsed = JSON.parse(text);
if (Array.isArray(parsed)) {
return parsed;
}
if (parsed && typeof parsed === 'object' && 'conv' in parsed && 'messages' in parsed) {
return [parsed];
}
throw new Error(
'Invalid file format: expected array of conversations or single conversation object'
);
}
/**
*
*
* Downloads
*
*
*/
/**
* Generates a sanitized filename for a conversation export
* @param conversation - The conversation metadata
* @param msgs - Optional array of messages belonging to the conversation
* @returns The generated filename string
*/
static generateConversationFilename(
conversation: { id?: string; name?: string },
msgs?: DatabaseMessage[]
): string {
const conversationName = (conversation.name ?? '').trim().toLowerCase();
const sanitizedName = conversationName
.replace(EXPORT_CONV.NON_ALPHANUMERIC_REGEX, EXPORT_CONV.NONALNUM_REPLACEMENT)
.replace(EXPORT_CONV.MULTIPLE_UNDERSCORE_REGEX, '_')
.substring(0, EXPORT_CONV.NAME_SUFFIX_MAX_LENGTH);
// If we have messages, use the timestamp of the newest message
const referenceDate = msgs?.length
? new Date(Math.max(...msgs.map((m) => m.timestamp)))
: new Date();
const iso = referenceDate.toISOString().slice(0, EXPORT_CONV.ISO_TIMESTAMP_SLICE);
const formattedDate = iso
.replace(EXPORT_CONV.ISO_DATE_TIME_SEPARATOR, EXPORT_CONV.ISO_DATE_TIME_SEPARATOR_REPLACEMENT)
.replaceAll(EXPORT_CONV.ISO_TIME_SEPARATOR, EXPORT_CONV.ISO_TIME_SEPARATOR_REPLACEMENT);
const trimmedConvId = conversation.id?.slice(0, EXPORT_CONV.ID_TRIM_LENGTH) ?? '';
return `${formattedDate}_conv_${trimmedConvId}_${sanitizedName}${FileExtensionText.JSONL}`;
}
/**
* Triggers a browser download of the provided exported conversation data
* @param data - The exported conversation payload (a single conversation with its messages)
@@ -262,6 +81,171 @@ export class ConversationTransferService {
ConversationTransferService.triggerDownload(blob, archiveName);
}
/**
* Generates a sanitized filename for a conversation export
* @param conversation - The conversation metadata
* @param msgs - Optional array of messages belonging to the conversation
* @returns The generated filename string
*/
static generateConversationFilename(
conversation: { id?: string; name?: string },
msgs?: DatabaseMessage[]
): string {
const conversationName = (conversation.name ?? '').trim().toLowerCase();
const sanitizedName = conversationName
.replace(EXPORT_CONV.NON_ALPHANUMERIC_REGEX, EXPORT_CONV.NONALNUM_REPLACEMENT)
.replace(EXPORT_CONV.MULTIPLE_UNDERSCORE_REGEX, '_')
.substring(0, EXPORT_CONV.NAME_SUFFIX_MAX_LENGTH);
// If we have messages, use the timestamp of the newest message
const referenceDate = msgs?.length
? new Date(Math.max(...msgs.map((m) => m.timestamp)))
: new Date();
const iso = referenceDate.toISOString().slice(0, EXPORT_CONV.ISO_TIMESTAMP_SLICE);
const formattedDate = iso
.replace(EXPORT_CONV.ISO_DATE_TIME_SEPARATOR, EXPORT_CONV.ISO_DATE_TIME_SEPARATOR_REPLACEMENT)
.replaceAll(EXPORT_CONV.ISO_TIME_SEPARATOR, EXPORT_CONV.ISO_TIME_SEPARATOR_REPLACEMENT);
const trimmedConvId = conversation.id?.slice(0, EXPORT_CONV.ID_TRIM_LENGTH) ?? '';
return `${formattedDate}_conv_${trimmedConvId}_${sanitizedName}${FileExtensionText.JSONL}`;
}
/**
* Parses an import file into conversations, accepting the current JSONL and
* ZIP formats as well as the legacy JSON format. The format comes from the
* contents, so an import works whatever the file is named.
* @param file - The user-selected file
* @returns The parsed conversations with their messages
*/
static async parseImportFile(file: File): Promise<ExportedConversation[]> {
const bytes = new Uint8Array(await file.arrayBuffer());
if (ZIP_MAGIC.every((byte, index) => bytes[index] === byte)) {
const entries = unzipSync(bytes);
const sessions: ExportedConversation[] = [];
for (const [entryName, entryBytes] of Object.entries(entries)) {
if (!entryName.toLowerCase().endsWith(FileExtensionText.JSONL)) continue;
sessions.push(...ConversationTransferService.parseSessionsJsonl(strFromU8(entryBytes)));
}
return sessions;
}
const text = strFromU8(bytes);
if (ConversationTransferService.isSessionsJsonl(text)) {
return ConversationTransferService.parseSessionsJsonl(text);
}
// Legacy JSON format: an array of conversations or a single conversation object.
const parsed = JSON.parse(text);
if (Array.isArray(parsed)) {
return parsed;
}
if (parsed && typeof parsed === 'object' && 'conv' in parsed && 'messages' in parsed) {
return [parsed];
}
throw new Error(
'Invalid file format: expected array of conversations or single conversation object'
);
}
/**
* Parses the JSONL session format produced by {@link serializeSessionToJsonl}.
* A `SessionRecordType.SESSION` line starts a new session; following
* `SessionRecordType.MESSAGE` lines are appended to it. Supports multiple
* sessions in a single file.
* @param text - The JSONL file contents
* @returns The parsed conversations with their messages
*/
static parseSessionsJsonl(text: string): ExportedConversation[] {
const sessions: ExportedConversation[] = [];
let current: ExportedConversation | null = null;
for (const line of text.split(NEWLINE)) {
const trimmed = line.trim();
if (!trimmed) continue;
const record = JSON.parse(trimmed);
if (record.type === SessionRecordType.SESSION) {
// Drop the discriminator and harness marker; the rest is the conversation.
const conv = { ...record };
delete conv.type;
delete conv.harness;
current = { conv: conv as DatabaseConversation, messages: [] };
sessions.push(current);
} else if (record.type === SessionRecordType.MESSAGE) {
if (!current) {
throw new Error('Invalid JSONL: message record before any session record');
}
const message = record.message as DatabaseMessage;
// `toolCalls` is parsed to an array on export; the DB stores it as a string.
if (message.toolCalls !== undefined && typeof message.toolCalls !== 'string') {
message.toolCalls = JSON.stringify(message.toolCalls);
}
current.messages.push(message);
}
// Ignore unknown record types for forward compatibility.
}
return sessions;
}
/**
* Serializes a session (a conversation with its messages) as JSONL.
* The first line is the session header (a `SessionRecordType.SESSION` record
* carrying the conversation properties); each subsequent line is a single message.
* @param data - The exported conversation payload
* @returns The JSONL string (one record per line)
*/
static serializeSessionToJsonl(data: ExportedConversation): string {
const { conv, messages } = data;
const sessionLine = JSON.stringify({
harness: EXPORT_CONV.HARNESS,
type: SessionRecordType.SESSION,
...conv
});
const messageLines = messages.map((message: DatabaseMessage) => {
// `toolCalls` is stored as a JSON string; drop it when empty, otherwise parse it.
const { toolCalls, ...rest } = message;
const normalized = toolCalls ? { ...rest, toolCalls: JSON.parse(toolCalls) } : rest;
return JSON.stringify({ message: normalized, type: SessionRecordType.MESSAGE });
});
return [sessionLine, ...messageLines].join(NEWLINE);
}
/**
* Reports whether the text is the JSONL session format, whose first non-empty
* line is a `SessionRecordType.SESSION` record. A legacy JSON export starts
* with an array or an object that has no such discriminator.
* @param text - The file contents
*/
private static isSessionsJsonl(text: string): boolean {
const trimmed = text.trimStart();
const lineEnd = trimmed.indexOf(NEWLINE);
const firstLine = lineEnd === -1 ? trimmed : trimmed.slice(0, lineEnd);
try {
return JSON.parse(firstLine).type === SessionRecordType.SESSION;
} catch {
// Not a standalone JSON record, so not the JSONL format.
return false;
}
}
/**
* Triggers a browser download of a blob under the given filename.
*/
+366 -399
View File
@@ -1,3 +1,11 @@
/**
* DatabaseService - IndexedDB persistence for conversations and messages
*
* Thin Dexie layer over the conversations/messages tables: CRUD, tree
* navigation (descendants, reparenting) and cascading deletes. No reactive
* state; consumed by conversationsStore and the chat flows.
*/
import { IDXDB_STORES, IDXDB_TABLES, STORAGE_APP_NAME } from '$lib/constants';
import { MessageRole } from '$lib/enums';
import type { McpServerOverride } from '$lib/types/database';
@@ -20,12 +28,99 @@ const db = new LlamaUiDatabase();
export class DatabaseService {
/**
* Deletes multiple conversations in a single transaction. Each deleted
* conversation has its direct children reparented to the nearest surviving
* ancestor (or promoted to top-level). Children also in `ids` are dropped
* entirely rather than reparented.
*
*
* Conversations
*
*
* @param ids - Conversation IDs to delete
*/
static async bulkDeleteConversations(ids: string[]): Promise<void> {
const cleanIds = ids.filter((id): id is string => typeof id === 'string' && id.length > 0);
if (cleanIds.length === 0) return;
const idSet = new Set(cleanIds);
await db.transaction(
'rw',
[db[IDXDB_TABLES.conversations], db[IDXDB_TABLES.messages]],
async () => {
// Pre-load each to-delete conversation so the per-id reparent
// walk-up doesn't ping-pong the same ancestry chain.
const prefetched = new Map<string, DatabaseConversation>();
let frontier = [...cleanIds];
const requested = new Set<string>(frontier);
while (frontier.length > 0) {
const fetched = await db[IDXDB_TABLES.conversations].bulkGet(frontier);
frontier = [];
for (let i = 0; i < fetched.length; i++) {
const conv = fetched[i];
if (!conv || !conv.id) continue;
prefetched.set(conv.id, conv);
const ancestor = conv.forkedFromConversationId;
if (ancestor && !prefetched.has(ancestor) && !requested.has(ancestor)) {
frontier.push(ancestor);
requested.add(ancestor);
}
}
}
for (const id of cleanIds) {
await this.reparentDirectChildren(id, idSet, prefetched);
}
await db[IDXDB_TABLES.conversations].bulkDelete(cleanIds);
await db[IDXDB_TABLES.messages].where('convId').anyOf(cleanIds).delete();
}
);
}
/**
* Toggles the pinned status of each conversation in `ids` inside a single
* transaction. Treats `pinned === undefined` as `false`, matching the
* semantics of {@link toggleConversationPin} where `!undefined` evaluates
* to `true`. Returns the resulting pinned state for every id that was
* updated; missing ids are omitted from the map.
*
* @param ids - Conversation IDs to toggle
* @returns Map of id -> new pinned state
*/
static async bulkToggleConversationPins(ids: string[]): Promise<Map<string, boolean>> {
const cleanIds = ids.filter((id): id is string => typeof id === 'string' && id.length > 0);
const result = new Map<string, boolean>();
if (cleanIds.length === 0) return result;
await db.transaction('rw', db[IDXDB_TABLES.conversations], async () => {
const convs = await db[IDXDB_TABLES.conversations].bulkGet(cleanIds);
const updates: DatabaseConversation[] = [];
for (let i = 0; i < cleanIds.length; i++) {
const conv = convs[i];
if (!conv) continue;
const newPinned = !conv.pinned;
updates.push({ ...conv, pinned: newPinned });
result.set(cleanIds[i], newPinned);
}
if (updates.length === 0) return;
await db[IDXDB_TABLES.conversations].bulkPut(updates);
});
return result;
}
/**
* Creates a new conversation.
@@ -51,14 +146,6 @@ export class DatabaseService {
return conversation;
}
/**
*
*
* Messages
*
*
*/
/**
* Creates a new message branch by adding a message and updating parent/child relationships.
* Also updates the conversation's currNode to point to the new message.
@@ -96,13 +183,7 @@ export class DatabaseService {
// Update parent's children array if parent exists
if (parentId !== null) {
const parentMessage = await db[IDXDB_TABLES.messages].get(parentId);
if (parentMessage) {
await db[IDXDB_TABLES.messages].update(parentId, {
children: [...parentMessage.children, newMessage.id]
});
}
await this.addChildToParent(parentId, newMessage.id);
}
await this.updateConversation(message.convId, {
@@ -178,9 +259,7 @@ export class DatabaseService {
};
await db[IDXDB_TABLES.messages].add(systemMessage);
await db[IDXDB_TABLES.messages].update(parentId, {
children: [...parentMessage.children, systemMessage.id]
});
await this.addChildToParent(parentId, systemMessage.id);
return systemMessage;
});
@@ -230,121 +309,6 @@ export class DatabaseService {
);
}
/**
* Reparents direct children of `parentId` to the nearest surviving
* ancestor (or promotes them to top-level when the immediate parent was
* top-level). Walking skips any ancestor listed in `excludeIds`, since
* those will be deleted in the same batch leaving a grandchild pointing
* at an `excludeIds` entry would orphan it. Children whose own id is in
* `excludeIds` are dropped from the updates (the bulk-delete pass will
* remove them). `prefetched` may carry a pre-fetched ancestor map to
* avoid repeat reads inside a bulk transaction.
*/
private static async reparentDirectChildren(
parentId: string,
excludeIds: ReadonlySet<string> = new Set(),
prefetched?: ReadonlyMap<string, DatabaseConversation>
): Promise<void> {
const conv = prefetched?.get(parentId) ?? (await db[IDXDB_TABLES.conversations].get(parentId));
if (!conv) return;
let newParent = conv.forkedFromConversationId;
const visited = new Set<string>([parentId]);
while (newParent && excludeIds.has(newParent)) {
if (visited.has(newParent)) {
newParent = undefined;
break;
}
visited.add(newParent);
const next =
prefetched?.get(newParent) ?? (await db[IDXDB_TABLES.conversations].get(newParent));
if (!next) {
newParent = undefined;
break;
}
newParent = next.forkedFromConversationId;
}
const directChildren = await db[IDXDB_TABLES.conversations]
.filter((c) => c.forkedFromConversationId === parentId)
.toArray();
const updates: DatabaseConversation[] = [];
for (const child of directChildren) {
if (excludeIds.has(child.id)) continue;
updates.push({ ...child, forkedFromConversationId: newParent });
}
if (updates.length === 0) return;
await db[IDXDB_TABLES.conversations].bulkPut(updates);
}
/**
* Deletes multiple conversations in a single transaction. Each deleted
* conversation has its direct children reparented to the nearest surviving
* ancestor (or promoted to top-level). Children also in `ids` are dropped
* entirely rather than reparented.
*
* @param ids - Conversation IDs to delete
*/
static async bulkDeleteConversations(ids: string[]): Promise<void> {
const cleanIds = ids.filter((id): id is string => typeof id === 'string' && id.length > 0);
if (cleanIds.length === 0) return;
const idSet = new Set(cleanIds);
await db.transaction(
'rw',
[db[IDXDB_TABLES.conversations], db[IDXDB_TABLES.messages]],
async () => {
// Pre-load each to-delete conversation so the per-id reparent
// walk-up doesn't ping-pong the same ancestry chain.
const prefetched = new Map<string, DatabaseConversation>();
let frontier = [...cleanIds];
const requested = new Set<string>(frontier);
while (frontier.length > 0) {
const fetched = await db[IDXDB_TABLES.conversations].bulkGet(frontier);
frontier = [];
for (let i = 0; i < fetched.length; i++) {
const conv = fetched[i];
if (!conv || !conv.id) continue;
prefetched.set(conv.id, conv);
const ancestor = conv.forkedFromConversationId;
if (ancestor && !prefetched.has(ancestor) && !requested.has(ancestor)) {
frontier.push(ancestor);
requested.add(ancestor);
}
}
}
for (const id of cleanIds) {
await this.reparentDirectChildren(id, idSet, prefetched);
}
await db[IDXDB_TABLES.conversations].bulkDelete(cleanIds);
await db[IDXDB_TABLES.messages].where('convId').anyOf(cleanIds).delete();
}
);
}
/**
* Deletes a message and removes it from its parent's children array.
*
@@ -356,17 +320,8 @@ export class DatabaseService {
if (!message) return;
// Remove this message from its parent's children array
if (message.parent) {
const parent = await db[IDXDB_TABLES.messages].get(message.parent);
await this.removeChildFromParent(messageId);
if (parent) {
parent.children = parent.children.filter((childId: string) => childId !== messageId);
await db[IDXDB_TABLES.messages].put(parent);
}
}
// Delete the message
await db[IDXDB_TABLES.messages].delete(messageId);
});
}
@@ -389,20 +344,10 @@ export class DatabaseService {
.where('convId')
.equals(conversationId)
.toArray();
// Find all descendant messages
const descendants = findDescendantMessages(allMessages, messageId);
const allToDelete = [messageId, ...descendants];
// Get the message to delete for parent cleanup
const message = await db[IDXDB_TABLES.messages].get(messageId);
if (message && message.parent) {
const parent = await db[IDXDB_TABLES.messages].get(message.parent);
if (parent) {
parent.children = parent.children.filter((childId: string) => childId !== messageId);
await db[IDXDB_TABLES.messages].put(parent);
}
}
await this.removeChildFromParent(messageId);
// Delete all messages in the branch
await db[IDXDB_TABLES.messages].bulkDelete(allToDelete);
@@ -411,243 +356,6 @@ export class DatabaseService {
});
}
/**
* Gets all conversations, sorted by last modified time (newest first).
*
* @returns Array of conversations
*/
static async getAllConversations(): Promise<DatabaseConversation[]> {
return await db[IDXDB_TABLES.conversations].orderBy('lastModified').reverse().toArray();
}
/**
* Gets a conversation by ID.
*
* @param id - Conversation ID
* @returns The conversation if found, otherwise undefined
*/
static async getConversation(id: string): Promise<DatabaseConversation | undefined> {
return await db[IDXDB_TABLES.conversations].get(id);
}
/**
* Gets all messages in a conversation, sorted by timestamp (oldest first).
*
* @param convId - Conversation ID
* @returns Array of messages in the conversation
*/
static async getConversationMessages(convId: string): Promise<DatabaseMessage[]> {
return await db[IDXDB_TABLES.messages].where('convId').equals(convId).sortBy('timestamp');
}
/**
* Loads multiple conversations with all of their messages in two bulk
* reads. Missing conversations are silently omitted from the result.
*
* @param convIds - Conversation IDs to load
* @returns Map of id -> { conv, messages }. Messages are sorted ascending by timestamp.
*/
static async getConversationsWithMessages(
convIds: string[]
): Promise<Map<string, ExportedConversation>> {
const result = new Map<string, ExportedConversation>();
const cleanIds = convIds.filter((id): id is string => typeof id === 'string' && id.length > 0);
if (cleanIds.length === 0) return result;
const [convs, allMessages] = await Promise.all([
db[IDXDB_TABLES.conversations].bulkGet(cleanIds),
db[IDXDB_TABLES.messages].where('convId').anyOf(cleanIds).toArray()
]);
const messagesByConv = new Map<string, DatabaseMessage[]>();
for (const msg of allMessages) {
const bucket = messagesByConv.get(msg.convId);
if (bucket) bucket.push(msg);
else messagesByConv.set(msg.convId, [msg]);
}
for (let i = 0; i < cleanIds.length; i++) {
const conv = convs[i];
if (!conv) continue;
const messages = (messagesByConv.get(conv.id) ?? []).sort(
(a, b) => a.timestamp - b.timestamp
);
result.set(conv.id, { conv, messages });
}
return result;
}
/**
* Updates a conversation. `lastModified` is never stamped implicitly;
* pass it in `updates` to bump the conversation in recency ordering.
*
* @param id - Conversation ID
* @param updates - Partial updates to apply
* @returns Promise that resolves when the conversation is updated
*/
static async updateConversation(
id: string,
updates: Partial<Omit<DatabaseConversation, 'id'>>
): Promise<void> {
await db[IDXDB_TABLES.conversations].update(id, updates);
}
/**
*
*
* Navigation
*
*
*/
/**
* Toggles the pinned status of a conversation.
*
* @param id - Conversation ID
* @returns The new pinned status
*/
static async toggleConversationPin(id: string): Promise<boolean> {
const conversation = await db[IDXDB_TABLES.conversations].get(id);
if (!conversation) {
throw new Error(`Conversation ${id} not found`);
}
const newPinnedState = !conversation.pinned;
await this.updateConversation(id, { pinned: newPinnedState });
return newPinnedState;
}
/**
* Toggles the pinned status of each conversation in `ids` inside a single
* transaction. Treats `pinned === undefined` as `false`, matching the
* semantics of {@link toggleConversationPin} where `!undefined` evaluates
* to `true`. Returns the resulting pinned state for every id that was
* updated; missing ids are omitted from the map.
*
* @param ids - Conversation IDs to toggle
* @returns Map of id -> new pinned state
*/
static async bulkToggleConversationPins(ids: string[]): Promise<Map<string, boolean>> {
const cleanIds = ids.filter((id): id is string => typeof id === 'string' && id.length > 0);
const result = new Map<string, boolean>();
if (cleanIds.length === 0) return result;
await db.transaction('rw', db[IDXDB_TABLES.conversations], async () => {
const convs = await db[IDXDB_TABLES.conversations].bulkGet(cleanIds);
const updates: DatabaseConversation[] = [];
for (let i = 0; i < cleanIds.length; i++) {
const conv = convs[i];
if (!conv) continue;
const newPinned = !conv.pinned;
updates.push({ ...conv, pinned: newPinned });
result.set(cleanIds[i], newPinned);
}
if (updates.length === 0) return;
await db[IDXDB_TABLES.conversations].bulkPut(updates);
});
return result;
}
/**
* Updates the conversation's current node (active branch).
* This determines which conversation path is currently being viewed.
*
* @param convId - Conversation ID
* @param nodeId - Message ID to set as current node
*/
static async updateCurrentNode(convId: string, nodeId: string): Promise<void> {
await this.updateConversation(convId, {
currNode: nodeId
});
}
/**
* Updates a message.
*
* @param id - Message ID
* @param updates - Partial updates to apply
* @returns Promise that resolves when the message is updated
*/
static async updateMessage(
id: string,
updates: Partial<Omit<DatabaseMessage, 'id'>>
): Promise<void> {
await db[IDXDB_TABLES.messages].update(id, updates);
}
/**
*
*
* Import
*
*
*/
/**
* Imports multiple conversations and their messages.
* Skips conversations that already exist.
*
* @param data - Array of { conv, messages } objects
* @returns The conversations written to the database and the ones skipped
*/
static async importConversations(
data: { conv: DatabaseConversation; messages: DatabaseMessage[] }[]
): Promise<{ imported: DatabaseConversation[]; skipped: DatabaseConversation[] }> {
const imported: DatabaseConversation[] = [];
const skipped: DatabaseConversation[] = [];
return await db.transaction(
'rw',
[db[IDXDB_TABLES.conversations], db[IDXDB_TABLES.messages]],
async () => {
for (const item of data) {
const { conv, messages } = item;
const existing = await db[IDXDB_TABLES.conversations].get(conv.id);
if (existing) {
skipped.push(conv);
continue;
}
await db[IDXDB_TABLES.conversations].add(conv);
for (const msg of messages) {
await db[IDXDB_TABLES.messages].put(msg);
}
imported.push(conv);
}
return { imported, skipped };
}
);
}
/**
*
*
* Forking
*
*
*/
/**
* Forks a conversation at a specific message, creating a new conversation
* containing all messages from the root up to (and including) the target message.
@@ -726,13 +434,272 @@ export class DatabaseService {
};
await db[IDXDB_TABLES.conversations].add(newConv);
for (const msg of clonedMessages) {
await db[IDXDB_TABLES.messages].add(msg);
}
await db[IDXDB_TABLES.messages].bulkAdd(clonedMessages);
return newConv;
}
);
}
/**
* Gets all conversations, sorted by last modified time (newest first).
*
* @returns Array of conversations
*/
static async getAllConversations(): Promise<DatabaseConversation[]> {
return await db[IDXDB_TABLES.conversations].orderBy('lastModified').reverse().toArray();
}
/**
* Gets a conversation by ID.
*
* @param id - Conversation ID
* @returns The conversation if found, otherwise undefined
*/
static async getConversation(id: string): Promise<DatabaseConversation | undefined> {
return await db[IDXDB_TABLES.conversations].get(id);
}
/**
* Gets all messages in a conversation, sorted by timestamp (oldest first).
*
* @param convId - Conversation ID
* @returns Array of messages in the conversation
*/
static async getConversationMessages(convId: string): Promise<DatabaseMessage[]> {
return await db[IDXDB_TABLES.messages].where('convId').equals(convId).sortBy('timestamp');
}
/**
* Loads multiple conversations with all of their messages in two bulk
* reads. Missing conversations are silently omitted from the result.
*
* @param convIds - Conversation IDs to load
* @returns Map of id -> { conv, messages }. Messages are sorted ascending by timestamp.
*/
static async getConversationsWithMessages(
convIds: string[]
): Promise<Map<string, ExportedConversation>> {
const result = new Map<string, ExportedConversation>();
const cleanIds = convIds.filter((id): id is string => typeof id === 'string' && id.length > 0);
if (cleanIds.length === 0) return result;
const [convs, allMessages] = await Promise.all([
db[IDXDB_TABLES.conversations].bulkGet(cleanIds),
db[IDXDB_TABLES.messages].where('convId').anyOf(cleanIds).toArray()
]);
const messagesByConv = new Map<string, DatabaseMessage[]>();
for (const msg of allMessages) {
const bucket = messagesByConv.get(msg.convId);
if (bucket) bucket.push(msg);
else messagesByConv.set(msg.convId, [msg]);
}
for (let i = 0; i < cleanIds.length; i++) {
const conv = convs[i];
if (!conv) continue;
const messages = (messagesByConv.get(conv.id) ?? []).sort(
(a, b) => a.timestamp - b.timestamp
);
result.set(conv.id, { conv, messages });
}
return result;
}
/**
* Imports multiple conversations and their messages.
* Skips conversations that already exist.
*
* @param data - Array of { conv, messages } objects
* @returns The conversations written to the database and the ones skipped
*/
static async importConversations(
data: { conv: DatabaseConversation; messages: DatabaseMessage[] }[]
): Promise<{ imported: DatabaseConversation[]; skipped: DatabaseConversation[] }> {
const imported: DatabaseConversation[] = [];
const skipped: DatabaseConversation[] = [];
return await db.transaction(
'rw',
[db[IDXDB_TABLES.conversations], db[IDXDB_TABLES.messages]],
async () => {
for (const item of data) {
const { conv, messages } = item;
const existing = await db[IDXDB_TABLES.conversations].get(conv.id);
if (existing) {
skipped.push(conv);
continue;
}
await db[IDXDB_TABLES.conversations].add(conv);
for (const msg of messages) {
await db[IDXDB_TABLES.messages].put(msg);
}
imported.push(conv);
}
return { imported, skipped };
}
);
}
/**
* Toggles the pinned status of a conversation.
*
* @param id - Conversation ID
* @returns The new pinned status
*/
static async toggleConversationPin(id: string): Promise<boolean> {
const conversation = await db[IDXDB_TABLES.conversations].get(id);
if (!conversation) {
throw new Error(`Conversation ${id} not found`);
}
const newPinnedState = !conversation.pinned;
await this.updateConversation(id, { pinned: newPinnedState });
return newPinnedState;
}
/**
* Updates a conversation. `lastModified` is never stamped implicitly;
* pass it in `updates` to bump the conversation in recency ordering.
*
* @param id - Conversation ID
* @param updates - Partial updates to apply
* @returns Promise that resolves when the conversation is updated
*/
static async updateConversation(
id: string,
updates: Partial<Omit<DatabaseConversation, 'id'>>
): Promise<void> {
await db[IDXDB_TABLES.conversations].update(id, updates);
}
/**
* Updates the conversation's current node (active branch).
* This determines which conversation path is currently being viewed.
*
* @param convId - Conversation ID
* @param nodeId - Message ID to set as current node
*/
static async updateCurrentNode(convId: string, nodeId: string): Promise<void> {
await this.updateConversation(convId, {
currNode: nodeId
});
}
/**
* Updates a message.
*
* @param id - Message ID
* @param updates - Partial updates to apply
* @returns Promise that resolves when the message is updated
*/
static async updateMessage(
id: string,
updates: Partial<Omit<DatabaseMessage, 'id'>>
): Promise<void> {
await db[IDXDB_TABLES.messages].update(id, updates);
}
/**
* Appends a child id to a parent message's children array.
*/
private static async addChildToParent(parentId: string, childId: string): Promise<void> {
const parent = await db[IDXDB_TABLES.messages].get(parentId);
if (!parent) return;
await db[IDXDB_TABLES.messages].update(parentId, {
children: [...parent.children, childId]
});
}
/**
* Removes a child id from its parent message's children array.
*/
private static async removeChildFromParent(messageId: string): Promise<void> {
const message = await db[IDXDB_TABLES.messages].get(messageId);
if (!message?.parent) return;
const parent = await db[IDXDB_TABLES.messages].get(message.parent);
if (!parent) return;
parent.children = parent.children.filter((childId: string) => childId !== messageId);
await db[IDXDB_TABLES.messages].put(parent);
}
/**
* Reparents direct children of `parentId` to the nearest surviving
* ancestor (or promotes them to top-level when the immediate parent was
* top-level). Walking skips any ancestor listed in `excludeIds`, since
* those will be deleted in the same batch leaving a grandchild pointing
* at an `excludeIds` entry would orphan it. Children whose own id is in
* `excludeIds` are dropped from the updates (the bulk-delete pass will
* remove them). `prefetched` may carry a pre-fetched ancestor map to
* avoid repeat reads inside a bulk transaction.
*/
private static async reparentDirectChildren(
parentId: string,
excludeIds: ReadonlySet<string> = new Set(),
prefetched?: ReadonlyMap<string, DatabaseConversation>
): Promise<void> {
const conv = prefetched?.get(parentId) ?? (await db[IDXDB_TABLES.conversations].get(parentId));
if (!conv) return;
let newParent = conv.forkedFromConversationId;
const visited = new Set<string>([parentId]);
while (newParent && excludeIds.has(newParent)) {
if (visited.has(newParent)) {
newParent = undefined;
break;
}
visited.add(newParent);
const next =
prefetched?.get(newParent) ?? (await db[IDXDB_TABLES.conversations].get(newParent));
if (!next) {
newParent = undefined;
break;
}
newParent = next.forkedFromConversationId;
}
const directChildren = await db[IDXDB_TABLES.conversations]
.filter((c) => c.forkedFromConversationId === parentId)
.toArray();
const updates: DatabaseConversation[] = [];
for (const child of directChildren) {
if (excludeIds.has(child.id)) continue;
updates.push({ ...child, forkedFromConversationId: newParent });
}
if (updates.length === 0) return;
await db[IDXDB_TABLES.conversations].bulkPut(updates);
}
}
+24 -14
View File
@@ -53,9 +53,9 @@
* - Reasoning content stripping from prompt history to avoid KV cache pollution
* - Error translation (network, timeout, server errors user-friendly messages)
*
* @see chatStore in stores/chat.svelte.ts primary consumer for chat state management
* @see agenticStore in stores/agentic.svelte.ts uses ChatService for agentic loop streaming
* @see conversationsStore in stores/conversations.svelte.ts provides message context
* @see chatStore in stores/chat/index.svelte.ts primary consumer for chat state management
* @see agenticStore in stores/agentic/index.svelte.ts uses ChatService for agentic loop streaming
* @see conversationsStore in stores/conversations/index.svelte.ts provides message context
*/
export { ChatService } from './chat.service';
@@ -98,8 +98,8 @@ export { ChatService } from './chat.service';
* enabling conversation branching and alternative response paths. The conversation's
* `currNode` tracks the currently active branch endpoint.
*
* @see conversationsStore in stores/conversations.svelte.ts reactive layer on top of DatabaseService
* @see chatStore in stores/chat.svelte.ts uses DatabaseService directly for message CRUD during streaming
* @see conversationsStore in stores/conversations/index.svelte.ts reactive layer on top of DatabaseService
* @see chatStore in stores/chat/index.svelte.ts uses DatabaseService directly for message CRUD during streaming
*/
export { DatabaseService } from './database.service';
@@ -143,7 +143,7 @@ export { ConversationTransferService } from './conversation-transfer.service';
* - `POST /models/load` Load a model (ROUTER mode only)
* - `POST /models/unload` Unload a model (ROUTER mode only)
*
* @see modelsStore in stores/models.svelte.ts primary consumer for reactive model state
* @see modelsStore in stores/models/index.svelte.ts primary consumer for reactive model state
*/
export { ModelsService } from './models.service';
@@ -174,8 +174,8 @@ export { ModelsService } from './models.service';
* - `&autoload=false` Prevents model auto-loading when querying props
*
* @see serverStore in stores/server.svelte.ts consumes global server props
* @see modelsStore in stores/models.svelte.ts consumes per-model props for modalities
* @see settingsStore in stores/settings.svelte.ts syncs default generation params from props
* @see modelsStore in stores/models/index.svelte.ts consumes per-model props for modalities
* @see settingsStore in stores/settings/index.svelte.ts syncs default generation params from props
*/
export { PropsService } from './props.service';
@@ -217,7 +217,7 @@ export { PropsService } from './props.service';
* - `ParameterSyncService` class static methods for sync logic
* - `SYNCABLE_PARAMETERS` mapping of UI setting keys to server parameter keys
*
* @see settingsStore in stores/settings.svelte.ts primary consumer for settings sync
* @see settingsStore in stores/settings/index.svelte.ts primary consumer for settings sync
* @see SettingsChatParameterSourceIndicator displays parameter source badges in UI
*/
export { ParameterSyncService } from './parameter-sync.service';
@@ -241,7 +241,7 @@ export { ParameterSyncService } from './parameter-sync.service';
* - Manages connection lifecycle, health checks, reconnection
* - Handles tool name conflict resolution and server coordination
*
* - **mcpResourceStore**: Reactive resource state
* - **mcpResourceStore** (composed as mcpStore.resources): Reactive resource state
* - Receives resource data fetched via MCPService
* - Manages resource caching, subscriptions, and attachments
*
@@ -263,9 +263,9 @@ export { ParameterSyncService } from './parameter-sync.service';
* 2. **StreamableHTTP** modern HTTP-based, supports CORS proxy
* 3. **SSE** legacy fallback, supports CORS proxy
*
* @see mcpStore in stores/mcp.svelte.ts reactive business logic facade on top of MCPService
* @see mcpResourceStore in stores/mcp-resources.svelte.ts reactive resource state management
* @see agenticStore in stores/agentic.svelte.ts uses MCPService (via mcpStore) for tool execution
* @see mcpStore in stores/mcp/index.svelte.ts reactive business logic facade on top of MCPService
* @see mcpStore.resources in stores/mcp/resources.svelte.ts reactive resource state management
* @see agenticStore in stores/agentic/index.svelte.ts uses MCPService (via mcpStore) for tool execution
* @see MCP Protocol Specification: https://modelcontextprotocol.io/specification/2025-06-18
*/
export { MCPService } from './mcp.service';
@@ -286,7 +286,7 @@ export { MCPService } from './mcp.service';
* - **agenticStore**: Dispatches ToolSource.BROWSER calls here
*
* @see buildSandboxToolDefinition in utils/sandbox-tool - tool schema sent to the LLM
* @see agenticStore in stores/agentic.svelte.ts - tool dispatch
* @see agenticStore in stores/agentic/index.svelte.ts - tool dispatch
*/
export { SandboxService } from './sandbox.service';
@@ -340,3 +340,13 @@ export { RouterService } from './router.service';
* @see migration.service.ts full implementation (non-destructive)
*/
export { MigrationService } from './migration.service';
/**
* **SettingsService** - localStorage persistence layer for settings
*
* Stateless read/write of the settings config and user-override keys. Business
* logic (default merging, mobile defaults, theme migration) stays in the store.
*
* @see settingsStore in stores/settings/index.svelte.ts - reactive state + business logic
*/
export { SettingsService } from './settings.service';
File diff suppressed because it is too large Load Diff
+6 -15
View File
@@ -1,20 +1,11 @@
/**
* Migration Service - Unified data migration hook
* MigrationService - Unified data migration hook
*
* Centralizes all data migrations (localStorage, IndexedDB, legacy formats) into a single
* initialization point. Each migration copies data to new format WITHOUT deleting the old.
*
* **Architecture:**
* - Migrations are defined as objects with `id` and `run()` methods
* - Migration state is tracked in localStorage to avoid re-running
* - `runAllMigrations()` should be called once at app startup
* - All migrations are NON-DESTRUCTIVE - legacy data is preserved for downgrade compatibility
*
* **Current Migrations:**
* 1. localStorage prefix: Copy LlamaCppWebui.* LlamaUi.* (both preserved)
* 2. IndexedDB database: Copy LlamacppWebui LlamaUi (both preserved)
* 3. Legacy message format: Transform in-place (preserves structure, migrates markers)
* 4. Theme key: Copy standalone `theme` config object (both preserved)
* Centralizes all data migrations (localStorage, IndexedDB, legacy formats)
* into a single initialization point. Each migration copies data to the new
* format WITHOUT deleting the old, and state is tracked in localStorage so
* `runAllMigrations()` (called once at startup) never re-runs a completed
* migration. All migrations are non-destructive for downgrade compatibility.
*/
import {
+119 -148
View File
@@ -1,25 +1,55 @@
/**
* ModelsService - Stateless model management API layer
*
* Wraps the /models endpoints (list, load, unload) and the /models/sse
* status feed in MODEL and ROUTER modes. No reactive state; consumed by
* modelsStore and its status manager.
*/
import { base } from '$app/paths';
import {
API_MODELS,
MODEL_ID,
SSE_DATA_PREFIX,
SSE_LINE_SEPARATOR,
SSE_RECORD_SEPARATOR
} from '$lib/constants';
import { API_MODELS, MODEL_ID } from '$lib/constants';
import { ServerModelStatus } from '$lib/enums';
import type { ParsedModelId } from '$lib/types/models';
import { apiFetch, apiPost, normalizeModelName } from '$lib/utils';
import {
apiFetch,
apiPost,
extractSseDataPayload,
normalizeModelName,
splitSseRecords
} from '$lib/utils';
import { getAuthHeaders } from '$lib/utils/api-headers';
export class ModelsService {
private static readonly SSE_RECONNECT_MS = 1000;
/**
* Check if a model is loaded based on its metadata.
*
* @param model - Model data entry from the API response
* @returns True if the model status is LOADED
*/
static isModelLoaded(model: ApiModelDataEntry): boolean {
return model.status.value === ServerModelStatus.LOADED;
}
/**
*
*
* Listing
* Load/Unload
*
*
*/
/**
* Check if a model is currently loading.
*
* @param model - Model data entry from the API response
* @returns True if the model status is LOADING
*/
static isModelLoading(model: ApiModelDataEntry): boolean {
return model.status.value === ServerModelStatus.LOADING;
}
/**
* Fetch list of models from OpenAI-compatible endpoint.
* Works in both MODEL and ROUTER modes.
@@ -41,14 +71,6 @@ export class ModelsService {
return apiFetch<ApiRouterModelsListResponse>(API_MODELS.LIST);
}
/**
*
*
* Load/Unload
*
*
*/
/**
* Load a model (ROUTER mode only).
* Sends POST request to `/models/load`. Note: the endpoint returns success
@@ -68,137 +90,6 @@ export class ModelsService {
return apiPost<ApiRouterModelsLoadResponse>(API_MODELS.LOAD, payload);
}
/**
* Unload a model (ROUTER mode only).
* Sends POST request to `/models/unload`. Note: the endpoint returns success
* before unloading completes use polling to await actual unload status.
*
* @param modelId - Model identifier to unload
* @returns Unload response from the server
*/
static async unload(modelId: string): Promise<ApiRouterModelsUnloadResponse> {
return apiPost<ApiRouterModelsUnloadResponse>(API_MODELS.UNLOAD, { model: modelId });
}
/**
*
*
* Status
*
*
*/
/**
* Check if a model is loaded based on its metadata.
*
* @param model - Model data entry from the API response
* @returns True if the model status is LOADED
*/
static isModelLoaded(model: ApiModelDataEntry): boolean {
return model.status.value === ServerModelStatus.LOADED;
}
/**
* Check if a model is currently loading.
*
* @param model - Model data entry from the API response
* @returns True if the model status is LOADING
*/
static isModelLoading(model: ApiModelDataEntry): boolean {
return model.status.value === ServerModelStatus.LOADING;
}
/**
*
*
* Status Feed
*
*
*/
private static readonly SSE_RECONNECT_MS = 1000;
/**
* Read the /models/sse feed and invoke onEvent for each parsed envelope.
* Reconnects on network drops until the signal aborts. Splits the byte
* stream into SSE records on the blank line boundary; the payload rides in
* the data lines as a JSON envelope with its own model, event and data fields.
*/
static async watchModelEvents(
signal: AbortSignal,
onEvent: (event: ApiModelsSseEvent) => void
): Promise<void> {
const decoder = new TextDecoder();
while (!signal.aborted) {
try {
const response = await fetch(`${base}${API_MODELS.SSE}`, {
headers: getAuthHeaders(),
signal
});
if (response.ok && response.body) {
const reader = response.body.getReader();
let buffer = '';
while (!signal.aborted) {
const { done, value } = await reader.read();
if (done) break;
buffer += decoder.decode(value, { stream: true });
let boundary = buffer.indexOf(SSE_RECORD_SEPARATOR);
while (boundary !== -1) {
const event = ModelsService.parseStatusRecord(buffer.slice(0, boundary));
if (event) onEvent(event);
buffer = buffer.slice(boundary + SSE_RECORD_SEPARATOR.length);
boundary = buffer.indexOf(SSE_RECORD_SEPARATOR);
}
}
}
} catch {
// network drop or abort falls through to the reconnect delay
}
if (signal.aborted) return;
await new Promise((resolve) => setTimeout(resolve, ModelsService.SSE_RECONNECT_MS));
}
}
/**
* Parse one SSE record into its JSON envelope, or null when the record
* carries no data payload or malformed JSON.
*/
private static parseStatusRecord(record: string): ApiModelsSseEvent | null {
const payload = record
.split(SSE_LINE_SEPARATOR)
.filter((line) => line.startsWith(SSE_DATA_PREFIX))
.map((line) => line.slice(SSE_DATA_PREFIX.length).trim())
.join(SSE_LINE_SEPARATOR);
if (payload.length === 0) return null;
try {
return JSON.parse(payload) as ApiModelsSseEvent;
} catch {
return null;
}
}
/**
*
*
* Parsing
*
*
*/
/**
* Parse a model ID string into its structured components.
*
@@ -311,4 +202,84 @@ export class ModelsService {
return result;
}
/**
* Unload a model (ROUTER mode only).
* Sends POST request to `/models/unload`. Note: the endpoint returns success
* before unloading completes use polling to await actual unload status.
*
* @param modelId - Model identifier to unload
* @returns Unload response from the server
*/
static async unload(modelId: string): Promise<ApiRouterModelsUnloadResponse> {
return apiPost<ApiRouterModelsUnloadResponse>(API_MODELS.UNLOAD, { model: modelId });
}
/**
* Read the /models/sse feed and invoke onEvent for each parsed envelope.
* Reconnects on network drops until the signal aborts. Splits the byte
* stream into SSE records on the blank line boundary; the payload rides in
* the data lines as a JSON envelope with its own model, event and data fields.
*/
static async watchModelEvents(
signal: AbortSignal,
onEvent: (event: ApiModelsSseEvent) => void
): Promise<void> {
const decoder = new TextDecoder();
while (!signal.aborted) {
try {
const response = await fetch(`${base}${API_MODELS.SSE}`, {
headers: getAuthHeaders(),
signal
});
if (response.ok && response.body) {
const reader = response.body.getReader();
let buffer = '';
while (!signal.aborted) {
const { done, value } = await reader.read();
if (done) break;
buffer += decoder.decode(value, { stream: true });
const { records, rest } = splitSseRecords(buffer);
buffer = rest;
for (const record of records) {
const event = ModelsService.parseStatusRecord(record);
if (event) onEvent(event);
}
}
}
} catch {
// network drop or abort falls through to the reconnect delay
}
if (signal.aborted) return;
await new Promise((resolve) => setTimeout(resolve, ModelsService.SSE_RECONNECT_MS));
}
}
/**
* Parse one SSE record into its JSON envelope, or null when the record
* carries no data payload or malformed JSON.
*/
private static parseStatusRecord(record: string): ApiModelsSseEvent | null {
const payload = extractSseDataPayload(record);
if (payload.length === 0) return null;
try {
return JSON.parse(payload) as ApiModelsSseEvent;
} catch {
return null;
}
}
}
@@ -1,26 +1,71 @@
import { SETTINGS_KEYS, SYNCABLE_PARAMETERS } from '$lib/constants';
/**
* ParameterSyncService - Syncs sampling parameters with the server
*
* Decides for each sampling parameter whether the user's setting is an
* override of the server default, and normalizes floating-point values.
* No reactive state; consumed by settingsStore.
*/
import { SETTINGS_KEYS, SETTINGS_REGISTRY } from '$lib/constants';
import { ParameterSource, SyncableParameterType } from '$lib/enums';
import type { ParameterInfo, ParameterRecord, ParameterValue } from '$lib/types';
import type { ParameterInfo, ParameterRecord, ParameterValue, SyncableParameter } from '$lib/types';
import { normalizeFloatingPoint } from '$lib/utils';
/** Mapping of UI setting keys to server parameter keys, derived from the registry. */
export const SYNCABLE_PARAMETERS: SyncableParameter[] = SETTINGS_REGISTRY.flatMap(
(section) => section.settings
)
.filter((s) => s.sync !== undefined)
.map((s) => ({
canSync: true,
key: s.key,
serverKey: s.sync!.serverKey,
type: s.sync!.paramType
}));
export class ParameterSyncService {
/**
* Check if a parameter can be synced from server.
*
*
* Extraction
*
*
* @param key - The parameter key to check
* @returns True if the parameter is in the syncable parameters list
*/
static canSyncParameter(key: string): boolean {
return SYNCABLE_PARAMETERS.some((param) => param.key === key && param.canSync);
}
/**
* Round floating-point numbers to avoid JavaScript precision issues.
* E.g., 0.1 + 0.2 = 0.30000000000000004 0.3
* Create a diff between current settings and server defaults.
* Shows which parameters differ from server values, useful for debugging
* and for the "Reset to defaults" functionality.
*
* @param value - Parameter value to normalize
* @returns Precision-normalized value
* @param currentSettings - Current parameter values in the settings store
* @param serverDefaults - Default values extracted from server props
* @returns Record of parameter diffs with current value, server value, and whether they differ
*/
private static roundFloatingPoint(value: ParameterValue): ParameterValue {
return normalizeFloatingPoint(value) as ParameterValue;
static createParameterDiff(
currentSettings: ParameterRecord,
serverDefaults: ParameterRecord
): Record<string, { current: ParameterValue; server: ParameterValue; differs: boolean }> {
const diff: Record<
string,
{ current: ParameterValue; server: ParameterValue; differs: boolean }
> = {};
for (const key of this.getSyncableParameterKeys()) {
const currentValue = currentSettings[key];
const serverValue = serverDefaults[key];
if (serverValue !== undefined) {
diff[key] = {
current: currentValue,
differs: currentValue !== serverValue,
server: serverValue
};
}
}
return diff;
}
/**
@@ -59,49 +104,6 @@ export class ParameterSyncService {
return extracted;
}
/**
*
*
* Merging
*
*
*/
/**
* Merge server defaults with current user settings.
* User overrides always take priority only parameters not in `userOverrides`
* set will be updated from server defaults.
*
* @param currentSettings - Current parameter values in the settings store
* @param serverDefaults - Default values extracted from server props
* @param userOverrides - Set of parameter keys explicitly overridden by the user
* @returns Merged parameter record with user overrides preserved
*/
static mergeWithServerDefaults(
currentSettings: ParameterRecord,
serverDefaults: ParameterRecord,
userOverrides: Set<string> = new Set()
): ParameterRecord {
const merged = { ...currentSettings };
for (const [key, serverValue] of Object.entries(serverDefaults)) {
// Only update if user hasn't explicitly overridden this parameter
if (!userOverrides.has(key)) {
merged[key] = this.roundFloatingPoint(serverValue);
}
}
return merged;
}
/**
*
*
* Info
*
*
*/
/**
* Get parameter information including source and values.
* Used by SettingsChatParameterSourceIndicator to display the correct badge
@@ -132,16 +134,6 @@ export class ParameterSyncService {
};
}
/**
* Check if a parameter can be synced from server.
*
* @param key - The parameter key to check
* @returns True if the parameter is in the syncable parameters list
*/
static canSyncParameter(key: string): boolean {
return SYNCABLE_PARAMETERS.some((param) => param.key === key && param.canSync);
}
/**
* Get all syncable parameter keys.
*
@@ -151,6 +143,33 @@ export class ParameterSyncService {
return SYNCABLE_PARAMETERS.filter((param) => param.canSync).map((param) => param.key);
}
/**
* Merge server defaults with current user settings.
* User overrides always take priority only parameters not in `userOverrides`
* set will be updated from server defaults.
*
* @param currentSettings - Current parameter values in the settings store
* @param serverDefaults - Default values extracted from server props
* @param userOverrides - Set of parameter keys explicitly overridden by the user
* @returns Merged parameter record with user overrides preserved
*/
static mergeWithServerDefaults(
currentSettings: ParameterRecord,
serverDefaults: ParameterRecord,
userOverrides: Set<string> = new Set()
): ParameterRecord {
const merged = { ...currentSettings };
for (const [key, serverValue] of Object.entries(serverDefaults)) {
// Only update if user hasn't explicitly overridden this parameter
if (!userOverrides.has(key)) {
merged[key] = this.roundFloatingPoint(serverValue);
}
}
return merged;
}
/**
* Validate a server parameter value against its expected type.
*
@@ -176,44 +195,13 @@ export class ParameterSyncService {
}
/**
* Round floating-point numbers to avoid JavaScript precision issues.
* E.g., 0.1 + 0.2 = 0.30000000000000004 0.3
*
*
* Diff
*
*
* @param value - Parameter value to normalize
* @returns Precision-normalized value
*/
/**
* Create a diff between current settings and server defaults.
* Shows which parameters differ from server values, useful for debugging
* and for the "Reset to defaults" functionality.
*
* @param currentSettings - Current parameter values in the settings store
* @param serverDefaults - Default values extracted from server props
* @returns Record of parameter diffs with current value, server value, and whether they differ
*/
static createParameterDiff(
currentSettings: ParameterRecord,
serverDefaults: ParameterRecord
): Record<string, { current: ParameterValue; server: ParameterValue; differs: boolean }> {
const diff: Record<
string,
{ current: ParameterValue; server: ParameterValue; differs: boolean }
> = {};
for (const key of this.getSyncableParameterKeys()) {
const currentValue = currentSettings[key];
const serverValue = serverDefaults[key];
if (serverValue !== undefined) {
diff[key] = {
current: currentValue,
differs: currentValue !== serverValue,
server: serverValue
};
}
}
return diff;
private static roundFloatingPoint(value: ParameterValue): ParameterValue {
return normalizeFloatingPoint(value) as ParameterValue;
}
}
+8 -8
View File
@@ -1,14 +1,14 @@
/**
* PropsService - Fetches server properties from /props
*
* Returns global server settings and capabilities, including per-model
* modalities in MODEL mode. No reactive state; consumed by serverStore and
* the model props manager.
*/
import { apiFetchWithParams } from '$lib/utils';
export class PropsService {
/**
*
*
* Fetching
*
*
*/
/**
* Fetches global server properties from the `/props` endpoint.
* In MODEL mode, returns modalities for the single loaded model.
@@ -1,3 +1,10 @@
/**
* ReadMediaService - Reads local media files for the read_media tool
*
* Encodes image and audio files as base64 data URLs with the metadata the
* model needs. No reactive state; consumed by toolsStore.
*/
import { ToolsService } from './tools.service';
import {
FILE_EXTENSION_SEPARATOR,
@@ -40,7 +47,7 @@ function fileExtension(path: string): string {
* actually use the result - the server has no idea which model is selected.
*
* @see buildReadMediaToolDefinition in constants/read-media.ts - tool schema sent to the LLM
* @see agenticStore in stores/agentic.svelte.ts - tool dispatch and attachment extraction
* @see agenticStore in stores/agentic/index.svelte.ts - tool dispatch and attachment extraction
*/
export class ReadMediaService {
static async executeTool(
@@ -1,3 +1,10 @@
/**
* RouterService - Builds app route paths
*
* Returns chat and settings route strings from a single source of truth
* (ROUTES). No state.
*/
import { ROUTES } from '$lib/constants';
export class RouterService {
@@ -1,3 +1,10 @@
/**
* Sandbox harness - builds the srcdoc document for the sandboxed iframe
*
* Produces the HTML/CSP/worker shim that runs untrusted model code in an
* opaque origin. Consumed by sandbox.service.
*/
import WORKER_SHIM from './sandbox-worker.js?raw';
import { NEWLINE } from '$lib/constants';
+9 -1
View File
@@ -1,3 +1,11 @@
/**
* SandboxService - Runs untrusted code in a sandboxed worker
*
* Executes model-generated code inside a CSP-restricted, opaque-origin
* iframe worker with output and timeout limits. No reactive state; consumed
* by toolsStore for code-execution tools.
*/
import { buildSandboxHarness } from './sandbox-harness';
import {
NEWLINE,
@@ -8,7 +16,7 @@ import {
SANDBOX_TOOL_NAME,
SANDBOX_TRUNCATION_NOTICE
} from '$lib/constants';
import { settingsStore } from '$lib/stores/settings.svelte';
import { settingsStore } from '$lib/stores/settings/index.svelte';
import type { ToolExecutionResult } from '$lib/types';
/** Cached harnesses keyed by whether nerdamer is included. */
@@ -0,0 +1,76 @@
import { browser } from '$app/environment';
import { CONFIG_LOCALSTORAGE_KEY, USER_OVERRIDES_LOCALSTORAGE_KEY } from '$lib/constants';
/**
* SettingsService - localStorage persistence layer for settings
*
* Stateless read/write of the settings config and user-override keys. Business
* logic (default merging, mobile defaults, theme migration) stays in the store.
*
* **Architecture & Relationships:**
* - **settingsStore**: Primary consumer - loads config on init and persists on change
*
* @see settingsStore in stores/settings/index.svelte.ts - reactive state + business logic
*/
export class SettingsService {
/**
* Read the raw config and user overrides from localStorage.
* @returns Parsed values, or empty defaults when nothing is stored or parsing fails.
*/
static loadConfig(): {
config: Record<string, unknown>;
userOverrides: string[];
isFirstVisit: boolean;
} {
if (!browser) {
return { config: {}, isFirstVisit: false, userOverrides: [] };
}
try {
const storedConfigRaw = localStorage.getItem(CONFIG_LOCALSTORAGE_KEY);
const isFirstVisit = storedConfigRaw === null;
const config = JSON.parse(storedConfigRaw || '{}') as Record<string, unknown>;
const userOverrides = JSON.parse(
localStorage.getItem(USER_OVERRIDES_LOCALSTORAGE_KEY) || '[]'
) as string[];
return { config, isFirstVisit, userOverrides };
} catch (error) {
console.warn('Failed to parse config from localStorage, using defaults:', error);
return { config: {}, isFirstVisit: false, userOverrides: [] };
}
}
/**
* Migrate the legacy un-namespaced "theme" localStorage key.
* Returns the legacy theme value (and removes the key) when present, else null.
*/
static migrateLegacyTheme(): string | null {
if (!browser) return null;
const legacyTheme = localStorage.getItem('theme');
if (legacyTheme) {
localStorage.removeItem('theme');
return legacyTheme;
}
return null;
}
/**
* Persist the config and user overrides to localStorage.
*/
static saveConfig(config: Record<string, unknown>, userOverrides: string[]): void {
if (!browser) return;
try {
localStorage.setItem(CONFIG_LOCALSTORAGE_KEY, JSON.stringify(config));
localStorage.setItem(USER_OVERRIDES_LOCALSTORAGE_KEY, JSON.stringify(userOverrides));
} catch (error) {
console.error('Failed to save config to localStorage:', error);
}
}
}

Some files were not shown because too many files have changed in this diff Show More