* server: SSE replay buffer, survives client disconnect
Opt in on POST /v1/chat/completions when the client sends
X-Stream-Resume: 1 and a non empty X-Conversation-Id. The conv id is
the session identity end to end, no extra opaque token. The drain
runs detached server side and buffers SSE bytes, the generation
survives HTTP disconnect, F5, or lets users switch from iOS Safari
to another app without losing the actively generated response.
Routes:
GET /v1/stream/<conv_id>?from=N replay
GET /v1/streams[?conversation_id=X] list, drives sidebar spinners
DELETE /v1/stream/<conv_id> Stop, idempotent
Router parent fans out to children for list and delete, probes on GET
to route to the owner, fans out DELETE on POST so "one session per
conv" holds across model swaps.
WebUI: the layout snapshots /v1/streams at mount and on
visibilitychange, the sidebar reflects live inferences across all
convs. The chat page reattaches on mount, append vs fresh is detected
from existing content so continue mid stream keeps its prefix.
update_slots: on llama_memory_seq_rm refusal at a deep position, full
clear of the seq and reprefill from zero instead of GGML_ABORT.
OAI strict path unchanged when the opt in headers are absent.
* server: create stream session only after post_tasks succeeds
* server, ui: drop X-Stream-Resume, X-Conversation-Id alone enables the replay buffer
* server: drop magic 17, derive the X-Conversation-Id header length from sizeof at build time
* refactor: address review feedback from ngxson
* server-context: cleaning
* server-stream: fix use-after-free on rd
Guard stop_producer with a shared alive flag, flipped by on_stream_end
before rd dies. Prevents a late cancel (session eviction by a later
POST on the same conv_id, or a DELETE arriving after the producer
ended) from touching a destroyed rd.
* ui: fix cross-conversation contamination
Scope streaming flags per conv so one finishing does not unflag the
others, guard discoverActiveStream against concurrent runs to avoid
duplicate attaches, and stop racing syncRemoteRunningStreams for the
sidebar set.
* server-http: keep request alive in detached SSE drain
The response next() lambda may reach into *request via &req long
after on_complete reset the request shared_ptr. Capture request in
the detached thread so it outlives the drain.
* ui: address review feedback from coder543
Forward Authorization to /v1/stream and /v1/streams fetches, the resumable routes
must obey --api-key like the rest of the API.
Wrap reader.read() in a try/catch, the underlying connection drop rejects with
TypeError instead of resolving done=true, treat it as a premature end of stream
so the existing resume loop kicks in.
Freeze the model at session start in chatStreamingStates.model and thread it
through cancel and resume, the dropdown selection may have changed since the
POST and the server side identity is fixed at that time.
* format
* ui: remove unused selectedModelName
* server-stream: poll session->is_cancelled() in stream_aware_should_stop
Address review feedback from coder543. The cancel propagation through
rd.stop() relies on the slot eventually processing the cancel task and
posting a result that notifies the recv condvar, remove_waiting_task_ids
does not notify directly. Add a defensive poll on session->is_cancelled()
so the producer-side next() loop exits on its next iteration after
cancel() without waiting for the cancel task to round trip through a slot.
* server-stream, ui: replace GET /v1/streams with POST /v1/streams/lookup
Address review feedback from coder543. Listing live sessions leaks the
conversation_id of every concurrent user, which defeats the random UUID
unguessability. The new route takes {conversation_ids: [...]} in the
body and returns matches only for the ids the caller already owns, so
foreign UUIDs stay private. The router fans out the same POST to every
child and aggregates, the WebUI passes the convs visible in its sidebar.
* ui: read conv ids from IndexedDB in syncRemoteRunningStreams
The conversations store is not hydrated yet at +layout onMount, so the
sidebar spinners stayed off for background convs until the user clicked
on them. Read straight from the DB to dodge the init race.
* server-models: deduplicate stream lookup timeouts behind one constant
* ui: extract visibility kick grace into a stream constant, bump to 1000 ms
* make it safer & more simple
* server-stream: survive client disconnect via stream_pipe::finish_producer
After the RAII rewrite the generation stopped the moment the client
disconnected. httplib bails its content provider on the is_peer_alive
check at the top of write_content_chunked, so returning true from the
provider never keeps it producing: the response resets, rd is destroyed
and its task gets cancelled.
Reinstate the disconnect survival inside the pipe. stream_pipe gains
finish_producer, which pumps the response next() into the ring buffer
until the generation ends, and mark_producer_done for the clean wire
end. server-http only triggers them: mark before sink.done on a clean
close, finish in on_complete when the peer left early. No detach, no
stream logic in server-http beyond the trigger, and the strict OAI path
is untouched when no pipe is attached.
Known limitation: finish_producer pumps synchronously on the http
worker, so a disconnected stream keeps its worker busy until the
generation ends. A follow-up will move the drain off the http worker so
no worker is held.
* server-stream: drain disconnected streams on a manager owned thread
The previous commit pumped the post disconnect drain synchronously in
on_complete, on the http worker, so a disconnected stream kept its
worker busy until the generation ended. Under a wave of reloads or tab
closes that pins workers from the pool.
Move the drain off the http worker. on_complete now hands the response
to stream_session_manager::adopt_orphan, which pumps it to completion on
a manager owned thread and releases the worker at once. One thread per
disconnected stream still generating, stored in a list, joined and
reaped on the next adopt, by the GC, and at shutdown. No detach, the
thread lifecycle is fully owned by the manager. needs_drain gates the
handoff so a cleanly finished stream never spawns a thread, and the
strict OAI path stays untouched when no pipe is attached.
stop_gc now cancels sessions before finalizing them, so an in flight
drain sees is_cancelled and exits instead of blocking the shutdown join
until the generation ends naturally.
* ui: add missing JSDoc
* server-stream: drain on the http worker, drop the manager thread
Address @ngxson review: httplib runs a large dynamic pool and a worker
blocked in next() sits on a condvar instead of burning cpu, so draining
the rest of the generation on that worker is fine and much simpler than
a dedicated thread.
on_complete calls finish_producer directly again. Removes adopt_orphan,
the orphan thread list and its reaping, the stop_gc session cancel that
only existed to unblock those threads, and the now dead drain_shutdown
flag.
* server-stream: split stream_pipe into producer and consumer classes
Address @ngxson review: one class covering both ends was messy. stream_pipe
is now a base holding the session and is_cancelled, with stream_pipe_producer
(write, mark_producer_done, finish_producer, cleanup, finalizes on destruct)
and stream_pipe_consumer (read only, no finalize) deriving from it.
Drops the is_producer_ discriminator and its runtime guards, the type now
encodes the role. res.spipe is retyped to shared_ptr<stream_pipe_producer>
since it is only ever a producer. No behavior change.
* server-stream: rename producer methods to unix pipe semantics
Address @ngxson review: mark_producer_done becomes done(), finish_producer
becomes close(), matching a unix pipe write end. The producer_done_ member
follows as done_. write() is unchanged. No behavior change.
* server, ui: route resumable streams via a conv map, persist resume identity
Address ngxson review: drop the polling probe, proxy_post records a conv_id ->
model map and the stream routes resolve the owning child with one lookup. The
map is the single source of truth, the ::model suffix stays for child session
uniqueness but the router never parses it.
UI: the server keys a session by the POST time identity (conv::model), but reload
probed with the bare conv id and missed model tagged sessions, so F5 stopped the
stream and sidebar spinners stayed off. Persist the model and rebuild the exact
identity on resume, single conv and bulk sidebar both send it.
Add unit coverage for the identity round trip.
* ui: resolve continue target by id to stop cross-conversation flash on switch
* ui: skip stream resume when the abort is intentional
* server: move the conv id to model map into a self contained tracker
Address review from ngxson: server_models held two mutexes side by side, the
global one and a bare conv_model_mu guarding a loose map, which made the locking
hard to follow. Wrap the map and its lock in a small conv_model_tracker struct
that owns its mutex, one mutex per struct. The remember, lookup and forget
methods move inline into the tracker, server_models exposes a single conv_models
member and the routes call models.conv_models.lookup and friends. No behavior
change, the map stays the single source of truth for routing resumable streams
to a child.
* ui: replace stream magic values with enums and shared constants
Address review from allozaur: lift the inline literals around the resumable
stream code into named symbols so the intent is explicit and reusable.
* ui: fold the stream resume and discovery helpers into ChatService
Address review from allozaur: drop the two standalone stream-*.service files.
They were used only by the chat service and store, carried no shared state, and
did not follow the static class pattern the other services use, so a separate
abstraction was not warranted. Move the helpers onto ChatService as static
methods. No behavior change, tests now exercise them through ChatService.
* docs: document the SSE replay buffer in server README-dev
Add the resumable streaming section, list stream_session_manager in the
backend component inventory, and link PR 23226 in the related PRs.
* ui: align attachServerStream call with onCompletionId param in handleStreamResponse
* server-http: rename del_ to del to match get and post
* ui: address review feedback from allozaur
* ui: drop duplicate SSE constants, keep sse.ts canonical
* ui: use svelte:document for the visibilitychange listener
address review from allozaur: replace the manual document.addEventListener
in onMount with a declarative <svelte:document onvisibilitychange>. svelte
handles attach, detach and SSR, so the typeof document guard and the onMount
cleanup go away. onMount keeps only the first load snapshot.
* server: trim redundant stream drain comments
Address review from ngxson
* server: balance and clean up stream comments
remove redundant comments and tighten the verbose ones across the resumable
stream code, keeping the concurrency and lifetime rationale that is not obvious
from the code. also fix two stale comments in server.cpp and server-models.h
that still described the old ::model suffix probe and fan out routing, now
replaced by the conv_id -> model map
Address review from ngxson
* ui: balance and clean up stream comments
dedup repeated rationale (frozen conv::model identity, the lookup privacy note,
the abort patterns) down to one canonical spot, tighten the verbose blocks, and
keep the concurrency and resume-offset reasoning. fix stale comments in
stream-identity.ts and chat.service.ts that still described the old loopback
probe and fan out routing, now the conv_id -> model map.
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* ui: show model load progress on the selector trigger
Mirror the in-dropdown stage progress as a thin bar on the selector
trigger, so the active model's load percent stays visible when the menu
is closed. Same status gating and composite fraction as the dropdown
row, so both bars track the selected model in sync.
Suggested-by: Julien Chaumond <@julien-c>
* ui: show model load progress bar on the in-conversation model selector
* ui: tune model load indicator to a pulsing highlight (suggested by @ngxson)
Also wire the indicator onto the mobile sheet trigger, which was missing
it since mobile uses the sheet instead of the dropdown.
* ui: thin (@allozaur) pulsating (@ngxson) model load bar
* ui: model status and load progress via /models/sse feed
* ui: centralize SSE wire-format delimiters into shared constants for the chat and /models/sse parsers
* ui: type /models/sse event names as a ServerModelsSseEventType enum
Address review from allozaur
* server: avoid forwarding auth headers in CORS proxy
* format
* fix test
* fix e2e test
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* ui : add model selector storybook stories
Covers list, favorites, single-model, all status states
(loading/loaded/sleeping/failed/idle), and selection states.
* ui : improve model selector mobile UX with hover media queries
Use @media (hover:none) to show action buttons directly on touch
devices and color-code them by model status (amber=sleeping,
green=loaded, muted=idle). Status dots hidden on touch. Desktop
hover behavior unchanged.
* webui: export conversations as jsonl
each session is one jsonl file, a session header line followed by one line per message
exporting multiple conversations bundles them into a zip, one jsonl file each
* webui: import jsonl and zip conversation exports
parse the new jsonl session format and zip archives on import
keep supporting the legacy json format
* UI : fix SSE transport detection and routing through CORS proxy. Assisted-by: Antigravity
* ui : replace magic strings with constants in MCP transport handling
* ui: add source toggle to mermaid and svg blocks
Add a toggle button next to copy and preview that switches a rendered
mermaid or svg block to its source code and back. The button is shared by
both block types and the rendered view stays the default.
The source view reuses the code block scroll container and the highlighted
code element captured at transform time, so it matches the app code blocks
without highlighting again.
Make tall diagrams scroll like text code blocks: safe centering keeps the
diagram centered when it fits and falls back to start alignment when it
overflows, so the top stays reachable instead of clipping above.
Keep the block header opaque and layered above the scrolled diagram, and
ignore header clicks in the zoom handler, so a button click never falls
through to the zoom dialog.
* ui: transparent diagram block header, address review from @allozaur
* ui: add svg block visualizer based on allozaur's mermaid PR
* ui: rationalise diagram block styling and pre transforms shared by mermaid and svg
* ui: live render streaming svg blocks
* ui: also render svg authored in xml code fences
* ui: refactor svg block rendering, address review from allozaur
- Move the svg size ceiling and DOMPurify config out of sanitize-svg.ts into /constants.
- Rename the svg-diagram class to svg-block so the name no longer implies diagrams only.
- Replace the svg, xml and svg tag magic strings in the markdown pipeline with shared constants.
- Promote the data-svg-rendered marker and its sibling data attributes to constants.
* ui: render svg blocks in a shadow root for animation and live zoom
Mount each sanitized svg inside an open shadow root so author <style> and
keyframe or smil animations run while staying scoped to the host element.
Relax the sanitizer to forbid only foreignObject and script, which lets
animation, href and external resource refs through for wider compatibility.
Render the inline block and the zoom dialog from the same reactive source,
so a streaming svg keeps drawing live inside the open zoom popup.
* Add boilerplate for file types
* Add heic-to and implement conversion
* Load heic library from CDN
* Use jpg instead of png for conversion
* Move const to constants file
* ui: make mobile layout keyboard-aware via interactive-widget and dvh shell anchor
* ui: fix duplicate PWA refresh popup by scoping the storage check to non-PWA pages
* server: clean up static assets handling
* nits
* simplify file name handling, use static file name everywhere
* cmake/ui : bundle UI assets in an archive
* ui : run prettier on post-build.js
---------
Co-authored-by: Alde Rojas <hello@alde.dev>
* ui: bake jpeg exif orientation into uploaded images
stb_image in mtmd ignores exif metadata, so rotated smartphone photos
reach the model with raw pixel orientation. The webui now reads the
exif orientation tag at send time and feeds it into the existing
capImageDataURLSize canvas pass: the browser applies the rotation when
decoding, so capped images come out upright for free, and images under
the cap threshold get a single plain redraw when orientation > 1.
At most one re-encode ever happens per image. Upright jpegs with
capping disabled pass through untouched, bit perfect.
Adds jpeg-orientation.ts with a minimal exif parser working on a
bounded base64 prefix (both endianness, returns 1 on any malformed
input) and unit tests against handcrafted jpeg byte streams.
* ui: move jpeg exif constants into lib/constants
* ui: add browser test for jpeg orientation and capping
Covers capImageDataURLSize end to end in chromium with real Pillow
generated jpeg fixtures across exif orientations 1/3/5/6/8: upright
quadrant colors checked pixel-wise, expected dimensions with and
without capping, no orientation tag left in the output, and strict
passthrough when nothing needs rewriting.
* webui: implement pinned conversations support
* webui: linter/prettier pass
* Fix the unused handleMobileSidebarItemClick from the component.
* the search should find pinned conversations as well
Co-authored-by: Pascal <admin@serveurperso.com>
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* ui: add opt-in run_javascript frontend tool
Expose a run_javascript tool to the model, executed entirely in the
browser through the existing agentic loop. Code runs in a Web Worker
inside a sandboxed iframe with an opaque origin, isolated from the
WebUI and its API. Console output, errors and the return value are
fed back as the tool result. The parent enforces a hard timeout by
removing the iframe, which terminates the worker.
Disabled by default, toggle in Settings > Developer.
* ui: address review feedback from allozaur
Use the JsonSchemaType enum for the tool definition parameter types
instead of raw string literals, extending it with STRING and NUMBER.
Move the worker shim and the iframe harness html into their own files
so the service no longer carries inline source blobs.
Replace the remaining magic strings with constants: SANDBOX_EMPTY_OUTPUT
and SANDBOX_TRUNCATION_NOTICE, and reuse NEWLINE_SEPARATOR for joins.
* ui: move sandbox worker shim to a raw imported file
Replace the inline worker template string with a real sandbox-worker.js
imported as raw text, and build the iframe harness from it in
sandbox-harness.ts. The raw worker ships as a string, not a module, so
it is excluded from eslint and the typecheck program.
* use child snippets for landing and chat message elements
* make ... icon visible in conversation history menu
* conversation history forward tab fix
* add snippet fix for fork icon in conversation history
* focus/keyboard fix for attachment x icon and scroll left/right
* formatting
* fix scroll down issue
* simply Statistics and pointer events in scrolldown
* create storybook tests and move to folder
* improve tests to actually assert on element
* chore(ui): pin package versions to currently installed
- Update all dependencies and devDependencies to match exactly what's in package-lock.json
- This ensures reproducible builds by locking to specific versions rather than semver ranges
* chore: Update packages
* chore: Move remaining dependencies to devDependencies
* fix: Add missing `mermaid` package
* chore: Update `cookie` package to `v1.1.1`
* chore: Formatting
* test: Update test configs
* webui: fix tool selector toggle/counter, key tools by stable identity
Key the disabled set, counts and toggles by a stable per-tool key
instead of bare function name, deduped from one canonical list. Per-tool
checkboxes become presentational (single row handler, no nested button),
category checkboxes drop the tristate (n/total carries partial). One
getEnabledToolsForLLM keeps normalized MCP schemas and dedupes by name.
* ui: use SvelteSet and SvelteMap for local tool collections to satisfy svelte/prefer-svelte-reactivity
Previously error to string conversion was split in two different files,
with one converting errors into strings, and another function analyzing
those strings to generate yet another string.
Now the the error handling for network fetches has been centralised and
uses directly HTTP error codes whereas possible to generate the
human-readable error strings.
It also fixes an issue where all JSON errors reported from the backend,
such as "Invalid API key", would get turned incorrectly in to
"Failed to connect to server" due to poor matching logic in the
now-gone getErrorMessage function.
* feat: Add "Thinking" toggle and status icon + redesign Chat Form Actions Add panel
* test: Update test reference
* fix: Icon
* fix: E2E test command
* fix: wait for greeting h1 to be visible in e2e test
* fix: remove duplicate PDF option in attachment dropdown
* fix: use label-based group toggle to avoid stale references
* refactor: inline MCP server and tool toggles in mobile sheet
* fix: serve correct build directory in e2e playwright config
* feat: add reasoning effort levels selector in model dropdown
* feat: Reasoning effort
* refactor: Make server origin configurable via environment variable
* feat: Add chat template thinking detector utility
* feat: Add thinking support detection to models store
* refactor: Update model selector components with thinking detection and message-specific indicators
* feat: Update chat form components for model selection and thinking support
* feat: Improve Reasoning controls UI
* refactor: Apply suggestions from code review
Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
* fix: Model tags
* refactor: Cleanup
* refactor: Remove unneeded components
* refactor: Cleanup
* server: real-time reasoning interruption via control endpoint
Builds on the manual reasoning budget trigger from #23949. Adds a
CONTROL task that mirrors the CANCEL path on the live slot and calls
common_sampler_reasoning_budget_force to end thinking mid-generation.
POST /v1/chat/completions/control with { id_slot, action }, opt-in
reasoning_control arms the budget sampler on demand. Router and single
model. Minimal WebUI button as a skeleton for further UI work.
* ui: track reasoning phase via explicit streaming state
Add isReasoning to the chat store, mirroring the isLoading pattern:
per conversation map, private setter, public accessor and reactive
export. Set from the stream callbacks, true on reasoning chunks, false
on the first content chunk, reset on stream end and resynced on
conversation switch. The skip button now keys off isReasoning so it
shows only during the thinking phase, not the whole generation.
* ui: extract control endpoint and action into constants
Move the chat completion routes, the slots route and the reasoning
control action out of chat.service into api-endpoints and a dedicated
control-actions module. No behavior change, drops the magic strings so
the control protocol has a single source of truth.
* server: target reasoning control by completion id
Address @ngxson review on the control endpoint.
Switch from id_slot to the chat completion id to avoid a TOCTOU: the
slot can be reassigned between the lookup and the control request, so
matching the live completion (oaicompat_cmpl_id) is safe and a finished
one simply matches nothing. Rename the action to reasoning_end, guard
it on the reasoning_control flag of the target slot, and reduce the
response to {success} with an optional message.
* ui: target reasoning control by completion id
Keep the streamed completion id on the message and post it back to the
control endpoint instead of probing /slots. Drops the slot discovery
and the TOCTOU that came with it. Action renamed to reasoning_end,
response read as {success}.
* server: address review from @ngxson
Move the control fields into task_params and drop the redundant
comments on the control path.
* server: document the reasoning control endpoint
* Update tools/ui/src/lib/types/database.d.ts
Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
* ui: rename cmplId to completionId
Per @allozaur review, clearer name for the streamed completion id.
* ui: wire completion id capture through the agentic flow
The webui streams through the agentic flow, which relayed onModel but
not onCompletionId, so the completion id never reached the message and
the control request was never sent. Relay it through the flow and its
callbacks type, declare id on the chunk type, and log an explicit error
when the button fires without a usable id.
* ui: target reasoning control model from the message
The model is a property of the completion, so read it from the streaming
message like the id, not from the model dropdown which is unrelated UI
state. Makes the request self-consistent by construction instead of just
unlikely to drift.
---------
Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
* webui: add custom CSS injection via config
register a customCSS setting in the Developer section under Custom JSON,
syncable so it rides the existing ui-config pass through. inject the value
into a single style element in the head, reactive on the setting. lets an
operator theme a prebuilt binary through --ui-config without rebuilding,
and lets a user set it from the settings panel.
* ui: address review from @niutech and @allozaur, rename custom JSON key and CSS field
* ui: address review from @allozaur, move custom CSS injection to a style tag in svelte:head
* ui: inject custom CSS through a svelte action instead of a bound element
move the textContent write into a use: action on the head style node.
the action is the idiomatic way to touch a node, so the no-dom-manipulating
lint rule is satisfied without a disable. value stays text through
textContent, never parsed as HTML.
* Update tools/ui/src/lib/constants/settings-keys.ts
Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
* ui: address review from @allozaur, rename custom config key to customJson with migration
rename the custom config key to customJson across the type, the chat
request builder, the settings save check and the custom tools reader,
keeping the custom API param name unchanged. add a non destructive
migration that copies the legacy custom key to customJson at startup.
only render the head style tag when custom CSS is set.
---------
Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
When model props are fetched asynchronously from the server,
modelPropsVersion is incremented to trigger reactivity, but
only the vision effect was listening to it.
* allow caching of ui elements in llama-server
* use fnv_hash
* Update tools/server/server-http.cpp
etag has to be set always
Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com>
---------
Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com>
* webui: Add max image size option
* remove magic numbers
* support all image formats
* use const
* Move regex to match b64 images to constants
* use SETTINGS_KEYS to get max image resolution setting
* Do not touch the image if already under the size threshold
Sanjay Ahari