A `setParams` / `setParamsByID` key ending in `?` is set-if-undefined:
the value is applied only when the request doesn't already carry that
parameter. Plain keys keep forcing their values, and a config that
doesn't use the suffix behaves exactly as before.
Fixes: #1052
Hide tailnet server token in the UI and add an easy to copy/paste
Peer config that is easy to send to connect to the server.
- Add missing configuration overview documentation
- Fix out of date docs
- Fix minor bug for listing tailcat server models
Fixes#1092
Refresh documentation suggestions and streamline the Help experience.
- align sidebar footer links and add topic refresh action
- remove the speculative-decoding article and stale search hint
- increase agent loop limit while hiding iteration display
The Docs Agent's get_config tool would fail on very large
configs with lots of models. Replace it with a jq expression
so the agent can request very specific parts of the configuration
without adding too many extra tokens to the context.
- add jq expression support to Doc Agent's Tools
- move Help out of the Playground into the main sidebar
- add more tips to the Help front page
Updates #1085
Expose llama-swap through Tailcat virtual TCP listeners and route peer
requests through Tailcat transports.
- validate server identities, caller allowlists, and published model IDs
- restrict the default remote HTTP surface and retain Tailcat request sources
- add UI status, activity attribution, configuration docs, and schema support
- make Tailcat transport diagnostics opt-in with server.tailcat.debug
fixes: #1073
Add an Docs agent to the playground that can help the user with more
advanced configuration.
- add mcp 2026-07-28 (stateless mcp) framework
- add MCP tools for doc search and config search
- add initial set of guides for key topics
- add Docs to the Playground
- removed out of data documentation and plans
An optional `hooks.on_startup.profile` setting for activating a profile
on startup and after a configuration reload:
```yaml
profiles:
coding:
pins:
llm-code: "gpt-oss-120b"
hooks:
on_startup:
profile: "coding"
```
Another attempt to close#992 after unsuccessful #993 ;)
Co-authored-by: David Soušek <david.sousek@intelogy.co.uk>
Some OpenAI-compatible gateways read the context size from a
**context_window** field on the models endpoint (e.g. Bifrost, following
GROQ's convention) instead of **context_length**.
llama-swap only exposed the latter, so such gateways silently dropped
the context size configured in model capabilities.
Add context_window to the /v1/models and /models responses, mirroring
the capabilities-based context_length value, so both field names carry
the same number. The field is renderer-owned like context_length: a
context_window key in a model's metadata block is dropped when
capabilities are rendered, matching the existing behavior.
Tests assert the mirrored value alongside context_length, including the
metadata precedence and passthrough cases.
---
Replaces #1062 (closed when the head repository was reworked).
Co-authored-by: Chris <chrispaulm@users.noreply.github.com>
capabilities.in and capabilities.out accepted text, audio and image but not video, so a multimodal model that takes video input could not declare it and /v1/models reported it as image-only.
video is added to the shared validModalities set, which makes it valid on both in and out; tests assert both sides explicitly so that stays a decision rather than a side effect of the two lists sharing one map.
Updated together so no layer disagrees: the validModalities map, both validation error messages, both config-schema.json enums, and the valid-values comments in config.example.yaml.
Fixes: #1014
A client that hangs up before a response is written was logged and
recorded as a successful 200 with a 0-byte body: the cancellation
branches return without touching the ResponseWriter, so the access log's
seeded 200 was what got reported, and the metrics path filed it through
the "empty body, recording minimal metrics" success arm. Aborted
requests were invisible to status-code monitoring. Once the model was
loaded, the same hangup surfaced as a 502, blaming a healthy upstream.
Add swaputil.StatusClientClosedRequest (nginx's non-standard 499) as the
sentinel. It is recorded only, never written to the connection: the
client is already gone, and on a streamed response a late WriteHeader
would just log "superfluous response.WriteHeader".
- add StatusMarker/MarkClientClosed to swaputil; the response recorders
implement it and forward outward so log and metrics agree
- derive the sentinel in the access-log and metrics middleware, which
covers every cancellation branch instead of each one separately
- classify cancellation in the model and peer proxy ErrorHandlers so a
client hangup no longer reports 502
- record cancelled requests with a client-disconnected ErrorMsg at debug
level and without a capture, since an impatient caller is normal
traffic rather than a server fault
- tag health check polls so a booting upstream is not logged as a proxy
error once per second
A response that already started keeps the status the client actually
received.
fix: #1029
vLLM started with --per-request-spec-decode-metrics reports draft token
counts in the response's metrics.speculative_decoding object. Map those
onto the existing draft token fields so the activity table's drafted
acceptance rate works for vLLM the same way it already does for
llama-server's timings.
- read num_draft_tokens/num_accepted_draft_tokens from
metrics.speculative_decoding
- require both counters so the rate is never derived from a partial
object
- cover non-streaming, streaming and partial-object responses in tests
fix: #1032
Error responses used a string-valued "error" field, so OpenAI clients
that read body["error"]["message"] hit a string where they expected an
object. Home Assistant's OpenAI-style conversation integration turns
that into an opaque 500 instead of a retryable "server busy" signal
when concurrencyLimit sheds a request.
Every JSON error body now uses the OpenAI envelope, with error.type and
error.code derived from the HTTP status:
{"src":"llama-swap",
"error":{"message":"Too many requests","type":"rate_limit_error",
"param":null,"code":"concurrency_limit"}}
- add ErrorEnvelope/ErrorDetail and NewErrorEnvelope() in swaputil
- render SendResponse's JSON branch and ConcurrencyLimitError.Body()
through the envelope, keeping status codes and Retry-After unchanged
- route the two remaining http.Error() call sites (process not ready,
peer proxy failure) through SendResponse so they use the envelope too
- keep the top-level "src" marker and the text/plain and text/html
response formats as they were
fix: #1037
Co-authored-by: Claude <noreply@anthropic.com>
Add an opt-in setting to surface capability badges next to each model on the Models list.
All capability tags from the Details tab are shown in canonical order,
each with a distinct muted pastel color; the context window badge sits
last (rightmost). Disabled by default to keep the list dense; toggle in Settings.
LLM Disclosure: Yes, GLM-5.2 helped along
Fixes#1006
Some tools are trying to auto-detect max context size for models
using llama.cpp's /models endpoint instead of /props since it is
exposed in both places.
Add the meta.n_ctx to the /models response to improve compatibility
with other tools.
updates: #999
Add a dedicated /comfyui/ passthrough backed by the reserved
comfyui_auto local model.
- preserve escaped upstream paths such as encoded workflow separators
- allow only the root path to start an unloaded ComfyUI model
- enforce a minimum concurrency limit of 50 for the reserved model
- add example upstream.ignorePaths to prevent unintentional swaps by
ComfyUI
- add model.workarounds.ignoreWebsockets so ComfyUI does not block
swapping
fixes: #1001fixes: #1000
Detect the hardware available to the inference host and expose it
through the API and UI.
- Detect operating system, CPU, memory, and accelerator details
- Provide a hardware snapshot endpoint with platform-specific collectors
- Show an overview and copyable text summary in the UI
update: #977
Surface selector strategy and targets via the existing /v1/models
listing (meta.llamaswap) and render them in a Selectors card under the
Profiles card.
- add strategy, targets, and spillover to selector records in /v1/models
- map selector metadata onto Model in the playground models loader
- render Selectors card: id - name, description, linked targets
- move the unlisted-models toggle into the Local models header
- rename "Profile models" section to "Profiles"
fix: #974
Compile matrix DSL expressions into an immutable AST/DAG and evaluate
only the requested and running models when choosing evictions. This
removes eager Cartesian expansion and the maxDSLExpansions limit.
- move DSL parsing and symbolic evaluation into internal/matrix
- resolve aliases and link named references during compilation
- project each runtime query onto target and running-model bitsets
- memoize AST nodes and prune duplicate or dominated mask states
- preserve ordered tie-breaking and reconstruct full witness sets
- use symbolic containment for spillover selector validation
- compile programmatic matrix configurations on router creation
- document the design, complexity, and concurrency guarantees
- test one million theoretical combinations and over 64 models
- benchmark 100, 1K, 10K, and 100K theoretical combinations
Benchmarks ran on QEMU Virtual CPU 2.5+ with three 500 ms samples.
At 10K theoretical combinations:
- validation: 9.71 ms, 6.53 MB, and 81,401 allocs before; 12.38 us, 14.3
KB, and 122 allocs after
- solver setup: 1.31 ms and 1.37 MB before; 22.4 ns and 16 B after
- eviction solve: 809 us and 16 B before; 9.04 us and 7.97 KB after
At 100K theoretical combinations with the symbolic implementation:
- validation: 15.10 us, 16.1 KB, and 149 allocs
- solver setup: 23.6 ns, 16 B, and one allocation
- eviction solve: 16.17 us, 13.7 KB, and 193 allocs
At 100 combinations, solve time increases from 3.07 us to 4.72 us, and
allocation rises from 16 B to 3.82 KB. At 1K combinations, solve time
drops from 50.8 us to 7.23 us.
Written to GPT 5.6-Sol (high). Verified and reviewed with Opus 5 and
Kimi K3.
I don't understand what's going because these models know more than me
on writing a DSL compiler.
fixes#951
Resolve configuration macros against an untyped YAML representation
before decoding the typed Config. This materializes YAML anchors and
preserves scalar types without maintaining field-specific replacement
paths.
- apply environment, global, and model macros in their defined scopes
- centralize MODEL_ID, PORT, PID, key, and unknown-macro handling
- remove the capability-specific raw decoder and replacement loops
- document macro ordering, scope, and runtime behavior
- add coverage for anchors, scope isolation, validation, and runtime
macros
fixes#919
Address peer models by fully qualified names across routing, selectors,
model listings, and the UI.
- support fully qualified peer/model routing names
- support peer models in selector spillover targets
- show peer models in Playground model pickers
fixes#944
## Summary
Fixes issue #946 where a request arriving during a process stop would
hang forever. The router was reading process state and making start/stop
decisions based on that snapshot, which could become stale by the time
the decision was acted upon. This change moves the start decision into
the process's own state machine where it can be made atomically.
## Key Changes
- **Renamed `runReq` to `startReq`** with a new `block` field to
distinguish between `Run()` (blocking until process terminates) and the
new `EnsureReady()` (non-blocking, answers when ready or failed)
- **Added `EnsureReady()` method** to the `Process` interface and
`ProcessCommand` that brings a process to ready state without blocking
on termination. The decision of whether to start is made inside the
process's run loop, eliminating race conditions
- **Updated process state machine** to:
- Call `notifyWaiters()` on all terminal state transitions (not just
`StateStarting` exits) to prevent subscribers from hanging
- Handle `EnsureReady` requests that arrive during `StateStopping` by
waiting for the stop to complete before starting
- Notify waiters when a process exits unexpectedly or is stopped
- **Updated router's `doSwap()`** to call `EnsureReady()` instead of
reading `State()` and conditionally calling `Run()`. This ensures the
start decision is made atomically within the process's run loop
- **Added comprehensive tests**:
- `TestProcessCommand_EnsureReadyDuringStop`: Regression test verifying
requests during stop wait for completion then start
- `TestProcessCommand_EnsureReadyIsIdempotent`: Verifies idempotent
behavior on ready/shutdown states
- `TestBaseRouter_RequestDuringStop`: Router-level test ensuring
requests during TTL unload don't hang
- Updated `fakeProcess` to mirror `ProcessCommand` behavior with proper
synchronization
- **Updated design documentation** to clarify that advisory state reads
are fine, but mutations must never be gated on state snapshots
## Implementation Details
The core fix is that `EnsureReady()` sends a request to the process's
run loop, which then decides whether to start based on the current
state. This is safe because:
- The run loop is single-threaded and owns the state
- A stop in flight keeps the loop parked in `killProcess()`, so start
requests cannot be received until the stop completes
- The send on `startCh` is the synchronization point that naturally
serializes against in-flight stops
This follows the principle: **advisory reads of process state are fine,
but reads that gate mutations must happen inside the process's own state
machine**.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
- use shared.ReplaceRequestModel everywhere
- remove magic number for multipart form parsing
- add form.RemoveAll() to clean up temp files
Updates #933Fixes#937
Supercedes #939
Add configurable model ID profiles that can be switched without
restarting.
- rewrite request and upstream model IDs through active profile pins
- expose profile selection through API, SSE, and UI
- validate and document profile configuration
Updates #933
Supercedes: #774
Track request and streamed response metadata for active model requests.
- publish keyed updates through a non-blocking recoverable outbox
- add compact configurable activity columns and clock-safe elapsed time
- tag playground requests with a page-scoped session ID
Fixes#912
Reduce initial bundle size and build time by lazy-loading route
components and the always-mounted Playground component, and by
importing only the highlight.js core + a curated language subset and
only the chart.js line-chart pieces instead of the full registerables.
- App.svelte: route components loaded via svelte-spa-router's wrap(),
Playground dynamically imported after mount
- markdown.ts: highlight.js/lib/core + ~35 registered languages instead
of the full ~190-language bundle
- PerformanceChart.svelte: register only LineController/LineElement/etc.
instead of chart.js's full registerables
- vite.config.ts: raise chunkSizeWarningLimit for the now-deferred
Playground chunk
Initial JS chunk drops from ~2.1MB to ~174KB (gzip ~660KB to ~49KB),
and `vite build` time drops from ~16s to ~8s.
Resolve capability macros from an untyped representation so YAML anchors
are materialized before typed values are decoded.
- preserve macro value types for capability fields
- support anchors shared across model configurations
- reject unresolved capability macros
fixes#917
Model macros (e.g. ${default_ctx}) were not resolved inside the
capabilities block because ModelCapConfig has typed fields (int, bool)
that YAML cannot parse macro reference strings into.
Add a custom UnmarshalYAML on ModelCapConfig that captures the raw
yaml.Node, and a ResolveMacros method that marshals it back to YAML,
substitutes all macros (LIFO order), and re-decodes into the typed
struct. Existing Capabilities.Validate() call is subsumed by
ResolveMacros.
- new ModelCapConfig.rawNode field (unexported)
- ModelCapConfig.UnmarshalYAML captures raw node; best-effort direct
decode; resets fields to zero on failure (macro not yet known)
- ModelCapConfig.ResolveMacros: marshal -> LIFO substitute -> decode
- LoadConfigFromReader calls ResolveMacros after the main LIFO loop,
before PORT allocation and unknown-macro validation
- Removes the now-redundant standalone Capabilities.Validate() call
Closes#900
Store activity metrics in SQLite and use the store as the source for
activity APIs.
- add SQLite-backed activity store with migrations and activity stats
queries
- add new store.path configuration option
- add activity pagination and stats API endpoints
- update the activity UI to fetch paginated rows and server-side stats
- add optional store.path configuration for durable activity history
Fixes#310, #729
Track active model-dispatched requests with cancellable contexts and
stream request details to the UI.
- add inflight request snapshots and cancel endpoint
- include supported /upstream inference requests
- render a collapsible inflight table above completed activity
Fixes#893
Reject over-limit requests during scheduler admission so loading SSE
output is not started before a 429 can be returned.
- add admission handshake before loading stream setup
- reserve concurrency slots for queued and swap-waiting requests
- cover queued waiter rejection and streaming 429 regression
fixes#887
add /props to modelGetRotes so that 3p tools like huggingface's official
Pi agent plugin can use this to determine context length for the hosted
model. This mimics the behavior of how llama-server operates in routing
mode when the model name is pased.
This is a partial implementation of the llama.cpp's /props endpoint. A
full implementation was not done to both provide the majority of the
value
while also keeping the code change extremely minimal.
Similarities/differences vs llama-server in router mode:
- /props?model=<model_name> - llama-server and llama-swap have same
behavior
- /props?model=<model_name>&autoload=false - llama-server throws error
if model is not loaded, but llama-swap ignores parameter and loads the
model anyways.
- /props?model=NOT_A_VALID_MODEL - llama-server and llama-swap both
throw errors
- /props - llama-server prints some metadata bout the llama-server
process. llama-swap gives error.
implements #854
A slow web UI log subscriber could stall request processing. Write runs
on the upstream process's stdout drain, and it published to a bounded,
backpressuring event bus. When a browser tab fell behind on verbose
llama.cpp output, the event queue filled, Broadcast blocked Write, the
OS
stdout pipe backed up, and llama.cpp stalled until the tab was closed.
Hand log data to a dedicated broadcaster goroutine over a buffered
channel and make Write's send non-blocking, dropping the live broadcast
instead of blocking. Dropped bytes are accumulated and reported
in-stream
as a "— N bytes dropped —" marker when delivery resumes. GetHistory()
remains lossless for reconnecting clients.
- non-blocking, drop-on-full hand-off in Write
- broadcaster goroutine owns the event bus
- in-stream dropped-bytes marker
fixes#875
---------
Co-authored-by: Claude <noreply@anthropic.com>
- Replace the ui_dist placeholder.txt hack with a build-tag guard.
- Update Makefile and goreleaser to use new tags
- Small UI and build improvements while I'm at it
Set up shadcn-svelte components and adopt its design-token system as the
base for modernizing the UI.
What started as a curiosity on what it could take to switch to shadcn
became a complete UI refactor. Used a combination of Opus 4.8, GLM 5.2
for changes.
Over time the llama-swap configuration file can get really long and
challenging to work with. The -config-dir flag is used for a directory
of configuration YAML fragments.
These fragments are merged together and into a full configuration and
tested for validity. All previous configuration functionality remains
unchanged.
