Key changes:
- add cuda13 container for Ampere to Blackwell GPUs
- support DGX Spark (GB10) with arm64 cuda 13 manifests
- enable ik_llama on vulkan container
- map `LLAMA_SWAP_*` env vars to llama-server CLI flags with backwards support
- refactor GHA/docker build to accept variants
closes: #905, #1066
supercedes: PR #841, PR #1084
A `setParams` / `setParamsByID` key ending in `?` is set-if-undefined:
the value is applied only when the request doesn't already carry that
parameter. Plain keys keep forcing their values, and a config that
doesn't use the suffix behaves exactly as before.
Fixes: #1052
Hide tailnet server token in the UI and add an easy to copy/paste
Peer config that is easy to send to connect to the server.
- Add missing configuration overview documentation
- Fix out of date docs
- Fix minor bug for listing tailcat server models
Fixes#1092
Refresh documentation suggestions and streamline the Help experience.
- align sidebar footer links and add topic refresh action
- remove the speculative-decoding article and stale search hint
- increase agent loop limit while hiding iteration display
The Docs Agent's get_config tool would fail on very large
configs with lots of models. Replace it with a jq expression
so the agent can request very specific parts of the configuration
without adding too many extra tokens to the context.
- add jq expression support to Doc Agent's Tools
- move Help out of the Playground into the main sidebar
- add more tips to the Help front page
Updates #1085
Expose llama-swap through Tailcat virtual TCP listeners and route peer
requests through Tailcat transports.
- validate server identities, caller allowlists, and published model IDs
- restrict the default remote HTTP surface and retain Tailcat request sources
- add UI status, activity attribution, configuration docs, and schema support
- make Tailcat transport diagnostics opt-in with server.tailcat.debug
fixes: #1073
- Add option to specify CUDA version and architectures during image
building. If not provided, it should behave just like now.
- Include Stable Diffusion UI in SD binaries.
- Fix ik_llama.cpp build for arm64 architecture:
https://github.com/ikawrakow/ik_llama.cpp/issues/1684.
Big refactor to split the build stages for the unified container to be built in parallel.
- parallel building of binaries to speed up full container build (~6+hr to 1.5hr)
- split cuda and vulkan pipelines to be independent (vulkan is much faster)
- establish pattern for building building binaries for final image (easier to add new resources)
- use llama-swap-build for build containers to avoid untagged clean up script
Fixes: #1069
Add an Docs agent to the playground that can help the user with more
advanced configuration.
- add mcp 2026-07-28 (stateless mcp) framework
- add MCP tools for doc search and config search
- add initial set of guides for key topics
- add Docs to the Playground
- removed out of data documentation and plans
An optional `hooks.on_startup.profile` setting for activating a profile
on startup and after a configuration reload:
```yaml
profiles:
coding:
pins:
llm-code: "gpt-oss-120b"
hooks:
on_startup:
profile: "coding"
```
Another attempt to close#992 after unsuccessful #993 ;)
Co-authored-by: David Soušek <david.sousek@intelogy.co.uk>
Some OpenAI-compatible gateways read the context size from a
**context_window** field on the models endpoint (e.g. Bifrost, following
GROQ's convention) instead of **context_length**.
llama-swap only exposed the latter, so such gateways silently dropped
the context size configured in model capabilities.
Add context_window to the /v1/models and /models responses, mirroring
the capabilities-based context_length value, so both field names carry
the same number. The field is renderer-owned like context_length: a
context_window key in a model's metadata block is dropped when
capabilities are rendered, matching the existing behavior.
Tests assert the mirrored value alongside context_length, including the
metadata precedence and passthrough cases.
---
Replaces #1062 (closed when the head repository was reworked).
Co-authored-by: Chris <chrispaulm@users.noreply.github.com>
add a -api-key option that the user can optionally provide to give
wol-proxy a private api key it will use for authenticating its
health checks against the llama-swap server.
This is api key is not used as part of any client request, and
clients connecting to the wol-proxy will still need to provide their
own api keys matching one of the ones in the destination llama-swap's
config.yaml
Fixes: #1051
If a server suspends without hanging up the connection, the wol-proxy
may be left with an open, but unresponsive connection.
this PR adds a simple check to ensure connection remains responsive
if not, we assume system suspended and attempt to re-send the wol
packet. While this is assumption, it is relatively safe since sending
extra unneeded wol packets would have little negative impact if that
happened.
fixes#1056
capabilities.in and capabilities.out accepted text, audio and image but not video, so a multimodal model that takes video input could not declare it and /v1/models reported it as image-only.
video is added to the shared validModalities set, which makes it valid on both in and out; tests assert both sides explicitly so that stays a decision rather than a side effect of the two lists sharing one map.
Updated together so no layer disagrees: the validModalities map, both validation error messages, both config-schema.json enums, and the valid-values comments in config.example.yaml.
Fixes: #1014
Adds a -validate boolean flag that loads the config via the existing
config.LoadConfigSources and exits without starting the server,
detecting hardware, or binding a listener. Exits 0 when the config loads
cleanly and 1 when it does not, printing a one-line result either way.
The logic lives in runValidate(configPath, configDir string, out
io.Writer) int rather than inline in main() so it can be tested
directly.
Fixes: #1034
A client that hangs up before a response is written was logged and
recorded as a successful 200 with a 0-byte body: the cancellation
branches return without touching the ResponseWriter, so the access log's
seeded 200 was what got reported, and the metrics path filed it through
the "empty body, recording minimal metrics" success arm. Aborted
requests were invisible to status-code monitoring. Once the model was
loaded, the same hangup surfaced as a 502, blaming a healthy upstream.
Add swaputil.StatusClientClosedRequest (nginx's non-standard 499) as the
sentinel. It is recorded only, never written to the connection: the
client is already gone, and on a streamed response a late WriteHeader
would just log "superfluous response.WriteHeader".
- add StatusMarker/MarkClientClosed to swaputil; the response recorders
implement it and forward outward so log and metrics agree
- derive the sentinel in the access-log and metrics middleware, which
covers every cancellation branch instead of each one separately
- classify cancellation in the model and peer proxy ErrorHandlers so a
client hangup no longer reports 502
- record cancelled requests with a client-disconnected ErrorMsg at debug
level and without a capture, since an impatient caller is normal
traffic rather than a server fault
- tag health check polls so a booting upstream is not logged as a proxy
error once per second
A response that already started keeps the status the client actually
received.
fix: #1029
vLLM started with --per-request-spec-decode-metrics reports draft token
counts in the response's metrics.speculative_decoding object. Map those
onto the existing draft token fields so the activity table's drafted
acceptance rate works for vLLM the same way it already does for
llama-server's timings.
- read num_draft_tokens/num_accepted_draft_tokens from
metrics.speculative_decoding
- require both counters so the rate is never derived from a partial
object
- cover non-streaming, streaming and partial-object responses in tests
fix: #1032
Error responses used a string-valued "error" field, so OpenAI clients
that read body["error"]["message"] hit a string where they expected an
object. Home Assistant's OpenAI-style conversation integration turns
that into an opaque 500 instead of a retryable "server busy" signal
when concurrencyLimit sheds a request.
Every JSON error body now uses the OpenAI envelope, with error.type and
error.code derived from the HTTP status:
{"src":"llama-swap",
"error":{"message":"Too many requests","type":"rate_limit_error",
"param":null,"code":"concurrency_limit"}}
- add ErrorEnvelope/ErrorDetail and NewErrorEnvelope() in swaputil
- render SendResponse's JSON branch and ConcurrencyLimitError.Body()
through the envelope, keeping status codes and Retry-After unchanged
- route the two remaining http.Error() call sites (process not ready,
peer proxy failure) through SendResponse so they use the envelope too
- keep the top-level "src" marker and the text/plain and text/html
response formats as they were
fix: #1037
Co-authored-by: Claude <noreply@anthropic.com>
Improve vllm-wrapper support for native vLLM installations and systemd-managed vLLM processes.
It introduces three related features:
Passing the vLLM startup command as separate arguments after --.
Forwarding logs from a systemd user unit through the wrapper with --journal-unit.
Terminating the active proxy after a successful sleep request with --stop-pid.
Add an opt-in setting to surface capability badges next to each model on the Models list.
All capability tags from the Details tab are shown in canonical order,
each with a distinct muted pastel color; the context window badge sits
last (rightmost). Disabled by default to keep the list dense; toggle in Settings.
LLM Disclosure: Yes, GLM-5.2 helped along
Fixes#1006
Some tools are trying to auto-detect max context size for models
using llama.cpp's /models endpoint instead of /props since it is
exposed in both places.
Add the meta.n_ctx to the /models response to improve compatibility
with other tools.
updates: #999
Add a dedicated /comfyui/ passthrough backed by the reserved
comfyui_auto local model.
- preserve escaped upstream paths such as encoded workflow separators
- allow only the root path to start an unloaded ComfyUI model
- enforce a minimum concurrency limit of 50 for the reserved model
- add example upstream.ignorePaths to prevent unintentional swaps by
ComfyUI
- add model.workarounds.ignoreWebsockets so ComfyUI does not block
swapping
fixes: #1001fixes: #1000
Detect the hardware available to the inference host and expose it
through the API and UI.
- Detect operating system, CPU, memory, and accelerator details
- Provide a hardware snapshot endpoint with platform-specific collectors
- Show an overview and copyable text summary in the UI
update: #977
Surface selector strategy and targets via the existing /v1/models
listing (meta.llamaswap) and render them in a Selectors card under the
Profiles card.
- add strategy, targets, and spillover to selector records in /v1/models
- map selector metadata onto Model in the playground models loader
- render Selectors card: id - name, description, linked targets
- move the unlisted-models toggle into the Local models header
- rename "Profile models" section to "Profiles"
fix: #974
Show active profile mappings separately from configured local and peer
models.
- display profile pin targets and disabled mappings
- group local and peer models into dedicated sections
- identify the selected profile with an active status badge
fix: #961
Compile matrix DSL expressions into an immutable AST/DAG and evaluate
only the requested and running models when choosing evictions. This
removes eager Cartesian expansion and the maxDSLExpansions limit.
- move DSL parsing and symbolic evaluation into internal/matrix
- resolve aliases and link named references during compilation
- project each runtime query onto target and running-model bitsets
- memoize AST nodes and prune duplicate or dominated mask states
- preserve ordered tie-breaking and reconstruct full witness sets
- use symbolic containment for spillover selector validation
- compile programmatic matrix configurations on router creation
- document the design, complexity, and concurrency guarantees
- test one million theoretical combinations and over 64 models
- benchmark 100, 1K, 10K, and 100K theoretical combinations
Benchmarks ran on QEMU Virtual CPU 2.5+ with three 500 ms samples.
At 10K theoretical combinations:
- validation: 9.71 ms, 6.53 MB, and 81,401 allocs before; 12.38 us, 14.3
KB, and 122 allocs after
- solver setup: 1.31 ms and 1.37 MB before; 22.4 ns and 16 B after
- eviction solve: 809 us and 16 B before; 9.04 us and 7.97 KB after
At 100K theoretical combinations with the symbolic implementation:
- validation: 15.10 us, 16.1 KB, and 149 allocs
- solver setup: 23.6 ns, 16 B, and one allocation
- eviction solve: 16.17 us, 13.7 KB, and 193 allocs
At 100 combinations, solve time increases from 3.07 us to 4.72 us, and
allocation rises from 16 B to 3.82 KB. At 1K combinations, solve time
drops from 50.8 us to 7.23 us.
Written to GPT 5.6-Sol (high). Verified and reviewed with Opus 5 and
Kimi K3.
I don't understand what's going because these models know more than me
on writing a DSL compiler.
fixes#951
Resolve configuration macros against an untyped YAML representation
before decoding the typed Config. This materializes YAML anchors and
preserves scalar types without maintaining field-specific replacement
paths.
- apply environment, global, and model macros in their defined scopes
- centralize MODEL_ID, PORT, PID, key, and unknown-macro handling
- remove the capability-specific raw decoder and replacement loops
- document macro ordering, scope, and runtime behavior
- add coverage for anchors, scope isolation, validation, and runtime
macros
fixes#919
