Big refactor to split the build stages for the unified container to be built in parallel.
- parallel building of binaries to speed up full container build (~6+hr to 1.5hr)
- split cuda and vulkan pipelines to be independent (vulkan is much faster)
- establish pattern for building building binaries for final image (easier to add new resources)
- use llama-swap-build for build containers to avoid untagged clean up script
Fixes: #1069
Add an Docs agent to the playground that can help the user with more
advanced configuration.
- add mcp 2026-07-28 (stateless mcp) framework
- add MCP tools for doc search and config search
- add initial set of guides for key topics
- add Docs to the Playground
- removed out of data documentation and plans
An optional `hooks.on_startup.profile` setting for activating a profile
on startup and after a configuration reload:
```yaml
profiles:
coding:
pins:
llm-code: "gpt-oss-120b"
hooks:
on_startup:
profile: "coding"
```
Another attempt to close#992 after unsuccessful #993 ;)
Co-authored-by: David Soušek <david.sousek@intelogy.co.uk>
Some OpenAI-compatible gateways read the context size from a
**context_window** field on the models endpoint (e.g. Bifrost, following
GROQ's convention) instead of **context_length**.
llama-swap only exposed the latter, so such gateways silently dropped
the context size configured in model capabilities.
Add context_window to the /v1/models and /models responses, mirroring
the capabilities-based context_length value, so both field names carry
the same number. The field is renderer-owned like context_length: a
context_window key in a model's metadata block is dropped when
capabilities are rendered, matching the existing behavior.
Tests assert the mirrored value alongside context_length, including the
metadata precedence and passthrough cases.
---
Replaces #1062 (closed when the head repository was reworked).
Co-authored-by: Chris <chrispaulm@users.noreply.github.com>
add a -api-key option that the user can optionally provide to give
wol-proxy a private api key it will use for authenticating its
health checks against the llama-swap server.
This is api key is not used as part of any client request, and
clients connecting to the wol-proxy will still need to provide their
own api keys matching one of the ones in the destination llama-swap's
config.yaml
Fixes: #1051
If a server suspends without hanging up the connection, the wol-proxy
may be left with an open, but unresponsive connection.
this PR adds a simple check to ensure connection remains responsive
if not, we assume system suspended and attempt to re-send the wol
packet. While this is assumption, it is relatively safe since sending
extra unneeded wol packets would have little negative impact if that
happened.
fixes#1056
capabilities.in and capabilities.out accepted text, audio and image but not video, so a multimodal model that takes video input could not declare it and /v1/models reported it as image-only.
video is added to the shared validModalities set, which makes it valid on both in and out; tests assert both sides explicitly so that stays a decision rather than a side effect of the two lists sharing one map.
Updated together so no layer disagrees: the validModalities map, both validation error messages, both config-schema.json enums, and the valid-values comments in config.example.yaml.
Fixes: #1014
Adds a -validate boolean flag that loads the config via the existing
config.LoadConfigSources and exits without starting the server,
detecting hardware, or binding a listener. Exits 0 when the config loads
cleanly and 1 when it does not, printing a one-line result either way.
The logic lives in runValidate(configPath, configDir string, out
io.Writer) int rather than inline in main() so it can be tested
directly.
Fixes: #1034
A client that hangs up before a response is written was logged and
recorded as a successful 200 with a 0-byte body: the cancellation
branches return without touching the ResponseWriter, so the access log's
seeded 200 was what got reported, and the metrics path filed it through
the "empty body, recording minimal metrics" success arm. Aborted
requests were invisible to status-code monitoring. Once the model was
loaded, the same hangup surfaced as a 502, blaming a healthy upstream.
Add swaputil.StatusClientClosedRequest (nginx's non-standard 499) as the
sentinel. It is recorded only, never written to the connection: the
client is already gone, and on a streamed response a late WriteHeader
would just log "superfluous response.WriteHeader".
- add StatusMarker/MarkClientClosed to swaputil; the response recorders
implement it and forward outward so log and metrics agree
- derive the sentinel in the access-log and metrics middleware, which
covers every cancellation branch instead of each one separately
- classify cancellation in the model and peer proxy ErrorHandlers so a
client hangup no longer reports 502
- record cancelled requests with a client-disconnected ErrorMsg at debug
level and without a capture, since an impatient caller is normal
traffic rather than a server fault
- tag health check polls so a booting upstream is not logged as a proxy
error once per second
A response that already started keeps the status the client actually
received.
fix: #1029
vLLM started with --per-request-spec-decode-metrics reports draft token
counts in the response's metrics.speculative_decoding object. Map those
onto the existing draft token fields so the activity table's drafted
acceptance rate works for vLLM the same way it already does for
llama-server's timings.
- read num_draft_tokens/num_accepted_draft_tokens from
metrics.speculative_decoding
- require both counters so the rate is never derived from a partial
object
- cover non-streaming, streaming and partial-object responses in tests
fix: #1032
Error responses used a string-valued "error" field, so OpenAI clients
that read body["error"]["message"] hit a string where they expected an
object. Home Assistant's OpenAI-style conversation integration turns
that into an opaque 500 instead of a retryable "server busy" signal
when concurrencyLimit sheds a request.
Every JSON error body now uses the OpenAI envelope, with error.type and
error.code derived from the HTTP status:
{"src":"llama-swap",
"error":{"message":"Too many requests","type":"rate_limit_error",
"param":null,"code":"concurrency_limit"}}
- add ErrorEnvelope/ErrorDetail and NewErrorEnvelope() in swaputil
- render SendResponse's JSON branch and ConcurrencyLimitError.Body()
through the envelope, keeping status codes and Retry-After unchanged
- route the two remaining http.Error() call sites (process not ready,
peer proxy failure) through SendResponse so they use the envelope too
- keep the top-level "src" marker and the text/plain and text/html
response formats as they were
fix: #1037
Co-authored-by: Claude <noreply@anthropic.com>
Improve vllm-wrapper support for native vLLM installations and systemd-managed vLLM processes.
It introduces three related features:
Passing the vLLM startup command as separate arguments after --.
Forwarding logs from a systemd user unit through the wrapper with --journal-unit.
Terminating the active proxy after a successful sleep request with --stop-pid.
Add an opt-in setting to surface capability badges next to each model on the Models list.
All capability tags from the Details tab are shown in canonical order,
each with a distinct muted pastel color; the context window badge sits
last (rightmost). Disabled by default to keep the list dense; toggle in Settings.
LLM Disclosure: Yes, GLM-5.2 helped along
Fixes#1006
Some tools are trying to auto-detect max context size for models
using llama.cpp's /models endpoint instead of /props since it is
exposed in both places.
Add the meta.n_ctx to the /models response to improve compatibility
with other tools.
updates: #999
Add a dedicated /comfyui/ passthrough backed by the reserved
comfyui_auto local model.
- preserve escaped upstream paths such as encoded workflow separators
- allow only the root path to start an unloaded ComfyUI model
- enforce a minimum concurrency limit of 50 for the reserved model
- add example upstream.ignorePaths to prevent unintentional swaps by
ComfyUI
- add model.workarounds.ignoreWebsockets so ComfyUI does not block
swapping
fixes: #1001fixes: #1000
Detect the hardware available to the inference host and expose it
through the API and UI.
- Detect operating system, CPU, memory, and accelerator details
- Provide a hardware snapshot endpoint with platform-specific collectors
- Show an overview and copyable text summary in the UI
update: #977
Surface selector strategy and targets via the existing /v1/models
listing (meta.llamaswap) and render them in a Selectors card under the
Profiles card.
- add strategy, targets, and spillover to selector records in /v1/models
- map selector metadata onto Model in the playground models loader
- render Selectors card: id - name, description, linked targets
- move the unlisted-models toggle into the Local models header
- rename "Profile models" section to "Profiles"
fix: #974
Show active profile mappings separately from configured local and peer
models.
- display profile pin targets and disabled mappings
- group local and peer models into dedicated sections
- identify the selected profile with an active status badge
fix: #961
Compile matrix DSL expressions into an immutable AST/DAG and evaluate
only the requested and running models when choosing evictions. This
removes eager Cartesian expansion and the maxDSLExpansions limit.
- move DSL parsing and symbolic evaluation into internal/matrix
- resolve aliases and link named references during compilation
- project each runtime query onto target and running-model bitsets
- memoize AST nodes and prune duplicate or dominated mask states
- preserve ordered tie-breaking and reconstruct full witness sets
- use symbolic containment for spillover selector validation
- compile programmatic matrix configurations on router creation
- document the design, complexity, and concurrency guarantees
- test one million theoretical combinations and over 64 models
- benchmark 100, 1K, 10K, and 100K theoretical combinations
Benchmarks ran on QEMU Virtual CPU 2.5+ with three 500 ms samples.
At 10K theoretical combinations:
- validation: 9.71 ms, 6.53 MB, and 81,401 allocs before; 12.38 us, 14.3
KB, and 122 allocs after
- solver setup: 1.31 ms and 1.37 MB before; 22.4 ns and 16 B after
- eviction solve: 809 us and 16 B before; 9.04 us and 7.97 KB after
At 100K theoretical combinations with the symbolic implementation:
- validation: 15.10 us, 16.1 KB, and 149 allocs
- solver setup: 23.6 ns, 16 B, and one allocation
- eviction solve: 16.17 us, 13.7 KB, and 193 allocs
At 100 combinations, solve time increases from 3.07 us to 4.72 us, and
allocation rises from 16 B to 3.82 KB. At 1K combinations, solve time
drops from 50.8 us to 7.23 us.
Written to GPT 5.6-Sol (high). Verified and reviewed with Opus 5 and
Kimi K3.
I don't understand what's going because these models know more than me
on writing a DSL compiler.
fixes#951
Resolve configuration macros against an untyped YAML representation
before decoding the typed Config. This materializes YAML anchors and
preserves scalar types without maintaining field-specific replacement
paths.
- apply environment, global, and model macros in their defined scopes
- centralize MODEL_ID, PORT, PID, key, and unknown-macro handling
- remove the capability-specific raw decoder and replacement loops
- document macro ordering, scope, and runtime behavior
- add coverage for anchors, scope isolation, validation, and runtime
macros
fixes#919
Address peer models by fully qualified names across routing, selectors,
model listings, and the UI.
- support fully qualified peer/model routing names
- support peer models in selector spillover targets
- show peer models in Playground model pickers
fixes#944
## Summary
Fixes issue #946 where a request arriving during a process stop would
hang forever. The router was reading process state and making start/stop
decisions based on that snapshot, which could become stale by the time
the decision was acted upon. This change moves the start decision into
the process's own state machine where it can be made atomically.
## Key Changes
- **Renamed `runReq` to `startReq`** with a new `block` field to
distinguish between `Run()` (blocking until process terminates) and the
new `EnsureReady()` (non-blocking, answers when ready or failed)
- **Added `EnsureReady()` method** to the `Process` interface and
`ProcessCommand` that brings a process to ready state without blocking
on termination. The decision of whether to start is made inside the
process's run loop, eliminating race conditions
- **Updated process state machine** to:
- Call `notifyWaiters()` on all terminal state transitions (not just
`StateStarting` exits) to prevent subscribers from hanging
- Handle `EnsureReady` requests that arrive during `StateStopping` by
waiting for the stop to complete before starting
- Notify waiters when a process exits unexpectedly or is stopped
- **Updated router's `doSwap()`** to call `EnsureReady()` instead of
reading `State()` and conditionally calling `Run()`. This ensures the
start decision is made atomically within the process's run loop
- **Added comprehensive tests**:
- `TestProcessCommand_EnsureReadyDuringStop`: Regression test verifying
requests during stop wait for completion then start
- `TestProcessCommand_EnsureReadyIsIdempotent`: Verifies idempotent
behavior on ready/shutdown states
- `TestBaseRouter_RequestDuringStop`: Router-level test ensuring
requests during TTL unload don't hang
- Updated `fakeProcess` to mirror `ProcessCommand` behavior with proper
synchronization
- **Updated design documentation** to clarify that advisory state reads
are fine, but mutations must never be gated on state snapshots
## Implementation Details
The core fix is that `EnsureReady()` sends a request to the process's
run loop, which then decides whether to start based on the current
state. This is safe because:
- The run loop is single-threaded and owns the state
- A stop in flight keeps the loop parked in `killProcess()`, so start
requests cannot be received until the stop completes
- The send on `startCh` is the synchronization point that naturally
serializes against in-flight stops
This follows the principle: **advisory reads of process state are fine,
but reads that gate mutations must happen inside the process's own state
machine**.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
- use shared.ReplaceRequestModel everywhere
- remove magic number for multipart form parsing
- add form.RemoveAll() to clean up temp files
Updates #933Fixes#937
Supercedes #939
Add configurable model ID profiles that can be switched without
restarting.
- rewrite request and upstream model IDs through active profile pins
- expose profile selection through API, SSE, and UI
- validate and document profile configuration
Updates #933
Supercedes: #774
