Add an Docs agent to the playground that can help the user with more
advanced configuration.
- add mcp 2026-07-28 (stateless mcp) framework
- add MCP tools for doc search and config search
- add initial set of guides for key topics
- add Docs to the Playground
- removed out of data documentation and plans
An optional `hooks.on_startup.profile` setting for activating a profile
on startup and after a configuration reload:
```yaml
profiles:
coding:
pins:
llm-code: "gpt-oss-120b"
hooks:
on_startup:
profile: "coding"
```
Another attempt to close#992 after unsuccessful #993 ;)
Co-authored-by: David Soušek <david.sousek@intelogy.co.uk>
capabilities.in and capabilities.out accepted text, audio and image but not video, so a multimodal model that takes video input could not declare it and /v1/models reported it as image-only.
video is added to the shared validModalities set, which makes it valid on both in and out; tests assert both sides explicitly so that stays a decision rather than a side effect of the two lists sharing one map.
Updated together so no layer disagrees: the validModalities map, both validation error messages, both config-schema.json enums, and the valid-values comments in config.example.yaml.
Fixes: #1014
Add a dedicated /comfyui/ passthrough backed by the reserved
comfyui_auto local model.
- preserve escaped upstream paths such as encoded workflow separators
- allow only the root path to start an unloaded ComfyUI model
- enforce a minimum concurrency limit of 50 for the reserved model
- add example upstream.ignorePaths to prevent unintentional swaps by
ComfyUI
- add model.workarounds.ignoreWebsockets so ComfyUI does not block
swapping
fixes: #1001fixes: #1000
Resolve configuration macros against an untyped YAML representation
before decoding the typed Config. This materializes YAML anchors and
preserves scalar types without maintaining field-specific replacement
paths.
- apply environment, global, and model macros in their defined scopes
- centralize MODEL_ID, PORT, PID, key, and unknown-macro handling
- remove the capability-specific raw decoder and replacement loops
- document macro ordering, scope, and runtime behavior
- add coverage for anchors, scope isolation, validation, and runtime
macros
fixes#919
Address peer models by fully qualified names across routing, selectors,
model listings, and the UI.
- support fully qualified peer/model routing names
- support peer models in selector spillover targets
- show peer models in Playground model pickers
fixes#944
Add configurable model ID profiles that can be switched without
restarting.
- rewrite request and upstream model IDs through active profile pins
- expose profile selection through API, SSE, and UI
- validate and document profile configuration
Updates #933
Supercedes: #774
Track request and streamed response metadata for active model requests.
- publish keyed updates through a non-blocking recoverable outbox
- add compact configurable activity columns and clock-safe elapsed time
- tag playground requests with a page-scoped session ID
Fixes#912
Store activity metrics in SQLite and use the store as the source for
activity APIs.
- add SQLite-backed activity store with migrations and activity stats
queries
- add new store.path configuration option
- add activity pagination and stats API endpoints
- update the activity UI to fetch paginated rows and server-side stats
- add optional store.path configuration for durable activity history
Fixes#310, #729
Add upstream.ignorePaths config to prevent model swaps for static-asset
requests made through the /upstream/<model>/<path> passthrough endpoint.
- add UpstreamConfig with compiled *regexp.Regexp slice; invalid regex
returns an error at load time
- apply a default pattern matching common static-asset suffixes
(.js/.json/.css/.png/.gif/.jpg/.jpeg/.ico/.txt) when unset
- in handleUpstream, return 409 Conflict when a path matches and the
local model is not already loaded; peer and already-loaded models fall
through to normal dispatch
- update config-schema.json and config.example.yaml
Updates discussion: #868
internal/config,server: implement model capabilities
- define the capabilities of a model using a simple config block on the
model
- v1/models renders out capabilities to be compatible with openrouter,
huggingface chat, and mistral formats for broader compatibility
- add support for capabilities in UI
Fixes#734
- introduce internal/router/scheduler to decouple routing, swapping and
queuing into interface contracts.
- introduce a new `routing` configuration section that supersedes
`matrix` and `group` while maintaining backwards compatibility
- add FIFO scheduler with prioritized queuing
- add internal/router/design.md as developer documentation on
implementing new schedulers and routers
Fixes#797
- update README.md with new docker instructions
- update docs/configuration.md
- update .github/workflows to have pinned action versions
- gofmt events package
- fix small bugs in CI scripts
- reduce config options for internal/perf/monitor and config. A ring buffer is used to keep 1hr of entries at max 5s granularity. For long term stats use prometheus monitoring on /metrics
Fixes#744
Add a comprehensive performance monitoring system that collects CPU, memory, swap, load average, network IO, and GPU stats. Provides both a REST API for the UI and a Prometheus /metrics endpoint.
Backend changes:
- New internal/perf package with configurable interval-based stats collection
- GPU monitoring via LACT (Unix socket) and nvidia-smi fallback on Linux
- Ring buffer (internal/ring) for time-series stat storage
- Prometheus /metrics endpoint with all system and GPU metrics
- Moved LogMonitor to internal/logmon package
- New PerformanceConfig for hot-reloadable monitoring settings
- REST /api/performance endpoint replacing SSE streaming
UI changes:
- New Performance page with real-time charts for CPU, memory, GPU, and network
- Reusable PerformanceChart component
- LLAMA_SWAP_URL environment variable support
- Improved capture dialog display
Other:
- Example Grafana dashboard for Prometheus metrics
- monitor-test standalone binary
- Config schema and example updates
fixes#596
Add a new swap matrix to supersede groups for running concurrent models.
The matrix uses a solver that picks the lowest cost evictions to make a
requested model available. This simple approach along with a very basic
DSL grammar can enable very complex swapping scenarios.
- add DSL parser for set expressions with & (AND), | (OR), (), +ref
- add MatrixConfig structs, validation, and topological sort for +ref
- add MatrixSolver with cost-minimizing swap decisions
- add Matrix runtime integrating solver with Process lifecycle
- integrate matrix into ProxyManager with if-branches at all endpoints
- update config.example.yaml and config-schema.json with matrix schema
- config enforces groups XOR matrix (cannot use both)
fixes#643
Add configurable HTTP timeout settings to both models and peers to support installations that requires longer timeouts than the current hardcoded defaults.
Closes#618
Add a new configuration parameter globalTTL that all models will
inherit. The default value is 0 which matches the currently
functionality to never automatically unload a model.
The model.ttl's default has changed to -1, which means use the global
TTL value. Any model.ttl >=0 is now value with 0 meaning never unload.
This allows a model to override a globalTTL > 0 and be configured to
never unload.
Fixes#459Closes#512
Add setParamsByID filter that applies different request parameters based
on the requested model ID, enabling per-alias behaviour for a single
loaded model.
- add SetParamsByID field to Filters struct and SanitizedSetParamsByID
method
- substitute ${MODEL_ID} and other macros in setParamsByID keys and
values
- validate no unknown macros remain in keys or values after substitution
- apply setParamsByID in proxyInferenceHandler after setParams (can
override it)
- update config-schema.json with setParamsByID definition
- update UI to show aliases and make them selectable in the Playground
closes#534
Add saving request and response headers and bodies that go through
llama-swap in memory.
- captureBuffer added to configuration. Captures are enabled by default.
- 5MB of memory is allocated for req/response captures in a ring buffer.
Setting captureBuffer to 0 will disable captures.
- UI elements to view captured data added to Activity page. Includes
some
QOL features like json formatting and recombining SSE chat streams
- capture saving is done at the byte level and has minimal impact on
llama-swap performance
Fixes#464
Ref #503
* config: support environment variable macros in peer apiKeys
Add ${env.VAR_NAME} substitution for peer apiKey fields, consistent
with existing env macro support for model fields and global apiKeys.
- Add env macro substitution for peers.{name}.apiKey in LoadConfigFromReader
- Add tests for peer apiKey env substitution
- Update config.example.yaml to show env macro usage
* config: support macros in peer apiKey and filters
Extend macro substitution to peer configuration fields:
- peers.{name}.apiKey supports both global macros and env macros
- peers.{name}.filters.stripParams supports both macro types
- peers.{name}.filters.setParams supports both macro types
Also renamed validateMetadataForUnknownMacros to validateNestedForUnknownMacros
for reuse across model metadata and peer filters validation.
* config: add environment variable macros
Add support for ${env.VAR_NAME} syntax to pull values from system
environment variables during config loading.
- env macros processed before regular macros (allows macros to reference env vars)
- works in cmd, cmdStop, proxy, checkEndpoint, filters.stripParams, metadata
- returns error if env var is not set
- add comprehensive tests
fixes#462
* docs: add env macro example to config.example.yaml
This unifies the filtering capabilities for models and peers
- stripParams: removes params in the request
- setParams: sets params in the request
fixes#453
This PR allows a single llama-swap to be the central proxy for models served by other inference servers. The peer servers can be another llama-swap or any API that supports the /v1/* inference endpoint.
Updates: #433, #299Closes: #296
Add configuration support for api keys that are enforced by llama-swap. Keys are stripped before sending them to upstream servers.
Updates: #433, #50 and #251
The new logToStdout option controls what is logged to stdout. The
default has been changed to just the proxy logs, which contain swap and
http request logs.
There are four supported settings: none, proxy, upstream, both. The
"both" setting is the legacy setting where everything was spewed to
stdout.
introduces a new configuration option logTimeFormat that allows customizing the timestamp in log messages using golang's built in time format constants. The default remains no timestamp.
Swapping models can take a long time and leave a lot of silence while the model is loading. Rather than silently load the model in the background, this PR allows llama-swap to send status updates in the reasoning_content of a streaming chat response.
Fixes: #366
Add full macro-in-macro support so any user defined macro can contain another one as long as it was previously declared in the configuration file.
Fixes#336
Supercedes #335
Changes:
- add Metadata key to ModelConfig
- include metadata in /v1/models under meta.llamaswap key
- add recursive macro substitution into Metadata
- change macros at global and model level to be any scalar type
Note:
This is the first mostly AI generated change to llama-swap. See #333 for notes about the workflow and approach to AI going forward.
* proxy/config: add model level macros
Add macros to model configuration. Model macros override macros that are
defined at the global configuration level. They follow the same naming
and value rules as the global macros.
* proxy/config: fix bug with macro reserved name checking
The PORT reserved name was not properly checked
* proxy/config: add tests around model.filters.stripParams
- add check that model.filters.stripParams has no invalid macros
- renamed strip_params to stripParams for camel case consistency
- add legacy code compatibility so model.filters.strip_params continues to work
* proxy/config: add duplicate removal to model.filters.stripParams
* clean up some doc nits
Add barebones but working implementation of model preload
* add config test for Preload hook
* improve TestProxyManager_StartupHooks
* docs for new hook configuration
* add a .dev to .gitignore
llama-swap can strip specific keys in JSON requests. This is useful for removing the ability for clients to set sampling parameters like temperature, top_k, top_p, etc.
