mirror of
https://github.com/mostlygeek/llama-swap.git
synced 2026-09-07 16:37:22 +02:00
Benson WongandGitHub
41f143110d
Hide tailnet server token in the UI and add an easy to copy/paste Peer config that is easy to send to connect to the server. - Add missing configuration overview documentation - Fix out of date docs - Fix minor bug for listing tailcat server models Fixes #1092
77 lines
2.0 KiB
Markdown
77 lines
2.0 KiB
Markdown
# Hello!
|
|
|
|
Here you will find the knowledge base for llama-swap. It's
|
|
easy to get started in llama-swap with just a few lines
|
|
of YAML. However, the real power comes from the dozens
|
|
of configuration options to control routing and resource
|
|
loading exactly as you want it.
|
|
|
|
llama-swap doesn't come with traditional documentation.
|
|
Instead, it includes a documentation agent that reads from
|
|
the knowledge base to answer your questions directly.
|
|
|
|
Three steps to get started:
|
|
|
|
1. Download gemma-4-12B
|
|
2. Install llama-server
|
|
3. Write your first configuration file and start llama-swap
|
|
|
|
## Downloading gemma-4-12B
|
|
|
|
Docs is evaluated against a gemma-4-12B Q4_K_M from Unsloth.
|
|
It is a small and capable model.
|
|
|
|
Download it form https://huggingface.co/unsloth/gemma-4-12b-it-GGUF
|
|
|
|
```bash
|
|
uvx hf download unsloth/gemma-4-12b-it-GGUF gemma-4-12b-it-Q4_K_M.gguf --local-dir .
|
|
|
|
# download MTP
|
|
uvx hf download unsloth/gemma-4-12b-it-GGUF MTP/mtp-gemma-4-12b-it-Q8_0.gguf --local-dir .
|
|
```
|
|
|
|
If you have a smaller GPU or none at all, use the lighter
|
|
[gemma-4-E4B](https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF) instead.
|
|
|
|
## Installing llama-server
|
|
|
|
(find instructions for your os) - to be written.
|
|
|
|
## Installing llama-swap
|
|
|
|
(to be written)
|
|
|
|
## config.yaml
|
|
|
|
Use this minimal configuration to get gemma-4-12B running to
|
|
power the Docs agent.
|
|
|
|
```yaml
|
|
models:
|
|
gemma-4-12B:
|
|
cmd: |
|
|
/path/to/llama-server-latest
|
|
--host 127.0.0.1 --port ${PORT}
|
|
--log-verbosity 4 --log-colors on
|
|
--temp 1.0 --top-p 0.95 --top-k 64
|
|
|
|
# model params
|
|
--model /path/to/gemma-4-12b-it-Q4_K_M.gguf
|
|
--model-draft /path/to/mtp-gemma-4-12b-it-Q8_0.gguf
|
|
|
|
# enable MTP
|
|
--spec-type draft-mtp
|
|
--spec-draft-n-max 4 --spec-draft-p-min 0.75
|
|
capabilities:
|
|
tools: true
|
|
filters:
|
|
stripParams: "temperature, top_k, top_p, repeat_penalty, min_p, presence_penalty"
|
|
```
|
|
|
|
## Run llama-swap
|
|
|
|
Start up llama-swap and visit http://localhost:8080
|
|
|
|
```bash
|
|
llama-swap -config config.yaml -listen localhost:8080
|
|
``` |