# 1. Configure and build
cmake -B build -DGGML_CUDA=ON
cmake --build build -j8

# 2. First profiling run. Send one or more representative requests, then stop the server.
./build/bin/llama-server \
  --cpu-moe \
  --moe-layer-perf-out moe-hot-cache.json \
  --ctx-size <ctx_tokens> \
  --device CUDA0 \
  --n-gpu-layers 999 \
  <your model and server args>

# 3. Normal hot-cache run. Keep ctx/model args the same as the profiling run.
./build/bin/llama-server \
  --cpu-moe \
  --moe-hot-cache moe-hot-cache.json \
  --moe-hot-cache-max-mib -1 \
  --moe-hot-cache-auto-reserve-mib 1024 \
  --moe-hot-cache-update-rate 0.10 \
  --ctx-size <ctx_tokens> \
  --device CUDA0 \
  --n-gpu-layers 999 \
  <your model and server args>

# 4. Final speed run after tuning. Perf counters start disabled.
./build/bin/llama-server \
  --no-perf \
  --cpu-moe \
  --moe-hot-cache moe-hot-cache.json \
  --moe-hot-cache-max-mib -1 \
  --moe-hot-cache-auto-reserve-mib 1024 \
  --ctx-size <ctx_tokens> \
  --device CUDA0 \
  --n-gpu-layers 999 \
  <your model and server args>

For a primary GPU that owns the normal graph/KV cache plus two expert-only GPUs, keep the normal device list scoped to the primary GPU and name expert GPUs through the hot-cache lane arguments:

./build/bin/llama-server \
  --cpu-moe \
  --moe-hot-cache moe-hot-cache.json \
  --moe-hot-cache-max-mib 2048 \
  --moe-hot-cache-device CUDA0 \
  --moe-hot-cache-second-device CUDA1 \
  --moe-hot-cache-second-max-mib 8192 \
  --moe-hot-cache-second-auto-reserve-mib 1024 \
  --moe-hot-cache-third-device CUDA2 \
  --moe-hot-cache-third-max-mib 8192 \
  --moe-hot-cache-third-auto-reserve-mib 1024 \
  --moe-hot-cache-device-strategy hot-even \
  --ctx-size <ctx_tokens> \
  --device CUDA0 \
  --n-gpu-layers 999 \
  <your model and server args>

For a pure speed measurement after tuning, use --no-perf. That removes the perf callback overhead, but it also disables live MoE counters and leaves dynamic hot-cache updates without new counter data.

Open it with the activity button next to the chat input actions. The dropdown next to that button controls the runtime MoE perf mode: Full, Update, or Off. When the server starts with --no-perf, the dropdown starts in Off.

The page also includes cache action buttons. Apply cache applies the displayed snapshot only to the running hot cache. Apply to disk applies the same snapshot and asks the server to persist it into the file passed with --moe-hot-cache, first copying the existing file to a numbered backup. Save JSON only saves a browser-side copy of the displayed /moe-layer-perf snapshot. Browsers that support the native save-file picker let you choose the file name and destination; otherwise the browser handles it as a normal download according to its download settings.

The timing cards are ordered by execution flow: summary, routing/prep, parallel region, hot and cold lanes, synchronization, then merge. Do not add all timing cards together. Some values are nested or overlapping measurements. Parallel wall is the wall time of the hot/cold region, Hot lane and Cold lane are lane wall times, Overlap is time hidden by parallel execution, and Join wait is time one lane waits for the other.

The JSON body uses the same schema as --moe-hot-cache and /moe-layer-perf. It may contain first-run experts arrays or runtime hot_experts and cold_experts arrays. The existing cache layout stays fixed: the endpoint can replace experts inside the already allocated per-layer hot slots, but it cannot increase the total cache size or create new hot slots for a layer that was not part of the initial cache.

# Preferred: apply an expert/perf JSON file directly.
curl -sS -X POST http://127.0.0.1:8080/moe-hot-cache \
  -H 'Content-Type: application/json' \
  --data-binary @moe-hot-cache.json

# Router mode: target one loaded model explicitly.
curl -sS -X POST 'http://127.0.0.1:28002/moe-hot-cache?model=unsloth/Qwen3.6-35B-A3B-GGUF:Q6_K_XL' \
  -H 'Content-Type: application/json' \
  --data-binary @moe-hot-cache.json

# Apply the server's current /moe-layer-perf snapshot.
# In router mode, pass ?model=... if more than one model is loaded.
curl -sS http://127.0.0.1:8080/moe-hot-cache

# GET can also carry a small URL-encoded JSON value.
# Prefer POST for real expert lists because they are usually large.
curl -sS --get http://127.0.0.1:8080/moe-hot-cache \
  --data-urlencode 'json={"schema":"llama.cpp.moe_layer_opt_perf.v1","layers":[{"layer":0,"experts":[[12,100],[42,90]]}]}'

# Copy the live perf endpoint into the manual apply endpoint.
curl -sS http://127.0.0.1:8080/moe-layer-perf \
  | curl -sS -X POST http://127.0.0.1:8080/moe-hot-cache \
      -H 'Content-Type: application/json' \
      --data-binary @-

A successful response returns HTTP 200 and a compact status object:

{
  "success": true,
  "active": true,
  "update_rate": 1.0,
  "exchanged": 47,
  "candidates": 131,
  "max_exchange": 131,
  "hot_experts": 469,
  "layers_changed": 29
}

While a manual apply request is pending, the automatic post-request hot-cache update is skipped for that cycle. This prevents the automatic --moe-hot-cache-update-rate update from changing the cache immediately before the manual list is applied.

Expected log lines:

MoE hot-cache manual apply requested; auto update will be skipped until it is handled
MoE hot-cache auto update skipped: manual /moe-hot-cache request pending
MoE hot-cache apply: mode = manual, rate = 100.00%, exchanged = ...

Hot-cache workflow arguments

Perf, logging, and live switching

Related llama.cpp arguments that affect hot-cache runs

The measured setup used unsloth/Qwen3.6-35B-A3B-GGUF:Q6_K_XL with a 100000 token context and the coding prompt Erstelle mir ein Snake Spiel in HTML und Java Script. The standard llama.cpp router setup averaged 22.2 t/s. With MoE hot-cache enabled, auto-sizing selected 2367/10172 observed experts, used 6554 MiB of VRAM, and corresponded to about 9.2 full MoE layers.

Practical Next Steps