Optimising local LLM speed across hardware with open agent skill

A climber ascending a steep mountain ridge shaped like a circuit board, with layered rock strata resembling memory stacks and data flows, under a sky showing multiple peaks labelled 'Decode', 'Prefill

A new open agent skill from Sybil Solutions enables systematic, reproducible optimisation of local large language model (LLM) inference across diverse hardware and quantisation schemes. It codifies a methodical approach to improving decode and prefill speeds, time to first token (TTFT), KV cache efficiency, and startup times—without exceeding memory limits or sacrificing stability.

The skill is designed for engineers and AI teams running models on consumer or datacentre GPUs, including Intel Arc, NVIDIA RTX and H200 systems. It has already demonstrated significant gains: up to 2.1× faster prefill at 256K context, decode throughput increased from 43.6 to 53 tok/s (median) for one user, and startup times reduced from 113 to 43 seconds on a laptop-class Intel Arc B390 iGPU.

How the optimisation skill works

The skill applies a hill-climbing methodology: it systematically tests one variable at a time, measures performance against interleaved control runs, logs all results—including failures—and only accepts improvements that pass strict stability gates. This avoids noise-induced false positives and ensures gains are reproducible.

It begins with computing theoretical ceilings based on hardware bandwidth, FLOPs, and model size. For example, on the Intel Arc B390, the decode ceiling was calculated at 88–97 tok/s; the final result of 53 tok/s represents roughly 55% of that limit, with the remainder attributed to GEMV kernel inefficiencies. This early ceiling check prevents wasted effort chasing physically unattainable targets.

The workflow is encoded in SKILL.md and supported by scripts for benchmarking, ceiling calculation, and run queuing. Benchmarks measure real-world metrics: startup time, TTFT, single- and multi-user decode, prefill at multiple lengths, long-context retrieval, and memory use—including GPU driver residency on unified-memory systems.

Validated gains across hardware and stacks

Results have been documented across several configurations. On an Intel Arc Pro B70, Qwen3.8-27B at 4 bpw achieved 365 tok/s under 16-user load using vLLM with custom XPU kernels—tripling performance over llama.cpp’s Q4_K_M quant at 25.0 tok/s. Prefill speeds increased from 650 to 911 tok/s at 254K context.

The same model on the same card with SGLang achieved 82.6 tok/s, compared to vLLM’s 75.7 tok/s on the same day, showing stack-specific sensitivity. On an RTX 3090, Qwen3.6-35B-A3B at 3 bpw reached 357.9 tok/s in code generation.

Even high-end systems benefit: a 4× H200 NVL setup running DeepSeek-V4.1-Flash initially crashed under load. After optimisation using speculative decoding and proper configuration, it achieved 229.1 tok/s decode and 17,546 tok/s prefill—passing all six stability and performance gates.

All results are tied to specific hardware, quantisation, engine versions, and measurement methods, with raw data linked in public repositories like local-ai-registry and trellis-serve.

Practical steps for developers and teams

To use the skill, developers clone the GitHub repository and place the inference-optimisation folder in Claude Code’s skills directory, or point any agent to SKILL.md or PROMPT.md. The provided scripts—bench.py, ceiling.py, queue.sh, mem.sh—are engine-agnostic and rely only on standard Python and bash.

Key practices are enforced: token counts come from server usage stats (not streamed chunks), memory includes unified-memory GPU residency, and improvements must outperform both preceding and following control runs. Stability is verified via long-context needle retrieval and concurrent full-KV soak tests.

Publishing requires a clean, pinned build that passes six acceptance gates, ensuring reproducibility. The method explicitly discourages chasing isolated wins, given ±5–10% measurement noise; instead, it promotes interleaved testing and median-based evaluation.

What it means

This skill shifts LLM optimisation from ad-hoc tuning to a scientific, auditable process. For businesses in the UK, UAE, US, and Kazakhstan running local models—whether for compliance, cost, or latency reasons—it reduces dependency on cloud APIs and opaque vendor stacks.

The transparency of methods and results allows teams to assess whether performance claims are credible and transferable. It also lowers the barrier to entry: startups and research groups can achieve near-optimal performance without deep kernel expertise.

However, gains are bounded by hardware physics and software maturity. The 55% ceiling attainment on Intel highlights room for improvement in kernel optimisation, particularly for GEMV operations. Users should expect diminishing returns as they approach theoretical limits.

The MIT-licensed project welcomes contributions, especially new levers, hardware-specific gotchas, or validated results—provided they include full measurement details and raw data links.

The systematic approach offers a template for other performance-critical AI engineering tasks, reinforcing that sustainable speedups come not from magic flags, but from disciplined iteration.

Sources

Leave a Reply

Your email address will not be published. Required fields are marked *