My Inference Engine

What I built

A C++ terminal application that loads a GGUF language model and generates text using its own CPU inference implementation. The current backend targets Qwen2, including Qwen2.5 instruction-tuned models.

I wanted to understand the work behind a generated token: how model weights are represented, how attention uses earlier tokens, and how numerical operations become a text-generation loop. This is a correctness-first reference engine, not a wrapper around an external inference runtime or a claim to match an optimized serving system.

From model file to generated text

GGUF file → memory-mapped weights and validated metadata
Prompt → instruction formatting → token IDs
Token history → transformer layers → vocabulary logits
Greedy selection → token-piece decoding → terminal output

The implementation covers the full path:

• Model loading: A custom GGUF reader parses metadata and tensor layouts, checks bounds and shapes, and exposes views over the mapped file. The engine selects its backend from general.architecture, keeping Qwen-specific configuration outside the shared lifecycle code.
• Transformer operations: CPU implementations handle matrix–vector products, RMSNorm, rotary position embeddings, causal grouped-query attention, SwiGLU, residual connections, and the final vocabulary projection.
• Quantized weights: Packed Q5_0, Q8_0, Q4_K, and Q6_K formats are supported alongside F32 tensors. Embedding lookup decodes the selected row, while projection kernels accumulate products from packed blocks without expanding an entire weight matrix to F32 on each call.
• Generation: The CLI formats a single-user instruction prompt, processes its tokens, selects the highest-logit next token, and prints decoded pieces until EOS or the requested token limit.

Keeping the boundaries explicit

The project uses a pinned llama.cpp dependency for GGUF-aware tokenization and token-piece decoding only. Model-layer execution stays in this project's code. That lets me focus on inference mechanics without confusing tokenizer compatibility with a working transformer forward pass.

Model ownership and inference state are separate. Token history, logits, activations, and temporary attention buffers are allocated during state initialization, rather than allocating during each token-forward call. The API also defines failure behavior: invalid arguments, exhausted context, malformed tensors, and non-finite outputs should produce errors, and a failed forward must not advance the committed token position.

Testing correctness

The CTest suite includes model-free checks for tensor operations, packed-format decoding and dot products, quantized matrix–vector dispatch, F32 and mixed-weight forward passes, prompt formatting, greedy selection, and application control flow.

Optional integration checks accept a local GGUF file. They validate model structure, forward execution, and visible generated text through the real CLI. The text-generation check also rejects leaked instruction-protocol markers and verifies the token limit and completion metrics. These are correctness checks, not throughput benchmarks.

Current limits and optimization path

The baseline recomputes the complete token history on every forward call. Its key/value buffers are temporary workspace, not a persistent KV cache. Prompt tokens are also forwarded one at a time, repeating work before the first generated token.

That makes the next structural improvements clear: evaluate the prompt once during prefill, then retain per-layer keys and values so decoding only computes the new token's projections and feed-forward work. SIMD kernels, parallel matrix rows, and a Metal backend are later proposals—not current capabilities.

Greedy sampling is implemented; nonzero-temperature and top-p sampling are not. The CLI reports generation time, tokens per second, CPU usage, and resident memory where supported, but its generation timer starts after prompt prefill. I would measure loading, prefill, time to first token, and subsequent decoding separately before using those numbers to justify an optimization.