Evaluator aware decoding

LLMs are bad at arithmetic, the standard fix is tool calling: the model asks for a calculator, the application runs it, and the model continues in a new turn. That works, but for tiny deterministic operations it is a surprisingly heavyweight loop. One alternative is evaluator aware decoding. The model generates structure, and the inference engine evaluates deterministic spans while decoding, injecting the result before generation continues. No tool-call boundary, no second turn, no chance for the model to misquote the result.

The core idea

Consider a model producing this stream:

The expression is <calc>17 * 23 + 91</calc>

The moment the runtime sees </calc>, it has a complete, parseable expression. Instead of finishing the generation and handing control back to an orchestrator, the engine can:

1. Detect the completed span in the token stream.
2. Evaluate it locally: 17 * 23 + 91 = 482.
3. Tokenize <result>482</result> and inject those tokens into the sequence, running them through the model as forced tokens so the KV cache stays consistent.
4. Resume normal sampling, with the correct value already in context.

The model chose the expression; the engine owns the answer. The final text reads as one continuous generation:

The expression is <calc>17 * 23 + 91</calc><result>482</result>, so there are 482 items.

Why bother, when tool calling exists?

Tool calling is the right abstraction for application level capabilities: search, databases, email, anything slow, stateful, or security sensitive. But for small pure computations it has three costs that decode time evaluation avoids.

Latency. A tool call is a loop boundary: model output, parse, application executes, second model continuation. For arithmetic, unit conversion, date math, or checksum computation, the round trip dominates the actual work. Decode time evaluation happens inside the same generation loop, often within a single decode step’s budget.

An extra model turn. Tool calling serializes messages and complicates streaming: the client sees the model stop, a tool result arrive, and generation restart. With decode time injection the stream never pauses in a user visible way; the result simply appears as soon as the expression closes.

Correctness is advisory, not enforced. A tool result goes back into context, but the model can still round it wrong, contradict it two sentences later, or ignore it. When the engine injects the value as forced tokens, the exact evaluated result is guaranteed to appear in the output. Correctness moves from “the model probably copies this” to “the runtime enforces this.”

There is also an efficiency argument: the model spends its capacity on what it is good at (decomposing the problem, extracting the expression, explaining), and stops burning probability mass trying to imitate an ALU.

This is a grammar problem

The reason I find this framing compelling is that it slots neatly into constrained decoding. If you are already using grammars, you can enforce syntactic shape:

root ::= "The expression is <calc>" ws? expression ws? "</calc>" (result | tail)

A conventional grammar guarantees the shape: answer is numeric, tags are balanced. An evaluator aware grammar can go further and guarantee a semantic invariant:

syntax constraint:   answer must be a number
semantic constraint: answer must equal eval(expression)

In JSON form:

{"expression": "17 * 23 + 91", "answer": 482}

The grammar says answer is a number. The evaluator says it is exactly 482. Grammar enforcement upgrades from “valid shape” to “valid shape plus computed invariants.”

A concrete prototype on llama.cpp

https://github.com/undo-git-pull/decode-eval-example

I built a small demo directly on the llama.cpp API to make this concrete. It runs in three modes:

• no-grammar: baseline free-form generation.
• grammar-no-eval: a GBNF grammar forces the model to emit <calc>...</calc> and then compute the answer itself.
• grammar-eval: same grammar, but the runtime watches the stream for a completed <calc> span, evaluates the expression with a small local parser, and injects <result>VALUE</result> as forced tokens before sampling resumes.

The core loop is simple: after each accepted token, scan the decoded text past the last evaluation point for a closed <calc> tag. On a hit, evaluate, tokenize the <result> block, decode those tokens through the model to extend the KV cache, and continue sampling. Failures inject <result_error>...</result_error> instead, so the model can recover in-band. Instrumentation tracks injected token count and per injection decode latency, which for local arithmetic is negligible next to a normal forward pass, let alone a full tool-call round trip.

Two implementation details matter more than they look:

1. Injection must go through the model. You cannot just splice text into the output string; the injected tokens have to be decoded so subsequent sampling conditions on them. Otherwise the model continues as if the result never happened.
2. The grammar and the evaluator must agree. In eval mode, the grammar should not allow the model to write its own <result> block, and after injection the grammar state must be advanced (or designed) so the forced tokens are legal. Result injection is effectively a grammar transition the model did not choose.

Not just arithmetic: string operations

Nothing about the technique is specific to math. Any pure function over a completed span works, and string operations are a perfect fit because they target another famous LLM blind spot: tokenization makes character level questions like “how many r’s are in strawberry” unreliable, since the model never sees the letters.

The same span protocol handles it:

The count is <strop>count("strawberry", "r")</strop><result>3</result>, so ...

The engine parses the operation, runs it in a few nanoseconds of native code, and forces the exact answer. A small evaluator registry covers a lot of ground:

EVALUATORS = {
    "count": lambda s, ch: s.count(ch),
    "len": len,
    "reverse": lambda s: s[::-1],
    "upper": str.upper,
    "substr": lambda s, i, j: s[int(i):int(j)],
}

Character counting, reversal, slicing, case transforms, checksums over strings: all deterministic, all pure, all trivially time bounded, so they sit safely inside the decode loop with no security implications.

Test results

Results running the prototype with google_gemma-4-E4B-it-Q4_K_M on 50 samples with randomly generated numbers, each of the form “a box has ROWS rows of ITEMS items, then ADDED more are added”:

Mode	Accuracy
no-grammar	2.0%
grammar-no-eval	2.0%
grammar-eval	100.0%

The middle row is the interesting one. Forcing the model to extract a well-formed expression via grammar did not help accuracy at all: the model reliably produced the right expression and then computed it wrong. Structure alone fixes shape, not truth. Once the engine owns evaluation, accuracy goes to 100% by construction, because the correct value is injected as forced tokens and the model cannot deviate from it. The residual failure mode shifts from “model can’t do arithmetic” (common) to “model extracted the wrong expression” (which, on this set, never happened).

That 2% to 100% jump on a small quantized model is the whole pitch: you get calculator reliability out of a model class that is otherwise hopeless at the task, without a tool call round trip.

Prior work: Toolformer, and why this is different

The closest ancestor of this idea is Toolformer (Meta, 2023). Toolformer fine-tuned a model to emit special API-call markup mid-generation; at inference time, decoding was interrupted when the model produced the call, the tool ran, and the result was inserted into the sequence before decoding resumed. Its calculator alone more than doubled accuracy on arithmetic benchmarks. The interrupt evaluate inject loop is therefore not new, and the post above is best read as a modern reworking of it rather than an invention from scratch.

The differences are what make it practical today. First, Toolformer required self-supervised fine-tuning to teach the model when and how to emit API calls; evaluator aware decoding is training free, because the grammar forces a stock instruct model into the span format. Second, Toolformer’s tool results were merely inserted into context, so the model remained free to round them, restate them incorrectly, or contradict them downstream. Here the grammar prohibits the model from writing its own result at all, and the engine forces the exact evaluated tokens, upgrading the result from a suggestion in context to an enforced invariant of the output. Third, Toolformer relied on the model spontaneously deciding to call a tool, which its authors had to encourage with decoding tricks like accepting the call token whenever it appeared in the top ten candidates; a grammar makes span emission deterministic. In short: Toolformer taught the model to ask; evaluator aware decoding makes the runtime answer, and makes the answer binding.

Where XGrammar-2 could take this

The recent XGrammar-2 release is, I think, the natural home for a production version of this idea. XGrammar-2 introduces Structural Tag, a composable JSON DSL where JSON Schema, regex, literal strings, and token IDs are all first-class atomic types that can be nested to describe arbitrarily complex output structures, and it is integrated into SGLang, vLLM, TensorRT-LLM, and MLC-LLM.

Today a Structural Tag describes purely syntactic structure: a tag has a begin, a content constraint, and an end. Evaluator aware decoding is a simple extension: an evaluated tag whose content, once complete, is passed to a registered pure function, with the return value forced into a sibling span. Sketching it in Structural Tag style:

{
  "type": "evaluated_tag",
  "begin": "<calc>",
  "content": { "type": "regex", "pattern": "[0-9+\\-*/(). ]+" },
  "end": "</calc>",
  "evaluator": "arithmetic",
  "result": { "begin": "<result>", "end": "</result>" }
}

Everything XGrammar-2 already built compounds here. The token mask cache makes the surrounding constraints near-free. The repetition and cross-grammar caching mean an evaluated tag embedded in a 100 tool agent grammar does not blow up compilation. And the speculative decoding support raises an interesting design question: a draft token that closes an evaluated span pins the tokens that must follow it, so verification of <result> content becomes exact string matching rather than model verification, which speculative pipelines are already good at exploiting via forced-token paths.

The security posture also stays sane if evaluators are restricted to what this technique is actually good for: deterministic, pure, local, time bounded functions. Arithmetic, string operations, unit and date math, checksums, canonicalization, schema derived computations. Anything with I/O, state, or authorization belongs in tool calling, where the application layer can log, sandbox, and gate it.

Why isn’t this in production APIs yet?

If the idea is this simple, why do vLLM, SGLang, TensorRT-LLM, and the major API providers not offer it? Three realistic reasons.

Batched decoding hates per sequence stalls. Production servers decode hundreds of sequences in lockstep, one big forward pass per step, heavily pipelined with CUDA graphs and overlapped scheduling. Evaluator aware decoding means that at any step, some sequences have just closed a span and need host code to parse, evaluate, tokenize, and schedule forced tokens before their next sample. Either the whole batch stalls on the slowest evaluator, reintroducing exactly the host device synchronization these engines spent years removing, or the sequence is preempted and re-admitted, which adds scheduling machinery and latency that can exceed a tool round trip. Injected tokens also change sequence lengths mid-flight, complicating paged KV accounting and invalidating speculative-decoding draft state. Grammar masks avoid all of this because they are computed from pure automaton state ahead of the step; an evaluator puts arbitrary content-dependent host code on the critical path.

No standard operator set. Tool calling scales because the provider ships zero semantics; customers define the tools. A decode time evaluator forces the provider to answer: evaluate what, exactly? A fixed builtin set makes the provider own every semantic edge case forever: float formatting, overflow behavior, division by zero, timezone rules, grapheme versus code point counting. Every answer becomes a cross-version compatibility contract, and there is no MCP-equivalent standard for an expression language evaluated inside the decode loop. Letting customers register evaluators instead means running tenant code inside the inference engine, which is the next problem.

Security in a multi-tenant engine. The inference process is the most trusted, most shared component in the stack: it holds the weights and the KV caches of every batched tenant. An evaluator moves computation over attacker influenced content into that process. Expression parsers are classic exploit surface: exponentiation bombs, pathological nesting, catastrophic regex backtracking, allocation blowups. Real sandboxing (separate process, seccomp, WASM) restores safety but its invocation overhead eats the latency win that justified the feature. And because the evaluator’s output becomes forced tokens, a buggy or manipulated evaluator steers generation with grammar level authority.

None of this applies with the same force to a single user local runtime, where the batch is small, the tenant is you, and the evaluator set can be a handful of audited pure functions. That is why likes of llama.cpp is the natural home for the prototype, and why provider adoption probably waits for a Structural Tag style standard with a tiny, fixed, provably time bounded operator set.