Running Kimi and GLM at Scale: KV Cache Quantization, INT4 Compression, and Safer Serving

Why these models are hard to serve

Large models consume GPU memory in two big ways:

• the model weights
• the KV cache

The weights are the static part. They sit there, large and expensive, waiting to be used.

The KV cache is the dynamic part. As the model processes tokens, it stores attention keys and values so it can continue a conversation or long document without re-reading everything from scratch. Useful, yes. Also hungry.

For long-context models, the KV cache often fills memory before the weights become the real problem. So if you want better throughput, more concurrency, or lower cost, the KV cache is an obvious place to look.

FP8 KV cache: same memory, more room

By default, KV caches are often stored in BF16. Cloudflare describes storing them in FP8 instead, specifically to cut the memory footprint in half.

That matters because halving cache size means you can keep far more context resident on the GPU. For a long-context model, that is not a cute optimization. That is the difference between “yes, we can admit this request” and “please wait while memory says no.”

On Kimi K2.6, the available context held in memory roughly doubles, from about 686,000 tokens to around 1.37 million. Same GPU. More breathing room.

The tradeoff: a little slower per token, faster overall

FP8 cache quantization is not free. The system has to do a bit more work when reading the cache, so at a fixed concurrency level, BF16 can be a few percent faster per token.

But that’s the wrong place to stop the analysis.

The bigger win is that FP8 lets more requests stay resident at once. BF16 may be slightly quicker in a narrow head-to-head, but it hits a memory wall sooner. FP8 keeps scaling further, which means higher peak throughput and better hardware utilization.

This is the classic infrastructure lesson: the fastest single lane is not always the fastest highway.

Cloudflare reports that BF16 runs out of cache capacity at 32 concurrent requests, while FP8 can continue to 64. That leads to higher overall tokens per second and lower cost per token, even if BF16 has a small edge at lower concurrency.

Why prefill and decode should not be treated the same

One useful design detail here is that Workers AI separates prefill and decode into different pools.

That matters because prefill and decode stress the hardware differently:

• Prefill is more compute-bound
• Decode is more memory-bound

So the “best” precision format depends on the phase. For prefill, BF16 still makes sense because the slight throughput advantage is useful and memory pressure is less dominant. For decode, FP8 KV cache helps more because decode is where memory constraints pinch hardest.

In plain English: don’t apply one optimization everywhere just because it looks tidy on a slide.

What about model quality?

This is where people get suspicious, reasonably.

If you compress the cache, are you quietly nudging the model into worse answers? Based on the available description, Cloudflare found FP8 and BF16 KV caches to be indistinguishable across its evaluation suite.

That’s the ideal outcome for this kind of optimization: less memory, more throughput, no noticeable quality penalty.

INT4 weight compression: attack the other half of memory

If KV cache is one half of the memory problem, weights are the other.

For GLM 5.2, Cloudflare describes compressing model weights from FP8 down to INT4. The result is a much smaller checkpoint and lower per-GPU memory use in an 8-way tensor-parallel deployment.

Smaller weights mean more free memory for KV cache. So even before speed enters the discussion, you get a packing benefit. More room for context. More room for concurrency. Fewer memory-induced compromises.

This is the kind of model optimization that changes what fits on the same hardware.

Why INT4 helps decode more than prefill

The speed story for INT4 is phase-specific too.

During decode, generating each token requires streaming model weights from GPU memory. That makes decode heavily limited by memory bandwidth. If the weights are smaller, less data has to move. Less movement means faster tokens.

That’s why INT4 improves decode throughput, especially at low concurrency where per-request latency matters more. Cloudflare reports gains across concurrency levels, with the largest relative boost appearing at the low end.

Prefill is different. It is compute-bound, and INT4 weights need to be expanded before the main math happens. That extra step adds overhead, so prefill gets slower rather than faster.

Again, this is where disaggregated serving earns its keep:

• use FP8 for prefill
• use INT4 for decode

No need to pick one compromise for the whole pipeline.

Compression without obvious quality loss

Weight compression usually raises the same question as cache quantization: did we just save memory by making the model worse?

Based on the described benchmarks, Cloudflare found GLM INT4 quality stayed within a very small margin of the FP8 model, with accuracy described as indistinguishable in practice.

That’s the pattern to watch for in serving infrastructure. The good optimizations are the ones users don’t notice.

More shared hardware means more shared-risk bookkeeping

Now for the unglamorous but important part.

When you compress weights and quantize KV cache, you can fit more concurrent requests on the same GPU. Great for efficiency. Also a sharper test of your request isolation.

Modern inference serving uses techniques like paged attention, continuous batching, and cache reuse. Those work because the system is very carefully tracking which request owns which cache pages. If that bookkeeping goes wrong, even rarely, one request could read from the wrong physical page.

At enough scale, “extremely unlikely” starts sounding less comforting.

KV cache integrity checks: trust, then verify

Cloudflare added KV cache integrity checking as a defense layer.

The basic idea is simple:

• each physical cache page gets a tag
• the tag changes whenever that page is reallocated
• the server records which pages and tags each request is supposed to use
• before supported decode operations read from the cache, the mappings are checked

If the expected mapping and the actual mapping do not match, the request is aborted instead of returning data from the wrong page.

That’s a very practical definition of safety: fail closed, not weird.

Why the safety check matters

Open-model serving gets discussed in terms of latency, throughput, and cost. Fair enough. But when many requests share the same physical memory structures, correctness becomes part of the product.

The integrity check is not there because cache corruption is normal. It is there because infrastructure at scale should assume rare edge cases are not fictional.

The nice part is the reported overhead is small. Cloudflare says the impact stays under 1% on throughput and tail latency in the measured setup. That makes the check hard to dismiss as “too expensive for production.”

In other words, the seatbelt does not appear to slow the car much.

Why SGLang shows up in this story

All of this work is described as running and being benchmarked with SGLang, the open-source inference serving framework.

That matters because performance stories often hide behind internal tooling. Here, the framing suggests the optimizations are built in close collaboration with the serving layer, with patches and features upstreamed back into the open-source ecosystem.

For readers comparing AI infrastructure stacks, that’s useful context. The model is not the whole product. The serving framework often decides whether a model feels practical or painful.

The deeper lesson: optimize by phase, not by slogan

The cleanest insight here is not “FP8 good” or “INT4 good.”

It’s that different parts of inference want different things.

• Decode wants less memory traffic
• Prefill wants efficient compute
• Shared caches want stronger correctness checks as concurrency rises

So the winning setup is layered, not uniform. Quantize the KV cache where memory is the bottleneck. Compress weights where bandwidth dominates. Add integrity checks where shared-state complexity increases.

This is less “one weird trick” and more “respect the physics.”

What AI teams should take from this

If you’re evaluating model serving platforms, or designing your own stack, this example is a good reminder to ask better questions than just “what model does it host?”

Ask things like:

• How does the platform manage KV cache growth?
• Are prefill and decode handled differently?
• What compression methods are used, and where?
• How are shared cache pages validated under concurrency?
• What happens when an integrity check fails?

Those questions sound operational because they are. They also shape user experience more than many benchmark charts do.

The practical takeaway

Running Kimi and GLM 5.2 at scale is not about finding one magic compression setting. It’s about combining memory savings, bandwidth wins, and safety checks in the right places.

The useful pattern is simple: shrink what can be shrunk, keep high precision where it still pays off, and never let shared-memory efficiency outrun shared-memory safety. That’s how you serve bigger models without turning your GPU fleet into a very expensive stress ball.