Engineering

Technical deep dives into how AI systems are built and run: models, retrieval, agents, evaluation, fine-tuning, and infrastructure.

17 posts

Subscribe via RSS

Engineering • Sep 05, 2026 •12 min read

KV Cache Math for Million Token Agent Runs

KV cache math explains why every decoded token reads the whole context, and what eviction and quantization can cut from million-token agent bills today.

#kv-cache #attention-sparsity +3

Text-to-SQL evaluation tests whether AI-generated SQL returns defensible answers on real warehouse data instead of just matching public benchmark scores.

Engineering • Aug 27, 2026 •12 min read

Text-to-SQL Evaluation That Catches Silent Wrong Answers

Public benchmarks say 89%, your warehouse says otherwise. Build a text-to-SQL evaluation with schema-specific oracles that catches silent wrong answers.

#text-to-sql #nl2sql +4

A GPU cluster in a data center represents the hardware baseline behind MoE serving cost, where residency, communication, and utilization set the real invoice for a sparse model.

Engineering • Aug 26, 2026 •12 min read

MoE Serving Cost Math for 6 of 125B Active Parameters

MoE serving cost for a 6-of-125B model is not 6B per token. All 125B stay in VRAM, so run the builder math on residency, routing, and break-even.

#moe #inference-cost +4

Speculative decoding turns idle CPU compute into nearly four times faster token generation for large language model inference.

Engineering • Aug 24, 2026 •11 min read

Why Speculative Decoding Pays Nearly 4x on CPUs

Speculative decoding turns idle CPU cores into 4x faster LLM generation. Learn why it works, when gains collapse, and when CPU beats GPU or API.

#speculative-decoding #cpu-inference +3

Deep learning GPU training where gradient accumulation splits an effective batch into micro-batches that change wall-clock time on rented hardware.

Engineering • Aug 19, 2026 •12 min read

The Real Cost of Gradient Accumulation on T4 and L4

Gradient accumulation can make identical batches train at different speeds. Learn why micro-batch shape drives T4 vs L4 wall-clock time and throughput.

#gradient-accumulation #gpu-benchmarks +4

Agentic loop token costs compound across turns as static payloads and conversation history are re-sent on every API call.

Engineering • Aug 07, 2026 •12 min read

Agentic Loop Token Costs Are an Architecture Problem

Agentic loop token costs come from context accumulation, tool bloat, and retries. Learn to map hidden API waste before downgrading your model.

#agentic-loop-token-costs #llm-api-token-optimization +6

Megakernels in LLM inference fuse separate GPU kernel launches into one persistent execution context to eliminate host overhead during low-latency token decoding.

Engineering • Aug 06, 2026 •10 min read

Megakernels in LLM Inference When Fusion Actually Wins

Megakernels in LLM inference trade off launch overhead against SM occupancy. Learn when fused kernels beat CUDA graphs for low-latency agentic workloads.

#megakernels-in-llm-inference #fused-kernels-vs-cuda-graphs +6

Prefill decode disaggregation isolates compute phases across separate GPU pools in a datacenter to optimize large language model serving.

Engineering • Aug 03, 2026 •11 min read

Disaggregated GPU Inference Hits the KV Cache Wall

Disaggregated GPU inference splits prefill and decode for higher throughput, but each request moves 2.6 GB of KV cache across the datacenter.

#llm-serving #gpu-inference +3

BM25, dense, and SPLADE retrieval baselines compared for production RAG, each with distinct tradeoffs in memory footprint, query latency, and failure behavior.

Engineering • Jul 27, 2026 •13 min read

BM25 vs Dense Retrieval and SPLADE for Production RAG

BM25 vs dense retrieval on a consumer GPU. Each baseline wins one RAG scenario and breaks another. Match your index to memory, latency, and failure mode.

#rag-retrieval #bm25 +4