Engineering

Technical deep dives into how AI systems are built and run: models, retrieval, agents, evaluation, fine-tuning, and infrastructure.

14 posts

Subscribe via RSS

Speculative decoding turns idle CPU compute into nearly four times faster token generation for large language model inference.

Engineering • Aug 24, 2026 •11 min read

Why Speculative Decoding Pays Nearly 4x on CPUs

Speculative decoding turns idle CPU cores into 4x faster LLM generation. Learn why it works, when gains collapse, and when CPU beats GPU or API.

#speculative-decoding #cpu-inference +3

Deep learning GPU training where gradient accumulation splits an effective batch into micro-batches that change wall-clock time on rented hardware.

Engineering • Aug 19, 2026 •12 min read

The Real Cost of Gradient Accumulation on T4 and L4

Gradient accumulation can make identical batches train at different speeds. Learn why micro-batch shape drives T4 vs L4 wall-clock time and throughput.

#gradient-accumulation #gpu-benchmarks +4

Agentic loop token costs compound across turns as static payloads and conversation history are re-sent on every API call.

Engineering • Aug 07, 2026 •12 min read

Agentic Loop Token Costs Are an Architecture Problem

Agentic loop token costs come from context accumulation, tool bloat, and retries. Learn to map hidden API waste before downgrading your model.

#agentic-loop-token-costs #llm-api-token-optimization +6

Megakernels in LLM inference fuse separate GPU kernel launches into one persistent execution context to eliminate host overhead during low-latency token decoding.

Engineering • Aug 06, 2026 •10 min read

Megakernels in LLM Inference When Fusion Actually Wins

Megakernels in LLM inference trade off launch overhead against SM occupancy. Learn when fused kernels beat CUDA graphs for low-latency agentic workloads.

#megakernels-in-llm-inference #fused-kernels-vs-cuda-graphs +6

Prefill decode disaggregation isolates compute phases across separate GPU pools in a datacenter to optimize large language model serving.

Engineering • Aug 03, 2026 •11 min read

Disaggregated GPU Inference Hits the KV Cache Wall

Disaggregated GPU inference splits prefill and decode for higher throughput, but each request moves 2.6 GB of KV cache across the datacenter.

#llm-serving #gpu-inference +3

BM25, dense, and SPLADE retrieval baselines compared for production RAG, each with distinct tradeoffs in memory footprint, query latency, and failure behavior.

Engineering • Jul 27, 2026 •13 min read

BM25 vs Dense Retrieval and SPLADE for Production RAG

BM25 vs dense retrieval on a consumer GPU. Each baseline wins one RAG scenario and breaks another. Match your index to memory, latency, and failure mode.

#rag-retrieval #bm25 +4

Financial review of AI agent cost per resolution showing why unit economics determine whether an agent ships to production.

Engineering • Jul 19, 2026 •12 min read

AI Agent Cost Per Resolution Decides If It Ships

AI agent cost per resolution, not eval accuracy, decides if your agent ships or dies. Token pricing hides the unit economics of retries and failures.

#ai-agents #agent-evaluation +4

Netflix's in-house LLM serving platform illustrates what large-scale self-hosted inference infrastructure looks like when workload thresholds justify the investment.

Engineering • Jul 18, 2026 •12 min read

Self-Host LLM Inference, the Netflix Decision Framework

Learn when to self-host LLM inference instead of paying per token. Netflix's production stack reveals the real cost, latency, and control tradeoffs.

#llm-inference #self-hosting +4

Illustration of a server network designed to orchestrate open source LLMs efficiently.

Engineering • Jul 16, 2026 •12 min read

Orchestrate Open Source LLMs vs Frontier Models

Should you orchestrate open source LLMs or call a single frontier model? We break down Sakana Fugu's claims and the real cost and latency tradeoffs.

#ai-engineering #llm-orchestration +3

Abstract visualization of llm agent scaffolding showing structured layers that organize and constrain software logic.

Engineering • Jul 11, 2026 •13 min read

How LLM Agent Scaffolding Fixes Failing Code Review Agents

LLM agent scaffolding constrains what models see and call. Learn why GitHub Copilot code review regressed with more tools and how routing helps.

#llm-agent-scaffolding #agentic-workflow-failures +6

Implementing typed schema contracts enforces strict data boundaries to successfully stop rag hallucination in automated pipelines.

Engineering • Jul 04, 2026 •12 min read

Stop RAG Hallucination with Typed Schema Contracts

Stop RAG hallucination with typed schema contracts. Build programmatic answer contracts, validate field-level citations, and handle missing data.

#stop-rag-hallucination #structured-outputs +6

domain-specific LLM evaluation in a practical business context.

Engineering • Jul 03, 2026 •11 min read

Domain-Specific LLM Evaluation Demands Leakage-Free Data

GPT and Claude failed Bridgewater's private financial evals. Discover what this reveals about LLM benchmark leakage and how to build robust holdout sets for domain-specific testing.

#domain-specific-llm-evaluation #llm-benchmark-leakage +6