Agentic Loop Token Costs Are an Architecture Problem
Agentic loop token costs come from context accumulation, tool bloat, and retries. Learn to map hidden API waste before downgrading your model.
Category
Engineering
Technical deep dives into how AI systems are built and run: models, retrieval, agents, evaluation, fine-tuning, and infrastructure.
24 posts
Megakernels in LLM Inference When Fusion Actually Wins
Megakernels in LLM inference trade off launch overhead against SM occupancy. Learn when fused kernels beat CUDA graphs for low-latency agentic workloads.
Disaggregated GPU Inference Hits the KV Cache Wall
Disaggregated GPU inference splits prefill and decode for higher throughput, but each request moves 2.6 GB of KV cache across the datacenter.
BM25 vs Dense Retrieval and SPLADE for Production RAG
BM25 vs dense retrieval on a consumer GPU. Each baseline wins one RAG scenario and breaks another. Match your index to memory, latency, and failure mode.
AI Agent Cost Per Resolution Decides If It Ships
AI agent cost per resolution, not eval accuracy, decides if your agent ships or dies. Token pricing hides the unit economics of retries and failures.
Self-Host LLM Inference, the Netflix Decision Framework
Learn when to self-host LLM inference instead of paying per token. Netflix's production stack reveals the real cost, latency, and control tradeoffs.
Orchestrate Open Source LLMs vs Frontier Models
Should you orchestrate open source LLMs or call a single frontier model? We break down Sakana Fugu's claims and the real cost and latency tradeoffs.
How LLM Agent Scaffolding Fixes Failing Code Review Agents
LLM agent scaffolding constrains what models see and call. Learn why GitHub Copilot code review regressed with more tools and how routing helps.
Stop RAG Hallucination with Typed Schema Contracts
Stop RAG hallucination with typed schema contracts. Build programmatic answer contracts, validate field-level citations, and handle missing data.
Domain-Specific LLM Evaluation Demands Leakage-Free Data
GPT and Claude failed Bridgewater's private financial evals. Discover what this reveals about LLM benchmark leakage and how to build robust holdout sets for domain-specific testing.
Secure AI Coding Agents From Supply Chain Attacks
Learn how to secure AI coding agents against supply chain attacks. Discover how to prevent prompt injection malware execution using sandboxing and strict file permissions.
Human-in-the-Loop Systems: Balancing AI Speed and Operational Safety
Discover why full automation fails at scale and learn 5 architectural patterns for human-in-the-loop systems to balance AI speed with operational safety.