A bilingual engineering journal about AI systems, retrieval, agents, multimodal intelligence, and inference infrastructure. https://optiverse-tech.com en-us Sat, 09 May 2026 17:32:46 GMT https://optiverse-tech.com/blog/moe-sharding/ https://optiverse-tech.com/blog/moe-sharding/ Fri, 03 Apr 2026 17:36:25 GMT In Post I-00, we listed five ways that LLM inference differs from conventional model serving. The first four -- variable-length computation, two-phase resource profiles, growing memory requirements, and cache-aware routing -- have each received a full post in this track. The fifth was stated in a... Inference Infrastructure Huang Tzu Lin https://optiverse-tech.com/blog/prefix-aware-routing/ https://optiverse-tech.com/blog/prefix-aware-routing/ Fri, 03 Apr 2026 17:36:25 GMT In Post I-02, we saw that PagedAttention enables different requests to share physical KV cache blocks on the same replica. Two requests with the same system prompt can point to the same physical blocks rather than storing duplicate copies. That sharing mechanism is real and it works -- but only i... Inference Infrastructure Huang Tzu Lin https://optiverse-tech.com/blog/prefill-decode-disaggregation/ https://optiverse-tech.com/blog/prefill-decode-disaggregation/ Fri, 03 Apr 2026 17:36:25 GMT Post I-00 established that LLM inference has two phases with fundamentally different resource profiles. Prefill processes all input tokens in parallel and is compute-bound -- the GPU's arithmetic units are the bottleneck. Decode generates tokens one at a time and is memory-bandwidth-bound -- the ... Inference Infrastructure Huang Tzu Lin https://optiverse-tech.com/blog/paged-kv-cache/ https://optiverse-tech.com/blog/paged-kv-cache/ Fri, 03 Apr 2026 17:36:25 GMT In Post I-00, we traced a single API call through the inference pipeline and introduced the KV cache: the data structure that stores attention key-value vectors so the model does not recompute them at every decode step. The KV cache grows with every generated token, and it must reside in GPU memo... Inference Infrastructure Huang Tzu Lin https://optiverse-tech.com/blog/continuous-batching/ https://optiverse-tech.com/blog/continuous-batching/ Fri, 03 Apr 2026 17:36:25 GMT Post I-00 traced a single request through the inference pipeline: prefill processed all input tokens in parallel, decode generated output tokens one at a time, and the KV cache grew with every step. At the end of that trace, we noted that 49 other agents were submitting queries at roughly the sam... Inference Infrastructure Huang Tzu Lin https://optiverse-tech.com/blog/what-happens-after-you-call-the-api/ https://optiverse-tech.com/blog/what-happens-after-you-call-the-api/ Fri, 03 Apr 2026 17:36:25 GMT You have built a travel copilot. A user types a query, your application sends it to an LLM provider's API, and a few seconds later a response streams back. From the application developer's perspective, that is one function call. From the infrastructure's perspective, that function call triggers a... Inference Infrastructure Huang Tzu Lin https://optiverse-tech.com/blog/agent-loops-react-tools-failure-modes/ https://optiverse-tech.com/blog/agent-loops-react-tools-failure-modes/ Sat, 21 Mar 2026 05:32:32 GMT Most engineering discussions about agents start too early with the word agent and too late with the operational loop. In practice, the important design question is simpler: once a model can take more than one step, how does the system decide what to do next, what tools it may call, what it is all... Agents & Autonomy Huang Tzu Lin https://optiverse-tech.com/blog/building-travel-copilot-end-to-end-architecture/ https://optiverse-tech.com/blog/building-travel-copilot-end-to-end-architecture/ Fri, 20 Mar 2026 15:39:09 GMT By the time a team reaches an advanced AI travel copilot, the hard question is no longer "Which model should we use?" It is "What has to happen, in what order, with what evidence, with what state, and under whose approval before this system can be trusted in production?" That is an architectural ... AI System Architecture Huang Tzu Lin https://optiverse-tech.com/blog/multimodal-systems-vlms-figure-grounding/ https://optiverse-tech.com/blog/multimodal-systems-vlms-figure-grounding/ Fri, 20 Mar 2026 15:39:09 GMT Text-only systems break as soon as the evidence stops being mostly text, which is exactly what happens in accessible travel planning when photos, floor plans, route maps, captions, and measurements all shape the answer. Document & Multimodal Intelligence Huang Tzu Lin https://optiverse-tech.com/blog/document-intelligence-ocr-layout-tables/ https://optiverse-tech.com/blog/document-intelligence-ocr-layout-tables/ Fri, 20 Mar 2026 15:39:09 GMT Most teams first meet document processing through OCR. The problem seems straightforward: convert pages into text, index the text, and let retrieval or an LLM answer questions from it. Document & Multimodal Intelligence Huang Tzu Lin https://optiverse-tech.com/blog/when-rag-not-enough-cag-hybrid-retrieval/ https://optiverse-tech.com/blog/when-rag-not-enough-cag-hybrid-retrieval/ Fri, 20 Mar 2026 15:39:09 GMT Basic retrieval-augmented generation, or RAG, is still the default grounding pattern for most production systems. If you have a large corpus, frequent updates, and a need to show where an answer came from, retrieval remains the cleanest starting point. But there is a practical limit case where si... Retrieval & Knowledge Huang Tzu Lin https://optiverse-tech.com/blog/assistants-agents-autonomy-spectrum/ https://optiverse-tech.com/blog/assistants-agents-autonomy-spectrum/ Fri, 20 Mar 2026 15:39:09 GMT Agent has become one of the most overloaded terms in AI. Product teams use it to describe everything from a chat interface with retrieval to a long-running process that can plan, call tools, and take actions on its own. That vocabulary drift creates a practical problem: teams start arguing about ... Agents & Autonomy Huang Tzu Lin https://optiverse-tech.com/blog/memory-state-and-knowledge/ https://optiverse-tech.com/blog/memory-state-and-knowledge/ Fri, 20 Mar 2026 15:39:09 GMT A travel-planning copilot for a mid-size agency is asked a straightforward question: "Did this hotel fail an accessibility review before for wheelchair users?" Retrieval & Knowledge Huang Tzu Lin https://optiverse-tech.com/blog/grounding-with-rag/ https://optiverse-tech.com/blog/grounding-with-rag/ Fri, 20 Mar 2026 15:39:09 GMT Large language models are useful because they can synthesize, explain, and transform information in fluent language. They are unreliable when we ask them to know something current, something private, or something that needs verifiable support. A model may have seen similar material during trainin... Retrieval & Knowledge Huang Tzu Lin https://optiverse-tech.com/blog/reliable-llm-pipelines-and-control-logic/ https://optiverse-tech.com/blog/reliable-llm-pipelines-and-control-logic/ Fri, 20 Mar 2026 15:39:09 GMT Useful AI systems usually fail for ordinary software reasons before they fail for exotic model reasons. A prototype looks impressive when a single prompt produces a plausible answer, but production systems do not consume plausibility. They consume records, decisions, and actions that need to be r... AI System Architecture Huang Tzu Lin https://optiverse-tech.com/blog/from-models-to-compound-ai-systems/ https://optiverse-tech.com/blog/from-models-to-compound-ai-systems/ Fri, 20 Mar 2026 15:39:09 GMT Most failures in real AI products do not come from the model suddenly becoming unintelligent. They come from asking a model to do work that actually belongs to a larger system: fetch the right data, interpret a messy document, check a schema, track state across steps, and show evidence for the an... AI System Architecture Huang Tzu Lin https://optiverse-tech.com/blog/ai-powered-customer-support-chatbot-to-intelligent-system/ https://optiverse-tech.com/blog/ai-powered-customer-support-chatbot-to-intelligent-system/ Fri, 20 Mar 2026 15:39:09 GMT Customer support is a useful capstone example because one message can require retrieval, tool use, memory, routing, and approval boundaries at the same time. LLM Fundamentals Huang Tzu Lin https://optiverse-tech.com/blog/ai-assistants-agents-everything-in-between/ https://optiverse-tech.com/blog/ai-assistants-agents-everything-in-between/ Fri, 20 Mar 2026 15:39:08 GMT A useful AI system is defined less by whether it is called an assistant or an agent than by how much control it has over the next step. LLM Fundamentals Huang Tzu Lin https://optiverse-tech.com/blog/why-llms-need-help-hallucinations-grounding-systems/ https://optiverse-tech.com/blog/why-llms-need-help-hallucinations-grounding-systems/ Fri, 20 Mar 2026 15:39:08 GMT Large language models produce fluent, confident text. That confidence is the problem. A model can sound authoritative about a property listing that no longer exists, a tax rate that changed last quarter, or a school rating from three years ago. It has no mechanism to check. It was not designed to... LLM Fundamentals Huang Tzu Lin https://optiverse-tech.com/blog/prompts-context-windows-and-how-you-talk-to-an-llm/ https://optiverse-tech.com/blog/prompts-context-windows-and-how-you-talk-to-an-llm/ Fri, 20 Mar 2026 15:39:08 GMT In the previous post, we sent a single line to an LLM — "Plan a trip to Helsinki" — and got back an itinerary full of specific-sounding details: restaurant names, transit directions, day-trip logistics. It was fluent and plausible, but several of those details turned out to be wrong. The model wa... LLM Fundamentals Huang Tzu Lin https://optiverse-tech.com/blog/what-large-language-models-actually-do/ https://optiverse-tech.com/blog/what-large-language-models-actually-do/ Fri, 20 Mar 2026 15:39:08 GMT You type a sentence into an AI application. Seconds later, it returns several paragraphs of fluent, well-organized text that reads like it was written by a knowledgeable human. That experience is now routine. What is not routine — and what matters if you plan to build anything on top of these sys... LLM Fundamentals Huang Tzu Lin