AI Agent Architecture: How Agent Systems Really Work

Your vendor says it sells an AI agent. So does the next one, and the nine after that. Gartner concluded in 2025 that of the thousands of vendors marketing "agentic AI", only about 130 genuinely qualify — a practice the firm calls "agent washing". If you cannot describe an AI agent’s architecture, you cannot tell the difference.

That is fixable in about seven minutes, because the underlying design is simpler than the marketing suggests.

Anthropic’s engineering team draws a line that most vendor decks blur. In Building Effective Agents, a workflow is a system "where LLMs and tools are orchestrated through predefined code paths" — your software decides what happens next. An agent is a system "where LLMs dynamically direct their own processes and tool usage" — the model decides.

Both sit on one building block: the augmented LLM — a language model given retrieval, tools and memory. Strip away the diagrams and an agent is that model running in a loop. It reasons, calls a tool, reads the result, and repeats until the task is done or a limit is hit.

The 2022 paper ReAct (Yao et al.) showed that models perform better when reasoning and action interleave — think, act, observe, repeat — rather than reasoning once and answering. Cited roughly 15,000 times, it is the ancestor of nearly every agent shipped today.

So the honest definition is unglamorous: an agent is a language model in a loop, allowed to use tools.

Lilian Weng’s widely-cited agent framework overview breaks the rest into three subsystems around the model. None of them are intelligence — they are plumbing.

• Planning. Breaking a task into subgoals, and criticising one’s own work to retry a failed path. Prompt engineering plus bookkeeping, not a separate model.
• Memory. Short-term memory is the context window — the text the model can currently see, and it is finite. Long-term memory is usually a vector database (a search index over your documents) queried on demand.
• Tools. Typed function calls to everything the model cannot know or do: your calendar, a SQL database, a code runner, a web search.

This is why agent projects are integration projects. The model arrives finished; most of the work is connecting it to systems never designed to be driven by something that can be confidently wrong.

Between a chatbot and full autonomy sit five patterns, and the instinct to skip them is the most expensive mistake in the field.

Anthropic names them:

• prompt chaining (steps in sequence, with checks between),
• routing (classify the input, send it to a specialist),
• parallelization (run independent calls simultaneously, then merge),
• orchestrator-workers (a lead model decomposes the task and delegates), and
• evaluator-optimizer (one call drafts, another critiques, in a loop).

The difference from an agent is who writes the plan. In these patterns your engineers do, in advance. The model executes steps. That is why workflows are predictable and agents are not — and why Anthropic’s first rule is to find the simplest solution possible and only add autonomy when the task genuinely requires it.

The test is simple: if you can enumerate the steps in advance, you do not need an agent. A refund-approval flow has a known shape. "Investigate why our churn spiked in Q2" does not.

The frameworks you will evaluate do not disagree about the loop. They disagree about where control lives.

LangGraph models an agent as a graph whose nodes and edges you define explicitly, with checkpointing so a run can be paused, inspected and replayed. It is the choice when you need auditability — and the steepest learning curve, because you design the control flow yourself.

CrewAI organises agents as role-based teams: a researcher, a writer, a reviewer, each with a stated job. You describe outcomes more than mechanics, which makes it the fastest route from idea to working demo.

OpenAI Agents SDK is deliberately minimal — agents, handoffs, guardrails — and its handoff model is the cleanest way to pass a task between specialists. The trade-off is gravity: it is built around OpenAI’s APIs.

Microsoft Agent Framework is the 2025 convergence of AutoGen and Semantic Kernel, pairing AutoGen’s conversational multi-agent abstractions with Semantic Kernel’s enterprise state management — the default answer inside Azure estates.

Model Context Protocol (MCP) is not a framework but a standard — a common way to expose tools and data to any model. With over 17,000 public servers listed and a 2026 specification rewritten for stateless, cacheable operation, it is becoming the connective tissue that makes tools portable between all of the above. If you build one asset with lasting value, build MCP tool definitions — frameworks come and go, and your integrations survive them.

One decision cuts across all of these: the model itself is the component you will swap most often. Framework lock-in is annoying; model lock-in is expensive. Endpoints that speak the OpenAI and Anthropic API shapes — such as AKI.IO, which hosts open-weight models on EU infrastructure — let you change the model, the price and the data-protection regime without rewriting the agent.

The strongest argument against multi-agent architectures comes from Cognition, the company behind the coding agent Devin. In Don’t Build Multi-Agents, Walden Yan argues that sub-agents inevitably make conflicting assumptions because they cannot see each other’s reasoning. His two principles: share full agent traces, not just individual messages, and actions carry implicit decisions, and conflicting decisions carry bad results.

This is not a fringe view. A UC Berkeley-led study published at NeurIPS 2025, Why Do Multi-Agent LLM Systems Fail? (Cemri et al.), analysed over 1,600 execution traces across seven leading frameworks — including LangGraph, CrewAI and AutoGen — and found task failure rates between 41% and 86.7%. Its taxonomy of 14 failure modes attributes most of them to system design and to inter-agent misalignment (agents misunderstanding each other’s intent), not to the underlying models.

The market is already correcting. Gartner predicts that over 40% of agentic AI projects will be cancelled by the end of 2027, citing escalating costs, unclear business value and inadequate risk controls.

The pattern across all three: teams bought autonomy before they had a task that needed it.

Agent architecture is a solved problem; knowing when to use it is not. Three things you can do this week:

1. Take one live "agent" initiative and classify it. Workflow or agent? If the steps can be enumerated in advance, rebuild it as a workflow and bank the reliability.
2. Audit the tools, not the model. When Anthropic’s team improved its results on a software-engineering benchmark, it reports spending more time optimising tool definitions than prompts. Your integrations are the moat.
3. Demand the architecture diagram before the PO. Any vendor that cannot show you the loop, the guardrails and the stopping conditions is selling you a chatbot.

If you are prototyping: one model, three tools, a hard step limit, and a log of every decision. That is an agent. Everything else is decoration.