How to Build AI Agents for Production: A Practical Guide

An AI agent is a language model running in a loop: it reads the goal and everything that has happened so far, decides whether to call a tool, observes the result, and repeats until it can answer or hits a limit. Building one that works in production is mostly ordinary engineering around that loop: a small set of well-described tools, explicit state that survives restarts, hard budgets enforced in code, permission checks outside the model, and an evaluation suite of real tasks. Start with a fixed workflow, and give the model control of the sequence only where the path genuinely cannot be known in advance.

Key takeaways

• Many "agent" projects are better as workflows: your code fixes the sequence and the model handles one or two steps. Use an agent when the next step depends on what the model finds.
• Reliability comes from tool design more than prompt design. Few tools, clear names, typed arguments, compact results and errors the model can act on.
• Put every limit in code: steps, tokens, wall-clock time, spend, and human approval for irreversible actions.
• Persist run state outside the context window so a run can pause, resume and be audited.
• Evaluate on tasks with checkable end states, run several times each, not on whether the transcript reads well.

What an AI agent is, and what it is not

The core loop is small:

Every modern model API supports native tool calling: you pass tool names, descriptions and JSON Schemas for their arguments, and the model returns structured calls instead of free text. The older pattern of prompting the model to print "Thought / Action / Observation" lines and parsing them with regular expressions is no longer necessary, and it was always fragile.

What separates agents from other LLM applications is who owns the control flow.

Move down the table only when the row above cannot do the job. Every row down adds cost, latency and new ways to fail.

Workflow or agent? Decide first

Ask four questions before writing any agent code.

1. Can you draw the flowchart? If a domain expert can sketch the steps on a whiteboard, build that flowchart as code and use the model inside the boxes.
2. Does the number of steps vary with the input? Debugging a failed data sync might take two lookups or twelve. That is agent territory.
3. Must it recover from surprises? An agent can read a tool error and try a different approach. A workflow needs every branch written in advance.
4. What does a wrong step cost? If a mistake is cheap and reversible (a wrong search), autonomy is fine. If it sends money, email or deletes data, put a human or a policy check in front of that tool regardless of pattern.

A common and sensible hybrid: a workflow on the outside with an agent inside one step. For example, a support pipeline whose code handles authentication, classification and handoff, with an agent only inside the "investigate technical issue" branch.

The agent loop in code

Here is a provider-agnostic loop with the production details that tutorials skip: budgets, argument validation, tool errors returned as observations, approval for side effects, capped tool output and a tool log. call_model is a thin adapter over your provider's tool-calling API that returns the text, any tool calls, and token usage.

"""agent_loop.py: a minimal agent loop with production guardrails."""
import json
import time
from dataclasses import dataclass, field
from typing import Any, Callable
 
from jsonschema import validate  # pip install jsonschema
 
 
@dataclass
class Tool:
    name: str
    description: str
    parameters: dict  # JSON Schema for the arguments
    fn: Callable[..., Any]
    side_effects: bool = False  # True for anything that writes, sends, pays or deletes
 
 
@dataclass
class Budget:
    max_steps: int = 12
    max_tokens: int = 60_000
    max_seconds: float = 90.0
 
 
@dataclass
class RunState:
    messages: list = field(default_factory=list)
    steps: int = 0
    tokens: int = 0
    started: float = field(default_factory=time.monotonic)
    tool_log: list = field(default_factory=list)
 
 
def clip(text: str, limit: int = 4000) -> str:
    return text if len(text) <= limit else text[:limit] + f"... [truncated {len(text) - limit} chars]"
 
 
def execute(call, registry: dict, approve) -> dict:
    tool = registry.get(call.name)
    if tool is None:
        return {"ok": False, "error": f"Unknown tool {call.name}. Available: {sorted(registry)}"}
    try:
        args = json.loads(call.arguments) if isinstance(call.arguments, str) else call.arguments
        validate(args, tool.parameters)
    except Exception as exc:
        return {"ok": False, "error": f"Invalid arguments: {exc}. Correct them and retry."}
    if tool.side_effects and (approve is None or not approve(call.name, args)):
        return {"ok": False, "error": "Action not approved. Do not retry it; explain what you intended."}
    try:
        return {"ok": True, "result": tool.fn(**args)}
    except Exception as exc:
        # Tool failures become observations the model can reason about, not crashes.
        return {"ok": False, "error": f"{type(exc).__name__}: {exc}"}
 
 
def run_agent(goal: str, tools: list[Tool], call_model, system: str,
              budget: Budget = Budget(), approve=None) -> dict:
    registry = {t.name: t for t in tools}
    state = RunState(messages=[{"role": "system", "content": system},
                               {"role": "user", "content": goal}])
    last_call = None
    while True:
        if state.steps >= budget.max_steps:
            return stopped(state, "step_limit")
        if state.tokens >= budget.max_tokens:
            return stopped(state, "token_limit")
        if time.monotonic() - state.started > budget.max_seconds:
            return stopped(state, "time_limit")
 
        reply = call_model(state.messages, tools)
        state.steps += 1
        state.tokens += reply.usage_tokens
        state.messages.append(reply.as_message())
 
        if not reply.tool_calls:
            return {"status": "done", "answer": reply.text,
                    "steps": state.steps, "tokens": state.tokens, "tool_log": state.tool_log}
 
        for call in reply.tool_calls:
            signature = (call.name, json.dumps(call.arguments, sort_keys=True))
            if signature == last_call:
                return stopped(state, "repeated_tool_call")
            last_call = signature
            result = execute(call, registry, approve)
            state.tool_log.append({"step": state.steps, "tool": call.name,
                                   "args": call.arguments, "ok": result["ok"]})
            state.messages.append({"role": "tool", "tool_call_id": call.id,
                                   "content": clip(json.dumps(result, default=str))})
 
 
def stopped(state: RunState, reason: str) -> dict:
    # Hitting a budget is an outcome the caller must handle, not an exception to swallow.
    return {"status": "stopped", "reason": reason, "steps": state.steps,
            "tokens": state.tokens, "tool_log": state.tool_log}

Four details carry most of the value. Side-effecting tools are denied unless an approver says yes, so forgetting to wire approval fails safe. Invalid arguments and tool exceptions go back to the model as readable errors, which is how agents recover. Identical repeated calls stop the run, which catches the most common loop. And every exit path returns a status and reason that the caller can act on.

Tool design is most of the work

When an agent misbehaves, the cause is more often a tool than the model. The model only knows what your names, descriptions and results tell it.

Rules we apply to every tool:

• Write the description for a new hire. What it does, when to use it, when not to, and what the arguments mean, with units (amount_cents, not amount).
• Use enums and required fields in the schema. Every free-text argument is a place for the model to improvise.
• Return stable IDs and small results. Paginate lists. The model can always ask for more; it cannot un-read a 30,000-token dump.
• Enforce policy in the tool, not the prompt. "Refunds over the policy limit need a manager" is a check in refund_order, not a sentence in the system prompt.
• Make writes idempotent and give risky tools a dry_run mode that returns what would happen.
• Scope credentials to the user. The tool should act with the permissions of the person the agent is working for, not a service account that can see everything.

When several agents or applications need the same tools, expose them once through the Model Context Protocol instead of re-implementing them per framework. Our MCP guide covers building and deploying MCP servers.

Agent patterns that work

Multi-agent only when it earns it

Multi-agent designs look elegant in diagrams. In production each agent rebuilds context from scratch, handoffs lose information, token cost multiplies, and when the final answer is wrong it is hard to tell which agent caused it. They earn their place in two situations: genuinely parallel, independent subtasks, and isolation, where one part of the work needs permissions or data the rest must not touch. Otherwise, one agent with good tools beats a committee.

State and memory

"Memory" covers three different things that should live in different places.

Practical rules:

• Compact the working context. Summarize old tool results and keep their IDs. Every step resends the whole transcript, so total tokens grow much faster than the step count.
• Checkpoint after every step. A run waiting four hours for a human approval must survive a deploy. LangGraph's checkpointers do this for you; in plain code it is one row per run in Postgres.
• Write long-term memory deliberately. Store facts with their source and an expiry, not every message. Anything written to memory from untrusted input is a prompt injection that persists across sessions.

Choosing a framework

Frameworks change quickly, so choose on properties rather than popularity:

• Durable state and resume after failure
• Human-in-the-loop interrupts that pause a run and wait
• Streaming of intermediate steps to the user interface
• Tracing hooks that export to your observability stack
• Tool schemas generated from typed functions
• MCP client support
• An easy escape hatch to plain code

LangGraph models agents as explicit state machines with checkpointing and suits long-running, branching flows with approvals. The agent SDKs from model providers integrate tightly with their own models and hosted tools. CrewAI and similar frameworks organize multi-agent work around roles. A plain loop like the one above is also a legitimate production choice for a single agent: it is about a hundred lines, has no dependency churn, and every behavior is visible in your own code. Boring where possible, exotic where it earns it.

Limits and guardrails

Put every limit in code, where the model cannot talk its way past it.

• Budgets per run: steps, tokens, wall-clock time and money. Also a daily budget per user and per tenant, because one bad prompt stuck in a loop can run up a large bill overnight.
• Approval gates for irreversible actions. Show the approver the exact tool and arguments, not the model's summary of them.
• Least-privilege credentials per tool, scoped to the acting user, short-lived where possible.
• Untrusted tool output. Web pages, emails, tickets and documents can contain text written to steer your agent. Never let tool output grant new permissions or change the goal.
• A kill switch per tool and per agent that on-call can flip without a deploy.

This is the short version. The full threat model, from prompt injection to tool authorization and sandboxing, is in our guide to securing AI agents in production, and the pre-launch list is in the AI agent hardening checklist.

Evaluating agents

Reading transcripts is not evaluation. Agents need a task suite.

• Real tasks with checkable end states. Start with a few dozen tasks taken from real requests. Each one defines what must be true afterwards: a refund record exists with the right amount, the ticket has the right label, no email went to the wrong address.
• A sandbox with fake backends. Tools hit test doubles or a seeded staging environment, so runs are safe and repeatable.
• Several trials per task. The same task can pass on one run and fail on the next. Track pass rate per task, not a single pass or fail.
• Metrics that matter: success rate, steps per success, cost per success, tool error rate, unsafe action attempts, and how often the agent correctly hands off to a human.
• Re-run on every change to the model, system prompt or any tool description. Tool descriptions are prompts.

A task test reads like an integration test:

def test_refund_for_damaged_item(sandbox, call_model):
    result = run_agent(
        goal="Customer 88 says order 4471 arrived broken and wants a refund.",
        tools=sandbox.tools, call_model=call_model, system=SUPPORT_PROMPT,
        approve=sandbox.policy_approver,
    )
    assert result["status"] == "done"
    refund = sandbox.db.refund_for("4471")
    assert refund is not None and refund.amount_cents == sandbox.order("4471").total_cents
    assert sandbox.emails_sent_to("customer-88") == 1
    assert result["steps"] <= 8

For observability, trace each run as one trace with a span per model call and per tool call: input size, output, tool arguments, result status, latency, tokens and cost. OpenTelemetry has semantic conventions for generative AI spans, and LLM tracing tools such as Langfuse, LangSmith and Arize Phoenix render these as step-by-step timelines. Without traces, a failed agent run is close to impossible to debug.

What goes wrong in production

The failures below are the ones that wake people up. Plan for each before launch.

• Loops. The agent calls the same failing tool with the same arguments, forever. Detect repeats and stop, and make tool errors specific enough that a retry with different arguments is possible.
• Silent partial completion. The agent finishes three of five steps, then hits the budget. Without checkpointed state and a "stopped, reason" result, nobody knows the job is half done. Design compensating actions for multi-step writes.
• Duplicate side effects. A timeout triggers a retry and the customer gets two refunds. Idempotency keys on every write tool, enforced server-side.
• Context bloat. A tool that returns full records fills the context window by step six. Quality drops, latency climbs, and the cost per run jumps. Cap and summarize tool output.
• Model upgrades change behavior. A new model version picks tools differently or becomes more or less willing to ask for approval. Pin model versions and run the task suite before switching.
• Injection through tools. A web page, document or ticket carries instructions aimed at the agent. Keep untrusted content in tool results, never in the system prompt, and keep consequential actions behind policy checks that ignore what the model says.
• Spend spikes. One user, one loop, one weekend. Per-user and per-tenant budgets with alerts, not just per-run limits.
• Nobody owns it. An agent is a production service. It needs a runbook, dashboards, alerts on success rate and cost, and a named on-call owner.

If your agent mostly answers questions from documents, get retrieval right first. Our RAG architecture guide covers chunking, hybrid search, reranking and evaluation in depth.

Frequently Asked Questions

What is an AI agent?

An AI agent is a language model running in a loop that decides which tools to call, observes the results and repeats until it can finish the task or hits a limit. The model chooses the control flow at run time instead of following a fixed sequence written by a developer. Tools are what let it act: search, query a database, call an API or hand off to a human.

What is the difference between an AI agent and a workflow?

In a workflow, your code decides the sequence of steps and the model fills in individual steps, such as classifying or drafting. In an agent, the model decides which step comes next based on what it has seen so far. Workflows are cheaper, faster and easier to test, so use an agent only where the path genuinely cannot be known in advance.

Which framework should I use to build AI agents?

Pick on durable state, human approval interrupts, tracing and how easily you can drop down to plain code, not on demo speed. LangGraph, the agent SDKs from model providers and role-based frameworks such as CrewAI all work. A plain loop of about a hundred lines plus a tracing library is also a legitimate production choice for a single agent.

How do you stop an AI agent from looping forever?

Enforce budgets in code, outside the model: a maximum number of steps, a token budget, a wall-clock timeout and a spend limit per run and per user. Also detect repeated identical tool calls and stop early. When a budget is hit, return a clear stopped status with the reason so the caller or a human can take over.

When do you need a multi-agent system?

Rarely at the start. Multiple agents make sense when subtasks are truly independent and can run in parallel, or when they need different permissions that should not share one context. Otherwise one agent with well-designed tools is cheaper, faster and much easier to debug.

How do you evaluate an AI agent?

Build a suite of real tasks with end states you can check in a sandbox, such as a refund record with the right amount or a ticket with the right label. Run each task several times because agent behavior varies, and track success rate, steps, cost per success and unsafe action attempts. Re-run the suite whenever the model, prompt or a tool description changes.

Need help building AI agents?

We build agents that act on real systems, then stay on the pager for them. Senior engineers design the tools, permissions, limits and evaluation suite so the agent behaves on day ninety the way it did in the demo. If we built it, we're on the pager.

• AI and machine learning services
• POC-to-production engineering
• Talk to an engineer