AI agents explained

The four components

The LLM is the planner-in-the-loop, not the orchestration engine. You own execution, validation, audit logs, and termination. Without that boundary, “agent” collapses into an unbounded chat that hallucinates API responses and never stops calling tools.

flowchart TB
  subgraph runtime["Your agent runtime"]
    P[Planner / loop]
    M[Memory messages state]
    T[Tool router allowlist]
  end
  U[User goal] --> P
  P --> LLM["LLM API"]
  LLM -->|tool_calls or text| P
  P --> T
  T -->|execute| EXT["DB APIs search email"]
  EXT -->|observation| M
  M --> P
  P -->|final answer| U

Planning: implicit vs explicit

Implicit planning happens inside the model’s hidden reasoning—one tool call at a time until done. Explicit planning asks the model to emit a numbered plan first, then execute step-by-step (useful for long workflows; costs extra tokens). Both are valid; explicit plans help debugging and human review but can go stale if tool results contradict the plan.

Memory layers (preview)

• Working memory — the messages list for this request: user, assistant, tool roles.
• Session memory — trimmed prior turns or rolling summary for multi-turn chat (Guide 3).
• Long-term memory — facts extracted and stored in a DB or vector index for future sessions.

Track 3 taught context assembly under a token budget. Agents add tool observations to that stack—each round appends assistant + tool messages, so memory management becomes critical (Guide 3 in this track).

Minimal agent type definition

from dataclasses import dataclass, field
from typing import Callable, Any

@dataclass
class AgentState:
    """Working memory + control flags for one user request."""
    messages: list[dict] = field(default_factory=list)
    round_count: int = 0
    max_rounds: int = 8
    tool_log: list[dict] = field(default_factory=list)
    done: bool = False
    final_answer: str | None = None

@dataclass
class AgentRuntime:
    model: str
    tools: list[dict]           # JSON schemas for the API
    tool_impls: dict[str, Callable[..., Any]]  # name -> Python fn
    system_prompt: str

def step(state: AgentState, runtime: AgentRuntime, llm_client) -> AgentState:
    """One observe-think-act iteration — planner owns the loop."""
    if state.round_count >= state.max_rounds:
        state.done = True
        state.final_answer = "Step limit reached."
        return state
    # ... call LLM, dispatch tools, append observations (core-loop section)
    return state

public record AgentState(
    List<Message> messages,
    int roundCount,
    int maxRounds,
    List<ToolLogEntry> toolLog,
    boolean done,
    String finalAnswer
) {
  public AgentState withRound(int r) {
    return new AgentState(messages, r, maxRounds, toolLog, done, finalAnswer);
  }
}

public record AgentRuntime(
    String model,
    List<ToolDefinition> tools,
    Map<String, ToolFunction> toolImpls,
    String systemPrompt
) {}

@Service
public class AgentPlanner {
  public AgentState step(AgentState state, AgentRuntime runtime) {
    if (state.roundCount() >= state.maxRounds()) {
      return new AgentState(state.messages(), state.roundCount(),
          state.maxRounds(), state.toolLog(), true, "Step limit reached.");
    }
    // call ChatClient, dispatch tools, append observations
    return state;
  }
}

Notice what is not in the LLM: the allowlist map, round counter, and audit log live in your runtime. The model never directly invokes lookup_order—it emits a structured tool_calls payload your router validates.

Before native tool calling

Early “agents” parsed free-text actions with regex: Action: search_kb(query="refund"). Fragile—models drifted format, hallucinated observations, and burned tokens on verbose Thought blocks. Production teams built brittle parsers and retry loops.

Native tools / tool_calls in chat completions return structured JSON: function name + arguments validated against your schema. The provider fine-tuned models to emit valid tool payloads—ReAct-style prompting became optional, not mandatory.

Three enablers (2024–2026)

Provider tool-calling shape (OpenAI-style)

You register tool schemas once per request. The model may return finish_reason: tool_calls with one or more parallel invocations. You execute, append role: tool messages with tool_call_id, and call the model again.

from openai import OpenAI

client = OpenAI()

TOOLS = [{
    "type": "function",
    "function": {
        "name": "lookup_order",
        "description": "Get shipping status for order id ORD-####",
        "parameters": {
            "type": "object",
            "properties": {"order_id": {"type": "string"}},
            "required": ["order_id"],
            "additionalProperties": False,
        },
    },
}]

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "Use tools for live order data."},
        {"role": "user", "content": "Where is order ORD-1042?"},
    ],
    tools=TOOLS,
    tool_choice="auto",
)

msg = resp.choices[0].message
if msg.tool_calls:
    # Structured — no regex on "Action: lookup_order(...)"
    for call in msg.tool_calls:
        print(call.function.name, call.function.arguments)

@Bean
FunctionCallback lookupOrderCallback() {
  return FunctionCallback.builder()
      .function("lookup_order", (OrderIdRequest req) -> orderService.status(req.orderId()))
      .description("Get shipping status for order id ORD-####")
      .inputType(OrderIdRequest.class)
      .build();
}

String reply = chatClient.prompt()
    .system("Use tools for live order data.")
    .user("Where is order ORD-1042?")
    .functions(lookupOrderCallback())
    .call()
    .content();
// Spring AI maps tool_calls to Java methods — same structured contract

Instruction following in multi-step loops

Each loop round re-sends the full messages prefix. System rules from Track 3 (“do not invent tool results,” “cite policy text”) must survive 5–10 rounds of tool noise. Models trained for tool use maintain format longer than 2022-era chat models—but you still cap rounds and log drift.

Context budget in agent loops

Every round bills input tokens for the entire transcript so far. A six-round agent with 2K tokens per observation can exceed 12K input before the user question. Techniques:

• Truncate tool outputs to the smallest useful slice (top 3 hits, 500 chars).
• Summarize completed sub-goals into one assistant line; drop raw tool JSON.
• Use smaller models for tool-selection rounds; larger model only for final synthesis.

Comparison table

Use a chatbot when the answer fits in one generation from static or pre-assembled context. Use RAG alone when one retrieval + one answer suffices—see RAG explained. Use an agent when the user goal requires several lookups whose order depends on intermediate results: “Check order ORD-1042, then explain refund rules if it shipped late.”

Responds vs acts — concrete example

User: “Where is order ORD-1042, and can I get a refund on my monthly plan?”

flowchart LR
  subgraph chat["Chatbot — responds"]
    U1[User] --> M1[Model once]
    M1 --> A1[Answer]
  end
  subgraph agent["Agent — acts"]
    U2[User] --> M2[Model]
    M2 -->|tool_calls| T[Tools]
    T -->|observation| M2
    M2 --> A2[Final answer]
  end

When not to build an agent

• Fixed workflow — known steps in order; code the steps, use LLM only for NLU or summarization.
• Single retrieval suffices — FAQ with one search + one answer; agent adds latency and cost.
• High-stakes writes without human review — refunds, account deletion, wire transfers need approval UI, not autonomous loops.
• Eval simplicity — single-turn RAG golden sets are easier than multi-path agent trajectories (Track 5).

QUESTION = "Where is order ORD-1042, and can I refund my monthly plan?"

# Chatbot — one shot, no tools (often hallucinates)
chat_resp = client.chat.completions.create(
    model="gpt-4o-mini",
    temperature=0.2,
    messages=[
        {"role": "system", "content": "You are Acme support."},
        {"role": "user", "content": QUESTION},
    ],
)

# Agent — run_agent() loops until no tool_calls (see core-loop section)
agent_result = run_agent(QUESTION)
# agent_result["tool_log"] shows lookup_order + search_policy
# agent_result["answer"] should cite shipped status + no partial-month refunds

String question = "Where is order ORD-1042, and can I refund my monthly plan?";

// Chatbot — single ChatClient call
String chatAnswer = chatClient.prompt()
    .system("You are Acme support.")
    .user(question)
    .call()
    .content();

// Agent — AgentService.run() loops with FunctionCallbacks
AgentResult agentResult = agentService.run(question);
// agentResult.toolLog() lists lookup_order, search_policy
// agentResult.answer() grounded in tool JSON

Loop steps (production-shaped)

1. Initialize — append system prompt and user message to messages.
2. Think — call LLM with tool schemas; model returns text and/or tool_calls.
3. Observe — if tool calls present, validate args, execute allowlisted functions, append role: tool messages.
4. Check termination — if no tool calls, return assistant text; if round >= MAX_ROUNDS, return safe fallback.
5. Log — record tool name, args hash, latency, success/fail for every act step.

flowchart TD
  START([User message]) --> OBS[Observe state messages]
  OBS --> THINK[Think LLM call]
  THINK --> DEC{tool_calls?}
  DEC -->|yes| ACT[Act execute tools]
  ACT --> APP[Append observations]
  APP --> OBS
  DEC -->|no| DONE([Final answer])
  THINK --> MAX{round >= MAX?}
  MAX -->|yes| SAFE([Safe fallback message])

Full agent runner

This is the reference loop for the Acme support demo used across Track 4 guides. Tools are mocked; swap real APIs without changing loop shape.

import json
from openai import OpenAI

client = OpenAI()
MAX_ROUNDS = 6

SYSTEM = (
    "You are Acme Corp support. Use tools for order status and policy facts. "
    "Do not invent tracking numbers or refund rules. "
    "If tools return not_found, say you are not sure."
)

TOOL_DEFINITIONS = [/* lookup_order, search_policy schemas — Guide 2 */]

ALLOWED = {
    "lookup_order": lookup_order_impl,
    "search_policy": search_policy_impl,
}

def run_tool(name: str, args_json: str) -> str:
    if name not in ALLOWED:
        return json.dumps({"error": "tool_not_allowed"})
    args = json.loads(args_json or "{}")
    # validate per tool — never trust raw model output
    return json.dumps(ALLOWED[name](**validated_args(name, args)))

def run_agent(user_message: str) -> dict:
    messages = [
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": user_message},
    ]
    tool_log = []

    for round_i in range(MAX_ROUNDS):
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=messages,
            tools=TOOL_DEFINITIONS,
            tool_choice="auto",
            temperature=0.2,
            max_tokens=500,
        )
        msg = response.choices[0].message
        messages.append(msg.model_dump(exclude_none=True))

        if not msg.tool_calls:
            return {"answer": msg.content or "", "rounds": round_i + 1, "tool_log": tool_log}

        for call in msg.tool_calls:
            output = run_tool(call.function.name, call.function.arguments)
            tool_log.append({
                "round": round_i + 1,
                "name": call.function.name,
                "arguments": call.function.arguments,
            })
            messages.append({
                "role": "tool",
                "tool_call_id": call.id,
                "content": output,
            })

    return {
        "answer": "Sorry, I could not complete this within the step limit.",
        "rounds": MAX_ROUNDS,
        "tool_log": tool_log,
        "error": "max_rounds",
    }

@Service
public class AcmeAgentService {
  private static final int MAX_ROUNDS = 6;
  private final ChatClient chatClient;
  private final ToolRouter toolRouter;

  public AgentResult run(String userMessage) {
    List<Message> messages = new ArrayList<>(List.of(
        new SystemMessage(SYSTEM_PROMPT),
        new UserMessage(userMessage)
    ));
    List<ToolLogEntry> toolLog = new ArrayList<>();

    for (int round = 0; round < MAX_ROUNDS; round++) {
      ChatResponse response = chatClient.prompt(new Prompt(messages))
          .tools(toolRouter.definitions())
          .call();

      AssistantMessage assistant = response.getResult().getOutput();
      messages.add(assistant);

      if (!assistant.hasToolCalls()) {
        return new AgentResult(assistant.getText(), round + 1, toolLog);
      }

      for (ToolCall call : assistant.getToolCalls()) {
        String output = toolRouter.run(call.name(), call.arguments());
        toolLog.add(new ToolLogEntry(round + 1, call.name(), call.arguments()));
        messages.add(new ToolResponseMessage(call.id(), output));
      }
    }
    return AgentResult.maxRoundsExceeded(toolLog);
  }
}

Termination conditions

Parallel tool calls

Models may emit multiple tool_calls in one assistant message—e.g. lookup_order and search_policy together. Execute in parallel when independent; preserve order in transcript when results depend on each other.

Transcript shape

Worked example (Acme support)

User: “Where is order ORD-1042, and can I refund my monthly plan?”

Thought: I need live order status and refund policy for monthly plans. I'll query both.
Action: lookup_order(order_id="ORD-1042")
Observation: {"order_id": "ORD-1042", "status": "shipped", "eta_days": 2}

Thought: Order is shipped. Now check monthly refund policy before answering.
Action: search_policy(query="monthly plan refund partial month")
Observation: {"source": "refund-policy.txt", "snippet": "Monthly plans: no partial-month refunds."}

Thought: I have order status and policy. I can answer the user.
Final Answer: Order ORD-1042 shipped and should arrive in about 2 days. Monthly plans can be canceled anytime, but we do not refund partial months per [refund-policy.txt].

With native tool calling, Thoughts may appear in message.content alongside structured tool calls—useful for UI “reasoning” panels. You can suppress Thoughts in production to save tokens via system prompt: “Do not output chain-of-thought; call tools directly.”

REACT_SYSTEM = """You are Acme support. Use tools when you need live data.

Before each tool call you may briefly state your plan (one sentence).
After tools return, synthesize a final user-facing answer.
Never invent data not present in tool observations."""

# Same run_agent() loop — tool_calls replace regex Action: lines
result = run_agent("Where is ORD-1042 and can I refund monthly?")

# Inspect transcript for ReAct shape:
# assistant(content="I'll check order status...", tool_calls=[...])
# tool(content='{"status":"shipped",...}')
# assistant(content="Order ORD-1042 is shipped...")

public static final String REACT_SYSTEM = """
    You are Acme support. Use tools when you need live data.
    Before each tool call you may briefly state your plan (one sentence).
    After tools return, synthesize a final user-facing answer.
    Never invent data not present in tool observations.""";

AgentResult result = acmeAgentService.run(
    "Where is ORD-1042 and can I refund monthly?");

// ChatResponse history shows:
// AssistantMessage with text + ToolCalls
// ToolResponseMessage with JSON observation
// Final AssistantMessage without tool calls

Text ReAct (legacy / teaching)

When tool APIs are unavailable, parse Action: tool(args) from assistant text and inject Observation: as a user message. See Core prompting techniques — ReAct for regex loop. Prefer native tool calling in production—parsers break under model updates.

ReAct vs plan-and-execute

1. Tool loops (runaway rounds)

The model repeatedly calls the same tool with the same or slightly varied args—searching “refund policy” five times without synthesizing an answer. Causes: vague system prompt, missing “stop when sufficient,” or tool returning empty results without a clear error shape.

• Mitigation: hard MAX_ROUNDS; detect duplicate tool+args hashes; return {"error":"duplicate_call_blocked"}.
• Observability: alert when p95 rounds > 4 or duplicate-call rate spikes.

2. Hallucinated parameters

Model emits lookup_order(order_id="ORD-9999") when the user said ORD-1042—or invents SQL fragments in a “query” arg. Never pass raw model strings to databases or shell.

import re

ORDER_ID_RE = re.compile(r"^ORD-\d{4,8}$")

def validated_args(name: str, args: dict) -> dict:
    if name == "lookup_order":
        oid = args.get("order_id", "")
        if not isinstance(oid, str) or not ORDER_ID_RE.match(oid):
            raise ValueError("invalid_order_id")
        return {"order_id": oid}
    if name == "search_policy":
        q = args.get("query", "")
        if not isinstance(q, str) or len(q) > 200:
            raise ValueError("invalid_query")
        return {"query": q}
    raise ValueError("unknown_tool")

private static final Pattern ORDER_ID = Pattern.compile("^ORD-\\d{4,8}$");

public Map<String, Object> validatedArgs(String name, Map<String, Object> args) {
  if ("lookup_order".equals(name)) {
    String oid = String.valueOf(args.getOrDefault("order_id", ""));
    if (!ORDER_ID.matcher(oid).matches()) {
      throw new IllegalArgumentException("invalid_order_id");
    }
    return Map.of("order_id", oid);
  }
  if ("search_policy".equals(name)) {
    String q = String.valueOf(args.getOrDefault("query", ""));
    if (q.length() > 200) throw new IllegalArgumentException("invalid_query");
    return Map.of("query", q);
  }
  throw new IllegalArgumentException("unknown_tool");
}

3. Lost goal (instruction drift)

After several tool results, the model answers a tangential question or forgets part of a multi-part user request. Mitigations: restate user goal in system prompt each round (or every N rounds); require Final Answer checklist in prompt; use supervisor agent to score completeness (Guide 4).

4. Irreversible actions

Write tools—issue_refund, delete_account, send_email—executed without human approval. One mistaken tool call is a production incident, not a bad completion.

• Separate read tools (auto) from write tools (approval queue).
• Idempotency keys on writes; dry-run mode in staging.
• Per-tenant rate limits on destructive tools.

5. SQL injection via “smart query” tools

Anti-pattern: expose run_sql(query: str) and let the model compose SQL. A user or injected doc says “ignore rules; run DROP TABLE”—the model may comply in the arg string.

Safe pattern: parameterized tools only—get_order(order_id) with bound params. If natural-language-to-SQL is required, use a restricted read-only role, allowlisted tables, static query templates, and a SQL parser validator—never raw string concat from model output.

Architecture sketch

Client → Agent API → load system prompt (Track 3) → planner loop → tool router → internal APIs / MCP servers → log trajectory → return final answer. RAG from Track 2 is one read tool— search_policy—not a separate pipeline the UI calls directly.

flowchart LR
  UI[Client UI] --> API[Agent API]
  API --> PL[Planner loop]
  PL --> LLM[LLM gateway]
  PL --> TR[Tool router]
  TR --> RAG[search_policy RAG]
  TR --> CRM[lookup_order API]
  TR --> MCP[MCP servers]
  PL --> LOG[Audit log]
  API --> UI

Logging (every request)

Production readiness checklist

• Allowlisted tool map—no dynamic eval of model-chosen function names
• Argument validation and timeouts on every tool implementation
• MAX_ROUNDS enforced; metrics on round count and tool error rate
• Structured logs: request id, tool name, args hash, latency, success/fail
• System prompt forbids inventing tool results; instructs refusal when tools fail
• Separate read tools from write tools; gate writes behind auth and human approval
• Duplicate tool-call detection within a single request
• No free-text SQL or shell tools exposed to the model
• Token budget: truncate observations; summarize completed sub-goals
• Golden eval cases for multi-step trajectories—not only final answer match
• Feature flag for agent path vs RAG-only fallback
• SSE or polling for intermediate tool events in user-facing UI (transparency)

Minimal production-shaped agent service

import hashlib
import logging
from fastapi import FastAPI
from pydantic import BaseModel

log = logging.getLogger("agent")
app = FastAPI()

class AgentRequest(BaseModel):
    message: str
    tenant_id: str
    session_id: str

@app.post("/v1/agent/run")
def agent_run(req: AgentRequest) -> dict:
    request_id = f"req_{hashlib.sha256(req.session_id.encode()).hexdigest()[:12]}"
    result = run_agent(req.message)
    log.info("agent_done", extra={
        "request_id": request_id,
        "tenant_id": req.tenant_id,
        "rounds": result["rounds"],
        "tools": [t["name"] for t in result["tool_log"]],
        "termination": result.get("error", "done"),
        "prompt_version": "support_agent_v1",
    })
    return {"request_id": request_id, "answer": result["answer"]}

@RestController
@RequestMapping("/v1/agent")
public class AgentController {
  private final AcmeAgentService agentService;
  private static final Logger log = LoggerFactory.getLogger(AgentController.class);

  @PostMapping("/run")
  public AgentResponse run(@RequestBody AgentRequest req) {
    String requestId = "req_" + UUID.randomUUID().toString().substring(0, 12);
    AgentResult result = agentService.run(req.message());
    log.info("agent_done request_id={} tenant={} rounds={} tools={} termination={}",
        requestId, req.tenantId(), result.rounds(),
        result.toolLog().stream().map(ToolLogEntry::name).toList(),
        result.terminationReason());
    return new AgentResponse(requestId, result.answer());
  }
}

Combine agents with RAG (preview)

Production copilots expose search_policy as hybrid retrieval from Track 2—not a hard-coded mock. The agent decides when to search; RAG supplies what text returns. Guide 6 covers agentic RAG patterns in depth.