What I Learned Building an AI Agent from Scratch

I spent two days building a general-purpose AI agent in TypeScript from first principles. No high-level agent frameworks—just an LLM API, a tool-aware loop, and the patterns that make agents reliable. Here’s what stuck with me.

The Core Idea: It’s Just a Loop

An agent is essentially a loop that:

1. Sends the current conversation (system + history + latest user message) to the LLM.
2. Streams the response and collects tool calls if the model requests them.
3. If the model is done (no tool calls), you break and return the reply.
4. Otherwise, you execute each tool, append the results to the conversation as tool messages, and go back to step 1.

So the “agent” is: conversation state + tool execution + one while (true). Everything else (context window, evals, human approval) is built on top of that.

while (true) {
  const result = streamText({ model, messages, tools, ... });

  // Collect text + tool calls from the stream
  for await (const chunk of result.fullStream) { ... }

  const finishReason = await result.finishReason;
  if (finishReason !== "tool-calls" || toolCalls.length === 0) {
    break; // Model is done
  }

  // Execute tools and append results to messages
  for (const tc of toolCalls) {
    const result = await executeTool(tc.toolName, tc.args);
    messages.push({ role: "tool", content: [{ type: "tool-result", ... }] });
  }
}

No framework is required for this. You just need a model that supports tool calling and a way to run tools and feed results back.

Tools: Describe, Schema, Execute

Each tool has three parts:

• Description – What the tool does (the model uses this to choose tools).
• Input schema – We used Zod so the SDK can validate and type the arguments.
• Execute – An async function that takes the parsed args and returns a string (or something serializable).

Example: a file read tool might take { path: string }, read the file, and return its contents or an error message. The agent doesn’t “see” your filesystem; it only sees the string you return. That’s the contract: tools are functions that turn (name, args) into a string the model can reason about.

We had file tools (read, write, list, delete), and the same pattern extends to shell commands, web search, or any API—define the schema, implement execute, add it to the tool map. The loop doesn’t care what the tool does, only that it returns a string to put back into the conversation.

Message Compatibility and Filtering

In practice, not every message shape is safe to send back to the API. Some provider tools or special responses produce message formats that break the next request. So we added a filter step: before building the next request, we only keep messages that we know our model and SDK handle (e.g. plain text assistant content, standard tool results). Filtering keeps the conversation array in a “compatible” shape and avoids obscure API errors.

Context and Tokens

The conversation grows every turn (user, assistant, tool, assistant, tool, …). So we need to stay under the model’s context window. Two things we did:

• Token estimation – A simple heuristic (e.g. characters / 3.75) to approximate token count per message and for the whole thread. Not exact, but good enough to know when we’re getting close to the limit.
• Compaction – A placeholder for “summarize old messages into a shorter history” so we can keep the tail of the conversation and a summary of the rest. We didn’t fully implement it in my version, but the idea is: when estimated tokens exceed a threshold, run a separate LLM call to summarize the middle of the thread, then replace that range with a short summary message and continue. That way the agent can still use “memory” without blowing the context.

Together, estimate tokens → compact when needed → then run the agent loop keeps the agent usable for long interactions.

Evals: Single-Step and Multi-Step

We had two levels of evaluation:

• Single-turn – Given one user message and maybe a context, did the model call the right tool with the right arguments? Good for testing tool choice and input shaping in isolation.
• Multi-turn – Run the full agent loop with a short script of user messages and check: were the right tools called in the right order? Were dangerous or irrelevant tools not called? And did the final answer make sense (e.g. with an LLM-as-judge)?

Tools were mocked in evals so that file reads, shell commands, etc. return fixed values. That made runs deterministic and fast. The evaluators then checked tool order, “forbidden” tools, and sometimes used another LLM to score the final response. That combination caught a lot of mistakes that you’d never see from a single prompt test.

Human-in-the-Loop (HITL) for Risky Tools

For tools that can change or break things (e.g. shell commands, bulk file writes), we added a tool approval step. Before executing such a tool, the agent pauses and asks the user (in our case, in a terminal UI): “Run this tool with these args? Yes / No.” Only if the user approves do we call executeTool and append the result.

Mechanically, that meant a callback like onToolApproval(toolName, args) that returns a Promise<boolean>. The UI (we used Ink for a React-based CLI) showed the tool name and a short preview of the arguments, and resolved the promise when the user chose Yes or No. The agent loop stays the same; the only change is “before executing this tool, await approval.” Same pattern could be used for a web UI or a Slack button.

Streaming and Callbacks

We streamed the model output and exposed everything via callbacks: onToken, onToolCallStart, onToolCallEnd, onComplete. That kept the core loop simple and let the CLI (or any client) update the UI in real time—tokens as they arrived, tool calls as they started and finished. We also plugged in telemetry (e.g. Laminar) so we could trace token usage and tool calls in production. So: streaming + callbacks + optional tracing gave a good dev and ops story without coupling the loop to a specific UI.

What I’d Do Next

• Finish context compaction – Implement the summarization step and wire it to the token estimator so long chats don’t hit the context limit.
• More tools – Web search, code execution, browser handoff; each is just another tool with a description, schema, and execute function.
• Stronger evals – More multi-turn scenarios and edge cases (e.g. model tries to use the wrong tool, or gives up too early).
• Composability – Use this agent as a tool inside another system, or call other agents as tools. The same loop and tool contract make that straightforward.

Overall, the big takeaway was: the agent is a small, clear loop plus a disciplined tool contract. Once that’s in place, everything else—context, evals, guardrails, UX—sits around it without changing the core. If you’re comfortable with TypeScript and an LLM API, you can build something like this in a couple of days and then keep extending it.