Crispin: Building an AI agent I could trust

Crispin could write to Notion and Google Calendar. The difficult part was making sure that a confident answer corresponded to a completed action.

The problem

Crispin once told me it had captured everything in a long message. It had not.

The message contained a list of tasks and ideas. The agent created the first few items in Notion, stopped making tool calls, and then replied as though the entire request had succeeded. There was no exception and no obvious failure state. The response sounded right; the underlying system was wrong.

Crispin was a working private AI assistant in Telegram. A user could send it tasks, notes, plans or questions in ordinary language. It organised information in Notion, read and changed Google Calendar, and delivered proactive briefings and reminders.

The user experience was intentionally simple. A single request behind it could involve intent classification, retrieval, structured extraction, external API calls, state changes and a final conversational response. Fluent language at the end could conceal a failure at any earlier stage.

The first version used one broad agent with more than 20 tools and a large system prompt. That worked while the product was small. As workflows grew, it became difficult to tell whether a failure belonged to intent, routing, tool choice, arguments, application state or the final explanation.

I needed a system whose behaviour I could inspect, reproduce and improve—not one that merely produced useful-looking answers.

What I built

I replaced the broad agent with a thin router and focused specialists for creation, editing, retrieval, calendar operations, strategy, nudges, reminders and administration. Each specialist received only the tools and context it needed. This reduced the model’s decision surface and created boundaries I could test independently.

Around that architecture, I built more than 300 automated cases at four levels:

• Routing evals checked whether the router selected the correct specialist and passed through the full request.
• Specialist evals inspected tool selection, arguments and return behaviour inside one focused agent.
• Full-chain evals exercised the real router and specialists together, checking route, tool use, specialist return and final response separately.
• Multi-turn evals ran the application path against a database, carried state between turns and checked both data changes and leakage of internal instructions or tool output.

The system used the real prompt assembly and field-for-field tool contracts wherever the production path mattered. A response saying “I’ve added that” did not count as success unless the recorded action supported it.

Where reality disagreed

Three failures changed the architecture most.

The model stopped midway through deterministic work

The original capture flow made one tool call for every item a user wanted to save. Short messages worked. In longer brain dumps, the model could stop after several calls and produce a normal completion, leaving later items uncaptured.

Prompt instructions to count, process and verify every item reduced neither the uncertainty nor the silent partial-success state. The model was being asked to interpret the request and execute a deterministic loop.

A green eval did not represent the product

An early calendar suite appeared reassuring, but its mock tools and prompt construction had drifted from the live application. The model performed well in the test environment while the real workflow still failed: some requests skipped creation, while others resolved relative dates incorrectly.

The test had become an approximation of a system that no longer existed. Its pass state provided false confidence.

Faster inference preserved fluency, not grounded behaviour

I benchmarked lower reasoning settings across different specialists. Constrained routing and some calendar cases became faster without losing cases in their recorded suites. Strategy and nudge workflows degraded differently: they could still produce plausible answers while skipping the tools needed to inspect the user’s workspace.

That was worse than awkward prose. The product’s value depended on personal, grounded information. A fast answer that silently skipped retrieval was not an acceptable optimisation.

How I investigated it

I used evals as a debugging instrument, not as a single accuracy score. Each case needed to locate a failure layer: routing, tool choice, arguments, state transition or final response.

For long captures, I added ordinary and stress cases, including a 30-item message. They checked the number and types of items produced and whether relationships between goals, projects and tasks were preserved.

For calendar work, I rebuilt the suite around the real schemas and prompt assembly. The cases inspected tool calls and arguments rather than accepting a confident final message as proof of completion.

For inference settings, I compared workflows separately. I looked beyond final prose to whether the specialist gathered the evidence its answer required. This exposed groundedness failures that a response-only check would have missed.

The important discipline was production equivalence. When an eval passed and the real workflow failed, I treated the eval as the first thing to debug.

What changed

For capture, the model now makes two structured decisions regardless of message length: return searches to check for duplicates, then return classified items to create. Application code performs the loop.

Inside the batch tool, each item is validated and written independently. Transient failures can be retried. Invalid metadata can be stripped without losing the underlying item. A failure on one item does not prevent later items from being attempted, and the result explicitly reports created and failed items.

For calendar work, I split creation, reading and editing into separate specialists. A higher-level creation tool now fetches events, checks conflicts, searches for available slots and creates the event. A generated calendar reference table handles relative date interpretation without asking the model to calculate day-of-week arithmetic from scratch.

The model still interprets the user’s language and supplies structured fields. Code owns iteration, validation, retries, conflict detection and confirmation that an action happened.

Improved coverage also allowed me to remove complexity. I replaced a broad agent framework configuration with a leaner agent and custom middleware, then reduced a calendar-routing prompt section from 93 lines to 28 while the relevant routing and full-chain suites continued to pass.

Other prompt responsibilities moved into stronger system boundaries:

• preserving nudge output became code-level passthrough;
• including action links became automatic extraction from tool results;
• preventing false calendar confirmation became a structured tool response;
• a fragile onboarding state machine became state inferred from persisted data.

The largest reliability gains came when I stopped treating every failure as a prompt problem.

What I learned

In this system, the model was useful for understanding ambiguous language, classifying intent, choosing among focused capabilities, reading structured context and composing concise responses.

Code was more reliable for iteration, arithmetic, validation, retries, access control, state transitions, conflict detection and confirming that work actually happened.

That boundary is not fixed in advance. It emerged by observing the real product, reproducing failures and changing the layer responsible for each guarantee.

A fluent response is not evidence of a completed action. Trust comes from inspecting behaviour and making the system—not the prose—carry the guarantee.

This changed how I now approach agent work: ship the useful workflow early, make its failures observable, and let those failures decide whether the next change belongs in the prompt, tool, data model, architecture or deterministic code.