Four Agents Walk Into a Service Bay

"It makes a noise sometimes"

Most service visits start with a vague sentence. A technician needs facts.

The whole system, in one animation

Three acts. Every later section zooms into one box.

Four agents, one conductor

One agent talks. Three are its tools. Code decides what happens next.

Safety first: the concern queue

Three problems in one breath? A priority queue, safety first:

Knowing when you're done

Fewer questions, same facts: every answer should fill as many fields as it can.

Memory is a cache with manners

Memory is layered caching, nearest first:

One turn, against the clock

Same work, three schedules:

async def handle_turn(session, message):
    message = guardrails.check(redact_pii(message))      # nothing raw reaches a model
    state = await sessions.load(session.id)              # DynamoDB: the source of truth
    if state.already_answered(message.turn_id):          # a retried or duplicate turn
        return state.last_reply
    concern = state.queue.current()                      # highest-priority open concern
    fields, history, dealer = await gather_with_deadline(
        rubric.extract(concern, message),                # what did that answer tell us?
        vehicle_history.lookup(state.vin),               # past repairs, cached per VIN
        dealership.check(state.vin, concern),            # warranty, parts, advisories
        deadline=2.0)                                    # a slow tool is skipped, not waited on
    concern.merge(fields)
    reply = await conversation.next_turn(concern, history, dealer)   # streams to the browser
    await sessions.save(state, turn_id=message.turn_id)  # saved every turn: resumable
    return reply

Monday, 9 am

Links open in waves; the model has a quota. Drag the wave:

def admit(call, bucket, queue):
    if bucket.take(call.tokens):          # under quota: go now
        return run(call)
    if call.session.is_active:            # mid-conversation beats new arrivals
        return queue.push_front(call)
    if queue.wait_seconds() < 20:         # a short wait is better than an error
        return queue.push(call)
    return offer_callback(call.session)   # shed gracefully: "we'll text you when it's quiet"

A token bucket, a queue that favours live chats, and graceful load shedding.

When the model says no

Design as if the model will be slow, wrong or gone. Break something:

Every failure lands on one ladder:

Sessions live in a DynamoDB global table; hand-offs retry in Step Functions.

Quality is a metric too

Fast and error-free isn't enough. Ship a bad prompt:

Every turn is a trace in AgentCore observability and CloudWatch.

Ship prompts like code

Guardrails both ways

Ten decisions, one page

My calls for this recreation, with what each one costs.

1. How do the agents work together?
   group chatone big agentone talker, three tools

   Every turn stays explainable. Cost: the talker must stay fast.

2. Who decides what happens next?
   the modela state machine

   “Done” is a fact, not an opinion. Cost: new flows need code.

3. How do messages travel?
   pollingserver-sent eventsWebSocket

   Two-way, streaming, managed. Cost: reconnects to handle.

4. Where does session state live?
   in memorya relational databaseDynamoDB, saved every turn

   Resumable, and global for failover. Cost: a write per turn.

5. How do tools run?
   one after anotherin parallel, with a deadline

   The turn waits for the slowest, capped. Cost: sometimes no history.

6. What does the model remember?
   the full transcriptfacts + recent turns, tiered

   Small prompts are fast and cheap. Cost: extraction must be right.

7. What happens at the quota?
   let it throttlebuy more quotatoken bucket, queue, shed

   Live chats never fail for new ones. Cost: some wait, or get a text.

8. What if the model is down?
   show an errorwait it outthe failover ladder

   The last rung needs no model. Cost: plainer questions.

9. How does a summary reach the dealership?
   call it inlineStep Functions + retries + DLQ

   A slow dealer system never blocks a customer. Cost: delivery is async.

10. How do prompts ship?
    edit in placemanual spot checksgolden set, judge, canary, alias

    Regressions stop before customers see them. Cost: a golden set to curate.

The well-earned drink

Every Brew Miles trip ends with one:

1. Let models talk and let code decide; the most important agent is a state machine.
2. Fan out with deadlines, stream the reply, and save every turn by its ID.
3. Treat quota like capacity: a token bucket, a queue, and a graceful "later".
4. Build a failover ladder whose last rung needs no model.
5. Put quality on the dashboard, and ship prompts like code.