How Do You Test a Phone Call?

You make one. Every hour.

Testing a web app is a solved problem. Render the page, click the button, assert the thing. Testing a voice agent, though, is a different story. One call passes through STT, an LLM, TTS, and a real telecom network, live, in sequence. Any link in that chain can quietly break, and the product becomes an awkward silence on a phone line.

Unit tests don’t catch a carrier misbehaving in the middle of the night. Status pages don’t either; every dependency can be “up” while calls are broken end to end.

So we built Canary!

Coal miners kept canaries to notice bad air before it noticed them. Ours makes phone calls. Every hour, a robot caller dials our production stack, over real phone lines, and checks that everything works the way a real caller would experience it.

We didn’t start by “covering everything”. We started from our clients instead. The first canaries were built around real client flows, the same journeys their callers take every day. If those work, the platform works. From there, Canary grew to cover the platform’s most vital paths. And it doesn’t just check that a call happened: it checks the whole journey, that the agent spoke, the conversation went where it should, and the data landed where it was supposed to, right down to the record in a client’s CRM.

The same machinery scales up, too. Because Canary already knows how to be a caller, it can just as easily be many callers at once. That turns the pipeline into our load-testing rig: waves of concurrent calls against the platform, so we find any capacity limits in a drill we run, not on the day a client launches a big campaign.

And when something breaks, we don’t find out from a client. Canary alerts us and someone gets paged before anyone else notices.

Voice AI demos are easy. What our clients get, though, is the guarantee that the call actually gets answered, correctly, at 4am on a Friday. Canary is how we know. It sings every hour, and the whole point is that nobody thinks about it until it stops.

One check, moment by moment

Why this bird exists

1. It is three in the morning. A Sarj voice agent is still answering calls, and nobody is watching it work.
2. So Sarj keeps a tester. Meet Canary: a small program that calls the live service on a schedule and reports what it finds.
3. The Sarj platform is a neighbourhood of workspaces. Each customer’s agents, scenarios and phone numbers live behind their own walls.
4. Canary has a workspace of its own. It holds the test scenarios and the settings that decide what each run does.
5. A workspace is a boundary for ownership, not a place a call passes through. On the Sarj platform we call it an organization.
6. The workspace keeps four checks: an inbound call, an outbound call, a batch of one call, and a Zoho record update. One run does all four.
7. Each check ends as one small piece of evidence, collected here at the inspection desk. Time to follow the first check.

Follow a telephone check

1. Two routes leave the workspace. The first is the telephone route: the outbound check places a real call, over a real phone line, to a test number.
2. Canary asks the Sarj platform to place the call, exactly the way a customer’s outbound call begins.
3. The call leaves through the telephone connection, called a trunk, and out across the ordinary phone network. The test line rings.
4. The test line answers. It exists only to be called: it talks back for about ninety seconds, then hangs up.
5. Canary is not listening to the audio. Every few seconds it asks the platform for the call record: ringing, in progress, still waiting.
6. The record says completed. Canary checks two things: the call really ended as completed, and it lasted longer than zero seconds.
7. When the run pinned a trunk and a from-number, Canary also checks the record shows that exact route was used, not some other line.
8. Canary carries the result back to the desk: one card, telephone call completed, with the reason attached whenever it fails.
9. The card makes one narrow claim: a call can still leave Sarj and finish. It says nothing about what either side heard.

Follow a different route

1. Back at the junction. The second route skips the telephone network completely: the inbound check goes straight to the agent.
2. Canary plays a recorded caller and joins the agent’s audio room directly. No phone line, no trunk, no ringing.
3. First, a deliberate pause. That quiet is the agent’s cue to deliver its opening line.
4. The agent speaks. Canary sees a finished piece of transcript arrive from the other side of the room. That is signal one.
5. A second later, Canary plays a short prerecorded clip, the way a caller would say hello and ask a question.
6. Inside the agent, speech becomes text, a language model chooses the next line, and that line becomes speech again.
7. Then a second piece of transcript arrives, inside the check’s time limit: the agent answered. Signal two. The check passes.
8. That is the whole claim: the agent spoke, then answered. Not whether the words made sense, not how the audio sounded.
9. One more scenario uses this same direct route. This recorded caller asks the agent to update a customer record in Zoho.
10. The conversation ends normally. But Canary still has one more thing to check, and it is not in this room.
11. The Zoho check is supposed to update a dedicated test record in Zoho CRM: a lead called Canary Tester.
12. Canary reads the Zoho record back. Its last-modified time is 09:09, after this run started at 09:07, allowing two minutes of clock drift.
13. Fresh enough: the update landed in Zoho. This proves the record changed after the call. It does not prove every field is correct.

Why the checks matter

1. Now the same Zoho scenario again, with one difference: this time the update never reaches the record.
2. From inside the call, nothing looks wrong. The agent spoke, the agent answered, the conversation finished normally.
3. Canary reads the Zoho record back. The last-modified time is still 14:02 yesterday. It predates this run.
4. That missing update makes the check fail, even though the conversation was fine. This is the failure a happy transcript hides.
5. Every check fails in its own words: a call that ends as failed, an agent that never answers inside the limit, a record that never moves.
6. At the desk the two outcomes are easy to tell apart. Fresh timestamp: passed. Stale timestamp: failed, with the reason written on the card.
7. A failed run writes one log line marked as a canary alert. The watchtower rings: an alert named IntegrationCanaryFailed reaches the engineers on duty.
8. An hour later the next run passes. The bell goes quiet, and the row of hourly results grows by one green mark.
9. Now imagine the runs simply stop arriving. No failure line is written, so the first alert has nothing to ring about.
10. After three hours without a fully successful run, a second alert fires: IntegrationCanaryDead. Silence means repeated failures, or a monitor that stopped.

The wider system

1. Back at the desk, the other checks have returned too: the inbound response, the outbound call with its pinned route, the batch, and the Zoho record. Four cards, one report.
2. The batch check is the quiet one. It puts a single number in a list and passes when the platform works through it and that one call completes.
3. Which checks run is a setting called CANARY_TAGS, not a rule in the code. A green run proves the checks that were switched on, nothing about the rest.
4. One bird asks whether the service works. Load testing asks how many callers it can hold at once, through a separate door: POST /v1/load/start.
5. Raise the slider to release callers. The request names a target, a ramp and a plateau: callers join five a second by default, then hold the line.
6. Every caller is a task tagged with the run id, readable afterwards. This is never the hourly batch check, which calls one number once.
7. Step back. A workspace with test scenarios, two routes, a desk of narrow claims, a record that must move, and a tower that listens for silence.
8. Canary flies home. The next check runs at seven minutes past the hour, whether anyone is watching or not.
9. Dawn over the night shift. Canary was built by the Sarj engineering team, who would rather be woken by a bird than by a customer.

Glossary