benchcall reads your Vapi or Retell agent's own config, generates an adversarial test suite tailored to it, runs simulated callers against it live, and judges every transcript with quoted evidence. It all runs on your API keys, and transcripts never leave your machine.
↑ real result: a major platform's own template agent, tested unmodified
Get started free → How it works Prompt leaderboardbenchcall reads your agent's prompt, tools, and flow, then writes the test suite a paranoid QA engineer would: invented-price probes, identity leaks, prompt injection, impossible dates, angry callers.
LLM callers with personas run every scenario against your live agent. A judge grades each explicit criterion pass/fail with a quote from the transcript, never a vibes score. You grade the judge, too.
Change a prompt, get a diff of exactly which behaviors broke, with alerts when it happens overnight. Failed tests become suggested prompt patches you review before applying.
Client-ready evidence reports and a live "tested" badge, white-labeled with your agency's name. Show clients their agent works, every month, automatically.
git clone https://github.com/Sammsamy/benchcall cd benchcall && npm install && npm run build cp .env.example .env # add ONE LLM key (OpenRouter works) node runner/dist/cli.js run \ --config examples/sample-agent.json \ --suite examples/sample-suite.json
The runner is MIT open source. Suite generation tailored to your agent runs on our hosted service, free tier, bring your own LLM key, we never mark up inference.
I'm Fuzlullah Syed, but everyone calls me Fuz. I'm a third year medical student, and I built benchcall with my brother, a team of two. Clinics like the ones I train in are starting to hand their phones to AI agents, and someone should be testing them. Voice is where we started. The engine underneath is built to benchmark any kind of agent, and that's where we're headed.