ReceptionBenchRESEARCH / 0.1
AN OPEN BENCHMARK IN DEVELOPMENT

Good conversation.
But did the task
actually happen?

Measure AI receptionists by the business outcomes they deliver. The right message. The right person. A connection that works.

Follow the evidence Run the local core
TRIAL / ILLUSTRATIVE
01 / CALLER GOAL

“Send me the
booking link.”

02 / CHECK THE OUTCOME

Correct link. Correct caller.

SIMULATED RECEIPT Evidence before verdict.
FIRST DOMAIN

English-language reception / Synthetic salon businesses

See what’s available
01 / ANATOMY OF AN OUTCOME

A claim is a starting point.
A receipt is evidence.

Explore three illustrative trials. Change the case to see why successful tool calls and confident replies are only part of the story.

Simulated evidence Educational examples · no vendor scores
CALLER INTENT

“Send me the link to book a haircut.”

SYSTEM SAYS

“I’ve sent that to you.”

ILLUSTRATIVE VERDICTPass

The right link reached the consenting caller.

EVIDENCE TRAILSYNTHETIC / NOT A LIVE RUN
  1. ✓
    Consent recorded

    Caller agrees to receive one booking text.

  2. ✓
    Correct link received

    Controlled message sink receives the approved booking URL.

  3. ✓
    Recipient and count match

    One message, correct synthetic caller, inside the observation window.

REQUIRED OUTCOME

Consent + correct recipient + received link + no forbidden effects

A booking-link handoff needs consent and a receipt at the caller’s controlled destination. It does not establish that an appointment was created.

02 / THE METHOD

Measure the task.
Keep the proof.

A vendor-neutral contract defines the outcome. Adapters connect systems to that contract without prescribing their internal tools.

01

Set the business context

Fresh synthetic business facts, a caller goal and an explicit policy. Expected outcomes stay with the evaluator, outside the system under test.

02

Observe the real effect

Check the recipient, state change or media path. A sent booking link is a handoff; creating an appointment requires a separate profile and booking evidence.

03

Report the whole result

Retain failures, missing evidence and unsupported tasks. Show coverage and critical violations alongside success counts. Rescore recorded evidence deterministically.

There can be several correct ways to complete a task.
The outcome matters more than the tool sequence.

03 / FIELD NOTES

Useful today.
Honest about tomorrow.

The offline core helps developers inspect and challenge the verifier. Real voice evaluation needs additional transport, outcome observers and independent operation.

REFERENCE SYSTEM DISCLOSURE

Receptionaly is the initial reference system. A second vendor adapter and independent hidden-test custody are pending. This project does not currently offer independent certification or a verified vendor leaderboard.

  1. M0–M1
    AVAILABLE LOCALLY

    Audit + offline core

    Ten synthetic cases, fake adapter, deterministic evaluator, evidence replay and reports. Simulated receipts only.

  2. M2
    IN DEVELOPMENT

    Private reference adapter

    Isolated fixture mapping, lifecycle conformance and local integration. Conformance is distinct from a scored voice-system benchmark.

  3. M3–M4
    PLANNED

    Voice transport + pilot

    Controlled counterparts, authoritative delivery and media evidence, reviewed scenarios and repeated trials.

  4. M5–M6
    PLANNED

    Independent beta + release

    Second vendor and domain, independent hidden custody, licensing clearance and reproducible public reporting.

04 / AT THE BENCH

Start with a fake system.
Learn something real.

Run the local offline core, inspect an evidence bundle and replay its verdicts. No credentials or live calls required.

FROM AN EXISTING CHECKOUT

A small, inspectable loop.

Use Node 18.20.8, npm 10.8.2 and sqlite3 on your PATH. Dependency installation may need network access; subsequent offline runs do not.

Choose a fresh output directory for each run. This demonstration evaluates the fake adapter and verifier, not a production receptionist.

LOCAL / OFFLINE SIMULATION
npm ci --ignore-scripts
npm run build
npm test
node dist/src/cli.js doctor
node dist/src/cli.js adapter check --adapter fake
node dist/src/cli.js plan --out run-plan.json
node dist/src/cli.js run --plan run-plan.json \
  --offline --out runs/demo
node dist/src/cli.js replay \
  --bundle runs/demo/bundle.json
05 / BUILD THE NEXT PIECE

Bring your system.
Question the benchmark.

We need adapter authors, scenario contributors and critical reviewers. Prepare a participation brief, then use the contribution documents in this checkout.

YOUR PARTICIPATION BRIEFLOCAL ONLY

Creates a text file in your browser. Nothing is submitted, stored by this site or sent to a server.

A good first contribution

  1. Declare the scope.What can your system do, and what evidence can establish it?
  2. Keep fixtures synthetic.No customer records, private prompts or proprietary source in a contribution.
  3. Challenge a verdict.A minimal false-pass or missing-evidence example makes the verifier stronger.
Help shape the benchmark.

Open an issue with your proposed adapter, evidence case or participation brief. The contribution guide explains synthetic data, evidence and review requirements. Downloading a brief does not submit it or register your system.

Read the contribution guide ↗

Download the general brief