Why human testing and LLM evals aren’t enough.
Evaluating the brain doesn’t test the mouth or the ears. You need full-pipe telephony testing.
Manual human QA
- InconsistentHumans can't replicate the exact same interruption timing twice.
- UnscalableYou cannot run 1,000 concurrent human calls to test load.
- ExpensivePaying people to dial phone numbers is a waste of engineering budget.
LLM-to-LLM text evals
- Ignores latencyText evals don't measure text-to-speech delay.
- Misses interruptionsEndpointing failures only happen over audio.
- No PSTN realitiesIgnores packet loss, background noise, and bad reception.
Vera telephony testing
- Full-pipeTests STT, the LLM, and TTS over actual SIP/PSTN networks.
- ProgrammableDefine the dialect, interruption frequency, and mood of the caller.
- ScalableHit your IVR with 10 or 10,000 calls simultaneously.