Why we publish a benchmark
"More accurate than Otter" is a slogan. A reproducible fixture and an open test script is data. This page is updated monthly, and all fixtures and scripts are downloadable and re-runnable from github.com/crossmeet/benchmarks (repo bootstrapping).
Current status: fixtures and tooling are being productionized. Cells marked TBD · in progress below are our honest signal that the numbers aren't ready yet — we'd rather ship empty cells than fabricate flattering ones. When complete, the PR diff will show exactly how each number was produced.
Test setup
| Item | Detail |
|---|---|
| Fixtures | BBC / NPR / TED public speeches + redacted real-world cross-border meeting recordings · ~30 minutes mixed content |
| Metrics | WER (Word Error Rate, lower is better) · BLEU (translation quality, higher is better) · First-token latency (ASR → on-screen) |
| Competitors | CrossMeet (multiple engine combinations) · Otter.ai Pro · JotMe Pro · OpenAI Whisper API (raw) · Aliyun Qwen3-ASR |
| Test rig | Windows 11 · RTX 4070 12GB · 100Mbps network |
| Cadence | Monthly |
EN → ZH simul-interpret
| Tool | WER ↓ | BLEU ↑ | First-token latency ↓ |
|---|---|---|---|
| CrossMeet · Whisper-faster + Claude 3.7 | TBD · in progress | TBD · in progress | ~100ms |
| CrossMeet · Qwen3-ASR + GPT-4 | TBD · in progress | TBD · in progress | TBD |
| Otter.ai Pro | N/A (transcription only, no translation) | N/A | TBD |
| JotMe Pro | TBD | TBD | TBD |
| Whisper API (raw, no post-processing) | TBD | N/A | TBD |
ZH → EN simul-interpret
| Tool | WER ↓ | BLEU ↑ | First-token latency ↓ |
|---|---|---|---|
| CrossMeet · Aliyun ASR + Claude 3.7 | TBD · in progress | TBD · in progress | TBD |
| CrossMeet · Qwen3-ASR + GPT-4 | TBD · in progress | TBD · in progress | TBD |
| Otter.ai Pro | × Does not support ZH transcription | × | × |
| JotMe Pro | TBD | TBD | TBD |
Common language-pair coverage
Sources: vendor public documentation · May 2026 · facts only.
| Pair | CrossMeet | Otter | JotMe | Wordly |
|---|---|---|---|---|
| EN ↔ ZH | √ | × | √ | √ |
| EN ↔ JA | √ | × | √ | √ |
| EN ↔ KO | √ | × | √ | √ |
| EN ↔ DE | √ | × | √ | √ |
| EN ↔ FR | √ | √ | √ | √ |
| EN ↔ ES | √ | √ | √ | √ |
| EN ↔ RU | √ | × | × | √ |
| EN ↔ AR | √ | × | × | √ |
| EN ↔ PT | √ | × | √ | √ |
| EN ↔ IT | √ | × | √ | √ |
Methodology · adversarial validation
"You wrote the test that you win — how is that fair?"
Fair point. The fixtures and scripts are all open on GitHub, so anyone can re-run them. We also welcome competitors to submit PRs adding their own test data — real numbers can survive scrutiny; fabricated ones can't.
Every monthly release ships as a PR diff so the audit trail is human-readable.
Limitations
- Current fixtures skew towards business conversation. Literary translation, poetry, and legal text are not yet covered.
- Windows only. macOS / Linux runners coming later.
- We don't test extremely under-powered hardware (old CPUs, no GPU) — local engines slow down significantly without GPU acceleration.
- We don't test extremely poor networks — cloud engines depend on network quality.
GitHub repository
- Test scripts and fixtures: github.com/crossmeet/benchmarks (placeholder · repo bootstrapping)
- Monthly reports are archived as markdown at
reports/<YYYY-MM>.md. Watch the repo to subscribe.
If you have suggested metrics, language pairs you'd like covered, or want to contribute fixtures, contact research@crossmeet.com.