Which TTS system do listeners prefer?
Paid US-based listeners compare matching audio from your candidate systems without seeing system names. Get preference rates, confidence intervals and individual judgements through the API.
Get an exact price quote before starting a study.
How it works
Audio in. Preference evidence out.
One API call creates and prices a study. Start it when the quote looks right; results arrive when the listeners are done, usually within hours to a couple of days.
The same text rendered by system A and system B, as URLs. Label the sides; raters never see the labels.
Each pair is heard in full by several paid listeners, sides shuffled, exposure capped per person.
Win rate with a 95% interval, per-pair counts, and a CSV of every judgement for your own analysis.
The API
Two calls you will actually make.
curl -sX POST https://api.woodstitch.com/v1/evals \
-H "Authorization: Bearer $WOODSTITCH_KEY" \
-H "Content-Type: application/json" \
-d '{
"template": "tts_preference.v1",
"ratings_per_pair": 5,
"pairs": [
{"id": "p001", "passage": "p001",
"a": "https://your-bucket/cartesia/p001.wav",
"b": "https://your-bucket/eleven/p001.wav",
"system_a": "cartesia", "system_b": "elevenlabs"}
],
"dry_run": true
}'
# → {"quote": {"total_usd": 1.50}, "pair_seconds": [...], "warnings": [], "nothing_charged_yet": true}
Remove dry_run to create the study, then POST /v1/evals/{id}/start
with a spending ceiling. Signed or expiring URLs are fine: clips are fetched once, during this call.
curl -s https://api.woodstitch.com/v1/evals/7/results \
-H "Authorization: Bearer $WOODSTITCH_KEY"
{
"metrics": {
"human_pref/b_win_rate": 0.612,
"human_pref/b_win_rate_ci95_low": 0.551,
"human_pref/b_win_rate_ci95_high": 0.671,
"human_pref/tie_rate": 0.084,
"human_pref/pairs": 200,
"human_pref/judgements": 1000
},
"counts": {"a_wins": 346, "b_wins": 570, "ties": 84},
"per_pair": [{"id": "p001", "a_wins": 1, "b_wins": 4, "ties": 0}, ...]
}
Illustrative numbers for a 200-pair study. b_win_rate is the share of
preference for side b, ties counted half each way; the interval is bootstrapped
over pairs. /export?format=csv gives one row per judgement.
Full reference →
Method
Inspect the method before you trust the number.
Public research and customer studies run on the same platform under the same published rules.
No system names. Clip order is shuffled per listener, and a judgement cannot be submitted until both clips have played to the end.
Paid per second of audio heard. Location is checked by IP, connection type and device timezone on every visit.
Per-listener caps on how often anyone hears a system, and no-repeat rules per passage.
Every estimate carries a 95% interval. One that spans 0.5 does not establish a preference, and the result says so.
Limitations are written down too, including that there are no attention-check trials yet. See the platform methodology.
Proof
We publish our own listening studies.
Same platform, same listeners, same rules, in public.
Can listeners distinguish AI narration from human narration?
That report shows how we run a study, not how your system will score. We also build a narration product and disclose it on every report. Customer studies are never published and never enter our public research.
Questions
What engineers ask first.
Participating listeners, inside the study, with no system name, after accepting a published rater agreement. Clips are never published. Need a written retention commitment? Email us before you start.
English-speaking, US-based people paid per second of audio heard. Other languages or regions: email us and we will say plainly whether we can staff it.
No 1–5 MOS template. Paired preference answers which system people prefer. A separate detection template asks whether a clip passes as human; it is in the docs.
Try it on your own audio.
Run a small pilot on your own clips before a larger study. Need help designing the comparison? support@woodstitch.com