Can listeners distinguish AI narration from human narration?
A blinded listening study of ElevenLabs v4 and human-read audiobook excerpts. Interim report: 176 of 1,050 planned judgments.
Question Can paid, untrained listeners tell a current commercial text-to-speech system from a human narrator, and does a longer excerpt help?
Study 35 listeners made 176 blinded, forced-choice judgments (human or AI) on audiobook excerpts of 15, 30 and 60 seconds read by six ElevenLabs v4 voices and ten human narrators, then rated their confidence. The human recordings were level- and spectrally adjusted toward the AI output before use (§4.2). This is an interim snapshot of a study planned at 1,050 judgments.
Findings Accuracy was 48.9% (95% CI 42.2–56.1% resampling listeners; 34.5–63.2% resampling listeners and speakers). We did not establish above-chance classification. Accuracy at 60 s was 0.2 points lower than at 15 s (CI −17 to +15): no clip-length effect was detected, and a moderate one cannot be ruled out. The “very confident” category had the lowest observed accuracy (16 of 48, 33.3%); evidence for an overall confidence–accuracy relationship was inconclusive. The AI and human stimulus sets differed acoustically, most on pause-length variability; measured pause features had little marginal association with listeners’ answers, and pitch range a small positive one. Why is untested.
Scope One system, calm nonfiction prose, volunteer (not professional) narrators as the human baseline, processed human recordings, untrained listeners hearing each clip once with no reference, inference conditional on this stimulus set. The acoustic analysis is exploratory and post hoc.
Key findings
- Listeners classified 86 of 176 judgments correctly (48.9%). Above-chance classification was not established.
- Accuracy did not differ detectably between 15 s and 60 s excerpts (−0.2 points; CI −17 to +15).
- “Very confident” judgments were correct 16 of 48 times (33.3%): the lowest of the four confidence levels. The listener-resampled interval (20.0–46.9%) excludes 50%; the listener-and-speaker interval (12.2–60.3%) does not.
- Three of six AI voices were called human at least as often as most human narrators; one narrator was called AI on all eight of her trials. Per-speaker counts are small.
- Acoustically, the AI clips pause longer and more variably than the (processed) human clips and use a wider pitch range. Listeners’ answers were weakly associated with pitch range and essentially unassociated with the pause measures.
1What we tested
Audiobook producers, platforms and listeners increasingly ask whether synthetic narration is detectable by ear. Most public claims are anecdotal, or rest on older systems. We asked a narrower, answerable question: when an ordinary listener hears a single excerpt from a current commercial system, blind and without a reference, can they say whether it was read by a person — and does hearing more of it help?
We tested one system, ElevenLabs v4,1 against human narrators from the Hi-Fi TTS corpus,2 on ten passages of public-domain nonfiction prose. Each recording was cut at 15, 30 and 60 seconds so that clip length could be examined within the same material. Listeners also rated their confidence, which lets us ask whether people know when they are guessing. Finally, we measured each clip acoustically to ask two separate questions: what objectively distinguishes the two sets, and what listeners were responding to when they said “human”.
2Results
2.1 Overall accuracy was near chance
Listeners classified 86 of 176 judgments correctly (48.9%). Resampling listeners, the 95% interval is 42.2–56.1%; resampling listeners and speakers independently, it is 34.5–63.2% (§4.4). Both include 50%. The result is symmetric across ground truth — 48.5% on AI clips and 49.4% on human clips (Table 1) — so it is not explained by a lean toward one answer: listeners answered “human” on 50.6% of judgments against a true human share of 44.9%. Sensitivity measured as d′3 was −0.06 (listener-resampled CI −0.39 to 0.31; listener-and-speaker −0.89 to 0.70), where zero denotes no discrimination. No listener who completed six or more trials exceeded 5 correct of 10. We did not establish above-chance classification on these stimuli.
Table 1. Classification accuracy by ground truth. Intervals are 95% bootstrap intervals resampling listeners (5,000 replicates), conditional on this stimulus set; Wilson intervals on the raw counts are within 0.5 pp.
| Clip was | Trials | Correct | Accuracy | 95% CI |
|---|---|---|---|---|
| AI (ElevenLabs v4) | 97 | 47 | 48.5% | 38.3 – 59.4 |
| Human narrator | 79 | 39 | 49.4% | 38.9 – 59.8 |
| All trials | 176 | 86 | 48.9% | 42.2 – 56.1 |
2.2 We did not detect an effect of clip length
Accuracy was 46.8% at 15 seconds, 53.6% at 30 seconds and 46.6% at 60 seconds (Table 2; Figure 1). The 60-second condition gave listeners four times the audio of the 15-second condition; the estimated difference is −0.2 percentage points with a listener-resampled 95% interval of −17.3 to +15.1. The data do not detect a clip-length effect and cannot rule out a moderate one in either direction. The 30-second value lies inside its own interval and does not indicate a trend. These intervals are conditional on the selected speakers and passages.
Table 2. Accuracy by clip length. Each listener heard a mix of lengths; no listener heard the same passage or the same speaker twice.
| Clip length | Trials | Correct | Accuracy | 95% CI |
|---|---|---|---|---|
| 15 seconds | 62 | 29 | 46.8% | 36.7 – 58.5 |
| 30 seconds | 56 | 30 | 53.6% | 39.7 – 69.4 |
| 60 seconds | 58 | 27 | 46.6% | 34.7 – 58.1 |
2.3 The “very confident” category had the lowest observed accuracy
After each verdict, listeners chose one of four confidence labels. Judgments rated “very confident” were correct 16 of 48 times (33.3%; listener-resampled CI 20.0–46.9%, which excludes 50%; listener-and-speaker CI 12.2–60.3%, which does not). The pattern across the four levels is not monotone (Table 3): “just guessing” scored highest, the two middle levels were near 50%, and “very confident” lowest. Pooling the two lower and two upper levels gives 55.3% versus 44.0%, a difference of 11.3 points whose interval (−3.2 to +27.0) includes zero. Evidence for an overall confidence–accuracy relationship is therefore inconclusive; the very-confident result is reported as a subgroup finding. Of the 32 very-confident errors, 20 were an AI clip called human.
Table 3. Accuracy by stated confidence. The four labels are those shown to listeners.
| Stated confidence | Trials | Correct | Accuracy | 95% CI | Answered “human” |
|---|---|---|---|---|---|
| Just guessing | 34 | 21 | 61.8% | 41.4 – 81.1 | 50% |
| Slightly confident | 42 | 21 | 50.0% | 35.8 – 65.6 | 40% |
| Confident | 52 | 28 | 53.8% | 44.0 – 63.6 | 48% |
| Very confident | 48 | 16 | 33.3% | 20.0 – 46.9 | 62% |
Figure 1. Accuracy by condition. Points are observed proportions; bars are 95% intervals from a bootstrap that resamples listeners, conditional on this stimulus set. The dashed line marks chance. The only condition whose listener-resampled interval excludes 50% is “very confident”; its listener-and-speaker interval (12.2–60.3%) does not.
2.4 Some voices were classified as human more often than others
Table 4 lists all sixteen speakers by the share of trials on which listeners answered “human”, regardless of the truth. Three AI voices (Eric, Matilda, Chris) were called human at least as often as most of the narrators, and one narrator (speaker 12787) was called AI on all eight of her trials. Trials per speaker range from 5 to 22 and every interval overlaps several others; the table describes the sample and should not be read as a ranking of voices.
Table 4. Share of trials on which each speaker was judged human. Narrator IDs are Hi-Fi TTS speaker IDs. Wilson 95% intervals.
| Speaker | Trials | Judged human | 95% CI |
|---|---|---|---|
| AIEric | 12 | 83% | 55 – 95 |
| HUMANNarrator 9136 | 6 | 83% | 44 – 97 |
| HUMANNarrator 11614 | 11 | 82% | 52 – 95 |
| HUMANNarrator 6670 | 5 | 80% | 38 – 96 |
| AIMatilda | 10 | 70% | 40 – 89 |
| HUMANNarrator 6671 | 6 | 67% | 30 – 90 |
| HUMANNarrator 8051 | 8 | 62% | 31 – 86 |
| AIChris | 14 | 57% | 33 – 79 |
| AIGeorge | 22 | 50% | 31 – 69 |
| AIAlice | 19 | 42% | 23 – 64 |
| HUMANNarrator 9017 | 10 | 40% | 17 – 69 |
| HUMANNarrator 92 | 8 | 38% | 14 – 69 |
| HUMANNarrator 6097 | 8 | 38% | 14 – 69 |
| AIBella | 20 | 30% | 15 – 52 |
| HUMANNarrator 11697 | 9 | 22% | 6 – 55 |
| HUMANNarrator 12787 | 8 | 0% | 0 – 32 |
2.5 Exploratory acoustic comparison of the stimulus sets
We extracted eleven acoustic measures from every clip as served (§4.5) and asked two descriptive questions: on what dimensions do the AI clips differ from the human clips, and which measures are associated with a listener answering “human”? Both analyses were specified after the behavioural results were known and are exploratory. They describe these stimulus sets, not an intrinsic property of AI versus human speech: each narrator read only one passage, so speaker and passage are confounded on the human side, and the pause detector is energy-based rather than linguistic. The spectral adjustment applied to the human recordings (§4.2) does not account for the contrasts: recomputing every measure on the unprocessed source recordings gives the same values to within 0.01 (human pause s.d. 0.045 s before and after; AI 0.142 s; pitch range 10.46 versus 10.45 semitones).
What differs. The two sets differ most on pause structure (Table 5). The AI clips pause for longer (mean 0.47 s versus 0.32 s), their longest pauses are longer (0.70 s versus 0.40 s), and their pause lengths vary far more (standard deviation 0.14 s versus 0.04 s; standardised difference d = 3.85 on speaker means). The AI clips also use a wider pitch range (15.4 versus 10.5 semitones). Median pitch, loudness and dynamic range do not differ. On these measures the AI output is more varied and more deliberate than these narrators, not flatter. A different human baseline — professional narrators, or the same readers in different material — could narrow or reverse several rows.
Table 5. Acoustic measures on the clips as served, speaker-level means (6 AI voices, 10 narrators; 210 clips). d is Cohen’s d on the 16 speaker means; rows in bold have Welch t > 2.1. Descriptive only.
| Measure | AI voices | Human narrators | d |
|---|---|---|---|
| Pause variability (s.d. of pause length, s) | 0.14 | 0.04 | 3.85 |
| Longest pause (s) | 0.70 | 0.40 | 2.99 |
| Mean pause (s) | 0.47 | 0.32 | 2.70 |
| Pitch range, 5th–95th percentile (semitones) | 15.4 | 10.5 | 2.00 |
| Pitch variability (s.d., semitones) | 4.6 | 3.3 | 1.97 |
| Speech fraction (time not pausing) | 0.82 | 0.88 | −1.78 |
| Pauses per minute | 14.7 | 10.0 | 0.91 |
| Speaking rate (syllable nuclei / s) | 4.90 | 4.64 | 0.44 |
| Dynamic range (dB) | 20.9 | 20.6 | 0.14 |
| Median pitch (Hz) | 171 | 168 | 0.06 |
| Loudness (dBFS) | −22.9 | −22.5 | −0.33 |
Association with listeners’ answers. Table 6 relates the same measures to the listener’s answer across the 176 judgments. All associations are small. Listeners answered “human” somewhat more often for clips with a wider pitch range (r = 0.15; 60% above the median versus 41% below), and slightly more often for slower, lower-pitched and quieter clips. The pause measures, which separate the two sets most strongly, showed essentially no marginal association (r = 0.02–0.03). The direction of the pitch-range association was similar within the AI clips and within the human clips. These are marginal correlations on non-independent judgments; they do not establish which cues listeners used, and we draw no causal conclusion from them.
Table 6. Association between each acoustic measure and answering “human” (176 judgments). r is the point-biserial correlation; “below / above median” is the share answered human in each half. Judgments are not independent (listeners, speakers and passages recur) and eleven measures were examined, so no significance threshold is claimed.
| Measure | r | Below median | Above median |
|---|---|---|---|
| Pitch range | +0.15 | 41% | 60% |
| Loudness | −0.15 | 55% | 47% |
| Pitch variability | +0.14 | 41% | 60% |
| Speaking rate | −0.12 | 55% | 47% |
| Median pitch | −0.12 | 57% | 44% |
| Pauses per minute | +0.09 | 47% | 55% |
| Pause variability | +0.03 | 51% | 51% |
| Mean pause | +0.03 | 53% | 48% |
| Longest pause | +0.02 | 51% | 51% |
3Interpretation and limitations
The defensible central finding is narrow: on these stimuli, these listeners did not classify above chance, more audio did not detectably help, and stated confidence was a poor guide to accuracy. The acoustic comparison adds a descriptive observation — the sets differ most on pause rhythm, and listeners’ answers were not associated with it — but not an explanation.
One hypothesis consistent with the pattern, which this study did not test, is that listeners apply a prior about synthetic voices (flat, quick, bright) in reverse, taking range and an unhurried pace as evidence of a person; since the AI clips here have the wider pitch range, such a prior would point the wrong way, and would produce confident errors of exactly the kind observed (AI called human). Alternatives are equally open: pause variability may not be perceptible in a single short hearing; listeners may be responding to features we did not measure; or the associations may reflect particular speakers rather than any general cue. Distinguishing these would need a design that manipulates cues directly.
What this study does and does not show
It shows that in this sample — one system, calm nonfiction, a single unreferenced hearing, processed human recordings — paid untrained listeners did not classify above chance, that more audio did not detectably help, and that the most confident judgments were the least accurate.
It does not show that synthetic speech is undetectable in general. Trained listeners, repeated exposure to one voice, a human reference clip, emotionally demanding text, or an automated detector working on spectral detail could all change the outcome, and we did not test any of them. The acoustic difference in Table 5 is a difference between two processed stimulus sets; it does not establish an intrinsic AI–human difference, nor a cue listeners could use with training. The intervals are not adjusted for the fact that this is an interim snapshot chosen during data collection.
Specific limitations:
- Interim snapshot. The study is planned at 1,050 judgments (210 clips × 5). This report uses the 176 available when we began writing; the headline estimate had been inspected once before, at 174 judgments the same day. The intervals are not adjusted for data-dependent inspection. A versioned update will be issued at the planned sample size whatever it shows.
- Single system. All synthetic clips are ElevenLabs v4 with default settings. Results do not transfer to other systems or versions.
- Human baseline. The narrators are LibriVox volunteers via Hi-Fi TTS, selected for recording quality, not professional audiobook narrators. Professional delivery typically has more pausing and wider range, which would likely reduce several contrasts in Table 5 and might change detectability in either direction.
- Design imbalance. 180 AI clips versus 30 human; six AI voices versus ten narrators; 5–22 trials per speaker. Per-speaker estimates are imprecise.
- Repeated measures and generalisation. Listeners contributed up to 10 judgments each, and each speaker and passage recurs across judgments. Listener-resampled intervals are conditional on this stimulus set. Intervals that also resample speakers are reported for the main estimates and are much wider (§4.4); with 16 speakers and 10 passages, intervals that resample passages as well are close to uninformative and we do not rely on them.
- Exploratory analysis. The acoustic analyses were not prespecified and were run after the behavioural results were known. Eleven measures were examined on non-independent judgments; no significance claim is made. Features come from our own extraction pipeline (§4.5), not a validated phonetic toolkit, and the pause detector is energy-based. The human recordings were spectrally adjusted before measurement (§4.2).
- Material. One genre, ten passages. Dialogue, verse and emotionally loaded prose are where synthetic narration is most often reported to fail and were not tested.
- Engagement. There were no catch trials. Interaction signals were captured for 91 of the 176 judgments (§4.3); they are consistent with the audio having been heard, and are not extrapolated to the rest.
4Methods
4.1 Stimuli
Ten passages of public-domain nonfiction prose were selected from the Hi-Fi TTS corpus2 (NVIDIA; derived from LibriVox), each read by one of ten narrators, giving ten human source recordings of about 60 seconds. The same ten passages were synthesised with six ElevenLabs v4 voices (Alice, Bella, Chris, Eric, George, Matilda) using the vendor’s default voice settings, giving 60 AI source recordings. Every source recording was cut at 15, 30 and 60 seconds from its start, for 210 clips (180 AI, 30 human). Passages were chosen to be calm expository prose without humour, grief or technical vocabulary.
4.2 Levelling and spectral adjustment
All clips were rendered as 44.1 kHz mono 16-bit PCM. A single gain adjustment of +0.34 dB was applied to every clip; this sets the level of the set as a whole and leaves between-clip level differences unchanged (the AI and human clips differ by 0.4 dB on average, Table 5). On the earlier clip set, a classifier using only long-term spectral tilt separated human from AI recordings at AUC 0.815 — a microphone and recording-chain difference, not a property of the voices. To remove it, a straight-line tilt equalisation in log-frequency (−0.11 dB per decade, capped at ±4 dB) was fitted on the 60-second clips and applied to the human recordings only; the AI clips were not altered. After adjustment the same tilt classifier, evaluated on the same clips (not held out), scored AUC 0.51. Listeners therefore heard spectrally adjusted human recordings against unprocessed vendor output. Other spectral or temporal recording-chain differences were not tested for and may remain. Source recordings were screened for background music and for spoken LibriVox preambles.
4.3 Listeners and task
Recruitment, payment, blinding and assignment follow the Woodstitch platform methodology; only what is specific to this study is given here. 35 raters on the Woodstitch platform participated between 30 October and 3 November 2026 (data snapshot 3 November 2026, 23:30 UTC), contributing a median of 5 judgments (range 1–10). Raters are paid per second of audio heard; payment does not depend on the answer given, so there is no incentive to answer in any particular way. Each trial presented one clip with no reference and a forced choice, Human reading or AI-generated, followed by a mandatory confidence rating with the labels Just guessing, Slightly confident, Confident, Very confident. The player permits one playback per clip, with pause and resume but no restart; the instruction shown was “Play the clip once in full to answer · you can pause, but not replay”. Listeners were not told the proportion of AI clips. The assignment rules prevented any listener from hearing the same passage or the same speaker twice. Clip order within a sitting was randomised. No feedback on correctness was given. Of 182 trials served, 0 were skipped and 176 received a verdict with a confidence rating; the remaining 6 were in progress at the snapshot. Browser interaction signals were captured for 91 of the 176 judgments: in 85 of those the recorded listening time reached at least 90% of the clip, 3 reported an audio error, 6 showed the tab losing focus at some point, and mean time with the tab hidden was 1.1 s. These signals are not available for the other 85 judgments and are not extrapolated to them.
4.4 Statistical analysis
Accuracy is the proportion of judgments on which the verdict matched the ground truth. Primary 95% intervals are percentile intervals from a non-parametric bootstrap that resamples listeners with replacement (5,000 replicates),4 because judgments from one listener are not independent; these intervals are conditional on the selected speakers and passages. To indicate uncertainty from the choice of speakers, a second bootstrap resamples listeners and speakers independently (4,000 replicates), weighting each judgment by the product of its listener’s and speaker’s resampled multiplicities; replicates in which a subgroup is empty are dropped, and d′ rates are clipped to [0.001, 0.999] before the normal inverse. Independent resampling of crossed clusters is not guaranteed to be calibrated, especially with sparse cells and with speaker confounded with passage on the human side, so these wider intervals are reported as indicative rather than exact. Wilson score intervals5 on raw counts are reported for per-speaker rates. Sensitivity d′ was computed as z(hit rate) − z(false-alarm rate), treating “AI” as the signal (hit rate 48.5%, false-alarm rate 50.6%).3 No judgments were excluded and no listener was removed. No adjustment is made for this being an interim snapshot (§3).
4.5 Acoustic measures
Clips were downsampled to 16 kHz and analysed with librosa 0.10.6 Pitch (F0) was estimated with pYIN7 over 60–400 Hz in 10 ms frames; pitch variability and range are the standard deviation and the 5th–95th percentile span of voiced-frame F0 in semitones. Speech frames were those within 35 dB of the clip’s peak short-time RMS; pauses are runs of non-speech frames of at least 250 ms between the first and last speech frame. Speaking rate is the number of onset-strength peaks within speech frames per second of speech, a syllable-nucleus proxy. Loudness is overall RMS in dBFS; dynamic range is the 95th minus 10th percentile of frame level during speech. Group differences in Table 5 use speaker-level means to avoid treating the three cuts of one recording as independent. Associations in Table 6 are point-biserial correlations at the judgment level and are reported without significance tests. Measures were taken on the clips as served, i.e. after the adjustments in §4.2, and repeated on the unprocessed source recordings as a check; the two agree to within 0.01 on every measure.
5Data and reproducibility
The per-judgment dataset (trial, clip identifier, ground truth, clip length, verdict, confidence; listener identifiers replaced by anonymous indices), the per-clip acoustic feature table and the analysis scripts are available on request from hello@woodstitch.com. The audio clips remain in use in a running study and will be released when it closes. The task interface and assignment rules are documented in the Woodstitch API reference.
6Funding and competing interests
This study was funded by SquireWorks Audiobooks, which produces audiobooks using ElevenLabs voices among other systems, and was conducted on Woodstitch, a human-evaluation platform operated by the author. The author therefore has a commercial interest in synthetic narration being acceptable to listeners. The finding reported here is favourable to that interest. ElevenLabs had no involvement in the design, execution, analysis or reporting of the study and was not informed of it. The author declares no other competing interests.
7References
- ElevenLabs. Eleven v4 text-to-speech model. https://elevenlabs.io/ (accessed October 2026).
- Bakhturina E, Lavrukhin V, Ginsburg B, Zhang Y. Hi-Fi Multi-Speaker English TTS Dataset. Proc. Interspeech 2021. arXiv:2104.01497.
- Macmillan NA, Creelman CD. Detection Theory: A User’s Guide, 2nd ed. Lawrence Erlbaum, 2005.
- Efron B, Tibshirani RJ. An Introduction to the Bootstrap. Chapman & Hall, 1993.
- Wilson EB. Probable inference, the law of succession, and statistical inference. J. Am. Stat. Assoc. 1927;22(158):209–212.
- McFee B, Raffel C, Liang D, et al. librosa: Audio and music signal analysis in Python. Proc. SciPy 2015.
- Mauch M, Dixon S. pYIN: A fundamental frequency estimator using probabilistic threshold distributions. Proc. ICASSP 2014.
8Version history
- 1.0 (interim) — 3 November 2026; revised 4 November 2026 after external methodological review: added listener-and-speaker intervals, restated the length and confidence findings as estimates with intervals, described the human-side spectral adjustment explicitly, reframed the acoustic analysis as descriptive and the listener-cue account as an untested hypothesis, added engagement signals and the interim-snapshot status. Numerical results unchanged. A versioned update will be issued at the planned 1,050 judgments.