How Woodstitch listening studies are run

The procedures shared by every study we publish: who the listeners are and how they are paid, how clips are blinded and assigned, how audio is prepared, what statistics we report by default, and the rules for what gets published. Each study report describes only what is specific to that study and refers here for the rest.

1. 1What Woodstitch is
2. 2Listeners
3. 3Tasks
4. 4Blinding and assignment
5. 5Audio preparation
6. 6Statistics
7. 7Publication rules
8. 8Known limitations of the platform
9. 9Changes to this document

1What Woodstitch is

Woodstitch is a human-evaluation platform for speech audio. A panel of paid listeners hears short clips in a browser and answers one question per clip or pair of clips — which of two they prefer, or whether a clip was read by a person or generated by a machine. The platform is used for two things: private evaluations commissioned by companies building speech systems, and the public studies published at woodstitch.com/research. This document describes the second use. Private evaluations never contribute data to a public result, and the existence of a private evaluation is not disclosed.

Woodstitch is operated by the author, who also produces audiobooks commercially using synthetic voices. That interest is stated on every report, and Section 7 describes the rules that are meant to keep it from shaping what is published.

2Listeners

2.1 Recruitment and eligibility

Listeners are recruited through social media and word of mouth, and sign in with an existing Google account. We ask for one account per person and verify location at sign-in and on each visit; accounts that appear to arrive through a proxy or from outside the regions a study is open to are placed under review until a person has looked at them. We do not currently collect demographic information. Listeners agree to a published rater agreement covering payment, data use and conduct before their first task.

Panel size is modest — in the tens of active listeners at any time — and growing. Studies state how many listeners contributed and how many judgments each made.

2.2 Payment

Listeners are paid per second of audio they hear, at a fixed rate equivalent to US$12 per hour of audio. Nothing is paid for time spent deciding: the deciding interval is under the listener's control, and paying for it would reward leaving a page open. Because deciding time is unpaid, the effective rate per hour of work is lower than $12 and depends on clip length — for a 30-second clip followed by 15 seconds of deciding it is about $8 per hour of work. Payment does not depend on the answer given, and no feedback on correctness is shown, so there is no incentive to answer in any particular way; this does not by itself ensure attention (see 2.4). Earnings accrue to a balance that can be withdrawn once it reaches a threshold.

2.3 What is recorded

For each judgment we store the answer, the time from clip end to answer, and any confidence rating the task asks for. Where the browser permits, we also capture interaction signals — recorded listening time per clip, audio errors, whether the tab lost focus and for how long. These signals are not available for every judgment, and reports state their coverage. We do not record audio or video of the listener. Listener identities are replaced by anonymous indices in any data we share.

2.4 Exclusions

We do not currently run catch trials or automated attention checks. Exclusion is by review of an account rather than of an individual judgment: an account flagged by the location check, or found to be a duplicate, has its judgments withheld from analysis while it is under review and removed if the flag is upheld. No public report to date has excluded any judgment on quality grounds, and every report states its exclusions explicitly, including when there are none. We intend to add catch trials; when we do, the report for any study that uses them will say so.

3Tasks

Two task formats are in use. A study uses one of them and states which.

Table 1. Task formats.

The question shown above the clips is fixed for a study and is reproduced verbatim in its report, because small changes in wording change results. Judgments are served in sets of five on one question so that a listener is not switching between studies on every trial. Listeners may skip a trial if the audio fails to play; skips are recorded and reported.

4Blinding and assignment

4.1 Blinding

Listeners are never shown which system, voice or source produced a clip. Audio is stored under a content hash and served from an opaque address, so the page source, the network request and the file name reveal nothing about the condition. Listeners are not told the proportion of conditions in a study — for a detection study, not told how many clips are synthetic.

4.2 Side balance

In paired studies each system appears on side A in half of its pairs, or as close to half as the design permits; where a design cannot be balanced exactly, the report gives the achieved A-side counts. Reports also give the overall share of A choices, so that any position bias is visible rather than assumed away.

4.3 No-repeat rules

Each study carries assignment rules that hold across a listener's whole history on that study, not just within one sitting:

• A listener never hears the same passage twice, so a verdict cannot be informed by a previous reading of the text.
• A listener never hears the same speaker twice in a detection study, so familiarity with a voice cannot build up.
• In paired studies, a cap limits how often a listener meets the same system, so that no one hears one system often enough to start recognising it rather than judging it.
• A listener may contribute at most a fixed number of judgments to a study (typically 10–20), stated in the report.

4.4 Allocation

Each item (clip or pair) is judged by a fixed number of distinct listeners, typically three for preference and five for detection. Items are drawn for a listener at random from those with capacity remaining that the rules above allow; when several studies are running, a listener is routed to a study with probability proportional to how many judgments they still may contribute to it. Items are not retired early because of how they are scoring.

5Audio preparation

All clips in a study share one format (sample rate, bit depth, channels), stated in the report. Beyond that, how much the audio is altered depends on what the study is measuring, and the report says which of the following was done:

• Nothing. When the system's output level is itself part of what is under test — a vendor's default rendering, say — clips are served as produced, and measured loudness is reported so readers can judge whether level explains the result.
• A single shared gain offset. All clips are shifted by one common amount so that the set as a whole sits at a target level. This leaves level differences between clips unchanged, and reports give the measured levels so readers can judge whether level explains a result. We do not normalise clip by clip, because that removes between-clip level differences, which are part of what is being judged.
• Spectral adjustment. For detection studies, where the human and synthetic recordings come from different recording chains, a mild straight-line tilt equalisation may be applied to one condition so that a classifier using only long-term spectral tilt can no longer separate the conditions. When this is done the report states which condition was adjusted, how the correction was fitted, the classifier's separability before and after (area under the ROC curve) and whether it was evaluated on held-out clips. This addresses one recording-chain difference; it does not establish that no others remain, and listeners in such a study hear adjusted recordings.
• Screening. Source recordings are screened for background music, clipping and spoken preambles that would identify a source, and failing clips are replaced.

6Statistics

Reports use the following unless they state a reason not to.

• Unit of analysis. One judgment. Because listeners contribute several judgments and are not interchangeable, primary intervals come from a non-parametric bootstrap that resamples listeners with replacement (5,000 replicates). These intervals are conditional on the study's particular speakers and passages. Where a report also wants to indicate uncertainty from the choice of speakers or clips, it reports a second interval that resamples those as well, and says that independent resampling of crossed clusters is indicative rather than exactly calibrated. Reports state which inference each interval supports.
• Proportions. Wilson score intervals on raw counts are given where a group is too small to bootstrap, and are reported alongside bootstrap intervals for the main results so readers can see they agree.
• Paired preference with several systems. A Bradley–Terry model is fitted to decisive judgments (ties excluded and their count reported), giving each system a strength on a log scale centred on zero, with bootstrap intervals. Win rates are shown alongside, because they are easier to read and because the two orderings differ when some systems met stronger opponents.
• Detection. Accuracy by ground truth, the share of “human” answers (response bias), and sensitivity d′ computed from hit and false-alarm rates.
• Decisiveness. We report what fraction of judgments were “No difference” or skipped. A high rate is itself a finding.
• Exploratory analyses — anything specified after the main results were seen — are labelled as such, and the number of comparisons made is stated.
• Language. A result whose interval includes the null is reported as “we did not detect”, not as “there is no effect”. A difference whose interval excludes zero is reported as an association unless the design supports a causal reading.

7Publication rules

• When a study is published. Each study has a planned number of judgments, or a calendar cutoff, fixed before collection starts and stated in the report. The report of record is issued when that point is reached, whatever the result. A report issued earlier is labelled an interim report, states the planned size and how far collection had got, states how many times the headline estimate had been inspected, and notes that its intervals are not adjusted for data-dependent inspection.
• Versioning. A report is versioned. Every published numerical snapshot is archived and remains available; later numbers are never substituted silently. A new version is issued at the planned sample size and whenever a conclusion changes.
• Audio. Clips from a study are released when the study closes, subject to the licence of the source material. Clips are not released while a study is still collecting, because a listener who has heard them cannot be blinded.
• Vendors. Companies whose systems appear in a public study are not involved in its design, execution or analysis, and are not informed of it in advance.
• Corrections. Errors found after publication are corrected on the page with a dated note. We would rather be corrected than appear consistent.

8Known limitations of the platform

• The panel is small and not a population sample. Listeners are people who chose to sign up to rate audio for money, reached mainly through social media. They are not weighted to any demographic, and we do not currently know their age, gender, first language or listening habits. Results describe this panel.
• English only, so far. All studies to date are in English with English-language material.
• No catch trials yet. See 2.4. We cannot presently distinguish an inattentive listener from an attentive one who finds the task hard, except through implausibly fast answers or playback that did not reach the end of a clip.
• Single playback. The interface allows one playback per clip in both task formats (pause and resume, no restart). This makes the task harder than everyday listening, where a person can go back, and results should be read as single-hearing results.
• Single hearing, browser playback. Listeners use their own devices and headphones or speakers in uncontrolled environments. This is the condition under which most audio is actually heard, but it adds noise relative to a laboratory.
• Repeated measures are only partly modelled. Default intervals account for listeners and are conditional on a study's speakers and passages. Intervals that also resample speakers are reported where a study generalises beyond its stimuli, and are much wider; with the small numbers of speakers and passages in most studies, resampling passages as well is close to uninformative.

9Changes to this document

• 1.0 — 4 November 2026. First version, written alongside the first detection report and revised the same day after external methodological review (publication rule changed from “estimate has stopped moving” to a fixed planned size with labelled interim reports; payment, spectral-adjustment and interval wording corrected).