VoiceArena.com TTS Inline Steerability

An event tag is correct when the listener names the exact sound the model was given. In same-family scoring it is correct when the named sound is in the same family: Laughter = laugh, chuckle, giggle; Exhale = sigh, breath; Cry = cry. Other sound, nothing there, read out as words and garbled are always wrong, in both modes.

A style span is correct when the requested style is among the styles ticked. Ticking an extra style alongside the right one does not fail the span, and the problem answer is not part of the score — listeners agree on it far too poorly (0.03) to be worth anything. Style spans score the same in both modes.

Where a span sits inside another span, only the inner style has to be picked. 600 of the 1,195 spans tested are nested.

Every clip was listened to five separate times, by five different people. No one heard the same clip twice. A tag counts as correct when at least three of the five mark it correct.

The panel is an odd number on purpose, because an odd panel cannot tie: five votes always split three to two or better, so there is always a majority and never a rule to invent. 13,651 of the 14,400 tags carry all five rounds. 670 carry four and 79 carry three, and those use the majority of the votes they have.

What the fifth round bought: every tag now clears the agreement bar, ten of the eleven reach the reliable standard of 0.80, and the two-against-two splits that affected one tag in ten at four rounds are gone — 69 tags out of 14,400 still have no majority, all of them on clips that did not receive a fifth listener.

Every clip is rated in five rounds. Each round is a different person, working alone, who never sees anyone else's answers. A tag counts as correct when at least three of the five agree.

Why not one listener. Two of our listeners give the same answer 74.2% of the time, but much of that is luck — people landing on the same answer by accident. So we measure agreement with Krippendorff's alpha, which removes chance agreement. On that scale 1.0 is perfect and 0 is no better than guessing. One listener alone scores 0.637, and at that level only five of our eleven tags are usable and none are reliable.

Where the pass mark comes from. It is Krippendorff's published standard, not ours: 0.800 and above is reliable, 0.667 to 0.800 is tentative and is flagged, and anything below 0.667 is not used for ranking.

Why five. The Spearman-Brown formula, published in 1910 and still the standard way to work out how many raters a task needs, converts one rater's reliability into that of a panel. Run backwards it gives the score a single listener must reach:

Ours score 0.637, so five clears the bar with room. We stop at five because the gain flattens: the second listener adds 0.133 to the average, the sixth adds 0.016, at identical cost. At five listeners ten of our eleven tags reach the reliable band; only soft falls short, and it is flagged on the board.

Why an odd number. An odd panel cannot tie. All five rounds have now landed. An odd panel removes the two-against-two splits: 69 tags out of 14,400 still have no majority, all of them on clips that did not receive a fifth listener.

Krippendorff, Content Analysis, 2nd ed. (2004); Spearman (1910) and Brown (1910), British Journal of Psychology 3; Wong and Paritosh, k-Rater Reliability, ACL 2022.

Events carries six event tags per sentence. Styles carries five style spans. Mixed carries eleven tags — six events and five spans in the same sentence — and is the hardest set for every model on the board.

Every ranked model receives the same sentences in every set, so nothing needs adjusting for difficulty. A model that does not document a tag is never sent its sentences and never given a substitute, so the Gemini models appear on the Events set alone.

Tag accuracy is correct tags divided by all tags. Task Success Rate is the share of sentences in which every tag was correct — one tag wrong anywhere and the whole sentence fails, however good the rest of it was. Tag accuracy tells you how often one instruction is performed; Task Success Rate tells you how often a whole clip is usable as it is, with no retake.

The letter beneath each Task Success Rate figure says which set it covers. On All sets it is every sentence pooled: total sentences with no mistake, divided by total sentences. Pooling and averaging the three sets give the same answer here to within 0.03 of a point, and pooling is used because it is how tag accuracy on All sets is already computed, and because it still works for a model that was only sent one set.

The sets are not equally hard and the per-set figures in Results below show why: a sentence carries six tags in Events, five in Styles and eleven in Mixed. On Mixed only Eleven v4 manages any fully correct sentences, at 2.0%.

Sort by tag accuracy. Compare each model with the leader of its tier using McNemar’s test, which looks only at the tags where one model was right and the other wrong and ignores every tag they both got right or both missed. At p ≥ 0.05 the model shares the leader’s rank.

The range after each rank comes from resampling the sentences 4,000 times and recording where the model lands each time. (1–2) means the model could plausibly be first or second; (5–5) means its position is not in doubt.