Optimized for Dark Mode

The website is optimized for dark mode to enhance your user experience. Switch to dark mode to enjoy it.

Can Audio Decision Models Actually Hear?

94 % on spoken benchmarks, but does it actually hear, or just transcribe? I tested two audio decision models on voices, echoes and sounds a transcript can't capture.

Sebastian Bodza

TLDR

  • Clef-Omni answers spoken requests on benchmarks almost as well as text (BANKING77 94.0 vs. 94.7 %), but so does Whisper plus the same model. So that result alone does not show that it hears anything.
  • Clef-Omni recognizes voices and sounds, but tracks speakers poorly. It names sound events 96 % of the time and identifies who said a fact 59 % of the time (chance 36 %, Whisper transcript 37 %). With two voices it gets 87 % right, with four only 40 % (chance 25 %). Whisper plus a diarizer reaches 96 %.
  • Clef-Omni calls all 546 CREMA-D clips “neutral”, regardless of tone. It performs at chance on physics tasks. It can tell one voice from several, but it cannot count them.
  • d1-omni-600M mostly stays near chance on tasks without words.

What is a decision model?

A decision model reads an input and answers a question within a format you define. The answer can be a choice between options, a score, or a yes/no probability. You describe the task and possible answers in natural language, so the same model can evaluate different questions without a separate classifier for each one. The returned probabilities tell you when to automate a decision and when to send it for review.

Available question types depend on the model. Jev, for example, supports four question types. Choice selects from options you describe, such as which team should handle a ticket. Score rates an input on ordered levels, such as urgency. Noul returns the probability that a yes/no statement holds, such as whether a refund was requested. Box, available since Jev 1.14, locates a described object in an image.

How a decision model works
You define the task and the answer format with each request.
Your input and questionText, images or audioQuestion and answer definitionsDecision modelEvaluate input against the questionScore the requested outputsChoiceWhich team should handle this?Billing0.92Technical0.06Account0.02ScoreHow urgent is this ticket?LowMediumHighScore of 1.1 on levels from 0 to 2NoulDoes the customer want a refund?0.96probability of yesBoxWhere is the invoice in this image?Bounding boxImage coordinates
Examples of structured outputs. Available types and input formats depend on the model. Values are illustrative.

Making decisions with a model is nothing new. Traditional machine-learning classifiers and transformers with classification heads have long been used for this, commonly trained on labeled examples for a specific task and a fixed set of categories. LLMs made it possible to describe new tasks and categories in a prompt.

Earlier libraries checked responses against a schema and retried when they did not match. Newer strict Structured Outputs enforce the schema during decoding. Both approaches generate the answer token by token, adding latency and cost.

For a choice question, the decision models tested here read the input, question and allowed options, then return a probability for each option in a single pass. An LLM generates its answer token by token.

Generate an answer or score the choices
The LLM builds a JSON response token by token. The decision model scores the choices in a single pass and returns the best match.
PROMPT“I ordered a new card twoweeks ago and it stillhasn’t arrived.”intent?77 optionsescalate?yes / noTransformerprompt + KV cacheLM headscores vocabularynext token, reuse cached stateQwen3.8-27B · JSON output{"intent":"card_arrival","escalate":false}<end>prompt + decoding steps15 / 15estimated response time1,342 ms · doneSame model architectureTransformerreads prompt onceDecision headscores every optionClef 27B · decisionintentcard_arrival91 %card_delivery_estimate6 %lost_or_stolen_card2 %+ 74 more1 %escalateno83 %yes17 %passes1 / 1reported decision latency430 ms · done
Timing from OpenRouter (served by Cloudflare), October 25, 2026. Clef 27B reports 430 ms latency. Qwen3.8-27B takes 930 ms to the first token and runs at 34 tokens/s, about 1,342 ms for this 14-token answer. Playback slowed 10×.

Jev, from TypeSafe, helped popularize models built specifically for making decisions. It is a general-purpose decision model that can handle different topics and tasks without task-specific training or fine-tuning. You supply the question and describe the possible answers with each request. TypeSafe calls these System One models and trains them to return decisions with probabilities directly.

Jev can make these decisions much faster and at lower cost than an LLM generating an answer. In OpenRouter's ticket-routing test, Jev responded in 194 ms versus about 1.1 seconds for the LLM and cost roughly 0.025 USD versus 0.092 USD per 1,000 tickets. That made it about six times faster and four times cheaper in that comparison.

Less time, lower cost

Response time

About 6× faster

Cost per 1,000 tickets

About 4× lower cost

Jev 1.13 vs GPT 5.6 Luna in OpenRouter’s ticket-routing test. Bars show median response time and cost relative to the LLM. Shorter is better.

OpenAI introduced its Decisions API at DevDay, just two weeks after Jev's launch. Within three weeks, Cloudflare had also released its open-weight Clef models. Liquid AI's open-weight d1 models followed on October 7.

The audio variants are the focus here. Clef-Omni builds on Qwen3-Omni, while d1-omni uses an LFM2.5 encoder. Both add a decision head that scores the supplied choices.

Adding audio

You can transcribe a recording with Whisper and let a decision model read the text. This preserves the words but leaves out tone, speaker identity and background sounds. Passing the audio directly to the model gives it access to that information too. A model may accept audio and still rely only on the words. I test whether the recording helps it answer questions it gets wrong from the transcript alone.

Two ways to “accept audio”
From the outside both take a WAV and return one probability per option. Only the lower path keeps what the transcript throws away.
spoken requestSpeech-to-textASR, then language“It’s eleven o’clock.”transcriptpitchspeakeremotionroomtiminglostTranscriberAudio encoderframes, not wordswordspitchspeakeremotionroomtimingaudio tokensListenerLLMmodelJoint headreads hidden statesneutralangersadhappy
Clef-Omni follows the second path. It uses the Qwen3-Omni-30B-A3B thinker with its audio encoder and a small transformer “joint head” that scores every option in one pass. It generates no text. The first path uses Whisper, followed by the same model reading the transcript.

The two contenders

The Decision Index lists several decision models with audio input. I picked two at opposite ends of the size range, Clef-Omni at 30B and d1-omni-600M at 0.6B.

How Clef-Omni and d1-omni process audio
Both encode the audio, read it with the question, then score the choices.
Clef-OmniCloudflare

30B parameters · 19.8 GB in MLX 4-bit

  1. AuT audio encoder
    Turns speech, music and sounds into features.
  2. Qwen3-Omni thinker3B active
    Reads the audio with the question and choices.
  3. Joint schema head
    Scores all questions and choices together.

One probability for each choice

d1-omni-600MLiquid AI

0.6B parameters · 1.2 GB in MLX fp16

  1. FastConformer encoder112M
    Turns the recording into features for LFM2.5.
  2. LFM2.5 encoder~350M
    Reads the audio with the question and choices.
  3. Decision head
    Scores each question separately, so questions can be batched.

One probability for each choice

Simplified from the Clef-Omni and d1-omni inference code.

Clef-Omni builds on Qwen3-Omni, which supports speech recognition and translation, music and sound analysis, and audio captioning. Qwen also provides examples with speech, music and environmental sounds mixed together. That gives us a reason to expect Clef-Omni to understand more than words. d1-omni inherits its speech-focused encoder from LFM2-Audio, which uses Canary's FastConformer. Its later audio decision training covered spoken requests to an assistant.

I ran both models on a MacBook Pro (M5 Pro, 64 GB) using MLX. Clef-Omni needs about 70 GB in full precision, so I used the community 4-bit version to fit it in memory. d1-omni is small enough to run in fp16 without quantization.

Does the model hear or transcribe?

Getting a spoken request right does not tell us whether a model hears more than the words. I started with two tests that separate understanding what someone says from recognizing how they say it.

BANKING77 contains customer requests grouped into 77 banking intents, such as reporting a lost card or asking when a card will arrive. I gave Clef-Omni the same requests as text and as speech. The words contain everything needed to choose the right intent.

“How do I locate my card?”

The expected intent is card_arrival. Clef-Omni chooses card_arrival from both the text and the recording.

0:00 / 0:00

A test request spoken by OmniVoice, from the benchmark run.

CREMA-D tests emotion recognition. In the subset used here, 91 actors say the same sentence, “It's eleven o'clock”, in six different emotions. The model has to identify the emotion from the voice because the words stay the same.

Listen to one actor say “I think I've seen this before” in all six emotions in these CREMA-D recordings. At least in this sample, you can clearly hear the different emotions, yet Clef-Omni labels every recording “neutral”.

Actor plays anger. Model answers “neutral”.

0:00 / 0:00
One model, two answers to “can it hear?”
The banking task can be solved from the words. The emotion task requires the speaker's tone.
The words carry the answer
BANKING77, 77 intents
Text94.7
Audio94.0
The gap is -0.7 points. Speech costs almost nothing.
Only the voice carries the answer
CREMA-D, 546 clips. Each square is one clip, grouped by the emotion the actor played.
anger
disgust
fear
happiness
neutral
sadness
“neutral”, correct“neutral”, wrong
546 / 546 answered “neutral”, at 94.2 % mean confidence.
The full-precision model on an H200 gives the same picture, with 94.2 vs. 94.1 macro-F1 on all 3,080 requests and “neutral” on all 546 clips.

Clef-Omni handles the spoken bank requests almost as well as the text, but calls every emotion clip “neutral”. To check what the audio contributes, I also give each model a Whisper transcript of the recording (faster-whisper base.en). This keeps the question and choices the same while removing the sound. If the model uses more than words, it should outperform its own transcript on tasks that need the voice.

I test this in three steps, each with a clear expectation.

  1. Words carry the answer. On ten text benchmarks turned into speech, text, audio and the Whisper transcript should score about the same.
  2. Only the voice carries the answer. On four emotion sets, Whisper should be at chance. A listener should score clearly above it, ideally near its text level.
  3. Our own tests cover speakers, rooms, sounds and physics. The Whisper transcript sets the baseline, or chance where nothing is spoken. Anything the model scores above that baseline comes from hearing.

Step 1. Words carry the answer

Compare text, audio and Whisper on BANKING77
BANKING77, 77 intents. A transcriber and a listener should both land near the text score.
textaudioWhisper transcript
0255075100
Clef · Text
94.7
Audio
94.0
Whisper
92.3
d1 · Text
84.3
Audio
7.3
Whisper
83.0
300 test requests, the same for every bar. Whisper = faster-whisper base.en transcribes the clip, the same model decides on that text.

Clef-Omni gets almost as many requests right from speech as from text. d1-omni scores 84 % as text and 83 % behind Whisper, but only 7 % from its own audio path.

I tested the same requests as text and speech across ten decision benchmarks (BANKING77, CLINC150 + OOS, ANLI, When2Call, CLadder, GSM8K, HellaSwag, WinoGrande, iSarcasmEval, VAST), with 300 requests each. Only the input is spoken. The instruction and options stay text. Six text-to-speech models share the work on every benchmark. Each one copies its voices from a different set of real speakers, so no voice shows up in two models. Numbers with three or more digits go to OmniVoice, the only one that read “1234” right five times out of five. I transcribe every clip with Whisper to check it (word error rate ≤ 0.5, no extra words, plausible speed) and generate it again if it fails. A clip that keeps failing gets a new voice, then another text-to-speech model. Clips that still fail stay in the dataset with a flag, and the benchmark runs leave them out. All spoken benchmarks are in the Decider Audio collection on Hugging Face.

I also compared how confident the model was when reading and hearing the same request, and how often its answer changed between the two.

How confident is the model with text and audio?
Text and audio confidence for the same requests, grouped by confidence in the text answer.
textaudio
BANKING77
< .5.5–.7.7–.9≥ .9
CLINC150 + OOS
< .5.5–.7.7–.9≥ .9
When2Call
< .5.5–.7.7–.9≥ .9
ANLI
< .5.5–.7.7–.9≥ .9
CLadder
< .5.5–.7.7–.9≥ .9
GSM8K (10 options)
< .5.5–.7.7–.9≥ .9
HellaSwag
< .5.5–.7.7–.9≥ .9
WinoGrande
< .5.5–.7.7–.9≥ .9
iSarcasmEval
< .5.5–.7.7–.9≥ .9
VAST stance
< .5.5–.7.7–.9≥ .9
Bars show the average probability assigned to the chosen answer. Faded bars contain fewer than 30 requests. Grid lines mark 50 % and 100 %.

Clef-Omni makes the same choice on 98–99 % of BANKING77, CLINC150 and HellaSwag requests. CLadder and GSM8K lose four points. On CLadder Whisper loses the same, so invented words are hard to hear for both. On GSM8K Whisper loses less than one point, so Clef-Omni hears numbers worse than Whisper does. ANLI is the outlier, falling from 68 % to 60 %. It is the only benchmark where each request is split into two clips, a premise and a hypothesis. With Whisper transcripts of the same clips, Clef-Omni still gets 66 %, so the words get through. What gets lost is which clip is which. When I put the hypothesis clip first and say so in the instruction, accuracy drops to 45 %. Clef-Omni goes by the order of the clips and mostly ignores the labels. The TTS voice hardly matters.

Does the voice that read the request matter?
Clef-Omni, spoken minus written, in points of the benchmark's own metric, per text-to-speech model.
PocketSupertonicSoproQwen3-TTSChatterboxOmniVoicemeanBANKING77
+1.3
+0.2
−0.9
−1.5
−1.1
+1.4
−0.1
CLINC150 + OOS
−1.0
+0.3
+0.3
−0.9
−0.7
+1.5
−0.1
When2Call
+2.5
−0.5
+2.5
+2.8
+0.8
−0.3
+1.3
ANLI
−7.6
+1.2
−3.7
−2.0
−2.7
−4.6
−3.2
mean
−1.2
+0.3
−0.4
−0.4
−0.9
−0.5
audio worseaudio better(full color at ±5 points)
~500–900 requests per cell. A single cell moves by about ±2 points from sampling noise alone, so read rows and columns, not single squares. Text scores are computed on the same requests each TTS model rendered.

Could d1-omni have a bug? As text it gets 84 % on BANKING77, close to the 86 % Liquid AI reports on MASSIVE intent. It is at chance on ANLI, CLadder and WinoGrande. Three checks show that my MLX port is not why the spoken results collapse.

  • The examples from the model card give the right answers. My MLX audio preprocessing produces nearly the same input as Liquid AI's original code.
  • With four options instead of 77, spoken BANKING77 rises to 32 % (chance 25 %), compared with 95 % from text. The audio path picks up the topic but loses the sentence.
  • Liquid AI's own PyTorch code picks the same answer as my MLX port on 99–100 % of 5,426 requests, so it also lands at 7 %. The card says its audio training covered spoken requests to an assistant.

Step 1 shows that spoken words get through. Clef-Omni loses only 0.7 points on BANKING77, which is within noise. But a model that only transcribes would pass this step just as well, so it cannot tell us whether the model actually hears.

Step 2. Only the voice carries the answer

I use four emotion sets in which the words provide different amounts of help. CREMA-D has 91 actors say “It's eleven o'clock” in six emotions. VocalAffectBench uses neutral sentences across seven emotions. Neither set carries the answer in the words. IEMOCAP uses improvised dialogue, where the words sometimes help. MUStARD uses sitcom sarcasm, where they often do.

Compare direct audio with Whisper on CREMA-D
CREMA-D, six emotions, every actor says the same neutral sentence. A transcriber must sit at chance. A listener should score above it.
textaudioWhisper transcript
0255075100
Clef · Audio
16.7
Whisper
16.7
d1 · Audio
16.9
Whisper
16.7
The dashed line marks chance (16.7 %).
All 546 clips (91 actors × 6 emotions, one sentence). There is no text version because the text is identical for every clip.

Both models are exactly at chance on CREMA-D, with audio and with Whisper. Where do the answers go?

Where the answers go when the voice is the only cue
The go-to answer is the label a model gives most often, like “neutral”.
go-to answer, correctgo-to answer, wrongother answer, correctother answer, wrong
CREMA-D · 6 emotionsgo-to “neutral” · 17 % correct · chance 17 %
17 %
83 %
VocalAffectBench · 7 emotionsgo-to “neutral” · 16 % correct · chance 14 %
14 %
81 %
IEMOCAP · 4 emotionsgo-to “neutral” · 43 % correct · chance 25 %
28 %
55 %
15 %
MUStARD · sarcasmgo-to “not sarcastic” · 73 % correct · chance 50 %
37 %
14 %
36 %
13 %
All CREMA-D, VocalAffectBench and MUStARD clips, plus 300 IEMOCAP clips. Clef-Omni in 4-bit matches the full-precision run. In CREMA-D and VocalAffectBench the words give no hint.

Clef-Omni confidently labels all 546 CREMA-D clips “neutral”, with 94 % mean confidence. It gives the same label to 267 of 280 VocalAffectBench clips. On IEMOCAP's four classes it gets 43 % (Whisper 37 %, chance 25 %), mostly by calling neutral turns neutral. d1-omni collapses onto “angry” for VocalAffectBench and “sarcastic” for MUStARD.

At least Cloudflare's legal team can relax about the EU AI Act.

Roll Safe meme. Can't violate the EU AI Act's emotion ban if you can't detect emotions.

MUStARD is the exception. Clef-Omni scores 72.9 % with audio against 64.6 % from each line's transcript (balanced, chance 50 %). About 8 points come from the sound, although some of that may be audience laughter rather than tone of voice.

So both models fail step 2. For Clef-Omni there are two possible reasons. Either its audio encoder passes on only the words, or the voice gets through but the model never learned to use it for emotion. Step 3 tells them apart. If Clef-Omni does better with audio than with its Whisper transcript on any task the words cannot solve, such as telling speakers apart, the voice must be getting through. Then the second reason is the right one.

Step 3. Speakers, rooms and sounds

A recording carries more than emotion. It reveals who speaks, how many people are present, and what the room and surrounding sounds are like. For most of this, existing benchmarks can be solved from a transcript, so I built my own. They follow real use cases (meetings, calls, audio quality, everyday sounds), plus harder music and physics tests. I generate the audio with cloned voices, simulated rooms and small physics models, so every label is known exactly. These tests are part of the same collection, and each group below links to its dataset.

  • Fact attribution tests use 2–4 speakers who trade personal facts. Who said they grew up in Denver?
  • Speaker count tests ask how many people are talking, from one to five.
  • The five audio-native tasks test speaker matching, quote attribution, reverb, degradation and recording quality.
  • The nine acoustic physics tasks cover Doppler, bounces, struck material and size, pouring, clock strikes and tempo.
  • The six sound scene tasks use everyday sounds from ESC-50 (dog barks, rain, sirens), footsteps and foley (sound effects like a creaking door). They test sound identification and order, along with missing or repeated events.
  • The four bird soundscape tasks cover bird detection, species identification, counting and matching calls from the same species.
  • The four Bach chorale tasks test wrong notes, cadence, major or minor keys, and phrase count.
  • The chord tests play piano covers of pop songs (POP909) and ask which chord is played in a given bar.

Each group has 400 test items. Fact attribution has 1,198, and each audio-native task has 400.

Try a few physics clips yourself. It is harder than it sounds.

Listen to nine questions without a single word
One real test clip per task. Play it, pick an answer, then reveal. The spectrogram runs from 40 Hz (bottom) to 10 kHz (top).
Every clip is synthesized from a small physical model (numpy/scipy), so the label is exact. The 7,200 rows use 24 kHz mono audio, shown here as 64 kbit/s MP3.

Who said the fact?

The fact-attribution test simulates a meeting. Two to four speakers introduce themselves by name, then trade personal facts in pairs. One says, “I moved here ten years ago.” Another replies, “Oh nice. I moved here five years ago.” Who said they moved here five years ago? Turns are randomly ordered and the introductions contain no facts, so only the voice links a fact to a name. The test has 1,198 questions, and every recording fits into the 30 seconds both models can take in.

Audio, Whisper and diarization by number of speakers
Share of facts matched to the right speaker. The dashed tick is chance.
WhisperaudioWhisper + diarization
0 %25 %50 %75 %100 %
2 speakers
3 speakers
4 speakers
All
1,198 questions, about 400 per speaker count, conversations of about 23 s. NVIDIA Streaming Sortformer v2 provides the diarization. With true speaker labels, Clef-Omni scores 100 % and d1-omni 38.3 %.

Clef-Omni scores 59 %. From its Whisper transcript it scores 37 %, which is exactly chance for every number of speakers. The original script without speaker names scores the same (36 %). So the voice does reach the decision, which answers the open question from step 2. But it tracks speakers poorly. It scores 87 % with two voices and drops close to chance with four.

The classic pipeline wins by a wide margin. Whisper with speaker labels from NVIDIA's Sortformer diarizer brings the same model to 96 %, still 91 % with four voices. With the correct speaker labels it reaches 100 %, so the reasoning is easy. The hard part for the audio path is telling the voices apart and keeping track of them over 20 seconds. d1-omni is at chance in every condition, even with the correct labels. It fails at linking names to facts, not only at hearing.

Speaker count, same speaker and room

How many people are talking?
Conversations of 6–9 turns with 1–5 cloned voices. Rows show the true count. Columns show the model's answer. Each row sums to 100 %.
Real
Predicted
12345
1
91
4
1
0
4
2
18
2
40
6
33
3
9
3
34
7
47
4
9
2
32
8
48
5
4
3
35
22
37
↙ Voices merge below the diagonal.One voice splits above the diagonal. ↗
400 test conversations, about 80 per count. No speaker says their name and all turns come from the same set of sentences, so a transcript cannot answer this. From the Whisper transcript, Clef-Omni answers “one” for 314 of 400 and scores 19 %.

Clef-Omni can tell one voice from several, but it cannot count them. It gets 70 of 77 monologues right, and it answers “one” for only 33 of 323 conversations with more voices. From the Whisper transcript it answers “one” 314 times out of 400, so this comes from hearing. Beyond that it scores 20 % with two to five voices, which is chance. It mostly answers “three” or “five”, whatever the true count. d1-omni answers “one” for 362 of 400 conversations.

The other speaker and room tasks come from the audio-native set. All recordings are equally loud, so volume is no hint.

  • Same speaker. Two clips. Is it the same person both times? Some pairs are two different people who sound very much alike.
  • Who said the quote? Two to four people say their names, then talk in random order. Who said this sentence?
  • Echo. The same voice in two rooms. Which room echoes more?
  • Defect. One clip. Is it clean, or does it have noise, hum, distortion, phone sound, bad compression or gaps?
  • Better recording. The same clip twice, once a little damaged and once badly. Which one sounds better?

These tasks work a little. On each of them, Clef-Omni scores higher with audio than with its Whisper transcript.

TaskAudioWhisperChance
Is it the same speaker?67 %48 %50 %
Who said the quote?51 %37 %36 %
Which room has more echo?63 %54 %50 %
What is wrong with the recording? (7 options)28 %15 %14 %
Which recording sounds better?60 %46 %50 %

The same-speaker test includes pairs of very similar voices. So Clef-Omni hears a bit of the voice and the room, but not much. A small dedicated speaker model (20 million parameters) does far better.

Physics without words

Nine physical events, two listeners
Share of correct answers on generated clips. The dashed tick is chance.
Clef-Omnid1-omni-600Mchance
0 %50 %100 %
Doppler directionapproaching / passing / receding
Doppler speedwhich pass is faster
Bounce countfour consecutive numbers
Which ballrubber / table-tennis / steel
Struck materialwood / metal / glass
Struck sizewhich object is larger
Fill or drainresonance rises or falls
Clock strikesfour consecutive numbers
Tempo changefaster / slower / steady
44–45 clips per task, the same for both models. The clips contain no speech, so there is no Whisper run. A dot left of the tick means the model keeps picking a wrong answer.

Without words, Clef-Omni mostly guesses. Over all nine tasks it scores 38 %, and guessing gives 37 %. Counting clock strikes works a little (49 % against 25 % by chance). So does telling which of two objects is bigger (59 % against 50 %).

Doppler speed is odd. Two vehicles drive past, and the model has to say which one was faster. Clef-Omni is right only 11 % of the time, far worse than a coin flip. So it does hear a difference, but it almost always picks the wrong vehicle.

d1-omni guesses on every task but one. It can tell whether a vehicle is coming closer, driving past or moving away (51 % against 33 % by chance).

Two tasks have a shortcut. Bigger objects also sound louder, so I randomized the volume in the size task. In the material task, glass always sounds higher than wood and metal, so pitch alone can give the answer away.

Scenes, birds and music

TaskClef-Omnid1-omniChance
Name the everyday sound96 %27 %25 %
Which sound is missing?93 %18 %25 %
Put three sounds in order82 %26 %30 %
What surface are the footsteps on?45 %22 %25 %
Is a bird calling?71 %42 %50 %
Which bird species?71 %38 %25 %
How many phrases in a Bach piece?70 %21 %25 %
Spot the wrong note in Bach38 %38 %50 %
Which chord is playing?10 %4 %0.8 %

Clef-Omni clearly hears everyday sounds. It names them almost every time. It also finds the sound that is missing and puts three sounds in the right order, which takes more than recognizing a single sound. Footsteps are harder. Birds work well.

Music is mixed. Clef-Omni can count the phrases in a Bach piece. Spotting a wrong note fails for both models. Both score below a coin flip, so they probably lean toward the same answer. Chords are hard. There are more than 100 possible chords, so 10 % is far better than guessing, but the model is still wrong nine times out of ten.

d1-omni stays at or near chance on almost every task.

Summary

The chart shows every task, from words at the top to no words at the bottom. The dashed tick marks chance. Each dot is a model's score when it gets the audio. Each ring is its score when it only gets the Whisper transcript. If the dot sits to the right of the ring, the model hears more than the words.

The hearing ladder
Share of correct answers. The dashed tick is chance.
Whisperaudio
0 %50 %100 %
The wordsa transcript holds the answer
Bank customer requestBANKING77
Assistant requestCLINC150, incl. off-topic
When to call a toolWhen2Call
Does the claim follow?ANLI
Math word problemsGSM8K
Cause and effectCLadder
Words and tonethe tone can flip the answer
Sarcasm in TV scenesMUStARD
Sarcasm, read aloudiSarcasmEval
Only the voicethe words are neutral by design
Emotion, 4 optionsIEMOCAP
Emotion in the voiceVocalAffectBench
Emotion, neutral wordsCREMA-D
Who and how manytelling voices apart
How many speakers?1 to 5 voices
Same speaker?
Who said the quote?
Who said the fact?
Room and recordingecho and audio quality
Which room has more echo?
Which audio defect?7 options
Better recording?
Physical eventsno speech at all
Vehicle coming or going?Doppler
How many bounces?
Wood, metal or glass?struck object
How many clock strikes?
Scenes, birds, musicno speech at all
Everyday soundsESC-50
Birdssoundscapes
Bach choralesphrases, wrong notes
Which chord?POP909
For MUStARD, the show's own transcript stands in for Whisper. Rows without speech have no ring. 300–690 items per row, the same for both models.
  1. On word-based tasks, Clef-Omni scores the same with audio as with the Whisper transcript. Hearing the voice adds nothing. d1-omni scores much lower with audio, because it understands speech worse than Whisper does.
  2. On voice-only tasks, both models just guess the emotion.
  3. On our own tests, Clef-Omni beats its Whisper transcript on every speaker and room task. It also does better than chance on everyday sounds, birds and counting phrases in Bach. It only guesses on physics and on the exact number of voices. d1-omni mostly guesses whenever there are no words.
Does it only read the words, or does it hear?
What each test should show in both cases. The first two tests look the same either way.
Test
If it only reads the words
If it hears
What happened
Verdict
Spoken version of a text test
Same score as text
Same score as text
94.0 % from audio, 94.7 % from text (BANKING77)
Can't tell
Same request as text and as audio
Same answer, except where Whisper mishears
Same answer
Same answer 98 % of the time on BANKING77, 74 % on ANLI
Can't tell
Same words, different emotions
Always the same emotion
The right emotion
“Neutral” for all 546 clips
Doesn't hear
Sarcasm with and without audio
Same score as the transcript
Better than the transcript
72.9 % with audio, 64.6 % from the words alone
Hears
How many speakers?
Chance
Better than chance
34 % (chance 20 %). It tells one voice from several, but cannot count them.
Barely
Who said the fact?
Chance, the transcript has no names
Better than chance
59 % (chance 36 %). 87 % with two speakers, 40 % with four.
Barely
Physics, no speech at all
Chance
Better than chance
38 % (chance 37 %)
Doesn't hear
Clef-Omni in 4-bit on a MacBook, 300–690 requests per row. The full-size model on an H200 also says “neutral” for every emotion clip.

What this means

Clef-Omni does hear. It picks up speakers, rooms and everyday sounds, which no transcript can explain. But it hears poorly, and it does not hear tone at all. The voice information seems to reach the model, but it never learned to use it for emotion.

In practice, it comes down to four rules.

  • If the answer is in the words, give Clef-Omni the audio directly. It does as well as Whisper plus a text model, with one model instead of two.
  • If the answer depends on who is speaking, label the speakers first. Whisper with speaker labels from Sortformer scores 37 points higher than Clef-Omni listening by itself.
  • If the answer depends on how something is said, don't rely on it. It calls every emotion “neutral”.
  • For d1-omni, always transcribe with Whisper first.