Can Audio Decision Models Actually Hear?
94 % on spoken benchmarks, but does it actually hear, or just transcribe? I tested two audio decision models on voices, echoes and sounds a transcript can't capture.
On this page
TLDR
- Clef-Omni answers spoken requests on benchmarks almost as well as text (BANKING77 94.0 vs. 94.7 %), but so does Whisper plus the same model. So that result alone does not show that it hears anything.
- Clef-Omni recognizes voices and sounds, but tracks speakers poorly. It names sound events 96 % of the time and identifies who said a fact 59 % of the time (chance 36 %, Whisper transcript 37 %). With two voices it gets 87 % right, with four only 40 % (chance 25 %). Whisper plus a diarizer reaches 96 %.
- Clef-Omni calls all 546 CREMA-D clips “neutral”, regardless of tone. It performs at chance on physics tasks. It can tell one voice from several, but it cannot count them.
- d1-omni-600M mostly stays near chance on tasks without words.
What is a decision model?
A decision model reads an input and answers a question within a format you define. The answer can be a choice between options, a score, or a yes/no probability. You describe the task and possible answers in natural language, so the same model can evaluate different questions without a separate classifier for each one. The returned probabilities tell you when to automate a decision and when to send it for review.
Available question types depend on the model. Jev, for example, supports four question types. Choice selects from options you describe, such as which team should handle a ticket. Score rates an input on ordered levels, such as urgency. Noul returns the probability that a yes/no statement holds, such as whether a refund was requested. Box, available since Jev 1.14, locates a described object in an image.
Making decisions with a model is nothing new. Traditional machine-learning classifiers and transformers with classification heads have long been used for this, commonly trained on labeled examples for a specific task and a fixed set of categories. LLMs made it possible to describe new tasks and categories in a prompt.
Earlier libraries checked responses against a schema and retried when they did not match. Newer strict Structured Outputs enforce the schema during decoding. Both approaches generate the answer token by token, adding latency and cost.
For a choice question, the decision models tested here read the input, question and allowed options, then return a probability for each option in a single pass. An LLM generates its answer token by token.
Jev, from TypeSafe, helped popularize models built specifically for making decisions. It is a general-purpose decision model that can handle different topics and tasks without task-specific training or fine-tuning. You supply the question and describe the possible answers with each request. TypeSafe calls these System One models and trains them to return decisions with probabilities directly.
Jev can make these decisions much faster and at lower cost than an LLM generating an answer. In OpenRouter's ticket-routing test, Jev responded in 194 ms versus about 1.1 seconds for the LLM and cost roughly 0.025 USD versus 0.092 USD per 1,000 tickets. That made it about six times faster and four times cheaper in that comparison.
Response time
About 6× faster
Cost per 1,000 tickets
About 4× lower cost
OpenAI introduced its Decisions API at DevDay, just two weeks after Jev's launch. Within three weeks, Cloudflare had also released its open-weight Clef models. Liquid AI's open-weight d1 models followed on October 7.
The audio variants are the focus here. Clef-Omni builds on Qwen3-Omni, while d1-omni uses an LFM2.5 encoder. Both add a decision head that scores the supplied choices.
Adding audio
You can transcribe a recording with Whisper and let a decision model read the text. This preserves the words but leaves out tone, speaker identity and background sounds. Passing the audio directly to the model gives it access to that information too. A model may accept audio and still rely only on the words. I test whether the recording helps it answer questions it gets wrong from the transcript alone.
The two contenders
The Decision Index lists several decision models with audio input. I picked two at opposite ends of the size range, Clef-Omni at 30B and d1-omni-600M at 0.6B.
30B parameters · 19.8 GB in MLX 4-bit
- AuT audio encoderTurns speech, music and sounds into features.
- Qwen3-Omni thinker3B activeReads the audio with the question and choices.
- Joint schema headScores all questions and choices together.
One probability for each choice
0.6B parameters · 1.2 GB in MLX fp16
- FastConformer encoder112MTurns the recording into features for LFM2.5.
- LFM2.5 encoder~350MReads the audio with the question and choices.
- Decision headScores each question separately, so questions can be batched.
One probability for each choice
Clef-Omni builds on Qwen3-Omni, which supports speech recognition and translation, music and sound analysis, and audio captioning. Qwen also provides examples with speech, music and environmental sounds mixed together. That gives us a reason to expect Clef-Omni to understand more than words. d1-omni inherits its speech-focused encoder from LFM2-Audio, which uses Canary's FastConformer. Its later audio decision training covered spoken requests to an assistant.
I ran both models on a MacBook Pro (M5 Pro, 64 GB) using MLX. Clef-Omni needs about 70 GB in full precision, so I used the community 4-bit version to fit it in memory. d1-omni is small enough to run in fp16 without quantization.
Does the model hear or transcribe?
Getting a spoken request right does not tell us whether a model hears more than the words. I started with two tests that separate understanding what someone says from recognizing how they say it.
BANKING77 contains customer requests grouped into 77 banking intents, such as reporting a lost card or asking when a card will arrive. I gave Clef-Omni the same requests as text and as speech. The words contain everything needed to choose the right intent.
“How do I locate my card?”
The expected intent is card_arrival. Clef-Omni chooses card_arrival from both the text and the recording.
A test request spoken by OmniVoice, from the benchmark run.
CREMA-D tests emotion recognition. In the subset used here, 91 actors say the same sentence, “It's eleven o'clock”, in six different emotions. The model has to identify the emotion from the voice because the words stay the same.
Listen to one actor say “I think I've seen this before” in all six emotions in these CREMA-D recordings. At least in this sample, you can clearly hear the different emotions, yet Clef-Omni labels every recording “neutral”.
Clef-Omni handles the spoken bank requests almost as well as the text, but calls every emotion clip “neutral”. To check what the audio contributes, I also give each model a Whisper transcript of the recording (faster-whisper base.en). This keeps the question and choices the same while removing the sound. If the model uses more than words, it should outperform its own transcript on tasks that need the voice.
I test this in three steps, each with a clear expectation.
- Words carry the answer. On ten text benchmarks turned into speech, text, audio and the Whisper transcript should score about the same.
- Only the voice carries the answer. On four emotion sets, Whisper should be at chance. A listener should score clearly above it, ideally near its text level.
- Our own tests cover speakers, rooms, sounds and physics. The Whisper transcript sets the baseline, or chance where nothing is spoken. Anything the model scores above that baseline comes from hearing.
Step 1. Words carry the answer
Clef-Omni gets almost as many requests right from speech as from text. d1-omni scores 84 % as text and 83 % behind Whisper, but only 7 % from its own audio path.
I tested the same requests as text and speech across ten decision benchmarks (BANKING77, CLINC150 + OOS, ANLI, When2Call, CLadder, GSM8K, HellaSwag, WinoGrande, iSarcasmEval, VAST), with 300 requests each. Only the input is spoken. The instruction and options stay text. Six text-to-speech models share the work on every benchmark. Each one copies its voices from a different set of real speakers, so no voice shows up in two models. Numbers with three or more digits go to OmniVoice, the only one that read “1234” right five times out of five. I transcribe every clip with Whisper to check it (word error rate ≤ 0.5, no extra words, plausible speed) and generate it again if it fails. A clip that keeps failing gets a new voice, then another text-to-speech model. Clips that still fail stay in the dataset with a flag, and the benchmark runs leave them out. All spoken benchmarks are in the Decider Audio collection on Hugging Face.
I also compared how confident the model was when reading and hearing the same request, and how often its answer changed between the two.
Clef-Omni makes the same choice on 98–99 % of BANKING77, CLINC150 and HellaSwag requests. CLadder and GSM8K lose four points. On CLadder Whisper loses the same, so invented words are hard to hear for both. On GSM8K Whisper loses less than one point, so Clef-Omni hears numbers worse than Whisper does. ANLI is the outlier, falling from 68 % to 60 %. It is the only benchmark where each request is split into two clips, a premise and a hypothesis. With Whisper transcripts of the same clips, Clef-Omni still gets 66 %, so the words get through. What gets lost is which clip is which. When I put the hypothesis clip first and say so in the instruction, accuracy drops to 45 %. Clef-Omni goes by the order of the clips and mostly ignores the labels. The TTS voice hardly matters.
Could d1-omni have a bug? As text it gets 84 % on BANKING77, close to the 86 % Liquid AI reports on MASSIVE intent. It is at chance on ANLI, CLadder and WinoGrande. Three checks show that my MLX port is not why the spoken results collapse.
- The examples from the model card give the right answers. My MLX audio preprocessing produces nearly the same input as Liquid AI's original code.
- With four options instead of 77, spoken BANKING77 rises to 32 % (chance 25 %), compared with 95 % from text. The audio path picks up the topic but loses the sentence.
- Liquid AI's own PyTorch code picks the same answer as my MLX port on 99–100 % of 5,426 requests, so it also lands at 7 %. The card says its audio training covered spoken requests to an assistant.
Step 1 shows that spoken words get through. Clef-Omni loses only 0.7 points on BANKING77, which is within noise. But a model that only transcribes would pass this step just as well, so it cannot tell us whether the model actually hears.
Step 2. Only the voice carries the answer
I use four emotion sets in which the words provide different amounts of help. CREMA-D has 91 actors say “It's eleven o'clock” in six emotions. VocalAffectBench uses neutral sentences across seven emotions. Neither set carries the answer in the words. IEMOCAP uses improvised dialogue, where the words sometimes help. MUStARD uses sitcom sarcasm, where they often do.
Both models are exactly at chance on CREMA-D, with audio and with Whisper. Where do the answers go?
Clef-Omni confidently labels all 546 CREMA-D clips “neutral”, with 94 % mean confidence. It gives the same label to 267 of 280 VocalAffectBench clips. On IEMOCAP's four classes it gets 43 % (Whisper 37 %, chance 25 %), mostly by calling neutral turns neutral. d1-omni collapses onto “angry” for VocalAffectBench and “sarcastic” for MUStARD.
At least Cloudflare's legal team can relax about the EU AI Act.
MUStARD is the exception. Clef-Omni scores 72.9 % with audio against 64.6 % from each line's transcript (balanced, chance 50 %). About 8 points come from the sound, although some of that may be audience laughter rather than tone of voice.
So both models fail step 2. For Clef-Omni there are two possible reasons. Either its audio encoder passes on only the words, or the voice gets through but the model never learned to use it for emotion. Step 3 tells them apart. If Clef-Omni does better with audio than with its Whisper transcript on any task the words cannot solve, such as telling speakers apart, the voice must be getting through. Then the second reason is the right one.
Step 3. Speakers, rooms and sounds
A recording carries more than emotion. It reveals who speaks, how many people are present, and what the room and surrounding sounds are like. For most of this, existing benchmarks can be solved from a transcript, so I built my own. They follow real use cases (meetings, calls, audio quality, everyday sounds), plus harder music and physics tests. I generate the audio with cloned voices, simulated rooms and small physics models, so every label is known exactly. These tests are part of the same collection, and each group below links to its dataset.
- Fact attribution tests use 2–4 speakers who trade personal facts. Who said they grew up in Denver?
- Speaker count tests ask how many people are talking, from one to five.
- The five audio-native tasks test speaker matching, quote attribution, reverb, degradation and recording quality.
- The nine acoustic physics tasks cover Doppler, bounces, struck material and size, pouring, clock strikes and tempo.
- The six sound scene tasks use everyday sounds from ESC-50 (dog barks, rain, sirens), footsteps and foley (sound effects like a creaking door). They test sound identification and order, along with missing or repeated events.
- The four bird soundscape tasks cover bird detection, species identification, counting and matching calls from the same species.
- The four Bach chorale tasks test wrong notes, cadence, major or minor keys, and phrase count.
- The chord tests play piano covers of pop songs (POP909) and ask which chord is played in a given bar.
Each group has 400 test items. Fact attribution has 1,198, and each audio-native task has 400.
Try a few physics clips yourself. It is harder than it sounds.
Who said the fact?
The fact-attribution test simulates a meeting. Two to four speakers introduce themselves by name, then trade personal facts in pairs. One says, “I moved here ten years ago.” Another replies, “Oh nice. I moved here five years ago.” Who said they moved here five years ago? Turns are randomly ordered and the introductions contain no facts, so only the voice links a fact to a name. The test has 1,198 questions, and every recording fits into the 30 seconds both models can take in.
Clef-Omni scores 59 %. From its Whisper transcript it scores 37 %, which is exactly chance for every number of speakers. The original script without speaker names scores the same (36 %). So the voice does reach the decision, which answers the open question from step 2. But it tracks speakers poorly. It scores 87 % with two voices and drops close to chance with four.
The classic pipeline wins by a wide margin. Whisper with speaker labels from NVIDIA's Sortformer diarizer brings the same model to 96 %, still 91 % with four voices. With the correct speaker labels it reaches 100 %, so the reasoning is easy. The hard part for the audio path is telling the voices apart and keeping track of them over 20 seconds. d1-omni is at chance in every condition, even with the correct labels. It fails at linking names to facts, not only at hearing.
Speaker count, same speaker and room
Clef-Omni can tell one voice from several, but it cannot count them. It gets 70 of 77 monologues right, and it answers “one” for only 33 of 323 conversations with more voices. From the Whisper transcript it answers “one” 314 times out of 400, so this comes from hearing. Beyond that it scores 20 % with two to five voices, which is chance. It mostly answers “three” or “five”, whatever the true count. d1-omni answers “one” for 362 of 400 conversations.
The other speaker and room tasks come from the audio-native set. All recordings are equally loud, so volume is no hint.
- Same speaker. Two clips. Is it the same person both times? Some pairs are two different people who sound very much alike.
- Who said the quote? Two to four people say their names, then talk in random order. Who said this sentence?
- Echo. The same voice in two rooms. Which room echoes more?
- Defect. One clip. Is it clean, or does it have noise, hum, distortion, phone sound, bad compression or gaps?
- Better recording. The same clip twice, once a little damaged and once badly. Which one sounds better?
These tasks work a little. On each of them, Clef-Omni scores higher with audio than with its Whisper transcript.
| Task | Audio | Whisper | Chance |
|---|---|---|---|
| Is it the same speaker? | 67 % | 48 % | 50 % |
| Who said the quote? | 51 % | 37 % | 36 % |
| Which room has more echo? | 63 % | 54 % | 50 % |
| What is wrong with the recording? (7 options) | 28 % | 15 % | 14 % |
| Which recording sounds better? | 60 % | 46 % | 50 % |
The same-speaker test includes pairs of very similar voices. So Clef-Omni hears a bit of the voice and the room, but not much. A small dedicated speaker model (20 million parameters) does far better.
Physics without words
Without words, Clef-Omni mostly guesses. Over all nine tasks it scores 38 %, and guessing gives 37 %. Counting clock strikes works a little (49 % against 25 % by chance). So does telling which of two objects is bigger (59 % against 50 %).
Doppler speed is odd. Two vehicles drive past, and the model has to say which one was faster. Clef-Omni is right only 11 % of the time, far worse than a coin flip. So it does hear a difference, but it almost always picks the wrong vehicle.
d1-omni guesses on every task but one. It can tell whether a vehicle is coming closer, driving past or moving away (51 % against 33 % by chance).
Two tasks have a shortcut. Bigger objects also sound louder, so I randomized the volume in the size task. In the material task, glass always sounds higher than wood and metal, so pitch alone can give the answer away.
Scenes, birds and music
| Task | Clef-Omni | d1-omni | Chance |
|---|---|---|---|
| Name the everyday sound | 96 % | 27 % | 25 % |
| Which sound is missing? | 93 % | 18 % | 25 % |
| Put three sounds in order | 82 % | 26 % | 30 % |
| What surface are the footsteps on? | 45 % | 22 % | 25 % |
| Is a bird calling? | 71 % | 42 % | 50 % |
| Which bird species? | 71 % | 38 % | 25 % |
| How many phrases in a Bach piece? | 70 % | 21 % | 25 % |
| Spot the wrong note in Bach | 38 % | 38 % | 50 % |
| Which chord is playing? | 10 % | 4 % | 0.8 % |
Clef-Omni clearly hears everyday sounds. It names them almost every time. It also finds the sound that is missing and puts three sounds in the right order, which takes more than recognizing a single sound. Footsteps are harder. Birds work well.
Music is mixed. Clef-Omni can count the phrases in a Bach piece. Spotting a wrong note fails for both models. Both score below a coin flip, so they probably lean toward the same answer. Chords are hard. There are more than 100 possible chords, so 10 % is far better than guessing, but the model is still wrong nine times out of ten.
d1-omni stays at or near chance on almost every task.
Summary
The chart shows every task, from words at the top to no words at the bottom. The dashed tick marks chance. Each dot is a model's score when it gets the audio. Each ring is its score when it only gets the Whisper transcript. If the dot sits to the right of the ring, the model hears more than the words.
- On word-based tasks, Clef-Omni scores the same with audio as with the Whisper transcript. Hearing the voice adds nothing. d1-omni scores much lower with audio, because it understands speech worse than Whisper does.
- On voice-only tasks, both models just guess the emotion.
- On our own tests, Clef-Omni beats its Whisper transcript on every speaker and room task. It also does better than chance on everyday sounds, birds and counting phrases in Bach. It only guesses on physics and on the exact number of voices. d1-omni mostly guesses whenever there are no words.
What this means
Clef-Omni does hear. It picks up speakers, rooms and everyday sounds, which no transcript can explain. But it hears poorly, and it does not hear tone at all. The voice information seems to reach the model, but it never learned to use it for emotion.
In practice, it comes down to four rules.
- If the answer is in the words, give Clef-Omni the audio directly. It does as well as Whisper plus a text model, with one model instead of two.
- If the answer depends on who is speaking, label the speakers first. Whisper with speaker labels from Sortformer scores 37 points higher than Clef-Omni listening by itself.
- If the answer depends on how something is said, don't rely on it. It calls every emotion “neutral”.
- For d1-omni, always transcribe with Whisper first.
Links
- Cloudflare/clef-omni and mlx-community/clef-omni-4bit
- LiquidAI/d1-omni-600M and its MLX fp16 build
- Decision Index
- Decider Audio collection with all 18 audio benchmarks from this post
- faster-whisper
About the author
Sebastian Bodza
AI engineer focused on LLMs, TTS, agent workflows, and practical GenAI systems built for real production constraints.
Related reading
Why LLM Benchmarks Can Be Misleading - AWQ vs. GPTQ
Why official LLM benchmarks can mislead you: AWQ vs GPTQ results, custom tests, and what matters more than leaderboard scores.
From Llama to LLaSA - GRPO with WER and custom repetition penalty
How I used GRPO, Whisper WER, and a custom repetition penalty to improve LLaSA-style TTS quality with audio tokens.
Faster LLMs with Quantization - How to get faster inference times with quantization
A practical look at LLM quantization tradeoffs, benchmark results, and what actually improves vLLM inference speed in production.