Ask your audio about what was said, how it was said, and who said it. Every answer comes with a probability you can trust.
Try it yourself: the Dodona demo. Record your voice or pick an example, type your own questions, and see the probabilities. No install, free GPU.
Dodona answers yes/no, multiple-choice and scale questions about a recording (is the speaker angry? was the word "refund" said? how many people are talking?) with probabilities that mean what they say: when Dodona answers 80%, it is right about 80% of the time. It reads the audio once and answers every question on top of it, by reading the probability of each answer letter (no text is generated): forty questions take about four times as long as one, not forty times.
Every question has one of three types, as in decision models for text, with audio in place of
text: noul (yes/no), choice (one of several options) and score (a choice whose options
are ordered from lowest to highest; the answer is the expected level). Dodona is two LoRA adapters on
Qwen2-Audio-7B-Instruct and a calibration
layer; its answers are the calibrated mean of the two adapters.
You write the question in plain language and pick its type; the answer always has that shape, with a probability, and nothing to parse (a plain audio LLM asked the usual way left 1.3% of its answers without a usable letter, experiment 064).
Use the dodona package (GitHub): it loads both
adapters, averages them and applies the calibration. Loading one adapter with PEFT alone gives
uncalibrated answers.
pip install git+https://github.com/bahadobay/dodona-audio
from dodona import Dodona
model = Dodona() # this repo's adapters + Qwen2-Audio-7B (~16 GB)
result = model.ask("angry.wav", { # CREMA-D 1006_WSI_ANG_XX.wav, an actor, angrily
# choice: this wording is a built-in question, so Dodona supplies the options:
# angry, happy, sad, neutral, fear, disgust (model.questions() describes each built-in's options)
"emotion": {"type": "choice", "instructions": "Which emotion is the speaker expressing?"},
"angry": {"type": "noul", "instructions": "The speaker sounds angry in this recording."},
"upset": {"type": "score", "instructions": "How upset is the speaker?",
"criteria": ["calm", "annoyed", "very upset"]}, # a question you write yourself
})
result["answers"]
{
"emotion": {"type": "choice", "choice": "angry",
"probabilities": {"angry": 0.969, "happy": 0.01, "sad": 0.004, "neutral": 0.005,
"fear": 0.004, "disgust": 0.008}, "status": "calibrated"},
"angry": {"type": "noul", "noul": 0.978, "where": [0.0, 2.736], "status": "calibrated"},
"upset": {"type": "score", "score": 1.27, "probabilities": {"0": 0.07, "1": 0.589, "2": 0.341},
"legend": {"0": "calm", "1": "annoyed", "2": "very upset"}, "status": "off catalogue"}
}
The clip's actor is from the test set, never heard in training (CREMA-D's 12 sentences are read
by every actor, so the sentence itself was); 10 of 10 people who rated the voice alone heard
anger. emotion is a choice: it picks one of six emotions (angry, happy, sad, neutral, fear,
disgust, which Dodona supplies because the wording is one it was trained on). angry is a
noul: yes or no. Both use the exact wording of a built-in question, so they are tested and
their status is calibrated. upset is a
question we wrote ourselves: score lists the options from lowest to highest, Dodona reads the
probability of each, and the score is the expected level (here 0 x 0.07 + 1 x 0.589 + 2 x 0.341,
about 1.27: just above "annoyed"). Questions in your own words are answered too, with
status off catalogue: calibrated by a general map, not one fitted to that question.
Audio is read in 10-second windows. For the whole recording, a yes/no question reports its
highest window and where it is ("the angriest 10 seconds", a statement about those 10
seconds, not about the whole call); choice and score questions report the windows' average.
Every window's own answers are in result["windows"]. More in EXAMPLES.md.
Better than its backbone. The same test windows, the same questions, Qwen2-Audio-7B-Instruct without the adapters (experiment 064; skill = mean AUROC, UAR for multiple choice). This gives the backbone its best chance: its answers are read the way Dodona reads its own (the probability of each answer letter, no text generated), with a calibration map fitted for it:
| backbone | Dodona | |
|---|---|---|
| emotion (acted) | 0.848 | 0.967 |
| tone (speaking rate, hesitation, speakers, overlap) | 0.456 | 0.821 |
| content (topics, questions, numbers) | 0.779 | 0.941 |
| sounds, dialogue acts, voices | 0.693 | 0.869 |
| was a word said | 0.891 | 0.961 |
| new wordings of trained questions | 0.647 | 0.944 |
Calibration error (ECE, each model with its own calibration map): emotion 0.069 → 0.018, tone 0.232 → 0.066, content 0.078 → 0.015, sounds/acts/voices 0.205 → 0.034.
Asked the usual way instead (each question its own prompt, the answer generated and its letter parsed; balanced accuracy on a sample of 100 windows per corpus), the backbone falls further behind: emotion 0.656 against Dodona's 0.946, tone 0.433 against 0.764, content 0.594 against 0.838, and 1.3% of its answers had no usable letter.
Against a transcript. Whether a given word was said, Dodona matches a speech recogniser (Whisper large-v3-turbo + exact match: balanced accuracy 0.904 against Dodona's 0.907). The two miss different words: of the 333 said words the transcript missed (AMI test meetings, 100 words, 600 windows), Dodona says yes (0.5 or more) to 48%, and to 0.3% of words that were not said. Counting a word as said when either says so raises balanced accuracy from 0.904 to 0.949. For topics (was money mentioned?) a transcript plus a keyword list is more accurate (0.943 against 0.809; about a third of the gap is the threshold: topics are rare, so use a threshold below 0.5, e.g. 0.2, to catch them). The topic labels are themselves defined by keyword lists, so the transcript matches the label rule by construction; how Dodona compares on topics defined by meaning is not measured. What Dodona adds, in the same pass, is how it was said and who said it (tone, emotion, voices, non-speech sounds) with calibrated probabilities.
Speed. One 10-second window on an NVIDIA L40S: one adapter answers 1 question in 0.10 s, 40 in 0.40 s, 100 in 0.76 s. Asking the backbone the usual way (one generated answer per question) takes 0.10 s per question, 4.1 s for 40. So: the same speed for one question, about 10× faster at 40 questions (one adapter). The package reads both adapters, one after the other: 53 questions take 1.41 s per 10-second window, about 7× faster than real time. If the question is only whether a word was said, a transcript is faster still: Whisper large-v3-turbo takes 0.14 s per 10-second window.
Noise. Skill drop from clean audio (white noise / a second voice in the background):
| white 20 dB | white 10 dB | white 5 dB | babble 10 dB | |
|---|---|---|---|---|
| tone | −0.016 | −0.035 | −0.053 | −0.079 |
| content | −0.006 | −0.029 | −0.072 | −0.032 |
| emotion | −0.045 | −0.098 | −0.131 | −0.067 |
Catalogue yes/no questions (104 question–corpus pairs), cross-fitted on held-out test windows:
| shown | windows | true |
|---|---|---|
| 0–10% | 78,759 | 0.01 (shown 0.02) |
| 10–30% | 6,453 | 0.17 (shown 0.18) |
| 30–50% | 2,271 | 0.39 (shown 0.37) |
| 50–70% | 2,915 | 0.56 (shown 0.57) |
| 70–90% | 3,679 | 0.79 (shown 0.80) |
| 90–100% | 8,646 | 0.95 (shown 0.95) |
Mean ECE: yes/no 0.024, multiple choice 0.029, words never seen in training 0.013, scales 0.099 (weak, see below).
Every answer carries a status:
model.questions()) or the word shortcut;| corpus | what | licence |
|---|---|---|
| AMI Meeting Corpus | 32 English meetings (~16 h), 10 s windows, transcripts and annotations | CC BY 4.0 |
| CREMA-D | 91 actors, 12 sentences, 6 acted emotions | ODbL / DbCL |
| Common Voice Scripted 27.0 (en) | read sentences, self-reported gender and age, ~30 h | CC0 |
| Common Voice Spontaneous (en) | spontaneous speech | CC0 |
Test sets never enter training (AMI meetings, CREMA-D actors, Common Voice speakers and sentences are disjoint). Every question and its label rule: TRAINING_DATA.md.
Dodona was built by one person on a single cloud GPU, with about $210 of cloud compute in all (AWS's $200 starter credits, then a few dollars more), research runs included. Openly licensed, labelled speech is scarce, and that shapes what it can claim:
over_sixty_speaks is not verified: 12 positive test clips).speaking_rate, is_overlapping_speech, a_teenager_speaks, backchannels, long_silence
and over_sixty_speaks are answered but marked not verified.off catalogue. On 12 new content topics in read speech (experiment 078), the ranking holds
(AUROC 0.96; every topic 0.89 to 0.99), whether or not the topic resembles a trained one.
Of the answers shown at 90% or more, 89 to 94% named the topic when the
sentences were read by hand (keyword labels alone say 63%, because they miss words such as
"car" or "politics"). Trust such an answer less when the two adapters disagree: their
disagreement tracks the error (Spearman 0.73). A question no recording can settle ("the
speaker owns a bicycle") gets a confident no (mean 0.05), not a guess at 50%. New tone or
acoustic concepts are not measured.n_speakers counts the voices in a 10-second window (balanced accuracy 0.72 to
0.74), not in a whole meeting; to know whether more than one person speaks, ask
several_speakers (AUROC 0.96 to 0.97); to count everyone, use a diarization tool.dodona.model.ROWS = 32
(measured on a 48 GB L40S); on a smaller GPU lower ROWS (not measured).Do not use Dodona to make decisions about people. In particular, under the EU AI Act, inferring emotions of people in the workplace or in education is prohibited. Dodona's emotion answers are about acted speech and are not evidence of anyone's state of mind.
Adapters, calibration and code: Apache-2.0. Base model: Qwen2-Audio-7B-Instruct (Apache-2.0). See NOTICE for the training data's licences.