Speech prosody across languages: what transfers and what breaks

Prosody support splits into three layers: transcription, conversational recognition with turn detection, and speech synthesis. The Nova-3 transcription model, Flux Multilingual model, and Aura-2 synthesis model cover one layer each.

Which languages appear in each layer determines each market's support level for your voice agent, either full conversation or transcription alone. Nova-3 now transcribes Mandarin, Cantonese, and Thai, and Flux Multilingual covers ten languages with more planned, so tone languages are a transcription-layer decision today.

In a tone language, pitch carries word meaning, and multilingual speech recognition models learn that job during training. In this guide, you'll map pitch behavior to model coverage, then see which target languages converse and which only transcribe.

These points determine where each language lands, in the conversational layer or the transcription layer only.

• Nova-3 transcribes Mandarin, Cantonese, and Thai, but Flux Multilingual's ten languages include no tone language, so those markets transcribe without conversing.
• Turn detection and lexical prosody may depend on overlapping F0 information.
• Politeness register lives in delivery, and its acoustic cues can reverse across languages.

Three models mean three language lists. Miss one and your agent ships into a market it can't serve.

Each model owns one layer, so compare layer fit, not model quality. Seven operational criteria separate the layers.

Choose Nova-3 for transcription, Flux Multilingual for conversational recognition, and Aura-2 for multilingual synthesis, then confirm their coverage overlaps.

A tone language assigns a contour to every syllable, while a pitch-accent language marks one location per word. A model resolves the system it trained on, not the other. Mandarin has four such contours, and a Japanese word derives its whole melody from that one location.

The difference sets how much of the pitch trajectory a multilingual speech recognition model must resolve, and at what granularity, before it can commit a word.

Change a Mandarin syllable's pitch contour and you change the word itself. Four contours applied to the same segments produce four different words. A University of Illinois resource gives the classic set, where mā with a high level tone means 'mother' and má with a rising tone means 'hemp'.

A low-dipping mǎ means 'horse' and a high-falling mà means 'scold'. In Chao's five-level notation, those contours are 55, 35, 214, and 51. A model that misreads one syllable's trajectory returns the wrong word. On Deepgram's stack, Mandarin transcription landed before conversation did.

Unlike Mandarin's syllable-level contours, one accent location shapes a Japanese word. Kawahara's handbook chapter puts it directly. Know where the accent sits, and the word's whole tonal contour follows. A model that misplaces the accent locus by one mora can return the wrong word.

Because the two systems are learned, not configured, the distinction decides which layer can serve a language. Resolving either system takes training data with per-syllable contours or single accent loci, and vocabulary swaps and runtime flags can't supply it. If you've tried fixing a tone error with a keyword list, you know how that goes.

Some languages mark emphasis on the phrase head, others cue it with boundary tones at phrase edges, and some do both. So when you port English-style focus handling to Korean, you're porting the grammar that controls focus.

English concentrates emphasis into a single stressed syllable's pitch and loudness, and deaccents everything after it. For a model, that's convenient, because emphasis shows up as a localized, detectable pitch event on one word.

Instead of accenting a focused item, Korean restructures phrasing. A focused word starts a new accentual phrase, and everything after it is dephrased or squeezed into a reduced pitch range. In Japanese, a 2025 study in Languages found silence and duration were the strongest predictors of narrow focus. They operated independently of pitch accent.

If your agent equates emphasis with pitch prominence, it misses focus in languages that cue emphasis at phrase edges. Listener expectations differ too. A 2025 JIPA study found German learners rated post-focal words as less prominent, while Italian listeners heard them as equally prominent to broad-focus words. So the same prominence pattern can land as emphasis in one market and as flat delivery in another, which puts native listeners in your test plan.

For formal interactions, tempo and timing shape how respectful speech sounds alongside honorific verb endings. Loudness does too. A synthesis voice can produce every polite word correctly and still land wrong.

Formal and informal speech differ acoustically even when the words are identical. A 2025 Linguistics Vanguard study found Korean deferential speech carries lower pitch and lower intensity, while Japanese deferential speech shows higher pitch and higher intensity.

The same team's interactional data adds that the Korean quieting concentrates on final syllables. A register model trained on one language's polite acoustics risks inverting the intended cue in the other.

Keigo lives in pitch and intensity as much as in word choice. Correct honorific verb endings delivered with mismatched acoustics send mixed signals. A transcript keeps the wording and loses the delivery, so text alone won't tell you whether the register landed. On the synthesis side, default prosody can undo politeness you got right in the words.

Training determines per-language prosody. The Aura-2 language expansion puts it plainly. Aura-2 "adapts to each language's unique phonology, whether it involves pitch accents, liaisons, or compound stress rules."

For Japanese specifically, the post calls out heavy reliance on pitch accent. Judge the output directly because Deepgram leaves the mechanism unpublished. For a register-sensitive market, listen for tempo, voice quality, and vocabulary in formal contexts.

In a live tone-language conversation, pitch has to carry word identity and signal end of turn at once, and the model has to decide which job it's reading. The conflict limits real-time multilingual speech recognition more than raw language counts do.

Pitch contributes to turn prediction in Mandarin, while lexical tone also uses F0. Both jobs read the same track, so a model tuned to endpoint on it is competing with the cue that identifies the word. English endpointing can lean on pitch freely, since it carries no lexical load there.

Deepgram's own engineering notes show how sharp the trade is even without tone. An early Flux training curriculum leaned too heavily on silence as the signal that a turn had ended. Silence is treacherous too. Formal speech runs slower and leaves longer pauses, so a silence-based endpoint risks firing mid-turn on precisely the most polite callers.

On two Mandarin benchmarks, additional right context improved streaming recognition. A 2024 streaming ASR study found character error rate keeps improving until about 256 ms of lookahead on AISHELL-1 and around 448 ms on AISHELL-2. Those figures are benchmark- and model-specific. A conversational model has to commit before that window closes, while a transcription-only pipeline can usually tolerate the wait.

Deepgram reports mean Word Error Rate improvements in batch and streaming across supported languages, with the largest gains on code-switching datasets. The Nova-3 Multilingual update came from training-side fixes, mainly curriculum ordering and data curation. The prosodic load differs by pair, which is inference rather than measurement.

An English-German switch stays inside one system, while a Mandarin-English switch flips pitch's job mid-utterance from lexical to intonational.

The three layers don't cover the same languages, and the conversational layer is the narrow one. Nova-3 transcribes Mandarin, Cantonese, and Thai in batch and streaming. Flux Multilingual runs on ten languages, none of them tonal: English, Spanish, French, German, Hindi, Russian, Portuguese, Japanese, Italian, and Dutch.

Flux TTS went generally available on August 12, 2026, and its voices speak English only, so multilingual synthesis stays on Aura-2. Aura-2 synthesizes Japanese, which therefore clears all three layers, while the tone languages clear transcription only.

Sort your target languages by what pitch does in them before you scope a single integration. The coverage question is per layer, and for an agent the conversational one decides it.

Put each target market into one of three groups, then check it against each model's approved footprint. Mandarin uses pitch lexically, as do Cantonese and Thai. Japanese fixes one accent locus per word. English and European languages use pitch intonationally. The tone-language group carries the dual-use pitch signal traced above, and it currently sits in transcription only.

A language can clear one check and fail the other, so run both. If it appears only in the transcription layer, use it for transcription workloads such as analytics and captioning. It can also produce compliance records. Conversational deployment needs multilingual speech recognition trained for turn detection in that language, and vocabulary features can't substitute for that training.

Run your hardest audio through the supported layers and let the transcripts argue. Try Nova-3 on your tone-language audio and Flux Multilingual on the ten it covers, then compare turn behavior against your current stack. Create a free account and put your $200 free credits toward finding out which of your languages converse and which only transcribe.

Keyterm Prompting sharpens vocabulary rather than prosody, so tone accuracy comes from the model's training. It boosts recognition of domain-specific terms, up to 500 tokens per request, on both Nova-3 and Flux. Test your term list on supported audio, and judge vocabulary and prosody separately.

It only works on flux-general-multi and only spans the languages that model already covers. It won't add prosodic coverage the model wasn't trained for. Within those bounds, the language_hint parameter biases Flux Multilingual toward specific languages when automatic selection is uncertain.

No. Prosody handling belongs to the model, which behaves the same self-hosted as in the cloud. What changes is which models you have, since the hosted and self-hosted catalogs differ. Flux requires its own Engine node, so plan separate nodes for transcription and conversational workloads.

Crowdsource it. Panels of native-language raters are cheap to assemble and enough for a register check. For close comparisons, play two samples back to back and ask which sounds more appropriate, rather than scoring each one alone.

Treat it as a separate production-readiness decision. Validate transcription, turn behavior, and synthesis independently, and keep the language out of production until every required layer passes your acceptance tests.