Nova-2 Streaming Language detection

[robfig]

It would be very convenient to support language auto-detection. We have customers that hold different meetings in multiple languages (e.g. English & Spanish), so even if we supported a per-account setting it would not be correct in general. I'd like to avoid requiring customers to select a language each time they turn on closed captioning.
Thank you
Rob

[team-deepgram]

Thanks for asking your question about Deepgram! If you didn't already include it in your post, please be sure to add as much detail as possible so we can assist you efficiently, such as:

• The request_id if you have a question about your requests or transcription responses.
• The features you used or the full api.deepgram.com URL you sent your request to, including parameters.
• Any code snippets you can share.

[jjmaldonis]

Hey @robfig, we do have support for language detection! You can find the relevant docs here: https://developers.deepgram.com/docs/language-detection

Just add detect_language=true as a new query parameter to enable language detection, and remove the language=... parameter.

The response will come back with additional information:

"detected_language": "fr",
"language_confidence": 0.0

For more advanced functionality, such as how to restrict languages can be detected, make sure to read to the bottom of the page.

[robfig]

Thanks! It looks like that's available for pre-recorded only. Is that correct? If so, any plans to make it work for streaming?

[jjmaldonis]

Ahh yes, it is currently only available for pre-recorded. We do have plans to support language detection for streaming. In the short term, there are a few possible workarounds depending on your use case:

1. Send the first X seconds of the stream to the pre-recorded endpoint to get the language. We typically recommend a minimum of 30 seconds of speech for accuracy language detection. Some languages are easier to detect than others, so this number varies depending on which languages can be detected.
   a. Useful for use cases like call centers where the transcript of the first 30 seconds of the call is not time-critical.
   b. The downside is this has a long latency, but it's a cost-effective solution.
2. Open multiple websockets with different languages and show the user the result with the highest confidence scores. Once you're confident the user is speaking a given language, you can close the other websockets.
   a. Useful for uses cases like voicebots where the response time needs to be fast.
   b. The downside is this is more expensive, but the latency is excellent.

[robfig]

Interesting! Good ideas, thank you. I think the first would work for us

[redders123]

@jjmaldonis we have a conversational AI for language learning, and previously tested Whisper vs Nova-2. The main drawback for Nova-2 was the inability for it to understand when you speak in English instead of your target language. We'd also consider switching if Nova-2 starts supporting multilingual usage. Also since this week, we've noticed that the Whisper model we run via Deepgram is often translating what we say in another language to English. This is not desired and we don't know why it's just started happening. Did you change the large Whisper model being used, perhaps updating to V3? I came across an article on your site suggesting V2 is better (https://deepgram.com/learn/whisper-v3-results). Trying to work out a solution.

[jjmaldonis]

Hey @redders123, starting a new thread here.

For readers, this answer is directed at those using Deepgram's pre-recorded endpoint rather than live streaming audio.

For the use case you described where multiple languages are spoken in the same call, our current recommendation is to identify areas of the call where the confidence is abnormally low. If 90% of a call can be transcribed with high confidence in the user's target language, then the other 10% of the call will have low confidence (assuming words are spoken). You can extract the portion of the call with low confidence and retranscribe it using the detect_language=true query parameter.

To answer your other question, Whisper will return a transcript in the language you specify with language=true independent of the language of the audio. So if you specify language=en, the transcript will be returned in English even if the audio is in another language. Nothing has changed with Whisper recently so my first guess is the language query parameter is now being used slightly differently in your app.

[redders123]

thanks a lot @jjmaldonis I checked with the dev team and we actually switched to OpenAI for Whisper (apologies) as it was working faster than Deepgram. Maybe that has changed recently (keen to switch back to Deepgram if it works well enough). We set the language to the target language e.g. Spanish, that's why it's so odd that it sometimes translates to English.

[TasseDeCafe]

Hi @jjmaldonis, I'd also be interested in being able to detect multiple languages either in a pre-recorded audio or in a stream. I'll try to implement the workarounds you provided, but please let me know if you implement a detect_language query in the streaming API, or some other solution that would considerable improve transcription of audio recordings in multiple languages (2 would already help a lot).

[janak2]

@jjmaldonis We also need to be able to detect multiple languages over streaming with low latency for a call center app. Adding detect_language to websocket would be great!

[fred-snapstream]

Would like to double down on the importance of auto lang detection for streaming in our use cases. And at NAB this year seeing lots of AI transcription / translation services which support language detection for live streaming.

[sb1992]

Folks just to add here, inability of language detection is a bummer. Especially when we working in german and a few english or spanish words in the same call give very poor results. As much as i love deepgram, is making me use azure just due to this feature.

[KryTisch]

same here! streaming recognition would be a must in the future!