What should I know before using multilingual source ASR in live Services and Events?

Multilingual source language support in ASR transcription is a new feature that can be used in Services and Events. There are certain behaviors and limitations that you should be aware of to ensure a successful service or event. Many of these limitations are specific to the Soniox engine which is the only speech engine supporting multilingual source at this time. We expect to expand options in the near future and will update this article as new developments arise.

Key limitation 1 - The Soniox speech engine will not restrict which languages are transcribed. If any of the supported 60 languages are identified, it will transcribe in those languages. This applies regardless of the language selected - a single language, multilingual source in auto mode, or multilingual with a set of expected languages.

Key limitation 2 - When Soniox speech engine is used in combination with Soniox translation, source language sentences identified for a language won't be displayed in a translation output with the same language. For example, in an event using Multilingual source with English and French translations - any English spoken in the source will only show up in the Multilingual output and not in the English translation.

Key limitation 3 - Multilingual source only supports display of languages written in left to right direction. Arabic, Hebrew and other right to left languages may be detected but their output will be displayed incorrectly in the left to right direction. We hope to address this in a future release.

Language support

Multilingual source is able to detect and transcribe speech in 60 different languages. It can switch between languages seamlessly, typically at a sentence level. Multilingual source defaults to automatic language detection.

In Events, you must choose Soniox as the speech engine before selecting Multilingual as the source language, in the Events > Settings tab.

Expected languages

We strongly recommend using the expected languages feature to give the speech engine hints about the languages expected in the source for best results. Even if you are uncertain about the exact mix of languages expected you should specify two or more dominant languages expected in the source.

For example, if you expect your event to include a mix of English, German and Japanese speech, you should select these as expected languages. As noted above, this won't prevent speech in other languages from being transcribed.

608 and DVB-TXT support

Multilingual source can be used with 608 and DVB-TXT outputs. We recommended only using these outputs if you know that the languages expected in the source have character sets that are supported in 608 and DVB-TXT. See more information about supported languages and character sets in 608 and DVB-TXT.

For example, multilingual source might be used for a program where a mix of English and Spanish is expected and would work well for 608 and DVB-TXT. If Japanese dialogue happened to come up its characters would be unsupported (showing up as 'empty boxes') in both the 608 and DVB-TXT outputs.

Labeling Multilingual source in Events

You can set a custom label for the multilingual source in Events pages and widgets. The default label is 'Multilingual' but can be set in the Event > Settings tab after Multilingual source has been selected. Other popular options include - Original, Source, Source languages, Floor language, etc.

Dictionaries and Postprocessing Dictionaries

At this time, dictionaries and postprocessing dictionaries cannot be created for multilingual source language at the organization settings level. We plan to support this in the near future.
Dictionaries can be used in multilingual source ASR Events and must be authored in the event where they will be applied. You can use a mix of terms across expected languages in different scripts. Dictionaries for multilingual source events only support languages written in the left to right direction, like English and not right to left languages like Arabic.

Glossaries

Glossaries can be authored and used with multilingual source ASR Events and Services. You can use a mix of terms across expected languages using different scripts, however it is important to note that glossary terms are weighed alongside other factors when producing translations and mixing of terms across languages and different scripts may reduce the effectiveness of a supplied glossary or produce unexpected results. Consider use of glossaries with multilingual text to be experimental.