AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Code-Switching Detection via Language-Selective Dimensions in Speech Foundation Models

Collected Oct 14, 2026

Apple Machine Learning Research published a paper titled "Code-Switching Detection via Language-Selective Dimensions in Speech Foundation Models" by Jie Chi, Zakaria Aldeneh, Masha Fedzechkina, Maureen de Seyssel and Natalie Schluter. The work targets code-switching (CS), the alternation between languages within a single utterance, and proposes a lightweight way to find CS utterances in large speech collections so they can be added to pretraining data.

The motivation is a data problem. CS is common in multilingual speech but underrepresented in pretraining corpora, and increasing its coverage means locating relevant utterances inside large speech collections. The obvious route is to transcribe the speech first and then detect language alternation in the text, but that route depends on obtaining accurate multilingual transcriptions, which is itself hard. The paper instead detects CS directly from speech.

The method uses frame-level language information already present in pretrained speech encoders to decide whether both languages appear within an utterance. The authors identify language-selective hidden-state dimensions using monolingual recordings, then aggregate the frame-level activations of those dimensions into a CS prediction. A small CS set is used to tune a decision rule. This suggests the approach treats the encoder's internal representations as a ready-made language signal rather than training a new detector from scratch, which is what keeps it lightweight.

For readers outside speech research, a few terms are worth unpacking. Speech foundation models such as Whisper and HuBERT are trained on large amounts of audio and produce hidden states, vector representations for short frames of audio, at each time step. A frame is a brief slice of the waveform. If some of those dimensions respond mainly to one language, then watching how they fire over time can reveal when a speaker switches languages. Aggregating at the frame level means combining those per-frame signals into a single utterance-level decision.

The headline result is a mean F1 of 0.798 across 14 language pairs in CS-FLEURS when using the Whisper-small encoder. For comparison, a CNN reached 0.776 and a HuBERT linear baseline trained for monolingual language identification reached 0.751. F1 is a combined measure of precision and recall, so higher is better; the gap over the HuBERT baseline, roughly 0.047, is the clearest evidence that a model pretrained with a language-identification objective does not capture the same frame-level language information that Whisper-small's encoder does. The authors conclude that pretrained speech encoders already encode the information needed to identify CS utterances, and that the method provides a lightweight tool for building more representative multilingual corpora.

The evaluation uses CS-FLEURS across 14 language pairs, with a small CS set reserved for tuning the decision rule rather than training the encoder. That design matters: no new large-scale training run is required, only the selection of dimensions from monolingual recordings and a rule fit on a modest amount of CS audio. In practice, that could make the approach attractive for teams that want to mine CS utterances from existing speech archives without building labeled CS corpora first. A likely trade-off is that results depend on the encoder and on the language pairs available; the reported numbers are averages across 14 pairs, so per-pair performance may vary more than the mean suggests. On the related work side, the same research group has explored Mandarin-English CS-ASR with iterative pseudo-labeling and earlier end-to-end speech translation for English/Spanish CS conversations.

Why it matters: teams curating multilingual speech training data currently face a choice between expensive multilingual transcription and manual listening, and this method offers a cheaper way to flag code-switched utterances. If the reported F1 holds up beyond the studied language pairs, the practical change is that dataset builders can scan large archives with a small, tuned decision rule layered on an existing encoder. A reasonable inference is that better CS coverage would feed back into multilingual ASR quality, though the paper reports detection performance rather than downstream transcription gains.

Read at Apple Machine Learning Research

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

Code-switching (CS), the alternation between languages within an utterance, is common in multilingual speech but underrepresented in pretraining corpora. Increasing CS coverage requires finding relevant utterances in large speech collections so that they can be included in training data. One way to find such utterances is to transcribe the speech and detect language alternation in the text, but this depends on obtaining accurate multilingual transcriptions. We propose a lightweight method that detects CS directly f