Types of Disfluencies

Speech disfluencies are spontaneous disruptions in fluent speech. They include fillers, repetitions, restarts, elongations, and corrections. Each type serves a subtle communicative function — sometimes signalling hesitation, other times planning or emotional emphasis.

• Fillers: Common in almost every language, fillers such as “uh,” “um,” or “er” signal cognitive planning time. They allow speakers to hold their conversational turn while they think ahead. Some languages use unique equivalents — for instance, “ano” in Japanese or “eh” in Afrikaans — which makes multilingual data collection essential for inclusive modelling.
• Repetitions: These occur when speakers repeat words or phrases (“I, I just thought maybe…”). Repetitions can indicate uncertainty or a reorganisation of thoughts, but they can also occur due to speech motor control patterns, especially in fast or emotional speech.
• Restarts: A restart happens when a speaker abandons a phrase and begins again (“Can you – could you tell me the time?”). Restarts signal self-correction and are rich in cues about intent and conversational repair.
• Elongations and pauses: Extending sounds (“sooo… maybe”) or inserting pauses provides temporal space to maintain control of a conversation. In tonal languages, elongations may carry additional prosodic meaning, affecting how they are interpreted by automatic speech recognition (ASR) systems.

These variations are far from random. They are deeply tied to cognitive load, linguistic rhythm, and even social context. When training voice UIs, developers must decide whether to normalise, remove, or explicitly label these features depending on the application — for instance, whether the system is for dictation, voice search, or conversational AI.

Why They Matter

For humans, disfluencies are part of natural communication. For machines, they introduce ambiguity. The ability of a voice UI to handle such moments determines how natural and intuitive a conversation feels.

1. Turn-taking and timing: Disfluencies often signal a speaker’s intent to continue talking. Without proper modelling, a voice assistant might incorrectly interpret a pause or “um” as the end of a command, leading to early barge-in responses or interruptions.
2. Intent inference: In many cases, a disfluency indicates that the speaker is revising their intention. Consider “Play – actually, pause the music.” A system that recognises the restart pattern can adaptively ignore the first command and prioritise the revised one.
3. User frustration: Poor handling of hesitations can result in repeated queries, errors, or unnatural exchanges. If a user must consciously speak in a machine-friendly way, engagement drops sharply.
4. Accessibility and inclusion: Users with speech disorders, non-native accents, or anxiety often produce higher rates of disfluency. Accurately processing these patterns ensures that systems remain inclusive and equitable.

In short, modelling disfluency is not just a technical challenge — it’s a design imperative for building empathy into machine interaction. Systems that “listen” like humans do enable smoother conversations, better trust, and higher satisfaction.

Data Strategy

The success of disfluency handling begins with the data. Training a robust ASR or NLU (natural language understanding) model requires a careful balance between clean data and real-world speech complexity.

1. Labelling schemes: Annotators need clear guidelines on how to tag disfluencies. Should “uh” be labelled as a filler token or as background noise? Should a restart be marked as a self-repair boundary? Consistent annotation ensures that models learn the intended behaviour rather than noise patterns.
2. Preserving vs normalising disfluencies: In some pipelines, disfluencies are removed to improve word error rate (WER). In others — particularly for conversational AI — they are preserved to teach systems how to respond naturally. A hybrid strategy often works best: training ASR to recognise disfluencies but allowing NLU to filter them contextually.
3. Synthetic augmentation: Creating artificial data that includes controlled disfluencies helps balance datasets. For instance, generating versions of an utterance with inserted fillers or restarts can improve model robustness, especially when real-world disfluent data is limited.
4. Cross-lingual considerations: Disfluency patterns vary widely by language and culture. For example, code-switching contexts (common in South Africa and many multilingual regions) may include filler words from multiple languages within one sentence. Building cross-lingual disfluency models requires data sourced ethically from diverse speakers.

A sound data strategy therefore demands not only volume but nuance — capturing how people actually speak rather than how we imagine they should.

Modelling Approaches

Disfluency handling has evolved from basic pre-processing to integrated deep-learning models that treat disfluency as a feature rather than a flaw.

1. End-to-end ASR with disfluency tags: Modern ASR architectures, such as transformer-based models, can predict disfluency tokens alongside words. This allows the model to represent fillers explicitly while still maintaining transcription accuracy.
2. NLU post-processing: Alternatively, a secondary NLU layer can remove or interpret disfluencies once the speech is transcribed. For instance, a dialogue manager can learn to disregard a filler before executing a command. This separation of recognition and interpretation reduces confusion in downstream tasks.
3. Prosodic features: Beyond text, disfluencies often manifest in rhythm, pitch, and duration. Incorporating prosodic embeddings helps models distinguish between meaningful pauses and natural breathing.
4. Real-time constraints: Voice UIs must process input almost instantaneously. This limits the computational complexity of disfluency detection. Lightweight architectures or on-device models are often preferred to minimise latency while maintaining accuracy.
5. Adaptive learning: Some systems personalise their response based on individual speaking style. Over time, the assistant learns how a particular user hesitates or corrects themselves, improving predictive accuracy.

Each approach reflects a trade-off between computational efficiency, linguistic realism, and user experience. The frontier of disfluency modelling lies in blending these methods into systems that adaptively understand speech as fluidly as humans do.

UX and Metrics

Traditional ASR metrics such as word error rate are insufficient for evaluating performance in conversational systems. Disfluency handling demands a broader perspective — one that considers how users feel about the interaction and whether tasks are completed successfully.

1. Task completion rates: If a system can correctly interpret a disfluent command (“Set – uh, I mean cancel – the alarm”), the task completion rate rises even if the literal transcription contains errors.
2. Correction rates: How often must a user restate or clarify themselves? High correction rates usually point to poor disfluency handling.
3. User satisfaction: Subjective ratings, gathered through post-interaction surveys or behavioural analytics, reveal whether the voice UI feels “natural.” Users often describe successful systems as “understanding me even when I hesitate.”
4. Accessibility metrics: Tracking performance across diverse user groups — including non-native speakers and individuals with stutters or speech impairments — ensures equitable design outcomes.
5. Real-world robustness: Evaluations should measure performance in noisy or emotionally charged contexts, where disfluencies increase.

A voice UI that models hesitation intelligently becomes less of a tool and more of a companion — one that listens with patience and context-awareness.