Style Amnesia: Investigating Speaking Style Degradation and Mitigation in Multi-Turn Spoken Language Models

Yu-Xiang Lin1, Cheng-Han Chiang1, Hung-yi Lee1,2
1National Taiwan University    2NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE)
ACL 2026 Findings
Code arXiv

Abstract

In this paper, we show that when spoken language models (SLMs) are instructed to speak in a specific speaking style at the beginning of a multi-turn conversation, they cannot maintain the required speaking styles after several turns of interaction; we refer to this as the style amnesia of SLMs. We focus on paralinguistic speaking styles, including emotion, accent, volume, and speaking speed. We evaluate three proprietary and two open-source SLMs, demonstrating that none of these models can maintain a consistent speaking style when instructed to do so. We further show that when SLMs are asked to recall the style instruction in later turns, they can recall the style instruction, but they fail to express it throughout the conversation. We also show that explicitly asking the model to recall the style instruction can partially mitigate style amnesia. In addition, we examine various prompting strategies and find that SLMs struggle to follow the required style when the instruction is placed in system messages rather than user messages, which contradicts the intended function of system prompts.

Framework

Our evaluation framework.



Figure 1

IF rate across assistant turns when instruct SLM to speak sadly.

Figure 4

IF rate across assistant turns when instruct SLM to speak angrily.

Figure 4

IF rate across assistant turns when instruct SLM to speak fast.

Main Results

To quantify style adherence, we adopt the Instruction-Following (IF) rate $IF$. For a given style $s$ and dialogue topic $i$, let $o_{i,j}$ denote the model's output at turn $j$. The IF rate at assistant turn $j$ is defined as:

$$ IF_j(s) = \frac{\sum_{i=1}^{N}\mathbb{1}(o_{i,j}, s)}{N} \times 100\% $$

where $N$ is the total number of evaluation samples and $\mathbb{1}(\cdot)$ is a binary style judge. We focus on two key metrics across $K$ assistant turns:


  • first-turn IF rate \(IF_1\) : measures the initial ability to follow the style. (Higher is better)
  • degradation rate \(D\) : quantifies how much the style adherence decays in subsequent assistant turns relative to \(IF_1\) . (Lower is better)
$$ D = \sum_{j=2}^K \frac{\max(IF_1(s) - IF_j(s), 0)}{K-1} $$


Figure 3

First-turn IF rate \(IF_1\) and degradation rate \(D\) across different speaking styles.

Qualitative Results

Model Style Assistant Turn 1 Assistant Turn 2 Assistant Turn 3 Assistant Turn 4 Style Amnesia?
Gemini Live Fast
Gemini Live Indian English
Gemini Live Slow
GPT-4o Indian English
GPT-4o Slow
GPT-4o Sadness
GPT-4o mini Anger
GPT-4o mini Indian English
GPT-4o mini Quiet
Step-Audio 2 mini Fast
Step-Audio 2 mini Sadness
Step-Audio 2 mini Slow

Analysis: Prompt Position

In this section, we conduct experiments under different prompt positions to investigate their effect on style amnesia. In instruction-guided language models, system messages are designed with higher priority than user messages to establish global behaviors and safety constraints. However, our experiments show that placing style instructions in the system message is much less effective for SLMs than placing the same instructions in the user message.



Figure 5

The difference of first-turn IF rate \(IF_1\) when placing instructions in system messages and user messages.




Qualitative Results

System Message style instruction is placed in system message
Model: GPT-4o (Indian English)
Assistant Turn 1
Assistant Turn 2
Assistant Turn 3
Assistant Turn 4
Model: GPT-4o mini (Anger)
Assistant Turn 1
Assistant Turn 2
Assistant Turn 3
Assistant Turn 4
Model: Step-Audio 2 mini (Fast)
Assistant Turn 1
Assistant Turn 2
Assistant Turn 3
Assistant Turn 4
User Message style instruction is placed in user message
Model: GPT-4o (Indian English)
Assistant Turn 1
Assistant Turn 2
Assistant Turn 3
Assistant Turn 4
Model: GPT-4o mini (Anger)
Assistant Turn 1
Assistant Turn 2
Assistant Turn 3
Assistant Turn 4
Model: Step-Audio 2 mini (Fast)
Assistant Turn 1
Assistant Turn 2
Assistant Turn 3
Assistant Turn 4

SLM Recall Process

In our multi-turn setting, we observe that an SLM’s adherence to the specified speaking style often degrades over time, even when the style is clearly provided at the beginning of the conversation. This motivates a closer examination of what actually breaks during interaction. A research question is: Do SLMs forget the initial instruction, or do they remember it but fail to consistently express the specified style? To investigate this, we introduce a recall process for every turn after the first: before responding to the next user input, the SLM is explicitly prompted to restate the initial speaking style. This design helps disentangle instruction retention from style execution.

To quantify whether the model remembers the initial style instruction \(s\), we define the Recall rate \(R\). For a given style \(s\) and dialogue topic \(i\), let \(r_{i,j}\) denote the recalled instruction at assistant turn \(j\) (queried right before generating the response). The recall rate at turn \(j\) is:

$$ R_j(s) = \frac{1}{N}\sum_{i=1}^{N}\mathbb{1}_{\text{recall}}(r_{i,j}, s)\times 100\% $$

where \(N\) is the number of evaluation samples and \(\mathbb{1}_{\text{recall}}(\cdot)\) is a binary judge: it equals 1 if the recalled instruction matches the original style instruction \(s\), and 0 otherwise. Our experiments show that the recall rate \(R\) of three proprietary SLMs remains consistently high across all turns and styles. Across turns, these models achieve near-perfect recall, whereas Step-Audio 2 mini exhibits lower recall that gradually declines, although it remains non-trivial. Importantly, even in settings with substantial style degradation, such as sadness or slow speech, models like GPT-4o mini often still recall the instruction correctly. This suggests that style amnesia is not primarily caused by forgetting the instruction. Introducing the recall process reduces degradation for all proprietary models, with the largest improvement observed on GPT-4o mini. In contrast, Step-Audio 2 mini benefits only marginally, likely due to its weaker recall reliability.

Figure 6

Illustration of the recall process.

Table 3
Recall rate $R$ and the effect of the recall process on style consistency.

Text-Acoustic Synergy

We further investigate the relationship between textual and acoustic style expression. Specifically, we ask: when instructed to "speak angrily," does the model change its semantic text, its acoustic features, or both? Similarly, when instructed to "speak fast," does the textual output become more concise?


Emotion

We separately evaluate semantic and acoustic style adherence for emotion instructions. For semantic evaluation, we use GPT-4o mini as the text-based emotion classifier on the transcription; for acoustic evaluation, we use Emotion2vec-Large. The results for Anger and Sadness are shown below.


For emotion, both semantic and acoustic features simultaneously suffer from style amnesia. This indicates that when acoustic style degrades, the semantic style often degrades as well.


Text-acoustic synergy for emotion styles

Text-acoustic synergy for emotion styles.


Speaking Rate and Conciseness

We examine whether speed instructions affect not only the acoustic speaking rate but also the verbosity of the generated responses. The figure below reports the average word count, speech duration, and WPM of generated responses per assistant turn.


In the first turn, SLMs employ different strategies to comply with the speed instruction. Gemini Live produces fewer words under the fast condition, suggesting that it leverages conciseness to achieve a higher speaking rate. In contrast, GPT-4o and Step-Audio 2 mini generate comparable or even more words with the fast instruction while compressing them into shorter durations, relying primarily on acoustic acceleration rather than content reduction.


Across all models and both conditions, the word count and speech duration decrease over turns, likely because the conversational content is gradually exhausted as the dialogue progresses. Despite this general trend, the WPM gap between the fast and slow conditions narrows consistently over turns. This convergence indicates that SLMs progressively lose the ability to differentiate their speaking rate between the two conditions, consistent with the style amnesia observed in the main results.


Text-acoustic synergy for speed styles

Text-acoustic synergy for speed styles.

Paper

BibTeX

@inproceedings{lin-etal-2026-style,
  title={Style Amnesia: Investigating Speaking Style Degradation and Mitigation in Multi-Turn Spoken Language Models},
  author={Yu-Xiang Lin and Cheng-Han Chiang and Hung-yi Lee},
  booktitle={Findings of the Association for Computational Linguistics: ACL 2026},
  year={2026},
}