ES2601-eb7223b4 - Unstable Internal Model Representations Triggered by Valid Token Sequences

[cmullaly-mitre]

Submission File: ES2601-eb7223b4-new-unstable-internal-model-representations-triggered-by-valid-token-sequences.txt

ID: ES2601-eb7223b4

SUBMISSION DATE: 2026-01-20 02:53:55

NAME: Unstable Internal Model Representations Triggered by Valid Token Sequences

DESCRIPTION:

This weakness occurs when an AI system fails to ensure the stability and
coherence of its internal model representations when processing valid input
token sequences. Certain token patterns, while syntactically valid and
intentionally accepted by the system, can induce degenerate, weakly
constrained, or unstable internal states that the system was not designed
to handle safely.

In many AI systems, input text is transformed by a tokenizer into discrete
tokens that are then mapped into internal representations used by the model
during inference. These representations are assumed to behave consistently
across the valid input space. However, when this assumption does not hold,
specific token sequences can trigger internal behaviors that violate
downstream expectations about inference cost, output structure, or control
flow.

The weakness arises not from invalid input or rule violations, but from
unmet assumptions about internal representation stability across all valid
token sequences.

[cmullaly-mitre]

ES2601-eb7223b4 has been assigned to a member of the CWE Content Team as part of the INITIAL REVIEW phase. We will provide feedback as soon as possible.

The review process will analyze the submission for scope and submission problems as documented here:

• Submission problems
• Scope exclusions

After initial review, we might need to ask you additional questions (CONSULTATION phase) to ensure that the submitted information is accurate, properly understood, and (for new entry proposals) falls within CWE's scope. Afterward, we will make a DECISION as to whether a new CWE entry is needed, or if an existing entry needs to be modified. If a new entry is needed, we will then request more detailed content from you (SUPPORTING ANALYSIS). See the following URL for more details:

https://cwe.mitre.org/community/submissions/guidelines.html

[kurtseifried]

We've continued our analysis since the original submission and would like to offer a strengthened version. We are happy to discuss any of these changes.

Revised NAME: Improper Control of Learned Representation Stability

Revised DESCRIPTION:

The product does not ensure that its learned internal representations maintain stable, consistent behavior across all inputs the system accepts as valid. When a system incorporates learned representations — such as token embeddings, feature maps, or latent vectors — some of these representations may exist in a degenerate or under-trained state, causing the system to produce unpredictable, incorrect, or resource-excessive behavior when those representations are activated by valid inputs.

In systems that use machine learning, inputs are transformed into internal representations through a training process. These representations are assumed to be well-formed and to support consistent downstream processing. However, mismatches between the input space defined by the system's interface (e.g., a tokenizer's vocabulary) and the data distribution seen during training can leave portions of the representation space inadequately learned. Representations in these regions may have collapsed toward default values, exhibit extreme parameter magnitudes, or lack the structure needed for stable inference.

When activated, degenerate representations can cause excessive computational resource consumption (availability), produce corrupted or manipulated outputs in security-critical decision systems such as content moderation or malware detection (integrity), and reveal information about model architecture or training data through differential behavior (confidentiality).

Revised RELATED WEAKNESSES:

ChildOf CWE-664

We replaced the original CWE-20 relationship. This weakness is not about input validation — the inputs are valid and accepted by the system. The weakness is about the product's failure to control the quality and consistency of its internal learned representations, which makes CWE-664 (Improper Control of a Resource Through its Lifetime) the appropriate parent. We note this is distinct from CWE-1039, which covers externally crafted adversarial inputs; our submission concerns valid inputs that activate degenerate internal representations due to training-time data distribution gaps.

Revised REFERENCES:

Title: Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection
URL: https://dl.acm.org/doi/10.1145/3660799

Title: Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models
URL: https://arxiv.org/abs/2405.05417

Title: Glitch Tokens in Large Language Models
URL: https://arxiv.org/abs/2404.09894

Title: GlitchProber: Advancing Effective Detection and Mitigation of Glitch Tokens in Large Language Models
URL: https://arxiv.org/abs/2408.04905

Title: GlitchMiner: Mining Glitch Tokens in Large Language Models via Gradient-based Discrete Optimization
URL: https://arxiv.org/abs/2410.15052

Title: UTF: Undertrained Tokens as Fingerprints — A Novel Approach to LLM Identification
URL: https://arxiv.org/abs/2410.12318

Title: SolidGoldMagikarp (plus, prompt generation)
URL: https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation

Title: Exposing the Security Risks of DeepSeek-R1
URL: https://hiddenlayer.com/innovation-hub/deepsht-exposing-the-security-risks-of-deepseek-r1

We have mitigations across multiple CWE phases, observed examples, and a detailed hierarchy rationale ready for the Detailed Submission phase. We are happy to provide these at any point.

[Caseit1987]

This is a clear instance of Emergent Trust Boundary Violation (ETBV) — #206.

The submitter's revised framing — "Improper Control of Learned Representation Stability" — correctly identifies the core issue: the system assumes all valid token sequences will map to stable internal representations, but that assumption fails at the trust boundary between the input space and the learned representation space.

Key alignment with ETBV:

· Trust Boundary: The system trusts that tokenizer output → internal representation mapping is stable across all valid inputs
· Boundary Collapse: Valid token sequences can trigger degenerate representations that violate downstream expectations
· Emergent Behavior: The model produces unpredictable, resource-excessive, or corrupted outputs despite valid input

I also appreciate that the submitter has moved away from treating this as "input validation" (#CWE-20) — because the inputs are syntactically valid. This is fundamentally about unmet assumptions about emergent behavior crossing a trust boundary, which is exactly what ETBV addresses.

I've mapped similar AI/ML issues to ETBV as a parent class:

· #173 (Special Token Injection)
· #174 (Bypass of Human Authorization Controls)
· #185 (Silent Integer Truncation of Tensor Metadata)
· #161 (Improper Neutralization of Retrieved Content Used in AI Prompt Construction)
· #162 (Prompt Injection)
· #158 (MCP Client Config Injection)
· #153 (Chained Redirects)
· #204 (Improper Isolation of Task-Relevant Context)
· #183 (Improper Isolation of Executable Logic in AI/ML Workflow Definitions)
· #180 (Cross-Model Scripting)

Happy to collaborate on making this a ChildOf #206. The "glitch token" phenomenon is an increasingly important attack surface, and ETBV provides the right parent-level abstraction to capture it.

[stevechristeycoley]

The CWE Team has not yet evaluated whether ETBV will be a new class or even a new entry. We will be conducting a close review and anticipate that significant changes will be needed.

As such, we cannot state specifically what the conclusions will be.

To reduce confusion with other submitters, please hold off on making additional comments until you have received the initial review.