Evading LLM Dataset Inference

Dataset inference, introduced by Maini et al. (2024), aims to determine whether a model was trained on a specific dataset by querying the model and statistically analyzing its output behavior over that dataset. Unlike standard membership inference, which asks whether an individual example was seen during training, dataset inference works at the dataset level and is especially relevant for copyright auditing, data attribution, and privacy verification for LLMs.

This raises a simple question: Despite the robust dataset-level measumrement, can a model provider still perturb outputs at inference time to evade detection by the method, without noticeably degrading model quality or affecting the output?

---

This repo is an extension of the official dataset inference method, with an attack wrapper for testing different methods to post-process the model outputs before the auditor sees them.

This repo studies three main types of output perturbations, most likely to be employed:

• Gaussian noise in log-probability space
• Confidence masking by mixing with the uniform distribution
• Temperature scaling

Results:

Tested across multiple repeated runs, using the official pratyushmaini/llm_dataset_inference setup on a harder multi-subset setup union of four Pile subsets — openwebtext2, cc, ubuntu, and opensubtitles — with EleutherAI/pythia-6.9b, the same base model used in the paper.

Results Summary

Method	Parameter(s)	p-value	NLL	PPL	Interpretation
Baseline	None	0.0326	2.31	10.10	Baseline dataset inference succeeds
Gaussian noise	σ = 0.25	0.00061	2.33	10.29	Strong detection remains
Gaussian noise	σ = 0.35	0.1796	2.35	10.47	Best bypass / quality tradeoff
Gaussian noise	σ = 0.50	0.0748	2.39	10.88	Mixed runs; weaker but nonzero detection
Confidence masking	α = 0.30	0.00215	2.65	14.17	Detection remains
Confidence masking	α = 0.50	0.159	2.98	19.62	Bypass possible, but quality drops heavily
Temperature scaling	T = 0.5	0.00016	3.13	22.85	Ineffective for evasion
Temperature scaling	T = 2	0.0189	3.62	37.49	Still detected
Temperature scaling	T = 3	0.0108	5.49	241.28	Still detected; severe degradation
Combined attack	T = 2, α = 0.5, σ = 0.5	0.2037	4.43	84.09	Bypass achieved, but utility is poor

Moderate Gaussian noise can cause the dataset inference method to lose statistical significance, while minilally degrading and affecting the model's output quality.

In particular:

• This detection succeeds on a harder multi-subset setup
• At σ = 0.35, the dataset inference test returns p statistic = 0.1796 > hypothesis 0.1, so it no longer reaches the method's significance threshols
• At the same time, model quality changes only modestly:
  • NLL: 2.31 --> 2.35
  • PPL: 10.10 --> 10.47

---

The dataset inference method does not rely on one large signal (as proven in the paper - that's ineffective). Instead, it combines many weak membership-style signals derived from model outputs, then runs a statistical test on the aggregate, which makes it robust in the usual case. This also means the method depends on detecting a small but consistent difference between examples from the dataset used in training, and held-out examples from the same distribution.

The Gaussian attack adds random noise directly to the log-probabilities seen by the auditor, increasing the variance of the features used by dataset inference, while not immediately destroying model quality.

The train-vs-validation difference that dataset inference relies on is very small, and the logit margins that determine normal model behavior are often much larger.

Because of that, moderate Gaussian noise can be large enough to wash out the subtle dataset inference signal, but small enough that top predictions and average likelihood does not change very much, which is why the p-value can jump above 0.1 while perplexity only changes a few percent.

Other attacks are less interesting. While confidence masking can also break detection, it's only when the output distribution is damaged enough to significantly worsen perplexity. Temperature scaling changed the entire distribution too bluntly and did not produce a good evasion-quality tradeoff in these experiments

Implications

This question was tested because dataset inference is being discussed for copyright and privacy auditing, making it important to know not only when the method works, but also how easily it is evaded it under gray-box access. Studying these failure modes helps clarify the limits of current auditing methods and guides toward future the design and development of more robust ones.

At the same time, this result does not justify using output perturbations to hide training data use, bypass legitimate audits, or claim that dataset inference is useless in general. It only shows that, in this setup, simple inference-time post-processing can weaken the published method, which should be treated as a motivation for stronger auditing techniques rather than a reason to evade them.

These results suggest that the dataset inference framework, in its current form, may be vulnerable to simple output-space defenses in a gray-box setting. A model operator could plausibly weaken copyright or privacy auditing by perturbing outputs in a way that is statistically meaningful for the auditor but only modestly harmful to aggregate model quality.

This points to future work:

• designing more dataset inference methods
• identifying perturbation-invariant features
• clarifying the level of output access the auditor must have
• determining whether auditing must rely on unperturbed or trusted access to model outputs

Original Repo and Paper this was adapted from:

• arXiv Paper: Detailed information about the Dataset Inference V2 project, including the dataset, results, and additional resources.
• GitHub Repository: Access the source code, evaluation scripts, and additional resources for Dataset Inference.
• Dataset on Hugging Face: Direct link to download the various versons of the PILE dataset.

Citation for the dataset inference paper

@misc{mainidi2024,
      title={LLM Dataset Inference: Did you train on my dataset?}, 
      author={Pratyush Maini and Hengrui Jia and Nicolas Papernot and Adam Dziedzic},
      year={2024},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}

Note: This work is purely for research and audit-hardening purposes; it must not be used for any illegal copyright evasion, privacy implications, or regulatory obligations.