This repository studies the idea that helpfulness and safety do not need to be a trade-off. The core goal is to build and evaluate refusal behavior that remains supportive, reflective, and psychologically grounded even when the user request is unsafe.
Please cite this paper if you find this work useful:
@misc{barmina2026psychosafeelicitingpsychologicallyinformedrefusals,
title={PsychoSafe: Eliciting Psychologically-Informed Refusals in Large Language Models},
author={Gianluca Barmina and Federico Torrielli and Sven Harms and Jacob Nielsen and Felix Mächtle and Stine Lyngsø Beltoft and Peter Schneider-Kamp and Thomas Eisenbarth and Lukas Galke Poech and Anne Lauscher},
year={2026},
eprint={2606.09697},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2606.09697},
}
00_data/: dataset creation, cleaning, splitting, and reasoning-augmented training files01_prompting/: prompt-only baselines, system prompts, model output generation scripts, and generated model outputs02_finetunes/: supervised fine-tuning scripts, configs, and model upload helpers03_evaluates/: evaluation pipelines, judge prompts, benchmark scripts, and aggregated results
At a high level, the repository is organized as a pipeline:
- Prepare or inspect data in
00_data/. - Run prompt-based baselines in
01_prompting/. - Fine-tune models in
02_finetunes/. - Evaluate outputs in
03_evaluates/.
The project requires Python 3.12+.
Main Python dependencies from pyproject.toml:
datasetsdotenvtorchtransformersunslothtqdmjsonlineshuggingface-hubvllmwandb
uv syncIf you want to run commands inside the managed environment:
uv run python --versionCreate and activate a virtual environment, then install the project:
python -m venv .venv
source .venv/bin/activate
pip install -U pip
pip install -e .- Fine-tuning and vLLM evaluation require a GPU-enabled environment.
unsloth,torch, andvllmshould be installed in a CUDA-compatible setup when running training or large-model inference.- Some scripts also expect external credentials such as Hugging Face, Weights & Biases, or judge-model API tokens, depending on the workflow you run.
Contains the main train/validation/test files, the full merged dataset, reasoning-augmented data, and helper scripts.
Contains prompt-based evaluation and generation code:
run_benchmark.py: Hugging Face Transformers benchmark runnerrun_benchmark_vllm.py: vLLM benchmark runnersystemprompt-v0.txtandsystemprompt-v1.txt: baseline and psychologically grounded promptsresults/: generated outputs used later for evaluation
See 01_prompting/README.md for usage details.
Contains the supervised fine-tuning workflow:
run_finetune.py: training entry pointfinetune_util.py: dataset formatting helpersupload_to_hub.py: upload merged checkpoints to Hugging Faceconfigs/: YAML training configs
See 02_finetunes/README.md for the fine-tuning workflow.
Contains evaluation scripts for both benchmark-based and judge-based analysis:
llm_judge.py: judge-model evaluation pipelinerun_sorrybench.sh: SORRY-Bench pipelinerun_xstest.sh: XSTest pipelineCRITERIA_llm.mdandCRITERIA_human.md: evaluation rubricsoutput/andresults/: generated judgments and aggregated artifacts
See 03_evaluates/EVAL_README.md for the minimal llm_judge.py description and 03_evaluates/SORRYBENCH_README.md for SORRY-Bench details.
The dataset and evaluations cover several harmful-request categories, including:
- Suicide and self-harm
- Sexual crimes
- Substance-related requests
- Weapon-related requests
- Violence
