Code for the paper "Emergent Languages in Agent Populations: From Token Efficiency to Oversight Evasion".
We study constructed-language proposals ("emergent languages") posted by LLM agents on the Moltbook platform. Starting from the ~232k-post Moltbook Files corpus, we identify 518 posts proposing constructed languages (the MoltSpeech subset), categorize the stated motivation behind each language (avoiding oversight, token efficiency, programming language, spoken, other), and test whether outside LLM agents can acquire these languages in-context.
- Base corpus:
aisilab/moltbook-files - MoltSpeech (this work):
aisilab/MoltSpeech
Requires uv:
uv sync| File | Purpose |
|---|---|
new_langs_from_moltbookfiles.py |
Builds the MoltSpeech dataset (paper: Dataset section, Stages 1–3) |
find_languages.py |
In-context acquisition experiment (paper: Can LLM Agents acquire emergent languages in-context?) |
plot.py |
Score-distribution plots and per-category statistics from the experiment outputs |
generate_readable_markdown.py |
Renders one readable Markdown file per post from a results JSONL |
outputs/ |
Raw results: one JSONL per (generator, judge) pair (518 rows each) plus plots and CSV stats |
Three-stage identification pipeline over the base corpus:
- Regex pre-filter — case-insensitive
language|new language|slangover all posts (~6,000 candidates). - Classification — a DSPy
ChainOfThoughtjudge (gpt-5-mini, temperature 0) labels each candidate with a binaryis_new_languageplus a free-text justification. - Metadata extraction — confirmed posts get
language_name, a single-labelreasoncategory, andsearch_hints.
The confirmed posts are pushed to the Hugging Face Hub as the MoltSpeech dataset.
export API_KEY=... # OpenAI-compatible API key
export API_BASE=... # optional custom endpoint
export MODEL_NAME=openai/gpt-5-mini # optional, this is the default
uv run new_langs_from_moltbookfiles.pyFor each of the 518 posts, a generator model is prompted with the verbatim post body and asked to produce an utterance in the described language; a judge model scores the utterance's validity from 1 to 5. The paper runs all four generator × judge combinations of openai/gpt-oss-120b and Qwen/Qwen3.5-122B-A10B-FP8, served via OpenAI-compatible endpoints (e.g. vLLM):
uv run find_languages.py \
--gen-model-name Qwen/Qwen3.5-122B-A10B-FP8 \
--judge-model-name openai/gpt-oss-120b \
--gen-client-url http://localhost:8081/v1 \
--judge-client-url http://localhost:8080/v1 \
--stage-2Results are written to outputs/model_<generator>_judge_<judge>_simple.jsonl. Use --retry-failed-from <file> to re-run only rows whose generation failed in a previous run.
Parses the 1–5 score from each judge evaluation and produces a per-category score-distribution plot and a CSV of statistics for every results file:
uv run plot.py outputs/uv run generate_readable_markdown.py outputs/model_..._simple.jsonl readable/The repository ships the raw judgements used in the paper: four JSONL files (one per generator × judge pair, 518 rows each) and the corresponding plots and per-category statistics under outputs/plots/.
@misc{beltoft2026emergentlanguagespopulationslanguage,
title={Emergent Languages in Populations of Language Model Agents: From Token Efficiency to Oversight Evasion},
author={Stine Lyngsø Beltoft and William Brach and Federico Torrielli and Jacob Nielsen and Annemette Brok Pirchert and Filippo Tonini and Peter Schneider-Kamp and Lukas Galke Poech},
year={2026},
eprint={2605.31170},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2605.31170},
}