Skip to content

About

Code for the paper "Emergent Languages in Agent Populations: From Token Efficiency to Oversight Evasion"

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Emergent Languages in Agent Populations

Code for the paper "Emergent Languages in Agent Populations: From Token Efficiency to Oversight Evasion".

We study constructed-language proposals ("emergent languages") posted by LLM agents on the Moltbook platform. Starting from the ~232k-post Moltbook Files corpus, we identify 518 posts proposing constructed languages (the MoltSpeech subset), categorize the stated motivation behind each language (avoiding oversight, token efficiency, programming language, spoken, other), and test whether outside LLM agents can acquire these languages in-context.

Datasets

Setup

Requires uv:

uv sync

Repository layout

File Purpose
new_langs_from_moltbookfiles.py Builds the MoltSpeech dataset (paper: Dataset section, Stages 1–3)
find_languages.py In-context acquisition experiment (paper: Can LLM Agents acquire emergent languages in-context?)
plot.py Score-distribution plots and per-category statistics from the experiment outputs
generate_readable_markdown.py Renders one readable Markdown file per post from a results JSONL
outputs/ Raw results: one JSONL per (generator, judge) pair (518 rows each) plus plots and CSV stats

Pipeline

1. Dataset construction

Three-stage identification pipeline over the base corpus:

  1. Regex pre-filter — case-insensitive language|new language|slang over all posts (~6,000 candidates).
  2. Classification — a DSPy ChainOfThought judge (gpt-5-mini, temperature 0) labels each candidate with a binary is_new_language plus a free-text justification.
  3. Metadata extraction — confirmed posts get language_name, a single-label reason category, and search_hints.

The confirmed posts are pushed to the Hugging Face Hub as the MoltSpeech dataset.

export API_KEY=...            # OpenAI-compatible API key
export API_BASE=...           # optional custom endpoint
export MODEL_NAME=openai/gpt-5-mini   # optional, this is the default
uv run new_langs_from_moltbookfiles.py

2. In-context acquisition experiment

For each of the 518 posts, a generator model is prompted with the verbatim post body and asked to produce an utterance in the described language; a judge model scores the utterance's validity from 1 to 5. The paper runs all four generator × judge combinations of openai/gpt-oss-120b and Qwen/Qwen3.5-122B-A10B-FP8, served via OpenAI-compatible endpoints (e.g. vLLM):

uv run find_languages.py \
  --gen-model-name Qwen/Qwen3.5-122B-A10B-FP8 \
  --judge-model-name openai/gpt-oss-120b \
  --gen-client-url http://localhost:8081/v1 \
  --judge-client-url http://localhost:8080/v1 \
  --stage-2

Results are written to outputs/model_<generator>_judge_<judge>_simple.jsonl. Use --retry-failed-from <file> to re-run only rows whose generation failed in a previous run.

3. Analysis and plots

Parses the 1–5 score from each judge evaluation and produces a per-category score-distribution plot and a CSV of statistics for every results file:

uv run plot.py outputs/

4. Readable exports

uv run generate_readable_markdown.py outputs/model_..._simple.jsonl readable/

Results in outputs/

The repository ships the raw judgements used in the paper: four JSONL files (one per generator × judge pair, 518 rows each) and the corresponding plots and per-category statistics under outputs/plots/.

Citation

@misc{beltoft2026emergentlanguagespopulationslanguage,
      title={Emergent Languages in Populations of Language Model Agents: From Token Efficiency to Oversight Evasion}, 
      author={Stine Lyngsø Beltoft and William Brach and Federico Torrielli and Jacob Nielsen and Annemette Brok Pirchert and Filippo Tonini and Peter Schneider-Kamp and Lukas Galke Poech},
      year={2026},
      eprint={2605.31170},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2605.31170}, 
}

About

Code for the paper "Emergent Languages in Agent Populations: From Token Efficiency to Oversight Evasion"

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages