SNEU-Guide: Traceable Evidence-State Evaluation for Chinese Tour-Guide Training

Topo-SCTB-Net

Topo-SCTB-Net is a research codebase for structured guide-training evaluation: RoBERTa or dummy encoder + Soft SNEU Evidence-State Graph + topology-guided counterfactual state boundary learning + monotonic calibrated scoring. The repository does not bundle the dataset; the SNEU-Guide benchmark is publicly available on Hugging Face (see Dataset). It does not ship trained checkpoints or reported experiment metrics.

Introduction

Chinese tour-guide training requires more than fluent response scoring: supervisors must verify scenic facts, relation consistency, service and safety norms, and visitor-specific adaptation while tracing each decision to inspectable evidence. Existing text classifiers and LLM judges often collapse these requirements into holistic scores or free-form rationales, making feedback difficult to audit.

We propose SNEU-Guide, a rubric-grounded protocol built around Structured Normative Evidence Units (SNEU), and Topo-SCTB-Net, a query-node heterogeneous graph model for traceable guide-response evaluation. A SNEU record exposes facts, relations, norms, audience conditions, counterfactual boundary edits, and score rules as structured model inputs. Topo-SCTB-Net links response spans and rubric slots to state-specific query nodes, learns signed topology-guided counterfactual boundaries, calibrates monotonic scores, and accepts feedback only when its cited evidence resolves in the graph.

Key Highlights

• Traceable evaluation: Four assessment states (fact, relation, norm, audience) are predicted from dedicated query nodes instead of a single holistic score.
• Strong accuracy: 78.9 average macro-F1 on a Chinese tour-guide training dataset; outperforms the strongest local LLM judge by 3.5 F1 points.
• Calibrated scoring: Score MAE reduced to 0.308 with monotonic calibration.
• Grounded feedback: 84.9% grounded feedback accuracy and only 8.7% unsupported feedback rate.
• Efficient: 0.11 s per case, substantially faster than local LLM judges (2.11–3.08 s).
• Signed counterfactual learning: Harmful, corrective, and neutral edits are modeled as localized utility shifts with topology-aware boundaries.

Overview

Topo-SCTB-Net normalizes each case into a SNEU record and linked response spans, then builds a typed evidence-state graph. Four query nodes (fact, relation, norm, audience) read decisions from admissible evidence paths defined by SNEU compatibility and rule priors. Relation-aware message passing combines neural affinity with rule priors, graph-noise consistency stabilizes predictions under perturbed graphs, and signed topology-guided counterfactual boundary learning supervises local utility shifts. A monotonic scorer maps query-node distributions to rubric scores, and a feedback module emits repair suggestions only when citations resolve to a response span or valid SNEU slot.

Results

The published paper reports the following on a Chinese tour-guide training dataset (the SNEU-Guide benchmark is now publicly available on Hugging Face; the repository itself does not bundle data or checkpoints, as described below).

Topo-SCTB-Net reaches the upper-left region of the accuracy–auditability trade-off: higher average macro-F1 (78.9) with lower unsupported feedback rate (8.7%), compared with text encoders, graph baselines, and local LLM judges.

Dataset

The SNEU-Guide benchmark is now publicly available on Hugging Face: NOONELIKEYOU8/SNEU-Guide. It is an early public benchmark release for SNEU-Guide and Topo-SCTB-Net, a traceable evidence-state evaluation framework for Chinese tour-guide training, and is the stable release location for subsequent dataset updates associated with the paper.

Release contents

File	Rows	Description
samples.jsonl	4,800	Chinese guide-response cases, four state labels, evidence spans, and risk spans
sneu.jsonl	4,800	Structured Normative Evidence Unit records
scores.jsonl	4,800	Knowledge, expression, service, and final scores
feedback_eval.jsonl	4,800	Generated feedback records and evaluation fields
counterfactual_pairs.jsonl	720	Seen, unseen, and human-test counterfactual pairs

The 4,800 response cases were generated through the project's local Ollama data-generation workflow and retain provenance fields such as generator_seed, scenario_plan_id, and system_name. They must not be described as raw student practicum responses or as the complete human-annotated research corpus.

Labels

• Fact and relation: supported, partial, missing, contradicted
• Norm: appropriate, risky, violated, uncertain
• Audience: fit, partially_fit, misfit, uncertain

Each state has 4,080 positive/default cases and 240 cases for each of its three alternative labels. Scenario buckets contain 3,200 seen, 800 unseen_question, 400 unseen_spot, and 400 unseen_both cases. Counterfactual pairs contain 240 rows in each of test_cf_seen, test_cf_unseen, and test_cf_human.

Loading

from datasets import load_dataset

samples = load_dataset("NOONELIKEYOU8/SNEU-Guide", "samples")
sneu = load_dataset("NOONELIKEYOU8/SNEU-Guide", "sneu")
counterfactuals = load_dataset("NOONELIKEYOU8/SNEU-Guide", "counterfactual_pairs")

Intended use and limitations

The release supports reproducibility testing, schema development, state-classification experiments, counterfactual evaluation, and evidence-grounded feedback research. Because it is a generated benchmark release, it should not be used to estimate real student performance, instructor behavior, or population-level tourism-training outcomes. Users should independently review source provenance and applicable tourism or scenic-spot policies before operational deployment.

License

The dataset is released under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license, which is separate from the MIT license covering the Topo-SCTB-Net code.

Modules

• Stage 1 evidence extraction: topo_sctb/extraction implements rule extraction, BIO tagging, span classification, and span merging.
• Stage 2 state reasoning: topo_sctb/graph, topo_sctb/models/topo_sctb_net.py, and topo_sctb/losses/topo_ctb_loss.py implement SNEU graphs, soft alignment, graph reasoning, topology weights, graph noise consistency, and TopoCTB.
• Stage 3 scoring: topo_sctb/models/monotonic_scorer.py implements fixed, unconstrained, and monotonic calibrated score prediction.
• Baselines: topo_sctb/baselines includes text, SNEU concat, graph, counterfactual, retrieval/cross-encoder components, and local LLM judge baselines.
• Experiments: topo_sctb/experiments provides training, evaluation, ablation, LLM baseline, and table-generation CLIs.

Install

pip install -r requirements.txt

Initialize Empty Data

python scripts/init_empty_dataset.py --out_dir data/empty

This creates empty JSONL files and JSON Schema files under data/schema. The files are intentionally blank until real annotated records are added.

Validate Data

python scripts/validate_data.py \
  --sneu data/empty/sneu.jsonl \
  --samples data/empty/samples.jsonl \
  --cf data/empty/counterfactual_pairs.jsonl \
  --scores data/empty/scores.jsonl \
  --feedback data/empty/feedback_eval.jsonl

Empty files pass with warnings. Non-empty files are validated against the schema, label space, cross-file references, and span offsets.

Human CSV Annotation Workflow

Use data/csv_unified_templates as the recommended annotation handoff folder. Annotators fill annotations.csv and optionally counterfactuals.csv; all internal IDs are generated by the converter.

python scripts/csv_to_dataset.py \
  --csv_dir data/csv_unified_templates \
  --out_dir data/empty \
  --split_dir data/splits

The converter builds SNEU records, samples, spans, counterfactual pairs, scores, feedback evaluation records, and split files, then runs the same validation checks used by validate_data.py.

Generate CSV Annotations With Ollama

The local generation flow asks Ollama for structured JSON only. Python then validates enums and scores, computes evidence_spans / risk_spans from exact response.find(text) matches, serializes compact metadata_json, and writes only valid rows to the unified annotations.csv column order. This avoids brittle model-generated CSV and prevents bad span offsets from being imported.

ollama pull modelscope.cn/unsloth/Qwen3.6-27B-GGUF:latest
OLLAMA_CONTEXT_LENGTH=8192 ollama serve

Smoke test:

python scripts/generate_csv_with_ollama.py \
  --model modelscope.cn/unsloth/Qwen3.6-27B-GGUF:latest \
  --out-csv data/csv_unified_templates/annotations.generated.csv \
  --n-rows 5 \
  --batch-size 2 \
  --split train \
  --annotator-id ollama_qwen36 \
  --review-status pending_review

Validate the generated CSV:

python scripts/validate_csv_annotations.py \
  data/csv_unified_templates/annotations.generated.csv

Convert reviewed annotations into JSONL data:

python scripts/csv_to_dataset.py \
  --csv_dir data/csv_unified_templates \
  --out_dir data/empty \
  --split_dir data/splits

Generate Planned Ollama Dataset

For larger training/evaluation data, use the preset catalog-driven generator. It loads data/csv_unified_templates/scenario_catalog.json, fixes province, spot, scenario, split, labels, scores, and metadata in Python, then asks Ollama to write only JSON natural-language fields for one spot at a time. Each Ollama call generates 40 rows for the same spot. Python computes spans, validates rows, writes CSV, and optionally converts the CSV into JSONL data.

Dry-run and plan validation:

python scripts/generate_planned_ollama_dataset.py --validate-plan-only

Smoke test one spot:

python scripts/generate_planned_ollama_dataset.py \
  --limit-spots 1 \
  --skip-counterfactuals \
  --skip-convert

Full planned run:

python scripts/generate_planned_ollama_dataset.py

The planned generator defaults to --rows-per-call 10, --temperature 0.2, --top-p 0.85, and --repeat-penalty 1.08. These values keep each structured JSON response short enough for reliable parsing while reducing repetitive phrasing.

Default outputs:

• data/generated_ollama_planned_csv/annotations.csv
• data/generated_ollama_planned_csv/counterfactuals.csv
• data/generated_ollama_planned_dataset/
• data/generated_ollama_planned_splits/

The full plan produces 4800 annotation rows from 120 catalog spots and about 720 counterfactual rows. Existing CSVs are resumed only when completed spot batches and headers are valid.

Train And Evaluate

Stage 1:

python -m topo_sctb.experiments.train_extractor --config topo_sctb/configs/extractor_bio.yaml
python -m topo_sctb.experiments.eval_extractor --config topo_sctb/configs/extractor_bio.yaml

Stage 2:

python -m topo_sctb.experiments.train_state --config topo_sctb/configs/state_full.yaml
python -m topo_sctb.experiments.eval_state --config topo_sctb/configs/state_full.yaml --checkpoint outputs/state_full/best.pt --split test_seen
python -m topo_sctb.experiments.eval_counterfactual --config topo_sctb/configs/state_full.yaml --checkpoint outputs/state_full/best.pt --cf_split test_cf_human

Stage 3:

python -m topo_sctb.experiments.train_scorer --config topo_sctb/configs/scorer.yaml
python -m topo_sctb.experiments.eval_scorer --config topo_sctb/configs/scorer.yaml --checkpoint outputs/scorer/best.pt

Baselines

Small-model baselines:

python -m topo_sctb.experiments.run_small_baselines --config topo_sctb/configs/experiments.yaml

Local LLM baselines:

python -m topo_sctb.experiments.run_llm_baseline --config topo_sctb/configs/llm_direct.yaml
python -m topo_sctb.experiments.run_llm_baseline --config topo_sctb/configs/llm_sneu.yaml
python -m topo_sctb.experiments.run_llm_baseline --config topo_sctb/configs/llm_chain_of_rubric.yaml

LLM backends are mock, ollama, transformers, and vllm. The mock backend is for CI and parser checks. Ollama uses http://localhost:11434 by default. Transformer and vLLM backends load local model weights only.

Tables

python -m topo_sctb.experiments.make_tables --results_dir outputs --out_dir outputs/tables

The command always writes the seven CSV tables with headers. Missing metrics are NA; no synthetic metric values are produced.

Empty-Data Behavior

• validate_data.py exits 0 with warnings.
• Training commands exit non-zero with a clear message and do not create a checkpoint.
• Evaluation commands write NA metrics for empty splits.
• make_tables.py writes CSV headers and NA placeholders.
• Unit tests construct minimal in-memory objects and do not depend on real records.

Dummy Encoder

Set model.use_dummy_encoder: true in any config for no-network CPU tests. For full experiments with local weights, set it to false and ensure the configured model is available locally.

What Is Waiting For Real Data

The SNEU-Guide benchmark is now publicly available on Hugging Face (see Dataset). The repository code is ready to load those records but does not bundle the data, ship trained checkpoints, or include reported experiment metrics.

Citation

This paper has been accepted by NLPCC 2026. If you find this work helpful, please cite:

@inproceedings{qu2026senu,
  title={SNEU-Guide: Traceable Evidence-State Evaluation for Chinese Tour-Guide Training},
  author={Qu, Guanheng and Qiu, Jingxiang and Li, Na and Liu, Jiangming},
  booktitle={Proceedings of the 15th CCF International Conference on Natural Language Processing and Chinese Computing (NLPCC)},
  month={November},
  year={2026},
  address={Macau, China},
  pages={},
  note={Accepted for publication},
  doi = {},
}