Skip to content

README.md

Latest commit

 

History

History
124 lines (84 loc) · 5.5 KB

README.md

File metadata and controls

124 lines (84 loc) · 5.5 KB
Raw

CRC Automatic Classification

This repository contains tools and notebooks for the automatic classification of CRC (Convention on the Rights of the Child) recommendations. It replaces a manual process with a structured, reproducible pipeline, classifying recommendations across three key dimensions:

  1. Clusters Framework – 12 thematic clusters and 52 sub-clusters
  2. Modified Determinant Framework – 8 bottlenecks of child rights realization
  3. ECA Child Rights Monitoring (CRM) Framework – classification into detailed subdomains

This project supports UNICEF’s child rights monitoring efforts in Europe and Central Asia, enabling faster, more consistent integration of CRC recommendations into dashboards and analytical tools.

🌟 Features

  • Automated Extraction from the OHCHR Universal Human Rights Index (UHRI) API
  • Multi-Model Classification of bottlenecks, sub-clusters, and subdomains
  • OpenAI Integration to complement traditional ML models
  • Voting System to improve prediction robustness
  • Manual Validation for quality control where no majority is reached
  • Cross-Cutting Theme Tagging (gender, disability, adolescents, etc.)

🎯 Project Goals

  • Automate classification of CRC recommendations
  • Provide structured outputs for child rights monitoring
  • Enhance the CRC Recommendations Dashboard on the TransMonEE website

📁 Project Structure

├── setup                                      # Shared configuration, SharePoint authentication and helper functions
├── 0. Training/                               # Model training and evaluation notebooks
├── 1. Extraction/
│   └── 1.1 Extract new recommendations.ipynb  # Pulls recommendation data from the UHRI API
├── 2. Preprocessing/
│   ├── 2.1 Initial preprocessing of new recs.ipynb
│   └── 2.2 NLP preprocessing of new recs.ipynb # Cleans and prepares text features for classification
├── 3. Classification/
│   ├── 3.1 Classification (OpenAI API - bottlenecks).ipynb
│   ├── 3.2 Classification (ML models - bottlenecks and subclusters).ipynb
│   └── 3.3 Classification (ML models - subdomains).ipynb
├── 4. Transformation/
│   └── 4.1 Mapping and cross-cutting tagging.ipynb
├── CEDAW/                                     # Adapted workflow for CEDAW recommendation classification
├── CRPD/                                      # Experimental workflow for CRPD recommendation classification
├── LACRO Recs/                                # Custom extraction and transformation workflow for the LACRO region
├── README.md
├── requirements.txt
└── LICENSE

🚀 Environment Setup

This project is designed to run entirely within Databricks.

How to Get Started

  1. Clone the repository into your Databricks workspace using the Repos feature.

  2. Configure Databricks secrets for:

    • SharePoint authentication
    • OpenAI API access (if using OpenAI classification notebooks)

  3. Open the workflow notebooks in sequence and run them as described in the Usage section.

📂 Data and Model Storage

Operational data files, reviewed classification outputs, reference datasets, and trained model artefacts are stored externally in SharePoint rather than within the Git repository.

This repository contains the notebooks, workflow logic, and configuration required to run the classification pipeline.

Access to the associated SharePoint directories is required to run the full workflow.

Large generated files such as trained model artefacts (.joblib) are intentionally excluded from the repository.

📦 Usage

The classification process is organized into sequential notebooks, with manual validation and review required between specific steps to ensure quality and accuracy.

Step-by-Step Workflow

Step Notebook Purpose Manual Review
1 1.1 Extract new CRC recommendations Extract CRC recommendations from the UHRI API. -
2 2.1 Initial preprocessing of new recs Clean and structure the recommendations. ✅ Review for typos, formatting issues
3 2.2 NLP preprocessing of new recs Clean and prepare text features for classification. —
4 3.1 Classification (OpenAI API - bottlenecks) Classify bottlenecks using OpenAI's model. —
5 3.2 Classification (ML models - bottlenecks and subclusters) Predict bottlenecks and sub-clusters using ML. ✅ Review rows without a majority prediction
6 3.3 Classification (ML models - subdomains) Predict subdomains using ML and sub-cluster input. ✅ Review rows without a majority prediction
7 4.1 Mapping and cross-cutting tagging Final mapping to clusters/domains and tagging cross-cutting issues. —

After tagging and transformation, the final dataset is exported to Excel and then manually copied into the main CRC database, which powers the CRC Recommendations Dashboard.

Ensure:

  • Data is reviewed before integration
  • Format and columns match the main database
  • A backup is saved before making changes

🔐 Data Privacy and Quality

  • All data used is publicly available from the UHRI.
  • The pipeline includes pre-checks for incomplete or malformed data.
  • Model outputs are reviewed where consensus is not reached.

📌 License

This project is licensed under the MIT License.