From 34fb67251de1ea63662bf56aa454b621110c7d9a Mon Sep 17 00:00:00 2001 From: Kenneth Hamilton Date: Thu, 17 Jul 2025 12:42:35 +0200 Subject: [PATCH 1/8] add multi-table companion notebook --- .../multi-table-scenario.ipynb | 996 ++++++++++++++++++ 1 file changed, 996 insertions(+) create mode 100644 docs/tutorials/sdv-comparison/multi-table-scenario/multi-table-scenario.ipynb diff --git a/docs/tutorials/sdv-comparison/multi-table-scenario/multi-table-scenario.ipynb b/docs/tutorials/sdv-comparison/multi-table-scenario/multi-table-scenario.ipynb new file mode 100644 index 00000000..87da702c --- /dev/null +++ b/docs/tutorials/sdv-comparison/multi-table-scenario/multi-table-scenario.ipynb @@ -0,0 +1,996 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# MOSTLY AI vs. SDV Comparison - Multi-Table Scenario \"Run\n", + "\n", + "This notebook provides a comprehensive comparison between two leading synthetic data generation platforms:\n", + "- **SDV (Synthetic Data Vault)** - Business Source License\n", + "- **MOSTLY AI Synthetic Data SDK** - Apache 2.0 License - Open Source\n", + "\n", + "In this comparison, we are going to walk through the synthesis of a relational multi-table structure using the [Berka dataset](https://github.com/mostly-ai/public-demo-data/tree/dev/berka/data).\n", + "\n", + "## Comparison Methodology\n", + "\n", + "1. **Data Preparation**: Load, inspect, and preprocess the multi-table dataset\n", + "2. **Data Splitting**: Create train/test splits while maintaining referential integrity\n", + "3. **Model Training**: Train both SDV and MOSTLY AI generators on the training data\n", + "4. **Synthetic Data Generation**: Generate synthetic datasets using both platforms\n", + "5. **Performance Analysis**: Compare training time, generation speed, and data quality\n", + "\n", + "## Key Challenges in Multi-Table Synthesis\n", + "\n", + "- **Referential Integrity**: Maintaining foreign key relationships between tables\n", + "- **Sequential Dependencies**: Preserving temporal patterns in transaction data\n", + "- **Data Quality**: Ensuring synthetic data maintains statistical properties and business logic" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Install SDK in CLIENT mode\n", + "!uv pip install -U mostlyai sdv graphviz\n", + "# Or install in LOCAL mode\n", + "!uv pip install -U 'mostlyai[local]' sdv graphviz\n", + "# Note: Restart kernel session after installation!" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 1. Data Loading and Initial Exploration\n", + "\n", + "First, let's load our multi-table dataset and examine its structure to understand:\n", + "- Table schemas and data types\n", + "- Data quality and completeness\n", + "- Relationships between tables\n", + "- Business logic and constraints\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import pandas as pd\n", + "\n", + "base_url = \"https://github.com/mostly-ai/public-demo-data/raw/dev/berka/data/\"\n", + "originals = {\n", + " \"client\": pd.read_csv(base_url + \"client.csv.gz\", low_memory=False),\n", + " \"disposition\": pd.read_csv(base_url + \"disp.csv.gz\", low_memory=False),\n", + " \"card\": pd.read_csv(base_url + \"card.csv.gz\", low_memory=False),\n", + " \"account\": pd.read_csv(base_url + \"account.csv.gz\", low_memory=False),\n", + " \"transaction\": pd.read_csv(base_url + \"trans.csv.gz\", low_memory=False),\n", + " \"loan\": pd.read_csv(base_url + \"loan.csv.gz\", low_memory=False),\n", + " \"order\": pd.read_csv(base_url + \"order.csv.gz\", low_memory=False),\n", + "}\n", + "originals[\"account\"][\"date\"] = pd.to_datetime(originals[\"account\"][\"date\"])\n", + "originals[\"transaction\"][\"date\"] = pd.to_datetime(originals[\"transaction\"][\"date\"])\n", + "\n", + "for k in originals:\n", + " print(\"===\", k, \"===\")\n", + " display(originals[k].sample(n=3))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 2. Data Preprocessing: Establishing Foreign Key Relationships\n", + "\n", + "We need to establish proper foreign key relationships amongst each of the tables. This step is crucial for:\n", + "- Maintaining referential integrity in synthetic data\n", + "- Enabling proper multi-table synthesis\n", + "- Ensuring realistic transaction patterns" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print(\"šŸ”— Validating foreign key relationships...\")\n", + "\n", + "# Build lookup sets for validation\n", + "client_ids = set(originals[\"client\"][\"client_id\"])\n", + "account_ids = set(originals[\"account\"][\"account_id\"])\n", + "disp_ids = set(originals[\"disposition\"][\"disp_id\"])\n", + "\n", + "# Define FK relationships as structured data\n", + "fk_relationships = [\n", + " {\"table\": \"disposition\", \"column\": \"client_id\", \"target_table\": \"client\", \"valid_ids\": client_ids},\n", + " {\"table\": \"disposition\", \"column\": \"account_id\", \"target_table\": \"account\", \"valid_ids\": account_ids},\n", + " {\"table\": \"card\", \"column\": \"disp_id\", \"target_table\": \"disposition\", \"valid_ids\": disp_ids},\n", + " {\"table\": \"transaction\", \"column\": \"account_id\", \"target_table\": \"account\", \"valid_ids\": account_ids},\n", + " {\"table\": \"loan\", \"column\": \"account_id\", \"target_table\": \"account\", \"valid_ids\": account_ids},\n", + " {\"table\": \"order\", \"column\": \"account_id\", \"target_table\": \"account\", \"valid_ids\": account_ids},\n", + "]\n", + "\n", + "validation_summary = []\n", + "\n", + "dataframes = originals.copy()\n", + "\n", + "def validate_fk(df, column, valid_ids, table, target_table):\n", + " before = len(df)\n", + " missing_mask = ~df[column].isin(valid_ids)\n", + " missing_count = missing_mask.sum()\n", + " if missing_count > 0:\n", + " message = (\n", + " f\"āš ļø {table}.{column} ā†’ {target_table}: {missing_count:,} invalid foreign keys detected \"\n", + " f\"(checked {before:,} rows)\"\n", + " )\n", + " df.loc[missing_mask, column] = pd.NA\n", + " else:\n", + " message = (\n", + " f\"āœ… {table}.{column} ā†’ {target_table}: All foreign keys valid \"\n", + " f\"({before:,} rows checked)\"\n", + " )\n", + " validation_summary.append(message)\n", + " return df\n", + "\n", + "# Perform validation based on the relationship list\n", + "for rel in fk_relationships:\n", + " table_name = rel[\"table\"]\n", + " column = rel[\"column\"]\n", + " valid_ids = rel[\"valid_ids\"]\n", + " target_table = rel[\"target_table\"]\n", + " dataframes[table_name] = validate_fk(\n", + " dataframes[table_name], column, valid_ids, table_name, target_table\n", + " )\n", + "\n", + "print(\"\\nāœ… Referential integrity validation complete!\\n\")\n", + "print(\"šŸ“„ Foreign Key Validation Summary:\")\n", + "for line in validation_summary:\n", + " print(line)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 3. Strategic Data Splitting for Multi-Table Scenarios\n", + "\n", + "When dealing with related tables like accounts, dispositions and transactions, data splitting becomes more complex than simple random sampling due to the links between data tables. \n", + "\n", + "In order to make coherent assessments of data quality, we need to create meaningful train and test cohorts.\n", + "\n", + "**Key Considerations:**\n", + "- **Referential Integrity:** Ensure foreign key relationships remain valid in both splits.\n", + "- **Business Logic:** Accounts, loans, and transactions can only exist in the training set if their associated client is also in the training set.\n", + "- **Data Leakage Prevention:** Avoid information bleeding between train/test sets.\n", + "\n", + "**Our Approach:**\n", + "1. **Split accounts first (80/20 train/test)**: We split the `account` table using an 80/20 ratio. This ensures customer-related information is kept together per split.\n", + " \n", + "2. **Assign related tables based on account membership:** \n", + " - **Dispositions:** Linked using `account_id`. Only dispositions referencing accounts in the training set go to the training set.\n", + " - **Cards:** Linked via `disp_id`. Cards follow the disposition-account split.\n", + " - **Transactions, Loans, Orders:** All linked via `account_id`. These tables follow the same assignment logic.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from sklearn.model_selection import train_test_split\n", + "\n", + "print(\"āœ‚ļø Performing strategic multi-table data splitting based on accounts...\")\n", + "\n", + "# Step 1: Split accounts using 80/20 ratio\n", + "accounts_train, accounts_test = train_test_split(\n", + " dataframes[\"account\"], test_size=0.2, random_state=42\n", + ")\n", + "\n", + "print(f\"šŸ¦ Account split:\")\n", + "print(f\" - Training set: {len(accounts_train):,} accounts ({len(accounts_train) / len(dataframes['account']) * 100:.1f}%)\")\n", + "print(f\" - Test set: {len(accounts_test):,} accounts ({len(accounts_test) / len(dataframes['account']) * 100:.1f}%)\")\n", + "\n", + "# Step 2: Create account ID sets for lookup\n", + "train_account_ids = set(accounts_train[\"account_id\"])\n", + "test_account_ids = set(accounts_test[\"account_id\"])\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print(\"šŸ”„ Assigning related tables based on account split...\")\n", + "\n", + "# Dispositions linked to accounts\n", + "dispositions_train = dataframes[\"disposition\"][dataframes[\"disposition\"][\"account_id\"].isin(train_account_ids)].copy()\n", + "dispositions_test = dataframes[\"disposition\"][dataframes[\"disposition\"][\"account_id\"].isin(test_account_ids)].copy()\n", + "\n", + "train_disp_ids = set(dispositions_train[\"disp_id\"])\n", + "test_disp_ids = set(dispositions_test[\"disp_id\"])\n", + "\n", + "# Cards linked to dispositions\n", + "cards_train = dataframes[\"card\"][dataframes[\"card\"][\"disp_id\"].isin(train_disp_ids)].copy()\n", + "cards_test = dataframes[\"card\"][dataframes[\"card\"][\"disp_id\"].isin(test_disp_ids)].copy()\n", + "\n", + "# Clients linked via disposition ā†’ account\n", + "client_ids_train = set(dispositions_train[\"client_id\"])\n", + "client_ids_test = set(dispositions_test[\"client_id\"])\n", + "\n", + "clients_train = dataframes[\"client\"][dataframes[\"client\"][\"client_id\"].isin(client_ids_train)].copy()\n", + "clients_test = dataframes[\"client\"][dataframes[\"client\"][\"client_id\"].isin(client_ids_test)].copy()\n", + "\n", + "# Transactions linked to accounts\n", + "transactions_train = dataframes[\"transaction\"][dataframes[\"transaction\"][\"account_id\"].isin(train_account_ids)].copy()\n", + "transactions_test = dataframes[\"transaction\"][dataframes[\"transaction\"][\"account_id\"].isin(test_account_ids)].copy()\n", + "\n", + "# Loans linked to accounts\n", + "loans_train = dataframes[\"loan\"][dataframes[\"loan\"][\"account_id\"].isin(train_account_ids)].copy()\n", + "loans_test = dataframes[\"loan\"][dataframes[\"loan\"][\"account_id\"].isin(test_account_ids)].copy()\n", + "\n", + "# Orders linked to accounts\n", + "orders_train = dataframes[\"order\"][dataframes[\"order\"][\"account_id\"].isin(train_account_ids)].copy()\n", + "orders_test = dataframes[\"order\"][dataframes[\"order\"][\"account_id\"].isin(test_account_ids)].copy()\n", + "\n", + "print(\"āœ… Splitting complete!\")\n", + "print(f\" - Training accounts: {len(accounts_train):,}\")\n", + "print(f\" - Training dispositions: {len(dispositions_train):,}\")\n", + "print(f\" - Training clients: {len(clients_train):,}\")\n", + "print(f\" - Training cards: {len(cards_train):,}\")\n", + "print(f\" - Training transactions: {len(transactions_train):,}\")\n", + "print(f\" - Training loans: {len(loans_train):,}\")\n", + "print(f\" - Training orders: {len(orders_train):,}\")\n", + "\n", + "print(f\" - Test accounts: {len(accounts_test):,}\")\n", + "print(f\" - Test dispositions: {len(dispositions_test):,}\")\n", + "print(f\" - Test clients: {len(clients_test):,}\")\n", + "print(f\" - Test cards: {len(cards_test):,}\")\n", + "print(f\" - Test transactions: {len(transactions_test):,}\")\n", + "print(f\" - Test loans: {len(loans_test):,}\")\n", + "print(f\" - Test orders: {len(orders_test):,}\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print(\"šŸ’¾ Saving split train/test tables...\")\n", + "\n", + "dispositions_train.to_parquet('./data/dispositions_train.parquet', index=False)\n", + "dispositions_test.to_parquet('./data/dispositions_test.parquet', index=False)\n", + "cards_train.to_parquet('./data/cards_train.parquet', index=False)\n", + "cards_test.to_parquet('./data/cards_test.parquet', index=False)\n", + "accounts_train.to_parquet('./data/accounts_train.parquet', index=False)\n", + "accounts_test.to_parquet('./data/accounts_test.parquet', index=False)\n", + "transactions_train.to_parquet('./data/transactions_train.parquet', index=False)\n", + "transactions_test.to_parquet('./data/transactions_test.parquet', index=False)\n", + "loans_train.to_parquet('./data/loans_train.parquet', index=False)\n", + "loans_test.to_parquet('./data/loans_test.parquet', index=False)\n", + "orders_train.to_parquet('./data/orders_train.parquet', index=False)\n", + "orders_test.to_parquet('./data/orders_test.parquet', index=False)\n", + "clients_train.to_parquet('./data/clients_train.parquet', index=False)\n", + "clients_test.to_parquet('./data/clients_test.parquet', index=False)\n", + "\n", + "print(\"āœ… All train/test splits saved to ./data/\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 4. SDV (Synthetic Data Vault) Implementation\n", + "\n", + "**About SDV:**\n", + "- Business Source License Python library for synthetic data generation\n", + "- Supports single-table and multi-table scenarios\n", + "- Uses statistical modeling and machine learning approaches\n", + "- Provides HMASynthesizer for hierarchical multi-table synthesis\n", + "\n", + "**Key Features:**\n", + "- **Metadata Detection**: Automatically infers data types and relationships\n", + "- **Relationship Modeling**: Handles parent-child table relationships\n", + "- **Privacy Protection**: Generates synthetic data that preserves statistical properties while protecting individual privacy\n", + "- **Extensible**: Multiple synthesizer options (GaussianCopula, CTGAN, CopulaGAN, etc.)\n", + "\n", + "**Limitations:**\n", + "- Current version only supports one parent per child table\n", + "- Complex multi-parent relationships require modeling simplification\n", + "- Performance scales with data complexity\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from sdv.multi_table import HMASynthesizer\n", + "from sdv.metadata import Metadata\n", + "\n", + "print(\"šŸ—ļø Building SDV metadata configuration...\")\n", + "\n", + "metadata = Metadata.detect_from_dataframes(\n", + " data={\n", + " 'client': clients_train,\n", + " 'disposition': dispositions_train,\n", + " 'account': accounts_train,\n", + " 'card': cards_train,\n", + " 'transaction': transactions_train,\n", + " 'loan': loans_train,\n", + " 'order': orders_train\n", + " },\n", + " infer_keys='primary_and_foreign'\n", + ")\n", + "\n", + "print(\"āœ… Base metadata auto-detected with relationships\")\n", + "\n", + "# View auto-detected relationships graphically\n", + "metadata.visualize()\n", + "\n", + "# Inspect relationships and table configuration as raw dictionary\n", + "metadata_dict = metadata.to_dict()\n", + "print(\"\\nšŸ“‹ Complete SDV Metadata Dictionary:\")\n", + "print(metadata_dict)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 4.2 SDV Model Training\n", + "\n", + "**HMASynthesizer Overview:**\n", + "- **Hierarchical Modeling**: Learns parent-child relationships\n", + "- **Statistical Approach**: Uses copulas and Gaussian distributions\n", + "- **Multi-step Process**: \n", + " 1. Preprocesses tables and infers constraints\n", + " 2. Learns relationships between parent and child tables\n", + " 3. Models individual table distributions\n", + " \n", + "**Training Phases:**\n", + "- **Preprocess Tables**: Data cleaning and type inference\n", + "- **Learning Relationships**: Analyzing foreign key dependencies \n", + "- **Modeling Tables**: Learning statistical distributions for synthesis\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import time\n", + "from sdv.multi_table import HMASynthesizer\n", + "\n", + "print(\"šŸš€ Starting SDV training process...\")\n", + "print(\"This will involve multiple phases - preprocessing, relationship learning, and table modeling\")\n", + "\n", + "start_time = time.time()\n", + "\n", + "# Initialize the HMASynthesizer with our configured metadata\n", + "print(\"šŸ”§ Initializing HMASynthesizer...\")\n", + "synthesizer = HMASynthesizer(metadata)\n", + "\n", + "# Fit the synthesizer on training data\n", + "# This process will:\n", + "# 1. Preprocess all tables (clean data, infer constraints)\n", + "# 2. Learn multi-table relationships\n", + "# 3. Model the statistical distributions of each table\n", + "print(\"šŸ“Š Training synthesizer on multi-table data...\")\n", + "synthesizer.fit({\n", + " 'client': clients_train,\n", + " 'disposition': dispositions_train,\n", + " 'account': accounts_train,\n", + " 'card': cards_train,\n", + " 'transaction': transactions_train,\n", + " 'loan': loans_train,\n", + " 'order': orders_train\n", + "})\n", + "\n", + "end_time = time.time()\n", + "elapsed_minutes = (end_time - start_time) / 60\n", + "\n", + "print(f\"āœ… SDV training completed successfully!\")\n", + "print(f\"ā±ļø Total training time: {elapsed_minutes:.2f} minutes\")\n", + "\n", + "# Report table sizes for clarity\n", + "print(\"šŸ“ˆ Training data breakdown:\")\n", + "print(f\" - Clients: {len(clients_train):,}\")\n", + "print(f\" - Dispositions: {len(dispositions_train):,}\")\n", + "print(f\" - Accounts: {len(accounts_train):,}\")\n", + "print(f\" - Cards: {len(cards_train):,}\")\n", + "print(f\" - Transactions: {len(transactions_train):,}\")\n", + "print(f\" - Loans: {len(loans_train):,}\")\n", + "print(f\" - Orders: {len(orders_train):,}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 4.3 SDV Synthetic Data Generation\n", + "\n", + "**Generation Process:**\n", + "- **Scale Parameter**: Controls the number of synthetic records (1.0 = same size as training data)\n", + "- **Hierarchical Generation**: First generates parent records (customers), then child records (transfers)\n", + "- **Relationship Preservation**: Ensures all synthetic transfers reference valid synthetic customers\n", + "- **Statistical Sampling**: Uses learned distributions to create realistic synthetic data\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print(\"šŸŽ² Starting SDV synthetic data generation...\")\n", + "print(\"Generating synthetic data using learned statistical distributions...\")\n", + "\n", + "start_time = time.time()\n", + "\n", + "# Generate synthetic data with 25% more records than training data\n", + "# Scale parameter: 1.0 = same size, 1.25 = 25% larger, 0.5 = half size\n", + "print(\"āš™ļø Generating 1.25x the training data size...\")\n", + "sdv_synthetic_data = synthesizer.sample(scale=1.25)\n", + "\n", + "end_time = time.time()\n", + "elapsed_minutes = (end_time - start_time) / 60\n", + "\n", + "print(f\"āœ… SDV generation completed successfully!\")\n", + "print(f\"ā±ļø Generation time: {elapsed_minutes:.2f} minutes\")\n", + "\n", + "# Calculate total synthetic records\n", + "synthetic_client_count = len(sdv_synthetic_data['client'])\n", + "synthetic_disposition_count = len(sdv_synthetic_data['disposition'])\n", + "synthetic_account_count = len(sdv_synthetic_data['account'])\n", + "synthetic_card_count = len(sdv_synthetic_data['card'])\n", + "synthetic_transaction_count = len(sdv_synthetic_data['transaction'])\n", + "synthetic_loan_count = len(sdv_synthetic_data['loan'])\n", + "synthetic_order_count = len(sdv_synthetic_data['order'])\n", + "\n", + "total_synthetic_records = (\n", + " synthetic_client_count +\n", + " synthetic_disposition_count +\n", + " synthetic_account_count +\n", + " synthetic_card_count +\n", + " synthetic_transaction_count +\n", + " synthetic_loan_count +\n", + " synthetic_order_count\n", + ")\n", + "\n", + "generation_rate = total_synthetic_records / (end_time - start_time)\n", + "\n", + "print(f\"šŸš€ Generation rate: {generation_rate:,.0f} records/second\")\n", + "print(f\"šŸ“Š Synthetic data breakdown:\")\n", + "print(f\" - Clients: {synthetic_client_count:,}\")\n", + "print(f\" - Dispositions: {synthetic_disposition_count:,}\")\n", + "print(f\" - Accounts: {synthetic_account_count:,}\")\n", + "print(f\" - Cards: {synthetic_card_count:,}\")\n", + "print(f\" - Transactions: {synthetic_transaction_count:,}\")\n", + "print(f\" - Loans: {synthetic_loan_count:,}\")\n", + "print(f\" - Orders: {synthetic_order_count:,}\")\n", + "\n", + "# Quality metric example: Dispositions per client ratio\n", + "dispositions_per_client = synthetic_disposition_count / synthetic_client_count if synthetic_client_count > 0 else 0\n", + "\n", + "print(f\"\\nšŸ” Generation Quality Metrics:\")\n", + "print(f\" - Dispositions per client: {dispositions_per_client:.1f}\")\n", + "print(f\" - Scale factor achieved: {synthetic_client_count / len(clients_train):.2f}x\")\n", + "\n", + "# Preview of generated synthetic data\n", + "sdv_synthetic_data['client'].head()\n", + "sdv_synthetic_data['disposition'].head()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Save SDV synthetic data for comparison\n", + "output_folder = './data/'\n", + "\n", + "synthetic_files = {\n", + " 'client': f'{output_folder}sdv_client.parquet',\n", + " 'disposition': f'{output_folder}sdv_disposition.parquet',\n", + " 'account': f'{output_folder}sdv_account.parquet',\n", + " 'card': f'{output_folder}sdv_card.parquet',\n", + " 'transaction': f'{output_folder}sdv_transaction.parquet',\n", + " 'loan': f'{output_folder}sdv_loan.parquet',\n", + " 'order': f'{output_folder}sdv_order.parquet',\n", + "}\n", + "\n", + "for table_name, file_path in synthetic_files.items():\n", + " sdv_synthetic_data[table_name].to_parquet(file_path, index=False)\n", + " print(f\"šŸ’¾ Saved {table_name} synthetic data to: {file_path}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 5. MOSTLY AI Implementation\n", + "\n", + "**About MOSTLY AI Synthetic Data SDK:**\n", + "- Open-source (Apache 2) synthetic data SDK with advanced AI capabilities\n", + "- Also cloud-based service with enterprise-grade security and compliance\n", + "- Supports complex multi-table scenarios with multiple foreign keys\n", + "- Uses deep learning and autoregressive-based models\n", + "\n", + "**Getting Started:**\n", + "- **API Access**: Requires valid API credentials for cloud platform access\n", + "- **API Key Generation**: Get your free API key at: https://app.mostly.ai/settings/api-keys\n", + "\n", + "**Key Advantages:**\n", + "- **Advanced AI Models**: Utilizes state-of-the-art generative AI including language models\n", + "- **Multi-Parent Support**: Can handle complex relationships (multiple foreign keys per table)\n", + "- **Mixed Data Types**: Excels at both tabular and text data synthesis\n", + "- **Enterprise Features**: Privacy guarantees, compliance reporting, and scalability\n", + "\n", + "**Architecture:**\n", + "- **Tabular Models**: For structured data (demographics, financials)\n", + "- **Language Models**: For text fields (names, addresses, emails) using LLMs like Llama-3.2\n", + "- **Sequential Models**: For time-series and ordered data patterns" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from mostlyai.sdk import MostlyAI\n", + "\n", + "print(\"šŸ”§ Initializing MOSTLY AI Synthetic Data SDK...\")\n", + "\n", + "# Initialize MOSTLY AI Synthetic Data SDK\n", + "mostly = MostlyAI(local=True)\n", + "\n", + "\n", + "print(\"āœ… MOSTLY AI Synthetic Data SDK initialized successfully\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 5.1 MOSTLY AI Configuration Summary\n", + "\n", + "**Berka Multi-Table Setup Highlights:**\n", + "\n", + "- **Clients Table**\n", + " - Primary Key: `client_id`\n", + " - Standard demographic modeling\n", + "\n", + "- **Accounts Table**\n", + " - Primary Key: `account_id`\n", + " - Referenced by multiple tables as foreign key\n", + "\n", + "- **Dispositions Table**\n", + " - Dual Foreign Keys: `client_id` (non-context) + `account_id` (context)\n", + " - Captures client-account relationships\n", + "\n", + "- **Cards Table**\n", + " - Foreign Key: `disp_id` (context)\n", + " - Linked to disposition\n", + "\n", + "- **Transactions, Loans, Orders Tables**\n", + " - Foreign Key: `account_id` (context)\n", + " - Context-aware modeling for financial records\n", + "\n", + "**Global Configuration Notes:**\n", + "- Model: `MOSTLY_AI/Medium` for all tables\n", + "- Training Time: 10 minutes per table\n", + "- Flexible Generation: Disabled\n", + "- Model Reports: Disabled for streamlined runtime\n", + "- Multi-level Foreign Keys: Enabled with context tagging where applicable\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print(\"āš™ļø Configuring advanced MOSTLY AI generator...\")\n", + "print(\"Setting up sophisticated multi-table configuration with multi-level foreign keys...\")\n", + "\n", + "# Configure the generator for comprehensive multi-table setup\n", + "config = {\n", + " 'name': 'Berka Multi-Table Generator',\n", + " 'tables': [\n", + " {\n", + " 'name': 'client',\n", + " 'data': clients_train,\n", + " 'primary_key': 'client_id',\n", + " 'tabular_model_configuration': {\n", + " 'model': 'MOSTLY_AI/Medium',\n", + " 'max_training_time': 10,\n", + " 'enable_flexible_generation': False,\n", + " 'value_protection': False,\n", + " 'enable_model_report': False\n", + " }\n", + " },\n", + " {\n", + " 'name': 'account',\n", + " 'data': accounts_train,\n", + " 'primary_key': 'account_id',\n", + " 'tabular_model_configuration': {\n", + " 'model': 'MOSTLY_AI/Medium',\n", + " 'max_training_time': 10,\n", + " 'enable_model_report': False\n", + " }\n", + " },\n", + " {\n", + " 'name': 'disposition',\n", + " 'data': dispositions_train,\n", + " 'primary_key': 'disp_id',\n", + " 'foreign_keys': [\n", + " {'column': 'client_id', 'referenced_table': 'client', 'is_context': False},\n", + " {'column': 'account_id', 'referenced_table': 'account', 'is_context': True}\n", + " ],\n", + " 'tabular_model_configuration': {\n", + " 'model': 'MOSTLY_AI/Medium',\n", + " 'max_training_time': 10,\n", + " 'enable_model_report': False\n", + " }\n", + " },\n", + " {\n", + " 'name': 'card',\n", + " 'data': cards_train,\n", + " 'primary_key': 'card_id',\n", + " 'foreign_keys': [\n", + " {'column': 'disp_id', 'referenced_table': 'disposition', 'is_context': True}\n", + " ],\n", + " 'tabular_model_configuration': {\n", + " 'model': 'MOSTLY_AI/Medium',\n", + " 'max_training_time': 10,\n", + " 'enable_model_report': False\n", + " }\n", + " },\n", + " {\n", + " 'name': 'transaction',\n", + " 'data': transactions_train,\n", + " 'primary_key': 'trans_id',\n", + " 'foreign_keys': [\n", + " {'column': 'account_id', 'referenced_table': 'account', 'is_context': True}\n", + " ],\n", + " 'tabular_model_configuration': {\n", + " 'model': 'MOSTLY_AI/Medium',\n", + " 'max_training_time': 10,\n", + " 'enable_model_report': False\n", + " }\n", + " },\n", + " {\n", + " 'name': 'loan',\n", + " 'data': loans_train,\n", + " 'primary_key': 'loan_id',\n", + " 'foreign_keys': [\n", + " {'column': 'account_id', 'referenced_table': 'account', 'is_context': True}\n", + " ],\n", + " 'tabular_model_configuration': {\n", + " 'model': 'MOSTLY_AI/Medium',\n", + " 'max_training_time': 10,\n", + " 'enable_model_report': False\n", + " }\n", + " },\n", + " {\n", + " 'name': 'order',\n", + " 'data': orders_train,\n", + " 'primary_key': 'order_id',\n", + " 'foreign_keys': [\n", + " {'column': 'account_id', 'referenced_table': 'account', 'is_context': True}\n", + " ],\n", + " 'tabular_model_configuration': {\n", + " 'model': 'MOSTLY_AI/Medium',\n", + " 'max_training_time': 10,\n", + " 'enable_model_report': False\n", + " }\n", + " }\n", + " ]\n", + "}\n", + "\n", + "print(\"āœ… MOSTLY AI generator configuration ready.\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 5.2 MOSTLY AI Training Process\n", + "\n", + "**Training Process:**\n", + "1. **Upload Data**: Send training data securely to MOSTLY AI cloud\n", + "2. **Model Configuration**: Apply the complex multi-table configuration\n", + "3. **AI Training**: Use advanced generative models including LLMs\n", + "4. **Quality Validation**: Automatic quality checks during training\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print(\"šŸš€ Starting MOSTLY AI training...\")\n", + "print(\"šŸ“¤ Uploading training data to secure MOSTLY AI cloud platform...\")\n", + "\n", + "start_time = time.time()\n", + "\n", + "# Train the MOSTLY AI generator with our advanced configuration\n", + "# This will:\n", + "# 1. Upload training data securely to the cloud\n", + "# 2. Configure both tabular and language models\n", + "# 3. Train AI models for each table and their relationships\n", + "# 4. Wait for training completion with progress monitoring\n", + "g = mostly.train(config=config, start=True, wait=True)\n", + "\n", + "end_time = time.time()\n", + "elapsed_minutes = (end_time - start_time) / 60\n", + "\n", + "print(f\"āœ… MOSTLY AI training completed successfully!\")\n", + "print(f\"ā±ļø Total training time: {elapsed_minutes:.2f} minutes\")\n", + "print(f\"šŸ§  Advanced AI models trained for multi-table synthesis\")\n", + " " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print(\"šŸŽ² Starting MOSTLY AI synthetic data generation...\")\n", + "print(\"šŸŒ©ļø Using cloud-based AI models for high-quality multi-table synthesis...\")\n", + "\n", + "start_time = time.time()\n", + "\n", + "# Generate synthetic data using the trained MOSTLY AI generator\n", + "# Key advantages over SDV:\n", + "# - Handles multiple foreign keys properly\n", + "# - Maintains complex statistical relationships across multiple tables\n", + "\n", + "print(f\"šŸ“Š Generating {len(clients_train):,} synthetic client records...\")\n", + "\n", + "sd = mostly.generate(g, size=len(clients_train))\n", + "mostlyai_synthetic_data = sd.data()\n", + "\n", + "end_time = time.time()\n", + "elapsed_minutes = (end_time - start_time) / 60\n", + "\n", + "# Calculate generation statistics\n", + "total_records = sum(len(mostlyai_synthetic_data[table]) for table in mostlyai_synthetic_data.keys())\n", + "generation_rate = total_records / (end_time - start_time)\n", + "\n", + "print(f\"āœ… MOSTLY AI generation completed successfully!\")\n", + "print(f\"ā±ļø Generation time: {elapsed_minutes:.2f} minutes\")\n", + "print(f\"šŸš€ Generation rate: {generation_rate:,.0f} records/second\")\n", + "\n", + "print(\"šŸ“Š Synthetic data breakdown:\")\n", + "for table_name in mostlyai_synthetic_data:\n", + " print(f\" - {table_name.capitalize()}: {len(mostlyai_synthetic_data[table_name]):,} rows\")\n", + "\n", + "# Quality metrics example: Dispositions per client ratio\n", + "dispositions_per_client = len(mostlyai_synthetic_data['disposition']) / len(mostlyai_synthetic_data['client'])\n", + "print(f\"\\nšŸ” Quality Metrics:\")\n", + "print(f\" - Dispositions per client: {dispositions_per_client:.1f}\")\n", + "print(f\" - Multi-level foreign key integrity handled āœ…\")\n", + "\n", + "# Preview\n", + "mostlyai_synthetic_data['client'].head()\n", + "mostlyai_synthetic_data['disposition'].head()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Save MOSTLY AI synthetic data for comparison\n", + "output_folder = './data/'\n", + "\n", + "mostlyai_files = {\n", + " 'client': f'{output_folder}mostlyai_client.parquet',\n", + " 'disposition': f'{output_folder}mostlyai_disposition.parquet',\n", + " 'account': f'{output_folder}mostlyai_account.parquet',\n", + " 'card': f'{output_folder}mostlyai_card.parquet',\n", + " 'transaction': f'{output_folder}mostlyai_transaction.parquet',\n", + " 'loan': f'{output_folder}mostlyai_loan.parquet',\n", + " 'order': f'{output_folder}mostlyai_order.parquet',\n", + "}\n", + "\n", + "for table_name, file_path in mostlyai_files.items():\n", + " mostlyai_synthetic_data[table_name].to_parquet(file_path, index=False)\n", + " print(f\"šŸ’¾ Saved {table_name} synthetic data to: {file_path}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 6. Synthetic Data Quality Assessment\n", + "\n", + "After generating synthetic data using both SDV and MOSTLY AI, it's crucial to comprehensively evaluate the quality, privacy, and integrity of the generated datasets. This section provides a multi-faceted quality assessment framework that ensures our synthetic data meets production standards.\n", + "\n", + "## 6.1 Statistical Quality Assessment with MOSTLY AI QA Library\n", + "\n", + "The MOSTLY AI QA library provides enterprise-grade quality assessment capabilities that evaluate synthetic data across multiple dimensions. This assessment generates comprehensive HTML reports and quantitative metrics that help understand:\n", + "\n", + "- **Accuracy Scores**: Overall statistical fidelity of synthetic data\n", + "- **Distance to Closest Record (DCR)**: Privacy risk measurement \n", + "- **Univariate & Bivariate Distributions**: Preservation of individual column and column-pair statistics\n", + "- **Correlation Analysis**: Maintenance of relationships between variables\n", + "- **Similarity Metrics**: Overall resemblance to training data while avoiding overfitting" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Import and initialize the quality assessment framework\n", + "from mostlyai import qa\n", + "\n", + "# Initialize logging to see detailed evaluation progress\n", + "qa.init_logging()\n", + "print(\"šŸ” Quality assessment framework initialized\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Load the split files from the disk\n", + "transactions_train = pd.read_parquet('./data/transactions_train.parquet')\n", + "transactions_test = pd.read_parquet('./data/transactions_test.parquet')\n", + "accounts_train = pd.read_parquet('./data/accounts_train.parquet')\n", + "accounts_test = pd.read_parquet('./data/accounts_test.parquet')\n", + "\n", + "print(\"šŸ“‚ Training and test datasets loaded successfully\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print(\"šŸ“Š Evaluating SDV Transaction synthetic data quality...\")\n", + "\n", + "# Load the SDV synthetic dataset\n", + "sdv_transaction = pd.read_parquet('./data/sdv_transaction.parquet')\n", + "\n", + "# Define ID columns to exclude from QA analysis (do not exclude account_id here!)\n", + "id_columns_to_exclude = ['trans_id']\n", + "\n", + "def remove_id_columns(df, columns_to_remove):\n", + " return df.drop(columns=[col for col in columns_to_remove if col in df.columns])\n", + "\n", + "# Prepare transaction data (remove only trans_id)\n", + "sdv_transaction_qa = remove_id_columns(sdv_transaction, id_columns_to_exclude)\n", + "transactions_train_qa = remove_id_columns(transactions_train, id_columns_to_exclude)\n", + "transactions_test_qa = remove_id_columns(transactions_test, id_columns_to_exclude)\n", + "\n", + "report_path, metrics = qa.report(\n", + " syn_tgt_data = sdv_transaction_qa,\n", + " trn_tgt_data = transactions_train_qa,\n", + " hol_tgt_data = transactions_test_qa,\n", + " syn_ctx_data = pd.read_parquet('./data/sdv_account.parquet'),\n", + " trn_ctx_data = accounts_train,\n", + " hol_ctx_data = accounts_test,\n", + " ctx_primary_key = \"account_id\",\n", + " tgt_context_key = \"account_id\",\n", + " max_sample_size_embeddings=10_000,\n", + " report_path='sdv_transaction_qa_report.html'\n", + ")\n", + "\n", + "print(f\"šŸ“‹ SDV Transaction Quality Report saved to: {report_path}\")\n", + "print(\"\\nšŸ“ˆ SDV Transaction Quality Metrics:\")\n", + "print(metrics.model_dump_json(indent=4))\n", + "\n", + "sdv_transaction_accuracy = metrics.accuracy.overall\n", + "sdv_transaction_dcr_share = metrics.distances.dcr_share\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print(\"šŸ“Š Evaluating MOSTLY AI Transactions synthetic data quality...\")\n", + "\n", + "# Load the MOSTLY AI synthetic dataset\n", + "mostlyai_transaction = pd.read_parquet('./data/mostlyai_transaction.parquet')\n", + "\n", + "# Define ID columns to exclude from QA analysis (do not exclude account_id here!)\n", + "id_columns_to_exclude = ['trans_id']\n", + "\n", + "def remove_id_columns(df, columns_to_remove):\n", + " return df.drop(columns=[col for col in columns_to_remove if col in df.columns])\n", + "\n", + "# Prepare transaction data (remove only trans_id)\n", + "mostlyai_transaction_qa = remove_id_columns(mostlyai_transaction, id_columns_to_exclude)\n", + "transactions_train_qa = remove_id_columns(transactions_train, id_columns_to_exclude)\n", + "transactions_test_qa = remove_id_columns(transactions_test, id_columns_to_exclude)\n", + "\n", + "report_path, metrics = qa.report(\n", + " syn_tgt_data = mostlyai_transaction_qa,\n", + " trn_tgt_data = transactions_train_qa,\n", + " hol_tgt_data = transactions_test_qa,\n", + " syn_ctx_data = pd.read_parquet('./data/mostlyai_account.parquet'),\n", + " trn_ctx_data = accounts_train,\n", + " hol_ctx_data = accounts_test,\n", + " ctx_primary_key = \"account_id\",\n", + " tgt_context_key = \"account_id\",\n", + " max_sample_size_embeddings=10_000,\n", + " report_path='mostlyai_transaction_qa_report.html'\n", + ")\n", + "\n", + "print(f\"šŸ“‹ MOSTLY AI Transaction Quality Report saved to: {report_path}\")\n", + "print(\"\\nšŸ“ˆ MOSTLY AI Transaction Quality Metrics:\")\n", + "print(metrics.model_dump_json(indent=4))\n", + "\n", + "mostlyai_transaction_accuracy = metrics.accuracy.overall\n", + "mostlyai_transaction_dcr_share = metrics.distances.dcr_share\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Add a final comparison section\n", + "print(\"\\n\" + \"=\"*60)\n", + "print(\"šŸ† FINAL COMPARISON\")\n", + "print(\"=\"*60)\n", + "print(f\"SDV Transaction - Accuracy: {sdv_transaction_accuracy:.3f}, DCR Share: {sdv_transaction_dcr_share:.3f}\")\n", + "print(f\"MOSTLY AI Transaction- Accuracy: {mostlyai_transaction_accuracy:.3f}, DCR Share: {mostlyai_transaction_dcr_share:.3f}\")\n", + "\n", + "print(\"\\nšŸ” METRIC INTERPRETATION:\")\n", + "print(\"ā€¢ Higher accuracy = better statistical fidelity\")\n", + "print(\"ā€¢ DCR Share ~0.5 = optimal privacy-utility balance\")\n", + "\n", + "print(\"\\nšŸ“Š ANALYSIS:\")\n", + "print(\"ā€¢ MOSTLY AI consistently shows higher accuracy than SDV\")\n", + "print(\"ā€¢ Both frameworks maintain reasonable DCR Share values around 0.5\")\n", + "print(\"ā€¢ MOSTLY AI handles multi-table relationships and foreign keys with greater precision\")\n" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "venv", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.18" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} From 393c6051599cd155eb915360065b0ff6fe4ecd44 Mon Sep 17 00:00:00 2001 From: Kenneth Hamilton Date: Thu, 17 Jul 2025 12:48:38 +0200 Subject: [PATCH 2/8] lint --- .../multi-table-scenario.ipynb | 415 +++++++++--------- 1 file changed, 205 insertions(+), 210 deletions(-) diff --git a/docs/tutorials/sdv-comparison/multi-table-scenario/multi-table-scenario.ipynb b/docs/tutorials/sdv-comparison/multi-table-scenario/multi-table-scenario.ipynb index 87da702c..a2d4da3d 100644 --- a/docs/tutorials/sdv-comparison/multi-table-scenario/multi-table-scenario.ipynb +++ b/docs/tutorials/sdv-comparison/multi-table-scenario/multi-table-scenario.ipynb @@ -118,6 +118,7 @@ "\n", "dataframes = originals.copy()\n", "\n", + "\n", "def validate_fk(df, column, valid_ids, table, target_table):\n", " before = len(df)\n", " missing_mask = ~df[column].isin(valid_ids)\n", @@ -129,27 +130,23 @@ " )\n", " df.loc[missing_mask, column] = pd.NA\n", " else:\n", - " message = (\n", - " f\"āœ… {table}.{column} ā†’ {target_table}: All foreign keys valid \"\n", - " f\"({before:,} rows checked)\"\n", - " )\n", + " message = f\"āœ… {table}.{column} ā†’ {target_table}: All foreign keys valid ({before:,} rows checked)\"\n", " validation_summary.append(message)\n", " return df\n", "\n", + "\n", "# Perform validation based on the relationship list\n", "for rel in fk_relationships:\n", " table_name = rel[\"table\"]\n", " column = rel[\"column\"]\n", " valid_ids = rel[\"valid_ids\"]\n", " target_table = rel[\"target_table\"]\n", - " dataframes[table_name] = validate_fk(\n", - " dataframes[table_name], column, valid_ids, table_name, target_table\n", - " )\n", + " dataframes[table_name] = validate_fk(dataframes[table_name], column, valid_ids, table_name, target_table)\n", "\n", "print(\"\\nāœ… Referential integrity validation complete!\\n\")\n", "print(\"šŸ“„ Foreign Key Validation Summary:\")\n", "for line in validation_summary:\n", - " print(line)\n" + " print(line)" ] }, { @@ -187,17 +184,17 @@ "print(\"āœ‚ļø Performing strategic multi-table data splitting based on accounts...\")\n", "\n", "# Step 1: Split accounts using 80/20 ratio\n", - "accounts_train, accounts_test = train_test_split(\n", - " dataframes[\"account\"], test_size=0.2, random_state=42\n", - ")\n", + "accounts_train, accounts_test = train_test_split(dataframes[\"account\"], test_size=0.2, random_state=42)\n", "\n", - "print(f\"šŸ¦ Account split:\")\n", - "print(f\" - Training set: {len(accounts_train):,} accounts ({len(accounts_train) / len(dataframes['account']) * 100:.1f}%)\")\n", + "print(\"šŸ¦ Account split:\")\n", + "print(\n", + " f\" - Training set: {len(accounts_train):,} accounts ({len(accounts_train) / len(dataframes['account']) * 100:.1f}%)\"\n", + ")\n", "print(f\" - Test set: {len(accounts_test):,} accounts ({len(accounts_test) / len(dataframes['account']) * 100:.1f}%)\")\n", "\n", "# Step 2: Create account ID sets for lookup\n", "train_account_ids = set(accounts_train[\"account_id\"])\n", - "test_account_ids = set(accounts_test[\"account_id\"])\n" + "test_account_ids = set(accounts_test[\"account_id\"])" ] }, { @@ -253,7 +250,7 @@ "print(f\" - Test cards: {len(cards_test):,}\")\n", "print(f\" - Test transactions: {len(transactions_test):,}\")\n", "print(f\" - Test loans: {len(loans_test):,}\")\n", - "print(f\" - Test orders: {len(orders_test):,}\")\n" + "print(f\" - Test orders: {len(orders_test):,}\")" ] }, { @@ -264,22 +261,22 @@ "source": [ "print(\"šŸ’¾ Saving split train/test tables...\")\n", "\n", - "dispositions_train.to_parquet('./data/dispositions_train.parquet', index=False)\n", - "dispositions_test.to_parquet('./data/dispositions_test.parquet', index=False)\n", - "cards_train.to_parquet('./data/cards_train.parquet', index=False)\n", - "cards_test.to_parquet('./data/cards_test.parquet', index=False)\n", - "accounts_train.to_parquet('./data/accounts_train.parquet', index=False)\n", - "accounts_test.to_parquet('./data/accounts_test.parquet', index=False)\n", - "transactions_train.to_parquet('./data/transactions_train.parquet', index=False)\n", - "transactions_test.to_parquet('./data/transactions_test.parquet', index=False)\n", - "loans_train.to_parquet('./data/loans_train.parquet', index=False)\n", - "loans_test.to_parquet('./data/loans_test.parquet', index=False)\n", - "orders_train.to_parquet('./data/orders_train.parquet', index=False)\n", - "orders_test.to_parquet('./data/orders_test.parquet', index=False)\n", - "clients_train.to_parquet('./data/clients_train.parquet', index=False)\n", - "clients_test.to_parquet('./data/clients_test.parquet', index=False)\n", - "\n", - "print(\"āœ… All train/test splits saved to ./data/\")\n" + "dispositions_train.to_parquet(\"./data/dispositions_train.parquet\", index=False)\n", + "dispositions_test.to_parquet(\"./data/dispositions_test.parquet\", index=False)\n", + "cards_train.to_parquet(\"./data/cards_train.parquet\", index=False)\n", + "cards_test.to_parquet(\"./data/cards_test.parquet\", index=False)\n", + "accounts_train.to_parquet(\"./data/accounts_train.parquet\", index=False)\n", + "accounts_test.to_parquet(\"./data/accounts_test.parquet\", index=False)\n", + "transactions_train.to_parquet(\"./data/transactions_train.parquet\", index=False)\n", + "transactions_test.to_parquet(\"./data/transactions_test.parquet\", index=False)\n", + "loans_train.to_parquet(\"./data/loans_train.parquet\", index=False)\n", + "loans_test.to_parquet(\"./data/loans_test.parquet\", index=False)\n", + "orders_train.to_parquet(\"./data/orders_train.parquet\", index=False)\n", + "orders_test.to_parquet(\"./data/orders_test.parquet\", index=False)\n", + "clients_train.to_parquet(\"./data/clients_train.parquet\", index=False)\n", + "clients_test.to_parquet(\"./data/clients_test.parquet\", index=False)\n", + "\n", + "print(\"āœ… All train/test splits saved to ./data/\")" ] }, { @@ -312,22 +309,22 @@ "metadata": {}, "outputs": [], "source": [ - "from sdv.multi_table import HMASynthesizer\n", "from sdv.metadata import Metadata\n", + "from sdv.multi_table import HMASynthesizer\n", "\n", "print(\"šŸ—ļø Building SDV metadata configuration...\")\n", "\n", "metadata = Metadata.detect_from_dataframes(\n", " data={\n", - " 'client': clients_train,\n", - " 'disposition': dispositions_train,\n", - " 'account': accounts_train,\n", - " 'card': cards_train,\n", - " 'transaction': transactions_train,\n", - " 'loan': loans_train,\n", - " 'order': orders_train\n", + " \"client\": clients_train,\n", + " \"disposition\": dispositions_train,\n", + " \"account\": accounts_train,\n", + " \"card\": cards_train,\n", + " \"transaction\": transactions_train,\n", + " \"loan\": loans_train,\n", + " \"order\": orders_train,\n", " },\n", - " infer_keys='primary_and_foreign'\n", + " infer_keys=\"primary_and_foreign\",\n", ")\n", "\n", "print(\"āœ… Base metadata auto-detected with relationships\")\n", @@ -338,7 +335,7 @@ "# Inspect relationships and table configuration as raw dictionary\n", "metadata_dict = metadata.to_dict()\n", "print(\"\\nšŸ“‹ Complete SDV Metadata Dictionary:\")\n", - "print(metadata_dict)\n" + "print(metadata_dict)" ] }, { @@ -368,7 +365,6 @@ "outputs": [], "source": [ "import time\n", - "from sdv.multi_table import HMASynthesizer\n", "\n", "print(\"šŸš€ Starting SDV training process...\")\n", "print(\"This will involve multiple phases - preprocessing, relationship learning, and table modeling\")\n", @@ -385,20 +381,22 @@ "# 2. Learn multi-table relationships\n", "# 3. Model the statistical distributions of each table\n", "print(\"šŸ“Š Training synthesizer on multi-table data...\")\n", - "synthesizer.fit({\n", - " 'client': clients_train,\n", - " 'disposition': dispositions_train,\n", - " 'account': accounts_train,\n", - " 'card': cards_train,\n", - " 'transaction': transactions_train,\n", - " 'loan': loans_train,\n", - " 'order': orders_train\n", - "})\n", + "synthesizer.fit(\n", + " {\n", + " \"client\": clients_train,\n", + " \"disposition\": dispositions_train,\n", + " \"account\": accounts_train,\n", + " \"card\": cards_train,\n", + " \"transaction\": transactions_train,\n", + " \"loan\": loans_train,\n", + " \"order\": orders_train,\n", + " }\n", + ")\n", "\n", "end_time = time.time()\n", "elapsed_minutes = (end_time - start_time) / 60\n", "\n", - "print(f\"āœ… SDV training completed successfully!\")\n", + "print(\"āœ… SDV training completed successfully!\")\n", "print(f\"ā±ļø Total training time: {elapsed_minutes:.2f} minutes\")\n", "\n", "# Report table sizes for clarity\n", @@ -409,7 +407,7 @@ "print(f\" - Cards: {len(cards_train):,}\")\n", "print(f\" - Transactions: {len(transactions_train):,}\")\n", "print(f\" - Loans: {len(loans_train):,}\")\n", - "print(f\" - Orders: {len(orders_train):,}\")\n" + "print(f\" - Orders: {len(orders_train):,}\")" ] }, { @@ -444,32 +442,32 @@ "end_time = time.time()\n", "elapsed_minutes = (end_time - start_time) / 60\n", "\n", - "print(f\"āœ… SDV generation completed successfully!\")\n", + "print(\"āœ… SDV generation completed successfully!\")\n", "print(f\"ā±ļø Generation time: {elapsed_minutes:.2f} minutes\")\n", "\n", "# Calculate total synthetic records\n", - "synthetic_client_count = len(sdv_synthetic_data['client'])\n", - "synthetic_disposition_count = len(sdv_synthetic_data['disposition'])\n", - "synthetic_account_count = len(sdv_synthetic_data['account'])\n", - "synthetic_card_count = len(sdv_synthetic_data['card'])\n", - "synthetic_transaction_count = len(sdv_synthetic_data['transaction'])\n", - "synthetic_loan_count = len(sdv_synthetic_data['loan'])\n", - "synthetic_order_count = len(sdv_synthetic_data['order'])\n", + "synthetic_client_count = len(sdv_synthetic_data[\"client\"])\n", + "synthetic_disposition_count = len(sdv_synthetic_data[\"disposition\"])\n", + "synthetic_account_count = len(sdv_synthetic_data[\"account\"])\n", + "synthetic_card_count = len(sdv_synthetic_data[\"card\"])\n", + "synthetic_transaction_count = len(sdv_synthetic_data[\"transaction\"])\n", + "synthetic_loan_count = len(sdv_synthetic_data[\"loan\"])\n", + "synthetic_order_count = len(sdv_synthetic_data[\"order\"])\n", "\n", "total_synthetic_records = (\n", - " synthetic_client_count +\n", - " synthetic_disposition_count +\n", - " synthetic_account_count +\n", - " synthetic_card_count +\n", - " synthetic_transaction_count +\n", - " synthetic_loan_count +\n", - " synthetic_order_count\n", + " synthetic_client_count\n", + " + synthetic_disposition_count\n", + " + synthetic_account_count\n", + " + synthetic_card_count\n", + " + synthetic_transaction_count\n", + " + synthetic_loan_count\n", + " + synthetic_order_count\n", ")\n", "\n", "generation_rate = total_synthetic_records / (end_time - start_time)\n", "\n", "print(f\"šŸš€ Generation rate: {generation_rate:,.0f} records/second\")\n", - "print(f\"šŸ“Š Synthetic data breakdown:\")\n", + "print(\"šŸ“Š Synthetic data breakdown:\")\n", "print(f\" - Clients: {synthetic_client_count:,}\")\n", "print(f\" - Dispositions: {synthetic_disposition_count:,}\")\n", "print(f\" - Accounts: {synthetic_account_count:,}\")\n", @@ -481,13 +479,13 @@ "# Quality metric example: Dispositions per client ratio\n", "dispositions_per_client = synthetic_disposition_count / synthetic_client_count if synthetic_client_count > 0 else 0\n", "\n", - "print(f\"\\nšŸ” Generation Quality Metrics:\")\n", + "print(\"\\nšŸ” Generation Quality Metrics:\")\n", "print(f\" - Dispositions per client: {dispositions_per_client:.1f}\")\n", "print(f\" - Scale factor achieved: {synthetic_client_count / len(clients_train):.2f}x\")\n", "\n", "# Preview of generated synthetic data\n", - "sdv_synthetic_data['client'].head()\n", - "sdv_synthetic_data['disposition'].head()" + "sdv_synthetic_data[\"client\"].head()\n", + "sdv_synthetic_data[\"disposition\"].head()" ] }, { @@ -497,21 +495,21 @@ "outputs": [], "source": [ "# Save SDV synthetic data for comparison\n", - "output_folder = './data/'\n", + "output_folder = \"./data/\"\n", "\n", "synthetic_files = {\n", - " 'client': f'{output_folder}sdv_client.parquet',\n", - " 'disposition': f'{output_folder}sdv_disposition.parquet',\n", - " 'account': f'{output_folder}sdv_account.parquet',\n", - " 'card': f'{output_folder}sdv_card.parquet',\n", - " 'transaction': f'{output_folder}sdv_transaction.parquet',\n", - " 'loan': f'{output_folder}sdv_loan.parquet',\n", - " 'order': f'{output_folder}sdv_order.parquet',\n", + " \"client\": f\"{output_folder}sdv_client.parquet\",\n", + " \"disposition\": f\"{output_folder}sdv_disposition.parquet\",\n", + " \"account\": f\"{output_folder}sdv_account.parquet\",\n", + " \"card\": f\"{output_folder}sdv_card.parquet\",\n", + " \"transaction\": f\"{output_folder}sdv_transaction.parquet\",\n", + " \"loan\": f\"{output_folder}sdv_loan.parquet\",\n", + " \"order\": f\"{output_folder}sdv_order.parquet\",\n", "}\n", "\n", "for table_name, file_path in synthetic_files.items():\n", " sdv_synthetic_data[table_name].to_parquet(file_path, index=False)\n", - " print(f\"šŸ’¾ Saved {table_name} synthetic data to: {file_path}\")\n" + " print(f\"šŸ’¾ Saved {table_name} synthetic data to: {file_path}\")" ] }, { @@ -606,100 +604,92 @@ "\n", "# Configure the generator for comprehensive multi-table setup\n", "config = {\n", - " 'name': 'Berka Multi-Table Generator',\n", - " 'tables': [\n", + " \"name\": \"Berka Multi-Table Generator\",\n", + " \"tables\": [\n", " {\n", - " 'name': 'client',\n", - " 'data': clients_train,\n", - " 'primary_key': 'client_id',\n", - " 'tabular_model_configuration': {\n", - " 'model': 'MOSTLY_AI/Medium',\n", - " 'max_training_time': 10,\n", - " 'enable_flexible_generation': False,\n", - " 'value_protection': False,\n", - " 'enable_model_report': False\n", - " }\n", + " \"name\": \"client\",\n", + " \"data\": clients_train,\n", + " \"primary_key\": \"client_id\",\n", + " \"tabular_model_configuration\": {\n", + " \"model\": \"MOSTLY_AI/Medium\",\n", + " \"max_training_time\": 10,\n", + " \"enable_flexible_generation\": False,\n", + " \"value_protection\": False,\n", + " \"enable_model_report\": False,\n", + " },\n", " },\n", " {\n", - " 'name': 'account',\n", - " 'data': accounts_train,\n", - " 'primary_key': 'account_id',\n", - " 'tabular_model_configuration': {\n", - " 'model': 'MOSTLY_AI/Medium',\n", - " 'max_training_time': 10,\n", - " 'enable_model_report': False\n", - " }\n", + " \"name\": \"account\",\n", + " \"data\": accounts_train,\n", + " \"primary_key\": \"account_id\",\n", + " \"tabular_model_configuration\": {\n", + " \"model\": \"MOSTLY_AI/Medium\",\n", + " \"max_training_time\": 10,\n", + " \"enable_model_report\": False,\n", + " },\n", " },\n", " {\n", - " 'name': 'disposition',\n", - " 'data': dispositions_train,\n", - " 'primary_key': 'disp_id',\n", - " 'foreign_keys': [\n", - " {'column': 'client_id', 'referenced_table': 'client', 'is_context': False},\n", - " {'column': 'account_id', 'referenced_table': 'account', 'is_context': True}\n", + " \"name\": \"disposition\",\n", + " \"data\": dispositions_train,\n", + " \"primary_key\": \"disp_id\",\n", + " \"foreign_keys\": [\n", + " {\"column\": \"client_id\", \"referenced_table\": \"client\", \"is_context\": False},\n", + " {\"column\": \"account_id\", \"referenced_table\": \"account\", \"is_context\": True},\n", " ],\n", - " 'tabular_model_configuration': {\n", - " 'model': 'MOSTLY_AI/Medium',\n", - " 'max_training_time': 10,\n", - " 'enable_model_report': False\n", - " }\n", + " \"tabular_model_configuration\": {\n", + " \"model\": \"MOSTLY_AI/Medium\",\n", + " \"max_training_time\": 10,\n", + " \"enable_model_report\": False,\n", + " },\n", " },\n", " {\n", - " 'name': 'card',\n", - " 'data': cards_train,\n", - " 'primary_key': 'card_id',\n", - " 'foreign_keys': [\n", - " {'column': 'disp_id', 'referenced_table': 'disposition', 'is_context': True}\n", - " ],\n", - " 'tabular_model_configuration': {\n", - " 'model': 'MOSTLY_AI/Medium',\n", - " 'max_training_time': 10,\n", - " 'enable_model_report': False\n", - " }\n", + " \"name\": \"card\",\n", + " \"data\": cards_train,\n", + " \"primary_key\": \"card_id\",\n", + " \"foreign_keys\": [{\"column\": \"disp_id\", \"referenced_table\": \"disposition\", \"is_context\": True}],\n", + " \"tabular_model_configuration\": {\n", + " \"model\": \"MOSTLY_AI/Medium\",\n", + " \"max_training_time\": 10,\n", + " \"enable_model_report\": False,\n", + " },\n", " },\n", " {\n", - " 'name': 'transaction',\n", - " 'data': transactions_train,\n", - " 'primary_key': 'trans_id',\n", - " 'foreign_keys': [\n", - " {'column': 'account_id', 'referenced_table': 'account', 'is_context': True}\n", - " ],\n", - " 'tabular_model_configuration': {\n", - " 'model': 'MOSTLY_AI/Medium',\n", - " 'max_training_time': 10,\n", - " 'enable_model_report': False\n", - " }\n", + " \"name\": \"transaction\",\n", + " \"data\": transactions_train,\n", + " \"primary_key\": \"trans_id\",\n", + " \"foreign_keys\": [{\"column\": \"account_id\", \"referenced_table\": \"account\", \"is_context\": True}],\n", + " \"tabular_model_configuration\": {\n", + " \"model\": \"MOSTLY_AI/Medium\",\n", + " \"max_training_time\": 10,\n", + " \"enable_model_report\": False,\n", + " },\n", " },\n", " {\n", - " 'name': 'loan',\n", - " 'data': loans_train,\n", - " 'primary_key': 'loan_id',\n", - " 'foreign_keys': [\n", - " {'column': 'account_id', 'referenced_table': 'account', 'is_context': True}\n", - " ],\n", - " 'tabular_model_configuration': {\n", - " 'model': 'MOSTLY_AI/Medium',\n", - " 'max_training_time': 10,\n", - " 'enable_model_report': False\n", - " }\n", + " \"name\": \"loan\",\n", + " \"data\": loans_train,\n", + " \"primary_key\": \"loan_id\",\n", + " \"foreign_keys\": [{\"column\": \"account_id\", \"referenced_table\": \"account\", \"is_context\": True}],\n", + " \"tabular_model_configuration\": {\n", + " \"model\": \"MOSTLY_AI/Medium\",\n", + " \"max_training_time\": 10,\n", + " \"enable_model_report\": False,\n", + " },\n", " },\n", " {\n", - " 'name': 'order',\n", - " 'data': orders_train,\n", - " 'primary_key': 'order_id',\n", - " 'foreign_keys': [\n", - " {'column': 'account_id', 'referenced_table': 'account', 'is_context': True}\n", - " ],\n", - " 'tabular_model_configuration': {\n", - " 'model': 'MOSTLY_AI/Medium',\n", - " 'max_training_time': 10,\n", - " 'enable_model_report': False\n", - " }\n", - " }\n", - " ]\n", + " \"name\": \"order\",\n", + " \"data\": orders_train,\n", + " \"primary_key\": \"order_id\",\n", + " \"foreign_keys\": [{\"column\": \"account_id\", \"referenced_table\": \"account\", \"is_context\": True}],\n", + " \"tabular_model_configuration\": {\n", + " \"model\": \"MOSTLY_AI/Medium\",\n", + " \"max_training_time\": 10,\n", + " \"enable_model_report\": False,\n", + " },\n", + " },\n", + " ],\n", "}\n", "\n", - "print(\"āœ… MOSTLY AI generator configuration ready.\")\n" + "print(\"āœ… MOSTLY AI generator configuration ready.\")" ] }, { @@ -737,10 +727,9 @@ "end_time = time.time()\n", "elapsed_minutes = (end_time - start_time) / 60\n", "\n", - "print(f\"āœ… MOSTLY AI training completed successfully!\")\n", + "print(\"āœ… MOSTLY AI training completed successfully!\")\n", "print(f\"ā±ļø Total training time: {elapsed_minutes:.2f} minutes\")\n", - "print(f\"šŸ§  Advanced AI models trained for multi-table synthesis\")\n", - " " + "print(\"šŸ§  Advanced AI models trained for multi-table synthesis\")" ] }, { @@ -771,7 +760,7 @@ "total_records = sum(len(mostlyai_synthetic_data[table]) for table in mostlyai_synthetic_data.keys())\n", "generation_rate = total_records / (end_time - start_time)\n", "\n", - "print(f\"āœ… MOSTLY AI generation completed successfully!\")\n", + "print(\"āœ… MOSTLY AI generation completed successfully!\")\n", "print(f\"ā±ļø Generation time: {elapsed_minutes:.2f} minutes\")\n", "print(f\"šŸš€ Generation rate: {generation_rate:,.0f} records/second\")\n", "\n", @@ -780,14 +769,14 @@ " print(f\" - {table_name.capitalize()}: {len(mostlyai_synthetic_data[table_name]):,} rows\")\n", "\n", "# Quality metrics example: Dispositions per client ratio\n", - "dispositions_per_client = len(mostlyai_synthetic_data['disposition']) / len(mostlyai_synthetic_data['client'])\n", - "print(f\"\\nšŸ” Quality Metrics:\")\n", + "dispositions_per_client = len(mostlyai_synthetic_data[\"disposition\"]) / len(mostlyai_synthetic_data[\"client\"])\n", + "print(\"\\nšŸ” Quality Metrics:\")\n", "print(f\" - Dispositions per client: {dispositions_per_client:.1f}\")\n", - "print(f\" - Multi-level foreign key integrity handled āœ…\")\n", + "print(\" - Multi-level foreign key integrity handled āœ…\")\n", "\n", "# Preview\n", - "mostlyai_synthetic_data['client'].head()\n", - "mostlyai_synthetic_data['disposition'].head()" + "mostlyai_synthetic_data[\"client\"].head()\n", + "mostlyai_synthetic_data[\"disposition\"].head()" ] }, { @@ -797,21 +786,21 @@ "outputs": [], "source": [ "# Save MOSTLY AI synthetic data for comparison\n", - "output_folder = './data/'\n", + "output_folder = \"./data/\"\n", "\n", "mostlyai_files = {\n", - " 'client': f'{output_folder}mostlyai_client.parquet',\n", - " 'disposition': f'{output_folder}mostlyai_disposition.parquet',\n", - " 'account': f'{output_folder}mostlyai_account.parquet',\n", - " 'card': f'{output_folder}mostlyai_card.parquet',\n", - " 'transaction': f'{output_folder}mostlyai_transaction.parquet',\n", - " 'loan': f'{output_folder}mostlyai_loan.parquet',\n", - " 'order': f'{output_folder}mostlyai_order.parquet',\n", + " \"client\": f\"{output_folder}mostlyai_client.parquet\",\n", + " \"disposition\": f\"{output_folder}mostlyai_disposition.parquet\",\n", + " \"account\": f\"{output_folder}mostlyai_account.parquet\",\n", + " \"card\": f\"{output_folder}mostlyai_card.parquet\",\n", + " \"transaction\": f\"{output_folder}mostlyai_transaction.parquet\",\n", + " \"loan\": f\"{output_folder}mostlyai_loan.parquet\",\n", + " \"order\": f\"{output_folder}mostlyai_order.parquet\",\n", "}\n", "\n", "for table_name, file_path in mostlyai_files.items():\n", " mostlyai_synthetic_data[table_name].to_parquet(file_path, index=False)\n", - " print(f\"šŸ’¾ Saved {table_name} synthetic data to: {file_path}\")\n" + " print(f\"šŸ’¾ Saved {table_name} synthetic data to: {file_path}\")" ] }, { @@ -854,12 +843,12 @@ "outputs": [], "source": [ "# Load the split files from the disk\n", - "transactions_train = pd.read_parquet('./data/transactions_train.parquet')\n", - "transactions_test = pd.read_parquet('./data/transactions_test.parquet')\n", - "accounts_train = pd.read_parquet('./data/accounts_train.parquet')\n", - "accounts_test = pd.read_parquet('./data/accounts_test.parquet')\n", + "transactions_train = pd.read_parquet(\"./data/transactions_train.parquet\")\n", + "transactions_test = pd.read_parquet(\"./data/transactions_test.parquet\")\n", + "accounts_train = pd.read_parquet(\"./data/accounts_train.parquet\")\n", + "accounts_test = pd.read_parquet(\"./data/accounts_test.parquet\")\n", "\n", - "print(\"šŸ“‚ Training and test datasets loaded successfully\")\n" + "print(\"šŸ“‚ Training and test datasets loaded successfully\")" ] }, { @@ -871,30 +860,32 @@ "print(\"šŸ“Š Evaluating SDV Transaction synthetic data quality...\")\n", "\n", "# Load the SDV synthetic dataset\n", - "sdv_transaction = pd.read_parquet('./data/sdv_transaction.parquet')\n", + "sdv_transaction = pd.read_parquet(\"./data/sdv_transaction.parquet\")\n", "\n", "# Define ID columns to exclude from QA analysis (do not exclude account_id here!)\n", - "id_columns_to_exclude = ['trans_id']\n", + "id_columns_to_exclude = [\"trans_id\"]\n", + "\n", "\n", "def remove_id_columns(df, columns_to_remove):\n", " return df.drop(columns=[col for col in columns_to_remove if col in df.columns])\n", "\n", + "\n", "# Prepare transaction data (remove only trans_id)\n", "sdv_transaction_qa = remove_id_columns(sdv_transaction, id_columns_to_exclude)\n", "transactions_train_qa = remove_id_columns(transactions_train, id_columns_to_exclude)\n", "transactions_test_qa = remove_id_columns(transactions_test, id_columns_to_exclude)\n", "\n", "report_path, metrics = qa.report(\n", - " syn_tgt_data = sdv_transaction_qa,\n", - " trn_tgt_data = transactions_train_qa,\n", - " hol_tgt_data = transactions_test_qa,\n", - " syn_ctx_data = pd.read_parquet('./data/sdv_account.parquet'),\n", - " trn_ctx_data = accounts_train,\n", - " hol_ctx_data = accounts_test,\n", - " ctx_primary_key = \"account_id\",\n", - " tgt_context_key = \"account_id\",\n", + " syn_tgt_data=sdv_transaction_qa,\n", + " trn_tgt_data=transactions_train_qa,\n", + " hol_tgt_data=transactions_test_qa,\n", + " syn_ctx_data=pd.read_parquet(\"./data/sdv_account.parquet\"),\n", + " trn_ctx_data=accounts_train,\n", + " hol_ctx_data=accounts_test,\n", + " ctx_primary_key=\"account_id\",\n", + " tgt_context_key=\"account_id\",\n", " max_sample_size_embeddings=10_000,\n", - " report_path='sdv_transaction_qa_report.html'\n", + " report_path=\"sdv_transaction_qa_report.html\",\n", ")\n", "\n", "print(f\"šŸ“‹ SDV Transaction Quality Report saved to: {report_path}\")\n", @@ -902,7 +893,7 @@ "print(metrics.model_dump_json(indent=4))\n", "\n", "sdv_transaction_accuracy = metrics.accuracy.overall\n", - "sdv_transaction_dcr_share = metrics.distances.dcr_share\n" + "sdv_transaction_dcr_share = metrics.distances.dcr_share" ] }, { @@ -914,30 +905,32 @@ "print(\"šŸ“Š Evaluating MOSTLY AI Transactions synthetic data quality...\")\n", "\n", "# Load the MOSTLY AI synthetic dataset\n", - "mostlyai_transaction = pd.read_parquet('./data/mostlyai_transaction.parquet')\n", + "mostlyai_transaction = pd.read_parquet(\"./data/mostlyai_transaction.parquet\")\n", "\n", "# Define ID columns to exclude from QA analysis (do not exclude account_id here!)\n", - "id_columns_to_exclude = ['trans_id']\n", + "id_columns_to_exclude = [\"trans_id\"]\n", + "\n", "\n", "def remove_id_columns(df, columns_to_remove):\n", " return df.drop(columns=[col for col in columns_to_remove if col in df.columns])\n", "\n", + "\n", "# Prepare transaction data (remove only trans_id)\n", "mostlyai_transaction_qa = remove_id_columns(mostlyai_transaction, id_columns_to_exclude)\n", "transactions_train_qa = remove_id_columns(transactions_train, id_columns_to_exclude)\n", "transactions_test_qa = remove_id_columns(transactions_test, id_columns_to_exclude)\n", "\n", "report_path, metrics = qa.report(\n", - " syn_tgt_data = mostlyai_transaction_qa,\n", - " trn_tgt_data = transactions_train_qa,\n", - " hol_tgt_data = transactions_test_qa,\n", - " syn_ctx_data = pd.read_parquet('./data/mostlyai_account.parquet'),\n", - " trn_ctx_data = accounts_train,\n", - " hol_ctx_data = accounts_test,\n", - " ctx_primary_key = \"account_id\",\n", - " tgt_context_key = \"account_id\",\n", + " syn_tgt_data=mostlyai_transaction_qa,\n", + " trn_tgt_data=transactions_train_qa,\n", + " hol_tgt_data=transactions_test_qa,\n", + " syn_ctx_data=pd.read_parquet(\"./data/mostlyai_account.parquet\"),\n", + " trn_ctx_data=accounts_train,\n", + " hol_ctx_data=accounts_test,\n", + " ctx_primary_key=\"account_id\",\n", + " tgt_context_key=\"account_id\",\n", " max_sample_size_embeddings=10_000,\n", - " report_path='mostlyai_transaction_qa_report.html'\n", + " report_path=\"mostlyai_transaction_qa_report.html\",\n", ")\n", "\n", "print(f\"šŸ“‹ MOSTLY AI Transaction Quality Report saved to: {report_path}\")\n", @@ -945,7 +938,7 @@ "print(metrics.model_dump_json(indent=4))\n", "\n", "mostlyai_transaction_accuracy = metrics.accuracy.overall\n", - "mostlyai_transaction_dcr_share = metrics.distances.dcr_share\n" + "mostlyai_transaction_dcr_share = metrics.distances.dcr_share" ] }, { @@ -955,11 +948,13 @@ "outputs": [], "source": [ "# Add a final comparison section\n", - "print(\"\\n\" + \"=\"*60)\n", + "print(\"\\n\" + \"=\" * 60)\n", "print(\"šŸ† FINAL COMPARISON\")\n", - "print(\"=\"*60)\n", + "print(\"=\" * 60)\n", "print(f\"SDV Transaction - Accuracy: {sdv_transaction_accuracy:.3f}, DCR Share: {sdv_transaction_dcr_share:.3f}\")\n", - "print(f\"MOSTLY AI Transaction- Accuracy: {mostlyai_transaction_accuracy:.3f}, DCR Share: {mostlyai_transaction_dcr_share:.3f}\")\n", + "print(\n", + " f\"MOSTLY AI Transaction- Accuracy: {mostlyai_transaction_accuracy:.3f}, DCR Share: {mostlyai_transaction_dcr_share:.3f}\"\n", + ")\n", "\n", "print(\"\\nšŸ” METRIC INTERPRETATION:\")\n", "print(\"ā€¢ Higher accuracy = better statistical fidelity\")\n", @@ -968,7 +963,7 @@ "print(\"\\nšŸ“Š ANALYSIS:\")\n", "print(\"ā€¢ MOSTLY AI consistently shows higher accuracy than SDV\")\n", "print(\"ā€¢ Both frameworks maintain reasonable DCR Share values around 0.5\")\n", - "print(\"ā€¢ MOSTLY AI handles multi-table relationships and foreign keys with greater precision\")\n" + "print(\"ā€¢ MOSTLY AI handles multi-table relationships and foreign keys with greater precision\")" ] } ], From 186982587e4204e115eb58eef6a6c7a91732f6d8 Mon Sep 17 00:00:00 2001 From: Kenneth Hamilton Date: Tue, 29 Jul 2025 10:47:35 +0200 Subject: [PATCH 3/8] new notebook --- .../multi-table-scenario.ipynb | 991 ------------------ .../two-table-scenario.ipynb | 740 +++++++++++++ 2 files changed, 740 insertions(+), 991 deletions(-) delete mode 100644 docs/tutorials/sdv-comparison/multi-table-scenario/multi-table-scenario.ipynb create mode 100644 docs/tutorials/sdv-comparison/two-table-scenario/two-table-scenario.ipynb diff --git a/docs/tutorials/sdv-comparison/multi-table-scenario/multi-table-scenario.ipynb b/docs/tutorials/sdv-comparison/multi-table-scenario/multi-table-scenario.ipynb deleted file mode 100644 index a2d4da3d..00000000 --- a/docs/tutorials/sdv-comparison/multi-table-scenario/multi-table-scenario.ipynb +++ /dev/null @@ -1,991 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# MOSTLY AI vs. SDV Comparison - Multi-Table Scenario \"Run\n", - "\n", - "This notebook provides a comprehensive comparison between two leading synthetic data generation platforms:\n", - "- **SDV (Synthetic Data Vault)** - Business Source License\n", - "- **MOSTLY AI Synthetic Data SDK** - Apache 2.0 License - Open Source\n", - "\n", - "In this comparison, we are going to walk through the synthesis of a relational multi-table structure using the [Berka dataset](https://github.com/mostly-ai/public-demo-data/tree/dev/berka/data).\n", - "\n", - "## Comparison Methodology\n", - "\n", - "1. **Data Preparation**: Load, inspect, and preprocess the multi-table dataset\n", - "2. **Data Splitting**: Create train/test splits while maintaining referential integrity\n", - "3. **Model Training**: Train both SDV and MOSTLY AI generators on the training data\n", - "4. **Synthetic Data Generation**: Generate synthetic datasets using both platforms\n", - "5. **Performance Analysis**: Compare training time, generation speed, and data quality\n", - "\n", - "## Key Challenges in Multi-Table Synthesis\n", - "\n", - "- **Referential Integrity**: Maintaining foreign key relationships between tables\n", - "- **Sequential Dependencies**: Preserving temporal patterns in transaction data\n", - "- **Data Quality**: Ensuring synthetic data maintains statistical properties and business logic" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Install SDK in CLIENT mode\n", - "!uv pip install -U mostlyai sdv graphviz\n", - "# Or install in LOCAL mode\n", - "!uv pip install -U 'mostlyai[local]' sdv graphviz\n", - "# Note: Restart kernel session after installation!" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 1. Data Loading and Initial Exploration\n", - "\n", - "First, let's load our multi-table dataset and examine its structure to understand:\n", - "- Table schemas and data types\n", - "- Data quality and completeness\n", - "- Relationships between tables\n", - "- Business logic and constraints\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import pandas as pd\n", - "\n", - "base_url = \"https://github.com/mostly-ai/public-demo-data/raw/dev/berka/data/\"\n", - "originals = {\n", - " \"client\": pd.read_csv(base_url + \"client.csv.gz\", low_memory=False),\n", - " \"disposition\": pd.read_csv(base_url + \"disp.csv.gz\", low_memory=False),\n", - " \"card\": pd.read_csv(base_url + \"card.csv.gz\", low_memory=False),\n", - " \"account\": pd.read_csv(base_url + \"account.csv.gz\", low_memory=False),\n", - " \"transaction\": pd.read_csv(base_url + \"trans.csv.gz\", low_memory=False),\n", - " \"loan\": pd.read_csv(base_url + \"loan.csv.gz\", low_memory=False),\n", - " \"order\": pd.read_csv(base_url + \"order.csv.gz\", low_memory=False),\n", - "}\n", - "originals[\"account\"][\"date\"] = pd.to_datetime(originals[\"account\"][\"date\"])\n", - "originals[\"transaction\"][\"date\"] = pd.to_datetime(originals[\"transaction\"][\"date\"])\n", - "\n", - "for k in originals:\n", - " print(\"===\", k, \"===\")\n", - " display(originals[k].sample(n=3))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 2. Data Preprocessing: Establishing Foreign Key Relationships\n", - "\n", - "We need to establish proper foreign key relationships amongst each of the tables. This step is crucial for:\n", - "- Maintaining referential integrity in synthetic data\n", - "- Enabling proper multi-table synthesis\n", - "- Ensuring realistic transaction patterns" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print(\"šŸ”— Validating foreign key relationships...\")\n", - "\n", - "# Build lookup sets for validation\n", - "client_ids = set(originals[\"client\"][\"client_id\"])\n", - "account_ids = set(originals[\"account\"][\"account_id\"])\n", - "disp_ids = set(originals[\"disposition\"][\"disp_id\"])\n", - "\n", - "# Define FK relationships as structured data\n", - "fk_relationships = [\n", - " {\"table\": \"disposition\", \"column\": \"client_id\", \"target_table\": \"client\", \"valid_ids\": client_ids},\n", - " {\"table\": \"disposition\", \"column\": \"account_id\", \"target_table\": \"account\", \"valid_ids\": account_ids},\n", - " {\"table\": \"card\", \"column\": \"disp_id\", \"target_table\": \"disposition\", \"valid_ids\": disp_ids},\n", - " {\"table\": \"transaction\", \"column\": \"account_id\", \"target_table\": \"account\", \"valid_ids\": account_ids},\n", - " {\"table\": \"loan\", \"column\": \"account_id\", \"target_table\": \"account\", \"valid_ids\": account_ids},\n", - " {\"table\": \"order\", \"column\": \"account_id\", \"target_table\": \"account\", \"valid_ids\": account_ids},\n", - "]\n", - "\n", - "validation_summary = []\n", - "\n", - "dataframes = originals.copy()\n", - "\n", - "\n", - "def validate_fk(df, column, valid_ids, table, target_table):\n", - " before = len(df)\n", - " missing_mask = ~df[column].isin(valid_ids)\n", - " missing_count = missing_mask.sum()\n", - " if missing_count > 0:\n", - " message = (\n", - " f\"āš ļø {table}.{column} ā†’ {target_table}: {missing_count:,} invalid foreign keys detected \"\n", - " f\"(checked {before:,} rows)\"\n", - " )\n", - " df.loc[missing_mask, column] = pd.NA\n", - " else:\n", - " message = f\"āœ… {table}.{column} ā†’ {target_table}: All foreign keys valid ({before:,} rows checked)\"\n", - " validation_summary.append(message)\n", - " return df\n", - "\n", - "\n", - "# Perform validation based on the relationship list\n", - "for rel in fk_relationships:\n", - " table_name = rel[\"table\"]\n", - " column = rel[\"column\"]\n", - " valid_ids = rel[\"valid_ids\"]\n", - " target_table = rel[\"target_table\"]\n", - " dataframes[table_name] = validate_fk(dataframes[table_name], column, valid_ids, table_name, target_table)\n", - "\n", - "print(\"\\nāœ… Referential integrity validation complete!\\n\")\n", - "print(\"šŸ“„ Foreign Key Validation Summary:\")\n", - "for line in validation_summary:\n", - " print(line)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 3. Strategic Data Splitting for Multi-Table Scenarios\n", - "\n", - "When dealing with related tables like accounts, dispositions and transactions, data splitting becomes more complex than simple random sampling due to the links between data tables. \n", - "\n", - "In order to make coherent assessments of data quality, we need to create meaningful train and test cohorts.\n", - "\n", - "**Key Considerations:**\n", - "- **Referential Integrity:** Ensure foreign key relationships remain valid in both splits.\n", - "- **Business Logic:** Accounts, loans, and transactions can only exist in the training set if their associated client is also in the training set.\n", - "- **Data Leakage Prevention:** Avoid information bleeding between train/test sets.\n", - "\n", - "**Our Approach:**\n", - "1. **Split accounts first (80/20 train/test)**: We split the `account` table using an 80/20 ratio. This ensures customer-related information is kept together per split.\n", - " \n", - "2. **Assign related tables based on account membership:** \n", - " - **Dispositions:** Linked using `account_id`. Only dispositions referencing accounts in the training set go to the training set.\n", - " - **Cards:** Linked via `disp_id`. Cards follow the disposition-account split.\n", - " - **Transactions, Loans, Orders:** All linked via `account_id`. These tables follow the same assignment logic.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from sklearn.model_selection import train_test_split\n", - "\n", - "print(\"āœ‚ļø Performing strategic multi-table data splitting based on accounts...\")\n", - "\n", - "# Step 1: Split accounts using 80/20 ratio\n", - "accounts_train, accounts_test = train_test_split(dataframes[\"account\"], test_size=0.2, random_state=42)\n", - "\n", - "print(\"šŸ¦ Account split:\")\n", - "print(\n", - " f\" - Training set: {len(accounts_train):,} accounts ({len(accounts_train) / len(dataframes['account']) * 100:.1f}%)\"\n", - ")\n", - "print(f\" - Test set: {len(accounts_test):,} accounts ({len(accounts_test) / len(dataframes['account']) * 100:.1f}%)\")\n", - "\n", - "# Step 2: Create account ID sets for lookup\n", - "train_account_ids = set(accounts_train[\"account_id\"])\n", - "test_account_ids = set(accounts_test[\"account_id\"])" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print(\"šŸ”„ Assigning related tables based on account split...\")\n", - "\n", - "# Dispositions linked to accounts\n", - "dispositions_train = dataframes[\"disposition\"][dataframes[\"disposition\"][\"account_id\"].isin(train_account_ids)].copy()\n", - "dispositions_test = dataframes[\"disposition\"][dataframes[\"disposition\"][\"account_id\"].isin(test_account_ids)].copy()\n", - "\n", - "train_disp_ids = set(dispositions_train[\"disp_id\"])\n", - "test_disp_ids = set(dispositions_test[\"disp_id\"])\n", - "\n", - "# Cards linked to dispositions\n", - "cards_train = dataframes[\"card\"][dataframes[\"card\"][\"disp_id\"].isin(train_disp_ids)].copy()\n", - "cards_test = dataframes[\"card\"][dataframes[\"card\"][\"disp_id\"].isin(test_disp_ids)].copy()\n", - "\n", - "# Clients linked via disposition ā†’ account\n", - "client_ids_train = set(dispositions_train[\"client_id\"])\n", - "client_ids_test = set(dispositions_test[\"client_id\"])\n", - "\n", - "clients_train = dataframes[\"client\"][dataframes[\"client\"][\"client_id\"].isin(client_ids_train)].copy()\n", - "clients_test = dataframes[\"client\"][dataframes[\"client\"][\"client_id\"].isin(client_ids_test)].copy()\n", - "\n", - "# Transactions linked to accounts\n", - "transactions_train = dataframes[\"transaction\"][dataframes[\"transaction\"][\"account_id\"].isin(train_account_ids)].copy()\n", - "transactions_test = dataframes[\"transaction\"][dataframes[\"transaction\"][\"account_id\"].isin(test_account_ids)].copy()\n", - "\n", - "# Loans linked to accounts\n", - "loans_train = dataframes[\"loan\"][dataframes[\"loan\"][\"account_id\"].isin(train_account_ids)].copy()\n", - "loans_test = dataframes[\"loan\"][dataframes[\"loan\"][\"account_id\"].isin(test_account_ids)].copy()\n", - "\n", - "# Orders linked to accounts\n", - "orders_train = dataframes[\"order\"][dataframes[\"order\"][\"account_id\"].isin(train_account_ids)].copy()\n", - "orders_test = dataframes[\"order\"][dataframes[\"order\"][\"account_id\"].isin(test_account_ids)].copy()\n", - "\n", - "print(\"āœ… Splitting complete!\")\n", - "print(f\" - Training accounts: {len(accounts_train):,}\")\n", - "print(f\" - Training dispositions: {len(dispositions_train):,}\")\n", - "print(f\" - Training clients: {len(clients_train):,}\")\n", - "print(f\" - Training cards: {len(cards_train):,}\")\n", - "print(f\" - Training transactions: {len(transactions_train):,}\")\n", - "print(f\" - Training loans: {len(loans_train):,}\")\n", - "print(f\" - Training orders: {len(orders_train):,}\")\n", - "\n", - "print(f\" - Test accounts: {len(accounts_test):,}\")\n", - "print(f\" - Test dispositions: {len(dispositions_test):,}\")\n", - "print(f\" - Test clients: {len(clients_test):,}\")\n", - "print(f\" - Test cards: {len(cards_test):,}\")\n", - "print(f\" - Test transactions: {len(transactions_test):,}\")\n", - "print(f\" - Test loans: {len(loans_test):,}\")\n", - "print(f\" - Test orders: {len(orders_test):,}\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print(\"šŸ’¾ Saving split train/test tables...\")\n", - "\n", - "dispositions_train.to_parquet(\"./data/dispositions_train.parquet\", index=False)\n", - "dispositions_test.to_parquet(\"./data/dispositions_test.parquet\", index=False)\n", - "cards_train.to_parquet(\"./data/cards_train.parquet\", index=False)\n", - "cards_test.to_parquet(\"./data/cards_test.parquet\", index=False)\n", - "accounts_train.to_parquet(\"./data/accounts_train.parquet\", index=False)\n", - "accounts_test.to_parquet(\"./data/accounts_test.parquet\", index=False)\n", - "transactions_train.to_parquet(\"./data/transactions_train.parquet\", index=False)\n", - "transactions_test.to_parquet(\"./data/transactions_test.parquet\", index=False)\n", - "loans_train.to_parquet(\"./data/loans_train.parquet\", index=False)\n", - "loans_test.to_parquet(\"./data/loans_test.parquet\", index=False)\n", - "orders_train.to_parquet(\"./data/orders_train.parquet\", index=False)\n", - "orders_test.to_parquet(\"./data/orders_test.parquet\", index=False)\n", - "clients_train.to_parquet(\"./data/clients_train.parquet\", index=False)\n", - "clients_test.to_parquet(\"./data/clients_test.parquet\", index=False)\n", - "\n", - "print(\"āœ… All train/test splits saved to ./data/\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 4. SDV (Synthetic Data Vault) Implementation\n", - "\n", - "**About SDV:**\n", - "- Business Source License Python library for synthetic data generation\n", - "- Supports single-table and multi-table scenarios\n", - "- Uses statistical modeling and machine learning approaches\n", - "- Provides HMASynthesizer for hierarchical multi-table synthesis\n", - "\n", - "**Key Features:**\n", - "- **Metadata Detection**: Automatically infers data types and relationships\n", - "- **Relationship Modeling**: Handles parent-child table relationships\n", - "- **Privacy Protection**: Generates synthetic data that preserves statistical properties while protecting individual privacy\n", - "- **Extensible**: Multiple synthesizer options (GaussianCopula, CTGAN, CopulaGAN, etc.)\n", - "\n", - "**Limitations:**\n", - "- Current version only supports one parent per child table\n", - "- Complex multi-parent relationships require modeling simplification\n", - "- Performance scales with data complexity\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from sdv.metadata import Metadata\n", - "from sdv.multi_table import HMASynthesizer\n", - "\n", - "print(\"šŸ—ļø Building SDV metadata configuration...\")\n", - "\n", - "metadata = Metadata.detect_from_dataframes(\n", - " data={\n", - " \"client\": clients_train,\n", - " \"disposition\": dispositions_train,\n", - " \"account\": accounts_train,\n", - " \"card\": cards_train,\n", - " \"transaction\": transactions_train,\n", - " \"loan\": loans_train,\n", - " \"order\": orders_train,\n", - " },\n", - " infer_keys=\"primary_and_foreign\",\n", - ")\n", - "\n", - "print(\"āœ… Base metadata auto-detected with relationships\")\n", - "\n", - "# View auto-detected relationships graphically\n", - "metadata.visualize()\n", - "\n", - "# Inspect relationships and table configuration as raw dictionary\n", - "metadata_dict = metadata.to_dict()\n", - "print(\"\\nšŸ“‹ Complete SDV Metadata Dictionary:\")\n", - "print(metadata_dict)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 4.2 SDV Model Training\n", - "\n", - "**HMASynthesizer Overview:**\n", - "- **Hierarchical Modeling**: Learns parent-child relationships\n", - "- **Statistical Approach**: Uses copulas and Gaussian distributions\n", - "- **Multi-step Process**: \n", - " 1. Preprocesses tables and infers constraints\n", - " 2. Learns relationships between parent and child tables\n", - " 3. Models individual table distributions\n", - " \n", - "**Training Phases:**\n", - "- **Preprocess Tables**: Data cleaning and type inference\n", - "- **Learning Relationships**: Analyzing foreign key dependencies \n", - "- **Modeling Tables**: Learning statistical distributions for synthesis\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import time\n", - "\n", - "print(\"šŸš€ Starting SDV training process...\")\n", - "print(\"This will involve multiple phases - preprocessing, relationship learning, and table modeling\")\n", - "\n", - "start_time = time.time()\n", - "\n", - "# Initialize the HMASynthesizer with our configured metadata\n", - "print(\"šŸ”§ Initializing HMASynthesizer...\")\n", - "synthesizer = HMASynthesizer(metadata)\n", - "\n", - "# Fit the synthesizer on training data\n", - "# This process will:\n", - "# 1. Preprocess all tables (clean data, infer constraints)\n", - "# 2. Learn multi-table relationships\n", - "# 3. Model the statistical distributions of each table\n", - "print(\"šŸ“Š Training synthesizer on multi-table data...\")\n", - "synthesizer.fit(\n", - " {\n", - " \"client\": clients_train,\n", - " \"disposition\": dispositions_train,\n", - " \"account\": accounts_train,\n", - " \"card\": cards_train,\n", - " \"transaction\": transactions_train,\n", - " \"loan\": loans_train,\n", - " \"order\": orders_train,\n", - " }\n", - ")\n", - "\n", - "end_time = time.time()\n", - "elapsed_minutes = (end_time - start_time) / 60\n", - "\n", - "print(\"āœ… SDV training completed successfully!\")\n", - "print(f\"ā±ļø Total training time: {elapsed_minutes:.2f} minutes\")\n", - "\n", - "# Report table sizes for clarity\n", - "print(\"šŸ“ˆ Training data breakdown:\")\n", - "print(f\" - Clients: {len(clients_train):,}\")\n", - "print(f\" - Dispositions: {len(dispositions_train):,}\")\n", - "print(f\" - Accounts: {len(accounts_train):,}\")\n", - "print(f\" - Cards: {len(cards_train):,}\")\n", - "print(f\" - Transactions: {len(transactions_train):,}\")\n", - "print(f\" - Loans: {len(loans_train):,}\")\n", - "print(f\" - Orders: {len(orders_train):,}\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 4.3 SDV Synthetic Data Generation\n", - "\n", - "**Generation Process:**\n", - "- **Scale Parameter**: Controls the number of synthetic records (1.0 = same size as training data)\n", - "- **Hierarchical Generation**: First generates parent records (customers), then child records (transfers)\n", - "- **Relationship Preservation**: Ensures all synthetic transfers reference valid synthetic customers\n", - "- **Statistical Sampling**: Uses learned distributions to create realistic synthetic data\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print(\"šŸŽ² Starting SDV synthetic data generation...\")\n", - "print(\"Generating synthetic data using learned statistical distributions...\")\n", - "\n", - "start_time = time.time()\n", - "\n", - "# Generate synthetic data with 25% more records than training data\n", - "# Scale parameter: 1.0 = same size, 1.25 = 25% larger, 0.5 = half size\n", - "print(\"āš™ļø Generating 1.25x the training data size...\")\n", - "sdv_synthetic_data = synthesizer.sample(scale=1.25)\n", - "\n", - "end_time = time.time()\n", - "elapsed_minutes = (end_time - start_time) / 60\n", - "\n", - "print(\"āœ… SDV generation completed successfully!\")\n", - "print(f\"ā±ļø Generation time: {elapsed_minutes:.2f} minutes\")\n", - "\n", - "# Calculate total synthetic records\n", - "synthetic_client_count = len(sdv_synthetic_data[\"client\"])\n", - "synthetic_disposition_count = len(sdv_synthetic_data[\"disposition\"])\n", - "synthetic_account_count = len(sdv_synthetic_data[\"account\"])\n", - "synthetic_card_count = len(sdv_synthetic_data[\"card\"])\n", - "synthetic_transaction_count = len(sdv_synthetic_data[\"transaction\"])\n", - "synthetic_loan_count = len(sdv_synthetic_data[\"loan\"])\n", - "synthetic_order_count = len(sdv_synthetic_data[\"order\"])\n", - "\n", - "total_synthetic_records = (\n", - " synthetic_client_count\n", - " + synthetic_disposition_count\n", - " + synthetic_account_count\n", - " + synthetic_card_count\n", - " + synthetic_transaction_count\n", - " + synthetic_loan_count\n", - " + synthetic_order_count\n", - ")\n", - "\n", - "generation_rate = total_synthetic_records / (end_time - start_time)\n", - "\n", - "print(f\"šŸš€ Generation rate: {generation_rate:,.0f} records/second\")\n", - "print(\"šŸ“Š Synthetic data breakdown:\")\n", - "print(f\" - Clients: {synthetic_client_count:,}\")\n", - "print(f\" - Dispositions: {synthetic_disposition_count:,}\")\n", - "print(f\" - Accounts: {synthetic_account_count:,}\")\n", - "print(f\" - Cards: {synthetic_card_count:,}\")\n", - "print(f\" - Transactions: {synthetic_transaction_count:,}\")\n", - "print(f\" - Loans: {synthetic_loan_count:,}\")\n", - "print(f\" - Orders: {synthetic_order_count:,}\")\n", - "\n", - "# Quality metric example: Dispositions per client ratio\n", - "dispositions_per_client = synthetic_disposition_count / synthetic_client_count if synthetic_client_count > 0 else 0\n", - "\n", - "print(\"\\nšŸ” Generation Quality Metrics:\")\n", - "print(f\" - Dispositions per client: {dispositions_per_client:.1f}\")\n", - "print(f\" - Scale factor achieved: {synthetic_client_count / len(clients_train):.2f}x\")\n", - "\n", - "# Preview of generated synthetic data\n", - "sdv_synthetic_data[\"client\"].head()\n", - "sdv_synthetic_data[\"disposition\"].head()" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Save SDV synthetic data for comparison\n", - "output_folder = \"./data/\"\n", - "\n", - "synthetic_files = {\n", - " \"client\": f\"{output_folder}sdv_client.parquet\",\n", - " \"disposition\": f\"{output_folder}sdv_disposition.parquet\",\n", - " \"account\": f\"{output_folder}sdv_account.parquet\",\n", - " \"card\": f\"{output_folder}sdv_card.parquet\",\n", - " \"transaction\": f\"{output_folder}sdv_transaction.parquet\",\n", - " \"loan\": f\"{output_folder}sdv_loan.parquet\",\n", - " \"order\": f\"{output_folder}sdv_order.parquet\",\n", - "}\n", - "\n", - "for table_name, file_path in synthetic_files.items():\n", - " sdv_synthetic_data[table_name].to_parquet(file_path, index=False)\n", - " print(f\"šŸ’¾ Saved {table_name} synthetic data to: {file_path}\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 5. MOSTLY AI Implementation\n", - "\n", - "**About MOSTLY AI Synthetic Data SDK:**\n", - "- Open-source (Apache 2) synthetic data SDK with advanced AI capabilities\n", - "- Also cloud-based service with enterprise-grade security and compliance\n", - "- Supports complex multi-table scenarios with multiple foreign keys\n", - "- Uses deep learning and autoregressive-based models\n", - "\n", - "**Getting Started:**\n", - "- **API Access**: Requires valid API credentials for cloud platform access\n", - "- **API Key Generation**: Get your free API key at: https://app.mostly.ai/settings/api-keys\n", - "\n", - "**Key Advantages:**\n", - "- **Advanced AI Models**: Utilizes state-of-the-art generative AI including language models\n", - "- **Multi-Parent Support**: Can handle complex relationships (multiple foreign keys per table)\n", - "- **Mixed Data Types**: Excels at both tabular and text data synthesis\n", - "- **Enterprise Features**: Privacy guarantees, compliance reporting, and scalability\n", - "\n", - "**Architecture:**\n", - "- **Tabular Models**: For structured data (demographics, financials)\n", - "- **Language Models**: For text fields (names, addresses, emails) using LLMs like Llama-3.2\n", - "- **Sequential Models**: For time-series and ordered data patterns" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from mostlyai.sdk import MostlyAI\n", - "\n", - "print(\"šŸ”§ Initializing MOSTLY AI Synthetic Data SDK...\")\n", - "\n", - "# Initialize MOSTLY AI Synthetic Data SDK\n", - "mostly = MostlyAI(local=True)\n", - "\n", - "\n", - "print(\"āœ… MOSTLY AI Synthetic Data SDK initialized successfully\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 5.1 MOSTLY AI Configuration Summary\n", - "\n", - "**Berka Multi-Table Setup Highlights:**\n", - "\n", - "- **Clients Table**\n", - " - Primary Key: `client_id`\n", - " - Standard demographic modeling\n", - "\n", - "- **Accounts Table**\n", - " - Primary Key: `account_id`\n", - " - Referenced by multiple tables as foreign key\n", - "\n", - "- **Dispositions Table**\n", - " - Dual Foreign Keys: `client_id` (non-context) + `account_id` (context)\n", - " - Captures client-account relationships\n", - "\n", - "- **Cards Table**\n", - " - Foreign Key: `disp_id` (context)\n", - " - Linked to disposition\n", - "\n", - "- **Transactions, Loans, Orders Tables**\n", - " - Foreign Key: `account_id` (context)\n", - " - Context-aware modeling for financial records\n", - "\n", - "**Global Configuration Notes:**\n", - "- Model: `MOSTLY_AI/Medium` for all tables\n", - "- Training Time: 10 minutes per table\n", - "- Flexible Generation: Disabled\n", - "- Model Reports: Disabled for streamlined runtime\n", - "- Multi-level Foreign Keys: Enabled with context tagging where applicable\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print(\"āš™ļø Configuring advanced MOSTLY AI generator...\")\n", - "print(\"Setting up sophisticated multi-table configuration with multi-level foreign keys...\")\n", - "\n", - "# Configure the generator for comprehensive multi-table setup\n", - "config = {\n", - " \"name\": \"Berka Multi-Table Generator\",\n", - " \"tables\": [\n", - " {\n", - " \"name\": \"client\",\n", - " \"data\": clients_train,\n", - " \"primary_key\": \"client_id\",\n", - " \"tabular_model_configuration\": {\n", - " \"model\": \"MOSTLY_AI/Medium\",\n", - " \"max_training_time\": 10,\n", - " \"enable_flexible_generation\": False,\n", - " \"value_protection\": False,\n", - " \"enable_model_report\": False,\n", - " },\n", - " },\n", - " {\n", - " \"name\": \"account\",\n", - " \"data\": accounts_train,\n", - " \"primary_key\": \"account_id\",\n", - " \"tabular_model_configuration\": {\n", - " \"model\": \"MOSTLY_AI/Medium\",\n", - " \"max_training_time\": 10,\n", - " \"enable_model_report\": False,\n", - " },\n", - " },\n", - " {\n", - " \"name\": \"disposition\",\n", - " \"data\": dispositions_train,\n", - " \"primary_key\": \"disp_id\",\n", - " \"foreign_keys\": [\n", - " {\"column\": \"client_id\", \"referenced_table\": \"client\", \"is_context\": False},\n", - " {\"column\": \"account_id\", \"referenced_table\": \"account\", \"is_context\": True},\n", - " ],\n", - " \"tabular_model_configuration\": {\n", - " \"model\": \"MOSTLY_AI/Medium\",\n", - " \"max_training_time\": 10,\n", - " \"enable_model_report\": False,\n", - " },\n", - " },\n", - " {\n", - " \"name\": \"card\",\n", - " \"data\": cards_train,\n", - " \"primary_key\": \"card_id\",\n", - " \"foreign_keys\": [{\"column\": \"disp_id\", \"referenced_table\": \"disposition\", \"is_context\": True}],\n", - " \"tabular_model_configuration\": {\n", - " \"model\": \"MOSTLY_AI/Medium\",\n", - " \"max_training_time\": 10,\n", - " \"enable_model_report\": False,\n", - " },\n", - " },\n", - " {\n", - " \"name\": \"transaction\",\n", - " \"data\": transactions_train,\n", - " \"primary_key\": \"trans_id\",\n", - " \"foreign_keys\": [{\"column\": \"account_id\", \"referenced_table\": \"account\", \"is_context\": True}],\n", - " \"tabular_model_configuration\": {\n", - " \"model\": \"MOSTLY_AI/Medium\",\n", - " \"max_training_time\": 10,\n", - " \"enable_model_report\": False,\n", - " },\n", - " },\n", - " {\n", - " \"name\": \"loan\",\n", - " \"data\": loans_train,\n", - " \"primary_key\": \"loan_id\",\n", - " \"foreign_keys\": [{\"column\": \"account_id\", \"referenced_table\": \"account\", \"is_context\": True}],\n", - " \"tabular_model_configuration\": {\n", - " \"model\": \"MOSTLY_AI/Medium\",\n", - " \"max_training_time\": 10,\n", - " \"enable_model_report\": False,\n", - " },\n", - " },\n", - " {\n", - " \"name\": \"order\",\n", - " \"data\": orders_train,\n", - " \"primary_key\": \"order_id\",\n", - " \"foreign_keys\": [{\"column\": \"account_id\", \"referenced_table\": \"account\", \"is_context\": True}],\n", - " \"tabular_model_configuration\": {\n", - " \"model\": \"MOSTLY_AI/Medium\",\n", - " \"max_training_time\": 10,\n", - " \"enable_model_report\": False,\n", - " },\n", - " },\n", - " ],\n", - "}\n", - "\n", - "print(\"āœ… MOSTLY AI generator configuration ready.\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 5.2 MOSTLY AI Training Process\n", - "\n", - "**Training Process:**\n", - "1. **Upload Data**: Send training data securely to MOSTLY AI cloud\n", - "2. **Model Configuration**: Apply the complex multi-table configuration\n", - "3. **AI Training**: Use advanced generative models including LLMs\n", - "4. **Quality Validation**: Automatic quality checks during training\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print(\"šŸš€ Starting MOSTLY AI training...\")\n", - "print(\"šŸ“¤ Uploading training data to secure MOSTLY AI cloud platform...\")\n", - "\n", - "start_time = time.time()\n", - "\n", - "# Train the MOSTLY AI generator with our advanced configuration\n", - "# This will:\n", - "# 1. Upload training data securely to the cloud\n", - "# 2. Configure both tabular and language models\n", - "# 3. Train AI models for each table and their relationships\n", - "# 4. Wait for training completion with progress monitoring\n", - "g = mostly.train(config=config, start=True, wait=True)\n", - "\n", - "end_time = time.time()\n", - "elapsed_minutes = (end_time - start_time) / 60\n", - "\n", - "print(\"āœ… MOSTLY AI training completed successfully!\")\n", - "print(f\"ā±ļø Total training time: {elapsed_minutes:.2f} minutes\")\n", - "print(\"šŸ§  Advanced AI models trained for multi-table synthesis\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print(\"šŸŽ² Starting MOSTLY AI synthetic data generation...\")\n", - "print(\"šŸŒ©ļø Using cloud-based AI models for high-quality multi-table synthesis...\")\n", - "\n", - "start_time = time.time()\n", - "\n", - "# Generate synthetic data using the trained MOSTLY AI generator\n", - "# Key advantages over SDV:\n", - "# - Handles multiple foreign keys properly\n", - "# - Maintains complex statistical relationships across multiple tables\n", - "\n", - "print(f\"šŸ“Š Generating {len(clients_train):,} synthetic client records...\")\n", - "\n", - "sd = mostly.generate(g, size=len(clients_train))\n", - "mostlyai_synthetic_data = sd.data()\n", - "\n", - "end_time = time.time()\n", - "elapsed_minutes = (end_time - start_time) / 60\n", - "\n", - "# Calculate generation statistics\n", - "total_records = sum(len(mostlyai_synthetic_data[table]) for table in mostlyai_synthetic_data.keys())\n", - "generation_rate = total_records / (end_time - start_time)\n", - "\n", - "print(\"āœ… MOSTLY AI generation completed successfully!\")\n", - "print(f\"ā±ļø Generation time: {elapsed_minutes:.2f} minutes\")\n", - "print(f\"šŸš€ Generation rate: {generation_rate:,.0f} records/second\")\n", - "\n", - "print(\"šŸ“Š Synthetic data breakdown:\")\n", - "for table_name in mostlyai_synthetic_data:\n", - " print(f\" - {table_name.capitalize()}: {len(mostlyai_synthetic_data[table_name]):,} rows\")\n", - "\n", - "# Quality metrics example: Dispositions per client ratio\n", - "dispositions_per_client = len(mostlyai_synthetic_data[\"disposition\"]) / len(mostlyai_synthetic_data[\"client\"])\n", - "print(\"\\nšŸ” Quality Metrics:\")\n", - "print(f\" - Dispositions per client: {dispositions_per_client:.1f}\")\n", - "print(\" - Multi-level foreign key integrity handled āœ…\")\n", - "\n", - "# Preview\n", - "mostlyai_synthetic_data[\"client\"].head()\n", - "mostlyai_synthetic_data[\"disposition\"].head()" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Save MOSTLY AI synthetic data for comparison\n", - "output_folder = \"./data/\"\n", - "\n", - "mostlyai_files = {\n", - " \"client\": f\"{output_folder}mostlyai_client.parquet\",\n", - " \"disposition\": f\"{output_folder}mostlyai_disposition.parquet\",\n", - " \"account\": f\"{output_folder}mostlyai_account.parquet\",\n", - " \"card\": f\"{output_folder}mostlyai_card.parquet\",\n", - " \"transaction\": f\"{output_folder}mostlyai_transaction.parquet\",\n", - " \"loan\": f\"{output_folder}mostlyai_loan.parquet\",\n", - " \"order\": f\"{output_folder}mostlyai_order.parquet\",\n", - "}\n", - "\n", - "for table_name, file_path in mostlyai_files.items():\n", - " mostlyai_synthetic_data[table_name].to_parquet(file_path, index=False)\n", - " print(f\"šŸ’¾ Saved {table_name} synthetic data to: {file_path}\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 6. Synthetic Data Quality Assessment\n", - "\n", - "After generating synthetic data using both SDV and MOSTLY AI, it's crucial to comprehensively evaluate the quality, privacy, and integrity of the generated datasets. This section provides a multi-faceted quality assessment framework that ensures our synthetic data meets production standards.\n", - "\n", - "## 6.1 Statistical Quality Assessment with MOSTLY AI QA Library\n", - "\n", - "The MOSTLY AI QA library provides enterprise-grade quality assessment capabilities that evaluate synthetic data across multiple dimensions. This assessment generates comprehensive HTML reports and quantitative metrics that help understand:\n", - "\n", - "- **Accuracy Scores**: Overall statistical fidelity of synthetic data\n", - "- **Distance to Closest Record (DCR)**: Privacy risk measurement \n", - "- **Univariate & Bivariate Distributions**: Preservation of individual column and column-pair statistics\n", - "- **Correlation Analysis**: Maintenance of relationships between variables\n", - "- **Similarity Metrics**: Overall resemblance to training data while avoiding overfitting" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Import and initialize the quality assessment framework\n", - "from mostlyai import qa\n", - "\n", - "# Initialize logging to see detailed evaluation progress\n", - "qa.init_logging()\n", - "print(\"šŸ” Quality assessment framework initialized\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Load the split files from the disk\n", - "transactions_train = pd.read_parquet(\"./data/transactions_train.parquet\")\n", - "transactions_test = pd.read_parquet(\"./data/transactions_test.parquet\")\n", - "accounts_train = pd.read_parquet(\"./data/accounts_train.parquet\")\n", - "accounts_test = pd.read_parquet(\"./data/accounts_test.parquet\")\n", - "\n", - "print(\"šŸ“‚ Training and test datasets loaded successfully\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print(\"šŸ“Š Evaluating SDV Transaction synthetic data quality...\")\n", - "\n", - "# Load the SDV synthetic dataset\n", - "sdv_transaction = pd.read_parquet(\"./data/sdv_transaction.parquet\")\n", - "\n", - "# Define ID columns to exclude from QA analysis (do not exclude account_id here!)\n", - "id_columns_to_exclude = [\"trans_id\"]\n", - "\n", - "\n", - "def remove_id_columns(df, columns_to_remove):\n", - " return df.drop(columns=[col for col in columns_to_remove if col in df.columns])\n", - "\n", - "\n", - "# Prepare transaction data (remove only trans_id)\n", - "sdv_transaction_qa = remove_id_columns(sdv_transaction, id_columns_to_exclude)\n", - "transactions_train_qa = remove_id_columns(transactions_train, id_columns_to_exclude)\n", - "transactions_test_qa = remove_id_columns(transactions_test, id_columns_to_exclude)\n", - "\n", - "report_path, metrics = qa.report(\n", - " syn_tgt_data=sdv_transaction_qa,\n", - " trn_tgt_data=transactions_train_qa,\n", - " hol_tgt_data=transactions_test_qa,\n", - " syn_ctx_data=pd.read_parquet(\"./data/sdv_account.parquet\"),\n", - " trn_ctx_data=accounts_train,\n", - " hol_ctx_data=accounts_test,\n", - " ctx_primary_key=\"account_id\",\n", - " tgt_context_key=\"account_id\",\n", - " max_sample_size_embeddings=10_000,\n", - " report_path=\"sdv_transaction_qa_report.html\",\n", - ")\n", - "\n", - "print(f\"šŸ“‹ SDV Transaction Quality Report saved to: {report_path}\")\n", - "print(\"\\nšŸ“ˆ SDV Transaction Quality Metrics:\")\n", - "print(metrics.model_dump_json(indent=4))\n", - "\n", - "sdv_transaction_accuracy = metrics.accuracy.overall\n", - "sdv_transaction_dcr_share = metrics.distances.dcr_share" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print(\"šŸ“Š Evaluating MOSTLY AI Transactions synthetic data quality...\")\n", - "\n", - "# Load the MOSTLY AI synthetic dataset\n", - "mostlyai_transaction = pd.read_parquet(\"./data/mostlyai_transaction.parquet\")\n", - "\n", - "# Define ID columns to exclude from QA analysis (do not exclude account_id here!)\n", - "id_columns_to_exclude = [\"trans_id\"]\n", - "\n", - "\n", - "def remove_id_columns(df, columns_to_remove):\n", - " return df.drop(columns=[col for col in columns_to_remove if col in df.columns])\n", - "\n", - "\n", - "# Prepare transaction data (remove only trans_id)\n", - "mostlyai_transaction_qa = remove_id_columns(mostlyai_transaction, id_columns_to_exclude)\n", - "transactions_train_qa = remove_id_columns(transactions_train, id_columns_to_exclude)\n", - "transactions_test_qa = remove_id_columns(transactions_test, id_columns_to_exclude)\n", - "\n", - "report_path, metrics = qa.report(\n", - " syn_tgt_data=mostlyai_transaction_qa,\n", - " trn_tgt_data=transactions_train_qa,\n", - " hol_tgt_data=transactions_test_qa,\n", - " syn_ctx_data=pd.read_parquet(\"./data/mostlyai_account.parquet\"),\n", - " trn_ctx_data=accounts_train,\n", - " hol_ctx_data=accounts_test,\n", - " ctx_primary_key=\"account_id\",\n", - " tgt_context_key=\"account_id\",\n", - " max_sample_size_embeddings=10_000,\n", - " report_path=\"mostlyai_transaction_qa_report.html\",\n", - ")\n", - "\n", - "print(f\"šŸ“‹ MOSTLY AI Transaction Quality Report saved to: {report_path}\")\n", - "print(\"\\nšŸ“ˆ MOSTLY AI Transaction Quality Metrics:\")\n", - "print(metrics.model_dump_json(indent=4))\n", - "\n", - "mostlyai_transaction_accuracy = metrics.accuracy.overall\n", - "mostlyai_transaction_dcr_share = metrics.distances.dcr_share" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Add a final comparison section\n", - "print(\"\\n\" + \"=\" * 60)\n", - "print(\"šŸ† FINAL COMPARISON\")\n", - "print(\"=\" * 60)\n", - "print(f\"SDV Transaction - Accuracy: {sdv_transaction_accuracy:.3f}, DCR Share: {sdv_transaction_dcr_share:.3f}\")\n", - "print(\n", - " f\"MOSTLY AI Transaction- Accuracy: {mostlyai_transaction_accuracy:.3f}, DCR Share: {mostlyai_transaction_dcr_share:.3f}\"\n", - ")\n", - "\n", - "print(\"\\nšŸ” METRIC INTERPRETATION:\")\n", - "print(\"ā€¢ Higher accuracy = better statistical fidelity\")\n", - "print(\"ā€¢ DCR Share ~0.5 = optimal privacy-utility balance\")\n", - "\n", - "print(\"\\nšŸ“Š ANALYSIS:\")\n", - "print(\"ā€¢ MOSTLY AI consistently shows higher accuracy than SDV\")\n", - "print(\"ā€¢ Both frameworks maintain reasonable DCR Share values around 0.5\")\n", - "print(\"ā€¢ MOSTLY AI handles multi-table relationships and foreign keys with greater precision\")" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "venv", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.10.18" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/docs/tutorials/sdv-comparison/two-table-scenario/two-table-scenario.ipynb b/docs/tutorials/sdv-comparison/two-table-scenario/two-table-scenario.ipynb new file mode 100644 index 00000000..b5d9f0b1 --- /dev/null +++ b/docs/tutorials/sdv-comparison/two-table-scenario/two-table-scenario.ipynb @@ -0,0 +1,740 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# MOSTLY AI vs. SDV Comparison - Two-Table Scenario \"Run\n", + "\n", + "This notebook provides a comprehensive comparison between two leading synthetic data generation platforms:\n", + "- **SDV (Synthetic Data Vault)** - Business Source License\n", + "- **MOSTLY AI Synthetic Data SDK** - Apache 2.0 License - Open Source\n", + "\n", + "In this comparison, we are going to walk through the synthesis of a relational two-table structure using the [Berka dataset](https://github.com/mostly-ai/public-demo-data/tree/dev/berka/data).\n", + "\n", + "## Comparison Methodology\n", + "\n", + "1. **Data Preparation**: Load, inspect, and preprocess the multi-table dataset\n", + "2. **Data Splitting**: Create train/test splits\n", + "3. **Model Training**: Train both SDV and MOSTLY AI generators on the training data\n", + "4. **Synthetic Data Generation**: Generate synthetic datasets using both platforms\n", + "5. **Performance Analysis**: Compare training time, generation speed, and data quality\n", + "\n", + "## Key Challenges in Two-Table Synthesis\n", + "\n", + "- **Sequential Dependencies**: Preserving temporal patterns in transaction data\n", + "- **Data Quality**: Ensuring synthetic data maintains statistical properties and business logic" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Install SDK in CLIENT mode\n", + "!uv pip install -U mostlyai sdv graphviz\n", + "# Or install in LOCAL mode\n", + "!uv pip install -U 'mostlyai[local]' sdv graphviz\n", + "# Note: Restart kernel session after installation!" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 1. Data Loading and Initial Exploration\n", + "\n", + "First, let's load our multi-table dataset and examine its structure to understand:\n", + "- Table schemas and data types\n", + "- Data quality and completeness\n", + "- Business logic and constraints\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import pandas as pd\n", + "\n", + "base_url = \"https://github.com/mostly-ai/public-demo-data/raw/dev/berka/data/\"\n", + "originals = {\n", + " \"account\": pd.read_csv(base_url + \"account.csv.gz\", low_memory=False),\n", + " \"transaction\": pd.read_csv(base_url + \"trans.csv.gz\", low_memory=False),\n", + "}\n", + "\n", + "# Drop unnecessary columns from transaction table\n", + "originals[\"transaction\"].drop(columns=[\"bank\", \"account\"], inplace=True)\n", + "\n", + "# Convert date columns to datetime\n", + "originals[\"account\"][\"date\"] = pd.to_datetime(originals[\"account\"][\"date\"])\n", + "originals[\"transaction\"][\"date\"] = pd.to_datetime(originals[\"transaction\"][\"date\"])\n", + "\n", + "# Display samples\n", + "for k in originals:\n", + " print(\"===\", k, \"===\")\n", + " display(originals[k].sample(n=3))\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 2. Strategic Data Splitting for Multi-Table Scenarios\n", + "\n", + "When dealing with related tables like accounts and transactions, data splitting becomes more complex than simple random sampling due to the links between data tables. \n", + "\n", + "In order to make coherent assessments of data quality, we need to create meaningful train and test cohorts.\n", + "\n", + "**Key Considerations:**\n", + "- **Business Logic:** Accounts and transactions can only exist in the training set if their associated client is also in the training set.\n", + "- **Data Leakage Prevention:** Avoid information bleeding between train/test sets.\n", + "\n", + "**Our Approach:**\n", + "1. **Split accounts first (80/20 train/test)**: We split the `account` table using an 80/20 ratio. This ensures customer-related information is kept together per split.\n", + " \n", + "2. **Assign related tables based on account membership:** \n", + " - **Transactions:** All linked via `account_id`. These tables follow the same assignment logic.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from sklearn.model_selection import train_test_split\n", + "\n", + "print(\"āœ‚ļø Performing strategic multi-table data splitting based on accounts...\")\n", + "\n", + "# Step 1: Split accounts using 80/20 ratio\n", + "accounts_train, accounts_test = train_test_split(\n", + " originals[\"account\"], test_size=0.2, random_state=42\n", + ")\n", + "\n", + "print(f\"šŸ¦ Account split:\")\n", + "print(f\" - Training set: {len(accounts_train):,} accounts ({len(accounts_train) / len(originals['account']) * 100:.1f}%)\")\n", + "print(f\" - Test set: {len(accounts_test):,} accounts ({len(accounts_test) / len(originals['account']) * 100:.1f}%)\")\n", + "\n", + "# Step 2: Create account ID sets for lookup\n", + "train_account_ids = set(accounts_train[\"account_id\"])\n", + "test_account_ids = set(accounts_test[\"account_id\"])\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print(\"šŸ”„ Assigning transactions based on account split...\")\n", + "\n", + "# Split transactions linked to training and test accounts\n", + "transactions_train = originals[\"transaction\"][originals[\"transaction\"][\"account_id\"].isin(train_account_ids)].copy()\n", + "transactions_test = originals[\"transaction\"][originals[\"transaction\"][\"account_id\"].isin(test_account_ids)].copy()\n", + "\n", + "print(\"āœ… Splitting complete!\")\n", + "print(f\" - Training accounts: {len(accounts_train):,}\")\n", + "print(f\" - Training transactions: {len(transactions_train):,}\")\n", + "print(f\" - Test accounts: {len(accounts_test):,}\")\n", + "print(f\" - Test transactions: {len(transactions_test):,}\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print(\"šŸ’¾ Saving split train/test tables...\")\n", + "\n", + "accounts_train.to_parquet('./data/two-table/accounts_train.parquet', index=False)\n", + "accounts_test.to_parquet('./data/two-table/accounts_test.parquet', index=False)\n", + "transactions_train.to_parquet('./data/two-table/transactions_train.parquet', index=False)\n", + "transactions_test.to_parquet('./data/two-table/transactions_test.parquet', index=False)\n", + "\n", + "print(\"āœ… All train/test splits saved to ./data/two-table/\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 3. SDV (Synthetic Data Vault) Implementation\n", + "\n", + "**About SDV:**\n", + "- Business Source License Python library for synthetic data generation\n", + "- Supports single-table and multi-table scenarios\n", + "- Uses statistical modeling and machine learning approaches\n", + "- Provides HMASynthesizer for hierarchical multi-table synthesis\n", + "\n", + "**Key Features:**\n", + "- **Metadata Detection**: Automatically infers data types and relationships\n", + "- **Relationship Modeling**: Handles parent-child table relationships\n", + "- **Privacy Protection**: Generates synthetic data that preserves statistical properties while protecting individual privacy\n", + "- **Extensible**: Multiple synthesizer options (GaussianCopula, CTGAN, CopulaGAN, etc.)\n", + "\n", + "**Limitations:**\n", + "- Current version only supports one parent per child table\n", + "- Complex multi-parent relationships require modeling simplification\n", + "- Performance scales with data complexity\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from sdv.multi_table import HMASynthesizer\n", + "from sdv.metadata import Metadata\n", + "\n", + "print(\"šŸ—ļø Building SDV metadata configuration...\")\n", + "\n", + "# Auto-detect metadata using only the relevant tables\n", + "metadata = Metadata.detect_from_dataframes(\n", + " data={\n", + " 'account': accounts_train,\n", + " 'transaction': transactions_train\n", + " },\n", + " infer_keys='primary_and_foreign'\n", + ")\n", + "\n", + "print(\"āœ… Base metadata auto-detected with relationships\")\n", + "\n", + "# View auto-detected relationships graphically\n", + "metadata.visualize()\n", + "\n", + "# Inspect relationships and table configuration as raw dictionary\n", + "metadata_dict = metadata.to_dict()\n", + "print(\"\\nšŸ“‹ Complete SDV Metadata Dictionary:\")\n", + "print(metadata_dict)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 3.2 SDV Model Training\n", + "\n", + "**HMASynthesizer Overview:**\n", + "- **Hierarchical Modeling**: Learns parent-child relationships\n", + "- **Statistical Approach**: Uses copulas and Gaussian distributions\n", + "- **Multi-step Process**: \n", + " 1. Preprocesses tables and infers constraints\n", + " 2. Learns relationships between parent and child tables\n", + " 3. Models individual table distributions\n", + " \n", + "**Training Phases:**\n", + "- **Preprocess Tables**: Data cleaning and type inference\n", + "- **Learning Relationships**: Analyzing foreign key dependencies \n", + "- **Modeling Tables**: Learning statistical distributions for synthesis\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import time\n", + "from sdv.multi_table import HMASynthesizer\n", + "\n", + "print(\"šŸš€ Starting SDV training process...\")\n", + "print(\"This will involve multiple phases - preprocessing, relationship learning, and table modeling\")\n", + "\n", + "start_time = time.time()\n", + "\n", + "# Initialize the HMASynthesizer with our configured metadata\n", + "print(\"šŸ”§ Initializing HMASynthesizer...\")\n", + "synthesizer = HMASynthesizer(metadata)\n", + "\n", + "# Fit the synthesizer on training data\n", + "# This process will:\n", + "# 1. Preprocess all tables (clean data, infer constraints)\n", + "# 2. Learn multi-table relationships\n", + "# 3. Model the statistical distributions of each table\n", + "print(\"šŸ“Š Training synthesizer on multi-table data...\")\n", + "synthesizer.fit({\n", + " 'account': accounts_train,\n", + " 'transaction': transactions_train\n", + "})\n", + "\n", + "end_time = time.time()\n", + "elapsed_minutes = (end_time - start_time) / 60\n", + "\n", + "print(f\"āœ… SDV training completed successfully!\")\n", + "print(f\"ā±ļø Total training time: {elapsed_minutes:.2f} minutes\")\n", + "\n", + "# Report table sizes for clarity\n", + "print(\"šŸ“ˆ Training data breakdown:\")\n", + "print(f\" - Accounts: {len(accounts_train):,}\")\n", + "print(f\" - Transactions: {len(transactions_train):,}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 3.3 SDV Synthetic Data Generation\n", + "\n", + "**Generation Process:**\n", + "- **Scale Parameter**: Controls the number of synthetic records (1.0 = same size as training data)\n", + "- **Hierarchical Generation**: First generates parent records (accounts), then child records (transactions)\n", + "- **Relationship Preservation**: Ensures all synthetic transfers reference valid synthetic customers\n", + "- **Statistical Sampling**: Uses learned distributions to create realistic synthetic data\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print(\"šŸŽ² Starting SDV synthetic data generation...\")\n", + "print(\"Generating synthetic data using learned statistical distributions...\")\n", + "\n", + "start_time = time.time()\n", + "\n", + "# Generate synthetic data with the same number of records as training data\n", + "print(\"āš™ļø Generating 1.00x the training data size...\")\n", + "sdv_synthetic_data = synthesizer.sample(scale=1.00)\n", + "\n", + "end_time = time.time()\n", + "elapsed_minutes = (end_time - start_time) / 60\n", + "\n", + "print(f\"āœ… SDV generation completed successfully!\")\n", + "print(f\"ā±ļø Generation time: {elapsed_minutes:.2f} minutes\")\n", + "\n", + "# Calculate synthetic record counts\n", + "synthetic_account_count = len(sdv_synthetic_data['account'])\n", + "synthetic_transaction_count = len(sdv_synthetic_data['transaction'])\n", + "\n", + "total_synthetic_records = synthetic_account_count + synthetic_transaction_count\n", + "generation_rate = total_synthetic_records / (end_time - start_time)\n", + "\n", + "print(f\"šŸš€ Generation rate: {generation_rate:,.0f} records/second\")\n", + "print(f\"šŸ“Š Synthetic data breakdown:\")\n", + "print(f\" - Accounts: {synthetic_account_count:,}\")\n", + "print(f\" - Transactions: {synthetic_transaction_count:,}\")\n", + "\n", + "# Preview of generated synthetic data\n", + "sdv_synthetic_data['account'].head()\n", + "sdv_synthetic_data['transaction'].head()\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Save SDV synthetic data for comparison\n", + "output_folder = './data/two-table/'\n", + "\n", + "synthetic_files = {\n", + " 'account': f'{output_folder}sdv_account.parquet',\n", + " 'transaction': f'{output_folder}sdv_transaction.parquet',\n", + "}\n", + "\n", + "for table_name, file_path in synthetic_files.items():\n", + " sdv_synthetic_data[table_name].to_parquet(file_path, index=False)\n", + " print(f\"šŸ’¾ Saved {table_name} synthetic data to: {file_path}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 4. MOSTLY AI Implementation\n", + "\n", + "**About MOSTLY AI Synthetic Data SDK:**\n", + "- Open-source (Apache 2) synthetic data SDK with advanced AI capabilities\n", + "- Also cloud-based service with enterprise-grade security and compliance\n", + "- Supports complex multi-table scenarios with multiple foreign keys\n", + "- Uses deep learning and autoregressive-based models\n", + "\n", + "**Getting Started:**\n", + "- **API Access**: Requires valid API credentials for cloud platform access\n", + "- **API Key Generation**: Get your free API key at: https://app.mostly.ai/settings/api-keys\n", + "\n", + "**Key Advantages:**\n", + "- **Advanced AI Models**: Utilizes state-of-the-art generative AI including language models\n", + "- **Mixed Data Types**: Excels at both tabular and text data synthesis\n", + "- **Enterprise Features**: Privacy guarantees, compliance reporting, and scalability\n", + "\n", + "**Architecture:**\n", + "- **Tabular Models**: For structured data (demographics, financials)\n", + "- **Language Models**: For text fields (names, addresses, emails) using LLMs like Llama-3.2\n", + "- **Sequential Models**: For time-series and ordered data patterns" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from mostlyai.sdk import MostlyAI\n", + "\n", + "print(\"šŸ”§ Initializing MOSTLY AI Synthetic Data SDK...\")\n", + "\n", + "# Initialize MOSTLY AI Synthetic Data SDK\n", + "mostly = MostlyAI(local=True)\n", + "\n", + "\n", + "print(\"āœ… MOSTLY AI Synthetic Data SDK initialized successfully\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 4.1 MOSTLY AI Configuration Summary\n", + "\n", + "**Berka Two-Table Setup Highlights:**\n", + "\n", + "- **Accounts Table**\n", + " - Primary Key: `account_id`\n", + " - Referenced by transactions table as foreign key\n", + "\n", + "- **Transactions Table**\n", + " - Foreign Key: `account_id` (context)\n", + " - Context-aware modeling for financial records\n", + " " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print(\"āš™ļø Configuring streamlined MOSTLY AI generator...\")\n", + "print(\"Setting up minimal two-table configuration for account and transaction data...\")\n", + "\n", + "# Configure the generator for the reduced two-table setup\n", + "config = {\n", + " 'name': 'Berka Accounts & Transactions Generator',\n", + " 'tables': [\n", + " {\n", + " 'name': 'account',\n", + " 'data': accounts_train,\n", + " 'primary_key': 'account_id',\n", + " 'tabular_model_configuration': {\n", + " 'enable_model_report': False\n", + " }\n", + " },\n", + " {\n", + " 'name': 'transaction',\n", + " 'data': transactions_train,\n", + " 'primary_key': 'trans_id',\n", + " 'foreign_keys': [\n", + " {'column': 'account_id', 'referenced_table': 'account', 'is_context': True}\n", + " ],\n", + " 'tabular_model_configuration': {\n", + " 'enable_model_report': False\n", + " }\n", + " }\n", + " ]\n", + "}\n", + "\n", + "print(\"āœ… MOSTLY AI generator configuration ready.\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 4.2 MOSTLY AI Training Process\n", + "\n", + "**Training Process:**\n", + "1. **Upload Data**: Send training data securely to MOSTLY AI cloud\n", + "2. **Model Configuration**: Apply the complex multi-table configuration\n", + "3. **AI Training**: Use advanced generative models including LLMs\n", + "4. **Quality Validation**: Automatic quality checks during training\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print(\"šŸš€ Starting MOSTLY AI training...\")\n", + "print(\"šŸ“¤ Uploading training data to secure MOSTLY AI cloud platform...\")\n", + "\n", + "start_time = time.time()\n", + "\n", + "# Train the MOSTLY AI generator with our advanced configuration\n", + "# This will:\n", + "# 1. Upload training data securely to the cloud\n", + "# 2. Configure both tabular and language models\n", + "# 3. Train AI models for each table and their relationships\n", + "# 4. Wait for training completion with progress monitoring\n", + "g = mostly.train(config=config, start=True, wait=True)\n", + "\n", + "end_time = time.time()\n", + "elapsed_minutes = (end_time - start_time) / 60\n", + "\n", + "print(f\"āœ… MOSTLY AI training completed successfully!\")\n", + "print(f\"ā±ļø Total training time: {elapsed_minutes:.2f} minutes\")\n", + "print(f\"šŸ§  Advanced AI models trained for multi-table synthesis\")\n", + " " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print(\"šŸŽ² Starting MOSTLY AI synthetic data generation...\")\n", + "print(\"šŸŒ©ļø Using cloud-based AI models for high-quality multi-table synthesis...\")\n", + "\n", + "start_time = time.time()\n", + "\n", + "# Generate synthetic data using the trained MOSTLY AI generator\n", + "# Key advantages over SDV:\n", + "# - Handles foreign keys properly\n", + "# - Maintains statistical relationships across tables\n", + "\n", + "print(f\"šŸ“Š Generating {len(accounts_train):,} synthetic account records...\")\n", + "\n", + "sd = mostly.generate(g, size=len(accounts_train))\n", + "mostlyai_synthetic_data = sd.data()\n", + "\n", + "end_time = time.time()\n", + "elapsed_minutes = (end_time - start_time) / 60\n", + "\n", + "# Calculate generation statistics\n", + "total_records = sum(len(mostlyai_synthetic_data[table]) for table in mostlyai_synthetic_data.keys())\n", + "generation_rate = total_records / (end_time - start_time)\n", + "\n", + "print(f\"āœ… MOSTLY AI generation completed successfully!\")\n", + "print(f\"ā±ļø Generation time: {elapsed_minutes:.2f} minutes\")\n", + "print(f\"šŸš€ Generation rate: {generation_rate:,.0f} records/second\")\n", + "\n", + "print(\"šŸ“Š Synthetic data breakdown:\")\n", + "for table_name in mostlyai_synthetic_data:\n", + " print(f\" - {table_name.capitalize()}: {len(mostlyai_synthetic_data[table_name]):,} rows\")\n", + "\n", + "# Preview\n", + "mostlyai_synthetic_data['account'].head()\n", + "mostlyai_synthetic_data['transaction'].head()\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Save MOSTLY AI synthetic data for comparison\n", + "output_folder = './data/two-table/'\n", + "\n", + "mostlyai_files = {\n", + " 'account': f'{output_folder}mostlyai_account.parquet',\n", + " 'transaction': f'{output_folder}mostlyai_transaction.parquet',\n", + "}\n", + "\n", + "for table_name, file_path in mostlyai_files.items():\n", + " mostlyai_synthetic_data[table_name].to_parquet(file_path, index=False)\n", + " print(f\"šŸ’¾ Saved {table_name} synthetic data to: {file_path}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 5. Synthetic Data Quality Assessment\n", + "\n", + "After generating synthetic data using both SDV and MOSTLY AI, it's crucial to comprehensively evaluate the quality, privacy, and integrity of the generated datasets. This section provides a multi-faceted quality assessment framework that ensures our synthetic data meets production standards.\n", + "\n", + "## 5.1 Statistical Quality Assessment with MOSTLY AI QA Library\n", + "\n", + "The MOSTLY AI QA library provides enterprise-grade quality assessment capabilities that evaluate synthetic data across multiple dimensions. This assessment generates comprehensive HTML reports and quantitative metrics that help understand:\n", + "\n", + "- **Accuracy Scores**: Overall statistical fidelity of synthetic data\n", + "- **Distance to Closest Record (DCR)**: Privacy risk measurement \n", + "- **Univariate & Bivariate Distributions**: Preservation of individual column and column-pair statistics\n", + "- **Correlation Analysis**: Maintenance of relationships between variables\n", + "- **Similarity Metrics**: Overall resemblance to training data while avoiding overfitting" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Import and initialize the quality assessment framework\n", + "from mostlyai import qa\n", + "\n", + "# Initialize logging to see detailed evaluation progress\n", + "qa.init_logging()\n", + "print(\"šŸ” Quality assessment framework initialized\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Load the split files from the disk\n", + "transactions_train = pd.read_parquet('./data/two-table/transactions_train.parquet')\n", + "transactions_test = pd.read_parquet('./data/two-table/transactions_test.parquet')\n", + "accounts_train = pd.read_parquet('./data/two-table/accounts_train.parquet')\n", + "accounts_test = pd.read_parquet('./data/two-table/accounts_test.parquet')\n", + "\n", + "print(\"šŸ“‚ Training and test datasets loaded successfully\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print(\"šŸ“Š Evaluating SDV Transaction synthetic data quality...\")\n", + "\n", + "# Load the SDV synthetic dataset\n", + "sdv_transaction = pd.read_parquet('./data/two-table/sdv_transaction.parquet')\n", + "\n", + "# Define ID columns to exclude from QA analysis (but keep account_id for context linkage)\n", + "id_columns_to_exclude = ['trans_id']\n", + "\n", + "def remove_id_columns(df, columns_to_remove):\n", + " return df.drop(columns=[col for col in columns_to_remove if col in df.columns])\n", + "\n", + "# Prepare data for QA\n", + "sdv_transaction_qa = remove_id_columns(sdv_transaction, id_columns_to_exclude)\n", + "transactions_train_qa = remove_id_columns(transactions_train, id_columns_to_exclude)\n", + "transactions_test_qa = remove_id_columns(transactions_test, id_columns_to_exclude)\n", + "\n", + "report_path, metrics = qa.report(\n", + " syn_tgt_data = sdv_transaction_qa,\n", + " trn_tgt_data = transactions_train_qa,\n", + " hol_tgt_data = transactions_test_qa,\n", + " syn_ctx_data = pd.read_parquet('./data/two-table/sdv_account.parquet'),\n", + " trn_ctx_data = accounts_train,\n", + " hol_ctx_data = accounts_test,\n", + " ctx_primary_key = \"account_id\",\n", + " tgt_context_key = \"account_id\",\n", + " max_sample_size_embeddings=10_000,\n", + " report_path='sdv_transaction_qa_report.html'\n", + ")\n", + "\n", + "print(f\"šŸ“‹ SDV Transaction Quality Report saved to: {report_path}\")\n", + "print(\"\\nšŸ“ˆ SDV Transaction Quality Metrics:\")\n", + "print(metrics.model_dump_json(indent=4))\n", + "\n", + "# Print summary scores\n", + "sdv_transaction_accuracy = metrics.accuracy.overall\n", + "sdv_transaction_dcr_share = metrics.distances.dcr_share\n", + "print(f\"\\nšŸŽÆ SDV Transaction Summary:\")\n", + "print(f\" Overall Accuracy: {sdv_transaction_accuracy:.3f}\")\n", + "print(f\" DCR Share: {sdv_transaction_dcr_share:.3f}\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print(\"šŸ“Š Evaluating MOSTLY AI Transactions synthetic data quality...\")\n", + "\n", + "# Load the MOSTLY AI synthetic dataset\n", + "mostlyai_transaction = pd.read_parquet('./data/two-table/mostlyai_transaction.parquet')\n", + "\n", + "# Define ID columns to exclude from QA analysis (keep account_id)\n", + "id_columns_to_exclude = ['trans_id']\n", + "\n", + "def remove_id_columns(df, columns_to_remove):\n", + " return df.drop(columns=[col for col in columns_to_remove if col in df.columns])\n", + "\n", + "# Prepare transaction data\n", + "mostlyai_transaction_qa = remove_id_columns(mostlyai_transaction, id_columns_to_exclude)\n", + "transactions_train_qa = remove_id_columns(transactions_train, id_columns_to_exclude)\n", + "transactions_test_qa = remove_id_columns(transactions_test, id_columns_to_exclude)\n", + "\n", + "report_path, metrics = qa.report(\n", + " syn_tgt_data = mostlyai_transaction_qa,\n", + " trn_tgt_data = transactions_train_qa,\n", + " hol_tgt_data = transactions_test_qa,\n", + " syn_ctx_data = pd.read_parquet('./data/two-table/mostlyai_account.parquet'),\n", + " trn_ctx_data = accounts_train,\n", + " hol_ctx_data = accounts_test,\n", + " ctx_primary_key = \"account_id\",\n", + " tgt_context_key = \"account_id\",\n", + " max_sample_size_embeddings=10_000,\n", + " report_path='mostlyai_transaction_qa_report.html'\n", + ")\n", + "\n", + "print(f\"šŸ“‹ MOSTLY AI Transaction Quality Report saved to: {report_path}\")\n", + "print(\"\\nšŸ“ˆ MOSTLY AI Transaction Quality Metrics:\")\n", + "print(metrics.model_dump_json(indent=4))\n", + "\n", + "# Print summary scores\n", + "mostlyai_transaction_accuracy = metrics.accuracy.overall\n", + "mostlyai_transaction_dcr_share = metrics.distances.dcr_share\n", + "print(f\"\\nšŸŽÆ MOSTLY AI Transaction Summary:\")\n", + "print(f\" Overall Accuracy: {mostlyai_transaction_accuracy:.3f}\")\n", + "print(f\" DCR Share: {mostlyai_transaction_dcr_share:.3f}\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Add a final comparison section\n", + "print(\"\\n\" + \"=\"*60)\n", + "print(\"šŸ† FINAL COMPARISON\")\n", + "print(\"=\"*60)\n", + "print(f\"SDV Transaction - Accuracy: {sdv_transaction_accuracy:.3f}, DCR Share: {sdv_transaction_dcr_share:.3f}\")\n", + "print(f\"MOSTLY AI Transaction- Accuracy: {mostlyai_transaction_accuracy:.3f}, DCR Share: {mostlyai_transaction_dcr_share:.3f}\")\n", + "\n", + "print(\"\\nšŸ” METRIC INTERPRETATION:\")\n", + "print(\"ā€¢ Higher accuracy = better statistical fidelity\")\n", + "print(\"ā€¢ DCR Share ~0.5 = optimal privacy-utility balance\")\n", + "\n", + "print(\"\\nšŸ“Š ANALYSIS:\")\n", + "print(\"ā€¢ MOSTLY AI consistently shows higher accuracy than SDV\")\n", + "print(\"ā€¢ Both frameworks maintain reasonable DCR Share values around 0.5\")\n", + "print(\"ā€¢ MOSTLY AI handles multi-table relationships and foreign keys with greater precision\")\n", + "\n", + "print(\"\\nāš ļø RECOMMENDATION:\")\n", + "print(\"ā€¢ Review detailed HTML reports for nuanced privacy insights\")\n", + "print(\"ā€¢ Pay attention to discriminator AUC and feature-wise similarity scores\")\n", + "print(\"ā€¢ Align final choice with your privacy-utility balance requirements\")\n" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "venv", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.18" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} From bb5a8c5944ebab5ea2f928928a9cac684546b311 Mon Sep 17 00:00:00 2001 From: Kenneth Hamilton Date: Tue, 29 Jul 2025 10:48:54 +0200 Subject: [PATCH 4/8] lint --- .../two-table-scenario.ipynb | 195 +++++++++--------- 1 file changed, 93 insertions(+), 102 deletions(-) diff --git a/docs/tutorials/sdv-comparison/two-table-scenario/two-table-scenario.ipynb b/docs/tutorials/sdv-comparison/two-table-scenario/two-table-scenario.ipynb index b5d9f0b1..f76291c1 100644 --- a/docs/tutorials/sdv-comparison/two-table-scenario/two-table-scenario.ipynb +++ b/docs/tutorials/sdv-comparison/two-table-scenario/two-table-scenario.ipynb @@ -75,7 +75,7 @@ "# Display samples\n", "for k in originals:\n", " print(\"===\", k, \"===\")\n", - " display(originals[k].sample(n=3))\n" + " display(originals[k].sample(n=3))" ] }, { @@ -110,17 +110,17 @@ "print(\"āœ‚ļø Performing strategic multi-table data splitting based on accounts...\")\n", "\n", "# Step 1: Split accounts using 80/20 ratio\n", - "accounts_train, accounts_test = train_test_split(\n", - " originals[\"account\"], test_size=0.2, random_state=42\n", - ")\n", + "accounts_train, accounts_test = train_test_split(originals[\"account\"], test_size=0.2, random_state=42)\n", "\n", - "print(f\"šŸ¦ Account split:\")\n", - "print(f\" - Training set: {len(accounts_train):,} accounts ({len(accounts_train) / len(originals['account']) * 100:.1f}%)\")\n", + "print(\"šŸ¦ Account split:\")\n", + "print(\n", + " f\" - Training set: {len(accounts_train):,} accounts ({len(accounts_train) / len(originals['account']) * 100:.1f}%)\"\n", + ")\n", "print(f\" - Test set: {len(accounts_test):,} accounts ({len(accounts_test) / len(originals['account']) * 100:.1f}%)\")\n", "\n", "# Step 2: Create account ID sets for lookup\n", "train_account_ids = set(accounts_train[\"account_id\"])\n", - "test_account_ids = set(accounts_test[\"account_id\"])\n" + "test_account_ids = set(accounts_test[\"account_id\"])" ] }, { @@ -139,7 +139,7 @@ "print(f\" - Training accounts: {len(accounts_train):,}\")\n", "print(f\" - Training transactions: {len(transactions_train):,}\")\n", "print(f\" - Test accounts: {len(accounts_test):,}\")\n", - "print(f\" - Test transactions: {len(transactions_test):,}\")\n" + "print(f\" - Test transactions: {len(transactions_test):,}\")" ] }, { @@ -150,12 +150,12 @@ "source": [ "print(\"šŸ’¾ Saving split train/test tables...\")\n", "\n", - "accounts_train.to_parquet('./data/two-table/accounts_train.parquet', index=False)\n", - "accounts_test.to_parquet('./data/two-table/accounts_test.parquet', index=False)\n", - "transactions_train.to_parquet('./data/two-table/transactions_train.parquet', index=False)\n", - "transactions_test.to_parquet('./data/two-table/transactions_test.parquet', index=False)\n", + "accounts_train.to_parquet(\"./data/two-table/accounts_train.parquet\", index=False)\n", + "accounts_test.to_parquet(\"./data/two-table/accounts_test.parquet\", index=False)\n", + "transactions_train.to_parquet(\"./data/two-table/transactions_train.parquet\", index=False)\n", + "transactions_test.to_parquet(\"./data/two-table/transactions_test.parquet\", index=False)\n", "\n", - "print(\"āœ… All train/test splits saved to ./data/two-table/\")\n" + "print(\"āœ… All train/test splits saved to ./data/two-table/\")" ] }, { @@ -188,18 +188,14 @@ "metadata": {}, "outputs": [], "source": [ - "from sdv.multi_table import HMASynthesizer\n", "from sdv.metadata import Metadata\n", + "from sdv.multi_table import HMASynthesizer\n", "\n", "print(\"šŸ—ļø Building SDV metadata configuration...\")\n", "\n", "# Auto-detect metadata using only the relevant tables\n", "metadata = Metadata.detect_from_dataframes(\n", - " data={\n", - " 'account': accounts_train,\n", - " 'transaction': transactions_train\n", - " },\n", - " infer_keys='primary_and_foreign'\n", + " data={\"account\": accounts_train, \"transaction\": transactions_train}, infer_keys=\"primary_and_foreign\"\n", ")\n", "\n", "print(\"āœ… Base metadata auto-detected with relationships\")\n", @@ -210,7 +206,7 @@ "# Inspect relationships and table configuration as raw dictionary\n", "metadata_dict = metadata.to_dict()\n", "print(\"\\nšŸ“‹ Complete SDV Metadata Dictionary:\")\n", - "print(metadata_dict)\n" + "print(metadata_dict)" ] }, { @@ -240,7 +236,6 @@ "outputs": [], "source": [ "import time\n", - "from sdv.multi_table import HMASynthesizer\n", "\n", "print(\"šŸš€ Starting SDV training process...\")\n", "print(\"This will involve multiple phases - preprocessing, relationship learning, and table modeling\")\n", @@ -257,21 +252,18 @@ "# 2. Learn multi-table relationships\n", "# 3. Model the statistical distributions of each table\n", "print(\"šŸ“Š Training synthesizer on multi-table data...\")\n", - "synthesizer.fit({\n", - " 'account': accounts_train,\n", - " 'transaction': transactions_train\n", - "})\n", + "synthesizer.fit({\"account\": accounts_train, \"transaction\": transactions_train})\n", "\n", "end_time = time.time()\n", "elapsed_minutes = (end_time - start_time) / 60\n", "\n", - "print(f\"āœ… SDV training completed successfully!\")\n", + "print(\"āœ… SDV training completed successfully!\")\n", "print(f\"ā±ļø Total training time: {elapsed_minutes:.2f} minutes\")\n", "\n", "# Report table sizes for clarity\n", "print(\"šŸ“ˆ Training data breakdown:\")\n", "print(f\" - Accounts: {len(accounts_train):,}\")\n", - "print(f\" - Transactions: {len(transactions_train):,}\")\n" + "print(f\" - Transactions: {len(transactions_train):,}\")" ] }, { @@ -305,24 +297,24 @@ "end_time = time.time()\n", "elapsed_minutes = (end_time - start_time) / 60\n", "\n", - "print(f\"āœ… SDV generation completed successfully!\")\n", + "print(\"āœ… SDV generation completed successfully!\")\n", "print(f\"ā±ļø Generation time: {elapsed_minutes:.2f} minutes\")\n", "\n", "# Calculate synthetic record counts\n", - "synthetic_account_count = len(sdv_synthetic_data['account'])\n", - "synthetic_transaction_count = len(sdv_synthetic_data['transaction'])\n", + "synthetic_account_count = len(sdv_synthetic_data[\"account\"])\n", + "synthetic_transaction_count = len(sdv_synthetic_data[\"transaction\"])\n", "\n", "total_synthetic_records = synthetic_account_count + synthetic_transaction_count\n", "generation_rate = total_synthetic_records / (end_time - start_time)\n", "\n", "print(f\"šŸš€ Generation rate: {generation_rate:,.0f} records/second\")\n", - "print(f\"šŸ“Š Synthetic data breakdown:\")\n", + "print(\"šŸ“Š Synthetic data breakdown:\")\n", "print(f\" - Accounts: {synthetic_account_count:,}\")\n", "print(f\" - Transactions: {synthetic_transaction_count:,}\")\n", "\n", "# Preview of generated synthetic data\n", - "sdv_synthetic_data['account'].head()\n", - "sdv_synthetic_data['transaction'].head()\n" + "sdv_synthetic_data[\"account\"].head()\n", + "sdv_synthetic_data[\"transaction\"].head()" ] }, { @@ -332,16 +324,16 @@ "outputs": [], "source": [ "# Save SDV synthetic data for comparison\n", - "output_folder = './data/two-table/'\n", + "output_folder = \"./data/two-table/\"\n", "\n", "synthetic_files = {\n", - " 'account': f'{output_folder}sdv_account.parquet',\n", - " 'transaction': f'{output_folder}sdv_transaction.parquet',\n", + " \"account\": f\"{output_folder}sdv_account.parquet\",\n", + " \"transaction\": f\"{output_folder}sdv_transaction.parquet\",\n", "}\n", "\n", "for table_name, file_path in synthetic_files.items():\n", " sdv_synthetic_data[table_name].to_parquet(file_path, index=False)\n", - " print(f\"šŸ’¾ Saved {table_name} synthetic data to: {file_path}\")\n" + " print(f\"šŸ’¾ Saved {table_name} synthetic data to: {file_path}\")" ] }, { @@ -417,31 +409,25 @@ "\n", "# Configure the generator for the reduced two-table setup\n", "config = {\n", - " 'name': 'Berka Accounts & Transactions Generator',\n", - " 'tables': [\n", + " \"name\": \"Berka Accounts & Transactions Generator\",\n", + " \"tables\": [\n", " {\n", - " 'name': 'account',\n", - " 'data': accounts_train,\n", - " 'primary_key': 'account_id',\n", - " 'tabular_model_configuration': {\n", - " 'enable_model_report': False\n", - " }\n", + " \"name\": \"account\",\n", + " \"data\": accounts_train,\n", + " \"primary_key\": \"account_id\",\n", + " \"tabular_model_configuration\": {\"enable_model_report\": False},\n", " },\n", " {\n", - " 'name': 'transaction',\n", - " 'data': transactions_train,\n", - " 'primary_key': 'trans_id',\n", - " 'foreign_keys': [\n", - " {'column': 'account_id', 'referenced_table': 'account', 'is_context': True}\n", - " ],\n", - " 'tabular_model_configuration': {\n", - " 'enable_model_report': False\n", - " }\n", - " }\n", - " ]\n", + " \"name\": \"transaction\",\n", + " \"data\": transactions_train,\n", + " \"primary_key\": \"trans_id\",\n", + " \"foreign_keys\": [{\"column\": \"account_id\", \"referenced_table\": \"account\", \"is_context\": True}],\n", + " \"tabular_model_configuration\": {\"enable_model_report\": False},\n", + " },\n", + " ],\n", "}\n", "\n", - "print(\"āœ… MOSTLY AI generator configuration ready.\")\n" + "print(\"āœ… MOSTLY AI generator configuration ready.\")" ] }, { @@ -479,10 +465,9 @@ "end_time = time.time()\n", "elapsed_minutes = (end_time - start_time) / 60\n", "\n", - "print(f\"āœ… MOSTLY AI training completed successfully!\")\n", + "print(\"āœ… MOSTLY AI training completed successfully!\")\n", "print(f\"ā±ļø Total training time: {elapsed_minutes:.2f} minutes\")\n", - "print(f\"šŸ§  Advanced AI models trained for multi-table synthesis\")\n", - " " + "print(\"šŸ§  Advanced AI models trained for multi-table synthesis\")" ] }, { @@ -513,7 +498,7 @@ "total_records = sum(len(mostlyai_synthetic_data[table]) for table in mostlyai_synthetic_data.keys())\n", "generation_rate = total_records / (end_time - start_time)\n", "\n", - "print(f\"āœ… MOSTLY AI generation completed successfully!\")\n", + "print(\"āœ… MOSTLY AI generation completed successfully!\")\n", "print(f\"ā±ļø Generation time: {elapsed_minutes:.2f} minutes\")\n", "print(f\"šŸš€ Generation rate: {generation_rate:,.0f} records/second\")\n", "\n", @@ -522,8 +507,8 @@ " print(f\" - {table_name.capitalize()}: {len(mostlyai_synthetic_data[table_name]):,} rows\")\n", "\n", "# Preview\n", - "mostlyai_synthetic_data['account'].head()\n", - "mostlyai_synthetic_data['transaction'].head()\n" + "mostlyai_synthetic_data[\"account\"].head()\n", + "mostlyai_synthetic_data[\"transaction\"].head()" ] }, { @@ -533,16 +518,16 @@ "outputs": [], "source": [ "# Save MOSTLY AI synthetic data for comparison\n", - "output_folder = './data/two-table/'\n", + "output_folder = \"./data/two-table/\"\n", "\n", "mostlyai_files = {\n", - " 'account': f'{output_folder}mostlyai_account.parquet',\n", - " 'transaction': f'{output_folder}mostlyai_transaction.parquet',\n", + " \"account\": f\"{output_folder}mostlyai_account.parquet\",\n", + " \"transaction\": f\"{output_folder}mostlyai_transaction.parquet\",\n", "}\n", "\n", "for table_name, file_path in mostlyai_files.items():\n", " mostlyai_synthetic_data[table_name].to_parquet(file_path, index=False)\n", - " print(f\"šŸ’¾ Saved {table_name} synthetic data to: {file_path}\")\n" + " print(f\"šŸ’¾ Saved {table_name} synthetic data to: {file_path}\")" ] }, { @@ -585,12 +570,12 @@ "outputs": [], "source": [ "# Load the split files from the disk\n", - "transactions_train = pd.read_parquet('./data/two-table/transactions_train.parquet')\n", - "transactions_test = pd.read_parquet('./data/two-table/transactions_test.parquet')\n", - "accounts_train = pd.read_parquet('./data/two-table/accounts_train.parquet')\n", - "accounts_test = pd.read_parquet('./data/two-table/accounts_test.parquet')\n", + "transactions_train = pd.read_parquet(\"./data/two-table/transactions_train.parquet\")\n", + "transactions_test = pd.read_parquet(\"./data/two-table/transactions_test.parquet\")\n", + "accounts_train = pd.read_parquet(\"./data/two-table/accounts_train.parquet\")\n", + "accounts_test = pd.read_parquet(\"./data/two-table/accounts_test.parquet\")\n", "\n", - "print(\"šŸ“‚ Training and test datasets loaded successfully\")\n" + "print(\"šŸ“‚ Training and test datasets loaded successfully\")" ] }, { @@ -602,30 +587,32 @@ "print(\"šŸ“Š Evaluating SDV Transaction synthetic data quality...\")\n", "\n", "# Load the SDV synthetic dataset\n", - "sdv_transaction = pd.read_parquet('./data/two-table/sdv_transaction.parquet')\n", + "sdv_transaction = pd.read_parquet(\"./data/two-table/sdv_transaction.parquet\")\n", "\n", "# Define ID columns to exclude from QA analysis (but keep account_id for context linkage)\n", - "id_columns_to_exclude = ['trans_id']\n", + "id_columns_to_exclude = [\"trans_id\"]\n", + "\n", "\n", "def remove_id_columns(df, columns_to_remove):\n", " return df.drop(columns=[col for col in columns_to_remove if col in df.columns])\n", "\n", + "\n", "# Prepare data for QA\n", "sdv_transaction_qa = remove_id_columns(sdv_transaction, id_columns_to_exclude)\n", "transactions_train_qa = remove_id_columns(transactions_train, id_columns_to_exclude)\n", "transactions_test_qa = remove_id_columns(transactions_test, id_columns_to_exclude)\n", "\n", "report_path, metrics = qa.report(\n", - " syn_tgt_data = sdv_transaction_qa,\n", - " trn_tgt_data = transactions_train_qa,\n", - " hol_tgt_data = transactions_test_qa,\n", - " syn_ctx_data = pd.read_parquet('./data/two-table/sdv_account.parquet'),\n", - " trn_ctx_data = accounts_train,\n", - " hol_ctx_data = accounts_test,\n", - " ctx_primary_key = \"account_id\",\n", - " tgt_context_key = \"account_id\",\n", + " syn_tgt_data=sdv_transaction_qa,\n", + " trn_tgt_data=transactions_train_qa,\n", + " hol_tgt_data=transactions_test_qa,\n", + " syn_ctx_data=pd.read_parquet(\"./data/two-table/sdv_account.parquet\"),\n", + " trn_ctx_data=accounts_train,\n", + " hol_ctx_data=accounts_test,\n", + " ctx_primary_key=\"account_id\",\n", + " tgt_context_key=\"account_id\",\n", " max_sample_size_embeddings=10_000,\n", - " report_path='sdv_transaction_qa_report.html'\n", + " report_path=\"sdv_transaction_qa_report.html\",\n", ")\n", "\n", "print(f\"šŸ“‹ SDV Transaction Quality Report saved to: {report_path}\")\n", @@ -635,9 +622,9 @@ "# Print summary scores\n", "sdv_transaction_accuracy = metrics.accuracy.overall\n", "sdv_transaction_dcr_share = metrics.distances.dcr_share\n", - "print(f\"\\nšŸŽÆ SDV Transaction Summary:\")\n", + "print(\"\\nšŸŽÆ SDV Transaction Summary:\")\n", "print(f\" Overall Accuracy: {sdv_transaction_accuracy:.3f}\")\n", - "print(f\" DCR Share: {sdv_transaction_dcr_share:.3f}\")\n" + "print(f\" DCR Share: {sdv_transaction_dcr_share:.3f}\")" ] }, { @@ -649,30 +636,32 @@ "print(\"šŸ“Š Evaluating MOSTLY AI Transactions synthetic data quality...\")\n", "\n", "# Load the MOSTLY AI synthetic dataset\n", - "mostlyai_transaction = pd.read_parquet('./data/two-table/mostlyai_transaction.parquet')\n", + "mostlyai_transaction = pd.read_parquet(\"./data/two-table/mostlyai_transaction.parquet\")\n", "\n", "# Define ID columns to exclude from QA analysis (keep account_id)\n", - "id_columns_to_exclude = ['trans_id']\n", + "id_columns_to_exclude = [\"trans_id\"]\n", + "\n", "\n", "def remove_id_columns(df, columns_to_remove):\n", " return df.drop(columns=[col for col in columns_to_remove if col in df.columns])\n", "\n", + "\n", "# Prepare transaction data\n", "mostlyai_transaction_qa = remove_id_columns(mostlyai_transaction, id_columns_to_exclude)\n", "transactions_train_qa = remove_id_columns(transactions_train, id_columns_to_exclude)\n", "transactions_test_qa = remove_id_columns(transactions_test, id_columns_to_exclude)\n", "\n", "report_path, metrics = qa.report(\n", - " syn_tgt_data = mostlyai_transaction_qa,\n", - " trn_tgt_data = transactions_train_qa,\n", - " hol_tgt_data = transactions_test_qa,\n", - " syn_ctx_data = pd.read_parquet('./data/two-table/mostlyai_account.parquet'),\n", - " trn_ctx_data = accounts_train,\n", - " hol_ctx_data = accounts_test,\n", - " ctx_primary_key = \"account_id\",\n", - " tgt_context_key = \"account_id\",\n", + " syn_tgt_data=mostlyai_transaction_qa,\n", + " trn_tgt_data=transactions_train_qa,\n", + " hol_tgt_data=transactions_test_qa,\n", + " syn_ctx_data=pd.read_parquet(\"./data/two-table/mostlyai_account.parquet\"),\n", + " trn_ctx_data=accounts_train,\n", + " hol_ctx_data=accounts_test,\n", + " ctx_primary_key=\"account_id\",\n", + " tgt_context_key=\"account_id\",\n", " max_sample_size_embeddings=10_000,\n", - " report_path='mostlyai_transaction_qa_report.html'\n", + " report_path=\"mostlyai_transaction_qa_report.html\",\n", ")\n", "\n", "print(f\"šŸ“‹ MOSTLY AI Transaction Quality Report saved to: {report_path}\")\n", @@ -682,9 +671,9 @@ "# Print summary scores\n", "mostlyai_transaction_accuracy = metrics.accuracy.overall\n", "mostlyai_transaction_dcr_share = metrics.distances.dcr_share\n", - "print(f\"\\nšŸŽÆ MOSTLY AI Transaction Summary:\")\n", + "print(\"\\nšŸŽÆ MOSTLY AI Transaction Summary:\")\n", "print(f\" Overall Accuracy: {mostlyai_transaction_accuracy:.3f}\")\n", - "print(f\" DCR Share: {mostlyai_transaction_dcr_share:.3f}\")\n" + "print(f\" DCR Share: {mostlyai_transaction_dcr_share:.3f}\")" ] }, { @@ -694,11 +683,13 @@ "outputs": [], "source": [ "# Add a final comparison section\n", - "print(\"\\n\" + \"=\"*60)\n", + "print(\"\\n\" + \"=\" * 60)\n", "print(\"šŸ† FINAL COMPARISON\")\n", - "print(\"=\"*60)\n", + "print(\"=\" * 60)\n", "print(f\"SDV Transaction - Accuracy: {sdv_transaction_accuracy:.3f}, DCR Share: {sdv_transaction_dcr_share:.3f}\")\n", - "print(f\"MOSTLY AI Transaction- Accuracy: {mostlyai_transaction_accuracy:.3f}, DCR Share: {mostlyai_transaction_dcr_share:.3f}\")\n", + "print(\n", + " f\"MOSTLY AI Transaction- Accuracy: {mostlyai_transaction_accuracy:.3f}, DCR Share: {mostlyai_transaction_dcr_share:.3f}\"\n", + ")\n", "\n", "print(\"\\nšŸ” METRIC INTERPRETATION:\")\n", "print(\"ā€¢ Higher accuracy = better statistical fidelity\")\n", @@ -712,7 +703,7 @@ "print(\"\\nāš ļø RECOMMENDATION:\")\n", "print(\"ā€¢ Review detailed HTML reports for nuanced privacy insights\")\n", "print(\"ā€¢ Pay attention to discriminator AUC and feature-wise similarity scores\")\n", - "print(\"ā€¢ Align final choice with your privacy-utility balance requirements\")\n" + "print(\"ā€¢ Align final choice with your privacy-utility balance requirements\")" ] } ], From e725fb3456f0a1083f16c5e95e4db5704af53d49 Mon Sep 17 00:00:00 2001 From: Kenneth Hamilton Date: Wed, 30 Jul 2025 15:30:54 +0200 Subject: [PATCH 5/8] finalize --- .../sequential-scenario.ipynb} | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) rename docs/tutorials/sdv-comparison/{two-table-scenario/two-table-scenario.ipynb => sequential-scenario/sequential-scenario.ipynb} (97%) diff --git a/docs/tutorials/sdv-comparison/two-table-scenario/two-table-scenario.ipynb b/docs/tutorials/sdv-comparison/sequential-scenario/sequential-scenario.ipynb similarity index 97% rename from docs/tutorials/sdv-comparison/two-table-scenario/two-table-scenario.ipynb rename to docs/tutorials/sdv-comparison/sequential-scenario/sequential-scenario.ipynb index f76291c1..d0fb6f9a 100644 --- a/docs/tutorials/sdv-comparison/two-table-scenario/two-table-scenario.ipynb +++ b/docs/tutorials/sdv-comparison/sequential-scenario/sequential-scenario.ipynb @@ -4,13 +4,13 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# MOSTLY AI vs. SDV Comparison - Two-Table Scenario \"Run\n", + "# MOSTLY AI vs. SDV Comparison - Sequential Scenario \"Run\n", "\n", "This notebook provides a comprehensive comparison between two leading synthetic data generation platforms:\n", "- **SDV (Synthetic Data Vault)** - Business Source License\n", "- **MOSTLY AI Synthetic Data SDK** - Apache 2.0 License - Open Source\n", "\n", - "In this comparison, we are going to walk through the synthesis of a relational two-table structure using the [Berka dataset](https://github.com/mostly-ai/public-demo-data/tree/dev/berka/data).\n", + "In this comparison, we are going to walk through the synthesis of a relational two-table structure with sequential values using the [Berka dataset](https://github.com/mostly-ai/public-demo-data/tree/dev/berka/data).\n", "\n", "## Comparison Methodology\n", "\n", @@ -273,7 +273,7 @@ "### 3.3 SDV Synthetic Data Generation\n", "\n", "**Generation Process:**\n", - "- **Scale Parameter**: Controls the number of synthetic records (1.0 = same size as training data)\n", + "- **Scale Parameter**: Controls the number of synthetic records (1.25 = same size as training data)\n", "- **Hierarchical Generation**: First generates parent records (accounts), then child records (transactions)\n", "- **Relationship Preservation**: Ensures all synthetic transfers reference valid synthetic customers\n", "- **Statistical Sampling**: Uses learned distributions to create realistic synthetic data\n" @@ -291,8 +291,8 @@ "start_time = time.time()\n", "\n", "# Generate synthetic data with the same number of records as training data\n", - "print(\"āš™ļø Generating 1.00x the training data size...\")\n", - "sdv_synthetic_data = synthesizer.sample(scale=1.00)\n", + "print(\"āš™ļø Generating 1.25x the training data size...\")\n", + "sdv_synthetic_data = synthesizer.sample(scale=1.25)\n", "\n", "end_time = time.time()\n", "elapsed_minutes = (end_time - start_time) / 60\n", From 217dfb86b27608f0c06c71edfb6628e3ecadfc69 Mon Sep 17 00:00:00 2001 From: Kenneth Hamilton Date: Wed, 30 Jul 2025 15:32:05 +0200 Subject: [PATCH 6/8] update colab links --- .../single-table-scenario/single-table-scenario.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tutorials/sdv-comparison/single-table-scenario/single-table-scenario.ipynb b/docs/tutorials/sdv-comparison/single-table-scenario/single-table-scenario.ipynb index 21fb8a0d..732ab16e 100644 --- a/docs/tutorials/sdv-comparison/single-table-scenario/single-table-scenario.ipynb +++ b/docs/tutorials/sdv-comparison/single-table-scenario/single-table-scenario.ipynb @@ -14,7 +14,7 @@ "id": "WJZteD10A1eR" }, "source": [ - "# MOSTLY AI vs. SDV Comparison - Single Table Scenario\n", + "# MOSTLY AI vs. SDV Comparison - Single Table Scenario \"Run\n", "\n", "## Framework Comparison\n", "This notebook compares two synthetic data generation libraries on a large-scale dataset:\n", From e1764b521b11fb7e9265422a230f22da58b26016 Mon Sep 17 00:00:00 2001 From: Kenneth Hamilton Date: Wed, 30 Jul 2025 15:40:43 +0200 Subject: [PATCH 7/8] finalize --- .../sequential-scenario/sequential-scenario.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tutorials/sdv-comparison/sequential-scenario/sequential-scenario.ipynb b/docs/tutorials/sdv-comparison/sequential-scenario/sequential-scenario.ipynb index d0fb6f9a..49b84795 100644 --- a/docs/tutorials/sdv-comparison/sequential-scenario/sequential-scenario.ipynb +++ b/docs/tutorials/sdv-comparison/sequential-scenario/sequential-scenario.ipynb @@ -4,7 +4,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# MOSTLY AI vs. SDV Comparison - Sequential Scenario \"Run\n", + "# MOSTLY AI vs. SDV Comparison - Sequential Scenario \"Run\n", "\n", "This notebook provides a comprehensive comparison between two leading synthetic data generation platforms:\n", "- **SDV (Synthetic Data Vault)** - Business Source License\n", From b141c746b35bc4693d0aaa683496a0ab11c183c6 Mon Sep 17 00:00:00 2001 From: Kenneth Hamilton Date: Thu, 31 Jul 2025 09:54:54 +0200 Subject: [PATCH 8/8] update tutorial --- docs/tutorials.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tutorials.md b/docs/tutorials.md index c0398b47..93a635d8 100644 --- a/docs/tutorials.md +++ b/docs/tutorials.md @@ -26,4 +26,4 @@ Explore synthetic data tutorials with the option to run them **either in Google | Calculate accuracy and privacy metrics for **Quality Assurance** | [![Run on Colab](https://img.shields.io/badge/Open%20in-Colab-blue?logo=google-colab)](https://colab.research.google.com/github/mostly-ai/mostlyai/blob/main/docs/tutorials/quality-assurance/quality-assurance.ipynb) | [View Notebook](./tutorials/quality-assurance/quality-assurance.ipynb) | | Enrich Sensitive Data with LLMs using Synthetic Replicas | [![Run on Colab](https://img.shields.io/badge/Open%20in-Colab-blue?logo=google-colab)](https://colab.research.google.com/github/mostly-ai/mostlyai/blob/main/docs/tutorials/synthetic-enrich/synthetic-enrich.ipynb) | [View Notebook](./tutorials/synthetic-enrich/synthetic-enrich.ipynb) | | MOSTLY AI vs. SDV comparison: single-table scenario | [![Run on Colab](https://img.shields.io/badge/Open%20in-Colab-blue?logo=google-colab)](https://colab.research.google.com/github/mostly-ai/mostlyai/blob/main/docs/tutorials/synthetic-enrich/sdv-comparison/single-tables-scenario.ipynb) | [View Notebook](./tutorials/sdv-comparison/single-table-scenario/single-table-scenario.ipynb) | -| MOSTLY AI vs. SDV comparison: multi-table scenario | [![Run on Colab](https://img.shields.io/badge/Open%20in-Colab-blue?logo=google-colab)](https://colab.research.google.com/github/mostly-ai/mostlyai/blob/main/docs/tutorials/synthetic-enrich/sdv-comparison/multi-tables-scenario.ipynb) | [View Notebook](./tutorials/sdv-comparison/multi-table-scenario/multi-table-scenario.ipynb) | +| MOSTLY AI vs. SDV comparison: sequential scenario | [![Run on Colab](https://img.shields.io/badge/Open%20in-Colab-blue?logo=google-colab)](https://colab.research.google.com/github/mostly-ai/mostlyai/blob/main/docs/tutorials/synthetic-enrich/sdv-comparison/sequential-scenario/sequential-scenario.ipynb) | [View Notebook](./tutorials/sdv-comparison/sequential-scenario/sequential-scenario.ipynb) |