diff --git a/docs/synthetic-enrich/synthetic-enrich.ipynb b/docs/synthetic-enrich/synthetic-enrich.ipynb new file mode 100644 index 00000000..2482dd1e --- /dev/null +++ b/docs/synthetic-enrich/synthetic-enrich.ipynb @@ -0,0 +1,655 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "LVzkBFce0_4x" + }, + "source": [ + "# Enrich Sensitive Data with LLMs using Synthetic Replicas \"Run\n", + "\n", + "This notebook shows how to safely enrich sensitive datasets with new LLM-generated columns - without sharing any private data. We first create a privacy-safe synthetic replica of the original dataset. The synthetic replica is enriched via an LLM. We then train a generator on this enriched replica. Finally, the generator applies the same enrichment to the original sensitive data - without the data ever leaving your environment.\n", + "\n", + "πŸ“‹ Steps\n", + "\n", + "1. Create a synthetic replica of your dataset\n", + "2. Use an LLM to add new columns to the replica\n", + "3. Train a generator on the enriched replica\n", + "4. Generate enriched original data using the trained generator\n", + "\n", + "πŸ” Key Benefits\n", + "\n", + "* No data exposure: Original data stays secure.\n", + "* Enrichment at scale: LLMs enrich synthetic data; the generator brings that intelligence back.\n", + "* Reusable logic: Once trained, the generator acts as a secure enrichment adapter - no repeated LLM calls needed.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "XyI2Ayj72OX9" + }, + "source": [ + "## Install Dependencies" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "2tc8hHbjz3aK" + }, + "outputs": [], + "source": [ + "# Install SDK in CLIENT mode\n", + "!uv pip install -U mostlyai mostlyai-mock\n", + "# Or install in LOCAL mode\n", + "!uv pip install -U 'mostlyai[local]' mostlyai-mock\n", + "# Note: Restart kernel session after installation!" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "B424tIf329Xv" + }, + "source": [ + "## Load Original Data\n", + "\n", + "Fetch a sample of the census dataset that will be used as our sensitive proprietary data that we want to enrich while keeping it private." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "OaaLzwJO4KFP", + "outputId": "ec8efb89-a74b-4e39-cf4c-3512655367e3" + }, + "outputs": [], + "source": [ + "# load sample of original data\n", + "import pandas as pd\n", + "\n", + "df_orig = pd.read_csv(\"https://github.com/mostly-ai/public-demo-data/raw/dev/census/census.csv.gz\", nrows=2000)\n", + "df_orig.head()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "9ebt84Nk4N5G" + }, + "source": [ + "## Initialize SDK\n", + "\n", + "The SDK will handle model training and synthetic data generation, while DataLLM will provide the LLM enrichment capabilities." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 52 + }, + "id": "2l66fJ9p4ccz", + "outputId": "7e3e61a0-233d-4339-a665-3d9dc7d16c04" + }, + "outputs": [], + "source": [ + "from mostlyai.sdk import MostlyAI\n", + "\n", + "# initialize SDK\n", + "mostly = MostlyAI()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "3nu3GHMX5GRx" + }, + "source": [ + "## Train a Generator on the Original Data\n", + "\n", + "Train a generator on the sensitive original data." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "kNZ_35TI5KRD" + }, + "outputs": [], + "source": [ + "# train generator on original data\n", + "g = mostly.train(data=df_orig)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "6NUTEH0y56Nx" + }, + "source": [ + "## Generate Synthetic Data\n", + "\n", + "Create synthetic data that will serve us as a proxy of the sensitive original dataset. This synthetic data will be shared with the LLM for enrichment." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 151, + "referenced_widgets": [ + "0d7508d3c7824a388dad8c211f06fc9e", + "42e68ad3cc324f50bd6246dc0942988e" + ] + }, + "id": "XSdvVU9y6LhB", + "outputId": "322dd567-d2ed-42e9-d632-f665697030a7" + }, + "outputs": [], + "source": [ + "# generate synthetic data\n", + "df_syn = mostly.probe(g, size=len(df_orig))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "HhJoUYoDyV9q" + }, + "source": [ + "## Enrich via MOCK" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "snSNuS2F6PbT" + }, + "source": [ + "Use the LLM to enrich the synthetic data with 2 new columns, namely: **work category**, and **career stage**. This is where we expose data to the LLM, but only the synthetic proxy data, not our sensitive original data." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "PfLcu2RbyUsy" + }, + "outputs": [], + "source": [ + "# Set your OpenAI API key as environment variable, required by mostlyai-mock\n", + "# import os\n", + "# os.environ[\"OPENAI_API_KEY\"] = \"YOUR_KEY_HERE\"\n", + "\n", + "from mostlyai import mock\n", + "\n", + "mock_config_tables = {\n", + " \"census\": {\n", + " \"prompt\": \"U.S. Census data with demographic and employment-related columns\",\n", + " \"columns\": {\n", + " \"specific_job_title\": {\n", + " \"prompt\": (\n", + " \"Generate a realistic, specific job title for a person \"\n", + " \"based on their occupation, education, and income level. \"\n", + " \"The job title should be more specific than the general \"\n", + " \"occupation category.\"\n", + " ),\n", + " \"dtype\": \"string\",\n", + " },\n", + " \"work_category\": {\n", + " \"prompt\": \"\"\"categorize the occupation into work category, considering the actual job duties and level.\n", + " Examples of correct categorizations:\n", + " - Handlers-cleaners β†’ Manual Labor\n", + " - Machine-op-inspct β†’ Manual Labor\n", + " - Craft-repair β†’ Manual Labor\n", + " - Transport-moving β†’ Manual Labor\n", + " - Farming-fishing β†’ Manual Labor\n", + " - Exec-managerial β†’ Management\n", + " - Prof-specialty β†’ Professional\n", + " - Tech-support β†’ Technical\n", + " - Sales β†’ Service Work\n", + " - Other-service β†’ Service Work\n", + " \n", + " Categories and their meanings:\n", + " - Manual Labor: physical work, manufacturing, construction, cleaning, transportation, farming, machine operation, craft work, manual repairs, physical labor\n", + " - Service Work: customer service, retail, hospitality, food service, personal care, non-physical service roles\n", + " - Professional: doctors, lawyers, engineers, scientists, specialized knowledge workers\n", + " - Management: supervisors, executives, administrators, team leaders\n", + " - Technical: IT, technical support, specialized technical skills, maintenance\"\"\",\n", + " \"dtype\": \"category\",\n", + " \"values\": [\"Manual Labor\", \"Service Work\", \"Professional\", \"Management\", \"Technical\"],\n", + " },\n", + " },\n", + " }\n", + "}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# this will run for ~5min\n", + "df_syn_enriched = mock.sample(\n", + " tables=mock_config_tables,\n", + " existing_data={\"census\": df_syn},\n", + " model=\"openai/gpt-4.1-nano\",\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 206 + }, + "id": "I8fGcP__1EFf", + "outputId": "feee5ed8-7a22-4d76-ee94-03259f8f0ec4" + }, + "outputs": [], + "source": [ + "df_syn_enriched.head()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "mJJr9yOU67Mb" + }, + "source": [ + "## Train a Generator on the Enriched Synthetic Data\n", + "\n", + "Train a generator on the enriched synthetic data to encode the LLM intelligence into a reusable, privacy-safe enrichment model. This enables it to later apply the same intelligence to sensitive original data." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 384, + "referenced_widgets": [ + "dcb5bfef4ff14d1d9216d9ea39796fa1", + "5c015acb44a24393975aa21a8b3605e5" + ] + }, + "id": "W2WM7dS_7JVu", + "outputId": "2985ceae-f1fb-4689-a535-85b5077dfd63" + }, + "outputs": [], + "source": [ + "# train generator on enriched synthetic data\n", + "config = {\n", + " \"name\": \"Enriched Census\",\n", + " \"tables\": [\n", + " {\n", + " \"name\": \"Census\",\n", + " \"data\": df_syn_enriched,\n", + " \"tabular_model_configuration\": {\n", + " \"value_protection\": False, # not needed as training data is not private\n", + " },\n", + " }\n", + " ],\n", + "}\n", + "g = mostly.train(config=config)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "EInzGRFXKUr0" + }, + "source": [ + "## Use Generator to Enrich Original Data\n", + "\n", + "Now we use the generator trained on enriched synthetic data to add the same new features to the original sensitive data. We do this by fixing the original data as the seed input to the generator, which then produces the enriched version with the same feature transformation. This approach ensures that the original data's structure and relationships are preserved while the new features are generated consistently with the same patterns learned from the synthetic data. Your sensitive data remains untouched - yet is now enhanced with the same intelligent enrichments, thanks to the generator’s learned transformations." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 135, + "referenced_widgets": [ + "bd5be9699cc74a9fabd9f8ff3b4142cb", + "80b67ef39b164de6b37a79b28fe06c50" + ] + }, + "id": "xX9JfwzTKUEP", + "outputId": "cdde6c72-9b09-4d27-8bc4-67d5374a88fd" + }, + "outputs": [], + "source": [ + "# generate enriched original data using original data as seed\n", + "df_orig_enriched = mostly.probe(g, seed=df_orig)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# display sample of enriched original data\n", + "df_orig_enriched.sample(n=10)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "xbe1HoQT8Gri" + }, + "source": [ + "## Conclusion\n", + "\n", + "This tutorial demonstrated how to securely enrich sensitive proprietary data by:\n", + "\n", + "1. Creating a synthetic replica\n", + "2. Enriching the proxy with an LLM\n", + "3. Training a generator on the enriched replica\n", + "4. Applying the enrichment to the sensitive data\n", + "\n", + "The sensitive data never leaves your secure environment, maintaining privacy while enabling LLM-based enrichment." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.8" + }, + "widgets": { + "application/vnd.jupyter.widget-state+json": { + "0d7508d3c7824a388dad8c211f06fc9e": { + "model_module": "@jupyter-widgets/output", + "model_module_version": "1.0.0", + "model_name": "OutputModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/output", + "_model_module_version": "1.0.0", + "_model_name": "OutputModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/output", + "_view_module_version": "1.0.0", + "_view_name": "OutputView", + "layout": "IPY_MODEL_42e68ad3cc324f50bd6246dc0942988e", + "msg_id": "", + "outputs": [ + { + "data": { + "text/html": "
Overall job progress                                 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━        100%  0:00:20 \nStep data:tabular GENERATE_DATA_TABULAR              ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━        100%  0:00:00 \nStep data:tabular CREATE_DATA_REPORT_TABULAR         ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━        100%  0:00:18 \nStep common FINALIZE_GENERATION                      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━        100%  0:00:02 \nStep common DELIVER_DATA                             ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━        100%  0:00:00 \n
\n", + "text/plain": "\u001b[1mOverall job progress\u001b[0m \u001b[38;2;36;219;149m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:20 \u001b[0m\nStep data:tabular \u001b[38;2;128;128;128mGENERATE_DATA_TABULAR\u001b[0m \u001b[38;2;36;219;149m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:00 \u001b[0m\nStep data:tabular \u001b[38;2;128;128;128mCREATE_DATA_REPORT_TABULAR\u001b[0m \u001b[38;2;36;219;149m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:18 \u001b[0m\nStep common \u001b[38;2;128;128;128mFINALIZE_GENERATION\u001b[0m \u001b[38;2;36;219;149m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:02 \u001b[0m\nStep common \u001b[38;2;128;128;128mDELIVER_DATA\u001b[0m \u001b[38;2;36;219;149m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:00 \u001b[0m\n" + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "tabbable": null, + "tooltip": null + } + }, + "42e68ad3cc324f50bd6246dc0942988e": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "2.0.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "2.0.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "2.0.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border_bottom": null, + "border_left": null, + "border_right": null, + "border_top": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "5c015acb44a24393975aa21a8b3605e5": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "2.0.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "2.0.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "2.0.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border_bottom": null, + "border_left": null, + "border_right": null, + "border_top": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "80b67ef39b164de6b37a79b28fe06c50": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "2.0.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "2.0.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "2.0.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border_bottom": null, + "border_left": null, + "border_right": null, + "border_top": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "bd5be9699cc74a9fabd9f8ff3b4142cb": { + "model_module": "@jupyter-widgets/output", + "model_module_version": "1.0.0", + "model_name": "OutputModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/output", + "_model_module_version": "1.0.0", + "_model_name": "OutputModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/output", + "_view_module_version": "1.0.0", + "_view_name": "OutputView", + "layout": "IPY_MODEL_80b67ef39b164de6b37a79b28fe06c50", + "msg_id": "", + "outputs": [ + { + "data": { + "text/html": "
Overall job progress                               ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━          100%  0:00:02 \nStep Census:tabular GENERATE_DATA_TABULAR          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━          100%  0:00:00 \nStep common FINALIZE_GENERATION                    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━          100%  0:00:02 \nStep common DELIVER_DATA                           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━          100%  0:00:00 \n
\n", + "text/plain": "\u001b[1mOverall job progress\u001b[0m \u001b[38;2;36;219;149m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:02 \u001b[0m\nStep Census:tabular \u001b[38;2;128;128;128mGENERATE_DATA_TABULAR\u001b[0m \u001b[38;2;36;219;149m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:00 \u001b[0m\nStep common \u001b[38;2;128;128;128mFINALIZE_GENERATION\u001b[0m \u001b[38;2;36;219;149m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:02 \u001b[0m\nStep common \u001b[38;2;128;128;128mDELIVER_DATA\u001b[0m \u001b[38;2;36;219;149m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:00 \u001b[0m\n" + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "tabbable": null, + "tooltip": null + } + }, + "dcb5bfef4ff14d1d9216d9ea39796fa1": { + "model_module": "@jupyter-widgets/output", + "model_module_version": "1.0.0", + "model_name": "OutputModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/output", + "_model_module_version": "1.0.0", + "_model_name": "OutputModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/output", + "_view_module_version": "1.0.0", + "_view_name": "OutputView", + "layout": "IPY_MODEL_5c015acb44a24393975aa21a8b3605e5", + "msg_id": "", + "outputs": [ + { + "data": { + "text/html": "
Overall job progress                               ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━          100%  0:00:38 \nStep Census:tabular PULL_TRAINING_DATA             ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━          100%  0:00:00 \nStep Census:tabular ANALYZE_TRAINING_DATA          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━          100%  0:00:00 \nStep Census:tabular ENCODE_TRAINING_DATA           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━          100%  0:00:00 \nStep Census:tabular TRAIN_MODEL πŸ’Ž                 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━          100%  0:00:38 \n                                                                                                                   \n                                                                                                                   \n                                                                                                                   \n                                         Training log for `Census:tabular`                                         \n                                                                                                                   \n                Epochs                  Samples                           Elapsed Time                   Val Loss  \n ───────────────────────────────────────────────────────────────────────────────────────────────────────────────── \n                 12.00                    9,375                                    25s                    26.0087  \n                 13.00                   10,156                                    27s                    26.1905  \n                 14.00                   10,937                                    29s                    26.0955  \n                 15.00                   11,718                                    31s                    26.4537  \n                 16.00                   12,499                                    34s                    26.1656  \n                 17.00                   13,280                                    36s                    26.6919  \n                                                                                                                   \n
\n", + "text/plain": "\u001b[1mOverall job progress\u001b[0m \u001b[38;2;36;219;149m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:38 \u001b[0m\nStep Census:tabular \u001b[38;2;128;128;128mPULL_TRAINING_DATA\u001b[0m \u001b[38;2;36;219;149m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:00 \u001b[0m\nStep Census:tabular \u001b[38;2;128;128;128mANALYZE_TRAINING_DATA\u001b[0m \u001b[38;2;36;219;149m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:00 \u001b[0m\nStep Census:tabular \u001b[38;2;128;128;128mENCODE_TRAINING_DATA\u001b[0m \u001b[38;2;36;219;149m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:00 \u001b[0m\nStep Census:tabular \u001b[38;2;128;128;128mTRAIN_MODEL πŸ’Ž\u001b[0m \u001b[38;2;36;219;149m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[33m0:00:38 \u001b[0m\n \n \n \n\u001b[3m Training log for `Census:tabular` \u001b[0m\n \n Epochs Samples Elapsed Time Val Loss \n ───────────────────────────────────────────────────────────────────────────────────────────────────────────────── \n \u001b[38;2;20;181;125;48;2;240;255;247m \u001b[0m\u001b[38;2;20;181;125;48;2;240;255;247m 12.00\u001b[0m\u001b[38;2;20;181;125;48;2;240;255;247m \u001b[0m\u001b[48;2;240;255;247m \u001b[0m\u001b[38;2;20;181;125;48;2;240;255;247m \u001b[0m\u001b[38;2;20;181;125;48;2;240;255;247m 9,375\u001b[0m\u001b[38;2;20;181;125;48;2;240;255;247m \u001b[0m\u001b[48;2;240;255;247m \u001b[0m\u001b[38;2;20;181;125;48;2;240;255;247m \u001b[0m\u001b[38;2;20;181;125;48;2;240;255;247m 25s\u001b[0m\u001b[38;2;20;181;125;48;2;240;255;247m \u001b[0m\u001b[48;2;240;255;247m \u001b[0m\u001b[38;2;20;181;125;48;2;240;255;247m \u001b[0m\u001b[38;2;20;181;125;48;2;240;255;247m 26.0087\u001b[0m\u001b[38;2;20;181;125;48;2;240;255;247m \u001b[0m \n \u001b[90m \u001b[0m\u001b[90m 13.00\u001b[0m\u001b[90m \u001b[0m \u001b[90m \u001b[0m\u001b[90m 10,156\u001b[0m\u001b[90m \u001b[0m \u001b[90m \u001b[0m\u001b[90m 27s\u001b[0m\u001b[90m \u001b[0m \u001b[90m \u001b[0m\u001b[90m 26.1905\u001b[0m\u001b[90m \u001b[0m \n \u001b[90m \u001b[0m\u001b[90m 14.00\u001b[0m\u001b[90m \u001b[0m \u001b[90m \u001b[0m\u001b[90m 10,937\u001b[0m\u001b[90m \u001b[0m \u001b[90m \u001b[0m\u001b[90m 29s\u001b[0m\u001b[90m \u001b[0m \u001b[90m \u001b[0m\u001b[90m 26.0955\u001b[0m\u001b[90m \u001b[0m \n \u001b[90m \u001b[0m\u001b[90m 15.00\u001b[0m\u001b[90m \u001b[0m \u001b[90m \u001b[0m\u001b[90m 11,718\u001b[0m\u001b[90m \u001b[0m \u001b[90m \u001b[0m\u001b[90m 31s\u001b[0m\u001b[90m \u001b[0m \u001b[90m \u001b[0m\u001b[90m 26.4537\u001b[0m\u001b[90m \u001b[0m \n \u001b[90m \u001b[0m\u001b[90m 16.00\u001b[0m\u001b[90m \u001b[0m \u001b[90m \u001b[0m\u001b[90m 12,499\u001b[0m\u001b[90m \u001b[0m \u001b[90m \u001b[0m\u001b[90m 34s\u001b[0m\u001b[90m \u001b[0m \u001b[90m \u001b[0m\u001b[90m 26.1656\u001b[0m\u001b[90m \u001b[0m \n \u001b[90m \u001b[0m\u001b[90m 17.00\u001b[0m\u001b[90m \u001b[0m \u001b[90m \u001b[0m\u001b[90m 13,280\u001b[0m\u001b[90m \u001b[0m \u001b[90m \u001b[0m\u001b[90m 36s\u001b[0m\u001b[90m \u001b[0m \u001b[90m \u001b[0m\u001b[90m 26.6919\u001b[0m\u001b[90m \u001b[0m \n \n" + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "tabbable": null, + "tooltip": null + } + } + } + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/docs/tutorials.md b/docs/tutorials.md index d01f5678..5cc80847 100644 --- a/docs/tutorials.md +++ b/docs/tutorials.md @@ -24,3 +24,4 @@ Explore synthetic data tutorials with the option to run them **either in Google | Develop a **fake or real discriminator** with Synthetic Data | [![Run on Colab](https://img.shields.io/badge/Open%20in-Colab-blue?logo=google-colab)](https://colab.research.google.com/github/mostly-ai/mostlyai/blob/main/docs/tutorials/fake-or-real/fake-or-real.ipynb) | [View Notebook](./tutorials/fake-or-real/fake-or-real.ipynb) | | Close gaps in your data with **Smart Imputation** | [![Run on Colab](https://img.shields.io/badge/Open%20in-Colab-blue?logo=google-colab)](https://colab.research.google.com/github/mostly-ai/mostlyai/blob/main/docs/tutorials/smart-imputation/smart-imputation.ipynb) | [View Notebook](./tutorials/smart-imputation/smart-imputation.ipynb) | | Calculate accuracy and privacy metrics for **Quality Assurance** | [![Run on Colab](https://img.shields.io/badge/Open%20in-Colab-blue?logo=google-colab)](https://colab.research.google.com/github/mostly-ai/mostlyai/blob/main/docs/tutorials/quality-assurance/quality-assurance.ipynb) | [View Notebook](./tutorials/quality-assurance/quality-assurance.ipynb) | +| Enrich Sensitive Data with LLMs using Synthetic Replicas | [![Run on Colab](https://img.shields.io/badge/Open%20in-Colab-blue?logo=google-colab)](https://colab.research.google.com/github/mostly-ai/mostlyai/blob/main/docs/tutorials/synthetic-enrich/synthetic-enrich.ipynb) | [View Notebook](./tutorials/synthetic-enrich/synthetic-enrich.ipynb) |