python/mlcroissant/recipes/introduction.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AriH9CP6AKhs"
      },
      "source": [
        "# Tutorial for `mlcroissant` 🥐"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hh-0cehIAErA"
      },
      "source": [
        "## Introduction\n",
        "\n",
        "Croissant 🥐 is a high-level format for machine learning datasets that combines metadata, resource file descriptions, data structure, and default ML semantics into a single file.\n",
        "\n",
        "Croissant builds on schema.org, and its `sc:Dataset` vocabulary, a widely used format to represent datasets on the Web, and make them searchable.\n",
        "\n",
        "The [`mlcroissant`](https://github.com/mlcommons/croissant/python/mlcroissant) Python library empowers developers to interact with Croissant:\n",
        "\n",
        "- Programmatically write your JSON-LD Croissant files.\n",
        "- Verify your JSON-LD Croissant files.\n",
        "- Load data from Croissant datasets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0qpWrlwV-x52"
      },
      "outputs": [],
      "source": [
        "# Install mlcroissant from the source\n",
        "!apt-get install -y python3-dev graphviz libgraphviz-dev pkg-config\n",
        "!pip install \"git+https://github.com/${GITHUB_REPOSITORY:-mlcommons/croissant}.git@${GITHUB_HEAD_REF:-main}#subdirectory=python/mlcroissant&egg=mlcroissant[dev]\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xwrol5JR_GTY"
      },
      "source": [
        "## Example\n",
        "\n",
        "Let's try on a very concrete dataset: OpenAI's [`gpt-3`](https://github.com/openai/gpt-3) dataset for LLMs!\n",
        "\n",
        "In the tutorial, we will generate programmatically the Croissant JSON-LD file describing the dataset. Then we will verify the file and yield data from the dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7OyQffJv-zso"
      },
      "outputs": [],
      "source": [
        "import mlcroissant as mlc\n",
        "\n",
        "# FileObjects and FileSets define the resources of the dataset.\n",
        "distribution = [\n",
        "    # gpt-3 is hosted on a GitHub repository:\n",
        "    mlc.FileObject(\n",
        "        id=\"github-repository\",\n",
        "        name=\"github-repository\",\n",
        "        description=\"OpenAI repository on GitHub.\",\n",
        "        content_url=\"https://github.com/openai/gpt-3\",\n",
        "        encoding_formats=[\"git+https\"],\n",
        "        sha256=\"main\",\n",
        "    ),\n",
        "    # Within that repository, a FileSet lists all JSONL files:\n",
        "    mlc.FileSet(\n",
        "        id=\"jsonl-files\",\n",
        "        name=\"jsonl-files\",\n",
        "        description=\"JSONL files are hosted on the GitHub repository.\",\n",
        "        contained_in=[\"github-repository\"],\n",
        "        encoding_formats=[\"application/jsonlines\"],\n",
        "        includes=\"data/*.jsonl\",\n",
        "    ),\n",
        "]\n",
        "record_sets = [\n",
        "    # RecordSets contains records in the dataset.\n",
        "    mlc.RecordSet(\n",
        "        id=\"jsonl\",\n",
        "        name=\"jsonl\",\n",
        "        # Each record has one or many fields...\n",
        "        fields=[\n",
        "            # Fields can be extracted from the FileObjects/FileSets.\n",
        "            mlc.Field(\n",
        "                id=\"jsonl/context\",\n",
        "                name=\"context\",\n",
        "                description=\"\",\n",
        "                data_types=mlc.DataType.TEXT,\n",
        "                source=mlc.Source(\n",
        "                    file_set=\"jsonl-files\",\n",
        "                    # Extract the field from the column of a FileObject/FileSet:\n",
        "                    extract=mlc.Extract(column=\"context\"),\n",
        "                ),\n",
        "            ),\n",
        "            mlc.Field(\n",
        "                id=\"jsonl/completion\",\n",
        "                name=\"completion\",\n",
        "                description=\"The expected completion of the promt.\",\n",
        "                data_types=mlc.DataType.TEXT,\n",
        "                source=mlc.Source(\n",
        "                    file_set=\"jsonl-files\",\n",
        "                    extract=mlc.Extract(column=\"completion\"),\n",
        "                ),\n",
        "            ),\n",
        "            mlc.Field(\n",
        "                id=\"jsonl/task\",\n",
        "                name=\"task\",\n",
        "                description=(\n",
        "                    \"The machine learning task appearing as the name of the\"\n",
        "                    \" file.\"\n",
        "                ),\n",
        "                data_types=mlc.DataType.TEXT,\n",
        "                source=mlc.Source(\n",
        "                    file_set=\"jsonl-files\",\n",
        "                    extract=mlc.Extract(\n",
        "                        file_property=mlc._src.structure_graph.nodes.source.FileProperty.filename\n",
        "                    ),\n",
        "                    # Extract the field from a regex on the filename:\n",
        "                    transforms=[mlc.Transform(regex=\"^(.*)\\\\.jsonl$\")],\n",
        "                ),\n",
        "            ),\n",
        "        ],\n",
        "    )\n",
        "]\n",
        "\n",
        "# Metadata contains information about the dataset.\n",
        "metadata = mlc.Metadata(\n",
        "    name=\"gpt-3\",\n",
        "    # Descriptions can contain plain text or markdown.\n",
        "    description=(\n",
        "        \"Recent work has demonstrated substantial gains on many NLP tasks and\"\n",
        "        \" benchmarks by pre-training on a large corpus of text followed by\"\n",
        "        \" fine-tuning on a specific task. While typically task-agnostic in\"\n",
        "        \" architecture, this method still requires task-specific fine-tuning\"\n",
        "        \" datasets of thousands or tens of thousands of examples. By contrast,\"\n",
        "        \" humans can generally perform a new language task from only a few\"\n",
        "        \" examples or from simple instructions \\u2013 something which current\"\n",
        "        \" NLP systems still largely struggle to do. Here we show that scaling\"\n",
        "        \" up language models greatly improves task-agnostic, few-shot\"\n",
        "        \" performance, sometimes even reaching competitiveness with prior\"\n",
        "        \" state-of-the-art fine-tuning approaches. Specifically, we train\"\n",
        "        \" GPT-3, an autoregressive language model with 175 billion parameters,\"\n",
        "        \" 10x more than any previous non-sparse language model, and test its\"\n",
        "        \" performance in the few-shot setting. For all tasks, GPT-3 is applied\"\n",
        "        \" without any gradient updates or fine-tuning, with tasks and few-shot\"\n",
        "        \" demonstrations specified purely via text interaction with the model.\"\n",
        "        \" GPT-3 achieves strong performance on many NLP datasets, including\"\n",
        "        \" translation, question-answering, and cloze tasks, as well as several\"\n",
        "        \" tasks that require on-the-fly reasoning or domain adaptation, such as\"\n",
        "        \" unscrambling words, using a novel word in a sentence, or performing\"\n",
        "        \" 3-digit arithmetic. At the same time, we also identify some datasets\"\n",
        "        \" where GPT-3's few-shot learning still struggles, as well as some\"\n",
        "        \" datasets where GPT-3 faces methodological issues related to training\"\n",
        "        \" on large web corpora. Finally, we find that GPT-3 can generate\"\n",
        "        \" samples of news articles which human evaluators have difficulty\"\n",
        "        \" distinguishing from articles written by humans. We discuss broader\"\n",
        "        \" societal impacts of this finding and of GPT-3 in general.\"\n",
        "    ),\n",
        "    cite_as=(\n",
        "        \"@article{brown2020language, title={Language Models are Few-Shot\"\n",
        "        \" Learners}, author={Tom B. Brown and Benjamin Mann and Nick Ryder and\"\n",
        "        \" Melanie Subbiah and Jared Kaplan and Prafulla Dhariwal and Arvind\"\n",
        "        \" Neelakantan and Pranav Shyam and Girish Sastry and Amanda Askell and\"\n",
        "        \" Sandhini Agarwal and Ariel Herbert-Voss and Gretchen Krueger and Tom\"\n",
        "        \" Henighan and Rewon Child and Aditya Ramesh and Daniel M. Ziegler and\"\n",
        "        \" Jeffrey Wu and Clemens Winter and Christopher Hesse and Mark Chen and\"\n",
        "        \" Eric Sigler and Mateusz Litwin and Scott Gray and Benjamin Chess and\"\n",
        "        \" Jack Clark and Christopher Berner and Sam McCandlish and Alec Radford\"\n",
        "        \" and Ilya Sutskever and Dario Amodei}, year={2020},\"\n",
        "        \" eprint={2005.14165}, archivePrefix={arXiv}, primaryClass={cs.CL} }\"\n",
        "    ),\n",
        "    url=\"https://github.com/openai/gpt-3\",\n",
        "    distribution=distribution,\n",
        "    record_sets=record_sets,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2RUVgWI-DldZ"
      },
      "source": [
        "When creating `Metadata`:\n",
        "- We also check for errors in the configuration.\n",
        "- We generate warnings if the configuration doesn't follow guidelines and best practices.\n",
        "\n",
        "For instance, in this case:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AENcJUwMCd1B"
      },
      "outputs": [],
      "source": [
        "print(metadata.issues.report())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vES3KHaND4P2"
      },
      "source": [
        "`Property \"https://schema.org/license\" is recommended`...\n",
        "\n",
        "We can see at a glance that we miss an important metadata to build datasets for responsible AI: the license!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S0BEhzqiEjd0"
      },
      "source": [
        "## Build the Croissant file and yield data\n",
        "\n",
        "Let's write the Croissant JSON-LD to a file on disk!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-XCycu81ECVq"
      },
      "outputs": [],
      "source": [
        "import json\n",
        "\n",
        "with open(\"croissant.json\", \"w\") as f:\n",
        "  content = metadata.to_json()\n",
        "  content = json.dumps(content, indent=2)\n",
        "  print(content)\n",
        "  f.write(content)\n",
        "  f.write(\"\\n\")  # Terminate file with newline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ypb_ll3SE6UU"
      },
      "source": [
        "From this JSON-LD file, we can easily create a dataset..."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_JNyQFuAEiIs"
      },
      "outputs": [],
      "source": [
        "dataset = mlc.Dataset(jsonld=\"croissant.json\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ldwdIGPoFT_p"
      },
      "source": [
        "...and yield records from this dataset:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MHdVY4TBEqZ8"
      },
      "outputs": [],
      "source": [
        "records = dataset.records(record_set=\"jsonl\")\n",
        "\n",
        "for i, record in enumerate(records):\n",
        "  print(record)\n",
        "  if i > 10:\n",
        "    break"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8a2sCy0GFYCQ"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}