notebooks/data-sources/TestGrid/metrics/test_pass_failures.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c2f9c2c3",
   "metadata": {},
   "source": [
    "# Quantify test pass/failures\n",
    "\n",
    "In this notebook, the key perfomance indicators that we would like to create greater visbility into and track over time is the percent of tests that passed/failed. This can help track and measure the effectiveness and success of our testing process.\n",
    "\n",
    "By measuring this metric, we can observe the trend of test failures over time and a decrease in the percent of failures over time (or increase in percent of test passes), should correlate to improved test efficiencies, enhanced testing process and error free releases.  In this notebook, we derive the following metrics from the TestGrid dataset:\n",
    "\n",
    "* total number of test cases\n",
    "* number of test cases passed\n",
    "* number of test cases failed\n",
    "* percent of tests that pass\n",
    "* percent of tests that fail\n",
    "\n",
    "_Linked Issue: [issue 1](https://github.com/aicoe-aiops/ocp-ci-analysis/issues/114)_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9802577e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T15:41:55.449933Z",
     "start_time": "2021-04-01T15:41:54.534127Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Import libraries\n",
    "import gzip\n",
    "import json\n",
    "import os\n",
    "import pandas as pd\n",
    "import datetime\n",
    "\n",
    "from ipynb.fs.defs.metric_template import (\n",
    "    testgrid_labelwise_encoding,\n",
    "    CephCommunication,\n",
    "    save_to_disk,\n",
    "    read_from_disk,\n",
    ")\n",
    "\n",
    "from dotenv import load_dotenv, find_dotenv\n",
    "\n",
    "load_dotenv(find_dotenv())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "aced4d13",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T15:41:55.455115Z",
     "start_time": "2021-04-01T15:41:55.451878Z"
    }
   },
   "outputs": [],
   "source": [
    "## Specify variables\n",
    "\n",
    "METRIC_NAME = \"test_pass_failures\"\n",
    "\n",
    "# Specify the path for input grid data\n",
    "INPUT_DATA_PATH = \"../../../../data/raw/testgrid_183.json.gz\"\n",
    "\n",
    "# Specify the path for output metric data\n",
    "OUTPUT_DATA_PATH = f\"../../../../data/processed/metrics/{METRIC_NAME}\"\n",
    "\n",
    "## CEPH Bucket variables\n",
    "## Create a .env file on your local with the correct configs\n",
    "s3_endpoint_url = os.getenv(\"S3_ENDPOINT\")\n",
    "s3_access_key = os.getenv(\"S3_ACCESS_KEY\")\n",
    "s3_secret_key = os.getenv(\"S3_SECRET_KEY\")\n",
    "s3_bucket = os.getenv(\"S3_BUCKET\")\n",
    "s3_path = os.getenv(\"S3_PROJECT_KEY\", \"ai4ci/testgrid/metrics\")\n",
    "s3_input_data_path = \"raw_data\"\n",
    "\n",
    "# Specify whether or not we are running this as a notebook or part of an automation pipeline.\n",
    "AUTOMATION = os.getenv(\"IN_AUTOMATION\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "0ec5f72c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T15:42:09.552625Z",
     "start_time": "2021-04-01T15:41:55.457211Z"
    }
   },
   "outputs": [],
   "source": [
    "## Import data\n",
    "timestamp = datetime.datetime.today()\n",
    "\n",
    "if AUTOMATION:\n",
    "    filename = f\"testgrid_{timestamp.day}{timestamp.month}.json\"\n",
    "    cc = CephCommunication(s3_endpoint_url, s3_access_key, s3_secret_key, s3_bucket)\n",
    "    s3_object = cc.s3_resource.Object(s3_bucket, f\"{s3_input_data_path}/{filename}\")\n",
    "    file_content = s3_object.get()[\"Body\"].read().decode(\"utf-8\")\n",
    "    testgrid_data = json.loads(file_content)\n",
    "\n",
    "else:\n",
    "    with gzip.open(INPUT_DATA_PATH, \"rb\") as read_file:\n",
    "        testgrid_data = json.load(read_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2afe72d",
   "metadata": {},
   "source": [
    "## Metric Calculation\n",
    "We find all the tests which are failing i.e. have a status code of 12"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "781dbcdb",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T15:42:33.307447Z",
     "start_time": "2021-04-01T15:42:09.554876Z"
    }
   },
   "outputs": [],
   "source": [
    "failures_list = testgrid_labelwise_encoding(testgrid_data, 12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4bbc41ee",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T15:42:33.314076Z",
     "start_time": "2021-04-01T15:42:33.309915Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "27192485"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(failures_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "420d20ea",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T15:42:46.458026Z",
     "start_time": "2021-04-01T15:42:33.331028Z"
    }
   },
   "outputs": [],
   "source": [
    "# Convert to dataframe\n",
    "failures_df = pd.DataFrame(\n",
    "    failures_list,\n",
    "    columns=[\"timestamp\", \"tab\", \"grid\", \"test\", \"test_duration\", \"failure\"],\n",
    ")\n",
    "failures_df = failures_df.drop(columns=\"test_duration\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "6779d9d5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>tab</th>\n",
       "      <th>grid</th>\n",
       "      <th>test</th>\n",
       "      <th>failure</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2021-03-15 23:40:20</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2021-03-15 00:01:06</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2021-03-13 20:51:32</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2021-03-13 07:51:20</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2021-03-13 06:43:20</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            timestamp                          tab  \\\n",
       "0 2021-03-15 23:40:20  \"redhat-assisted-installer\"   \n",
       "1 2021-03-15 00:01:06  \"redhat-assisted-installer\"   \n",
       "2 2021-03-13 20:51:32  \"redhat-assisted-installer\"   \n",
       "3 2021-03-13 07:51:20  \"redhat-assisted-installer\"   \n",
       "4 2021-03-13 06:43:20  \"redhat-assisted-installer\"   \n",
       "\n",
       "                                                grid     test  failure  \n",
       "0  periodic-ci-openshift-release-master-nightly-4...  Overall    False  \n",
       "1  periodic-ci-openshift-release-master-nightly-4...  Overall    False  \n",
       "2  periodic-ci-openshift-release-master-nightly-4...  Overall    False  \n",
       "3  periodic-ci-openshift-release-master-nightly-4...  Overall    False  \n",
       "4  periodic-ci-openshift-release-master-nightly-4...  Overall    False  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "failures_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d03123b",
   "metadata": {},
   "source": [
    "We now find all the tests which are passing i.e. have a status code of 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "46333ea3",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T15:43:09.974015Z",
     "start_time": "2021-04-01T15:42:46.460727Z"
    }
   },
   "outputs": [],
   "source": [
    "passing_list = testgrid_labelwise_encoding(testgrid_data, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "1608ab18",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T15:43:09.980705Z",
     "start_time": "2021-04-01T15:43:09.976642Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "27192485"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(passing_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "4bc1517c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T15:43:21.985721Z",
     "start_time": "2021-04-01T15:43:09.982091Z"
    }
   },
   "outputs": [],
   "source": [
    "# Convert to dataframe\n",
    "passing_df = pd.DataFrame(\n",
    "    passing_list,\n",
    "    columns=[\"timestamp\", \"tab\", \"grid\", \"test\", \"test_duration\", \"passing\"],\n",
    ")\n",
    "passing_df = passing_df.drop(columns=\"test_duration\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "a1d94400",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>tab</th>\n",
       "      <th>grid</th>\n",
       "      <th>test</th>\n",
       "      <th>passing</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2021-03-15 23:40:20</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2021-03-15 00:01:06</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2021-03-13 20:51:32</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2021-03-13 07:51:20</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2021-03-13 06:43:20</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            timestamp                          tab  \\\n",
       "0 2021-03-15 23:40:20  \"redhat-assisted-installer\"   \n",
       "1 2021-03-15 00:01:06  \"redhat-assisted-installer\"   \n",
       "2 2021-03-13 20:51:32  \"redhat-assisted-installer\"   \n",
       "3 2021-03-13 07:51:20  \"redhat-assisted-installer\"   \n",
       "4 2021-03-13 06:43:20  \"redhat-assisted-installer\"   \n",
       "\n",
       "                                                grid     test  passing  \n",
       "0  periodic-ci-openshift-release-master-nightly-4...  Overall     True  \n",
       "1  periodic-ci-openshift-release-master-nightly-4...  Overall     True  \n",
       "2  periodic-ci-openshift-release-master-nightly-4...  Overall     True  \n",
       "3  periodic-ci-openshift-release-master-nightly-4...  Overall     True  \n",
       "4  periodic-ci-openshift-release-master-nightly-4...  Overall     True  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "passing_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "3b0ea648",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T15:43:48.354260Z",
     "start_time": "2021-04-01T15:43:21.987158Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>tab</th>\n",
       "      <th>grid</th>\n",
       "      <th>test</th>\n",
       "      <th>failure</th>\n",
       "      <th>passing</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2021-03-15 23:40:20</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2021-03-15 00:01:06</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2021-03-13 20:51:32</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2021-03-13 07:51:20</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2021-03-13 06:43:20</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            timestamp                          tab  \\\n",
       "0 2021-03-15 23:40:20  \"redhat-assisted-installer\"   \n",
       "1 2021-03-15 00:01:06  \"redhat-assisted-installer\"   \n",
       "2 2021-03-13 20:51:32  \"redhat-assisted-installer\"   \n",
       "3 2021-03-13 07:51:20  \"redhat-assisted-installer\"   \n",
       "4 2021-03-13 06:43:20  \"redhat-assisted-installer\"   \n",
       "\n",
       "                                                grid     test  failure  \\\n",
       "0  periodic-ci-openshift-release-master-nightly-4...  Overall    False   \n",
       "1  periodic-ci-openshift-release-master-nightly-4...  Overall    False   \n",
       "2  periodic-ci-openshift-release-master-nightly-4...  Overall    False   \n",
       "3  periodic-ci-openshift-release-master-nightly-4...  Overall    False   \n",
       "4  periodic-ci-openshift-release-master-nightly-4...  Overall    False   \n",
       "\n",
       "   passing  \n",
       "0     True  \n",
       "1     True  \n",
       "2     True  \n",
       "3     True  \n",
       "4     True  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "combined_df = pd.merge(\n",
    "    failures_df,\n",
    "    passing_df,\n",
    "    on=[\"timestamp\", \"tab\", \"grid\", \"test\"],\n",
    ")\n",
    "combined_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "712d1836",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T15:43:51.594251Z",
     "start_time": "2021-04-01T15:43:48.356023Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of tests: 27219757\n",
      "Total number of failing tests: 200541\n",
      "Test failure percentage: 0.736748\n",
      "Total number of passing tests: 14281659\n",
      "Test pass percentage: 52.467989\n"
     ]
    }
   ],
   "source": [
    "## The following implements test pass/failure percentage metrics\n",
    "## Moving forward, this will be aggregated in Superset\n",
    "## For the sake of completeness, it is implmented here\n",
    "\n",
    "no_tests = combined_df.test.count()\n",
    "print(\"Total number of tests: %i\" % (no_tests))\n",
    "no_failures = combined_df.failure.sum()\n",
    "print(\"Total number of failing tests: %i\" % (no_failures))\n",
    "test_failures_percentage = (\n",
    "    (combined_df.failure.sum() / combined_df.test.count())\n",
    ") * 100\n",
    "print(\"Test failure percentage: %f\" % (test_failures_percentage))\n",
    "no_pass = combined_df.passing.sum()\n",
    "print(\"Total number of passing tests: %i\" % (no_pass))\n",
    "test_pass_percentage = ((combined_df.passing.sum() / combined_df.passing.count())) * 100\n",
    "print(\"Test pass percentage: %f\" % (test_pass_percentage))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba9d4560",
   "metadata": {},
   "source": [
    "## Save results to Ceph or locally\n",
    "* Use the following helper function to save the data frame in a parquet format on the Ceph bucket if we are running in automation, and locally if not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c14d7e00",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T15:43:53.427951Z",
     "start_time": "2021-04-01T15:43:51.595971Z"
    }
   },
   "outputs": [],
   "source": [
    "timestamp = datetime.datetime.now()\n",
    "\n",
    "if AUTOMATION:\n",
    "    cc = CephCommunication(s3_endpoint_url, s3_access_key, s3_secret_key, s3_bucket)\n",
    "    cc.upload_to_ceph(\n",
    "        combined_df.head(1000000),\n",
    "        s3_path,\n",
    "        f\"{METRIC_NAME}/{METRIC_NAME}-{timestamp.year}-{timestamp.month}-{timestamp.day}.parquet\",\n",
    "    )\n",
    "else:\n",
    "    save_to_disk(\n",
    "        combined_df.head(1000000),\n",
    "        OUTPUT_DATA_PATH,\n",
    "        f\"{METRIC_NAME}-{timestamp.year}-{timestamp.month}-{timestamp.day}.parquet\",\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "bbce1a7f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-04-01T15:43:53.914079Z",
     "start_time": "2021-04-01T15:43:53.429787Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>tab</th>\n",
       "      <th>grid</th>\n",
       "      <th>test</th>\n",
       "      <th>failure</th>\n",
       "      <th>passing</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2021-03-15 23:40:20</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2021-03-15 00:01:06</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2021-03-13 20:51:32</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2021-03-13 07:51:20</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2021-03-13 06:43:20</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>999995</th>\n",
       "      <td>2021-03-12 11:16:51</td>\n",
       "      <td>\"redhat-openshift-informing\"</td>\n",
       "      <td>release-openshift-origin-installer-launch-gcp</td>\n",
       "      <td>openshift-tests.[sig-network] NetworkPolicy [L...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>999996</th>\n",
       "      <td>2021-03-12 11:16:36</td>\n",
       "      <td>\"redhat-openshift-informing\"</td>\n",
       "      <td>release-openshift-origin-installer-launch-gcp</td>\n",
       "      <td>openshift-tests.[sig-network] NetworkPolicy [L...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>999997</th>\n",
       "      <td>2021-03-12 11:08:17</td>\n",
       "      <td>\"redhat-openshift-informing\"</td>\n",
       "      <td>release-openshift-origin-installer-launch-gcp</td>\n",
       "      <td>openshift-tests.[sig-network] NetworkPolicy [L...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>999998</th>\n",
       "      <td>2021-03-12 10:34:23</td>\n",
       "      <td>\"redhat-openshift-informing\"</td>\n",
       "      <td>release-openshift-origin-installer-launch-gcp</td>\n",
       "      <td>openshift-tests.[sig-network] NetworkPolicy [L...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>999999</th>\n",
       "      <td>2021-03-12 10:30:18</td>\n",
       "      <td>\"redhat-openshift-informing\"</td>\n",
       "      <td>release-openshift-origin-installer-launch-gcp</td>\n",
       "      <td>openshift-tests.[sig-network] NetworkPolicy [L...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1000000 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                 timestamp                           tab  \\\n",
       "0      2021-03-15 23:40:20   \"redhat-assisted-installer\"   \n",
       "1      2021-03-15 00:01:06   \"redhat-assisted-installer\"   \n",
       "2      2021-03-13 20:51:32   \"redhat-assisted-installer\"   \n",
       "3      2021-03-13 07:51:20   \"redhat-assisted-installer\"   \n",
       "4      2021-03-13 06:43:20   \"redhat-assisted-installer\"   \n",
       "...                    ...                           ...   \n",
       "999995 2021-03-12 11:16:51  \"redhat-openshift-informing\"   \n",
       "999996 2021-03-12 11:16:36  \"redhat-openshift-informing\"   \n",
       "999997 2021-03-12 11:08:17  \"redhat-openshift-informing\"   \n",
       "999998 2021-03-12 10:34:23  \"redhat-openshift-informing\"   \n",
       "999999 2021-03-12 10:30:18  \"redhat-openshift-informing\"   \n",
       "\n",
       "                                                     grid  \\\n",
       "0       periodic-ci-openshift-release-master-nightly-4...   \n",
       "1       periodic-ci-openshift-release-master-nightly-4...   \n",
       "2       periodic-ci-openshift-release-master-nightly-4...   \n",
       "3       periodic-ci-openshift-release-master-nightly-4...   \n",
       "4       periodic-ci-openshift-release-master-nightly-4...   \n",
       "...                                                   ...   \n",
       "999995      release-openshift-origin-installer-launch-gcp   \n",
       "999996      release-openshift-origin-installer-launch-gcp   \n",
       "999997      release-openshift-origin-installer-launch-gcp   \n",
       "999998      release-openshift-origin-installer-launch-gcp   \n",
       "999999      release-openshift-origin-installer-launch-gcp   \n",
       "\n",
       "                                                     test  failure  passing  \n",
       "0                                                 Overall    False     True  \n",
       "1                                                 Overall    False     True  \n",
       "2                                                 Overall    False     True  \n",
       "3                                                 Overall    False     True  \n",
       "4                                                 Overall    False     True  \n",
       "...                                                   ...      ...      ...  \n",
       "999995  openshift-tests.[sig-network] NetworkPolicy [L...    False    False  \n",
       "999996  openshift-tests.[sig-network] NetworkPolicy [L...    False    False  \n",
       "999997  openshift-tests.[sig-network] NetworkPolicy [L...    False    False  \n",
       "999998  openshift-tests.[sig-network] NetworkPolicy [L...    False    False  \n",
       "999999  openshift-tests.[sig-network] NetworkPolicy [L...    False    False  \n",
       "\n",
       "[1000000 rows x 6 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Sanity check to see if the dataset is the same\n",
    "if AUTOMATION:\n",
    "    sanity_check = cc.read_from_ceph(\n",
    "        s3_path,\n",
    "        f\"{METRIC_NAME}/{METRIC_NAME}-{timestamp.year}-{timestamp.month}-{timestamp.day}.parquet\",\n",
    "    )\n",
    "else:\n",
    "    sanity_check = read_from_disk(\n",
    "        OUTPUT_DATA_PATH,\n",
    "        f\"{METRIC_NAME}-{timestamp.year}-{timestamp.month}-{timestamp.day}.parquet\",\n",
    "    )\n",
    "\n",
    "sanity_check"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0935505",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "This notebook computed the number of test passes, test failures and test pass/failure percentage metric. The dataframe saved on ceph can be used to generate aggregated views and visualizations."
   ]
  }
 ],
 "metadata": {
  "finalized": {
   "timestamp": 1612822647266,
   "trusted": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}