notebooks/data-sources/TestGrid/metrics/correlated_failures.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Correlated test failure sets per test and average size of correlation sets\n",
    "\n",
    "This notebook outputs 2 artifacts: \n",
    "\n",
    "1. A parquet file that provides, for a given test, all of the other tests that are highly correlated (correlation coefficient of 0.9 or above). This file omits any tests that do not have any highly correlated tests. So, if a test is not present on the list, then it has no highly correlated tests associated with it at this time and has been removed from the record. The calculation for correlation is performed on all available data exposed by the Red Hat test grid instance at the time the notebook is run.\n",
    "\n",
    "2. A summary metric that can be easily tracked over time that represents the average size of correlated test sets in the above parquet. \n",
    "\n",
    "\n",
    "__Note__: This notebook follows a very similar approach to an earlier [EDA notebook](https://github.com/aicoe-aiops/ocp-ci-analysis/blob/master/notebooks/data-sources/Sippy/sippy_failure_correlation.ipynb) where we correlated failures with a different dataset. For simplicity, much of the reasoning behind the decisions made in this notebook have been omited here, but can be found in the above linked notebook for interested readers :)   \n",
    "\n",
    "\n",
    "[related  issue #139](https://github.com/aicoe-aiops/ocp-ci-analysis/issues/139)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import libraries\n",
    "import gzip\n",
    "import json\n",
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import datetime\n",
    "\n",
    "from ipynb.fs.defs.metric_template import decode_run_length\n",
    "from ipynb.fs.defs.metric_template import CephCommunication\n",
    "from ipynb.fs.defs.metric_template import save_to_disk, read_from_disk\n",
    "from dotenv import load_dotenv, find_dotenv\n",
    "\n",
    "load_dotenv(find_dotenv())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Specify variables\n",
    "\n",
    "METRIC_NAME = \"correlation\"\n",
    "\n",
    "# Specify the path for input grid data,\n",
    "INPUT_DATA_PATH = \"../../../../data/raw/testgrid_183.json.gz\"\n",
    "\n",
    "# Specify the path for output metric data\n",
    "OUTPUT_DATA_PATH = f\"../../../../data/processed/metrics/{METRIC_NAME}\"\n",
    "\n",
    "## CEPH Bucket variables\n",
    "## Create a .env file on your local with the correct configs,\n",
    "s3_endpoint_url = os.getenv(\"S3_ENDPOINT\")\n",
    "s3_access_key = os.getenv(\"S3_ACCESS_KEY\")\n",
    "s3_secret_key = os.getenv(\"S3_SECRET_KEY\")\n",
    "s3_bucket = os.getenv(\"S3_BUCKET\")\n",
    "s3_path = os.getenv(\"S3_PROJECT_KEY\", \"ai4ci/testgrid/metrics\")\n",
    "s3_input_data_path = \"raw_data\"\n",
    "AUTOMATION = os.getenv(\"IN_AUTOMATION\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Import data\n",
    "timestamp = datetime.datetime.today()\n",
    "\n",
    "if AUTOMATION:\n",
    "    filename = f\"testgrid_{timestamp.day}{timestamp.month}.json\"\n",
    "    cc = CephCommunication(s3_endpoint_url, s3_access_key, s3_secret_key, s3_bucket)\n",
    "    s3_object = cc.s3_resource.Object(s3_bucket, f\"{s3_input_data_path}/{filename}\")\n",
    "    file_content = s3_object.get()[\"Body\"].read().decode(\"utf-8\")\n",
    "    testgrid_data = json.loads(file_content)\n",
    "\n",
    "else:\n",
    "    with gzip.open(INPUT_DATA_PATH, \"rb\") as read_file:\n",
    "        testgrid_data = json.load(read_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Calculation \n",
    "\n",
    "Here we iterate through each grid in our dataset and collect the the names of all the tests that fail during the same build. We will store this in the `failure_groups` list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "failure_groups = []\n",
    "\n",
    "for tab in list(testgrid_data.keys()):\n",
    "    for grid in testgrid_data[tab].keys():\n",
    "        current_grid = testgrid_data[tab][grid]\n",
    "\n",
    "        tests = [\n",
    "            current_grid[\"grid\"][i][\"name\"] for i in range(len(current_grid[\"grid\"]))\n",
    "        ]\n",
    "        # unroll the run-length encoding and set bool for flake or not (x==13)\n",
    "        decoded = [\n",
    "            (\n",
    "                np.array(decode_run_length(current_grid[\"grid\"][i][\"statuses\"])) == 12\n",
    "            ).tolist()\n",
    "            for i in range(len(current_grid[\"grid\"]))\n",
    "        ]\n",
    "\n",
    "        matrix = pd.DataFrame(zip(tests, decoded), columns=[\"test\", \"values\"])\n",
    "        matrix = pd.DataFrame(matrix[\"values\"].to_list(), index=matrix[\"test\"])\n",
    "\n",
    "        for c, items in matrix.iteritems():\n",
    "            if len(items[items].index) > 1:\n",
    "                failure_groups.append(items[items].index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "failure_groups = pd.Series(failure_groups)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20132"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(failure_groups)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we want to define a vocabulary for all of the unique tests in our dataset so that we can encode our failure sets using a binary encoding scheme."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "194477\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "8935"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab = set()\n",
    "count = 0\n",
    "for fg in failure_groups:\n",
    "    count += len(fg)\n",
    "    vocab.update(fg)\n",
    "\n",
    "vocab = list(vocab)\n",
    "print(count)\n",
    "len(vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Confirm that there are no duplicates in the vocab to ensure we have a unique set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(pd.Series(vocab).unique()) == len(vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll use the below function to create our binary encoded vectors for our correlation analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def encode_tests(job):\n",
    "    encoded = []\n",
    "    for v in vocab:\n",
    "        if v in job:\n",
    "            encoded.extend([1])\n",
    "        else:\n",
    "            encoded.extend([0])\n",
    "    return encoded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoded = failure_groups.apply(encode_tests)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...\n",
       "1    [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...\n",
       "2    [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...\n",
       "3    [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...\n",
       "4    [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...\n",
       "dtype: object"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoded.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>openshift-tests.[sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client [Top Level] [sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client should enforce policy based on NamespaceSelector with MatchExpressions[Feature:NetworkPolicy] [Skipped:Network/OpenShiftSDN/Multitenant] [Suite:openshift/conformance/parallel] [Suite:k8s] [5]</th>\n",
       "      <th>openshift-tests.[sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: dir] [Testpattern: Dynamic PV (ntfs)][sig-windows] subPath should be able to unmount after the subpath directory is deleted [Suite:openshift/conformance/parallel] [Suite:k8s]</th>\n",
       "      <th>openshift-tests.[sig-cli] Kubectl client Simple pod [Top Level] [sig-cli] Kubectl client Simple pod should support exec [Suite:openshift/conformance/parallel] [Suite:k8s]</th>\n",
       "      <th>openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/parallel] [12]</th>\n",
       "      <th>openshift-tests.[sig-network] DNS should provide DNS for services  [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s] [5]</th>\n",
       "      <th>operator.Run multi-stage test e2e-gcp-ovn-upgrade - e2e-gcp-ovn-upgrade-ipi-deprovision-deprovision container test</th>\n",
       "      <th>openshift-tests.[Conformance][Area:Networking][Feature:Router] The HAProxy router should respond with 503 to unrecognized hosts [Suite:openshift/conformance/parallel/minimal]</th>\n",
       "      <th>openshift-tests.[sig-auth][Feature:OpenShiftAuthorization] The default cluster RBAC policy should have correct RBAC rules [Suite:openshift/conformance/parallel] [1]</th>\n",
       "      <th>openshift-tests.[sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: tmpfs] [Testpattern: Dynamic PV (ext3)] volumes [Top Level] [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: tmpfs] [Testpattern: Dynamic PV (ext3)] volumes should allow exec of files on the volume [Suite:openshift/conformance/parallel] [Suite:k8s]</th>\n",
       "      <th>openshift-tests.[sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client [Top Level] [sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client should enforce multiple ingress policies with ingress allow-all policy taking precedence [Feature:NetworkPolicy] [Skipped:Network/OpenShiftSDN/Multitenant] [Suite:openshift/conformance/parallel] [Suite:k8s] [14]</th>\n",
       "      <th>...</th>\n",
       "      <th>openshift-tests.[sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: blockfs] [Testpattern: Dynamic PV (default fs)] provisioning [Top Level] [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: blockfs] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with mount options [Suite:openshift/conformance/parallel] [Suite:k8s]</th>\n",
       "      <th>operator.Run multi-stage test e2e-gcp-ovn-upgrade - e2e-gcp-ovn-upgrade-gather-network container test</th>\n",
       "      <th>openshift-tests.[sig-storage] In-tree Volumes [Driver: hostPath] [Testpattern: Inline-volume (ext3)] volumes [Top Level] [sig-storage] In-tree Volumes [Driver: hostPath] [Testpattern: Inline-volume (ext3)] volumes should store data [Suite:openshift/conformance/parallel] [Suite:k8s]</th>\n",
       "      <th>openshift-tests.[sig-network] Services should create endpoints for unready pods [Suite:openshift/conformance/parallel] [Suite:k8s] [13]</th>\n",
       "      <th>openshift-tests.[sig-devex][Feature:ImageEcosystem][Slow] openshift images should be SCL enabled  using the SCL in s2i images \"registry.redhat.io/rhscl/python-36-rhel7\" should be SCL enabled</th>\n",
       "      <th>openshift-tests.[sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client should allow ingress access from updated namespace [Feature:NetworkPolicy] [Skipped:Network/OpenShiftSDN/Multitenant] [Suite:openshift/conformance/parallel] [Suite:k8s] [7]</th>\n",
       "      <th>openshift-tests.[sig-apps] CronJob should replace jobs when ReplaceConcurrent [Suite:openshift/conformance/parallel] [Suite:k8s]</th>\n",
       "      <th>openshift-tests.[k8s.io] Variable Expansion should allow substituting values in a container's command [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]</th>\n",
       "      <th>openshift-tests.[Feature:ProjectAPI]  TestProjectWatchWithSelectionPredicate [Top Level] [Feature:ProjectAPI]  TestProjectWatchWithSelectionPredicate should succeed [Suite:openshift/conformance/parallel]</th>\n",
       "      <th>openshift-tests.[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (ntfs)][sig-windows] subPath should be able to unmount after the subpath directory is deleted [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s]</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 8935 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   openshift-tests.[sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client [Top Level] [sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client should enforce policy based on NamespaceSelector with MatchExpressions[Feature:NetworkPolicy] [Skipped:Network/OpenShiftSDN/Multitenant] [Suite:openshift/conformance/parallel] [Suite:k8s] [5]  \\\n",
       "0                                                  0                                                                                                                                                                                                                                                                                                                                             \n",
       "1                                                  0                                                                                                                                                                                                                                                                                                                                             \n",
       "2                                                  0                                                                                                                                                                                                                                                                                                                                             \n",
       "3                                                  0                                                                                                                                                                                                                                                                                                                                             \n",
       "4                                                  0                                                                                                                                                                                                                                                                                                                                             \n",
       "\n",
       "   openshift-tests.[sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: dir] [Testpattern: Dynamic PV (ntfs)][sig-windows] subPath should be able to unmount after the subpath directory is deleted [Suite:openshift/conformance/parallel] [Suite:k8s]  \\\n",
       "0                                                  0                                                                                                                                                                                                               \n",
       "1                                                  0                                                                                                                                                                                                               \n",
       "2                                                  0                                                                                                                                                                                                               \n",
       "3                                                  0                                                                                                                                                                                                               \n",
       "4                                                  0                                                                                                                                                                                                               \n",
       "\n",
       "   openshift-tests.[sig-cli] Kubectl client Simple pod [Top Level] [sig-cli] Kubectl client Simple pod should support exec [Suite:openshift/conformance/parallel] [Suite:k8s]  \\\n",
       "0                                                  0                                                                                                                            \n",
       "1                                                  0                                                                                                                            \n",
       "2                                                  0                                                                                                                            \n",
       "3                                                  0                                                                                                                            \n",
       "4                                                  0                                                                                                                            \n",
       "\n",
       "   openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/parallel] [12]  \\\n",
       "0                                                  0                                                                                                                 \n",
       "1                                                  0                                                                                                                 \n",
       "2                                                  0                                                                                                                 \n",
       "3                                                  0                                                                                                                 \n",
       "4                                                  0                                                                                                                 \n",
       "\n",
       "   openshift-tests.[sig-network] DNS should provide DNS for services  [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s] [5]  \\\n",
       "0                                                  0                                                                                                 \n",
       "1                                                  0                                                                                                 \n",
       "2                                                  0                                                                                                 \n",
       "3                                                  0                                                                                                 \n",
       "4                                                  0                                                                                                 \n",
       "\n",
       "   operator.Run multi-stage test e2e-gcp-ovn-upgrade - e2e-gcp-ovn-upgrade-ipi-deprovision-deprovision container test  \\\n",
       "0                                                  0                                                                    \n",
       "1                                                  0                                                                    \n",
       "2                                                  0                                                                    \n",
       "3                                                  0                                                                    \n",
       "4                                                  0                                                                    \n",
       "\n",
       "   openshift-tests.[Conformance][Area:Networking][Feature:Router] The HAProxy router should respond with 503 to unrecognized hosts [Suite:openshift/conformance/parallel/minimal]  \\\n",
       "0                                                  0                                                                                                                                \n",
       "1                                                  0                                                                                                                                \n",
       "2                                                  0                                                                                                                                \n",
       "3                                                  0                                                                                                                                \n",
       "4                                                  0                                                                                                                                \n",
       "\n",
       "   openshift-tests.[sig-auth][Feature:OpenShiftAuthorization] The default cluster RBAC policy should have correct RBAC rules [Suite:openshift/conformance/parallel] [1]  \\\n",
       "0                                                  0                                                                                                                      \n",
       "1                                                  0                                                                                                                      \n",
       "2                                                  0                                                                                                                      \n",
       "3                                                  0                                                                                                                      \n",
       "4                                                  0                                                                                                                      \n",
       "\n",
       "   openshift-tests.[sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: tmpfs] [Testpattern: Dynamic PV (ext3)] volumes [Top Level] [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: tmpfs] [Testpattern: Dynamic PV (ext3)] volumes should allow exec of files on the volume [Suite:openshift/conformance/parallel] [Suite:k8s]  \\\n",
       "0                                                  0                                                                                                                                                                                                                                                                                                       \n",
       "1                                                  0                                                                                                                                                                                                                                                                                                       \n",
       "2                                                  0                                                                                                                                                                                                                                                                                                       \n",
       "3                                                  0                                                                                                                                                                                                                                                                                                       \n",
       "4                                                  0                                                                                                                                                                                                                                                                                                       \n",
       "\n",
       "   openshift-tests.[sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client [Top Level] [sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client should enforce multiple ingress policies with ingress allow-all policy taking precedence [Feature:NetworkPolicy] [Skipped:Network/OpenShiftSDN/Multitenant] [Suite:openshift/conformance/parallel] [Suite:k8s] [14]  \\\n",
       "0                                                  0                                                                                                                                                                                                                                                                                                                                                                 \n",
       "1                                                  0                                                                                                                                                                                                                                                                                                                                                                 \n",
       "2                                                  0                                                                                                                                                                                                                                                                                                                                                                 \n",
       "3                                                  0                                                                                                                                                                                                                                                                                                                                                                 \n",
       "4                                                  0                                                                                                                                                                                                                                                                                                                                                                 \n",
       "\n",
       "   ...  \\\n",
       "0  ...   \n",
       "1  ...   \n",
       "2  ...   \n",
       "3  ...   \n",
       "4  ...   \n",
       "\n",
       "   openshift-tests.[sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: blockfs] [Testpattern: Dynamic PV (default fs)] provisioning [Top Level] [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: blockfs] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with mount options [Suite:openshift/conformance/parallel] [Suite:k8s]  \\\n",
       "0                                                  0                                                                                                                                                                                                                                                                                                                                    \n",
       "1                                                  0                                                                                                                                                                                                                                                                                                                                    \n",
       "2                                                  0                                                                                                                                                                                                                                                                                                                                    \n",
       "3                                                  0                                                                                                                                                                                                                                                                                                                                    \n",
       "4                                                  0                                                                                                                                                                                                                                                                                                                                    \n",
       "\n",
       "   operator.Run multi-stage test e2e-gcp-ovn-upgrade - e2e-gcp-ovn-upgrade-gather-network container test  \\\n",
       "0                                                  0                                                       \n",
       "1                                                  0                                                       \n",
       "2                                                  0                                                       \n",
       "3                                                  0                                                       \n",
       "4                                                  0                                                       \n",
       "\n",
       "   openshift-tests.[sig-storage] In-tree Volumes [Driver: hostPath] [Testpattern: Inline-volume (ext3)] volumes [Top Level] [sig-storage] In-tree Volumes [Driver: hostPath] [Testpattern: Inline-volume (ext3)] volumes should store data [Suite:openshift/conformance/parallel] [Suite:k8s]  \\\n",
       "0                                                  0                                                                                                                                                                                                                                            \n",
       "1                                                  0                                                                                                                                                                                                                                            \n",
       "2                                                  0                                                                                                                                                                                                                                            \n",
       "3                                                  0                                                                                                                                                                                                                                            \n",
       "4                                                  0                                                                                                                                                                                                                                            \n",
       "\n",
       "   openshift-tests.[sig-network] Services should create endpoints for unready pods [Suite:openshift/conformance/parallel] [Suite:k8s] [13]  \\\n",
       "0                                                  0                                                                                         \n",
       "1                                                  0                                                                                         \n",
       "2                                                  0                                                                                         \n",
       "3                                                  0                                                                                         \n",
       "4                                                  0                                                                                         \n",
       "\n",
       "   openshift-tests.[sig-devex][Feature:ImageEcosystem][Slow] openshift images should be SCL enabled  using the SCL in s2i images \"registry.redhat.io/rhscl/python-36-rhel7\" should be SCL enabled  \\\n",
       "0                                                  0                                                                                                                                                \n",
       "1                                                  0                                                                                                                                                \n",
       "2                                                  0                                                                                                                                                \n",
       "3                                                  0                                                                                                                                                \n",
       "4                                                  0                                                                                                                                                \n",
       "\n",
       "   openshift-tests.[sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client should allow ingress access from updated namespace [Feature:NetworkPolicy] [Skipped:Network/OpenShiftSDN/Multitenant] [Suite:openshift/conformance/parallel] [Suite:k8s] [7]  \\\n",
       "0                                                  0                                                                                                                                                                                                                              \n",
       "1                                                  0                                                                                                                                                                                                                              \n",
       "2                                                  0                                                                                                                                                                                                                              \n",
       "3                                                  0                                                                                                                                                                                                                              \n",
       "4                                                  0                                                                                                                                                                                                                              \n",
       "\n",
       "   openshift-tests.[sig-apps] CronJob should replace jobs when ReplaceConcurrent [Suite:openshift/conformance/parallel] [Suite:k8s]  \\\n",
       "0                                                  0                                                                                  \n",
       "1                                                  0                                                                                  \n",
       "2                                                  0                                                                                  \n",
       "3                                                  0                                                                                  \n",
       "4                                                  0                                                                                  \n",
       "\n",
       "   openshift-tests.[k8s.io] Variable Expansion should allow substituting values in a container's command [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]  \\\n",
       "0                                                  0                                                                                                                                                  \n",
       "1                                                  0                                                                                                                                                  \n",
       "2                                                  0                                                                                                                                                  \n",
       "3                                                  0                                                                                                                                                  \n",
       "4                                                  0                                                                                                                                                  \n",
       "\n",
       "   openshift-tests.[Feature:ProjectAPI]  TestProjectWatchWithSelectionPredicate [Top Level] [Feature:ProjectAPI]  TestProjectWatchWithSelectionPredicate should succeed [Suite:openshift/conformance/parallel]  \\\n",
       "0                                                  0                                                                                                                                                             \n",
       "1                                                  0                                                                                                                                                             \n",
       "2                                                  0                                                                                                                                                             \n",
       "3                                                  0                                                                                                                                                             \n",
       "4                                                  0                                                                                                                                                             \n",
       "\n",
       "   openshift-tests.[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (ntfs)][sig-windows] subPath should be able to unmount after the subpath directory is deleted [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s]  \n",
       "0                                                  0                                                                                                                                                                                                       \n",
       "1                                                  0                                                                                                                                                                                                       \n",
       "2                                                  0                                                                                                                                                                                                       \n",
       "3                                                  0                                                                                                                                                                                                       \n",
       "4                                                  0                                                                                                                                                                                                       \n",
       "\n",
       "[5 rows x 8935 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_encoded = pd.DataFrame(encoded.array, columns=vocab)\n",
    "df_encoded.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Overall                                                                                   0.777369\n",
       "openshift-tests.Monitor cluster while tests execute                                       0.163272\n",
       "openshift-tests.[sig-network] pods should successfully create sandboxes by getting pod    0.159944\n",
       "dtype: float64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# percent that each test is present in the data; percent failure\n",
    "perc_present = df_encoded.sum() / len(df_encoded)\n",
    "perc_present.sort_values(ascending=False).head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Overall                                                                                   15650\n",
       "openshift-tests.Monitor cluster while tests execute                                        3287\n",
       "openshift-tests.[sig-network] pods should successfully create sandboxes by getting pod     3220\n",
       "dtype: int64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Total failure count present in the data; failure per test\n",
    "occurrence_count = df_encoded.sum()\n",
    "occurrence_count.sort_values(ascending=False).head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also want to make sure that our correlation values are not just due to unique failed test sets present in our dataset. We want to make sure our tests impact multiple jobs. For example, if we had a unique failed test set that only occurred in a single example, and shared no other failed tests among the vocabulary, then all of the tests would appear to be 100% correlated with each other, when in fact that is merely a consequence of insufficient data. In order to prevent that, we will ignore any tests that occur only in a single job. In order to do that we will use occurrence_count to create a filter vector for any test that occurs only once. Then filter them out of our working DF."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter_unique = list(occurrence_count[occurrence_count.values <= 1].index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_encoded = df_encoded.drop(filter_unique, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(20132, 7330)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_encoded.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this takes time with full dataset - ~ 2 hours may need to use different approach\n",
    "# todo try with dask\n",
    "corr_matrix = df_encoded.corr()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7330"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# For each feature, find the other features that are correlated by more than 0.9\n",
    "top_correlation = {}\n",
    "\n",
    "for c in corr_matrix.columns:\n",
    "    top_correlation[c] = []\n",
    "    series = corr_matrix.loc[c]\n",
    "\n",
    "    for i, s in enumerate(series):\n",
    "        if s > 0.90 and series.index[i] != c:\n",
    "            top_correlation[c].append((series.index[i], s))\n",
    "\n",
    "len(top_correlation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Examine example output\n",
    "\n",
    "Let's go ahead and take a look at which tests are highly correlated with the first test in our results list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3239 sets of correlated tests \n",
      "\n",
      "Feature of interest: openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/parallel] [12]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>test_name</th>\n",
       "      <th>correlation coefficient</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...</td>\n",
       "      <td>0.975876</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                test_name  \\\n",
       "0   openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...   \n",
       "1   openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...   \n",
       "2   openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...   \n",
       "3   openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...   \n",
       "4   openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...   \n",
       "5   openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...   \n",
       "6   openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...   \n",
       "7   openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...   \n",
       "8   openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...   \n",
       "9   openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...   \n",
       "10  openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...   \n",
       "11  openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...   \n",
       "12  openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...   \n",
       "\n",
       "    correlation coefficient  \n",
       "0                  1.000000  \n",
       "1                  1.000000  \n",
       "2                  1.000000  \n",
       "3                  1.000000  \n",
       "4                  1.000000  \n",
       "5                  1.000000  \n",
       "6                  1.000000  \n",
       "7                  1.000000  \n",
       "8                  1.000000  \n",
       "9                  1.000000  \n",
       "10                 1.000000  \n",
       "11                 0.975876  \n",
       "12                 1.000000  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# top_correlation has a number of empty rows as not all tests have high correlations with others,\n",
    "# lets grab only the sets that have at least 1 highly correlated test\n",
    "\n",
    "pd.set_option(\"display.max_colwidth\", 150)\n",
    "corr_sets = []\n",
    "for i in top_correlation.items():\n",
    "    if len(i[1]) >= 1:\n",
    "        corr_sets.append(i)\n",
    "print(f\"{len(corr_sets)} sets of correlated tests \\n\")\n",
    "print(f\"Feature of interest: {corr_sets[1][0]}\")\n",
    "pd.DataFrame(corr_sets[1][1], columns=[\"test_name\", \"correlation coefficient\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5 : the number of times this test failed in our data set\n"
     ]
    }
   ],
   "source": [
    "if not AUTOMATION:\n",
    "    test_name = \"openshift-tests.[k8s.io] Security Context When creating a container with runAsUser should run the container with uid 65534 [LinuxOnly] [NodeConformance] [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]\"  # noqa\n",
    "    num = occurrence_count.loc[test_name]\n",
    "    print(f\"{num} : the number of times this test failed in our data set\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>test_name</th>\n",
       "      <th>num_occurrences</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                test_name  \\\n",
       "0   openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...   \n",
       "1   openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...   \n",
       "2   openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...   \n",
       "3   openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...   \n",
       "4   openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...   \n",
       "5   openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...   \n",
       "6   openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...   \n",
       "7   openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...   \n",
       "8   openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...   \n",
       "9   openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...   \n",
       "10  openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...   \n",
       "11  openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...   \n",
       "12  openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...   \n",
       "\n",
       "    num_occurrences  \n",
       "0                20  \n",
       "1                20  \n",
       "2                20  \n",
       "3                20  \n",
       "4                20  \n",
       "5                20  \n",
       "6                20  \n",
       "7                20  \n",
       "8                20  \n",
       "9                20  \n",
       "10               20  \n",
       "11               21  \n",
       "12               20  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lst = []\n",
    "focus = corr_sets[1][1]\n",
    "for j in focus:\n",
    "    lst.append((j[0], occurrence_count.loc[j[0]]))\n",
    "\n",
    "pd.DataFrame(lst, columns=[\"test_name\", \"num_occurrences\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save to Ceph or local"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "save = pd.DataFrame(corr_sets, columns=[\"test_name\", \"correlated_tests\"])\n",
    "save[\"correlated_tests\"] = save[\"correlated_tests\"].apply(str)\n",
    "\n",
    "if AUTOMATION:\n",
    "    cc = CephCommunication(s3_endpoint_url, s3_access_key, s3_secret_key, s3_bucket)\n",
    "    cc.upload_to_ceph(\n",
    "        save,\n",
    "        s3_path,\n",
    "        f\"{METRIC_NAME}/{METRIC_NAME}-{timestamp.year}-{timestamp.month}-{timestamp.day}.parquet\",\n",
    "    )\n",
    "else:\n",
    "    save_to_disk(\n",
    "        save,\n",
    "        OUTPUT_DATA_PATH,\n",
    "        f\"{METRIC_NAME}-{timestamp.year}-{timestamp.month}-{timestamp.day}.parquet\",\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>test_name</th>\n",
       "      <th>correlated_tests</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>openshift-tests.[sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client [Top Level] [sig-network] NetworkPolicy [LinuxOnly...</td>\n",
       "      <td>[('openshift-tests.[sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client [Top Level] [sig-network] NetworkPolicy [LinuxO...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...</td>\n",
       "      <td>[('openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformanc...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>openshift-tests.[sig-auth][Feature:OpenShiftAuthorization] The default cluster RBAC policy should have correct RBAC rules [Suite:openshift/conform...</td>\n",
       "      <td>[('openshift-tests.[sig-api-machinery][Feature:ClusterResourceQuota] Cluster resource quota should control resource limits across namespaces [Suit...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>openshift-tests.[sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client [Top Level] [sig-network] NetworkPolicy [LinuxOnly...</td>\n",
       "      <td>[('openshift-tests.[sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client [Top Level] [sig-network] NetworkPolicy [LinuxO...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>openshift-tests.[sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client should allow ingress access from updated namespace...</td>\n",
       "      <td>[('openshift-tests.[sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client should allow ingress access from updated namesp...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                               test_name  \\\n",
       "0  openshift-tests.[sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client [Top Level] [sig-network] NetworkPolicy [LinuxOnly...   \n",
       "1  openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformance/p...   \n",
       "2  openshift-tests.[sig-auth][Feature:OpenShiftAuthorization] The default cluster RBAC policy should have correct RBAC rules [Suite:openshift/conform...   \n",
       "3  openshift-tests.[sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client [Top Level] [sig-network] NetworkPolicy [LinuxOnly...   \n",
       "4  openshift-tests.[sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client should allow ingress access from updated namespace...   \n",
       "\n",
       "                                                                                                                                        correlated_tests  \n",
       "0  [('openshift-tests.[sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client [Top Level] [sig-network] NetworkPolicy [LinuxO...  \n",
       "1  [('openshift-tests.[sig-network][Feature:Router] The HAProxy router should override the route host with a custom value [Suite:openshift/conformanc...  \n",
       "2  [('openshift-tests.[sig-api-machinery][Feature:ClusterResourceQuota] Cluster resource quota should control resource limits across namespaces [Suit...  \n",
       "3  [('openshift-tests.[sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client [Top Level] [sig-network] NetworkPolicy [LinuxO...  \n",
       "4  [('openshift-tests.[sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client should allow ingress access from updated namesp...  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Sanity check to see if the dataset is the same\n",
    "if AUTOMATION:\n",
    "    sanity_check = cc.read_from_ceph(\n",
    "        s3_path,\n",
    "        f\"{METRIC_NAME}/{METRIC_NAME}-{timestamp.year}-{timestamp.month}-{timestamp.day}.parquet\",\n",
    "    ).head()\n",
    "else:\n",
    "    sanity_check = read_from_disk(\n",
    "        OUTPUT_DATA_PATH,\n",
    "        f\"{METRIC_NAME}-{timestamp.year}-{timestamp.month}-{timestamp.day}.parquet\",\n",
    "    ).head()\n",
    "\n",
    "sanity_check"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Lets also capture the average size of correlated failure groups to track over time "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "average_corr = save[\"correlated_tests\"].apply(len).mean()\n",
    "metric_to_save = pd.DataFrame(\n",
    "    [[timestamp, average_corr]],\n",
    "    columns=[\"timestamp\", \"average_number_of_correlated_failures\"],\n",
    ")\n",
    "\n",
    "\n",
    "if AUTOMATION:\n",
    "    cc.upload_to_ceph(\n",
    "        metric_to_save,\n",
    "        s3_path,\n",
    "        f\"avg_{METRIC_NAME}/avg_{METRIC_NAME}-{timestamp.year}-{timestamp.month}-{timestamp.day}.parquet\",\n",
    "    )\n",
    "else:\n",
    "    save_to_disk(\n",
    "        metric_to_save,\n",
    "        OUTPUT_DATA_PATH,\n",
    "        f\"avg_{METRIC_NAME}-{timestamp.year}-{timestamp.month}-{timestamp.day}.parquet\",\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>average_number_of_correlated_failures</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2021-04-27 21:58:21.538747</td>\n",
       "      <td>9224.075023</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   timestamp  average_number_of_correlated_failures\n",
       "0 2021-04-27 21:58:21.538747                            9224.075023"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Sanity check to see if the dataset is the same\n",
    "\n",
    "if AUTOMATION:\n",
    "    sanity_check = cc.read_from_ceph(\n",
    "        s3_path,\n",
    "        f\"avg_{METRIC_NAME}/avg_{METRIC_NAME}-{timestamp.year}-{timestamp.month}-{timestamp.day}.parquet\",\n",
    "    ).head()\n",
    "else:\n",
    "    sanity_check = read_from_disk(\n",
    "        OUTPUT_DATA_PATH,\n",
    "        f\"avg_{METRIC_NAME}-{timestamp.year}-{timestamp.month}-{timestamp.day}.parquet\",\n",
    "    ).head()\n",
    "\n",
    "sanity_check"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "\n",
    "This notebook collected all sets of highly correlated tests, i.e, sets of tests that most commonly fail together and stored that data in ceph as well as locally. A user can now pull this data and, given a test name of interest, be provided a list of all other highly correlated tests. \n",
    "\n",
    "\n",
    "This notebook also computed a numerical value to summarize and quantify these correlations in aggregate: the average size of failure correlation sets. This value is also stored both locally and in ceph. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}