Refactor project and add metrics MLCube

Author: davidjurado

brats/metrics/.gitignore

@@ -0,0 +1,2 @@
+__pycache__/
+mlcube/workspace/results.yaml

brats/metrics/README.md

@@ -0,0 +1,98 @@
+# BraTS Challenge 2020 - MLCube integration - Metrics
+
+Original implementation: ["BraTS Instructions Repo"](https://github.com/BraTS/Instructions)
+
+## Dataset
+
+Please refer to the [BraTS challenge page](http://braintumorsegmentation.org/) and follow the instructions in the data section.
+
+## Project setup
+
+```bash
+# Create Python environment and install MLCube Docker runner 
+virtualenv -p python3 ./env && source ./env/bin/activate && pip install mlcube-docker
+
+# Fetch the boston housing example from GitHub
+git clone https://github.com/mlcommons/mlcube_examples && cd ./mlcube_examples
+git fetch origin pull/39/head:feature/brats && git checkout feature/brats
+cd ./brats/metrics/mlcube
+```
+
+## Important files
+
+These are the most important files on this project:
+
+```bash
+
+├── mlcube
+│   ├── mlcube.yaml                             # MLCube configuration file, it defines the project, author, platform, docker and tasks.
+│   └── workspace
+│       ├── data
+│       │   ├── ground_truth
+│       │   │   └── BraTS_example_seg.nii.gz    # Ground truth example file
+│       │   └── predictions
+│       │       └── BraTS_example_seg.nii.gz    # Prediction example file
+│       ├── parameters.yaml
+│       └── results.yaml                        # Final output file containing result metrics.
+└── project
+    ├── Dockerfile                              # Docker file with instructions to create the image for the project.
+    ├── metrics.py                              # Python file that contains the main logic of the project.
+    ├── mlcube.py                               # Python entrypoint used by MLCube, contains the logic for MLCube tasks.
+    └── requirements.txt                        # Python requirements needed to run the project inside Docker.
+```
+
+## How to modify this project
+
+You can change each file described above in order to add your own implementation.
+
+### Requirements file
+
+In this file (`requirements.txt`) you can add all the python dependencies needed for running your implementation, these dependencies will be installed during the creation of the docker image, this happens when you run the ```mlcube run ...``` command.
+
+### Dockerfile
+
+You can use both, CPU or GPU version for the dockerfile (`Dockerfile_CPU`, `Dockerfile_GPU`), also, you can add or modify any steps inside the file, this comes handy when you need to install some OS dependencies or even when you want to change the base docker image, inside the file you can find some information about the existing steps.
+
+### Parameters file
+
+This is a yaml file (`parameters.yaml`)that contains all extra parameters that aren't files or directories, for example, here you can place all the hyperparameters that you will use for training a model. This file will be passed as an **input parameter** in the MLCube tasks and then it will be read inside the MLCube container.
+
+### MLCube yaml file
+
+In this file (`mlcube.yaml`) you can find the instructions about the docker image and platform that will be used, information about the project (name, description, authors), and also the tasks defined for the project.
+
+In the existing implementation you will find 1 task:
+
+* evaluate:
+
+    This task takes the following parameters:
+
+  * Input parameters:
+    * predictions: Folder path containing predictions
+    * ground_truth: Folder path containing ground truth data
+    * parameters_file: Extra parameters
+  * Output parameters:
+    * output_path: File path where output metrics will be stored
+
+    This task takes the input predictions and ground truth data, perform the evaluation and then save the output result in the output_path.
+
+### MLCube python file
+
+The `mlcube.py` file is the handler file and entrypoint described in the dockerfile, here you can find all the logic related to how to process each MLCube task. If you want to add a new task first you must define it inside the `mlcube.yaml` file with its input and output parameters and then you need to add the logic to handle this new task inside the `mlcube.py` file.
+
+### Metrics file
+
+The `metrics.py` file contains the main logic of the project, you can modify this file and write your implementation here to calculate different metrics, this metrics file is called from the `mlcube.py` file and there are other ways to link your implementation and shown in the [MLCube examples repo](https://github.com/mlcommons/mlcube_examples).
+
+## Tasks execution
+
+```bash
+# Run evaluate task.
+mlcube run --mlcube=mlcube_cpu.yaml --task=evaluate
+```
+
+We are targeting pull-type installation, so MLCube images should be available on Docker Hub. If not, try this:
+
+```Bash
+mlcube run ... -Pdocker.build_strategy=always
+```

brats/metrics/mlcube/mlcube.yaml

@@ -0,0 +1,22 @@
+name: MLCommons Brats metrics
+description: MLCommons Brats integration for metrics
+authors: 
+ - {name: "MLCommons Best Practices Working Group"}
+
+platform:
+  accelerator_count: 0
+
+docker:
+  # Image name.
+  image: mlcommons/brats_metrics:0.0.1
+  # Docker build context relative to $MLCUBE_ROOT. Default is `build`.
+  build_context: "../project"
+  # Docker file name within docker build context, default is `Dockerfile`.
+  build_file: "Dockerfile"
+
+tasks:
+  evaluate:
+  # Executes a number of metrics specified by the params file
+    parameters:
+      inputs: {predictions: data/predictions/, ground_truth: data/ground_truth/, parameters_file: parameters.yaml}
+      outputs: {output_path: {type: "file", default: "results.yaml"}}

brats/metrics/mlcube/workspace/data/ground_truth/BraTS_example_seg.nii.gz

@@ -0,0 +1,2 @@
+treshold: 0.5
+eps: 0

brats/metrics/mlcube/workspace/data/predictions/BraTS_example_seg.nii.gz

@@ -0,0 +1,172 @@
+"""Logic file"""
+import argparse
+import glob
+import yaml
+from pkgutil import get_data
+import nibabel as nib
+import numpy as np
+
+
+def dice_coef_metric(
+    probabilities: np.ndarray, truth: np.ndarray, treshold: float = 0.5, eps: float = 0
+) -> np.ndarray:
+    """
+    Calculate Dice score for data batch.
+    Params:
+        probobilities: model outputs after activation function.
+        truth: truth values.
+        threshold: threshold for probabilities.
+        eps: additive to refine the estimate.
+        Returns: dice score aka f1.
+    """
+    scores = []
+    num = probabilities.shape[0]
+    predictions = probabilities >= treshold
+    assert predictions.shape == truth.shape
+    for i in range(num):
+        prediction = predictions[i]
+        truth_ = truth[i]
+        intersection = 2.0 * (truth_ * prediction).sum()
+        union = truth_.sum() + prediction.sum()
+        if truth_.sum() == 0 and prediction.sum() == 0:
+            scores.append(1.0)
+        else:
+            scores.append((intersection + eps) / union)
+    return np.mean(scores)
+
+
+def jaccard_coef_metric(
+    probabilities: np.ndarray, truth: np.ndarray, treshold: float = 0.5, eps: float = 0
+) -> np.ndarray:
+    """
+    Calculate Jaccard index for data batch.
+    Params:
+        probobilities: model outputs after activation function.
+        truth: truth values.
+        threshold: threshold for probabilities.
+        eps: additive to refine the estimate.
+        Returns: jaccard score aka iou."
+    """
+    scores = []
+    num = probabilities.shape[0]
+    predictions = probabilities >= treshold
+    assert predictions.shape == truth.shape
+
+    for i in range(num):
+        prediction = predictions[i]
+        truth_ = truth[i]
+        intersection = (prediction * truth_).sum()
+        union = (prediction.sum() + truth_.sum()) - intersection + eps
+        if truth_.sum() == 0 and prediction.sum() == 0:
+            scores.append(1.0)
+        else:
+            scores.append((intersection + eps) / union)
+    return np.mean(scores)
+
+
+def preprocess_mask_labels(mask: np.ndarray):
+
+    mask_WT = mask.copy()
+    mask_WT[mask_WT == 1] = 1
+    mask_WT[mask_WT == 2] = 1
+    mask_WT[mask_WT == 4] = 1
+
+    mask_TC = mask.copy()
+    mask_TC[mask_TC == 1] = 1
+    mask_TC[mask_TC == 2] = 0
+    mask_TC[mask_TC == 4] = 1
+
+    mask_ET = mask.copy()
+    mask_ET[mask_ET == 1] = 0
+    mask_ET[mask_ET == 2] = 0
+    mask_ET[mask_ET == 4] = 1
+
+    mask = np.stack([mask_WT, mask_TC, mask_ET])
+    mask = np.moveaxis(mask, (0, 1, 2, 3), (0, 3, 2, 1))
+
+    return mask
+
+
+def load_img(file_path):
+    data = nib.load(file_path)
+    data = np.asarray(data.dataobj)
+    return data
+
+
+def get_data_arr(predictions_path, ground_truth_path):
+    predictions = glob.glob(predictions_path + "/*")
+    ground_truth = glob.glob(ground_truth_path + "/*")
+    if not len(predictions) == len(ground_truth):
+        raise ValueError(
+            "Number of predictions should be the same of ground truth labels"
+        )
+    gt_arr, prediction_arr = [], []
+    for gt_path, prediction_path in zip(ground_truth, predictions):
+        gt = load_img(gt_path)
+        gt = preprocess_mask_labels(gt)
+        prediction = load_img(prediction_path)
+        prediction = preprocess_mask_labels(prediction)
+        gt_arr.append(gt)
+        prediction_arr.append(prediction)
+    gt_arr = np.concatenate(gt_arr)
+    prediction_arr = np.concatenate(prediction_arr)
+    return gt_arr, prediction_arr
+
+
+def create_metrics_file(output_file, results):
+    with open(output_file, "w") as f:
+        yaml.dump(results, f)
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--ground_truth",
+        type=str,
+        required=True,
+        help="Directory containing the ground truth data",
+    )
+    parser.add_argument(
+        "--predictions",
+        type=str,
+        required=True,
+        help="Directory containing the predictions",
+    )
+    parser.add_argument(
+        "--output_file",
+        "--output-file",
+        type=str,
+        required=True,
+        help="file to store metrics results as YAML",
+    )
+    parser.add_argument(
+        "--parameters_file",
+        "--parameters-file",
+        type=str,
+        required=True,
+        help="File containing parameters for evaluation",
+    )
+    args = parser.parse_args()
+
+    with open(args.parameters_file, "r") as f:
+        params = yaml.full_load(f)
+
+    gt_arr, pred_arr = get_data_arr(args.predictions, args.ground_truth)
+
+    treshold = float(params["treshold"])
+    eps = float(params["eps"])
+
+    dice_coef = dice_coef_metric(pred_arr, gt_arr, treshold, eps)
+    jaccard_coef = jaccard_coef_metric(pred_arr, gt_arr, treshold, eps)
+
+    results = {
+        "dice_coef": str(dice_coef),
+        "jaccard_coef": str(jaccard_coef),
+    }
+
+    print(results)
+    create_metrics_file(args.output_file, results)
+
+
+if __name__ == "__main__":
+    main()

brats/metrics/mlcube/workspace/parameters.yaml

@@ -0,0 +1,43 @@
+"""MLCube handler file"""
+import os
+import typer
+import subprocess
+
+
+app = typer.Typer()
+
+
+class EvaluateTask(object):
+    """Runs evaluation metrics given the predictions and label files
+    Args:
+        object ([type]): [description]
+    """
+
+    @staticmethod
+    def run(
+        ground_truth: str, predictions: str, parameters_file: str, output_file: str
+    ) -> None:
+        cmd = f"python3 metrics.py --ground_truth={ground_truth} --predictions={predictions} --parameters_file={parameters_file} --output_file={output_file}"
+        splitted_cmd = cmd.split()
+
+        process = subprocess.Popen(splitted_cmd, cwd=".")
+        process.wait()
+
+
+@app.command("evaluate")
+def evaluate(
+    ground_truth: str = typer.Option(..., "--ground_truth"),
+    predictions: str = typer.Option(..., "--predictions"),
+    parameters_file: str = typer.Option(..., "--parameters_file"),
+    output_path: str = typer.Option(..., "--output_path"),
+):
+    EvaluateTask.run(ground_truth, predictions, parameters_file, output_path)
+
+
+@app.command("test")
+def test():
+    pass
+
+
+if __name__ == "__main__":
+    app()

brats/project/Dockerfile_CPU brats/metrics/project/Dockerfilebrats/project/Dockerfile_CPU renamed to brats/metrics/project/Dockerfile

@@ -0,0 +1,4 @@
+PyYAML
+typer
+numpy
+nibabel