initial commit

hkchengrex · Mar 28, 2020 · 19b7a56 · 19b7a56
1 parent c867754
commit 19b7a56
Show file tree

Hide file tree

Showing 12 changed files with 268 additions and 2 deletions.
diff --git a/README.md b/README.md
@@ -1,2 +1,61 @@
-# CascadePSP
-[CVPR2020] CascadePSP: Toward Class-Agnostic and Very High-Resolution Segmentation via Global and Local Refinement
+# CascadePSP: Toward Class-Agnostic and Very High-Resolution Segmentation via Global and Local Refinement
+
+Ho Kei Cheng, Jihoon Chung, Yu-Wing Tai, Chi-Keung Tang
+
+[[Paper]]()
+
+[[Supplementary Information (Comparisons with DenseCRF included!)]]()
+
+[[Supplementary image results]]()
+
+## Introduction
+
+CascadePSP is a deep learning model for high-resolution segmentation refinement.
+This repository contains our PyTorch implementation with both training and testing functionalities. We also provide the annotated UHD dataset **BIG** and the pretrained model.
+
+Here are some refinement results on high-resolution images.
+![teaser](docs/images/teaser.jpg)
+
+## Network Overview
+
+### Global Step & Local Step
+
+| Global Step | Local Step |
+|:-:|:-:|
+| ![Global Step](docs/images/global.jpg) | ![Local Step](docs/images/local.jpg) |
+
+
+### Refinement Module
+
+![Refinement Module](docs/images/rm.png)
+
+## Table of Contents
+
+Running:
+
+- [Installation](docs/installation.md)
+- [Training](docs/training.md)
+- [Testing on Semantic Segmentation](docs/testing_segmentation.md)
+- [Testing on Scene Parsing](docs/testing_scene_parsing.md)
+
+Downloads:
+
+- [Pretrained Models](docs/models.md)
+- [BIG Dataset and Relabeled PASCAL VOC 2012](docs/dataset.md)
+
+## Credit
+
+PSPNet implementation: https://github.com/Lextal/pspnet-pytorch
+
+SyncBN implementation: https://github.com/vacancy/Synchronized-BatchNorm-PyTorch
+
+If you find our work useful in your research, please cite the following:
+
+```
+@inproceedings{CascadePSP2020,
+  title={CascadePSP: Toward Class-Agnostic and Very High-Resolution Segmentation via Global and Local Refinement},
+  author={Cheng, Ho Kei and Chung, Jihoon and Tai, Yu-Wing and Tang, Chi-Keung},
+  booktitle={CVPR},
+  year={2020}
+}
+```
diff --git a/docs/dataset.md b/docs/dataset.md
@@ -0,0 +1,42 @@
+# Dataset
+
+Here we provide our annotated dataset for evaluation, as well as segmentation results from other models.
+
+## BIG
+BIG is a high-resolution segmentation dataset that has been hand-annotated by us. 
+BIG contains 50 validation objects, and 100 test objects with resolution ranges from 2048\*1600 to 5000\*3600. 
+
+- [One Drive](https://hkustconnect-my.sharepoint.com/:u:/g/personal/jchungaa_connect_ust_hk/EeTPE6gisqBBndX2ABIy2QEBTZR_OxPrpaCdKhuP8Q95QA?e=6rCUSQ)
+
+## Relabeled PASCAL VOC 2012
+We have relabeled 500 images from PASCAL VOC 2012 to have more accurate boundaries. 
+Below shows an example of our relabeled segmentation.
+
+![](images/relabeled_pascal.png)
+
+- [One Drive](https://hkustconnect-my.sharepoint.com/:u:/g/personal/jchungaa_connect_ust_hk/EbtbHa40zNJDpNlD3UbDadQB4eG_dNfFI7YDit3OYOXAkw?e=Gmuaym)
+
+## Segmentation Results
+
+For convenience, we provide segmentation results from other models for evaluation. 
+We tried our best to match the performance in their original papers, and use official code whenever available. 
+<!-- These are NOT an official result from the authors of the paper.  -->
+<!-- We recommend you to get the segmentation results manually from the original author's code release to test our model.  -->
+<!-- We also include multi-scale evaluation  -->
+
+| Segmentation |             |   |                         |                               |
+|--------------|-------------|---|-------------------------|-------------------------------|
+| BIG (Test)   | DeeplabV3+  |   | [Download](https://hkustconnect-my.sharepoint.com/:f:/g/personal/jchungaa_connect_ust_hk/Em8xxjDNRVNFpZaWwJV49NkBXxQwXd_AAIahQniAnq5IkQ?e=OwheVV) | [Source](https://github.com/tensorflow/models/tree/master/research/deeplab) |
+|              | RefineNet   |   | [Download](https://hkustconnect-my.sharepoint.com/:f:/g/personal/jchungaa_connect_ust_hk/Em8xxjDNRVNFpZaWwJV49NkBXxQwXd_AAIahQniAnq5IkQ?e=OwheVV) | [Source](https://github.com/guosheng/refinenet) |
+|              | PSPNet      |   | [Download](https://hkustconnect-my.sharepoint.com/:f:/g/personal/jchungaa_connect_ust_hk/Em8xxjDNRVNFpZaWwJV49NkBXxQwXd_AAIahQniAnq5IkQ?e=OwheVV) | [Source](https://github.com/hszhao/PSPNet) |
+|              | FCN-8s      |   | [Download](https://hkustconnect-my.sharepoint.com/:f:/g/personal/jchungaa_connect_ust_hk/Em8xxjDNRVNFpZaWwJV49NkBXxQwXd_AAIahQniAnq5IkQ?e=OwheVV) | [Source](https://github.com/developmentseed/caffe-fcn/tree/master/fcn-8s) |
+| PASCAL       | DeeplabV3+  |   | [Download](https://hkustconnect-my.sharepoint.com/:f:/g/personal/jchungaa_connect_ust_hk/EhTt-3DzfdZHoRsjQEC8_xABjjQEHbK9rKgXE78btCfE0g?e=EvsRGH) | [Source](https://github.com/tensorflow/models/tree/master/research/deeplab) |
+|              | RefineNet   |   | [Download](https://hkustconnect-my.sharepoint.com/:f:/g/personal/jchungaa_connect_ust_hk/EhTt-3DzfdZHoRsjQEC8_xABjjQEHbK9rKgXE78btCfE0g?e=EvsRGH) | [Source](https://github.com/guosheng/refinenet) |
+|              | PSPNet      |   | [Download](https://hkustconnect-my.sharepoint.com/:f:/g/personal/jchungaa_connect_ust_hk/EhTt-3DzfdZHoRsjQEC8_xABjjQEHbK9rKgXE78btCfE0g?e=EvsRGH) | [Source](https://github.com/hszhao/PSPNet) |
+|              | FCN-8s      |   | [Download](https://hkustconnect-my.sharepoint.com/:f:/g/personal/jchungaa_connect_ust_hk/EhTt-3DzfdZHoRsjQEC8_xABjjQEHbK9rKgXE78btCfE0g?e=EvsRGH) | [Source](https://github.com/developmentseed/caffe-fcn/tree/master/fcn-8s) |
+
+| Scene Parsing |           |   |                         |                               |
+|---------------|-----------|---|-------------------------|-------------------------------|
+| ADE20K        | RefineNet |   | [Download](https://hkustconnect-my.sharepoint.com/:f:/g/personal/jchungaa_connect_ust_hk/EvIgfKbjdNdJkjchYL5GBgcBzNX5n4DoLWoLx2dJjFBWgA?e=wGGxNt) | [Source](https://github.com/guosheng/refinenet) |
+|               | EncNet    |   | [Download](https://hkustconnect-my.sharepoint.com/:f:/g/personal/jchungaa_connect_ust_hk/EvIgfKbjdNdJkjchYL5GBgcBzNX5n4DoLWoLx2dJjFBWgA?e=wGGxNt) | [Source](https://github.com/zhanghang1989/PyTorch-Encoding) |
+|               | PSPNet    |   | [Download](https://hkustconnect-my.sharepoint.com/:f:/g/personal/jchungaa_connect_ust_hk/EvIgfKbjdNdJkjchYL5GBgcBzNX5n4DoLWoLx2dJjFBWgA?e=wGGxNt) | [Source](https://github.com/hszhao/PSPNet) |
diff --git a/docs/images/global.jpg b/docs/images/global.jpg
diff --git a/docs/images/local.jpg b/docs/images/local.jpg
diff --git a/docs/images/relabeled_pascal.png b/docs/images/relabeled_pascal.png
diff --git a/docs/images/rm.png b/docs/images/rm.png
diff --git a/docs/images/teaser.jpg b/docs/images/teaser.jpg
diff --git a/docs/installation.md b/docs/installation.md
@@ -0,0 +1,13 @@
+# Installation
+
+## Dependencies
+
+CascadePSP is tested on PyTorch 1.0 though higher version would likely work as well. 
+For installing PyTorch, please refer to the [Pytorch website](https://pytorch.org/).  
+
+We recommend using the anaconda distribution which should contain most of the required dependencies. Other dependencies can be installed by:
+
+``` bash
+pip install progressbar2
+conda install cv2
+```
diff --git a/docs/models.md b/docs/models.md
@@ -0,0 +1,7 @@
+# Model Zoo
+
+<!-- We provide the CascadePSP model that has been trained in merged dataset that contains MSRA-10K, DUT-OMRON, ECSSD, and FSS-1000. -->
+
+Checkpoint name       | Comment |  File Size | 
+----------------------| -----------------|  --------- |
+ [Model](https://hkustconnect-my.sharepoint.com/:u:/g/personal/hkchengad_connect_ust_hk/ESR9WDbHDeBNsCKqpR5KA7EBADMEgbt94nX11qzitNwNfQ?e=deEUOG) | This is the model that we used to generate all of our results in the paper. | 259MB |
diff --git a/docs/testing_scene_parsing.md b/docs/testing_scene_parsing.md
@@ -0,0 +1,38 @@
+# Testing on Scene Parsing
+
+Pretrained models can be downloaded [here](models.md). 
+For convenience, we offer pre-processed scene parsing inputs from other segmentation models [here](dataset.md).
+
+## Test set Structure
+Evaluation on scene parsing dataset is more complicated. Read this [document](testing_segmentation.md) about testing on segmentation first for starters. 
+
+We need to perform the following steps:
+
+1. Obtain initial segmentations from other models.
+2. Break down the scene parse into individual components using the method described in the paper.
+3. Process each component separately using CascadePSP.
+4. Combine the processed components to a final scene parse.
+
+You can skip step 1 and 2 by downloading our pre-processed dataset. 
+
+## Testing
+
+To run step 3, append an extra flag `--ade` to `eval.py`.
+``` bash
+# From CascadePSP/
+python eval.py \
+    --dataset testset_directory \
+    --model model.model \
+    --output output_directory \
+    --ade
+```
+
+And to run step 4, 
+
+``` 
+python eval_post_ade.py \
+    --mask_dir [Output directory in step3] \
+    --seg_dir [Directory with the original initial segmentations] \
+    --split_dir [Directory with the broken-down initial segmentations] \
+    --output output_directory
+```
diff --git a/docs/testing_segmentation.md b/docs/testing_segmentation.md
@@ -0,0 +1,47 @@
+# Testing on Semantic Segmentation
+
+Pretrained models can be downloaded [here](models.md). 
+For convenience, we offer pre-processed segmentation inputs from other segmentation models [here](dataset.md).
+
+## Test set Structure
+
+Our test script expects the following structure:
+
+```
++ testset_directory
+  - imagename_gt.png
+  - imagename_seg.png
+  - imagename_im.jpg
+```
+
+Where `_gt`, `_seg`, and `_im` denote the input segmentation, ground-truth segmentation, and RGB image respectively. Segmentations should be in binary format (i.e. only one object at a time).
+
+## Testing
+
+To refine on high-resolution segmentations using both the Global and Local step (i.e. for the BIG dataset), use the following:
+``` bash
+# From CascadePSP/
+python eval.py \
+    --dataset testset_directory \
+    --model model.model \
+    --output output_directory
+```
+
+To refine on low-resolution segmentations, we can skip the Local step (though using both will not deteriorate the result) by appending a `--global_only` flag, i.e.: 
+
+``` bash
+# From CascadePSP/
+python eval.py \
+    --dataset testset_directory \
+    --model model.model \
+    --output output_directory \
+    --global_only
+```
+
+You can obtain the accurate metrics (i.e. IoU and mBA) by running a separate script -- this allows you to test your own results easily:
+
+``` bash
+# From CascadePSP/
+python eval_post.py \
+    --dir results_directory
+```
diff --git a/docs/training.md b/docs/training.md
@@ -0,0 +1,60 @@
+# Training
+
+### Download the dataset 
+
+We have prepared a script for downloading the training dataset. 
+The script below downloads and merges the following datasets: MSRA-10K, DUT-OMRON, ECSSD, and FSS-1000.
+
+```
+# From cascadepsp/scripts/
+python download_training_dataset.py
+```
+
+Note that the following script will create a dataset folder as follows:
+```
++ cascadepsp/data/
+  + DUTS-TE/
+     - image_name_01.jpg
+     - image_name_01.png
+     - ...
+  + DUTS-TR/
+     - image_name_01.jpg
+     - image_name_01.png
+     - ...
+  + ecssd/
+     - image_name_01.jpg
+     - image_name_01.png
+     - ...
+  + fss/
+    + class_name/
+         - image_name_01.jpg
+         - image_name_01.png
+     - ...
+  + MSRA_10K/
+     - image_name_01.jpg
+     - image_name_01.png
+     - ...
+```
+
+### Running the Training
+
+Training can be done with following command with some distinguishable id:
+
+```
+# From cascadepsp/
+python train.py some_unique_id
+```
+
+Note that you can change the hyperparameter by specifying arguments, e.g. to change batch size.
+
+```
+# From cascadepsp/
+python train.py -b 10 some_unique_id
+```
+Please check [hyper_para.py](../util/hyper_para.py) for more options.
+
+### After the Training
+
+Tensorboard log file will be stored in `cascadepsp/log/some_unique_id_timestamp`
+
+Model will be saved in `cascadepsp/weights/some_unique_id_timestamp`