diff --git a/README.md b/README.md index 1db19d2..e123fce 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,61 @@ -# CascadePSP -[CVPR2020] CascadePSP: Toward Class-Agnostic and Very High-Resolution Segmentation via Global and Local Refinement +# CascadePSP: Toward Class-Agnostic and Very High-Resolution Segmentation via Global and Local Refinement + +Ho Kei Cheng, Jihoon Chung, Yu-Wing Tai, Chi-Keung Tang + +[[Paper]]() + +[[Supplementary Information (Comparisons with DenseCRF included!)]]() + +[[Supplementary image results]]() + +## Introduction + +CascadePSP is a deep learning model for high-resolution segmentation refinement. +This repository contains our PyTorch implementation with both training and testing functionalities. We also provide the annotated UHD dataset **BIG** and the pretrained model. + +Here are some refinement results on high-resolution images. +![teaser](docs/images/teaser.jpg) + +## Network Overview + +### Global Step & Local Step + +| Global Step | Local Step | +|:-:|:-:| +| ![Global Step](docs/images/global.jpg) | ![Local Step](docs/images/local.jpg) | + + +### Refinement Module + +![Refinement Module](docs/images/rm.png) + +## Table of Contents + +Running: + +- [Installation](docs/installation.md) +- [Training](docs/training.md) +- [Testing on Semantic Segmentation](docs/testing_segmentation.md) +- [Testing on Scene Parsing](docs/testing_scene_parsing.md) + +Downloads: + +- [Pretrained Models](docs/models.md) +- [BIG Dataset and Relabeled PASCAL VOC 2012](docs/dataset.md) + +## Credit + +PSPNet implementation: https://github.com/Lextal/pspnet-pytorch + +SyncBN implementation: https://github.com/vacancy/Synchronized-BatchNorm-PyTorch + +If you find our work useful in your research, please cite the following: + +``` +@inproceedings{CascadePSP2020, + title={CascadePSP: Toward Class-Agnostic and Very High-Resolution Segmentation via Global and Local Refinement}, + author={Cheng, Ho Kei and Chung, Jihoon and Tai, Yu-Wing and Tang, Chi-Keung}, + booktitle={CVPR}, + year={2020} +} +``` diff --git a/docs/dataset.md b/docs/dataset.md new file mode 100644 index 0000000..c677f1f --- /dev/null +++ b/docs/dataset.md @@ -0,0 +1,42 @@ +# Dataset + +Here we provide our annotated dataset for evaluation, as well as segmentation results from other models. + +## BIG +BIG is a high-resolution segmentation dataset that has been hand-annotated by us. +BIG contains 50 validation objects, and 100 test objects with resolution ranges from 2048\*1600 to 5000\*3600. + +- [One Drive](https://hkustconnect-my.sharepoint.com/:u:/g/personal/jchungaa_connect_ust_hk/EeTPE6gisqBBndX2ABIy2QEBTZR_OxPrpaCdKhuP8Q95QA?e=6rCUSQ) + +## Relabeled PASCAL VOC 2012 +We have relabeled 500 images from PASCAL VOC 2012 to have more accurate boundaries. +Below shows an example of our relabeled segmentation. + +![](images/relabeled_pascal.png) + +- [One Drive](https://hkustconnect-my.sharepoint.com/:u:/g/personal/jchungaa_connect_ust_hk/EbtbHa40zNJDpNlD3UbDadQB4eG_dNfFI7YDit3OYOXAkw?e=Gmuaym) + +## Segmentation Results + +For convenience, we provide segmentation results from other models for evaluation. +We tried our best to match the performance in their original papers, and use official code whenever available. + + + + +| Segmentation | | | | | +|--------------|-------------|---|-------------------------|-------------------------------| +| BIG (Test) | DeeplabV3+ | | [Download](https://hkustconnect-my.sharepoint.com/:f:/g/personal/jchungaa_connect_ust_hk/Em8xxjDNRVNFpZaWwJV49NkBXxQwXd_AAIahQniAnq5IkQ?e=OwheVV) | [Source](https://github.com/tensorflow/models/tree/master/research/deeplab) | +| | RefineNet | | [Download](https://hkustconnect-my.sharepoint.com/:f:/g/personal/jchungaa_connect_ust_hk/Em8xxjDNRVNFpZaWwJV49NkBXxQwXd_AAIahQniAnq5IkQ?e=OwheVV) | [Source](https://github.com/guosheng/refinenet) | +| | PSPNet | | [Download](https://hkustconnect-my.sharepoint.com/:f:/g/personal/jchungaa_connect_ust_hk/Em8xxjDNRVNFpZaWwJV49NkBXxQwXd_AAIahQniAnq5IkQ?e=OwheVV) | [Source](https://github.com/hszhao/PSPNet) | +| | FCN-8s | | [Download](https://hkustconnect-my.sharepoint.com/:f:/g/personal/jchungaa_connect_ust_hk/Em8xxjDNRVNFpZaWwJV49NkBXxQwXd_AAIahQniAnq5IkQ?e=OwheVV) | [Source](https://github.com/developmentseed/caffe-fcn/tree/master/fcn-8s) | +| PASCAL | DeeplabV3+ | | [Download](https://hkustconnect-my.sharepoint.com/:f:/g/personal/jchungaa_connect_ust_hk/EhTt-3DzfdZHoRsjQEC8_xABjjQEHbK9rKgXE78btCfE0g?e=EvsRGH) | [Source](https://github.com/tensorflow/models/tree/master/research/deeplab) | +| | RefineNet | | [Download](https://hkustconnect-my.sharepoint.com/:f:/g/personal/jchungaa_connect_ust_hk/EhTt-3DzfdZHoRsjQEC8_xABjjQEHbK9rKgXE78btCfE0g?e=EvsRGH) | [Source](https://github.com/guosheng/refinenet) | +| | PSPNet | | [Download](https://hkustconnect-my.sharepoint.com/:f:/g/personal/jchungaa_connect_ust_hk/EhTt-3DzfdZHoRsjQEC8_xABjjQEHbK9rKgXE78btCfE0g?e=EvsRGH) | [Source](https://github.com/hszhao/PSPNet) | +| | FCN-8s | | [Download](https://hkustconnect-my.sharepoint.com/:f:/g/personal/jchungaa_connect_ust_hk/EhTt-3DzfdZHoRsjQEC8_xABjjQEHbK9rKgXE78btCfE0g?e=EvsRGH) | [Source](https://github.com/developmentseed/caffe-fcn/tree/master/fcn-8s) | + +| Scene Parsing | | | | | +|---------------|-----------|---|-------------------------|-------------------------------| +| ADE20K | RefineNet | | [Download](https://hkustconnect-my.sharepoint.com/:f:/g/personal/jchungaa_connect_ust_hk/EvIgfKbjdNdJkjchYL5GBgcBzNX5n4DoLWoLx2dJjFBWgA?e=wGGxNt) | [Source](https://github.com/guosheng/refinenet) | +| | EncNet | | [Download](https://hkustconnect-my.sharepoint.com/:f:/g/personal/jchungaa_connect_ust_hk/EvIgfKbjdNdJkjchYL5GBgcBzNX5n4DoLWoLx2dJjFBWgA?e=wGGxNt) | [Source](https://github.com/zhanghang1989/PyTorch-Encoding) | +| | PSPNet | | [Download](https://hkustconnect-my.sharepoint.com/:f:/g/personal/jchungaa_connect_ust_hk/EvIgfKbjdNdJkjchYL5GBgcBzNX5n4DoLWoLx2dJjFBWgA?e=wGGxNt) | [Source](https://github.com/hszhao/PSPNet) | \ No newline at end of file diff --git a/docs/images/global.jpg b/docs/images/global.jpg new file mode 100644 index 0000000..5798660 Binary files /dev/null and b/docs/images/global.jpg differ diff --git a/docs/images/local.jpg b/docs/images/local.jpg new file mode 100644 index 0000000..07e8655 Binary files /dev/null and b/docs/images/local.jpg differ diff --git a/docs/images/relabeled_pascal.png b/docs/images/relabeled_pascal.png new file mode 100644 index 0000000..b7b07c4 Binary files /dev/null and b/docs/images/relabeled_pascal.png differ diff --git a/docs/images/rm.png b/docs/images/rm.png new file mode 100644 index 0000000..6dbce96 Binary files /dev/null and b/docs/images/rm.png differ diff --git a/docs/images/teaser.jpg b/docs/images/teaser.jpg new file mode 100644 index 0000000..8254e62 Binary files /dev/null and b/docs/images/teaser.jpg differ diff --git a/docs/installation.md b/docs/installation.md new file mode 100644 index 0000000..a3c6185 --- /dev/null +++ b/docs/installation.md @@ -0,0 +1,13 @@ +# Installation + +## Dependencies + +CascadePSP is tested on PyTorch 1.0 though higher version would likely work as well. +For installing PyTorch, please refer to the [Pytorch website](https://pytorch.org/). + +We recommend using the anaconda distribution which should contain most of the required dependencies. Other dependencies can be installed by: + +``` bash +pip install progressbar2 +conda install cv2 +``` diff --git a/docs/models.md b/docs/models.md new file mode 100644 index 0000000..19d14f3 --- /dev/null +++ b/docs/models.md @@ -0,0 +1,7 @@ +# Model Zoo + + + +Checkpoint name | Comment | File Size | +----------------------| -----------------| --------- | + [Model](https://hkustconnect-my.sharepoint.com/:u:/g/personal/hkchengad_connect_ust_hk/ESR9WDbHDeBNsCKqpR5KA7EBADMEgbt94nX11qzitNwNfQ?e=deEUOG) | This is the model that we used to generate all of our results in the paper. | 259MB | \ No newline at end of file diff --git a/docs/testing_scene_parsing.md b/docs/testing_scene_parsing.md new file mode 100644 index 0000000..bee08f4 --- /dev/null +++ b/docs/testing_scene_parsing.md @@ -0,0 +1,38 @@ +# Testing on Scene Parsing + +Pretrained models can be downloaded [here](models.md). +For convenience, we offer pre-processed scene parsing inputs from other segmentation models [here](dataset.md). + +## Test set Structure +Evaluation on scene parsing dataset is more complicated. Read this [document](testing_segmentation.md) about testing on segmentation first for starters. + +We need to perform the following steps: + +1. Obtain initial segmentations from other models. +2. Break down the scene parse into individual components using the method described in the paper. +3. Process each component separately using CascadePSP. +4. Combine the processed components to a final scene parse. + +You can skip step 1 and 2 by downloading our pre-processed dataset. + +## Testing + +To run step 3, append an extra flag `--ade` to `eval.py`. +``` bash +# From CascadePSP/ +python eval.py \ + --dataset testset_directory \ + --model model.model \ + --output output_directory \ + --ade +``` + +And to run step 4, + +``` +python eval_post_ade.py \ + --mask_dir [Output directory in step3] \ + --seg_dir [Directory with the original initial segmentations] \ + --split_dir [Directory with the broken-down initial segmentations] \ + --output output_directory +``` diff --git a/docs/testing_segmentation.md b/docs/testing_segmentation.md new file mode 100644 index 0000000..4e02dfb --- /dev/null +++ b/docs/testing_segmentation.md @@ -0,0 +1,47 @@ +# Testing on Semantic Segmentation + +Pretrained models can be downloaded [here](models.md). +For convenience, we offer pre-processed segmentation inputs from other segmentation models [here](dataset.md). + +## Test set Structure + +Our test script expects the following structure: + +``` ++ testset_directory + - imagename_gt.png + - imagename_seg.png + - imagename_im.jpg +``` + +Where `_gt`, `_seg`, and `_im` denote the input segmentation, ground-truth segmentation, and RGB image respectively. Segmentations should be in binary format (i.e. only one object at a time). + +## Testing + +To refine on high-resolution segmentations using both the Global and Local step (i.e. for the BIG dataset), use the following: +``` bash +# From CascadePSP/ +python eval.py \ + --dataset testset_directory \ + --model model.model \ + --output output_directory +``` + +To refine on low-resolution segmentations, we can skip the Local step (though using both will not deteriorate the result) by appending a `--global_only` flag, i.e.: + +``` bash +# From CascadePSP/ +python eval.py \ + --dataset testset_directory \ + --model model.model \ + --output output_directory \ + --global_only +``` + +You can obtain the accurate metrics (i.e. IoU and mBA) by running a separate script -- this allows you to test your own results easily: + +``` bash +# From CascadePSP/ +python eval_post.py \ + --dir results_directory +``` diff --git a/docs/training.md b/docs/training.md new file mode 100644 index 0000000..07ad71d --- /dev/null +++ b/docs/training.md @@ -0,0 +1,60 @@ +# Training + +### Download the dataset + +We have prepared a script for downloading the training dataset. +The script below downloads and merges the following datasets: MSRA-10K, DUT-OMRON, ECSSD, and FSS-1000. + +``` +# From cascadepsp/scripts/ +python download_training_dataset.py +``` + +Note that the following script will create a dataset folder as follows: +``` ++ cascadepsp/data/ + + DUTS-TE/ + - image_name_01.jpg + - image_name_01.png + - ... + + DUTS-TR/ + - image_name_01.jpg + - image_name_01.png + - ... + + ecssd/ + - image_name_01.jpg + - image_name_01.png + - ... + + fss/ + + class_name/ + - image_name_01.jpg + - image_name_01.png + - ... + + MSRA_10K/ + - image_name_01.jpg + - image_name_01.png + - ... +``` + +### Running the Training + +Training can be done with following command with some distinguishable id: + +``` +# From cascadepsp/ +python train.py some_unique_id +``` + +Note that you can change the hyperparameter by specifying arguments, e.g. to change batch size. + +``` +# From cascadepsp/ +python train.py -b 10 some_unique_id +``` +Please check [hyper_para.py](../util/hyper_para.py) for more options. + +### After the Training + +Tensorboard log file will be stored in `cascadepsp/log/some_unique_id_timestamp` + +Model will be saved in `cascadepsp/weights/some_unique_id_timestamp` \ No newline at end of file