A Robust Real-world Change Detection Transformer

Cheng Wang¹, Xinggang Wang¹, Shusheng Yang¹, Zelin Liu¹, Bingxian Shi², Guoli Wang³, Qian Zhang³, Geng Wei², Peng Guo^1,📧, Wenyu Liu¹

¹ Huazhong University of Science and Technology, ² Nanning Normal University, ³ Horizon Robotics, ^📧 corresponding author

Accepted to IEEE Transactions on Image Processing

Installation

conda create -n cyws python=3.9
conda activate cyws
conda install -c pytorch pytorch=1.10.1 torchvision=0.11.2 cudatoolkit=11.3.1
conda install -c conda-forge pytorch-lightning=1.5.8
pip install kornia@git+https://github.com/kornia/kornia@77589a58be6c603b7afd755d261783bd0c152a97
pip install matplotlib Shapely==1.8.0 easydict loguru scipy h5py bytecode
python -m pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu113/torch1.10/index.html
pip install mmcv-full==1.7.0 -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.10/index.html
pip install mmdet==2.28.2 wandb
pip install segmentation-models-pytorch@git+https://github.com/ragavsachdeva/segmentation_models.pytorch.git@0092ee4d6f851d89a4a401bb2dfa6187660b8dd3
pip install imageio==2.13.5

In addition, please compile the deformable attention module following Deformable DETR:

cd ./models/defdetr_utils/models/ops
sh ./make.sh
# unit test (should see all checking is True)
python test.py

Datasets

Please use the links below to download the datasets used in this work. The corresponding checksums can be found here.

We recommend placing all datasets under the changeDet/ directory.

COCO-Inpainted

Download as .tar

coco_inpainted
└───train
│   │   data_split.pkl
│   │   list_of_indices.npy
│   │
│   └───images_and_masks
│   |   │   <index>.png (original coco image)
│   |   │   <index>_mask<id>.png (mask of inpainted objects)
│   |   │   ...
|   |
│   └───inpainted
│   |   │   <index>_mask<id>.png (inpainted image corresponding to the mask with the same name)
│   |   │   ...
|   |
│   └───metadata
│   |   │   <index>.npy (annotations)
│   |   │   ...
│   
└───test
│   └───small
│   │   |   data_split.pkl
│   │   |   list_of_indices.npy
│   │   └───images_and_masks/
│   │   └───inpainted/
│   │   └───metadata/
│   │   └───test_augmentation/
|   |
│   └───medium/
│   └───large/

Kubric-Change

Download as .tar

kubric_change
│   metadata.npy (this is generated automatically the first time you load the dataset)
│   <index>_0.png (image 1)
|   <index>_1.png (image 2)
|   mask_<index>_00000.png (change mask for image 1)
|   mask_<index>_00001.png (change mask for image 2)
|   ...

VIRAT-STD

Download original images using link provided by Jhamtani et al. + Download annotations as .npy.gz

std
│   annotations.npy (ours)
│   <index>.png (provided by Jhamtani et al.)
|   <index>_2.png (provided by Jhamtani et al.)
|   ...

Synthtext-Change

Download original bg images as .tar.gz + Download synthetic text images as .h5.gz

synthtext_change
└───bg_imgs/ (original bg images)
|   | ...
│   synthtext-change.h5 (images with synthetic text we generated)

Real-Change

Download as .zip

RealChange
└───Double/ (view changed images)
|   <index>_left.png
|   <index>_left.json
|   <index>_right.png
|   <index>_right.json
|   ...
└───Single/ (view unchanged images)
|   <index>_left.png
|   <index>_left.json
|   <index>_right.png
|   <index>_right.json
|   ...

Example Usage

Training:

python main.py --method defdetr --gpus 4 --config_file configs/detection_defdetr_res50_6_affine.yml --max_epochs 200

Testing:

python main.py --method defdetr --gpus 4 --config_file configs/detection_defdetr_res50_6_affine.yml --test_from_checkpoint <path>

Demo/Inference:

python demo_single_pair.py --load_weights_from <path_to_checkpoint> --config_file configs/detection_defdetr_res50_6_affine.yml

Acknowledgements

This work is heavily based on The Change You Want to See repository. We thank the authors for their contributions.