Merge branch 'main' of https://github.com/compphoto/Intrinsic into main

Author: CCareaga

LICENSE

@@ -0,0 +1,17 @@
+Copyright 2021, Chris Careaga, Yağız Aksoy, Computational Photography Laboratory. All rights reserved.
+
+This software is for academic use only. A redistribution of this 
+software, with or  without modifications, has to be for academic 
+use only, while giving the appropriate credit to the original 
+authors of the software. The methods implemented as a part of 
+this software may be covered under patents or patent applications.
+
+THIS SOFTWARE IS PROVIDED BY THE AUTHOR ''AS IS'' AND ANY EXPRESS OR IMPLIED
+WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR
+CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
+NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
+ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

README.md

@@ -1,50 +1,118 @@
 # Intrinsic Image Decomposition via Ordinal Shading
-Code for the paper: Intrinsic Image Decomposition via Ordinal Shading, [Chris Careaga](https://ccareaga.github.io/) and [Yağız Aksoy](https://yaksoy.github.io) , ACM Transactions on Graphics, 2023 
-### [Project Page](https://yaksoy.github.io/intrinsic) | [Paper]() | [Video]() | [Supplementary]() | [Data]()
-
-In this work, we achieve high-resolution intrinsic
-decomposition by breaking the problem into two parts. First, we present a
-dense ordinal shading formulation using a shift- and scale-invariant loss in
-order to estimate ordinal shading cues without restricting the predictions to
-obey the intrinsic model. We then combine low- and high-resolution ordinal
-estimations using a second network to generate a shading estimate with both
-global coherency and local details. We encourage the model to learn an ac-
-curate decomposition by computing losses on the estimated shading as well
-as the albedo implied by the intrinsic model. We develop a straightforward 
-method for generating dense pseudo ground truth using our model’s pre-
-dictions and multi-illumination data, enabling generalization to in-the-wild
-imagery.
+Code for the paper: Intrinsic Image Decomposition via Ordinal Shading, [Chris Careaga](https://ccareaga.github.io/) and [Yağız Aksoy](https://yaksoy.github.io), ACM Transactions on Graphics, 2023 
+### [Project Page](https://yaksoy.github.io/intrinsic) | [Paper](https://yaksoy.github.io/papers/TOG23-Intrinsic.pdf) | [Video](https://www.youtube.com/watch?v=pWtJd3hqL3c) | [Supplementary](https://yaksoy.github.io/papers/TOG23-Intrinsic-Supp.pdf) | [Data](https://github.com/compphoto/MIDIntrinsics)
+
+We propose a method for generating high-resolution intrinsic image decompositions, for in-the-wild images. Our method relies on a carefully formulated ordinal shading representation, and real-world supervision from multi-illumination data in order to predict highly accurate albedo and shading. 
+
+[![YouTube Video](./figures/thumbnail.jpg)](https://www.youtube.com/watch?v=pWtJd3hqL3c)
+
+
+Try out our pipeline on your own images! [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/compphoto/Intrinsic/blob/main/intrinsic_inference.ipynb)
+
+## Method
+The inherently under-constrained and scale-invariant nature of the intrinsic decomposition makes it a challenging problem. 
+Shading, which represents complex interactions in the scene, is difficult for neural networks to predict. 
+Compounded by the scarcity of dense ground-truth data, state-of-the-art models fail at high resolutions in real-world scenarios.
+
+![intro_itw_comp_avo](./figures/intro_itw_comp_avo.png)
+
+Our method focuses on generating high-resolution shading estimations, rather than attempting to estimate shading and albedo separately. 
+Since shading values are unbounded, we develop a representation of shading values called "inverse shading" which maps the shading values into the zero-one range.
+This creates a balanced distribution of values in a well-defined range that is desirable for training neural networks.
+
+![ordinal_shd_rep](./figures/ordinal_shd_rep.jpg)
+
+Rather than directly regressing the inverse shading values, we relax the problem and aim to predict *ordinal* shading values.
+To do this, we train our network using shift- and scale-invariant loss functions. 
+This simplifies the task of shading estimation as the model does not need to estimate precise values that satisfy the core intrinsic decomposition model
+
+![ord_behavior_itw](./figures/ord_behavior_itw.png)
+
+Our ordinal estimations exhibit specific behaviors at different resolutions. 
+At low resolutions, the model can generate globally coherent predictions, but the outputs lack details.
+At high resolutions, the model can predict fine local details, but at the cost of global coherency. 
+
+![pool_table](./figures/pool_table.png)
+
+To generate a final shading estimation we combine two ordinal estimations, at low and high resolutions, with the input image and send them through a second network.
+We use the final shading estimation, and the input image in order to compute our estimated albedo. This allows us to compute losses on both shading and albedo while
+using only a single network.
+
+![network_pipeline_circles](./figures/network_pipeline_circles.jpg)
+
+We train our method on multiple rendered datasets. In order to generate real-world supervision for our method we use multi-illumination data. 
+Using our pipeline we estimate the albedo for each image in a given multi-illumination scene. By taking the median across these albedo estimations, small errors are removed resulting in a single accurate albedo.
+We use these 25,000 pseudo-ground-truth pairs as training data and continue training our pipeline.
+
+![multi_illum_examples](./figures/multi_illum_examples.png)
+
+Our method can be used for complex image editing tasks such as recoloring and relighting
+
+![yellow_chair](./figures/yellow_chair.png)
+
 
 ## Setup
 Depending on how you would like to use the code in this repository there are two options to setup the code.
 In either case, you should first create a fresh virtual environment (`python3 -m venv intrinsic_env`) and start it (`source intrinsic_env/bin/activate`)
 
-#### Option 1
-If you would like to download the repository to run and make changes you can simply clone the repo:
+You can install this repository as a package using `pip`:
 ```
 git clone https://github.com/compphoto/Intrinsic
 cd Intrinsic
+pip install .
 ```
-then pip install all the dependencies of the repo:
-```
-pip install -r requirements.txt 
-```
-
-#### Option 2
-Alternatively, you can install this repository as a package using `setup.py`:
-```
-git clone https://github.com/compphoto/Intrinsic
-cd Intrinsic
-python setup.py
-```
+If you want to make changes to the code and have it reflected when you import the package use `pip install --editable`
 Or perform the same action without cloning the code using:
 ```
-pip install https://github.com/compphoto/Intrinsic/archive/master.zip
+pip install https://github.com/compphoto/Intrinsic/archive/main.zip
 ```
-This will allow you to import the repository as a python package, and use our pipeline as part of your codebase.
+This will allow you to import the repository as a Python package, and use our pipeline as part of your codebase.
 
 ## Inference
 To run our pipeline on your own images you can use the decompose script:
+```python
+from chrislib.general import view, tile_imgs, view_scale, uninvert
+from chrislib.data_util import load_image
+
+from intrinsic.pipeline import run_pipeline
+from intrinsic.model_util import load_models
+
+# load the models from the given paths
+models = load_models('final_weights.pt')
+
+# load an image (np float array in [0-1])
+image = load_image('/path/to/input/image')
+
+# run the model on the image using R_0 resizing
+results = run_pipeline(
+    models,
+    image,
+    resize_conf=0.0,
+    maintain_size=True
+)
+
+albedo = results['albedo']
+inv_shd = results['inv_shading']
+
+# compute shading from inverse shading
+shading = uninvert(inv_shd)
+
 ```
+This will run our pipeline and output the linear albedo and shading. You can run this in your browser as well! [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/compphoto/Intrinsic/blob/main/intrinsic_inference.ipynb)
+
+## Citation
 
 ```
+@ARTICLE{careagaIntrinsic,
+  author={Chris Careaga and Ya\u{g}{\i}z Aksoy},
+  title={Intrinsic Image Decomposition via Ordinal Shading},
+  journal={ACM Trans. Graph.},
+  year={2023},
+}
+```
+
+## License
+
+This implementation is provided for academic use only. Please cite our paper if you use this code or any of the models. 
+
+The methodology presented in this work is safeguarded under intellectual property protection. For inquiries regarding licensing opportunities, kindly reach out to SFU Technology Licensing Office &#60;tlo_dir <i>ατ</i> sfu <i>δøτ</i> ca&#62; and Dr. Yağız Aksoy &#60;yagiz <i>ατ</i> sfu <i>δøτ</i> ca&#62;.

figures/avocado.png

@@ -2,13 +2,12 @@
 from altered_midas.midas_net import MidasNet
 from altered_midas.midas_net_custom import MidasNet_small
 
-def load_models(ord_path, iid_path, device='cuda'):
+def load_models(path, device='cuda'):
     """Load the ordinal network and the intrinsic decomposition network
        into a dictionary that can be used to run our pipeline
 
     params:
-        ord_path (str): the path to the weights file for the ordinal model
-        iid_path (str): the path to the weights file for the intrinsic decomposition model
+        path (str or list): the path to the combined weights file, or to each individual weights file (ordinal first, then iid)
         device (str) optional: the device to run the model on (default "cuda")
 
     returns:
@@ -18,13 +17,27 @@ def load_models(ord_path, iid_path, device='cuda'):
     """
     models = {}
 
+    if isinstance(path, list):
+        ord_state_dict = torch.load(path[0])
+        iid_state_dict = torch.load(path[1])
+    else:
+        if path == 'paper_weights':
+            combined_dict = torch.hub.load_state_dict_from_url('https://github.com/compphoto/Intrinsic/releases/download/v1.0/final_weights.pt', map_location=device, progress=True)
+        elif path == 'rendered_only':
+            combined_dict = torch.hub.load_state_dict_from_url('https://github.com/compphoto/Intrinsic/releases/download/v1.0/rendered_only_weights.pt', map_location=device, progress=True)
+        else:
+            combined_dict = torch.load(path)
+
+        ord_state_dict = combined_dict['ord_state_dict']
+        iid_state_dict = combined_dict['iid_state_dict']
+
     ord_model = MidasNet()
-    ord_model.load_state_dict(torch.load(ord_path))
+    ord_model.load_state_dict(ord_state_dict)
     ord_model.eval()
     ord_model = ord_model.to(device)
 
     iid_model = MidasNet_small(exportable=False, input_channels=5, output_channels=1)
-    iid_model.load_state_dict(torch.load(iid_path))
+    iid_model.load_state_dict(iid_state_dict)
     iid_model.eval()
     iid_model = iid_model.to(device)
 

figures/intro_itw_comp_avo.png

@@ -0,0 +1,142 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "gpuType": "T4"
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    },
+    "accelerator": "GPU"
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Make sure to set the runtime to GPU: Runtime -> Change runtime type -> T4 GPU\n",
+        "\n",
+        "You can upload your own images, then change the relevant code cells to load it and send it through the model."
+      ],
+      "metadata": {
+        "id": "-lKPW1wZEulh"
+      }
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "_cPkUxN-ikld"
+      },
+      "outputs": [],
+      "source": [
+        "# install the intrinsic decomposition repo from github\n",
+        "!pip install https://github.com/compphoto/Intrinsic/archive/main.zip"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import torch\n",
+        "\n",
+        "# import some helper functions from chrislib (will be installed by the intrinsic repo)\n",
+        "from chrislib.general import show, view, uninvert\n",
+        "from chrislib.data_util import load_image\n",
+        "\n",
+        "# import model loading and running the pipeline\n",
+        "from intrinsic.pipeline import run_pipeline\n",
+        "from intrinsic.model_util import load_models"
+      ],
+      "metadata": {
+        "id": "J0gn82ZSjomn"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# download the pretrained weights and return the model (may take a bit to download weights)\n",
+        "intrinsic_model = load_models('paper_weights')"
+      ],
+      "metadata": {
+        "id": "Ap3HubpwC_KG"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# load an example image from the github repo\n",
+        "torch.hub.download_url_to_file('https://raw.githubusercontent.com/compphoto/Intrinsic/main/figures/avocado.png', 'avo.png')"
+      ],
+      "metadata": {
+        "id": "m_NYfDx0AhTw"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# load the image to run through the pipeline\n",
+        "img = load_image('/content/avo.png')"
+      ],
+      "metadata": {
+        "id": "ALb4Pjfvj-MU"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# run the image through the pipeline (use R0 resizing dicussed in the paper)\n",
+        "result = run_pipeline(\n",
+        "    intrinsic_model,\n",
+        "    img,\n",
+        "    resize_conf=0.0,\n",
+        "    maintain_size=True,\n",
+        "    linear=False,\n",
+        "    device='cuda'\n",
+        ")"
+      ],
+      "metadata": {
+        "id": "QW0TiFypkOj-"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# convert the inverse shading to regular shading for visualization\n",
+        "shd = uninvert(result['inv_shading'])\n",
+        "alb = result['albedo']"
+      ],
+      "metadata": {
+        "id": "XpYY2MNjkp2f"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# show the result (gamma corrects the linear intrinsic components and scales to [0-1])\n",
+        "show([img, view(shd), view(alb)], size=(20, 7))"
+      ],
+      "metadata": {
+        "id": "8KKbyoVLki9s"
+      },
+      "execution_count": null,
+      "outputs": []
+    }
+  ]
+}