[🐛 bug report] Vertex Positions Loss Backpropagating Incorrectly

[nauman-tintash]

Summary

When I keep vertex positions buffer as optimization parameters, the loss is apparently being backpropogated incorrectly. Even if I keep the reference image and new image exactly the same, over a few iterations, the vertex positions keep getting updated and the loss keeps increasing.

System configuration

• Platform: Ubuntu 20.4
• Compiler: CMAKE 3.16.3
• Python version: 3.8.5
• Mitsuba 2 version: mitsuba-2
• Branch : pathreparam-optix7
• Compiled variants:
  • scalar_rgb
  • gpu_autodiff_rgb

Description

Even if my source and target scene is exactly the same, if I keep the vertex positions buffer as differentiable parameter, the loss keep getting increased. Following is a sample of losses and corresponding images for the first 4 iterations from one example.

Reference Render:

Iteration 1 : (Loss = 4.21295e-16)

Iteration 2 : (Loss = 0.000416789)

Iteration 3 : (Loss = 0.0170517)

Following is my relevant code snippet :

scene = load_file('bagel/scene.xml')
image_ref = render(scene, spp=2)

params = traverse(scene)
params.keep(['object.vertex_positions_buf'])
opt = Adam(params, lr=.02)

for it in range(iterations):
      image = render(scene, spp=2)
      ob_val = ek.hsum(ek.sqr(image - image_ref)) / len(image)
      ek.backward(ob_val)
      opt.step()

[Speierers]

Hi @nauman-tintash ,

This is strange indeed. At iteration 1, the loss you are getting is very small and I am surprised the first gradient step causes such a difference on iteration 2. Could it be that your learning rate is too high?

Also it would be useful to verify that the two renders are indeed identical (e.g. use the same random numbers), otherwise a noise difference in the primal images would cause non-zero gradients. You could for instance run the following:

scene = load_file('bagel/scene.xml')
image_ref = render(scene, spp=2)
ob_val = ek.hsum(ek.sqr(image_ref - image_ref)) / len(image_ref )

to check whether ob_val ==0.0 or if this is simply due to numerical imprecisions.

Another thing that comes to my mind is that the pathreparam integrator produces biased gradients (see publication for more info on that). This bias might affect such a simple case.

One last thing: in the provided optimization examples using the pathreparam integrator, we usually constraint the optimization problem (e.g. optimize translation, displacement along normals,...) to avoid creating invalid meshes with overlapping triangles. It looks like you 3rd iteration is already having this issue.

[HsiangYangChu]

I have the same system configuration with @nauman-tintash, but I meet this error:

Traceback (most recent call last):
  File "sphere2cow.py", line 101, in <module>
    ek.backward(ob_val)
RuntimeError: set_gradient(): no gradients are associated with this variable (a prior call to requires_gradient() is required.)

and my code is:

import torch
import enoki as ek
import mitsuba 
mitsuba.set_variant("gpu_autodiff_rgb")

from mitsuba.core import Thread, Bitmap, Struct, ScalarTransform4f
from mitsuba.core.xml import load_string
from mitsuba.python.util import traverse
from mitsuba.python.autodiff import render, write_bitmap, render_torch, Adam

sphere = """
<scene version='2.0.0'>
    <integrator type="path">
        <integer name="max_depth" value="3"/>
    </integrator>
    <emitter type="envmap" id="my_envmap">
        <string name="filename" value="museum.exr"/>
    </emitter>
    <shape type="obj">
        <string name="filename" value="./sphere.obj"/>
        <bsdf type="conductor"/>
    </shape>
    <sensor type="perspective">
        <transform name="to_world">
            <lookat origin="5, 5, 5"
                    target="0, 0, 0"
                    up="0, 1, 0"/>
        </transform>

        <float name="fov" value="40"/>

        <film type="hdrfilm">
            <string name="pixel_format" value="rgb"/>
            <integer name="width" value="512"/>
            <integer name="height" value="512"/>
        </film>

        <sampler type="independent">
            <integer name="sample_count" value="10"/>
        </sampler>
    </sensor>
</scene>
"""

cow = """
<scene version='2.0.0'>
    <integrator type="path">
        <integer name="max_depth" value="3"/>
    </integrator>
    <emitter type="envmap" id="my_envmap">
        <string name="filename" value="museum.exr"/>
    </emitter>
    <shape type="obj">
        <string name="filename" value="./cow_mesh/cow.obj"/>
        <bsdf type="conductor"/>
    </shape>
    <sensor type="perspective">
        <transform name="to_world">
            <lookat origin="5, 5, 5"
                    target="0, 0, 0"
                    up="0, 1, 0"/>
        </transform>

        <float name="fov" value="40"/>

        <film type="hdrfilm">
            <string name="pixel_format" value="rgb"/>
            <integer name="width" value="512"/>
            <integer name="height" value="512"/>
        </film>

        <sampler type="independent">
            <integer name="sample_count" value="10"/>
        </sampler>
    </sensor>
</scene>
"""

scene = load_string(cow)
image_ref = render(scene, spp=16)


scene = load_string(sphere)
camera = scene.sensors()[0]
crop_size = camera.film().crop_size()

params = traverse(scene)
print(params)
# varToOptimize = 'OBJMesh.vertex_positions_buf'
# ek.requires_gradient(params['OBJMesh.vertex_positions_buf'])
# params.set_dirty(varToOptimize)
params.keep(['OBJMesh.vertex_positions_buf'])
# params.update()
opt = Adam(params, lr=.02)

for it in range(100):
    # image = render(scene, optimizer=opt, unbiased=True, spp=8)
    image = render(scene, spp=2)
    write_bitmap('./output/out_%03i.png' % it, image, crop_size)
    ob_val = ek.hsum(ek.sqr(image - image_ref)) / len(image)
    ek.backward(ob_val)
    opt.step()

[Speierers]

You will most likely need to use the pathreparam integrator (here) in order to optimize geometry in Mitsuba 2. This integrator will account for visibility discontinuities when differentiating the light transport simulation. Maybe this will solve you problem.

[HsiangYangChu]

You will most likely need to use the pathreparam integrator (here) in order to optimize geometry in Mitsuba 2. This integrator will account for visibility discontinuities when differentiating the light transport simulation. Maybe this will solve you problem.
I've done what you said, but it's still:

Traceback (most recent call last):
  File "sphere2cow.py", line 104, in <module>
    ek.backward(ob_val)
RuntimeError: set_gradient(): no gradients are associated with this variable (a prior call to requires_gradient() is required.)

This is my new code:

import torch
import enoki as ek
import mitsuba 
mitsuba.set_variant("gpu_autodiff_rgb")

from mitsuba.core import Thread, Bitmap, Struct, ScalarTransform4f
from mitsuba.core.xml import load_string
from mitsuba.python.util import traverse
from mitsuba.python.autodiff import render, write_bitmap, render_torch, Adam

sphere = """
<scene version='2.2.1'>
    <integrator type="pathreparam">
        <integer name="max_depth" value="3"/>
    </integrator>
    <emitter type="envmap" id="my_envmap">
        <string name="filename" value="museum.exr"/>
    </emitter>
    <shape type="obj">
        <string name="filename" value="./sphere.obj"/>
        <bsdf type="diffuse"/>
    </shape>
    <sensor type="perspective">
        <transform name="to_world">
            <lookat origin="5, 5, 5"
                    target="0, 0, 0"
                    up="0, 1, 0"/>
        </transform>

        <float name="fov" value="40"/>

        <film type="hdrfilm">
            <string name="pixel_format" value="rgb"/>
            <integer name="width" value="512"/>
            <integer name="height" value="512"/>
        </film>

        <sampler type="independent">
            <integer name="sample_count" value="10"/>
        </sampler>
    </sensor>
</scene>
"""

cow = """
<scene version='2.2.1'>
    <integrator type="path">
        <integer name="max_depth" value="2"/>
    </integrator>
    <emitter type="envmap" id="my_envmap">
        <string name="filename" value="museum.exr"/>
    </emitter>
    <shape type="obj">
        <string name="filename" value="./cow_mesh/cow.obj"/>
        <bsdf type="conductor"/>
    </shape>
    <sensor type="perspective">
        <transform name="to_world">
            <lookat origin="5, 5, 5"
                    target="0, 0, 0"
                    up="0, 1, 0"/>
        </transform>

        <float name="fov" value="40"/>

        <film type="hdrfilm">
            <string name="pixel_format" value="rgb"/>
            <integer name="width" value="512"/>
            <integer name="height" value="512"/>
        </film>

        <sampler type="independent">
            <integer name="sample_count" value="10"/>
        </sampler>
    </sensor>
</scene>
"""

scene = load_string(cow)
image_ref = render(scene, spp=2)


scene = load_string(sphere)
camera = scene.sensors()[0]
crop_size = camera.film().crop_size()

params = traverse(scene)
print(params)
# varToOptimize = 'OBJMesh.bsdf.reflectance.value'
# varToOptimize = ['OBJMesh.bsdf.specular_reflectance.value', 'OBJMesh.bsdf.eta.value', 'OBJMesh.bsdf.k.value']
varToOptimize = 'OBJMesh.vertex_positions_buf'
# ek.requires_gradient(varToOptimize)
# params.set_dirty(varToOptimize)
params.keep(varToOptimize)
# params[varToOptimize] = [.9, .9, .9]
# params.update()
opt = Adam(params, lr=.02)

for it in range(100):
    # image = render(scene, optimizer=opt, unbiased=True, spp=8)
    image = render(scene, spp=2)
    write_bitmap('./output/out_%03i.png' % it, image, crop_size)
    ob_val = ek.hsum(ek.sqr(image - image_ref)) / len(image)
    ek.backward(ob_val)
    opt.step()

[Speierers]

Does it works with this new code?

[HsiangYangChu]

Does it works with this new code?

No! :(

Traceback (most recent call last):
  File "sphere2cow.py", line 104, in <module>
    ek.backward(ob_val)
RuntimeError: set_gradient(): no gradients are associated with this variable (a prior call to requires_gradient() is required.)

The log:

(mt) hsiangyangchu@zhuxiangyang-System-Product-Name:~/work/mitsuba-test/testgeo$ python sphere2cow_copy.py 
2021-10-11 15:53:43 INFO  main  [optix_api.cpp:56] Dynamic loading of the Optix library ..
2021-10-11 15:53:43 INFO  main  [xml.cpp:353] "<string>": in-memory version upgrade (v2.2.1 -> v2.1.0) ..
2021-10-11 15:53:43 INFO  main  [PluginManager] Loading plugin "plugins/path.so" ..
2021-10-11 15:53:43 INFO  main  [PluginManager] Loading plugin "plugins/envmap.so" ..
2021-10-11 15:53:43 INFO  main  [PluginManager] Loading plugin "plugins/uniform.so" ..
2021-10-11 15:53:43 INFO  main  [PluginManager] Loading plugin "plugins/conductor.so" ..
2021-10-11 15:53:43 INFO  main  [PluginManager] Loading plugin "plugins/obj.so" ..
2021-10-11 15:53:43 INFO  main  [PluginManager] Loading plugin "plugins/hdrfilm.so" ..
2021-10-11 15:53:43 INFO  main  [PluginManager] Loading plugin "plugins/gaussian.so" ..
2021-10-11 15:53:43 INFO  main  [PluginManager] Loading plugin "plugins/independent.so" ..
2021-10-11 15:53:43 INFO  main  [PluginManager] Loading plugin "plugins/perspective.so" ..
2021-10-11 15:53:43 INFO  main  [Scene] Building scene in OptiX ..
2021-10-11 15:53:43 INFO  main  [xml.cpp:353] "<string>": in-memory version upgrade (v2.2.1 -> v2.1.0) ..
2021-10-11 15:53:43 INFO  main  [PluginManager] Loading plugin "plugins/pathreparam.so" ..
2021-10-11 15:53:43 INFO  main  [PluginManager] Loading plugin "plugins/diffuse.so" ..
2021-10-11 15:53:43 INFO  main  [Scene] Building scene in OptiX ..
Traceback (most recent call last):
  File "sphere2cow_copy.py", line 97, in <module>
    ek.backward(ob_val)
RuntimeError: set_gradient(): no gradients are associated with this variable (a prior call to requires_gradient() is required.)

[Speierers]

It looks like the rendered image doesn't depend on the parameters that you are trying to optimize at all. Please check that the geometry OBJMesh is in sight.

Also you could use ek.foward() to produce a gradient image, this one shouldn't be black (see the tutorial in the doc).

[HsiangYangChu]

It looks like the rendered image doesn't depend on the parameters that you are trying to optimize at all. Please check that the geometry OBJMesh is in sight.

Also you could use ek.foward() to produce a gradient image, this one shouldn't be black (see the tutorial in the doc).

You mean the forward_diff.py in the doc?

[Speierers]

yes

[HsiangYangChu]

yes

I have a try, the result is:

(mt) hsiangyangchu@zhuxiangyang-System-Product-Name:~/work/mitsuba-test/testgeo$ python sphere2cow_forward.py 
2021-10-11 16:38:14 INFO  main  [optix_api.cpp:56] Dynamic loading of the Optix library ..
2021-10-11 16:38:14 INFO  main  [xml.cpp:353] "<string>": in-memory version upgrade (v2.2.1 -> v2.1.0) ..
2021-10-11 16:38:14 INFO  main  [PluginManager] Loading plugin "plugins/pathreparam.so" ..
2021-10-11 16:38:14 INFO  main  [PluginManager] Loading plugin "plugins/envmap.so" ..
2021-10-11 16:38:14 INFO  main  [PluginManager] Loading plugin "plugins/uniform.so" ..
2021-10-11 16:38:14 INFO  main  [PluginManager] Loading plugin "plugins/diffuse.so" ..
2021-10-11 16:38:14 INFO  main  [PluginManager] Loading plugin "plugins/obj.so" ..
2021-10-11 16:38:14 INFO  main  [PluginManager] Loading plugin "plugins/hdrfilm.so" ..
2021-10-11 16:38:14 INFO  main  [PluginManager] Loading plugin "plugins/gaussian.so" ..
2021-10-11 16:38:14 INFO  main  [PluginManager] Loading plugin "plugins/independent.so" ..
2021-10-11 16:38:14 INFO  main  [PluginManager] Loading plugin "plugins/perspective.so" ..
2021-10-11 16:38:14 INFO  main  [Scene] Building scene in OptiX ..
Traceback (most recent call last):
  File "sphere2cow_forward.py", line 52, in <module>
    image = render(scene, spp=8)
  File "/home/hsiangyangchu/mitsuba2/build/dist/python/mitsuba/python/autodiff.py", line 203, in render
    image = _render_helper(scene, spp=spp, sensor_index=sensor_index)
  File "/home/hsiangyangchu/mitsuba2/build/dist/python/mitsuba/python/autodiff.py", line 43, in _render_helper
    spec, mask, aovs = scene.integrator().sample(scene, sampler, rays)
RuntimeError: cuda_malloc(): out of memory!

My code is:

import torch
import enoki as ek
import mitsuba 
mitsuba.set_variant("gpu_autodiff_rgb")

from mitsuba.core import Float
from mitsuba.core.xml import load_string
from mitsuba.python.util import traverse
from mitsuba.python.autodiff import render, write_bitmap, render_torch, Adam

sphere = """
<scene version='2.2.1'>
    <integrator type="pathreparam">
        <integer name="max_depth" value="3"/>
    </integrator>
    <emitter type="envmap" id="my_envmap">
        <string name="filename" value="museum.exr"/>
    </emitter>
    <shape type="obj">
        <string name="filename" value="./sphere.obj"/>
        <bsdf type="diffuse"/>
    </shape>
    <sensor type="perspective">
        <transform name="to_world">
            <lookat origin="5, 5, 5"
                    target="0, 0, 0"
                    up="0, 1, 0"/>
        </transform>

        <float name="fov" value="40"/>

        <film type="hdrfilm">
            <string name="pixel_format" value="rgb"/>
            <integer name="width" value="512"/>
            <integer name="height" value="512"/>
        </film>

        <sampler type="independent">
            <integer name="sample_count" value="10"/>
        </sampler>
    </sensor>
</scene>
"""

scene = load_string(sphere)

params = traverse(scene)

param_0 = params['OBJMesh.vertex_positions_buf']
ek.set_requires_gradient(param_0)

image = render(scene, spp=8)

ek.set_gradient(param_0, [1, 1, 1], backward=False)

Float.forward()

image_grad = ek.gradient(image)

crop_size = scene.sensors()[0].film().crop_size()
fname = 'out.png'
write_bitmap(fname, image_grad, crop_size)
print('Wrote forward differentiation image to: {}'.format(fname))

[HsiangYangChu]

yes

Could you provide me with an example of geometry optimization?

[Speierers]

Please take a look at this one

[HsiangYangChu]

Please take a look at this one

It seems that the folder scene_folder is lost.

[binorchen]

Hello @HsiangYangChu @Speierers ,

Have you solved this problem? I am suffering exactly the same problem.

Thank you!