how to reconstruct geometry model according to given image by mitsuba

[chongxian]

• [❔ other question]

according to issue #26, I find the PR #44, and I download from https://github.com/loubetg/mitsuba2-reparam-tests to try the Geometric differentiation, but I can't let the empty_cube become the smooth_empty_cube by the inverse rendering. the paper "Reparameterizing Discontinuous Integrands for Differentiable Rendering" is very good,but the figure 8 is my attention, how I do can make the same result.
and the project offer some Optimization tests like optim_vertices.py which I try to make the empty_cube become the smooth_empty_cube.

[Speierers]

Hi @chongxian ,

Inverse rendering for geometry reconstruction is hard and require proper parameters tuning to get the optimization to converge to a reasonable solution. What is you current approach? I will need more information to be able to help you with this.

[Speierers]

Hi @chongxian ,

I will close this as it isn't really an issue. Please let me know if you have any other questions.

[binorchen]

Hello @Speierers and @chongxian ,

I am testing the same code optim_pose.py from https://github.com/loubetg/mitsuba2-reparam-tests

import numpy as np
import os
import mitsuba
import enoki as ek

mts_variant = 'rgb'
mitsuba.set_variant('gpu_autodiff_' + mts_variant)

from mitsuba.core import Transform4f, Bitmap, Float, Vector3f
from mitsuba.core.xml import load_string
from mitsuba.python.util import traverse
from mitsuba.python.autodiff import render, write_bitmap, SGD

# This test optimizes a colorful texture from a reference image.

path = "output/optim_pose/"

def make_scene(integrator, spp):
    return load_string("""
        <?xml version="1.0"?>
        <scene version="2.0.0">
            {integrator}
            <sensor type="perspective">
                <string name="fov_axis" value="smaller"/>
                <float name="near_clip" value="0.1"/>
                <float name="far_clip" value="2800"/>
                <float name="focus_distance" value="1000"/>
                <transform name="to_world">
                    <lookat origin="0, 0, 9" target="0, 0, 0" up="0, 1, 0"/>
                </transform>
                <float name="fov" value="15"/>
                <sampler type="independent">
                    <integer name="sample_count" value="{spp}"/>
                </sampler>
                <film type="hdrfilm">
                    <integer name="width" value="250"/>
                    <integer name="height" value="250"/>
                    <rfilter type="box" >
                        <float name="radius" value="0.5"/>
                    </rfilter>
                </film>
            </sensor>
            <shape type="obj" id="smooth_area_light_shape">
                <transform name="to_world">
                    <rotate x="1" angle="180"/>
                    <translate x="10.0" y="0.0" z="15.0"/>
                </transform>
                <string name="filename" value="data/meshes/xy_plane.obj"/>
                <emitter type="smootharea" id="smooth_area_light">
                    <spectrum name="radiance" value="100"/>
                </emitter>
            </shape>
            <shape type="obj" id="object">
                <string name="filename" value="data/meshes/smooth_empty_cube.obj"/>
                <bsdf type="diffuse" id="objectmat">
                </bsdf>
                <transform name="to_world">
                    <translate z="0.0"/>
                </transform>    
            </shape>
            <shape type="obj" id="planemesh">
                <string name="filename" value="data/meshes/xy_plane.obj"/>
                
                <bsdf type="diffuse" id="planemat">
                </bsdf>
                
                <transform name="to_world">
                    <translate z="-1"/>
                    <scale value="2.0"/>
                </transform>    
            </shape>
        </scene>
    """.format(integrator=integrator, spp=spp))

# Define integrators for this test

path_str =  """<integrator type="path">
                   <integer name="max_depth" value="2"/>
               </integrator>"""

path_reparam_str =  """<integrator type="pathreparam">
                           <integer name="max_depth" value="2"/>
                       </integrator>"""

if not os.path.isdir(path):
    os.makedirs(path)

# Render the target image

scene = make_scene(path_str, 32);
fsize = scene.sensors()[0].film().size()
image_ref = render(scene)
write_bitmap(path + "out_ref.exr", image_ref, fsize)
print("Writing " + path + "out_ref.exr")

# Define the differentiable scene for the optimization

del scene
scene = make_scene(path_reparam_str, 16);

properties = traverse(scene)

key = "object.vertex_positions"
properties.keep([key])
initial_positions = properties[key] + Vector3f(0.1,0.3,0.1)

P_translation = Vector3f(0.0);
ek.set_requires_gradient(P_translation)

params_optim = {"P_translation": P_translation}

# Instantiate an optimizer
opt = SGD(params_optim, lr=5.0, momentum=0.5)

for i in range(100):

    # Update the scene 
    print("P_translation: ", params_optim["P_translation"])
    properties[key] = Transform4f.translate(params_optim["P_translation"]).transform_point(initial_positions);
    properties.update()

    image = render(scene)

    image_np = image.numpy().reshape(fsize[1], fsize[0], 3)
    output_file = path + 'out_%03i.exr' % i
    print("Writing image %s" % (output_file))
    Bitmap(image_np).write(output_file)

    # Objective function
    loss = ek.hsum(ek.hsum(ek.sqr(image - image_ref))) / (fsize[1]*fsize[0]*3)
    print("Iteration %i: loss=%f" % (i, loss[0]))

    ek.backward(loss)
    opt.step()

But got:

terminate called after throwing an instance of 'std::runtime_error'
  what():  set_gradient(): no gradients are associated with this variable (a prior call to requires_gradient() is required.)
Aborted

on line:

ek.backward(loss)

Could you provide any hint?
Thanks a lot for your help!

[Speierers]

I suspect you might need to use ek.ravel and ek.unravel to convert the initial_position from a linear "flat" buffer to a Vector3f, and back.

I suggest you take a look at the rendered image at the first iteration after the update. It might be that the mesh is completely broken (e.g. out of frame) because the vertex positions were not handled properly.

[binorchen]

Hi @Speierers, Thanks for your prompt reply! Indeed, ek.ravel and ek.unravel are needed. I found a better official demo and it works well now

[Speierers]

Happy to hear this!