I can't get proper image when I put camera intrinsic matrix in FovPerspectiveCamera

[HATAEWOOK]

Hi, I'm trying to render 3d hand image using verts, faces tensors from MANO hand model.

# verts : [bs, 778, 3] faces : [bs, 1538,3] 
verts_rgb = torch.full([1,vert.shape[0], vert.shape[1], 3], 0.5, device=vert.device, requires_grad=True)
textures = TexturesVertex(verts_features=verts_rgb.squeeze().to(vert.device))
mesh = Meshes(verts=vert, faces=faces, textures=textures)

So, I can get hand 3d model image through FovPerspectiveCameras like this

device = torch.device("cuda:0")
R, T = look_at_view_transform(dist=250)
R = R.repeat(10, 1, 1)
T = T.repeat(10, 1)
camera = FoVPerspectiveCameras(zfar = 300, device=device, T=T[None, 0, ...], R=R[None, 0, ...]) 

lights = PointLights(device=device, location=[[0.0, 0.0, -3.0]])
raster_settings = RasterizationSettings(
    image_size=224, 
    blur_radius=0.0, 
    faces_per_pixel=1, 
)

renderer = MeshRenderer(
    rasterizer=MeshRasterizer(
        cameras=camera, 
        raster_settings=raster_settings
    ),
    shader=SoftPhongShader(
        device=device, 
        cameras=camera,
        lights=lights
    )
)

target_images = renderer(mesh, cameras=camera, lights=lights)
plt.imshow(target_images[0,:,:,:3].detach().cpu().numpy())
plt.show()

Here is the result.

And, I have camera intrinsic parameters Ks

K = torch.tensor([[[388.0, 0.0, 112.0], [0.0, 388.0, 112.0], [0.0, 0.0, 1.0]]])
Ks = K.repeat(bs, 1,1)
I = torch.tensor([[1,0,0,0],[0,1,0,0],[0,0,1,0]]).float()
Is = torch.unsqueeze(I,0).repeat(Ks.shape[0],1,1).cuda()
Ps = torch.bmm(Ks, Is)
K = torch.cat([Ps, torch.tensor([0,0,0,1]).unsqueeze(0).repeat(Ps.shape[0],1,1).to(vert.device)], dim=1)

So, I put this intrinsic matrix to camera.

device = torch.device("cuda:0")
R, T = look_at_view_transform(dist=250)
R = R.repeat(10, 1, 1)
T = T.repeat(10, 1)
camera = FoVPerspectiveCameras(zfar = 300, device=device, T=T[None, 0, ...], R=R[None, 0, ...], K=K[None, 0, ...]) 

lights = PointLights(device=device, location=[[0.0, 0.0, -3.0]])
raster_settings = RasterizationSettings(
    image_size=224, 
    blur_radius=0.0, 
    faces_per_pixel=1, 
)

renderer = MeshRenderer(
    rasterizer=MeshRasterizer(
        cameras=camera, 
        raster_settings=raster_settings
    ),
    shader=SoftPhongShader(
        device=device, 
        cameras=camera,
        lights=lights
    )
)

target_images = renderer(mesh, cameras=camera, lights=lights)
plt.imshow(target_images[0,:,:,:3].detach().cpu().numpy())
plt.show()

I got this image. even I adjust the dist of look_at_view_transform I only can get this image or white background image

How can I apply camera intrinsic matrix to renderer camera?

[tomer-grin]

I don't know how do you get the intrinsic matrix.
Is it coming from openCV?
inputting zfar and then the intrinsic matrix doesn't make much sense (zfar will be calculated according to your k-matrix).

You can use:
https://github.com/facebookresearch/pytorch3d/blob/2c64635daa2aa728f35ed4abe41c6942ae8c0d8b/pytorch3d/renderer/camera_conversions.py

[HATAEWOOK]

The intrinsic matrix is the ground truth from the dataset, by the way, I can get an image through the link that you give. Thank you!

[tomer-grin]

Great!
If you can add your code snippet for everyone to see and understand that would be also great :D