How WASD is derived from the camera pose

[liyaowei-stu]

Wonderful work! How WASD is derived from the camera pose?

[JingyeChen]

Hi yaowei, thanks for your interest in lingbot-world. The wasd operation can be intuitively derived from the camera's extrinsic metric based on pre-defined threshold.

[liyaowei-stu]

Hi Jingye👋, thanks for the clarification. I would like to confirm whether the WASD action only depends on the camera's translation along the z and x axes, i.e., Dolly In/Out: W/S and Truck Left/Right: A/D. Also, could you help me understand what an action like [0,0,1,1] represents, or point me to where the full action definitions are documented?

[sustecher]

The displacement of the camera pose signal has three dimensions: tx, ty, tz.
WASD can only cover two dimensions, so has one dimension been filtered out (e.g., the Z-axis)?
Referring to the sample input in the image below, could you share the logic for determining the threshold?

[liyaowei-stu]

Btw, using ViPE-extracted camera poses as input to LingBot-World-Cam, the generated videos show much slower camera translations and cannot move as quickly as in the original video; any idea what might be causing this?"

[zhw-zhang]

Hi Jingye👋, thanks for the clarification. I would like to confirm whether the WASD action only depends on the camera's translation along the z and x axes, i.e., Dolly In/Out: W/S and Truck Left/Right: A/D. Also, could you help me understand what an action like [0,0,1,1] represents, or point me to where the full action definitions are documented?你好，景业👋，感谢你的澄清。我想确认 WASD 动作是否仅依赖于相机沿 z 轴和 x 轴的平移，即推车进/出：W/S，转向车左右：A/D。另外，你能帮我理解像 [0,0,1,1] 这样的动作代表什么吗？或者指给我哪里有完整的动作定义？

I have the same question. It seems like the action inputs for lingbot-world-base-act aren't in the standard WASD + ↑↓→← format. It's a bit confusing—could you clarify how this part works?

[JingrunDu]

我也有类似的疑问。既然相机外参矩阵已经涵盖了视角移动的信息，那么在实时交互场景下，完全可以直接将 WASD 输入映射为外参矩阵的变化，并将其作为条件来生成视频。为什么还有必要专门训练一套模型参数，将 WASD 作为一个显式的条件输入呢？
I share a similar concern. Since the camera extrinsic matrix already encapsulates information about viewpoint movement, in a real-time interactive scenario, one could simply map WASD inputs directly to changes in the extrinsic matrix and use them as conditions to generate the video. Why is it necessary to train an additional set of model parameters to incorporate WASD as an explicit conditional input?

[BestWhere]

在我的理解以及测试中，LingBot-World-act中的action比如[1,1,1,1]，按照顺序是w、a、s、d（前、左、后、右）。但是貌似模型本身对这些action并不是很敏感，有时候方向让人很困惑:(

[zyjOrz]

I’ve run into similar situations too. When controlling LingBot-Base for actions or camera movement, it often doesn’t follow my instructions very well.

[sustecher]

是不是因为，不同视频的外参的平移量是不一致的，导致无法在推理阶段完全由WASD模拟出外参。
从lingbot的Demo看，像是已经有了一个视频（比如genie3），通过vipe推出相机外参数，控制信号的WASD也是从这个相机外参转换来的。参考视频->外参->WASD->生成视频。
似乎没有看到WASD->外参->生成视频，这样的推理流程。