Skip to content

[Improve] Improve camera backend and add motor read retries #2152

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
+30 −11
Open

[Improve] Improve camera backend and add motor read retries #2152

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8b4af6e
Fix: Use DirectShow for Windows (more compatible than MSMF)
whats2000 Oct 9, 2025
042f864
Fix: Add the retry for bus sync delay issue
whats2000 Oct 9, 2025
c3b5c25
Feat: Make camera width and height setting adjust from actual size au…
whats2000 Oct 9, 2025
10b836e
Fix: Fix the adjustment during record
whats2000 Oct 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 14 additions & 4 deletions src/lerobot/cameras/opencv/camera_opencv.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -233,15 +233,25 @@ def _validate_width_and_height(self) -> None:

actual_width = int(round(self.videocapture.get(cv2.CAP_PROP_FRAME_WIDTH)))
if not width_success or self.capture_width != actual_width:
raise RuntimeError(
f"{self} failed to set capture_width={self.capture_width} ({actual_width=}, {width_success=})."
logger.warning(
f"{self} failed to set capture_width={self.capture_width} ({actual_width=}, {width_success=}). "
f"Using camera's actual width={actual_width}."
)
self.capture_width = actual_width

actual_height = int(round(self.videocapture.get(cv2.CAP_PROP_FRAME_HEIGHT)))
if not height_success or self.capture_height != actual_height:
raise RuntimeError(
f"{self} failed to set capture_height={self.capture_height} ({actual_height=}, {height_success=})."
logger.warning(
f"{self} failed to set capture_height={self.capture_height} ({actual_height=}, {height_success=}). "
f"Using camera's actual height={actual_height}."
)
self.capture_height = actual_height

# Update output dimensions based on actual capture dimensions and rotation
if self.rotation in [cv2.ROTATE_90_CLOCKWISE, cv2.ROTATE_90_COUNTERCLOCKWISE]:
self.width, self.height = self.capture_height, self.capture_width
else:
self.width, self.height = self.capture_width, self.capture_height
Comment on lines +251 to +254
Copy link

Copilot AI Oct 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] This dimension update logic should be extracted into a separate method to improve code organization and reusability. The logic for determining output dimensions based on rotation could be used elsewhere in the camera handling code.

Suggested change
if self.rotation in [cv2.ROTATE_90_CLOCKWISE, cv2.ROTATE_90_COUNTERCLOCKWISE]:
self.width, self.height = self.capture_height, self.capture_width
else:
self.width, self.height = self.capture_width, self.capture_height
self.width, self.height = self._get_output_dimensions(self.capture_width, self.capture_height, self.rotation)
Returns the output dimensions (width, height) based on the capture dimensions and rotation.
"""
if rotation in [cv2.ROTATE_90_CLOCKWISE, cv2.ROTATE_90_COUNTERCLOCKWISE]:
return capture_height, capture_width
else:
return capture_width, capture_height

Copilot uses AI. Check for mistakes.

@staticmethod
def find_cameras() -> list[dict[str, Any]]:
Expand Down
2 changes: 1 addition & 1 deletion src/lerobot/cameras/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ def get_cv2_backend() -> int:
import cv2

if platform.system() == "Windows":
return cv2.CAP_MSMF # Use MSMF for Windows instead of AVFOUNDATION
return cv2.CAP_DSHOW # Use DirectShow for Windows (more compatible than MSMF)
# elif platform.system() == "Darwin": # macOS
# return cv2.CAP_AVFOUNDATION
else: # Linux and others
Expand Down
2 changes: 1 addition & 1 deletion src/lerobot/robots/so101_follower/so101_follower.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,7 @@ def get_observation(self) -> dict[str, Any]:

# Read arm position
start = time.perf_counter()
obs_dict = self.bus.sync_read("Present_Position")
obs_dict = self.bus.sync_read("Present_Position", num_retry=5)
obs_dict = {f"{motor}.pos": val for motor, val in obs_dict.items()}
dt_ms = (time.perf_counter() - start) * 1e3
logger.debug(f"{self} read state: {dt_ms:.1f}ms")
Expand Down
2 changes: 1 addition & 1 deletion src/lerobot/teleoperators/so101_leader/so101_leader.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@ def setup_motors(self) -> None:

def get_action(self) -> dict[str, float]:
start = time.perf_counter()
action = self.bus.sync_read("Present_Position")
action = self.bus.sync_read("Present_Position", num_retry=5)
action = {f"{motor}.pos": val for motor, val in action.items()}
dt_ms = (time.perf_counter() - start) * 1e3
logger.debug(f"{self} read action: {dt_ms:.1f}ms")
Expand Down