Add Api Examples #2204
Add Api Examples #2204
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR adds API examples for various policies and learning algorithms to support an upcoming tutorial release. The changes include refactoring common utility functions into reusable components and creating comprehensive example scripts demonstrating how to use different models (ACT, Diffusion, SmolVLA, PI0), reinforcement learning approaches (HIL-SERL), and asynchronous inference patterns.
Key changes:
- Extracted common observation preparation and action processing utilities to
src/lerobot/policies/utils.py - Added configurable observation dimensions to environment configurations
- Created 12 new tutorial example scripts covering model usage, training, and RL workflows
Reviewed Changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
src/lerobot/policies/utils.py |
New utility functions for observation preprocessing and action conversion |
src/lerobot/utils/control_utils.py |
Refactored to use new utility functions from policies/utils.py |
src/lerobot/scripts/lerobot_record.py |
Updated to use new make_robot_action utility function |
src/lerobot/envs/configs.py |
Added configurable observation height/width parameters |
tests/mocks/mock_robot.py |
Enhanced mock robot with proper motor bus integration |
examples/tutorial/**/*.py |
12 new example scripts demonstrating various use cases |
actor_learner_example.py |
Comprehensive HIL-SERL implementation example |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
| for _ in range(MAX_EPISODES): | ||
| for _ in range(MAX_STEPS_PER_EPISODE): | ||
| obs = robot.get_observation() | ||
| obs_frame = build_inference_frame(obs, dataset_features, device, task=task, robot_type=robot_type) |
There was a problem hiding this comment.
The function call has incorrect parameter order. According to the function signature in utils.py, the correct order should be build_inference_frame(obs, device, dataset_features, task=task, robot_type=robot_type).
| obs_frame = build_inference_frame(obs, dataset_features, device, task=task, robot_type=robot_type) | |
| obs_frame = build_inference_frame(obs, device, dataset_features, task=task, robot_type=robot_type) |
Copilot uses AI. Check for mistakes.
| for _ in range(MAX_EPISODES): | ||
| for _ in range(MAX_STEPS_PER_EPISODE): | ||
| obs = robot.get_observation() | ||
| obs_frame = build_inference_frame(obs, dataset_features, device, task=task, robot_type=robot_type) |
There was a problem hiding this comment.
The function call has incorrect parameter order. According to the function signature in utils.py, the correct order should be build_inference_frame(obs, device, dataset_features, task=task, robot_type=robot_type).
| obs_frame = build_inference_frame(obs, dataset_features, device, task=task, robot_type=robot_type) | |
| obs_frame = build_inference_frame(obs, device, dataset_features, task=task, robot_type=robot_type) |
Copilot uses AI. Check for mistakes.
| for _ in range(MAX_EPISODES): | ||
| for _ in range(MAX_STEPS_PER_EPISODE): | ||
| obs = robot.get_observation() | ||
| obs_frame = build_inference_frame(obs, dataset_metadata.features, device) |
There was a problem hiding this comment.
The function call has incorrect parameter order. According to the function signature in utils.py, the correct order should be build_inference_frame(obs, device, dataset_metadata.features).
| obs_frame = build_inference_frame(obs, dataset_metadata.features, device) | |
| obs_frame = build_inference_frame(obs, device, dataset_metadata.features) |
Copilot uses AI. Check for mistakes.
| for _ in range(MAX_EPISODES): | ||
| for _ in range(MAX_STEPS_PER_EPISODE): | ||
| obs = robot.get_observation() | ||
| obs_frame = build_inference_frame(obs, dataset_metadata.features, device) |
There was a problem hiding this comment.
The function call has incorrect parameter order. According to the function signature in utils.py, the correct order should be build_inference_frame(obs, device, dataset_metadata.features).
| obs_frame = build_inference_frame(obs, dataset_metadata.features, device) | |
| obs_frame = build_inference_frame(obs, device, dataset_metadata.features) |
Copilot uses AI. Check for mistakes.
imstevenpmwork
added
examples
enhancement
| model.config, | ||
| model_id, | ||
| # This overrides allows to run on MPS, otherwise defaults to CUDA (if available) | ||
| preprocessor_overrides={"device_processor": {"device": "mps"}}, |
There was a problem hiding this comment.
you should use device here (but cast as string)
There was a problem hiding this comment.
not sure I understand. You mean {"device": torch.device("mps")}? Kinda obscure without full documentation
There was a problem hiding this comment.
Nope, you have a variable device
Why not reuse this variable such as:
| preprocessor_overrides={"device_processor": {"device": "mps"}}, | |
| preprocessor_overrides={"device_processor": {"device": str(device)}}, |
There was a problem hiding this comment.
Interesting. Why would you do that tho? It is exactly the same as directly using the string mps
There was a problem hiding this comment.
The user just has to change the device variable that’s it. In your case, the user needs to update it both in the variable itself and in the overrides.
This is a minor detail.
fracapuano
force-pushed
the
feat/api-examples
branch
from
8dc86aa to
306429a
Compare
3 participants
Part of PR stack (the lower, the closer to main)
build_inference_frame/make_robot_actionutil to easily allow API-based Inference #2143What this does
The tutorial will be released very shortly (~tomorrow), and this PR adds the examples I have added in it to the library. Happy to tweak them, although they have all been tested end-to-end for functionality! Together with the behavior conversion and tutorial, I hope this is a good step towards making the library more hackable (and researcher friendly, 😉!)
FYI, I won't be able to change these examples in the tutorial but for second edition. That can be released soon (TBD), and I'd be happy to add there whatever ends up landing on
main