Camera Implementation [AARD-1925]

[ryanzhangofficial]

Task

Implement a camera "sensor" for code sim that sends a video stream to the robot that can be processed for vision (the equivalent of using a usb webcam).
Jira Ticket

Symptom

Part of the quest to make Code Simulation for Synthesis more robust. This PR is also a building block for the Coprocessor Implementation Ticket.

Solution

The solution implements a virtual camera sensor by rendering the 3D scene from the robot's perspective, capturing real-time video frames, and streaming them to the robot simulation via WebSocket communication. The rendered frames are processed and integrated with WPILib's standard CameraServer infrastructure, making the video stream automatically available in dashboard applications like Shuffleboard and accessible to robot code for vision processing using standard OpenCV and WPILib APIs.

Verification

• A lot of print statements
• Adding Synthesis Camera widget to shuffleboard
  To test, replace these two functions in Robot.java in JavaSample

 /** This function is called once when the robot is first started up. */
  @Override
  public void simulationInit() {
    // Start WebSocket server to receive camera frames from Fission
    System.out.println("🚀 Starting WebSocket server for Synthesis communication...");
    SynthesisWebSocketServer.getInstance().startServer();
    
    // Camera metadata simulation
    m_Camera.setConnected(true);
    m_Camera.setWidth(640);
    m_Camera.setHeight(480);
    m_Camera.setFPS(30);
    
    // Create custom video source - WPILib handles all the streaming!
    m_videoSource = CameraServer.putVideo("Synthesis Camera", 640, 480);
    
    // Register camera with frame handler to receive frames from simulation
    CameraFrameHandler.getInstance().registerCamera("USB Camera 0", m_videoSource);
    
    System.out.println("Camera simulation initialized using WPILib CameraServer!");
    System.out.println("Metadata: " + m_Camera.getWidth() + "x" + m_Camera.getHeight() + " @ " + m_Camera.getFPS() + "fps");
    System.out.println("WPILib automatically creates MJPEG stream and publishes to NetworkTables");
    System.out.println("Camera will appear in Shuffleboard/Glass as 'Synthesis Camera'");
    System.out.println("Camera frame bridge registered - waiting for frames from Synthesis...");
    System.out.println("MJPEG Stream should be available at: http://localhost:1181/?action=stream");
    System.out.println("Stream name in NetworkTables: 'Synthesis Camera'");
    
    // Send a test frame to verify the bridge works
    CameraFrameHandler.getInstance().sendTestFrame("USB Camera 0", 640, 480);
    System.out.println("Initial test frame sent to camera stream");
    
    // IMPORTANT: Test WebSocket message processing
    System.out.println("Testing WebSocket message handler...");
    WebSocketMessageHandler.getInstance().simulateTestMessage();
  }

  /** This function is called periodically whilst in simulation. */
  @Override
  public void simulationPeriodic() {
    // Read camera metadata from simulation
    boolean cameraConnected = m_Camera.isConnected();
    double cameraWidth = m_Camera.getWidth();
    double cameraHeight = m_Camera.getHeight();
    double cameraFPS = m_Camera.getFPS();
    
    // Display camera info on dashboard
    SmartDashboard.putBoolean("Camera Connected", cameraConnected);
    SmartDashboard.putNumber("Camera Width", cameraWidth);
    SmartDashboard.putNumber("Camera Height", cameraHeight);
    SmartDashboard.putNumber("Camera FPS", cameraFPS);
    SmartDashboard.putNumber("Registered Cameras", CameraFrameHandler.getInstance().getCameraCount());
    
    // DEBUGGING: Disable backup test pattern frames completely to see real 3D frames
    // Uncomment line below only if you need to test the camera bridge
    // if (Math.random() < 0.0001) { CameraFrameHandler.getInstance().sendTestFrame("USB Camera 0", (int)cameraWidth, (int)cameraHeight); }
  }

---

Before merging, ensure the following criteria are met:

• All acceptance criteria outlined in the ticket are met.
• Necessary test cases have been added and updated.
• A feature toggle or safe disable path has been added (if applicable).
• User-facing polish:
  • Ask: "Is this ready-looking?"
• Cross-linking between Jira and GitHub:
  • PR links to the relevant Jira issue.
  • Jira ticket has a comment referencing this PR.

[autodesk-chorus]

Chorus detected one or more security issues with this pull request. See the Checks tab for more details.

As a reminder, please follow the secure code review process as part of the Secure Coding Non-Negotiable requirement.

[rutmanz]

Can't seem to get the stream to show up

[rutmanz]

Have you been able to get this to work with someone else's computer?

[ryanzhangofficial]

Have you been able to get this to work with someone else's computer?

I'm able to reproduce the solution on both work and personal laptop. Alexey said he will also test on his laptop.

[AlexD717]

Got the camera feed to show up

[rutmanz]

@AlexD717 what process did you undergo to make this work?

[AlexD717]

@AlexD717 what process did you undergo to make this work?

I was having a lot of build issues on macos, so today I tried doing it on windows. I just installed Java 17 and wpilib. Ran the required ./gradlew build and other commands and it just showed up.