You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/software/SDK/Baballonia Overview.mdx
+19-74Lines changed: 19 additions & 74 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,94 +5,39 @@ sidebar_position: 1
5
5
6
6
# Overview of Baballonia SDK
7
7
8
-
The Baballonia SDK provides developers with the tools and interfaces necessary to integrate eye-tracking and face-tracking capabilities into their applications. This SDK is designed to work seamlessly with the Baballonia software, allowing for easy access to processed tracking data and raw camera streams.
8
+
The Baballonia SDK provides developers with the tools and interfaces necessary to integrate eye-tracking and face-tracking capabilities into applications. It works seamlessly with the Baballonia desktop software, giving access to processed tracking data as well as raw camera streams.
9
9
10
-
It enables developers to build custom plugins, create new capture sources, and develop integrations inside the Baballonia application—without needing to modify Baballonia’s core codebase itself.
10
+
With the SDK, developers can build custom plugins, implement new capture sources, and develop integrations inside the Baballonia application—without modifying the core codebase.
11
11
12
12
---
13
13
14
14
## SDK Components
15
15
16
-
### **1. Capture System (Camera Stream Behavior)**
16
+
### 1. `Capture` Class
17
17
18
-
At the core of the SDK is the Capture System: a unified API for accessing frames from any supported camera source. Rather than tying Baballonia to a single device type, each camera is represented as a subclass of the abstract `Capture` class.
18
+
`Capture` provides the framework that lets developers create their own camera capture sources. By extending the `Capture` class, you define exactly how your camera behaves and how it provides frames to Baballonia.
19
19
20
-
:::info The `Capture` Base Class
21
-
- Defines how a camera source starts and stops streaming
22
-
- Handles thread-safe delivery of OpenCV `Mat` frame objects
23
-
- Tracks device readiness and connection status
24
-
- Allows developers to safely produce and consume frames across multiple threads
25
-
:::
20
+
Baballonia currently supports:
26
21
27
-
This makes it possible for Baballonia to support:
22
+
- Standard USB webcams (UVC)
23
+
- Varjo eye-tracking cameras
24
+
- Android camera streams
25
+
- IP cameras
26
+
- V4L2 devices on Linux
27
+
- Custom or experimental capture sources
28
28
29
-
- Standard USB webcams (UVC)
30
-
- Varjo eye-tracking cameras
31
-
- Android phone camera streams
32
-
- V4L2 devices on Linux
33
-
- Custom or experimental capture sources
34
-
35
-
<h3><b><i>All through the same interface.</i></b></h3>
36
-
37
-
---
38
-
39
-
### **2. Capture Factories (Plugin System)**
40
-
41
-
The SDK includes the `ICaptureFactory` interface, which allows developers to create custom camera capture sources as plugins. By implementing this interface, developers can define how their specific camera hardware integrates with the Baballonia application.
42
-
43
-
:::info The `ICaptureFactory` interface defines:
44
-
- Creating a new capture source by returning a subclass of the `Capture` class
45
-
- Checking if your source is connectable via the `CanConnect` method
46
-
- Providing the provider name that identifies your capture source inside Baballonia
47
-
:::
48
-
49
-
Baballonia's core application queries all registered factories to determine which one can handle a given camera address.
50
-
51
-
This plugin-based structure lets you extend Baballonia with new device support **without modifying the main application**.
52
-
53
-
---
54
-
55
-
### **3. Thread-Safe Frame Transfer**
56
-
57
-
The `Capture` class includes methods like:
58
-
59
-
-`SetRawMat(Mat value)` — called by the capture thread to provide new frames
60
-
-`AcquireRawMat()` — called by the processing thread to consume frames
61
-
62
-
These methods transfer ownership of the frame safely, preventing race conditions and ensuring that each frame is only processed once. Optimized for performance, the SDK avoids unnecessary copies of frame data.
63
-
64
-
This design allows Baballonia to run real-time tracking models efficiently and reliably.
65
-
66
-
---
67
-
68
-
### **4. Tracking Data Pipeline**
69
-
70
-
While the Capture system provides raw video frames, the SDK also defines interfaces and data structures for:
71
-
72
-
- Processed lower-face expression data
73
-
- Eye gaze direction
74
-
- Blink and openness values
75
-
- Confidence metrics
76
-
- Filtered/smoothed measurement output
77
-
78
-
Baballonia’s core uses these structures to generate high-quality tracking data, which can then be streamed via OSC or consumed by external modules (such as VRCFaceTracking integrations).
29
+
This system allows all of these devices to work through a unified interface inside of Baballonia.
79
30
80
31
---
81
32
82
-
### **5. Software Extensibility**
33
+
### 2. `ICaptureFactory` Interface
83
34
84
-
The Baballonia SDK is built with extensibility in mind. Developers can create new modules that plug into the Baballonia application, enabling:
85
-
86
-
- Custom camera drivers
87
-
- Designing custom tracking pipelines
88
-
- Logging or manipulating frame data
89
-
- Integrating Baballonia into other applications (e.g., VRCFT)
90
-
- Creating new output modules (OSC, WebSocket, gRPC, etc.)
91
-
92
-
---
35
+
Every custom camera backend must provide a corresponding `ICaptureFactory`. A factory tells Baballonia:
93
36
94
-
### **Summary**
37
+
- Whether it can connect to a given camera address (`CanConnect`)
38
+
- How to construct the correct `Capture` implementation (`Create`)
39
+
- What provider name should appear in the UI and logs (`GetProviderName`)
95
40
96
-
The Baballonia SDK provides a robust foundation for camera capture, frame handling, and tracking integration. Its abstraction layers and plugin-friendly architecture make it easy to extend Baballonia with new devices, add custom processing, or integrate tracking data into external applications.
41
+
Baballonia never instantiates `Capture` classes directly.
97
42
98
-
Whether you're building support for a new camera, creating a custom VR integration, or experimenting with computer vision, the Baballonia SDK gives you the structure and flexibility for your applications.
43
+
The capture system and the `ICaptureFactory` are inseparable: a `Capture` implementation cannot function without a corresponding factory, and a factory has no purpose without a `Capture` implementation.
0 commit comments