Currently, the system can only retrieve audio input and produce audio output to the local speaker and microphone, so it cannot hear the game voice chat or speak to players.
This proposal introduces four components + some script to address this limitation.
Component 1: AudioIn
This component has the same interface (i.e. the ports shown on the node in the UI) as the existing Mic component. During setup, it has a drop down to select the source audio device, where the items in the dropdown is rendered dynamically similar to the current Camera component. The dropdown should have something like:
Component 2: AudioOut
The same as above, but for AudioOutput. This component is similar to the existing Speaker component.
So far the two components described above makes it possible to connect to game audio when both OpenNeuro and VrChat is running locally, but we also need to address the case where VrChat and OpenNeuro is running on different machines.
To do this we need a simple python script that sets up a fastapi websocket server (or webrtc, but I don't exactly know how it works), the server is on the same machine as the game and sends out game Audio and receives AI speech input and proxies it to the game's mic. Then, 2 components: RemoteAudioIn and RemoteAudioOut which talks to this server.
Game audio setup: you should install VBCable https://vb-audio.com/Cable/ and practice setting it up in game. Although I haven't tested myself, VrChat supposedly have options for selecting audio device for its mic input, as well as its game sound output. You can select VBCable's virtual audio devices for those (hopefully). You need to install VBCables A+B, which gives you 2 cables, because you need one cable to connect the component-out to game-in, and another to connect the game-out to component-in.
Currently, the system can only retrieve audio input and produce audio output to the local speaker and microphone, so it cannot hear the game voice chat or speak to players.
This proposal introduces four components + some script to address this limitation.
Component 1: AudioIn
This component has the same interface (i.e. the ports shown on the node in the UI) as the existing Mic component. During setup, it has a drop down to select the source audio device, where the items in the dropdown is rendered dynamically similar to the current Camera component. The dropdown should have something like:
Component 2: AudioOut
The same as above, but for AudioOutput. This component is similar to the existing Speaker component.
So far the two components described above makes it possible to connect to game audio when both OpenNeuro and VrChat is running locally, but we also need to address the case where VrChat and OpenNeuro is running on different machines.
To do this we need a simple python script that sets up a fastapi websocket server (or webrtc, but I don't exactly know how it works), the server is on the same machine as the game and sends out game Audio and receives AI speech input and proxies it to the game's mic. Then, 2 components: RemoteAudioIn and RemoteAudioOut which talks to this server.
Game audio setup: you should install VBCable https://vb-audio.com/Cable/ and practice setting it up in game. Although I haven't tested myself, VrChat supposedly have options for selecting audio device for its mic input, as well as its game sound output. You can select VBCable's virtual audio devices for those (hopefully). You need to install VBCables A+B, which gives you 2 cables, because you need one cable to connect the component-out to game-in, and another to connect the game-out to component-in.