OpenAI-Realtime API support.

[drestr30]

Hello, I would like to ask if there is a future plan for integrating the new openAI realtime API (which works with sockets connections instead of HTTP request) to the langchain and langgraph frameworks.
Right now I'm building some voice agents with langgraph using the multi-agent architecture and using TTS and SST models to do voice processing, this approach instroduces a lot of lag into the voice interaction which is dramatically improved with the openAI realtime endpoint however I have not found documentation or a way to integrate this new model (sockets connection) to the langraph agent graph.
Please let me know if you already have an approach to solve this.
Thanks so much.

[tushar-madheshia]

Eagerly waiting for a reply on this.. @keenborder786

[surendrasalke]

Awaiting your response to this thread.

[ngDuyAnh]

Still waiting for this thread?

[edwinjosechittilappilly]

Looking forward to this support!

[acomito]

yes!

[uagarwal310]

Looking forward to the support on this, meanwhile found https://github.com/AK391/ai-gradio

Has anyone tried?

[dmitryrPlanner5D]

@drestr30
I found this langchain-ai/react-voice-agent repo which seems to be related to your request. It looks like a demo server integrating langchain tools with socket-based gpt-4o-realtime LLM.

[mahimairaja]

@dmitryrPlanner5D yes, but does lang graph supports for Web Socket to access the real time api?

[dmitryrPlanner5D]

@mahimairaja
As far as I know langgraph does not directly support web sockets,

but you can make an independent module with openai realtime llm (not in LangGraph) based on the repo above.
And this realtime module will call your main LangGraph logic as a tool.

Also I found this LlamaIndex realtime voice client implementation rather clean: https://github.com/run-llama/openai_realtime_client
And it looks like it is not hard to replace their LlamaIndex tools with LangChain ones or wrap LangGraph logic with LlamaIndex tool.