Text-only mode gives only empty responses

[andreysamode]

To reproduce:

git clone https://github.com/openai/openai-realtime-agents
cd openai-realtime-agents
rm -rf .next
rm -rf .node_modules
npm i
(add key to .env)
(in App.tsx, change "modalities: ["text", "audio"]," to "modalities: ["text"],"
next dev

Result:

The responses I get from the server are empty - just the timestamp:

Works fine with audio added to in modalities, both in text and audio.

Is there some extra bit of configuration that I need to do to make it work in a text-only mode?

[andreysamode]

To solve this, in src/app/hooks/useHandleServerEvent.ts, I changed:

    case "response.output_item.done": {
      const itemId = serverEvent.item?.id;
      if (itemId) {
        updateTranscriptItemStatus(itemId, "DONE");
      }
      break;
    }

to

    case "response.output_item.done": {
      const itemId = serverEvent.item?.id;
      if (itemId) {
        updateTranscriptItemStatus(itemId, "DONE");
        if (serverEvent.item?.content?.[0]?.type === "text") {
          const itemText = serverEvent.item?.content?.[0]?.text || "";
          updateTranscriptMessage(itemId, itemText, false);
        }
      }
      break;
    }

Not sure if it's the best way to do this, but it works.