useChat with generateText (maxRoundTrips)?

[benjaminalgreen]

Is it possible to useChat with generateText instead of streamText? Is there a best practice for appending generated text to the message stream and sending it back to the frontend?

Main reason for this is that I'm using the new maxroundtrips with generateText, but still want to have a chat UI. Alternatively, is there a good way to feed tool results into the message stream for streamText?

[lgrammel]

Here is a guide for using maxToolRoundtrips with useChat: https://sdk.vercel.ai/docs/ai-sdk-ui/chatbot-with-tool-calling

[benjaminalgreen]

Thanks @lgrammel, I'm on Sveltekit so I don't believe that approach works for me. Is there a workaround with tools on the backend?

[lgrammel]

ideally the functionality would be added to SvelteKit. Unfortunately I don't have the bandwidth to cover all frameworks so community contributions are appreciated

[benjaminalgreen]

No problem! I'm not a great programmer so can't help you there. Can generateText be used with useChat? Otherwise I'll revert to the workaround I saw of manually appending toolResults to messages.

[aretrace]

Hi @lgrammel,

I'm also interested in using useChat with generateText, it doesn't seem they fit together in a meaningful fashion (unlike streamText that fits like a glove).

Quick preliminary escape hatch I've come up with:

<div>
  {messages.map((m) => (
    <div key={m.id}>
      {m.role === "user" && "You: " + m.content}
      {m.role === "assistant" && "AI: " + JSON.parse(m.content).messages[0].content[0].text}
    </div>
  ))}
</div>

What do you think? What would you recommend todo?

P.S.
Is generateText simply not designed to be used with useChat? If so, why not? 😢
And, may there be any future where Vercel could reconcile these two APIs? 🥺

[shaurya157]

Bump on this, I'm interested as well.

[ian]

Came here to say the same thing. It's not ideal, but react-native/expo doesn't support streaming so it's easiest to just use generateText for now. But it seems that useChat doesn't play nice out of the box.

[nicoalbanese]

Hey Ian - expo 52 now supports streaming. We just put out a quickstart guide showing how you can using expo/fetch alongside useChat to enable streaming.

[ian]

Thanks @nicoalbanese - excited to give this a try, will report back.

[ian]

Confirmed working with [email protected], expo@52 🎉

[jacobsieradzki]

+1. I’m using a custom provider for a custom LLM that doesn’t support tool streaming yet, so being able to use generateText with useChat would be super helpful!

[lgrammel]

We just added a simulateStreaming model setting to openai (and compatible) chat models that you can use if your provider is openai compatible.

[MarvinS1991]

Hi @lgrammel, where can I get a documentation for the simulateStreaming function?

[lgrammel]

https://sdk.vercel.ai/docs/reference/ai-sdk-core/simulate-readable-stream#simulatereadablestream (name changed to simulateReadableStream)