Context management for long conversations

[mnort9]

I'm trying to figure out the most RubyLLM way to compact the context during long conversations. For a basic "lazy" approach, I want to rescue a token limit error, compact the context and continue the response.

One option is creating a new chat with the compacted context, but I don't love this approach since it should still show as a single chat thread to the user.

Another option is creating a new system message with the compacted context and only passing new messages since the new system message to the llm. I don't see a built-in way to scope the messages sent to the llm other than using raw content blocks, but I'm happy to contribute if it's something that would be considered as in scope for RubyLLM.

Any other ideas I might be overlooking?

[crmne]

I think creating a new chat makes a lot of sense.

You could add a pointer to the compacted chat in the previous one and the view layer would handle showing both seamlessly?

[mnort9]

Definitely an option. I was just thinking it will would isolate the logic a little better if I only had to scope the messages for a response and not worry about it in the view layer.

For example, something like:

new_system_msg = chat.add_message(role: :system, content: "Compacted context")
chat.with_messages(chat.messages.since(new_system_msg)) # Exclude older messages
chat.ask "User message"