dry_agent ChatOps LLM agent

[EnigmaCurry]

A hypothetical management interface I dreamed up:

dry_agent

dry_agent is a ChatOps bot and agent for d.rymcg.tech. The bot sits in
a Matrix room which you can chat with to manage your Docker server.

The user is able to ask questions and perform the following
tasks:

• "Which services are running?"
• "Start whoami and immich and then turn them off after 45 minutes."

Stretch goals include:

• "Configure a new postgres database." and answer followup questions
  posed by the bot.

The chat bot may run on any machine where it has access to an LLM,
used to build a structured JSON
message based on
information provided by the user. Once the user confirms the action,
it posts the message to an MQTT server that the agent receives from.

The agent must run on a secure workstation that has access to an
unlocked SSH key that controls your Docker server, and receives its
instructions via MQTT, which may include starting and stopping
services as well as status requests.

Structured Responses

{
  "message_type": "string",  // "chat", "action", or "confirmation"
  "content": {},             // Content varies based on message_type
  "conversation_id": "string" // To track multi-turn interactions
}

General chat responses (no Docker action)

{
  "message_type": "chat",
  "content": {
    "text": "I'm your Docker management assistant. I can help you start, stop, and check the status of your containers."
  },
  "conversation_id": "abc123"
}

Docker actions

{
  "message_type": "action",
  "content": {
    "action_type": "string",  // "status", "start", "stop", "restart", "configure"
    "services": ["service1", "service2"],  // Array of service names
    "parameters": {  // Optional parameters specific to action_type
      "timeout": 45,  // Minutes
      "schedule": "2023-11-15T14:30:00Z"  // ISO 8601 timestamp
    },
    "confirmation_required": true  // Whether user confirmation is needed
  },
  "conversation_id": "abc123"
}

Confirmations

{
  "message_type": "confirmation",
  "content": {
    "action_id": "action123",
    "description": "I'm about to start whoami and immich containers, then stop them after 45 minutes. Is this correct?",
    "action_details": {
      // Original action content copied here for reference
    }
  },
  "conversation_id": "abc123"
}

Examples

"Which services are running?"

{
  "message_type": "action",
  "content": {
    "action_type": "status",
    "services": ["whoami", "immich"],
    "parameters": {},
    "confirmation_required": false
  },
  "conversation_id": "abc123"
}

"Start whoami and immich and then turn them off after 45 minutes."

{
  "message_type": "confirmation",
  "content": {
    "action_id": "action456",
    "description": "I'll start the whoami and immich containers, and automatically stop them after 45 minutes. Would you like me to proceed?",
    "action_details": {
      "action_type": "start_with_timeout",
      "services": ["whoami", "immich"],
      "parameters": {
        "timeout": 45
      }
    }
  },
  "conversation_id": "abc123"
}

"Configure a new postgres database."

{
  "message_type": "action",
  "content": {
    "action_type": "configure",
    "services": ["postgres"],
    "parameters": {
      "configuration_type": "new_database",
      "needs_additional_info": true,
      "required_parameters": ["hostname", "user", "password"]
    },
    "confirmation_required": true
  },
  "conversation_id": "abc123"
}

System prompt

You are dry_agent, a ChatOps bot that helps users manage Docker services via Matrix chat. Your responses must be formatted as structured JSON that clearly identifies whether you're providing:

1. A general chat response (message_type: "chat")
2. A container action request (message_type: "action")
3. A confirmation request (message_type: "confirmation")

Message Structure Rules:

• Always assess if the user is requesting a Docker-related action or just chatting
• Use "chat" type for general conversation, questions, or when clarification is needed
• Use "action" type only when the user clearly requests a specific Docker operation
• Use "confirmation" type when additional verification is needed before executing actions
• Always include "conversation_id" in your responses (use "[CONVERSATION_ID]" for now)

Valid Actions:

• status: Check running containers
• start: Launch containers
• stop: Halt containers
• restart: Restart containers
• start_with_timeout: Start containers and stop after specified minutes
• configure: Set up new services (requires additional information)

JSON Response Format:

For chat responses:

{
  "message_type": "chat",
  "content": {
    "text": "Your friendly message here"
  },
  "conversation_id": "[CONVERSATION_ID]"
}

[mcmikemn]

Wow! Ambitious. Cool.

I'm concerned that the nature of LLMs today will make this a bit "fuzzy" (like sometimes it will understand "start drawio" and sometimes it won't). But if this did work as described, then I could have services I rarely use shut down, and when I want to use them I chat them into action and then use them.

Your example suggested I could also make them active for a period of time, which is definitely nice, but not as nice as shutting down a service after a period of inactivity. E.g., "start drawio for 45 minutes", but I'm still working on my document in drawio after 45 minutes - does the container shut down?

[EnigmaCurry]

good point about the timeout I hadn't considered that, maybe it could be like "Start immich and whoami and remind me to turn them off tomorrow". It could be possible to add an actual usage tracker but the original plan does not cover that.

Its my understanding that the structured json output will help to prevent mistakes. It would need to match the request with actual configured services, so if it didn't understand that "drawio" means a service that actually exists, it would have to give an error response. The extra confirmation would only happen after it determined that it could fullfill the action.

[mcmikemn]

Yeah, there could be separate functionality (a separate project) to shut down a container after X. Even if that doesn't exist, then at least the containers aren't active until I chat them up, and then I can manually shut them down when I'm done.

Your idea will probably work pretty well. It's just been my experience that LLMs are kind of arbitrary. Given the same prompt, sometimes they have signifcantly different responses. So even though it's supposed to be outputting JSON, maybe it'll cough up bogus JSO sometimes, or may other times the JSON will be accurate byt the content will be hallucination-affected.

[EnigmaCurry]

If that turns out to be the case, we could implement a non-LLM stricter language

literally: "start drawio" "stop drawio" and if "drawio" isn't a service it fails.