Skip to content

Conversation

jandubois
Copy link
Member

@jandubois jandubois commented Sep 20, 2025

I had an idea how to test limactl-mcp with BATS, so I wrote this PoC. I think it works well, but must obviously be fleshed out some more.

This PR depends on #3744 and assumes it has been merged already.

@jandubois
Copy link
Member Author

One thing I found while writing the test was that run_shell_command returns an object that has additional JSON embedded as a string:

$ echo '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"run_shell_command","arguments":{"command":["cat","os-release"],"directory":"/etc"}}}' >&"${MCP[1]}"

$ read -t 1 -r line <&"${MCP[0]}"; jq . <<<"$line"
{
  "jsonrpc": "2.0",
  "id": 3,
  "result": {
    "content": [
      {
        "type": "text",
        "text": "{\"stdout\":\"PRETTY_NAME=\\\"Ubuntu 25.04\\\"\\nNAME=\\\"Ubuntu\\\"\\nVERSION_ID=\\\"25.04\\\"\\nVERSION=\\\"25.04 (Plucky Puffin)\\\"\\nVERSION_CODENAME=plucky\\nID=ubuntu\\nID_LIKE=debian\\nHOME_URL=\\\"https://www.ubuntu.com/\\\"\\nSUPPORT_URL=\\\"https://help.ubuntu.com/\\\"\\nBUG_REPORT_URL=\\\"https://bugs.launchpad.net/ubuntu/\\\"\\nPRIVACY_POLICY_URL=\\\"https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\\\"\\nUBUNTU_CODENAME=plucky\\nLOGO=ubuntu-logo\\n\",\"stderr\":\"\",\"exit_code\":0}"
      }
    ]
  }
}

So you need to extract .text, unmarshall the result into JSON and then extract .stdout, which is quite cumbersome. I don't know if this will bother an LLM, but doesn't this also increase the number of tokens used for the content by escaping all the quotes and backslashes?

Is there a reason this can't be a normal nested object?

@jandubois jandubois force-pushed the bats-mcp branch 4 times, most recently from f56729d to 1f10d05 Compare September 21, 2025 03:27
@jandubois
Copy link
Member Author

Beyond the encoded JSON thing, I don't understand the schema of the output: Why is content an array? I don't understand how there can be more than one set of results given that the tool only accepts a single command.

And what does type: text mean? Especially given that text is a json string. Could it also be object? Isn't this all too complex for a tool command?

So I would expect the output of run_shell_command to look more like this:

{
  "jsonrpc": "2.0",
  "id": 3,
  "result": {
    "stdout": "...",
    "stderr": "",
    "exit_code": 0
  }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant