Move command requests to card message matrix events and add support for multiple commands per message

[lukemelia]

This PR is breaking in a sense -- previous commands added to matrix rooms will no longer be visible since this PR changes the way we are encoding command requests (tool calls) into matrix events.

The motivation for this change is that tool-call-enabled LLMs have the ability make multiple tool calls in a single response, and our modeling made that quite awkward. This PR make that case normal with full support.

This PR also changes the way that debouncing/throttling works in the aibot. It handles response streaming in better, ensuring an event is sent to matrix at most every 250ms and the response is encoded and delivered in full.

Finally, type-checking and linting in the aibot package is improved in this PR too. It was a little loose before.

UPDATE: This PR has some issues consuming the stream from OpenRouter. I'm investigating. Resolved!

• add host tests for multiple tool calls
• add aibot tests for multiple tool calls with results
• test coverage / implementation for command failures
• Ensure code TODOs are addressed

[github-actions]

Host Test Results

    1 files  ±0      1 suites  ±0   25m 25s ⏱️ + 1m 21s
795 tests +2  793 ✔️ +2  2 💤 ±0  0 ❌ ±0 
800 runs  +2  798 ✔️ +2  2 💤 ±0  0 ❌ ±0 

Results for commit 8b4e305. ± Comparison against base commit 0e42fb1.

♻️ This comment has been updated with latest results.

[backspace]

I’m approving after having read this over with the caveat that I couldn’t get it running locally, the bot was crashing like this:

/Users/b/Documents/Cardstack/Code/boxel-motion/packages/ai-bot/helpers.ts:196
      let { attachedCardsEventIds } = event.content.data;
            ^
TypeError: Cannot destructure property 'attachedCardsEventIds' of 'event.content.data' as it is undefined.
    at constructHistory (/Users/b/Documents/Cardstack/Code/boxel-motion/packages/ai-bot/helpers.ts:196:13)

Is this the breaking change of the PR description, do I need to clear my Matrix history?

[lukemelia]

Is this the breaking change of the PR description, do I need to clear my Matrix history?

It's not supposed to crash. Thanks for the error message. I'll see if I can fix it.

[IanCal]

I ran this and managed to get 3.5 sonnet to return two calls for a command on a skill - I noticed there wasn't a description with the options, it replied when the first one was applied (unsure if those two were intentional here, I know we spoke about different options) and it didn't like sending the reaction event for the second command

Looks like we are using m.annotation, and while that doesn't seem to be against the spec it's flagged by synapse as two reactions -

[IanCal]

replacing m.annotation with a custom thing seems to work for that issue.

[jurgenwerk]

[backspace]

I copied Matic’s example and it seems to have worked:

For that screenshot I pressed “Apply” on the first. The blankness in the response seems confusing but I did inspect the Matrix message and saw the command was properly in an array.

Is there a more thorough way to exercise this, to get multiple commands in a response?

[lukemelia]

Is there a more thorough way to exercise this, to get multiple commands in a response?

Interesting -- that prompt was enough to get multiple commands in one message for me

[jurgenwerk]

I tested this again and I'm a little confused about how multiple commands per message should work. I would expect there would be multiple commands within the same message, but when I test it, the commands will be in multiple messages. Is this expected? Or is there an issue with the test prompt?

Here is my prompt:

[jurgenwerk]

Should there be a description of the action to be applied?