Merge branch 'main' into cs-7989-display-attached-file-in-message-bubble

Author: FadhlanR

Diff for: docs/README.md

@@ -1,4 +1,3 @@
-
 # Documentation about Boxel
 
 ## Prerequisites
@@ -9,13 +8,13 @@ It also has a feature to save an .svg image, which be opened in Chrome for a dec
 
 ## Concepts
 
-The following are important concepts: 
+The following are important concepts:
 
-- [Card and Field Definition Relationships](card-def-field-def-relationships.md): There is a subtle distinction between card and fields to consider when creating cards. 
-- [Inheritance](card-inheritance.md): Cards can be extended based upon user's custom needs -- no reinventing the wheel.  
-- [Rendering](card-rendering.md): Cards can be rendered easily in the browser. Each card renders differently based upon how it is related and what context it exists in.  
+- [Card and Field Definition Relationships](card-def-field-def-relationships.md): There is a subtle distinction between card and fields to consider when creating cards.
+- [Inheritance](card-inheritance.md): Cards can be extended based upon user's custom needs -- no reinventing the wheel.
+- [Rendering](card-rendering.md): Cards can be rendered easily in the browser. Each card renders differently based upon how it is related and what context it exists in.
 - [Serialization and Deserialization](card-serialization-deserialization.md): Cards have to be adapted to a consistent JSON format before being sent over-the-wire to other consumers.
-- [Computed Fields](computed-fields.md): Computed fields work too! We can compute on the data that is already contained in a card to build more complex logic.   
-- [Indexing](indexing.md): Indexing powers the re-rendering of cards when it's dependencies get updated. 
+- [Computed Fields](computed-fields.md): Computed fields work too! We can compute on the data that is already contained in a card to build more complex logic.
+- [Indexing](indexing.md): Indexing powers the re-rendering of cards when it's dependencies get updated.
 - [Realm](realm.md): Realms are storage for cards that have their own underlying permissions and indexer.
-- [Search](search.md): Every Card is searchable within and across realms.
+- [Search](search.md): Every Card is searchable within and across realms.

Diff for: docs/queue.md

@@ -1,29 +1,33 @@
 Our queue system is postgres based job queue system (resurrected from hub v2). This leverages postgres pub/sub capabilities to create a job queue system that implements our `Queue` interface (we previously created a mock queue that implements this same interface for browser testing).
 
-The queue is controlled by the `jobs` table. We can monitor and control our queue using this table. The following is an example query of this table after running our tests. 
+The queue is controlled by the `jobs` table. We can monitor and control our queue using this table. The following is an example query of this table after running our tests.
 
 ```
 > SELECT * FROM JOBS;
- 
- id | category  | args |  status  |         created_at         |        finished_at         |      queue      | result 
+
+ id | category  | args |  status  |         created_at         |        finished_at         |      queue      | result
 ----+-----------+------+----------+----------------------------+----------------------------+-----------------+--------
   1 | increment | 17   | resolved | 2024-04-19 16:57:43.305961 | 2024-04-19 16:57:43.311274 | increment-queue | 18
 ```
 
 On system start up we can register job handlers whose responsibility it is to run queued jobs (these handlers can horizontally scale if we so choose). A handler registration looks like this:
+
 ```ts
 queue.register('increment', async (a: number) => a + 1);
 ```
-This is a real simple example that just adds 1 to the job's input arguments. A handler ran return an async result as JSONB value which is stored in the `jobs.result` column of the `jobs` table. This handler defines  a "category" called `increment` for this function that it has registered. A handler processes the queue by looking for the oldest job that isn't running and handles that first.
+
+This is a real simple example that just adds 1 to the job's input arguments. A handler ran return an async result as JSONB value which is stored in the `jobs.result` column of the `jobs` table. This handler defines a "category" called `increment` for this function that it has registered. A handler processes the queue by looking for the oldest job that isn't running and handles that first.
 
 Clients of the queue that wish to run jobs can do so by specifying the category of job that they wish to run, the queue that they wish to use, and input arguments for the job (the input arguments can be a JSONB value which is stored in the `jobs.args` column of the `jobs` table).
+
 ```ts
 let job = await queue.publish<number>('increment', 17, {
   queueName: 'increment-queue',
 });
 ```
+
 The caller is handed a `job` object. This object has an `id` property and a `done` property that returns a promise for the job's return value (which is a parameterized type) when the job is completed. Note that the `queueName` is optional. If no name is supplied then the queue name `"default"` is used. The `queueName` is used to control job concurrency. Jobs are processed in each `queueName` serially.
 
-When a job is first published to a queue it is assigned a status of `unfulfilled`. When a job has completed successfully it is assigned a status of `resolved`. If a job throws an error it is assigned a status of `rejected` and the error is serialized in the `jobs.result` column. 
+When a job is first published to a queue it is assigned a status of `unfulfilled`. When a job has completed successfully it is assigned a status of `resolved`. If a job throws an error it is assigned a status of `rejected` and the error is serialized in the `jobs.result` column.
 
-Using SQL you can monitor the progress of the jobs in the queue, as well as, you can manipulate the results of the queue processing by setting `jobs.status`, `jobs.result`, and `jobs.queueName` using SQL. 
+Using SQL you can monitor the progress of the jobs in the queue, as well as, you can manipulate the results of the queue processing by setting `jobs.status`, `jobs.result`, and `jobs.queueName` using SQL.

Diff for: package.json

@@ -27,13 +27,11 @@
       "@embroider/util": "1.13.1",
       "@glimmer/tracking>@glimmer/validator": "0.84.3",
       "jsesc": "^3.0.0",
-      "ember-modifier": "^4.1.0",
-      "prettier": "github:cardstack/prettier#glimmer-style-tag-in-template-support"
+      "ember-modifier": "^4.1.0"
     },
     "peerDependencyRules": {
       "allowedVersions": {
         "mustache": "3",
-        "prettier@github:cardstack/prettier#glimmer-style-tag-in-template-support": "3.1.0-dev",
         "ember-qunit@5.1.2>ember-source": "*"
       }
     },
@@ -70,7 +68,7 @@
     "eslint-plugin-prefer-let": "^3.0.1",
     "eslint-plugin-prettier": "^5.0.0",
     "hcl2-parser": "^1.0.3",
-    "prettier": "^2.7.1",
+    "prettier": "^3.5.1",
     "prettier-plugin-ember-template-tag": "^1.1.0",
     "typescript": "~5.1.6"
   },

Diff for: packages/ai-bot/helpers.ts

@@ -76,10 +76,10 @@ export class HistoryConstructionError extends Error {
   }
 }
 
-export function getPromptParts(
+export async function getPromptParts(
   eventList: DiscreteMatrixEvent[],
   aiBotUserId: string,
-): PromptParts {
+): Promise<PromptParts> {
   let cardFragments: Map<string, CardFragmentContent> =
     extractCardFragmentsFromEvents(eventList);
   let history: DiscreteMatrixEvent[] = constructHistory(
@@ -89,7 +89,7 @@ export function getPromptParts(
   let skills = getEnabledSkills(eventList, cardFragments);
   let tools = getTools(history, aiBotUserId);
   let toolChoice = getToolChoice(history, aiBotUserId);
-  let messages = getModifyPrompt(history, aiBotUserId, tools, skills);
+  let messages = await getModifyPrompt(history, aiBotUserId, tools, skills);
   let model = getModel(eventList);
   return {
     tools,
@@ -321,6 +321,113 @@ export function getRelevantCards(
   };
 }
 
+export async function loadCurrentlyAttachedFiles(
+  history: DiscreteMatrixEvent[],
+  aiBotUserId: string,
+): Promise<
+  {
+    url: string;
+    name: string;
+    contentType?: string;
+    content: string | undefined;
+    error: string | undefined;
+  }[]
+> {
+  let lastMessageEventByUser = history.findLast(
+    (event) => event.sender !== aiBotUserId,
+  );
+
+  let mostRecentUserMessageContent = lastMessageEventByUser?.content as {
+    msgtype?: string;
+    data?: {
+      attachedFiles?: { url: string; name: string; contentType?: string }[];
+    };
+  };
+
+  if (
+    !mostRecentUserMessageContent ||
+    mostRecentUserMessageContent.msgtype !== APP_BOXEL_MESSAGE_MSGTYPE
+  ) {
+    return [];
+  }
+
+  // We are only interested in downloading the most recently attached files -
+  // downloading older ones is not needed since the prompt that is being constructed
+  // should operate on fresh data
+  if (!mostRecentUserMessageContent.data?.attachedFiles?.length) {
+    return [];
+  }
+
+  let attachedFiles = mostRecentUserMessageContent.data.attachedFiles;
+
+  return Promise.all(
+    attachedFiles.map(
+      async (attachedFile: {
+        url: string;
+        name: string;
+        contentType?: string;
+      }) => {
+        try {
+          let content: string | undefined;
+          let error: string | undefined;
+          if (attachedFile.contentType?.startsWith('text/')) {
+            let response = await (globalThis as any).fetch(attachedFile.url);
+            if (!response.ok) {
+              throw new Error(`HTTP error. Status: ${response.status}`);
+            }
+            content = await response.text();
+          } else {
+            error = `Unsupported file type: ${attachedFile.contentType}. For now, only text files are supported.`;
+          }
+
+          return {
+            url: attachedFile.url,
+            name: attachedFile.name,
+            contentType: attachedFile.contentType,
+            content,
+            error,
+          };
+        } catch (error) {
+          log.error(`Failed to fetch file ${attachedFile.url}:`, error);
+          Sentry.captureException(error, {
+            extra: { fileUrl: attachedFile.url, fileName: attachedFile.name },
+          });
+          return {
+            url: attachedFile.url,
+            name: attachedFile.name,
+            contentType: attachedFile.contentType,
+            content: undefined,
+            error: `Error loading attached file: ${(error as Error).message}`,
+          };
+        }
+      },
+    ),
+  );
+}
+
+export function attachedFilesToPrompt(
+  attachedFiles: {
+    url: string;
+    name: string;
+    contentType?: string;
+    content: string | undefined;
+    error: string | undefined;
+  }[],
+): string {
+  if (!attachedFiles.length) {
+    return 'No attached files';
+  }
+  return attachedFiles
+    .map((f) => {
+      if (f.error) {
+        return `${f.name}: ${f.error}`;
+      }
+
+      return `${f.name}: ${f.content}`;
+    })
+    .join('\n');
+}
+
 export function getTools(
   history: DiscreteMatrixEvent[],
   aiBotUserId: string,
@@ -431,7 +538,7 @@ function toPromptMessageWithToolResult(
   };
 }
 
-export function getModifyPrompt(
+export async function getModifyPrompt(
   history: DiscreteMatrixEvent[],
   aiBotUserId: string,
   tools: Tool[] = [],
@@ -501,15 +608,18 @@ export function getModifyPrompt(
     history,
     aiBotUserId,
   );
-  let systemMessage =
-    MODIFY_SYSTEM_MESSAGE +
-    `
-  The user currently has given you the following data to work with:
-  Cards:\n`;
-  systemMessage += attachedCardsToMessage(
-    mostRecentlyAttachedCard,
-    attachedCards,
-  );
+
+  let attachedFiles = await loadCurrentlyAttachedFiles(history, aiBotUserId);
+
+  let systemMessage = `${MODIFY_SYSTEM_MESSAGE}
+The user currently has given you the following data to work with:
+
+Cards: ${attachedCardsToMessage(mostRecentlyAttachedCard, attachedCards)}
+
+Attached files:
+${attachedFilesToPrompt(attachedFiles)}
+`;
+
   if (skillCards.length) {
     systemMessage += SKILL_INSTRUCTIONS_MESSAGE;
     systemMessage += skillCardsToMessage(skillCards);

Diff for: packages/ai-bot/main.ts

@@ -214,7 +214,7 @@ Common issues are:
         let eventList = (initial!.messages?.chunk ||
           []) as DiscreteMatrixEvent[];
         try {
-          promptParts = getPromptParts(eventList, aiBotUserId);
+          promptParts = await getPromptParts(eventList, aiBotUserId);
         } catch (e) {
           log.error(e);
           responder.finalize(