[Security Assistant] Conversation pagination refactor

[stephmilovic]

Summary

This pull request introduces several improvements and optimizations to the assistant UI and backend logic to address performance issues, particularly Out of Memory (OOM) errors, and enhance the overall user experience. Key updates include pagination support, reduced data fetches, and cleaner logic for conversation management.

Key Changes:

• Pagination for Conversations: Refactors the assistant UI to support pagination, reducing memory usage by loading only a subset of conversations at a time. As the user scrolls, more conversations are loaded
• Group conversations by date: Categorizes the chat history based on updatedAt date of the conversation.
• Optimized Fetching: Updates the fetch logic to prevent loading message data for every conversation request, making requests smaller and more efficient.
• Conversation Creation Logic: Removes default conversations and ensures a conversation is not created until the first message is sent. This applies to various triggers such as new chats, attack discovery, alert/event invocations, and data quality dashboard, and detection rule actions.
• Assistant Settings Refactor: Updates the logic in the assistant settings to align with the new pagination structure.
• Content-Specific Hooks: Eliminates unnecessary usage of useSettingsUpdater and replaces it with more appropriate hooks like useSystemPromptUpdater and useConversationsUpdater, streamlining state management.
• Unique Conversations for Triggered Actions: Creates a unique conversation for each new chat initiated from various sources, including alerts, events, DQD, the rules table toolbar, rule status failed callouts, and rule creation forms. This ensures more accurate conversation tracking and better user interaction.

Issue Links:

• Resolves #192714
• Resolves multiple SDH related to OOM errors.

Additional Notes:

• This work aims to mitigate performance bottlenecks caused by fetching excessive data and creating unnecessary conversations.
• Pagination is implemented to improve the scalability of the assistant UI, ensuring it can handle larger datasets more effectively.

Testing

• This PR needs lots of manual testing. Perform all of the AI Assistant Settings actions, create conversations, delete conversations, invoke conversations from all the different possible invocation areas.
• Use the create conversation script to create conversations over wide date range to ensure both pagination and group conversations by date are working. Note that you will be unable to update these conversations from AI Assistant Settings due to the user profile_uid bug.
  • node x-pack/solutions/security/plugins/elastic_assistant/scripts/create_conversations --count 150

[stephmilovic]

@elasticmachine merge upstream

[spong]

Checked out, tested locally, and code reviewed -- LGTM!

Thanks for this giant effort and refactoring @stephmilovic! Really appreciate you taking the time to pair and go over the changes and last couple issues we found. The code base is in a much better spot with all of your changes here! 🙂 🚀

[stephmilovic]

@elasticmachine merge upstream

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 93d8e62

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
automaticImport	732	723	-9
securitySolution	7106	7099	-7
total			-16

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/elastic-assistant	144	135	-9

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
automaticImport	1.1MB	1.1MB	-154.0B
securitySolution	8.9MB	8.9MB	-9.2KB
total			-9.3KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
securitySolution	85.9KB	85.6KB	-364.0B

Unknown metric groups

API count

id	before	after	diff
@kbn/elastic-assistant	172	164	-8

ESLint disabled line counts

id	before	after	diff
@kbn/elastic-assistant	14	13	-1

Total ESLint disabled count

id	before	after	diff
@kbn/elastic-assistant	15	14	-1

History

• 💔 Build #280430 failed 8347cd8
• 💔 Build #280328 failed f7d1eb1
• 💔 Build #279789 failed 3eb4780
• 💔 Build #279729 failed bb88cd9
• 💔 Build #279707 failed 3dc557f
• 💔 Build #279585 failed c3b0d44

cc @stephmilovic

[kibanamachine]

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/13640012888

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	8.x	Backport failed because of merge conflicts You might need to backport the following PRs to 8.x: - [AI Assistant] Fixed DataClient find method to pass fields param to esClient.msearch (#212465) - [EUI Visual Refresh][Security Gen AI] Add support for borealis theme (#202393)

Manual backport

To create the backport manually run:

node scripts/backport --pr 211831

Questions ?

Please refer to the Backport tool documentation

[stephmilovic]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation