Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Obs AI Assistant] Add API test for get_alerts_dataset_info tool #212858

Merged
merged 15 commits into from
Mar 5, 2025
Merged

[Obs AI Assistant] Add API test for get_alerts_dataset_info tool #212858

merged 15 commits into from
Mar 5, 2025

Conversation

sorenlouv
Copy link
Member

@sorenlouv sorenlouv commented Mar 3, 2025 •
edited by kibanamachine
Loading

Follow-up to: #212077

This PR includes an API test that covers get_alerts_dataset_info and would have caught the bug fixed in #212077.

It also contains the following bug fixes:

  • Fix system message in select_relevant_fields
  • Change prompt in select_relevant_fields so that the LLM consistently uses the right format when responding.

@sorenlouv sorenlouv requested a review from a team as a code owner March 3, 2025 09:05
@botelastic botelastic bot added ci:project-deploy-observability Create an Observability project Team:Obs AI Assistant Observability AI Assistant labels Mar 3, 2025
@elasticmachine
Copy link
Contributor

Pinging @elastic/obs-ai-assistant (Team:Obs AI Assistant)

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Mar 3, 2025

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

  • /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

@sorenlouv sorenlouv added release_note:skip Skip the PR/issue when compiling release notes backport:prev-minor Backport to (9.0) the previous minor version (i.e. one version back from main) backport:prev-major Backport to (8.x, 8.18, 8.17, 8.16) the previous major branch and other branches in development labels Mar 3, 2025
@sorenlouv sorenlouv force-pushed the add-test-get_alerts_dataset_info branch from c2aadc3 to 8da67c6 Compare March 3, 2025 09:07
@sorenlouv
Copy link
Member Author

@arturoliduena All comments addressed. I've kicked off the flaky test runner 🤞

@kibanamachine
Copy link
Contributor

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7969

[❌] x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts: 25/50 tests passed.

see run history

@kibanamachine
Copy link
Contributor

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#7968

[✅] x-pack/test/observability_ai_assistant_functional/enterprise/config.ts: 50/50 tests passed.
[✅] x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.apm.serverless.config.ts: 50/50 tests passed.
[✅] x-pack/test_serverless/api_integration/test_suites/observability/ai_assistant/config.ts: 50/50 tests passed.

see run history

@kibanamachine
Copy link
Contributor

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7970

[❌] x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts: 25/50 tests passed.

see run history

@sorenlouv
Skip flaky synthetics test
0e7cad9 
1)    Stateful Observability - Deployment-agnostic API integration tests
       SyntheticsAPITests
         getSyntheticsMonitors
           get many monitors
             without params:
       Error: {"statusCode":500,"error":"Internal Server Error","message":"Manifest file manifest.yml not found in paths."}
       + expected - actual
       -500
       +200
       at Assertion.assert (expect.js:100:11)
       at Assertion.eql (expect.js:244:8)
       at saveMonitor (get_monitor.ts:52:26)
       at processTicksAndRejections (node:internal/process/task_queues:95:5)
       at async Promise.all (index 3)
       at Context.<anonymous> (get_monitor.ts:85:30)
       at Object.apply (wrap_function.js:74:16)
@sorenlouv sorenlouv requested a review from a team as a code owner March 4, 2025 13:57
viduni94
viduni94 reviewed Mar 4, 2025
View reviewed changes
x-pack/test/api_integration/deployment_agnostic/apis/observability/synthetics/get_monitor.ts Outdated
@@ -80,7 +80,7 @@ export default function ({ getService }: DeploymentAgnosticFtrProviderContext) {
});

describe('get many monitors', () => {
it('without params', async () => {
it.skip('without params', async () => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this intentionally skipped?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it looked flaky. But I'll revert it because it didn't seem to fix the flakiness. Will keep digging

@kibanamachine
Copy link
Contributor

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7972

[❌] x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts: 0/100 tests passed.

see run history

@kibanamachine
Copy link
Contributor

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7976

[❌] x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts: 0/100 tests passed.

see run history

@kibanamachine
Copy link
Contributor

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7977

[❌] x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.stateful.config.ts: 72/100 tests passed.

see run history

@elasticmachine
Copy link
Contributor

elasticmachine commented Mar 4, 2025 •
edited
Loading

💛 Build succeeded, but was flaky

  • Buildkite Build
  • Commit: 4634839
  • Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-212858-4634839dbb3d

Failed CI Steps

Metrics [docs]

✅ unchanged

History

@sorenlouv sorenlouv merged commit 0fb83ef into elastic:main Mar 5, 2025
10 checks passed
@sorenlouv sorenlouv deleted the add-test-get_alerts_dataset_info branch March 5, 2025 08:09
@kibanamachine
Copy link
Contributor

Starting backport for target branches: 8.16, 8.17, 8.18, 8.x, 9.0

https://github.com/elastic/kibana/actions/runs/13671303397

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Mar 5, 2025
@sorenlouv
[Obs AI Assistant] Add API test for get_alerts_dataset_info tool (e…
b633bcd 
…lastic#212858)

Follow-up to: elastic#212077

This PR includes an API test that covers `get_alerts_dataset_info` and
would have caught the bug fixed in
elastic#212077.

It also contains the following bug fixes:

- Fix system message in `select_relevant_fields`
- Change prompt in `select_relevant_fields` so that the LLM consistently
uses the right format when responding.

(cherry picked from commit 0fb83ef)
@kibanamachine kibanamachine mentioned this pull request Mar 5, 2025
Merged
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Mar 5, 2025
@sorenlouv
[Obs AI Assistant] Add API test for get_alerts_dataset_info tool (e…
2a69a7b 
…lastic#212858)

Follow-up to: elastic#212077

This PR includes an API test that covers `get_alerts_dataset_info` and
would have caught the bug fixed in
elastic#212077.

It also contains the following bug fixes:

- Fix system message in `select_relevant_fields`
- Change prompt in `select_relevant_fields` so that the LLM consistently
uses the right format when responding.

(cherry picked from commit 0fb83ef)
@kibanamachine kibanamachine mentioned this pull request Mar 5, 2025
Merged
@kibanamachine
Copy link
Contributor

💔 Some backports could not be created

Status Branch Result
❌ 8.16 Backport failed because of merge conflicts
❌ 8.17 Backport failed because of merge conflicts
❌ 8.18 Backport failed because of merge conflicts
✅ 8.x
✅ 9.0

Note: Successful backport PRs will be merged automatically after passing CI.

Manual backport

To create the backport manually run:

node scripts/backport --pr 212858

Questions ?

Please refer to the Backport tool documentation

kibanamachine added a commit that referenced this pull request Mar 5, 2025
@kibanamachine @sorenlouv
[9.0] [Obs AI Assistant] Add API test for get_alerts_dataset_info t…
c6c2cea 
…ool (#212858) (#213203)

# Backport

This will backport the following commits from `main` to `9.0`:
- [[Obs AI Assistant] Add API test for `get_alerts_dataset_info` tool
(#212858)](#212858)

<!--- Backport version: 9.6.6 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Søren
Louv-Jansen","email":"soren.louv@elastic.co"},"sourceCommit":{"committedDate":"2025-03-05T08:09:22Z","message":"[Obs
AI Assistant] Add API test for `get_alerts_dataset_info` tool
(#212858)\n\nFollow-up to:
https://github.com/elastic/kibana/pull/212077\n\nThis PR includes an API
test that covers `get_alerts_dataset_info` and\nwould have caught the
bug fixed in\nhttps://github.com//pull/212077.\n\nIt also
contains the following bug fixes:\n\n- Fix system message in
`select_relevant_fields`\n- Change prompt in `select_relevant_fields` so
that the LLM consistently\nuses the right format when
responding.","sha":"0fb83efd82ae3ebd8a9fe27813e436b80cd240d3","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport:prev-minor","backport:prev-major","Team:Obs
AI Assistant","ci:project-deploy-observability","v9.1.0"],"title":"[Obs
AI Assistant] Add API test for `get_alerts_dataset_info`
tool","number":212858,"url":"https://github.com/elastic/kibana/pull/212858","mergeCommit":{"message":"[Obs
AI Assistant] Add API test for `get_alerts_dataset_info` tool
(#212858)\n\nFollow-up to:
https://github.com/elastic/kibana/pull/212077\n\nThis PR includes an API
test that covers `get_alerts_dataset_info` and\nwould have caught the
bug fixed in\nhttps://github.com//pull/212077.\n\nIt also
contains the following bug fixes:\n\n- Fix system message in
`select_relevant_fields`\n- Change prompt in `select_relevant_fields` so
that the LLM consistently\nuses the right format when
responding.","sha":"0fb83efd82ae3ebd8a9fe27813e436b80cd240d3"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/212858","number":212858,"mergeCommit":{"message":"[Obs
AI Assistant] Add API test for `get_alerts_dataset_info` tool
(#212858)\n\nFollow-up to:
https://github.com/elastic/kibana/pull/212077\n\nThis PR includes an API
test that covers `get_alerts_dataset_info` and\nwould have caught the
bug fixed in\nhttps://github.com//pull/212077.\n\nIt also
contains the following bug fixes:\n\n- Fix system message in
`select_relevant_fields`\n- Change prompt in `select_relevant_fields` so
that the LLM consistently\nuses the right format when
responding.","sha":"0fb83efd82ae3ebd8a9fe27813e436b80cd240d3"}}]}]
BACKPORT-->

Co-authored-by: Søren Louv-Jansen <soren.louv@elastic.co>
kibanamachine added a commit that referenced this pull request Mar 5, 2025
@kibanamachine @sorenlouv
[8.x] [Obs AI Assistant] Add API test for get_alerts_dataset_info t…
9c61f24 
…ool (#212858) (#213202)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Obs AI Assistant] Add API test for `get_alerts_dataset_info` tool
(#212858)](#212858)

<!--- Backport version: 9.6.6 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Søren
Louv-Jansen","email":"soren.louv@elastic.co"},"sourceCommit":{"committedDate":"2025-03-05T08:09:22Z","message":"[Obs
AI Assistant] Add API test for `get_alerts_dataset_info` tool
(#212858)\n\nFollow-up to:
https://github.com/elastic/kibana/pull/212077\n\nThis PR includes an API
test that covers `get_alerts_dataset_info` and\nwould have caught the
bug fixed in\nhttps://github.com//pull/212077.\n\nIt also
contains the following bug fixes:\n\n- Fix system message in
`select_relevant_fields`\n- Change prompt in `select_relevant_fields` so
that the LLM consistently\nuses the right format when
responding.","sha":"0fb83efd82ae3ebd8a9fe27813e436b80cd240d3","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport:prev-minor","backport:prev-major","Team:Obs
AI Assistant","ci:project-deploy-observability","v9.1.0"],"title":"[Obs
AI Assistant] Add API test for `get_alerts_dataset_info`
tool","number":212858,"url":"https://github.com/elastic/kibana/pull/212858","mergeCommit":{"message":"[Obs
AI Assistant] Add API test for `get_alerts_dataset_info` tool
(#212858)\n\nFollow-up to:
https://github.com/elastic/kibana/pull/212077\n\nThis PR includes an API
test that covers `get_alerts_dataset_info` and\nwould have caught the
bug fixed in\nhttps://github.com//pull/212077.\n\nIt also
contains the following bug fixes:\n\n- Fix system message in
`select_relevant_fields`\n- Change prompt in `select_relevant_fields` so
that the LLM consistently\nuses the right format when
responding.","sha":"0fb83efd82ae3ebd8a9fe27813e436b80cd240d3"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/212858","number":212858,"mergeCommit":{"message":"[Obs
AI Assistant] Add API test for `get_alerts_dataset_info` tool
(#212858)\n\nFollow-up to:
https://github.com/elastic/kibana/pull/212077\n\nThis PR includes an API
test that covers `get_alerts_dataset_info` and\nwould have caught the
bug fixed in\nhttps://github.com//pull/212077.\n\nIt also
contains the following bug fixes:\n\n- Fix system message in
`select_relevant_fields`\n- Change prompt in `select_relevant_fields` so
that the LLM consistently\nuses the right format when
responding.","sha":"0fb83efd82ae3ebd8a9fe27813e436b80cd240d3"}}]}]
BACKPORT-->

Co-authored-by: Søren Louv-Jansen <soren.louv@elastic.co>
@sorenlouv sorenlouv removed the backport:prev-major Backport to (8.x, 8.18, 8.17, 8.16) the previous major branch and other branches in development label Mar 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@viduni94 viduni94 viduni94 left review comments

@arturoliduena arturoliduena arturoliduena approved these changes

Assignees
No one assigned
Labels
backport:prev-minor Backport to (9.0) the previous minor version (i.e. one version back from main) ci:project-deploy-observability Create an Observability project release_note:skip Skip the PR/issue when compiling release notes Team:Obs AI Assistant Observability AI Assistant v8.19.0 v9.0.0 v9.1.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@sorenlouv @elasticmachine @kibanamachine @viduni94 @arturoliduena