[ML] Trained Models: Count tests run against trained models #212927

rbrtj · 2025-03-03T14:30:47Z

Part of: #200725
This PR adds UI Counters for tests (success and failed) run against trained models.

elasticmachine · 2025-03-03T14:30:51Z

Pinging @elastic/ml-ui (:ml)

PR Overview

This PR adds UI counters to track the number of tests run against trained models (both success and failure) by updating usage events and passing the MlUsageCollection dependency to various inference modules.

Added new ML usage event constants for test events in the constants file.
Updated the inference classes and their constructors to accept and use MlUsageCollection.
Modified UI components and service hooks to supply the new dependency to model inference instances.

Reviewed Changes

File	Description
x-pack/platform/plugins/shared/ml/common/constants/usage_collection.ts	Adds new usage event constants for test counters.
x-pack/platform/plugins/shared/ml/public/application/model_management/test_models/models/inference_base.ts	Updates runInfer method to count test events using mlUsageCollection.
x-pack/platform/plugins/shared/ml/public/application/services/usage_collection.ts	Exports the MlUsageCollection type to support dependency injection.
x-pack/platform/plugins/shared/ml/public/application/model_management/test_models/selected_model.tsx	Retrieves and passes mlUsageCollection to the inference model constructors.
Other inference modules (ner, text classification, lang ident, text embedding, question answering, zero shot, text expansion, fill mask)	Their constructors are updated to receive mlUsageCollection and pass it to the superclass.

Copilot reviewed 12 out of 12 changed files in this pull request and generated no comments.

Comments suppressed due to low confidence (2)

x-pack/platform/plugins/shared/ml/common/constants/usage_collection.ts:15

[nitpick] The constant name is plural (TESTED_TRAINED_MODELS) but its value is singular ('tested_trained_model'); consider aligning these for consistency.

TESTED_TRAINED_MODELS: 'tested_trained_model',

x-pack/platform/plugins/shared/ml/public/application/model_management/test_models/models/inference_base.ts:317

Instead of using hardcoded event strings, consider using the defined constants (e.g., ML_USAGE_EVENT.TESTED_TRAINED_MODELS) to ensure consistency across the codebase.

this.mlUsageCollection.count('tested_trained_model');

PR Overview

This PR integrates UI counters for tracking the number of tests run against trained models by adding new usage event constants and wiring up usage collection calls into the inference workflows.

Added two new usage event constants (“tested_trained_model” and “test_failed_trained_model”) to support tracking the tests.
Injected and utilized the mlUsageCollection dependency into various inference models to record UI counter events on success and error paths.
Updated the usage collection service to export a type for mlUsageCollection, ensuring type consistency across the plugin.

Reviewed Changes

File	Description
x-pack/platform/plugins/shared/ml/common/constants/usage_collection.ts	Added new usage event constants for tracking trained model tests.
x-pack/platform/plugins/shared/ml/public/application/model_management/test_models/models/inference_base.ts	Integrated mlUsageCollection counting in the runInfer method for both success and failure scenarios.
x-pack/platform/plugins/shared/ml/public/application/services/usage_collection.ts	Exported the MlUsageCollection type from the usage collection provider.
x-pack/platform/plugins/shared/ml/public/application/model_management/test_models/selected_model.tsx	Updated usage context extraction and passed mlUsageCollection to inference model constructors.
Other model files (text_embedding, ner, text_classification, lang_ident, question_answering, zero_shot_classification, text_expansion, fill_mask)	Updated constructors to accept the mlUsageCollection dependency and pass it on to their super class.

Copilot reviewed 12 out of 12 changed files in this pull request and generated no comments.

jgowdyelastic · 2025-03-04T08:52:09Z

...m/plugins/shared/ml/public/application/model_management/test_models/models/inference_base.ts

@@ -84,7 +85,8 @@ export abstract class InferenceBase<TInferResponse> {
    protected readonly trainedModelsApi: ReturnType<typeof trainedModelsApiProvider>,
    protected readonly model: estypes.MlTrainedModelConfig,
    protected readonly inputType: INPUT_TYPE,
-    protected readonly deploymentId: string
+    protected readonly deploymentId: string,
+    protected readonly mlUsageCollection: MlUsageCollection


This isn't used in any of the derived classes and so could be private

Done in 5704fb9

jgowdyelastic · 2025-03-04T08:58:59Z

...m/plugins/shared/ml/public/application/model_management/test_models/models/inference_base.ts

@@ -312,6 +314,8 @@ export abstract class InferenceBase<TInferResponse> {
        DEFAULT_INFERENCE_TIME_OUT
      )) as unknown as TRawInferResponse;

+      this.mlUsageCollection.count('tested_trained_model');


Although it's good to have counts for when testing has been performed. I think it would be a lot more valuable to also track additional information, such as the model type and pytorch task type.

I assume we need event-based telemetry for that

After short discussion with @peteharverson, we agreed to collect EBT events with all relevant data, updated in: 5704fb9

jgowdyelastic

LGTM

elasticmachine · 2025-03-06T16:14:37Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 870de87

Failed CI Steps

Test Failures

[job] [logs] FTR Configs #45 / discover/context_awareness extension getAdditionalCellActions data view mode should render additional cell actions for logs data source
[job] [logs] FTR Configs #4 / Fleet Endpoints fleet_proxies_crud PUT /proxies/{itemId} should allow to update an existing fleet proxy

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`ml`	5.3MB	5.3MB	+415.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`ml`	79.0KB	79.6KB	+588.0B

History

💚 Build #280835 succeeded e7ed1a4
💔 Build #280785 failed b58ae2b
💚 Build #280353 succeeded 255bdae

cc @rbrtj

ui counters for tests run against trained models

7a4643b

rbrtj added :ml release_note:skip Skip the PR/issue when compiling release notes Feature:3rd Party Models ML 3rd party models Team:ML Team label for ML (also use :ml) backport:version Backport to applied version labels v9.1.0 v8.19.0 labels Mar 3, 2025

rbrtj requested review from darnautov, jgowdyelastic and Copilot March 3, 2025 14:30

rbrtj self-assigned this Mar 3, 2025

rbrtj requested a review from a team as a code owner March 3, 2025 14:30

Copilot AI reviewed Mar 3, 2025

View reviewed changes

use singular in event names

255bdae

rbrtj requested a review from Copilot March 3, 2025 14:36

Copilot AI reviewed Mar 3, 2025

View reviewed changes

jgowdyelastic reviewed Mar 4, 2025

View reviewed changes

Merge branch 'main' into ml-count-tests-run-against-trained-models

b86796c

rbrtj marked this pull request as draft March 4, 2025 13:58

rbrtj added 3 commits March 4, 2025 15:42

use ebt for tested models telemetry

5704fb9

cleanup usage collection types

4945f6c

remove another unused usage collection type

5a45d76

rbrtj marked this pull request as ready for review March 4, 2025 14:44

add telemetry method

b58ae2b

rbrtj requested a review from jgowdyelastic March 4, 2025 15:10

update trained models service test

e7ed1a4

jgowdyelastic approved these changes Mar 6, 2025

View reviewed changes

Merge branch 'main' into ml-count-tests-run-against-trained-models

870de87

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML] Trained Models: Count tests run against trained models #212927

[ML] Trained Models: Count tests run against trained models #212927

rbrtj commented Mar 3, 2025

elasticmachine commented Mar 3, 2025

jgowdyelastic Mar 4, 2025

rbrtj Mar 4, 2025

jgowdyelastic Mar 4, 2025 •

edited

Loading

darnautov Mar 4, 2025

rbrtj Mar 4, 2025 •

edited

Loading

jgowdyelastic left a comment

elasticmachine commented Mar 6, 2025

[ML] Trained Models: Count tests run against trained models #212927

Are you sure you want to change the base?

[ML] Trained Models: Count tests run against trained models #212927

Conversation

rbrtj commented Mar 3, 2025

elasticmachine commented Mar 3, 2025

Choose a reason for hiding this comment

PR Overview

Reviewed Changes

Choose a reason for hiding this comment

PR Overview

Reviewed Changes

jgowdyelastic Mar 4, 2025

Choose a reason for hiding this comment

rbrtj Mar 4, 2025

Choose a reason for hiding this comment

jgowdyelastic Mar 4, 2025 • edited Loading

Choose a reason for hiding this comment

darnautov Mar 4, 2025

Choose a reason for hiding this comment

rbrtj Mar 4, 2025 • edited Loading

Choose a reason for hiding this comment

jgowdyelastic left a comment

Choose a reason for hiding this comment

elasticmachine commented Mar 6, 2025

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Async chunks

Page load bundle

History

jgowdyelastic Mar 4, 2025 •

edited

Loading

rbrtj Mar 4, 2025 •

edited

Loading