feat(slo): cleanup temp summary documents task

[kdelemme]

Resolves #210194

🏇🏻 Summary

These temporary documents are primarily used to return a placeholder SLO Instance after it has been created in the UI, until the two layers of transform generate one or more actual SLO Instances for that SLO Definition. However, we deduplicate these documents only when both a temporary and a non-temporary summary document are returned in the same search query. This situation can never occur if there are more SLO Instances than the page size, resulting in temporary SLO Instances never being deleted.

This PR introduces a new task that runs every hour to delete temporary summary documents that are no longer needed, as real summary documents have been generated by the transforms. As a circuit breaker, we count the number of temp summary documents and return early if nothing is found.

[elasticmachine]

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)

[shahzad31]

ideally this should only need to be started if user adds SLOs for the first time.

[kdelemme]

Or in the case they have some existing SLOs and are only migrating from a previous version? Handling every cases might become tricky.
There is a feature flag that can be used to turn off this task, if someone's complaining about this task and don't use SLOs, it can be disabled easily.

[shahzad31]

how can it be disabled in cloud? cloud has an allow list for which kibana configs can be added.

[kdelemme]

Then the question is how can we update that allow list?

[shahzad31]

i am not sure myself, on cloud first region you can add all configs but not on other regions. but i think it's highly unlikely it will be needed in this case.

[shahzad31]

LGTM, with suggestions !!

IMO all those log.info should be log.debug, it's an unnecessary info for typical kibana user.

[kdelemme]

Thanks @shahzad31, I'll update the logger level used and the terminate after 👍🏻

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: c2e2a84

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
observability	1405	1406	+1
slo	1006	1007	+1
synthetics	970	971	+1
total			+3

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/slo-schema	183	187	+4

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
observability	1.3MB	1.3MB	+248.0B
slo	795.3KB	795.6KB	+246.0B
synthetics	788.4KB	788.7KB	+248.0B
total			+742.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
slo	42.2KB	42.2KB	+16.0B

Unknown metric groups

API count

id	before	after	diff
@kbn/slo-schema	183	187	+4

History

• 💔 Build #277349 failed 9f06a98
• 💚 Build #277042 succeeded 86c3927
• 💛 Build #276065 was flaky b2845e2
• 💚 Build #275754 succeeded 752ce38
• 💚 Build #275649 succeeded 01b88e4
• 💚 Build #275375 succeeded eb5128f

[ymao1]

Response Ops changes LGTM. New task type with 1h cadence and 2m timeout.