[Fleet] Use streaming for package install instead of an assetsMap with everything loaded in memory

[nchaulet]

Summary

Related to #206842

That PR change how we install packages to not rely on a big assets map that contains all the assets, but by streaming the related assets when needed.

To not change to much the code, I still load smaller assets map for templates, pipelines, but in the future we could improve that by streaming here too.

Benchmark

I Tested installing a few packages, before that PR and it seems it consume a little less external memory, and it's a little slower.

Before

Memory metrics Installing aws package

After

Memory metrics Installing aws package

Todo

Kibana assets

• saved objects

ES assets

• ilm
• datastream ilm
• ingest pipelines
• transforms
• ml model
• templates

Package assets

• save package archive

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[nchaulet]

👀

[juliaElastic]

are you going to update this?

[nchaulet]

@elasticmachine merge upstream

[nchaulet]

@elasticmachine merge upstream

[juliaElastic]

I Tested installing a few packages, before that PR and it seems it consume a little less external memory, and it's a little slower.

Can you give a few examples on the difference in external memory and duration of package install?

[juliaElastic]

LGTM

[jen-huang]

Have we ran the install all packages script with these changes? Could be interesting to benchmark

[nchaulet]

Have we ran the install all packages script with these changes? Could be interesting to benchmark

@jen-huang here the results of the benchmark

With that PR

info INSTALLING packages
 info ✅ 1password-1.32.0  took 5.157s : 200
 info ✅ aws-2.41.1  took 15.304s : 200
 info ✅ awsfargate-1.3.0  took 3.83s : 200
 info ✅ abnormal_security-1.4.0  took 3.081s : 200
 info ✅ ti_abusech-2.6.0  took 6.243s : 200
 info ✅ entityanalytics_ad-0.7.0  took 2.022s : 200
 info ✅ activemq-1.8.0  took 3.091s : 200
 info ✅ admin_by_request_epm-0.2.0  took 3.078s : 200
 info ✅ airflow-0.10.0  took 3.066s : 200
 info ✅ akamai-2.28.0  took 3.064s : 200
 info ✅ ti_otx-1.28.0  took 3.185s : 200
 info ✅ aws_bedrock-1.1.0  took 2.036s : 200
 info ✅ awsfirehose-1.5.2  took 2.053s : 200
 info ✅ aws_mq-0.2.0  took 2.044s : 200
 info ✅ amazon_security_lake-2.4.0  took 4.166s : 200
 info ✅ apache-1.27.0  took 3.05s : 200
 info ✅ apache_spark-1.4.0  took 3.064s : 200
 info ✅ apache_tomcat-1.9.0  took 3.096s : 200
 info ❌ netscout-0.1.2  took 0.32s : {"body":{"statusCode":400,"error":"Bad Request","message":"Invalid top-level package manifest at top-level directory netscout-0.1.2 (package name: netscout): one or more fields missing of name, version, title, owner."},"status":400,"took":0.32,"didError":true}
 info ✅ atlassian_bitbucket-2.4.0  took 1.698s : 200
 info ✅ atlassian_confluence-1.28.0  took 2.062s : 200
 info ✅ atlassian_jira-1.29.0  took 2.064s : 200
 info ✅ auditd_manager-1.18.0  took 2.042s : 200
 info ✅ auth0-1.21.0  took 2.038s : 200
 info ✅ azure_app_service-0.6.0  took 2.064s : 200
 info ✅ azure_application_insights-1.8.0  took 2.039s : 200
 info ✅ azure_billing-1.8.0  took 2.05s : 200
 info ✅ azure_frontdoor-2.2.0  took 2.04s : 200
 info ✅ azure_functions-0.9.0  took 2.052s : 200
 info ✅ azure-1.22.1  took 3.124s : 200
 info ✅ azure_network_watcher_nsg-1.3.0  took 1.992s : 200
 info ✅ azure_network_watcher_vnet-1.3.0  took 2.061s : 200
 info ✅ azure_openai-1.6.0  took 2.047s : 200
 info ✅ azure_metrics-1.7.0  took 3.16s : 200
 info ✅ bbot-1.3.0  took 2.017s : 200
 info ✅ barracuda_cloudgen_firewall-1.15.0  took 2.066s : 200
 info ✅ barracuda-1.18.0  took 2.056s : 200
 info ✅ beyondinsight_password_safe-0.2.0  took 2.051s : 200
 info ❌ f5-0.1.0  took 0.289s : {"body":{"statusCode":400,"error":"Bad Request","message":"Invalid top-level package manifest at top-level directory f5-0.1.0 (package name: f5): one or more fields missing of name, version, title, owner."},"status":400,"took":0.289,"didError":true}
 info ✅ bitdefender-2.4.0  took 3.809s : 200
 info ✅ bitwarden-1.17.0  took 2.05s : 200
 info ❌ bluecoat-0.1.2  took 0.297s : {"body":{"statusCode":400,"error":"Bad Request","message":"Invalid top-level package manifest at top-level directory bluecoat-0.1.2 (package name: bluecoat): one or more fields missing of name, version, title, owner."},"status":400,"took":0.297,"didError":true}
 info ✅ box_events-2.12.0  took 2.766s : 200
 info ✅ hid_bravura_monitor-1.20.0  took 3.207s : 200
 info ✅ cisa_kevs-1.5.0  took 2.049s : 200
 info ✅ canva-0.5.0  took 2.057s : 200
 info ✅ cassandra-1.18.0  took 2.076s : 200
 info ✅ ceph-1.8.0  took 3.1s : 200
 info ✅ checkpoint-0.5.0  took 2.03s : 200
 info ✅ checkpoint_email-0.5.0  took 2.051s : 200
 info ✅ checkpoint_harmony_endpoint-0.4.0  took 2.058s : 200
 info ✅ cisco_meraki-1.28.0  took 2.034s : 200
 info ✅ cisco_meraki_metrics-0.3.1  took 2.036s : 200
 info ✅ cisco_secure_endpoint-2.29.0  took 2.068s : 200
 info ✅ cisco_umbrella-1.29.0  took 2.055s : 200
 info ✅ citrix_adc-1.15.0  took 3.181s : 200
 info ✅ claroty_ctd-0.5.0  took 2.021s : 200
 info ✅ cloud_asset_inventory-0.8.0  took 2.05s : 200
 info ✅ cloudflare-2.30.0  took 2.042s : 200
 info ✅ cloudflare_logpush-1.34.1  took 4.147s : 200
 info ✅ cockroachdb-1.13.0  took 3.018s : 200
 info ✅ ti_cif3-1.17.0  took 4.099s : 200
 info ✅ containerd-0.5.0  took 2.046s : 200
 info ✅ coredns-0.10.0  took 2.034s : 200
 info ✅ corelight-0.8.0  took 4.12s : 200
 info ✅ couchdb-1.5.0  took 2.028s : 200
 info ✅ couchbase-1.9.0  took 3.092s : 200
 info ✅ cribl-0.5.0  took 2.007s : 200
 info ✅ crowdstrike-1.52.1  took 3.107s : 200
 info ✅ ti_crowdstrike-2.4.0  took 4.241s : 200
 info ✅ httpjson-1.22.0  took 2.042s : 200
 info ✅ cel-1.15.0  took 2.032s : 200
 info ✅ aws_logs-1.7.0  took 3.121s : 200
 info ✅ azure_blob_storage-2.2.0  took 2.049s : 200
 info ✅ azure_logs-0.2.0  took 2.049s : 200
 info ✅ filestream-1.0.1  took 2.066s : 200
 info ✅ google_cloud_storage-2.2.0  took 2.057s : 200
 info ✅ gcp_pubsub-2.2.0  took 2.034s : 200
 info ✅ http_endpoint-2.5.0  took 2.042s : 200
 info ✅ journald-1.1.2  took 2.054s : 200
 info ✅ kafka_log-1.8.1  took 2.061s : 200
 info ✅ log-1.1.2  took 2.053s : 200
 info ✅ ti_custom-0.8.0  took 3.112s : 200
 info ✅ websocket-0.2.0  took 2.016s : 200
 info ✅ windows_etw-0.3.0  took 2.051s : 200
 info ✅ winlog-2.4.0  took 2.064s : 200
 info ❌ unifiedlogs-0.3.0  took 0.097s : {"body":{"statusCode":404,"error":"Not Found","message":"[unifiedlogs] package not found in registry"},"status":404,"took":0.097,"didError":true}
 info ✅ cyberark_epm-0.2.0  took 2.999s : 200
 info ✅ cyberarkpas-2.27.0  took 3.059s : 200
 info ✅ cyberark_pta-1.12.0  took 3.106s : 200
 info ✅ cybereason-1.3.0  took 4.064s : 200
 info ✅ ti_cybersixgill-1.33.0  took 3.138s : 200
 info ✅ cylance-0.23.0  took 2.04s : 200
 info ✅ darktrace-1.23.0  took 3.094s : 200
 info ✅ digital_guardian-1.5.0  took 3.06s : 200
 info ✅ docker-2.14.0  took 3.149s : 200
 info ✅ ti_domaintools-0.2.0  took 3.114s : 200
 info ✅ eset_protect-1.6.0  took 2.051s : 200
 info ✅ ti_eset-1.6.0  took 8.273s : 200
 info ✅ ti_eclecticiq-1.4.0  took 3.018s : 200
 info ✅ apm-9.0.0-preview-1738343125  took 2.045s : 200
 info ❌ elastic_connectors-1.0.2  took 0.431s : {"body":{"statusCode":403,"error":"Forbidden","message":"elastic_connectors installation is not authorized"},"status":403,"took":0.431,"didError":true}
 info ✅ endpoint-9.1.0-prerelease.0  took 4.807s : 200
 info ✅ elastic_package_registry-0.3.1  took 2.023s : 200
 info ✅ synthetics-1.4.1  took 4.213s : 200
 info ✅ envoyproxy-0.3.0  took 2.006s : 200
 info ✅ f5_bigip-1.26.0  took 3.092s : 200
 info ✅ falco-1.3.0  took 2.031s : 200
 info ✅ fireeye-1.25.0  took 2.059s : 200
 info ✅ first_epss-0.4.0  took 3.102s : 200
 info ✅ fleet_server-1.6.0  took 2.058s : 200
 info ✅ forcepoint_web-1.13.0  took 3.087s : 200
 info ✅ forgerock-1.21.0  took 3.209s : 200
 info ✅ gcp_metrics-0.3.0  took 2.024s : 200
 info ✅ gcp_vertexai-1.1.0  took 2.046s : 200
 info ✅ gigamon-1.4.0  took 2.048s : 200
 info ✅ github-2.4.0  took 6.435s : 200
 info ✅ gitlab-2.2.0  took 3.09s : 200
 info ✅ golang-1.7.0  took 3.089s : 200
 info ✅ gcp-2.41.0  took 4.197s : 200
 info ✅ santa-3.24.0  took 1.937s : 200
 info ✅ google_scc-1.8.0  took 3.21s : 200
 info ✅ google_workspace-2.31.0  took 4.201s : 200
 info ✅ haproxy-1.16.0  took 3.052s : 200
 info ✅ hadoop-1.8.1  took 3.088s : 200
 info ✅ ibmmq-1.6.0  took 3.065s : 200
 info ✅ iis-1.22.0  took 3.096s : 200
 info ✅ imperva_cloud_waf-1.8.0  took 2.046s : 200
 info ❌ imperva-0.1.2  took 0.327s : {"body":{"statusCode":400,"error":"Bad Request","message":"Invalid top-level package manifest at top-level directory imperva-0.1.2 (package name: imperva): one or more fields missing of name, version, title, owner."},"status":400,"took":0.327,"didError":true}
 info ✅ influxdb-0.11.0  took 1.727s : 200
 info ✅ infoblox_bloxone_ddi-1.21.0  took 3.077s : 200
 info ❌ infoblox-0.1.2  took 0.329s : {"body":{"statusCode":400,"error":"Bad Request","message":"Invalid top-level package manifest at top-level directory infoblox-0.1.2 (package name: infoblox): one or more fields missing of name, version, title, owner."},"status":400,"took":0.329,"didError":true}
 info ✅ infoblox_nios-1.28.0  took 4.064s : 200
 info ✅ istio-0.7.0  took 2.051s : 200
 info ✅ jamf_compliance_reporter-1.16.0  took 3.197s : 200
 info ✅ jamf_pro-0.5.0  took 3.045s : 200
 info ✅ jamf_protect-2.10.0  took 3.153s : 200
 info ✅ jolokia-0.7.0  took 1.963s : 200
 info ✅ jumpcloud-1.15.0  took 2.054s : 200
 info ❌ juniper-0.1.0  took 0.333s : {"body":{"statusCode":400,"error":"Bad Request","message":"Invalid top-level package manifest at top-level directory juniper-0.1.0 (package name: juniper): one or more fields missing of name, version, title, owner."},"status":400,"took":0.333,"didError":true}
 info ✅ kafka-1.18.0  took 2.77s : 200
 info ✅ keycloak-1.26.0  took 2.046s : 200
 info ✅ kubernetes-1.80.1  took 5.19s : 200
 info ✅ kubernetes_otel-1.1.0  took 1.938s : 200
 info ✅ lastpass-1.20.0  took 3.111s : 200
 info ✅ linux-0.7.1  took 3.095s : 200
 info ✅ lumos-1.5.0  took 2.002s : 200
 info ✅ lyve_cloud-1.17.0  took 3.097s : 200
 info ✅ ti_misp-1.38.0  took 4.264s : 200
 info ✅ ti_maltiverse-1.5.0  took 4.157s : 200
 info ✅ ti_mandiant_advantage-1.9.0  took 2.047s : 200
 info ✅ mattermost-2.4.0  took 2.044s : 200
 info ✅ memcached-1.7.0  took 2.041s : 200
 info ✅ menlo-1.5.0  took 2.052s : 200
 info ✅ microsoft_dhcp-1.26.0  took 2.062s : 200
 info ✅ microsoft_dnsserver-1.2.0  took 3.074s : 200
 info ✅ microsoft_defender_cloud-2.4.0  took 2.044s : 200
 info ✅ microsoft_defender_endpoint-2.30.0  took 2.06s : 200
 info ✅ entityanalytics_entra_id-1.6.0  took 2.069s : 200
 info ✅ microsoft_exchange_online_message_trace-1.27.0  took 2.036s : 200
 info ✅ microsoft_exchange_server-1.3.1  took 3.103s : 200
 info ✅ m365_defender-2.23.0  took 3.171s : 200
 info ✅ o365_metrics-0.6.2  took 9.499s : 200
 info ✅ microsoft_sqlserver-2.12.0  took 4.145s : 200
 info ✅ microsoft_sentinel-0.4.0  took 4.329s : 200
 info ✅ mimecast-2.6.2  took 3.17s : 200
 info ✅ mongodb-1.19.0  took 3.056s : 200
 info ✅ mongodb_atlas-0.2.1  took 4.306s : 200
 info ✅ mysql-1.26.0  took 3.061s : 200
 info ✅ mysql_enterprise-1.16.0  took 2.068s : 200
 info ✅ nats-1.9.0  took 3.047s : 200
 info ✅ nagios_xi-1.5.0  took 9.187s : 200
 info ✅ netskope-1.23.0  took 3.106s : 200
 info ✅ nginx-1.25.0  took 3.173s : 200
 info ✅ nginx_ingress_controller-1.11.0  took 3.065s : 200
 info ❌ nginx_ingress_controller_otel-0.1.0  took 0.106s : {"body":{"statusCode":404,"error":"Not Found","message":"[nginx_ingress_controller_otel] package not found in registry"},"status":404,"took":0.106,"didError":true}
 info ✅ o365-0.2.6  took 2.96s : 200
 info ✅ okta-3.5.0  took 3.121s : 200
 info ✅ entityanalytics_okta-2.1.0  took 3.19s : 200
 info ✅ openai-0.3.0  took 3.161s : 200
 info ✅ ti_opencti-2.6.0  took 4.083s : 200
 info ✅ opencanary-0.4.0  took 2.037s : 200
 info ✅ oracle-1.29.0  took 3.109s : 200
 info ✅ oracle_weblogic-1.9.0  took 3.1s : 200
 info ✅ osquery_manager-1.16.0  took 3.145s : 200
 info ✅ php_fpm-1.5.0  took 3.059s : 200
 info ✅ panw_cortex_xdr-2.1.0  took 3.094s : 200
 info ✅ panw-0.5.0  took 3.108s : 200
 info ✅ panw_metrics-0.2.0  took 3.174s : 200
 info ✅ prisma_access-1.5.0  took 4.103s : 200
 info ✅ prisma_cloud-2.1.0  took 3.189s : 200
 info ✅ ping_federate-0.2.0  took 3.043s : 200
 info ✅ ping_one-1.19.0  took 3.09s : 200
 info ✅ pps-0.5.0  took 2.052s : 200
 info ✅ postgresql-1.27.0  took 3.111s : 200
 info ✅ prometheus-1.23.0  took 3.074s : 200
 info ✅ prometheus_input-0.7.0  took 2.038s : 200
 info ✅ proofpoint_on_demand-1.5.0  took 4.268s : 200
 info ✅ proofpoint_tap-1.27.0  took 3.16s : 200
 info ✅ pulse_connect_secure-2.5.0  took 2.026s : 200
 info ✅ qualys_vmdr-6.2.0  took 3.112s : 200
 info ✅ rabbitmq-1.18.0  took 3.142s : 200
 info ❌ radware-0.1.2  took 0.396s : {"body":{"statusCode":400,"error":"Bad Request","message":"Invalid top-level package manifest at top-level directory radware-0.1.2 (package name: radware): one or more fields missing of name, version, title, owner."},"status":400,"took":0.396,"didError":true}
 info ✅ rapid7_insightvm-1.16.0  took 2.755s : 200
 info ✅ ti_rapid7_threat_command-2.4.0  took 7.165s : 200
 info ✅ ti_recordedfuture-1.29.0  took 4.112s : 200
 info ✅ redis-1.19.0  took 3.195s : 200
 info ✅ redisenterprise-0.12.0  took 3.077s : 200
 info ✅ rubrik-0.8.0  took 3.105s : 200
 info ✅ sql-0.6.0  took 2.018s : 200
 info ✅ stan-1.9.0  took 3.1s : 200
 info ✅ sailpoint_identity_sc-0.2.0  took 2.121s : 200
 info ✅ salesforce-1.1.0  took 3.212s : 200
 info ✅ cloud_security_posture-1.13.0-preview04  took 4.205s : 200
 info ✅ sentinel_one-1.29.1  took 3.084s : 200
 info ✅ sentinel_one_cloud_funnel-1.9.0  took 4.205s : 200
 info ✅ servicenow-0.10.0  took 4.041s : 200
 info ✅ slack-1.25.0  took 2.048s : 200
 info ❌ sonicwall-0.1.2  took 0.34s : {"body":{"statusCode":400,"error":"Bad Request","message":"Invalid top-level package manifest at top-level directory sonicwall-0.1.2 (package name: sonicwall): one or more fields missing of name, version, title, owner."},"status":400,"took":0.34,"didError":true}
 info ✅ sophos_central-1.19.0  took 2.732s : 200
 info ✅ spring_boot-1.8.0  took 3.1s : 200
 info ✅ spycloud-1.3.0  took 3.071s : 200
 info ❌ squid-0.1.2  took 0.316s : {"body":{"statusCode":400,"error":"Bad Request","message":"Invalid top-level package manifest at top-level directory squid-0.1.2 (package name: squid): one or more fields missing of name, version, title, owner."},"status":400,"took":0.316,"didError":true}
 info ✅ statsd_input-0.5.0  took 1.72s : 200
 info ✅ sublime_security-1.6.0  took 3.127s : 200
 info ✅ suricata-0.5.0  took 3.096s : 200
 info ✅ symantec_endpoint-2.19.0  took 3.091s : 200
 info ✅ symantec_endpoint_security-1.8.0  took 3.091s : 200
 info ✅ sysdig-0.3.0  took 3.077s : 200
 info ✅ system-1.67.0  took 4.255s : 200
 info ✅ tychon-0.3.0  took 29.538s : 200
 info ✅ tanium-1.15.0  took 2.767s : 200
 info ✅ teleport-1.3.0  took 3.06s : 200
 info ✅ tenable_sc-1.29.0  took 3.144s : 200
 info ✅ tenable_io-3.8.0  took 3.073s : 200
 info ✅ ti_util-1.7.0  took 2.075s : 200
 info ✅ threat_map-0.2.0  took 2.062s : 200
 info ✅ ti_threatconnect-1.8.0  took 5.253s : 200
 info ✅ ti_threatq-1.31.0  took 4.148s : 200
 info ✅ thycotic_ss-1.12.0  took 3.085s : 200
 info ✅ tines-1.15.0  took 3.141s : 200
 info ✅ tomcat-1.12.0  took 3.082s : 200
 info ✅ traefik-2.5.0  took 3.108s : 200
 info ✅ trellix_edr_cloud-1.7.0  took 3.083s : 200
 info ✅ trellix_epo_cloud-1.15.0  took 3.164s : 200
 info ✅ trendmicro-2.7.0  took 3.174s : 200
 info ✅ trend_micro_vision_one-1.26.0  took 3.085s : 200
 info ✅ carbonblack_edr-1.21.0  took 2.04s : 200
 info ✅ vsphere-1.18.1  took 3.117s : 200
 info ✅ vectra_detect-1.13.0  took 3.068s : 200
 info ✅ websphere_application_server-1.6.0  took 3.108s : 200
 info ✅ windows-2.5.0  took 4.192s : 200
 info ✅ wiz-2.8.1  took 5.367s : 200
 info ✅ zeronetworks-1.18.0  took 3.063s : 200
 info ✅ zerofox-1.28.0  took 3.061s : 200
 info ✅ zookeeper-1.14.0  took 3.187s : 200
 info ✅ zoom-1.22.0  took 3.079s : 200
 info ✅ zscaler_zia-3.7.1  took 4.205s : 200
 info ❌ zscaler-0.1.2  took 0.255s : {"body":{"statusCode":400,"error":"Bad Request","message":"Invalid top-level package manifest at top-level directory zscaler-0.1.2 (package name: zscaler): one or more fields missing of name, version, title, owner."},"status":400,"took":0.255,"didError":true}
 info ✅ zscaler_zpa-1.21.0  took 2.804s : 200
 info ✅ authentik-1.3.0  took 3.088s : 200
 info ✅ blacklens-0.3.0  took 3.089s : 200
 info ✅ etcd-1.4.0  took 3.069s : 200
ERROR There were 13 errors, exiting with code

Before

 info INSTALLING packages
 info ✅ 1password-1.32.0  took 5.285s : 200
 info ✅ aws-2.41.1  took 6.407s : 200
 info ✅ awsfargate-1.3.0  took 1.829s : 200
 info ✅ abnormal_security-1.4.0  took 3.193s : 200
 info ✅ ti_abusech-2.6.0  took 6.18s : 200
 info ✅ entityanalytics_ad-0.7.0  took 1.997s : 200
 info ✅ activemq-1.8.0  took 2.128s : 200
 info ✅ admin_by_request_epm-0.2.0  took 1.988s : 200
 info ✅ airflow-0.10.0  took 2.029s : 200
 info ✅ akamai-2.28.0  took 2.038s : 200
 info ✅ ti_otx-1.28.0  took 4.319s : 200
 info ✅ aws_bedrock-1.1.0  took 2.035s : 200
 info ✅ awsfirehose-1.5.2  took 2.034s : 200
 info ✅ aws_mq-0.2.0  took 2.039s : 200
 info ✅ amazon_security_lake-2.4.0  took 4.126s : 200
 info ✅ apache-1.27.0  took 2.014s : 200
 info ✅ apache_spark-1.4.0  took 2.038s : 200
 info ✅ apache_tomcat-1.9.0  took 2.067s : 200
 info ❌ netscout-0.1.2  took 0.106s : {"body":{"statusCode":400,"error":"Bad Request","message":"Invalid top-level package manifest at top-level directory netscout-0.1.2 (package name: netscout): one or more fields missing of name, version, title, owner."},"status":400,"took":0.106,"didError":true}
 info ✅ atlassian_bitbucket-2.4.0  took 1.89s : 200
 info ✅ atlassian_confluence-1.28.0  took 2.047s : 200
 info ✅ atlassian_jira-1.29.0  took 2.032s : 200
 info ✅ auditd_manager-1.18.0  took 4.061s : 200
 info ✅ auth0-1.21.0  took 3.077s : 200
 info ✅ azure_app_service-0.6.0  took 2.044s : 200
 info ✅ azure_application_insights-1.8.0  took 3.048s : 200
 info ✅ azure_billing-1.8.0  took 2.025s : 200
 info ✅ azure_frontdoor-2.2.0  took 2.15s : 200
 info ✅ azure_functions-0.9.0  took 2.047s : 200
 info ✅ azure-1.22.1  took 2.098s : 200
 info ✅ azure_network_watcher_nsg-1.3.0  took 1.974s : 200
 info ✅ azure_network_watcher_vnet-1.3.0  took 2.041s : 200
 info ✅ azure_openai-1.6.0  took 2.049s : 200
 info ✅ azure_metrics-1.7.0  took 2.105s : 200
 info ✅ bbot-1.3.0  took 1.972s : 200
 info ✅ barracuda_cloudgen_firewall-1.15.0  took 2.037s : 200
 info ✅ barracuda-1.18.0  took 3.212s : 200
 info ✅ beyondinsight_password_safe-0.2.0  took 2.044s : 200
 info ❌ f5-0.1.0  took 0.109s : {"body":{"statusCode":400,"error":"Bad Request","message":"Invalid top-level package manifest at top-level directory f5-0.1.0 (package name: f5): one or more fields missing of name, version, title, owner."},"status":400,"took":0.109,"didError":true}
 info ✅ bitdefender-2.4.0  took 1.938s : 200
 info ✅ bitwarden-1.17.0  took 2.05s : 200
 info ❌ bluecoat-0.1.2  took 0.093s : {"body":{"statusCode":400,"error":"Bad Request","message":"Invalid top-level package manifest at top-level directory bluecoat-0.1.2 (package name: bluecoat): one or more fields missing of name, version, title, owner."},"status":400,"took":0.093,"didError":true}
 info ✅ box_events-2.12.0  took 2.964s : 200
 info ✅ hid_bravura_monitor-1.20.0  took 3.078s : 200
 info ✅ cisa_kevs-1.5.0  took 2.023s : 200
 info ✅ canva-0.5.0  took 2.038s : 200
 info ✅ cassandra-1.18.0  took 2.056s : 200
 info ✅ ceph-1.8.0  took 2.067s : 200
 info ✅ checkpoint-0.5.0  took 2.006s : 200
 info ✅ checkpoint_email-0.5.0  took 2.035s : 200
 info ✅ checkpoint_harmony_endpoint-0.4.0  took 3.082s : 200
 info ✅ cisco_meraki-1.28.0  took 2.03s : 200
 info ✅ cisco_meraki_metrics-0.3.1  took 2.02s : 200
 info ✅ cisco_secure_endpoint-2.29.0  took 2.095s : 200
 info ✅ cisco_umbrella-1.29.0  took 1.986s : 200
 info ✅ citrix_adc-1.15.0  took 2.086s : 200
 info ✅ claroty_ctd-0.5.0  took 3.124s : 200
 info ✅ cloud_asset_inventory-0.8.0  took 3.122s : 200
 info ✅ cloudflare-2.30.0  took 2.051s : 200
 info ✅ cloudflare_logpush-1.34.1  took 3.148s : 200
 info ✅ cockroachdb-1.13.0  took 1.96s : 200
 info ✅ ti_cif3-1.17.0  took 4.155s : 200
 info ✅ containerd-0.5.0  took 3.15s : 200
 info ✅ coredns-0.10.0  took 2.027s : 200
 info ✅ corelight-0.8.0  took 3.053s : 200
 info ✅ couchdb-1.5.0  took 2.04s : 200
 info ✅ couchbase-1.9.0  took 2.086s : 200
 info ✅ cribl-0.5.0  took 2.002s : 200
 info ✅ crowdstrike-1.52.1  took 3.133s : 200
 info ✅ ti_crowdstrike-2.4.0  took 5.049s : 200
 info ✅ httpjson-1.22.0  took 2.059s : 200
 info ✅ cel-1.15.0  took 2.026s : 200
 info ✅ aws_logs-1.7.0  took 2.043s : 200
 info ✅ azure_blob_storage-2.2.0  took 2.031s : 200
 info ✅ azure_logs-0.2.0  took 2.059s : 200
 info ✅ filestream-1.0.1  took 2.034s : 200
 info ✅ google_cloud_storage-2.2.0  took 2.027s : 200
 info ✅ gcp_pubsub-2.2.0  took 2.037s : 200
 info ✅ http_endpoint-2.5.0  took 2.038s : 200
 info ✅ journald-1.1.2  took 2.03s : 200
 info ✅ kafka_log-1.8.1  took 2.06s : 200
 info ✅ log-1.1.2  took 2.028s : 200
 info ✅ ti_custom-0.8.0  took 4.092s : 200
 info ✅ websocket-0.2.0  took 2.004s : 200
 info ✅ windows_etw-0.3.0  took 2.044s : 200
 info ✅ winlog-2.4.0  took 2.038s : 200
 info ❌ unifiedlogs-0.3.0  took 0.06s : {"body":{"statusCode":404,"error":"Not Found","message":"[unifiedlogs] package not found in registry"},"status":404,"took":0.06,"didError":true}
 info ✅ cyberark_epm-0.2.0  took 2s : 200
 info ✅ cyberarkpas-2.27.0  took 2.025s : 200
 info ✅ cyberark_pta-1.12.0  took 2.041s : 200
 info ✅ cybereason-1.3.0  took 2.071s : 200
 info ✅ ti_cybersixgill-1.33.0  took 4.129s : 200
 info ✅ cylance-0.23.0  took 2.028s : 200
 info ✅ darktrace-1.23.0  took 3.374s : 200
 info ✅ digital_guardian-1.5.0  took 3.043s : 200
 info ✅ docker-2.14.0  took 2.088s : 200
 info ✅ ti_domaintools-0.2.0  took 7.064s : 200
 info ✅ eset_protect-1.6.0  took 2.046s : 200
 info ✅ ti_eset-1.6.0  took 8.223s : 200
 info ✅ ti_eclecticiq-1.4.0  took 3.132s : 200
 info ✅ apm-9.0.0-preview-1738343125  took 2.005s : 200
 info ❌ elastic_connectors-1.0.2  took 0.18s : {"body":{"statusCode":403,"error":"Forbidden","message":"elastic_connectors installation is not authorized"},"status":403,"took":0.18,"didError":true}
 info ✅ endpoint-9.1.0-prerelease.0  took 5.125s : 200
 info ✅ elastic_package_registry-0.3.1  took 1.945s : 200
 info ✅ synthetics-1.4.1  took 3.11s : 200
 info ✅ envoyproxy-0.3.0  took 1.993s : 200
 info ✅ f5_bigip-1.26.0  took 3.072s : 200
 info ✅ falco-1.3.0  took 2.026s : 200
 info ✅ fireeye-1.25.0  took 2.036s : 200
 info ✅ first_epss-0.4.0  took 2.045s : 200
 info ✅ fleet_server-1.6.0  took 2.092s : 200
 info ✅ forcepoint_web-1.13.0  took 2.032s : 200
 info ✅ forgerock-1.21.0  took 3.095s : 200
 info ✅ gcp_metrics-0.3.0  took 1.987s : 200
 info ✅ gcp_vertexai-1.1.0  took 2.05s : 200
 info ✅ gigamon-1.4.0  took 2.047s : 200
 info ✅ github-2.4.0  took 6.241s : 200
 info ✅ gitlab-2.2.0  took 3.033s : 200
 info ✅ golang-1.7.0  took 2.024s : 200
 info ✅ gcp-2.41.0  took 3.136s : 200
 info ✅ santa-3.24.0  took 1.954s : 200
 info ✅ google_scc-1.8.0  took 3.086s : 200
 info ✅ google_workspace-2.31.0  took 3.16s : 200
 info ✅ haproxy-1.16.0  took 3.038s : 200
 info ✅ hadoop-1.8.1  took 2.051s : 200
 info ✅ ibmmq-1.6.0  took 2.06s : 200
 info ✅ iis-1.22.0  took 3.058s : 200
 info ✅ imperva_cloud_waf-1.8.0  took 2.009s : 200
 info ❌ imperva-0.1.2  took 0.101s : {"body":{"statusCode":400,"error":"Bad Request","message":"Invalid top-level package manifest at top-level directory imperva-0.1.2 (package name: imperva): one or more fields missing of name, version, title, owner."},"status":400,"took":0.101,"didError":true}
 info ✅ influxdb-0.11.0  took 1.985s : 200
 info ✅ infoblox_bloxone_ddi-1.21.0  took 3.029s : 200
 info ❌ infoblox-0.1.2  took 0.115s : {"body":{"statusCode":400,"error":"Bad Request","message":"Invalid top-level package manifest at top-level directory infoblox-0.1.2 (package name: infoblox): one or more fields missing of name, version, title, owner."},"status":400,"took":0.115,"didError":true}
 info ✅ infoblox_nios-1.28.0  took 1.919s : 200
 info ✅ istio-0.7.0  took 2.042s : 200
 info ✅ jamf_compliance_reporter-1.16.0  took 2.076s : 200
 info ✅ jamf_pro-0.5.0  took 1.996s : 200
 info ✅ jamf_protect-2.10.0  took 3.279s : 200
 info ✅ jolokia-0.7.0  took 1.936s : 200
 info ✅ jumpcloud-1.15.0  took 2.049s : 200
 info ❌ juniper-0.1.0  took 0.188s : {"body":{"statusCode":400,"error":"Bad Request","message":"Invalid top-level package manifest at top-level directory juniper-0.1.0 (package name: juniper): one or more fields missing of name, version, title, owner."},"status":400,"took":0.188,"didError":true}
 info ✅ kafka-1.18.0  took 2.943s : 200
 info ✅ keycloak-1.26.0  took 1.991s : 200
 info ✅ kubernetes-1.80.1  took 3.241s : 200
 info ✅ kubernetes_otel-1.1.0  took 1.924s : 200
 info ✅ lastpass-1.20.0  took 3.07s : 200
 info ✅ linux-0.7.1  took 3.156s : 200
 info ✅ lumos-1.5.0  took 1.987s : 200
 info ✅ lyve_cloud-1.17.0  took 4.046s : 200
 info ✅ ti_misp-1.38.0  took 4.148s : 200
 info ✅ ti_maltiverse-1.5.0  took 3.059s : 200
 info ✅ ti_mandiant_advantage-1.9.0  took 3.056s : 200
 info ✅ mattermost-2.4.0  took 2.04s : 200
 info ✅ memcached-1.7.0  took 2.04s : 200
 info ✅ menlo-1.5.0  took 2.041s : 200
 info ✅ microsoft_dhcp-1.26.0  took 2.033s : 200
 info ✅ microsoft_dnsserver-1.2.0  took 3.118s : 200
 info ✅ microsoft_defender_cloud-2.4.0  took 3.101s : 200
 info ✅ microsoft_defender_endpoint-2.30.0  took 3.143s : 200
 info ✅ entityanalytics_entra_id-1.6.0  took 3.14s : 200
 info ✅ microsoft_exchange_online_message_trace-1.27.0  took 2.015s : 200
 info ✅ microsoft_exchange_server-1.3.1  took 3.161s : 200
 info ✅ m365_defender-2.23.0  took 3.074s : 200
 info ✅ o365_metrics-0.6.2  took 7.276s : 200
 info ✅ microsoft_sqlserver-2.12.0  took 2.974s : 200
 info ✅ microsoft_sentinel-0.4.0  took 4.083s : 200
 info ✅ mimecast-2.6.2  took 3.14s : 200
 info ✅ mongodb-1.19.0  took 2.998s : 200
 info ✅ mongodb_atlas-0.2.1  took 3.062s : 200
 info ✅ mysql-1.26.0  took 3.041s : 200
 info ✅ mysql_enterprise-1.16.0  took 2.009s : 200
 info ✅ nats-1.9.0  took 3.202s : 200
 info ✅ nagios_xi-1.5.0  took 3.098s : 200
 info ✅ netskope-1.23.0  took 3.112s : 200
 info ✅ nginx-1.25.0  took 3.08s : 200
 info ✅ nginx_ingress_controller-1.11.0  took 3.044s : 200
 info ❌ nginx_ingress_controller_otel-0.1.0  took 0.061s : {"body":{"statusCode":404,"error":"Not Found","message":"[nginx_ingress_controller_otel] package not found in registry"},"status":404,"took":0.061,"didError":true}
 info ✅ o365-0.2.6  took 1.97s : 200
 info ✅ okta-3.5.0  took 3.07s : 200
 info ✅ entityanalytics_okta-2.1.0  took 3.082s : 200
 info ✅ openai-0.3.0  took 3.095s : 200
 info ✅ ti_opencti-2.6.0  took 4.237s : 200
 info ✅ opencanary-0.4.0  took 2.014s : 200
 info ✅ oracle-1.29.0  took 3.189s : 200
 info ✅ oracle_weblogic-1.9.0  took 3.067s : 200
 info ✅ osquery_manager-1.16.0  took 3.145s : 200
 info ✅ php_fpm-1.5.0  took 3.03s : 200
 info ✅ panw_cortex_xdr-2.1.0  took 3.21s : 200
 info ✅ panw-0.5.0  took 3.063s : 200
 info ✅ panw_metrics-0.2.0  took 3.077s : 200
 info ✅ prisma_access-1.5.0  took 3.053s : 200
 info ✅ prisma_cloud-2.1.0  took 3.163s : 200
 info ✅ ping_federate-0.2.0  took 3.069s : 200
 info ✅ ping_one-1.19.0  took 7.553s : 200
 info ✅ pps-0.5.0  took 3.074s : 200
 info ✅ postgresql-1.27.0  took 3.083s : 200
 info ✅ prometheus-1.23.0  took 3.033s : 200
 info ✅ prometheus_input-0.7.0  took 2.022s : 200
 info ✅ proofpoint_on_demand-1.5.0  took 3.138s : 200
 info ✅ proofpoint_tap-1.27.0  took 3.18s : 200
 info ✅ pulse_connect_secure-2.5.0  took 2.027s : 200
 info ✅ qualys_vmdr-6.2.0  took 3.078s : 200
 info ✅ rabbitmq-1.18.0  took 3.068s : 200
 info ❌ radware-0.1.2  took 0.121s : {"body":{"statusCode":400,"error":"Bad Request","message":"Invalid top-level package manifest at top-level directory radware-0.1.2 (package name: radware): one or more fields missing of name, version, title, owner."},"status":400,"took":0.121,"didError":true}
 info ✅ rapid7_insightvm-1.16.0  took 3.075s : 200
 info ✅ ti_rapid7_threat_command-2.4.0  took 7.23s : 200
 info ✅ ti_recordedfuture-1.29.0  took 5.25s : 200
 info ✅ redis-1.19.0  took 7.203s : 200
 info ✅ redisenterprise-0.12.0  took 3.107s : 200
 info ✅ rubrik-0.8.0  took 3.162s : 200
 info ✅ sql-0.6.0  took 2.02s : 200
 info ✅ stan-1.9.0  took 3.121s : 200
 info ✅ sailpoint_identity_sc-0.2.0  took 2.055s : 200
 info ✅ salesforce-1.1.0  took 3.118s : 200
 info ✅ cloud_security_posture-1.13.0-preview04  took 3.197s : 200
 info ✅ sentinel_one-1.29.1  took 2.98s : 200
 info ✅ sentinel_one_cloud_funnel-1.9.0  took 3.127s : 200
 info ✅ servicenow-0.10.0  took 3.158s : 200
 info ✅ slack-1.25.0  took 3.124s : 200
 info ❌ sonicwall-0.1.2  took 0.191s : {"body":{"statusCode":400,"error":"Bad Request","message":"Invalid top-level package manifest at top-level directory sonicwall-0.1.2 (package name: sonicwall): one or more fields missing of name, version, title, owner."},"status":400,"took":0.191,"didError":true}
 info ✅ sophos_central-1.19.0  took 2.959s : 200
 info ✅ spring_boot-1.8.0  took 3.121s : 200
 info ✅ spycloud-1.3.0  took 3.094s : 200
 info ❌ squid-0.1.2  took 0.116s : {"body":{"statusCode":400,"error":"Bad Request","message":"Invalid top-level package manifest at top-level directory squid-0.1.2 (package name: squid): one or more fields missing of name, version, title, owner."},"status":400,"took":0.116,"didError":true}
 info ✅ statsd_input-0.5.0  took 1.946s : 200
 info ✅ sublime_security-1.6.0  took 3.165s : 200
 info ✅ suricata-0.5.0  took 3.122s : 200
 info ✅ symantec_endpoint-2.19.0  took 3.135s : 200
 info ✅ symantec_endpoint_security-1.8.0  took 3.11s : 200
 info ✅ sysdig-0.3.0  took 3.093s : 200
 info ✅ system-1.67.0  took 4.225s : 200
 info ✅ tychon-0.3.0  took 28.783s : 200
 info ✅ tanium-1.15.0  took 2.822s : 200
 info ✅ teleport-1.3.0  took 3.057s : 200
 info ✅ tenable_sc-1.29.0  took 3.244s : 200
 info ✅ tenable_io-3.8.0  took 3.152s : 200
 info ✅ ti_util-1.7.0  took 3.074s : 200
 info ✅ threat_map-0.2.0  took 2.081s : 200
 info ✅ ti_threatconnect-1.8.0  took 5.205s : 200
 info ✅ ti_threatq-1.31.0  took 4.25s : 200
 info ✅ thycotic_ss-1.12.0  took 3.112s : 200
 info ✅ tines-1.15.0  took 3.13s : 200
 info ✅ tomcat-1.12.0  took 2.057s : 200
 info ✅ traefik-2.5.0  took 3.131s : 200
 info ✅ trellix_edr_cloud-1.7.0  took 3.092s : 200
 info ✅ trellix_epo_cloud-1.15.0  took 3.073s : 200
 info ✅ trendmicro-2.7.0  took 3.187s : 200
 info ✅ trend_micro_vision_one-1.26.0  took 3.167s : 200
 info ✅ carbonblack_edr-1.21.0  took 3.11s : 200
 info ✅ vsphere-1.18.1  took 3.13s : 200
 info ✅ vectra_detect-1.13.0  took 4.203s : 200
 info ✅ websphere_application_server-1.6.0  took 3.112s : 200
 info ✅ windows-2.5.0  took 3.151s : 200
 info ✅ wiz-2.8.1  took 6.11s : 200
 info ✅ zeronetworks-1.18.0  took 3.069s : 200
 info ✅ zerofox-1.28.0  took 3.09s : 200
 info ✅ zookeeper-1.14.0  took 3.213s : 200
 info ✅ zoom-1.22.0  took 3.099s : 200
 info ✅ zscaler_zia-3.7.1  took 3.148s : 200
 info ❌ zscaler-0.1.2  took 0.127s : {"body":{"statusCode":400,"error":"Bad Request","message":"Invalid top-level package manifest at top-level directory zscaler-0.1.2 (package name: zscaler): one or more fields missing of name, version, title, owner."},"status":400,"took":0.127,"didError":true}
 info ✅ zscaler_zpa-1.21.0  took 3.98s : 200
 info ✅ authentik-1.3.0  took 3.123s : 200
 info ✅ blacklens-0.3.0  took 3.089s : 200
 info ✅ etcd-1.4.0  took 3.1s : 200
ERROR There were 13 errors, exiting with code

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: e8cd6f4

Metrics [docs]

✅ unchanged

History

• 💚 Build #280476 succeeded 227a756
• 💔 Build #280325 failed f94a365
• 💔 Build #280120 failed e7d06f4
• 💔 Build #280115 failed 05ae827
• 💔 Build #279533 failed 8817801
• 💔 Build #279476 failed 0c1e4b6

cc @nchaulet

[jen-huang]

@jen-huang here the results of the benchmark

I compared them and on average, there doesn't seem to be much difference between the two results (overall average increase of 0.129s). The only one that shows a big difference is aws-2.41.1 which you already called out in the memory metrics. I think it's fine as the tradeoff is saving memory. Thanks for running the benchmarks!