From 67cdb93f5b800caac80672c942d04afe4d7aa4d8 Mon Sep 17 00:00:00 2001 From: Dmitrii Shevchenko Date: Tue, 5 Nov 2024 13:11:47 +0100 Subject: [PATCH] [Fleet] [Security Solution] Install prebuilt rules package using stream-based approach (#195888) **Resolves: https://github.com/elastic/kibana/issues/192350** ## Summary Implemented stream-based installation of the detection rules package. **Background**: The installation of the detection rules package was causing OOM (Out of Memory) errors in Serverless environments where the available memory is limited to 1GB. The root cause of the errors was that during installation, the package was being read and unzipped entirely into memory. Given the large package size, this led to OOMs. To address these memory issues, the following changes were made: 1. Added a branching logic to the `installPackageFromRegistry` and `installPackageByUpload` methods, where based on the package name is decided to use streaming or not. Only one `security_detection_engine` package is currently hardcoded to use streaming. 2. In the state machine then defined a separate set of steps for the stream-based package installation. It is reduced to cover only Kibana assets installation at this stage. 3. A new `stepInstallKibanaAssetsWithStreaming` step is added to handle assets installation. While this method still reads the package archive into memory (since unzipping from a readable stream is [not possible due to the design of the .zip format](https://github.com/thejoshwolfe/yauzl?tab=readme-ov-file#no-streaming-unzip-api)), the package is unzipped using streams after being read into a buffer. This allows only a small portion of the archive (100 saved objects at a time) to be unpacked into memory, reducing memory usage. 4. The new method also includes several optimizations, such as only removing previously installed assets if they are missing in the new package and using `savedObjectClient.bulkCreate` instead of the less efficient `savedObjectClient.import`. ### Test environment 1. Prebuilt detection rules package with ~20k saved objects; 118MB zipped. 5. Local package registry. 6. Production build of Kibana running locally with a 700MB max old space limit, pointed to that registry. Setting up a test environment is not completely straightforward. Here's a rough outline of the steps:
How to test this PR 1. Create a package containing a large number of prebuilt rules. 1. I used the `package-storage` repository to find one of the previously released prebuilt rules packages. 2. Multiplied the number of assets in the package to 20k historical versions. 4. Built the package using `elastic-package build`. 2. Start a local package registry serving the built package using `elastic-package stack up --services package-registry`. 4. Create a production build of Kibana. To speed up the process, unnecessary artifacts can be skipped: ``` node scripts/build --skip-cdn-assets --skip-docker-ubi --skip-docker-ubuntu --skip-docker-wolfi --skip-docker-fips ``` 7. Provide the built Kibana with a config pointing to the local registry. The config is located in `build/default/kibana-9.0.0-SNAPSHOT-darwin-aarch64/config/kibana.yml`. You can use the following config: ``` csp.strict: false xpack.security.encryptionKey: 've4Vohnu oa0Fu9ae Eethee8c oDieg4do Nohrah1u ao9Hu2oh Aeb4Ieyi Aew1aegi' xpack.encryptedSavedObjects.encryptionKey: 'Shah7nai Eew6izai Eir7OoW0 Gewi2ief eiSh8woo shoogh7E Quae6hal ce6Oumah' xpack.fleet.internal.registry.kibanaVersionCheckEnabled: false xpack.fleet.registryUrl: https://localhost:8080 elasticsearch: username: 'kibana_system' password: 'changeme' hosts: 'http://localhost:9200' ``` 8. Override the Node options Kibana starts with to allow it to connect to the local registry and set the memory limit. For this, you need to edit the `build/default/kibana-9.0.0-SNAPSHOT-darwin-aarch64/bin/kibana` file: ``` NODE_OPTIONS="--no-warnings --max-http-header-size=65536 --unhandled-rejections=warn --dns-result-order=ipv4first --openssl-legacy-provider --max_old_space_size=700 --inspect" NODE_ENV=production NODE_EXTRA_CA_CERTS=~/.elastic-package/profiles/default/certs/ca-cert.pem exec "${NODE}" "${DIR}/src/cli/dist" "${@}" ``` 9. Navigate to the build folder: `build/default/kibana-9.0.0-SNAPSHOT-darwin-aarch64`. 10. Start Kibana using `./bin/kibana`. 11. Kibana is now running in debug mode, with the debugger started on port 9229. You can connect to it using VS Code's debug config or Chrome's DevTools. 12. Now you can install prebuilt detection rules by calling the `POST /internal/detection_engine/prebuilt_rules/_bootstrap` endpoint, which uses the new streaming installation under the hood.
### Test results locally **Without the streaming approach** Guaranteed OOM. Even smaller packages, up to 10k rules, caused sporadic OOM errors. So for comparison, tested the package installation without memory limits. ![Screenshot 2024-10-14 at 14 15 26](https://github.com/user-attachments/assets/131cb877-2404-4638-b619-b1370a53659f) 1. Heap memory usage spikes up to 2.5GB 5. External memory consumes up to 450 Mb, which is four times the archive size 13. RSS (Resident Set Size) exceeds 4.5GB **With the streaming approach** No OOM errors observed. The memory consumption chart looks like the following: ![Screenshot 2024-10-14 at 11 15 21](https://github.com/user-attachments/assets/b47ba8c9-2ba7-42de-b921-c33104d4481e) 1. Heap memory remains stable, around 450MB, without any spikes. 2. External memory jumps to around 250MB at the beginning of the installation, then drops to around 120MB, which is roughly equal to the package archive size. I couldn't determine why the external memory consumption exceeds the package size by 2x when the installation starts. I checked the code for places where the package might be loaded into memory twice but found nothing suspicious. This might be worth investigating further. 3. RSS remains stable, peaking slightly above 1GB. I believe this is the upper limit for a package that can be handled without errors in a Serverless environment, where the memory limit is dictated by pod-level settings rather than Node settings and is set to 1GB. I'll verify this on a real Serverless instance to confirm. ### Test results on Serverless ![Screenshot 2024-10-31 at 12 31 34](https://github.com/user-attachments/assets/d20d2860-fa96-4e56-be2b-7b3c0b5c7b77) --- .../plugins/fleet/common/types/models/epm.ts | 15 ++ .../server/routes/epm/file_handler.test.ts | 4 +- .../fleet/server/routes/epm/file_handler.ts | 4 +- .../routes/epm/kibana_assets_handler.ts | 2 + .../services/epm/archive/archive_iterator.ts | 83 +++++++++ .../server/services/epm/archive/extract.ts | 16 +- .../server/services/epm/archive/index.ts | 100 +++++++---- .../server/services/epm/archive/parse.ts | 5 +- .../server/services/epm/archive/storage.ts | 2 +- .../elasticsearch/ingest_pipeline/install.ts | 3 +- .../elasticsearch/transform/mappings.test.ts | 4 + .../services/epm/kibana/assets/install.ts | 14 +- .../kibana/assets/install_with_streaming.ts | 115 +++++++++++++ .../server/services/epm/package_service.ts | 8 +- .../server/services/epm/packages/assets.ts | 2 +- .../server/services/epm/packages/get.test.ts | 3 + .../services/epm/packages/install.test.ts | 18 ++ .../server/services/epm/packages/install.ts | 32 +++- .../_state_machine_package_install.test.ts | 7 + .../_state_machine_package_install.ts | 34 +++- .../step_create_restart_installation.test.ts | 6 + .../step_delete_previous_pipelines.test.ts | 4 + .../steps/step_install_ilm_policies.test.ts | 4 + ...p_install_index_template_pipelines.test.ts | 8 + .../steps/step_install_kibana_assets.test.ts | 159 +++++++++++++++++- .../steps/step_install_kibana_assets.ts | 63 +++++++ .../steps/step_install_mlmodel.test.ts | 3 + .../steps/step_install_transforms.test.ts | 3 + .../step_remove_legacy_templates.test.ts | 3 + .../steps/step_save_archive_entries.test.ts | 3 + .../steps/step_save_archive_entries.ts | 23 ++- .../steps/step_save_system_object.test.ts | 4 + .../step_update_current_write_indices.test.ts | 3 + .../update_latest_executed_state.test.ts | 6 + .../server/services/epm/packages/remove.ts | 1 + .../server/services/epm/registry/index.ts | 17 +- .../experimental_datastream_features.ts | 2 + .../fleet_api_integration/apis/epm/index.js | 1 + .../apis/epm/install_by_upload.ts | 2 +- .../apis/epm/install_with_streaming.ts | 65 +++++++ .../security_detection_engine-8.16.0.zip | Bin 0 -> 63220 bytes 41 files changed, 768 insertions(+), 83 deletions(-) create mode 100644 x-pack/plugins/fleet/server/services/epm/archive/archive_iterator.ts create mode 100644 x-pack/plugins/fleet/server/services/epm/kibana/assets/install_with_streaming.ts create mode 100644 x-pack/test/fleet_api_integration/apis/epm/install_with_streaming.ts create mode 100644 x-pack/test/fleet_api_integration/apis/fixtures/test_packages/security_detection_engine/security_detection_engine-8.16.0.zip diff --git a/x-pack/plugins/fleet/common/types/models/epm.ts b/x-pack/plugins/fleet/common/types/models/epm.ts index 3aa65dc3adcd4..827130d802f22 100644 --- a/x-pack/plugins/fleet/common/types/models/epm.ts +++ b/x-pack/plugins/fleet/common/types/models/epm.ts @@ -124,10 +124,25 @@ export type InstallablePackage = RegistryPackage | ArchivePackage; export type AssetsMap = Map; +export interface ArchiveEntry { + path: string; + buffer?: Buffer; +} + +export interface ArchiveIterator { + traverseEntries: (onEntry: (entry: ArchiveEntry) => Promise) => Promise; + getPaths: () => Promise; +} + export interface PackageInstallContext { packageInfo: InstallablePackage; + /** + * @deprecated Use `archiveIterator` to access the package archive entries + * without loading them all into memory at once. + */ assetsMap: AssetsMap; paths: string[]; + archiveIterator: ArchiveIterator; } export type ArchivePackage = PackageSpecManifest & diff --git a/x-pack/plugins/fleet/server/routes/epm/file_handler.test.ts b/x-pack/plugins/fleet/server/routes/epm/file_handler.test.ts index 1eb8387f69751..5690c32c2d7fd 100644 --- a/x-pack/plugins/fleet/server/routes/epm/file_handler.test.ts +++ b/x-pack/plugins/fleet/server/routes/epm/file_handler.test.ts @@ -15,7 +15,7 @@ import { getBundledPackageByPkgKey } from '../../services/epm/packages/bundled_p import { getFile, getInstallation } from '../../services/epm/packages/get'; import type { FleetRequestHandlerContext } from '../..'; import { appContextService } from '../../services'; -import { unpackBufferEntries } from '../../services/epm/archive'; +import { unpackArchiveEntriesIntoMemory } from '../../services/epm/archive'; import { getAsset } from '../../services/epm/archive/storage'; import { getFileHandler } from './file_handler'; @@ -29,7 +29,7 @@ jest.mock('../../services/epm/packages/get'); const mockedGetBundledPackageByPkgKey = jest.mocked(getBundledPackageByPkgKey); const mockedGetInstallation = jest.mocked(getInstallation); const mockedGetFile = jest.mocked(getFile); -const mockedUnpackBufferEntries = jest.mocked(unpackBufferEntries); +const mockedUnpackBufferEntries = jest.mocked(unpackArchiveEntriesIntoMemory); const mockedGetAsset = jest.mocked(getAsset); function mockContext() { diff --git a/x-pack/plugins/fleet/server/routes/epm/file_handler.ts b/x-pack/plugins/fleet/server/routes/epm/file_handler.ts index 0f22a31c1aa72..994f52a71c224 100644 --- a/x-pack/plugins/fleet/server/routes/epm/file_handler.ts +++ b/x-pack/plugins/fleet/server/routes/epm/file_handler.ts @@ -17,7 +17,7 @@ import { defaultFleetErrorHandler } from '../../errors'; import { getAsset } from '../../services/epm/archive/storage'; import { getBundledPackageByPkgKey } from '../../services/epm/packages/bundled_packages'; import { pkgToPkgKey } from '../../services/epm/registry'; -import { unpackBufferEntries } from '../../services/epm/archive'; +import { unpackArchiveEntriesIntoMemory } from '../../services/epm/archive'; const CACHE_CONTROL_10_MINUTES_HEADER: HttpResponseOptions['headers'] = { 'cache-control': 'max-age=600', @@ -69,7 +69,7 @@ export const getFileHandler: FleetRequestHandler< pkgToPkgKey({ name: pkgName, version: pkgVersion }) ); if (bundledPackage) { - const bufferEntries = await unpackBufferEntries( + const bufferEntries = await unpackArchiveEntriesIntoMemory( await bundledPackage.getBuffer(), 'application/zip' ); diff --git a/x-pack/plugins/fleet/server/routes/epm/kibana_assets_handler.ts b/x-pack/plugins/fleet/server/routes/epm/kibana_assets_handler.ts index 8fe83f98669d1..ad0bec6397ee8 100644 --- a/x-pack/plugins/fleet/server/routes/epm/kibana_assets_handler.ts +++ b/x-pack/plugins/fleet/server/routes/epm/kibana_assets_handler.ts @@ -22,6 +22,7 @@ import type { FleetRequestHandler, InstallKibanaAssetsRequestSchema, } from '../../types'; +import { createArchiveIteratorFromMap } from '../../services/epm/archive/archive_iterator'; export const installPackageKibanaAssetsHandler: FleetRequestHandler< TypeOf, @@ -69,6 +70,7 @@ export const installPackageKibanaAssetsHandler: FleetRequestHandler< packageInfo, paths: installedPkgWithAssets.paths, assetsMap: installedPkgWithAssets.assetsMap, + archiveIterator: createArchiveIteratorFromMap(installedPkgWithAssets.assetsMap), }, }); diff --git a/x-pack/plugins/fleet/server/services/epm/archive/archive_iterator.ts b/x-pack/plugins/fleet/server/services/epm/archive/archive_iterator.ts new file mode 100644 index 0000000000000..369b32412bd82 --- /dev/null +++ b/x-pack/plugins/fleet/server/services/epm/archive/archive_iterator.ts @@ -0,0 +1,83 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import type { AssetsMap, ArchiveIterator, ArchiveEntry } from '../../../../common/types'; + +import { traverseArchiveEntries } from '.'; + +/** + * Creates an iterator for traversing and extracting paths from an archive + * buffer. This iterator is intended to be used for memory efficient traversal + * of archive contents without extracting the entire archive into memory. + * + * @param archiveBuffer - The buffer containing the archive data. + * @param contentType - The content type of the archive (e.g., + * 'application/zip'). + * @returns ArchiveIterator instance. + * + */ +export const createArchiveIterator = ( + archiveBuffer: Buffer, + contentType: string +): ArchiveIterator => { + const paths: string[] = []; + + const traverseEntries = async ( + onEntry: (entry: ArchiveEntry) => Promise + ): Promise => { + await traverseArchiveEntries(archiveBuffer, contentType, async (entry) => { + await onEntry(entry); + }); + }; + + const getPaths = async (): Promise => { + if (paths.length) { + return paths; + } + + await traverseEntries(async (entry) => { + paths.push(entry.path); + }); + + return paths; + }; + + return { + traverseEntries, + getPaths, + }; +}; + +/** + * Creates an archive iterator from the assetsMap. This is a stop-gap solution + * to provide a uniform interface for traversing assets while assetsMap is still + * in use. It works with a map of assets loaded into memory and is not intended + * for use with large archives. + * + * @param assetsMap - A map where the keys are asset paths and the values are + * asset buffers. + * @returns ArchiveIterator instance. + * + */ +export const createArchiveIteratorFromMap = (assetsMap: AssetsMap): ArchiveIterator => { + const traverseEntries = async ( + onEntry: (entry: ArchiveEntry) => Promise + ): Promise => { + for (const [path, buffer] of assetsMap) { + await onEntry({ path, buffer }); + } + }; + + const getPaths = async (): Promise => { + return [...assetsMap.keys()]; + }; + + return { + traverseEntries, + getPaths, + }; +}; diff --git a/x-pack/plugins/fleet/server/services/epm/archive/extract.ts b/x-pack/plugins/fleet/server/services/epm/archive/extract.ts index 84aa161385cb3..9f5f90959d144 100644 --- a/x-pack/plugins/fleet/server/services/epm/archive/extract.ts +++ b/x-pack/plugins/fleet/server/services/epm/archive/extract.ts @@ -11,13 +11,12 @@ import * as tar from 'tar'; import yauzl from 'yauzl'; import { bufferToStream, streamToBuffer } from '../streams'; - -import type { ArchiveEntry } from '.'; +import type { ArchiveEntry } from '../../../../common/types'; export async function untarBuffer( buffer: Buffer, filter = (entry: ArchiveEntry): boolean => true, - onEntry = (entry: ArchiveEntry): void => {} + onEntry = async (entry: ArchiveEntry): Promise => {} ) { const deflatedStream = bufferToStream(buffer); // use tar.list vs .extract to avoid writing to disk @@ -37,7 +36,7 @@ export async function untarBuffer( export async function unzipBuffer( buffer: Buffer, filter = (entry: ArchiveEntry): boolean => true, - onEntry = (entry: ArchiveEntry): void => {} + onEntry = async (entry: ArchiveEntry): Promise => {} ): Promise { const zipfile = await yauzlFromBuffer(buffer, { lazyEntries: true }); zipfile.readEntry(); @@ -45,9 +44,12 @@ export async function unzipBuffer( const path = entry.fileName; if (!filter({ path })) return zipfile.readEntry(); - const entryBuffer = await getZipReadStream(zipfile, entry).then(streamToBuffer); - onEntry({ buffer: entryBuffer, path }); - zipfile.readEntry(); + try { + const entryBuffer = await getZipReadStream(zipfile, entry).then(streamToBuffer); + await onEntry({ buffer: entryBuffer, path }); + } finally { + zipfile.readEntry(); + } }); return new Promise((resolve, reject) => zipfile.on('end', resolve).on('error', reject)); } diff --git a/x-pack/plugins/fleet/server/services/epm/archive/index.ts b/x-pack/plugins/fleet/server/services/epm/archive/index.ts index 5943f8f838fcb..ed9ff2a5e4b72 100644 --- a/x-pack/plugins/fleet/server/services/epm/archive/index.ts +++ b/x-pack/plugins/fleet/server/services/epm/archive/index.ts @@ -5,13 +5,20 @@ * 2.0. */ -import type { AssetParts, AssetsMap } from '../../../../common/types'; +import type { + ArchiveEntry, + ArchiveIterator, + AssetParts, + AssetsMap, +} from '../../../../common/types'; import { PackageInvalidArchiveError, PackageUnsupportedMediaTypeError, PackageNotFoundError, } from '../../../errors'; +import { createArchiveIterator } from './archive_iterator'; + import { deletePackageInfo } from './cache'; import type { SharedKey } from './cache'; import { getBufferExtractor } from './extract'; @@ -20,66 +27,85 @@ export * from './cache'; export { getBufferExtractor, untarBuffer, unzipBuffer } from './extract'; export { generatePackageInfoFromArchiveBuffer } from './parse'; -export interface ArchiveEntry { - path: string; - buffer?: Buffer; -} - export async function unpackBufferToAssetsMap({ - name, - version, contentType, archiveBuffer, + useStreaming, }: { - name: string; - version: string; contentType: string; archiveBuffer: Buffer; -}): Promise<{ paths: string[]; assetsMap: AssetsMap }> { - const assetsMap = new Map(); - const paths: string[] = []; - const entries = await unpackBufferEntries(archiveBuffer, contentType); - - entries.forEach((entry) => { - const { path, buffer } = entry; - if (buffer) { - assetsMap.set(path, buffer); - paths.push(path); - } - }); - - return { assetsMap, paths }; + useStreaming: boolean | undefined; +}): Promise<{ paths: string[]; assetsMap: AssetsMap; archiveIterator: ArchiveIterator }> { + const archiveIterator = createArchiveIterator(archiveBuffer, contentType); + let paths: string[] = []; + let assetsMap: AssetsMap = new Map(); + if (useStreaming) { + paths = await archiveIterator.getPaths(); + // We keep the assetsMap empty as we don't want to load all the assets in memory + assetsMap = new Map(); + } else { + const entries = await unpackArchiveEntriesIntoMemory(archiveBuffer, contentType); + + entries.forEach((entry) => { + const { path, buffer } = entry; + if (buffer) { + assetsMap.set(path, buffer); + paths.push(path); + } + }); + } + + return { paths, assetsMap, archiveIterator }; } -export async function unpackBufferEntries( +/** + * This function extracts all archive entries into memory. + * + * NOTE: This is potentially dangerous for large archives and can cause OOM + * errors. Use 'traverseArchiveEntries' instead to iterate over the entries + * without storing them all in memory at once. + * + * @param archiveBuffer + * @param contentType + * @returns All the entries in the archive buffer + */ +export async function unpackArchiveEntriesIntoMemory( archiveBuffer: Buffer, contentType: string ): Promise { + const entries: ArchiveEntry[] = []; + const addToEntries = async (entry: ArchiveEntry) => void entries.push(entry); + await traverseArchiveEntries(archiveBuffer, contentType, addToEntries); + + // While unpacking a tar.gz file with unzipBuffer() will result in a thrown + // error, unpacking a zip file with untarBuffer() just results in nothing. + if (entries.length === 0) { + throw new PackageInvalidArchiveError( + `Archive seems empty. Assumed content type was ${contentType}, check if this matches the archive type.` + ); + } + return entries; +} + +export async function traverseArchiveEntries( + archiveBuffer: Buffer, + contentType: string, + onEntry: (entry: ArchiveEntry) => Promise +) { const bufferExtractor = getBufferExtractor({ contentType }); if (!bufferExtractor) { throw new PackageUnsupportedMediaTypeError( `Unsupported media type ${contentType}. Please use 'application/gzip' or 'application/zip'` ); } - const entries: ArchiveEntry[] = []; try { const onlyFiles = ({ path }: ArchiveEntry): boolean => !path.endsWith('/'); - const addToEntries = (entry: ArchiveEntry) => entries.push(entry); - await bufferExtractor(archiveBuffer, onlyFiles, addToEntries); + await bufferExtractor(archiveBuffer, onlyFiles, onEntry); } catch (error) { throw new PackageInvalidArchiveError( `Error during extraction of package: ${error}. Assumed content type was ${contentType}, check if this matches the archive type.` ); } - - // While unpacking a tar.gz file with unzipBuffer() will result in a thrown error in the try-catch above, - // unpacking a zip file with untarBuffer() just results in nothing. - if (entries.length === 0) { - throw new PackageInvalidArchiveError( - `Archive seems empty. Assumed content type was ${contentType}, check if this matches the archive type.` - ); - } - return entries; } export const deletePackageCache = ({ name, version }: SharedKey) => { diff --git a/x-pack/plugins/fleet/server/services/epm/archive/parse.ts b/x-pack/plugins/fleet/server/services/epm/archive/parse.ts index 530ca804f24eb..8cccfe9982457 100644 --- a/x-pack/plugins/fleet/server/services/epm/archive/parse.ts +++ b/x-pack/plugins/fleet/server/services/epm/archive/parse.ts @@ -40,7 +40,7 @@ import { import { PackageInvalidArchiveError } from '../../../errors'; import { pkgToPkgKey } from '../registry'; -import { unpackBufferEntries } from '.'; +import { traverseArchiveEntries } from '.'; const readFileAsync = promisify(readFile); export const MANIFEST_NAME = 'manifest.yml'; @@ -160,9 +160,8 @@ export async function generatePackageInfoFromArchiveBuffer( contentType: string ): Promise<{ paths: string[]; packageInfo: ArchivePackage }> { const assetsMap: AssetsBufferMap = {}; - const entries = await unpackBufferEntries(archiveBuffer, contentType); const paths: string[] = []; - entries.forEach(({ path: bufferPath, buffer }) => { + await traverseArchiveEntries(archiveBuffer, contentType, async ({ path: bufferPath, buffer }) => { paths.push(bufferPath); if (buffer && filterAssetPathForParseAndVerifyArchive(bufferPath)) { assetsMap[bufferPath] = buffer; diff --git a/x-pack/plugins/fleet/server/services/epm/archive/storage.ts b/x-pack/plugins/fleet/server/services/epm/archive/storage.ts index dd6321445df75..8f6f151383d5a 100644 --- a/x-pack/plugins/fleet/server/services/epm/archive/storage.ts +++ b/x-pack/plugins/fleet/server/services/epm/archive/storage.ts @@ -15,6 +15,7 @@ import { SavedObjectsErrorHelpers } from '@kbn/core/server'; import { ASSETS_SAVED_OBJECT_TYPE } from '../../../../common'; import type { + ArchiveEntry, InstallablePackage, InstallSource, PackageAssetReference, @@ -24,7 +25,6 @@ import { PackageInvalidArchiveError, PackageNotFoundError } from '../../../error import { appContextService } from '../../app_context'; import { setPackageInfo } from '.'; -import type { ArchiveEntry } from '.'; import { filterAssetPathForParseAndVerifyArchive, parseAndVerifyArchive } from './parse'; const ONE_BYTE = 1024 * 1024; diff --git a/x-pack/plugins/fleet/server/services/epm/elasticsearch/ingest_pipeline/install.ts b/x-pack/plugins/fleet/server/services/epm/elasticsearch/ingest_pipeline/install.ts index a456734747324..5a4672f67fe53 100644 --- a/x-pack/plugins/fleet/server/services/epm/elasticsearch/ingest_pipeline/install.ts +++ b/x-pack/plugins/fleet/server/services/epm/elasticsearch/ingest_pipeline/install.ts @@ -16,14 +16,13 @@ import type { PackageInfo, } from '../../../../types'; import { getAssetFromAssetsMap, getPathParts } from '../../archive'; -import type { ArchiveEntry } from '../../archive'; import { FLEET_FINAL_PIPELINE_CONTENT, FLEET_FINAL_PIPELINE_ID, FLEET_FINAL_PIPELINE_VERSION, } from '../../../../constants'; import { getPipelineNameForDatastream } from '../../../../../common/services'; -import type { PackageInstallContext } from '../../../../../common/types'; +import type { ArchiveEntry, PackageInstallContext } from '../../../../../common/types'; import { appendMetadataToIngestPipeline } from '../meta'; import { retryTransientEsErrors } from '../retry'; diff --git a/x-pack/plugins/fleet/server/services/epm/elasticsearch/transform/mappings.test.ts b/x-pack/plugins/fleet/server/services/epm/elasticsearch/transform/mappings.test.ts index f34015bf77697..de962850fba8c 100644 --- a/x-pack/plugins/fleet/server/services/epm/elasticsearch/transform/mappings.test.ts +++ b/x-pack/plugins/fleet/server/services/epm/elasticsearch/transform/mappings.test.ts @@ -5,6 +5,8 @@ * 2.0. */ +import { createArchiveIteratorFromMap } from '../../archive/archive_iterator'; + import { loadMappingForTransform } from './mappings'; describe('loadMappingForTransform', () => { @@ -13,6 +15,7 @@ describe('loadMappingForTransform', () => { { packageInfo: {} as any, assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), paths: [], }, 'test' @@ -49,6 +52,7 @@ describe('loadMappingForTransform', () => { ), ], ]), + archiveIterator: createArchiveIteratorFromMap(new Map()), paths: [ '/package/ti_opencti/2.1.0/elasticsearch/transform/latest_ioc/fields/ecs.yml', '/package/ti_opencti/2.1.0/elasticsearch/transform/latest_ioc/fields/ecs-extra.yml', diff --git a/x-pack/plugins/fleet/server/services/epm/kibana/assets/install.ts b/x-pack/plugins/fleet/server/services/epm/kibana/assets/install.ts index 276478099daf8..bf5684f29c205 100644 --- a/x-pack/plugins/fleet/server/services/epm/kibana/assets/install.ts +++ b/x-pack/plugins/fleet/server/services/epm/kibana/assets/install.ts @@ -325,16 +325,16 @@ export async function deleteKibanaAssetsAndReferencesForSpace({ await saveKibanaAssetsRefs(savedObjectsClient, pkgName, [], true); } +const kibanaAssetTypes = Object.values(KibanaAssetType); +export const isKibanaAssetType = (path: string) => { + const parts = getPathParts(path); + + return parts.service === 'kibana' && (kibanaAssetTypes as string[]).includes(parts.type); +}; + export function getKibanaAssets( packageInstallContext: PackageInstallContext ): Record { - const kibanaAssetTypes = Object.values(KibanaAssetType); - const isKibanaAssetType = (path: string) => { - const parts = getPathParts(path); - - return parts.service === 'kibana' && (kibanaAssetTypes as string[]).includes(parts.type); - }; - const result = Object.fromEntries( kibanaAssetTypes.map((type) => [type, []]) ) as Record; diff --git a/x-pack/plugins/fleet/server/services/epm/kibana/assets/install_with_streaming.ts b/x-pack/plugins/fleet/server/services/epm/kibana/assets/install_with_streaming.ts new file mode 100644 index 0000000000000..fca6cf27a0cd7 --- /dev/null +++ b/x-pack/plugins/fleet/server/services/epm/kibana/assets/install_with_streaming.ts @@ -0,0 +1,115 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +import type { SavedObject, SavedObjectsClientContract } from '@kbn/core/server'; + +import type { Installation, PackageInstallContext } from '../../../../../common/types'; +import type { KibanaAssetReference, KibanaAssetType } from '../../../../types'; +import { getPathParts } from '../../archive'; + +import { saveKibanaAssetsRefs } from '../../packages/install'; + +import type { ArchiveAsset } from './install'; +import { + KibanaSavedObjectTypeMapping, + createSavedObjectKibanaAsset, + isKibanaAssetType, + toAssetReference, +} from './install'; +import { getSpaceAwareSaveobjectsClients } from './saved_objects'; + +interface InstallKibanaAssetsWithStreamingArgs { + pkgName: string; + packageInstallContext: PackageInstallContext; + spaceId: string; + savedObjectsClient: SavedObjectsClientContract; + installedPkg?: SavedObject | undefined; +} + +const MAX_ASSETS_TO_INSTALL_IN_PARALLEL = 100; + +export async function installKibanaAssetsWithStreaming({ + spaceId, + packageInstallContext, + savedObjectsClient, + pkgName, + installedPkg, +}: InstallKibanaAssetsWithStreamingArgs): Promise { + const { archiveIterator } = packageInstallContext; + + const { savedObjectClientWithSpace } = getSpaceAwareSaveobjectsClients(spaceId); + + const assetRefs: KibanaAssetReference[] = []; + let batch: ArchiveAsset[] = []; + + await archiveIterator.traverseEntries(async ({ path, buffer }) => { + if (!buffer || !isKibanaAssetType(path)) { + return; + } + const savedObject = JSON.parse(buffer.toString('utf8')) as ArchiveAsset; + const assetType = getPathParts(path).type as KibanaAssetType; + const soType = KibanaSavedObjectTypeMapping[assetType]; + if (savedObject.type !== soType) { + return; + } + + batch.push(savedObject); + assetRefs.push(toAssetReference(savedObject)); + + if (batch.length >= MAX_ASSETS_TO_INSTALL_IN_PARALLEL) { + await bulkCreateSavedObjects({ + savedObjectsClient: savedObjectClientWithSpace, + kibanaAssets: batch, + refresh: false, + }); + batch = []; + } + }); + + // install any remaining assets + if (batch.length) { + await bulkCreateSavedObjects({ + savedObjectsClient: savedObjectClientWithSpace, + kibanaAssets: batch, + // Use wait_for with the last batch to ensure all assets are readable once the install is complete + refresh: 'wait_for', + }); + } + + // Update the installation saved object with installed kibana assets + await saveKibanaAssetsRefs(savedObjectsClient, pkgName, assetRefs); + + return assetRefs; +} + +async function bulkCreateSavedObjects({ + savedObjectsClient, + kibanaAssets, + refresh, +}: { + kibanaAssets: ArchiveAsset[]; + savedObjectsClient: SavedObjectsClientContract; + refresh?: boolean | 'wait_for'; +}) { + if (!kibanaAssets.length) { + return []; + } + + const toBeSavedObjects = kibanaAssets.map((asset) => createSavedObjectKibanaAsset(asset)); + + const { saved_objects: createdSavedObjects } = await savedObjectsClient.bulkCreate( + toBeSavedObjects, + { + // We only want to install new saved objects without overwriting existing ones + overwrite: false, + managed: true, + refresh, + } + ); + + return createdSavedObjects; +} diff --git a/x-pack/plugins/fleet/server/services/epm/package_service.ts b/x-pack/plugins/fleet/server/services/epm/package_service.ts index 661475dfadc09..a097db584b460 100644 --- a/x-pack/plugins/fleet/server/services/epm/package_service.ts +++ b/x-pack/plugins/fleet/server/services/epm/package_service.ts @@ -39,7 +39,10 @@ import type { InstallResult } from '../../../common'; import { appContextService } from '..'; -import type { CustomPackageDatasetConfiguration, EnsurePackageResult } from './packages/install'; +import { + type CustomPackageDatasetConfiguration, + type EnsurePackageResult, +} from './packages/install'; import type { FetchFindLatestPackageOptions } from './registry'; import { getPackageFieldsMetadata } from './registry'; @@ -56,6 +59,7 @@ import { } from './packages'; import { generatePackageInfoFromArchiveBuffer } from './archive'; import { getEsPackage } from './archive/storage'; +import { createArchiveIteratorFromMap } from './archive/archive_iterator'; export type InstalledAssetType = EsAssetReference; @@ -381,12 +385,14 @@ class PackageClientImpl implements PackageClient { } const { assetsMap } = esPackage; + const archiveIterator = createArchiveIteratorFromMap(assetsMap); const { installedTransforms } = await installTransforms({ packageInstallContext: { assetsMap, packageInfo, paths, + archiveIterator, }, esClient: this.internalEsClient, savedObjectsClient: this.internalSoClient, diff --git a/x-pack/plugins/fleet/server/services/epm/packages/assets.ts b/x-pack/plugins/fleet/server/services/epm/packages/assets.ts index a82b5c0d103b2..3bb84c0d23163 100644 --- a/x-pack/plugins/fleet/server/services/epm/packages/assets.ts +++ b/x-pack/plugins/fleet/server/services/epm/packages/assets.ts @@ -5,9 +5,9 @@ * 2.0. */ +import type { ArchiveEntry } from '../../../../common/types'; import type { AssetsMap, PackageInfo } from '../../../types'; import { getAssetFromAssetsMap } from '../archive'; -import type { ArchiveEntry } from '../archive'; const maybeFilterByDataset = (packageInfo: Pick, datasetName: string) => diff --git a/x-pack/plugins/fleet/server/services/epm/packages/get.test.ts b/x-pack/plugins/fleet/server/services/epm/packages/get.test.ts index 2dc295762e33a..5711c8fcccaf4 100644 --- a/x-pack/plugins/fleet/server/services/epm/packages/get.test.ts +++ b/x-pack/plugins/fleet/server/services/epm/packages/get.test.ts @@ -27,6 +27,8 @@ import { auditLoggingService } from '../../audit_logging'; import * as Registry from '../registry'; +import { createArchiveIteratorFromMap } from '../archive/archive_iterator'; + import { getInstalledPackages, getPackageInfo, getPackages, getPackageUsageStats } from './get'; jest.mock('../registry'); @@ -915,6 +917,7 @@ owner: elastic`, MockRegistry.getPackage.mockResolvedValue({ paths: [], assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), packageInfo: { name: 'my-package', version: '1.0.0', diff --git a/x-pack/plugins/fleet/server/services/epm/packages/install.test.ts b/x-pack/plugins/fleet/server/services/epm/packages/install.test.ts index 709e0d84d70fc..6b3a31eda649e 100644 --- a/x-pack/plugins/fleet/server/services/epm/packages/install.test.ts +++ b/x-pack/plugins/fleet/server/services/epm/packages/install.test.ts @@ -442,6 +442,24 @@ describe('install', () => { expect(response.status).toEqual('installed'); }); + + it('should use streaming installation for the detection rules package', async () => { + jest.spyOn(licenseService, 'hasAtLeast').mockReturnValue(true); + + const response = await installPackage({ + spaceId: DEFAULT_SPACE_ID, + installSource: 'registry', + pkgkey: 'security_detection_engine', + savedObjectsClient: savedObjectsClientMock.create(), + esClient: {} as ElasticsearchClient, + }); + + expect(response.error).toBeUndefined(); + + expect(installStateMachine._stateMachineInstallPackage).toHaveBeenCalledWith( + expect.objectContaining({ useStreaming: true }) + ); + }); }); describe('upload', () => { diff --git a/x-pack/plugins/fleet/server/services/epm/packages/install.ts b/x-pack/plugins/fleet/server/services/epm/packages/install.ts index 1ea6f29cad839..ebe5acc35178d 100644 --- a/x-pack/plugins/fleet/server/services/epm/packages/install.ts +++ b/x-pack/plugins/fleet/server/services/epm/packages/install.ts @@ -76,6 +76,7 @@ import { deleteVerificationResult, unpackBufferToAssetsMap, } from '../archive'; +import { createArchiveIteratorFromMap } from '../archive/archive_iterator'; import { toAssetReference } from '../kibana/assets/install'; import type { ArchiveAsset } from '../kibana/assets/install'; import type { PackageUpdateEvent } from '../../upgrade_sender'; @@ -107,6 +108,12 @@ import { removeInstallation } from './remove'; export const UPLOAD_RETRY_AFTER_MS = 10000; // 10s const MAX_ENSURE_INSTALL_TIME = 60 * 1000; +const PACKAGES_TO_INSTALL_WITH_STREAMING = [ + // The security_detection_engine package contains a large number of assets and + // is not suitable for regular installation as it might cause OOM errors. + 'security_detection_engine', +]; + export async function isPackageInstalled(options: { savedObjectsClient: SavedObjectsClientContract; pkgName: string; @@ -449,6 +456,7 @@ async function installPackageFromRegistry({ // TODO: change epm API to /packageName/version so we don't need to do this const { pkgName, pkgVersion: version } = Registry.splitPkgKey(pkgkey); let pkgVersion = version ?? ''; + const useStreaming = PACKAGES_TO_INSTALL_WITH_STREAMING.includes(pkgName); // if an error happens during getInstallType, report that we don't know let installType: InstallType = 'unknown'; @@ -478,11 +486,12 @@ async function installPackageFromRegistry({ } // get latest package version and requested version in parallel for performance - const [latestPackage, { paths, packageInfo, assetsMap, verificationResult }] = + const [latestPackage, { paths, packageInfo, assetsMap, archiveIterator, verificationResult }] = await Promise.all([ latestPkg ? Promise.resolve(latestPkg) : queryLatest(), Registry.getPackage(pkgName, pkgVersion, { ignoreUnverified: force && !neverIgnoreVerificationError, + useStreaming, }), ]); @@ -490,6 +499,7 @@ async function installPackageFromRegistry({ packageInfo, assetsMap, paths, + archiveIterator, }; // let the user install if using the force flag or needing to reinstall or install a previous version due to failed update @@ -542,6 +552,7 @@ async function installPackageFromRegistry({ ignoreMappingUpdateErrors, skipDataStreamRollover, retryFromLastState, + useStreaming, }); } catch (e) { sendEvent({ @@ -580,6 +591,7 @@ async function installPackageWithStateMachine(options: { ignoreMappingUpdateErrors?: boolean; skipDataStreamRollover?: boolean; retryFromLastState?: boolean; + useStreaming?: boolean; }): Promise { const packageInfo = options.packageInstallContext.packageInfo; @@ -599,6 +611,7 @@ async function installPackageWithStateMachine(options: { skipDataStreamRollover, packageInstallContext, retryFromLastState, + useStreaming, } = options; let { telemetryEvent } = options; const logger = appContextService.getLogger(); @@ -696,6 +709,7 @@ async function installPackageWithStateMachine(options: { ignoreMappingUpdateErrors, skipDataStreamRollover, retryFromLastState, + useStreaming, }) .then(async (assets) => { logger.debug(`Removing old assets from previous versions of ${pkgName}`); @@ -785,6 +799,7 @@ async function installPackageByUpload({ } const { packageInfo } = await generatePackageInfoFromArchiveBuffer(archiveBuffer, contentType); const pkgName = packageInfo.name; + const useStreaming = PACKAGES_TO_INSTALL_WITH_STREAMING.includes(pkgName); // Allow for overriding the version in the manifest for cases where we install // stack-aligned bundled packages to support special cases around the @@ -807,17 +822,17 @@ async function installPackageByUpload({ packageInfo, }); - const { assetsMap, paths } = await unpackBufferToAssetsMap({ - name: packageInfo.name, - version: pkgVersion, + const { paths, assetsMap, archiveIterator } = await unpackBufferToAssetsMap({ archiveBuffer, contentType, + useStreaming, }); const packageInstallContext: PackageInstallContext = { packageInfo: { ...packageInfo, version: pkgVersion }, assetsMap, paths, + archiveIterator, }; // update the timestamp of latest installation setLastUploadInstallCache(); @@ -837,6 +852,7 @@ async function installPackageByUpload({ authorizationHeader, ignoreMappingUpdateErrors, skipDataStreamRollover, + useStreaming, }); } catch (e) { return { @@ -1004,12 +1020,14 @@ export async function installCustomPackage( acc.set(asset.path, asset.content); return acc; }, new Map()); - const paths = [...assetsMap.keys()]; + const paths = assets.map((asset) => asset.path); + const archiveIterator = createArchiveIteratorFromMap(assetsMap); const packageInstallContext: PackageInstallContext = { assetsMap, paths, packageInfo, + archiveIterator, }; return await installPackageWithStateMachine({ packageInstallContext, @@ -1341,16 +1359,20 @@ export async function installAssetsForInputPackagePolicy(opts: { ignoreUnverified: force, }); + const archiveIterator = createArchiveIteratorFromMap(pkg.assetsMap); packageInstallContext = { assetsMap: pkg.assetsMap, packageInfo: pkg.packageInfo, paths: pkg.paths, + archiveIterator, }; } else { + const archiveIterator = createArchiveIteratorFromMap(installedPkgWithAssets.assetsMap); packageInstallContext = { assetsMap: installedPkgWithAssets.assetsMap, packageInfo: installedPkgWithAssets.packageInfo, paths: installedPkgWithAssets.paths, + archiveIterator, }; } diff --git a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/_state_machine_package_install.test.ts b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/_state_machine_package_install.test.ts index 174076a9e9b1b..73b78a6cc4aa0 100644 --- a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/_state_machine_package_install.test.ts +++ b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/_state_machine_package_install.test.ts @@ -38,6 +38,8 @@ import { updateCurrentWriteIndices } from '../../elasticsearch/template/template import { installIndexTemplatesAndPipelines } from '../install_index_template_pipeline'; +import { createArchiveIteratorFromMap } from '../../archive/archive_iterator'; + import { handleState } from './state_machine'; import { _stateMachineInstallPackage } from './_state_machine_package_install'; import { cleanupLatestExecutedState } from './steps'; @@ -110,6 +112,7 @@ describe('_stateMachineInstallPackage', () => { logger: loggerMock.create(), packageInstallContext: { assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), paths: [], packageInfo: { title: 'title', @@ -172,6 +175,7 @@ describe('_stateMachineInstallPackage', () => { logger: loggerMock.create(), packageInstallContext: { assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), paths: [], packageInfo: { title: 'title', @@ -208,6 +212,7 @@ describe('_stateMachineInstallPackage', () => { logger: loggerMock.create(), packageInstallContext: { assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), paths: [], packageInfo: { title: 'title', @@ -257,6 +262,7 @@ describe('_stateMachineInstallPackage', () => { logger: loggerMock.create(), packageInstallContext: { assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), paths: [], packageInfo: { title: 'title', @@ -336,6 +342,7 @@ describe('_stateMachineInstallPackage', () => { owner: { github: 'elastic/fleet' }, } as any, assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), paths: [], }, installType: 'install', diff --git a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/_state_machine_package_install.ts b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/_state_machine_package_install.ts index 1f10d40feba38..c941b6d60d63b 100644 --- a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/_state_machine_package_install.ts +++ b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/_state_machine_package_install.ts @@ -48,11 +48,13 @@ import { updateLatestExecutedState, cleanupLatestExecutedState, cleanUpKibanaAssetsStep, + cleanUpUnusedKibanaAssetsStep, cleanupILMPoliciesStep, cleanUpMlModelStep, cleanupIndexTemplatePipelinesStep, cleanupTransformsStep, cleanupArchiveEntriesStep, + stepInstallKibanaAssetsWithStreaming, } from './steps'; import type { StateMachineDefinition, StateMachineStates } from './state_machine'; import { handleState } from './state_machine'; @@ -73,6 +75,7 @@ export interface InstallContext extends StateContext { skipDataStreamRollover?: boolean; retryFromLastState?: boolean; initialState?: INSTALL_STATES; + useStreaming?: boolean; indexTemplates?: IndexTemplateEntry[]; packageAssetRefs?: PackageAssetReference[]; @@ -83,7 +86,7 @@ export interface InstallContext extends StateContext { /** * This data structure defines the sequence of the states and the transitions */ -const statesDefinition: StateMachineStates = { +const regularStatesDefinition: StateMachineStates = { create_restart_installation: { nextState: INSTALL_STATES.INSTALL_KIBANA_ASSETS, onTransition: stepCreateRestartInstallation, @@ -152,6 +155,31 @@ const statesDefinition: StateMachineStates = { }, }; +const streamingStatesDefinition: StateMachineStates = { + create_restart_installation: { + nextState: INSTALL_STATES.INSTALL_KIBANA_ASSETS, + onTransition: stepCreateRestartInstallation, + onPostTransition: updateLatestExecutedState, + }, + install_kibana_assets: { + onTransition: stepInstallKibanaAssetsWithStreaming, + nextState: INSTALL_STATES.SAVE_ARCHIVE_ENTRIES, + onPostTransition: updateLatestExecutedState, + }, + save_archive_entries_from_assets_map: { + onPreTransition: cleanupArchiveEntriesStep, + onTransition: stepSaveArchiveEntries, + nextState: INSTALL_STATES.UPDATE_SO, + onPostTransition: updateLatestExecutedState, + }, + update_so: { + onPreTransition: cleanUpUnusedKibanaAssetsStep, + onTransition: stepSaveSystemObject, + nextState: 'end', + onPostTransition: updateLatestExecutedState, + }, +}; + /* * _stateMachineInstallPackage installs packages using the generic state machine in ./state_machine * installStates is the data structure providing the state machine definition @@ -166,6 +194,10 @@ export async function _stateMachineInstallPackage( const logger = appContextService.getLogger(); let initialState = INSTALL_STATES.CREATE_RESTART_INSTALLATION; + const statesDefinition = context.useStreaming + ? streamingStatesDefinition + : regularStatesDefinition; + // if retryFromLastState, restart install from last install state // if force is passed, the install should be executed from the beginning if (retryFromLastState && !force && installedPkg?.attributes?.latest_executed_state?.name) { diff --git a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_create_restart_installation.test.ts b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_create_restart_installation.test.ts index 2b653728d6574..e5a7fed55fe87 100644 --- a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_create_restart_installation.test.ts +++ b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_create_restart_installation.test.ts @@ -31,6 +31,8 @@ import { auditLoggingService } from '../../../../audit_logging'; import { restartInstallation, createInstallation } from '../../install'; import type { Installation } from '../../../../../../common'; +import { createArchiveIteratorFromMap } from '../../../archive/archive_iterator'; + import { stepCreateRestartInstallation } from './step_create_restart_installation'; jest.mock('../../../../audit_logging'); @@ -84,6 +86,7 @@ describe('stepCreateRestartInstallation', () => { logger, packageInstallContext: { assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), paths: [], packageInfo: { title: 'title', @@ -120,6 +123,7 @@ describe('stepCreateRestartInstallation', () => { logger, packageInstallContext: { assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), paths: [], packageInfo: { title: 'title', @@ -164,6 +168,7 @@ describe('stepCreateRestartInstallation', () => { logger, packageInstallContext: { assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), paths: [], packageInfo: { title: 'title', @@ -208,6 +213,7 @@ describe('stepCreateRestartInstallation', () => { logger, packageInstallContext: { assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), paths: [], packageInfo: { title: 'title', diff --git a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_delete_previous_pipelines.test.ts b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_delete_previous_pipelines.test.ts index 7d8a251433bb5..06201770ee2e2 100644 --- a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_delete_previous_pipelines.test.ts +++ b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_delete_previous_pipelines.test.ts @@ -24,6 +24,8 @@ import { deletePreviousPipelines, } from '../../../elasticsearch/ingest_pipeline'; +import { createArchiveIteratorFromMap } from '../../../archive/archive_iterator'; + import { stepDeletePreviousPipelines } from './step_delete_previous_pipelines'; jest.mock('../../../elasticsearch/ingest_pipeline'); @@ -84,6 +86,7 @@ describe('stepDeletePreviousPipelines', () => { owner: { github: 'elastic/fleet' }, } as any, assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), paths: [], }; appContextService.start( @@ -276,6 +279,7 @@ describe('stepDeletePreviousPipelines', () => { owner: { github: 'elastic/fleet' }, } as any, assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), paths: [], }; appContextService.start( diff --git a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_install_ilm_policies.test.ts b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_install_ilm_policies.test.ts index 2cf9b23bb9adb..4c106a0c68f15 100644 --- a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_install_ilm_policies.test.ts +++ b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_install_ilm_policies.test.ts @@ -24,6 +24,8 @@ import { installIlmForDataStream } from '../../../elasticsearch/datastream_ilm/i import { ElasticsearchAssetType } from '../../../../../types'; import { deleteILMPolicies, deletePrerequisiteAssets } from '../../remove'; +import { createArchiveIteratorFromMap } from '../../../archive/archive_iterator'; + import { stepInstallILMPolicies, cleanupILMPoliciesStep } from './step_install_ilm_policies'; jest.mock('../../../archive/storage'); @@ -56,6 +58,7 @@ const packageInstallContext = { owner: { github: 'elastic/fleet' }, } as any, assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), paths: [], }; let soClient: jest.Mocked; @@ -239,6 +242,7 @@ describe('stepInstallILMPolicies', () => { owner: { github: 'elastic/fleet' }, } as any, assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), paths: [], }, installType: 'install', diff --git a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_install_index_template_pipelines.test.ts b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_install_index_template_pipelines.test.ts index d258747edc6ef..1c368cfd998d3 100644 --- a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_install_index_template_pipelines.test.ts +++ b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_install_index_template_pipelines.test.ts @@ -37,6 +37,8 @@ const mockDeletePrerequisiteAssets = deletePrerequisiteAssets as jest.MockedFunc typeof deletePrerequisiteAssets >; +import { createArchiveIteratorFromMap } from '../../../archive/archive_iterator'; + import { stepInstallIndexTemplatePipelines, cleanupIndexTemplatePipelinesStep, @@ -122,6 +124,7 @@ describe('stepInstallIndexTemplatePipelines', () => { owner: { github: 'elastic/fleet' }, } as any, assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), paths: [], }; appContextService.start( @@ -281,6 +284,7 @@ describe('stepInstallIndexTemplatePipelines', () => { ], } as any, assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), paths: [], }; appContextService.start( @@ -431,6 +435,7 @@ describe('stepInstallIndexTemplatePipelines', () => { ], } as any, assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), paths: [], }; appContextService.start( @@ -521,6 +526,7 @@ describe('stepInstallIndexTemplatePipelines', () => { ], } as any, assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), paths: [], }; appContextService.start( @@ -574,6 +580,7 @@ describe('stepInstallIndexTemplatePipelines', () => { owner: { github: 'elastic/fleet' }, } as any, assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), paths: [], }; appContextService.start( @@ -647,6 +654,7 @@ describe('cleanupIndexTemplatePipelinesStep', () => { ], } as any, assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), paths: [], }; const mockInstalledPackageSo: SavedObject = { diff --git a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_install_kibana_assets.test.ts b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_install_kibana_assets.test.ts index 52c93c61c16e1..cf9d953868b6a 100644 --- a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_install_kibana_assets.test.ts +++ b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_install_kibana_assets.test.ts @@ -23,8 +23,25 @@ import { deleteKibanaAssets } from '../../remove'; import { KibanaSavedObjectType, type Installation } from '../../../../../types'; -import { stepInstallKibanaAssets, cleanUpKibanaAssetsStep } from './step_install_kibana_assets'; +import { createArchiveIteratorFromMap } from '../../../archive/archive_iterator'; +import { + stepInstallKibanaAssets, + cleanUpKibanaAssetsStep, + stepInstallKibanaAssetsWithStreaming, + cleanUpUnusedKibanaAssetsStep, +} from './step_install_kibana_assets'; + +jest.mock('../../../kibana/assets/saved_objects', () => { + return { + getSpaceAwareSaveobjectsClients: jest.fn().mockReturnValue({ + savedObjectClientWithSpace: jest.fn(), + savedObjectsImporter: jest.fn(), + savedObjectTagAssignmentService: jest.fn(), + savedObjectTagClient: jest.fn(), + }), + }; +}); jest.mock('../../../kibana/assets/install'); jest.mock('../../remove', () => { return { @@ -58,6 +75,7 @@ const packageInstallContext = { } as any, paths: ['some/path/1', 'some/path/2'], assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), }; describe('stepInstallKibanaAssets', () => { @@ -82,6 +100,7 @@ describe('stepInstallKibanaAssets', () => { logger: loggerMock.create(), packageInstallContext: { assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), paths: [], packageInfo: { title: 'title', @@ -102,7 +121,7 @@ describe('stepInstallKibanaAssets', () => { }); await expect(installationPromise).resolves.not.toThrowError(); - expect(mockedInstallKibanaAssetsAndReferencesMultispace).toBeCalledTimes(1); + expect(installKibanaAssetsAndReferencesMultispace).toBeCalledTimes(1); }); esClient = elasticsearchServiceMock.createClusterClient().asInternalUser; appContextService.start(createAppContextStartContractMock()); @@ -121,6 +140,7 @@ describe('stepInstallKibanaAssets', () => { logger: loggerMock.create(), packageInstallContext: { assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), paths: [], packageInfo: { title: 'title', @@ -144,6 +164,60 @@ describe('stepInstallKibanaAssets', () => { }); }); +describe('stepInstallKibanaAssetsWithStreaming', () => { + beforeEach(async () => { + soClient = savedObjectsClientMock.create(); + esClient = elasticsearchServiceMock.createClusterClient().asInternalUser; + appContextService.start(createAppContextStartContractMock()); + }); + + it('should rely on archiveIterator instead of in-memory assetsMap', async () => { + const assetsMap = new Map(); + assetsMap.get = jest.fn(); + assetsMap.set = jest.fn(); + + const archiveIterator = { + traverseEntries: jest.fn(), + getPaths: jest.fn(), + }; + + const result = await stepInstallKibanaAssetsWithStreaming({ + savedObjectsClient: soClient, + esClient, + logger: loggerMock.create(), + packageInstallContext: { + assetsMap, + archiveIterator, + paths: [], + packageInfo: { + title: 'title', + name: 'xyz', + version: '4.5.6', + description: 'test', + type: 'integration', + categories: ['cloud', 'custom'], + format_version: 'string', + release: 'experimental', + conditions: { kibana: { version: 'x.y.z' } }, + owner: { github: 'elastic/fleet' }, + }, + }, + installType: 'install', + installSource: 'registry', + spaceId: DEFAULT_SPACE_ID, + }); + + expect(result).toEqual({ installedKibanaAssetsRefs: [] }); + + // Verify that assetsMap was not used + expect(assetsMap.get).not.toBeCalled(); + expect(assetsMap.set).not.toBeCalled(); + + // Verify that archiveIterator was used + expect(archiveIterator.traverseEntries).toBeCalled(); + }); +}); + describe('cleanUpKibanaAssetsStep', () => { const mockInstalledPackageSo: SavedObject = { id: 'mocked-package', @@ -302,3 +376,84 @@ describe('cleanUpKibanaAssetsStep', () => { expect(mockedDeleteKibanaAssets).not.toBeCalled(); }); }); + +describe('cleanUpUnusedKibanaAssetsStep', () => { + const mockInstalledPackageSo: SavedObject = { + id: 'mocked-package', + attributes: { + name: 'test-package', + version: '1.0.0', + install_status: 'installing', + install_version: '1.0.0', + install_started_at: new Date().toISOString(), + install_source: 'registry', + verification_status: 'verified', + installed_kibana: [] as any, + installed_es: [] as any, + es_index_patterns: {}, + }, + type: PACKAGES_SAVED_OBJECT_TYPE, + references: [], + }; + + const installationContext = { + savedObjectsClient: soClient, + savedObjectsImporter: jest.fn(), + esClient, + logger: loggerMock.create(), + packageInstallContext, + installType: 'install' as const, + installSource: 'registry' as const, + spaceId: DEFAULT_SPACE_ID, + retryFromLastState: true, + initialState: 'install_kibana_assets' as any, + }; + + beforeEach(async () => { + soClient = savedObjectsClientMock.create(); + esClient = elasticsearchServiceMock.createClusterClient().asInternalUser; + appContextService.start(createAppContextStartContractMock()); + }); + + it('should not clean up assets if they all present in the new package', async () => { + const installedAssets = [{ type: KibanaSavedObjectType.dashboard, id: 'dashboard-1' }]; + await cleanUpUnusedKibanaAssetsStep({ + ...installationContext, + installedPkg: { + ...mockInstalledPackageSo, + attributes: { + ...mockInstalledPackageSo.attributes, + installed_kibana: installedAssets, + }, + }, + installedKibanaAssetsRefs: installedAssets, + }); + + expect(mockedDeleteKibanaAssets).not.toBeCalled(); + }); + + it('should clean up assets that are not present in the new package', async () => { + const installedAssets = [ + { type: KibanaSavedObjectType.dashboard, id: 'dashboard-1' }, + { type: KibanaSavedObjectType.dashboard, id: 'dashboard-2' }, + ]; + const newAssets = [{ type: KibanaSavedObjectType.dashboard, id: 'dashboard-1' }]; + await cleanUpUnusedKibanaAssetsStep({ + ...installationContext, + installedPkg: { + ...mockInstalledPackageSo, + attributes: { + ...mockInstalledPackageSo.attributes, + installed_kibana: installedAssets, + }, + }, + installedKibanaAssetsRefs: newAssets, + }); + + expect(mockedDeleteKibanaAssets).toBeCalledWith({ + installedObjects: [installedAssets[1]], + spaceId: 'default', + packageInfo: packageInstallContext.packageInfo, + }); + }); +}); diff --git a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_install_kibana_assets.ts b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_install_kibana_assets.ts index b5a1fff91d3b8..aabd23f2eb9cc 100644 --- a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_install_kibana_assets.ts +++ b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_install_kibana_assets.ts @@ -11,7 +11,9 @@ import { withPackageSpan } from '../../utils'; import type { InstallContext } from '../_state_machine_package_install'; import { deleteKibanaAssets } from '../../remove'; +import type { KibanaAssetReference } from '../../../../../../common/types'; import { INSTALL_STATES } from '../../../../../../common/types'; +import { installKibanaAssetsWithStreaming } from '../../../kibana/assets/install_with_streaming'; export async function stepInstallKibanaAssets(context: InstallContext) { const { savedObjectsClient, logger, installedPkg, packageInstallContext, spaceId } = context; @@ -37,6 +39,26 @@ export async function stepInstallKibanaAssets(context: InstallContext) { return { kibanaAssetPromise }; } +export async function stepInstallKibanaAssetsWithStreaming(context: InstallContext) { + const { savedObjectsClient, installedPkg, packageInstallContext, spaceId } = context; + const { packageInfo } = packageInstallContext; + const { name: pkgName } = packageInfo; + + const installedKibanaAssetsRefs = await withPackageSpan( + 'Install Kibana assets with streaming', + () => + installKibanaAssetsWithStreaming({ + savedObjectsClient, + pkgName, + packageInstallContext, + installedPkg, + spaceId, + }) + ); + + return { installedKibanaAssetsRefs }; +} + export async function cleanUpKibanaAssetsStep(context: InstallContext) { const { logger, @@ -65,3 +87,44 @@ export async function cleanUpKibanaAssetsStep(context: InstallContext) { }); } } + +/** + * Cleans up Kibana assets that are no longer in the package. As opposite to + * `cleanUpKibanaAssetsStep`, this one is used after the package assets are + * installed. + * + * This function compares the currently installed Kibana assets with the assets + * in the previous package and removes any assets that are no longer present in the + * new installation. + * + */ +export async function cleanUpUnusedKibanaAssetsStep(context: InstallContext) { + const { logger, installedPkg, packageInstallContext, spaceId, installedKibanaAssetsRefs } = + context; + const { packageInfo } = packageInstallContext; + + if (!installedKibanaAssetsRefs) { + return; + } + + logger.debug('Clean up Kibana assets that are no longer in the package'); + + // Get the assets installed by the previous package + const previousAssetRefs = installedPkg?.attributes.installed_kibana ?? []; + + // Remove any assets that are not in the new package + const nextAssetRefKeys = new Set( + installedKibanaAssetsRefs.map((asset: KibanaAssetReference) => `${asset.id}-${asset.type}`) + ); + const assetsToRemove = previousAssetRefs.filter( + (existingAsset) => !nextAssetRefKeys.has(`${existingAsset.id}-${existingAsset.type}`) + ); + + if (assetsToRemove.length === 0) { + return; + } + + await withPackageSpan('Clean up Kibana assets that are no longer in the package', async () => { + await deleteKibanaAssets({ installedObjects: assetsToRemove, spaceId, packageInfo }); + }); +} diff --git a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_install_mlmodel.test.ts b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_install_mlmodel.test.ts index 1afb436eb4361..df939f3a458b6 100644 --- a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_install_mlmodel.test.ts +++ b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_install_mlmodel.test.ts @@ -22,6 +22,8 @@ import { createAppContextStartContractMock } from '../../../../../mocks'; import { installMlModel } from '../../../elasticsearch/ml_model'; import { deleteMLModels, deletePrerequisiteAssets } from '../../remove'; +import { createArchiveIteratorFromMap } from '../../../archive/archive_iterator'; + import { stepInstallMlModel, cleanUpMlModelStep } from './step_install_mlmodel'; jest.mock('../../../elasticsearch/ml_model'); @@ -53,6 +55,7 @@ const packageInstallContext = { } as any, paths: ['some/path/1', 'some/path/2'], assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), }; let soClient: jest.Mocked; let esClient: jest.Mocked; diff --git a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_install_transforms.test.ts b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_install_transforms.test.ts index 1ac2383950b05..3bf07d52c6cbf 100644 --- a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_install_transforms.test.ts +++ b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_install_transforms.test.ts @@ -22,6 +22,8 @@ import { createAppContextStartContractMock } from '../../../../../mocks'; import { installTransforms } from '../../../elasticsearch/transform/install'; import { cleanupTransforms } from '../../remove'; +import { createArchiveIteratorFromMap } from '../../../archive/archive_iterator'; + import { stepInstallTransforms, cleanupTransformsStep } from './step_install_transforms'; jest.mock('../../../elasticsearch/transform/install'); @@ -52,6 +54,7 @@ const packageInstallContext = { } as any, paths: ['some/path/1', 'some/path/2'], assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), }; describe('stepInstallTransforms', () => { diff --git a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_remove_legacy_templates.test.ts b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_remove_legacy_templates.test.ts index 39e7159596ba8..7fa00a1c57f57 100644 --- a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_remove_legacy_templates.test.ts +++ b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_remove_legacy_templates.test.ts @@ -24,6 +24,8 @@ import { appContextService } from '../../../../app_context'; import { createAppContextStartContractMock } from '../../../../../mocks'; import { removeLegacyTemplates } from '../../../elasticsearch/template/remove_legacy'; +import { createArchiveIteratorFromMap } from '../../../archive/archive_iterator'; + import { stepRemoveLegacyTemplates } from './step_remove_legacy_templates'; jest.mock('../../../elasticsearch/template/remove_legacy'); @@ -82,6 +84,7 @@ describe('stepRemoveLegacyTemplates', () => { } as any, paths: ['some/path/1', 'some/path/2'], assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), }; appContextService.start( createAppContextStartContractMock({ diff --git a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_save_archive_entries.test.ts b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_save_archive_entries.test.ts index b03c146640488..255572d57cf49 100644 --- a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_save_archive_entries.test.ts +++ b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_save_archive_entries.test.ts @@ -21,6 +21,8 @@ import { appContextService } from '../../../../app_context'; import { createAppContextStartContractMock } from '../../../../../mocks'; import { saveArchiveEntriesFromAssetsMap, removeArchiveEntries } from '../../../archive/storage'; +import { createArchiveIteratorFromMap } from '../../../archive/archive_iterator'; + import { stepSaveArchiveEntries, cleanupArchiveEntriesStep } from './step_save_archive_entries'; jest.mock('../../../archive/storage', () => { @@ -60,6 +62,7 @@ const packageInstallContext = { Buffer.from('{"content": "data"}'), ], ]), + archiveIterator: createArchiveIteratorFromMap(new Map()), }; const getMockInstalledPackageSo = ( installedEs: EsAssetReference[] = [] diff --git a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_save_archive_entries.ts b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_save_archive_entries.ts index b0d5bb67627a6..7db44bb243f85 100644 --- a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_save_archive_entries.ts +++ b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_save_archive_entries.ts @@ -14,17 +14,32 @@ import { withPackageSpan } from '../../utils'; import type { InstallContext } from '../_state_machine_package_install'; import { INSTALL_STATES } from '../../../../../../common/types'; +import { MANIFEST_NAME } from '../../../archive/parse'; export async function stepSaveArchiveEntries(context: InstallContext) { - const { packageInstallContext, savedObjectsClient, installSource } = context; + const { packageInstallContext, savedObjectsClient, installSource, useStreaming } = context; - const { packageInfo } = packageInstallContext; + const { packageInfo, archiveIterator } = packageInstallContext; + + let assetsMap = packageInstallContext?.assetsMap; + let paths = packageInstallContext?.paths; + // For stream based installations, we don't want to save any assets but + // manifest.yaml due to the large number of assets in the package. + if (useStreaming) { + assetsMap = new Map(); + await archiveIterator.traverseEntries(async (entry) => { + if (entry.path.endsWith(MANIFEST_NAME)) { + assetsMap.set(entry.path, entry.buffer); + } + }); + paths = Array.from(assetsMap.keys()); + } const packageAssetResults = await withPackageSpan('Update archive entries', () => saveArchiveEntriesFromAssetsMap({ savedObjectsClient, - assetsMap: packageInstallContext?.assetsMap, - paths: packageInstallContext?.paths, + assetsMap, + paths, packageInfo, installSource, }) diff --git a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_save_system_object.test.ts b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_save_system_object.test.ts index aecdd0b2552c4..8d80c236aefb0 100644 --- a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_save_system_object.test.ts +++ b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_save_system_object.test.ts @@ -21,6 +21,8 @@ import { createAppContextStartContractMock } from '../../../../../mocks'; import { auditLoggingService } from '../../../../audit_logging'; import { packagePolicyService } from '../../../../package_policy'; +import { createArchiveIteratorFromMap } from '../../../archive/archive_iterator'; + import { stepSaveSystemObject } from './step_save_system_object'; jest.mock('../../../../audit_logging'); @@ -67,6 +69,7 @@ describe('updateLatestExecutedState', () => { logger, packageInstallContext: { assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), paths: [], packageInfo: { title: 'title', @@ -133,6 +136,7 @@ describe('updateLatestExecutedState', () => { logger, packageInstallContext: { assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), paths: [], packageInfo: { title: 'title', diff --git a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_update_current_write_indices.test.ts b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_update_current_write_indices.test.ts index c7f3c040b7966..017805d34efef 100644 --- a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_update_current_write_indices.test.ts +++ b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/step_update_current_write_indices.test.ts @@ -22,6 +22,8 @@ import { appContextService } from '../../../../app_context'; import { createAppContextStartContractMock } from '../../../../../mocks'; import { updateCurrentWriteIndices } from '../../../elasticsearch/template/template'; +import { createArchiveIteratorFromMap } from '../../../archive/archive_iterator'; + import { stepUpdateCurrentWriteIndices } from './step_update_current_write_indices'; jest.mock('../../../elasticsearch/template/template'); @@ -86,6 +88,7 @@ describe('stepUpdateCurrentWriteIndices', () => { } as any, paths: ['some/path/1', 'some/path/2'], assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), }; appContextService.start( createAppContextStartContractMock({ diff --git a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/update_latest_executed_state.test.ts b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/update_latest_executed_state.test.ts index d963e5fea44c9..aea879aba5479 100644 --- a/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/update_latest_executed_state.test.ts +++ b/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/steps/update_latest_executed_state.test.ts @@ -32,6 +32,8 @@ import { auditLoggingService } from '../../../../audit_logging'; import type { PackagePolicySOAttributes } from '../../../../../types'; +import { createArchiveIteratorFromMap } from '../../../archive/archive_iterator'; + import { updateLatestExecutedState } from './update_latest_executed_state'; jest.mock('../../../../audit_logging'); @@ -61,6 +63,7 @@ describe('updateLatestExecutedState', () => { logger, packageInstallContext: { assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), paths: [], packageInfo: { title: 'title', @@ -116,6 +119,7 @@ describe('updateLatestExecutedState', () => { logger, packageInstallContext: { assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), paths: [], packageInfo: { title: 'title', @@ -153,6 +157,7 @@ describe('updateLatestExecutedState', () => { logger, packageInstallContext: { assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), paths: [], packageInfo: { title: 'title', @@ -198,6 +203,7 @@ describe('updateLatestExecutedState', () => { logger, packageInstallContext: { assetsMap: new Map(), + archiveIterator: createArchiveIteratorFromMap(new Map()), paths: [], packageInfo: { title: 'title', diff --git a/x-pack/plugins/fleet/server/services/epm/packages/remove.ts b/x-pack/plugins/fleet/server/services/epm/packages/remove.ts index ac3f5def5d09c..3892eaa951e5f 100644 --- a/x-pack/plugins/fleet/server/services/epm/packages/remove.ts +++ b/x-pack/plugins/fleet/server/services/epm/packages/remove.ts @@ -148,6 +148,7 @@ export async function deleteKibanaAssets({ const namespace = SavedObjectsUtils.namespaceStringToId(spaceId); + // TODO this should be the installed package info, not the package that is being installed const minKibana = packageInfo.conditions?.kibana?.version ? minVersion(packageInfo.conditions.kibana.version) : null; diff --git a/x-pack/plugins/fleet/server/services/epm/registry/index.ts b/x-pack/plugins/fleet/server/services/epm/registry/index.ts index bb4d612aa7de3..75b9869d0a7c6 100644 --- a/x-pack/plugins/fleet/server/services/epm/registry/index.ts +++ b/x-pack/plugins/fleet/server/services/epm/registry/index.ts @@ -54,6 +54,8 @@ import { resolveDataStreamFields, resolveDataStreamsMap, withPackageSpan } from import { verifyPackageArchiveSignature } from '../packages/package_verification'; +import type { ArchiveIterator } from '../../../../common/types'; + import { fetchUrl, getResponse, getResponseStream } from './requests'; import { getRegistryUrl } from './registry_url'; @@ -309,11 +311,12 @@ async function getPackageInfoFromArchiveOrCache( export async function getPackage( name: string, version: string, - options?: { ignoreUnverified?: boolean } + options?: { ignoreUnverified?: boolean; useStreaming?: boolean } ): Promise<{ paths: string[]; packageInfo: ArchivePackage; assetsMap: AssetsMap; + archiveIterator: ArchiveIterator; verificationResult?: PackageVerificationResult; }> { const verifyPackage = appContextService.getExperimentalFeatures().packageVerification; @@ -340,18 +343,18 @@ export async function getPackage( setVerificationResult({ name, version }, latestVerificationResult); } - const { assetsMap, paths } = await unpackBufferToAssetsMap({ - name, - version, + const contentType = ensureContentType(archivePath); + const { paths, assetsMap, archiveIterator } = await unpackBufferToAssetsMap({ archiveBuffer, - contentType: ensureContentType(archivePath), + contentType, + useStreaming: options?.useStreaming, }); if (!packageInfo) { packageInfo = await getPackageInfoFromArchiveOrCache(name, version, archiveBuffer, archivePath); } - return { paths, packageInfo, assetsMap, verificationResult }; + return { paths, packageInfo, assetsMap, archiveIterator, verificationResult }; } export async function getPackageFieldsMetadata( @@ -397,7 +400,7 @@ export async function getPackageFieldsMetadata( } } -function ensureContentType(archivePath: string) { +export function ensureContentType(archivePath: string) { const contentType = mime.lookup(archivePath); if (!contentType) { diff --git a/x-pack/plugins/fleet/server/services/package_policies/experimental_datastream_features.ts b/x-pack/plugins/fleet/server/services/package_policies/experimental_datastream_features.ts index edf31991634b9..cd1c26942aa0c 100644 --- a/x-pack/plugins/fleet/server/services/package_policies/experimental_datastream_features.ts +++ b/x-pack/plugins/fleet/server/services/package_policies/experimental_datastream_features.ts @@ -30,6 +30,7 @@ import { applyDocOnlyValueToMapping, forEachMappings, } from '../experimental_datastream_features_helper'; +import { createArchiveIteratorFromMap } from '../epm/archive/archive_iterator'; export async function handleExperimentalDatastreamFeatureOptIn({ soClient, @@ -75,6 +76,7 @@ export async function handleExperimentalDatastreamFeatureOptIn({ return prepareTemplate({ packageInstallContext: { assetsMap, + archiveIterator: createArchiveIteratorFromMap(assetsMap), packageInfo, paths, }, diff --git a/x-pack/test/fleet_api_integration/apis/epm/index.js b/x-pack/test/fleet_api_integration/apis/epm/index.js index 3caed7da79f65..cae50abdae762 100644 --- a/x-pack/test/fleet_api_integration/apis/epm/index.js +++ b/x-pack/test/fleet_api_integration/apis/epm/index.js @@ -31,6 +31,7 @@ export default function loadTests({ loadTestFile, getService }) { loadTestFile(require.resolve('./install_update')); loadTestFile(require.resolve('./install_tsds_disable')); loadTestFile(require.resolve('./install_tag_assets')); + loadTestFile(require.resolve('./install_with_streaming')); loadTestFile(require.resolve('./bulk_upgrade')); loadTestFile(require.resolve('./bulk_install')); loadTestFile(require.resolve('./update_assets')); diff --git a/x-pack/test/fleet_api_integration/apis/epm/install_by_upload.ts b/x-pack/test/fleet_api_integration/apis/epm/install_by_upload.ts index e6fa2930cf84d..e32328b4e22cc 100644 --- a/x-pack/test/fleet_api_integration/apis/epm/install_by_upload.ts +++ b/x-pack/test/fleet_api_integration/apis/epm/install_by_upload.ts @@ -195,7 +195,7 @@ export default function (providerContext: FtrProviderContext) { .send(buf) .expect(400); expect((res.error as HTTPError).text).to.equal( - '{"statusCode":400,"error":"Bad Request","message":"Archive seems empty. Assumed content type was application/gzip, check if this matches the archive type."}' + '{"statusCode":400,"error":"Bad Request","message":"Manifest file manifest.yml not found in paths."}' ); }); diff --git a/x-pack/test/fleet_api_integration/apis/epm/install_with_streaming.ts b/x-pack/test/fleet_api_integration/apis/epm/install_with_streaming.ts new file mode 100644 index 0000000000000..152e3dfd4c69d --- /dev/null +++ b/x-pack/test/fleet_api_integration/apis/epm/install_with_streaming.ts @@ -0,0 +1,65 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ +import type { Client } from '@elastic/elasticsearch'; +import { INGEST_SAVED_OBJECT_INDEX } from '@kbn/core-saved-objects-server'; +import { Installation } from '@kbn/fleet-plugin/server/types'; +import expect from 'expect'; +import { FtrProviderContext } from '../../../api_integration/ftr_provider_context'; +import { skipIfNoDockerRegistry, isDockerRegistryEnabledOrSkipped } from '../../helpers'; + +export default function (providerContext: FtrProviderContext) { + const { getService } = providerContext; + const es: Client = getService('es'); + const supertest = getService('supertest'); + const fleetAndAgents = getService('fleetAndAgents'); + + const uninstallPackage = async (pkg: string, version: string) => { + await supertest.delete(`/api/fleet/epm/packages/${pkg}/${version}`).set('kbn-xsrf', 'xxxx'); + }; + const installPackage = (pkg: string, version: string, opts?: { force?: boolean }) => { + return supertest + .post(`/api/fleet/epm/packages/${pkg}/${version}`) + .set('kbn-xsrf', 'xxxx') + .send({ force: !!opts?.force }); + }; + + const getInstallationSavedObject = async (pkg: string): Promise => { + const res: { _source?: { 'epm-packages': Installation } } = await es.transport.request({ + method: 'GET', + path: `/${INGEST_SAVED_OBJECT_INDEX}/_doc/epm-packages:${pkg}`, + }); + + return res?._source?.['epm-packages'] as Installation; + }; + + describe('Installs a package using stream-based approach', () => { + skipIfNoDockerRegistry(providerContext); + + before(async () => { + await fleetAndAgents.setup(); + }); + + describe('security_detection_engine package', () => { + after(async () => { + if (!isDockerRegistryEnabledOrSkipped(providerContext)) return; + await uninstallPackage('security_detection_engine', '8.16.0'); + }); + it('should install security-rule assets from the package', async () => { + await installPackage('security_detection_engine', '8.16.0').expect(200); + const installationSO = await getInstallationSavedObject('security_detection_engine'); + expect(installationSO?.installed_kibana).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + id: expect.any(String), + type: 'security-rule', + }), + ]) + ); + }); + }); + }); +} diff --git a/x-pack/test/fleet_api_integration/apis/fixtures/test_packages/security_detection_engine/security_detection_engine-8.16.0.zip b/x-pack/test/fleet_api_integration/apis/fixtures/test_packages/security_detection_engine/security_detection_engine-8.16.0.zip new file mode 100644 index 0000000000000000000000000000000000000000..b57713203546d8eb9b6f5c32f49c72d9bb6388ac GIT binary patch literal 63220 zcmeHwOKdAydY&fs#A6zhvEg1AFkqbF^?2^psQ7*g_jZGlc>4;i7bxDj?zG05BCAMN zt60UUDoR&lOp*qYz%a7PB0yG2fHvTbm+{t{*j_n61_lgoy~!qqVZdGm2=e{sQB|y> zB(_GkRWl~LbtTs0KmYm9|2~iZnv!4yvwhHla34_eN+C@*T_ikZtjdBQrw zp*ah5XNa#m*KF?NK^9C*=DVZdO83kycJ0oYKK4wL1uoNVo4F(YL|}STpXrXlhOT2+ zL)zoJ`h0kfsMO#DF7^gWO{T(w!qPy`*>goM7Ee=`GIE*<^iru z^$RnnU+R`EKu{-Uo;gE*nPGa?r5*qcYWKod-HixhIn30D6E@UAe<1TIm;kiN5!q~O zY6Wo%A_NfwNGydzVWhH*I2oiToc_JYuWGPJzme0pg*pjpTcCc7}N z7aPcd%)ARBBN9D{85OZSlUgCn+<^~~G(qtM5GH`+)ZhW49x%%%f{^TjfP?EK9*M4i z5ls&)$y?h>HW!MQ;4{wf;vpdK(V${T{G61TkogmxE3ghw$KWWkst5n(-~49`k*6Ex zR3~Nx>XPGfB#pBf9u!QkOR+oA1JKsLP1po*jLX~sw{xn^_*-mf>lTDM8WMppk~JU^-oxJ{SX*qOV_2rZ87+_i`lfAzCL;(a8@XQ2*GX=z>CE*4E+za0 z@f{;2HzkPzn-O5q7p}TJ`9%?e4?rBa(8K7HU|tN4jx6@T&-rs`LtS9MywM~i!h)+;<~pG4M7KwTJ5eVmsUwWYk(3{yT;_hj zVD(7UIg!v;*MXj%xNK~~mH-Fn`iKf(FDZCxEZE3%r_@ePxBb*xvCgA;Icj{Mg_2NE zgi)LlI+g>Y1&s}U<3q~10@MT|q^d>pk^X_+nZi@RP)W_epmhUbbS1|L=}~avL#0T< zj&apvJu5kh@OMBqQmZarFyF%D`97ZGmKZo&4(WUaOSIMp zCs+>W@{l52B4R!Qb^g+FZT^((vVsEXC+?tEAb%us5+h?PCl$Jr1A;Cp0L~9Gw5awfOlHDdpCcG(dj_C)-&7n7- zF{f9x#0>o!>@}iGfD2HBI}>aZFYP%(xLUFwa~AKr0$i!MfivABk4M_6=b>Kn85FF=cMzm z8JtUU03od;^3s76qqH!{9Dm{)C}11y3!_7WKstmV8Tu6d4;t|-a$j@Qf{;7Kgk-b{ zC~+bF1Ur)oNvMXnIlQny0k}Z6IR^Bz!T_6AL6DJAv*ge>JVC!9rXOQCmb2GOTbx$?LEJjNaxZ=!I51 z69{puLGaN>3*P1&Jp@rC&fL@IIFr{IvXKH3s(yhpH6~~zUWc^17$6Ad%(KkE_56u7 zi@8IJuC6!M9qWfeVQ?X$0VRK%MsH zNV+%TEk`%Oi&NMeb)Rbt0-1}WQ(&8$@(J{DY9g>;?v(?&p)15QF;AIvEc6HwQ}^LD zbA1nYNfAS`-3)>b26W*ELvg@n;6dTEbED0r%>@^c!GJq@3D=aoN%R8Bg8Go-%KOUl zp#*nj@8Fh^NJx-qiH4KFM?Ak36rq72_XszgYidb z0jf_!kSRItjw*HL6 z^IRTU?pZ$fuffuQ*Vp|E8$mcA4Tt5%sM)M(_4cT()fyE;YYo~h&8ReoL!(wNx6Pq| z6k>K5bGM0&C~@KYmM}+b2QCT}jt|()TB&Xrqq5c*woI*79wIGMZjFHNz|7(j z1niCoqex}4(+M12;%Gwo)EJQQ2Go&J)7x#}*(g^_fK{q!?a`>EwJPmFtF4=@QhCIo z_C0uo7}-6TcH4y!WpU`QeOM}hYa8WqX*j5A^-*o0)rM_d8?;AttzI3K^m4P*)Xnm< zTn^(IB~K!rdpdl^Dy4Fpbq!bdAa);`p6lyd?AX>v>=m*EF3e!=gghtE2))9nQ%(TN zMf!SMaDCVkv)MX;yyJL0lE|-s3t~$UiX8 zK_`j$8@U}~6y$rRChyjal12VxG5-SQAf+(c&T^Mz8f>oM`KR$BFS0eI4&C9S6vPcrE4=AZYH0Rw@ z4lR+J4*>aPa#V%m$$W%Qwun!8EJBVvDSNn8A_hV4V%Z((9H`hN90cgB{;MuKJ$~7L z)9H2D{wX`@9e;O!ue-#k#+J z((9g{vg2NE|M28szl&@8N4p2F_x6uIWIK2dx|JR5gRcR!f6S;s32MKK_g=EYZg2M$ zAa{255BB>XZ{=R@_m2qX%i|vFu#-*?%K7y{r^ilS_fC#ayJ&t7V2}2XUiQ#R_pp1^ zN2qXwYpnZSd|{`rItK^TRIc+H*!KuOwtIZ?ac}>_SAF*C_+YP#i#uIl*4a7eil)%1 z-Gk2l;TGHL9Ckh=?0W1NP~)iMcZ^8%h3IIVR<@lPNgPiXc6%dv99>bk69kQRBE)!W>ssO zMnkLBnnO(=REFBXY}8tZp)(#zKdRhks8y5#ypMjQ zK<{DnW2jKX1c1Ws3+l&QLO;T!to7qsKXTRjW$H(Y8h-QFey{qMzqGMIKQ~7Wg;k;k zG1pB2NZUPB8%`?UK@QtTt zVO(I)b%9gljLxCrfy*3173`bHP90La0y|XiD}6klk+kDHvVKx z!&H|)^sE{Az1yrO+x6w_rVJhCGXwq|e}S>!dS-4TQ-CDm&?Ik9UbijhBAS4VMKW^A zv1mDg3D1+~%8RqPZ5PX>O0AwivR0`7G$2{~sX$UNOgVG7R@b`IA&AYD=EcP$($`JwUY`SIA)WarFPP6RCveOr zch*x;W;t`ZvtDW>BnUWWlbp3y8r7`n&Ppq(kjjNj`tik9C1o|sg{%a#H^-tX2zWNMzfKiVk>J?){SO8;bk@p zt&B-cGpWxT8ALImCF>bPF_GD7Wo?e7(X6D*VK%2&fN5g9o0&7*%}QdtfnqjoO*1+3 z+sGg>6N(%tW|Wx8`G9)%h*?e;ZlIV|VkX4Am9@!$M!lKH4+6z(60_b&DWr_8-)qzx zsSu)(Rbr+TQs#)6ToTtPWR;jn2eY0zH@%+B6adAHa%MfHkTSM%uTiTd!tho`vu%x9 zGRAFY&S|ez6HYr&%qC~nstE(s$RIHj1_&r-l$gozqMkismJ^%_6f;W9a>AOoGM7}Z zG*UVbpT7sC^ z9-Xnob*Yrdtpmku60=lFEZu2lZHl~6O0H@LidiLQ!tZEgkeG?Y7EsJ8F%#OooFlrIkz!$oOd9ZlzjT ziHsb_M{**|ZLD!wVW>Hdk2BVOv0PtSW*+7Uy?e+^xr)Ujs}ZvLAj#DG;8?*Kd8^|1 zID=g*mvIK_$~bCNl-$mLazWSjS)Y8eUMIL3qdcx1he zpJ>5bybH~j*4yx;bV3u>zFH`WDv)9MX6+|NHW7;5gzRdR6%}i~6tq|JK1%RD?sAZb z&f)3EqQaGH^KXtY{2PAC-Sm z)lAkpLbU)Yfu#6?i`o6j+gw%8D!lJfw!5gC5TNLSETll{L81C>AH@-5ad*@^pyKZD z25~Av@fGVw|A`GYHt0tbp`c>(Ur$t?|E*PuP@oEzze@S};>y5!-Ok=&w=gwgB`u)) z2Xj=1qe?wiRt|{6I>-LP7a~A^vWx|Vl#)eH01HFIpL{*$)N4)kj z>Lh_jKm9LH|Mc&_vavxwH}mMXR$(h^I$niEJFCJ{+x4_YZT7a{Uyehj zeg47QDT?HZDl9MZ2&nUr>mh2zIpY`k*Zr4TEC0d!x#xHX#psM+@*+<#@+g}}RhHE^ zVw4^C?TdVgl~|*O|MPL6Aa~&y~*eZ)zG!{GkM?4$0YOrxzyKuLk*BPZ4U=E6-5OMTvpW?@}E}rfCuY zw;1o=M5yOQabGk>MPc3-s?iYjJVkqj=a3!T5>JhpWxUAW1Tl}Q)}DJ|zQ{YM`l-HY zlCYo>>I2&9CjHHM^hF*u6CKwuWWUM%doBv5qOF08`k5FlZQx>9XbAG`n*SS)SA4Yc z?>{fny8>ShO5psV=gwz&h7zNW4>dD=k>`I>x#y?t0tN}6Emo6%J&WPOfuOYdgaMty za(SyD*@#a>oTxgVP z)mF8}s->+O;Nq7Ak6-^X&&ho{(PyBhAl{Rgoo21prG64lQ9l>aENxY>s;rIq^ERLY zXp0?G>grd$!FI8SNq)9!AOLn-V)GCK7RZ3Mz+TllzQlk?WKPg%h)a2Ir?XSK3zu@T zd31EQ+%D8gjZ(Q%*{YQa?FLri*UAi!R*_F?;}pBL_=Y3lDnJI+Ru2T6Wq4G9daazs z<9Tr`8JGTePICA@skQdM{^dXV)t}$kpr4!7+S653(FJ!WlbUUrC2xEIa7x0?NpmNG ze{G4Ipx+yKNgp7OW{$LWWza%R(kA|DwFg?=Fx%B(t6ssc^K#)+lm`vf>hE%GkVk1+ z&l=1F6ky$E;u>*b9+liDu7_)%@aO0nee%+*=fkVt+Y0~8$Es4{&3p&C1cjq~UJstC zdzv;%K%<_xK{abf7W_@B>x}xC8vV%9&Ftwev?40L;yaWE<5kzmQ{fegC(!9sd2>WS zhA67M6Fx{~l_#b>W1`$H3GQ>j{jzL=ZdbsVxRol5tV ze|BIEO$P94tK5SdixzL(FdikU&dm3eBdf(Na3DxxrM-4L{Mo zLe*z&?$dSIr{x@gGIU@|<2;U2ibRp9bGatbmQFs6&P_ewWRR{HJ@-2_y-|wLp!+`x z^+8-Uh+^Wv3Sw^?=^Y@LF%sh{1%i1weOENi5;Gkq5O$ikkX{O2;Ur-0Am!YxkoB z4uX+59(Xbhh~F1%(eIStG%k6cKckpVidL%i{+wK72(D`;KK;6zPD=QCXdsAaqS` z=|KqQp2Pu?OBg~yrQeYXk6)#~zn4+)7d#Cay-~QVrdE_ zg93~k?wLl640)xwg}V^(Z$2_eV#-PKwj7yN6!W=~(6EuvCX+JF=YB*X#}EJD_y0o? z?3lQ$a10}xz-aB*7oF|w9QIAIsLWOvHc-D+_)gJG7)D1)Eaal%1Sm+fpU~VBC%bPUrPMQj zFo$PIb=v*yGe=`DZ4-Ux1>^Z05$)ZDKcH)@??yqQghP%uD%*4aQG|jf`vh5yIW@>5 zzPo3%MubsOh(?K=RG2DT9A_Y*#i*wRt|kBrTp52YVrmky z5k%a*qD(Fv`kBxCi95F;??_I(#r!VGKJZl(mI?&F+@VEjERpt&cl<`ljv>Z{=0hYo z#B8ZtFfwtcSS&I|GUPQKKT0%Cp!0c*g*HTkR{eV%;C{I zw;GBYF`|r(Z<2=NP+|xYX1-}6D-KJkNxZR#YeQ3d*DDMcc4W_K;&E@+ok*)4maReAdYxB17V{PWFHx zArbzGCwpYt{g&j7*jIXwSmdr)4}Y~I&;JQy!dLBC2mdV3#t_=>t#Ba zWB(_fxq0|yI^rCcdt`dlyydN#o?(=CA*2${|2oKkCT$a%GI%tT}=-53;Ko4XnkaSr2sjb-lcBSI-5pmp~Z(?Wq;ycncgY( zF4*jJ=vlKsOnUVM$-8`-j7W~TZ+&3dRzO6*v~)q&Sl6(b;Fjg}Vrd;_?ai_4U2kI^ z2#e>%fKOaMD7b!sm}py)g*f5NVVF4tc0OTOz@7)Q;Byoe*O_kvw#}Z>gABhgJ#~rg zW7?akb;y|r^G}#53$M$SuIg%+NUWWlo&pi{@GSp~^WRw>&&@D5=pA82?pdCoKUJoA^XS<+%XOV7=N1<9?>6iMMQF!1u`$Ee4PTU6?S*q z!aPwVnRt)y9qSmEI<_DgtTQ((EKZ_5PuPvY*U(@M!kP2^?|=WyQOa!bK70Sn=@>X^ z1M7(BjZ2gDA=xCmKHYmGtP_iM2v_@a&tb3yv8&Q%EOe7S?ebkC(3%{DReMEN#WTl* z(sk4UT;c0a&hnEBdwOxxPCJ&dwy}bXOTRyu-}~ zM^X+2j6m^{2|UwxSutR}?uX}lZ;pF=tZ1+@@5ZeZHS0+JT>ZQf*L?rhqA1kbG{B_6?r8b8m>fFA=4PQB$q=XiHC@j5FKW=|y@d z?yfxgK9^D}(M2L0eZo2LJ4hm)#`B?ZC)ev{YaZL=Qfe*P(~!{69F@EGexwz7h?m;r z0@vY&uJSfi^ei1Xxz8mzs0u27WyKyR{g>OYcvl-;(0OchY24DZ>RCAMN|i~1MC-0r z0dh~$aMvze0jyAp|Bkk$#IMrdi`!S!-SqQ|&qena#PZ?e=Nq=U+-Wpd+vc+IZ$E4Q z;lJ3}pr1#JpTC4{F6;PNGWI%tRucX*9Y3ph9ytP}TTHKD8Hn7^@~2u^t83+!SSd=O zHa)>ZeX%_e$$}`vM`*r%hR=!Tjqy+ld?-rw07+!kwih%*~-(hYs*%Z}-fU^?UB zzDN;#aCXMur#lkH2O$i)81p0P>)BcN?U0`Y_W{{alf$OVoAeKT2FMqoNuGd+L+1!h zBZTN#;Ig^&;iafZvlaPf1cE*|3V}jy@#XBy7~psxB&^i;-5Gw zjz+j6v5<}Q#-bis;_rpI{B@G$HtH--vMeSN(~-++DJW?TZfkz|;DqjV+~M_BJTBu- zcQuC3I>#gC1=cy9h~u#-wUs?x(N}_uNV5nSECD^C;nY8^buS5(&zWC z7y8KEjdzpg5f=JHNoeBZC9Y(6BpfYZd$^ONZ#-T&^gnJ=LQ3lHBXP@)3x0P?>+0=< z{#2;b@35b89|7MjAl8o3l3qH#%fngz`1dv-_gPU^ZznejEa3Id7TeXSvdwL)4HX(8 zX`z=d#PD>VCDyZ~3S`8P!N`Eh|np1bv&o4d~-H%T;zJo)gZ`hr7cjmP! z=^gs>>V!XDBS{`2efsPF+}SVHLMny*PDJ;i7B^T=4_kecTnG<-MFQmUiR1EW5I;j%3tuGH#An?urpcbv{~H&S=%Q zpOXkjCllsEx=2dX;pG3f{a^p&A5h|reoj96%0GMW=fGe~stkP-oFzWLyz#dk`b$wP z)oo3HkJl~|a8>o9@sI!F)Azoyu|YpuA>56P&p&&$3fzPJ-R{w8w-CGyLiSatt0sQ) z*M6`1mpE^oe!jEN#GjziJ5fJ6?xT@gTZq?iEG)F}kH5Y`3u0}W?T#0uP<4ao`}HN| zAnE$X293ZE|J4eO&@mV{4@11dAC-qxpU*%0^Ob?8i$&hn1*#S(KGcQZ_-utPBx(mP zGE%&l7gY~bpU*%0bXC}j6_&|;in|DYa#*NTnD|hi{_)?f(5H0au$$$MN{2<@_x}Cr z;8o93g~S$V7_Ux1H3-z_^Uwaa8_ol~FBKl`KqvjSdAox78dzWUDz&adFVzy5`djj#S4ef$6ALgH5d literal 0 HcmV?d00001