diff --git a/deploy-manage/cloud-organization/billing.md b/deploy-manage/cloud-organization/billing.md index 894d1d589..579cbc67e 100644 --- a/deploy-manage/cloud-organization/billing.md +++ b/deploy-manage/cloud-organization/billing.md @@ -2,6 +2,7 @@ mapped_urls: - https://www.elastic.co/guide/en/cloud/current/ec-billing.html - https://www.elastic.co/guide/en/serverless/current/general-manage-billing.html + - https://www.elastic.co/guide/en/serverless/current/general-billing-stop-project.html applies_to: deployment: ess: all @@ -21,7 +22,7 @@ Elastic charges a recurring fee for using our offerings on {{ecloud}}. In this s * [Elasticsearch projects](/deploy-manage/cloud-organization/billing/elasticsearch-billing-dimensions.md) * [Elastic Observability projects](/deploy-manage/cloud-organization/billing/elastic-observability-billing-dimensions.md) * [Elastic Security projects](/deploy-manage/cloud-organization/billing/security-billing-dimensions.md) - + For more information, you can also refer to our [{{ech}}](https://www.elastic.co/pricing) and [{{serverless-full}}](https://www.elastic.co/pricing/serverless-search) pricing pages. ## Billing models diff --git a/deploy-manage/deploy/cloud-enterprise/air-gapped-install.md b/deploy-manage/deploy/cloud-enterprise/air-gapped-install.md index 06fc1afe3..da252cc9d 100644 --- a/deploy-manage/deploy/cloud-enterprise/air-gapped-install.md +++ b/deploy-manage/deploy/cloud-enterprise/air-gapped-install.md @@ -1,26 +1,33 @@ --- +applies_to: + deployment: + ece: all mapped_urls: - - https://www.elastic.co/guide/en/elastic-stack/current/air-gapped-install.html - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-install-offline.html --- -# Air gapped install +# Air-gapped install [ece-install-offline] -% What needs to be done: Refine +Installing ECE on hosts without internet access is commonly referred to as an *offline* or *air-gapped* installation. ECE supports two air-gapped installation methods, depending on whether a private Docker registry is available. In both cases, you must download multiple Docker images and the installation script from Elastic, and load them onto your hosts or private registry. -% GitHub issue: https://github.com/elastic/docs-projects/issues/309 +::::{note} + The versioning of {{es}} and {{kib}} is synchronized and versions where the major, minor, and patch levels match can be used together. Differences in build versions indicated by a dash do not affect compatibility. +:::: -% Scope notes: In the issue +Before you start, you must: -% Use migrated content from existing pages that map to this page: +* Follow the same prerequisites described in [](./install.md#ece-install-prerequisites). This includes [](./identify-deployment-scenario.md) and [](./prepare-environment.md) steps. +* [Configure your operating system](./configure-operating-system.md) in all ECE hosts. +* Be part of the `docker` group to run the installation script. You should not install Elastic Cloud Enterprise as the `root` user. +* Set up and run a local copy of the Elastic Package Repository, otherwise your deployments with APM server and Elastic agent won’t work. Refer to the [Running EPR in air-gapped environments](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/air-gapped.md#air-gapped-diy-epr) documentation. -% - [ ] ./raw-migrated-files/stack-docs/elastic-stack/air-gapped-install.md -% - [ ] ./raw-migrated-files/cloud/cloud-enterprise/ece-install-offline.md -% Notes: 3 child docs +When you are ready to install ECE, you can proceed: -⚠️ **This page is a work in progress.** ⚠️ +* [With your private Docker registry](./ece-install-offline-with-registry.md) +* [Without a private Docker registry](./ece-install-offline-no-registry.md) -The documentation team is working to combine content pulled from the following pages: +After installing ECE in your hosts, you can continue with [](./post-installation-steps.md). -* [/raw-migrated-files/stack-docs/elastic-stack/air-gapped-install.md](/raw-migrated-files/stack-docs/elastic-stack/air-gapped-install.md) -* [/raw-migrated-files/cloud/cloud-enterprise/ece-install-offline.md](/raw-migrated-files/cloud/cloud-enterprise/ece-install-offline.md) \ No newline at end of file +::::{note} +Deployment End-of-life (EOL) information relies on the connection to [https://www.elastic.co/support/eol.json](https://www.elastic.co/support/eol.json). If EOL information is updated, Elastic may require you to reconnect to [https://www.elastic.co/support/eol.json](https://www.elastic.co/support/eol.json) over the internet to get this information reflected. +:::: diff --git a/deploy-manage/deploy/cloud-enterprise/alternative-install-ece-with-ansible.md b/deploy-manage/deploy/cloud-enterprise/alternative-install-ece-with-ansible.md index f8e95b1a2..adb2887f4 100644 --- a/deploy-manage/deploy/cloud-enterprise/alternative-install-ece-with-ansible.md +++ b/deploy-manage/deploy/cloud-enterprise/alternative-install-ece-with-ansible.md @@ -1,6 +1,10 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configure-ansible.html +navigation_title: Ansible playbook --- # Alternative: Install ECE with Ansible [ece-configure-ansible] diff --git a/deploy-manage/deploy/cloud-enterprise/assign-roles-to-hosts.md b/deploy-manage/deploy/cloud-enterprise/assign-roles-to-hosts.md index 2048ac20b..dd4f7d51e 100644 --- a/deploy-manage/deploy/cloud-enterprise/assign-roles-to-hosts.md +++ b/deploy-manage/deploy/cloud-enterprise/assign-roles-to-hosts.md @@ -1,4 +1,7 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-change-roles.html --- diff --git a/deploy-manage/deploy/cloud-enterprise/ce-add-support-for-node-roles-autoscaling.md b/deploy-manage/deploy/cloud-enterprise/ce-add-support-for-node-roles-autoscaling.md index d3c4db11e..8199d3510 100644 --- a/deploy-manage/deploy/cloud-enterprise/ce-add-support-for-node-roles-autoscaling.md +++ b/deploy-manage/deploy/cloud-enterprise/ce-add-support-for-node-roles-autoscaling.md @@ -1,14 +1,19 @@ --- +navigation_title: Data tiers and autoscaling support +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ce-add-support-for-node-roles-and-autoscaling.html --- # Updating custom templates to support node_roles and autoscaling [ce-add-support-for-node-roles-and-autoscaling] -Custom deployment templates should be updated in order to take advantage of new Elastic Cloud Enterprise features, such as [Data tiers](../../../manage-data/lifecycle/data-tiers.md) (that is, the new cold and frozen data tiers) and [Deployment autoscaling](../../autoscaling.md). By updating these templates we also ensure forward compatibility with future Elastic Cloud Enterprise versions that will require certain fields such as `node_roles` and `id` to be present in the deployment configuration. - -System owned deployment templates have already been updated to support both data tiers with `node_roles` and autoscaling. However, the custom templates that you created need to be manually updated by following the steps in this guide. +Templates created in older versions of ECE should be updated in order to take advantage of new Elastic Cloud Enterprise features, such as [Data tiers](../../../manage-data/lifecycle/data-tiers.md), and [Deployment autoscaling](../../autoscaling.md). By updating these templates we also ensure forward compatibility with future Elastic Cloud Enterprise versions that will require certain fields such as `node_roles` and `id` to be present in the deployment configuration. +::::{note} +System owned deployment templates are automatically updated during the ECE upgrade process to support both data tiers with `node_roles` and autoscaling. However, custom templates that you created must be manually updated by following the steps in this guide. +:::: ## Adding support for node_roles [ece_adding_support_for_node_roles] diff --git a/deploy-manage/deploy/cloud-enterprise/change-allocator-disconnect-timeout.md b/deploy-manage/deploy/cloud-enterprise/change-allocator-disconnect-timeout.md index 92b66c191..47fb379d7 100644 --- a/deploy-manage/deploy/cloud-enterprise/change-allocator-disconnect-timeout.md +++ b/deploy-manage/deploy/cloud-enterprise/change-allocator-disconnect-timeout.md @@ -1,4 +1,7 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-change-allocator-disconnect-timeout.html --- diff --git a/deploy-manage/deploy/cloud-enterprise/change-ece-api-url.md b/deploy-manage/deploy/cloud-enterprise/change-ece-api-url.md index 2d3be29af..958d83b01 100644 --- a/deploy-manage/deploy/cloud-enterprise/change-ece-api-url.md +++ b/deploy-manage/deploy/cloud-enterprise/change-ece-api-url.md @@ -1,4 +1,7 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-config-api-base-url.html --- diff --git a/deploy-manage/deploy/cloud-enterprise/change-endpoint-urls.md b/deploy-manage/deploy/cloud-enterprise/change-endpoint-urls.md index 6f7dc9b83..034a7637c 100644 --- a/deploy-manage/deploy/cloud-enterprise/change-endpoint-urls.md +++ b/deploy-manage/deploy/cloud-enterprise/change-endpoint-urls.md @@ -1,4 +1,7 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-administering-endpoints.html --- @@ -9,15 +12,11 @@ For applications without SSL or HTTPS protocol support, you can use a local endp By default, cluster and Kibana endpoint URLs are constructed according to the following pattern, where `CLUSTER_ID` and `LOCAL_HOST_IP` are values that depend on your specific installation: -::::{admonition} -```text +```sh http://CLUSTER_ID.LOCAL_HOST_IP.ip.es.io:9200 https://CLUSTER_ID.LOCAL_HOST_IP.ip.es.io:9243 ``` -:::: - - For example: ```sh @@ -29,7 +28,6 @@ https://2882c82e54d4361.us-west-5.aws.found.io:9243 To find your endpoints, select a deployment review the information on the **Elasticsearch** and **Kibana** pages. :::: - To change endpoints in the Cloud UI: 1. [Log into the Cloud UI](log-into-cloud-ui.md). diff --git a/deploy-manage/deploy/cloud-enterprise/configure-allocator-affinity.md b/deploy-manage/deploy/cloud-enterprise/configure-allocator-affinity.md index 076970f65..d224c8a66 100644 --- a/deploy-manage/deploy/cloud-enterprise/configure-allocator-affinity.md +++ b/deploy-manage/deploy/cloud-enterprise/configure-allocator-affinity.md @@ -1,4 +1,7 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configuring-allocator-affinity.html --- diff --git a/deploy-manage/deploy/cloud-enterprise/configure-deployment-templates.md b/deploy-manage/deploy/cloud-enterprise/configure-deployment-templates.md index f93aee41a..81ae3dddf 100644 --- a/deploy-manage/deploy/cloud-enterprise/configure-deployment-templates.md +++ b/deploy-manage/deploy/cloud-enterprise/configure-deployment-templates.md @@ -1,4 +1,7 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configuring-ece-templates.html --- diff --git a/deploy-manage/deploy/cloud-enterprise/configure-host-rhel-onprem.md b/deploy-manage/deploy/cloud-enterprise/configure-host-rhel-onprem.md deleted file mode 100644 index cf3a64ff1..000000000 --- a/deploy-manage/deploy/cloud-enterprise/configure-host-rhel-onprem.md +++ /dev/null @@ -1,351 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configure-hosts-rhel-centos-onprem.html ---- - -# Configure host RHEL onprem [ece-configure-hosts-rhel-centos-onprem] - - -## Red Hat Enterprise Linux 8 (RHEL 8), 9 (RHEL 9), and Rocky Linux 8 and 9 [ece-setup-rhel8-podman-onprem] - -The following instructions show you how to prepare your hosts on Red Hat Enterprise Linux 8 (RHEL 8), 9 (RHEL 9), and Rocky Linux 8 and 9. - -* [Prerequisites](#ece-prerequisites-rhel8-onprem) -* [Configure the host](#ece-configure-hosts-rhel8-podman-onprem) - - -### Prerequisites [ece-prerequisites-rhel8-onprem] - -Create a RHEL 8 (the version must be >= 8.5, but <9), RHEL 9, Rocky Linux 8, or Rocky Linux 9 instance. - -* For RHEL 8, follow your internal guidelines to add a vanilla RHEL 8 instance to your environment. Note that the version must be >= 8.5, but <9. - -Verify that required traffic is allowed. - - -### Configure the host [ece-configure-hosts-rhel8-podman-onprem] - -1. Install the OS packages `lvm2`, `iptables`, `sysstat`, and `net-tools` by executing: - - ```sh - sudo dnf install lvm2 iptables sysstat net-tools <1> - ``` - - 1. The ECE diagnostic script requires `net-tools`.
- - - ::::{note} - For RHEL 9 and Rocky Linux 9, also install the `containernetworking-plugins` package using:
- - ```sh - sudo dnf -y install containernetworking-plugins - ``` - - :::: - -2. Remove Docker and previously installed podman packages (if previously installed). - - ```sh - sudo dnf remove docker docker-ce podman podman-remote containerd.io - ``` - -3. As a sudoers user, edit the `/etc/selinux/config` file: - - 1. If you are not using SELinux, set it to permissive mode: - - ```text - SELINUX=permissive - ``` - - 2. If you are using SELinux, set it to enforcing mode: - - ::::{note} - Avoid customizing the host Docker path `/mnt/data/docker` when using SELinux. Otherwise the ECE installer script needs to be adjusted. - :::: - - - ```text - SELINUX=enforcing - ``` - -4. Install podman: - - * Install the latest available version `4.*` using dnf. - - ```sh - sudo dnf install podman-4.* podman-remote-4.* - ``` - - * To prevent automatic Podman major version updates, configure the Podman version to be locked while still allowing minor and patch updates. - - ```sh - ## Install versionlock - sudo dnf install 'dnf-command(versionlock)' - - ## Lock major version - sudo dnf versionlock add --raw 'podman-4.*' - sudo dnf versionlock add --raw 'podman-remote-4.*' - - ## Verify that podman-4.* and podman-remote-4.* appear in the output - sudo dnf versionlock list - ``` - -5. [This step is for RHEL 9 and Rocky Linux 9 only] Switch the network stack from Netavark to CNI: - - 1. If the */etc/containers/containers.conf* file does not exist, copy the */usr/share/containers/containers.conf* file to the */etc/containers/* directory (for example, using `cp /usr/share/containers/containers.conf /etc/containers/`). - 2. Open the */etc/containers/containers.conf* file. Navigate to the **network** section and make sure that the **network_backend** setting is set to `cni`. - 3. Reboot the system (`reboot`). - 4. Check that the network stack has changed to `cni`:
- - ```sh - cat /etc/containers/containers.conf - [...] - [network] - network_backend="cni" - [...] - ``` - -6. If podman requires a proxy in your infrastructure setup, modify the `/usr/share/containers/containers.conf` file and add the `HTTP_PROXY` and `HTTPS_PROXY` environment variables in the [engine] section. Please note that multiple env variables in that configuration file exists — use the one in the [engine] section. - - Example: - - ```text - [engine] - env = ["HTTP_PROXY=http://{proxy-ip}:{proxy-port}", "HTTPS_PROXY=http://{proxy-ip}:{proxy-port}"] - ``` - -7. Reload systemd configuration - - ```sh - sudo systemctl daemon-reload - ``` - -8. Create OS groups, if they do not exist yet - - Reference: [Users and permissions](ece-users-permissions.md) - - ```sh - sudo groupadd elastic - sudo groupadd podman - ``` - -9. Add user `elastic` to the `podman` group - - Reference: [Users and permissions](ece-users-permissions.md) - - ```sh - sudo useradd -g "elastic" -G "podman" elastic - ``` - -10. As a sudoers user, add the following line to /etc/sudoers.d/99-ece-users - - Reference: [Users and permissions](ece-users-permissions.md) - - ```text - elastic ALL=(ALL) NOPASSWD:ALL - ``` - -11. Add the required options to the kernel boot arguments - - ```sh - sudo /sbin/grubby --update-kernel=ALL --args='cgroup_enable=memory cgroup.memory=nokmem swapaccount=1' - ``` - -12. Create the directory - - ```sh - sudo mkdir -p /etc/systemd/system/podman.socket.d - ``` - -13. As a sudoers user, create the file `/etc/systemd/system/podman.socket.d/podman.conf` with the following content. Set the correct ownership and permission. - - ::::{important} - Both `ListenStream=` and `ListenStream=/var/run/docker.sock` parameters are required! - :::: - - - File content: - - ```text - [Socket] - ListenStream= - ListenStream=/var/run/docker.sock - SocketMode=770 - SocketUser=elastic - SocketGroup=podman - ``` - - File ownership and permission: - - ```sh - sudo chown root:root /etc/systemd/system/podman.socket.d/podman.conf - sudo chmod 0644 /etc/systemd/system/podman.socket.d/podman.conf - ``` - -14. As a sudoers user, create the (text) file `/usr/bin/docker` with the following content. Verify that the regular double quotes in the text file are used (ASCII code Hex 22) - - ```text - #!/bin/bash - podman-remote --url unix:///var/run/docker.sock "$@" - ``` - -15. Set the file permissions on `/usr/bin/docker` - - ```sh - sudo chmod 0755 /usr/bin/docker - ``` - -16. As a sudoers user, add the following two lines to section `[storage]` in the file `/etc/containers/storage.conf`. Verify that those parameters are only defined once. Either remove or comment out potentially existing parameters. - - ::::{note} - Avoid customizing the host Docker path `/mnt/data/docker` when using SELinux. Otherwise the ECE installer script needs to be adjusted. - :::: - - - ```text - runroot = "/mnt/data/docker/runroot/" - graphroot = "/mnt/data/docker" - ``` - -17. Enable podman so that itself and running containers start automatically after a reboot - - ```sh - sudo systemctl enable podman.service - sudo systemctl enable podman-restart.service - ``` - -18. Enable the `overlay` kernel module (check [Use the OverlayFS storage driver](https://docs.docker.com/storage/storagedriver/overlayfs-driver/)) that the Podman `overlay` storage driver uses (check [Working with the Container Storage library and tools in Red Hat Enterprise Linux](https://www.redhat.com/en/blog/working-container-storage-library-and-tools-red-hat-enterprise-linux#:~:text=Storage%20Configuration)). - - In the Docker world there are two overlay drivers, overlay and overlay2. Today most users use the overlay2 driver, so we just use that one, and called it overlay. Refer also to [Use the OverlayFS storage driver](https://docs.docker.com/storage/storagedriver/overlayfs-driver/). - - ```sh - echo "overlay" | sudo tee -a /etc/modules-load.d/overlay.conf - ``` - -19. Format the additional data partition - - ```sh - sudo mkfs.xfs /dev/nvme1n1 - ``` - -20. Create the `/mnt/data/` directory used as a mount point - - ```sh - sudo install -o elastic -g elastic -d -m 700 /mnt/data - ``` - -21. As a sudoers user, modify the entry for the XFS volume in the `/etc/fstab` file to add `pquota,prjquota`. The default filesystem path used by Elastic Cloud Enterprise is `/mnt/data`. - - ::::{note} - Replace `/dev/nvme1n1` in the following example with the corresponding device on your host, and add this example configuration as a single line to `/etc/fstab`. - :::: - - - ```text - /dev/nvme1n1 /mnt/data xfs defaults,nofail,x-systemd.automount,prjquota,pquota 0 2 - ``` - -22. Restart the local-fs target - - ```sh - sudo systemctl daemon-reload - sudo systemctl restart local-fs.target - ``` - -23. Set the permissions on the newly mounted device - - ```sh - ls /mnt/data - sudo chown elastic:elastic /mnt/data - ``` - -24. Create the `/mnt/data/docker` directory for the Docker service storage - - ::::{note} - Avoid customizing the host Docker path `/mnt/data/docker` when using SELinux. Otherwise the ECE installer script needs to be adjusted. - :::: - - - ```sh - sudo install -o elastic -g elastic -d -m 700 /mnt/data/docker - ``` - -25. If you want to use FirewallD, please ensure you meet the [networking prerequisites](ece-networking-prereq.md). Otherwise, you can disable it with: - - ```sh - sudo systemctl disable firewalld - ``` - - ::::{note} - If FirewallD does not exist on your VM, you can skip this step. - :::: - -26. Configure kernel parameters - - ```sh - cat <" - } - } - } - ``` - -30. Restart the podman service by running this command: - - ```sh - sudo systemctl daemon-reload - sudo systemctl restart podman - ``` - -31. Reboot the RHEL host - - ```sh - sudo reboot - ``` diff --git a/deploy-manage/deploy/cloud-enterprise/configure-host-rhel-cloud.md b/deploy-manage/deploy/cloud-enterprise/configure-host-rhel.md similarity index 92% rename from deploy-manage/deploy/cloud-enterprise/configure-host-rhel-cloud.md rename to deploy-manage/deploy/cloud-enterprise/configure-host-rhel.md index 99410a9f1..f9a1aa670 100644 --- a/deploy-manage/deploy/cloud-enterprise/configure-host-rhel-cloud.md +++ b/deploy-manage/deploy/cloud-enterprise/configure-host-rhel.md @@ -1,31 +1,32 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configure-hosts-rhel-centos-cloud.html + - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configure-hosts-rhel-centos-onprem.html +navigation_title: RHEL --- -# Configure host RHEL cloud [ece-configure-hosts-rhel-centos-cloud] +# Configure a RHEL host [ece-configure-hosts-rhel-centos] -## Red Hat Enterprise Linux 8 (RHEL 8), 9 (RHEL 9), and Rocky Linux 8 and 9 [ece-setup-rhel8-podman-cloud] The following instructions show you how to prepare your hosts on Red Hat Enterprise Linux 8 (RHEL 8), 9 (RHEL 9), and Rocky Linux 8 and 9. -* [Prerequisites](#ece-prerequisites-rhel8-cloud) -* [Configure the host](#ece-configure-hosts-rhel8-podman-cloud) +* [Prerequisites](#ece-prerequisites-rhel8) +* [Configure the host](#ece-configure-hosts-rhel8-podman) -### Prerequisites [ece-prerequisites-rhel8-cloud] +## Prerequisites [ece-prerequisites-rhel8] -Create a RHEL 8 (the version must be >= 8.5, but <9), RHEL 9, Rocky Linux 8, or Rocky Linux 9 VM. +Follow your internal guidelines to create a RHEL 8 (the version must be >= 8.5), RHEL 9, Rocky Linux 8, or Rocky Linux 9 server or VM in your environment. -* For RHEL 8, follow your internal guidelines to add a vanilla RHEL 8 VM to your environment. Note that the version must be >= 8.5, but <9. - -Verify that required traffic is allowed. Check the [Networking prerequisites](ece-networking-prereq.md) and [Google Cloud Platform (GCP)](/deploy-manage/deploy/cloud-enterprise/prepare-environment.md) guidelines for a list of ports that need to be open. The technical configuration highly depends on the underlying infrastructure. +Verify that required traffic is allowed. Check the [Networking prerequisites](ece-networking-prereq.md) for a list of ports that need to be open. The technical configuration depends on the underlying infrastructure. **Example:** For AWS, allowing traffic between hosts is implemented using security groups. - -### Configure the host [ece-configure-hosts-rhel8-podman-cloud] +## Configure the host [ece-configure-hosts-rhel8-podman] 1. Install the OS packages `lvm2`, `iptables`, `sysstat`, and `net-tools` by executing: diff --git a/deploy-manage/deploy/cloud-enterprise/configure-host-suse-onprem.md b/deploy-manage/deploy/cloud-enterprise/configure-host-suse-onprem.md deleted file mode 100644 index cd18a0c70..000000000 --- a/deploy-manage/deploy/cloud-enterprise/configure-host-suse-onprem.md +++ /dev/null @@ -1,344 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configure-hosts-sles12-onprem.html ---- - -# Configure host SUSE onprem [ece-configure-hosts-sles12-onprem] - -The following instructions show you how to prepare your hosts on SLES 12 SP5 or 15. - -* [Install Docker](#ece-install-docker-sles12-onprem) -* [Set up XFS on SLES](#ece-xfs-setup-sles12-onprem) -* [Update the configurations settings](#ece-update-config-sles-onprem) -* [Configure the Docker daemon options](#ece-configure-docker-daemon-sles12-onprem) - -If you want to install Elastic Cloud Enterprise on your own hosts, the steps for preparing your hosts can take a bit of time. There are two ways you can approach this: - -* **Think like a minimalist**: [Install the correct version of Docker](#ece-install-docker-sles12-onprem) on hosts that meet the [prerequisites](prepare-environment.md) for Elastic Cloud Enterprise, then skip ahead and [install Elastic Cloud Enterprise](install.md). Be aware that some checks during the installation can fail with this approach, which will mean doing further host preparation work before retrying the installation. -* **Cover your bases**: If you want to make absolutely sure that your installation of Elastic Cloud Enterprise can succeed on hosts that meet the [prerequisites](prepare-environment.md), or if any of the checks during the installation failed previously, run through the full preparation steps in this section and then and [install Elastic Cloud Enterprise](install.md). You’ll do a bit more work now, but life will be simpler later on. - -Regardless of which approach you take, the steps in this section need to be performed on every host that you want to use with Elastic Cloud Enterprise. - - -## Install Docker [ece-install-docker-sles12-onprem] - -::::{important} -Make sure to use a combination of Linux distribution and Docker version that is supported, following our official [Support matrix](https://www.elastic.co/support/matrix#elastic-cloud-enterprise). Using unsupported combinations can cause multiple issues with you ECE environment, such as failures to create system deployments, to upgrade workload deployments, proxy timeouts, and more. -:::: - - -1. Remove Docker and previously installed podman packages (if previously installed). - - ```sh - sudo zypper remove -y docker docker-ce podman podman-remote - ``` - -2. Update packages to the latest available versions - - ```sh - sudo zypper refresh - sudo zypper update -y - ``` - -3. Install Docker and other required packages: - - * For SLES 12: - - ```sh - sudo zypper install -y docker=24.0.7_ce-98.109.3 - ``` - - * For SLES 15: - - ```sh - sudo zypper install -y curl device-mapper lvm2 net-tools docker=24.0.7_ce-150000.198.2 net-tools - ``` - -4. Disable nscd, as it interferes with Elastic’s services: - - ```sh - sudo systemctl stop nscd - sudo systemctl disable nscd - ``` - - - -## Set up OS groups and user [ece_set_up_os_groups_and_user_2] - -1. If they don’t already exist, create the following OS groups: - - ```sh - sudo groupadd elastic - sudo groupadd docker - ``` - -2. Add the user to these groups: - - ```sh - sudo usermod -aG elastic,docker $USER - ``` - - - -## Set up XFS on SLES [ece-xfs-setup-sles12-onprem] - -XFS is required to support disk space quotas for Elasticsearch data directories. Some Linux distributions such as RHEL and Rocky Linux already provide XFS as the default file system. On SLES 12 and 15, you need to set up an XFS file system and have quotas enabled. - -Disk space quotas set a limit on the amount of disk space an Elasticsearch cluster node can use. Currently, quotas are calculated by a static ratio of 1:32, which means that for every 1 GB of RAM a cluster is given, a cluster node is allowed to consume 32 GB of disk space. - -::::{note} -Using LVM, `mdadm`, or a combination of the two for block device management is possible, but the configuration is not covered here, nor is it provided as part of supporting Elastic Cloud Enterprise. -:::: - - -::::{important} -You must use XFS and have quotas enabled on all allocators, otherwise disk usage won’t display correctly. -:::: - - -**Example:** Set up XFS on a single, pre-partitioned block device named `/dev/xvdg1`. Replace `/dev/xvdg1` in the following example with the corresponding device on your host. - -1. Format the partition: - - ```sh - sudo mkfs.xfs /dev/xvdg1 - ``` - -2. Create the `/mnt/data/` directory as a mount point: - - ```sh - sudo install -o $USER -g elastic -d -m 700 /mnt/data - ``` - -3. Add an entry to the `/etc/fstab` file for the new XFS volume. The default filesystem path used by Elastic Cloud Enterprise is `/mnt/data`. - - ```sh - /dev/xvdg1 /mnt/data xfs defaults,pquota,prjquota,x-systemd.automount 0 0 - ``` - -4. Regenerate the mount files: - - ```sh - sudo mount -a - ``` - - - -## Update the configurations settings [ece-update-config-sles-onprem] - -1. Stop the Docker service: - - ```sh - sudo systemctl stop docker - ``` - -2. Enable cgroup accounting for memory and swap space. - - 1. In the `/etc/default/grub` file, ensure that the `GRUB_CMDLINE_LINUX=` variable includes these values: - - ```sh - cgroup_enable=memory swapaccount=1 cgroup.memory=nokmem - ``` - - 2. Update your Grub configuration: - - ```sh - sudo update-bootloader - ``` - -3. Configure kernel parameters - - ```sh - cat <" - } - } - } - ``` - -6. If you did not create the mount point earlier (if you did not set up XFS), create the `/mnt/data/` directory as a mount point: - - ```sh - sudo install -o $USER -g elastic -d -m 700 /mnt/data - ``` - -7. If you [set up a new device with XFS](#ece-xfs-setup-sles12-onprem) earlier: - - 1. Mount the block device (change the device name if you use a different device than `/dev/xvdg1`): - - ```sh - sudo mount /dev/xvdg1 - ``` - - 2. Set the permissions on the newly mounted device: - - ```sh - sudo chown $USER:elastic /mnt/data - ``` - -8. Create the `/mnt/data/docker` directory for the Docker service storage: - - ```sh - sudo install -o $USER -g elastic -d -m 700 /mnt/data/docker - ``` - - - -## Configure the Docker daemon [ece-configure-docker-daemon-sles12-onprem] - -1. Edit `/etc/docker/daemon.json`, and make sure that the following configuration values are present:
- - ```json - { - "storage-driver": "overlay2", - "bip":"172.17.42.1/16", - "icc": false, - "log-driver": "json-file", - "log-opts": { - "max-size": "500m", - "max-file": "10" - }, - "data-root": "/mnt/data/docker" - } - ``` - -2. The user installing {{ece}} must have a User ID (UID) and Group ID (GID) of 1000 or higher. Make sure that the GID matches the ID of the `elastic`` group created earlier (likely to be 1000). You can set this using the following command: - - ```sh - sudo usermod -g $USER - ``` - -3. Apply the updated Docker daemon configuration: - - Reload the Docker daemon configuration: - - ```sh - sudo systemctl daemon-reload - ``` - - Restart the Docker service: - - ```sh - sudo systemctl restart docker - ``` - - Enable Docker to start on boot: - - ```sh - sudo systemctl enable docker - ``` - -4. Recommended: Tune your network settings. - - Create a `70-cloudenterprise.conf` file in the `/etc/sysctl.d/` file path that includes these network settings: - - ```sh - cat << SETTINGS | sudo tee /etc/sysctl.d/70-cloudenterprise.conf - net.ipv4.tcp_max_syn_backlog=65536 - net.core.somaxconn=32768 - net.core.netdev_max_backlog=32768 - net.ipv4.tcp_keepalive_time=1800 - net.netfilter.nf_conntrack_tcp_timeout_established=7200 - net.netfilter.nf_conntrack_max=262140 - SETTINGS - ``` - - 1. Ensure settings in /etc/sysctl.d/*.conf are applied on boot - - ```sh - SCRIPT_LOCATION="/var/lib/cloud/scripts/per-boot/00-load-sysctl-settings" - sudo sh -c "cat << EOF > ${SCRIPT_LOCATION} - #!/bin/bash - - set -x - - lsmod | grep ip_conntrack || modprobe ip_conntrack - - sysctl --system - EOF - " - sudo chmod +x ${SCRIPT_LOCATION} - ``` - -5. Reboot your system to ensure that all configuration changes take effect: - - ```sh - sudo reboot - ``` - -6. If the Docker daemon is not already running, start it: - - ```sh - sudo systemctl start docker - ``` - -7. After rebooting, verify that your Docker settings persist as expected: - - ```sh - sudo docker info | grep Root - ``` - - If the command returns `Docker Root Dir: /mnt/data/docker`, then your changes were applied successfully and persist as expected. - - If the command returns `Docker Root Dir: /var/lib/docker`, then you need to troubleshoot the previous configuration steps until the Docker settings are applied successfully before continuing with the installation process. For more information, check [Custom Docker daemon options](https://docs.docker.com/engine/admin/systemd/#/custom-docker-daemon-options) in the Docker documentation. - -8. Repeat these steps on other hosts that you want to use with Elastic Cloud Enterprise or follow the steps in the next section to start installing Elastic Cloud Enterprise. diff --git a/deploy-manage/deploy/cloud-enterprise/configure-host-suse-cloud.md b/deploy-manage/deploy/cloud-enterprise/configure-host-suse.md similarity index 80% rename from deploy-manage/deploy/cloud-enterprise/configure-host-suse-cloud.md rename to deploy-manage/deploy/cloud-enterprise/configure-host-suse.md index d397f7a1d..5fd99dd82 100644 --- a/deploy-manage/deploy/cloud-enterprise/configure-host-suse-cloud.md +++ b/deploy-manage/deploy/cloud-enterprise/configure-host-suse.md @@ -1,26 +1,31 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configure-hosts-sles12-cloud.html + - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configure-hosts-sles12-onprem.html +navigation_title: SUSE --- -# Configure host SUSE cloud [ece-configure-hosts-sles12-cloud] +# Configure a SUSE host [ece-configure-hosts-sles12] The following instructions show you how to prepare your hosts on SLES 12 SP5 or 15. -* [Install Docker](#ece-install-docker-sles12-cloud) -* [Set up XFS on SLES](#ece-xfs-setup-sles12-cloud) -* [Update the configurations settings](#ece-update-config-sles-cloud) -* [Configure the Docker daemon options](#ece-configure-docker-daemon-sles12-cloud) +* [Install Docker](#ece-install-docker-sles12) +* [Set up XFS on SLES](#ece-xfs-setup-sles12) +* [Update the configurations settings](#ece-update-config-sles) +* [Configure the Docker daemon options](#ece-configure-docker-daemon-sles12) -If you want to install Elastic Cloud Enterprise on your own hosts, the steps for preparing your hosts can take a bit of time. There are two ways you can approach this: +If you want to install {{ece}} (ECE) on your own hosts, the steps for preparing your hosts can take a bit of time. There are two ways you can approach this: -* **Think like a minimalist**: [Install the correct version of Docker](#ece-install-docker-sles12-cloud) on hosts that meet the [prerequisites](prepare-environment.md) for Elastic Cloud Enterprise, then skip ahead and [install Elastic Cloud Enterprise](install.md). Be aware that some checks during the installation can fail with this approach, which will mean doing further host preparation work before retrying the installation. -* **Cover your bases**: If you want to make absolutely sure that your installation of Elastic Cloud Enterprise can succeed on hosts that meet the [prerequisites](prepare-environment.md), or if any of the checks during the installation failed previously, run through the full preparation steps in this section and then and [install Elastic Cloud Enterprise](install.md). You’ll do a bit more work now, but life will be simpler later on. +* **Think like a minimalist**: [Install the correct version of Docker](#ece-install-docker-sles12) on hosts that meet the [prerequisites](prepare-environment.md) for ECE, then skip ahead and [install ECE](install.md#install-ece). Be aware that some checks during the installation can fail with this approach, which will mean doing further host preparation work before retrying the installation. +* **Cover your bases**: If you want to make absolutely sure that your installation of {{ece}} can succeed on hosts that meet the [prerequisites](prepare-environment.md), or if any of the checks during the installation failed previously, run through the full preparation steps in this section and then and [install ECE](install.md#install-ece). You’ll do a bit more work now, but life will be simpler later on. -Regardless of which approach you take, the steps in this section need to be performed on every host that you want to use with Elastic Cloud Enterprise. +Regardless of which approach you take, the steps in this section need to be performed on every host that you want to use with ECE. -## Install Docker [ece-install-docker-sles12-cloud] +## Install Docker [ece-install-docker-sles12] ::::{important} Make sure to use a combination of Linux distribution and Docker version that is supported, following our official [Support matrix](https://www.elastic.co/support/matrix#elastic-cloud-enterprise). Using unsupported combinations can cause multiple issues with you ECE environment, such as failures to create system deployments, to upgrade workload deployments, proxy timeouts, and more. @@ -80,14 +85,14 @@ Make sure to use a combination of Linux distribution and Docker version that is -## Set up XFS on SLES [ece-xfs-setup-sles12-cloud] +## Set up XFS on SLES [ece-xfs-setup-sles12] XFS is required to support disk space quotas for Elasticsearch data directories. Some Linux distributions such as RHEL and Rocky Linux already provide XFS as the default file system. On SLES 12 and 15, you need to set up an XFS file system and have quotas enabled. Disk space quotas set a limit on the amount of disk space an Elasticsearch cluster node can use. Currently, quotas are calculated by a static ratio of 1:32, which means that for every 1 GB of RAM a cluster is given, a cluster node is allowed to consume 32 GB of disk space. ::::{note} -Using LVM, `mdadm`, or a combination of the two for block device management is possible, but the configuration is not covered here, nor is it provided as part of supporting Elastic Cloud Enterprise. +Using LVM, `mdadm`, or a combination of the two for block device management is possible, but the configuration is not covered here, nor is it provided as part of supporting ECE. :::: @@ -110,7 +115,7 @@ You must use XFS and have quotas enabled on all allocators, otherwise disk usage sudo install -o $USER -g elastic -d -m 700 /mnt/data ``` -3. Add an entry to the `/etc/fstab` file for the new XFS volume. The default filesystem path used by Elastic Cloud Enterprise is `/mnt/data`. +3. Add an entry to the `/etc/fstab` file for the new XFS volume. The default filesystem path used by ECE is `/mnt/data`. ```sh /dev/xvdg1 /mnt/data xfs defaults,pquota,prjquota,x-systemd.automount 0 0 @@ -124,7 +129,7 @@ You must use XFS and have quotas enabled on all allocators, otherwise disk usage -## Update the configurations settings [ece-update-config-sles-cloud] +## Update the configurations settings [ece-update-config-sles] 1. Stop the Docker service: @@ -176,7 +181,7 @@ You must use XFS and have quotas enabled on all allocators, otherwise disk usage 4. Adjust the system limits. - Add the following configuration values to the `/etc/security/limits.conf` file. These values are derived from our experience with the Elastic Cloud hosted offering and should be used for Elastic Cloud Enterprise as well. + Add the following configuration values to the `/etc/security/limits.conf` file. These values are derived from our experience with the Elastic Cloud hosted offering and should be used for ECE as well. ::::{tip} If you are using a user name other than `elastic`, adjust the configuration values accordingly. @@ -221,7 +226,7 @@ You must use XFS and have quotas enabled on all allocators, otherwise disk usage sudo install -o $USER -g elastic -d -m 700 /mnt/data ``` -7. If you [set up a new device with XFS](#ece-xfs-setup-sles12-cloud) earlier: +7. If you [set up a new device with XFS](#ece-xfs-setup-sles12) earlier: 1. Mount the block device (change the device name if you use a different device than `/dev/xvdg1`): @@ -243,7 +248,7 @@ You must use XFS and have quotas enabled on all allocators, otherwise disk usage -## Configure the Docker daemon [ece-configure-docker-daemon-sles12-cloud] +## Configure the Docker daemon [ece-configure-docker-daemon-sles12] 1. Edit `/etc/docker/daemon.json`, and make sure that the following configuration values are present:
@@ -261,7 +266,7 @@ You must use XFS and have quotas enabled on all allocators, otherwise disk usage } ``` -2. The user installing {{ece}} must have a User ID (UID) and Group ID (GID) of 1000 or higher. Make sure that the GID matches the ID of the `elastic`` group created earlier (likely to be 1000). You can set this using the following command: +2. The user installing ECE must have a User ID (UID) and Group ID (GID) of 1000 or higher. Make sure that the GID matches the ID of the `elastic`` group created earlier (likely to be 1000). You can set this using the following command: ```sh sudo usermod -g $USER @@ -269,23 +274,23 @@ You must use XFS and have quotas enabled on all allocators, otherwise disk usage 3. Apply the updated Docker daemon configuration: - Reload the Docker daemon configuration: + * Reload the Docker daemon configuration: - ```sh - sudo systemctl daemon-reload - ``` + ```sh + sudo systemctl daemon-reload + ``` - Restart the Docker service: + * Restart the Docker service: - ```sh - sudo systemctl restart docker - ``` + ```sh + sudo systemctl restart docker + ``` - Enable Docker to start on boot: + * Enable Docker to start on boot: - ```sh - sudo systemctl enable docker - ``` + ```sh + sudo systemctl enable docker + ``` 4. Recommended: Tune your network settings. @@ -341,4 +346,4 @@ You must use XFS and have quotas enabled on all allocators, otherwise disk usage If the command returns `Docker Root Dir: /var/lib/docker`, then you need to troubleshoot the previous configuration steps until the Docker settings are applied successfully before continuing with the installation process. For more information, check [Custom Docker daemon options](https://docs.docker.com/engine/admin/systemd/#/custom-docker-daemon-options) in the Docker documentation. -8. Repeat these steps on other hosts that you want to use with Elastic Cloud Enterprise or follow the steps in the next section to start installing Elastic Cloud Enterprise. +8. Repeat these steps on other hosts that you want to use with ECE or follow the steps in the next section to start installing {{ece}}. diff --git a/deploy-manage/deploy/cloud-enterprise/configure-host-ubuntu-onprem.md b/deploy-manage/deploy/cloud-enterprise/configure-host-ubuntu-onprem.md deleted file mode 100644 index c0a23de7d..000000000 --- a/deploy-manage/deploy/cloud-enterprise/configure-host-ubuntu-onprem.md +++ /dev/null @@ -1,309 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configure-hosts-ubuntu-onprem.html ---- - -# Configure host Ubuntu onprem [ece-configure-hosts-ubuntu-onprem] - -The following instructions show you how to prepare your hosts on 20.04 LTS (Focal Fossa) and Ubuntu 22.04 LTS (Jammy Jellyfish). - -* [Install Docker 24.0](#ece-install-docker-ubuntu-onprem) -* [Set up XFS quotas](#ece-xfs-setup-ubuntu-onprem) -* [Update the configurations settings](#ece-update-config-ubuntu-onprem) -* [Configure the Docker daemon options](#ece-configure-docker-daemon-ubuntu-onprem) - - -## Install Docker [ece-install-docker-ubuntu-onprem] - -Install Docker LTS version 24.0 for Ubuntu 20.04 or 22.04. - -::::{important} -Make sure to use a combination of Linux distribution and Docker version that is supported, following our official [Support matrix](https://www.elastic.co/support/matrix#elastic-cloud-enterprise). Using unsupported combinations can cause multiple issues with you ECE environment, such as failures to create system deployments, to upgrade workload deployments, proxy timeouts, and more. -:::: - - -::::{note} -Docker 25 and higher are not compatible with ECE 3.7. -:::: - - -1. Install the Docker repository dependencies: - - ```sh - sudo apt-get install ca-certificates curl gnupg lsb-release - ``` - -2. Add Docker’s official GPG key: - - ```sh - sudo mkdir -m 0755 -p /etc/apt/keyrings - curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg - ``` - -3. Add the stable Docker repository: - - ```sh - echo \ - "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \ - $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null - ``` - -4. Install the correct version of the `docker-ce` package, for Ubuntu 20.04 LTS (Focal Fossa) or Ubuntu 22.04 LTS (Jammy Jellyfish): - - ```sh - sudo apt install -y docker-ce=5:24.0.* docker-ce-cli=5:24.0.* containerd.io - ``` - - - -## Set up XFS quotas [ece-xfs-setup-ubuntu-onprem] - -XFS is required to support disk space quotas for Elasticsearch data directories. Some Linux distributions such as RHEL and Rocky Linux already provide XFS as the default file system. On Ubuntu, you need to set up an XFS file system and have quotas enabled. - -Disk space quotas set a limit on the amount of disk space an Elasticsearch cluster node can use. Currently, quotas are calculated by a static ratio of 1:32, which means that for every 1 GB of RAM a cluster is given, a cluster node is allowed to consume 32 GB of disk space. - -::::{note} -Using LVM, `mdadm`, or a combination of the two for block device management is possible, but the configuration is not covered here, and it is not supported by Elastic Cloud Enterprise. -:::: - - -::::{important} -You must use XFS and have quotas enabled on all allocators, otherwise disk usage won’t display correctly. -:::: - - -**Example:** Set up XFS on a single, pre-partitioned block device named `/dev/xvdg1`. - -1. Format the partition: - - ```sh - sudo mkfs.xfs /dev/xvdg1 - ``` - -2. Create the `/mnt/data/` directory as a mount point: - - ```sh - sudo install -o $USER -g $USER -d -m 700 /mnt/data - ``` - -3. Add an entry to the `/etc/fstab` file for the new XFS volume. The default filesystem path used by Elastic Cloud Enterprise is `/mnt/data`. - - ```sh - /dev/xvdg1 /mnt/data xfs defaults,nofail,x-systemd.automount,prjquota,pquota 0 2 - ``` - -4. Regenerate the mount files: - - ```sh - sudo systemctl daemon-reload - sudo systemctl restart local-fs.target - ``` - - - -## Update the configurations settings [ece-update-config-ubuntu-onprem] - -1. Stop the Docker service: - - ```sh - sudo systemctl stop docker - ``` - -2. Enable cgroup accounting for memory and swap space. - - 1. In the `/etc/default/grub` file, ensure that the `GRUB_CMDLINE_LINUX=` variable includes these values: - - ```sh - cgroup_enable=memory swapaccount=1 cgroup.memory=nokmem - ``` - - 2. Update your Grub configuration: - - ```sh - sudo update-grub - ``` - -3. Configure kernel parameters - - ```sh - cat <" - } - } - } - ``` - -6. If you did not create the mount point earlier (if you did not set up XFS), create the `/mnt/data/` directory as a mount point: - - ```sh - sudo install -o $USER -g $USER -d -m 700 /mnt/data - ``` - -7. If you [set up a new device with XFS](#ece-xfs-setup-ubuntu-onprem) earlier: - - 1. Mount the block device (change the device name if you use a different device than `/dev/xvdg1`): - - ```sh - sudo mount /dev/xvdg1 /mnt/data - ``` - - 2. Set the permissions on the newly mounted device: - - ```sh - sudo chown $USER:$USER /mnt/data - ``` - -8. Create the `/mnt/data/docker` directory for the Docker service storage: - - ```sh - sudo install -o $USER -g $USER -d -m 700 /mnt/data/docker - ``` - - - -## Configure the Docker daemon options [ece-configure-docker-daemon-ubuntu-onprem] - -::::{tip} -Docker creates a bridge IP address that can conflict with IP addresses on your internal network. To avoid an IP address conflict, change the `--bip=172.17.42.1/16` parameter in our examples to something that you know will work. If there is no conflict, you can omit the `--bip` parameter. The `--bip` parameter is internal to the host and can be set to the same IP for each host in the cluster. More information on Docker daemon options can be found in the [dockerd command line reference](https://docs.docker.com/engine/reference/commandline/dockerd/). -:::: - - -::::{tip} -You can specify `--log-opt max-size` and `--log-opt max-file` to define the Docker daemon containers log rotation. -:::: - - -1. Update `/etc/systemd/system/docker.service.d/docker.conf`. If the file path and file do not exist, create them first. - - ```sh - [Unit] - Description=Docker Service - After=multi-user.target - - [Service] - Environment="DOCKER_OPTS=-H unix:///run/docker.sock --data-root /mnt/data/docker --storage-driver=overlay2 --bip=172.17.42.1/16 --raw-logs --log-opt max-size=500m --log-opt max-file=10 --icc=false" - ExecStart= - ExecStart=/usr/bin/dockerd $DOCKER_OPTS - ``` - -2. Apply the updated Docker daemon configuration: - - Reload the Docker daemon configuration: - - ```sh - sudo systemctl daemon-reload - ``` - - Restart the Docker service: - - ```sh - sudo systemctl restart docker - ``` - - Enable Docker to start on boot: - - ```sh - sudo systemctl enable docker - ``` - -3. Enable your user to communicate with the Docker subsystem by adding it to the `docker` group: - - ```sh - sudo usermod -aG docker $USER - ``` - -4. Recommended: Tune your network settings. - - Create a `70-cloudenterprise.conf` file in the `/etc/sysctl.d/` file path that includes these network settings: - - ```sh - cat << SETTINGS | sudo tee /etc/sysctl.d/70-cloudenterprise.conf - net.ipv4.tcp_max_syn_backlog=65536 - net.core.somaxconn=32768 - net.core.netdev_max_backlog=32768 - SETTINGS - ``` - -5. Pin the Docker version to ensure that the package does not get upgraded: - - ```sh - echo "docker-ce hold" | sudo dpkg --set-selections - echo "docker-ce-cli hold" | sudo dpkg --set-selections - echo "containerd.io hold" | sudo dpkg --set-selections - ``` - -6. Reboot your system to ensure that all configuration changes take effect: - - ```sh - sudo reboot - ``` - -7. After rebooting, verify that your Docker settings persist as expected: - - ```sh - sudo docker info | grep Root - ``` - - If the command returns `Docker Root Dir: /mnt/data/docker`, then your changes were applied successfully and persist as expected. - - If the command returns `Docker Root Dir: /var/lib/docker`, then you need to troubleshoot the previous configuration steps until the Docker settings are applied successfully before continuing with the installation process. For more information, check [Custom Docker daemon options](https://docs.docker.com/engine/admin/systemd/#/custom-docker-daemon-options) in the Docker documentation. - -8. Repeat these steps on other hosts that you want to use with Elastic Cloud Enterprise or follow the steps in the next section to start installing Elastic Cloud Enterprise. diff --git a/deploy-manage/deploy/cloud-enterprise/configure-host-ubuntu-cloud.md b/deploy-manage/deploy/cloud-enterprise/configure-host-ubuntu.md similarity index 94% rename from deploy-manage/deploy/cloud-enterprise/configure-host-ubuntu-cloud.md rename to deploy-manage/deploy/cloud-enterprise/configure-host-ubuntu.md index 86d667482..fd98d15e6 100644 --- a/deploy-manage/deploy/cloud-enterprise/configure-host-ubuntu-cloud.md +++ b/deploy-manage/deploy/cloud-enterprise/configure-host-ubuntu.md @@ -1,19 +1,24 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configure-hosts-ubuntu-cloud.html + - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configure-hosts-ubuntu-onprem.html +navigation_title: Ubuntu --- -# Configure host Ubuntu cloud [ece-configure-hosts-ubuntu-cloud] +# Configure an Ubuntu host [ece-configure-hosts-ubuntu] The following instructions show you how to prepare your hosts on 20.04 LTS (Focal Fossa) and Ubuntu 22.04 LTS (Jammy Jellyfish). -* [Install Docker 24.0](#ece-install-docker-ubuntu-cloud) -* [Set up XFS quotas](#ece-xfs-setup-ubuntu-cloud) -* [Update the configurations settings](#ece-update-config-ubuntu-cloud) -* [Configure the Docker daemon options](#ece-configure-docker-daemon-ubuntu-cloud) +* [Install Docker 24.0](#ece-install-docker-ubuntu) +* [Set up XFS quotas](#ece-xfs-setup-ubuntu) +* [Update the configurations settings](#ece-update-config-ubuntu) +* [Configure the Docker daemon options](#ece-configure-docker-daemon-ubuntu) -## Install Docker [ece-install-docker-ubuntu-cloud] +## Install Docker [ece-install-docker-ubuntu] Install Docker LTS version 24.0 for Ubuntu 20.04 or 22.04. @@ -56,7 +61,7 @@ Docker 25 and higher are not compatible with ECE 3.7. -## Set up XFS quotas [ece-xfs-setup-ubuntu-cloud] +## Set up XFS quotas [ece-xfs-setup-ubuntu] XFS is required to support disk space quotas for Elasticsearch data directories. Some Linux distributions such as RHEL and Rocky Linux already provide XFS as the default file system. On Ubuntu, you need to set up an XFS file system and have quotas enabled. @@ -101,7 +106,7 @@ You must use XFS and have quotas enabled on all allocators, otherwise disk usage -## Update the configurations settings [ece-update-config-ubuntu-cloud] +## Update the configurations settings [ece-update-config-ubuntu] 1. Stop the Docker service: @@ -197,7 +202,7 @@ You must use XFS and have quotas enabled on all allocators, otherwise disk usage sudo install -o $USER -g $USER -d -m 700 /mnt/data ``` -7. If you [set up a new device with XFS](#ece-xfs-setup-ubuntu-cloud) earlier: +7. If you [set up a new device with XFS](#ece-xfs-setup-ubuntu) earlier: 1. Mount the block device (change the device name if you use a different device than `/dev/xvdg1`): @@ -219,7 +224,7 @@ You must use XFS and have quotas enabled on all allocators, otherwise disk usage -## Configure the Docker daemon options [ece-configure-docker-daemon-ubuntu-cloud] +## Configure the Docker daemon options [ece-configure-docker-daemon-ubuntu] ::::{tip} Docker creates a bridge IP address that can conflict with IP addresses on your internal network. To avoid an IP address conflict, change the `--bip=172.17.42.1/16` parameter in our examples to something that you know will work. If there is no conflict, you can omit the `--bip` parameter. The `--bip` parameter is internal to the host and can be set to the same IP for each host in the cluster. More information on Docker daemon options can be found in the [dockerd command line reference](https://docs.docker.com/engine/reference/commandline/dockerd/). diff --git a/deploy-manage/deploy/cloud-enterprise/configure-operating-system-cloud.md b/deploy-manage/deploy/cloud-enterprise/configure-operating-system-cloud.md deleted file mode 100644 index 17ae9b46c..000000000 --- a/deploy-manage/deploy/cloud-enterprise/configure-operating-system-cloud.md +++ /dev/null @@ -1,17 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configure-os-cloud.html ---- - -# Configure your operating system cloud [ece-configure-os-cloud] - -Before installing Elastic Cloud Enterprise, you have to prepare your hosts with one of the following Linux distributions: - -* [Ubuntu 20.04 LTS (Focal Fossa) and Ubuntu 22.04 LTS (Jammy Jellyfish)](configure-host-ubuntu-cloud.md) -* [Red Hat Enterprise Linux (RHEL) 8 and 9](configure-host-rhel-cloud.md) -* [Rocky Linux 8 and 9](configure-host-rhel-cloud.md) -* [SUSE Linux Enterprise Server (SLES) 12 SP5 and 15](configure-host-suse-cloud.md) - - - - diff --git a/deploy-manage/deploy/cloud-enterprise/configure-operating-system-onprem.md b/deploy-manage/deploy/cloud-enterprise/configure-operating-system.md similarity index 51% rename from deploy-manage/deploy/cloud-enterprise/configure-operating-system-onprem.md rename to deploy-manage/deploy/cloud-enterprise/configure-operating-system.md index 599d6e225..6cf39a8f6 100644 --- a/deploy-manage/deploy/cloud-enterprise/configure-operating-system-onprem.md +++ b/deploy-manage/deploy/cloud-enterprise/configure-operating-system.md @@ -1,17 +1,17 @@ --- +applies_to: + deployment: + ece: all mapped_pages: + - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configure-os-cloud.html - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configure-os-onprem.html --- -# Configure your operating system onprem [ece-configure-os-onprem] +# Configure your operating system [ece-configure-os] Before installing Elastic Cloud Enterprise, you have to prepare your hosts with one of the following Linux distributions: -* [Ubuntu 20.04 LTS (Focal Fossa) and Ubuntu 22.04 LTS (Jammy Jellyfish)](configure-host-ubuntu-onprem.md) -* [Red Hat Enterprise Linux (RHEL) 8 and 9](configure-host-rhel-onprem.md) -* [Rocky Linux 8 and 9](configure-host-rhel-onprem.md) -* [SUSE Linux Enterprise Server (SLES) 12 SP5 and 15](configure-host-suse-onprem.md) - - - - +* [Ubuntu 20.04 LTS (Focal Fossa) and Ubuntu 22.04 LTS (Jammy Jellyfish)](configure-host-ubuntu.md) +* [Red Hat Enterprise Linux (RHEL) 8 and 9](configure-host-rhel.md) +* [Rocky Linux 8 and 9](configure-host-rhel.md) +* [SUSE Linux Enterprise Server (SLES) 12 SP5 and 15](configure-host-suse.md) diff --git a/deploy-manage/deploy/cloud-enterprise/configure.md b/deploy-manage/deploy/cloud-enterprise/configure.md index 7a71714ea..cc569297a 100644 --- a/deploy-manage/deploy/cloud-enterprise/configure.md +++ b/deploy-manage/deploy/cloud-enterprise/configure.md @@ -9,30 +9,38 @@ mapped_pages: # Configure ECE [ece-configuring-ece] -⚠️ **This page is a work in progress.** ⚠️ +Now that you have {{ece}} up and running, take a look at some of the additional features that you can configure: -Now that you have Elastic Cloud Enterprise up and running, take a look at some of the additional features that you can configure: +## Common ECE tasks -* [System deployment configuration](system-deployments-configuration.md) - Best practices for ECE system deployments to ensure a highly available and resilient setup. -* [Configure deployment templates](configure-deployment-templates.md) - Make the most out of deployment templates by configuring ECE for your hardware and creating custom deployment templates. -* [Manage snapshot repositories](../../tools/snapshot-and-restore/cloud-enterprise.md) - To back up your Elasticsearch clusters automatically, you need to configure a snapshot repository. +* [Assign roles to hosts](../../../deploy-manage/deploy/cloud-enterprise/assign-roles-to-hosts.md) - Make sure new hosts can be used for their intended purpose after you install ECE on them. +* [System deployments configuration](system-deployments-configuration.md) - Best practices for ECE system deployments to ensure a highly available and resilient setup. +* [Configure deployment templates](configure-deployment-templates.md) – Define the resources, topology, hardware, and configurations that will be applied to your deployments. +* [Manage Elastic Stack versions](./manage-elastic-stack-versions.md) - Upload or remove Elastic Stack packs. +* [Change the ECE API URL](./change-ece-api-url.md) - Configure the HTTPS URL used to access the ECE API. +* [Change endpoint URLs](change-endpoint-urls.md) - Configure the URLs to access {{es}} and {{kib}} deployments to match your [domain name](./ece-wildcard-dns.md) and [proxy certificate](../../security/secure-your-elastic-cloud-enterprise-installation/manage-security-certificates.md). +* [Enable custom endpoint aliases](./enable-custom-endpoint-aliases.md) - This feature allows to use aliases in the endpoint URLs instead of cluster UUIDs. + +Other sections of the documentation describe important ECE features to consider: + +* [Configure ECE users and roles](../../users-roles/cloud-enterprise-orchestrator.md) - Manage authentication and authorization at ECE platform level. +* [Manage security certificates](../../security/secure-your-elastic-cloud-enterprise-installation/manage-security-certificates.md) - Configure Cloud UI and Proxy TLS/SSL certificates. * [Manage licenses](../../license/manage-your-license-in-ece.md) - Keep Elastic Cloud Enterprise current with a valid license. -* [Change endpoint URLs](change-endpoint-urls.md) - Set where Elasticsearch and Kibana can be accessed from. +* [Manage snapshot repositories](../../tools/snapshot-and-restore/cloud-enterprise.md) - To back up your Elasticsearch clusters automatically, you need to configure a snapshot repository. + +## Advanced configuration procedures + * [Configure allocator affinity](configure-allocator-affinity.md) - Determine how ECE distributes your Elastic Stack deployments across allocators. * [Change allocator disconnect timeout](change-allocator-disconnect-timeout.md) - Configure how long ECE waits before considering allocators to be disconnected. +* [Migrate ECE to Podman hosts](./migrate-ece-to-podman-hosts.md) - If you are running a Docker based installation and you need to migrate to Podman. * [Migrate ECE on Podman hosts to SELinux in enforcing mode](migrate-ece-on-podman-hosts-to-selinux-enforce.md) - Migrate ECE to SELinux in `enforcing` mode using Podman. -## Administering your installation [ece-administering-ece] - -Now that you have Elastic Cloud Enterprise up and running, take a look at the things you can do to keep your installation humming along, from adding more capacity to dealing with hosts that require maintenance or have failed. They are all presented in the [](../../maintenance.md) section. - -* [Scale Out Your Installation](../../../deploy-manage/maintenance/ece/scale-out-installation.md) - Need to add more capacity? Here’s how. -* [Assign Roles to Hosts](../../../deploy-manage/deploy/cloud-enterprise/assign-roles-to-hosts.md) - Make sure new hosts can be used for their intended purpose after you install ECE on them. -* [Enable Maintenance Mode](../../../deploy-manage/maintenance/ece/enable-maintenance-mode.md) - Perform administrative actions on allocators safely by putting them into maintenance mode first. -* [Move Nodes From Allocators](../../../deploy-manage/maintenance/ece/move-nodes-instances-from-allocators.md) - Moves all Elasticsearch clusters and Kibana instances to another allocator, so that the allocator is no longer used for handling user requests. -* [Delete Hosts](../../../deploy-manage/maintenance/ece/delete-ece-hosts.md) - Remove a host from your ECE installation, either because it is no longer needed or because it is faulty. -* [Perform Host Maintenance](../../../deploy-manage/maintenance/ece/perform-ece-hosts-maintenance.md) - Apply operating system patches and other maintenance to hosts safely without removing them from your ECE installation. -* [Manage Elastic Stack Versions](../../../deploy-manage/deploy/cloud-enterprise/manage-elastic-stack-versions.md) - View, add, or update versions of the Elastic Stack that are available on your ECE installation. -* [Upgrade Your Installation](../../../deploy-manage/upgrade/orchestrator/upgrade-cloud-enterprise.md) - A new version of Elastic Cloud Enterprise is available and you want to upgrade. Here’s how. +## Maintenance activities +Refer to [ECE maintenance](../../maintenance/ece.md) for important maintenance activities, including adding capacity, applying OS patches, and addressing host failures. +* [Scale out your installation](../../../deploy-manage/maintenance/ece/scale-out-installation.md) - Need to add more capacity? Here’s how. +* [Enable maintenance mode](../../../deploy-manage/maintenance/ece/enable-maintenance-mode.md) - Perform administrative actions on allocators safely by putting them into maintenance mode first. +* [Move nodes from allocators](../../../deploy-manage/maintenance/ece/move-nodes-instances-from-allocators.md) - Moves all Elasticsearch clusters and Kibana instances to another allocator, so that the allocator is no longer used for handling user requests. +* [Perform host maintenance](../../../deploy-manage/maintenance/ece/perform-ece-hosts-maintenance.md) - Apply operating system patches and other maintenance to hosts safely without removing them from your ECE installation. +* [Delete hosts](../../../deploy-manage/maintenance/ece/delete-ece-hosts.md) - Remove a host from your ECE installation, either because it is no longer needed or because it is faulty. diff --git a/deploy-manage/deploy/cloud-enterprise/default-system-deployment-versions.md b/deploy-manage/deploy/cloud-enterprise/default-system-deployment-versions.md index a2df2dd8f..9ac6e41fd 100644 --- a/deploy-manage/deploy/cloud-enterprise/default-system-deployment-versions.md +++ b/deploy-manage/deploy/cloud-enterprise/default-system-deployment-versions.md @@ -1,4 +1,7 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-system-deployment-versions.html --- diff --git a/deploy-manage/deploy/cloud-enterprise/deploy-an-orchestrator.md b/deploy-manage/deploy/cloud-enterprise/deploy-an-orchestrator.md index a23a4873d..a9a70ba71 100644 --- a/deploy-manage/deploy/cloud-enterprise/deploy-an-orchestrator.md +++ b/deploy-manage/deploy/cloud-enterprise/deploy-an-orchestrator.md @@ -17,15 +17,17 @@ This section provides step-by-step guidance on: * [Prepare the environment](./prepare-environment.md): Follow the hardware, software, and networking prerequisites before the installation. -* [Install ECE](./install.md): Identify the deployment scenario that best fits your needs, choose an installation method, and complete the setup. - * [Install ECE on a public cloud](./install-ece-on-public-cloud.md) - * [Install ECE on your own premises](./install-ece-on-own-premises.md) +* [Install ECE orchestrator](./install.md): Identify the deployment scenario that best fits your needs, choose an installation method, and complete the setup. + * [](./configure-operating-system.md) + * [](./install-ece-procedures.md) * [Alternative: install ECE with Ansible](./alternative-install-ece-with-ansible.md) * [Air-gapped installations](./air-gapped-install.md): Review the different options for air-gapped environments. * [With your private Docker registry](./ece-install-offline-with-registry.md) * [Without any Docker registry](./ece-install-offline-no-registry.md) +* [](./post-installation-steps.md): Get ready for production by adding SSL certificates, configuring domain names, and completing other essential tasks. + * [Configure ECE](./configure.md): Explore the most common tasks to configure your ECE platform. * [System deployments configuration](./system-deployments-configuration.md) * [Configure deployment templates](./deployment-templates.md) @@ -36,8 +38,8 @@ This section provides step-by-step guidance on: After deploying the ECE platform, you may need to configure custom proxy certificates, manage snapshot repositories, or perform maintenance operations, among other tasks. Refer to the following sections for more details: -* [Secure your ECE installation](../../security/secure-your-elastic-cloud-enterprise-installation.md) -*[](/deploy-manage/security/secure-your-cluster-deployment.md) +* [Security considerations](../../security/secure-your-elastic-cloud-enterprise-installation.md) +* [Secure your deployments](/deploy-manage/security/secure-your-cluster-deployment.md) * [Users and roles](../../users-roles/cloud-enterprise-orchestrator.md) * [Manage snapshot repositories](../../tools/snapshot-and-restore.md) * [Manage licenses](../../license/manage-your-license-in-ece.md) diff --git a/deploy-manage/deploy/cloud-enterprise/deploy-large-installation-onprem.md b/deploy-manage/deploy/cloud-enterprise/deploy-large-installation-onprem.md deleted file mode 100644 index 4de56154a..000000000 --- a/deploy-manage/deploy/cloud-enterprise/deploy-large-installation-onprem.md +++ /dev/null @@ -1,93 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-install-large-onprem.html ---- - -# Deploy a large installation onprem [ece-install-large-onprem] - -This type of installation is recommended for deployments with significant overall search and indexing throughput. You need: - -* 3 hosts with at least 64 GB RAM each for directors and coordinators (ECE management services) -* 3 hosts for allocators, each with one of the following RAM configurations: - - * 1 x 256 GB RAM - * 2 x 128 GB RAM - * 4 x 64 GB RAM - -* 3 hosts with 16 GB RAM each for proxies -* 3 availability zones - -:::{image} ../../../images/cloud-enterprise-ece-pb-9.png -:alt: A large installation with nine to twelve hosts across three availability zones -::: - - -## Before you start [ece_before_you_start_6] - -Note that the large-sized Elastic Cloud Enterprise installation separates the allocator and proxy roles from the director and coordinator roles (ECE management services). - -**Check the recommended JVM Heap sizes** - -| Service | JVM Heap Size (Xms and Xmx) | -| --- | --- | -| `runner` | 1 GB | -| `allocator` | 4 GB | -| `zookeeper` | 24 GB | -| `director` | 1 GB | -| `constructor` | 4 GB | -| `admin-console` | 24 GB | - -::::{warning} -For production environments, you must define the memory settings for each role, except for the `proxy` role, as starting from ECE 2.4 the JVM proxy was replaced with a Golang-based proxy. If you don’t set any memory setting, the default values are used, which are inadequate for production environments and can lead to performance or stability issues. -:::: - - - -## Installation steps [ece_installation_steps_6] - -1. Install Elastic Cloud Enterprise on the first host to start a new installation with your first availability zone. This first host holds all roles to help bootstrap the rest of the installation, but you will remove some of its roles in a later step. - - ```sh - bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install --availability-zone MY_ZONE-1 --memory-settings '{"runner":{"xms":"1G","xmx":"1G"},"allocator":{"xms":"4G","xmx":"4G"},"zookeeper":{"xms":"24G","xmx":"24G"},"director":{"xms":"1G","xmx":"1G"},"constructor":{"xms":"4G","xmx":"4G"},"admin-console":{"xms":"24G","xmx":"24G"}}' - ``` - - After the installation completes, copy down the coordinator host IP address, user credentials, and roles token information. Keep this information safe. - -2. Generate a new roles token that persists for one hour on the first host, so that other hosts can join your installation with the right role permissions in subsequent steps (referred to as `MY_TOKEN`). The new token needs to enable the director, coordinator, and proxy roles. - - ```sh - curl -k -H 'Content-Type: application/json' -u admin:PASSWORD https://localhost:12443/api/v1/platform/configuration/security/enrollment-tokens -d '{ "persistent": false, "roles": ["director", "coordinator", "proxy"] }' - ``` - -3. Install Elastic Cloud Enterprise on a second and third host, placing them into a second and a third availability zone, and assign them the `director` and `coordinator` roles. Do not assign the `allocator` or the `proxy` role, as these hosts should not handle or route any user requests. Make sure you include the coordinator host IP information from step 1 and the new roles token from step 2. - - ```sh - bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install --coordinator-host HOST_IP --roles-token 'MY_TOKEN' --roles "director,coordinator" --availability-zone MY_ZONE-2 --memory-settings '{"runner":{"xms":"1G","xmx":"1G"},"zookeeper":{"xms":"24G","xmx":"24G"},"director":{"xms":"1G","xmx":"1G"},"constructor":{"xms":"4G","xmx":"4G"},"admin-console":{"xms":"24G","xmx":"24G"}}' - ``` - - ```sh - bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install --coordinator-host HOST_IP --roles-token 'MY_TOKEN' --roles "director,coordinator" --availability-zone MY_ZONE-3 --memory-settings '{"runner":{"xms":"1G","xmx":"1G"},"zookeeper":{"xms":"24G","xmx":"24G"},"director":{"xms":"1G","xmx":"1G"},"constructor":{"xms":"4G","xmx":"4G"},"admin-console":{"xms":"24G","xmx":"24G"}}' - ``` - -4. To handle the Elasticsearch and Kibana workload, install Elastic Cloud Enterprise on three or more hosts, distributing them evenly across the existing three availability zones, or on however many hosts you think you need initially, and assign them the `allocator` role. Make sure you include the coordinator host IP information and allocator roles token from step 1. - - ```sh - bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install --coordinator-host HOST_IP --roles-token 'ALLOCATOR_TOKEN' --roles "allocator" --availability-zone MY_ZONE-1 --memory-settings '{"runner":{"xms":"1G","xmx":"1G"},"allocator":{"xms":"4G","xmx":"4G"}}' - - bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install --coordinator-host HOST_IP --roles-token 'ALLOCATOR_TOKEN' --roles "allocator" --availability-zone MY_ZONE-2 --memory-settings '{"runner":{"xms":"1G","xmx":"1G"},"allocator":{"xms":"4G","xmx":"4G"}}' - - bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install --coordinator-host HOST_IP --roles-token 'ALLOCATOR_TOKEN' --roles "allocator" --availability-zone MY_ZONE-3 --memory-settings '{"runner":{"xms":"1G","xmx":"1G"},"allocator":{"xms":"4G","xmx":"4G"}}' - ``` - -5. To handle the routing of user requests to Elasticsearch, install Elastic Cloud Enterprise on a three additional hosts, distributing them evenly across the existing three availability zones, and assign them the `proxy` role. Do not assign any other roles, as these hosts should only route user requests. Make sure you include the coordinator host IP information from step 1 and the new roles token from step 2. - - ```sh - bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install --coordinator-host HOST_IP --roles-token 'MY_TOKEN' --roles "proxy" --availability-zone MY_ZONE-1 --memory-settings - '{"runner":{"xms":"1G","xmx":"1G"}}' - - bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install --coordinator-host HOST_IP --roles-token 'MY_TOKEN' --roles "proxy" --availability-zone MY_ZONE-2 --memory-settings - '{"runner":{"xms":"1G","xmx":"1G"}}' - - bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install --coordinator-host HOST_IP --roles-token 'MY_TOKEN' --roles "proxy" --availability-zone MY_ZONE-3 --memory-settings - '{"runner":{"xms":"1G","xmx":"1G"}}' - ``` diff --git a/deploy-manage/deploy/cloud-enterprise/deploy-large-installation-cloud.md b/deploy-manage/deploy/cloud-enterprise/deploy-large-installation.md similarity index 84% rename from deploy-manage/deploy/cloud-enterprise/deploy-large-installation-cloud.md rename to deploy-manage/deploy/cloud-enterprise/deploy-large-installation.md index 20e7e75a6..fe2e56c51 100644 --- a/deploy-manage/deploy/cloud-enterprise/deploy-large-installation-cloud.md +++ b/deploy-manage/deploy/cloud-enterprise/deploy-large-installation.md @@ -1,9 +1,13 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-install-large-cloud.html + - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-install-large-onprem.html --- -# Deploy a large installation cloud [ece-install-large-cloud] +# Deploy a large installation [ece-install-large] This type of installation is recommended for deployments with significant overall search and indexing throughput. You need: @@ -21,8 +25,7 @@ This type of installation is recommended for deployments with significant overal :alt: A large installation with nine to twelve hosts across three availability zones ::: - -## Before you start [ece_before_you_start_3] +## Important considerations [ece_before_you_start_3] Note that the large-sized Elastic Cloud Enterprise installation separates the allocator and proxy roles from the director and coordinator roles (ECE management services). @@ -41,7 +44,9 @@ Note that the large-sized Elastic Cloud Enterprise installation separates the al For production environments, you must define the memory settings for each role, except for the `proxy` role, as starting from ECE 2.4 the JVM proxy was replaced with a Golang-based proxy. If you don’t set any memory setting, the default values are used, which are inadequate for production environments and can lead to performance or stability issues. :::: +## Before you start +Make sure you have completed all prerequisites and environment preparations described in the [Installation overview](./install.md), and that the hosts are configured according to [](./configure-operating-system.md). ## Installation steps [ece_installation_steps_3] @@ -82,12 +87,19 @@ For production environments, you must define the memory settings for each role, 5. To handle the routing of user requests to Elasticsearch, install Elastic Cloud Enterprise on a three additional hosts, distributing them evenly across the existing three availability zones, and assign them the `proxy` role. Do not assign any other roles, as these hosts should only route user requests. Make sure you include the coordinator host IP information from step 1 and the new roles token from step 2. ```sh - bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install --coordinator-host HOST_IP --roles-token 'MY_TOKEN' --roles "proxy" --availability-zone MY_ZONE-1 --memory-settings - '{"runner":{"xms":"1G","xmx":"1G"}}' + bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install --coordinator-host HOST_IP --roles-token 'MY_TOKEN' --roles "proxy" --availability-zone MY_ZONE-1 --memory-settings '{"runner":{"xms":"1G","xmx":"1G"}}' + ``` - bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install --coordinator-host HOST_IP --roles-token 'MY_TOKEN' --roles "proxy" --availability-zone MY_ZONE-2 --memory-settings - '{"runner":{"xms":"1G","xmx":"1G"}}' + ```sh + bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install --coordinator-host HOST_IP --roles-token 'MY_TOKEN' --roles "proxy" --availability-zone MY_ZONE-2 --memory-settings '{"runner":{"xms":"1G","xmx":"1G"}}' + ``` - bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install --coordinator-host HOST_IP --roles-token 'MY_TOKEN' --roles "proxy" --availability-zone MY_ZONE-3 --memory-settings - '{"runner":{"xms":"1G","xmx":"1G"}}' + ```sh + bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install --coordinator-host HOST_IP --roles-token 'MY_TOKEN' --roles "proxy" --availability-zone MY_ZONE-3 --memory-settings '{"runner":{"xms":"1G","xmx":"1G"}}' ``` + +6. [Change the deployment configuration](working-with-deployments.md) for the `admin-console-elasticsearch`, `logging-and-metrics`, and `security` clusters to use three availability zones and resize the nodes to use at least 4 GB of RAM. This change makes sure that the clusters used by the administration console are highly available and provisioned sufficiently. + +7. [Log into the Cloud UI](log-into-cloud-ui.md) to provision your deployment. + +Once the installation is complete, you can continue with [](./post-installation-steps.md). \ No newline at end of file diff --git a/deploy-manage/deploy/cloud-enterprise/deploy-medium-installation-onprem.md b/deploy-manage/deploy/cloud-enterprise/deploy-medium-installation-onprem.md deleted file mode 100644 index 93997d08c..000000000 --- a/deploy-manage/deploy/cloud-enterprise/deploy-medium-installation-onprem.md +++ /dev/null @@ -1,75 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-install-medium-onprem.html ---- - -# Deploy a medium installation onprem [ece-install-medium-onprem] - -This type of installation is recommended for many production setups. You need: - -* 3 hosts with at least 32 GB RAM each for directors and coordinators (ECE management services), and proxies -* 3 hosts with 256 GB RAM each for allocators -* 3 availability zones - -:::{image} ../../../images/cloud-enterprise-ece-pb-6.png -:alt: A medium installation with nine to twelve hosts across three availability zones -::: - - -## Before you start [ece_before_you_start_5] - -* Monitor the load on proxies and make sure the volume of user requests routed by the proxies does not affect the resources available to the ECE management services. -* Note that the medium-sized Elastic Cloud Enterprise installation separates the allocator from the director and coordinator roles (ECE management services) and the proxy roles. - -**Check the recommended JVM Heap sizes** - -| Service | JVM Heap Size (Xms and Xmx) | -| --- | --- | -| `runner` | 1 GB | -| `allocator` | 4 GB | -| `zookeeper` | 8 GB | -| `director` | 1 GB | -| `constructor` | 4 GB | -| `admin-console` | 8 GB | - -::::{warning} -For production environments, you must define the memory settings for each role, except for the `proxy` role, as starting from ECE 2.4 the JVM proxy was replaced with a Golang-based proxy. If you don’t set any memory setting, the default values are used, which are inadequate for production environments and can lead to performance or stability issues. -:::: - - - -## Installation steps [ece_installation_steps_5] - -1. Install Elastic Cloud Enterprise on the first host to start a new installation with your first availability zone. This first host holds all roles to help bootstrap the rest of the installation, but you will remove some of its roles in a later step. - - ```sh - bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install --availability-zone MY_ZONE-1 --memory-settings '{"runner":{"xms":"1G","xmx":"1G"},"allocator":{"xms":"4G","xmx":"4G"},"zookeeper":{"xms":"8G","xmx":"8G"},"director":{"xms":"1G","xmx":"1G"},"constructor":{"xms":"4G","xmx":"4G"},"admin-console":{"xms":"8G","xmx":"8G"}}' - ``` - - After the installation completes, copy down the coordinator host IP address, user credentials, and roles token information. Keep this information safe. - -2. Generate a new roles token that persists for one hour on the first host, so that other hosts can join your installation with the right role permissions in the next step (referred to as `MY_TOKEN`). The new token needs to enable the director, coordinator and proxy roles. - - ```sh - curl -k -H 'Content-Type: application/json' -u admin:PASSWORD https://localhost:12443/api/v1/platform/configuration/security/enrollment-tokens -d '{ "persistent": false, "roles": ["director", "coordinator", "proxy"] }' - ``` - -3. Install Elastic Cloud Enterprise on a second and third host, placing them into a second and a third availability zone, and assign them the `director`, `coordinator`, and `proxy` roles. Do not assign the `allocator` role, as these hosts should not handle any user requests. Make sure you include the coordinator host IP information from step 1 and the new roles token from step 2. - - ```sh - bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install --coordinator-host HOST_IP --roles-token 'MY_TOKEN' --roles "director,coordinator,proxy" --availability-zone MY_ZONE-2 --memory-settings '{"runner":{"xms":"1G","xmx":"1G"},"zookeeper":{"xms":"8G","xmx":"8G"},"director":{"xms":"1G","xmx":"1G"},"constructor":{"xms":"4G","xmx":"4G"},"admin-console":{"xms":"8G","xmx":"8G"}}' - ``` - - ```sh - bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install --coordinator-host HOST_IP --roles-token 'MY_TOKEN' --roles "director,coordinator,proxy" --availability-zone MY_ZONE-3 --memory-settings '{"runner":{"xms":"1G","xmx":"1G"},"zookeeper":{"xms":"8G","xmx":"8G"},"director":{"xms":"1G","xmx":"1G"},"constructor":{"xms":"4G","xmx":"4G"},"admin-console":{"xms":"8G","xmx":"8G"}}' - ``` - -4. To handle the Elasticsearch and Kibana workload, install Elastic Cloud Enterprise on a fourth, fifth, and sixth host, distributing them evenly across the existing three availability zones and assign them the `allocator` role. Make sure you include the coordinator host IP information and allocator roles token from step 1. - - ```sh - bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install --coordinator-host HOST_IP --roles-token 'ALLOCATOR_TOKEN' --roles "allocator" --availability-zone MY_ZONE-1 --memory-settings '{"runner":{"xms":"1G","xmx":"1G"},"allocator":{"xms":"4G","xmx":"4G"}}' - - bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install --coordinator-host HOST_IP --roles-token 'ALLOCATOR_TOKEN' --roles "allocator" --availability-zone MY_ZONE-2 --memory-settings '{"runner":{"xms":"1G","xmx":"1G"},"allocator":{"xms":"4G","xmx":"4G"}}' - - bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install --coordinator-host HOST_IP --roles-token 'ALLOCATOR_TOKEN' --roles "allocator" --availability-zone MY_ZONE-3 --memory-settings '{"runner":{"xms":"1G","xmx":"1G"},"allocator":{"xms":"4G","xmx":"4G"}}' - ``` diff --git a/deploy-manage/deploy/cloud-enterprise/deploy-medium-installation-cloud.md b/deploy-manage/deploy/cloud-enterprise/deploy-medium-installation.md similarity index 78% rename from deploy-manage/deploy/cloud-enterprise/deploy-medium-installation-cloud.md rename to deploy-manage/deploy/cloud-enterprise/deploy-medium-installation.md index 990fd96f7..5e5d630d1 100644 --- a/deploy-manage/deploy/cloud-enterprise/deploy-medium-installation-cloud.md +++ b/deploy-manage/deploy/cloud-enterprise/deploy-medium-installation.md @@ -1,9 +1,13 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-install-medium-cloud.html + - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-install-medium-onprem.html --- -# Deploy a medium installation cloud [ece-install-medium-cloud] +# Deploy a medium installation [ece-install-medium] This type of installation is recommended for many production setups. You need: @@ -15,8 +19,7 @@ This type of installation is recommended for many production setups. You need: :alt: A medium installation with nine to twelve hosts across three availability zones ::: - -## Before you start [ece_before_you_start_2] +## Important considerations [ece_before_you_start_2] * Monitor the load on proxies and make sure the volume of user requests routed by the proxies does not affect the resources available to the ECE management services. * Note that the medium-sized Elastic Cloud Enterprise installation separates the allocator from the director and coordinator roles (ECE management services) and the proxy roles. @@ -36,7 +39,9 @@ This type of installation is recommended for many production setups. You need: For production environments, you must define the memory settings for each role, except for the `proxy` role, as starting from ECE 2.4 the JVM proxy was replaced with a Golang-based proxy. If you don’t set any memory setting, the default values are used, which are inadequate for production environments and can lead to performance or stability issues. :::: +## Before you start +Make sure you have completed all prerequisites and environment preparations described in the [Installation overview](./install.md), and that the hosts are configured according to [](./configure-operating-system.md). ## Installation steps [ece_installation_steps_2] @@ -64,12 +69,22 @@ For production environments, you must define the memory settings for each role, bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install --coordinator-host HOST_IP --roles-token 'MY_TOKEN' --roles "director,coordinator,proxy" --availability-zone MY_ZONE-3 --memory-settings '{"runner":{"xms":"1G","xmx":"1G"},"zookeeper":{"xms":"8G","xmx":"8G"},"director":{"xms":"1G","xmx":"1G"},"constructor":{"xms":"4G","xmx":"4G"},"admin-console":{"xms":"8G","xmx":"8G"}}' ``` -4. To handle the Elasticsearch and Kibana workload, install Elastic Cloud Enterprise on a fourth, fifth, and sixth host, distributing them evenly across the existing three availability zones and assign them the `allocator` role. Make sure you include the coordinator host IP information and allocator roles token from step 1. +4. To handle the Elasticsearch and Kibana workloads, install Elastic Cloud Enterprise on a fourth, fifth, and sixth host, distributing them evenly across the existing three availability zones and assign them the `allocator` role. Make sure you include the coordinator host IP information and allocator roles token from step 1. ```sh bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install --coordinator-host HOST_IP --roles-token 'ALLOCATOR_TOKEN' --roles "allocator" --availability-zone MY_ZONE-1 --memory-settings '{"runner":{"xms":"1G","xmx":"1G"},"allocator":{"xms":"4G","xmx":"4G"}}' + ``` + ```sh bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install --coordinator-host HOST_IP --roles-token 'ALLOCATOR_TOKEN' --roles "allocator" --availability-zone MY_ZONE-2 --memory-settings '{"runner":{"xms":"1G","xmx":"1G"},"allocator":{"xms":"4G","xmx":"4G"}}' - + ``` + + ```sh bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install --coordinator-host HOST_IP --roles-token 'ALLOCATOR_TOKEN' --roles "allocator" --availability-zone MY_ZONE-3 --memory-settings '{"runner":{"xms":"1G","xmx":"1G"},"allocator":{"xms":"4G","xmx":"4G"}}' ``` + +5. [Change the deployment configuration](working-with-deployments.md) for the `admin-console-elasticsearch`, `logging-and-metrics`, and `security` clusters to use three availability zones and resize the nodes to use at least 4 GB of RAM. This change makes sure that the clusters used by the administration console are highly available and provisioned sufficiently. + +6. [Log into the Cloud UI](log-into-cloud-ui.md) to provision your deployment. + +Once the installation is complete, you can continue with [](./post-installation-steps.md). \ No newline at end of file diff --git a/deploy-manage/deploy/cloud-enterprise/deploy-small-installation-onprem.md b/deploy-manage/deploy/cloud-enterprise/deploy-small-installation-onprem.md deleted file mode 100644 index 2c511663a..000000000 --- a/deploy-manage/deploy/cloud-enterprise/deploy-small-installation-onprem.md +++ /dev/null @@ -1,70 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-install-small-onprem.html ---- - -# Deploy a small installation onprem [ece-install-small-onprem] - -The type of installation is recommended for development, test, and small-scale use cases. You need: - -* 3 hosts with 128 GB RAM -* 3 availability zones - -:::{image} ../../../images/cloud-enterprise-ece-pb-3.png -:alt: A small baseline installation with three hosts across three availability zones -::: - - -## Before you start [ece_before_you_start_4] - -* This type of installation is **not recommended for high-traffic workloads**. -* You must not use **spinning disks** with small ECE installations, as these are not supported when you run allocators and ECE management services on the same server. -* Note that the small-size ECE installation keeps the directors and coordinators roles (ECE management services) on the same hosts as your allocators and proxies. - -**Check the recommended JVM Heap sizes** - -| Service | JVM Heap Size (Xms and Xmx) | -| --- | --- | -| `runner` | 1 GB | -| `allocator` | 4 GB | -| `zookeeper` | 4 GB | -| `director` | 1 GB | -| `constructor` | 4 GB | -| `admin-console` | 4 GB | - -::::{warning} -For production environments, you must define the memory settings for each role, except for the `proxy` role, as starting from ECE 2.4 the JVM proxy was replaced with a Golang-based proxy. If you don’t set any memory setting, the default values are used, which are inadequate for production environments and can lead to performance or stability issues. -:::: - - - -## Installation steps [ece_installation_steps_4] - -1. Install Elastic Cloud Enterprise on the first host to start a new installation with your first availability zone. This first host holds all roles to help bootstrap the rest of the installation. - - ```sh - bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install --availability-zone MY_ZONE-1 --memory-settings '{"runner":{"xms":"1G","xmx":"1G"},"allocator":{"xms":"4G","xmx":"4G"},"zookeeper":{"xms":"4G","xmx":"4G"},"director":{"xms":"1G","xmx":"1G"},"constructor":{"xms":"4G","xmx":"4G"},"admin-console":{"xms":"4G","xmx":"4G"}}' - ``` - - After the installation completes, copy down the coordinator host IP address, user credentials, and roles token information. Keep this information safe. - -2. Generate a new roles token that persists for one hour on the first host, so that other hosts can join your installation with the right role permissions in the next step (referred to as `MY_TOKEN`). The new token needs to enable all host roles, which none of the tokens automatically generated by the installation on the first host provide. - - ```sh - curl -k -H 'Content-Type: application/json' -u admin:PASSWORD https://localhost:12443/api/v1/platform/configuration/security/enrollment-tokens -d '{ "persistent": false, "roles": ["director", "coordinator", "proxy", "allocator"] }' - ``` - -3. Install Elastic Cloud Enterprise on a second and third host, placing them into a second and a third availability zone, and assign them the same roles and memory settings as the first host. Make sure you include the coordinator host IP information from step 1 and the new roles token from step 2. - - ```sh - bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install --coordinator-host HOST_IP --roles-token 'MY_TOKEN' --roles "director,coordinator,proxy,allocator" --availability-zone MY_ZONE-2 --memory-settings '{"runner":{"xms":"1G","xmx":"1G"},"allocator":{"xms":"4G","xmx":"4G"},"zookeeper":{"xms":"4G","xmx":"4G"},"director":{"xms":"1G","xmx":"1G"},"constructor":{"xms":"4G","xmx":"4G"},"admin-console":{"xms":"4G","xmx":"4G"}}' - ``` - - ```sh - bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install --coordinator-host HOST_IP --roles-token 'MY_TOKEN' --roles "director,coordinator,proxy,allocator" --availability-zone MY_ZONE-3 --memory-settings '{"runner":{"xms":"1G","xmx":"1G"},"allocator":{"xms":"4G","xmx":"4G"},"zookeeper":{"xms":"4G","xmx":"4G"},"director":{"xms":"1G","xmx":"1G"},"constructor":{"xms":"4G","xmx":"4G"},"admin-console":{"xms":"4G","xmx":"4G"}}' - ``` - -4. [Change the deployment configuration for the `admin-console-elasticsearch`, `logging-and-metrics`, and `security` clusters](working-with-deployments.md) to use three availability zones and resize the nodes to use at least 4 GB of RAM. This change makes sure that the clusters used by the administration console are highly available and provisioned sufficiently. -5. [Log into the Cloud UI](log-into-cloud-ui.md) to provision your deployment. - -If necessary, you can scale and deploy a [medium installation](deploy-medium-installation-cloud.md). diff --git a/deploy-manage/deploy/cloud-enterprise/deploy-small-installation-cloud.md b/deploy-manage/deploy/cloud-enterprise/deploy-small-installation.md similarity index 84% rename from deploy-manage/deploy/cloud-enterprise/deploy-small-installation-cloud.md rename to deploy-manage/deploy/cloud-enterprise/deploy-small-installation.md index 16541a1d4..9196d9245 100644 --- a/deploy-manage/deploy/cloud-enterprise/deploy-small-installation-cloud.md +++ b/deploy-manage/deploy/cloud-enterprise/deploy-small-installation.md @@ -1,9 +1,13 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-install-small-cloud.html + - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-install-small-onprem.html --- -# Deploy a small installation cloud [ece-install-small-cloud] +# Deploy a small installation [ece-install-small] The type of installation is recommended for development, test, and small-scale use cases. You need: @@ -14,8 +18,7 @@ The type of installation is recommended for development, test, and small-scale u :alt: A small baseline installation with three hosts across three availability zones ::: - -## Before you start [ece_before_you_start] +## Important considerations [ece_before_you_start] * This type of installation is **not recommended for high-traffic workloads**. * You must not use **spinning disks** with small ECE installations, as these are not supported when you run allocators and ECE management services on the same server. @@ -36,6 +39,9 @@ The type of installation is recommended for development, test, and small-scale u For production environments, you must define the memory settings for each role, except for the `proxy` role, as starting from ECE 2.4 the JVM proxy was replaced with a Golang-based proxy. If you don’t set any memory setting, the default values are used, which are inadequate for production environments and can lead to performance or stability issues. :::: +## Before you start + +Make sure you have completed all prerequisites and environment preparations described in the [Installation overview](./install.md), and that the hosts are configured according to [](./configure-operating-system.md). ## Installation steps [ece_installation_steps] @@ -64,7 +70,10 @@ For production environments, you must define the memory settings for each role, bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install --coordinator-host HOST_IP --roles-token 'MY_TOKEN' --roles "director,coordinator,proxy,allocator" --availability-zone MY_ZONE-3 --memory-settings '{"runner":{"xms":"1G","xmx":"1G"},"allocator":{"xms":"4G","xmx":"4G"},"zookeeper":{"xms":"4G","xmx":"4G"},"director":{"xms":"1G","xmx":"1G"},"constructor":{"xms":"4G","xmx":"4G"},"admin-console":{"xms":"4G","xmx":"4G"}}' ``` -4. [Change the deployment configuration for the `admin-console-elasticsearch`, `logging-and-metrics`, and `security` clusters](working-with-deployments.md) to use three availability zones and resize the nodes to use at least 4 GB of RAM. This change makes sure that the clusters used by the administration console are highly available and provisioned sufficiently. +4. [Change the deployment configuration](working-with-deployments.md) for the `admin-console-elasticsearch`, `logging-and-metrics`, and `security` clusters to use three availability zones and resize the nodes to use at least 4 GB of RAM. This change makes sure that the clusters used by the administration console are highly available and provisioned sufficiently. + 5. [Log into the Cloud UI](log-into-cloud-ui.md) to provision your deployment. -If necessary, you can scale and deploy a [medium installation](deploy-medium-installation-cloud.md). +If necessary, you can scale and deploy a [medium installation](deploy-medium-installation.md). + +Once the installation is complete, you can continue with [](./post-installation-steps.md). \ No newline at end of file diff --git a/deploy-manage/deploy/cloud-enterprise/ece-ce-add-support-for-integrations-server.md b/deploy-manage/deploy/cloud-enterprise/ece-ce-add-support-for-integrations-server.md index dbb9d75dc..0375fb3e2 100644 --- a/deploy-manage/deploy/cloud-enterprise/ece-ce-add-support-for-integrations-server.md +++ b/deploy-manage/deploy/cloud-enterprise/ece-ce-add-support-for-integrations-server.md @@ -1,4 +1,8 @@ --- +navigation_title: Integrations server support +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-ce-add-support-for-integrations-server.html --- @@ -42,7 +46,6 @@ To manually update your custom deployment templates to support Integrations Serv ] ``` - Send a `PUT` request with the updated template in the payload to replace the original template with the new one. Remember that: * The following request is just an example; other resources in the request payload should remain unchanged (they have been truncated in the example). diff --git a/deploy-manage/deploy/cloud-enterprise/ece-configure-templates-index-management.md b/deploy-manage/deploy/cloud-enterprise/ece-configure-templates-index-management.md index 0c0525241..a9495cbc1 100644 --- a/deploy-manage/deploy/cloud-enterprise/ece-configure-templates-index-management.md +++ b/deploy-manage/deploy/cloud-enterprise/ece-configure-templates-index-management.md @@ -1,9 +1,13 @@ --- +navigation_title: Configure index management +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configure-templates-index-management.html --- -# Configure index management for templates [ece-configure-templates-index-management] +# Configure index management for deployment templates [ece-configure-templates-index-management] If you create a deployment template that includes more than one data configuration, you must also specify how Elastic Cloud Enterprise should manage indices for your users when they create their deployments. For time-series use cases such as logging, metrics, and APM, providing a template that enables index management ensures that data is being stored in the most cost-effective way possible as it ages. diff --git a/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-configure-system-templates.md b/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-configure-system-templates.md index b9dd286c4..4d93ffff9 100644 --- a/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-configure-system-templates.md +++ b/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-configure-system-templates.md @@ -1,16 +1,21 @@ --- +navigation_title: Configure default templates +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configuring-ece-configure-system-templates.html --- -# Configure system deployment templates [ece-configuring-ece-configure-system-templates] +# Configure default system deployment templates [ece-configuring-ece-configure-system-templates] -While you can create new deployment templates for some use cases, if the system templates generally suit your needs but just require minor changes, you may choose to configure or modify the system templates. +While you can create new deployment templates for some use cases, if the default system templates meet your needs but require minor adjustments, you may choose to configure or modify them. For example, you want to use autoscaling with the system templates, but want to modify some of the default values for autoscaling in those templates. You might want to enable autoscaling by default for new deployments, or adjust the default value of the autoscaling maximum for the hot tier. -Note that you cannot edit system templates through the UI; they may only be configured through the API. - +::::{note} +You cannot edit system templates through the UI; they can only be configured through the API. +:::: ## Configure system deployment templates through the RESTful API [ece_configure_system_deployment_templates_through_the_restful_api] @@ -18,7 +23,6 @@ Note that you cannot edit system templates through the UI; they may only be conf The API user must have the `Platform admin` role in order to configure system templates. :::: - 1. Obtain the existing system deployment template you wish to modify. Note the `id` of the system deployment template as you will include this value in the API call to edit the template. ```sh diff --git a/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-create-templates.md b/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-create-templates.md index 20939bb79..7c60a6059 100644 --- a/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-create-templates.md +++ b/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-create-templates.md @@ -1,4 +1,8 @@ --- +navigation_title: Create templates +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configuring-ece-create-templates.html --- diff --git a/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-instance-configurations-create.md b/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-instance-configurations-create.md index 8b18432af..b5401110a 100644 --- a/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-instance-configurations-create.md +++ b/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-instance-configurations-create.md @@ -1,11 +1,14 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configuring-ece-instance-configurations-create.html --- # Create instance configurations [ece-configuring-ece-instance-configurations-create] -If you plan to [create your own templates](ece-configuring-ece-create-templates.md) and the default instance configurations that ship with ECE don’t quite suit your purpose, it’s generally easier and safer to create your own custom instance configurations first. Instance configurations match components of the Elastic Stack to allocators and tailor how memory and storage resources get sized relative to each other, and what sizes are available. +If you plan to [create your own templates](ece-configuring-ece-create-templates.md) and the [default instance configurations](./ece-configuring-ece-instance-configurations-default.md) that ship with ECE don’t quite suit your purpose, it’s generally easier and safer to create your own custom instance configurations first. Instance configurations match components of the Elastic Stack to allocators and tailor how memory and storage resources get sized relative to each other, and what sizes are available. ## Before you begin [ece_before_you_begin_2] diff --git a/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-instance-configurations-default.md b/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-instance-configurations-default.md index 3576abd18..caba12ad8 100644 --- a/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-instance-configurations-default.md +++ b/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-instance-configurations-default.md @@ -1,4 +1,7 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configuring-ece-instance-configurations-default.html --- diff --git a/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-instance-configurations-edit.md b/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-instance-configurations-edit.md index 85fc9167f..c12d2ff33 100644 --- a/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-instance-configurations-edit.md +++ b/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-instance-configurations-edit.md @@ -1,4 +1,7 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configuring-ece-instance-configurations-edit.html --- @@ -16,23 +19,21 @@ You might need to edit instance configurations under the following circumstances If you edit instance configurations, so that they match fewer allocators, instances of the Elastic Stack that were previously matched to those allocators might be relocated. Keep this in mind when making queries more restrictive. :::: - - ## Steps [ece_steps] 1. [Log into the Cloud UI](log-into-cloud-ui.md). 2. From the **Platform** menu, select **Templates**. -3. Select the **Instance configurations** tab to check the default instance configurations that ship with ECE. +3. Select the **Instance configurations** tab to check the [default instance configurations](./ece-configuring-ece-instance-configurations-default.md) that ship with ECE. 4. Choose one of the instance configurations and select **Edit instance configuration**. For example: Select to edit the `data.default` default instance configuration, so that you can specify where Elasticsearch data nodes for incoming data should be deployed. In a hot-warm architecture, this will determine where your hot data gets sent to. 5. In the **Input** section, construct a query that filters on specific allocator tags. - The following steps assume that no query exists, as is the case when you edit the default instance configurations for the first time after installing ECE version 2.0 or later. You can also edit an existing query by modifying the inner and outer clauses. + The following steps assume that no query exists, as is the case when you edit the [default instance configurations](./ece-configuring-ece-instance-configurations-default.md) for the first time after installing ECE version 2.0 or later. You can also edit an existing query by modifying the inner and outer clauses. ::::{tip} - An *outer clause* ANDs or ORs your main filtering criteria. You use outer clauses to find the allocators that you tagged earlier. An *inner clause* modifies an outer clause and let’s you refine your filtering criteria further. If you are unsure how the process works, try searching on some of the allocator tags that you added and check how the query results change. If you are editing the `data.default` default instance configuration, you want your query to return all allocators on which Elasticsearch data nodes for incoming data can be placed. + An *outer clause* ANDs or ORs your main filtering criteria. You use outer clauses to find the allocators that you tagged earlier. An *inner clause* modifies an outer clause and let’s you refine your filtering criteria further. If you are unsure how the process works, try searching on some of the allocator tags that you added and check how the query results change. If you are editing the `data.default` instance configuration, you want your query to return all allocators on which Elasticsearch data nodes for incoming data can be placed. :::: @@ -47,11 +48,11 @@ If you edit instance configurations, so that they match fewer allocators, instan 3. Check the list of allocators that get matched by your query: - * If you are satisfied that your query matches all the allocators where the component(s) of the Elastic Stack can be deployed, move on to the next step. For the `data.default` default instance configuration, this means all the allocators where Elasticsearch data nodes for incoming data should be deployed, for example. + * If you are satisfied that your query matches all the allocators where the component(s) of the Elastic Stack can be deployed, move on to the next step. For the `data.default` instance configuration, this means all the allocators where Elasticsearch data nodes for incoming data should be deployed, for example. * If you need to refine your query further, continue to adjust your outer or inner clauses. If you are unsure what to do, keep your initial query simple. You can always refine the query later on by re-editing the instance configuration. 6. Select **Save changes**. -7. If you are configuring the default instance configurations for the hot-warm template: Repeat steps 4 through 6 for the `data.highstorage`, `master`, `coordinating`, `kibana`, and `ml` instance configurations. +7. If you are configuring the [default instance configurations](./ece-configuring-ece-instance-configurations-default.md) for the hot-warm template: Repeat steps 4 through 6 for the `data.highstorage`, `master`, `coordinating`, `kibana`, and `ml` instance configurations. For example: For the `data.highstorage` instance configuration, your query should filter for allocators that use spindle-based storage. If you are using our [sample tags](ece-configuring-ece-tag-allocators.md#allocator-sample-tags), you could filter on either `SSD: false` or `highstorage: true`, depending on which tag you decided to use. For the `master` and `kibana` configurations, some multi-purpose hardware might work well. The `ml` instance configuration can benefit from hardware that provides higher CPU (`highCPU: true` in our sample tags). diff --git a/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-tag-allocators.md b/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-tag-allocators.md index 5e44fd9a2..7daf8d1c1 100644 --- a/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-tag-allocators.md +++ b/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-tag-allocators.md @@ -1,4 +1,7 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configuring-ece-tag-allocators.html --- diff --git a/deploy-manage/deploy/cloud-enterprise/ece-ha.md b/deploy-manage/deploy/cloud-enterprise/ece-ha.md index 2d57c6f76..8db8390d2 100644 --- a/deploy-manage/deploy/cloud-enterprise/ece-ha.md +++ b/deploy-manage/deploy/cloud-enterprise/ece-ha.md @@ -1,20 +1,26 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-ha.html --- # High availability [ece-ha] +Ensuring high availability in {{ece}} (ECE) requires careful planning and implementation across multiple areas, including availability zones, master nodes, replica shards, snapshot backups, and Zookeeper nodes. + +This section describes key considerations and best practices to prevent downtime and data loss at both the ECE platform level and within orchestrated deployments. ## Availability zones [ece-ece-ha-1-az] -Fault tolerance for Elastic Cloud Enterprise is based around the concept of *availability zones*. +Fault tolerance for ECE is based around the concept of *availability zones*. -An availability zone contains resources available to an Elastic Cloud Enterprise installation that are isolated from other availability zones to safeguard against potential failure. +An availability zone contains resources available to an ECE installation that are isolated from other availability zones to safeguard against potential failure. -Planning for a fault-tolerant installation with multiple availability zones means avoiding any single point of failure that could bring down Elastic Cloud Enterprise. +Planning for a fault-tolerant installation with multiple availability zones means avoiding any single point of failure that could bring down ECE. -The main difference between Elastic Cloud Enterprise installations that include two or three availability zones is that three availability zones enable Elastic Cloud Enterprise to create clusters with a *tiebreaker*. If you have only two availability zones in total in your installation, no tiebreaker is created. +The main difference between ECE installations that include two or three availability zones is that three availability zones enable ECE to create clusters with a *tiebreaker*. If you have only two availability zones in total in your installation, no tiebreaker is created. We recommend that for each deployment you use at least two availability zones for production and three for mission-critical systems. Using more than three availability zones for a deployment is not required nor supported. Availability zones are intended for high availability, not scalability. @@ -22,18 +28,16 @@ We recommend that for each deployment you use at least two availability zones fo {{es}} clusters that are set up to use only one availability zone are not [highly available](/deploy-manage/production-guidance/availability-and-resilience.md) and are at risk of data loss. To safeguard against data loss, you must use at least two {{ece}} availability zones. :::: - ::::{warning} Increasing the number of zones should not be used to add more resources. The concept of zones is meant for High Availability (2 zones) and Fault Tolerance (3 zones), but neither will work if the cluster relies on the resources from those zones to be operational. The recommendation is to scale up the resources within a single zone until the cluster can take the full load (add some buffer to be prepared for a peak of requests), then scale out by adding additional zones depending on your requirements: 2 zones for High Availability, 3 zones for Fault Tolerance. :::: - ## Master nodes [ece-ece-ha-2-master-nodes] -$$$ece-ha-tiebreaker$$$Tiebreakers are used in distributed clusters to avoid cases of [split brain](https://en.wikipedia.org/wiki/Split-brain_(computing)), where an {{es}} cluster splits into multiple, autonomous parts that continue to handle requests independently of each other, at the risk of affecting cluster consistency and data loss. A split-brain scenario is avoided by making sure that a minimum number of [master-eligible nodes](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md#master-node) must be present in order for any part of the cluster to elect a master node and accept user requests. To prevent multiple parts of a cluster from being eligible, there must be a [quorum-based majority](/deploy-manage/distributed-architecture/discovery-cluster-formation/modules-discovery-quorums.md) of `(n/2)+1` nodes, where `n` is the number of master-eligible nodes in the cluster. The minimum number of master nodes to reach quorum in a two-node cluster is the same as for a three-node cluster: two nodes must be available. +Tiebreakers are used in distributed clusters to avoid cases of [split brain](https://en.wikipedia.org/wiki/Split-brain_(computing)), where an {{es}} cluster splits into multiple, autonomous parts that continue to handle requests independently of each other, at the risk of affecting cluster consistency and data loss. A split-brain scenario is avoided by making sure that a minimum number of [master-eligible nodes](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md#master-node) must be present in order for any part of the cluster to elect a master node and accept user requests. To prevent multiple parts of a cluster from being eligible, there must be a [quorum-based majority](/deploy-manage/distributed-architecture/discovery-cluster-formation/modules-discovery-quorums.md) of `(n/2)+1` nodes, where `n` is the number of master-eligible nodes in the cluster. The minimum number of master nodes to reach quorum in a two-node cluster is the same as for a three-node cluster: two nodes must be available. -When you create a cluster with nodes in two availability zones when a third zone is available, Elastic Cloud Enterprise can create a tiebreaker in the third availability zone to help establish quorum in case of loss of an availability zone. The extra tiebreaker node that helps to provide quorum does not have to be a full-fledged and expensive node, as it does not hold data. For example: By tagging allocators hosts in Elastic Cloud Enterprise, can you create a cluster with eight nodes each in zones `ece-1a` and `ece-1b`, for a total of 16 nodes, and one tiebreaker node in zone `ece-1c`. This cluster can lose any of the three availability zones whilst maintaining quorum, which means that the cluster can continue to process user requests, provided that there is sufficient capacity available when an availability zone goes down. +When you create a cluster with nodes in two availability zones when a third zone is available, ECE can create a tiebreaker in the third availability zone to help establish quorum in case of loss of an availability zone. The extra tiebreaker node that helps to provide quorum does not have to be a full-fledged and expensive node, as it does not hold data. For example: By tagging allocators hosts in ECE, can you create a cluster with eight nodes each in zones `ece-1a` and `ece-1b`, for a total of 16 nodes, and one tiebreaker node in zone `ece-1c`. This cluster can lose any of the three availability zones whilst maintaining quorum, which means that the cluster can continue to process user requests, provided that there is sufficient capacity available when an availability zone goes down. By default, each node in an {{es}} cluster is a master-eligible node and a data node. In larger clusters, such as production clusters, it’s a good practice to split the roles, so that master nodes are not handling search or indexing work. When you create a cluster, you can specify to use dedicated [master-eligible nodes](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md#master-node), one per availability zone. @@ -41,8 +45,6 @@ By default, each node in an {{es}} cluster is a master-eligible node and a data Clusters that only have two or fewer master-eligible node are not [highly available](/deploy-manage/production-guidance/availability-and-resilience.md) and are at risk of data loss. You must have [at least three master-eligible nodes](/deploy-manage/distributed-architecture/discovery-cluster-formation/modules-discovery-quorums.md). :::: - - ## Replica shards [ece-ece-ha-3-replica-shards] With multiple {{es}} nodes in multiple availability zones you have the recommended hardware, the next thing to consider is having the recommended index replication. Each index, with the exception of searchable snapshot indexes, should have one or more replicas. Use the index settings API to find any indices with no replica: @@ -55,18 +57,18 @@ GET _all/_settings/index.number_of_replicas Indices with no replica, except for [searchable snapshot indices](/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md), are not highly available. You should use replicas to mitigate against possible data loss. :::: - +Refer to [](../../reference-architectures.md) for information about {{es}} architectures. ## Snapshot backups [ece-ece-ha-4-snapshot] You should configure and use [{{es}} snapshots](/deploy-manage/tools/snapshot-and-restore.md). Snapshots provide a way to backup and restore your {{es}} indices. They can be used to copy indices for testing, to recover from failures or accidental deletions, or to migrate data to other deployments. We recommend configuring an [{{ece}}-level repository](../../tools/snapshot-and-restore/cloud-enterprise.md) to apply across all deployments. See [Work with snapshots](../../tools/snapshot-and-restore.md) for more guidance. - ## Furthermore considerations [ece-ece-ha-5-other] * Make sure you have three Zookeepers - by default, on the Director host - for your ECE installation. Similar to three Elasticsearch master nodes can form a quorum, three Zookeepers can forum the quorum for high availability purposes. Backing up Zookeeper data directory is also recommended, read [this doc](../../../troubleshoot/deployments/cloud-enterprise/rebuilding-broken-zookeeper-quorum.md) for more guidance. + * Make sure that if you’re using a [private Docker registry server](ece-install-offline-with-registry.md) or are using any [custom bundles and plugins](../../../solutions/search/full-text/search-with-synonyms.md) hosted on a web server, that these are available to all ECE allocators, so that they can continue to be accessed in the event of a network partition or zone outage. + * Don’t delete containers unless guided by Elastic Support or there’s public documentation explicitly describing this as required action. Otherwise, it can cause issues and you may lose access or functionality of your {{ece}} platform. See [Troubleshooting container engines](../../../troubleshoot/deployments/cloud-enterprise/troubleshooting-container-engines.md) for more information. If in doubt, please [contact support for help](../../../troubleshoot/deployments/cloud-enterprise/ask-for-help.md). - diff --git a/deploy-manage/deploy/cloud-enterprise/ece-hardware-prereq.md b/deploy-manage/deploy/cloud-enterprise/ece-hardware-prereq.md index 6d5a1a6ad..8a34c97de 100644 --- a/deploy-manage/deploy/cloud-enterprise/ece-hardware-prereq.md +++ b/deploy-manage/deploy/cloud-enterprise/ece-hardware-prereq.md @@ -1,4 +1,7 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-hardware-prereq.html --- @@ -48,7 +51,7 @@ The size of your ECE deployment has a bearing on the JVM heap sizes that you sho | Minimum to install | 10 GB | 10 GB | 15 GB | 10 GB | | Minimum recommended | 1:4 RAM-to-storage ratio1 | 1:4 RAM-to-storage ratio1 | 1:4 RAM-to-storage ratio1 | Enough storage to support the RAM-to-storage ratio2 | -1 Control-plane services usually require about 1:4 RAM-to-storage ratio, this may vary. +1 Control-plane management services usually require about 1:4 RAM-to-storage ratio, this may vary. 2 For example, if you use a host with 256 GB of RAM and the default ratio of 1:32, your host must provide 8192 GB of disk space. diff --git a/deploy-manage/deploy/cloud-enterprise/ece-include-additional-kibana-plugin.md b/deploy-manage/deploy/cloud-enterprise/ece-include-additional-kibana-plugin.md index c235968cc..97964752d 100644 --- a/deploy-manage/deploy/cloud-enterprise/ece-include-additional-kibana-plugin.md +++ b/deploy-manage/deploy/cloud-enterprise/ece-include-additional-kibana-plugin.md @@ -1,4 +1,8 @@ --- +navigation_title: Include additional Kibana plugins +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-include-additional-kibana-plugin.html --- diff --git a/deploy-manage/deploy/cloud-enterprise/ece-install-offline-images.md b/deploy-manage/deploy/cloud-enterprise/ece-install-offline-images.md index e728f6908..f9019483e 100644 --- a/deploy-manage/deploy/cloud-enterprise/ece-install-offline-images.md +++ b/deploy-manage/deploy/cloud-enterprise/ece-install-offline-images.md @@ -1,4 +1,7 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-install-offline-images.html --- @@ -58,7 +61,7 @@ Enterprise Search is not available in versions 9.0+. ## All available Elastic Stack packs and Docker images [ece-full-download-list] -::::{dropdown} **Expand to view the full list** +::::{dropdown} Expand to view the full list | Required downloads | Minimum required ECE version | | --- | --- | | [Elasticsearch, Kibana, APM, and Enterprise Search stack pack: 8.17.1](https://download.elastic.co/cloud-enterprise/versions/8.17.1.zip) | ECE 3.0.0
(+ docker 20.10.10+ required for 8.16+) | diff --git a/deploy-manage/deploy/cloud-enterprise/ece-install-offline-no-registry.md b/deploy-manage/deploy/cloud-enterprise/ece-install-offline-no-registry.md index 5f0b2ded8..7d426c18e 100644 --- a/deploy-manage/deploy/cloud-enterprise/ece-install-offline-no-registry.md +++ b/deploy-manage/deploy/cloud-enterprise/ece-install-offline-no-registry.md @@ -1,9 +1,13 @@ --- +navigation_title: Without a private Docker registry +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-install-offline-no-registry.html --- -# Without a private Docker registry [ece-install-offline-no-registry] +# Air-gapped install without a private Docker registry [ece-install-offline-no-registry] To perform an offline installation without a private Docker registry, you have to download the available Docker Images on each host. @@ -65,6 +69,10 @@ To perform an offline installation without a private Docker registry, you have t 7. Copy the installation script to each host where you plan to install Elastic Cloud Enterprise or make it available on your network. 8. Invoke the installation script on each host: + ::::{note} + The installation commands for this method are the same as in a standard installation. Refer to [](./install-ece-procedures.md) for details on the installation steps and the parameters required by the installation script, which vary based on your installation size. + :::: + 1. On the first host: ```sh @@ -79,4 +87,4 @@ To perform an offline installation without a private Docker registry, you have t --roles-token 'TOKEN' ``` - + Once the installation is complete, refer to [](./log-into-cloud-ui.md) to access Cloud UI. \ No newline at end of file diff --git a/deploy-manage/deploy/cloud-enterprise/ece-install-offline-with-registry.md b/deploy-manage/deploy/cloud-enterprise/ece-install-offline-with-registry.md index a2faa50ed..a28dd3166 100644 --- a/deploy-manage/deploy/cloud-enterprise/ece-install-offline-with-registry.md +++ b/deploy-manage/deploy/cloud-enterprise/ece-install-offline-with-registry.md @@ -1,9 +1,13 @@ --- +navigation_title: With your private Docker registry +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-install-offline-with-registry.html --- -# With your private Docker registry [ece-install-offline-with-registry] +# Air-gapped install with a private Docker registry [ece-install-offline-with-registry] Installing ECE on multiple hosts with your own registry server is simpler, because you do not have to load the Docker images on each host. @@ -36,7 +40,7 @@ Installing ECE on multiple hosts with your own registry server is simpler, becau Enterprise Search is not available in versions 9.0+. ::: -3. Tag the Docker images with your private registry, where `REGISTRY` is `my.private.repo:5000`, for example: +3. Tag the Docker images with your private registry URL by replacing `REGISTRY` with your actual registry address, for example `my.private.repo:5000`: ```sh docker tag docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:3.8.1 REGISTRY/cloud-enterprise/elastic-cloud-enterprise:3.8.1 @@ -50,7 +54,7 @@ Installing ECE on multiple hosts with your own registry server is simpler, becau docker tag docker.elastic.co/cloud-release/enterprise-search-cloud:8.17.1 REGISTRY/cloud-release/enterprise-search-cloud:8.17.1 ``` -4. Push the Docker images to your private Docker registry, where `REGISTRY` is `my.private.repo:5000`, for example: +4. Push the Docker images to your private Docker registry, using the same tags from the previous step. Replace `REGISTRY` with your actual registry URL, for example `my.private.repo:5000`: ```sh docker push REGISTRY/cloud-enterprise/elastic-cloud-enterprise:3.8.1 @@ -70,8 +74,13 @@ Installing ECE on multiple hosts with your own registry server is simpler, becau curl -L -O https://download.elastic.co/cloud/elastic-cloud-enterprise.sh ``` -6. Copy the installation script to each host where you plan to install Elastic Cloud Enterprise. (Alternatively, you can place the installation script in a secure network location where your other hosts can access it.) -7. Invoke the installation script on each host with the `--docker-registry REGISTRY` parameter, where `REGISTRY` is `my.private.repo:5000`, for example: +6. Copy the installation script to each host where you plan to install Elastic Cloud Enterprise or make it available on your network. + +7. Invoke the installation script on each host with the `--docker-registry REGISTRY` parameter, replacing `REGISTRY` with your actual registry URL (for example `my.private.repo:5000`): + + ::::{note} + Refer to [](./install-ece-procedures.md) for more details on the parameters to pass to the installation script depending on the size of your installation. + :::: 1. On the first host: @@ -89,4 +98,4 @@ Installing ECE on multiple hosts with your own registry server is simpler, becau --docker-registry REGISTRY ``` - + Once the installation is complete, refer to [](./log-into-cloud-ui.md) to access Cloud UI. \ No newline at end of file diff --git a/deploy-manage/deploy/cloud-enterprise/ece-jvm.md b/deploy-manage/deploy/cloud-enterprise/ece-jvm.md index 6cc8797d3..64ed541a1 100644 --- a/deploy-manage/deploy/cloud-enterprise/ece-jvm.md +++ b/deploy-manage/deploy/cloud-enterprise/ece-jvm.md @@ -1,4 +1,7 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-jvm.html --- @@ -9,12 +12,11 @@ mapped_pages: ECE uses default JVM heap sizes for services that work for testing. Make sure to configure the JVM heap size that fits your use case. Not following the recommended settings may cause issues later on as volume of data and usage increases. :::: - When you install ECE specify the recommended JVM heap sizes with `--memory-settings JVM_SETTINGS` parameter, based on the use cases as described below: -* [Deploy a small installation](deploy-small-installation-onprem.md): For development, test, and small-scale use cases. -* [Deploy a medium installation](deploy-medium-installation-onprem.md): For many production setups. -* [Deploy a large installation](deploy-large-installation-onprem.md): For deployments with significant overall search and indexing throughput. +* [Deploy a small installation](deploy-small-installation.md): For development, test, and small-scale use cases. +* [Deploy a medium installation](deploy-medium-installation.md): For many production setups. +* [Deploy a large installation](deploy-large-installation.md): For deployments with significant overall search and indexing throughput. Other JVM heap sizes can be left at their defaults. diff --git a/deploy-manage/deploy/cloud-enterprise/ece-load-balancers.md b/deploy-manage/deploy/cloud-enterprise/ece-load-balancers.md index 220612d36..63340ca37 100644 --- a/deploy-manage/deploy/cloud-enterprise/ece-load-balancers.md +++ b/deploy-manage/deploy/cloud-enterprise/ece-load-balancers.md @@ -1,22 +1,25 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-load-balancers.html --- # Load balancers [ece-load-balancers] -Elastic Cloud Enterprise is designed to be used in conjunction with at least one load balancer. A load balancer is not included with Elastic Cloud Enterprise, so you need to provide one yourself and place it in front of the Elastic Cloud Enterprise proxies. +[{{ece}} architecture](./ece-architecture.md) is designed to be used in conjunction with at least one load balancer. A load balancer is not included with {{ece}}, so you need to provide one yourself and place it in front of the {{ece}} proxies. Use the following recommendations when configuring your load balancer: * **High availability**: The exact number of load balancers depends on the utilization rate for your clusters. In a highly available installation, use at least two load balancers for each availability zone in your installation. * **Inbound ports**: Load balancers require that inbound traffic is open on the ports used by Elasticsearch, Kibana, and the transport client. * **X-found-cluster**: ECE proxy uses the header `X-found-cluster` to know which cluster’s UUID (Universally Unique Identifier) the traffic needs to be routed to. If the load balancer rewrites a URL, make sure the HTTP header `X-Found-Cluster` gets added. For example: `X-found-cluster: d59109b8d542c5c4845679e597810796`. -* **X-Forwarded-For**: Configure load balancers to strip inbound `X-Forwarded-For` headers and to replace them with the client source IP as seen by the load balancer. This is required to prevent clients from spoofing their IP addresses. Elastic Cloud Enterprise uses `X-Forwarded-For` for logging client IP addresses and, if you have implemented IP filtering, for traffic management. +* **X-Forwarded-For**: Configure load balancers to strip inbound `X-Forwarded-For` headers and to replace them with the client source IP as seen by the load balancer. This is required to prevent clients from spoofing their IP addresses. {{ece}} uses `X-Forwarded-For` for logging client IP addresses and, if you have implemented IP filtering, for traffic management. * **HTTP**: Use *HTTP mode* for ports 9200/9243 (HTTP traffic to clusters) and also for ports 12400/12443 (adminconsole traffic). * **TCP**: Use *TCP mode* for ports 9300/9343 (transport client traffic to clusters) and the load balancer should enable the proxy protocol support. * **TCP**: Use *TCP mode* for port 9400 for TLS authenticated passthrough between clusters for cross-cluster search (CCS) and replication (CCR), if used. The load balancer should **not** enable the proxy protocol support. -* **TCP**: Use *HTTP mode* for port 9443 for API key authenticated traffic between clusters for cross-cluster search (CCS) and replication (CCR), if used. Make sure that all load balancers or proxies sending this traffic to deployments hosted on Elastic Cloud Enterprise are sending HTTP/1.1 traffic. +* **TCP**: Use *HTTP mode* for port 9443 for API key authenticated traffic between clusters for cross-cluster search (CCS) and replication (CCR), if used. Make sure that all load balancers or proxies sending this traffic to deployments hosted on {{ece}} are sending HTTP/1.1 traffic. * **Deployment traffic and Admin traffic**: Create separate load balancers for Deployment traffic (Elasticsearch and Kibana traffic) and Admin traffic (Cloud UI Console and Admin API). This separation allows you to migrate to a large installation topology without reconfiguring or creating an additional load balancer. * **Traffic across proxies**: Balance traffic evenly across all proxies. Proxies are constantly updated with the internal routing information on how to direct requests to clusters on allocators that are hosting their nodes across zones. Proxies prefer cluster nodes in their local zone and route requests primarily to nodes in their own zone. * **Network**: Use network that is fast enough from a latency and throughput perspective to be considered local for the Elasticsearch clustering requirement. There shouldn’t be a major advantage in "preferring local" from a load balancer perspective (rather than a proxy perspective), it might even lead to potential hot spotting on specific proxies, so it should be avoided. @@ -46,7 +49,7 @@ This returns a healthy response as: ## Proxy health check for ECE 2.1 and later [ece_proxy_health_check_for_ece_2_1_and_later] -For Elastic Cloud Enterprise 2.1 and later, the health check endpoint has changed. You can use `/_health` on proxy hosts with a result of either a 200 OK to indicate healthy or a 502 Bad Gateway response for unhealthy. A healthy response also means that internal routing tables in the proxy are valid and initialized, but not necessarily up-to-date. +For {{ece}} 2.1 and later, the health check endpoint has changed. You can use `/_health` on proxy hosts with a result of either a 200 OK to indicate healthy or a 502 Bad Gateway response for unhealthy. A healthy response also means that internal routing tables in the proxy are valid and initialized, but not necessarily up-to-date. ``` http://PROXY_ADDRESS:9200/_health diff --git a/deploy-manage/deploy/cloud-enterprise/ece-manage-capacity.md b/deploy-manage/deploy/cloud-enterprise/ece-manage-capacity.md index ab8440506..70b30415f 100644 --- a/deploy-manage/deploy/cloud-enterprise/ece-manage-capacity.md +++ b/deploy-manage/deploy/cloud-enterprise/ece-manage-capacity.md @@ -1,11 +1,14 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-manage-capacity.html --- -# Manage your installation capacity [ece-manage-capacity] +# Manage your allocators capacity [ece-manage-capacity] -In ECE, every host is a runner. Depending on the size of your platform, runners can have [one or more roles](ece-roles.md): Coordinator, director, proxy, and allocator. While planning the capacity of your ECE installation, you have to properly size the capacity for all roles. However, the allocator role deserves particular attention, as it hosts the Elasticsearch, Kibana, and APM nodes, and the relevant services. +In {{ece}} (ECE), every host is a runner. Depending on the size of your platform, runners can have [one or more roles](ece-roles.md): Coordinator, director, proxy, and allocator. While planning the capacity of your ECE installation, you have to properly size the capacity for all roles. However, the allocator role deserves particular attention, as it hosts the Elasticsearch, Kibana, and APM nodes, and the relevant services. This section focuses on the allocator role, and explains how to plan its capacity in terms of memory, CPU, `processors` setting, and storage. @@ -69,18 +72,12 @@ Consider a 32GB deployment hosted on a 128GB allocator. If you use the default system service reservation, the CPU quota is 29%: -
-
-\$CPU quota = 32 / (128 * 0.85) = 29%\$ -
-
+* CPU quota = 32 / (128 * 0.85) = 29% + If you use 12GB Allocator system service reservation, the CPU quota is 28%: -
-
-\$CPU quota = 32 / (128 - 12) = 28%\$ -
-
+* CPU quota = 32 / (128 - 12) = 28% + Those percentages represent the upper limit of the % of the total CPU resources available in a given 100ms period. diff --git a/deploy-manage/deploy/cloud-enterprise/ece-networking-prereq.md b/deploy-manage/deploy/cloud-enterprise/ece-networking-prereq.md index 02d8024ac..fa5ffd550 100644 --- a/deploy-manage/deploy/cloud-enterprise/ece-networking-prereq.md +++ b/deploy-manage/deploy/cloud-enterprise/ece-networking-prereq.md @@ -1,4 +1,7 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-networking-prereq.html --- diff --git a/deploy-manage/deploy/cloud-enterprise/ece-roles.md b/deploy-manage/deploy/cloud-enterprise/ece-roles.md index 565387cea..241e35377 100644 --- a/deploy-manage/deploy/cloud-enterprise/ece-roles.md +++ b/deploy-manage/deploy/cloud-enterprise/ece-roles.md @@ -1,4 +1,7 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-roles.html --- diff --git a/deploy-manage/deploy/cloud-enterprise/ece-software-prereq.md b/deploy-manage/deploy/cloud-enterprise/ece-software-prereq.md index 3ac08bad9..84976f0c4 100644 --- a/deploy-manage/deploy/cloud-enterprise/ece-software-prereq.md +++ b/deploy-manage/deploy/cloud-enterprise/ece-software-prereq.md @@ -1,4 +1,7 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-software-prereq.html --- diff --git a/deploy-manage/deploy/cloud-enterprise/ece-sysconfig.md b/deploy-manage/deploy/cloud-enterprise/ece-sysconfig.md index 7815d902a..0a038ef58 100644 --- a/deploy-manage/deploy/cloud-enterprise/ece-sysconfig.md +++ b/deploy-manage/deploy/cloud-enterprise/ece-sysconfig.md @@ -1,4 +1,7 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-sysconfig.html --- diff --git a/deploy-manage/deploy/cloud-enterprise/ece-users-permissions.md b/deploy-manage/deploy/cloud-enterprise/ece-users-permissions.md index 8a0c7014c..1720725b2 100644 --- a/deploy-manage/deploy/cloud-enterprise/ece-users-permissions.md +++ b/deploy-manage/deploy/cloud-enterprise/ece-users-permissions.md @@ -1,4 +1,7 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-users-permissions.html --- diff --git a/deploy-manage/deploy/cloud-enterprise/ece-wildcard-dns.md b/deploy-manage/deploy/cloud-enterprise/ece-wildcard-dns.md index 8cda2e978..475582087 100644 --- a/deploy-manage/deploy/cloud-enterprise/ece-wildcard-dns.md +++ b/deploy-manage/deploy/cloud-enterprise/ece-wildcard-dns.md @@ -1,4 +1,7 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-wildcard-dns.html --- diff --git a/deploy-manage/deploy/cloud-enterprise/enable-custom-endpoint-aliases.md b/deploy-manage/deploy/cloud-enterprise/enable-custom-endpoint-aliases.md index dbae739bd..1fe727478 100644 --- a/deploy-manage/deploy/cloud-enterprise/enable-custom-endpoint-aliases.md +++ b/deploy-manage/deploy/cloud-enterprise/enable-custom-endpoint-aliases.md @@ -1,4 +1,7 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configuring-deployment-aliases.html --- diff --git a/deploy-manage/deploy/cloud-enterprise/fresh-installation-of-ece-using-podman-hosts-cloud.md b/deploy-manage/deploy/cloud-enterprise/fresh-installation-of-ece-using-podman-hosts-cloud.md deleted file mode 100644 index c41e86540..000000000 --- a/deploy-manage/deploy/cloud-enterprise/fresh-installation-of-ece-using-podman-hosts-cloud.md +++ /dev/null @@ -1,84 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-install-using-podman-cloud.html ---- - -# Fresh installation of ECE using Podman hosts cloud [ece-install-using-podman-cloud] - -This section provides guidelines and recommendations to install ECE using a Podman-based environment. The recommended approach consists of two (2) high-level steps. - -**Step 1**: Install ECE. - -**Step 2**: Add additional Podman hosts - -::::{note} -When copy-pasting commands, verify that characters like quotes (“) are encoded correctly in the console where you copy the command to. -:::: - - -::::{note} -Steps that run commands starting with `sudo` can be run as any sudoers user. Otherwise, the corresponding user is mentioned as part of the step description. -:::: - - -::::{note} -Avoid customizing the host Docker path `/mnt/data/docker` when using SELinux. Otherwise the ECE installer script needs to be adjusted. -:::: - - -1. Install ECE - - Use the ECE installer script together with the `--podman` flag. - - Refer to the official [Install ECE online](install-ece-onprem.md) documentation to adapt the command line parameters to your environment. - - [JVM heap sizes](ece-jvm.md) describes recommended JVM options. - - ::::{important} - Important while running `./elastic-cloud-enterprise.sh` - - * Execute the installer script as user `elastic`. - * Ensure to use an installer script that supports podman. - * Make sure you use `--podman`. - * Use `--cloud-enterprise-version VERSION_NAME` to specify the correct version. - * If you are using SELinux, make sure you also use `--selinux`. - - :::: - -2. Add additional Podman hosts - - Refer to the official [Install Elastic Cloud Enterprise on an additional host](install-ece-on-additional-hosts.md) and [Install ECE online](install-ece-onprem.md) documentation to adapt the command line parameters to your environment including fetching the role token. - - [JVM heap sizes](ece-jvm.md) describes recommended JVM options. - - ::::{important} - Important while running `./elastic-cloud-enterprise.sh` - - * Execute the installer script as user `elastic`. - * Ensure to use an installer script that supports podman. - * Make sure you use `--podman`. - * If you are using SELinux, make sure you also use `--selinux`. - * To fetch a role token following the [Generate Roles Tokens](generate-roles-tokens.md) guidelines, you need to send a JSON token to the admin console. Double check the correct format of the roles. Roles are a list of individual strings in quotes, **NOT a single string**. - - **Example** - - ```json - { "persistent": true, "roles": [ "allocator","coordinator","director","proxy" ] } - ``` - - * The ECE version of the additional host must be the same as the version used in step 2. Use `--cloud-enterprise-version VERSION_NAME` to specify the correct version. - * Make sure to apply the roles to the additional host. The value for the `--roles` flag is a single string. - - **Example** - - ```sh - --roles "allocator,coordinator,director,proxy" - ``` - - - :::: - - - To add a new allocator, use `--roles "allocator"`. To add a new coordinator, director, proxy, and allocator, use `--roles "allocator,coordinator,director,proxy"` - - diff --git a/deploy-manage/deploy/cloud-enterprise/fresh-installation-of-ece-using-podman-hosts-onprem.md b/deploy-manage/deploy/cloud-enterprise/fresh-installation-of-ece-using-podman-hosts.md similarity index 69% rename from deploy-manage/deploy/cloud-enterprise/fresh-installation-of-ece-using-podman-hosts-onprem.md rename to deploy-manage/deploy/cloud-enterprise/fresh-installation-of-ece-using-podman-hosts.md index 290c64fa8..4a5770dcf 100644 --- a/deploy-manage/deploy/cloud-enterprise/fresh-installation-of-ece-using-podman-hosts-onprem.md +++ b/deploy-manage/deploy/cloud-enterprise/fresh-installation-of-ece-using-podman-hosts.md @@ -1,9 +1,14 @@ --- +applies_to: + deployment: + ece: all mapped_pages: + - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-install-using-podman-cloud.html - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-install-using-podman-onprem.html +navigation_title: Deploy using Podman --- -# Fresh installation of ECE using Podman hosts onprem [ece-install-using-podman-onprem] +# Fresh installation of ECE using Podman hosts [ece-install-using-podman] This section provides guidelines and recommendations to install ECE using a Podman-based environment. The recommended approach consists of two (2) high-level steps. @@ -11,26 +16,17 @@ This section provides guidelines and recommendations to install ECE using a Podm **Step 2**: Add additional Podman hosts -::::{note} -When copy-pasting commands, verify that characters like quotes (“) are encoded correctly in the console where you copy the command to. +::::{note} +* When copy-pasting commands, verify that characters like quotes (“) are encoded correctly in the console where you copy the command to. +* Steps that run commands starting with `sudo` can be run as any sudoers user. Otherwise, the corresponding user is mentioned as part of the step description. +* Avoid customizing the host Docker path `/mnt/data/docker` when using SELinux. Otherwise the ECE installer script needs to be adjusted. :::: - -::::{note} -Steps that run commands starting with `sudo` can be run as any sudoers user. Otherwise, the corresponding user is mentioned as part of the step description. -:::: - - -::::{note} -Avoid customizing the host Docker path `/mnt/data/docker` when using SELinux. Otherwise the ECE installer script needs to be adjusted. -:::: - - 1. Install ECE Use the ECE installer script together with the `--podman` flag. - Refer to the official [Install ECE online](install-ece-onprem.md) documentation to adapt the command line parameters to your environment. + Refer to the official [ECE installation](./install-ece-procedures.md) documentation to adapt the command line parameters to your environment. [JVM heap sizes](ece-jvm.md) describes recommended JVM options. @@ -42,12 +38,11 @@ Avoid customizing the host Docker path `/mnt/data/docker` when using SELinux. Ot * Make sure you use `--podman`. * Use `--cloud-enterprise-version VERSION_NAME` to specify the correct version. * If you are using SELinux, make sure you also use `--selinux`. - :::: 2. Add additional Podman hosts - Refer to the official [Install Elastic Cloud Enterprise on an additional host](install-ece-on-additional-hosts.md) and [Install ECE online](install-ece-onprem.md) documentation to adapt the command line parameters to your environment including fetching the role token. + Refer to the official [Install Elastic Cloud Enterprise on an additional host](install-ece-on-additional-hosts.md) and [ECE installation](./install-ece-procedures.md) documentation to adapt the command line parameters to your environment including fetching the role token. [JVM heap sizes](ece-jvm.md) describes recommended JVM options. @@ -74,8 +69,6 @@ Avoid customizing the host Docker path `/mnt/data/docker` when using SELinux. Ot ```sh --roles "allocator,coordinator,director,proxy" ``` - - :::: diff --git a/deploy-manage/deploy/cloud-enterprise/identify-deployment-scenario.md b/deploy-manage/deploy/cloud-enterprise/identify-deployment-scenario.md index c25b3a874..45de3ec9e 100644 --- a/deploy-manage/deploy/cloud-enterprise/identify-deployment-scenario.md +++ b/deploy-manage/deploy/cloud-enterprise/identify-deployment-scenario.md @@ -1,4 +1,7 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-deploy-scenario.html --- @@ -23,8 +26,7 @@ The type of deployment is recommended for development, test, and small-scale use * Avoid ECE installations with **spinning disks** as these are not supported when you run allocators and control plane on the same server. * Note that the small-size ECE installation keeps the directors and coordinators roles (ECE management services) on the same hosts as your allocators and proxies. -You can proceed with this scenario and install ECE with [Ansible](alternative-install-ece-with-ansible.md), on a [public cloud](install-ece-on-public-cloud.md), or on [your own premises](install-ece-on-own-premises.md). - +You can proceed with this scenario and [install ECE](./install.md). ## Medium deployment [ece_medium_deployment] @@ -47,8 +49,7 @@ This type of deployment is recommended for many production setups. You need: * Monitor the load on proxies and make sure the volume of user requests routed by the proxies does not affect the resources available to the ECE management services. * Note that the large-sized Elastic Cloud Enterprise installation separates the allocator and proxy roles from the director and coordinator roles (ECE management services). -You can proceed with this scenario and install ECE with [Ansible](alternative-install-ece-with-ansible.md), on a [public cloud](install-ece-on-public-cloud.md), or on [your own premises](install-ece-on-own-premises.md). - +You can proceed with this scenario and [install ECE](./install.md). ## Large deployment [ece_large_deployment] @@ -70,4 +71,4 @@ This type of deployment is recommended for deployments with significant overall Note that the large-sized Elastic Cloud Enterprise installation separates the allocator and proxy roles from the director and coordinator roles (ECE management services). -You can proceed with this scenario and install ECE with [Ansible](alternative-install-ece-with-ansible.md), on a [public cloud](install-ece-on-public-cloud.md), or on [your own premises](install-ece-on-own-premises.md). +You can proceed with this scenario and [install ECE](./install.md). \ No newline at end of file diff --git a/deploy-manage/deploy/cloud-enterprise/install-ece-cloud.md b/deploy-manage/deploy/cloud-enterprise/install-ece-cloud.md deleted file mode 100644 index 501b8fa54..000000000 --- a/deploy-manage/deploy/cloud-enterprise/install-ece-cloud.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-install-cloud.html ---- - -# Install ECE cloud [ece-install-cloud] - -Choose the Elastic Cloud Enterprise deployment scenario that best fits your business needs: - -* [Deploy a small installation](deploy-small-installation-cloud.md): For development, test, and small-scale use cases. -* [Deploy a medium installation](deploy-medium-installation-cloud.md): For many production setups. -* [Deploy a large installation](deploy-large-installation-cloud.md): For deployments with significant overall search and indexing throughput. -* [Deploy using Podman](fresh-installation-of-ece-using-podman-hosts-cloud.md): Fresh installation of ECE using Podman hosts. - - - - - diff --git a/deploy-manage/deploy/cloud-enterprise/install-ece-on-additional-hosts.md b/deploy-manage/deploy/cloud-enterprise/install-ece-on-additional-hosts.md index 9d1506d83..562f4a55c 100644 --- a/deploy-manage/deploy/cloud-enterprise/install-ece-on-additional-hosts.md +++ b/deploy-manage/deploy/cloud-enterprise/install-ece-on-additional-hosts.md @@ -1,4 +1,7 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-installing-additional.html --- diff --git a/deploy-manage/deploy/cloud-enterprise/install-ece-on-own-premises.md b/deploy-manage/deploy/cloud-enterprise/install-ece-on-own-premises.md deleted file mode 100644 index cf05d016b..000000000 --- a/deploy-manage/deploy/cloud-enterprise/install-ece-on-own-premises.md +++ /dev/null @@ -1,28 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-install-your-infra.html ---- - -# Install ECE on your own premises [ece-install-your-infra] - -Before you start, make sure that your existing infrastructure meets the [requirements](prepare-environment.md). - -ECE supports a [wide range of OS versions](https://www.elastic.co/support/matrix). Here are some OS-specific instructions for preparing your hosts; other versions will be similar: - -* [Ubuntu 20.04 LTS (Focal Fossa) and 22.04 LTS (Jammy Jellyfish)](configure-host-ubuntu-onprem.md) -* [Red Hat Enterprise Linux (RHEL) 8 and 9, and Rocky Linux 8 and 9](configure-host-rhel-onprem.md) -* [SUSE Linux Enterprise Server (SLES) 12 SP5 and 15](configure-host-suse-onprem.md) - -After your hosts are prepared, choose your preferred installation type: - -* [Install ECE online](install-ece-onprem.md) -* [Install ECE offline](air-gapped-install.md) - -::::{note} -In these pages we frequently refer to [Docker](https://www.docker.com/), as its currently the most common container engine, but these instructions are generally valid for [Podman](https://podman.io/) as well, with `podman` replacing `docker` in commands as appropriate. -:::: - - - - - diff --git a/deploy-manage/deploy/cloud-enterprise/install-ece-on-public-cloud.md b/deploy-manage/deploy/cloud-enterprise/install-ece-on-public-cloud.md deleted file mode 100644 index 36d082621..000000000 --- a/deploy-manage/deploy/cloud-enterprise/install-ece-on-public-cloud.md +++ /dev/null @@ -1,27 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-install-public.html ---- - -# Install ECE on a Public Cloud [ece-install-public] - -You can deploy ECE on any of the following cloud providers: - -* Amazon Web Services (AWS) -* Google Cloud Platform (GCP) -* Microsoft Azure - -with one of the following operating systems: - -* [Ubuntu 20.04 LTS (Focal Fossa) and Ubuntu 22.04 LTS (Jammy Jellyfish)](configure-host-ubuntu-cloud.md) -* [Red Hat Enterprise Linux (RHEL) 8 and 9](configure-host-rhel-cloud.md) -* [Rocky Linux 8 and 9](configure-host-rhel-cloud.md) -* [SUSE Linux Enterprise Server (SLES) 12 SP5 and 15](configure-host-suse-cloud.md) - -::::{important} -Cloud providers default provide automatic operating system patching for their virtual machines. We strongly recommend disabling this feature to avoid potential data loss and installation failure. All patching should be done via [Perform host maintenance](../../maintenance/ece/perform-ece-hosts-maintenance.md). -:::: - - - - diff --git a/deploy-manage/deploy/cloud-enterprise/install-ece-onprem.md b/deploy-manage/deploy/cloud-enterprise/install-ece-onprem.md deleted file mode 100644 index a7f52b8cf..000000000 --- a/deploy-manage/deploy/cloud-enterprise/install-ece-onprem.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-install-onprem.html ---- - -# Install ECE onprem [ece-install-onprem] - -Choose the Elastic Cloud Enterprise deployment scenario that best fits your business needs: - -* [Deploy a small installation](deploy-small-installation-onprem.md): For development, test, and small-scale use cases. -* [Deploy a medium installation](deploy-medium-installation-onprem.md): For many production setups. -* [Deploy a large installation](deploy-large-installation-onprem.md): For deployments with significant overall search and indexing throughput. -* [Deploy using Podman](fresh-installation-of-ece-using-podman-hosts-onprem.md): Fresh installation of ECE using Podman hosts. - - - - - diff --git a/deploy-manage/deploy/cloud-enterprise/install-ece-procedures.md b/deploy-manage/deploy/cloud-enterprise/install-ece-procedures.md new file mode 100644 index 000000000..d1b7833df --- /dev/null +++ b/deploy-manage/deploy/cloud-enterprise/install-ece-procedures.md @@ -0,0 +1,18 @@ +--- +navigation_title: Installation procedures +applies_to: + deployment: + ece: all +--- + +# ECE installation procedures + +Choose the guide for the Elastic Cloud Enterprise [deployment scenario](/deploy-manage/deploy/cloud-enterprise/identify-deployment-scenario.md) that best fits your business needs: + +* [Deploy a small installation](deploy-small-installation.md): For development, test, and small-scale use cases. +* [Deploy a medium installation](deploy-medium-installation.md): For many production setups. +* [Deploy a large installation](deploy-large-installation.md): For deployments with significant overall search and indexing throughput. + +For installations using Podman instead of Docker, refer to [](./fresh-installation-of-ece-using-podman-hosts.md). + +If you need to migrate an existing installation based on Docker to Podman, follow [](./migrate-ece-to-podman-hosts.md). \ No newline at end of file diff --git a/deploy-manage/deploy/cloud-enterprise/install.md b/deploy-manage/deploy/cloud-enterprise/install.md index 63076e676..4ebc1e30c 100644 --- a/deploy-manage/deploy/cloud-enterprise/install.md +++ b/deploy-manage/deploy/cloud-enterprise/install.md @@ -1,23 +1,68 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-installing.html + - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-install-public.html + - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-install-your-infra.html +navigation_title: Install ECE --- -# Install [ece-installing] +# Install Elastic Cloud Enterprise [ece-installing] -Before you start, make sure you [identify your deployment scenario](identify-deployment-scenario.md) and [prepare your hosts](prepare-environment.md). +You can deploy {{ece}} (ECE) on public or private clouds, virtual machines, or on-premises. -You can get ECE up and running using the official bash script on a [public cloud](install-ece-on-public-cloud.md) or on [your own premises](install-ece-on-own-premises.md). Alternatively, you can install ECE with the [Ansible](alternative-install-ece-with-ansible.md) playbook. The ECE Ansible playbook is a community project, supported by Elastic, aimed at installing ECE at scale. +In ECE, a host refers to any server, VM, or cloud instance where the ECE software is installed. An ECE platform consists of multiple hosts working together to orchestrate Elastic Stack applications. -Once you have installed ECE, check some final [post-installation steps](post-installation-steps.md) to get ready for production. +For public cloud deployments, you can choose from the following providers: -::::{tip} -This outline pertains to troubleshooting on the container engine level. The following outline is structured according to [Docker](https://www.docker.com/) as the most common engine but is also valid for [Podman](https://podman.io/), replacing out commands as needed. -:::: +* Amazon Web Services (AWS) +* Google Cloud Platform (GCP) +* Microsoft Azure +::::{tip} +If you already have an ECE platform up and running, and you want to add hosts to your installation, refer to [](./install-ece-on-additional-hosts.md). +:::: ::::{note} In these pages we frequently refer to [Docker](https://www.docker.com/), as its currently the most common container engine, but these instructions are generally valid for [Podman](https://podman.io/) as well, with `podman` replacing `docker` in commands as appropriate. :::: +## Prerequisites [ece-install-prerequisites] + +Before you start, make sure to [identify your deployment scenario](identify-deployment-scenario.md) and follow all the referenced sections in [](prepare-environment.md). Make sure that your selected infrastructure meets the requirements. + +## Configure your ECE hosts [ece-configure-hosts] + +After completing the prerequisites, proceed to configure your ECE hosts. This includes installing Docker or Podman, setting up XFS quotas, preparing mount points, and other required configurations. + +ECE supports a [wide range of OS versions](https://www.elastic.co/support/matrix#elastic-cloud-enterprise). Below are some OS-specific instructions for preparing your hosts, though other versions follow a similar process. Choose the appropriate guide for your operating system and follow the instructions: + +* [Ubuntu 20.04 LTS (Focal Fossa) and Ubuntu 22.04 LTS (Jammy Jellyfish)](configure-host-ubuntu.md) +* [Red Hat Enterprise Linux (RHEL) 8 and 9](configure-host-rhel.md) +* [Rocky Linux 8 and 9](configure-host-rhel.md) +* [SUSE Linux Enterprise Server (SLES) 12 SP5 and 15](configure-host-suse.md) + +::::{important} +Cloud providers default provide automatic operating system patching for their virtual machines. We strongly recommend disabling this feature to avoid potential data loss and installation failure. All patching should be done using the [Perform host maintenance](../../maintenance/ece/perform-ece-hosts-maintenance.md) instructions. +:::: + +## Install ECE [install-ece] + +To install ECE with the official bash script, follow the instructions for the [deployment scenario](./identify-deployment-scenario.md) that best fits your business needs: + + * [Deploy a small installation](deploy-small-installation.md): For development, test, and small-scale use cases. + * [Deploy a medium installation](deploy-medium-installation.md): For many production setups. + * [Deploy a large installation](deploy-large-installation.md): For deployments with significant overall search and indexing throughput. + * [Deploy using Podman](./fresh-installation-of-ece-using-podman-hosts.md): Fresh installation of ECE using Podman hosts. + +Alternatively, you can install ECE with the [Ansible](alternative-install-ece-with-ansible.md) playbook. The ECE Ansible playbook is a community project, supported by Elastic, aimed at installing ECE at scale. + +To install ECE in an air-gapped environment, refer to [](./air-gapped-install.md). + +## Post-installation steps + +Once you have installed ECE, check some final [post-installation steps](post-installation-steps.md) to get ready for production. + diff --git a/deploy-manage/deploy/cloud-enterprise/manage-elastic-stack-versions.md b/deploy-manage/deploy/cloud-enterprise/manage-elastic-stack-versions.md index 746ad592b..f6af860e7 100644 --- a/deploy-manage/deploy/cloud-enterprise/manage-elastic-stack-versions.md +++ b/deploy-manage/deploy/cloud-enterprise/manage-elastic-stack-versions.md @@ -1,4 +1,7 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-manage-elastic-stack.html --- diff --git a/deploy-manage/deploy/cloud-enterprise/migrate-ece-on-podman-hosts-to-selinux-enforce.md b/deploy-manage/deploy/cloud-enterprise/migrate-ece-on-podman-hosts-to-selinux-enforce.md index d1e5caf20..f56009f9e 100644 --- a/deploy-manage/deploy/cloud-enterprise/migrate-ece-on-podman-hosts-to-selinux-enforce.md +++ b/deploy-manage/deploy/cloud-enterprise/migrate-ece-on-podman-hosts-to-selinux-enforce.md @@ -1,11 +1,18 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-migrate-to-selinux-in-enforcing-mode.html --- # Migrate ECE on Podman hosts to SELinux enforce [ece-migrate-to-selinux-in-enforcing-mode] -This section provides guidelines and recommendations for migrating an existing platform on a Podman-based environment to use SELinux in `enforcing` mode. The recommended approach consists of four (4) high-level steps. Steps 2-4 need to be repeated for each host in your environment. +This section provides guidelines and recommendations for migrating an existing platform on a Podman-based environment to use SELinux in `enforcing` mode. + +[SELinux](https://www.redhat.com/en/topics/linux/what-is-selinux) (Security-Enhanced Linux) is a security module that enforces mandatory access controls, helping to protect systems from unauthorized access and privilege escalation. Running in enforcing mode ensures that security policies are strictly applied, which can improve security and compliance in hardened environments. + +The migration process consists of four high-level steps. Steps 2-4 need to be repeated for each host in your environment. **Step 1** Migrate existing ECE installation to version >=3.7.2 diff --git a/deploy-manage/deploy/cloud-enterprise/migrate-ece-to-podman-hosts.md b/deploy-manage/deploy/cloud-enterprise/migrate-ece-to-podman-hosts.md index 95e5ed343..ea8a24243 100644 --- a/deploy-manage/deploy/cloud-enterprise/migrate-ece-to-podman-hosts.md +++ b/deploy-manage/deploy/cloud-enterprise/migrate-ece-to-podman-hosts.md @@ -1,4 +1,7 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-migrate-to-podman.html --- @@ -23,22 +26,11 @@ Using Docker or Podman as container runtime is a configuration local to the host ::: ::::{note} -When copy-pasting commands, verify that characters like quotes (“) are encoded correctly in the console where you copy the command to. +* When copy-pasting commands, verify that characters like quotes (“) are encoded correctly in the console where you copy the command to. +* Steps that run commands starting with `sudo` can be run as any sudoers user. Otherwise, the corresponding user is mentioned as part of the step description. +* Avoid customizing the host Docker path `/mnt/data/docker` when using SELinux. Otherwise the ECE installer script needs to be adjusted. :::: - -::::{note} -Steps that run commands starting with `sudo` can be run as any sudoers user. -:::: - - -::::{note} -Avoid customizing the host Docker path `/mnt/data/docker` when using SELinux. Otherwise the ECE installer script needs to be adjusted. -:::: - - -Otherwise, when the file content changes, the corresponding user is mentioned as part of the step description. - 1. Make sure you are running a healthy x-node ECE environment ready to be upgraded. All nodes use the Docker container runtime. 2. Upgrade to ECE 3.3.0+ following the [Upgrade your installation](../../upgrade/orchestrator/upgrade-cloud-enterprise.md) guideline. Skip this step if your existing ECE installation already runs ECE >= 3.3.0. 3. Follow your internal guidelines to add an additional vanilla RHEL (Note that the version must be >= 8.5, but <9), or Rocky Linux 8 or 9 VM to your environment. @@ -393,7 +385,7 @@ Otherwise, when the file content changes, the corresponding user is mentioned as 1. Use the ECE installer script together with the `--podman` flag to add the additional host as a podman-based host. - Refer to the official [Install Elastic Cloud Enterprise on an additional host](install-ece-on-additional-hosts.md) and [Install ECE online](install-ece-onprem.md) documentation to adapt the command line parameters to your environment including fetching the role token. + Refer to the official [Install Elastic Cloud Enterprise on an additional host](install-ece-on-additional-hosts.md) and [Install ECE online](./install.md) documentation to adapt the command line parameters to your environment including fetching the role token. [JVM heap sizes](ece-jvm.md) describes recommended JVM options. diff --git a/deploy-manage/deploy/cloud-enterprise/post-installation-steps.md b/deploy-manage/deploy/cloud-enterprise/post-installation-steps.md index ee7bafd32..2580c1d89 100644 --- a/deploy-manage/deploy/cloud-enterprise/post-installation-steps.md +++ b/deploy-manage/deploy/cloud-enterprise/post-installation-steps.md @@ -1,26 +1,49 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-getting-started-post-installation.html --- # Post-installation steps [ece-getting-started-post-installation] -After your Elastic Cloud Enterprise installation is up, some additional steps might be required: +After your Elastic Cloud Enterprise installation is up, some additional steps might be required. For a complete list of configurable features in ECE, refer to [](./configure.md). + +::::{tip} +To start creating {{es}} deployments directly, refer to [](./working-with-deployments.md). +:::: + +* Add your own [load balancer](./ece-load-balancers.md). Load balancers are user supplied and we do not currently provide configuration steps for you. + +* In production systems, add your own [Cloud UI and Proxy certificates](../../security/secure-your-elastic-cloud-enterprise-installation/manage-security-certificates.md) to enable secure connections over HTTPS. The proxy certificate must be a wildcard certificate signed for the needed DNS records of your domain. + + ::::{note} + The default DNS resolution provided by Elastic is not intended for production use. Refer to [](./ece-wildcard-dns.md) for more information. + + If you intend to use [custom endpoint aliases](./enable-custom-endpoint-aliases.md) functionality, ensure you add the necessary Subject Alternative Name (SAN) entries to the proxy certificate. + :::: + +* Optionally, if you want the deployment endpoint links and Single-sign on to work with your domain name, configure it as the **deployment domain name** in the **Platform** > **Settings** section of the [Cloud UI](./log-into-cloud-ui.md). The domain name is used to generate the endpoint URLs and must align with your proxy certificate and DNS record. + + ::::{tip} + For example, if your proxy certificate is signed for `*.elastic-cloud-enterprise.example.com` and you have a wildcard DNS register pointing `*.elastic-cloud-enterprise.example.com` to your load balancer, you should configure `elastic-cloud-enterprise.example.com` as the **deployment domain name** in Platform → Settings. Refer to [](./change-endpoint-urls.md) for more details. + :::: + +* If you received a license from Elastic, [manage the licenses](../../license/manage-your-license-in-ece.md) for your Elastic Cloud Enterprise installation. -* Add your own load balancer. Load balancers are user supplied and we do not currently provide configuration steps for you. -* [Add more capacity](../../maintenance/ece/scale-out-installation.md) to your Elastic Cloud Enterprise installation, [resize your deployment](resize-deployment.md), [upgrade to a newer Elasticsearch version](../../upgrade/deployment-or-cluster.md), and [add some plugins](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch-plugins/cloud-enterprise/ece-add-plugins.md). -* [Configure ECE system deployments](system-deployments-configuration.md) to ensure a highly available and resilient setup. -* [Configure ECE for deployment templates](configure-deployment-templates.md) to indicate what kind of hardware you have available for Elastic Stack deployments. -* [Install your security certificates](../../security/secure-your-elastic-cloud-enterprise-installation/manage-security-certificates.md) to enable TLS/SSL authentication for secure connections over HTTPS. -* [Add a snapshot repository](../../tools/snapshot-and-restore/cloud-enterprise.md) to enable regular backups of your Elasticsearch clusters. * [Add more platform users](../../users-roles/cloud-enterprise-orchestrator/manage-users-roles.md) with role-based access control. + +* [Add a snapshot repository](../../tools/snapshot-and-restore/cloud-enterprise.md) to enable regular backups of your Elasticsearch clusters. + * Consider enabling encryption-at-rest (EAR) on your hosts. -* [Set up traffic filters](../../security/traffic-filtering.md) to restrict traffic to your deployment to only trusted IP addresses or VPCs. -* Learn how to work around host maintenance or a host failure by [moving nodes off of an allocator](../../maintenance/ece/move-nodes-instances-from-allocators.md). -* If you received a license from Elastic, [manage the licenses](../../license/manage-your-license-in-ece.md) for your Elastic Cloud Enterprise installation. + + :::{{note}} + Encryption-at-rest is not implemented out of the box in {{ece}}. [Learn more](/deploy-manage/security/secure-your-elastic-cloud-enterprise-installation.md#ece_encryption). + ::: + +* Learn about common maintenance activities—such as adding capacity, applying OS patches, and addressing host failures--at [](../../maintenance/ece.md). ::::{warning} During installation, the system generates secrets that are placed into the `/mnt/data/elastic/bootstrap-state/bootstrap-secrets.json` secrets file, unless you passed in a different path with the --host-storage-path parameter. Keep the information in the `bootstrap-secrets.json` file secure by removing it from its default location and placing it into a secure storage location. :::: - - diff --git a/deploy-manage/deploy/cloud-enterprise/prepare-environment.md b/deploy-manage/deploy/cloud-enterprise/prepare-environment.md index 3f30d671b..594a2730d 100644 --- a/deploy-manage/deploy/cloud-enterprise/prepare-environment.md +++ b/deploy-manage/deploy/cloud-enterprise/prepare-environment.md @@ -1,29 +1,34 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-prereqs.html --- # Prepare your environment [ece-prereqs] +In this section you'll find all the prerequisites and environment preparations required to properly plan and install {{ece}} (ECE). ## Requirements [ece-prepare-requirements] +In {{ece}}, an ECE host is the server, virtual machine, or cloud instance where the ECE software is installed. An ECE installation consists of a cluster of multiple hosts, forming the platform where Elastic Stack applications are orchestrated. + +To prepare your hosts for installation, the following prerequisites **must** be met: + ::::{important} These prerequisites are critical to establish a supported ECE configuration. Using unsupported combinations can cause a number of either intermediate or potentially permanent issues with your ECE environment, such as failures to create [system deployments](system-deployments-configuration.md), failures to upgrade workload deployments, proxy timeouts, data loss, and more. If upgrading ECE, read [upgrade your installation](../../upgrade/orchestrator/upgrade-cloud-enterprise.md) for guidance. :::: - -To prepare your hosts for their ECE installation, the following prerequisites **must** be met: - * [Hardware prerequisites](ece-hardware-prereq.md) * [Software prerequisites](ece-software-prereq.md) +* [System configuration prerequisites](ece-sysconfig.md) * [Networking prerequisites](ece-networking-prereq.md) * [Users and permissions prerequisites](ece-users-permissions.md) - ## Best practices and recommendations [ece-prepare-recommendations] -To prepare your hosts for ECE installation, the following best practices are recommended and should be considered: +Follow these best practices to properly prepare your ECE installation: * [High availability](ece-ha.md) - For production and mission-critical systems, high availability **must** be considered * [Separation of roles](ece-roles.md) - To group components on ECE and prevent conflicting workloads, consider role separation diff --git a/deploy-manage/deploy/cloud-enterprise/statistics-collected-by-cloud-enterprise.md b/deploy-manage/deploy/cloud-enterprise/statistics-collected-by-cloud-enterprise.md index a61fd6031..a14f02ae6 100644 --- a/deploy-manage/deploy/cloud-enterprise/statistics-collected-by-cloud-enterprise.md +++ b/deploy-manage/deploy/cloud-enterprise/statistics-collected-by-cloud-enterprise.md @@ -1,4 +1,7 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-phone-home.html --- diff --git a/deploy-manage/deploy/cloud-enterprise/system-deployments-configuration.md b/deploy-manage/deploy/cloud-enterprise/system-deployments-configuration.md index 6850273b4..f0a89b320 100644 --- a/deploy-manage/deploy/cloud-enterprise/system-deployments-configuration.md +++ b/deploy-manage/deploy/cloud-enterprise/system-deployments-configuration.md @@ -1,4 +1,7 @@ --- +applies_to: + deployment: + ece: all mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-system-clusters-configuration.html --- @@ -21,7 +24,7 @@ Admin console - `admin-console-elasticsearch` : Stores the state of your deployments, plans, and other operational data. If this cluster is not available, there will be several unexpected behaviors in the Cloud UI, such as stale or wrong status indicators for deployments, allocators, hosts, and more. Logging and metrics - `logging-and-metrics` -: As part of an ECE environment, a Beats sidecar with Filebeat and Metricbeat is installed on each ECE host. The logs and metrics collected by those beats are indexed in the `logging-and-metrics` cluster. This includes ECE service logs, such as proxy logs, director logs, and more. It also includes hosted deployments logs, security cluster audit logs, and metrics, such as CPU and disk usage. Data is collected from all hosts. This information is critical in order to be able to monitor ECE and troubleshoot issues. You can also use this data to configure watches to alert you in case of an issue, or machine learning jobs that can provide alerts based on anomalies or forecasting. +: As part of an ECE environment, a Beats sidecar with Filebeat and Metricbeat is installed on each ECE host. The logs and metrics collected by those beats are indexed in the `logging-and-metrics` cluster. This includes ECE service logs, such as proxy logs, director logs, and more. It also includes hosted deployments logs, security cluster audit logs, and metrics, such as CPU and disk usage. Data is collected from all hosts. This information is critical in order to be able to monitor ECE and troubleshoot issues. You can also use this data to configure watches to alert you in case of an issue, or machine learning jobs that can provide alerts based on anomalies or forecasting. Security - `security` : When you enable the user management feature, you trigger the creation of a third system deployment named `security`. This cluster stores all security-related configurations, such as native users and the related native realm, integration with SAML or LDAP as external authentication providers and their role mapping, and the realm ordering. The health of this cluster is critical to provide access to the ECE Cloud UI and REST API. To learn more, check [Configure role-based access control](../../users-roles/cloud-enterprise-orchestrator/manage-users-roles.md). Beginning with Elastic Cloud Enterprise 2.5.0 the `security` cluster is created automatically for you. It is recommended to use the [dedicated API](https://www.elastic.co/docs/api/doc/cloud-enterprise/operation/operation-update-security-deployment) to manage the cluster. @@ -39,9 +42,7 @@ For the `logging-and-metrics` cluster, you might want to also make sure that you For the `security` cluster, the number of zones must be set to 3 for high availability, otherwise you may encounter errors when trying to upgrade ECE versions. :::: - - -### Backup and restore [ece_backup_and_restore] +## Backup and restore [ece_backup_and_restore] ECE lets you manage snapshot repositories, so that you can back up and restore your clusters. This mechanism allows you to centrally manage your snapshot repositories, assigning them to deployments, and restoring snapshots to an existing or new deployment. Since the `admin-console-elasticsearch` and `security` clusters have a key role in making sure your ECE installation is operational, it’s important that you configure a snapshot repository after you complete your ECE installation and enable snapshots for both the `admin-console-elasticsearch` and `security` clusters, so that you can easily restore them if needed. @@ -49,8 +50,7 @@ As mentioned earlier, the `logging-and-metrics` cluster stores important informa To configure snapshot repositories, check [Add snapshot repository configurations](../../tools/snapshot-and-restore/cloud-enterprise.md). - -### Sizing [ece_sizing] +## Sizing [ece_sizing] Both the `admin-console-elasticsearch` and `security` clusters require relatively small amounts of RAM and almost no disk space, so increasing their size to 4 GB or 8 GB RAM per data node should be sufficient. @@ -62,8 +62,7 @@ When sizing your `logging-and-metrics` cluster, consider: * the number of ECE hosts, deployments, and log types you want to enable, such as slow logs or audit logs. * the desired retention period for the data. As with any other time-series data, you must properly manage your indices and delete old indices based on that retention period. - -### Access to system deployments [ece_access_to_system_deployments] +## Access to system deployments [ece_access_to_system_deployments] In the case of the `admin-console-elasticsearch` and `security` system deployments, the team managing ECE and assigned to the platform admin role should have permission to change each system deployment configuration and also to access each cluster itself. @@ -73,11 +72,8 @@ The `logging-and-metrics` cluster is different since, as an ECE admin, you likel The `logging-and-metrics` cluster is only intended for troubleshooting ECE deployment issues. If your use case involves modifying or normalizing logs from {{es}} or {{kib}}, use a separate [dedicated monitoring deployment](../../monitor/stack-monitoring/ece-stack-monitoring.md) instead. :::: - You can’t use ECE’s single sign-on (SSO) to access system deployments. ::::{note} Enabling integration with external authentication provider requires that you set the `system_owned` flag to `false` in order to change the elasticsearch.yaml configuration. Remember to set the flag back to `true` after you are done. :::: - - diff --git a/deploy-manage/deploy/cloud-enterprise/working-with-deployments.md b/deploy-manage/deploy/cloud-enterprise/working-with-deployments.md index f4d171762..41af26fb3 100644 --- a/deploy-manage/deploy/cloud-enterprise/working-with-deployments.md +++ b/deploy-manage/deploy/cloud-enterprise/working-with-deployments.md @@ -30,4 +30,7 @@ The documentation team is working to combine content pulled from the following p * [/raw-migrated-files/cloud/cloud-enterprise/ece-stack-getting-started.md](/raw-migrated-files/cloud/cloud-enterprise/ece-stack-getting-started.md) * [/raw-migrated-files/cloud/cloud-enterprise/ece-administering-deployments.md](/raw-migrated-files/cloud/cloud-enterprise/ece-administering-deployments.md) * [/raw-migrated-files/cloud/cloud-enterprise/ece-change-deployment.md](/raw-migrated-files/cloud/cloud-enterprise/ece-change-deployment.md) -* [/raw-migrated-files/cloud/cloud-enterprise/ece-monitoring-deployments.md](/raw-migrated-files/cloud/cloud-enterprise/ece-monitoring-deployments.md) \ No newline at end of file +* [/raw-migrated-files/cloud/cloud-enterprise/ece-monitoring-deployments.md](/raw-migrated-files/cloud/cloud-enterprise/ece-monitoring-deployments.md) + +% from the post-install instructions +% * [Set up traffic filters](../../security/traffic-filtering.md) to restrict traffic to your deployment to only trusted IP addresses or VPCs. diff --git a/deploy-manage/deploy/self-managed/air-gapped-install.md b/deploy-manage/deploy/self-managed/air-gapped-install.md index 37a2fb793..e3f666f9a 100644 --- a/deploy-manage/deploy/self-managed/air-gapped-install.md +++ b/deploy-manage/deploy/self-managed/air-gapped-install.md @@ -1,7 +1,6 @@ --- mapped_urls: - https://www.elastic.co/guide/en/elastic-stack/current/air-gapped-install.html - - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-install-offline.html --- # Air gapped install @@ -83,4 +82,3 @@ $$$air-gapped-kibana-product-documentation$$$ **This page is a work in progress.** The documentation team is working to combine content pulled from the following pages: * [/raw-migrated-files/stack-docs/elastic-stack/air-gapped-install.md](/raw-migrated-files/stack-docs/elastic-stack/air-gapped-install.md) -* [/raw-migrated-files/cloud/cloud-enterprise/ece-install-offline.md](/raw-migrated-files/cloud/cloud-enterprise/ece-install-offline.md) \ No newline at end of file diff --git a/deploy-manage/toc.yml b/deploy-manage/toc.yml index 1fa538f02..809b55934 100644 --- a/deploy-manage/toc.yml +++ b/deploy-manage/toc.yml @@ -117,49 +117,34 @@ toc: - file: deploy/cloud-enterprise/ece-software-prereq.md - file: deploy/cloud-enterprise/ece-sysconfig.md - file: deploy/cloud-enterprise/ece-networking-prereq.md + - file: deploy/cloud-enterprise/ece-users-permissions.md - file: deploy/cloud-enterprise/ece-ha.md - file: deploy/cloud-enterprise/ece-roles.md - file: deploy/cloud-enterprise/ece-load-balancers.md - - file: deploy/cloud-enterprise/ece-users-permissions.md - file: deploy/cloud-enterprise/ece-jvm.md - file: deploy/cloud-enterprise/ece-wildcard-dns.md - file: deploy/cloud-enterprise/ece-manage-capacity.md - file: deploy/cloud-enterprise/install.md children: - file: deploy/cloud-enterprise/identify-deployment-scenario.md - - file: deploy/cloud-enterprise/install-ece-on-public-cloud.md + - file: deploy/cloud-enterprise/configure-operating-system.md children: - - file: deploy/cloud-enterprise/configure-operating-system-cloud.md - children: - - file: deploy/cloud-enterprise/configure-host-ubuntu-cloud.md - - file: deploy/cloud-enterprise/configure-host-rhel-cloud.md - - file: deploy/cloud-enterprise/configure-host-suse-cloud.md - - file: deploy/cloud-enterprise/install-ece-cloud.md - children: - - file: deploy/cloud-enterprise/deploy-small-installation-cloud.md - - file: deploy/cloud-enterprise/deploy-medium-installation-cloud.md - - file: deploy/cloud-enterprise/deploy-large-installation-cloud.md - - file: deploy/cloud-enterprise/fresh-installation-of-ece-using-podman-hosts-cloud.md - - file: deploy/cloud-enterprise/install-ece-on-own-premises.md + - file: deploy/cloud-enterprise/configure-host-ubuntu.md + - file: deploy/cloud-enterprise/configure-host-rhel.md + - file: deploy/cloud-enterprise/configure-host-suse.md + - file: deploy/cloud-enterprise/install-ece-procedures.md children: - - file: deploy/cloud-enterprise/configure-operating-system-onprem.md - children: - - file: deploy/cloud-enterprise/configure-host-ubuntu-onprem.md - - file: deploy/cloud-enterprise/configure-host-rhel-onprem.md - - file: deploy/cloud-enterprise/configure-host-suse-onprem.md - - file: deploy/cloud-enterprise/install-ece-onprem.md - children: - - file: deploy/cloud-enterprise/deploy-small-installation-onprem.md - - file: deploy/cloud-enterprise/deploy-medium-installation-onprem.md - - file: deploy/cloud-enterprise/deploy-large-installation-onprem.md - - file: deploy/cloud-enterprise/fresh-installation-of-ece-using-podman-hosts-onprem.md - - file: deploy/cloud-enterprise/alternative-install-ece-with-ansible.md + - file: deploy/cloud-enterprise/deploy-small-installation.md + - file: deploy/cloud-enterprise/deploy-medium-installation.md + - file: deploy/cloud-enterprise/deploy-large-installation.md + - file: deploy/cloud-enterprise/fresh-installation-of-ece-using-podman-hosts.md + - file: deploy/cloud-enterprise/migrate-ece-to-podman-hosts.md - file: deploy/cloud-enterprise/log-into-cloud-ui.md + - file: deploy/cloud-enterprise/post-installation-steps.md - file: deploy/cloud-enterprise/install-ece-on-additional-hosts.md children: - file: deploy/cloud-enterprise/generate-roles-tokens.md - - file: deploy/cloud-enterprise/migrate-ece-to-podman-hosts.md - - file: deploy/cloud-enterprise/post-installation-steps.md + - file: deploy/cloud-enterprise/alternative-install-ece-with-ansible.md - file: deploy/cloud-enterprise/statistics-collected-by-cloud-enterprise.md - file: deploy/cloud-enterprise/air-gapped-install.md children: @@ -183,7 +168,6 @@ toc: - file: deploy/cloud-enterprise/ce-add-support-for-node-roles-autoscaling.md - file: deploy/cloud-enterprise/ece-ce-add-support-for-integrations-server.md - file: deploy/cloud-enterprise/ece-configuring-ece-instance-configurations-default.md - - file: deploy/cloud-enterprise/ece-include-additional-kibana-plugin.md - file: deploy/cloud-enterprise/change-ece-api-url.md - file: deploy/cloud-enterprise/change-endpoint-urls.md - file: deploy/cloud-enterprise/enable-custom-endpoint-aliases.md @@ -192,6 +176,7 @@ toc: - file: deploy/cloud-enterprise/configure-allocator-affinity.md - file: deploy/cloud-enterprise/change-allocator-disconnect-timeout.md - file: deploy/cloud-enterprise/manage-elastic-stack-versions.md + - file: deploy/cloud-enterprise/ece-include-additional-kibana-plugin.md - file: deploy/cloud-enterprise/migrate-ece-on-podman-hosts-to-selinux-enforce.md - file: deploy/cloud-enterprise/working-with-deployments.md children: diff --git a/deploy-manage/tools/cross-cluster-replication.md b/deploy-manage/tools/cross-cluster-replication.md index 694a65755..bfdf72589 100644 --- a/deploy-manage/tools/cross-cluster-replication.md +++ b/deploy-manage/tools/cross-cluster-replication.md @@ -1,10 +1,12 @@ --- +mapped_pages: + - https://www.elastic.co/guide/en/elasticsearch/reference/current/xpack-ccr.html applies_to: deployment: - eck: - ess: - ece: - self: + eck: + ess: + ece: + self: --- # Cross-cluster replication [xpack-ccr] diff --git a/deploy-manage/tools/snapshot-and-restore/create-snapshots.md b/deploy-manage/tools/snapshot-and-restore/create-snapshots.md index cfe941935..793d23271 100644 --- a/deploy-manage/tools/snapshot-and-restore/create-snapshots.md +++ b/deploy-manage/tools/snapshot-and-restore/create-snapshots.md @@ -1,10 +1,12 @@ --- +mapped_pages: + - https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-take-snapshot.html applies_to: deployment: - eck: - ess: - ece: - self: + eck: + ess: + ece: + self: --- # Create snapshots [snapshots-take-snapshot] diff --git a/deploy-manage/tools/snapshot-and-restore/ec-aws-custom-repository.md b/deploy-manage/tools/snapshot-and-restore/ec-aws-custom-repository.md index a1df80117..342096b76 100644 --- a/deploy-manage/tools/snapshot-and-restore/ec-aws-custom-repository.md +++ b/deploy-manage/tools/snapshot-and-restore/ec-aws-custom-repository.md @@ -2,9 +2,10 @@ mapped_urls: - https://www.elastic.co/guide/en/cloud/current/ec-aws-custom-repository.html - https://www.elastic.co/guide/en/cloud-heroku/current/ech-custom-repository.html + - https://www.elastic.co/guide/en/cloud-heroku/current/ech-aws-custom-repository.html applies_to: deployment: - ess: + ess: --- # Configure a snapshot repository using AWS S3 [ec-aws-custom-repository] diff --git a/deploy-manage/tools/snapshot-and-restore/minio-on-premise-repository.md b/deploy-manage/tools/snapshot-and-restore/minio-on-premise-repository.md index 2d957c8ac..e3d443606 100644 --- a/deploy-manage/tools/snapshot-and-restore/minio-on-premise-repository.md +++ b/deploy-manage/tools/snapshot-and-restore/minio-on-premise-repository.md @@ -24,7 +24,7 @@ Installing Minio for production requires a high-availability configuration where As mentioned in the Minio documentation, you will need to have 4-16 Minio drive mounts. There is no hard limit on the number of Minio nodes. It might be convenient to place the Minio node containers on your ECE hosts to ensure you have a suitable level of availability, but those can not be located on the same hosts as ECE proxies since they both listen on the same port. -The following illustration is a sample architecture for a [large ECE installation](../../deploy/cloud-enterprise/deploy-large-installation-onprem.md). Note that there is at least one MinIO container in *each* availability zone. +The following illustration is a sample architecture for a [large ECE installation](../../deploy/cloud-enterprise/deploy-large-installation.md). Note that there is at least one MinIO container in *each* availability zone. There are a number of different ways of orchestrating the Minio deployment (Docker Compose, Kubernetes, and so on). We suggest you use the method most familiar to you. diff --git a/deploy-manage/tools/snapshot-and-restore/repository-isolation-on-azure.md b/deploy-manage/tools/snapshot-and-restore/repository-isolation-on-azure.md index bcdae3c0e..2b54bb57d 100644 --- a/deploy-manage/tools/snapshot-and-restore/repository-isolation-on-azure.md +++ b/deploy-manage/tools/snapshot-and-restore/repository-isolation-on-azure.md @@ -1,7 +1,9 @@ --- +mapped_pages: + - https://www.elastic.co/guide/en/cloud/current/ec-snapshot-repository-azure-migration.html applies_to: deployment: - ess: + ess: --- # Repository isolation on Azure [ec-snapshot-repository-azure-migration] diff --git a/deploy-manage/tools/snapshot-and-restore/restore-snapshot.md b/deploy-manage/tools/snapshot-and-restore/restore-snapshot.md index e306dcf70..264a42f43 100644 --- a/deploy-manage/tools/snapshot-and-restore/restore-snapshot.md +++ b/deploy-manage/tools/snapshot-and-restore/restore-snapshot.md @@ -1,10 +1,12 @@ --- +mapped_pages: + - https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-restore-snapshot.html applies_to: deployment: - eck: - ess: - ece: - self: + eck: + ess: + ece: + self: --- # Restore a snapshot diff --git a/deploy-manage/tools/snapshot-and-restore/s3-repository.md b/deploy-manage/tools/snapshot-and-restore/s3-repository.md index 8ec2d4938..4024c8e9c 100644 --- a/deploy-manage/tools/snapshot-and-restore/s3-repository.md +++ b/deploy-manage/tools/snapshot-and-restore/s3-repository.md @@ -1,16 +1,16 @@ ---- +--- mapped_urls: - https://www.elastic.co/guide/en/elasticsearch/reference/current/repository-s3.html applies_to: deployment: - self: + self: --- # S3 repository [repository-s3] You can use AWS S3 as a repository for [Snapshot/Restore](../snapshot-and-restore.md). -::::{note} +::::{note} If you are looking for a hosted solution of Elasticsearch on AWS, please visit [https://www.elastic.co/cloud/](https://www.elastic.co/cloud/). :::: @@ -122,7 +122,7 @@ The following list contains the available client settings. Those that must be st `path_style_access` : Whether to force the use of the path style access pattern. If `true`, the path style access pattern will be used. If `false`, the access pattern will be automatically determined by the AWS Java SDK (See [AWS documentation](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Builder.md#setPathStyleAccessEnabled-java.lang.Boolean-) for details). Defaults to `false`. -::::{note} +::::{note} :name: repository-s3-path-style-deprecation In versions `7.0`, `7.1`, `7.2` and `7.3` all bucket operations used the [now-deprecated](https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/) path style access pattern. If your deployment requires the path style access pattern then you should set this setting to `true` when upgrading. @@ -168,7 +168,7 @@ The following settings are supported: `base_path` : Specifies the path to the repository data within its bucket. Defaults to an empty string, meaning that the repository is at the root of the bucket. The value of this setting should not start or end with a `/`. - ::::{note} + ::::{note} Don’t set `base_path` when configuring a snapshot repository for {{ECE}}. {{ECE}} automatically generates the `base_path` for each deployment so that multiple deployments may share the same bucket. :::: @@ -192,7 +192,7 @@ The following settings are supported: If `false`, the cluster can write to the repository and create snapshots in it. Defaults to `false`. - ::::{important} + ::::{important} If you register the same snapshot repository with multiple clusters, only one cluster should have write access to the repository. Having multiple clusters write to the repository at the same time risks corrupting the contents of the repository. :::: @@ -231,7 +231,7 @@ The following settings are supported: `get_register_retry_delay` : ([time value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#time-units)) Sets the time to wait before trying again if an attempt to read a [linearizable register](#repository-s3-linearizable-registers) fails. Defaults to `5s`. -::::{note} +::::{note} The option of defining client settings in the repository settings as documented below is considered deprecated, and will be removed in a future version. :::: @@ -352,7 +352,7 @@ You may further restrict the permissions by specifying a prefix within the bucke The bucket needs to exist to register a repository for snapshots. If you did not create the bucket then the repository registration will fail. -#### Using IAM roles for Kubernetes service accounts for authentication [iam-kubernetes-service-accounts] +#### Using IAM roles for Kubernetes service accounts for authentication [iam-kubernetes-service-accounts] If you want to use [Kubernetes service accounts](https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/) for authentication, you need to add a symlink to the `$AWS_WEB_IDENTITY_TOKEN_FILE` environment variable (which should be automatically set by a Kubernetes pod) in the S3 repository config directory, so the repository can have the read access for the service account (a repository can’t read any files outside its config directory). For example: @@ -361,7 +361,7 @@ mkdir -p "${ES_PATH_CONF}/repository-s3" ln -s $AWS_WEB_IDENTITY_TOKEN_FILE "${ES_PATH_CONF}/repository-s3/aws-web-identity-token-file" ``` -::::{important} +::::{important} The symlink must be created on all data and master eligible nodes and be readable by the `elasticsearch` user. By default, {{es}} runs as user `elasticsearch` using uid:gid `1000:0`. :::: diff --git a/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md b/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md index d1c02d8db..500b77ecd 100644 --- a/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md +++ b/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md @@ -1,10 +1,12 @@ --- +mapped_pages: + - https://www.elastic.co/guide/en/elasticsearch/reference/current/searchable-snapshots.html applies_to: deployment: - eck: - ess: - ece: - self: + eck: + ess: + ece: + self: --- # Searchable snapshots [searchable-snapshots] diff --git a/explore-analyze/alerts-cases/alerts/rule-action-variables.md b/explore-analyze/alerts-cases/alerts/rule-action-variables.md index 9c1747608..099fa7a98 100644 --- a/explore-analyze/alerts-cases/alerts/rule-action-variables.md +++ b/explore-analyze/alerts-cases/alerts/rule-action-variables.md @@ -8,7 +8,7 @@ mapped_pages: # Rule action variables [rule-action-variables] -Alerting rules can use the [Mustache](https://mustache.github.io/mustache.5.md) template syntax (`{{variable name}}`) to pass values when the actions run. +Alerting rules can use the [Mustache](https://mustache.github.io/mustache.5.html) template syntax (`{{variable name}}`) to pass values when the actions run. ## Common variables [common-rule-action-variables] diff --git a/explore-analyze/machine-learning/data-frame-analytics/ml-trained-models.md b/explore-analyze/machine-learning/data-frame-analytics/ml-trained-models.md index 66ff63e90..41200aab0 100644 --- a/explore-analyze/machine-learning/data-frame-analytics/ml-trained-models.md +++ b/explore-analyze/machine-learning/data-frame-analytics/ml-trained-models.md @@ -115,4 +115,4 @@ If you also want to copy the {{dfanalytics-job}} to the new cluster, you can exp ## Importing an external model to the {{stack}} [import-external-model-to-es] -It is possible to import a model to your {{es}} cluster even if the model is not trained by Elastic {{dfanalytics}}. Eland supports [importing models](asciidocalypse://docs/eland/docs/reference/machine-learning.md) directly through its APIs. Please refer to the latest [Eland documentation](https://eland.readthedocs.io/en/latest/index.md) for more information on supported model types and other details of using Eland to import models with. +It is possible to import a model to your {{es}} cluster even if the model is not trained by Elastic {{dfanalytics}}. Eland supports [importing models](asciidocalypse://docs/eland/docs/reference/machine-learning.md) directly through its APIs. Please refer to the latest [Eland documentation](https://eland.readthedocs.io/en/latest/index.html) for more information on supported model types and other details of using Eland to import models with. diff --git a/explore-analyze/query-filter/languages/sql-functions-datetime.md b/explore-analyze/query-filter/languages/sql-functions-datetime.md index eae19ce4a..f742c8973 100644 --- a/explore-analyze/query-filter/languages/sql-functions-datetime.md +++ b/explore-analyze/query-filter/languages/sql-functions-datetime.md @@ -530,7 +530,7 @@ DATE_FORMAT( **Output**: string -**Description**: Returns the date/datetime/time as a string using the format specified in the 2nd argument. The formatting pattern is one of the specifiers used in the [MySQL DATE_FORMAT() function](https://dev.mysql.com/doc/refman/8.0/en/date-and-time-functions.md#function_date-format). +**Description**: Returns the date/datetime/time as a string using the format specified in the 2nd argument. The formatting pattern is one of the specifiers used in the [MySQL DATE_FORMAT() function](https://dev.mysql.com/doc/refman/8.0/en/date-and-time-functions.html#function_date-format). ::::{note} If the 1st argument is of type `time`, then pattern specified by the 2nd argument cannot contain date related units (e.g. *dd*, *MM*, *yyyy*, etc.). If it contains such units an error is returned. Ranges for month and day specifiers (%c, %D, %d, %e, %m) start at one, unlike MySQL, where they start at zero, due to the fact that MySQL permits the storing of incomplete dates such as *2014-00-00*. Elasticsearch in this case returns an error. @@ -578,7 +578,7 @@ DATE_PARSE( **Output**: date -**Description**: Returns a date by parsing the 1st argument using the format specified in the 2nd argument. The parsing format pattern used is the one from [`java.time.format.DateTimeFormatter`](https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/time/format/DateTimeFormatter.md). +**Description**: Returns a date by parsing the 1st argument using the format specified in the 2nd argument. The parsing format pattern used is the one from [`java.time.format.DateTimeFormatter`](https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/time/format/DateTimeFormatter.html). ::::{note} If the parsing pattern does not contain all valid date units (e.g. *HH:mm:ss*, *dd-MM HH:mm:ss*, etc.) an error is returned as the function needs to return a value of `date` type which will contain date part. @@ -627,7 +627,7 @@ DATETIME_FORMAT( **Output**: string -**Description**: Returns the date/datetime/time as a string using the format specified in the 2nd argument. The formatting pattern used is the one from [`java.time.format.DateTimeFormatter`](https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/time/format/DateTimeFormatter.md). +**Description**: Returns the date/datetime/time as a string using the format specified in the 2nd argument. The formatting pattern used is the one from [`java.time.format.DateTimeFormatter`](https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/time/format/DateTimeFormatter.html). ::::{note} If the 1st argument is of type `time`, then pattern specified by the 2nd argument cannot contain date related units (e.g. *dd*, *MM*, *yyyy*, etc.). If it contains such units an error is returned. @@ -675,7 +675,7 @@ DATETIME_PARSE( **Output**: datetime -**Description**: Returns a datetime by parsing the 1st argument using the format specified in the 2nd argument. The parsing format pattern used is the one from [`java.time.format.DateTimeFormatter`](https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/time/format/DateTimeFormatter.md). +**Description**: Returns a datetime by parsing the 1st argument using the format specified in the 2nd argument. The parsing format pattern used is the one from [`java.time.format.DateTimeFormatter`](https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/time/format/DateTimeFormatter.html). ::::{note} If the parsing pattern contains only date or only time units (e.g. *dd/MM/yyyy*, *HH:mm:ss*, etc.) an error is returned as the function needs to return a value of `datetime` type which must contain both. @@ -732,7 +732,7 @@ TIME_PARSE( **Output**: time -**Description**: Returns a time by parsing the 1st argument using the format specified in the 2nd argument. The parsing format pattern used is the one from [`java.time.format.DateTimeFormatter`](https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/time/format/DateTimeFormatter.md). +**Description**: Returns a time by parsing the 1st argument using the format specified in the 2nd argument. The parsing format pattern used is the one from [`java.time.format.DateTimeFormatter`](https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/time/format/DateTimeFormatter.html). ::::{note} If the parsing pattern contains only date units (e.g. *dd/MM/yyyy*) an error is returned as the function needs to return a value of `time` type which will contain only time. diff --git a/explore-analyze/scripting/modules-scripting-expression.md b/explore-analyze/scripting/modules-scripting-expression.md index 2f6b74516..b0f8c98f2 100644 --- a/explore-analyze/scripting/modules-scripting-expression.md +++ b/explore-analyze/scripting/modules-scripting-expression.md @@ -22,7 +22,7 @@ This allows for very fast execution, even faster than if you had written a `nati Expressions support a subset of javascript syntax: a single expression. -See the [expressions module documentation](https://lucene.apache.org/core/10_0_0/expressions/index.md?org/apache/lucene/expressions/js/package-summary.md) for details on what operators and functions are available. +See the [expressions module documentation](https://lucene.apache.org/core/10_0_0/expressions/index.html?org/apache/lucene/expressions/js/package-summary.md) for details on what operators and functions are available. Variables in `expression` scripts are available to access: diff --git a/explore-analyze/visualize/custom-visualizations-with-vega.md b/explore-analyze/visualize/custom-visualizations-with-vega.md index a8b19822a..7c90ffa4c 100644 --- a/explore-analyze/visualize/custom-visualizations-with-vega.md +++ b/explore-analyze/visualize/custom-visualizations-with-vega.md @@ -271,7 +271,7 @@ In the **Vega-Lite** spec, add the `encoding` block: #### Extract the `time_buckets.buckets` inner array [_extract_the_time_buckets_buckets_inner_array] -In {{kib}} 7.9 and later, use the **Vega-Lite** [flatten transformation](https://vega.github.io/vega-lite/docs/flatten.md) to extract the `time_buckets.buckets` inner array. +In {{kib}} 7.9 and later, use the **Vega-Lite** [flatten transformation](https://vega.github.io/vega-lite/docs/flatten.html) to extract the `time_buckets.buckets` inner array. If you are using {{kib}} 7.8 and earlier, the flatten transformation is available only in **Vega**. @@ -1160,10 +1160,10 @@ padding: { } ``` -To learn more, read about [Vega autosize](https://vega.github.io/vega/docs/specification/#autosize) and [Vega-Lite autosize](https://vega.github.io/vega-lite/docs/size.md). +To learn more, read about [Vega autosize](https://vega.github.io/vega/docs/specification/#autosize) and [Vega-Lite autosize](https://vega.github.io/vega-lite/docs/size.html). ::::{note} -Autosize in Vega-Lite has [several limitations](https://vega.github.io/vega-lite/docs/size.md#limitations) which can affect the height and width of your visualization, but these limitations do not exist in Vega. If you need full control, convert your spec to Vega using the [browser console](#vega-browser-debugging-console) `VEGA_DEBUG.vega_spec` output. To disable these warnings, you can [add extra options to your spec](#vega-additional-configuration-options). +Autosize in Vega-Lite has [several limitations](https://vega.github.io/vega-lite/docs/size.html#limitations) which can affect the height and width of your visualization, but these limitations do not exist in Vega. If you need full control, convert your spec to Vega using the [browser console](#vega-browser-debugging-console) `VEGA_DEBUG.vega_spec` output. To disable these warnings, you can [add extra options to your spec](#vega-additional-configuration-options). :::: @@ -1418,7 +1418,7 @@ The visualization automatically injects a `"projection"`, which you can use to c ##### Additional tooltip styling [vega-tooltip] -{{kib}} has installed the [Vega tooltip plugin](https://vega.github.io/vega-lite/docs/tooltip.md), so tooltips can be defined in the ways documented there. Beyond that, {{kib}} also supports a configuration option for changing the tooltip position and padding: +{{kib}} has installed the [Vega tooltip plugin](https://vega.github.io/vega-lite/docs/tooltip.html), so tooltips can be defined in the ways documented there. Beyond that, {{kib}} also supports a configuration option for changing the tooltip position and padding: ```js { @@ -1558,7 +1558,7 @@ The [Vega Editor](https://vega.github.io/editor/) includes examples for Vega & V #### Vega-Lite resources [vega-lite-resources] -* [Tutorials](https://vega.github.io/vega-lite/tutorials/getting_started.md) +* [Tutorials](https://vega.github.io/vega-lite/tutorials/getting_started.html) * [Docs](https://vega.github.io/vega-lite/docs/) * [Examples](https://vega.github.io/vega-lite/examples/) diff --git a/explore-analyze/visualize/maps/maps-clean-data.md b/explore-analyze/visualize/maps/maps-clean-data.md index 74709e092..8c33063fb 100644 --- a/explore-analyze/visualize/maps/maps-clean-data.md +++ b/explore-analyze/visualize/maps/maps-clean-data.md @@ -13,7 +13,7 @@ Geospatial fields in {{es}} have certain restrictions that need to be addressed ## Convert to GeoJSON or Shapefile [_convert_to_geojson_or_shapefile] -Use [ogr2ogr](https://gdal.org/programs/ogr2ogr.md) (part of the [GDAL/OGR](https://gdal.org) suite) to convert datasets into a GeoJSON or Esri Shapefile. For example, use the following commands to convert a GPX file into JSON: +Use [ogr2ogr](https://gdal.org/programs/ogr2ogr.html) (part of the [GDAL/OGR](https://gdal.org) suite) to convert datasets into a GeoJSON or Esri Shapefile. For example, use the following commands to convert a GPX file into JSON: ```sh # Example GPX file from https://www.topografix.com/gpx_sample_files.asp @@ -33,7 +33,7 @@ $ ogr2ogr -f "GeoJSON" "routes.geo.json" "fells_loop.gpx" "routes" {{es}} only supports WGS84 Coordinate Reference System. Use `ogr2ogr` to convert from other coordinate systems to WGS84. -On the following example, `ogr2ogr` transforms a shapefile from [NAD83](https://epsg.org/crs_4269/NAD83.md) to [WGS84](https://epsg.org/crs_4326/WGS-84.md). The input CRS is detected automatically thanks to the `.prj` sidecar file in the source dataset. +On the following example, `ogr2ogr` transforms a shapefile from [NAD83](https://epsg.org/crs_4269/NAD83.html) to [WGS84](https://epsg.org/crs_4326/WGS-84.html). The input CRS is detected automatically thanks to the `.prj` sidecar file in the source dataset. ```sh # Example NAD83 file from https://www2.census.gov/geo/tiger/GENZ2018/shp/cb_2018_us_county_5m.zip @@ -86,7 +86,7 @@ A dataset containing records with a very large amount of parts as the one from t Some machine generated datasets are stored with more decimals than are strictly necessary. For reference, the GeoJSON RFC 7946 [coordinate precision section](https://datatracker.ietf.org/doc/html/rfc7946#section-11.2) specifies six digits to be a common default to around 10 centimeters on the ground. The file uploader in the Maps application will automatically reduce the precision to 6 decimals but for big datasets it is better to do this before uploading. -`ogr2ogr` generates GeoJSON files with 7 decimal degrees when requesting `RFC7946` compliant files but using the `COORDINATE_PRECISION` [GeoJSON layer creation option](https://gdal.org/drivers/vector/geojson.md#layer-creation-options) it can be downsized even more if that is OK for the usage of the data. +`ogr2ogr` generates GeoJSON files with 7 decimal degrees when requesting `RFC7946` compliant files but using the `COORDINATE_PRECISION` [GeoJSON layer creation option](https://gdal.org/drivers/vector/geojson.html#layer-creation-options) it can be downsized even more if that is OK for the usage of the data. ```sh # Example NAD83 file from https://www2.census.gov/geo/tiger/GENZ2018/shp/cb_2018_us_county_5m.zip @@ -171,7 +171,7 @@ $ du -h cb_2018_us_county_5m*.geo.json The Maps application expects valid GeoJSON or Shapefile datasets. Apart from the mentioned CRS requirement, geometries need to be valid. Both `ogr2ogr` and `mapshaper` have options to try to fix invalid geometries: -* OGR [`-makevalid`](https://gdal.org/programs/ogr2ogr.md#cmdoption-ogr2ogr-makevalid) option +* OGR [`-makevalid`](https://gdal.org/programs/ogr2ogr.html#cmdoption-ogr2ogr-makevalid) option * Mapshaper [`-clean`](https://github.com/mbloch/mapshaper/wiki/Command-Reference#-clean) command diff --git a/raw-migrated-files/cloud/cloud-enterprise/ece-install-offline.md b/raw-migrated-files/cloud/cloud-enterprise/ece-install-offline.md deleted file mode 100644 index a8af0e872..000000000 --- a/raw-migrated-files/cloud/cloud-enterprise/ece-install-offline.md +++ /dev/null @@ -1,23 +0,0 @@ -# Install ECE offline [ece-install-offline] - -Installing ECE on hosts without internet access is commonly referred to as an *offline* or *air-gapped installation*. Before you start, you must: - -* Download the Elasticsearch and Kibana images and installation script from Elastic and load them on your hosts, or push them to your private Docker registry. You need to download both the Elastic Stack pack and the Docker images for the same version. - - ``` - The versioning of Elasticsearch and Kibana is synchronized and versions where the major, minor, and patch levels match can be used together. Differences in build versions indicated by a dash do not affect compatibility. - ``` - -* Be part of the `docker` group to run the installation script. You should not install Elastic Cloud Enterprise as the `root` user. -* Set up your [wildcard DNS record](../../../deploy-manage/deploy/cloud-enterprise/ece-wildcard-dns.md). -* Set up and run a local copy of the Elastic Package Repository, otherwise your deployments with APM server and Elastic agent won’t work. Refer to the [Running EPR in airgapped environments](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/air-gapped.md#air-gapped-diy-epr) documentation. -* Deployment End-of-life (EOL) information relies on the connection to [https://www.elastic.co/support/eol.json](https://www.elastic.co/support/eol.json). If EOL information is updated, Elastic may require you to reconnect to [https://www.elastic.co/support/eol.json](https://www.elastic.co/support/eol.json) over the Internet to get this information reflected. - -When you are ready to install ECE, you can proceed: - -* [With your private Docker registry](../../../deploy-manage/deploy/cloud-enterprise/ece-install-offline-with-registry.md) -* [Without your private Docker registry](../../../deploy-manage/deploy/cloud-enterprise/ece-install-offline-no-registry.md) - - - - diff --git a/raw-migrated-files/docs-content/serverless/cspm-required-permissions.md b/raw-migrated-files/docs-content/serverless/cspm-required-permissions.md deleted file mode 100644 index b8716371a..000000000 --- a/raw-migrated-files/docs-content/serverless/cspm-required-permissions.md +++ /dev/null @@ -1,60 +0,0 @@ -# CSPM privilege requirements [cspm-required-permissions] - -This page lists required privilges for {{elastic-sec}}'s CSPM features. There are three access levels: read, write, and manage. Each access level and its requirements are described below. - - -## Read [_read] - -Users with these minimum permissions can view data on the **Findings** page and the Cloud Posture dashboard. - - -### {{es}} index privileges [_es_index_privileges] - -`Read` privileges for the following {{es}} indices: - -* `logs-cloud_security_posture.findings_latest-*` -* `logs-cloud_security_posture.scores-*` - - -### {{kib}} privileges [_kib_privileges] - -* `Security: Read` - - -## Write [_write] - -Users with these minimum permissions can view data on the **Findings** page and the Cloud Posture dashboard, create detection rules from the findings details flyout, and enable or disable benchmark rules. - - -### {{es}} index privileges [_es_index_privileges_2] - -`Read` privileges for the following {{es}} indices: - -* `logs-cloud_security_posture.findings_latest-*` -* `logs-cloud_security_posture.scores-*` - - -### {{kib}} privileges [_kib_privileges_2] - -* `Security: All` - - -## Manage [_manage] - -Users with these minimum permissions can view data on the **Findings** page and the Cloud Posture dashboard, create detection rules from the findings details flyout, enable or disable benchmark rules, and install, update, or uninstall CSPM integrations and assets. - - -### {{es}} index privileges [_es_index_privileges_3] - -`Read` privileges for the following {{es}} indices: - -* `logs-cloud_security_posture.findings_latest-*` -* `logs-cloud_security_posture.scores-*` - - -### {{kib}} privileges [_kib_privileges_3] - -* `Security: All` -* `Spaces: All` -* `Fleet: All` -* `Integrations: All` diff --git a/raw-migrated-files/docs-content/serverless/ingest-aws-securityhub-data.md b/raw-migrated-files/docs-content/serverless/ingest-aws-securityhub-data.md deleted file mode 100644 index d0b3b2efc..000000000 --- a/raw-migrated-files/docs-content/serverless/ingest-aws-securityhub-data.md +++ /dev/null @@ -1,16 +0,0 @@ -# Ingest AWS Security Hub data [ingest-aws-securityhub-data] - -In order to enrich your {{elastic-sec}} workflows with third-party cloud security posture data collected by AWS Security Hub: - -* Follow the steps to [set up the AWS Security Hub integration](https://docs.elastic.co/en/integrations/aws/securityhub). -* Make sure the integration version is at least 2.31.1. -* Ensure you have `read` privileges for the `security_solution-*.misconfiguration_latest` index. -* While configuring the AWS Security Hub integration, turn on **Collect AWS Security Hub Findings from AWS**. We recommend you also set the **Initial Interval** value to `2160h` (equivalent to 90 days) to ingest existing logs. - -:::{image} ../../../images/serverless-aws-config-finding-logs.png -:alt: AWS Security Hub integration settings showing the findings toggle -::: - -After you’ve completed these steps, AWS Security Hub data will appear on the **Misconfigurations** tab of the [**Findings**](../../../solutions/security/cloud/findings-page.md) page. - -Any available findings data will also appear in the entity details flyouts for related [alerts](../../../solutions/security/detect-and-alert/view-detection-alert-details.md#insights-section). If alerts are present for a user or host that has findings data from AWS Security Hub, the findings will appear on the [users](/solutions/security/explore/users-page.md#user-details-flyout), and [hosts](/solutions/security/explore/hosts-page.md#host-details-flyout) flyouts. diff --git a/raw-migrated-files/docs-content/serverless/ingest-falco.md b/raw-migrated-files/docs-content/serverless/ingest-falco.md deleted file mode 100644 index 7be40874d..000000000 --- a/raw-migrated-files/docs-content/serverless/ingest-falco.md +++ /dev/null @@ -1,114 +0,0 @@ -# Ingest CNCF Falco data [ingest-falco] - -CNCF Falco is an open-source runtime security tool that detects anomalous activity in Linux hosts, containers, Kubernetes, and cloud environments. You can ingest Falco alerts into {{es}} to view them on {{elastic-sec}}'s **Alerts** page and incorporate them into your security workflows by using Falcosidekick, a proxy forwarder that can send alerts from your Falco deployments to {{es}}. - -First, you’ll need to configure {{elastic-sec}} to receive data from Falco, then you’ll need to configure Falco and Falcosidekick to send data to {{es}}. - - -## Configure {{elastic-sec}} to receive Falco data [ingest-falco-setup-kibana] - -In {{elastic-sec}}: - -1. Click **Add integrations**. -2. Search the **Integrations** page for `Falco`, then select it. -3. Go to the Falco integration’s **Settings** tab. -4. Click **Install Falco**, then confirm by clicking **Install Falco** again. Installation should take less than a minute. - -{{elastic-sec}} is now ready to receive data from Falco. The Falco integration page now has an **Assets** tab where you can inspect the newly installed assets that help to ingest Falco data. - -Next, to make alerts from Falco appear on {{elastic-sec}}'s **Alerts** page: - -1. Find the **Detection rules (SIEM)** page in the navigation menu or by using the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). -2. Search for a rule named `External Alerts`. Install it if necessary, and enable it. - - -## Configure Falco and Falcosidekick [ingest-falco-setup-falco] - -You can either: - -* [Send Falco data to {{es}} from virtual machines (VMs)](../../../solutions/security/cloud/ingest-cncf-falco-data.md#ingest-falco-setup-falco-vm); or, -* [Send Falco data to {{es}} from Kubernetes](../../../solutions/security/cloud/ingest-cncf-falco-data.md#ingest-falco-setup-falco-kubernetes). - - -### Configure Falco and Falcosidekick for VMs [ingest-falco-setup-falco-vm] - -Multiple methods for configuring Falco to send data from VMs to {{es}} are available. This guide uses the [Falco sidekick on Docker using environment variables](https://github.com/falcosecurity/falcosidekick/blob/master/docs/outputs/elasticsearch.md) method. - - -### Configure Falco for VMs: [_configure_falco_for_vms] - -1. Refer to Falco’s documentation to [install Falco on the Linux VMs you wish to monitor](https://falco.org/docs/setup/packages/). -2. Once Falco is installed, update `/etc/falco/falco.yaml` as follows: - - 1. Enable JSON output: `json_output: true` - 2. Enable HTTP output: under `http_output`, for the `url` value, enter the `url:port` where Falcosidekick will listen. For example, if Falcosidekick is running on localhost: - - ``` - http_output: - enabled: true - url: "http://0.0.0.0:2801/" - ``` - - - -### Configure Falcosidekick for VMs: [falco-config-falco-for-vms] - -1. Refer to Falcosidekick’s documentation to [install Falcosidekick](https://github.com/falcosecurity/falcosidekick?tab=readme-ov-file#installation). -2. Use the [Falcosidekick on Docker using environment variables](https://github.com/falcosecurity/falcosidekick/blob/master/docs/outputs/elasticsearch.md) method and set your environment variables as follows: - - 1. `ELASTICSEARCH_HOSTPORT`: Your {{es}} endpoint URL, which can be found under **Connection details** on the upper right of the **Integrations** page in {{kib}}. - 2. `ELASTICSEARCH_INDEX`: The {{es}} index where you want to store Falco logs. - - ::::{important} - Your `ELASTICSEARCH_INDEX` value must match `logs-falco.alerts-*`. - :::: - - 3. `ELASTICSEARCH_SUFFIX`: The frequency with which you want the {{es}} index suffix to change. Either `daily`, `monthly`, `annually`, or `none`. - 4. `ELASTICSEARCH_APIKEY`: The recommended way to authenticate to {{es}}, by providing an [API key](../../../deploy-manage/api-keys/elasticsearch-api-keys.md). Note that support for this environment variable starts with Falcosidekick version 2.30. You can access the latest version on Falcosidekick’s [Docker Hub](https://hub.docker.com/r/falcosecurity/falcosidekick). - 5. `ELASTICSEARCH_USERNAME` and `ELASTICSEARCH_PASSWORD`: The username and password for an account on your {{es}} instance. Authentication using these environment variables is not supported on {{ecloud}} Serverless. - 6. `ELASTICSEARCH_MUTUALTLS` and `ELASTICSEARCH_CHECKCERT`: For security reasons, we recommend setting these to `true`. - - -For example: - -``` -docker run -d -p 2801:2801 - -e ELASTICSEARCH_HOSTPORT=https://test-falco.es.us-west2.gcp.elastic-cloud.com - -e ELASTICSEARCH_INDEX=logs-falco.alerts-all - -e ELASTICSEARCH_SUFFIX=none - -e ELASTICSEARCH_APIKEY=XXXXXXXXXXXXX - -e ELASTICSEARCH_MUTUALTLS=true - -e ELASTICSEARCH_CHECKCERT=true falcosecurity/falcosidekick -``` - -::::{important} -The {{es}} account used to authenticate Falcosidekick only needs sufficient privileges to create and write to new indices. We recommend following the principle of least privilege when provisioning this account. -:::: - - -After installing and configuring Falcosidekick, restart Falco with `sudo systemctl restart falco`. Falcosidekick should start sending alerts to {{es}}. - - -## Configure Falco and Falcosidekick for Kubernetes [ingest-falco-setup-falco-kubernetes] - -1. Add the Falco [Helm charts](https://github.com/falcosecurity/charts/blob/master/README.md): - - ``` - helm repo add falcosecurity https://falcosecurity.github.io/charts - helm repo update - ``` - -2. Next, install Falco and Falcosidekick using the `falcosecurity/falco` Helm chart with [appropriate values](https://github.com/falcosecurity/falcosidekick/blob/master/docs/outputs/elasticsearch.md) for each of the `falcosidekick.config.elasticsearch.*` fields: - - ``` - helm install falco falcosecurity/falco \ - --set falcosidekick.enabled=true \ - --set tty=true \ - --set driver.kind=modern_ebpf \ - --set collectors.kubernetes.enabled=true \ - --set falcosidekick.config.elasticsearch.hostport="https://" \ - --set falcosidekick.config.elasticsearch.username="" \ - --set falcosidekick.config.elasticsearch.password="" \ - --set falcosidekick.config.elasticsearch.index="logs-falco.alerts-all" \ - --set falcosidekick.config.elasticsearch.suffix="none" - ``` diff --git a/raw-migrated-files/docs-content/serverless/ingest-third-party-cloud-security-data.md b/raw-migrated-files/docs-content/serverless/ingest-third-party-cloud-security-data.md deleted file mode 100644 index 985fa7639..000000000 --- a/raw-migrated-files/docs-content/serverless/ingest-third-party-cloud-security-data.md +++ /dev/null @@ -1,24 +0,0 @@ -# Ingest third-party cloud security data [ingest-third-party-cloud-security-data] - -This section describes how to ingest cloud security data from third-party tools into {{es}}. Once ingested, this data can provide additional context and enrich your {{elastic-sec}} workflows. - -You can ingest both third-party cloud workload protection data and third-party security posture and vulnerability data. - - -## Ingest third-party workload protection data [_ingest_third_party_workload_protection_data] - -You can ingest third-party cloud security alerts into {{elastic-sec}} to view them on the [Alerts page](../../../solutions/security/detect-and-alert/manage-detection-alerts.md) and incorporate them into your triage and threat hunting workflows. - -* Learn to [ingest alerts from Sysdig Falco](../../../solutions/security/cloud/ingest-cncf-falco-data.md). - - -## Ingest third-party security posture and vulnerability data [_ingest_third_party_security_posture_and_vulnerability_data] - -You can ingest third-party data into {{elastic-sec}} to review and investigate it alongside data collected by {{elastic-sec}}'s native cloud security integrations. Once ingested, cloud security posture and vulnerability data appears on the [Findings](../../../solutions/security/cloud/findings-page.md) page and in the entity details flyouts for [alerts](../../../solutions/security/detect-and-alert/view-detection-alert-details.md#insights-section), [users](/solutions/security/explore/users-page.md#user-details-flyout), and [hosts](/solutions/security/explore/hosts-page.md#host-details-flyout) flyouts. - -* Learn to [ingest cloud security posture data from AWS Security Hub](../../../solutions/security/cloud/ingest-aws-security-hub-data.md). -* Learn to [ingest cloud security posture and vulnerability data from Wiz](../../../solutions/security/cloud/ingest-wiz-data.md). - - - - diff --git a/raw-migrated-files/docs-content/serverless/ingest-wiz-data.md b/raw-migrated-files/docs-content/serverless/ingest-wiz-data.md deleted file mode 100644 index b307b87af..000000000 --- a/raw-migrated-files/docs-content/serverless/ingest-wiz-data.md +++ /dev/null @@ -1,24 +0,0 @@ -# Ingest Wiz data [ingest-wiz-data] - -In order to enrich your {{elastic-sec}} workflows with third-party cloud security posture and vulnerability data collected by Wiz: - -* Follow the steps to [set up the Wiz integration](https://docs.elastic.co/en/integrations/wiz). -* Make sure the integration version is at least 2.0.1. -* Ensure you have `read` privileges for the following indices: `security_solution-*.misconfiguration_latest`, `security_solution-*.vulnerability_latest`. -* While configuring the Wiz integration, turn on **Cloud Configuration Finding logs** and **Vulnerability logs**. We recommend you also set the **Initial Interval** values for both settings to `2160h` (equivalent to 90 days) to ingest existing logs. - -:::{image} ../../../images/serverless-wiz-config-finding-logs.png -:alt: Wiz integration settings showing the findings toggle -::: - -:::{image} ../../../images/serverless-wiz-config-vuln-logs.png -:alt: Wiz integration settings showing the vulnerabilities toggle -::: - -After you’ve completed these steps, Wiz data will appear on the **[**Misconfiguations**](../../../solutions/security/cloud/findings-page.md) and [**Vulnerabilities**](../../../solutions/security/cloud/findings-page-3.md) tabs of the **Findings** page. - -:::{image} ../../../images/serverless-wiz-findings.png -:alt: Wiz data on the Findings page -::: - -Any available findings data will also appear in the entity details flyouts for related [alerts](../../../solutions/security/detect-and-alert/view-detection-alert-details.md#insights-section). If alerts are present for a user or host that has findings data from Wiz, the findings will appear on the [users](/solutions/security/explore/users-page.md#user-details-flyout), and [hosts](/solutions/security/explore/hosts-page.md#host-details-flyout) flyouts. diff --git a/raw-migrated-files/docs-content/serverless/security-benchmark-rules-kspm.md b/raw-migrated-files/docs-content/serverless/security-benchmark-rules-kspm.md deleted file mode 100644 index 7490285fa..000000000 --- a/raw-migrated-files/docs-content/serverless/security-benchmark-rules-kspm.md +++ /dev/null @@ -1,48 +0,0 @@ -# Benchmarks [security-benchmark-rules-kspm] - -The Benchmarks page lets you view the cloud security posture (CSP) benchmarks for the [Cloud security posture management](../../../solutions/security/cloud/cloud-security-posture-management.md) (CSPM) and [Kubernetes security posture management](../../../solutions/security/cloud/kubernetes-security-posture-management.md) (KSPM) integrations. - -:::{image} ../../../images/serverless--cloud-native-security-benchmark-rules.png -:alt: Benchmark rules page -:class: screenshot -::: - - -## What are benchmarks? [security-benchmark-rules-what-are-benchmarks-kspm] - -Each benchmark contains benchmark rules which are used by the CSPM and KSPM integrations to identify configuration risks in your cloud infrastructure. There are different benchmarks for different cloud services, such as AWS, GCP, or Azure. They are based on the Center for Internet Security’s (CIS) [secure configuration benchmarks](https://www.cisecurity.org/cis-benchmarks/). - -Each benchmark rule checks to see if a specific type of resource is configured according to a CIS Benchmark. The names of rules describe what they check, for example: - -* `Ensure Kubernetes Secrets are encrypted using Customer Master Keys (CMKs) managed in AWS KMS` -* `Ensure the default namespace is not in use` -* `Ensure IAM policies that allow full "*:*" administrative privileges are not attached` -* `Ensure the default namespace is not in use` - -When benchmark rules are evaluated, the resulting [findings](../../../solutions/security/cloud/findings-page.md) data appears on the [Cloud Security Posture dashboard](../../../solutions/security/dashboards/cloud-security-posture-dashboard.md). - -::::{note} -Benchmark rules are not editable. - -:::: - - - -## Review your benchmarks [security-benchmark-rules-review-your-benchmarks-kspm] - -Find **Benchmarks** in the navigation menu or use the global search field. From there, you can click a benchmark’s name to view the benchmark rules associated with it. You can click a benchmark rule’s name to see details including information about how to remediate it, and related links. - -Benchmark rules are enabled by default, but you can disable some of them — at the benchmark level — to suit your environment. This means for example that if you have two CSPM integrations using the `CIS AWS` benchmark, disabling a rule for that benchmark affects both integrations. To enable or disable a rule, use the **Enabled** toggle on the right of the rules table. - -::::{note} -Disabling a benchmark rule automatically disables any associated detection rules and alerts. Re-enabling a benchmark rule **does not** automatically re-enable them. - -:::: - - - -## How benchmark rules work [security-benchmark-rules-how-benchmark-rules-work-kspm] - -1. When a security posture management integration is deployed, and every four hours after that, {{agent}} fetches relevant cloud resources. -2. After resources are fetched, they are evaluated against all applicable enabled benchmark rules. -3. Finding values of `pass` or `fail` indicate whether the standards defined by benchmark rules were met. diff --git a/raw-migrated-files/docs-content/serverless/security-benchmark-rules.md b/raw-migrated-files/docs-content/serverless/security-benchmark-rules.md deleted file mode 100644 index af1dade04..000000000 --- a/raw-migrated-files/docs-content/serverless/security-benchmark-rules.md +++ /dev/null @@ -1,48 +0,0 @@ -# Benchmarks [security-benchmark-rules] - -The Benchmarks page lets you view the cloud security posture (CSP) benchmarks for the [Cloud security posture management](../../../solutions/security/cloud/cloud-security-posture-management.md) (CSPM) and [Kubernetes security posture management](../../../solutions/security/cloud/kubernetes-security-posture-management.md) (KSPM) integrations. - -:::{image} ../../../images/serverless--cloud-native-security-benchmark-rules.png -:alt: Benchmark rules page -:class: screenshot -::: - - -## What are benchmarks? [security-benchmark-rules-what-are-benchmarks] - -Each benchmark contains benchmark rules which are used by the CSPM and KSPM integrations to identify configuration risks in your cloud infrastructure. There are different benchmarks for different cloud services, such as AWS, GCP, or Azure. They are based on the Center for Internet Security’s (CIS) [secure configuration benchmarks](https://www.cisecurity.org/cis-benchmarks/). - -Each benchmark rule checks to see if a specific type of resource is configured according to a CIS Benchmark. The names of rules describe what they check, for example: - -* `Ensure Kubernetes Secrets are encrypted using Customer Master Keys (CMKs) managed in AWS KMS` -* `Ensure the default namespace is not in use` -* `Ensure IAM policies that allow full "*:*" administrative privileges are not attached` -* `Ensure the default namespace is not in use` - -When benchmark rules are evaluated, the resulting [findings](../../../solutions/security/cloud/findings-page.md) data appears on the [Cloud Security Posture dashboard](../../../solutions/security/dashboards/cloud-security-posture-dashboard.md). - -::::{note} -Benchmark rules are not editable. - -:::: - - - -## Review your benchmarks [security-benchmark-rules-review-your-benchmarks] - -Find **Benchmarks** in the navigation menu or use the global search field. From there, you can click a benchmark’s name to view the benchmark rules associated with it. You can click a benchmark rule’s name to see details including information about how to remediate it, and related links. - -Benchmark rules are enabled by default, but you can disable some of them — at the benchmark level — to suit your environment. This means for example that if you have two CSPM integrations using the `CIS AWS` benchmark, disabling a rule for that benchmark affects both integrations. To enable or disable a rule, use the **Enabled** toggle on the right of the rules table. - -::::{note} -Disabling a benchmark rule automatically disables any associated detection rules and alerts. Re-enabling a benchmark rule **does not** automatically re-enable them. - -:::: - - - -## How benchmark rules work [security-benchmark-rules-how-benchmark-rules-work] - -1. When a security posture management integration is deployed, and every four hours after that, {{agent}} fetches relevant cloud resources. -2. After resources are fetched, they are evaluated against all applicable enabled benchmark rules. -3. Finding values of `pass` or `fail` indicate whether the standards defined by benchmark rules were met. diff --git a/raw-migrated-files/docs-content/serverless/security-cloud-native-security-overview.md b/raw-migrated-files/docs-content/serverless/security-cloud-native-security-overview.md deleted file mode 100644 index 59a209d5f..000000000 --- a/raw-migrated-files/docs-content/serverless/security-cloud-native-security-overview.md +++ /dev/null @@ -1,41 +0,0 @@ -# Cloud Security [security-cloud-native-security-overview] - -Elastic Security for Cloud helps you improve your cloud security posture by comparing your cloud configuration to best practices, and scanning for vulnerabilities. It also helps you monitor and investigate your cloud workloads inside and outside Kubernetes. - -This page describes what each solution does and provides links to more information. - - -## Cloud Security Posture Management (CSPM) [security-cloud-native-security-overview-cloud-security-posture-management-cspm] - -Discovers and evaluates the services in your cloud environment — like storage, compute, IAM, and more — against configuration security guidelines defined by the [Center for Internet Security](https://www.cisecurity.org/) (CIS) to help you identify and remediate risks that could undermine the confidentiality, integrity, and availability of your cloud data. - -[Read the CSPM docs](../../../solutions/security/cloud/cloud-security-posture-management.md). - - -## Kubernetes Security Posture Management (KSPM) [security-cloud-native-security-overview-kubernetes-security-posture-management-kspm] - -Allows you to identify configuration risks in the various components that make up your Kubernetes cluster. It does this by evaluating your Kubernetes clusters against secure configuration guidelines defined by the Center for Internet Security (CIS) and generating findings with step-by-step instructions for remediating potential security risks. - -[Read the KSPM docs](../../../solutions/security/cloud/kubernetes-security-posture-management.md). - - -## Cloud Native Vulnerability Management (CNVM) [security-cloud-native-security-overview-cloud-native-vulnerability-management-cnvm] - -Scans your cloud workloads for known vulnerabilities. When it finds a vulnerability, it supports your risk assessment by quickly providing information such as the vulnerability’s CVSS and severity, which software versions it affects, and whether a fix is available. - -[Read the CNVM docs](../../../solutions/security/cloud/cloud-native-vulnerability-management.md). - - -## Cloud Workload Protection for VMs [security-cloud-native-security-overview-cloud-workload-protection-for-vms] - -Helps you monitor and protect your Linux VMs. It uses {{elastic-defend}} to instantly detect and prevent malicious behavior and malware, and captures workload telemetry data for process, file, and network activity. You can use this data with Elastic’s out-of-the-box detection rules and {{ml}} models. These detections generate alerts that quickly help you identify and remediate threats. - -[Read the CWP for VMs docs](../../../solutions/security/cloud/cloud-workload-protection-for-vms.md). - - - - - - - - diff --git a/raw-migrated-files/docs-content/serverless/security-cloud-posture-dashboard-dash-cspm.md b/raw-migrated-files/docs-content/serverless/security-cloud-posture-dashboard-dash-cspm.md deleted file mode 100644 index 6339805a0..000000000 --- a/raw-migrated-files/docs-content/serverless/security-cloud-posture-dashboard-dash-cspm.md +++ /dev/null @@ -1,49 +0,0 @@ ---- -navigation_title: "Cloud Security Posture" ---- - -# Cloud Security Posture dashboard [security-cloud-posture-dashboard-dash-cspm] - - -The Cloud Security Posture dashboard summarizes your cloud infrastructure’s overall performance against [security guidelines](../../../solutions/security/cloud/benchmarks.md) defined by the Center for Internet Security (CIS). To start collecting this data, refer to [Get started with Cloud Security Posture Management](../../../solutions/security/cloud/get-started-with-cspm-for-aws.md) or [Get started with Kubernetes Security Posture Management](../../../solutions/security/cloud/get-started-with-kspm.md). - -:::{image} ../../../images/serverless--dashboards-cloud-sec-dashboard.png -:alt: The cloud Security dashboard -:class: screenshot -::: - -The Cloud Security Posture dashboard shows: - -* Configuration risk metrics for all monitored cloud accounts and Kubernetes clusters -* Configuration risk metrics grouped by the applicable benchmark, for example, CIS GCP, CIS Azure, CIS Kubernetes, or CIS EKS -* Configuration risks grouped by CIS section (security guideline category) - - -## Cloud Security Posture dashboard UI [cloud-posture-dashboard-UI-cspm] - -At the top of the dashboard, you can switch between the Cloud accounts and Kubernetes cluster views. - -The top section of either view summarizes your overall cloud security posture (CSP) by aggregating data from all monitored resources. The summary cards on the left show the number of cloud accounts or clusters evaluated, and the number of resources evaluated. You can click **Enroll more accounts** or **Enroll more clusters** to deploy to additional cloud assets. Click **View all resources** to open the [Findings page](../../../solutions/security/cloud/findings-page.md). - -The remaining summary cards show your overall compliance score, and your compliance score for each CIS section. Click **View all failed findings** to view all failed findings, or click a CIS section name to view failed findings from only that section on the Findings page. - -Below the summary section, each row shows the CSP for a benchmark that applies to your monitored cloud resources. For example, if you are monitoring GCP and Azure cloud accounts, a row appears for CIS GCP and another appears for CIS Azure. Each row shows the CIS benchmark, the number of cloud accounts or Kubernetes clusters it applies to, its overall compliance score, and its compliance score grouped by CIS section. - -:::{image} ../../../images/serverless--dashboards-cloud-sec-dashboard-individual-row.png -:alt: A row representing a single cluster in the Cloud Security Posture dashboard -:class: screenshot -::: - - -## FAQ (Frequently Asked Questions) [cloud-posture-dashboard-faq-cspm] - -::::{dropdown} When do newly-enrolled clusters appear on the dashboard? -It can take up to 10 minutes for deployment, resource fetching, evaluation, and data processing before a newly-enrolled cluster appears on the dashboard. - -:::: - - -::::{dropdown} When do unenrolled clusters disappear from the dashboard? -A cluster will disappear as soon as the KSPM integration fetches data while that cluster is not enrolled. The fetch process repeats every four hours, which means a newly unenrolled cluster can take a maximum of four hours to disappear from the dashboard. - -:::: diff --git a/raw-migrated-files/docs-content/serverless/security-cloud-posture-dashboard-dash-kspm.md b/raw-migrated-files/docs-content/serverless/security-cloud-posture-dashboard-dash-kspm.md deleted file mode 100644 index 87685ee9e..000000000 --- a/raw-migrated-files/docs-content/serverless/security-cloud-posture-dashboard-dash-kspm.md +++ /dev/null @@ -1,49 +0,0 @@ ---- -navigation_title: "Cloud Security Posture" ---- - -# Cloud Security Posture dashboard [security-cloud-posture-dashboard-dash-kspm] - - -The Cloud Security Posture dashboard summarizes your cloud infrastructure’s overall performance against [security guidelines](../../../solutions/security/cloud/benchmarks.md) defined by the Center for Internet Security (CIS). To start collecting this data, refer to [Get started with Cloud Security Posture Management](../../../solutions/security/cloud/get-started-with-cspm-for-aws.md) or [Get started with Kubernetes Security Posture Management](../../../solutions/security/cloud/get-started-with-kspm.md). - -:::{image} ../../../images/serverless--dashboards-cloud-sec-dashboard.png -:alt: The cloud Security dashboard -:class: screenshot -::: - -The Cloud Security Posture dashboard shows: - -* Configuration risk metrics for all monitored cloud accounts and Kubernetes clusters -* Configuration risk metrics grouped by the applicable benchmark, for example, CIS GCP, CIS Azure, CIS Kubernetes, or CIS EKS -* Configuration risks grouped by CIS section (security guideline category) - - -## Cloud Security Posture dashboard UI [cloud-posture-dashboard-UI-kspm] - -At the top of the dashboard, you can switch between the Cloud accounts and Kubernetes cluster views. - -The top section of either view summarizes your overall cloud security posture (CSP) by aggregating data from all monitored resources. The summary cards on the left show the number of cloud accounts or clusters evaluated, and the number of resources evaluated. You can click **Enroll more accounts** or **Enroll more clusters** to deploy to additional cloud assets. Click **View all resources** to open the [Findings page](../../../solutions/security/cloud/findings-page.md). - -The remaining summary cards show your overall compliance score, and your compliance score for each CIS section. Click **View all failed findings** to view all failed findings, or click a CIS section name to view failed findings from only that section on the Findings page. - -Below the summary section, each row shows the CSP for a benchmark that applies to your monitored cloud resources. For example, if you are monitoring GCP and Azure cloud accounts, a row appears for CIS GCP and another appears for CIS Azure. Each row shows the CIS benchmark, the number of cloud accounts or Kubernetes clusters it applies to, its overall compliance score, and its compliance score grouped by CIS section. - -:::{image} ../../../images/serverless--dashboards-cloud-sec-dashboard-individual-row.png -:alt: A row representing a single cluster in the Cloud Security Posture dashboard -:class: screenshot -::: - - -## FAQ (Frequently Asked Questions) [cloud-posture-dashboard-faq-kspm] - -::::{dropdown} When do newly-enrolled clusters appear on the dashboard? -It can take up to 10 minutes for deployment, resource fetching, evaluation, and data processing before a newly-enrolled cluster appears on the dashboard. - -:::: - - -::::{dropdown} When do unenrolled clusters disappear from the dashboard? -A cluster will disappear as soon as the KSPM integration fetches data while that cluster is not enrolled. The fetch process repeats every four hours, which means a newly unenrolled cluster can take a maximum of four hours to disappear from the dashboard. - -:::: diff --git a/raw-migrated-files/docs-content/serverless/security-cloud-posture-dashboard-dash.md b/raw-migrated-files/docs-content/serverless/security-cloud-posture-dashboard-dash.md deleted file mode 100644 index 037253a99..000000000 --- a/raw-migrated-files/docs-content/serverless/security-cloud-posture-dashboard-dash.md +++ /dev/null @@ -1,49 +0,0 @@ ---- -navigation_title: "Cloud Security Posture" ---- - -# Cloud Security Posture dashboard [security-cloud-posture-dashboard-dash] - - -The Cloud Security Posture dashboard summarizes your cloud infrastructure’s overall performance against [security guidelines](../../../solutions/security/cloud/benchmarks.md) defined by the Center for Internet Security (CIS). To start collecting this data, refer to [Get started with Cloud Security Posture Management](../../../solutions/security/cloud/get-started-with-cspm-for-aws.md) or [Get started with Kubernetes Security Posture Management](../../../solutions/security/cloud/get-started-with-kspm.md). - -:::{image} ../../../images/serverless--dashboards-cloud-sec-dashboard.png -:alt: The cloud Security dashboard -:class: screenshot -::: - -The Cloud Security Posture dashboard shows: - -* Configuration risk metrics for all monitored cloud accounts and Kubernetes clusters -* Configuration risk metrics grouped by the applicable benchmark, for example, CIS GCP, CIS Azure, CIS Kubernetes, or CIS EKS -* Configuration risks grouped by CIS section (security guideline category) - - -## Cloud Security Posture dashboard UI [cloud-posture-dashboard-UI] - -At the top of the dashboard, you can switch between the Cloud accounts and Kubernetes cluster views. - -The top section of either view summarizes your overall cloud security posture (CSP) by aggregating data from all monitored resources. The summary cards on the left show the number of cloud accounts or clusters evaluated, and the number of resources evaluated. You can click **Enroll more accounts** or **Enroll more clusters** to deploy to additional cloud assets. Click **View all resources** to open the [Findings page](../../../solutions/security/cloud/findings-page.md). - -The remaining summary cards show your overall compliance score, and your compliance score for each CIS section. Click **View all failed findings** to view all failed findings, or click a CIS section name to view failed findings from only that section on the Findings page. - -Below the summary section, each row shows the CSP for a benchmark that applies to your monitored cloud resources. For example, if you are monitoring GCP and Azure cloud accounts, a row appears for CIS GCP and another appears for CIS Azure. Each row shows the CIS benchmark, the number of cloud accounts or Kubernetes clusters it applies to, its overall compliance score, and its compliance score grouped by CIS section. - -:::{image} ../../../images/serverless--dashboards-cloud-sec-dashboard-individual-row.png -:alt: A row representing a single cluster in the Cloud Security Posture dashboard -:class: screenshot -::: - - -## FAQ (Frequently Asked Questions) [cloud-posture-dashboard-faq] - -::::{dropdown} When do newly-enrolled clusters appear on the dashboard? -It can take up to 10 minutes for deployment, resource fetching, evaluation, and data processing before a newly-enrolled cluster appears on the dashboard. - -:::: - - -::::{dropdown} When do unenrolled clusters disappear from the dashboard? -A cluster will disappear as soon as the KSPM integration fetches data while that cluster is not enrolled. The fetch process repeats every four hours, which means a newly unenrolled cluster can take a maximum of four hours to disappear from the dashboard. - -:::: diff --git a/raw-migrated-files/docs-content/serverless/security-cloud-workload-protection.md b/raw-migrated-files/docs-content/serverless/security-cloud-workload-protection.md deleted file mode 100644 index 14486b6a4..000000000 --- a/raw-migrated-files/docs-content/serverless/security-cloud-workload-protection.md +++ /dev/null @@ -1,18 +0,0 @@ -# Cloud workload protection for VMs [security-cloud-workload-protection] - -Cloud workload protection helps you monitor and protect your Linux VMs. It uses the [{{elastic-defend}}](../../../solutions/security/configure-elastic-defend/install-elastic-defend.md) integration to capture cloud workload telemetry containing process, file, and network activity. - -Use this telemetry with out-of-the-box detection rules and machine learning models to automate processes that identify cloud threats. - - -## Use cases [security-cloud-workload-protection-use-cases] - -* **Runtime monitoring of cloud workloads:** Provides visibility into cloud workloads, context for detected threats, and the historical data needed for retroactive threat investigations. -* **Cloud-native threat detection and prevention:** Provides security coverage for Linux, containers, and serverless applications. Protects against known and unknown threats using on-host detections and protections against malicious behavior, memory threats, and malware. -* **Reducing the time to detect and remediate runtime threats:** Helps you resolve potential threats by showing alerts in context, making the data necessary for further investigations readily available, and providing remediation options. - -To continue setting up your cloud workload protection, learn more about: - -* [**Getting started with {{elastic-defend}}**](../../../solutions/security/configure-elastic-defend/install-elastic-defend.md): configure {{elastic-defend}} to protect your hosts. Be sure to select one of the "Cloud workloads" presets if you want to collect session data by default, including process, file, and network telemetry. -* [**Session view**](../../../solutions/security/investigate/session-view.md): examine Linux process data organized in a tree-like structure according to the Linux logical event model, with processes organized by parentage and time of execution. Use it to monitor and investigate session activity, and to understand user and service behavior on your Linux infrastructure. -* [**Environment variable capture**](../../../solutions/security/cloud/capture-environment-variables.md): Capture the environment variables associated with process events, such as `PATH`, `LD_PRELOAD`, or `USER`. diff --git a/raw-migrated-files/docs-content/serverless/security-cspm-findings-page-kspm-kspm.md b/raw-migrated-files/docs-content/serverless/security-cspm-findings-page-kspm-kspm.md deleted file mode 100644 index 607f0ee17..000000000 --- a/raw-migrated-files/docs-content/serverless/security-cspm-findings-page-kspm-kspm.md +++ /dev/null @@ -1,79 +0,0 @@ -# Findings page [security-cspm-findings-page-kspm-kspm] - -The **Misconfigurations** tab on the **Findings** page displays the configuration risks identified by the [CSPM](../../../solutions/security/cloud/cloud-security-posture-management.md) and [KSPM](../../../solutions/security/cloud/kubernetes-security-posture-management.md) integrations, as well as data from [third-party integrations](../../../solutions/security/cloud/ingest-third-party-cloud-security-data.md). - -:::{image} ../../../images/serverless--cloud-native-security-findings-page.png -:alt: Findings page -:class: screenshot -::: - - -## What are CSPM and KSPM findings? [cspm-findings-page-what-are-findings-kspm] - -CSPM and KSPM findings indicate whether a given resource passed or failed evaluation against a specific security guideline. Each finding includes metadata about the resource evaluated and the security guideline used to evaluate it. Each finding’s result (`pass` or `fail`) indicates whether a particular part of your infrastructure meets a security guideline. - - -## Group and filter findings [cspm-findings-page-group-filter-kspm] - -By default, the Findings page lists all findings, without grouping or filtering. - - -### Group findings [security-cspm-findings-page-group-findings-kspm] - -Click **Group findings by** to group your data by a field. Select one of the suggested fields or **Custom field** to choose your own. You can select up to three group fields at once. - -* When grouping is turned on, click a group to expand it and examine all sub-groups or findings within that group. -* To turn off grouping, click **Group findings by** and select **None**. - -::::{note} -Multiple groupings apply to your data in the order you selected them. For example, if you first select **Cloud account**, then select **Resource***, the top-level grouping will be based on ***Cloud account**, and its subordinate grouping will be based on **Resource**. - -:::: - - - -### Filter findings [cspm-findings-page-filter-findings-kspm] - -You can filter findings data in two ways: - -* **KQL search bar**: For example, search for `result.evaluation : failed` to view all failed findings. -* **In-table value filters**: Hover over a finding to display available inline actions. Use the **Filter In** (plus) and **Filter Out** (minus) buttons. - - -## Customize the Findings table [security-cspm-findings-page-customize-the-findings-table-kspm] - -You can use the toolbar buttons in the upper-left of the Findings table to select which columns appear: - -* **Columns**: Select the left-to-right order in which columns appear. -* **Sort fields**: Sort the table by one or more columns, or turn sorting off. -* **Fields**: Select which fields to display for each finding. Selected fields appear in the table and the **Columns** menu. - -::::{tip} -You can also click a column’s name to open a menu that allows you to perform multiple actions on the column. - -:::: - - - -## Remediate failed findings [cspm-findings-page-remediate-findings-kspm] - -To remediate failed findings and reduce your attack surface: - -1. First, [filter for failed findings](../../../solutions/security/cloud/findings-page-2.md#cspm-findings-page-filter-findings-kspm). -2. Click the arrow to the left of a failed finding to open the findings flyout. -3. Follow the steps under **Remediation**. - - ::::{note} - Remediation steps typically include commands for you to execute. These sometimes contain placeholder values that you must replace before execution. - - :::: - - - -## Generate alerts for failed Findings [cspm-create-rule-from-finding-kspm] - -You can create detection rules that detect specific failed findings directly from the Findings page. - -1. Click the arrow to the left of a Finding to open the findings flyout. -2. Click **Take action**, then **Create a detection rule**. This automatically creates a detection rule that creates alerts when the associated benchmark rule generates a failed finding. -3. To review or customize the new rule, click **View rule**. diff --git a/raw-migrated-files/docs-content/serverless/security-cspm-findings-page.md b/raw-migrated-files/docs-content/serverless/security-cspm-findings-page.md deleted file mode 100644 index 2a5ecae1f..000000000 --- a/raw-migrated-files/docs-content/serverless/security-cspm-findings-page.md +++ /dev/null @@ -1,79 +0,0 @@ -# Findings page [security-cspm-findings-page] - -The **Misconfigurations** tab on the **Findings** page displays the configuration risks identified by the [CSPM](../../../solutions/security/cloud/cloud-security-posture-management.md) and [KSPM](../../../solutions/security/cloud/kubernetes-security-posture-management.md) integrations, as well as data from [third-party integrations](../../../solutions/security/cloud/ingest-third-party-cloud-security-data.md). - -:::{image} ../../../images/serverless--cloud-native-security-findings-page.png -:alt: Findings page -:class: screenshot -::: - - -## What are CSPM and KSPM findings? [cspm-findings-page-what-are-findings] - -CSPM and KSPM findings indicate whether a given resource passed or failed evaluation against a specific security guideline. Each finding includes metadata about the resource evaluated and the security guideline used to evaluate it. Each finding’s result (`pass` or `fail`) indicates whether a particular part of your infrastructure meets a security guideline. - - -## Group and filter findings [cspm-findings-page-group-filter] - -By default, the Findings page lists all findings, without grouping or filtering. - - -### Group findings [security-cspm-findings-page-group-findings] - -Click **Group findings by** to group your data by a field. Select one of the suggested fields or **Custom field** to choose your own. You can select up to three group fields at once. - -* When grouping is turned on, click a group to expand it and examine all sub-groups or findings within that group. -* To turn off grouping, click **Group findings by** and select **None**. - -::::{note} -Multiple groupings apply to your data in the order you selected them. For example, if you first select **Cloud account**, then select **Resource***, the top-level grouping will be based on ***Cloud account**, and its subordinate grouping will be based on **Resource**. - -:::: - - - -### Filter findings [cspm-findings-page-filter-findings] - -You can filter findings data in two ways: - -* **KQL search bar**: For example, search for `result.evaluation : failed` to view all failed findings. -* **In-table value filters**: Hover over a finding to display available inline actions. Use the **Filter In** (plus) and **Filter Out** (minus) buttons. - - -## Customize the Findings table [security-cspm-findings-page-customize-the-findings-table] - -You can use the toolbar buttons in the upper-left of the Findings table to select which columns appear: - -* **Columns**: Select the left-to-right order in which columns appear. -* **Sort fields**: Sort the table by one or more columns, or turn sorting off. -* **Fields**: Select which fields to display for each finding. Selected fields appear in the table and the **Columns** menu. - -::::{tip} -You can also click a column’s name to open a menu that allows you to perform multiple actions on the column. - -:::: - - - -## Remediate failed findings [cspm-findings-page-remediate-findings] - -To remediate failed findings and reduce your attack surface: - -1. First, [filter for failed findings](../../../solutions/security/cloud/findings-page.md#cspm-findings-page-filter-findings). -2. Click the arrow to the left of a failed finding to open the findings flyout. -3. Follow the steps under **Remediation**. - - ::::{note} - Remediation steps typically include commands for you to execute. These sometimes contain placeholder values that you must replace before execution. - - :::: - - - -## Generate alerts for failed Findings [cspm-create-rule-from-finding] - -You can create detection rules that detect specific failed findings directly from the Findings page. - -1. Click the arrow to the left of a Finding to open the findings flyout. -2. Click **Take action**, then **Create a detection rule**. This automatically creates a detection rule that creates alerts when the associated benchmark rule generates a failed finding. -3. To review or customize the new rule, click **View rule**. diff --git a/raw-migrated-files/docs-content/serverless/security-cspm-get-started-azure.md b/raw-migrated-files/docs-content/serverless/security-cspm-get-started-azure.md deleted file mode 100644 index 68752f4ca..000000000 --- a/raw-migrated-files/docs-content/serverless/security-cspm-get-started-azure.md +++ /dev/null @@ -1,180 +0,0 @@ -# Get started with CSPM for Azure [security-cspm-get-started-azure] - - -## Overview [cspm-overview-azure] - -This page explains how to get started monitoring the security posture of your cloud assets using the Cloud Security Posture Management (CSPM) feature. - -::::{admonition} Requirements -:class: note - -* Minimum privileges vary depending on whether you need to read, write, or manage CSPM data and integrations. Refer to [CSPM privilege requirements](../../../solutions/security/cloud/cspm-privilege-requirements.md). -* CSPM only works in the `Default` {{kib}} space. Installing the CSPM integration on a different {{kib}} space will not work. -* CSPM is supported only on AWS, GCP, and Azure commercial cloud platforms, and AWS GovCloud. Other government cloud platforms are not supported ([request support](https://github.com/elastic/kibana/issues/new/choose)). -* The user who gives the CSPM integration permissions in Azure must be an Azure subscription `admin`. - -:::: - - - -## Set up CSPM for Azure [cspm-setup-azure] - -You can set up CSPM for Azure by by enrolling an Azure organization (management group) containing multiple subscriptions, or by enrolling a single subscription. Either way, first add the CSPM integration, then enable cloud account access. Two deployment technologies are available: agentless, and agent-based. [Agentless deployment](../../../solutions/security/cloud/get-started-with-cspm-for-azure.md#cspm-azure-agentless) allows you to collect cloud posture data without having to manage the deployment of an agent in your cloud. [Agent-based deployment](../../../solutions/security/cloud/get-started-with-cspm-for-azure.md#cspm-azure-agent-based) requires you to deploy and manage an agent in the cloud account you want to monitor. - - -## Agentless deployment [cspm-azure-agentless] - -[beta] - -1. Find **Integrations** in the navigation menu or use the global search field. -2. Search for `CSPM`, then click on the result. -3. Click **Add Cloud Security Posture Management (CSPM)**. -4. Select **Azure**, then either **Azure Organization** to onboard your whole organization, or **Single Subscription** to onboard an individual subscription. -5. Give your integration a name that matches the purpose or team of the Azure subscription/organization you want to monitor, for example, `dev-azure-account`. -6. Click **Advanced options**, then select **Agentless (BETA)**. -7. Next, you’ll need to authenticate to Azure by providing a **Client ID**, **Tenant ID**, and **Client Secret**. To learn how to generate them, refer to [Service principal with client secret](../../../solutions/security/cloud/get-started-with-cspm-for-azure.md#cspm-azure-client-secret). -8. Once you’ve provided the necessary credentials, click **Save and continue** to finish deployment. Your data should start to appear within a few minutes. - - -## Agent-based deployment [cspm-azure-agent-based] - - -### Add your CSPM integration [cspm-add-and-name-integration-azure] - -1. Find **Integrations** in the navigation menu or use the global search field. -2. Search for `CSPM`, then click on the result. -3. Click **Add Cloud Security Posture Management (CSPM)**. -4. Under **Configure integration**, select **Azure***, then select either ***Azure Organization** or **Single Subscription**, depending on which resources you want to monitor. -5. Give your integration a name that matches the purpose or team of the Azure resources you want to monitor, for example, `azure-CSPM-dev-1`. - - -### Set up cloud account access [cspm-set-up-cloud-access-section-azure] - -To set up CSPM for an Azure organization or subscription, you will need admin privileges for that organization or subscription. - -For most users, the simplest option is to use an Azure Resource Manager (ARM) template to automatically provision the necessary resources and permissions in Azure. If you prefer a more hands-on approach or require a specific configuration not supported by the ARM template, you can use one of the manual setup options described below. - - -### ARM template setup (recommended) [cspm-set-up-ARM] - -1. Under **Setup Access**, select **ARM Template**. -2. Under **Where to add this integration**: - - 1. Select **New Hosts**. - 2. Name the {{agent}} policy. Use a name that matches the resources you want to monitor, for example, `azure-dev-policy`. Click **Save and continue**. The **ARM Template deployment** window appears. - 3. In a new tab, log in to the Azure portal, then return to {{kib}} and click **Launch ARM Template**. This will open the ARM template in Azure. - 4. If you are deploying to an Azure organization, select the management group you want to monitor from the drop-down menu. Next, enter the subscription ID of the subscription where you want to deploy the VM that will scan your resources. - 5. Copy the `Fleet URL` and `Enrollment Token` that appear in {{kib}} to the corresponding fields in the ARM Template, then click **Review + create**. - 6. (Optional) Change the `Resource Group Name` parameter. Otherwise, the name of the resource group defaults to a timestamp prefixed with `cloudbeat-`. - -3. Return to {{kib}} and wait for the confirmation of data received from your new integration. Then you can click **View Assets** to see your data. - - -### Manual setup [cspm-set-up-manual-azure] - -For manual setup, multiple authentication methods are available: - -1. Managed identity (recommended) -2. Service principal with client secret -3. Service principal with client certificate - - -### Option 1: Managed identity (recommended) [cspm-azure-managed-identity-setup] - -This method involves creating an Azure VM (or using an existing one), giving it read access to the resources you want to monitor with CSPM, and installing {{agent}} on it. - -1. Go to the Azure portal to create a new Azure VM. -2. Follow the setup process, and make sure you enable **System assigned managed identity** under the **Management** tab. -3. Go to your Azure subscription list and select the subscription or management group you want to monitor with CSPM. -4. Go to **Access control (IAM)**, and select **Add Role Assignment**. -5. Select the `Reader` role, assign access to **Managed Identity**, then select your VM. - -After assigning the role: - -1. Return to the **Add CSPM** page in {{kib}}. -2. Under **Configure integration**, select **Azure***. Under ***Setup access**, select **Manual**. -3. Under **Where to add this integration**, select **New hosts**. -4. Click **Save and continue**, then follow the instructions to install {{agent}} on your Azure VM. - -Wait for the confirmation that {{kib}} received data from your new integration. Then you can click **View Assets** to see your data. - - -### Option 2: Service principal with client secret [cspm-azure-client-secret] - -Before using this method, you must have set up a [Microsoft Entra application and service principal that can access resources](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal#get-tenant-and-app-id-values-for-signing-in). - -1. On the **Add Cloud Security Posture Management (CSPM) integration** page, scroll to the **Setup access** section, then select **Manual**. -2. Under **Preferred manual method**, select **Service principal with Client Secret**. -3. Go to the **Registered apps** section of [Microsoft Entra ID](https://ms.portal.azure.com/#view/Microsoft_AAD_IAM/ActiveDirectoryMenuBlade/~/RegisteredApps). -4. Click on **New Registration**, name your app and click **Register**. -5. Copy your new app’s `Directory (tenant) ID` and `Application (client) ID`. Paste them into the corresponding fields in {{kib}}. -6. Return to the Azure portal. Select **Certificates & secrets**, then go to the **Client secrets** tab. Click **New client secret**. -7. Copy the new secret. Paste it into the corresponding field in {{kib}}. -8. Return to Azure. Go to your Azure subscription list and select the subscription or management group you want to monitor with CSPM. -9. Go to **Access control (IAM)** and select **Add Role Assignment**. -10. Select the `Reader` function role, assign access to **User, group, or service principal**, and select your new app. -11. Return to the **Add CSPM** page in {{kib}}. -12. Under **Where to add this integration**, select **New hosts**. -13. Click **Save and continue**, then follow the instructions to install {{agent}} on your selected host. - -Wait for the confirmation that {{kib}} received data from your new integration. Then you can click **View Assets** to see your data. - - -### Option 3: Service principal with client certificate [cspm-azure-client-certificate] - -Before using this method, you must have set up a [Microsoft Entra application and service principal that can access resources](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal#get-tenant-and-app-id-values-for-signing-in). - -1. On the **Add Cloud Security Posture Management (CSPM) integration** page, under **Setup access**, select **Manual**. -2. Under **Preferred manual method**, select **Service principal with client certificate**. -3. Go to the **Registered apps** section of [Microsoft Entra ID](https://ms.portal.azure.com/#view/Microsoft_AAD_IAM/ActiveDirectoryMenuBlade/~/RegisteredApps). -4. Click on **New Registration**, name your app and click **Register**. -5. Copy your new app’s `Directory (tenant) ID` and `Application (client) ID`. Paste them into the corresponding fields in {{kib}}. -6. Return to Azure. Go to your Azure subscription list and select the subscription or management group you want to monitor with CSPM. -7. Go to **Access control (IAM)** and select **Add Role Assignment**. -8. Select the `Reader` function role, assign access to **User, group, or service principal**, and select your new app. - -Next, create a certificate. If you intend to use a password-protected certificate, you must use a pkcs12 certificate. Otherwise, you must use a pem certificate. - -Create a pkcs12 certificate, for example: - -```shell -# Create PEM file -openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes - -# Create pkcs12 bundle using legacy flag (CLI will ask for export password) -openssl pkcs12 -legacy -export -out bundle.p12 -inkey key.pem -in cert.pem -``` - -Create a PEM certificate, for example: - -```shell -# Generate certificate signing request (csr) and key -openssl req -new -newkey rsa:4096 -nodes -keyout cert.key -out cert.csr - -# Generate PEM and self-sign with key -openssl x509 -req -sha256 -days 365 -in cert.csr -signkey cert.key -out signed.pem - -# Create bundle -cat cert.key > bundle.pem -cat signed.pem >> bundle.pem -``` - -1. Return to Azure. -2. Navigate to the **Certificates & secrets** menu. Select the **Certificates** tab. -3. Click **Upload certificate**. - - 1. If you’re using a PEM certificate that was created using the example commands above, upload `signed.pem`. - 2. If you’re using a pkcs12 certificate that was created using the example commands above, upload `cert.pem`. - -4. Upload the certificate bundle to the VM where you will deploy {{agent}}. - - 1. If you’re using a PEM certificate that was created using the example commands above, upload `bundle.pem`. - 2. If you’re using a pkcs12 certificate that was created using the example commands above, upload `bundle.p12`. - -5. Return to the **Add CSPM** page in {{kib}}. -6. For **Client Certificate Path**, enter the full path to the certificate that you uploaded to the host where you will install {{agent}}. -7. If you used a pkcs12 certificate, enter its password under **Client Certificate Password**. -8. Under **Where to add this integration**, select **New hosts**. -9. Click **Save and continue**, then follow the instructions to install {{agent}} on your selected host. - -Wait for the confirmation that {{kib}} received data from your new integration. Then you can click **View Assets** to see your data. diff --git a/raw-migrated-files/docs-content/serverless/security-cspm-get-started-gcp.md b/raw-migrated-files/docs-content/serverless/security-cspm-get-started-gcp.md deleted file mode 100644 index fbe4045c2..000000000 --- a/raw-migrated-files/docs-content/serverless/security-cspm-get-started-gcp.md +++ /dev/null @@ -1,186 +0,0 @@ -# Get started with CSPM for GCP [security-cspm-get-started-gcp] - - -## Overview [cspm-overview-gcp] - -This page explains how to get started monitoring the security posture of your cloud assets using the Cloud Security Posture Management (CSPM) feature. - -::::{admonition} Requirements -:class: note - -* Minimum privileges vary depending on whether you need to read, write, or manage CSPM data and integrations. Refer to [CSPM privilege requirements](../../../solutions/security/cloud/cspm-privilege-requirements.md). -* CSPM only works in the `Default` {{kib}} space. Installing the CSPM integration on a different {{kib}} space will not work. -* CSPM is supported only on AWS, GCP, and Azure commercial cloud platforms, and AWS GovCloud. Other government cloud platforms are not supported ([request support](https://github.com/elastic/kibana/issues/new/choose)). -* The user who gives the CSPM integration GCP permissions must be a GCP project `admin`. - -:::: - - - -## Set up CSPM for GCP [cspm-setup-gcp] - -You can set up CSPM for GCP either by enrolling a single project, or by enrolling an organization containing multiple projects. Either way, you need to first add the CSPM integration, then enable cloud account access. Two deployment technologies are available: agentless, and agent-based. [Agentless deployment](../../../solutions/security/cloud/get-started-with-cspm-for-gcp.md#cspm-gcp-agentless) allows you to collect cloud posture data without having to manage the deployment of an agent in your cloud. [Agent-based deployment](../../../solutions/security/cloud/get-started-with-cspm-for-gcp.md#cspm-gcp-agent-based) requires you to deploy and manage an agent in the cloud account you want to monitor. - - -## Agentless deployment [cspm-gcp-agentless] - -[beta] - -1. Find **Integrations** in the navigation menu or use the global search field. -2. Search for `CSPM`, then click on the result. -3. Click **Add Cloud Security Posture Management (CSPM)**. -4. Select **GCP**, then either **GCP Organization** to onboard your whole organization, or **Single Account** to onboard an individual account. -5. Give your integration a name that matches the purpose or team of the GCP subscription/organization you want to monitor, for example, `dev-gcp-account`. -6. Click **Advanced options**, then select **Agentless (BETA)**. -7. Next, you’ll need to authenticate to GCP. Expand the **Steps to Generate GCP Account Credentials** section, then follow the instructions that appear to automatically create the necessary credentials using Google Cloud Shell. -8. Once you’ve provided the necessary credentials, click **Save and continue** to finish deployment. Your data should start to appear within a few minutes. - - -## Agent-based deployment [cspm-gcp-agent-based] - - -### Add your CSPM integration [cspm-add-and-name-integration-gcp] - -1. Find **Integrations** in the navigation menu or use the global search field. -2. Search for `CSPM`, then click on the result. -3. Click **Add Cloud Security Posture Management (CSPM)**. -4. Under **Configure integration**, select **GCP***, then either ***GCP Organization** (recommended) or **Single Account**. -5. Give your integration a name that matches the purpose or team of the GCP account you want to monitor, for example, `dev-gcp-project`. - - -### Set up cloud account access [cspm-set-up-cloud-access-section-gcp] - -To set up CSPM for a GCP project, you need admin privileges for the project. - -For most users, the simplest option is to use a Google Cloud Shell script to automatically provision the necessary resources and permissions in your GCP account. This method, as well as two manual options, are described below. - - -## Cloud Shell script setup (recommended) [cspm-set-up-cloudshell] - -1. Under **Setup Access**, select **Google Cloud Shell**. Enter your GCP Project ID, and for GCP Organization deployments, your GCP Organization ID. -2. Under **Where to add this integration**: - - 1. Select **New Hosts**. - 2. Name the {{agent}} policy. Use a name that matches the purpose or team of the cloud account or accounts you want to monitor. For example, `dev-gcp-account`. - 3. Click **Save and continue**, then **Add {{agent}} to your hosts**. The **Add agent** wizard appears and provides {{agent}} binaries, which you can download and deploy to a VM in your GCP account. - -3. Click **Save and continue**. -4. Copy the command that appears, then click **Launch Google Cloud Shell**. It opens in a new window. -5. Check the box to trust Elastic’s `cloudbeat` repo, then click **Confirm** - - :::{image} ../../../images/serverless--cloud-native-security-cspm-cloudshell-trust.png - :alt: The cloud shell confirmation popup - :class: screenshot - ::: - -6. In Google Cloud Shell, execute the command you copied. Once it finishes, return to {{kib}} and wait for the confirmation of data received from your new integration. Then you can click **View Assets** to see your data. - -::::{note} -During Cloud Shell setup, the CSPM integration adds roles to Google’s default service account, which enables custom role creation and attachment of the service account to a compute instance. After setup, these roles are removed from the service account. If you attempt to delete the deployment but find the deployment manager lacks necessary permissions, consider adding the missing roles to the service account: [Project IAM Admin](https://cloud.google.com/iam/docs/understanding-roles#resourcemanager.projectIamAdmin), [Role Administrator](https://cloud.google.com/iam/docs/understanding-roles#iam.roleAdmin). - -:::: - - - -## Manual authentication (GCP organization) [cspm-manual-auth-org] - -To authenticate manually to monitor a GCP organization, you’ll need to create a new GCP service account, assign it the necessary roles, generate credentials, then provide those credentials to the CSPM integration. - -Use the following commands, after replacing `` with the name of your new service account, `` with your GCP organization’s ID, and `` with the GCP project ID of the project where you want to provision the compute instance that will run CSPM. - -Create a new service account: - -```shell -gcloud iam service-accounts create \ - --description="Elastic agent service account for CSPM" \ - --display-name="Elastic agent service account for CSPM" \ - --project= -``` - -Assign the necessary roles to the service account: - -```shell -gcloud organizations add-iam-policy-binding \ - --member=serviceAccount:@.iam.gserviceaccount.com \ - --role=roles/cloudasset.viewer - -gcloud organizations add-iam-policy-binding \ - --member=serviceAccount:@.iam.gserviceaccount.com \ - --role=roles/browser -``` - -The `Cloud Asset Viewer` role grants read access to cloud asset metadata. The `Browser` role grants read access to the project hierarchy. - -Download the credentials JSON (first, replace `` with the location where you want to save it): - -```shell -gcloud iam service-accounts keys create \ - --iam-account=@.iam.gserviceaccount.com -``` - -Keep the credentials JSON in a secure location; you will need it later. - -Provide credentials to the CSPM integration: - -1. On the CSPM setup screen under **Setup Access**, select **Manual**. -2. Enter your GCP **Organization ID**. Enter the GCP **Project ID** of the project where you want to provision the compute instance that will run CSPM. -3. Select **Credentials JSON**, and enter the value you generated earlier. -4. Under **Where to add this integration**, select **New Hosts**. -5. Name the {{agent}} policy. Use a name that matches the purpose or team of the cloud account or accounts you want to monitor. For example, `dev-gcp-account`. -6. Click **Save and continue**, then follow the instructions to install {{agent}} in your chosen GCP project. - -Wait for the confirmation that {{kib}} received data from your new integration. Then you can click **View Assets** to see your data. - - -## Manual authentication (GCP project) [cspm-manual-auth-proj] - -To authenticate manually to monitor an individual GCP project, you’ll need to create a new GCP service account, assign it the necessary roles, generate credentials, then provide those credentials to the CSPM integration. - -Use the following commands, after replacing `` with the name of your new service account, and `` with your GCP project ID. - -Create a new service account: - -```shell -gcloud iam service-accounts create \ - --description="Elastic agent service account for CSPM" \ - --display-name="Elastic agent service account for CSPM" \ - --project= -``` - -Assign the necessary roles to the service account: - -```shell -gcloud projects add-iam-policy-binding \ - --member=serviceAccount:@.iam.gserviceaccount.com \ - --role=roles/cloudasset.viewer - -gcloud projects add-iam-policy-binding \ - --member=serviceAccount:@.iam.gserviceaccount.com \ - --role=roles/browser -``` - -::::{note} -The `Cloud Asset Viewer` role grants read access to cloud asset metadata. The `Browser` role grants read access to the project hierarchy. - -:::: - - -Download the credentials JSON (first, replace `` with the location where you want to save it): - -```shell -gcloud iam service-accounts keys create \ - --iam-account=@.iam.gserviceaccount.com -``` - -Keep the credentials JSON in a secure location; you will need it later. - -Provide credentials to the CSPM integration: - -1. On the CSPM setup screen under **Setup Access**, select **Manual**. -2. Enter your GCP **Project ID**. -3. Select **Credentials JSON**, and enter the value you generated earlier. -4. Under **Where to add this integration**, select **New Hosts**. -5. Name the policy. Use a name that matches the purpose or team of the cloud account or accounts you want to monitor. For example, `dev-gcp-account`. -6. Click **Save and continue**, then follow the instructions to install the agent in your chosen GCP project. - -Wait for the confirmation that Kibana received data from your new integration. Then you can click **View Assets** to see your data. diff --git a/raw-migrated-files/docs-content/serverless/security-cspm-get-started.md b/raw-migrated-files/docs-content/serverless/security-cspm-get-started.md deleted file mode 100644 index 3abcc1075..000000000 --- a/raw-migrated-files/docs-content/serverless/security-cspm-get-started.md +++ /dev/null @@ -1,352 +0,0 @@ -# Get started with CSPM for AWS [security-cspm-get-started] - - -## Overview [cspm-overview] - -This page explains how to get started monitoring the security posture of your cloud assets using the Cloud Security Posture Management (CSPM) feature. - -::::{admonition} Requirements -:class: note - -* Minimum privileges vary depending on whether you need to read, write, or manage CSPM data and integrations. Refer to [CSPM privilege requirements](../../../solutions/security/cloud/cspm-privilege-requirements.md). -* CSPM only works in the `Default` {{kib}} space. Installing the CSPM integration on a different {{kib}} space will not work. -* CSPM is supported only on AWS, GCP, and Azure commercial cloud platforms, and AWS GovCloud. Other government cloud platforms are not supported ([request support](https://github.com/elastic/kibana/issues/new/choose)). -* The user who gives the CSPM integration AWS permissions must be an AWS account `admin`. - -:::: - - - -## Set up CSPM for AWS [cspm-setup] - -You can set up CSPM for AWS either by enrolling a single cloud account, or by enrolling an organization containing multiple accounts. Either way, first you will add the CSPM integration, then enable cloud account access. Two deployment technologies are available: agentless, and agent-based. [Agentless deployment](../../../solutions/security/cloud/get-started-with-cspm-for-aws.md#cspm-aws-agentless) allows you to collect cloud posture data without having to manage the deployment of an {{agent}} in your cloud. [Agent-based deployment](../../../solutions/security/cloud/get-started-with-cspm-for-aws.md#cspm-aws-agent-based) requires you to deploy and manage an {{agent}} in the cloud account you want to monitor. - - -## Agentless deployment [cspm-aws-agentless] - -[beta] - -1. Find **Integrations** in the navigation menu or use the global search field. -2. Search for `CSPM`, then click on the result. -3. Click **Add Cloud Security Posture Management (CSPM)**. -4. Select **AWS**, then either **AWS Organization** to onboard multiple accounts, or **Single Account** to onboard an individual account. -5. Give your integration a name that matches the purpose or team of the AWS account/organization you want to monitor, for example, `dev-aws-account`. -6. Click **Advanced options**, then select **Agentless (BETA)**. -7. Next, you’ll need to authenticate to AWS. Two methods are available: - - 1. Option 1: Direct access keys/CloudFormation (Recommended). Under **Preferred method** select **Direct access keys**. Expand the **Steps to Generate AWS Account Credentials** section, then follow the displayed instructions to automatically create the necessary credentials using CloudFormation. - - ::::{note} - If you don’t want to monitor every account in your organization, specify which to monitor using the `OrganizationalUnitIDs` field that appears after you click **Launch CloudFormation**. - :::: - - 2. Option 2: Temporary keys. To authenticate using temporary keys, refer to the instructions for [Temporary keys](../../../solutions/security/cloud/get-started-with-cspm-for-aws.md#cspm-use-temp-credentials). - -8. Once you’ve selected an authentication method and provided all necessary credentials, click **Save and continue** to finish deployment. Your data should start to appear within a few minutes. - - -## Agent-based deployment [cspm-aws-agent-based] - - -### Add the CSPM integration [cspm-add-and-name-integration] - -1. Find **Integrations** in the navigation menu or use the global search field. -2. Search for `CSPM`, then click on the result. -3. Click **Add Cloud Security Posture Management (CSPM)**. -4. Select **AWS**, then either **AWS Organization** to onboard multiple accounts, or **Single Account** to onboard an individual account. -5. Give your integration a name that matches the purpose or team of the AWS account/organization you want to monitor, for example, `dev-aws-account`. - - -### Set up cloud account access [cspm-set-up-cloud-access-section] - -The CSPM integration requires access to AWS’s built-in [`SecurityAudit` IAM policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_job-functions.md#jf_security-auditor) in order to discover and evaluate resources in your cloud account. There are several ways to provide access. - -For most use cases, the simplest option is to use AWS CloudFormation to automatically provision the necessary resources and permissions in your AWS account. This method, as well as several manual options, are described below. - - -### CloudFormation (recommended) [cspm-set-up-cloudformation] - -1. In the **Add Cloud Security Posture Management (CSPM) integration** menu, under **Setup Access**, select **CloudFormation**. -2. In a new browser tab or window, log in as an admin to the AWS account or organization you want to onboard. -3. Return to your {{kib}} tab. Click **Save and continue** at the bottom of the page. -4. Review the information, then click **Launch CloudFormation**. -5. A CloudFormation template appears in a new browser tab. -6. For organization-level deployments only, you must enter the ID of the organizational units where you want to deploy into the CloudFormation template’s `OrganizationalUnitIds` field. You can find organizational unit IDs in the AWS console under **AWS Organizations → AWS Accounts** (under each organization’s name). You can also use this field to specify which accounts in your organization to monitor, and which to skip. -7. (Optional) Switch to the AWS region where you want to deploy using the controls in the upper right corner. -8. Tick the checkbox under **Capabilities** to authorize the creation of necessary resources. - - :::{image} ../../../images/serverless--cloud-native-security-cspm-cloudformation-template.png - :alt: The Add permissions screen in AWS - :class: screenshot - ::: - -9. At the bottom of the template, select **Create stack**. - -When you return to {{kib}}, click **View assets** to review the data being collected by your new integration. - - -### Manual authentication for organization-level onboarding [cspm-setup-organization-manual] - -::::{note} -If you’re onboarding a single account instead of an organization, skip this section. - -:::: - - -When using manual authentication to onboard at the organization level, you need to configure the necessary permissions using the AWS console for the organization where you want to deploy: - -* In the organization’s management account (root account), create an IAM role called `cloudbeat-root` (the name is important). The role needs several policies: - - * The following inline policy: - - ::::{dropdown} Click to expand policy - ```json - { - "Version": "2012-10-17", - "Statement": [ - { - "Action": [ - "organizations:List*", - "organizations:Describe*" - ], - "Resource": "*", - "Effect": "Allow" - }, - { - "Action": [ - "sts:AssumeRole" - ], - "Resource": "*", - "Effect": "Allow" - } - ] - } - ``` - - :::: - - * The following trust policy: - - ::::{dropdown} Click to expand policy - ```json - { - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Principal": { - "AWS": "arn:aws:iam:::root" - }, - "Action": "sts:AssumeRole" - }, - { - "Effect": "Allow", - "Principal": { - "Service": "ec2.amazonaws.com" - }, - "Action": "sts:AssumeRole" - } - ] - } - ``` - - :::: - - * The AWS-managed `SecurityAudit` policy. - - -::::{important} -You must replace `` in the trust policy with your AWS account ID. - -:::: - - -* Next, for each account you want to scan in the organization, create an IAM role named `cloudbeat-securityaudit` with the following policies: - - * The AWS-managed `SecurityAudit` policy. - * The following trust policy: - - ::::{dropdown} Click to expand policy - ```json - { - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Principal": { - "AWS": "arn:aws:iam:::role/cloudbeat-root" - }, - "Action": "sts:AssumeRole" - } - ] - } - ``` - - :::: - - -::::{important} -You must replace `` in the trust policy with your AWS account ID. - -:::: - - -After creating the necessary roles, authenticate using one of the manual authentication methods. - -::::{important} -When deploying to an organization using any of the authentication methods below, you need to make sure that the credentials you provide grant permission to assume `cloudbeat-root` privileges. - -:::: - - - -### Manual authentication methods [cspm-set-up-manual] - -* [Default instance role (recommended)](../../../solutions/security/cloud/get-started-with-cspm-for-aws.md#cspm-use-instance-role) -* [Direct access keys](../../../solutions/security/cloud/get-started-with-cspm-for-aws.md#cspm-use-keys-directly) -* [Temporary security credentials](../../../solutions/security/cloud/get-started-with-cspm-for-aws.md#cspm-use-temp-credentials) -* [Shared credentials file](../../../solutions/security/cloud/get-started-with-cspm-for-aws.md#cspm-use-a-shared-credentials-file) -* [IAM role Amazon Resource Name (ARN)](../../../solutions/security/cloud/get-started-with-cspm-for-aws.md#cspm-use-iam-arn) - -::::{important} -Whichever method you use to authenticate, make sure AWS’s built-in [`SecurityAudit` IAM policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_job-functions.md#jf_security-auditor) is attached. - -:::: - - - -#### Option 1 - Default instance role [cspm-use-instance-role] - -::::{note} -If you are deploying to an AWS organization instead of an AWS account, you should already have [created a new role](../../../solutions/security/cloud/get-started-with-cspm-for-aws.md#cspm-setup-organization-manual), `cloudbeat-root`. Skip to step 2 "Attach your new IAM role to an EC2 instance", and attach this role. You can use either an existing or new EC2 instance. - -:::: - - -Follow AWS’s [IAM roles for Amazon EC2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.md) documentation to create an IAM role using the IAM console, which automatically generates an instance profile. - -1. Create an IAM role: - - 1. In AWS, go to your IAM dashboard. Click **Roles**, then **Create role**. - 2. On the **Select trusted entity** page, under **Trusted entity type**, select **AWS service**. - 3. Under **Use case**, select **EC2**. Click **Next**. - - :::{image} ../../../images/serverless--cloud-native-security-cspm-aws-auth-1.png - :alt: The Select trusted entity screen in AWS - :class: screenshot - ::: - - 4. On the **Add permissions** page, search for and select `SecurityAudit`. Click **Next**. - - :::{image} ../../../images/serverless--cloud-native-security-cspm-aws-auth-2.png - :alt: The Add permissions screen in AWS - :class: screenshot - ::: - - 5. On the **Name, review, and create** page, name your role, then click **Create role**. - -2. Attach your new IAM role to an EC2 instance: - - 1. In AWS, select an EC2 instance. - 2. Select **Actions → Security → Modify IAM role**. - - :::{image} ../../../images/serverless--cloud-native-security-cspm-aws-auth-3.png - :alt: The EC2 page in AWS - :class: screenshot - ::: - - 3. On the **Modify IAM role** page, search for and select your new IAM role. - 4. Click **Update IAM role**. - 5. Return to {{kib}} and [finish manual setup](../../../solutions/security/cloud/get-started-with-cspm-for-aws.md#cspm-finish-manual). - - -::::{important} -Make sure to deploy the CSPM integration to this EC2 instance. When completing setup in {{kib}}, in the **Setup Access** section, select **Assume role***. Leave ***Role ARN** empty for agentless deployments. For agent-based deployments, leave it empty unless you want to specify a role the {{agent}} should assume instead of the default role for your EC2 instance. Click **Save and continue**. - -:::: - - - -#### Option 2 - Direct access keys [cspm-use-keys-directly] - -Access keys are long-term credentials for an IAM user or AWS account root user. To use access keys as credentials, you must provide the `Access key ID` and the `Secret Access Key`. After you provide credentials, [finish manual setup](../../../solutions/security/cloud/get-started-with-cspm-for-aws.md#cspm-finish-manual). - -For more details, refer to [Access Keys and Secret Access Keys](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.md). - -::::{important} -You must select **Programmatic access** when creating the IAM user. - -:::: - - - -#### Option 3 - Temporary security credentials [cspm-use-temp-credentials] - -You can configure temporary security credentials in AWS to last for a specified duration. They consist of an access key ID, a secret access key, and a session token, which is typically found using `GetSessionToken`. - -Because temporary security credentials are short term, once they expire, you will need to generate new ones and manually update the integration’s configuration to continue collecting cloud posture data. Update the credentials before they expire to avoid data loss. - -::::{note} -IAM users with multi-factor authentication (MFA) enabled need to submit an MFA code when calling `GetSessionToken`. For more details, refer to AWS’s [Temporary Security Credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.md) documentation. - -:::: - - -You can use the AWS CLI to generate temporary credentials. For example, you could use the following command if you have MFA enabled: - -```console -sts get-session-token --serial-number arn:aws:iam::1234:mfa/your-email@example.com --duration-seconds 129600 --token-code 123456 -``` - -The output from this command includes the following fields, which you should provide when configuring the KSPM integration: - -* `Access key ID`: The first part of the access key. -* `Secret Access Key`: The second part of the access key. -* `Session Token`: The required token when using temporary security credentials. - -After you provide credentials, [finish manual setup](../../../solutions/security/cloud/get-started-with-cspm-for-aws.md#cspm-finish-manual). - - -#### Option 4 - Shared credentials file [cspm-use-a-shared-credentials-file] - -If you use different AWS credentials for different tools or applications, you can use profiles to define multiple access keys in the same configuration file. For more details, refer to AWS' [Shared Credentials Files](https://docs.aws.amazon.com/sdkref/latest/guide/file-format.md) documentation. - -Instead of providing the `Access key ID` and `Secret Access Key` to the integration, provide the information required to locate the access keys within the shared credentials file: - -* `Credential Profile Name`: The profile name in the shared credentials file. -* `Shared Credential File`: The directory of the shared credentials file. - -If you don’t provide values for all configuration fields, the integration will use these defaults: - -* If `Access key ID`, `Secret Access Key`, and `ARN Role` are not provided, then the integration will check for `Credential Profile Name`. -* If there is no `Credential Profile Name`, the default profile will be used. -* If `Shared Credential File` is empty, the default directory will be used. - - * For Linux or Unix, the shared credentials file is located at `~/.aws/credentials`. - - -After providing credentials, [finish manual setup](../../../solutions/security/cloud/get-started-with-cspm-for-aws.md#cspm-finish-manual). - - -#### Option 5 - IAM role Amazon Resource Name (ARN) [cspm-use-iam-arn] - -An IAM role Amazon Resource Name (ARN) is an IAM identity that you can create in your AWS account. You define the role’s permissions. Roles do not have standard long-term credentials such as passwords or access keys. Instead, when you assume a role, it provides temporary security credentials for your session. - -To use an IAM role ARN, select **Assume role** under **Preferred manual method**, enter the ARN, and continue to Finish manual setup. - - -### Finish manual setup [cspm-finish-manual] - -Once you’ve provided AWS credentials, under **Where to add this integration**: - -If you want to monitor an AWS account or organization where you have not yet deployed {{agent}}: - -* Select **New Hosts**. -* Name the {{agent}} policy. Use a name that matches the purpose or team of the cloud account or accounts you want to monitor. For example, `dev-aws-account`. -* Click **Save and continue**, then **Add {{agent}} to your hosts**. The **Add agent** wizard appears and provides {{agent}} binaries, which you can download and deploy to your AWS account. - -If you want to monitor an AWS account or organization where you have already deployed {{agent}}: - -* Select **Existing hosts**. -* Select an agent policy that applies the AWS account you want to monitor. -* Click **Save and continue**. diff --git a/raw-migrated-files/docs-content/serverless/security-cspm-security-posture-faq.md b/raw-migrated-files/docs-content/serverless/security-cspm-security-posture-faq.md deleted file mode 100644 index 7fcd4812c..000000000 --- a/raw-migrated-files/docs-content/serverless/security-cspm-security-posture-faq.md +++ /dev/null @@ -1,69 +0,0 @@ -# Frequently asked questions (FAQ) [security-cspm-security-posture-faq] - - -## CSPM FAQ [cspm-security-posture-faq] - -Frequently asked questions about the Cloud Security Posture Management (CSPM) integration and features. - -**How often is my cloud security posture evaluated?** - -Cloud accounts are evaluated when you first deploy the CSPM integration and every 24 hours afterward. - -**Can I onboard multiple accounts at one time?** - -Yes. Follow the onboarding instructions in the getting started guides for AWS, GCP, or Azure. - -**When do newly enrolled cloud accounts appear on the dashboard?** - -After you deploy the CSPM integration, it can take up to 10 minutes for resource fetching, evaluation, and data processing before a newly enrolled account appears on the Cloud Security Posture dashboard. - -**When do unenrolled cloud accounts disappear from the dashboard?** - -Newly unenrolled cloud accounts can take a maximum of 24 hours to disappear from the Cloud Security Posture dashboard. - - -## KSPM FAQ [security-cspm-security-posture-faq-kspm-faq] - -Frequently asked questions about the Kubernetes Security Posture Management (KSPM) integration and features. - -**What versions of Kubernetes are supported?** - -For self-managed/vanilla clusters, Kubernetes version 1.23 is supported. - -For EKS clusters, all Kubernetes versions available at the time of cluster deployment are supported. - -**Do benchmark rules support multiple Kubernetes deployment types?** Yes. There are different sets of benchmark rules for self-managed and third party-managed deployments. Refer to [Get started with KSPM](../../../solutions/security/cloud/get-started-with-kspm.md) for more information about setting up each deployment type. - -**Can I evaluate the security posture of my Amazon EKS clusters?** Yes. KSPM currently supports the security posture evaluation of Amazon EKS and unmanaged Kubernetes clusters. - -**How often is my cluster’s security posture evaluated?** Clusters are evaluated when you deploy a KSPM integration, and every four hours after that. - -**When do newly-enrolled clusters appear on the dashboard?** It can take up to 10 minutes for deployment, resource fetching, evaluation, and data processing to complete before a newly-enrolled cluster appears on the dashboard. - -**When do unenrolled clusters disappear from the dashboard?** A cluster will disappear as soon as the KSPM integration fetches data while that cluster is not enrolled. The fetch process repeats every four hours, which means a newly unenrolled cluster can take a maximum of four hours to disappear from the dashboard. - - -## Findings page [security-cspm-security-posture-faq-findings-page] - -**Are all the findings page current?** Yes. Only the most recent findings appear on the Findings page. - -**Can I build custom visualizations and dashboards that incorporate findings data?** Yes, you can use custom visualization capabilities with findings data. To learn more, refer to [Dashboards and visualizations](../../../explore-analyze/dashboards.md). - -**Where is Findings data saved?** You can access findings data using the following index patterns: - -* **Current findings:** `logs-cloud_security_posture.findings_latest-*` -* **Historical findings:** `logs-cloud_security_posture.findings-*` - - -## Benchmark rules [security-cspm-security-posture-faq-benchmark-rules] - -**How often are my resources evaluated against benchmark rules?** Resources are fetched and evaluated against benchmark rules when a security posture management integration is deployed. After that, the CSPM integration evaluates every 24 hours, and the KSPM integration evaluates every four hours. - -**Can I configure an integration’s fetch cycle?** No, the four-hour fetch cycle is not configurable. - -**Can I contribute to the CSP ruleset?** You can’t directly edit benchmark rules. The rules are defined [in this repository](https://github.com/elastic/csp-security-policies), where you can raise issues with certain rules. They are written in [Rego](https://www.openpolicyagent.org/docs/latest/policy-language/). - -**How can I tell which specific version of the CIS benchmarks is in use?** Refer to the `rule.benchmark.name` and `rule.benchmark.version` fields for documents in these datastreams: - -* `logs-cloud_security_posture.findings-default` -* `logs-cloud_security_posture.findings_latest-default` diff --git a/raw-migrated-files/docs-content/serverless/security-cspm.md b/raw-migrated-files/docs-content/serverless/security-cspm.md deleted file mode 100644 index 4f10bcf16..000000000 --- a/raw-migrated-files/docs-content/serverless/security-cspm.md +++ /dev/null @@ -1,28 +0,0 @@ -# Cloud security posture management [security-cspm] - -The Cloud Security Posture Management (CSPM) feature discovers and evaluates the services in your cloud environment — like storage, compute, IAM, and more — against configuration security guidelines defined by the [Center for Internet Security](https://www.cisecurity.org/) (CIS) to help you identify and remediate risks that could undermine the confidentiality, integrity, and availability of your cloud data. - -This feature currently supports agentless and agent-based deployments on Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure. For step-by-step getting started guides, refer to [Get started with CSPM for AWS](../../../solutions/security/cloud/get-started-with-cspm-for-aws.md), [Get started with CSPM for GCP](../../../solutions/security/cloud/get-started-with-cspm-for-gcp.md), or [Get started with CSPM for Azure](../../../solutions/security/cloud/get-started-with-cspm-for-azure.md). - -::::{admonition} Requirements -:class: note - -* CSPM only works in the `Default` {{kib}} space. Installing the CSPM integration on a different {{kib}} space will not work. -* CSPM is supported only on AWS, GCP, and Azure commercial cloud platforms, and AWS GovCloud. Other government cloud platforms are not supported ([request support](https://github.com/elastic/kibana/issues/new/choose)). - -:::: - - - -## How CSPM works [cspm-how-it-works] - -Using the read-only credentials you will provide during the setup process, it will evaluate the configuration of resources in your environment every 24 hours. After each evaluation, the integration sends findings to Elastic. A high-level summary of the findings appears on the [Cloud Security Posture dashboard](../../../solutions/security/dashboards/cloud-security-posture-dashboard.md), and detailed findings appear on the [Findings page](../../../solutions/security/cloud/findings-page.md). - - - - - - - - - diff --git a/raw-migrated-files/docs-content/serverless/security-environment-variable-capture.md b/raw-migrated-files/docs-content/serverless/security-environment-variable-capture.md deleted file mode 100644 index 4112b9d0d..000000000 --- a/raw-migrated-files/docs-content/serverless/security-environment-variable-capture.md +++ /dev/null @@ -1,34 +0,0 @@ -# Capture environment variables [security-environment-variable-capture] - -You can configure an {{agent}} policy to capture up to five environment variables (`env vars`). - -::::{note} -* Env var names must be no more than 63 characters, and env var values must be no more than 1023 characters. Values outside these limits are silently ignored. -* Env var names are case sensitive. - -:::: - - -To set up environment variable capture for an {{agent}} policy: - -1. Find **Policies** in the navigation menu or use the global search field. -2. Select an {{agent}} policy. -3. Click **Show advanced settings**. -4. Scroll down or search for `linux.advanced.capture_env_vars`, or `mac.advanced.capture_env_vars`. -5. Enter the names of env vars you want to capture, separated by commas. For example: `PATH,USER` -6. Click **Save**. - - -## Find captured environment variables [find-cap-env-vars] - -Captured environment variables are associated with process events, and appear in each event’s `process.env_vars` field. - -To view environment variables in the **Events** table: - -1. Click the **Events** tab on the **Hosts***, ***Network***, or ***Users** pages, then click **Fields** in the Events table. -2. Search for the `process.env_vars` field, select it, and click **Close**. A new column appears containing captured environment variable data. - -:::{image} ../../../images/serverless--cloud-native-security-env-var-capture-detail.png -:alt: The Events table with the "process.env_vars" column highlighted -:class: screenshot -::: diff --git a/raw-migrated-files/docs-content/serverless/security-get-started-with-kspm.md b/raw-migrated-files/docs-content/serverless/security-get-started-with-kspm.md deleted file mode 100644 index 7d3ba86a1..000000000 --- a/raw-migrated-files/docs-content/serverless/security-get-started-with-kspm.md +++ /dev/null @@ -1,423 +0,0 @@ -# Get started with KSPM [security-get-started-with-kspm] - -This page explains how to configure the Kubernetes Security Posture Management (KSPM) integration. - -::::{admonition} Requirements -:class: note - -* KSPM only works in the `Default` {{kib}} space. Installing the KSPM integration on a different {{kib}} space will not work. -* KSPM is not supported on EKS clusters in AWS GovCloud ([request support](https://github.com/elastic/kibana/issues/new/choose)). -* To view posture data, ensure you have the appropriate user role to read the following {{es}} indices: -* `logs-cloud_security_posture.findings_latest-*` -* `logs-cloud_security_posture.scores-*` -* `logs-cloud_security_posture.findings` - -:::: - - -The instructions differ depending on whether you’re installing on EKS or on unmanaged clusters. - -* Install on EKS-managed clusters: - - 1. [Name your integration and select a Kubernetes deployment type](../../../solutions/security/cloud/get-started-with-kspm.md#kspm-setup-eks-start) - 2. [Authenticate to AWS](../../../solutions/security/cloud/get-started-with-kspm.md#kspm-setup-eks-auth) - 3. [Finish configuring the KSPM integration](../../../solutions/security/cloud/get-started-with-kspm.md#kspm-setup-eks-finish) - 4. [Deploy the DaemonSet to your clusters](../../../solutions/security/cloud/get-started-with-kspm.md#kspm-setup-eks-modify-deploy) - -* Install on unmanaged clusters: - - 1. [Configure the KSPM integration](../../../solutions/security/cloud/get-started-with-kspm.md#kspm-setup-unmanaged) - 2. [Deploy the DaemonSet manifest to your clusters](../../../solutions/security/cloud/get-started-with-kspm.md#kspm-setup-unmanaged-modify-deploy) - - - -## Set up KSPM for Amazon EKS clusters [kspm-setup-eks-start] - - -### Name your integration and select a Kubernetes Deployment type [security-get-started-with-kspm-name-your-integration-and-select-a-kubernetes-deployment-type] - -1. Find **Cloud Security Posture** in the navigation menu or use the global search field. -2. Click **Add a KSPM integration**. -3. Read the integration’s description to understand how it works. Then, click [*Add Kubernetes Security Posture Management*](https://docs.elastic.co/en/integrations/cloud_security_posture). -4. Name your integration. Use a name that matches the purpose or team of the cluster(s) you want to monitor, for example, `IT-dev-k8s-clusters`. -5. Select **EKS** from the **Kubernetes Deployment** menu. A new section for AWS credentials will appear. - - -### Authenticate to AWS [kspm-setup-eks-auth] - -There are several options for how to provide AWS credentials: - -* [Use Kubernetes Service Account to assume IAM role](../../../solutions/security/cloud/get-started-with-kspm.md#kspm-use-irsa) -* [Use default instance role](../../../solutions/security/cloud/get-started-with-kspm.md#kspm-use-instance-role) -* [Use access keys directly](../../../solutions/security/cloud/get-started-with-kspm.md#kspm-use-keys-directly) -* [Use temporary security credentials](../../../solutions/security/cloud/get-started-with-kspm.md#kspm-use-temp-credentials) -* [Use a shared credentials file](../../../solutions/security/cloud/get-started-with-kspm.md#kspm-use-a-shared-credentials-file) -* [Use an IAM role ARN](../../../solutions/security/cloud/get-started-with-kspm.md#kspm-use-iam-arn) - -Regardless of which option you use, you’ll need to grant the following permissions: - -```console -ecr:GetRegistryPolicy, -eks:ListTagsForResource -elasticloadbalancing:DescribeTags -ecr-public:DescribeRegistries -ecr:DescribeRegistry -elasticloadbalancing:DescribeLoadBalancerPolicyTypes -ecr:ListImages -ecr-public:GetRepositoryPolicy -elasticloadbalancing:DescribeLoadBalancerAttributes -elasticloadbalancing:DescribeLoadBalancers -ecr-public:DescribeRepositories -eks:DescribeNodegroup -ecr:DescribeImages -elasticloadbalancing:DescribeLoadBalancerPolicies -ecr:DescribeRepositories -eks:DescribeCluster -eks:ListClusters -elasticloadbalancing:DescribeInstanceHealth -ecr:GetRepositoryPolicy -``` - -If you are using the AWS visual editor to create and modify your IAM Policies, you can copy and paste this IAM policy JSON object: - -::::{dropdown} Click to view JSON object -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Sid": "VisualEditor0", - "Effect": "Allow", - "Action": [ - "ecr:GetRegistryPolicy", - "eks:ListTagsForResource", - "elasticloadbalancing:DescribeTags", - "ecr-public:DescribeRegistries", - "ecr:DescribeRegistry", - "elasticloadbalancing:DescribeLoadBalancerPolicyTypes", - "ecr:ListImages", - "ecr-public:GetRepositoryPolicy", - "elasticloadbalancing:DescribeLoadBalancerAttributes", - "elasticloadbalancing:DescribeLoadBalancers", - "ecr-public:DescribeRepositories", - "eks:DescribeNodegroup", - "ecr:DescribeImages", - "elasticloadbalancing:DescribeLoadBalancerPolicies", - "ecr:DescribeRepositories", - "eks:DescribeCluster", - "eks:ListClusters", - "elasticloadbalancing:DescribeInstanceHealth", - "ecr:GetRepositoryPolicy" - ], - "Resource": "*" - } - ] -} -``` - -:::: - - - -#### Option 1 - [Recommended] Use Kubernetes Service Account to assume IAM role [kspm-use-irsa] - -Follow AWS’s [EKS Best Practices](https://aws.github.io/aws-eks-best-practices/security/docs/iam/#iam-roles-for-service-accounts-irsa) documentation to use the [IAM Role to Kubernetes Service-Account](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.md) (IRSA) feature to get temporary credentials and scoped permissions. - -::::{important} -During setup, do not fill in any option in the "Setup Access" section. Click **Save and continue**. - -:::: - - - -#### Option 2 - Use default instance role [kspm-use-instance-role] - -Follow AWS’s [IAM roles for Amazon EC2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.md) documentation to create an IAM role using the IAM console, which automatically generates an instance profile. - -::::{important} -During setup, do not fill in any option in the "Setup Access" section. Click **Save and continue**. - -:::: - - - -#### Option 3 - Use access keys directly [kspm-use-keys-directly] - -Access keys are long-term credentials for an IAM user or AWS account root user. To use access keys as credentials, you must provide the `Access key ID` and the `Secret Access Key`. - -For more details, refer to AWS' [Access Keys and Secret Access Keys](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.md) documentation. - -::::{important} -You must select "Programmatic access" when creating the IAM user. - -:::: - - - -#### Option 4 - Use temporary security credentials [kspm-use-temp-credentials] - -You can configure temporary security credentials in AWS to last for a specified duration. They consist of an access key ID, a secret access key, and a security token, which is typically found using `GetSessionToken`. - -Because temporary security credentials are short term, once they expire, you will need to generate new ones and manually update the integration’s configuration to continue collecting cloud posture data. Update the credentials before they expire to avoid data loss. - -::::{note} -IAM users with multi-factor authentication (MFA) enabled need to submit an MFA code when calling `GetSessionToken`. For more details, refer to AWS' [Temporary Security Credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.md) documentation. - -:::: - - -You can use the AWS CLI to generate temporary credentials. For example, you could use the following command if you have MFA enabled: - -```console -`sts get-session-token --serial-number arn:aws:iam::1234:mfa/your-email@example.com --duration-seconds 129600 --token-code 123456` -``` - -The output from this command includes the following fields, which you should provide when configuring the KSPM integration: - -* `Access key ID`: The first part of the access key. -* `Secret Access Key`: The second part of the access key. -* `Session Token`: A token required when using temporary security credentials. - - -#### Option 5 - Use a shared credentials file [kspm-use-a-shared-credentials-file] - -If you use different AWS credentials for different tools or applications, you can use profiles to define multiple access keys in the same configuration file. For more details, refer to AWS' [Shared Credentials Files](https://docs.aws.amazon.com/sdkref/latest/guide/file-format.md) documentation. - -Instead of providing the `Access key ID` and `Secret Access Key` to the integration, provide the information required to locate the access keys within the shared credentials file: - -* `Credential Profile Name`: The profile name in the shared credentials file. -* `Shared Credential File`: The directory of the shared credentials file. - -If you don’t provide values for all configuration fields, the integration will use these defaults: - -* If `Access key ID`, `Secret Access Key`, and `ARN Role` are not provided, then the integration will check for `Credential Profile Name`. -* If there is no `Credential Profile Name`, the default profile will be used. -* If `Shared Credential File` is empty, the default directory will be used. - - * For Linux or Unix, the shared credentials file is located at `~/.aws/credentials`. - - - -#### Option 6 - Use an IAM role Amazon Resource Name (ARN) [kspm-use-iam-arn] - -An IAM role Amazon Resource Name (ARN) is an IAM identity that you can create in your AWS account. You define the role’s permissions. Roles do not have standard long-term credentials such as passwords or access keys. Instead, when you assume a role, it provides temporary security credentials for your session. An IAM role’s ARN can be used to specify which AWS IAM role to use to generate temporary credentials. - -For more details, refer to AWS' [AssumeRole API](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.md) documentation. Follow AWS' instructions to [create an IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.md), and define the IAM role’s permissions using the JSON permissions policy above. - -To use an IAM role’s ARN, you need to provide either a [credential profile](../../../solutions/security/cloud/get-started-with-kspm.md#kspm-use-a-shared-credentials-file) or [access keys](../../../solutions/security/cloud/get-started-with-kspm.md#kspm-use-keys-directly) along with the `ARN role`. The `ARN Role` value specifies which AWS IAM role to use for generating temporary credentials. - -::::{note} -If `ARN Role` is present, the integration will check if `Access key ID` and `Secret Access Key` are present. If not, the package will check for a `Credential Profile Name`. If a `Credential Profile Name` is not present, the default credential profile will be used. - -:::: - - - -### Finish configuring the KSPM integration for EKS [kspm-setup-eks-finish] - -Once you’ve provided AWS credentials, finish configuring the KSPM integration: - -1. If you want to monitor Kubernetes clusters that aren’t yet enrolled in {{fleet}}, select **New Hosts** under “where to add this integration”. -2. Name the {{agent}} policy. Use a name that matches the purpose or team of the cluster(s) you want to monitor. For example, `IT-dev-k8s-clusters`. -3. Click **Save and continue**, then **Add agent to your hosts**. The **Add agent** wizard appears and provides a DaemonSet manifest `.yaml` file with pre-populated configuration information, such as the `Fleet ID` and `Fleet URL`. - - -### Deploy the KSPM integration to EKS clusters [kspm-setup-eks-modify-deploy] - -The **Add agent** wizard helps you deploy the KSPM integration on the Kubernetes clusters you wish to monitor. For each cluster: - -1. Download the manifest and make any necessary revisions to its configuration to suit the needs of your environment. -2. Apply the manifest using the `kubectl apply -f` command. For example: `kubectl apply -f elastic-agent-managed-kubernetes.yaml` - -After a few minutes, a message confirming the {{agent}} enrollment appears, followed by a message confirming that data is incoming. You can then click **View assets** to see where the newly-collected configuration information appears, including the [Findings page](../../../solutions/security/cloud/findings-page.md) and the [Cloud Security Posture dashboard](../../../solutions/security/dashboards/cloud-security-posture-dashboard.md). - - -## Set up KSPM for unmanaged Kubernetes clusters [kspm-setup-unmanaged] - -Follow these steps to deploy the KSPM integration to unmanaged clusters. Keep in mind credentials are NOT required for unmanaged deployments. - - -### Configure the KSPM integration [security-get-started-with-kspm-configure-the-kspm-integration] - -To install the integration on unmanaged clusters: - -1. Find **Connectors** in the navigation menu or use the global search field. -2. Click **Add a KSPM integration**. -3. Read the integration’s description to understand how it works. Then, click [*Add Kubernetes Security Posture Management*](https://docs.elastic.co/en/integrations/cloud_security_posture). -4. Name your integration. Use a name that matches the purpose or team of the cluster(s) you want to monitor, for example, `IT-dev-k8s-clusters`. -5. Select **Unmanaged Kubernetes** from the **Kubernetes Deployment** menu. -6. If you want to monitor Kubernetes clusters that aren’t yet enrolled in {{fleet}}, select **New Hosts** when choosing the {{agent}} policy. -7. Select the {{agent}} policy where you want to add the integration. -8. Click **Save and continue**, then **Add agent to your hosts**. The **Add agent** wizard appears and provides a DaemonSet manifest `.yaml` file with pre-populated configuration information, such as the `Fleet ID` and `Fleet URL`. - -:::{image} ../../../images/serverless--cloud-native-security-kspm-add-agent-wizard.png -:alt: The KSPM integration's Add agent wizard -:class: screenshot -::: - - -### Deploy the KSPM integration to unmanaged clusters [kspm-setup-unmanaged-modify-deploy] - -The **Add agent** wizard helps you deploy the KSPM integration on the Kubernetes clusters you wish to monitor. To do this, for each cluster: - -1. Download the manifest and make any necessary revisions to its configuration to suit the needs of your environment. -2. Apply the manifest using the `kubectl apply -f` command. For example: `kubectl apply -f elastic-agent-managed-kubernetes.yaml` - -After a few minutes, a message confirming the {{agent}} enrollment appears, followed by a message confirming that data is incoming. You can then click **View assets** to see where the newly-collected configuration information appears, including the [Findings page](../../../solutions/security/cloud/findings-page.md) and the [Cloud Security Posture dashboard](../../../solutions/security/dashboards/cloud-security-posture-dashboard.md). - - -### Set up KSPM on ECK deployments [kspm-eck] - -To run KSPM on an [ECK](/deploy-manage/deploy/cloud-on-k8s/deploy-an-orchestrator.md) deployment, you must edit the [Elastic Agent CRD](/deploy-manage/deploy/cloud-on-k8s/configuration-standalone.md) and [Elastic Agent Cluster-Role](/deploy-manage/deploy/cloud-on-k8s/configuration-standalone.md#k8s-elastic-agent-role-based-access-control) `.yaml` files. - -::::{dropdown} Patch Elastic Agent -Add `volumes` and `volumeMounts` to `podTemplate`: - -```yaml -podTemplate: - spec: - containers: - - name: agent - volumeMounts: - - name: proc - mountPath: /hostfs/proc - readOnly: true - - name: cgroup - mountPath: /hostfs/sys/fs/cgroup - readOnly: true - - name: varlibdockercontainers - mountPath: /var/lib/docker/containers - readOnly: true - - name: varlog - mountPath: /var/log - readOnly: true - - name: etc-full - mountPath: /hostfs/etc - readOnly: true - - name: var-lib - mountPath: /hostfs/var/lib - readOnly: true - - name: etc-mid - mountPath: /etc/machine-id - readOnly: true - volumes: - - name: proc - hostPath: - path: /proc - - name: cgroup - hostPath: - path: /sys/fs/cgroup - - name: varlibdockercontainers - hostPath: - path: /var/lib/docker/containers - - name: varlog - hostPath: - path: /var/log - - name: etc-full - hostPath: - path: /etc - - name: var-lib - hostPath: - path: /var/lib - # Mount /etc/machine-id from the host to determine host ID - # Needed for Elastic Security integration - - name: etc-mid - hostPath: - path: /etc/machine-id - type: File -``` - -:::: - - -::::{dropdown} Patch RBAC -Make sure that the `elastic-agent` service-account has the following Role and ClusterRole: - -```yaml -apiVersion: rbac.authorization.k8s.io/v1 -kind: RoleBinding -metadata: - namespace: default - name: elastic-agent -subjects: -- kind: ServiceAccount - name: elastic-agent - namespace: default -roleRef: - kind: Role - name: elastic-agent - apiGroup: rbac.authorization.k8s.io ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRole -metadata: - name: elastic-agent - labels: - k8s-app: elastic-agent -rules: -- apiGroups: [""] - resources: - - nodes - - namespaces - - events - - pods - - services - - configmaps - - serviceaccounts - - persistentvolumes - - persistentvolumeclaims - verbs: ["get", "list", "watch"] -- apiGroups: ["extensions"] - resources: - - replicasets - verbs: ["get", "list", "watch"] -- apiGroups: ["apps"] - resources: - - statefulsets - - deployments - - replicasets - - daemonsets - verbs: ["get", "list", "watch"] -- apiGroups: - - "" - resources: - - nodes/stats - verbs: - - get -- apiGroups: [ "batch" ] - resources: - - jobs - - cronjobs - verbs: [ "get", "list", "watch" ] -- nonResourceURLs: - - "/metrics" - verbs: - - get -- apiGroups: ["rbac.authorization.k8s.io"] - resources: - - clusterrolebindings - - clusterroles - - rolebindings - - roles - verbs: ["get", "list", "watch"] -- apiGroups: ["policy"] - resources: - - podsecuritypolicies - verbs: ["get", "list", "watch"] ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: Role -metadata: - name: elastic-agent - namespace: default - labels: - k8s-app: elastic-agent -rules: - - apiGroups: - - coordination.k8s.io - resources: - - leases - verbs: ["get", "create", "update"] -``` - -:::: diff --git a/raw-migrated-files/docs-content/serverless/security-interactive-investigation-guides.md b/raw-migrated-files/docs-content/serverless/security-interactive-investigation-guides.md deleted file mode 100644 index 9c5dd34a4..000000000 --- a/raw-migrated-files/docs-content/serverless/security-interactive-investigation-guides.md +++ /dev/null @@ -1,124 +0,0 @@ -# Launch Timeline from investigation guides [security-interactive-investigation-guides] - -Detection rule investigation guides suggest steps for triaging, analyzing, and responding to potential security issues. For custom rules, you can create an interactive investigation guide that includes buttons for launching runtime queries in [Timeline](../../../solutions/security/investigate/timeline.md), using alert data and hard-coded literal values. This allows you to start detailed Timeline investigations directly from an alert using relevant data. - -:::{image} ../../../images/serverless--detections-ig-alert-flyout.png -:alt: Alert details flyout with interactive investigation guide -:class: screenshot -::: - -Under the Investigation section, click **Show investigation guide** to open the **Investigation** tab in the left panel of the alert details flyout. - -:::{image} ../../../images/serverless--detections-ig-alert-flyout-invest-tab.png -:alt: Alert details flyout with interactive investigation guide -:class: screenshot -::: - -The **Investigation** tab displays query buttons, and each query button displays the number of event documents found. Click the query button to automatically load the query in Timeline, based on configuration settings in the investigation guide. - -:::{image} ../../../images/serverless--detections-ig-timeline.png -:alt: Timeline with query pre-loaded from investigation guide action -:class: screenshot -::: - - -## Add investigation guide actions to a rule [add-ig-actions-rule] - -::::{note} -You can only create interactive investigation guides with custom rules because Elastic prebuilt rules can’t be edited. However, you can duplicate a prebuilt rule, then configure the investigation guide for the duplicated rule. - -:::: - - -You can configure an interactive investigation guide when you [create a new rule](../../../solutions/security/detect-and-alert/create-detection-rule.md) or [edit an existing rule](../../../solutions/security/detect-and-alert/manage-detection-rules.md#edit-rules-settings). - -1. When configuring the rule’s settings (the **About rule** step for a new rule, or the **About*** tab for an existing rule), expand the ***Advanced settings**, then scroll down to the **Investigation guide** Markdown editor. - - :::{image} ../../../images/serverless--detections-ig-investigation-guide-editor.png - :alt: Investigation guide editor field - :class: screenshot - ::: - -2. Place the editor cursor where you want to add the query button in the investigation guide, then select the Investigate icon in the toolbar. The **Add investigation query** builder form appears. - - ![Add investigation guide UI](../../../images/serverless--detections-ig-investigation-query-builder.png "") - -3. Complete the query builder form to create an investigation query: - - 1. **Label**: Enter the text to appear on the query button. - 2. **Description**: (Optional) Enter additional text to include with the button. - 3. **Filters**: Select fields, operators, and values to build the query. Click **OR** or **AND** to create multiple filters and define their relationships. - - To use a field value from the alert as a query parameter, enter the field name surrounded by double curly brackets — such as `{{kibana.alert.example}}` — as a custom option for the filter value. - - ![Add investigation guide UI](../../../images/serverless--detections-ig-filters-field-custom-value.png "") - - 4. **Relative time range**: (Optional) Select a time range to limit the query, relative to the alert’s creation time. - -4. Click **Save changes**. The syntax is added to the investigation guide editor. - - ::::{note} - If you need to change the query button’s configuration, you can either edit the syntax directly in the editor (refer to the [syntax reference](../../../solutions/security/detect-and-alert/launch-timeline-from-investigation-guides.md#query-button-syntax) below), or delete the syntax and use the query builder form to recreate the query. - - :::: - -5. Save and enable the rule. - - -### Query button syntax [query-button-syntax] - -The following syntax defines a query button in an interactive investigation guide. - -| Field | Description | -| --- | --- | -| `!{investigate{ }}` | The container object holding all the query button’s configuration attributes. | -| `label` | Identifying text on the button. | -| `description` | Additional text included with the button. | -| `providers` | A two-level nested array that defines the query to run in Timeline. Similar to the structure of queries in Timeline, items in the outer level are joined by an `OR` relationship, and items in the inner level are joined by an `AND` relationship.

Each item in `providers` corresponds to a filter created in the query builder UI and is defined by these attributes:

* `field`: The name of the field to query.
* `excluded`: Whether the query result is excluded (such as **is not one of**) or included (*is one of*).
* `queryType`: The query type used to filter events, based on the filter’s operator. For example, `phrase` or `range`.
* `value`: The value to search for. Either a hard-coded literal value, or the name of an alert field (in double curly brackets) whose value you want to use as a query parameter.
* `valueType`: The data type of `value`, such as `string` or `boolean`.
| -| `relativeFrom`, `relativeTo` | (Optional) The start and end, respectively, of the relative time range for the query. Times are relative to the alert’s creation time, represented as `now` in [date math](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/common-options.md#date-math) format. For example, selecting **Last 15 minutes** in the query builder form creates the syntax `"relativeFrom": "now-15m", "relativeTo": "now"`. | - -::::{note} -Some characters must be escaped with a backslash, such as `\"` for a quotation mark and `\\` for a literal backslash. Divide Windows paths with double backslashes (for example, `C:\\Windows\\explorer.exe`), and paths that already include double backslashes might require four backslashes for each divider. A clickable error icon (![Error](../../../images/serverless-error.svg "")) displays below the Markdown editor if there are any syntax errors. - -:::: - - - -### Example syntax [security-interactive-investigation-guides-example-syntax] - -```json -!{investigate{ - "label": "Test action", - "description": "Click to investigate.", - "providers": [ - [ - {"field": "event.id", "excluded": false, "queryType": "phrase", "value": "{{event.id}}", "valueType": "string"} - ], - [ - {"field": "event.action", "excluded": false, "queryType": "phrase", "value": "rename", "valueType": "string"}, - {"field": "process.pid", "excluded": false, "queryType": "phrase", "value": "{{process.pid}}", "valueType": "string"} - ] - ], - "relativeFrom": "now-15m", - "relativeTo": "now" -}} -``` - -This example creates the following Timeline query, as illustrated below: - -`(event.id : )` `OR (event.action : "rename" AND process.pid : )` - -:::{image} ../../../images/serverless--detections-ig-timeline-query.png -:alt: Timeline query -:class: screenshot -::: - - -### Timeline template fields [security-interactive-investigation-guides-timeline-template-fields] - -When viewing an interactive investigation guide in contexts unconnected to a specific alert (such a rule’s details page), queries open as [Timeline templates](../../../solutions/security/investigate/timeline-templates.md), and `parameter` fields are treated as Timeline template fields. - -:::{image} ../../../images/serverless--detections-ig-timeline-template-fields.png -:alt: Timeline template -:class: screenshot -::: diff --git a/raw-migrated-files/docs-content/serverless/security-kspm.md b/raw-migrated-files/docs-content/serverless/security-kspm.md deleted file mode 100644 index 6a2bc97d7..000000000 --- a/raw-migrated-files/docs-content/serverless/security-kspm.md +++ /dev/null @@ -1,75 +0,0 @@ -# Kubernetes security posture management [security-kspm] - - -## Overview [kspm-overview] - -The Kubernetes Security Posture Management (KSPM) integration allows you to identify configuration risks in the various components that make up your Kubernetes cluster. It does this by evaluating your Kubernetes clusters against secure configuration guidelines defined by the Center for Internet Security (CIS) and generating findings with step-by-step instructions for remediating potential security risks. - -This integration supports Amazon EKS and unmanaged Kubernetes clusters. For setup instructions, refer to [Get started with KSPM](../../../solutions/security/cloud/get-started-with-kspm.md). - -::::{admonition} Requirements -:class: note - -* KSPM only works in the `Default` {{kib}} space. Installing the KSPM integration on a different {{kib}} space will not work. -* KSPM is not supported on EKS clusters in AWS GovCloud ([request support](https://github.com/elastic/kibana/issues/new/choose)). -* To view posture data, ensure you have the appropriate user role to read the following {{es}} indices: -* `logs-cloud_security_posture.findings_latest-*` -* `logs-cloud_security_posture.scores-*` -* `logs-cloud_security_posture.findings` - -:::: - - - -## How KSPM works [kspm-how-kspm-works] - -1. When you add a KSPM integration, it generates a Kubernetes manifest. When applied to a cluster, the manifest deploys an {{agent}} as a [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset) to ensure all nodes are evaluated. -2. Upon deployment, the integration immediately assesses the security posture of your Kubernetes resources. The evaluation process repeats every four hours. -3. After each evaluation, the integration sends findings to {{es}}. Findings appear on the [Cloud Security Posture dashboard](../../../solutions/security/dashboards/cloud-security-posture-dashboard.md) and the [findings](../../../solutions/security/cloud/findings-page.md) page. - - -## Use cases [kspm-use-cases] - -The KSPM integration helps you to: - -* Identify and remediate `failed` findings -* Identify the most misconfigured resources -* Identify risks in particular CIS benchmark sections - - -### Identify and remediate failed findings [kspm-remediate-failed-findings] - -To identify and remediate failed failed findings: - -1. Go to the [Cloud Security Posture dashboard](../../../solutions/security/dashboards/cloud-security-posture-dashboard.md). -2. Click **View all failed findings**, either for an individual cluster or for all monitored clusters. -3. Click a failed finding. The findings flyout opens. -4. Follow the steps under **Remediation** to correct the misconfiguration. - - ::::{note} - Remediation steps typically include commands for you to execute. These sometimes contain placeholder values that you must replace before execution. - - :::: - - - -### Identify the most misconfigured Kubernetes resources [kspm-identify-misconfigured-resources] - -To identify the Kubernetes resources generating the most failed findings: - -1. Go to the [Findings](../../../solutions/security/cloud/findings-page.md) page. -2. Click the **Group by** menu near the search box and select **Resource** to view a list of resources sorted by their total number of failed findings. -3. Click a resource ID to view the findings associated with that resource. - - -### Identify configuration risks by CIS section [kspm-identify-config-risks-by-section] - -To identify risks in particular CIS sections: - -1. Go to the [Cloud Security Posture dashboard](../../../solutions/security/dashboards/cloud-security-posture-dashboard.md). -2. In the Failed findings by CIS section widget, click the name of a CIS section to view all failed findings for that section. - -Alternatively: - -1. Go to the Findings page. -2. Filter by the `rule.section` field. For example, search for `rule.section : API Server` to view findings for benchmark rules in the API Server category. diff --git a/raw-migrated-files/docs-content/serverless/security-llm-connector-guides.md b/raw-migrated-files/docs-content/serverless/security-llm-connector-guides.md deleted file mode 100644 index 244fdb4b0..000000000 --- a/raw-migrated-files/docs-content/serverless/security-llm-connector-guides.md +++ /dev/null @@ -1,18 +0,0 @@ -# LLM connector guides [security-llm-connector-guides] - -This section contains instructions for setting up connectors for LLMs so you can use [Elastic AI Assistant](../../../solutions/security/ai/ai-assistant.md) and [Attack discovery](../../../solutions/security/ai/attack-discovery.md). - -Setup guides are available for the following LLM providers: - -* [Azure OpenAI](../../../solutions/security/ai/connect-to-azure-openai.md) -* [Amazon Bedrock](../../../solutions/security/ai/connect-to-amazon-bedrock.md) -* [OpenAI](../../../solutions/security/ai/connect-to-openai.md) -* [Google Vertex](../../../solutions/security/ai/connect-to-google-vertex.md) -* [LM Studio (custom local LLM)](../../../solutions/security/ai/connect-to-own-local-llm.md) - - - - - - - diff --git a/raw-migrated-files/docs-content/serverless/security-llm-performance-matrix.md b/raw-migrated-files/docs-content/serverless/security-llm-performance-matrix.md deleted file mode 100644 index 5783fe053..000000000 --- a/raw-migrated-files/docs-content/serverless/security-llm-performance-matrix.md +++ /dev/null @@ -1,37 +0,0 @@ -# Large language model performance matrix [security-llm-performance-matrix] - -This page describes the performance of various large language models (LLMs) for different use cases in {{elastic-sec}}, based on our internal testing. To learn more about these use cases, refer to [Attack discovery](../../../solutions/security/ai/attack-discovery.md) or [AI Assistant](../../../solutions/security/ai/ai-assistant.md). - -::::{note} -`Excellent` is the best rating, followed by `Great`, then by `Good`, and finally by `Poor`. -:::: - - - -## Proprietary models [_proprietary_models] - -Models from third-party LLM providers. - -| **Feature** | | **Assistant - General** | **Assistant - {{esql}} generation** | **Assistant - Alert questions** | **Assistant - Knowledge retrieval** | **Attack Discovery** | -| --- | --- | --- | --- | --- | --- | --- | -| **Model** | **Claude 3: Opus** | Excellent | Excellent | Excellent | Good | Great | -| | **Claude 3.5: Sonnet v2** | Excellent | Excellent | Excellent | Excellent | Great | -| | **Claude 3.5: Sonnet** | Excellent | Excellent | Excellent | Excellent | Excellent | -| | **Claude 3.5: Haiku** | Excellent | Excellent | Excellent | Excellent | Poor | -| | **Claude 3: Haiku** | Excellent | Excellent | Excellent | Excellent | Poor | -| | **GPT-4o** | Excellent | Excellent | Excellent | Excellent | Great | -| | **GPT-4o-mini** | Excellent | Great | Great | Great | Poor | -| | **Gemini 1.5 Pro 002** | Excellent | Excellent | Excellent | Excellent | Excellent | -| | **Gemini 1.5 Flash 002** | Excellent | Poor | Good | Excellent | Poor | - - -## Open-source models [_open_source_models] - -Models you can [deploy yourself](../../../solutions/security/ai/connect-to-own-local-llm.md). - -| **Feature** | | **Assistant - General** | **Assistant - {{esql}} generation** | **Assistant - Alert questions** | **Assistant - Knowledge retrieval** | **Attack Discovery** | -| --- | --- | --- | --- | --- | --- | --- | -| **Model** | **Mistral Nemo** | Good | Good | Great | Good | Poor | -| | **LLama 3.2** | Good | Poor | Good | Poor | Poor | -| | **LLama 3.1 405b** | Good | Great | Good | Good | Poor | -| | **LLama 3.1 70b** | Good | Good | Poor | Poor | Poor | diff --git a/raw-migrated-files/docs-content/serverless/security-posture-faq.md b/raw-migrated-files/docs-content/serverless/security-posture-faq.md deleted file mode 100644 index 65126c1ef..000000000 --- a/raw-migrated-files/docs-content/serverless/security-posture-faq.md +++ /dev/null @@ -1,67 +0,0 @@ -# Frequently asked questions (FAQ) [security-posture-faq] - - -## CSPM FAQ [cspm-faq] - -Frequently asked questions about the Cloud Security Posture Management (CSPM) integration and features. - -**How often is my cloud security posture evaluated?** - -Cloud accounts are evaluated when you first deploy the CSPM integration and every 24 hours afterward. - -**Can I onboard multiple accounts at one time?** - -Yes. Follow the onboarding instructions in the getting started guides for AWS, GCP, or Azure. - -**When do newly enrolled cloud accounts appear on the dashboard?** - -After you deploy the CSPM integration, it can take up to 10 minutes for resource fetching, evaluation, and data processing before a newly enrolled account appears on the Cloud Security Posture dashboard. - -**When do unenrolled cloud accounts disappear from the dashboard?** - -Newly unenrolled cloud accounts can take a maximum of 24 hours to disappear from the Cloud Security Posture dashboard. - - -## KSPM FAQ [kspm-faq] - -Frequently asked questions about the Kubernetes Security Posture Management (KSPM) integration and features. - -**What versions of Kubernetes are supported?** - -For self-managed/vanilla clusters, Kubernetes version 1.23 is supported. - -**Do benchmark rules support multiple Kubernetes deployment types?** Yes. There are different sets of benchmark rules for self-managed and third party-managed deployments. Refer to [Get started with KSPM](../../../solutions/security/cloud/get-started-with-kspm.md) for more information about setting up each deployment type. - -**Can I evaluate the security posture of my Amazon EKS clusters?** Yes. KSPM currently supports the security posture evaluation of Amazon EKS and unmanaged Kubernetes clusters. - -**How often is my cluster’s security posture evaluated?** Clusters are evaluated when you deploy a KSPM integration, and every four hours after that. - -**When do newly-enrolled clusters appear on the dashboard?** It can take up to 10 minutes for deployment, resource fetching, evaluation, and data processing to complete before a newly-enrolled cluster appears on the dashboard. - -**When do unenrolled clusters disappear from the dashboard?** A cluster will disappear as soon as the KSPM integration fetches data while that cluster is not enrolled. The fetch process repeats every four hours, which means a newly unenrolled cluster can take a maximum of four hours to disappear from the dashboard. - - -## Findings page [security-posture-faq-findings-page] - -**Are all the findings page current?** Yes. Only the most recent findings appear on the Findings page. - -**Can I build custom visualizations and dashboards that incorporate findings data?** Yes. You can use {{kib}}'s custom visualization capabilities with findings data. To learn more, refer to [Dashboards and visualizations](../../../explore-analyze/dashboards.md). - -**Where is Findings data saved?** You can access findings data using the following index patterns: - -* **Current findings:** `logs-cloud_security_posture.findings_latest-*` -* **Historical findings:** `logs-cloud_security_posture.findings-*` - - -## Benchmark rules [security-posture-faq-benchmark-rules] - -**How often are my resources evaluated against benchmark rules?** Resources are fetched and evaluated against benchmark rules when a security posture management integration is deployed. After that, the CSPM integration evaluates every 24 hours, and the KSPM integration evaluates every four hours. - -**Can I configure an integration’s fetch cycle?** No, the fetch cycle’s timing is not configurable. - -**Can I contribute to the CSP ruleset?** You can’t directly edit benchmark rules. The rules are defined [in this repository](https://github.com/elastic/csp-security-policies), where you can raise issues with certain rules. They are written in [Rego](https://www.openpolicyagent.org/docs/latest/policy-language/). - -**How can I tell which specific version of the CIS benchmarks is in use?** Refer to the `rule.benchmark.name` and `rule.benchmark.version` fields for documents in these datastreams: - -* `logs-cloud_security_posture.findings-default` -* `logs-cloud_security_posture.findings_latest-default` diff --git a/raw-migrated-files/docs-content/serverless/security-posture-management.md b/raw-migrated-files/docs-content/serverless/security-posture-management.md deleted file mode 100644 index 3f0f227a3..000000000 --- a/raw-migrated-files/docs-content/serverless/security-posture-management.md +++ /dev/null @@ -1,43 +0,0 @@ -# Security posture management overview [security-posture-management] - - -## Overview [_overview] - -Elastic’s [Cloud Security Posture Management](../../../solutions/security/cloud/cloud-security-posture-management.md) (CSPM) and [Kubernetes Security Posture Management](../../../solutions/security/cloud/kubernetes-security-posture-management.md) (KSPM) features help you discover and evaluate the services and resources in your cloud environment — like storage, compute, IAM, and more — against security guidelines defined by the Center for Internet Security (CIS). They help you identify and remediate configuration risks that could undermine the confidentiality, integrity, and availability of your cloud assets, such as publicly exposed storage buckets or overly permissive networking objects. - -The KSPM feature assesses the security of your Kubernetes assets, while the CSPM feature assesses the security of your AWS resources such as storage, compute, IAM, and more. - - -## Getting started [security-posture-management-get-started] - -For setup instructions, refer to: - -* [Get started with KSPM](../../../solutions/security/cloud/get-started-with-kspm.md) -* [Get started with CSPM](../../../solutions/security/cloud/get-started-with-cspm-for-aws.md) - - -## Use cases [security-posture-use-cases] - -Using the data generated by these features, you can: - -**Identify and secure misconfigured infrastructure:** - -1. Find **Cloud Security Posture** in the navigation menu or use the global search field. -2. Click **View all failed findings**, either for an individual resource or a group of resources. -3. Click a failed finding to open the Findings flyout. -4. Follow the steps under Remediation to fix the misconfiguration. - -**Identify the CIS Sections (security best practice categories) with which your resources are least compliant:** - -1. Find **Cloud Security Posture** in the navigation menu or use the global search field. -2. Do one of the following: - - 1. Under Failed findings by CIS section, click the name of a CIS section to view all failed findings from that section. - 2. Go to the **Findings** page and filter by the `rule.section` field. For example, search for `rule.section : API Server` to view findings from the API Server category. - - -**Identify your least compliant cloud resources** - -1. Go to the **Findings** page. -2. Click the **Group by** menu near the search box, and select **Resource** to sort resources by their number of failed findings. -3. Click a resource ID to view associated findings. diff --git a/raw-migrated-files/docs-content/serverless/security-vuln-management-dashboard-dash.md b/raw-migrated-files/docs-content/serverless/security-vuln-management-dashboard-dash.md deleted file mode 100644 index 2bfd1710b..000000000 --- a/raw-migrated-files/docs-content/serverless/security-vuln-management-dashboard-dash.md +++ /dev/null @@ -1,45 +0,0 @@ ---- -navigation_title: "Cloud Native Vulnerability Management dashboard" ---- - -# Cloud Native Vulnerability Management Dashboard [security-vuln-management-dashboard-dash] - - -The Cloud Native Vulnerability Management (CNVM) dashboard gives you an overview of vulnerabilities detected in your cloud infrastructure. - -:::{image} ../../../images/serverless--cloud-native-security-vuln-management-dashboard.png -:alt: The CNVM dashboard -:class: screenshot -::: - -::::{admonition} Requirements -:class: note - -* To collect this data, install the [Cloud Native Vulnerability Management](../../../solutions/security/cloud/get-started-with-cnvm.md) integration. - -:::: - - - -## CNVM dashboard UI [CNVM-dashboard-UI-dash] - -The summary cards at the top of the dashboard display the number of monitored cloud accounts, scanned virtual machines (VMs), and vulnerabilities (grouped by severity). - -The **Trend by severity** bar graph complements the summary cards by displaying the number of vulnerabilities found on your infrastructure over time, sorted by severity. It has a maximum time scale of 30 days. - -::::{admonition} Graph tips -:class: note - -* Click the severity levels legend on its right to hide/show each severity level. -* To display data from specific cloud accounts, select the account names from the **Accounts** drop-down menu. - -:::: - - -The page also includes three tables: - -* **Top 10 vulnerable resources** shows your VMs with the highest number of vulnerabilities. -* **Top 10 patchable vulnerabilities** shows the most common vulnerabilities in your environment that can be fixed by a software update. -* **Top 10 vulnerabilities** shows the most common vulnerabilities in your environment, with additional details. - -Click **View all vulnerabilities** at the bottom of a table to open the [Vulnerabilities Findings](../../../solutions/security/cloud/findings-page-3.md) page, where you can view additional details. diff --git a/raw-migrated-files/docs-content/serverless/security-vuln-management-get-started.md b/raw-migrated-files/docs-content/serverless/security-vuln-management-get-started.md deleted file mode 100644 index 1dc77d8ee..000000000 --- a/raw-migrated-files/docs-content/serverless/security-vuln-management-get-started.md +++ /dev/null @@ -1,79 +0,0 @@ -# Get started with CNVM [security-vuln-management-get-started] - -This page explains how to set up Cloud Native Vulnerability Management (CNVM). - -::::{admonition} Requirements -:class: note - -* CNVM only works in the `Default` {{kib}} space. Installing the CNVM integration on a different {{kib}} space will not work. -* Requires {{agent}} version 8.8 or higher. -* CNVM can only be deployed on ARM-based VMs. -* To view vulnerability scan findings, you need the appropriate user role to read the following indices: - - * `logs-cloud_security_posture.vulnerabilities-*` - * `logs-cloud_security_posture.vulnerabilities_latest-*` - -* You need an AWS user account with permissions to perform the following actions: run CloudFormation templates, create IAM Roles and InstanceProfiles, and create EC2 SecurityGroups and Instances. - -:::: - - -::::{note} -CNVM currently only supports AWS EC2 Linux workloads. - -:::: - - - -## Set up CNVM for AWS [vuln-management-setup] - -To set up the CNVM integration for AWS, install the integration on a new {{agent}} policy, sign into the AWS account you want to scan, and run the [CloudFormation](https://docs.aws.amazon.com/cloudformation/index.md) template. - -::::{important} -Do not add the integration to an existing {{agent}} policy. It should always be added to a new policy since it should not run on VMs with existing workloads. For more information, refer to [How CNVM works](../../../solutions/security/cloud/cloud-native-vulnerability-management.md#vuln-management-overview-how-it-works). - -:::: - - - -### Step 1: Add the CNVM integration [vuln-management-setup-step-1] - -1. Find **Integrations** in the navigation menu or use the global search field. -2. Search for **Cloud Native Vulnerability Management**, then click on the result. -3. Click **Add Cloud Native Vulnerability Management**. -4. Give your integration a name that matches its purpose or the AWS account region you want to scan for vulnerabilities (for example, `uswest2-aws-account`.) - - :::{image} ../../../images/serverless--dashboards-cnvm-setup-1.png - :alt: The CNVM integration setup page - :class: screenshot - ::: - -5. Click **Save and continue**. The integration will create a new {{agent}} policy. -6. Click **Add {{agent}} to your hosts**. - - -### Step 2: Sign in to the AWS management console [vuln-management-setup-step-2] - -1. Open a new browser tab and use it to sign into your AWS management console. -2. Switch to the cloud region with the workloads that you want to scan for vulnerabilities. - -::::{important} -The integration will only scan VMs in the region you select. To scan multiple regions, repeat this setup process for each region. - -:::: - - - -### Step 3: Run the CloudFormation template [vuln-management-setup-step-3] - -1. Switch back to the tab with Elastic Security. -2. Click **Launch CloudFormation**. The CloudFormation page appears. - - :::{image} ../../../images/serverless--dashboards-cnvm-cloudformation.png - :alt: The cloud formation template - :class: screenshot - ::: - -3. Click **Create stack**. To avoid authentication problems, you can only make configuration changes to the VM InstanceType, which you could make larger to increase scanning speed. -4. Wait for the confirmation that {{agent}} was enrolled. -5. Your data will start to appear on the **Vulnerabilities** tab of the [Findings page](../../../solutions/security/cloud/findings-page-3.md). diff --git a/raw-migrated-files/docs-content/serverless/security-vuln-management-overview.md b/raw-migrated-files/docs-content/serverless/security-vuln-management-overview.md deleted file mode 100644 index 5f567c432..000000000 --- a/raw-migrated-files/docs-content/serverless/security-vuln-management-overview.md +++ /dev/null @@ -1,38 +0,0 @@ -# Cloud native vulnerability management [security-vuln-management-overview] - -Elastic’s Cloud Native Vulnerability Management (CNVM) feature helps you identify known vulnerabilities in your cloud workloads. - -Setup uses infrastructure as code. For instructions, refer to [Get started with Cloud Native Vulnerability Management](../../../solutions/security/cloud/get-started-with-cnvm.md). - -::::{note} -CNVM currently only supports AWS EC2 Linux workloads. - -:::: - - -::::{admonition} Requirements -:class: note - -* CNVM only works in the `Default` {{kib}} space. Installing the CNVM integration on a different {{kib}} space will not work. -* To view vulnerability scan findings, you need the appropriate user role to read the following indices: - - * `logs-cloud_security_posture.vulnerabilities-*` - * `logs-cloud_security_posture.vulnerabilities_latest-*` - - -:::: - - - -## How CNVM works [vuln-management-overview-how-it-works] - -During setup, you will use an infrastructure as code provisioning template to create a new virtual machine (VM) in the cloud region you wish to scan. This VM installs {{agent}} and the Cloud Native Vulnerability Management (CNVM) integration, and conducts all vulnerability scanning. - -The CNVM integration uses [Trivy](https://github.com/aquasecurity/trivy), a comprehensive open-source security scanner, to scan cloud workloads and identify security vulnerabilities. During each scan, the VM running the integration takes a snapshot of all cloud workloads in its region using the snapshot APIs of the cloud service provider, and analyzes them for vulnerabilities using Trivy. Therefore, scanning does not use resources on the VMs being scanned. All resource usage occurs on the VM installed during CNVM setup. - -The scanning process begins immediately upon deployment, then repeats every twenty-four hours. After each scan, the integration sends the discovered vulnerabilities to {{es}}, where they appear in the **Vulnerabilities** tab of the [Findings page](../../../solutions/security/cloud/findings-page-3.md). - -::::{note} -Environments with more VMs take longer to scan. - -:::: diff --git a/raw-migrated-files/toc.yml b/raw-migrated-files/toc.yml index 4a58a0bb0..963189d26 100644 --- a/raw-migrated-files/toc.yml +++ b/raw-migrated-files/toc.yml @@ -41,7 +41,6 @@ toc: - file: cloud/cloud-enterprise/ece-getting-started-search-use-cases-node-logs.md - file: cloud/cloud-enterprise/ece-getting-started-search-use-cases-python-logs.md - file: cloud/cloud-enterprise/ece-ingest-guides.md - - file: cloud/cloud-enterprise/ece-install-offline.md - file: cloud/cloud-enterprise/ece-integrations-server-api-example.md - file: cloud/cloud-enterprise/ece-manage-apm-settings.md - file: cloud/cloud-enterprise/ece-manage-enterprise-search-settings.md @@ -154,11 +153,9 @@ toc: - file: docs-content/serverless/index.md children: - file: docs-content/serverless/intro.md - - file: docs-content/serverless/_cloud_native_vulnerability_management_dashboard.md - file: docs-content/serverless/ai-assistant-knowledge-base.md - file: docs-content/serverless/attack-discovery.md - file: docs-content/serverless/connect-to-byo-llm.md - - file: docs-content/serverless/cspm-required-permissions.md - file: docs-content/serverless/detections-logsdb-index-mode-impact.md - file: docs-content/serverless/elasticsearch-differences.md - file: docs-content/serverless/elasticsearch-explore-your-data.md @@ -169,10 +166,6 @@ toc: - file: docs-content/serverless/general-ml-nlp-auto-scale.md - file: docs-content/serverless/general-sign-up-trial.md - file: docs-content/serverless/index-management.md - - file: docs-content/serverless/ingest-aws-securityhub-data.md - - file: docs-content/serverless/ingest-falco.md - - file: docs-content/serverless/ingest-third-party-cloud-security-data.md - - file: docs-content/serverless/ingest-wiz-data.md - file: docs-content/serverless/intro.md - file: docs-content/serverless/observability-ai-assistant.md - file: docs-content/serverless/observability-apm-get-started.md @@ -191,35 +184,13 @@ toc: - file: docs-content/serverless/security-alert-suppression.md - file: docs-content/serverless/security-alerts-manage.md - file: docs-content/serverless/security-automatic-import.md - - file: docs-content/serverless/security-benchmark-rules-kspm.md - - file: docs-content/serverless/security-benchmark-rules.md - file: docs-content/serverless/security-building-block-rules.md - - file: docs-content/serverless/security-cloud-native-security-overview.md - - file: docs-content/serverless/security-cloud-posture-dashboard-dash-cspm.md - - file: docs-content/serverless/security-cloud-posture-dashboard-dash-kspm.md - - file: docs-content/serverless/security-cloud-posture-dashboard-dash.md - - file: docs-content/serverless/security-cloud-workload-protection.md - file: docs-content/serverless/security-connect-to-azure-openai.md - file: docs-content/serverless/security-connect-to-bedrock.md - file: docs-content/serverless/security-connect-to-google-vertex.md - file: docs-content/serverless/security-connect-to-openai.md - - file: docs-content/serverless/security-cspm-findings-page-kspm-kspm.md - - file: docs-content/serverless/security-cspm-findings-page.md - - file: docs-content/serverless/security-cspm-get-started-azure.md - - file: docs-content/serverless/security-cspm-get-started-gcp.md - - file: docs-content/serverless/security-cspm-get-started.md - - file: docs-content/serverless/security-cspm-security-posture-faq.md - - file: docs-content/serverless/security-cspm.md - file: docs-content/serverless/security-detection-engine-overview.md - file: docs-content/serverless/security-detections-requirements.md - - file: docs-content/serverless/security-environment-variable-capture.md - - file: docs-content/serverless/security-get-started-with-kspm.md - - file: docs-content/serverless/security-interactive-investigation-guides.md - - file: docs-content/serverless/security-kspm.md - - file: docs-content/serverless/security-llm-connector-guides.md - - file: docs-content/serverless/security-llm-performance-matrix.md - - file: docs-content/serverless/security-posture-faq.md - - file: docs-content/serverless/security-posture-management.md - file: docs-content/serverless/security-prebuilt-rules-management.md - file: docs-content/serverless/security-query-alert-indices.md - file: docs-content/serverless/security-reduce-notifications-alerts.md @@ -231,10 +202,7 @@ toc: - file: docs-content/serverless/security-tune-detection-signals.md - file: docs-content/serverless/security-view-alert-details.md - file: docs-content/serverless/security-visualize-alerts.md - - file: docs-content/serverless/security-vuln-management-dashboard-dash.md - file: docs-content/serverless/security-vuln-management-faq.md - - file: docs-content/serverless/security-vuln-management-get-started.md - - file: docs-content/serverless/security-vuln-management-overview.md - file: docs-content/serverless/spaces.md - file: docs-content/serverless/what-is-observability-serverless.md - file: elasticsearch-hadoop/elasticsearch-hadoop/index.md diff --git a/redirects.yml b/redirects.yml index 47f9ac8ef..c81e1c8ef 100644 --- a/redirects.yml +++ b/redirects.yml @@ -12,6 +12,7 @@ redirects: 'anonymous-authentication': 'basic-authentication': 'http-authentication': + 'deploy-manage/deploy/cloud-enterprise/deploy-large-installation-cloud.md': '!deploy-manage/deploy/cloud-enterprise/deploy-large-installation.md' ## reference 'reference/security/elastic-defend/index.md': 'solutions/security/configure-elastic-defend.md' @@ -31,4 +32,4 @@ redirects: 'reference/security/elastic-defend/create-defend-policy-api.md': 'solutions/security/configure-elastic-defend/create-an-elastic-defend-policy-using-api.md' 'reference/security/elastic-defend/offline-endpoint.md': 'solutions/security/configure-elastic-defend/configure-offline-endpoints-air-gapped-environments.md' 'reference/security/elastic-defend/uninstall-agent.md': 'solutions/security/configure-elastic-defend/uninstall-elastic-agent.md' - 'reference/security/fields-and-object-schemas/runtime-fields.md': 'solutions/security/get-started/create-runtime-fields-in-elastic-security.md' \ No newline at end of file + 'reference/security/fields-and-object-schemas/runtime-fields.md': 'solutions/security/get-started/create-runtime-fields-in-elastic-security.md' diff --git a/reference/data-analysis/observability/observability-aws-metrics-serverless.md b/reference/data-analysis/observability/observability-aws-metrics-serverless.md index 1d7a4ba63..21cedb95f 100644 --- a/reference/data-analysis/observability/observability-aws-metrics-serverless.md +++ b/reference/data-analysis/observability/observability-aws-metrics-serverless.md @@ -1,6 +1,7 @@ --- mapped_pages: - https://www.elastic.co/guide/en/serverless/current/observability-aws-metrics.html + - https://www.elastic.co/guide/en/observability/current/aws-metrics.html --- # AWS metrics [observability-aws-metrics] diff --git a/reference/data-analysis/observability/observability-container-metrics-serverless.md b/reference/data-analysis/observability/observability-container-metrics-serverless.md index 033030bd4..7b762f6ee 100644 --- a/reference/data-analysis/observability/observability-container-metrics-serverless.md +++ b/reference/data-analysis/observability/observability-container-metrics-serverless.md @@ -1,6 +1,7 @@ --- mapped_pages: - https://www.elastic.co/guide/en/serverless/current/observability-container-metrics.html + - https://www.elastic.co/guide/en/observability/current/container-metrics.html --- # Container metrics [observability-container-metrics] diff --git a/reference/data-analysis/observability/observability-host-metrics-serverless.md b/reference/data-analysis/observability/observability-host-metrics-serverless.md index 2c6d4245c..dffecdab9 100644 --- a/reference/data-analysis/observability/observability-host-metrics-serverless.md +++ b/reference/data-analysis/observability/observability-host-metrics-serverless.md @@ -1,6 +1,7 @@ --- mapped_pages: - https://www.elastic.co/guide/en/serverless/current/observability-host-metrics.html + - https://www.elastic.co/guide/en/observability/current/host-metrics.html --- # Host metrics [observability-host-metrics] diff --git a/reference/data-analysis/observability/observability-kubernetes-pod-metrics-serverless.md b/reference/data-analysis/observability/observability-kubernetes-pod-metrics-serverless.md index e5f32d026..621a3e969 100644 --- a/reference/data-analysis/observability/observability-kubernetes-pod-metrics-serverless.md +++ b/reference/data-analysis/observability/observability-kubernetes-pod-metrics-serverless.md @@ -1,6 +1,7 @@ --- mapped_pages: - https://www.elastic.co/guide/en/serverless/current/observability-kubernetes-pod-metrics.html + - https://www.elastic.co/guide/en/observability/current/kubernetes-pod-metrics.html --- # Kubernetes pod metrics [observability-kubernetes-pod-metrics] diff --git a/reference/ingestion-tools/fleet/fleet-roles-privileges.md b/reference/ingestion-tools/fleet/fleet-roles-privileges.md index 3443eadc5..0beaada25 100644 --- a/reference/ingestion-tools/fleet/fleet-roles-privileges.md +++ b/reference/ingestion-tools/fleet/fleet-roles-privileges.md @@ -5,9 +5,7 @@ mapped_pages: # Required roles and privileges [fleet-roles-and-privileges] -Beginning with {{stack}} version 8.1, you no longer require the built-in `elastic` superuser credentials to use {{fleet}} and Integrations. - -Assigning the {{kib}} feature privileges `Fleet` and `Integrations` grants access to these features: +Assigning the {{kib}} feature privileges `Fleet` and `Integrations` grants access to use {{fleet}} and Integrations. `all` : Grants full read-write access. @@ -15,39 +13,55 @@ Assigning the {{kib}} feature privileges `Fleet` and `Integrations` grants acces `read` : Grants read-only access. -The built-in `editor` role grants the following privileges, supporting full read-write access to {{fleet}} and Integrations: +`none` +: No access is granted. + +Take advantage of these privilege settings by: + +* [Using an {{es}} built-in role](#fleet-roles-and-privileges-built-in) +* [Creating a new role](#fleet-roles-and-privileges-create) + +## Built-in roles [fleet-roles-and-privileges-built-in] -* {{Fleet}}: `All` -* Integrations: `All` +{{es}} comes with built-in roles that include default privileges. -The built-in `viewer` role grants the following privileges, supporting read-only access to {{fleet}} and Integrations: +`editor` +: The built-in `editor` role grants the following privileges, supporting full read-write access to {{fleet}} and Integrations: +* {{Fleet}}: `all` +* Integrations: `all` -* {{Fleet}}:: `None` -* Integrations:: `Read` +`viewer` +: The built-in `viewer` role grants the following privileges, supporting read-only access to {{fleet}} and Integrations: -You can also create a new role that can be assigned to a user to grant access to {{fleet}} and Integrations. +* {{Fleet}}:: `read` +* Integrations:: `read` + +You can also create a new role that can be assigned to a user, in order to grant more specific levels of access to {{fleet}} and Integrations. ## Create a role for {{fleet}} [fleet-roles-and-privileges-create] -To create a new role with full access to use and manage {{fleet}} and Integrations: +To create a new role with access to {{fleet}} and Integrations: 1. In {{kib}}, go to **Management → Stack Management**. 2. In the **Security** section, select **Roles**. 3. Select **Create role**. 4. Specify a name for the role. 5. Leave the {{es}} settings at their defaults, or refer to [Security privileges](asciidocalypse://docs/reference/elasticsearch/security-privileges.md) for descriptions of the available settings. -6. In the {{kib}} section, select **Add Kibana privilege**. -7. In the **Spaces** menu, select *** All Spaces**. Since many Integrations assets are shared across spaces, the users needs the {{kib}} privileges in all spaces. +6. In the {{kib}} section, select **Assign to space**. +7. In the **Spaces** menu, select *** All Spaces**. Since many Integrations assets are shared across spaces, the users need the {{kib}} privileges in all spaces. 8. Expand the **Management** section. 9. Set **Fleet** privileges to **All**. -10. Set **Integrations** privileges to **All**. - -:::{image} images/kibana-fleet-privileges.png -:alt: Kibana privileges flyout showing Fleet and Integrations set to All -:class: screenshot -::: - -To create a read-only user for Integrations, follow the same steps as above but set the **Fleet** privileges to **None*** and the ***Integrations** privileges to **Read**. - -Read-only access to {{fleet}} is not currently supported but is planned for development in a later release. +10. Choose the access level that you'd like the role to have with respect to {{fleet}} and integrations: + 1. To grant the role full access to use and manage {{fleet}} and integrations, set both the **Fleet** and **Integrations** privileges to `All`. + :::{image} images/kibana-fleet-privileges-all.png + :alt: Kibana privileges flyout showing Fleet and Integrations access set to All + :class: screenshot + ::: + 2. Similarly, to create a read-only user for {{fleet}} and Integrations, set both the **Fleet** and **Integrations** privileges to `Read`. + :::{image} images/kibana-fleet-privileges-read.png + :alt: Kibana privileges flyout showing Fleet and Integrations access set to All + :class: screenshot + ::: + +Once you've created a new role you can assign it to any {{es}} user. You can edit the role at any time by returning to the **Roles** page in {{kib}}. \ No newline at end of file diff --git a/reference/ingestion-tools/fleet/images/kibana-fleet-privileges-all.png b/reference/ingestion-tools/fleet/images/kibana-fleet-privileges-all.png new file mode 100644 index 000000000..128b1862b Binary files /dev/null and b/reference/ingestion-tools/fleet/images/kibana-fleet-privileges-all.png differ diff --git a/reference/ingestion-tools/fleet/images/kibana-fleet-privileges-read.png b/reference/ingestion-tools/fleet/images/kibana-fleet-privileges-read.png new file mode 100644 index 000000000..7288e9974 Binary files /dev/null and b/reference/ingestion-tools/fleet/images/kibana-fleet-privileges-read.png differ diff --git a/reference/ingestion-tools/fleet/images/kibana-fleet-privileges.png b/reference/ingestion-tools/fleet/images/kibana-fleet-privileges.png deleted file mode 100644 index cea848dab..000000000 Binary files a/reference/ingestion-tools/fleet/images/kibana-fleet-privileges.png and /dev/null differ diff --git a/release-notes/elastic-security.md b/release-notes/elastic-security.md index 519295a55..94f22dac1 100644 --- a/release-notes/elastic-security.md +++ b/release-notes/elastic-security.md @@ -2,23 +2,24 @@ navigation_title: "Elastic Security" mapped_pages: - https://www.elastic.co/guide/en/security/master/release-notes-header-9.0.0.html + - https://www.elastic.co/guide/en/security/current/release-notes.html --- # Elastic Security release notes [elastic-security-release-notes] -Review the changes, fixes, and more in each version of Elastic Security. +Review the changes, fixes, and more in each version of Elastic Security. To check for security updates, go to [Security announcements for the Elastic stack](https://discuss.elastic.co/c/announcements/security-announcements/31). -% Release notes include only features, enhancements, and fixes. Add breaking changes, deprecations, and known issues to the applicable release notes sections. +% Release notes include only features, enhancements, and fixes. Add breaking changes, deprecations, and known issues to the applicable release notes sections. % ## version.next [elastic-security-next-release-notes] % **Release date:** Month day, year % ### Features and enhancements [elastic-security-next-features-enhancements] -% * +% * % ### Fixes [elastic-security-next-fixes] -% * +% * ## 9.0.0 [elastic-security-900-release-notes] **Release date:** March 25, 2025 diff --git a/solutions/observability/apps/configure-synthetics-settings.md b/solutions/observability/apps/configure-synthetics-settings.md index 2863da968..c94275c50 100644 --- a/solutions/observability/apps/configure-synthetics-settings.md +++ b/solutions/observability/apps/configure-synthetics-settings.md @@ -1,3 +1,9 @@ +--- +mapped_pages: + - https://www.elastic.co/guide/en/serverless/current/observability-synthetics-settings.html + - https://www.elastic.co/guide/en/observability/current/synthetics-settings.html +--- + # Configure Synthetics settings [synthetics-settings] There are several Synthetics settings you can adjust in Observability. diff --git a/solutions/observability/apps/create-custom-links.md b/solutions/observability/apps/create-custom-links.md index 9637ec715..6f64fa14d 100644 --- a/solutions/observability/apps/create-custom-links.md +++ b/solutions/observability/apps/create-custom-links.md @@ -138,7 +138,7 @@ This link creates a new task on the Engineering board in Jira. It populates the | Label | `Open a task in Jira` | | Link | `https://test-site-33.atlassian.net/secure/CreateIssueDetails!init.jspa?pid=10000&issuetype=10001&summary=Created+via+APM&description=Investigate+the+following+APM+trace%3A%0D%0A%0D%0Aservice.name%3A+{{service.name}}%0D%0Atransaction.id%3A+{{transaction.id}}%0D%0Acontainer.id%3A+{{container.id}}%0D%0Aurl.full%3A+{{url.full}}` | -See the [Jira application administration knowledge base](https://confluence.atlassian.com/jirakb/how-to-create-issues-using-direct-html-links-in-jira-server-159474.md) for a full list of supported query parameters. +See the [Jira application administration knowledge base](https://confluence.atlassian.com/jirakb/how-to-create-issues-using-direct-html-links-in-jira-server-159474.html) for a full list of supported query parameters. ### Dashboards [custom-links-examples-kib] diff --git a/solutions/observability/apps/create-upload-source-maps-rum.md b/solutions/observability/apps/create-upload-source-maps-rum.md index ceaf9ab18..2e02edcfd 100644 --- a/solutions/observability/apps/create-upload-source-maps-rum.md +++ b/solutions/observability/apps/create-upload-source-maps-rum.md @@ -58,7 +58,7 @@ It can also be any other unique string that indicates a specific version of your ## Generate a source map [apm-source-map-rum-generate] -To be compatible with Elastic APM, source maps must follow the [source map revision 3 proposal spec](https://sourcemaps.info/spec.md). +To be compatible with Elastic APM, source maps must follow the [source map revision 3 proposal spec](https://sourcemaps.info/spec.html). Source maps can be generated and configured in many different ways. For example, parcel automatically generates source maps by default. If you’re using webpack, some configuration may be needed to generate a source map: diff --git a/solutions/observability/apps/monitoring-aws-lambda-functions.md b/solutions/observability/apps/monitoring-aws-lambda-functions.md index 1aaf3feeb..c6a0e29ed 100644 --- a/solutions/observability/apps/monitoring-aws-lambda-functions.md +++ b/solutions/observability/apps/monitoring-aws-lambda-functions.md @@ -18,7 +18,7 @@ AWS Lambda uses a special execution model to provide a scalable, on-demand compu 1. To avoid data loss, APM data collected by APM agents needs to be flushed before the execution environment of a lambda function is frozen. 2. Flushing APM data must be fast so as not to impact the response times of lambda function requests. -To accomplish the above, Elastic APM agents instrument AWS Lambda functions and dispatch APM data via an [AWS Lambda extension](https://docs.aws.amazon.com/lambda/latest/dg/using-extensions.md). +To accomplish the above, Elastic APM agents instrument AWS Lambda functions and dispatch APM data via an [AWS Lambda extension](https://docs.aws.amazon.com/lambda/latest/dg/using-extensions.html). Normally, during the execution of a Lambda function, there’s only a single language process running in the AWS Lambda execution environment. With an AWS Lambda extension, Lambda users run a *second* process alongside their main service/application process. diff --git a/solutions/observability/apps/tutorial-monitor-java-application.md b/solutions/observability/apps/tutorial-monitor-java-application.md index 3a3ba106a..d80582f38 100644 --- a/solutions/observability/apps/tutorial-monitor-java-application.md +++ b/solutions/observability/apps/tutorial-monitor-java-application.md @@ -917,7 +917,7 @@ You have now learned about parsing logs in either {{beats}} or {{es}}. What if w Writing out logs as plain text works and is easy to read for humans. However, first writing them out as plain text, parsing them using the `dissect` processors, and then creating a JSON again sounds tedious and burns unneeded CPU cycles. -While log4j2 has a [JSONLayout](https://logging.apache.org/log4j/2.x/manual/layouts.md#JSONLayout), you can go further and use a Library called [ecs-logging-java](https://github.com/elastic/ecs-logging-java). The advantage of ECS logging is that it uses the [Elastic Common Schema](asciidocalypse://docs/ecs/docs/reference/index.md). ECS defines a standard set of fields used when storing event data in {{es}}, such as logs and metrics. +While log4j2 has a [JSONLayout](https://logging.apache.org/log4j/2.x/manual/layouts.html#JSONLayout), you can go further and use a Library called [ecs-logging-java](https://github.com/elastic/ecs-logging-java). The advantage of ECS logging is that it uses the [Elastic Common Schema](asciidocalypse://docs/ecs/docs/reference/index.md). ECS defines a standard set of fields used when storing event data in {{es}}, such as logs and metrics. 1. Instead of writing our logging standard, use an existing one. Let’s add the logging dependency to our Javalin application. @@ -1561,7 +1561,7 @@ A programmatic setup allows you to attach the agent via a line of java in your s This looks much better, having differences between endpoints. -4. Add another endpoint to see the power of transactions, which polls another HTTP service. You may have heard of [wttr.in](https://wttr.in/), a service to poll weather information from. Let’s implement a proxy HTTP method that forwards the request to that endpoint. Let’s use [Apache HTTP client](https://hc.apache.org/httpcomponents-client-4.5.x/quickstart.md), one of the most typical HTTP clients out there. +4. Add another endpoint to see the power of transactions, which polls another HTTP service. You may have heard of [wttr.in](https://wttr.in/), a service to poll weather information from. Let’s implement a proxy HTTP method that forwards the request to that endpoint. Let’s use [Apache HTTP client](https://hc.apache.org/httpcomponents-client-4.5.x/quickstart.html), one of the most typical HTTP clients out there. ```gradle implementation 'org.apache.httpcomponents:fluent-hc:4.5.12' diff --git a/solutions/observability/apps/upstream-opentelemetry-collectors-language-sdks.md b/solutions/observability/apps/upstream-opentelemetry-collectors-language-sdks.md index ac001277d..57e46bb27 100644 --- a/solutions/observability/apps/upstream-opentelemetry-collectors-language-sdks.md +++ b/solutions/observability/apps/upstream-opentelemetry-collectors-language-sdks.md @@ -262,6 +262,8 @@ Many L7 load balancers handle HTTP and gRPC traffic separately and rely on expli * Use the `otlp` exporter in the OTel collector. Set annotation `nginx.ingress.kubernetes.io/backend-protocol: "GRPC"` on the K8s Ingress object proxying to APM Server. * Use the `otlphttp` exporter in the OTel collector. Set annotation `nginx.ingress.kubernetes.io/backend-protocol: "HTTP"` (or `"HTTPS"` if APM Server expects TLS) on the K8s Ingress object proxying to APM Server. +The preferred approach is to deploy a L4 (TCP) load balancer (e.g. [NLB](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html) on AWS) in front of APM Server, which forwards raw TCP traffic transparently without protocol inspection. + For more information on how to configure an AWS ALB to support gRPC, see this AWS blog post: [Application Load Balancer Support for End-to-End HTTP/2 and gRPC](https://aws.amazon.com/blogs/aws/new-application-load-balancer-support-for-end-to-end-http-2-and-grpc/). For more information on how APM Server services gRPC requests, see [Muxing gRPC and HTTP/1.1](https://github.com/elastic/apm-server/blob/main/dev_docs/otel.md#muxing-grpc-and-http11). diff --git a/solutions/observability/cicd.md b/solutions/observability/cicd.md index 127a9c750..f4e076d0a 100644 --- a/solutions/observability/cicd.md +++ b/solutions/observability/cicd.md @@ -370,7 +370,7 @@ To learn more, see the [integration of Maven builds with Elastic {{observability The Ansible OpenTelemetry plugin integration provides visibility into all your Ansible playbooks. The plugin generates traces for each run and performance metrics to help you understand which Ansible tasks or roles are run the most, how often they fail, and how long they take to complete. -You can configure your Ansible playbooks with the [Ansible OpenTelemetry callback plugin](https://docs.ansible.com/ansible/latest/collections/community/general/opentelemetry_callback.md). It’s required to install the OpenTelemetry python libraries and configure the callback as stated in the [examples](https://docs.ansible.com/ansible/latest/collections/community/general/opentelemetry_callback.md#examples) section. +You can configure your Ansible playbooks with the [Ansible OpenTelemetry callback plugin](https://docs.ansible.com/ansible/latest/collections/community/general/opentelemetry_callback.html). It’s required to install the OpenTelemetry python libraries and configure the callback as stated in the [examples](https://docs.ansible.com/ansible/latest/collections/community/general/opentelemetry_callback.html#examples) section. The context propagation from the Jenkins job or pipeline is passed to the Ansible run. Therefore, everything that happens in the CI is also shown in the traces. @@ -492,7 +492,7 @@ pytest --otel-session-name='My_Test_cases' ### Concourse CI [ci-cd-concourse-ci] -To configure Concourse CI to send traces, refer to the [tracing](https://concourse-ci.org/tracing.md) docs. In the Concourse configuration, you just need to define `CONCOURSE_TRACING_OTLP_ADDRESS` and `CONCOURSE_TRACING_OTLP_HEADERS`. +To configure Concourse CI to send traces, refer to the [tracing](https://concourse-ci.org/tracing.html) docs. In the Concourse configuration, you just need to define `CONCOURSE_TRACING_OTLP_ADDRESS` and `CONCOURSE_TRACING_OTLP_HEADERS`. ```bash CONCOURSE_TRACING_OTLP_ADDRESS=elastic-apm-server.example.com:8200 diff --git a/solutions/observability/cloud/monitor-amazon-cloud-compute-ec2.md b/solutions/observability/cloud/monitor-amazon-cloud-compute-ec2.md index 0fc7cc5d4..0d6a780a1 100644 --- a/solutions/observability/cloud/monitor-amazon-cloud-compute-ec2.md +++ b/solutions/observability/cloud/monitor-amazon-cloud-compute-ec2.md @@ -15,7 +15,7 @@ Amazon EC2 instances can be run in various locations. The location is composed o Like most AWS services, Amazon EC2 sends its metrics to Amazon CloudWatch. The Elastic [Amazon EC2 integration](https://docs.elastic.co/en/integrations/aws/ec2) collects metrics from Amazon CloudWatch using {{agent}}. -CloudWatch, by default, uses basic monitoring that publishes metrics at five-minute intervals. You can enable detailed monitoring to increase that resolution to one-minute, at an additional cost. To learn how to enable detailed monitoring, refer to the [Amazon EC2 documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-cloudwatch-new.md). +CloudWatch, by default, uses basic monitoring that publishes metrics at five-minute intervals. You can enable detailed monitoring to increase that resolution to one-minute, at an additional cost. To learn how to enable detailed monitoring, refer to the [Amazon EC2 documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-cloudwatch-new.html). CloudWatch does not expose metrics related to EC2 instance memory. You can install {{agent}} on the EC2 instances to collect detailed system metrics. @@ -159,7 +159,7 @@ Here are the key status check metrics you should monitor and what to look for: `aws.ec2.metrics.StatusCheckFailed_Instance.avg` -: This check monitors the software and network configuration of the instance. Problems that can cause instance status checks to fail may include: incorrect networking or startup configuration, exhausted memory, corrupted file system, incompatible kernel, and so on. When an instance status check fails, you typically must address the problem yourself. You may need to reboot the instance or make instance configuration changes. To troubleshoot instances with failed status checks, refer to the [Amazon EC2 documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstances.md). +: This check monitors the software and network configuration of the instance. Problems that can cause instance status checks to fail may include: incorrect networking or startup configuration, exhausted memory, corrupted file system, incompatible kernel, and so on. When an instance status check fails, you typically must address the problem yourself. You may need to reboot the instance or make instance configuration changes. To troubleshoot instances with failed status checks, refer to the [Amazon EC2 documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstances.html). This check returns 0 (passed) if an instance passes the system status check or 1 (failed) if it fails. diff --git a/solutions/observability/cloud/monitor-amazon-kinesis-data-streams.md b/solutions/observability/cloud/monitor-amazon-kinesis-data-streams.md index 88df4db19..3e9877303 100644 --- a/solutions/observability/cloud/monitor-amazon-kinesis-data-streams.md +++ b/solutions/observability/cloud/monitor-amazon-kinesis-data-streams.md @@ -19,7 +19,7 @@ By default, Kinesis Data Streams sends stream-level (basic level) metrics to Clo aws kinesis enable-enhanced-monitoring --stream-name samplestream --shard-level-metrics ALL ``` -For more details, refer to the [EnableEnhancedMonitoring](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_EnableEnhancedMonitoring.md) documentation. +For more details, refer to the [EnableEnhancedMonitoring](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_EnableEnhancedMonitoring.html) documentation. ## Get started [get-started-kinesis] diff --git a/solutions/observability/cloud/monitor-amazon-simple-storage-service-s3.md b/solutions/observability/cloud/monitor-amazon-simple-storage-service-s3.md index 39abcabae..860e8ed17 100644 --- a/solutions/observability/cloud/monitor-amazon-simple-storage-service-s3.md +++ b/solutions/observability/cloud/monitor-amazon-simple-storage-service-s3.md @@ -21,7 +21,7 @@ With the Amazon S3 integration, you can collect these S3 metrics from CloudWatch ## Get started [get-started-s3] -If you plan to collect request metrics, enable them for the S3 buckets you want to monitor. To learn how, refer to the [AWS documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/configure-request-metrics-bucket.md). +If you plan to collect request metrics, enable them for the S3 buckets you want to monitor. To learn how, refer to the [AWS documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/configure-request-metrics-bucket.html). To collect S3 metrics, you typically need to install the Elastic [Amazon S3 integration](https://docs.elastic.co/en/integrations/aws/s3) and deploy an {{agent}} locally or on an EC2 instance. diff --git a/solutions/observability/cloud/monitor-amazon-web-services-aws-with-amazon-data-firehose.md b/solutions/observability/cloud/monitor-amazon-web-services-aws-with-amazon-data-firehose.md index 88463fe71..66933fc08 100644 --- a/solutions/observability/cloud/monitor-amazon-web-services-aws-with-amazon-data-firehose.md +++ b/solutions/observability/cloud/monitor-amazon-web-services-aws-with-amazon-data-firehose.md @@ -66,13 +66,13 @@ From the **Destination settings** panel, specify the following settings: ## Step 4: Send data to the Firehose delivery stream [firehose-step-four] -You can configure a variety of log sources to send data to Firehose streams directly for example VPC flow logs. Some services don’t support publishing logs directly to Firehose but they do support publishing logs to CloudWatch logs, such as CloudTrail and Lambda. Refer to the [AWS documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AWS-logs-and-resource-policy.md) for more information. +You can configure a variety of log sources to send data to Firehose streams directly for example VPC flow logs. Some services don’t support publishing logs directly to Firehose but they do support publishing logs to CloudWatch logs, such as CloudTrail and Lambda. Refer to the [AWS documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AWS-logs-and-resource-policy.html) for more information. For example, a typical workflow for sending CloudTrail logs to Firehose would be the following: -* Publish CloudTrail logs to a Cloudwatch log group. Refer to the AWS documentation [about publishing CloudTrail logs](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/monitor-cloudtrail-log-files-with-cloudwatch-logs.md). -* Create a subscription filter in the CloudWatch log group to the Firehose stream. Refer to the AWS documentation [about using subscription filters](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.md#FirehoseExample). +* Publish CloudTrail logs to a Cloudwatch log group. Refer to the AWS documentation [about publishing CloudTrail logs](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/monitor-cloudtrail-log-files-with-cloudwatch-logs.html). +* Create a subscription filter in the CloudWatch log group to the Firehose stream. Refer to the AWS documentation [about using subscription filters](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html#FirehoseExample). -We also added support for sending CloudWatch monitoring metrics to Elastic using Firehose. For example, you can configure metrics ingestion by creating a metric stream through CloudWatch. You can select an existing Firehose stream by choosing the option **Custom setup with Firehose**. For more information, refer to the AWS documentation [about the custom setup with Firehose](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-metric-streams-setup-datalake.md). +We also added support for sending CloudWatch monitoring metrics to Elastic using Firehose. For example, you can configure metrics ingestion by creating a metric stream through CloudWatch. You can select an existing Firehose stream by choosing the option **Custom setup with Firehose**. For more information, refer to the AWS documentation [about the custom setup with Firehose](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-metric-streams-setup-datalake.html). For more information on Amazon Data Firehose, you can also check the [Amazon Data Firehose Integrations documentation](https://docs.elastic.co/integrations/awsfirehose). diff --git a/solutions/observability/cloud/monitor-amazon-web-services-aws-with-beats.md b/solutions/observability/cloud/monitor-amazon-web-services-aws-with-beats.md index e5a4e8cd8..969eb1ec9 100644 --- a/solutions/observability/cloud/monitor-amazon-web-services-aws-with-beats.md +++ b/solutions/observability/cloud/monitor-amazon-web-services-aws-with-beats.md @@ -30,7 +30,7 @@ Create an [{{ech}}](https://cloud.elastic.co/registration?page=docs&placement=do With this tutorial, we assume that your logs and your infrastructure data are already shipped to CloudWatch. We are going to show you how you can stream your data from CloudWatch to {{es}}. If you don’t know how to put your AWS logs and infrastructure data in CloudWatch, Amazon provides a lot of documentation around this specific topic: * Collect your logs and infrastructure data from specific [AWS services](https://www.youtube.com/watch?v=vAnIhIwE5hY) -* Export your logs [to an S3 bucket](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3ExportTasksConsole.md) +* Export your logs [to an S3 bucket](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3ExportTasksConsole.html) ## Step 1: Create an S3 Bucket [aws-step-one] @@ -269,7 +269,7 @@ Edit the `modules.d/aws.yml` file with the following configurations. ``` 1. Enables the `ec2` fileset. -2. This is the AWS profile defined following the [AWS standard](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.md). +2. This is the AWS profile defined following the [AWS standard](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html). 3. Add the URL to the queue containing notifications around the bucket containing the EC2 logs @@ -293,7 +293,7 @@ Make sure that the AWS user used to collect the logs from S3 has at least the fo } ``` -You can now upload your logs to the S3 bucket. If you are using CloudWatch, make sure to edit the policy of your bucket as shown in [step 3 of the AWS user guide](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3ExportTasksConsole.md). This will help you avoid permissions issues. +You can now upload your logs to the S3 bucket. If you are using CloudWatch, make sure to edit the policy of your bucket as shown in [step 3 of the AWS user guide](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3ExportTasksConsole.html). This will help you avoid permissions issues. Start {{filebeat}} to collect the logs. @@ -344,7 +344,7 @@ Copy the URL of the queue you created. Edit the `modules.d/aws.yml`file with the ``` 1. Enables the `ec2` fileset. -2. This is the AWS profile defined following the [AWS standard](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.md). +2. This is the AWS profile defined following the [AWS standard](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html). 3. Add the URL to the queue containing notifications around the bucket containing the EC2 logs 4. Add the URL to the queue containing notifications around the bucket containing the S3 access logs @@ -532,7 +532,7 @@ To collect metrics from your AWS infrastructure, we’ll use the [{{metricbeat}} 1. Defines the module that is going to be used. 2. Defines the period at which the metrics are going to be collected 3. Defines the metricset that is going to be used. - 4. This is the AWS profile defined following the [AWS standard](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.md). + 4. This is the AWS profile defined following the [AWS standard](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html). Make sure that the AWS user used to collect the metrics from CloudWatch has at least the following permissions attached to it: diff --git a/solutions/observability/cloud/monitor-amazon-web-services-aws-with-elastic-agent.md b/solutions/observability/cloud/monitor-amazon-web-services-aws-with-elastic-agent.md index beeb407f1..1a5bb2228 100644 --- a/solutions/observability/cloud/monitor-amazon-web-services-aws-with-elastic-agent.md +++ b/solutions/observability/cloud/monitor-amazon-web-services-aws-with-elastic-agent.md @@ -35,15 +35,15 @@ Create an [{{ech}}](https://cloud.elastic.co/registration?page=docs&placement=do In this tutorial, we assume that: -* Your VPC flow logs are already exported to an S3 bucket. To learn how, refer to the AWS documentation about [publishing flow logs to an S3 bucket](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-s3.md). -* You have EC2 instances in your AWS account. By default, Amazon EC2 sends metric data to CloudWatch. If you don’t have an EC2 instance in your account, refer to the [AWS documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.md) to learn how to launch, connect to, and use a Linux instance. +* Your VPC flow logs are already exported to an S3 bucket. To learn how, refer to the AWS documentation about [publishing flow logs to an S3 bucket](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-s3.html). +* You have EC2 instances in your AWS account. By default, Amazon EC2 sends metric data to CloudWatch. If you don’t have an EC2 instance in your account, refer to the [AWS documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html) to learn how to launch, connect to, and use a Linux instance. ## Step 1: Create a queue and notifications for VPC flow logs [aws-elastic-agent-set-up-sqs-queue-and-notifications] In this step, you create an Amazon Simple Queue Service (SQS) queue and configure the S3 bucket containing your VPC flow logs to send a message to the SQS queue whenever new logs are present in the S3 bucket. -You should already have an S3 bucket that contains exported VPC flow logs. If you don’t, create one now. To learn how, refer to [publishing flow logs to an S3 bucket](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-s3.md). +You should already have an S3 bucket that contains exported VPC flow logs. If you don’t, create one now. To learn how, refer to [publishing flow logs to an S3 bucket](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-s3.html). ::::{note} **Why is an SQS queue needed?** diff --git a/solutions/observability/cloud/monitor-amazon-web-services-aws-with-elastic-serverless-forwarder.md b/solutions/observability/cloud/monitor-amazon-web-services-aws-with-elastic-serverless-forwarder.md index 14b0c1dc2..d1a578d17 100644 --- a/solutions/observability/cloud/monitor-amazon-web-services-aws-with-elastic-serverless-forwarder.md +++ b/solutions/observability/cloud/monitor-amazon-web-services-aws-with-elastic-serverless-forwarder.md @@ -33,7 +33,7 @@ Create an [{{ech}}](https://cloud.elastic.co/registration?page=docs&placement=do 2. Specify the AWS region in which you want it deployed. 3. Enter the bucket name. -For more details, refer to the Amazon documentation on how to [Create your first S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.md). +For more details, refer to the Amazon documentation on how to [Create your first S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html). ## Step 2: Enable AWS VPC flow logs to be sent to your S3 bucket [esf-step-two] @@ -44,7 +44,7 @@ For more details, refer to the Amazon documentation on how to [Create your first 4. For **Destination**, select **Send to an S3 bucket**. 5. For **S3 bucket ARN**, enter the name of the S3 bucket you created in the previous step. -For more details, refer to the Amazon documentation on how to [Create a flow log that publishes to Amazon S3](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-s3.md). +For more details, refer to the Amazon documentation on how to [Create a flow log that publishes to Amazon S3](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-s3.html). ## Step 3: Create an SQS queue and notifications for VPC flow logs [esf-step-three] @@ -87,7 +87,7 @@ The Amazon Simple Queue Service (SQS) event notification on Amazon S3 serves as 3. Go to the properties of the S3 bucket containing the VPC flow logs and enable event notification. -For more details, refer to the AWS documentation on how to [Configure a bucket for notifications](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ways-to-add-notification-config-to-bucket.md). +For more details, refer to the AWS documentation on how to [Configure a bucket for notifications](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ways-to-add-notification-config-to-bucket.html). ## Step 4: Install the Elastic AWS integration [esf-step-four] diff --git a/solutions/observability/cloud/monitor-aws-network-firewall-logs.md b/solutions/observability/cloud/monitor-aws-network-firewall-logs.md index 13bcd991b..a866f51c5 100644 --- a/solutions/observability/cloud/monitor-aws-network-firewall-logs.md +++ b/solutions/observability/cloud/monitor-aws-network-firewall-logs.md @@ -47,7 +47,7 @@ AWS PrivateLink is not supported. Make sure the deployment is on AWS, because th You can either use an existing AWS Network Firewall, or create a new one for testing purposes. -Creating a Network Firewall is not trivial and is beyond the scope of this guide. For more information, check the AWS documentation on the [Getting started with AWS Network Firewall](https://docs.aws.amazon.com/network-firewall/latest/developerguide/getting-started.md) guide. +Creating a Network Firewall is not trivial and is beyond the scope of this guide. For more information, check the AWS documentation on the [Getting started with AWS Network Firewall](https://docs.aws.amazon.com/network-firewall/latest/developerguide/getting-started.html) guide. ## Step 3: Create a stream in Amazon Data Firehose [firehose-firewall-step-three] diff --git a/solutions/observability/cloud/monitor-cloudtrail-logs.md b/solutions/observability/cloud/monitor-cloudtrail-logs.md index 2d8ce8bbb..410275e01 100644 --- a/solutions/observability/cloud/monitor-cloudtrail-logs.md +++ b/solutions/observability/cloud/monitor-cloudtrail-logs.md @@ -125,7 +125,7 @@ The Amazon Data Firehose delivery stream is ready to send logs to your Elastic C 1. Visit the log group with the CloudTrail events. - Open the log group where the CloudTrail service is sending the events. You must forward these events to an Elastic stack using the Amazon Data Firehose delivery stream. CloudWatch log group offers a [subscription filter](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Subscriptions.md) that allows you to choose log events from the log group and forward them to other services like Amazon Kinesis stream, an Amazon Data Firehose stream, or AWS Lambda. + Open the log group where the CloudTrail service is sending the events. You must forward these events to an Elastic stack using the Amazon Data Firehose delivery stream. CloudWatch log group offers a [subscription filter](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Subscriptions.html) that allows you to choose log events from the log group and forward them to other services like Amazon Kinesis stream, an Amazon Data Firehose stream, or AWS Lambda. 2. Create a subscription filter for Amazon Data Firehose by following these steps. diff --git a/solutions/observability/cloud/monitor-cloudwatch-logs.md b/solutions/observability/cloud/monitor-cloudwatch-logs.md index 6fcf86e7d..080d1c978 100644 --- a/solutions/observability/cloud/monitor-cloudwatch-logs.md +++ b/solutions/observability/cloud/monitor-cloudwatch-logs.md @@ -141,7 +141,7 @@ To send log events from CloudWatch to Firehose, open the log group where the Lam **Create a subscription filter for Amazon Data Firehose** -The [subscription filter](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Subscriptions.md) allows you to pick log events from the log group and forward them to other services, such as an Amazon Kinesis stream, an Amazon Data Firehose stream, or AWS Lambda. +The [subscription filter](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Subscriptions.html) allows you to pick log events from the log group and forward them to other services, such as an Amazon Kinesis stream, an Amazon Data Firehose stream, or AWS Lambda. 1. On the log group page, select **Subscription filters** and click the **Create Amazon Data Firehose subscription filter** button. diff --git a/solutions/observability/get-started/quickstart-monitor-hosts-with-opentelemetry.md b/solutions/observability/get-started/quickstart-monitor-hosts-with-opentelemetry.md index be3d2b918..b88cabbbb 100644 --- a/solutions/observability/get-started/quickstart-monitor-hosts-with-opentelemetry.md +++ b/solutions/observability/get-started/quickstart-monitor-hosts-with-opentelemetry.md @@ -50,7 +50,7 @@ In this quickstart guide, you’ll learn how to monitor your hosts using the Ela ## Limitations [_limitations] -Refer to [Elastic OpenTelemetry Collector limitations](https://github.com/elastic/opentelemetry/blob/main/docs/collector-limitations.md) for known limitations when using the EDOT Collector. +Refer to [Elastic OpenTelemetry Collector limitations](https://github.com/elastic/opentelemetry/blob/main/docs/EDOT-collector/edot-collector-limitations.md) for known limitations when using the EDOT Collector. ## Collect your data [_collect_your_data] diff --git a/solutions/observability/infra-and-hosts/add-symbols-for-native-frames.md b/solutions/observability/infra-and-hosts/add-symbols-for-native-frames.md index 6c11dedc7..12b209755 100644 --- a/solutions/observability/infra-and-hosts/add-symbols-for-native-frames.md +++ b/solutions/observability/infra-and-hosts/add-symbols-for-native-frames.md @@ -35,7 +35,7 @@ You also need to copy the **Symbols** endpoint from the deployment overview page ## Custom C, C++, Go and Rust applications [profiling-symbols-c] -C/C++ applications must be built with debug symbols (`-g`) for symbolization to work. Rust applications must be built with [`debug = 1`](https://doc.rust-lang.org/cargo/reference/profiles.md#debug) (or higher). Go binaries will not require any special compiler flags and come with debug information by default. The debug info doesn’t have to be deployed to production, but it does have to be present temporarily to push it to the Elastic cluster. +C/C++ applications must be built with debug symbols (`-g`) for symbolization to work. Rust applications must be built with [`debug = 1`](https://doc.rust-lang.org/cargo/reference/profiles.html#debug) (or higher). Go binaries will not require any special compiler flags and come with debug information by default. The debug info doesn’t have to be deployed to production, but it does have to be present temporarily to push it to the Elastic cluster. If you don’t mind deploying your applications with debug symbols, run: diff --git a/solutions/observability/observability-ai-assistant.md b/solutions/observability/observability-ai-assistant.md index 38862a062..c10467ce2 100644 --- a/solutions/observability/observability-ai-assistant.md +++ b/solutions/observability/observability-ai-assistant.md @@ -81,7 +81,7 @@ To set up the AI Assistant: * [OpenAI API keys](https://platform.openai.com/docs/api-reference) * [Azure OpenAI Service API keys](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/reference) - * [Amazon Bedrock authentication keys and secrets](https://docs.aws.amazon.com/bedrock/latest/userguide/security-iam.md) + * [Amazon Bedrock authentication keys and secrets](https://docs.aws.amazon.com/bedrock/latest/userguide/security-iam.html) * [Google Gemini service account keys](https://cloud.google.com/iam/docs/keys-list-get) 2. Create a connector for your AI provider. Refer to the connector documentation to learn how: diff --git a/solutions/search/inference-api.md b/solutions/search/inference-api.md index 3a9443d67..76bd3a22d 100644 --- a/solutions/search/inference-api.md +++ b/solutions/search/inference-api.md @@ -39,4 +39,67 @@ To add a new interference endpoint using the UI: 1. Select the **Add endpoint** button. 1. Select a service from the drop down menu. 1. Provide the required configuration details. -1. Select **Save** to create the endpoint. \ No newline at end of file +1. Select **Save** to create the endpoint. + +## Adaptive allocations [adaptive-allocations] + +Adaptive allocations allow inference services to dynamically adjust the number of model allocations based on the current load. + +When adaptive allocations are enabled: + +* The number of allocations scales up automatically when the load increases. +* Allocations scale down to a minimum of 0 when the load decreases, saving resources. + +For more information about adaptive allocations and resources, refer to the trained model autoscaling documentation. + +% TO DO: Add a link to trained model autoscaling when the page is available.% + +## Default {{infer}} endpoints [default-enpoints] + +Your {{es}} deployment contains preconfigured {{infer}} endpoints which makes them easier to use when defining `semantic_text` fields or using {{infer}} processors. The following list contains the default {infer} endpoints listed by `inference_id`: + +* `.elser-2-elasticsearch`: uses the [ELSER](../../explore-analyze/machine-learning/nlp/ml-nlp-elser.md) built-in trained model for `sparse_embedding` tasks (recommended for English language tex). The `model_id` is `.elser_model_2_linux-x86_64`. +* `.multilingual-e5-small-elasticsearch`: uses the [E5](../../explore-analyze/machine-learning/nlp/ml-nlp-e5.md) built-in trained model for `text_embedding` tasks (recommended for non-English language texts). The `model_id` is `.e5_model_2_linux-x86_64`. + +Use the `inference_id` of the endpoint in a [`semantic_text`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/semantic-text.md) field definition or when creating an [{{infer}} processor](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/inference-processor.md). The API call will automatically download and deploy the model which might take a couple of minutes. Default {{infer}} enpoints have adaptive allocations enabled. For these models, the minimum number of allocations is `0`. If there is no {{infer}} activity that uses the endpoint, the number of allocations will scale down to `0` automatically after 15 minutes. + +## Configuring chunking [infer-chunking-config] + +{{infer-cap}} endpoints have a limit on the amount of text they can process at once, determined by the model's input capacity. Chunking is the process of splitting the input text into pieces that remain within these limits. +It occurs when ingesting documents into [`semantic_text` fields](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/semantic-text.md). Chunking also helps produce sections that are digestible for humans. Returning a long document in search results is less useful than providing the most relevant chunk of text. + +Each chunk will include the text subpassage and the corresponding embedding generated from it. + +By default, documents are split into sentences and grouped in sections up to 250 words with 1 sentence overlap so that each chunk shares a sentence with the previous chunk. Overlapping ensures continuity and prevents vital contextual information in the input text from being lost by a hard break. + +{{es}} uses the [ICU4J](https://unicode-org.github.io/icu-docs/) library to detect word and sentence boundaries for chunking. [Word boundaries](https://unicode-org.github.io/icu/userguide/boundaryanalysis/#word-boundary) are identified by following a series of rules, not just the presence of a whitespace character. For written languages that do use whitespace such as Chinese or Japanese dictionary lookups are used to detect word boundaries. + +### Chunking strategies + +Two strategies are available for chunking: `sentence` and `word`. + +The `sentence` strategy splits the input text at sentence boundaries. Each chunk contains one or more complete sentences ensuring that the integrity of sentence-level context is preserved, except when a sentence causes a chunk to exceed a word count of `max_chunk_size`, in which case it will be split across chunks. The `sentence_overlap` option defines the number of sentences from the previous chunk to include in the current chunk which is either `0` or `1`. + +The `word` strategy splits the input text on individual words up to the `max_chunk_size` limit. The `overlap` option is the number of words from the previous chunk to include in the current chunk. + +The default chunking strategy is `sentence`. + +#### Example of configuring the chunking behavior + +The following example creates an {{infer}} endpoint with the `elasticsearch` service that deploys the ELSER model by default and configures the chunking behavior. + +```console +PUT _inference/sparse_embedding/small_chunk_size +{ + "service": "elasticsearch", + "service_settings": { + "num_allocations": 1, + "num_threads": 1 + }, + "chunking_settings": { + "strategy": "sentence", + "max_chunk_size": 100, + "sentence_overlap": 0 + } +} +``` diff --git a/solutions/search/inference-api/amazon-bedrock-inference-integration.md b/solutions/search/inference-api/amazon-bedrock-inference-integration.md index 4012eb9f0..79de7781d 100644 --- a/solutions/search/inference-api/amazon-bedrock-inference-integration.md +++ b/solutions/search/inference-api/amazon-bedrock-inference-integration.md @@ -86,10 +86,10 @@ You need to provide the access and secret keys only once, during the {{infer}} m `model` -: (Required, string) The base model ID or an ARN to a custom model based on a foundational model. The base model IDs can be found in the [Amazon Bedrock model IDs](https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.md) documentation. Note that the model ID must be available for the provider chosen, and your IAM user must have access to the model. +: (Required, string) The base model ID or an ARN to a custom model based on a foundational model. The base model IDs can be found in the [Amazon Bedrock model IDs](https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.html) documentation. Note that the model ID must be available for the provider chosen, and your IAM user must have access to the model. `region` -: (Required, string) The region that your model or ARN is deployed in. The list of available regions per model can be found in the [Model support by AWS region](https://docs.aws.amazon.com/bedrock/latest/userguide/models-regions.md) documentation. +: (Required, string) The region that your model or ARN is deployed in. The list of available regions per model can be found in the [Model support by AWS region](https://docs.aws.amazon.com/bedrock/latest/userguide/models-regions.html) documentation. `rate_limit` : (Optional, object) By default, the `amazonbedrock` service sets the number of requests allowed per minute to `240`. This helps to minimize the number of rate limit errors returned from Amazon Bedrock. To modify this, set the `requests_per_minute` setting of this object in your service settings: @@ -124,7 +124,7 @@ You need to provide the access and secret keys only once, during the {{infer}} m The following example shows how to create an {{infer}} endpoint called `amazon_bedrock_embeddings` to perform a `text_embedding` task type. -Choose chat completion and embeddings models that you have access to from the [Amazon Bedrock base models](https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.md). +Choose chat completion and embeddings models that you have access to from the [Amazon Bedrock base models](https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.html). ```console PUT _inference/text_embedding/amazon_bedrock_embeddings diff --git a/solutions/search/inference-api/chat-completion-inference-api.md b/solutions/search/inference-api/chat-completion-inference-api.md index 97ff911df..1369efee4 100644 --- a/solutions/search/inference-api/chat-completion-inference-api.md +++ b/solutions/search/inference-api/chat-completion-inference-api.md @@ -23,9 +23,9 @@ The {{infer}} APIs enable you to use certain services, such as built-in {{ml}} m ## {{api-request-title}} [chat-completion-inference-api-request] -`POST /_inference//_unified` +`POST /_inference//_stream` -`POST /_inference/chat_completion//_unified` +`POST /_inference/chat_completion//_stream` ## {{api-prereq-title}} [chat-completion-inference-api-prereqs] @@ -38,8 +38,8 @@ The {{infer}} APIs enable you to use certain services, such as built-in {{ml}} m The chat completion {{infer}} API enables real-time responses for chat completion tasks by delivering answers incrementally, reducing response times during computation. It only works with the `chat_completion` task type for `openai` and `elastic` {{infer}} services. -::::{note} -* The `chat_completion` task type is only available within the _unified API and only supports streaming. +::::{note} +* The `chat_completion` task type is only available within the `_stream` API and only supports streaming. * The Chat completion {{infer}} API and the Stream {{infer}} API differ in their response structure and capabilities. The Chat completion {{infer}} API provides more comprehensive customization options through more fields and function calling support. If you use the `openai` service or the `elastic` service, use the Chat completion {{infer}} API. :::: diff --git a/solutions/search/inference-api/elastic-inference-service-eis.md b/solutions/search/inference-api/elastic-inference-service-eis.md index 7b337615a..3eaf36f17 100644 --- a/solutions/search/inference-api/elastic-inference-service-eis.md +++ b/solutions/search/inference-api/elastic-inference-service-eis.md @@ -36,7 +36,7 @@ Creates an {{infer}} endpoint to perform an {{infer}} task with the `elastic` se ::::{note} -The `chat_completion` task type only supports streaming and only through the `_unified` API. +The `chat_completion` task type only supports streaming and only through the `_stream` API. For more information on how to use the `chat_completion` task type, please refer to the [chat completion documentation](/solutions/search/inference-api/chat-completion-inference-api.md). diff --git a/solutions/search/inference-api/openai-inference-integration.md b/solutions/search/inference-api/openai-inference-integration.md index 4bdefa8a8..712922678 100644 --- a/solutions/search/inference-api/openai-inference-integration.md +++ b/solutions/search/inference-api/openai-inference-integration.md @@ -37,7 +37,7 @@ Creates an {{infer}} endpoint to perform an {{infer}} task with the `openai` ser ::::{note} -The `chat_completion` task type only supports streaming and only through the `_unified` API. +The `chat_completion` task type only supports streaming and only through the `_stream` API. For more information on how to use the `chat_completion` task type, please refer to the [chat completion documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/chat-completion-inference-api.html). diff --git a/solutions/search/inference-api/watsonx-inference-integration.md b/solutions/search/inference-api/watsonx-inference-integration.md index 677be506d..2b5c745ff 100644 --- a/solutions/search/inference-api/watsonx-inference-integration.md +++ b/solutions/search/inference-api/watsonx-inference-integration.md @@ -33,9 +33,8 @@ You need an [IBM Cloud® Databases for Elasticsearch deployment](https://cloud.i Available task types: - * `text_embedding`. - - + * `text_embedding`, + * `rerank`. ## {{api-request-body-title}} [infer-service-watsonx-ai-api-request-body] @@ -50,9 +49,9 @@ You need an [IBM Cloud® Databases for Elasticsearch deployment](https://cloud.i `api_key` : (Required, string) A valid API key of your Watsonx account. You can find your Watsonx API keys or you can create a new one [on the API keys page](https://cloud.ibm.com/iam/apikeys). - ::::{important} - You need to provide the API key only once, during the {{infer}} model creation. The [Get {{infer}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-inference-get) does not retrieve your API key. After creating the {{infer}} model, you cannot change the associated API key. If you want to use a different API key, delete the {{infer}} model and recreate it with the same name and the updated API key. - :::: + ::::{important} + You need to provide the API key only once, during the {{infer}} model creation. The [Get {{infer}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-inference-get) does not retrieve your API key. After creating the {{infer}} model, you cannot change the associated API key. If you want to use a different API key, delete the {{infer}} model and recreate it with the same name and the updated API key. + :::: `api_version` @@ -70,13 +69,28 @@ You need an [IBM Cloud® Databases for Elasticsearch deployment](https://cloud.i `rate_limit` : (Optional, object) By default, the `watsonxai` service sets the number of requests allowed per minute to `120`. This helps to minimize the number of rate limit errors returned from Watsonx. To modify this, set the `requests_per_minute` setting of this object in your service settings: - ```text - "rate_limit": { - "requests_per_minute": <> - } - ``` +```json +"rate_limit": { + "requests_per_minute": <> +} +``` + +`task_settings` +: (Optional, object) Settings to configure the inference task. + + These settings are specific to the `` you specified. + +::::{dropdown} `task_settings` for the `rerank` task type +`truncate_input_tokens` +: (Optional, integer) Specifies the maximum number of tokens per input document before truncation. + +`return_documents` +: (Optional, boolean) Specify whether to return doc text within the results. +`top_n` +: (Optional, integer) The number of most relevant documents to return. Defaults to the number of input documents. +:::: ## Watsonx AI service example [inference-example-watsonx-ai] @@ -101,4 +115,35 @@ PUT _inference/text_embedding/watsonx-embeddings 3. The ID of your IBM Cloud project. 4. A valid API version parameter. You can find the active version data parameters [here](https://cloud.ibm.com/apidocs/watsonx-ai#active-version-dates). +The following example shows how to create an inference endpoint called `watsonx-rerank` to perform a `rerank` task type. + +```console + +PUT _inference/rerank/watsonx-rerank +{ + "service": "watsonxai", + "service_settings": { + "api_key": "", <1> + "url": "", <2> + "model_id": "cross-encoder/ms-marco-minilm-l-12-v2", + "project_id": "", <3> + "api_version": "2024-05-02" <4> + }, + "task_settings": { + "truncate_input_tokens": 50, <5> + "return_documents": true, <6> + "top_n": 3 <7> + } +} +``` + +1. A valid Watsonx API key. You can find on the [API keys page of your account](https://cloud.ibm.com/iam/apikeys). +2. The {{infer}} endpoint URL you created on Watsonx. +3. The ID of your IBM Cloud project. +4. A valid API version parameter. You can find the active version data parameters [here](https://cloud.ibm.com/apidocs/watsonx-ai#active-version-dates). +5. The maximum number of tokens per document before truncation. +6. Whether to return the document text in the results. +7. The number of top relevant documents to return. + + diff --git a/solutions/search/ranking/learning-to-rank-ltr.md b/solutions/search/ranking/learning-to-rank-ltr.md index fac42dcef..b3845dea4 100644 --- a/solutions/search/ranking/learning-to-rank-ltr.md +++ b/solutions/search/ranking/learning-to-rank-ltr.md @@ -90,7 +90,7 @@ The LTR space is evolving rapidly and many approaches and model types are being Note that {{es}} supports model inference but the training process itself must happen outside of {{es}}, using a GBDT model. Among the most popular LTR models used today, [LambdaMART](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/MSR-TR-2010-82.pdf) provides strong ranking performance with low inference latencies. It relies on GBDT models and is therefore a perfect fit for LTR in {{es}}. -[XGBoost](https://xgboost.readthedocs.io/en/stable/) is a well known library that provides an [implementation](https://xgboost.readthedocs.io/en/stable/tutorials/learning_to_rank.md) of LambdaMART, making it a popular choice for LTR. We offer helpers in [eland](https://eland.readthedocs.io/) to facilitate the integration of a trained [XBGRanker](https://xgboost.readthedocs.io/en/stable/python/python_api.md#xgboost.XGBRanker) model as your LTR model in {{es}}. +[XGBoost](https://xgboost.readthedocs.io/en/stable/) is a well known library that provides an [implementation](https://xgboost.readthedocs.io/en/stable/tutorials/learning_to_rank.html) of LambdaMART, making it a popular choice for LTR. We offer helpers in [eland](https://eland.readthedocs.io/) to facilitate the integration of a trained [XBGRanker](https://xgboost.readthedocs.io/en/stable/python/python_api.html#xgboost.XGBRanker) model as your LTR model in {{es}}. ::::{tip} Learn more about training in [Train and deploy a LTR model](learning-to-rank-model-training.md), or check out our [interactive LTR notebook](https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/08-learning-to-rank.ipynb) available in the `elasticsearch-labs` repo. diff --git a/solutions/search/ranking/learning-to-rank-model-training.md b/solutions/search/ranking/learning-to-rank-model-training.md index 26d2dadcd..a9ffbb863 100644 --- a/solutions/search/ranking/learning-to-rank-model-training.md +++ b/solutions/search/ranking/learning-to-rank-model-training.md @@ -146,11 +146,11 @@ This method will serialize the trained model and the Learning To Rank configurat The following types of models are currently supported for LTR with {{es}}: -* [`DecisionTreeRegressor`](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeRegressor.md) -* [`RandomForestRegressor`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.md) -* [`LGBMRegressor`](https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMRegressor.md) -* [`XGBRanker`](https://xgboost.readthedocs.io/en/stable/python/python_api.md#xgboost.XGBRanker) -* [`XGBRegressor`](https://xgboost.readthedocs.io/en/stable/python/python_api.md#xgboost.XGBRegressor) +* [`DecisionTreeRegressor`](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeRegressor.html) +* [`RandomForestRegressor`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html) +* [`LGBMRegressor`](https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMRegressor.html) +* [`XGBRanker`](https://xgboost.readthedocs.io/en/stable/python/python_api.html#xgboost.XGBRanker) +* [`XGBRegressor`](https://xgboost.readthedocs.io/en/stable/python/python_api.html#xgboost.XGBRegressor) More model types will be supported in the future. diff --git a/solutions/search/search-templates.md b/solutions/search/search-templates.md index 24f273651..8755a4c6b 100644 --- a/solutions/search/search-templates.md +++ b/solutions/search/search-templates.md @@ -21,7 +21,7 @@ To create or update a search template, use the [create stored script API](https: The request’s `source` supports the same parameters as the [search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search)'s request body.`source` also accepts [Mustache](https://mustache.github.io/) variables, from an open source project [mustache.java](https://github.com/spullara/mustache.java). -Typically [Mustache](https://mustache.github.io/) variables are enclosed in double curly brackets: `{{my-var}}`. When you run a templated search, {{es}} replaces these variables with values from `params`. To learn more about mustache syntax - see [Mustache.js manual](http://mustache.github.io/mustache.5.md) Search templates must use a `lang` of `mustache`. +Typically [Mustache](https://mustache.github.io/) variables are enclosed in double curly brackets: `{{my-var}}`. When you run a templated search, {{es}} replaces these variables with values from `params`. To learn more about mustache syntax - see [Mustache.js manual](http://mustache.github.io/mustache.5.html) Search templates must use a `lang` of `mustache`. The following request creates a search template with an `id` of `my-search-template`. diff --git a/solutions/search/semantic-search/cohere-es.md b/solutions/search/semantic-search/cohere-es.md index 2fda0c058..c88b0010f 100644 --- a/solutions/search/semantic-search/cohere-es.md +++ b/solutions/search/semantic-search/cohere-es.md @@ -258,7 +258,7 @@ Rerank the results using the new {{infer}} endpoint. ```py # Pass the query and the search results to the service -response = client.inference.inference( +response = client.inference.rerank( inference_id="cohere_rerank", body={ "query": query, diff --git a/solutions/search/semantic-search/semantic-search-inference.md b/solutions/search/semantic-search/semantic-search-inference.md index 8546c0b59..d97ada57c 100644 --- a/solutions/search/semantic-search/semantic-search-inference.md +++ b/solutions/search/semantic-search/semantic-search-inference.md @@ -25,7 +25,7 @@ The following examples use the: * models available through [Azure AI Studio](https://ai.azure.com/explore/models?selectedTask=embeddings) or [Azure OpenAI](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models) * `text-embedding-004` model for [Google Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/text-embeddings-api) * `mistral-embed` model for [Mistral](https://docs.mistral.ai/getting-started/models/) -* `amazon.titan-embed-text-v1` model for [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.md) +* `amazon.titan-embed-text-v1` model for [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.html) * `ops-text-embedding-zh-001` model for [AlibabaCloud AI](https://help.aliyun.com/zh/open-search/search-platform/developer-reference/text-embedding-api-details) You can use any Cohere and OpenAI models, they are all supported by the {{infer}} API. For a list of recommended models available on HuggingFace, refer to [the supported model list](../inference-api/huggingface-inference-integration.md#inference-example-hugging-face-supported-models). @@ -561,7 +561,7 @@ PUT amazon-bedrock-embeddings 1. The name of the field to contain the generated tokens. It must be referenced in the {{infer}} pipeline configuration in the next step. 2. The field to contain the tokens is a `dense_vector` field. -3. The output dimensions of the model. This value may be different depending on the underlying model used. See the [Amazon Titan model](https://docs.aws.amazon.com/bedrock/latest/userguide/titan-multiemb-models.md) or the [Cohere Embeddings model](https://docs.cohere.com/reference/embed) documentation. +3. The output dimensions of the model. This value may be different depending on the underlying model used. See the [Amazon Titan model](https://docs.aws.amazon.com/bedrock/latest/userguide/titan-multiemb-models.html) or the [Cohere Embeddings model](https://docs.cohere.com/reference/embed) documentation. 4. For Amazon Bedrock embeddings, the `dot_product` function should be used to calculate similarity for Amazon titan models, or `cosine` for Cohere models. 5. The name of the field from which to create the dense vector representation. In this example, the name of the field is `content`. It must be referenced in the {{infer}} pipeline configuration in the next step. 6. The field type which is text in this example. diff --git a/solutions/security/ai/connect-to-amazon-bedrock.md b/solutions/security/ai/connect-to-amazon-bedrock.md index ae59fd16b..543f3c088 100644 --- a/solutions/security/ai/connect-to-amazon-bedrock.md +++ b/solutions/security/ai/connect-to-amazon-bedrock.md @@ -174,7 +174,7 @@ Finally, configure the connector in {{kib}}: ::::{important} -If you’re using [provisioned throughput](https://docs.aws.amazon.com/bedrock/latest/userguide/prov-throughput.md), your ARN becomes the model ID, and the connector settings **URL** value must be [encoded](https://www.urlencoder.org/) to work. For example, if the non-encoded ARN is `arn:aws:bedrock:us-east-2:123456789102:provisioned-model/3Ztr7hbzmkrqy1`, the encoded ARN would be `arn%3Aaws%3Abedrock%3Aus-east-2%3A123456789102%3Aprovisioned-model%2F3Ztr7hbzmkrqy1`. +If you’re using [provisioned throughput](https://docs.aws.amazon.com/bedrock/latest/userguide/prov-throughput.html), your ARN becomes the model ID, and the connector settings **URL** value must be [encoded](https://www.urlencoder.org/) to work. For example, if the non-encoded ARN is `arn:aws:bedrock:us-east-2:123456789102:provisioned-model/3Ztr7hbzmkrqy1`, the encoded ARN would be `arn%3Aaws%3Abedrock%3Aus-east-2%3A123456789102%3Aprovisioned-model%2F3Ztr7hbzmkrqy1`. :::: diff --git a/solutions/security/ai/connect-to-google-vertex.md b/solutions/security/ai/connect-to-google-vertex.md index e6d4db585..3e34d0dde 100644 --- a/solutions/security/ai/connect-to-google-vertex.md +++ b/solutions/security/ai/connect-to-google-vertex.md @@ -13,7 +13,7 @@ mapped_urls: % - [x] ./raw-migrated-files/security-docs/security/connect-to-vertex.md % - [ ] ./raw-migrated-files/docs-content/serverless/security-connect-to-google-vertex.md -This page provides step-by-step instructions for setting up a Google Vertex AI connector for the first time. This connector type enables you to leverage Vertex AI’s large language models (LLMs) within {{elastic-sec}}. You’ll first need to enable Vertex AI, then generate an API key, and finally configure the connector in your {{elastic-sec}} project. +This page provides step-by-step instructions for setting up a Google Vertex AI connector for the first time. This connector type enables you to leverage Vertex AI’s large language models (LLMs) within {{elastic-sec}}. You’ll first need to enable Vertex AI, then generate a key, and finally configure the connector in your {{elastic-sec}} project. ::::{important} Before continuing, you should have an active project in one of Google Vertex AI’s [supported regions](https://cloud.google.com/vertex-ai/docs/general/locations#feature-availability). @@ -74,7 +74,7 @@ The following video demonstrates these steps. -## Generate an API key [_generate_an_api_key] +## Generate a key [_generate_an_api_key] 1. Return to Vertex AI’s **Credentials** menu and click **Manage service accounts**. 2. Search for the service account you just created, select it, then click the link that appears under **Email**. @@ -108,7 +108,7 @@ Finally, configure the connector in your Elastic deployment: 4. Under **URL**, enter the URL for your region. 5. Enter your **GCP Region** and **GCP Project ID**. 6. Under **Default model**, specify either `gemini-1.5.pro` or `gemini-1.5-flash`. [Learn more about the models](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models). -7. Under **Authentication**, enter your API key. +7. Under **Authentication**, enter your credentials JSON. 8. Click **Save**. The following video demonstrates these steps. diff --git a/solutions/security/cloud.md b/solutions/security/cloud.md index 146c6d926..5db7b36b4 100644 --- a/solutions/security/cloud.md +++ b/solutions/security/cloud.md @@ -6,13 +6,6 @@ mapped_urls: # Cloud Security -% What needs to be done: Align serverless/stateful - -% Use migrated content from existing pages that map to this page: - -% - [x] ./raw-migrated-files/security-docs/security/cloud-native-security-overview.md -% - [ ] ./raw-migrated-files/docs-content/serverless/security-cloud-native-security-overview.md - Elastic Security for Cloud helps you improve your cloud security posture by comparing your cloud configuration to best practices, and scanning for vulnerabilities. It also helps you monitor and investigate your cloud workloads inside and outside Kubernetes. This page describes what each solution does and provides links to more information. @@ -39,13 +32,6 @@ Scans your cloud workloads for known vulnerabilities. When it finds a vulnerabil [Read the CNVM docs](/solutions/security/cloud/cloud-native-vulnerability-management.md). -## Cloud Workload Protection for Kubernetes [_cloud_workload_protection_for_kubernetes] - -Provides cloud-native runtime protections for containerized environments by identifying and (optionally) blocking unexpected system behavior in Kubernetes containers. These capabilities are sometimes referred to as container drift detection and prevention. The solution also captures detailed process and file telemetry from monitored containers, allowing you to set up custom alerts and protection rules. - -[Read the CWP for Kubernetes docs](/solutions/security/cloud/cloud-workload-protection-for-kubernetes.md). - - ## Cloud Workload Protection for VMs [_cloud_workload_protection_for_vms] Helps you monitor and protect your Linux VMs. It uses {{elastic-defend}} to instantly detect and prevent malicious behavior and malware, and captures workload telemetry data for process, file, and network activity. You can use this data with Elastic’s out-of-the-box detection rules and {{ml}} models. These detections generate alerts that quickly help you identify and remediate threats. diff --git a/solutions/security/cloud/benchmarks-2.md b/solutions/security/cloud/benchmarks-2.md deleted file mode 100644 index 6b4c2f4a4..000000000 --- a/solutions/security/cloud/benchmarks-2.md +++ /dev/null @@ -1,59 +0,0 @@ ---- -mapped_urls: - - https://www.elastic.co/guide/en/security/current/benchmark-rules.html - - https://www.elastic.co/guide/en/serverless/current/security-benchmark-rules-kspm.html ---- - -# Benchmarks - -% What needs to be done: Lift-and-shift - -% Use migrated content from existing pages that map to this page: - -% - [x] ./raw-migrated-files/security-docs/security/benchmark-rules.md -% - [ ] ./raw-migrated-files/docs-content/serverless/security-benchmark-rules-kspm.md - -The Benchmarks page lets you view the cloud security posture (CSP) benchmark rules for the [Cloud security posture management](/solutions/security/cloud/cloud-security-posture-management.md) (CSPM) and [Kubernetes security posture management](/solutions/security/cloud/kubernetes-security-posture-management.md) (KSPM) integrations. - -:::{image} ../../../images/security-benchmark-rules.png -:alt: Benchmarks page -:class: screenshot -::: - - -## What are benchmark rules? [_what_are_benchmark_rules_2] - -Benchmark rules are used by the CSPM and KSPM integrations to identify configuration risks in your cloud infrastructure. Benchmark rules are based on the Center for Internet Security’s (CIS) [secure configuration benchmarks](https://www.cisecurity.org/cis-benchmarks/). - -Each benchmark rule checks to see if a specific type of resource is configured according to a CIS Benchmark. The names of rules describe what they check, for example: - -* `Ensure Kubernetes Secrets are encrypted using Customer Master Keys (CMKs) managed in AWS KMS` -* `Ensure the default namespace is not in use` -* `Ensure IAM policies that allow full "*:*" administrative privileges are not attached` -* `Ensure the default namespace is not in use` - -When benchmark rules are evaluated, the resulting [findings](/solutions/security/cloud/findings-page-2.md) data appears on the [Cloud Security Posture dashboard](/solutions/security/dashboards/cloud-security-posture-dashboard.md). - -::::{note} -Benchmark rules are not editable. -:::: - - - -## Review your benchmarks [_review_your_benchmarks_2] - -Find **Benchmarks** in the navigation menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). From there, you can click a benchmark’s name to view the benchmark rules associated with it. You can click a benchmark rule’s name to see details including information about how to remediate it, and related links. - -Benchmark rules are enabled by default, but you can disable some of them — at the benchmark level — to suit your environment. This means for example that if you have two integrations using the `CIS AWS` benchmark, disabling a rule for that benchmark affects both integrations. To enable or disable a rule, use the **Enabled** toggle on the right of the rules table. - -::::{note} -Disabling a benchmark rule automatically disables any associated detection rules and alerts. Re-enabling a benchmark rule **does not** automatically re-enable them. -:::: - - - -## How benchmark rules work [_how_benchmark_rules_work_2] - -1. When a security posture management integration is deployed, and every four hours after that, {{agent}} fetches relevant cloud resources. -2. After resources are fetched, they are evaluated against all applicable enabled benchmark rules. -3. Finding values of `pass` or `fail` indicate whether the standards defined by benchmark rules were met. diff --git a/solutions/security/cloud/benchmarks.md b/solutions/security/cloud/benchmarks.md index 07235b13b..67c56e829 100644 --- a/solutions/security/cloud/benchmarks.md +++ b/solutions/security/cloud/benchmarks.md @@ -2,6 +2,8 @@ mapped_urls: - https://www.elastic.co/guide/en/security/current/cspm-benchmark-rules.html - https://www.elastic.co/guide/en/serverless/current/security-benchmark-rules.html + - https://www.elastic.co/guide/en/serverless/current/security-benchmark-rules-kspm.html + - https://www.elastic.co/guide/en/security/current/benchmark-rules.html --- # Benchmarks diff --git a/solutions/security/cloud/capture-environment-variables.md b/solutions/security/cloud/capture-environment-variables.md index 80dea7db6..2a0475ffe 100644 --- a/solutions/security/cloud/capture-environment-variables.md +++ b/solutions/security/cloud/capture-environment-variables.md @@ -6,20 +6,6 @@ mapped_urls: # Capture environment variables -% What needs to be done: Align serverless/stateful - -% Use migrated content from existing pages that map to this page: - -% - [x] ./raw-migrated-files/security-docs/security/environment-variable-capture.md -% - [ ] ./raw-migrated-files/docs-content/serverless/security-environment-variable-capture.md - -::::{admonition} Requirements -* This feature requires {{stack}} version 8.6 or higher. -* In {{stack}} version 8.6, this feature is only available for Linux. - -:::: - - You can configure an {{agent}} policy to capture up to five environment variables (`env vars`). ::::{note} diff --git a/solutions/security/cloud/cloud-native-vulnerability-management-dashboard.md b/solutions/security/cloud/cloud-native-vulnerability-management-dashboard.md deleted file mode 100644 index cb3869221..000000000 --- a/solutions/security/cloud/cloud-native-vulnerability-management-dashboard.md +++ /dev/null @@ -1,52 +0,0 @@ ---- -mapped_urls: - - https://www.elastic.co/guide/en/security/current/vuln-management-dashboard.html - - https://www.elastic.co/guide/en/serverless/current/_cloud_native_vulnerability_management_dashboard.html ---- - -# Cloud Native Vulnerability Management Dashboard - -% What needs to be done: Align serverless/stateful - -% Scope notes: Duplicate of Cloud Native Vulnerability Management dashboard page in Dashboards section. Consider removing this page and keeping the one in Dashboards. - -% Use migrated content from existing pages that map to this page: - -% - [x] ./raw-migrated-files/security-docs/security/vuln-management-dashboard.md -% - [ ] ./raw-migrated-files/docs-content/serverless/_cloud_native_vulnerability_management_dashboard.md - -The Cloud Native Vulnerability Management (CNVM) dashboard gives you an overview of vulnerabilities detected in your cloud infrastructure. - -:::{image} ../../../images/security-vuln-management-dashboard.png -:alt: The CNVM dashboard -::: - -::::{admonition} Requirements -* To collect this data, install the [Cloud Native Vulnerability Management](/solutions/security/cloud/get-started-with-cnvm.md) integration. -* The CNVM dashboard is available to all Elastic Cloud users. For on-premises deployments, it requires an [Enterprise subscription](https://www.elastic.co/pricing). - -:::: - - - -## CNVM dashboard UI [CNVM-dashboard-UI] - -The summary cards at the top of the dashboard display the number of monitored cloud accounts, scanned virtual machines (VMs), and vulnerabilities (grouped by severity). - -The **Trend by severity** bar graph complements the summary cards by displaying the number of vulnerabilities found on your infrastructure over time, sorted by severity. It has a maximum time scale of 30 days. - -::::{admonition} Graph tips -* Click the severity levels legend on its right to hide/show each severity level. -* To display data from specific cloud accounts, select the account names from the **Accounts** drop-down menu. - -:::: - - -The page also includes three tables: - -* **Top 10 vulnerable resources** shows your VMs with the highest number of vulnerabilities. -* **Top 10 patchable vulnerabilities** shows the most common vulnerabilities in your environment that can be fixed by a software update. -* **Top 10 vulnerabilities** shows the most common vulnerabilities in your environment, with additional details. - -Click **View all vulnerabilities** at the bottom of a table to open the [Vulnerabilities Findings](/solutions/security/cloud/findings-page-3.md) page, where you can view additional details. - diff --git a/solutions/security/cloud/cloud-native-vulnerability-management.md b/solutions/security/cloud/cloud-native-vulnerability-management.md index c8acfc995..24eea054c 100644 --- a/solutions/security/cloud/cloud-native-vulnerability-management.md +++ b/solutions/security/cloud/cloud-native-vulnerability-management.md @@ -6,17 +6,6 @@ mapped_urls: # Cloud native vulnerability management -% What needs to be done: Align serverless/stateful - -% Use migrated content from existing pages that map to this page: - -% - [x] ./raw-migrated-files/security-docs/security/vuln-management-overview.md -% - [ ] ./raw-migrated-files/docs-content/serverless/security-vuln-management-overview.md - -% Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc): - -$$$vuln-management-overview-how-it-works$$$ - Elastic’s Cloud Native Vulnerability Management (CNVM) feature helps you identify known vulnerabilities in your cloud workloads. Setup uses infrastructure as code. For instructions, refer to [Get started with Cloud Native Vulnerability Management](/solutions/security/cloud/get-started-with-cnvm.md). diff --git a/solutions/security/cloud/cloud-security-posture-dashboard-2.md b/solutions/security/cloud/cloud-security-posture-dashboard-2.md deleted file mode 100644 index 2bfcd1c20..000000000 --- a/solutions/security/cloud/cloud-security-posture-dashboard-2.md +++ /dev/null @@ -1,67 +0,0 @@ ---- -mapped_urls: - - https://www.elastic.co/guide/en/security/current/cloud-nat-sec-posture-dashboard.html - - https://www.elastic.co/guide/en/serverless/current/security-cloud-posture-dashboard-dash-kspm.html ---- - -# Cloud Security Posture dashboard - -% What needs to be done: Align serverless/stateful - -% Scope notes: Duplicate of Cloud Security Posture dashboard page in Dashboards section. Consider removing this page and keeping the one in Dashboards. - -% Use migrated content from existing pages that map to this page: - -% - [x] ./raw-migrated-files/security-docs/security/cloud-nat-sec-posture-dashboard.md -% - [ ] ./raw-migrated-files/docs-content/serverless/security-cloud-posture-dashboard-dash-kspm.md - -The Cloud Security Posture dashboard summarizes your cloud infrastructure’s overall performance against [security guidelines](/solutions/security/cloud/benchmarks-2.md) defined by the Center for Internet Security (CIS). To start collecting this data, refer to [Get started with Cloud Security Posture Management](/solutions/security/cloud/get-started-with-cspm-for-aws.md) or [Get started with Kubernetes Security Posture Management](/solutions/security/cloud/get-started-with-kspm.md). - -:::{image} ../../../images/security-cloud-sec-dashboard.png -:alt: The cloud Security dashboard -:class: screenshot -::: - -The Cloud Security Posture dashboard shows: - -* Configuration risk metrics for all monitored cloud accounts and Kubernetes clusters -* Configuration risk metrics grouped by the applicable benchmark, for example, CIS GCP, CIS Azure, CIS Kubernetes, or CIS EKS -* Configuration risks grouped by CIS section (security guideline category) - -::::{admonition} Requirements -* The Cloud Security Posture dashboard is available to all Elastic Cloud users. For on-prem deployments, it requires an [Enterprise subscription](https://www.elastic.co/pricing). - -:::: - - - -## Cloud Security Posture dashboard UI [cloud-nat-sec-posture-dashboard-UI] - -At the top of the dashboard, you can switch between the cloud accounts and Kubernetes cluster views. - -The top section of either view summarizes your overall cloud security posture (CSP) by aggregating data from all monitored resources. The summary cards on the left show the number of cloud accounts or clusters evaluated, and the number of resources evaluated. You can click **Enroll more accounts** or **Enroll more clusters** to deploy to additional cloud assets. Click **View all resources** to open the [Findings page](/solutions/security/cloud/findings-page-2.md). - -The remaining summary cards show your overall compliance score, and your compliance score for each CIS section. Click **View all failed findings** to view all failed findings, or click a CIS section name to view failed findings from only that section on the Findings page. - -Below the summary section, each row shows the CSP for a benchmark that applies to your monitored cloud resources. For example, if you are monitoring EKS and Kubernetes clusters, a row appears for CIS EKS and another appears for CIS Kubernetes. Each row shows the CIS benchmark, the number of clusters it applies to, its overall compliance score, and its compliance score grouped by CIS section. - -:::{image} ../../../images/security-cloud-sec-dashboard-individual-row.png -:alt: A row representing a single cluster in the Cloud Security Posture dashboard -:class: screenshot -::: - - -## FAQ (Frequently Asked Questions) [cloud-nat-sec-posture-dashboard-faq] - -::::{dropdown} When do newly-enrolled assets appear on the dashboard? -It can take up to 10 minutes for deployment, resource fetching, evaluation, and data processing before a newly-enrolled AWS account or Kubernetes cluster appears on the dashboard. - -:::: - - -::::{dropdown} When do unenrolled clusters disappear from the dashboard? -A cluster will disappear as soon as your integration fetches data while that cluster is not enrolled. The fetch process repeats every four hours, which means a newly unenrolled cluster can take a maximum of four hours to disappear from the dashboard. - -:::: - - diff --git a/solutions/security/cloud/cloud-security-posture-dashboard.md b/solutions/security/cloud/cloud-security-posture-dashboard.md deleted file mode 100644 index 93d30531c..000000000 --- a/solutions/security/cloud/cloud-security-posture-dashboard.md +++ /dev/null @@ -1,67 +0,0 @@ ---- -mapped_urls: - - https://www.elastic.co/guide/en/security/current/cspm-posture-dashboard.html - - https://www.elastic.co/guide/en/serverless/current/security-cloud-posture-dashboard-dash-cspm.html ---- - -# Cloud Security Posture dashboard - -% What needs to be done: Align serverless/stateful - -% Scope notes: Duplicate of Cloud Security Posture dashboard page in Dashboards section. Consider removing this page and keeping the one in Dashboards. - -% Use migrated content from existing pages that map to this page: - -% - [x] ./raw-migrated-files/security-docs/security/cspm-posture-dashboard.md -% - [ ] ./raw-migrated-files/docs-content/serverless/security-cloud-posture-dashboard-dash-cspm.md - -The Cloud Security Posture dashboard summarizes your cloud infrastructure’s overall performance against [security guidelines](/solutions/security/cloud/benchmarks-2.md) defined by the Center for Internet Security (CIS). To get started monitoring your security posture, refer to [Get started with Cloud Security Posture Management](/solutions/security/cloud/get-started-with-cspm-for-aws.md) or [Get started with Kubernetes Security Posture Management](/solutions/security/cloud/get-started-with-kspm.md). - -:::{image} ../../../images/security-cloud-sec-dashboard.png -:alt: The cloud Security dashboard -:class: screenshot -::: - -The Cloud Security Posture dashboard shows: - -* Configuration risk metrics for all monitored cloud accounts and Kubernetes clusters -* Configuration risk metrics grouped by the applicable benchmark, for example CIS GCP, CIS Azure, CIS Kubernetes, or CIS EKS -* Configuration risks grouped by CIS section (security guideline category) - -::::{admonition} Requirements -* The Cloud Security Posture dashboard is available to all Elastic Cloud users. For on-prem deployments, it requires an [Enterprise subscription](https://www.elastic.co/pricing). - -:::: - - - -## Cloud Security Posture dashboard UI [cspm-posture-dashboard-UI] - -At the top of the dashboard, you can switch between the cloud accounts and Kubernetes cluster views. - -The top section of either view summarizes your overall cloud security posture (CSP) by aggregating data from all monitored resources. The summary cards on the left show the number of cloud accounts or clusters evaluated, and the number of resources evaluated. You can click **Enroll more accounts** or **Enroll more clusters** to deploy to additional cloud assets. Click **View all resources** to open the [Findings page](/solutions/security/cloud/findings-page-2.md). - -The remaining summary cards show your overall compliance score, and your compliance score for each CIS section. Click **View all failed findings** to view all failed findings, or click a CIS section name to view failed findings from only that section on the Findings page. - -Below the summary section, each row shows the CSP for a benchmark that applies to your monitored cloud resources. For example, if you are monitoring GCP and Azure cloud accounts, a row appears for CIS GCP and another appears for CIS Azure. Each row shows the CIS benchmark, the number of cloud accounts it applies to, its overall compliance score, and its compliance score grouped by CIS section. - -:::{image} ../../../images/security-cloud-sec-dashboard-individual-row.png -:alt: A row representing a single cluster in the Cloud Security Posture dashboard -:class: screenshot -::: - - -## FAQ (Frequently Asked Questions) [cspm-posture-dashboard-faq] - -::::{dropdown} When do newly-enrolled assets appear on the dashboard? -It can take up to 10 minutes for deployment, resource fetching, evaluation, and data processing before a newly-enrolled AWS account or Kubernetes cluster appears on the dashboard. - -:::: - - -::::{dropdown} When do unenrolled accounts disappear from the dashboard? -An account will disappear as soon as your integration fetches data while that account is not enrolled. The fetch process repeats every four hours, which means a newly unenrolled account can take a maximum of four hours to disappear from the dashboard. - -:::: - - diff --git a/solutions/security/cloud/cloud-security-posture-management.md b/solutions/security/cloud/cloud-security-posture-management.md index 46b983506..210d653e4 100644 --- a/solutions/security/cloud/cloud-security-posture-management.md +++ b/solutions/security/cloud/cloud-security-posture-management.md @@ -6,42 +6,21 @@ mapped_urls: # Cloud security posture management -% What needs to be done: Align serverless/stateful - -% Use migrated content from existing pages that map to this page: - -% - [x] ./raw-migrated-files/security-docs/security/cspm.md -% - [ ] ./raw-migrated-files/docs-content/serverless/security-cspm.md - The Cloud Security Posture Management (CSPM) feature discovers and evaluates the services in your cloud environment — like storage, compute, IAM, and more — against configuration security guidelines defined by the [Center for Internet Security](https://www.cisecurity.org/) (CIS) to help you identify and remediate risks that could undermine the confidentiality, integrity, and availability of your cloud data. This feature currently supports agentless and agent-based deployments on Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure. For step-by-step getting started guides, refer to [Get started with CSPM for AWS](/solutions/security/cloud/get-started-with-cspm-for-aws.md), [Get started with CSPM for GCP](/solutions/security/cloud/get-started-with-cspm-for-gcp.md), or [Get started with CSPM for Azure](/solutions/security/cloud/get-started-with-cspm-for-azure.md). ::::{admonition} Requirements -* CSPM is available to all {{ecloud}} users. On-premise deployments require an [Enterprise subscription](https://www.elastic.co/pricing). -* {{stack}} version 8.10 or greater. +* Minimum privileges vary depending on whether you need to read, write, or manage CSPM data and integrations. Refer to [CSPM privilege requirements](/solutions/security/cloud/cspm-privilege-requirements.md). +* The CSPM integration is available to all {{ecloud}} users. On-premise deployments require an [Enterprise subscription](https://www.elastic.co/pricing). * CSPM only works in the `Default` {{kib}} space. Installing the CSPM integration on a different {{kib}} space will not work. * CSPM is supported only on AWS, GCP, and Azure commercial cloud platforms, and AWS GovCloud. Other government cloud platforms are not supported. [Click here to request support](https://github.com/elastic/kibana/issues/new/choose). -* `Read` privileges for the following {{es}} indices: - - * `logs-cloud_security_posture.findings_latest-*` - * `logs-cloud_security_posture.scores-*` - -* The following {{kib}} privileges: - - * Security: `Read` - * Integrations: `Read` - * Saved Objects Management: `Read` - * Fleet: `All` - :::: - - ## How CSPM works [cspm-how-it-works] -Using the read-only credentials you will provide during the setup process, it will evaluate the configuration of resources in your environment every 24 hours. After each evaluation, the integration sends findings to Elastic. A high-level summary of the findings appears on the [Cloud Security Posture dashboard](/solutions/security/cloud/cloud-security-posture-dashboard-2.md), and detailed findings appear on the [Findings page](/solutions/security/cloud/findings-page-2.md). +Using the read-only credentials you will provide during the setup process, it will evaluate the configuration of resources in your environment every 24 hours. After each evaluation, the integration sends findings to Elastic. A high-level summary of the findings appears on the [Cloud Security Posture dashboard](/solutions/security/dashboards/cloud-security-posture-dashboard.md), and detailed findings appear on the [Findings page](/solutions/security/cloud/findings-page-2.md). diff --git a/solutions/security/cloud/cloud-workload-protection-for-kubernetes.md b/solutions/security/cloud/cloud-workload-protection-for-kubernetes.md deleted file mode 100644 index 4f85b8ecb..000000000 --- a/solutions/security/cloud/cloud-workload-protection-for-kubernetes.md +++ /dev/null @@ -1,63 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/security/current/d4c-overview.html ---- - -# Cloud workload protection for Kubernetes [d4c-overview] - -::::{warning} -This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features. -:::: - - -Elastic Cloud Workload Protection (CWP) for Kubernetes provides cloud-native runtime protections for containerized environments by identifying and optionally blocking unexpected system behavior in Kubernetes containers. - - -## Use cases [d4c-use-cases] - - -### Threat detection & threat hunting [_threat_detection_threat_hunting] - -CWP for Kubernetes sends system events from your containers to {{es}}. {{elastic-sec}}'s prebuilt security rules include many designed to detect malicious behavior in container runtimes. These can help you detect events that should never occur in containers, such as reverse shell executions, privilege escalation, container escape attempts, and more. - - -### Drift detection & prevention [_drift_detection_prevention] - -Cloud-native containers should be immutable, meaning that their file systems should not change during normal operations. By leveraging this principle, security teams can detect unusual system behavior with a high degree of accuracy — without relying on more resource-intensive techniques like memory scanning or attack signature detection. Elastic’s Drift Detection mechanism has a low rate of false positives, so you can deploy it in most environments without worrying about creating excessive alerts. - - -### Workload protection policies [_workload_protection_policies] - -CWP for Kubernetes uses a flexible policy language to restrict container workloads to a set of allowlisted capabilities chosen by you. When employed with Drift and Threat Detection, this can provide multiple layers of defense. - - -## Support matrix: [_support_matrix] - -| | EKS 1.24-1.27 (AL2022) | GKE 1.24-1.27 (COS) | -| --- | --- | --- | -| Process event exports | ✓ | ✓ | -| Network event exports | ✓ | ✓ | -| File event exports | ✓ | ✓ | -| File blocking | ✓ | ✓ | -| Process blocking | ✓ | ✓ | -| Network blocking | ✗ | ✗ | -| Drift prevention | ✓ | ✓ | -| Mount point awareness | ✓ | ✓ | - - -## How CWP for Kubernetes works [_how_cwp_for_kubernetes_works] - -CWP for Kubernetes uses a lightweight integration, Defend for Containers (D4C). When you set up the D4C integration, it gets deployed by {{agent}}. Specifically, the {{agent}} is installed as a DaemonSet on your Kubernetes clusters, where it enables D4C to use eBPF Linux Security Modules ([LSM](https://docs.kernel.org/bpf/prog_lsm.md)) and tracepoint probes to record system events. Events are evaluated against LSM hook points, enabling {{agent}} to evaluate system activity against your policy before allowing it to proceed. - -Your D4C integration policy determines which system behaviors (for example, process execution or file creation or deletion) will result in which actions. *Selectors* and *responses* define each policy. Selectors define the conditions which cause the associated responses to run. Responses are associated with one or more selectors, and specify one or more actions (such as `log`, `alert`, or `block`) that will occur when the conditions defined in an associated selector are met. - -The default D4C policy sends data about all running processes to your {{es}} cluster. This data is used by {{elastic-sec}}'s prebuilt detection rules to detect malicious behavior in container workloads. - -::::{important} -To learn more about D4C policies, including how to create your own, refer to the [D4C policies guide](container-workload-protection-policies.md). -:::: - - - - - diff --git a/solutions/security/cloud/cloud-workload-protection-for-vms.md b/solutions/security/cloud/cloud-workload-protection-for-vms.md index ac8f6f914..e41cecee8 100644 --- a/solutions/security/cloud/cloud-workload-protection-for-vms.md +++ b/solutions/security/cloud/cloud-workload-protection-for-vms.md @@ -6,12 +6,6 @@ mapped_urls: # Cloud workload protection for VMs -% What needs to be done: Align serverless/stateful - -% Use migrated content from existing pages that map to this page: - -% - [x] ./raw-migrated-files/security-docs/security/cloud-workload-protection.md -% - [ ] ./raw-migrated-files/docs-content/serverless/security-cloud-workload-protection.md Cloud workload protection helps you monitor and protect your Linux VMs. It uses the [{{elastic-defend}}](/solutions/security/configure-elastic-defend/install-elastic-defend.md) integration to capture cloud workload telemetry containing process, file, and network activity. @@ -28,5 +22,4 @@ To continue setting up your cloud workload protection, learn more about: * [**Getting started with {{elastic-defend}}**](/solutions/security/configure-elastic-defend/install-elastic-defend.md): configure {{elastic-defend}} to protect your hosts. Be sure to select one of the "Cloud workloads" presets if you want to collect session data by default, including process, file, and network telemetry. * [**Session view**](/solutions/security/investigate/session-view.md): examine Linux process data organized in a tree-like structure according to the Linux logical event model, with processes organized by parentage and time of execution. Use it to monitor and investigate session activity, and to understand user and service behavior on your Linux infrastructure. -* [**The Kubernetes dashboard**](/solutions/security/cloud/kubernetes-dashboard.md): Explore an overview of your protected Kubernetes clusters, and drill down into individual sessions within your Kubernetes infrastructure. * [**Environment variable capture**](/solutions/security/cloud/capture-environment-variables.md): Capture the environment variables associated with process events, such as `PATH`, `LD_PRELOAD`, or `USER`. diff --git a/solutions/security/cloud/frequently-asked-questions-faq-3.md b/solutions/security/cloud/cnvm-frequently-asked-questions-faq.md similarity index 92% rename from solutions/security/cloud/frequently-asked-questions-faq-3.md rename to solutions/security/cloud/cnvm-frequently-asked-questions-faq.md index 3c0eb16db..00e023ef9 100644 --- a/solutions/security/cloud/frequently-asked-questions-faq-3.md +++ b/solutions/security/cloud/cnvm-frequently-asked-questions-faq.md @@ -6,15 +6,6 @@ mapped_pages: # Frequently asked questions (FAQ) -% What needs to be done: Lift-and-shift - -% Use migrated content from existing pages that map to this page: - -% - [x] ./raw-migrated-files/security-docs/security/vuln-management-faq.md -% - [ ] ./raw-migrated-files/docs-content/serverless/security-vuln-management-faq.md - - - Frequently asked questions about the Cloud Native Vulnerability Management (CNVM) integration and features. **Which security data sources does the CNVM integration use to identify vulnerabilities?** diff --git a/solutions/security/cloud/container-workload-protection-policies.md b/solutions/security/cloud/container-workload-protection-policies.md deleted file mode 100644 index 4d3c3e6f7..000000000 --- a/solutions/security/cloud/container-workload-protection-policies.md +++ /dev/null @@ -1,96 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/security/current/d4c-policy-guide.html ---- - -# Container workload protection policies [d4c-policy-guide] - -To unlock the full functionality of the Defend for Containers (D4C) integration, you’ll need to understand its policy syntax. This will enable you to construct policies that precisely allow expected container behaviors and prevent unexpected behaviors — thereby hardening your container workloads' security posture. - -D4C integration policies consist of *selectors* and *responses*. Each policy must contain at least one selector and one response. Currently, the system supports two types of selectors and responses: `file` and `process`. Selectors define which system operations to match and can include multiple conditions (grouped using a logical `AND`) to precisely select events. Responses define which actions to take when a system operation matches the conditions specified in an associated selector. - -The default policy described on this page provides an example that’s useful for understanding D4C policies in general. Following the description, you’ll find a comprehensive glossary of selector conditions, response fields, and actions. - - -## Default policies: [d4c-default-policies] - -The default D4C integration policy includes two selector-response pairs. It is designed to implement core container workload protection capabilities: - -* **Threat Detection:** The first selector-response pair is designed to stream process telemetry data to your {{es}} cluster so {{elastic-sec}} can evaluate it to detect threats. Both the selector and response are named `allProcesses`. The selector selects all fork and exec events. The associated response specifies that selected events should be logged. -* **Drift Detection & Prevention:** The second selector-response pair is designed to create alerts when container drift is detected. Both the selector and response are named `executableChanges`. The selector selects all `createExecutable` and `modifyExecutable` events. The associated response specifies that the selected events should create alerts, which will be sent to your {{es}} cluster. You can modify the response to block drift operations by setting it to block. - -:::{image} ../../../images/security-d4c-policy-editor.png -:alt: The defend for containers policy editor with the default policies -::: - - -## Selectors [d4c-selectors-glossary] - -A selector requires a name and at least one operation. It will select all events of the specified operation types, unless you also include *conditions* to narrow down the selection. Some conditions are available for both `file` and `process` selectors, while others only available for one type of selector. - - -### Common conditions [_common_conditions] - -These conditions are available for both `file` and `process` selectors. - -| Name | Description | -| --- | --- | -| containerImageFullName | A list of full container image names to match on. For example: `docker.io/nginx`. | -| containerImageName | A list of container image names to match on. For example: `nginx`. | -| containerImageTag | A list of container image tags to match on. For example: `latest`. | -| kubernetesClusterId | A list of Kubernetes cluster IDs to match on. For consistency with KSPM, the `kube-system` namespace’s UID is used as a cluster ID. | -| kubernetesClusterName | A list of Kubernetes cluster names to match on. | -| kubernetesNamespace | A list of Kubernetes namespaces to match on. | -| kubernetesPodName | A list of Kubernetes pod names to match on. Trailing wildcards supported. | -| kubernetesPodLabel | A list of resource labels. Trailing wildcards supported (value only), for example: `key1:val*`. | - - -### File-selector conditions [_file_selector_conditions] - -These conditions are available only for `file` selectors. - -| Name | Description | -| --- | --- | -| operation | The list of system operations to match on. Options include `createExecutable`, `modifyExecutable`, `createFile`, `modifyFile`, `deleteFile`. | -| ignoreVolumeMounts | If set, ignores file operations on *all* volume mounts. | -| ignoreVolumeFiles | If set, ignores operations on file mounts only. For example: mounted files, `configMaps`, and secrets. | -| targetFilePath | A list of file paths to include. Paths are absolute and wildcards are supported. The `*` wildcard matches any sequence of characters within a single directory, while the `**` wildcard matches any sequence of characters across multiple directories and subdirectories. | - -::::{note} -In order to ensure precise targeting of file integrity monitoring operations, a `TargetFilePath` is required whenever the `deleteFile`, `modifyFile`, or `createFile` operations are used within a selector. -:::: - - - -### Process-selector conditions [_process_selector_conditions] - -These conditions are available only for `process` selectors. - -| Name | Description | -| --- | --- | -| operation | The list of system operations to match on. Options include `fork` and `exec`. | -| processExecutable | A list of executables (full path included) to match on. For example: `/usr/bin/cat`. Wildcard support is same as targetFilePath above. | -| processName | A list of process names (executable basename) to match on. For example: `bash`, `vi`, `cat`. | -| sessionLeaderInteractive | If set to `true`, will only match on interactive sessions (defined as sessions with a controlling TTY). | - - -### Response fields [_response_fields] - -A policy can include one or more responses. Each response is comprised of the following fields: - -| Field | Description | -| --- | --- | -| match | An array of one or more selectors of the same type (`file` or `process`). | -| exclude | Optional. An array of one or more selectors to use as exclusions to everything in `match`. | -| actions | An array of actions to perform when at least one `match` selector matches and none of the `exclude` selectors match. Options include `log`, `alert`, and `block`. | - - -### Response actions [_response_actions] - -D4C responses can include the following actions: - -| Action | Description | -| --- | --- | -| log | Sends events to the `logs-cloud_defend.file-*` data stream for file responses, and the `logs-cloud_defend.process-*` data stream for process responses. | -| alert | Writes events (file or process) to the logs-cloud_defend.alerts-* data stream. | -| block | Prevents the system operation from proceeding. This blocking action happens prior to the execution of the event. It is required that the alert action be set if block is enabled.

**Note:** Currently, block is only supported on file operations.
| diff --git a/solutions/security/cloud/frequently-asked-questions-faq.md b/solutions/security/cloud/cspm-frequently-asked-questions-faq.md similarity index 93% rename from solutions/security/cloud/frequently-asked-questions-faq.md rename to solutions/security/cloud/cspm-frequently-asked-questions-faq.md index e21587952..c2c8e8670 100644 --- a/solutions/security/cloud/frequently-asked-questions-faq.md +++ b/solutions/security/cloud/cspm-frequently-asked-questions-faq.md @@ -2,17 +2,12 @@ mapped_urls: - https://www.elastic.co/guide/en/security/current/cspm-security-posture-faq.html - https://www.elastic.co/guide/en/serverless/current/security-cspm-security-posture-faq.html + - https://www.elastic.co/guide/en/serverless/current/security-posture-faq.html + - https://www.elastic.co/guide/en/security/current/security-posture-faq.html --- # Frequently asked questions (FAQ) -% What needs to be done: Lift-and-shift - -% Use migrated content from existing pages that map to this page: - -% - [x] ./raw-migrated-files/security-docs/security/cspm-security-posture-faq.md -% - [ ] ./raw-migrated-files/docs-content/serverless/security-cspm-security-posture-faq.md - ## CSPM FAQ [_cspm_faq] diff --git a/solutions/security/cloud/cspm-privilege-requirements.md b/solutions/security/cloud/cspm-privilege-requirements.md index e5fe2be1e..b92a219bc 100644 --- a/solutions/security/cloud/cspm-privilege-requirements.md +++ b/solutions/security/cloud/cspm-privilege-requirements.md @@ -6,13 +6,6 @@ mapped_urls: # CSPM privilege requirements -% What needs to be done: Lift-and-shift - -% Use migrated content from existing pages that map to this page: - -% - [x] ./raw-migrated-files/security-docs/security/cspm-required-permissions.md -% - [ ] ./raw-migrated-files/docs-content/serverless/cspm-required-permissions.md - This page lists required privileges for {{elastic-sec}}'s CSPM features. There are three access levels: read, write, and manage. Each access level and its requirements are described below. diff --git a/solutions/security/cloud/enable-cloud-security-features.md b/solutions/security/cloud/enable-cloud-security-features.md index 7c7cb9725..4856e31b2 100644 --- a/solutions/security/cloud/enable-cloud-security-features.md +++ b/solutions/security/cloud/enable-cloud-security-features.md @@ -1,28 +1,21 @@ --- mapped_pages: - https://www.elastic.co/guide/en/serverless/current/security-enable-cloudsec.html +applies_to: + serverless: all --- -# Enable cloud security features [security-enable-cloudsec] +# Enable cloud security features in {{serverless-short}} [security-enable-cloudsec] -To use cloud security features in your {{elastic-sec}} project, you must have the `Cloud Protection Essentials` or `Cloud Protection Complete` options enabled for your project. + +To use cloud security features in your {{serverless-full}} project, you must have the `Cloud Protection Essentials` or `Cloud Protection Complete` options enabled for your project. To enable these options or check their current status: 1. Click your project name in the upper-left corner of {{kib}}. Select **Manage project**. - :::{image} ../../../images/serverless-manage-project.png - :alt: The project menu with the manage project button highlighted - :class: screenshot - ::: - 2. To the right of **Project features**, select **Edit**. - :::{image} ../../../images/serverless-project-features-edit.png - :alt: The project menu with the manage project button highlighted - :class: screenshot - ::: - 3. Enable the necessary options, then click **Save**. Continue with cloud security setup. diff --git a/solutions/security/cloud/findings-page-2.md b/solutions/security/cloud/findings-page-2.md index 3e688f7f9..cfd9d324c 100644 --- a/solutions/security/cloud/findings-page-2.md +++ b/solutions/security/cloud/findings-page-2.md @@ -6,16 +6,6 @@ mapped_urls: # Findings page -% What needs to be done: Align serverless/stateful - -% Use migrated content from existing pages that map to this page: - -% - [x] ./raw-migrated-files/security-docs/security/findings-page.md -% - [ ] ./raw-migrated-files/docs-content/serverless/security-cspm-findings-page-kspm-kspm.md - -% Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc): - -$$$cspm-findings-page-filter-findings-kspm$$$ The **Misconfigurations** tab on the Findings page displays the configuration risks identified by the [CSPM](/solutions/security/cloud/cloud-security-posture-management.md) and [KSPM](/solutions/security/cloud/kubernetes-security-posture-management.md) integrations. @@ -37,13 +27,14 @@ By default, the Findings page lists all findings, without grouping or filtering. ### Group findings [_group_findings_2] -1. Click **Group findings by** to group your data by a field. Select one of the suggested fields or **Custom field** to choose your own. You can select up to three group fields at once. -2. When grouping is turned on, click a group to expand it and examine all sub-groups or findings within that group. -3. To turn off grouping, click **Group findings by** and select **None**. +Click **Group findings by** to group your data by a field. Select one of the suggested fields or **Custom field** to choose your own. You can select up to three group fields at once. -::::{note} -Multiple groupings apply to your data in the order you selected them. For example, if you first select **Kubernetes cluster**, then select **Resource***, the top-level grouping will be based on ***Kubernetes cluster**, and its subordinate grouping will be based on **Resource**. -:::: +* When grouping is turned on, click a group to expand it and examine all sub-groups or findings within that group. +* To turn off grouping, click **Group findings by** and select **None**. + + ::::{note} + Multiple groupings apply to your data in the order you selected them. For example, if you first select **Cloud account**, then select **Resource**, the top-level grouping will be based on **Cloud account**, and its subordinate grouping will be based on **Resource**. + :::: @@ -77,9 +68,9 @@ To remediate failed findings and reduce your attack surface: 2. Click the arrow to the left of a failed finding to open the findings flyout. 3. Follow the steps under **Remediation**. - ::::{note} - Remediation steps typically include commands for you to execute. These sometimes contain placeholder values that you must replace before execution. - :::: + ::::{note} + Remediation steps typically include commands for you to execute. These sometimes contain placeholder values that you must replace before execution. + :::: diff --git a/solutions/security/cloud/findings-page-3.md b/solutions/security/cloud/findings-page-3.md index 715f4104e..a0ec1fcac 100644 --- a/solutions/security/cloud/findings-page-3.md +++ b/solutions/security/cloud/findings-page-3.md @@ -34,7 +34,7 @@ Click **Group vulnerabilities by** to group your data by a field. Select one of * To turn off grouping, click **Group vulnerabilities by:** and select **None**. ::::{note} -Multiple groupings apply to your data in the order you selected them. For example, if you first select **Cloud account**, then select **Resource***, the top-level grouping will be based on ***Cloud account**, and its subordinate grouping will be based on **Resource**, as demonstrated in the following screenshot. +Multiple groupings apply to your data in the order you selected them. For example, if you first select **Cloud account**, then select **Resource**, the top-level grouping will be based on **Cloud account**, and its subordinate grouping will be based on **Resource**, as demonstrated in the following screenshot. :::: diff --git a/solutions/security/cloud/findings-page.md b/solutions/security/cloud/findings-page.md index 40339c262..c97a3f907 100644 --- a/solutions/security/cloud/findings-page.md +++ b/solutions/security/cloud/findings-page.md @@ -6,13 +6,6 @@ mapped_urls: # Findings page -% What needs to be done: Align serverless/stateful - -% Use migrated content from existing pages that map to this page: - -% - [x] ./raw-migrated-files/security-docs/security/cspm-findings-page.md -% - [ ] ./raw-migrated-files/docs-content/serverless/security-cspm-findings-page.md - % Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc): $$$cspm-findings-page-filter-findings$$$ @@ -37,11 +30,14 @@ By default, the Findings page lists all findings, without grouping or filtering. ### Group findings [_group_findings] -Click **Group findings by** to group your data by a field. Select one of the suggested fields or **Custom field*** to choose your own. You can select up to three group fields at once. . When grouping is turned on, click a group to expand it and examine all sub-groups or findings within that group. . To turn off grouping, click ***Group findings by** and select **None**. +Click **Group findings by** to group your data by a field. Select one of the suggested fields or **Custom field** to choose your own. You can select up to three group fields at once. -::::{note} -Multiple groupings apply to your data in the order you selected them. For example, if you first select **Cloud account**, then select **Resource***, the top-level grouping will be based on ***Cloud account**, and its subordinate grouping will be based on **Resource**. -:::: +* When grouping is turned on, click a group to expand it and examine all sub-groups or findings within that group. +* To turn off grouping, click **Group findings by** and select **None**. + + ::::{note} + Multiple groupings apply to your data in the order you selected them. For example, if you first select **Cloud account**, then select **Resource**, the top-level grouping will be based on **Cloud account**, and its subordinate grouping will be based on **Resource**. + :::: @@ -75,9 +71,9 @@ To remediate failed findings and reduce your attack surface: 2. Click the arrow to the left of a failed finding to open the findings flyout. 3. Follow the steps under **Remediation**. - ::::{note} - Remediation steps typically include commands for you to execute. These sometimes contain placeholder values that you must replace before execution. - :::: +::::{note} +Remediation steps typically include commands for you to execute. These sometimes contain placeholder values that you must replace before execution. +:::: diff --git a/solutions/security/cloud/frequently-asked-questions-faq-2.md b/solutions/security/cloud/frequently-asked-questions-faq-2.md deleted file mode 100644 index 49147dd59..000000000 --- a/solutions/security/cloud/frequently-asked-questions-faq-2.md +++ /dev/null @@ -1,80 +0,0 @@ ---- -mapped_urls: - - https://www.elastic.co/guide/en/security/current/security-posture-faq.html - - https://www.elastic.co/guide/en/serverless/current/security-posture-faq.html ---- - -# Frequently asked questions (FAQ) - -% What needs to be done: Lift-and-shift - -% Use migrated content from existing pages that map to this page: - -% - [x] ./raw-migrated-files/security-docs/security/security-posture-faq.md -% - [ ] ./raw-migrated-files/docs-content/serverless/security-posture-faq.md - - -## CSPM FAQ [cspm-faq] - -Frequently asked questions about the Cloud Security Posture Management (CSPM) integration and features. - -**How often is my cloud security posture evaluated?** - -Cloud accounts are evaluated when you first deploy the CSPM integration and every 24 hours afterward. - -**Can I onboard multiple accounts at one time?** - -Yes. Follow the onboarding instructions in the getting started guides for AWS, GCP, or Azure. - -**When do newly enrolled cloud accounts appear on the dashboard?** - -After you deploy the CSPM integration, it can take up to 10 minutes for resource fetching, evaluation, and data processing before a newly enrolled account appears on the Cloud Security Posture dashboard. - -**When do unenrolled cloud accounts disappear from the dashboard?** - -Newly unenrolled cloud accounts can take a maximum of 24 hours to disappear from the Cloud Security Posture dashboard. - - -## KSPM FAQ [kspm-faq] - -Frequently asked questions about the Kubernetes Security Posture Management (KSPM) integration and features. - -**What versions of Kubernetes are supported?** - -For self-managed/vanilla and EKS clusters, Kubernetes version 1.23 is supported. - -**Do benchmark rules support multiple Kubernetes deployment types?** Yes. There are different sets of benchmark rules for self-managed and third party-managed deployments. Refer to [Get started with KSPM](/solutions/security/cloud/get-started-with-kspm.md) for more information about setting up each deployment type. - -**Can I evaluate the security posture of my Amazon EKS clusters?** Yes. KSPM currently supports the security posture evaluation of Amazon EKS and unmanaged Kubernetes clusters. - -**How often is my cluster’s security posture evaluated?** Clusters are evaluated when you deploy a KSPM integration, and every four hours after that. - -**When do newly-enrolled clusters appear on the dashboard?** It can take up to 10 minutes for deployment, resource fetching, evaluation, and data processing to complete before a newly-enrolled cluster appears on the dashboard. - -**When do unenrolled clusters disappear from the dashboard?** A cluster will disappear as soon as the KSPM integration fetches data while that cluster is not enrolled. The fetch process repeats every four hours, which means a newly unenrolled cluster can take a maximum of four hours to disappear from the dashboard. - - -## Findings page [_findings_page_2] - -**Are all the findings page current?** Yes. Only the most recent findings appear on the Findings page. - -**Can I build custom visualizations and dashboards that incorporate findings data?** Yes. You can use {{kib}}'s custom visualization capabilities with findings data. To learn more, refer to [Dashboards and visualizations](/explore-analyze/dashboards.md). - -**Where is Findings data saved?** You can access findings data using the following index patterns: - -* **Current findings:** `logs-cloud_security_posture.findings_latest-*` -* **Historical findings:** `logs-cloud_security_posture.findings-*` - - -## Benchmark rules [_benchmark_rules_2] - -**How often are my resources evaluated against benchmark rules?** Resources are fetched and evaluated against benchmark rules when a security posture management integration is deployed. After that, the CSPM integration evaluates every 24 hours, and the KSPM integration evaluates every four hours. - -**Can I configure an integration’s fetch cycle?** No, the four-hour fetch cycle is not configurable. - -**Can I contribute to the CSP ruleset?** You can’t directly edit benchmark rules. The rules are defined [in this repository](https://github.com/elastic/csp-security-policies), where you can raise issues with certain rules. They are written in [Rego](https://www.openpolicyagent.org/docs/latest/policy-language/). - -**How can I tell which specific version of the CIS benchmarks is in use?** Refer to the `rule.benchmark.name` and `rule.benchmark.version` fields for documents in these datastreams: - -* `logs-cloud_security_posture.findings-default` -* `logs-cloud_security_posture.findings_latest-default` diff --git a/solutions/security/cloud/get-started-with-cnvm.md b/solutions/security/cloud/get-started-with-cnvm.md index ced3448ce..923e32f10 100644 --- a/solutions/security/cloud/get-started-with-cnvm.md +++ b/solutions/security/cloud/get-started-with-cnvm.md @@ -6,12 +6,6 @@ mapped_urls: # Get started with CNVM -% What needs to be done: Align serverless/stateful - -% Use migrated content from existing pages that map to this page: - -% - [x] ./raw-migrated-files/security-docs/security/vuln-management-get-started.md -% - [ ] ./raw-migrated-files/docs-content/serverless/security-vuln-management-get-started.md This page explains how to set up Cloud Native Vulnerability Management (CNVM). @@ -38,7 +32,7 @@ CNVM currently only supports AWS EC2 Linux workloads. ## Set up CNVM for AWS [vuln-management-setup] -To set up the CNVM integration for AWS, install the integration on a new {{agent}} policy, sign into the AWS account you want to scan, and run the [CloudFormation](https://docs.aws.amazon.com/cloudformation/index.md) template. +To set up the CNVM integration for AWS, install the integration on a new {{agent}} policy, sign into the AWS account you want to scan, and run the [CloudFormation](https://docs.aws.amazon.com/cloudformation/index.html) template. ::::{important} Do not add the integration to an existing {{agent}} policy. It should always be added to a new policy since it should not run on VMs with existing workloads. For more information, refer to [How CNVM works](/solutions/security/cloud/cloud-native-vulnerability-management.md#vuln-management-overview-how-it-works). @@ -53,9 +47,9 @@ Do not add the integration to an existing {{agent}} policy. It should always be 3. Click **Add Cloud Native Vulnerability Management**. 4. Give your integration a name that matches its purpose or the AWS account region you want to scan for vulnerabilities (for example, `uswest2-aws-account`.) - :::{image} ../../../images/security-cnvm-setup-1.png - :alt: The CNVM integration setup page - ::: + :::{image} ../../../images/security-cnvm-setup-1.png + :alt: The CNVM integration setup page + ::: 5. Click **Save and continue**. The integration will create a new {{agent}} policy. 6. Click **Add {{agent}} to your hosts**. @@ -77,9 +71,9 @@ The integration will only scan VMs in the region you select. To scan multiple re 1. Switch back to the tab where you have {{kib}} open. 2. Click **Launch CloudFormation**. The CloudFormation page appears. - :::{image} ../../../images/security-cnvm-cloudformation.png - :alt: The cloud formation template - ::: + :::{image} ../../../images/security-cnvm-cloudformation.png + :alt: The cloud formation template + ::: 3. Click **Create stack**. To avoid authentication problems, you can only make configuration changes to the VM InstanceType, which you could make larger to increase scanning speed. 4. Wait for the confirmation that {{agent}} was enrolled. diff --git a/solutions/security/cloud/get-started-with-cspm-for-aws.md b/solutions/security/cloud/get-started-with-cspm-for-aws.md index e128f9abe..3a0f30fdf 100644 --- a/solutions/security/cloud/get-started-with-cspm-for-aws.md +++ b/solutions/security/cloud/get-started-with-cspm-for-aws.md @@ -6,34 +6,6 @@ mapped_urls: # Get started with CSPM for AWS -% What needs to be done: Align serverless/stateful - -% Use migrated content from existing pages that map to this page: - -% - [x] ./raw-migrated-files/security-docs/security/cspm-get-started.md -% - [ ] ./raw-migrated-files/docs-content/serverless/security-cspm-get-started.md - -% Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc): - -$$$cspm-aws-agent-based$$$ - -$$$cspm-aws-agentless$$$ - -$$$cspm-finish-manual$$$ - -$$$cspm-setup-organization-manual$$$ - -$$$cspm-use-a-shared-credentials-file$$$ - -$$$cspm-use-iam-arn$$$ - -$$$cspm-use-instance-role$$$ - -$$$cspm-use-keys-directly$$$ - -$$$cspm-use-temp-credentials$$$ - - ## Overview [cspm-overview] This page explains how to get started monitoring the security posture of your cloud assets using the Cloud Security Posture Management (CSPM) feature. @@ -71,14 +43,17 @@ This functionality is in beta and is subject to change. The design and code is l 1. Option 1: Direct access keys/CloudFormation (Recommended). Under **Preferred method**, select **Direct access keys**. Expand the **Steps to Generate AWS Account Credentials** section, then follow the displayed instructions to automatically create the necessary credentials using CloudFormation. - ::::{note} - If you don’t want to monitor every account in your organization, specify which to monitor using the `OrganizationalUnitIDs` field that appears after you click **Launch CloudFormation**. - :::: + ::::{note} + If you don’t want to monitor every account in your organization, specify which to monitor using the `OrganizationalUnitIDs` field that appears after you click **Launch CloudFormation**. + :::: 2. Option 2: Temporary keys. To authenticate using temporary keys, refer to the instructions for [temporary keys](/solutions/security/cloud/get-started-with-cspm-for-aws.md#cspm-use-temp-credentials). 8. Once you’ve selected an authentication method and provided all necessary credentials, click **Save and continue** to finish deployment. Your data should start to appear within a few minutes. +::::{admonition} Important +Agentless deployment does not work if you are using [Traffic filtering](/deploy-manage/security/traffic-filtering.md). +:::: ## Agent-based deployment [cspm-aws-agent-based] @@ -94,7 +69,7 @@ This functionality is in beta and is subject to change. The design and code is l ### Set up cloud account access [cspm-set-up-cloud-access-section] -The CSPM integration requires access to AWS’s built-in [`SecurityAudit` IAM policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_job-functions.md#jf_security-auditor) in order to discover and evaluate resources in your cloud account. There are several ways to provide access. +The CSPM integration requires access to AWS’s built-in [`SecurityAudit` IAM policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_job-functions.html#jf_security-auditor) in order to discover and evaluate resources in your cloud account. There are several ways to provide access. For most use cases, the simplest option is to use AWS CloudFormation to automatically provision the necessary resources and permissions in your AWS account. This method, as well as several manual options, are described below. @@ -110,9 +85,9 @@ For most use cases, the simplest option is to use AWS CloudFormation to automati 7. (Optional) Switch to the AWS region where you want to deploy using the controls in the upper right corner. 8. Tick the checkbox under **Capabilities** to authorize the creation of necessary resources. - :::{image} ../../../images/security-cspm-cloudformation-template.png - :alt: The Add permissions screen in AWS - ::: + :::{image} ../../../images/security-cspm-cloudformation-template.png + :alt: The Add permissions screen in AWS + ::: 9. At the bottom of the template, select **Create stack**. @@ -242,7 +217,7 @@ When deploying to an organization using any of the authentication methods below, * [IAM role Amazon Resource Name (ARN)](/solutions/security/cloud/get-started-with-cspm-for-aws.md#cspm-use-iam-arn) ::::{important} -Whichever method you use to authenticate, make sure AWS’s built-in [`SecurityAudit` IAM policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_job-functions.md#jf_security-auditor) is attached. +Whichever method you use to authenticate, make sure AWS’s built-in [`SecurityAudit` IAM policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_job-functions.html#jf_security-auditor) is attached. :::: @@ -254,7 +229,7 @@ If you are deploying to an AWS organization instead of an AWS account, you shoul :::: -Follow AWS’s [IAM roles for Amazon EC2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.md) documentation to create an IAM role using the IAM console, which automatically generates an instance profile. +Follow AWS’s [IAM roles for Amazon EC2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html) documentation to create an IAM role using the IAM console, which automatically generates an instance profile. 1. Create an IAM role: @@ -262,15 +237,15 @@ Follow AWS’s [IAM roles for Amazon EC2](https://docs.aws.amazon.com/AWSEC2/lat 2. On the **Select trusted entity** page, under **Trusted entity type**, select **AWS service**. 3. Under **Use case**, select **EC2**. Click **Next**. - :::{image} ../../../images/security-cspm-aws-auth-1.png - :alt: The Select trusted entity screen in AWS - ::: + :::{image} ../../../images/security-cspm-aws-auth-1.png + :alt: The Select trusted entity screen in AWS + ::: 4. On the **Add permissions** page, search for and select `SecurityAudit`. Click **Next**. - :::{image} ../../../images/security-cspm-aws-auth-2.png - :alt: The Add permissions screen in AWS - ::: + :::{image} ../../../images/security-cspm-aws-auth-2.png + :alt: The Add permissions screen in AWS + ::: 5. On the **Name, review, and create** page, name your role, then click **Create role**. @@ -279,9 +254,9 @@ Follow AWS’s [IAM roles for Amazon EC2](https://docs.aws.amazon.com/AWSEC2/lat 1. In AWS, select an EC2 instance. 2. Select **Actions > Security > Modify IAM role**. - :::{image} ../../../images/security-cspm-aws-auth-3.png - :alt: The EC2 page in AWS - ::: + :::{image} ../../../images/security-cspm-aws-auth-3.png + :alt: The EC2 page in AWS + ::: 3. On the **Modify IAM role** page, search for and select your new IAM role. 4. Click **Update IAM role**. @@ -298,7 +273,7 @@ Make sure to deploy the CSPM integration to this EC2 instance. When completing s Access keys are long-term credentials for an IAM user or AWS account root user. To use access keys as credentials, you must provide the `Access key ID` and the `Secret Access Key`. After you provide credentials, [finish manual setup](/solutions/security/cloud/get-started-with-cspm-for-aws.md#cspm-finish-manual). -For more details, refer to [Access Keys and Secret Access Keys](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.md). +For more details, refer to [Access Keys and Secret Access Keys](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html). ::::{important} You must select **Programmatic access** when creating the IAM user. @@ -313,7 +288,7 @@ You can configure temporary security credentials in AWS to last for a specified Because temporary security credentials are short term, once they expire, you will need to generate new ones and manually update the integration’s configuration to continue collecting cloud posture data. Update the credentials before they expire to avoid data loss. ::::{note} -IAM users with multi-factor authentication (MFA) enabled need to submit an MFA code when calling `GetSessionToken`. For more details, refer to AWS’s [Temporary Security Credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.md) documentation. +IAM users with multi-factor authentication (MFA) enabled need to submit an MFA code when calling `GetSessionToken`. For more details, refer to AWS’s [Temporary Security Credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html) documentation. :::: @@ -334,7 +309,7 @@ After you provide credentials, [finish manual setup](/solutions/security/cloud/g #### Option 4 - Shared credentials file [cspm-use-a-shared-credentials-file] -If you use different AWS credentials for different tools or applications, you can use profiles to define multiple access keys in the same configuration file. For more details, refer to AWS' [Shared Credentials Files](https://docs.aws.amazon.com/sdkref/latest/guide/file-format.md) documentation. +If you use different AWS credentials for different tools or applications, you can use profiles to define multiple access keys in the same configuration file. For more details, refer to AWS' [Shared Credentials Files](https://docs.aws.amazon.com/sdkref/latest/guide/file-format.html) documentation. Instead of providing the `Access key ID` and `Secret Access Key` to the integration, provide the information required to locate the access keys within the shared credentials file: diff --git a/solutions/security/cloud/get-started-with-cspm-for-azure.md b/solutions/security/cloud/get-started-with-cspm-for-azure.md index f626fe941..a90605c81 100644 --- a/solutions/security/cloud/get-started-with-cspm-for-azure.md +++ b/solutions/security/cloud/get-started-with-cspm-for-azure.md @@ -6,22 +6,6 @@ mapped_urls: # Get started with CSPM for Azure -% What needs to be done: Align serverless/stateful - -% Use migrated content from existing pages that map to this page: - -% - [x] ./raw-migrated-files/security-docs/security/cspm-get-started-azure.md -% - [ ] ./raw-migrated-files/docs-content/serverless/security-cspm-get-started-azure.md - -% Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc): - -$$$cspm-azure-agent-based$$$ - -$$$cspm-azure-agentless$$$ - -$$$cspm-azure-client-secret$$$ - - ## Overview [cspm-overview-azure] This page explains how to get started monitoring the security posture of your cloud assets using the Cloud Security Posture Management (CSPM) feature. @@ -58,6 +42,9 @@ This functionality is in beta and is subject to change. The design and code is l 7. Next, you’ll need to authenticate to Azure by providing a **Client ID**, **Tenant ID**, and **Client Secret**. To learn how to generate them, refer to [Service principal with client secret](/solutions/security/cloud/get-started-with-cspm-for-azure.md#cspm-azure-client-secret). 8. Once you’ve provided the necessary credentials, click **Save and continue** to finish deployment. Your data should start to appear within a few minutes. +::::{admonition} Important +Agentless deployment does not work if you are using [Traffic filtering](/deploy-manage/security/traffic-filtering.md). +:::: ## Agent-based deployment [cspm-azure-agent-based] @@ -67,7 +54,7 @@ This functionality is in beta and is subject to change. The design and code is l 1. Find **Integrations** in the navigation menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). 2. Search for `CSPM`, then click on the result. 3. Click **Add Cloud Security Posture Management (CSPM)**. -4. Under **Configure integration**, select **Azure***, then select either ***Azure Organization** or **Single Subscription**, depending on which resources you want to monitor. +4. Under **Configure integration**, select **Azure**, then select either **Azure Organization** or **Single Subscription**, depending on which resources you want to monitor. 5. Give your integration a name that matches the purpose or team of the Azure resources you want to monitor, for example, `azure-CSPM-dev-1`. diff --git a/solutions/security/cloud/get-started-with-cspm-for-gcp.md b/solutions/security/cloud/get-started-with-cspm-for-gcp.md index 18435e6b2..62e03b2b4 100644 --- a/solutions/security/cloud/get-started-with-cspm-for-gcp.md +++ b/solutions/security/cloud/get-started-with-cspm-for-gcp.md @@ -56,6 +56,9 @@ This functionality is in beta and is subject to change. The design and code is l 7. Next, you’ll need to authenticate to GCP. Expand the **Steps to Generate GCP Account Credentials** section, then follow the instructions that appear to automatically create the necessary credentials using Google Cloud Shell. 8. Once you’ve provided the necessary credentials, click **Save and continue** to finish deployment. Your data should start to appear within a few minutes. +::::{admonition} Important +Agentless deployment does not work if you are using [Traffic filtering](/deploy-manage/security/traffic-filtering.md). +:::: ## Agent-based deployment [cspm-gcp-agent-based] @@ -92,9 +95,9 @@ For most users, the simplest option is to use a Google Cloud Shell script to aut 4. Copy the command that appears, then click **Launch Google Cloud Shell**. It opens in a new window. 5. Check the box to trust Elastic’s `cloudbeat` repo, then click **Confirm** - :::{image} ../../../images/security-cspm-cloudshell-trust.png - :alt: The cloud shell confirmation popup - ::: +:::{image} ../../../images/security-cspm-cloudshell-trust.png +:alt: The cloud shell confirmation popup +::: 6. In Google Cloud Shell, execute the command you copied. Once it finishes, return to {{kib}} and wait for the confirmation of data received from your new integration. Then you can click **View Assets** to see your data. diff --git a/solutions/security/cloud/get-started-with-cwp-for-kubernetes.md b/solutions/security/cloud/get-started-with-cwp-for-kubernetes.md deleted file mode 100644 index fff175520..000000000 --- a/solutions/security/cloud/get-started-with-cwp-for-kubernetes.md +++ /dev/null @@ -1,91 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/security/current/d4c-get-started.html ---- - -# Get started with CWP for Kubernetes [d4c-get-started] - -::::{warning} -This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features. -:::: - - -This page describes how to set up Cloud Workload Protection (CWP) for Kubernetes. - -::::{admonition} Requirements -* Kubernetes node operating systems must have Linux kernels 5.10.16 or higher. -* {{stack}} version 8.8 or higher. - -:::: - - - -## Initial setup [_initial_setup] - -First, you’ll need to deploy Elastic’s Defend for Containers integration to the Kubernetes clusters you wish to monitor. - -1. Find **Container Workload Security** in the navigation menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). Click **Add D4C Integration**. -2. Name the integration. The default name, which you can change, is `cloud_defend-1`. -3. Optional — make any desired changes to the integration’s policy by adjusting the **Selectors** and **Responses** sections. (For more information, refer to the [Defend for Containers policy guide](container-workload-protection-policies.md)). You can also change these later. -4. Under **Where to add this integration**, select an existing or new agent policy. -5. Click **Save & Continue**, then **Add {{agent}} to your hosts**. -6. On the {{agent}} policy page, click **Add agent** to open the Add agent flyout. -7. In the flyout, go to step 3 (**Install {{agent}} on your host**) and select the **Kubernetes** tab. -8. Download or copy the manifest (`elastic-agent-managed-kubernetes.yml`). -9. Open the manifest using your favorite editor, and uncomment the `#capabilities` section: - - ```console - #capabilities: - # add: - # - BPF # (since Linux 5.8) allows loading of BPF programs, create most map types, load BTF, iterate programs and maps. - # - PERFMON # (since Linux 5.8) allows attaching of BPF programs used for performance metrics and observability operations. - # - SYS_RESOURCE # Allow use of special resources or raising of resource limits. Used by 'Defend for Containers' to modify 'rlimit_memlock' - ``` - -10. From the directory where you saved the manifest, run the command `kubectl apply -f elastic-agent-managed-kubernetes.yml`. -11. Wait for the **Confirm agent enrollment** dialogue to show that data has started flowing from your newly-installed agent, then click **Close**. - - -## Get started with threat detection [d4c-get-started-threat] - -One of the [default D4C policies](container-workload-protection-policies.md#d4c-default-policies) sends process telemetry events (`fork` and `exec`) to {{es}}. - -In order to detect threats using this data, you’ll need active [detection rules](../detect-and-alert.md). Elastic has prebuilt detection rules designed for this data. (You can also create your own [custom rules](../detect-and-alert/create-detection-rule.md).) - -To install and enable the prebuilt rules: - -1. Find **Detection rules (SIEM)** in the navigation menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). Click **Add Elastic rules**. -2. Click the **Tags** filter next to the search bar, and search for the `Data Source: Elastic Defend for Containers` tag. -3. Select all the displayed rules, then click **Install *x* selected rule(s)**. -4. Return to the **Rules** page. Click the **Tags** filter next to the search bar, and search for the `Data Source: Elastic Defend for Containers` tag. -5. Select all the rules with the tag, and then click **Bulk actions > Enable**. - - -## Get started with drift detection and prevention [d4c-get-started-drift] - -{{elastic-sec}} defines container drift as the creation or modification of an executable within a container. Blocking drift restricts the number of attack vectors available to bad actors by prohibiting them from using external tools. - -To enable drift detection, you can use the default D4C policy: - -1. Make sure the [default D4C policy](container-workload-protection-policies.md#d4c-default-policies) is active. -2. Make sure you enabled at least the "Container Workload Protection" rule, by following the steps to install prebuilt rules, above. - -To enable drift prevention, create a new policy: - -1. Find **Container Workload Security** in the navigation menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md), then select your integration. -2. Under **Selectors**, click **Add selector > File Selector**. By default, it selects the operations `createExecutable` and `modifyExecutable`. -3. Name the selector, for example: `blockDrift`. -4. Scroll down to the **Responses** section and click **Add response > File Response**. -5. Under **Match selectors**, add the name of your new selector, for example: `blockDrift`. -6. Select the **Alert** and **Block** actions. -7. Click **Save integration**. - -::::{important} -Before you enable blocking, we strongly recommend you observe a production workload that’s using the default D4C policy to ensure that the workload does not create or modify executables as part of its normal operation. -:::: - - - -## Policy validation [d4c-get-started-validation] - -To ensure the stability of your production workloads, you should test policy changes before implementing them in production workloads. We also recommend you test policy changes on a simulated environment with workloads similar to production. This approach allows you to test that policy changes prevent undesirable behavior without disrupting your production workloads. diff --git a/solutions/security/cloud/get-started-with-kspm.md b/solutions/security/cloud/get-started-with-kspm.md index be0acbc3b..10e8471a2 100644 --- a/solutions/security/cloud/get-started-with-kspm.md +++ b/solutions/security/cloud/get-started-with-kspm.md @@ -6,39 +6,6 @@ mapped_urls: # Get started with KSPM -% What needs to be done: Align serverless/stateful - -% Use migrated content from existing pages that map to this page: - -% - [x] ./raw-migrated-files/security-docs/security/get-started-with-kspm.md -% - [ ] ./raw-migrated-files/docs-content/serverless/security-get-started-with-kspm.md - -% Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc): - -$$$kspm-setup-eks-start$$$ - -$$$kspm-setup-eks-auth$$$ - -$$$kspm-setup-eks-finish$$$ - -$$$kspm-setup-eks-modify-deploy$$$ - -$$$kspm-setup-unmanaged$$$ - -$$$kspm-setup-unmanaged-modify-deploy$$$ - -$$$kspm-use-irsa$$$ - -$$$kspm-use-instance-role$$$ - -$$$kspm-use-keys-directly$$$ - -$$$kspm-use-temp-credentials$$$ - -$$$kspm-use-a-shared-credentials-file$$$ - -$$$kspm-use-iam-arn$$$ - This page explains how to configure the Kubernetes Security Posture Management (KSPM) integration. ::::{admonition} Requirements @@ -161,7 +128,7 @@ If you are using the AWS visual editor to create and modify your IAM Policies, y #### Option 1 - [Recommended] Use Kubernetes Service Account to assume IAM role [kspm-use-irsa] -Follow AWS’s [EKS Best Practices](https://aws.github.io/aws-eks-best-practices/security/docs/iam/#iam-roles-for-service-accounts-irsa) documentation to use the [IAM Role to Kubernetes Service-Account](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.md) (IRSA) feature to get temporary credentials and scoped permissions. +Follow AWS’s [EKS Best Practices](https://aws.github.io/aws-eks-best-practices/security/docs/iam/#iam-roles-for-service-accounts-irsa) documentation to use the [IAM Role to Kubernetes Service-Account](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) (IRSA) feature to get temporary credentials and scoped permissions. ::::{important} During setup, do not fill in any option in the "Setup Access" section. Click **Save and continue**. @@ -171,7 +138,7 @@ During setup, do not fill in any option in the "Setup Access" section. Click **S #### Option 2 - Use default instance role [kspm-use-instance-role] -Follow AWS’s [IAM roles for Amazon EC2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.md) documentation to create an IAM role using the IAM console, which automatically generates an instance profile. +Follow AWS’s [IAM roles for Amazon EC2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html) documentation to create an IAM role using the IAM console, which automatically generates an instance profile. ::::{important} During setup, do not fill in any option in the "Setup Access" section. Click **Save and continue**. @@ -183,7 +150,7 @@ During setup, do not fill in any option in the "Setup Access" section. Click **S Access keys are long-term credentials for an IAM user or AWS account root user. To use access keys as credentials, you must provide the `Access key ID` and the `Secret Access Key`. -For more details, refer to AWS' [Access Keys and Secret Access Keys](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.md) documentation. +For more details, refer to AWS' [Access Keys and Secret Access Keys](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html) documentation. ::::{important} You must select "Programmatic access" when creating the IAM user. @@ -198,7 +165,7 @@ You can configure temporary security credentials in AWS to last for a specified Because temporary security credentials are short term, once they expire, you will need to generate new ones and manually update the integration’s configuration to continue collecting cloud posture data. Update the credentials before they expire to avoid data loss. ::::{note} -IAM users with multi-factor authentication (MFA) enabled need to submit an MFA code when calling `GetSessionToken`. For more details, refer to AWS' [Temporary Security Credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.md) documentation. +IAM users with multi-factor authentication (MFA) enabled need to submit an MFA code when calling `GetSessionToken`. For more details, refer to AWS' [Temporary Security Credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html) documentation. :::: @@ -217,7 +184,7 @@ The output from this command includes the following fields, which you should pro #### Option 5 - Use a shared credentials file [kspm-use-a-shared-credentials-file] -If you use different AWS credentials for different tools or applications, you can use profiles to define multiple access keys in the same configuration file. For more details, refer to AWS' [Shared Credentials Files](https://docs.aws.amazon.com/sdkref/latest/guide/file-format.md) documentation. +If you use different AWS credentials for different tools or applications, you can use profiles to define multiple access keys in the same configuration file. For more details, refer to AWS' [Shared Credentials Files](https://docs.aws.amazon.com/sdkref/latest/guide/file-format.html) documentation. Instead of providing the `Access key ID` and `Secret Access Key` to the integration, provide the information required to locate the access keys within the shared credentials file: @@ -236,7 +203,7 @@ If you don’t provide values for all configuration fields, the integration will An IAM role Amazon Resource Name (ARN) is an IAM identity that you can create in your AWS account. You define the role’s permissions. Roles do not have standard long-term credentials such as passwords or access keys. Instead, when you assume a role, it provides temporary security credentials for your session. An IAM role’s ARN can be used to specify which AWS IAM role to use to generate temporary credentials. -For more details, refer to AWS' [AssumeRole API](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.md) documentation. Follow AWS' instructions to [create an IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.md), and define the IAM role’s permissions using the JSON permissions policy above. +For more details, refer to AWS' [AssumeRole API](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html) documentation. Follow AWS' instructions to [create an IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html), and define the IAM role’s permissions using the JSON permissions policy above. To use an IAM role’s ARN, you need to provide either a [credential profile](/solutions/security/cloud/get-started-with-kspm.md#kspm-use-a-shared-credentials-file) or [access keys](/solutions/security/cloud/get-started-with-kspm.md#kspm-use-keys-directly) along with the `ARN role`. The `ARN Role` value specifies which AWS IAM role to use for generating temporary credentials. diff --git a/solutions/security/cloud/ingest-aws-security-hub-data.md b/solutions/security/cloud/ingest-aws-security-hub-data.md index 920f72544..55e28383d 100644 --- a/solutions/security/cloud/ingest-aws-security-hub-data.md +++ b/solutions/security/cloud/ingest-aws-security-hub-data.md @@ -6,13 +6,6 @@ mapped_urls: # Ingest AWS Security Hub data -% What needs to be done: Lift-and-shift - -% Use migrated content from existing pages that map to this page: - -% - [x] ./raw-migrated-files/security-docs/security/ingest-aws-securityhub-data.md -% - [ ] ./raw-migrated-files/docs-content/serverless/ingest-aws-securityhub-data.md - In order to enrich your {{elastic-sec}} workflows with third-party cloud security posture data collected by AWS Security Hub: * Follow the steps to [set up the AWS Security Hub integration](https://docs.elastic.co/en/integrations/aws/securityhub). diff --git a/solutions/security/cloud/ingest-cncf-falco-data.md b/solutions/security/cloud/ingest-cncf-falco-data.md index 396cd27ed..f5c3577af 100644 --- a/solutions/security/cloud/ingest-cncf-falco-data.md +++ b/solutions/security/cloud/ingest-cncf-falco-data.md @@ -6,19 +6,6 @@ mapped_urls: # Ingest CNCF Falco data -% What needs to be done: Lift-and-shift - -% Use migrated content from existing pages that map to this page: - -% - [x] ./raw-migrated-files/security-docs/security/ingest-falco.md -% - [ ] ./raw-migrated-files/docs-content/serverless/ingest-falco.md - -% Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc): - -$$$ingest-falco-setup-falco-kubernetes$$$ - -$$$ingest-falco-setup-falco-vm$$$ - CNCF Falco is an open-source runtime security tool that detects anomalous activity in Linux hosts, containers, Kubernetes, and cloud environments. You can ingest Falco alerts into {{es}} to view them on {{elastic-sec}}'s Alerts page and incorporate them into your security workflows by using Falcosidekick, a proxy forwarder which can send alerts from your Falco deployments to {{es}}. First, you’ll need to configure {{elastic-sec}} to receive data from Falco, then you’ll need to configure Falco and Falcosidekick to send data to {{es}}. @@ -79,9 +66,9 @@ Multiple methods for configuring Falco to send data from VMs to {{es}} are avail 1. `ELASTICSEARCH_HOSTPORT`: Your {{es}} endpoint URL, which can be found under **Connection details** on the upper right of the **Integrations** page in {{kib}}. 2. `ELASTICSEARCH_INDEX`: The {{es}} index where you want to store Falco logs. - ::::{important} - Your `ELASTICSEARCH_INDEX` value must match `logs-falco.alerts-*`. - :::: + ::::{important} + Your `ELASTICSEARCH_INDEX` value must match `logs-falco.alerts-*`. + :::: 3. `ELASTICSEARCH_SUFFIX`: The frequency with which you want the {{es}} index suffix to change. Either `daily`, `monthly`, `annually`, or `none`. 4. `ELASTICSEARCH_APIKEY`: The recommended way to authenticate to {{es}}, by providing an [API key](/deploy-manage/api-keys/elasticsearch-api-keys.md). Note that support for this environment variable starts with Falcosidekick version 2.30. You can access the latest version on Falcosidekick’s [Docker Hub](https://hub.docker.com/r/falcosecurity/falcosidekick). @@ -113,7 +100,7 @@ After installing and configuring Falcosidekick, restart Falco with `sudo systemc 1. Add the Falco [Helm charts](https://github.com/falcosecurity/charts/blob/master/README.md): - ``` + ```bash helm repo add falcosecurity https://falcosecurity.github.io/charts helm repo update ``` diff --git a/solutions/security/cloud/ingest-third-party-cloud-security-data.md b/solutions/security/cloud/ingest-third-party-cloud-security-data.md index 59f545777..ad66d2d69 100644 --- a/solutions/security/cloud/ingest-third-party-cloud-security-data.md +++ b/solutions/security/cloud/ingest-third-party-cloud-security-data.md @@ -6,13 +6,6 @@ mapped_urls: # Ingest third-party cloud security data -% What needs to be done: Lift-and-shift - -% Use migrated content from existing pages that map to this page: - -% - [x] ./raw-migrated-files/security-docs/security/ingest-third-party-cloud-security-data.md -% - [ ] ./raw-migrated-files/docs-content/serverless/ingest-third-party-cloud-security-data.md - This section describes how to ingest cloud security data from third-party tools into {{es}}. Once ingested, this data can provide additional context and enrich your {{elastic-sec}} workflows. You can ingest both third-party cloud workload protection data and third-party security posture and vulnerability data. @@ -27,7 +20,7 @@ You can ingest third-party cloud security alerts into {{elastic-sec}} to view th ## Ingest third-party security posture and vulnerability data [_ingest_third_party_security_posture_and_vulnerability_data] -You can ingest third-party data into {{elastic-sec}} to review and investigate it alongside data collected by {{elastic-sec}}'s native cloud security integrations. Once ingested, cloud security posture and vulnerability data appears on the [Findings](/solutions/security/cloud/findings-page.md) page, on the [Cloud Posture dashboard](/solutions/security/cloud/cloud-security-posture-dashboard.md), and in the entity details flyouts for [alerts](/solutions/security/detect-and-alert/view-detection-alert-details.md#insights-section), [users](/solutions/security/explore/users-page.md#user-details-flyout), and [hosts](/solutions/security/explore/hosts-page.md#host-details-flyout). +You can ingest third-party data into {{elastic-sec}} to review and investigate it alongside data collected by {{elastic-sec}}'s native cloud security integrations. Once ingested, cloud security posture and vulnerability data appears on the [Findings](/solutions/security/cloud/findings-page.md) page, on the [Cloud Posture dashboard](/solutions/security/dashboards/cloud-security-posture-dashboard.md), and in the entity details flyouts for [alerts](/solutions/security/detect-and-alert/view-detection-alert-details.md#insights-section), [users](/solutions/security/explore/users-page.md#user-details-flyout), and [hosts](/solutions/security/explore/hosts-page.md#host-details-flyout). * Learn to [ingest cloud security posture data from AWS Security Hub](/solutions/security/cloud/ingest-aws-security-hub-data.md). * Learn to [ingest cloud security posture and vulnerability data from Wiz](/solutions/security/cloud/ingest-wiz-data.md). diff --git a/solutions/security/cloud/ingest-wiz-data.md b/solutions/security/cloud/ingest-wiz-data.md index 749964ed2..835bdd185 100644 --- a/solutions/security/cloud/ingest-wiz-data.md +++ b/solutions/security/cloud/ingest-wiz-data.md @@ -6,13 +6,6 @@ mapped_urls: # Ingest Wiz data -% What needs to be done: Lift-and-shift - -% Use migrated content from existing pages that map to this page: - -% - [x] ./raw-migrated-files/security-docs/security/ingest-wiz-data.md -% - [ ] ./raw-migrated-files/docs-content/serverless/ingest-wiz-data.md - In order to enrich your {{elastic-sec}} workflows with third-party cloud security posture and vulnerability data collected by Wiz: * Follow the steps to [set up the Wiz integration](https://docs.elastic.co/en/integrations/wiz). diff --git a/solutions/security/cloud/kubernetes-dashboard.md b/solutions/security/cloud/kubernetes-dashboard.md deleted file mode 100644 index 38a0dc5da..000000000 --- a/solutions/security/cloud/kubernetes-dashboard.md +++ /dev/null @@ -1,71 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/security/current/cloud-nat-sec-kubernetes-dashboard.html ---- - -# Kubernetes dashboard [cloud-nat-sec-kubernetes-dashboard] - -The Kubernetes dashboard provides insight into Linux process data from your Kubernetes clusters. It shows sessions in detail and in the context of your monitored infrastructure. - -:::{image} ../../../images/security-kubernetes-dashboard.png -:alt: The Kubernetes dashboard -::: - -The numbered sections are described below: - -1. The charts at the top of the dashboard provide an overview of your monitored Kubernetes infrastructure. You can hide them by clicking **Hide charts**. -2. The tree navigation menu allows you to navigate through your deployments and select the scope of the sessions table to the right. You can select any item in the menu to show its sessions. In Logical view, the menu is organized by Cluster, Namespace, Pod, and Container image. In Infrastructure view, it is organized by Cluster, Node, Pod, and Container image. -3. The sessions table displays sessions collected from the selected element of your Kubernetes infrastructure. You can view it in fullscreen by selecting the button in the table’s upper right corner. You can sort the table by any of its fields. - -You can filter the data using the KQL search bar and date picker at the top of the page. - -From the sessions table’s Actions column, you can take the following investigative actions: - -* View details -* [Open in Timeline](../investigate/timeline.md) -* [Run Osquery](../investigate/run-osquery-from-alerts.md) -* [Analyze event](../investigate/visual-event-analyzer.md) -* [Open Session View](../investigate/session-view.md) - -Session View displays Kubernetes metadata under the **Metadata** tab of the Detail panel: - -:::{image} ../../../images/security-metadata-tab.png -:alt: The Detail panel's metadata tab -::: - -The **Metadata** tab is organized into these expandable sections: - -* **Metadata:** `hostname`, `id`, `ip`, `mac`, `name`, Host OS information -* **Cloud:** `instance.name`, `provider`, `region`, `account.id`, `project.id` -* **Container:** `id`, `name`, `image.name`, `image.tag`, `image.hash.all` -* **Orchestrator:** `resource.ip`, `resource.name`, `resource.type`, `namespace`, `cluster.id`, `cluster.name`, `parent.type` - - -## Setup [_setup] - -To get data for this dashboard, set up [Cloud Workload Protection for Kubernetes](get-started-with-cwp-for-kubernetes.md) for the clusters you want to display on the dashboard. - -::::{admonition} Requirements -* Kubernetes node operating systems must have Linux kernels 5.10.16 or higher. -* {{stack}} version 8.8 or higher. - -:::: - - -**Support matrix**: This feature is currently available on GKE and EKS using Linux hosts and Kubernetes versions that match the following specifications: - -| | | | -| --- | --- | --- | -| | EKS 1.24-1.26 (AL2022) | GKE 1.24-1.26 (COS) | -| Process event exports | ✓ | ✓ | -| Network event exports | ✓ | ✓ | -| File event exports | ✓ | ✓ | -| File blocking | ✓ | ✓ | -| Process blocking | ✓ | ✓ | -| Network blocking | ✗ | ✗ | -| Drift prevention | ✓ | ✓ | -| Mount point awareness | ✓ | ✓ | - -::::{important} -This dashboard uses data from the `logs-*` index pattern, which is included by default in the [`securitySolution:defaultIndex` advanced setting](../get-started/configure-advanced-settings.md). To collect data from multiple {{es}} clusters (as in a cross-cluster deployment), update `logs-*` to `*:logs-*`. -:::: diff --git a/solutions/security/cloud/kubernetes-security-posture-management.md b/solutions/security/cloud/kubernetes-security-posture-management.md index 27a213b68..3569b0b26 100644 --- a/solutions/security/cloud/kubernetes-security-posture-management.md +++ b/solutions/security/cloud/kubernetes-security-posture-management.md @@ -32,7 +32,7 @@ This integration supports Amazon EKS and unmanaged Kubernetes clusters. For setu 1. When you add a KSPM integration, it generates a Kubernetes manifest. When applied to a cluster, the manifest deploys an {{agent}} as a [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset) to ensure all nodes are evaluated. 2. Upon deployment, the integration immediately assesses the security posture of your Kubernetes resources. The evaluation process repeats every four hours. -3. After each evaluation, the integration sends findings to {{es}}. Findings appear on the [Cloud Security Posture dashboard](/solutions/security/cloud/cloud-security-posture-dashboard-2.md) and the [findings](/solutions/security/cloud/findings-page-2.md) page. +3. After each evaluation, the integration sends findings to {{es}}. Findings appear on the [Cloud Security Posture dashboard](/solutions/security/dashboards/cloud-security-posture-dashboard.md) and the [findings](/solutions/security/cloud/findings-page-2.md) page. ## Use cases [kspm-use-cases] @@ -48,14 +48,14 @@ The KSPM integration helps you to: To identify and remediate failed failed findings: -1. Go to the [Cloud Security Posture dashboard](/solutions/security/cloud/cloud-security-posture-dashboard-2.md). +1. Go to the [Cloud Security Posture dashboard](/solutions/security/dashboards/cloud-security-posture-dashboard.md). 2. Click **View all failed findings**, either for an individual cluster or for all monitored clusters. 3. Click a failed finding. The findings flyout opens. 4. Follow the steps under **Remediation** to correct the misconfiguration. - ::::{note} - Remediation steps typically include commands for you to execute. These sometimes contain placeholder values that you must replace before execution. - :::: + ::::{note} + Remediation steps typically include commands for you to execute. These sometimes contain placeholder values that you must replace before execution. + :::: @@ -72,7 +72,7 @@ To identify the Kubernetes resources generating the most failed findings: To identify risks in particular CIS sections: -1. Go to the [Cloud Security Posture dashboard](/solutions/security/cloud/cloud-security-posture-dashboard-2.md). +1. Go to the [Cloud Security Posture dashboard](/solutions/security/dashboards/cloud-security-posture-dashboard.md). 2. In the Failed findings by CIS section widget, click the name of a CIS section to view all failed findings for that section. Alternatively: diff --git a/solutions/security/cloud/security-posture-management-overview.md b/solutions/security/cloud/security-posture-management-overview.md index 0e3b77007..c674735bb 100644 --- a/solutions/security/cloud/security-posture-management-overview.md +++ b/solutions/security/cloud/security-posture-management-overview.md @@ -6,14 +6,6 @@ mapped_urls: # Security posture management overview -% What needs to be done: Lift-and-shift - -% Use migrated content from existing pages that map to this page: - -% - [x] ./raw-migrated-files/security-docs/security/security-posture-management.md -% - [ ] ./raw-migrated-files/docs-content/serverless/security-posture-management.md - - ## Overview [_overview] Elastic’s [Cloud Security Posture Management](/solutions/security/cloud/cloud-security-posture-management.md) (CSPM) and [Kubernetes Security Posture Management](/solutions/security/cloud/kubernetes-security-posture-management.md) (KSPM) features help you discover and evaluate the services and resources in your cloud environment — like storage, compute, IAM, and more — against security guidelines defined by the Center for Internet Security (CIS). They help you identify and remediate configuration risks that could undermine the confidentiality, integrity, and availability of your cloud assets, such as publicly exposed storage buckets or overly permissive networking objects. diff --git a/solutions/security/configure-elastic-defend/configure-offline-endpoints-air-gapped-environments.md b/solutions/security/configure-elastic-defend/configure-offline-endpoints-air-gapped-environments.md index a6ed532f8..e220ebd27 100644 --- a/solutions/security/configure-elastic-defend/configure-offline-endpoints-air-gapped-environments.md +++ b/solutions/security/configure-elastic-defend/configure-offline-endpoints-air-gapped-environments.md @@ -45,7 +45,7 @@ docker run -v "$PWD"/nginx.conf:/etc/nginx/conf.d/default.conf:ro -p 80:80 nginx ``` ::::{important} -This example script is not appropriate for production environments. We recommend configuring the Nginx server to use [TLS](http://nginx.org/en/docs/http/configuring_https_servers.md) according to your IT policies. Refer to [Nginx documentation](https://docs.nginx.com/nginx/admin-guide/installing-nginx/) for more information on downloading and configuring Nginx. +This example script is not appropriate for production environments. We recommend configuring the Nginx server to use [TLS](http://nginx.org/en/docs/http/configuring_https_servers.html) according to your IT policies. Refer to [Nginx documentation](https://docs.nginx.com/nginx/admin-guide/installing-nginx/) for more information on downloading and configuring Nginx. :::: @@ -71,7 +71,7 @@ docker run -p 80:80 -v "$PWD"/httpd.conf:/usr/local/apache2/conf/httpd.conf http ``` ::::{important} -This example script is not appropriate for production environments. We recommend configuring httpd to use [TLS](https://httpd.apache.org/docs/trunk/ssl/ssl_howto.md) according to your IT policies. Refer to [Apache documentation](https://httpd.apache.org) for more information on downloading and configuring Apache httpd. +This example script is not appropriate for production environments. We recommend configuring httpd to use [TLS](https://httpd.apache.org/docs/trunk/ssl/ssl_howto.html) according to your IT policies. Refer to [Apache documentation](https://httpd.apache.org) for more information on downloading and configuring Apache httpd. :::: @@ -133,7 +133,7 @@ docker run -v "$PWD"/nginx.conf:/etc/nginx/conf.d/default.conf:ro -v "$PWD"/stat ``` ::::{important} -This example script is not appropriate for production environments. We recommend configuring the Nginx server to use [TLS](http://nginx.org/en/docs/http/configuring_https_servers.md) according to your IT policies. Refer to [Nginx documentation](https://docs.nginx.com/nginx/admin-guide/installing-nginx/) for more information on downloading and configuring Nginx. +This example script is not appropriate for production environments. We recommend configuring the Nginx server to use [TLS](http://nginx.org/en/docs/http/configuring_https_servers.html) according to your IT policies. Refer to [Nginx documentation](https://docs.nginx.com/nginx/admin-guide/installing-nginx/) for more information on downloading and configuring Nginx. :::: @@ -153,7 +153,7 @@ docker run -p 80:80 -v "$PWD/static":/usr/local/apache2/htdocs/ -v "$PWD"/my-htt ``` ::::{important} -This example script is not appropriate for production environments. We recommend configuring httpd to use [TLS](https://httpd.apache.org/docs/trunk/ssl/ssl_howto.md) according to your IT policies. Refer to [Apache documentation](https://httpd.apache.org) for more information on downloading and configuring Apache httpd. +This example script is not appropriate for production environments. We recommend configuring httpd to use [TLS](https://httpd.apache.org/docs/trunk/ssl/ssl_howto.html) according to your IT policies. Refer to [Apache documentation](https://httpd.apache.org) for more information on downloading and configuring Apache httpd. :::: diff --git a/solutions/security/dashboards/cloud-native-vulnerability-management-dashboard.md b/solutions/security/dashboards/cloud-native-vulnerability-management-dashboard.md index 7435501f9..e7ab530e5 100644 --- a/solutions/security/dashboards/cloud-native-vulnerability-management-dashboard.md +++ b/solutions/security/dashboards/cloud-native-vulnerability-management-dashboard.md @@ -2,17 +2,12 @@ mapped_urls: - https://www.elastic.co/guide/en/security/current/vuln-management-dashboard-dash.html - https://www.elastic.co/guide/en/serverless/current/security-vuln-management-dashboard-dash.html + - https://www.elastic.co/guide/en/serverless/current/_cloud_native_vulnerability_management_dashboard.html + - https://www.elastic.co/guide/en/security/current/vuln-management-dashboard.html --- # Cloud Native Vulnerability Management Dashboard -% What needs to be done: Align serverless/stateful - -% Use migrated content from existing pages that map to this page: - -% - [x] ./raw-migrated-files/security-docs/security/vuln-management-dashboard-dash.md -% - [ ] ./raw-migrated-files/docs-content/serverless/security-vuln-management-dashboard-dash.md - The Cloud Native Vulnerability Management (CNVM) dashboard gives you an overview of vulnerabilities detected in your cloud infrastructure. :::{image} ../../../images/security-vuln-management-dashboard.png diff --git a/solutions/security/dashboards/cloud-security-posture-dashboard.md b/solutions/security/dashboards/cloud-security-posture-dashboard.md index 7dcc81376..efddd0ad6 100644 --- a/solutions/security/dashboards/cloud-security-posture-dashboard.md +++ b/solutions/security/dashboards/cloud-security-posture-dashboard.md @@ -2,18 +2,15 @@ mapped_urls: - https://www.elastic.co/guide/en/security/current/cloud-posture-dashboard.html - https://www.elastic.co/guide/en/serverless/current/security-cloud-posture-dashboard-dash.html + - https://www.elastic.co/guide/en/serverless/current/security-cloud-posture-dashboard-dash-cspm.html + - https://www.elastic.co/guide/en/serverless/current/security-cloud-posture-dashboard-dash-kspm.html + - https://www.elastic.co/guide/en/security/current/cspm-posture-dashboard.html + - https://www.elastic.co/guide/en/security/current/cloud-nat-sec-posture-dashboard.html --- # Cloud Security Posture dashboard -% What needs to be done: Align serverless/stateful - -% Use migrated content from existing pages that map to this page: - -% - [x] ./raw-migrated-files/security-docs/security/cloud-posture-dashboard.md -% - [ ] ./raw-migrated-files/docs-content/serverless/security-cloud-posture-dashboard-dash.md - -The Cloud Security Posture dashboard summarizes your cloud infrastructure’s overall performance against [security guidelines](/solutions/security/cloud/benchmarks-2.md) defined by the Center for Internet Security (CIS). To start collecting this data, refer to [Get started with Cloud Security Posture Management](/solutions/security/cloud/get-started-with-cspm-for-aws.md) or [Get started with Kubernetes Security Posture Management](/solutions/security/cloud/get-started-with-kspm.md). +The Cloud Security Posture dashboard summarizes your cloud infrastructure’s overall performance against [security guidelines](/solutions/security/cloud/benchmarks.md) defined by the Center for Internet Security (CIS). To start collecting this data, refer to [Get started with Cloud Security Posture Management](/solutions/security/cloud/get-started-with-cspm-for-aws.md) or [Get started with Kubernetes Security Posture Management](/solutions/security/cloud/get-started-with-kspm.md). :::{image} ../../../images/security-cloud-sec-dashboard.png :alt: The cloud Security dashboard diff --git a/solutions/security/dashboards/kubernetes-dashboard.md b/solutions/security/dashboards/kubernetes-dashboard.md deleted file mode 100644 index c20046c8f..000000000 --- a/solutions/security/dashboards/kubernetes-dashboard.md +++ /dev/null @@ -1,71 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/security/current/kubernetes-dashboard.html ---- - -# Kubernetes dashboard [kubernetes-dashboard] - -The Kubernetes dashboard provides insight into Linux process data from your Kubernetes clusters. It shows sessions in detail and in the context of your monitored infrastructure. - -:::{image} ../../../images/security-kubernetes-dashboard.png -:alt: The Kubernetes dashboard -::: - -The numbered sections are described below: - -1. The charts at the top of the dashboard provide an overview of your monitored Kubernetes infrastructure. You can hide them by clicking **Hide charts**. -2. The tree navigation menu allows you to navigate through your deployments and select the scope of the sessions table to the right. You can select any item in the menu to show its sessions. In Logical view, the menu is organized by Cluster, Namespace, Pod, and Container image. In Infrastructure view, it is organized by Cluster, Node, Pod, and Container image. -3. The sessions table displays sessions collected from the selected element of your Kubernetes infrastructure. You can view it in fullscreen by selecting the button in the table’s upper right corner. You can sort the table by any of its fields. - -You can filter the data using the KQL search bar and date picker at the top of the page. - -From the sessions table’s Actions column, you can take the following investigative actions: - -* View details -* [Open in Timeline](../investigate/timeline.md) -* [Run Osquery](../investigate/run-osquery-from-alerts.md) -* [Analyze event](../investigate/visual-event-analyzer.md) -* [Open Session View](../investigate/session-view.md) - -Session View displays Kubernetes metadata under the **Metadata** tab of the Detail panel: - -:::{image} ../../../images/security-metadata-tab.png -:alt: The Detail panel's metadata tab -::: - -The **Metadata** tab is organized into these expandable sections: - -* **Metadata:** `hostname`, `id`, `ip`, `mac`, `name`, Host OS information -* **Cloud:** `instance.name`, `provider`, `region`, `account.id`, `project.id` -* **Container:** `id`, `name`, `image.name`, `image.tag`, `image.hash.all` -* **Orchestrator:** `resource.ip`, `resource.name`, `resource.type`, `namespace`, `cluster.id`, `cluster.name`, `parent.type` - - -## Setup [_setup_2] - -To get data for this dashboard, set up [Cloud Workload Protection for Kubernetes](../cloud/get-started-with-cwp-for-kubernetes.md) for the clusters you want to display on the dashboard. - -::::{admonition} Requirements -* Kubernetes node operating systems must have Linux kernels 5.10.16 or higher. -* {{stack}} version 8.8 or higher. - -:::: - - -**Support matrix**: This feature is currently available on GKE and EKS using Linux hosts and Kubernetes versions that match the following specifications: - -| | | | -| --- | --- | --- | -| | EKS 1.24-1.26 (AL2022) | GKE 1.24-1.26 (COS) | -| Process event exports | ✓ | ✓ | -| Network event exports | ✗ | ✗ | -| File event exports | ✓ | ✓ | -| File blocking | ✓ | ✓ | -| Process blocking | ✓ | ✓ | -| Network blocking | ✗ | ✗ | -| Drift prevention | ✓ | ✓ | -| Mount point awareness | ✓ | ✓ | - -::::{important} -This dashboard uses data from the `logs-*` index pattern, which is included by default in the [`securitySolution:defaultIndex` advanced setting](../get-started/configure-advanced-settings.md). To collect data from multiple {{es}} clusters (as in a cross-cluster deployment), update `logs-*` to `*:logs-*`. -:::: diff --git a/solutions/security/get-started/elastic-security-ui.md b/solutions/security/get-started/elastic-security-ui.md index 2fd241214..ba08496e2 100644 --- a/solutions/security/get-started/elastic-security-ui.md +++ b/solutions/security/get-started/elastic-security-ui.md @@ -109,7 +109,7 @@ Expand this section to access the following pages: * [Rules](/solutions/security/detect-and-alert/manage-detection-rules.md): Create and manage rules to monitor suspicious events. -* [Benchmarks](/solutions/security/cloud/benchmarks-2.md): View, set up, or configure cloud security benchmarks. +* [Benchmarks](/solutions/security/cloud/benchmarks.md): View, set up, or configure cloud security benchmarks. * [Shared Exception Lists](/solutions/security/detect-and-alert/rule-exceptions.md#shared-exception-list-intro): View and manage rule exceptions and shared exception lists. diff --git a/solutions/security/investigate/session-view.md b/solutions/security/investigate/session-view.md index 055bcf1d6..b58a33631 100644 --- a/solutions/security/investigate/session-view.md +++ b/solutions/security/investigate/session-view.md @@ -23,10 +23,6 @@ Session View has the following features: * **Alerts:** Process, file, and network alerts in the context of the events which caused them. * **Terminal output:** Terminal output associated with each process in the session. -::::{note} -To view Linux session data from your Kubernetes infrastructure, you’ll need to set up the [Kubernetes dashboard](/solutions/security/dashboards/kubernetes-dashboard.md). -:::: - ## Enable Session View data [enable-session-view] diff --git a/solutions/toc.yml b/solutions/toc.yml index 6c909cdc9..81372e147 100644 --- a/solutions/toc.yml +++ b/solutions/toc.yml @@ -580,26 +580,21 @@ toc: - file: security/cloud/cspm-privilege-requirements.md - file: security/cloud/findings-page.md - file: security/cloud/benchmarks.md - - file: security/cloud/cloud-security-posture-dashboard.md - - file: security/cloud/frequently-asked-questions-faq.md + - file: security/dashboards/cloud-security-posture-dashboard.md + - file: security/cloud/cspm-frequently-asked-questions-faq.md - file: security/cloud/kubernetes-security-posture-management.md children: - file: security/cloud/get-started-with-kspm.md - file: security/cloud/findings-page-2.md - - file: security/cloud/benchmarks-2.md - - file: security/cloud/cloud-security-posture-dashboard-2.md - - file: security/cloud/frequently-asked-questions-faq-2.md + - file: security/cloud/benchmarks.md + - file: security/dashboards/cloud-security-posture-dashboard.md + - file: security/cloud/cspm-frequently-asked-questions-faq.md - file: security/cloud/cloud-native-vulnerability-management.md children: - file: security/cloud/get-started-with-cnvm.md - file: security/cloud/findings-page-3.md - - file: security/cloud/cloud-native-vulnerability-management-dashboard.md - - file: security/cloud/frequently-asked-questions-faq-3.md - - file: security/cloud/cloud-workload-protection-for-kubernetes.md - children: - - file: security/cloud/get-started-with-cwp-for-kubernetes.md - - file: security/cloud/container-workload-protection-policies.md - - file: security/cloud/kubernetes-dashboard.md + - file: security/dashboards/cloud-native-vulnerability-management-dashboard.md + - file: security/cloud/cnvm-frequently-asked-questions-faq.md - file: security/cloud/cloud-workload-protection-for-vms.md children: - file: security/cloud/capture-environment-variables.md @@ -635,7 +630,6 @@ toc: children: - file: security/dashboards/overview-dashboard.md - file: security/dashboards/detection-response-dashboard.md - - file: security/dashboards/kubernetes-dashboard.md - file: security/dashboards/cloud-security-posture-dashboard.md - file: security/dashboards/entity-analytics-dashboard.md - file: security/dashboards/data-quality-dashboard.md diff --git a/troubleshoot/deployments/cloud-enterprise/common-issues.md b/troubleshoot/deployments/cloud-enterprise/common-issues.md index 25c97e570..b8e11e645 100644 --- a/troubleshoot/deployments/cloud-enterprise/common-issues.md +++ b/troubleshoot/deployments/cloud-enterprise/common-issues.md @@ -10,7 +10,7 @@ This set of common symptoms and resolutions can help you to diagnose unexpected ## Emergency token not spinning up the coordinator role [ece_emergency_token_not_spinning_up_the_coordinator_role] -**Symptom:** You have no access to API and UI because all coordinators are lost. More than half of the director hosts are available. If you have 5 directors, 3 directors must be available. If you lost more than half of the directors, contact the support. If all directors are lost, [re-install ECE](../../../deploy-manage/deploy/cloud-enterprise/install-ece-cloud.md). +**Symptom:** You have no access to API and UI because all coordinators are lost. More than half of the director hosts are available. If you have 5 directors, 3 directors must be available. If you lost more than half of the directors, contact the support. If all directors are lost, [re-install ECE](../../../deploy-manage/deploy/cloud-enterprise/install.md). **Resolution:** Use the emergency token provided during the installation of the genesis ECE nodes. You must explicitly specify the roles with the parameter `--roles`, for example `"coordinator,director,proxy"`. Otherwise, the host does not run any role. diff --git a/troubleshoot/deployments/cloud-enterprise/troubleshooting-container-engines.md b/troubleshoot/deployments/cloud-enterprise/troubleshooting-container-engines.md index ba1b83ddb..59b54f85a 100644 --- a/troubleshoot/deployments/cloud-enterprise/troubleshooting-container-engines.md +++ b/troubleshoot/deployments/cloud-enterprise/troubleshooting-container-engines.md @@ -16,7 +16,7 @@ Do not restart the Docker daemon unless directly prescribed by Elastic Support u ## Use supported configuration [ece-troubleshooting-containers-supported] -Make sure to use a combination of [Linux operating systems](../../../deploy-manage/deploy/cloud-enterprise/configure-operating-system-cloud.md) and container engine version that is supported, following our official [Support matrix](https://www.elastic.co/support/matrix#elastic-cloud-enterprise). Using unsupported combinations can cause a plethora of either intermediate or potentially permanent issues with you {{ece}} environment, such as failures to create [system deployments](../../../deploy-manage/deploy/cloud-enterprise/system-deployments-configuration.md), to upgrade workload deployments, proxy timeouts, data loss, and more. +Make sure to use a combination of [Linux operating systems](../../../deploy-manage/deploy/cloud-enterprise/configure-operating-system.md) and container engine version that is supported, following our official [Support matrix](https://www.elastic.co/support/matrix#elastic-cloud-enterprise). Using unsupported combinations can cause a plethora of either intermediate or potentially permanent issues with you {{ece}} environment, such as failures to create [system deployments](../../../deploy-manage/deploy/cloud-enterprise/system-deployments-configuration.md), to upgrade workload deployments, proxy timeouts, data loss, and more. ## Troubleshoot unhealthy containers [ece-troubleshooting-containers-unhealthy] diff --git a/troubleshoot/monitoring/unavailable-shards.md b/troubleshoot/monitoring/unavailable-shards.md index 3730767e0..f44ae531c 100644 --- a/troubleshoot/monitoring/unavailable-shards.md +++ b/troubleshoot/monitoring/unavailable-shards.md @@ -1,7 +1,7 @@ --- navigation_title: "Unavailable shards" mapped_urls: - - + - https://www.elastic.co/guide/en/cloud/current/ec-scenario_why_are_shards_unavailable.html - https://www.elastic.co/guide/en/cloud-heroku/current/echscenario_why_are_shards_unavailable.html - https://www.elastic.co/guide/en/cloud-heroku/current/ech-analyze_shards_with-api.html - https://www.elastic.co/guide/en/cloud-heroku/current/ech-analyze_shards_with-kibana.html