diff --git a/deploy-manage/_snippets/field-doc-sec-limitations.md b/deploy-manage/_snippets/field-doc-sec-limitations.md index 46c88d957..a2413e809 100644 --- a/deploy-manage/_snippets/field-doc-sec-limitations.md +++ b/deploy-manage/_snippets/field-doc-sec-limitations.md @@ -1,4 +1,4 @@ -Field and document security is subject to the following limitations: +Field and document security is subject to the following limitations: ### Document level security limitations @@ -6,7 +6,7 @@ When a user’s role enables [document level security](/deploy-manage/users-role * Document level security doesn’t affect global index statistics that relevancy scoring uses. This means that scores are computed without taking the role query into account. Documents that don’t match the role query are never returned. * The `has_child` and `has_parent` queries aren’t supported as query parameters in the role definition. The `has_child` and `has_parent` queries can be used in the search API with document level security enabled. -* [Date math](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/common-options.md#date-math) expressions cannot contain `now` in [range queries with date fields](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-range-query.md#ranges-on-dates). +* [Date math](elasticsearch://reference/elasticsearch/rest-apis/common-options.md#date-math) expressions cannot contain `now` in [range queries with date fields](elasticsearch://reference/query-languages/query-dsl-range-query.md#ranges-on-dates). * Any query that makes remote calls to fetch query data isn’t supported, including the following queries: * `terms` query with terms lookup @@ -16,9 +16,9 @@ When a user’s role enables [document level security](/deploy-manage/users-role * If suggesters are specified and document level security is enabled, the specified suggesters are ignored. * A search request cannot be profiled if document level security is enabled. * The [terms enum API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-terms-enum) does not return terms if document level security is enabled. -* The [`multi_match`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-multi-match-query.md) query does not support specifying fields using wildcards. +* The [`multi_match`](elasticsearch://reference/query-languages/query-dsl-multi-match-query.md) query does not support specifying fields using wildcards. -:::{note} +:::{note} While document-level security prevents users from viewing restricted documents, it’s still possible to write search requests that return aggregate information about the entire index. A user whose access is restricted to specific documents in an index could still learn about field names and terms that only exist in inaccessible documents, and count how many inaccessible documents contain a given term. ::: diff --git a/deploy-manage/cloud-organization/tools-and-apis.md b/deploy-manage/cloud-organization/tools-and-apis.md index e1a502ebf..c09f3d93c 100644 --- a/deploy-manage/cloud-organization/tools-and-apis.md +++ b/deploy-manage/cloud-organization/tools-and-apis.md @@ -18,7 +18,7 @@ Most Elastic resources can be accessed and managed through RESTful APIs. While t Elasticsearch APIs : This set of APIs allows you to interact directly with the Elasticsearch nodes in your deployment. You can ingest data, run search queries, check the health of your clusters, manage snapshots, and more. - To use these APIs on {{ecloud}} read our topic [Access the API console](asciidocalypse://docs/cloud/docs/reference/cloud-hosted/ec-api-console.md), and to learn about all of the available endpoints check the [Elasticsearch API reference documentation](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/index.md). + To use these APIs on {{ecloud}} read our topic [Access the API console](asciidocalypse://docs/cloud/docs/reference/cloud-hosted/ec-api-console.md), and to learn about all of the available endpoints check the [Elasticsearch API reference documentation](elasticsearch://reference/elasticsearch/rest-apis/index.md). Some [restrictions](../deploy/elastic-cloud/restrictions-known-problems.md#ec-restrictions-apis-elasticsearch) apply when using the Elasticsearch APIs on {{ecloud}}. @@ -37,7 +37,7 @@ Other Products * [APM event intake API Reference](/solutions/observability/apps/elastic-apm-events-intake-api.md) * [App Search API Reference](https://www.elastic.co/guide/en/app-search/current/api-reference.html) * [Elastic Security APIs](https://www.elastic.co/guide/en/security/current/security-apis.html) - * [Fleet APIs](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/fleet-api-docs.md) + * [Fleet APIs](/reference/ingestion-tools/fleet/fleet-api-docs.md) * [Logstash APIs](https://www.elastic.co/guide/en/logstash/current/monitoring-logstash.html) * [Workplace Search API Reference](https://www.elastic.co/guide/en/workplace-search/current/workplace-search-api-overview.html) diff --git a/deploy-manage/deploy/cloud-enterprise.md b/deploy-manage/deploy/cloud-enterprise.md index 41258c4f4..359074524 100644 --- a/deploy-manage/deploy/cloud-enterprise.md +++ b/deploy-manage/deploy/cloud-enterprise.md @@ -32,7 +32,7 @@ Refer to [](./cloud-enterprise/ece-architecture.md) for details about the ECE pl - **Air-gapped installations**: Support for off-line installations. - **Microservices architecture**: All services are containerized through Docker. -Check the [glossary](asciidocalypse:///docs-content/docs/reference/glossary.md) to get familiar with the terminology for ECE as well as other Elastic products and solutions. +Check the [glossary](/reference/glossary/index.md) to get familiar with the terminology for ECE as well as other Elastic products and solutions. ## Section overview diff --git a/deploy-manage/deploy/cloud-enterprise/add-custom-bundles-plugins.md b/deploy-manage/deploy/cloud-enterprise/add-custom-bundles-plugins.md index 74e917ddd..d97006ba1 100644 --- a/deploy-manage/deploy/cloud-enterprise/add-custom-bundles-plugins.md +++ b/deploy-manage/deploy/cloud-enterprise/add-custom-bundles-plugins.md @@ -1,8 +1,8 @@ --- -mapped_pages: +mapped_pages: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-add-custom-bundle-plugin.html navigation_title: "Custom bundles and plugins" -applies_to: +applies_to: deployment: ece: --- @@ -360,7 +360,7 @@ You do not need to do this step if you are using default filename and password ( } ``` -4. To use this bundle, you can refer it in the [GeoIP processor](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/geoip-processor.md) of an ingest pipeline as `MyGeoLite2-City.mmdb` under `database_file` such as: +4. To use this bundle, you can refer it in the [GeoIP processor](elasticsearch://reference/ingestion-tools/enrich-processor/geoip-processor.md) of an ingest pipeline as `MyGeoLite2-City.mmdb` under `database_file` such as: ```sh ... diff --git a/deploy-manage/deploy/cloud-enterprise/add-plugins.md b/deploy-manage/deploy/cloud-enterprise/add-plugins.md index 632fc4395..8d267c7b0 100644 --- a/deploy-manage/deploy/cloud-enterprise/add-plugins.md +++ b/deploy-manage/deploy/cloud-enterprise/add-plugins.md @@ -16,7 +16,7 @@ Adding plugins to a deployment is as simple as selecting it from the list of ava Additional plugins might be available. If a plugin is listed for your version of {{es}}, it can be used. -You can also [create](asciidocalypse://elasticsearch/docs/extend/create-elasticsearch-plugins.md) and add custom plugins. +You can also [create](elasticsearch://extend/index.md) and add custom plugins. To add plugins when creating a new deployment: @@ -32,7 +32,7 @@ To add plugins to an existing deployment: 1. [Log into the Cloud UI](/deploy-manage/deploy/cloud-enterprise/log-into-cloud-ui.md). 2. On the **Deployments** page, select your deployment. - + Narrow the list by name, ID, or choose from several other filters. To further define the list, use a combination of filters. 3. From your deployment menu, go to the **Edit** page. diff --git a/deploy-manage/deploy/cloud-enterprise/assign-roles-to-hosts.md b/deploy-manage/deploy/cloud-enterprise/assign-roles-to-hosts.md index dd4f7d51e..21a2045f6 100644 --- a/deploy-manage/deploy/cloud-enterprise/assign-roles-to-hosts.md +++ b/deploy-manage/deploy/cloud-enterprise/assign-roles-to-hosts.md @@ -26,14 +26,14 @@ Each Elastic Cloud Enterprise runner can take on several roles: `director` : Manages the ZooKeeper datastore. This role is typically shared with the coordinator role. In production deployments it can be separated from a coordinator. -::::{important} +::::{important} Once the `director` role is assigned to a runner, the Zookeeper service starts on that host. The Zookeeper service continues even after the director role is removed from the runner. Therefore, if you remove the `director` role from any host that has ever had it, we highly recommend also [deleting the runner](../../maintenance/ece/delete-ece-hosts.md) and re-installing it. :::: Each role is associated with a set of Docker containers that provide the specific functionality. -There are some additional roles shown in the Cloud UI, such as the [beats-runner](asciidocalypse://docs/docs-content/docs/reference/glossary/index.md#glossary-beats-runner) and [services-forwarder](asciidocalypse://docs/docs-content/docs/reference/glossary/index.md#glossary-services-forwarder) roles, that are required by Elastic Cloud Enterprise and that you cannot modify. +There are some additional roles shown in the Cloud UI, such as the [beats-runner](/reference/glossary/index.md#glossary-beats-runner) and [services-forwarder](/reference/glossary/index.md#glossary-services-forwarder) roles, that are required by Elastic Cloud Enterprise and that you cannot modify. To assign roles to hosts: diff --git a/deploy-manage/deploy/cloud-enterprise/ce-add-support-for-node-roles-autoscaling.md b/deploy-manage/deploy/cloud-enterprise/ce-add-support-for-node-roles-autoscaling.md index 8199d3510..bdb668f1a 100644 --- a/deploy-manage/deploy/cloud-enterprise/ce-add-support-for-node-roles-autoscaling.md +++ b/deploy-manage/deploy/cloud-enterprise/ce-add-support-for-node-roles-autoscaling.md @@ -22,7 +22,7 @@ The `node_roles` field defines the roles that an Elasticsearch topology element There are a number of fields that need to be added to each Elasticsearch node in order to support `node_roles`: * **id**: Unique identifier of the topology element. This field, along with the `node_roles`, identifies an Elasticsearch topology element. -* **node_roles**: The list of node roles. Allowable roles are: `master`, `ingest`, `ml`, `data_hot`, `data_content`, `data_warm`, `data_cold`, `data_frozen`, `remote_cluster_client`, and `transform`. For details, check [Node roles](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md#node-roles). +* **node_roles**: The list of node roles. Allowable roles are: `master`, `ingest`, `ml`, `data_hot`, `data_content`, `data_warm`, `data_cold`, `data_frozen`, `remote_cluster_client`, and `transform`. For details, check [Node roles](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md#node-roles). * **topology_element_control**: Controls for the topology element. * **min**: The absolute minimum size limit for a topology element. If the value is `0`, that means the topology element can be disabled. diff --git a/deploy-manage/deploy/cloud-enterprise/ece-configure-templates-index-management.md b/deploy-manage/deploy/cloud-enterprise/ece-configure-templates-index-management.md index a9495cbc1..ff260f489 100644 --- a/deploy-manage/deploy/cloud-enterprise/ece-configure-templates-index-management.md +++ b/deploy-manage/deploy/cloud-enterprise/ece-configure-templates-index-management.md @@ -43,7 +43,7 @@ To configure index management when you create a deployment template: Index curation : Creates new indices on hot nodes first and moves them to warm nodes later on, based on the data views (formerly *index patterns*) you specify. Also manages replica counts for you, so that all shards of an index can fit on the right data nodes. Compared to index lifecycle management, index curation for time-based indices supports only one action, to move indices from nodes on one data configuration to another, but it is more straightforward to set up initially and all setup can be done directly from the Cloud UI. - If your user need to delete indices once they are no longer useful to them, they can run [Curator](asciidocalypse://docs/curator/docs/reference/index.md) on-premise to manage indices for Elasticsearch clusters hosted on Elastic Cloud Enterprise. + If your user need to delete indices once they are no longer useful to them, they can run [Curator](curator://reference/index.md) on-premise to manage indices for Elasticsearch clusters hosted on Elastic Cloud Enterprise. To configure index curation: diff --git a/deploy-manage/deploy/cloud-enterprise/ece-ha.md b/deploy-manage/deploy/cloud-enterprise/ece-ha.md index 8db8390d2..f3dec4dd4 100644 --- a/deploy-manage/deploy/cloud-enterprise/ece-ha.md +++ b/deploy-manage/deploy/cloud-enterprise/ece-ha.md @@ -35,11 +35,11 @@ Increasing the number of zones should not be used to add more resources. The con ## Master nodes [ece-ece-ha-2-master-nodes] -Tiebreakers are used in distributed clusters to avoid cases of [split brain](https://en.wikipedia.org/wiki/Split-brain_(computing)), where an {{es}} cluster splits into multiple, autonomous parts that continue to handle requests independently of each other, at the risk of affecting cluster consistency and data loss. A split-brain scenario is avoided by making sure that a minimum number of [master-eligible nodes](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md#master-node) must be present in order for any part of the cluster to elect a master node and accept user requests. To prevent multiple parts of a cluster from being eligible, there must be a [quorum-based majority](/deploy-manage/distributed-architecture/discovery-cluster-formation/modules-discovery-quorums.md) of `(n/2)+1` nodes, where `n` is the number of master-eligible nodes in the cluster. The minimum number of master nodes to reach quorum in a two-node cluster is the same as for a three-node cluster: two nodes must be available. +Tiebreakers are used in distributed clusters to avoid cases of [split brain](https://en.wikipedia.org/wiki/Split-brain_(computing)), where an {{es}} cluster splits into multiple, autonomous parts that continue to handle requests independently of each other, at the risk of affecting cluster consistency and data loss. A split-brain scenario is avoided by making sure that a minimum number of [master-eligible nodes](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md#master-node) must be present in order for any part of the cluster to elect a master node and accept user requests. To prevent multiple parts of a cluster from being eligible, there must be a [quorum-based majority](/deploy-manage/distributed-architecture/discovery-cluster-formation/modules-discovery-quorums.md) of `(n/2)+1` nodes, where `n` is the number of master-eligible nodes in the cluster. The minimum number of master nodes to reach quorum in a two-node cluster is the same as for a three-node cluster: two nodes must be available. When you create a cluster with nodes in two availability zones when a third zone is available, ECE can create a tiebreaker in the third availability zone to help establish quorum in case of loss of an availability zone. The extra tiebreaker node that helps to provide quorum does not have to be a full-fledged and expensive node, as it does not hold data. For example: By tagging allocators hosts in ECE, can you create a cluster with eight nodes each in zones `ece-1a` and `ece-1b`, for a total of 16 nodes, and one tiebreaker node in zone `ece-1c`. This cluster can lose any of the three availability zones whilst maintaining quorum, which means that the cluster can continue to process user requests, provided that there is sufficient capacity available when an availability zone goes down. -By default, each node in an {{es}} cluster is a master-eligible node and a data node. In larger clusters, such as production clusters, it’s a good practice to split the roles, so that master nodes are not handling search or indexing work. When you create a cluster, you can specify to use dedicated [master-eligible nodes](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md#master-node), one per availability zone. +By default, each node in an {{es}} cluster is a master-eligible node and a data node. In larger clusters, such as production clusters, it’s a good practice to split the roles, so that master nodes are not handling search or indexing work. When you create a cluster, you can specify to use dedicated [master-eligible nodes](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md#master-node), one per availability zone. ::::{warning} Clusters that only have two or fewer master-eligible node are not [highly available](/deploy-manage/production-guidance/availability-and-resilience.md) and are at risk of data loss. You must have [at least three master-eligible nodes](/deploy-manage/distributed-architecture/discovery-cluster-formation/modules-discovery-quorums.md). diff --git a/deploy-manage/deploy/cloud-enterprise/ece-manage-capacity.md b/deploy-manage/deploy/cloud-enterprise/ece-manage-capacity.md index 70b30415f..ea5d33b79 100644 --- a/deploy-manage/deploy/cloud-enterprise/ece-manage-capacity.md +++ b/deploy-manage/deploy/cloud-enterprise/ece-manage-capacity.md @@ -18,13 +18,13 @@ This section focuses on the allocator role, and explains how to plan its capacit * [Storage](#ece-alloc-storage) -## Memory [ece-alloc-memory] +## Memory [ece-alloc-memory] You should plan your deployment size based on the amount of data you ingest. Memory is the main scaling unit for a deployment. Other units, like CPU and disks, are proportional to the memory size. The memory available for an allocator is called *capacity*. During installation, the allocator capacity defaults to 85% of the host physical memory, as the rest is reserved for ECE system services. -::::{note} +::::{note} ECE does not support hot-adding of resources to a running node. When increasing CPU/memory allocated to a ECE node, a restart is needed to utilize the additional resources. :::: @@ -41,13 +41,13 @@ curl -X PUT \ For more information on how to use API keys for authentication, check the section [Access the API from the Command Line](asciidocalypse://docs/cloud/docs/reference/cloud-enterprise/ece-api-command-line.md). -::::{important} +::::{important} Prior to ECE 3.5.0, regardless of the use of this API, the [CPU quota](#ece-alloc-cpu) used the memory specified at installation time. :::: -### Examples [ece_examples] +### Examples [ece_examples] Here are some examples to make Elastic deployments and ECE system services run smoothly on your host: @@ -59,14 +59,14 @@ Note that the recommended reservations above are not guaranteed upper limits, if These fluctuations should not be a concern in practice. To get actual limits that could be used in alerts, you could add 4GB to the recommended values above. -## CPU quotas [ece-alloc-cpu] +## CPU quotas [ece-alloc-cpu] ECE uses CPU quotas to assign shares of the allocator host to the instances that are running on it. To calculate the CPU quota, use the following formula: `CPU quota = DeploymentRAM / HostCapacity` -### Examples [ece_examples_2] +### Examples [ece_examples_2] Consider a 32GB deployment hosted on a 128GB allocator. @@ -81,19 +81,19 @@ If you use 12GB Allocator system service reservation, the CPU quota is 28%: Those percentages represent the upper limit of the % of the total CPU resources available in a given 100ms period. -## Processors setting [ece-alloc-processors-setting] +## Processors setting [ece-alloc-processors-setting] In addition to the [CPU quotas](#ece-alloc-cpu), the `processors` setting also plays a relevant role. -The allocated `processors` setting originates from Elasticsearch and is responsible for calculating your [thread pools](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/thread-pool-settings.md#node.processors). While the CPU quota defines the percentage of the total CPU resources of an allocator that are assigned to an instance, the allocated `processors` define how the thread pools are calculated in Elasticsearch, and therefore how many concurrent search and indexing requests an instance can process. In other words, the CPU ratio defines how fast a single task can be completed, while the `processors` setting defines how many different tasks can be completed at the same time. +The allocated `processors` setting originates from Elasticsearch and is responsible for calculating your [thread pools](elasticsearch://reference/elasticsearch/configuration-reference/thread-pool-settings.md#node.processors). While the CPU quota defines the percentage of the total CPU resources of an allocator that are assigned to an instance, the allocated `processors` define how the thread pools are calculated in Elasticsearch, and therefore how many concurrent search and indexing requests an instance can process. In other words, the CPU ratio defines how fast a single task can be completed, while the `processors` setting defines how many different tasks can be completed at the same time. We rely on Elasticsearch and the `-XX:ActiveProcessorCount` JVM setting to automatically detect the allocated `processors`. -In earlier versions of ECE and Elasticsearch, the [Elasticsearch processors](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/thread-pool-settings.md#node.processors) setting was used to configure the allocated `processors` according to the following formula: +In earlier versions of ECE and Elasticsearch, the [Elasticsearch processors](elasticsearch://reference/elasticsearch/configuration-reference/thread-pool-settings.md#node.processors) setting was used to configure the allocated `processors` according to the following formula: `Math.min(16,Math.max(2,(16*instanceCapacity*1.0/1024/64).toInt))` -The following table gives an overview of the allocated `processors` that are used to calculate the Elasticsearch [thread pools](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/thread-pool-settings.md) based on the preceding formula: +The following table gives an overview of the allocated `processors` that are used to calculate the Elasticsearch [thread pools](elasticsearch://reference/elasticsearch/configuration-reference/thread-pool-settings.md) based on the preceding formula: | instance size | vCPU | | --- | --- | @@ -107,18 +107,18 @@ The following table gives an overview of the allocated `processors` that are use This table also provides a rough indication of what the auto-detected value could be on newer versions of ECE and Elasticsearch. -## Storage [ece-alloc-storage] +## Storage [ece-alloc-storage] ECE has specific [hardware prerequisites](ece-hardware-prereq.md) for storage. Disk space is consumed by system logs, container overhead, and deployment data. The main factor for selecting a disk quota is the deployment data, that is, data from your Elasticsearch, Kibana, and APM nodes. The biggest portion of data is consumed by the Elasticsearch nodes. -::::{note} +::::{note} ECE uses [XFS](ece-software-prereq.md#ece-xfs) to enforce specific disk space quotas to control the disk consumption for the deployment nodes running on your allocator. :::: -::::{important} +::::{important} You must use XFS and have quotas enabled on all allocators, otherwise disk usage won’t display correctly. :::: @@ -141,7 +141,7 @@ You can change the value of the disk multiplier at different levels: 3. Adjust the disk quota to your needs. -::::{important} +::::{important} The override only persists during the lifecycle of the instance container. If a new container is created, for example during a `grow_and_shrink` plan or a vacate operation, the quota is reset to its default. To increase the storage ratio in a persistent way, [edit the instance configurations](ece-configuring-ece-instance-configurations-edit.md). :::: diff --git a/deploy-manage/deploy/cloud-enterprise/ece-regional-deployment-aliases.md b/deploy-manage/deploy/cloud-enterprise/ece-regional-deployment-aliases.md index e0e152b0c..77a19aea3 100644 --- a/deploy-manage/deploy/cloud-enterprise/ece-regional-deployment-aliases.md +++ b/deploy-manage/deploy/cloud-enterprise/ece-regional-deployment-aliases.md @@ -10,7 +10,7 @@ Custom aliases for your deployment endpoints on Elastic Cloud Enterprise allow y Before setting up your custom alias, your platform administrator must enable the feature. Check [Enable custom endpoint aliases](enable-custom-endpoint-aliases.md) for more information. -## Create a custom endpoint alias for a deployment [ece-create-regional-deployment-alias] +## Create a custom endpoint alias for a deployment [ece-create-regional-deployment-alias] To add an alias to an existing deployment: @@ -19,14 +19,14 @@ To add an alias to an existing deployment: 3. Under **Custom endpoint alias**, select **Edit**. 4. Define a new alias. Make sure you choose something meaningful to you. - ::::{tip} + ::::{tip} Make the alias as unique as possible to avoid collisions. Aliases might have been already claimed by other users for deployments in the region. :::: 5. Select **Update alias**. -## Remove a custom endpoint alias [ece-delete-regional-deployment-alias] +## Remove a custom endpoint alias [ece-delete-regional-deployment-alias] To remove an alias from your deployment, or if you want to re-assign an alias to another deployment, follow these steps: @@ -37,17 +37,17 @@ To remove an alias from your deployment, or if you want to re-assign an alias to 5. Select **Update alias**. -## Using the custom endpoint URL [ece-using-regional-deployment-alias] +## Using the custom endpoint URL [ece-using-regional-deployment-alias] To use your new custom endpoint URL to access your Elastic products, note that each has its own alias to use in place of the default application UUID. For example, if you configured the custom endpoint alias for your deployment to be `test-alias`, the corresponding alias for the Elasticsearch cluster in that deployment is `test-alias.es`. -::::{note} +::::{note} You can get the application-specific custom endpoint alias by selecting **Copy endpoint** for that product. It should contain a subdomain for each application type, for example `es`, `kb`, `apm`, or `ent`. :::: -### With the REST Client [ece-rest-regional-deployment-alias] +### With the REST Client [ece-rest-regional-deployment-alias] * As part of the host name: @@ -61,7 +61,7 @@ You can get the application-specific custom endpoint alias by selecting **Copy e For more information on setting up a load balancer to ensure proper routing, check [Load balancers](ece-load-balancers.md). -### With the `TransportClient` [ece-transport-regional-deployment-alias] +### With the `TransportClient` [ece-transport-regional-deployment-alias] While the `TransportClient` is deprecated, your custom endpoint aliases still work with it. Similar to the REST Client, there are two ways to use your custom endpoint alias with the `TransportClient`: @@ -99,5 +99,5 @@ While the `TransportClient` is deprecated, your custom endpoint aliases still wo ``` -For more information on configuring the `TransportClient`, see [Configure the Java Transport Client](asciidocalypse://docs/elasticsearch-java/docs/reference/index.md). +For more information on configuring the `TransportClient`, see [Configure the Java Transport Client](elasticsearch-java://reference/index.md). diff --git a/deploy-manage/deploy/cloud-enterprise/post-installation-steps.md b/deploy-manage/deploy/cloud-enterprise/post-installation-steps.md index 2580c1d89..e1ece1349 100644 --- a/deploy-manage/deploy/cloud-enterprise/post-installation-steps.md +++ b/deploy-manage/deploy/cloud-enterprise/post-installation-steps.md @@ -37,13 +37,13 @@ To start creating {{es}} deployments directly, refer to [](./working-with-deploy * [Add a snapshot repository](../../tools/snapshot-and-restore/cloud-enterprise.md) to enable regular backups of your Elasticsearch clusters. * Consider enabling encryption-at-rest (EAR) on your hosts. - + :::{{note}} Encryption-at-rest is not implemented out of the box in {{ece}}. [Learn more](/deploy-manage/security/secure-your-elastic-cloud-enterprise-installation.md#ece_encryption). ::: * Learn about common maintenance activities—such as adding capacity, applying OS patches, and addressing host failures--at [](../../maintenance/ece.md). -::::{warning} +::::{warning} During installation, the system generates secrets that are placed into the `/mnt/data/elastic/bootstrap-state/bootstrap-secrets.json` secrets file, unless you passed in a different path with the --host-storage-path parameter. Keep the information in the `bootstrap-secrets.json` file secure by removing it from its default location and placing it into a secure storage location. :::: diff --git a/deploy-manage/deploy/cloud-enterprise/tools-apis.md b/deploy-manage/deploy/cloud-enterprise/tools-apis.md index e427c5852..5e27b6f21 100644 --- a/deploy-manage/deploy/cloud-enterprise/tools-apis.md +++ b/deploy-manage/deploy/cloud-enterprise/tools-apis.md @@ -6,12 +6,11 @@ ⚠️ **This page is a work in progress.** ⚠️ -You can use these tools and APIs to interact with the following {{ece}} features: +You can use these tools and APIs to interact with the following {{ece}} features: -* [{{ecloud}} Control (ecctl)](asciidocalypse://docs/ecctl/docs/reference/index.md): Wraps typical operations commonly needed by operators within a single command line tool. -* [ECE scripts](asciidocalypse://docs/cloud/docs/reference/cloud-enterprise/scripts.md): Use the `elastic-cloud-enterprise.sh` script to install {{ece}} or modify your installation. +* [{{ecloud}} Control (ecctl)](ecctl://reference/index.md): Wraps typical operations commonly needed by operators within a single command line tool. +* [ECE scripts](asciidocalypse://docs/cloud/docs/reference/cloud-enterprise/scripts.md): Use the `elastic-cloud-enterprise.sh` script to install {{ece}} or modify your installation. * [ECE diagnostics tool](/troubleshoot/deployments/cloud-enterprise/run-ece-diagnostics-tool.md): Collect logs and metrics that you can send to Elastic Support for troubleshooting and investigation purposes. - - \ No newline at end of file + diff --git a/deploy-manage/deploy/cloud-on-k8s/advanced-configuration-logstash.md b/deploy-manage/deploy/cloud-on-k8s/advanced-configuration-logstash.md index eb5304d47..994ea4785 100644 --- a/deploy-manage/deploy/cloud-on-k8s/advanced-configuration-logstash.md +++ b/deploy-manage/deploy/cloud-on-k8s/advanced-configuration-logstash.md @@ -37,7 +37,7 @@ spec: You can specify sensitive settings with Kubernetes secrets. ECK automatically injects these settings into the keystore before it starts Logstash. The ECK operator continues to watch the secrets for changes and will restart Logstash Pods when it detects a change. -The Logstash Keystore can be password protected by setting an environment variable called `LOGSTASH_KEYSTORE_PASS`. Check out [Logstash Keystore](asciidocalypse://docs/logstash/docs/reference/keystore.md#keystore-password) documentation for details. +The Logstash Keystore can be password protected by setting an environment variable called `LOGSTASH_KEYSTORE_PASS`. Check out [Logstash Keystore](logstash://reference/keystore.md#keystore-password) documentation for details. ```yaml apiVersion: v1 diff --git a/deploy-manage/deploy/cloud-on-k8s/advanced-elasticsearch-node-scheduling.md b/deploy-manage/deploy/cloud-on-k8s/advanced-elasticsearch-node-scheduling.md index f53377b5c..ffc15f25a 100644 --- a/deploy-manage/deploy/cloud-on-k8s/advanced-elasticsearch-node-scheduling.md +++ b/deploy-manage/deploy/cloud-on-k8s/advanced-elasticsearch-node-scheduling.md @@ -19,7 +19,7 @@ You can combine these features to deploy a production-grade Elasticsearch cluste ## Define Elasticsearch nodes roles [k8s-define-elasticsearch-nodes-roles] -You can configure Elasticsearch nodes with [one or multiple roles](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md). +You can configure Elasticsearch nodes with [one or multiple roles](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md). ::::{tip} You can use [YAML anchors](https://yaml.org/spec/1.2/spec.html#id2765878) to declare the configuration change once and reuse it across all the node sets. @@ -198,7 +198,7 @@ This example restricts Elasticsearch nodes so they are only scheduled on Kuberne ## Topology spread constraints and availability zone awareness [k8s-availability-zone-awareness] -Starting with ECK 2.0 the operator can make Kubernetes Node labels available as Pod annotations. It can be used to make information, such as logical failure domains, available in a running Pod. Combined with [Elasticsearch shard allocation awareness](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#allocation-awareness) and [Kubernetes topology spread constraints](https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/), you can create an availability zone-aware Elasticsearch cluster. +Starting with ECK 2.0 the operator can make Kubernetes Node labels available as Pod annotations. It can be used to make information, such as logical failure domains, available in a running Pod. Combined with [Elasticsearch shard allocation awareness](elasticsearch://reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md) and [Kubernetes topology spread constraints](https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/), you can create an availability zone-aware Elasticsearch cluster. ### Exposing Kubernetes node topology labels in Pods [k8s-availability-zone-awareness-downward-api] @@ -253,13 +253,13 @@ This example relies on: * Kubernetes nodes in each zone being labeled accordingly. `topology.kubernetes.io/zone` [is standard](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#interlude-built-in-node-labels), but any label can be used. * [Pod topology spread constraints](https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/) to spread the Pods across availability zones in the Kubernetes cluster. -* Elasticsearch configured to [allocate shards based on node attributes](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#allocation-awareness). Here we specified `node.attr.zone`, but any attribute name can be used. `node.attr.rack_id` is another common example. +* Elasticsearch configured to [allocate shards based on node attributes](elasticsearch://reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md). Here we specified `node.attr.zone`, but any attribute name can be used. `node.attr.rack_id` is another common example. ## Hot-warm topologies [k8s-hot-warm-topologies] -By combining [Elasticsearch shard allocation awareness](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#allocation-awareness) with [Kubernetes node affinity](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity-beta-feature), you can set up an Elasticsearch cluster with hot-warm topology: +By combining [Elasticsearch shard allocation awareness](elasticsearch://reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md) with [Kubernetes node affinity](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity-beta-feature), you can set up an Elasticsearch cluster with hot-warm topology: ```yaml apiVersion: elasticsearch.k8s.elastic.co/v1 diff --git a/deploy-manage/deploy/cloud-on-k8s/configuration-examples-standalone.md b/deploy-manage/deploy/cloud-on-k8s/configuration-examples-standalone.md index a673a2e54..342cc86bd 100644 --- a/deploy-manage/deploy/cloud-on-k8s/configuration-examples-standalone.md +++ b/deploy-manage/deploy/cloud-on-k8s/configuration-examples-standalone.md @@ -8,7 +8,7 @@ mapped_pages: # Configuration examples [k8s-elastic-agent-configuration-examples] -This section contains manifests that illustrate common use cases, and can be your starting point in exploring Elastic Agent deployed with ECK. These manifests are self-contained and work out-of-the-box on any non-secured Kubernetes cluster. They all contain a three-node Elasticsearch cluster and a single Kibana instance. Add the corresponding integration package to Kibana to install the dashboards, visualizations and other assets for each of these examples as described in [the Elastic Agent documentation](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/install-elastic-agents.md). +This section contains manifests that illustrate common use cases, and can be your starting point in exploring Elastic Agent deployed with ECK. These manifests are self-contained and work out-of-the-box on any non-secured Kubernetes cluster. They all contain a three-node Elasticsearch cluster and a single Kibana instance. Add the corresponding integration package to Kibana to install the dashboards, visualizations and other assets for each of these examples as described in [the Elastic Agent documentation](/reference/ingestion-tools/fleet/install-elastic-agents.md). ::::{warning} The examples in this section are for illustration purposes only and should not be considered to be production-ready. Some of these examples use the `node.store.allow_mmap: false` setting which has performance implications and should be tuned for production workloads, as described in [Virtual memory](virtual-memory.md). diff --git a/deploy-manage/deploy/cloud-on-k8s/configuration-fleet.md b/deploy-manage/deploy/cloud-on-k8s/configuration-fleet.md index 35c3857f5..05dd88bd1 100644 --- a/deploy-manage/deploy/cloud-on-k8s/configuration-fleet.md +++ b/deploy-manage/deploy/cloud-on-k8s/configuration-fleet.md @@ -10,7 +10,7 @@ mapped_pages: {{fleet}}-managed {{agents}} must connect to {{fleet-server}} to receive their configurations. You can deploy {{fleet-server}} instances using ECKs Agent CRD with the appropriate configuration, as shown in [Fleet mode and Fleet Server](#k8s-elastic-agent-fleet-configuration-fleet-mode-and-fleet-server). -To know more about {{fleet}} architecture and related components, check the {{fleet}} [documentation](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/fleet-server.md). +To know more about {{fleet}} architecture and related components, check the {{fleet}} [documentation](/reference/ingestion-tools/fleet/fleet-server.md). ## {{fleet}} mode and {{fleet-server}} [k8s-elastic-agent-fleet-configuration-fleet-mode-and-fleet-server] @@ -241,7 +241,7 @@ spec: ... ``` -Please note that the environment variables related to policy selection mentioned in the {{agent}} [docs](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/agent-environment-variables.md) like `FLEET_SERVER_POLICY_ID` will be managed by the ECK operator. +Please note that the environment variables related to policy selection mentioned in the {{agent}} [docs](/reference/ingestion-tools/fleet/agent-environment-variables.md) like `FLEET_SERVER_POLICY_ID` will be managed by the ECK operator. ## Running as a non-root user [k8s-elastic-agent-running-as-a-non-root-user] diff --git a/deploy-manage/deploy/cloud-on-k8s/configuration-standalone.md b/deploy-manage/deploy/cloud-on-k8s/configuration-standalone.md index b2656f8c7..645e90b23 100644 --- a/deploy-manage/deploy/cloud-on-k8s/configuration-standalone.md +++ b/deploy-manage/deploy/cloud-on-k8s/configuration-standalone.md @@ -108,7 +108,7 @@ stringData: period: 10s ``` -You can use the Fleet application in Kibana to generate the configuration for Elastic Agent, even when running in standalone mode. Check the [Elastic Agent standalone](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/install-standalone-elastic-agent.md) documentation. Adding the corresponding integration package to Kibana also adds the related dashboards and visualizations. +You can use the Fleet application in Kibana to generate the configuration for Elastic Agent, even when running in standalone mode. Check the [Elastic Agent standalone](/reference/ingestion-tools/fleet/install-standalone-elastic-agent.md) documentation. Adding the corresponding integration package to Kibana also adds the related dashboards and visualizations. ## Use multiple Elastic Agent outputs [k8s-elastic-agent-multi-output] diff --git a/deploy-manage/deploy/cloud-on-k8s/create-custom-images.md b/deploy-manage/deploy/cloud-on-k8s/create-custom-images.md index 59e1a7ebe..606c59c5b 100644 --- a/deploy-manage/deploy/cloud-on-k8s/create-custom-images.md +++ b/deploy-manage/deploy/cloud-on-k8s/create-custom-images.md @@ -8,7 +8,7 @@ mapped_pages: # Create custom images [k8s-custom-images] -You can create your own custom application images (Elasticsearch, Kibana, APM Server, Beats, Elastic Agent, Elastic Maps Server, and Logstash) instead of using the base images provided by Elastic. You might want to do this to have a canonical image with all the necessary plugins pre-loaded rather than [installing them through an init container](init-containers-for-plugin-downloads.md) each time a Pod starts. You must use the official image as the base for custom images. For example, if you want to create an Elasticsearch 8.16.1 image with the [ICU Analysis Plugin](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch-plugins/analysis-icu.md), you can do the following: +You can create your own custom application images (Elasticsearch, Kibana, APM Server, Beats, Elastic Agent, Elastic Maps Server, and Logstash) instead of using the base images provided by Elastic. You might want to do this to have a canonical image with all the necessary plugins pre-loaded rather than [installing them through an init container](init-containers-for-plugin-downloads.md) each time a Pod starts. You must use the official image as the base for custom images. For example, if you want to create an Elasticsearch 8.16.1 image with the [ICU Analysis Plugin](elasticsearch://reference/elasticsearch-plugins/analysis-icu.md), you can do the following: 1. Create a `Dockerfile` containing: diff --git a/deploy-manage/deploy/cloud-on-k8s/custom-configuration-files-plugins.md b/deploy-manage/deploy/cloud-on-k8s/custom-configuration-files-plugins.md index 4e68b62a3..d0480c52d 100644 --- a/deploy-manage/deploy/cloud-on-k8s/custom-configuration-files-plugins.md +++ b/deploy-manage/deploy/cloud-on-k8s/custom-configuration-files-plugins.md @@ -56,7 +56,7 @@ Refer to [Creating custom images](create-custom-images.md) for instructions on h ## Use init containers for plugins installation -The following example describes option 2, using a repository plugin. To install the plugin before the Elasticsearch nodes start, use an init container to run the [plugin installation tool](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch-plugins/installation.md). +The following example describes option 2, using a repository plugin. To install the plugin before the Elasticsearch nodes start, use an init container to run the [plugin installation tool](elasticsearch://reference/elasticsearch-plugins/installation.md). ```yaml spec: @@ -107,7 +107,7 @@ To install custom configuration files you can: 1. Add the configuration data into a ConfigMap or Secret. 2. Use volumes and volume mounts in your manifest to mount the contents of the ConfigMap or Secret as files in your {{es}} nodes. -The next example shows how to add a synonyms file for the [synonym token filter](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-synonym-tokenfilter.md) in Elasticsearch. But you can **use the same approach for any kind of file you want to mount into the configuration directory of Elasticsearch**, like adding CA certificates of external systems. +The next example shows how to add a synonyms file for the [synonym token filter](elasticsearch://reference/data-analysis/text-analysis/analysis-synonym-tokenfilter.md) in Elasticsearch. But you can **use the same approach for any kind of file you want to mount into the configuration directory of Elasticsearch**, like adding CA certificates of external systems. 1. Create the ConfigMap or Secret with the data: diff --git a/deploy-manage/deploy/cloud-on-k8s/elastic-stack-configuration-policies.md b/deploy-manage/deploy/cloud-on-k8s/elastic-stack-configuration-policies.md index 89016b078..21a1fe322 100644 --- a/deploy-manage/deploy/cloud-on-k8s/elastic-stack-configuration-policies.md +++ b/deploy-manage/deploy/cloud-on-k8s/elastic-stack-configuration-policies.md @@ -34,7 +34,7 @@ Starting from ECK `2.6.1` and Elasticsearch `8.6.1`, Elastic Stack configuration Additionally with ECK `2.11.0` it is possible to configure Kibana as well using Elastic Stack configuration policies, the following settings can be configured for Kibana: -* [Kibana Configuration](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md) (configuration settings for Kibana that will go into `kibana.yml`) +* [Kibana Configuration](kibana://reference/configuration-reference/general-settings.md) (configuration settings for Kibana that will go into `kibana.yml`) * [Kibana Secure Settings](k8s-kibana-secure-settings.md) A policy can be applied to one or more Elasticsearch clusters or Kibana instances in any namespace managed by the ECK operator. Configuration policy settings applied by the ECK operator are immutable through the Elasticsearch REST API. It is currently not allowed to configure an Elasticsearch cluster or Kibana instance with more than one policy. diff --git a/deploy-manage/deploy/cloud-on-k8s/elasticsearch-deployment-quickstart.md b/deploy-manage/deploy/cloud-on-k8s/elasticsearch-deployment-quickstart.md index 4a7ac2b47..22a88db27 100644 --- a/deploy-manage/deploy/cloud-on-k8s/elasticsearch-deployment-quickstart.md +++ b/deploy-manage/deploy/cloud-on-k8s/elasticsearch-deployment-quickstart.md @@ -108,7 +108,7 @@ NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE quickstart-es-http ClusterIP 10.15.251.145 9200/TCP 34m ``` -In order to make requests to the [{{es}} API](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/index.md): +In order to make requests to the [{{es}} API](elasticsearch://reference/elasticsearch/rest-apis/index.md): 1. Get the credentials. diff --git a/deploy-manage/deploy/cloud-on-k8s/fleet-managed-elastic-agent.md b/deploy-manage/deploy/cloud-on-k8s/fleet-managed-elastic-agent.md index e12036cc8..2f018e793 100644 --- a/deploy-manage/deploy/cloud-on-k8s/fleet-managed-elastic-agent.md +++ b/deploy-manage/deploy/cloud-on-k8s/fleet-managed-elastic-agent.md @@ -8,7 +8,7 @@ mapped_pages: # Fleet-managed Elastic Agent [k8s-elastic-agent-fleet] -This section describes how to configure and deploy {{agent}} in [{{fleet}}-managed](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/install-elastic-agents.md) mode with ECK. Check the [Standalone section](standalone-elastic-agent.md) if you want to run {{agent}} in the [standalone mode](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/install-standalone-elastic-agent.md). +This section describes how to configure and deploy {{agent}} in [{{fleet}}-managed](/reference/ingestion-tools/fleet/install-elastic-agents.md) mode with ECK. Check the [Standalone section](standalone-elastic-agent.md) if you want to run {{agent}} in the [standalone mode](/reference/ingestion-tools/fleet/install-standalone-elastic-agent.md). * [Quickstart](quickstart-fleet.md) * [Configuration](configuration-fleet.md) diff --git a/deploy-manage/deploy/cloud-on-k8s/k8s-kibana-advanced-configuration.md b/deploy-manage/deploy/cloud-on-k8s/k8s-kibana-advanced-configuration.md index 89ad871f1..31c4e89b3 100644 --- a/deploy-manage/deploy/cloud-on-k8s/k8s-kibana-advanced-configuration.md +++ b/deploy-manage/deploy/cloud-on-k8s/k8s-kibana-advanced-configuration.md @@ -59,7 +59,7 @@ Check [Set compute resources for Kibana, Elastic Maps Server, APM Server and Log You can add your own {{kib}} settings to the `spec.config` section. -The following example demonstrates how to set the [`elasticsearch.requestHeadersWhitelist`](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md#elasticsearch-requestHeadersWhitelist) configuration option. +The following example demonstrates how to set the [`elasticsearch.requestHeadersWhitelist`](kibana://reference/configuration-reference/general-settings.md#elasticsearch-requestheaderswhitelist) configuration option. ```yaml apiVersion: kibana.k8s.elastic.co/v1 @@ -81,8 +81,8 @@ spec: To deploy more than one instance of {{kib}}, the instances must share a matching set of encryption keys. The following keys are automatically generated by the operator: -* [`xpack.security.encryptionKey`](asciidocalypse://docs/kibana/docs/reference/configuration-reference/security-settings.md#xpack-security-encryptionKey) -* [`xpack.reporting.encryptionKey`](asciidocalypse://docs/kibana/docs/reference/configuration-reference/reporting-settings.md#encryption-keys) +* [`xpack.security.encryptionKey`](kibana://reference/configuration-reference/security-settings.md#xpack-security-encryptionkey) +* [`xpack.reporting.encryptionKey`](kibana://reference/configuration-reference/reporting-settings.md#encryption-keys) * [`xpack.encryptedSavedObjects.encryptionKey`](/deploy-manage/security/secure-saved-objects.md) ::::{tip} diff --git a/deploy-manage/deploy/cloud-on-k8s/k8s-openshift-agent.md b/deploy-manage/deploy/cloud-on-k8s/k8s-openshift-agent.md index 52ac57e07..055bbe427 100644 --- a/deploy-manage/deploy/cloud-on-k8s/k8s-openshift-agent.md +++ b/deploy-manage/deploy/cloud-on-k8s/k8s-openshift-agent.md @@ -8,7 +8,7 @@ mapped_pages: # Grant host access permission to Elastic Agent [k8s-openshift-agent] -Deploying Elastic Agent on Openshift may require additional permissions depending on the type of [integration](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/index.md) Elastic Agent is supposed to run. In any case, Elastic Agent uses a [hostPath](https://kubernetes.io/docs/concepts/storage/volumes/#hostpath) volume as its data directory on OpenShift to maintain a stable identity. Therefore, the Service Account used for Elastic Agent needs permissions to use hostPath volumes. +Deploying Elastic Agent on Openshift may require additional permissions depending on the type of [integration](/reference/ingestion-tools/fleet/index.md) Elastic Agent is supposed to run. In any case, Elastic Agent uses a [hostPath](https://kubernetes.io/docs/concepts/storage/volumes/#hostpath) volume as its data directory on OpenShift to maintain a stable identity. Therefore, the Service Account used for Elastic Agent needs permissions to use hostPath volumes. The following example assumes that Elastic Agent is deployed in the Namespace `elastic` with the ServiceAccount `elastic-agent`. You can replace these values according to your environment. diff --git a/deploy-manage/deploy/cloud-on-k8s/logstash-plugins.md b/deploy-manage/deploy/cloud-on-k8s/logstash-plugins.md index 0fce2f887..75bb1b963 100644 --- a/deploy-manage/deploy/cloud-on-k8s/logstash-plugins.md +++ b/deploy-manage/deploy/cloud-on-k8s/logstash-plugins.md @@ -8,7 +8,7 @@ mapped_pages: # Logstash plugins [k8s-logstash-plugins] -The power of {{ls}} is in the plugins--[inputs](asciidocalypse://docs/logstash/docs/reference/input-plugins.md), [outputs](asciidocalypse://docs/logstash/docs/reference/output-plugins.md), [filters](asciidocalypse://docs/logstash/docs/reference/filter-plugins.md), and [codecs](asciidocalypse://docs/logstash/docs/reference/codec-plugins.md). +The power of {{ls}} is in the plugins--[inputs](logstash://reference/input-plugins.md), [outputs](logstash://reference/output-plugins.md), [filters](logstash://reference/filter-plugins.md), and [codecs](logstash://reference/codec-plugins.md). In {{ls}} on ECK, you can use the same plugins that you use for other {{ls}} instances—​including Elastic-supported, community-supported, and custom plugins. However, you may have other factors to consider, such as how you configure your {{k8s}} resources, how you specify additional resources, and how you scale your {{ls}} installation. @@ -90,7 +90,7 @@ spec: **Static read-only files** -Some plugins require or allow access to small static read-only files. You can use these for a variety of reasons. Examples include adding custom `grok` patterns for [`logstash-filter-grok`](asciidocalypse://docs/logstash/docs/reference/plugins-filters-grok.md) to use for lookup, source code for [`logstash-filter-ruby`], a dictionary for [`logstash-filter-translate`](asciidocalypse://docs/logstash/docs/reference/plugins-filters-translate.md) or the location of a SQL statement for [`logstash-input-jdbc`](asciidocalypse://docs/logstash/docs/reference/plugins-inputs-jdbc.md). Make these files available to the {{ls}} resource in your manifest. +Some plugins require or allow access to small static read-only files. You can use these for a variety of reasons. Examples include adding custom `grok` patterns for [`logstash-filter-grok`](logstash://reference/plugins-filters-grok.md) to use for lookup, source code for [`logstash-filter-ruby`], a dictionary for [`logstash-filter-translate`](logstash://reference/plugins-filters-translate.md) or the location of a SQL statement for [`logstash-input-jdbc`](logstash://reference/plugins-inputs-jdbc.md). Make these files available to the {{ls}} resource in your manifest. ::::{tip} In the plugin documentation, these plugin settings are typically identified by `path` or an `array` of `paths`. @@ -99,7 +99,7 @@ In the plugin documentation, these plugin settings are typically identified by ` To use these in your manifest, create a ConfigMap or Secret representing the asset, a Volume in your `podTemplate.spec` containing the ConfigMap or Secret, and mount that Volume with a VolumeMount in your `podTemplateSpec.container` section of your {{ls}} resource. -This example illustrates configuring a ConfigMap from a ruby source file, and including it in a [`logstash-filter-ruby`](asciidocalypse://docs/logstash/docs/reference/plugins-filters-ruby.md) plugin. +This example illustrates configuring a ConfigMap from a ruby source file, and including it in a [`logstash-filter-ruby`](logstash://reference/plugins-filters-ruby.md) plugin. First, create the ConfigMap. @@ -143,7 +143,7 @@ spec: ### Larger read-only assets (1 MiB+) [k8s-logstash-working-with-plugins-large-ro] -Some plugins require or allow access to static read-only files that exceed the 1 MiB (mebibyte) limit imposed by ConfigMap and Secret. For example, you may need JAR files to load drivers when using a JDBC or JMS plugin, or a large [`logstash-filter-translate`](asciidocalypse://docs/logstash/docs/reference/plugins-filters-translate.md) dictionary. +Some plugins require or allow access to static read-only files that exceed the 1 MiB (mebibyte) limit imposed by ConfigMap and Secret. For example, you may need JAR files to load drivers when using a JDBC or JMS plugin, or a large [`logstash-filter-translate`](logstash://reference/plugins-filters-translate.md) dictionary. You can add files using: @@ -239,7 +239,7 @@ After you build and deploy the custom image, include it in the {{ls}} manifest. ### Writable storage [k8s-logstash-working-with-plugins-writable] -Some {{ls}} plugins need access to writable storage. This could be for checkpointing to keep track of events already processed, a place to temporarily write events before sending a batch of events, or just to actually write events to disk in the case of [`logstash-output-file`](asciidocalypse://docs/logstash/docs/reference/plugins-outputs-file.md). +Some {{ls}} plugins need access to writable storage. This could be for checkpointing to keep track of events already processed, a place to temporarily write events before sending a batch of events, or just to actually write events to disk in the case of [`logstash-output-file`](logstash://reference/plugins-outputs-file.md). {{ls}} on ECK by default supplies a small 1.5 GiB (gibibyte) default persistent volume to each pod. This volume is called `logstash-data` and is located at `/usr/logstash/data`, and is typically the default location for most plugin use cases. This volume is stable across restarts of {{ls}} pods and is suitable for many use cases. @@ -333,7 +333,7 @@ spec: ::::{admonition} Horizontal scaling for {{ls}} plugins * Not all {{ls}} deployments can be scaled horizontally by increasing the number of {{ls}} Pods defined in the {{ls}} resource. Depending on the types of plugins in a {{ls}} installation, increasing the number of pods may cause data duplication, data loss, incorrect data, or may waste resources with pods unable to be utilized correctly. -* The ability of a {{ls}} installation to scale horizontally is bound by its most restrictive plugin(s). Even if all pipelines are using [`logstash-input-elastic_agent`](asciidocalypse://docs/logstash/docs/reference/plugins-inputs-elastic_agent.md) or [`logstash-input-beats`](asciidocalypse://docs/logstash/docs/reference/plugins-inputs-beats.md) which should enable full horizontal scaling, introducing a more restrictive input or filter plugin forces the restrictions for pod scaling associated with that plugin. +* The ability of a {{ls}} installation to scale horizontally is bound by its most restrictive plugin(s). Even if all pipelines are using [`logstash-input-elastic_agent`](logstash://reference/plugins-inputs-elastic_agent.md) or [`logstash-input-beats`](logstash://reference/plugins-inputs-beats.md) which should enable full horizontal scaling, introducing a more restrictive input or filter plugin forces the restrictions for pod scaling associated with that plugin. :::: @@ -345,12 +345,12 @@ spec: * They **must** specify `pipeline.workers=1` for any pipelines that use them. * The number of pods cannot be scaled above 1. -Examples of aggregating filters include [`logstash-filter-aggregate`](asciidocalypse://docs/logstash/docs/reference/plugins-filters-aggregate.md), [`logstash-filter-csv`](asciidocalypse://docs/logstash/docs/reference/plugins-filters-csv.md) when `autodetect_column_names` set to `true`, and any [`logstash-filter-ruby`](asciidocalypse://docs/logstash/docs/reference/plugins-filters-ruby.md) implementations that perform aggregations. +Examples of aggregating filters include [`logstash-filter-aggregate`](logstash://reference/plugins-filters-aggregate.md), [`logstash-filter-csv`](logstash://reference/plugins-filters-csv.md) when `autodetect_column_names` set to `true`, and any [`logstash-filter-ruby`](logstash://reference/plugins-filters-ruby.md) implementations that perform aggregations. ### Input plugins: events pushed to {{ls}} [k8s-logstash-inputs-data-pushed] -{{ls}} installations with inputs that enable {{ls}} to receive data should be able to scale freely and have load spread across them horizontally. These plugins include [`logstash-input-beats`](asciidocalypse://docs/logstash/docs/reference/plugins-inputs-beats.md), [`logstash-input-elastic_agent`](asciidocalypse://docs/logstash/docs/reference/plugins-inputs-elastic_agent.md), [`logstash-input-tcp`](asciidocalypse://docs/logstash/docs/reference/plugins-inputs-tcp.md), and [`logstash-input-http`](asciidocalypse://docs/logstash/docs/reference/plugins-inputs-http.md). +{{ls}} installations with inputs that enable {{ls}} to receive data should be able to scale freely and have load spread across them horizontally. These plugins include [`logstash-input-beats`](logstash://reference/plugins-inputs-beats.md), [`logstash-input-elastic_agent`](logstash://reference/plugins-inputs-elastic_agent.md), [`logstash-input-tcp`](logstash://reference/plugins-inputs-tcp.md), and [`logstash-input-http`](logstash://reference/plugins-inputs-http.md). ### Input plugins: {{ls}} maintains state [k8s-logstash-inputs-local-checkpoints] @@ -361,16 +361,16 @@ Note that plugins that retrieve data from external sources, and require some lev Input plugins that include configuration settings such as `sincedb`, `checkpoint` or `sql_last_run_metadata` may fall into this category. -Examples of these plugins include [`logstash-input-jdbc`](asciidocalypse://docs/logstash/docs/reference/plugins-inputs-jdbc.md) (which has no automatic way to split queries across {{ls}} instances), [`logstash-input-s3`](asciidocalypse://docs/logstash/docs/reference/plugins-inputs-s3.md) (which has no way to split which buckets to read across {{ls}} instances), or [`logstash-input-file`](asciidocalypse://docs/logstash/docs/reference/plugins-inputs-file.md). +Examples of these plugins include [`logstash-input-jdbc`](logstash://reference/plugins-inputs-jdbc.md) (which has no automatic way to split queries across {{ls}} instances), [`logstash-input-s3`](logstash://reference/plugins-inputs-s3.md) (which has no way to split which buckets to read across {{ls}} instances), or [`logstash-input-file`](logstash://reference/plugins-inputs-file.md). ### Input plugins: external source stores state [k8s-logstash-inputs-external-state] {{ls}} installations that use input plugins that retrieve data from an external source, and **rely on the external source to store state** can scale based on the parameters of the external source. -For example, a {{ls}} installation that uses a [`logstash-input-kafka`](asciidocalypse://docs/logstash/docs/reference/plugins-inputs-kafka.md) plugin to retrieve data can scale the number of pods up to the number of partitions used, as a partition can have at most one consumer belonging to the same consumer group. Any pods created beyond that threshold cannot be scheduled to receive data. +For example, a {{ls}} installation that uses a [`logstash-input-kafka`](logstash://reference/plugins-inputs-kafka.md) plugin to retrieve data can scale the number of pods up to the number of partitions used, as a partition can have at most one consumer belonging to the same consumer group. Any pods created beyond that threshold cannot be scheduled to receive data. -Examples of these plugins include [`logstash-input-kafka`](asciidocalypse://docs/logstash/docs/reference/plugins-inputs-kafka.md), [`logstash-input-azure_event_hubs`](asciidocalypse://docs/logstash/docs/reference/plugins-inputs-azure_event_hubs.md), and [`logstash-input-kinesis`](asciidocalypse://docs/logstash/docs/reference/plugins-inputs-kinesis.md). +Examples of these plugins include [`logstash-input-kafka`](logstash://reference/plugins-inputs-kafka.md), [`logstash-input-azure_event_hubs`](logstash://reference/plugins-inputs-azure_event_hubs.md), and [`logstash-input-kinesis`](logstash://reference/plugins-inputs-kinesis.md). @@ -390,12 +390,12 @@ Use these guidelines *in addition* to the general guidelines provided in [Scalin ### {{ls}} integration plugin [k8s-logstash-plugin-considerations-ls-integration] -When your pipeline uses the [`Logstash integration`](asciidocalypse://docs/logstash/docs/reference/plugins-integrations-logstash.md) plugin, add `keepalive=>false` to the [logstash-output](asciidocalypse://docs/logstash/docs/reference/plugins-outputs-logstash.md) definition to ensure that load balancing works correctly rather than keeping affinity to the same pod. +When your pipeline uses the [`Logstash integration`](logstash://reference/plugins-integrations-logstash.md) plugin, add `keepalive=>false` to the [logstash-output](logstash://reference/plugins-outputs-logstash.md) definition to ensure that load balancing works correctly rather than keeping affinity to the same pod. ### Elasticsearch output plugin [k8s-logstash-plugin-considerations-es-output] -The [`elasticsearch output`](asciidocalypse://docs/logstash/docs/reference/plugins-outputs-elasticsearch.md) plugin requires certain roles to be configured in order to enable {{ls}} to communicate with {{es}}. +The [`elasticsearch output`](logstash://reference/plugins-outputs-elasticsearch.md) plugin requires certain roles to be configured in order to enable {{ls}} to communicate with {{es}}. You can customize roles in {{es}}. Check out [creating custom roles](../../users-roles/cluster-or-deployment-auth/native.md) @@ -419,7 +419,7 @@ stringData: ### Elastic_integration filter plugin [k8s-logstash-plugin-considerations-integration-filter] -The [`elastic_integration filter`](asciidocalypse://docs/logstash/docs/reference/plugins-filters-elastic_integration.md) plugin allows the use of [`ElasticsearchRef`](configuration-logstash.md#k8s-logstash-esref) and environment variables. +The [`elastic_integration filter`](logstash://reference/plugins-filters-elastic_integration.md) plugin allows the use of [`ElasticsearchRef`](configuration-logstash.md#k8s-logstash-esref) and environment variables. ```json elastic_integration { @@ -448,7 +448,7 @@ stringData: ### Elastic Agent input and Beats input plugins [k8s-logstash-plugin-considerations-agent-beats] -When you use the [Elastic Agent input](asciidocalypse://docs/logstash/docs/reference/plugins-inputs-elastic_agent.md) or the [Beats input](asciidocalypse://docs/logstash/docs/reference/plugins-inputs-beats.md), set the [`ttl`](asciidocalypse://docs/beats/docs/reference/filebeat/logstash-output.md#_ttl) value on the Agent or Beat to ensure that load is distributed appropriately. +When you use the [Elastic Agent input](logstash://reference/plugins-inputs-elastic_agent.md) or the [Beats input](logstash://reference/plugins-inputs-beats.md), set the [`ttl`](asciidocalypse://docs/beats/docs/reference/filebeat/logstash-output.md#_ttl) value on the Agent or Beat to ensure that load is distributed appropriately. @@ -456,7 +456,7 @@ When you use the [Elastic Agent input](asciidocalypse://docs/logstash/docs/refer If you need plugins in addition to those included in the standard {{ls}} distribution, you can add them. Create a custom Docker image that includes the installed plugins, using the `bin/logstash-plugin install` utility to add more plugins to the image so that they can be used by {{ls}} pods. -This sample Dockerfile installs the [`logstash-filter-tld`](asciidocalypse://docs/logstash/docs/reference/plugins-filters-tld.md) plugin to the official {{ls}} Docker image: +This sample Dockerfile installs the [`logstash-filter-tld`](logstash://reference/plugins-filters-tld.md) plugin to the official {{ls}} Docker image: ```shell FROM docker.elastic.co/logstash/logstash:8.16.1 diff --git a/deploy-manage/deploy/cloud-on-k8s/nodes-orchestration.md b/deploy-manage/deploy/cloud-on-k8s/nodes-orchestration.md index ac99ae6c5..a24aad4d0 100644 --- a/deploy-manage/deploy/cloud-on-k8s/nodes-orchestration.md +++ b/deploy-manage/deploy/cloud-on-k8s/nodes-orchestration.md @@ -177,7 +177,7 @@ Advanced users may force an upgrade by manually deleting Pods themselves. The de Operations that reduce the number of nodes in the cluster cannot make progress without user intervention, if the Elasticsearch index replica settings are incompatible with the intended downscale. Specifically, if the Elasticsearch index settings demand a higher number of shard copies than data nodes in the cluster after the downscale operation, ECK cannot migrate the data away from the node about to be removed. You can address this in the following ways: * Adjust the Elasticsearch [index settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings) to a number of replicas that allow the desired node removal. -* Use [`auto_expand_replicas`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md) to automatically adjust the replicas to the number of data nodes in the cluster. +* Use [`auto_expand_replicas`](elasticsearch://reference/elasticsearch/index-settings/index-modules.md) to automatically adjust the replicas to the number of data nodes in the cluster. ## Advanced control during rolling upgrades [k8s-advanced-upgrade-control] diff --git a/deploy-manage/deploy/cloud-on-k8s/quickstart-fleet.md b/deploy-manage/deploy/cloud-on-k8s/quickstart-fleet.md index e79d53a8c..c9bd8b614 100644 --- a/deploy-manage/deploy/cloud-on-k8s/quickstart-fleet.md +++ b/deploy-manage/deploy/cloud-on-k8s/quickstart-fleet.md @@ -214,5 +214,5 @@ ECK automatically configures secure connections between all components. {{fleet} kubectl logs -f elastic-agent-quickstart-agent-xbcxr ``` -4. Configure the policy used by {{agents}}. Check [{{agent}} policies](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/agent-policy.md) for more details. +4. Configure the policy used by {{agents}}. Check [{{agent}} policies](/reference/ingestion-tools/fleet/agent-policy.md) for more details. diff --git a/deploy-manage/deploy/cloud-on-k8s/readiness-probe.md b/deploy-manage/deploy/cloud-on-k8s/readiness-probe.md index 8b39c91c2..58a25f00c 100644 --- a/deploy-manage/deploy/cloud-on-k8s/readiness-probe.md +++ b/deploy-manage/deploy/cloud-on-k8s/readiness-probe.md @@ -46,6 +46,6 @@ Note that this requires restarting the Pods. % this feature might have disappeared, we will need to investigate this a bit more, as the link below doesn't work anymore but it does for 8.15 for example. -We do not recommend overriding the default readiness probe on Elasticsearch 8.2.0 and later. ECK configures a socket based readiness probe using the Elasticsearch [readiness port feature](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/jvm-settings.md#readiness-tcp-port) which is not influenced by the load on the Elasticsearch cluster. +We do not recommend overriding the default readiness probe on Elasticsearch 8.2.0 and later. ECK configures a socket based readiness probe using the Elasticsearch [readiness port feature](elasticsearch://reference/elasticsearch/jvm-settings.md#readiness-tcp-port) which is not influenced by the load on the Elasticsearch cluster. diff --git a/deploy-manage/deploy/cloud-on-k8s/requests-routing-to-elasticsearch-nodes.md b/deploy-manage/deploy/cloud-on-k8s/requests-routing-to-elasticsearch-nodes.md index 7626121c8..903b665d2 100644 --- a/deploy-manage/deploy/cloud-on-k8s/requests-routing-to-elasticsearch-nodes.md +++ b/deploy-manage/deploy/cloud-on-k8s/requests-routing-to-elasticsearch-nodes.md @@ -8,7 +8,7 @@ mapped_pages: # Requests routing to Elasticsearch nodes [k8s-traffic-splitting] -The default Kubernetes service created by ECK, named `-es-http`, is configured to include all the Elasticsearch nodes in that cluster. This configuration is good to get started and is adequate for most use cases. However, if you are operating an Elasticsearch cluster with [different node types](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md) and want control over which nodes handle which types of traffic, you should create additional Kubernetes services yourself. +The default Kubernetes service created by ECK, named `-es-http`, is configured to include all the Elasticsearch nodes in that cluster. This configuration is good to get started and is adequate for most use cases. However, if you are operating an Elasticsearch cluster with [different node types](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md) and want control over which nodes handle which types of traffic, you should create additional Kubernetes services yourself. As an alternative, you can use features provided by third-party software such as service meshes and ingress controllers to achieve more advanced traffic management configurations. Check the [recipes directory](https://github.com/elastic/cloud-on-k8s/tree/2.16/config/recipes) in the ECK source repository for a few examples. diff --git a/deploy-manage/deploy/cloud-on-k8s/securing-logstash-api.md b/deploy-manage/deploy/cloud-on-k8s/securing-logstash-api.md index 371177c3a..c50c29ace 100644 --- a/deploy-manage/deploy/cloud-on-k8s/securing-logstash-api.md +++ b/deploy-manage/deploy/cloud-on-k8s/securing-logstash-api.md @@ -45,7 +45,7 @@ spec: 1. Store the username and password in a Secret. 2. Map the username and password to the environment variables of the Pod. -3. At Logstash startup, `${API_USERNAME}` and `${API_PASSWORD}` are replaced by the value of environment variables. Check [using environment variables](asciidocalypse://docs/logstash/docs/reference/environment-variables.md) for more details. +3. At Logstash startup, `${API_USERNAME}` and `${API_PASSWORD}` are replaced by the value of environment variables. Check [using environment variables](logstash://reference/environment-variables.md) for more details. An alternative is to set up [keystore](advanced-configuration-logstash.md#k8s-logstash-keystore) to resolve `${API_USERNAME}` and `${API_PASSWORD}` diff --git a/deploy-manage/deploy/cloud-on-k8s/standalone-elastic-agent.md b/deploy-manage/deploy/cloud-on-k8s/standalone-elastic-agent.md index 00dbeb3e1..7b1bc3251 100644 --- a/deploy-manage/deploy/cloud-on-k8s/standalone-elastic-agent.md +++ b/deploy-manage/deploy/cloud-on-k8s/standalone-elastic-agent.md @@ -8,7 +8,7 @@ mapped_pages: # Standalone Elastic Agent [k8s-elastic-agent] -This section describes how to configure and deploy Elastic Agent in [standalone mode](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/install-standalone-elastic-agent.md) with ECK. Check the [Fleet section](fleet-managed-elastic-agent.md) if you want to manage your Elastic Agents with [Fleet](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/install-elastic-agents.md). +This section describes how to configure and deploy Elastic Agent in [standalone mode](/reference/ingestion-tools/fleet/install-standalone-elastic-agent.md) with ECK. Check the [Fleet section](fleet-managed-elastic-agent.md) if you want to manage your Elastic Agents with [Fleet](/reference/ingestion-tools/fleet/install-elastic-agents.md). * [Quickstart](quickstart-standalone.md) * [Configuration](configuration-standalone.md) diff --git a/deploy-manage/deploy/cloud-on-k8s/transport-settings.md b/deploy-manage/deploy/cloud-on-k8s/transport-settings.md index e0887773e..8197b55c7 100644 --- a/deploy-manage/deploy/cloud-on-k8s/transport-settings.md +++ b/deploy-manage/deploy/cloud-on-k8s/transport-settings.md @@ -8,7 +8,7 @@ mapped_pages: # Transport settings [k8s-transport-settings] -The transport module in Elasticsearch is used for internal communication between nodes within the cluster as well as communication between remote clusters. Check the [Elasticsearch documentation](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md) for details. For customization options of the HTTP layer, check [Services](accessing-services.md) and [TLS certificates](/deploy-manage/security/secure-http-communications.md). +The transport module in Elasticsearch is used for internal communication between nodes within the cluster as well as communication between remote clusters. Check the [Elasticsearch documentation](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md) for details. For customization options of the HTTP layer, check [Services](accessing-services.md) and [TLS certificates](/deploy-manage/security/secure-http-communications.md). ## Customize the Transport Service [k8s_customize_the_transport_service] diff --git a/deploy-manage/deploy/cloud-on-k8s/virtual-memory.md b/deploy-manage/deploy/cloud-on-k8s/virtual-memory.md index fad49f599..003d163d9 100644 --- a/deploy-manage/deploy/cloud-on-k8s/virtual-memory.md +++ b/deploy-manage/deploy/cloud-on-k8s/virtual-memory.md @@ -14,7 +14,7 @@ The kernel setting `vm.max_map_count=262144` can be set on the host directly, by For more information, check the Elasticsearch documentation on [Virtual memory](/deploy-manage/deploy/self-managed/vm-max-map-count.md). -Optionally, you can select a different type of file system implementation for the storage. For possible options, check the [store module documentation](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/store.md). +Optionally, you can select a different type of file system implementation for the storage. For possible options, check the [store module documentation](elasticsearch://reference/elasticsearch/index-settings/store.md). ```yaml spec: diff --git a/deploy-manage/deploy/elastic-cloud/add-plugins-extensions.md b/deploy-manage/deploy/elastic-cloud/add-plugins-extensions.md index 40655a968..23f677a7a 100644 --- a/deploy-manage/deploy/elastic-cloud/add-plugins-extensions.md +++ b/deploy-manage/deploy/elastic-cloud/add-plugins-extensions.md @@ -16,16 +16,16 @@ Plugins extend the core functionality of {{es}}. There are many suitable plugins * Analysis plugins, to provide analyzers targeted at languages other than English. * Scripting plugins, to provide additional scripting languages. -Plugins can come from different sources: the official ones created or at least maintained by Elastic, community-sourced plugins from other users, and plugins that you provide. Some of the official plugins are always provided with our service, and can be [enabled per deployment](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch-plugins/cloud/ec-adding-elastic-plugins.md). +Plugins can come from different sources: the official ones created or at least maintained by Elastic, community-sourced plugins from other users, and plugins that you provide. Some of the official plugins are always provided with our service, and can be [enabled per deployment](elasticsearch://reference/elasticsearch-plugins/cloud/ec-adding-elastic-plugins.md). There are two ways to add plugins to a hosted deployment in {{ecloud}}: -* [Enable one of the official plugins already available in {{ecloud}}](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch-plugins/cloud/ec-adding-elastic-plugins.md). +* [Enable one of the official plugins already available in {{ecloud}}](elasticsearch://reference/elasticsearch-plugins/cloud/ec-adding-elastic-plugins.md). * [Upload a custom plugin and then enable it per deployment](upload-custom-plugins-bundles.md). -Custom plugins can include the official {{es}} plugins not provided with {{ecloud}}, any of the community-sourced plugins, or [plugins that you write yourself](asciidocalypse://docs/elasticsearch/docs/extend/index.md). Uploading custom plugins is available only to Gold, Platinum, and Enterprise subscriptions. For more information, check [Upload custom plugins and bundles](upload-custom-plugins-bundles.md). +Custom plugins can include the official {{es}} plugins not provided with {{ecloud}}, any of the community-sourced plugins, or [plugins that you write yourself](elasticsearch://extend/index.md). Uploading custom plugins is available only to Gold, Platinum, and Enterprise subscriptions. For more information, check [Upload custom plugins and bundles](upload-custom-plugins-bundles.md). -To learn more about the official and community-sourced plugins, refer to [{{es}} Plugins and Integrations](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch-plugins/index.md). +To learn more about the official and community-sourced plugins, refer to [{{es}} Plugins and Integrations](elasticsearch://reference/elasticsearch-plugins/index.md). For a detailed guide with examples of using the {{ecloud}} API to create, get information about, update, and delete extensions and plugins, check [Managing plugins and extensions through the API](manage-plugins-extensions-through-api.md). diff --git a/deploy-manage/deploy/elastic-cloud/available-stack-versions.md b/deploy-manage/deploy/elastic-cloud/available-stack-versions.md index fa8628954..b1ff431bd 100644 --- a/deploy-manage/deploy/elastic-cloud/available-stack-versions.md +++ b/deploy-manage/deploy/elastic-cloud/available-stack-versions.md @@ -28,7 +28,7 @@ You might sometimes notice additional versions listed in the user interface beyo Whenever a new Elastic Stack version is released, we do our best to provide the new version on our hosted service at the same time. We send you an email and add a notice to the console, recommending an upgrade. You’ll need to decide whether to upgrade to the new version with new features and bug fixes or to stay with a version you know works for you a while longer. -There can be [breaking changes](asciidocalypse://docs/elasticsearch/docs/release-notes/breaking-changes.md) in some new versions of Elasticsearch that break what used to work in older versions. Before upgrading, you’ll want to check if the new version introduces any changes that might affect your applications. A breaking change might be a function that was previously deprecated and that has been removed in the latest version, for example. If you have an application that depends on the removed function, the application will need to be updated to continue working with the new version of Elasticsearch. +There can be [breaking changes](elasticsearch://release-notes/breaking-changes.md) in some new versions of Elasticsearch that break what used to work in older versions. Before upgrading, you’ll want to check if the new version introduces any changes that might affect your applications. A breaking change might be a function that was previously deprecated and that has been removed in the latest version, for example. If you have an application that depends on the removed function, the application will need to be updated to continue working with the new version of Elasticsearch. To learn more about upgrading to newer versions of the Elastic Stack on our hosted service, check [Upgrade Versions](../../upgrade/deployment-or-cluster.md). diff --git a/deploy-manage/deploy/elastic-cloud/azure-native-isv-service.md b/deploy-manage/deploy/elastic-cloud/azure-native-isv-service.md index fd394f1a2..900f32d35 100644 --- a/deploy-manage/deploy/elastic-cloud/azure-native-isv-service.md +++ b/deploy-manage/deploy/elastic-cloud/azure-native-isv-service.md @@ -240,7 +240,7 @@ $$$azure-integration-cli-api$$$What other methods are available to deploy {{es}} * The {{ecloud}} [console](https://cloud.elastic.co?page=docs&placement=docs-body) * The {{ecloud}} [REST API](asciidocalypse://docs/cloud/docs/reference/cloud-hosted/ec-api-restful.md) - * The {{ecloud}} [command line tool](asciidocalypse://docs/ecctl/docs/reference/index.md) + * The {{ecloud}} [command line tool](ecctl://reference/index.md) * The {{ecloud}} [Terraform provider](https://registry.terraform.io/providers/elastic/ec/latest/docs) Note that when you use any of the {{ecloud}} methods, the {{es}} deployment will not be available in Azure. @@ -430,7 +430,7 @@ $$$azure-integration-vm-extensions$$$How can I monitor my Azure virtual machines **Managing the Elastic Agent VM extension** - Once installed on the virtual machine, you can manage Elastic Agent either from Fleet or locally on the host where it’s installed. We recommend managing the VM extension through Fleet, because it makes handling and upgrading the agents considerably easier. For more information on the Elastic Agent, check [Manage your Elastic Agents](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/install-elastic-agents.md). + Once installed on the virtual machine, you can manage Elastic Agent either from Fleet or locally on the host where it’s installed. We recommend managing the VM extension through Fleet, because it makes handling and upgrading the agents considerably easier. For more information on the Elastic Agent, check [Manage your Elastic Agents](/reference/ingestion-tools/fleet/install-elastic-agents.md). **Operating system compatibility matrix** diff --git a/deploy-manage/deploy/elastic-cloud/differences-from-other-elasticsearch-offerings.md b/deploy-manage/deploy/elastic-cloud/differences-from-other-elasticsearch-offerings.md index d4b8d56f7..9ec7081d2 100644 --- a/deploy-manage/deploy/elastic-cloud/differences-from-other-elasticsearch-offerings.md +++ b/deploy-manage/deploy/elastic-cloud/differences-from-other-elasticsearch-offerings.md @@ -42,7 +42,7 @@ To ensure optimal performance, follow these recommendations for sizing individua For large datasets that exceed the recommended maximum size for a single index, consider splitting your data across smaller indices and using an alias to search them collectively. -These recommendations do not apply to indices using better binary quantization (BBQ). Refer to [vector quantization](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/dense-vector.md#dense-vector-quantization) in the core {{es}} docs for more information. +These recommendations do not apply to indices using better binary quantization (BBQ). Refer to [vector quantization](elasticsearch://reference/elasticsearch/mapping-reference/dense-vector.md#dense-vector-quantization) in the core {{es}} docs for more information. ## API availability [elasticsearch-differences-serverless-apis-availability] @@ -94,7 +94,7 @@ When attempting to use an unavailable API, you’ll receive a clear error messag ## Settings availability [elasticsearch-differences-serverless-settings-availability] -In {{es-serverless}}, you can only configure [index-level settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md). Cluster-level settings and node-level settings are not required by end users and the `elasticsearch.yml` file is fully managed by Elastic. +In {{es-serverless}}, you can only configure [index-level settings](elasticsearch://reference/elasticsearch/index-settings/index.md). Cluster-level settings and node-level settings are not required by end users and the `elasticsearch.yml` file is fully managed by Elastic. Available settings : **Index-level settings**: Settings that control how {{es}} documents are processed, stored, and searched are available to end users. These include: @@ -153,7 +153,7 @@ The following features are planned for future support in all {{serverless-full}} The following features are not available in {{es-serverless}} and are not planned for future support: * [Custom plugins and bundles](/deploy-manage/deploy/elastic-cloud/upload-custom-plugins-bundles.md) -* [{{es}} for Apache Hadoop](asciidocalypse://docs/elasticsearch-hadoop/docs/reference/elasticsearch-for-apache-hadoop.md) -* [Scripted metric aggregations](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-scripted-metric-aggregation.md) +* [{{es}} for Apache Hadoop](elasticsearch-hadoop://reference/index.md) +* [Scripted metric aggregations](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-scripted-metric-aggregation.md) * Managed web crawler: You can use the [self-managed web crawler](https://github.com/elastic/crawler) instead. -* Managed Search connectors: You can use [self-managed Search connectors](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/search-connectors/self-managed-connectors.md) instead. \ No newline at end of file +* Managed Search connectors: You can use [self-managed Search connectors](elasticsearch://reference/ingestion-tools/search-connectors/self-managed-connectors.md) instead. \ No newline at end of file diff --git a/deploy-manage/deploy/elastic-cloud/ech-restrictions.md b/deploy-manage/deploy/elastic-cloud/ech-restrictions.md index 9dc8ec9e4..66d09f2e5 100644 --- a/deploy-manage/deploy/elastic-cloud/ech-restrictions.md +++ b/deploy-manage/deploy/elastic-cloud/ech-restrictions.md @@ -70,7 +70,7 @@ Currently you can’t use SSO to login directly from {{ecloud}} into Kibana endp ## Kibana [ech-restrictions-kibana] * The maximum size of a single {{kib}} instance is 8GB. This means, {{kib}} instances can be scaled up to 8GB before they are scaled out. For example, when creating a deployment with a {{kib}} instance of size 16GB, then 2x8GB instances are created. If you face performance issues with {{kib}} PNG or PDF reports, the recommendations are to create multiple, smaller dashboards to export the data, or to use a third party browser extension for exporting the dashboard in the format you need. -* Running an external Kibana in parallel to Elasticsearch Add-On for Heroku’s Kibana instances may cause errors, for example [`Unable to decrypt attribute`](../../../explore-analyze/alerts-cases/alerts/alerting-common-issues.md#rule-cannot-decrypt-api-key), due to a mismatched [`xpack.encryptedSavedObjects.encryptionKey`](asciidocalypse://docs/kibana/docs/reference/configuration-reference/security-settings.md#security-encrypted-saved-objects-settings) as Elasticsearch Add-On for Heroku does not [allow users to set](edit-stack-settings.md) nor expose this value. While workarounds are possible, this is not officially supported nor generally recommended. +* Running an external Kibana in parallel to Elasticsearch Add-On for Heroku’s Kibana instances may cause errors, for example [`Unable to decrypt attribute`](../../../explore-analyze/alerts-cases/alerts/alerting-common-issues.md#rule-cannot-decrypt-api-key), due to a mismatched [`xpack.encryptedSavedObjects.encryptionKey`](kibana://reference/configuration-reference/security-settings.md#security-encrypted-saved-objects-settings) as Elasticsearch Add-On for Heroku does not [allow users to set](edit-stack-settings.md) nor expose this value. While workarounds are possible, this is not officially supported nor generally recommended. ## APM Agent central configuration with PrivateLink or traffic filters [ech-restrictions-apm-traffic-filters] @@ -116,7 +116,7 @@ There are situations where you may need or want to move your installed {{agents} In {{ecloud}}, you can migrate your {{agents}} by taking a snapshot of your source deployment, and restoring it on a target deployment. -To make a seamless migration, after restoring from a snapshot there are some additional steps required, such as updating settings and resetting the agent policy. Check [Migrate Elastic Agents](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/migrate-elastic-agent.md) for details. +To make a seamless migration, after restoring from a snapshot there are some additional steps required, such as updating settings and resetting the agent policy. Check [Migrate Elastic Agents](/reference/ingestion-tools/fleet/migrate-elastic-agent.md) for details. ## Regions and Availability Zones [ech-regions-and-availability-zone] diff --git a/deploy-manage/deploy/elastic-cloud/ech-version-policy.md b/deploy-manage/deploy/elastic-cloud/ech-version-policy.md index ff660ca6d..79b75d280 100644 --- a/deploy-manage/deploy/elastic-cloud/ech-version-policy.md +++ b/deploy-manage/deploy/elastic-cloud/ech-version-policy.md @@ -25,7 +25,7 @@ You might sometimes notice additional versions listed in the user interface beyo Whenever a new Elastic Stack version is released, we do our best to provide the new version on our hosted service at the same time. We send you an email and add a notice to the console, recommending an upgrade. You’ll need to decide whether to upgrade to the new version with new features and bug fixes or to stay with a version you know works for you a while longer. -There can be [breaking changes](asciidocalypse://docs/elasticsearch/docs/release-notes/breaking-changes.md) in some new versions of Elasticsearch that break what used to work in older versions. Before upgrading, you’ll want to check if the new version introduces any changes that might affect your applications. A breaking change might be a function that was previously deprecated and that has been removed in the latest version, for example. If you have an application that depends on the removed function, the application will need to be updated to continue working with the new version of Elasticsearch. +There can be [breaking changes](elasticsearch://release-notes/breaking-changes.md) in some new versions of Elasticsearch that break what used to work in older versions. Before upgrading, you’ll want to check if the new version introduces any changes that might affect your applications. A breaking change might be a function that was previously deprecated and that has been removed in the latest version, for example. If you have an application that depends on the removed function, the application will need to be updated to continue working with the new version of Elasticsearch. To learn more about upgrading to newer versions of the Elastic Stack on our hosted service, check [Upgrade Versions](../../upgrade/deployment-or-cluster.md). diff --git a/deploy-manage/deploy/elastic-cloud/ech-whats-new.md b/deploy-manage/deploy/elastic-cloud/ech-whats-new.md index eaa7c4049..1973c1195 100644 --- a/deploy-manage/deploy/elastic-cloud/ech-whats-new.md +++ b/deploy-manage/deploy/elastic-cloud/ech-whats-new.md @@ -15,14 +15,14 @@ Check the Release Notes to get the recent updates for each product. Elasticsearch -* [Elasticsearch 8.x Release Notes](asciidocalypse://docs/elasticsearch/docs/release-notes/index.md) +* [Elasticsearch 8.x Release Notes](elasticsearch://release-notes/index.md) * [Elasticsearch 7.x Release Notes](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/es-release-notes.html) * [Elasticsearch 6.x Release Notes](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/es-release-notes.html) * [Elasticsearch 5.x Release Notes](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/es-release-notes.html) Kibana -* [Kibana 8.x Release Notes](asciidocalypse://docs/kibana/docs/release-notes/kibana.md) +* [Kibana 8.x Release Notes](kibana://release-notes/index.md) * [Kibana 7.x Release Notes](https://www.elastic.co/guide/en/kibana/7.17/release-notes.html) * [Kibana 6.x Release Notes](https://www.elastic.co/guide/en/kibana/6.8/release-notes.html) * [Kibana 5.x Release Notes](https://www.elastic.co/guide/en/kibana/5.6/release-notes.html) diff --git a/deploy-manage/deploy/elastic-cloud/edit-stack-settings.md b/deploy-manage/deploy/elastic-cloud/edit-stack-settings.md index 408788433..f0cd3bf09 100644 --- a/deploy-manage/deploy/elastic-cloud/edit-stack-settings.md +++ b/deploy-manage/deploy/elastic-cloud/edit-stack-settings.md @@ -57,9 +57,9 @@ From the {{ecloud}} Console you can customize {{es}}, {{kib}}, and related produ Change how {{es}} runs by providing your own user settings. {{ech}} appends these settings to each node’s `elasticsearch.yml` configuration file. -{{ech}} automatically rejects `elasticsearch.yml` settings that could break your cluster. +{{ech}} automatically rejects `elasticsearch.yml` settings that could break your cluster. -For a list of supported settings, check [Supported {{es}} settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/elastic-cloud-hosted-elasticsearch-settings.md). +For a list of supported settings, check [Supported {{es}} settings](elasticsearch://reference/elasticsearch/configuration-reference/elastic-cloud-hosted-elasticsearch-settings.md). ::::{warning} You can also update [dynamic cluster settings](../../../deploy-manage/deploy/self-managed/configure-elasticsearch.md#dynamic-cluster-setting) using {{es}}'s [update cluster settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings). However, {{ech}} doesn’t reject unsafe setting changes made using this API. Use it with caution. @@ -88,7 +88,7 @@ In some cases, you may get a warning saying "User settings are different across Be aware that some settings that could break your cluster if set incorrectly and that the syntax might change between major versions. -For a list of supported settings, check [Kibana settings](asciidocalypse://docs/kibana/docs/reference/cloud/elastic-cloud-kibana-settings.md). +For a list of supported settings, check [Kibana settings](kibana://reference/cloud/elastic-cloud-kibana-settings.md). To change Kibana settings: diff --git a/deploy-manage/deploy/elastic-cloud/manage-integrations-server.md b/deploy-manage/deploy/elastic-cloud/manage-integrations-server.md index c526976be..45289da53 100644 --- a/deploy-manage/deploy/elastic-cloud/manage-integrations-server.md +++ b/deploy-manage/deploy/elastic-cloud/manage-integrations-server.md @@ -8,7 +8,7 @@ mapped_pages: # Manage your Integrations server [ec-manage-integrations-server] -For deployments that are version 8.0 and later, you have the option to add a combined [Application Performance Monitoring (APM) Server](/solutions/observability/apps/application-performance-monitoring-apm.md) and [Fleet Server](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/index.md) to your deployment. APM allows you to monitor software services and applications in real time, turning that data into documents stored in the Elasticsearch cluster. Fleet allows you to centrally manage Elastic Agents on many hosts. +For deployments that are version 8.0 and later, you have the option to add a combined [Application Performance Monitoring (APM) Server](/solutions/observability/apps/application-performance-monitoring-apm.md) and [Fleet Server](/reference/ingestion-tools/fleet/index.md) to your deployment. APM allows you to monitor software services and applications in real time, turning that data into documents stored in the Elasticsearch cluster. Fleet allows you to centrally manage Elastic Agents on many hosts. As part of provisioning, the APM Server and Fleet Server are already configured to work with Elasticsearch and Kibana. At the end of provisioning, you are shown the secret token to configure communication between the APM Server and the backend [APM Agents](https://www.elastic.co/guide/en/apm/agent/index.html). The APM Agents get deployed within your services and applications. diff --git a/deploy-manage/deploy/elastic-cloud/manage-plugins-extensions-through-api.md b/deploy-manage/deploy/elastic-cloud/manage-plugins-extensions-through-api.md index 00165ca3c..7d7539ac4 100644 --- a/deploy-manage/deploy/elastic-cloud/manage-plugins-extensions-through-api.md +++ b/deploy-manage/deploy/elastic-cloud/manage-plugins-extensions-through-api.md @@ -22,21 +22,21 @@ This guide provides a full list of tasks for managing [plugins and extensions](a * [Delete an extension](#ec-extension-guide-delete) -## Create an extension [ec-extension-guide-create] +## Create an extension [ec-extension-guide-create] There are two methods to create an extension. You can: 1. Stream the file from a publicly-accessible download URL. 2. Upload the file from a local file path. -::::{note} +::::{note} For plugins larger than 200MB the download URL option **must** be used. Plugins larger than 8GB cannot be uploaded with either method. :::: These two examples are for the `plugin` extension type. For bundles, change `extension_type` to `bundle`. -For plugins, `version` must match (exactly) the `elasticsearch.version` field defined in the plugin’s `plugin-descriptor.properties` file. Check [Help for plugin authors](asciidocalypse://docs/elasticsearch/docs/extend/index.md#plugin-authors) for details. For plugins larger than 5GB, the `plugin-descriptor.properties` file needs to be at the top of the archive. This ensures that the our verification process is able to detect that it is an Elasticsearch plugin; otherwise the plugin will be rejected by the API. This order can be achieved by specifying at time of creating the ZIP file: `zip -r name-of-plugin.zip plugin-descriptor.properties *`. +For plugins, `version` must match (exactly) the `elasticsearch.version` field defined in the plugin’s `plugin-descriptor.properties` file. Check [Help for plugin authors](elasticsearch://extend/index.md) for details. For plugins larger than 5GB, the `plugin-descriptor.properties` file needs to be at the top of the archive. This ensures that the our verification process is able to detect that it is an Elasticsearch plugin; otherwise the plugin will be rejected by the API. This order can be achieved by specifying at time of creating the ZIP file: `zip -r name-of-plugin.zip plugin-descriptor.properties *`. For bundles, we recommend setting `version` using wildcard notation that matches the major version of the Elasticsearch deployment. For example, if Elasticsearch is on version 8.4.3, simply set `8.*` as the version. The value `8.*` means that the bundle is compatible with all 8.x versions of Elasticsearch. @@ -58,12 +58,12 @@ curl -X POST \ The single POST request creates an extension with the metadata, validates, and streams the file from the `download_url` specified. The accepted protocols for `download_url` are `http` and `https`. -::::{note} +::::{note} The `download_url` must be directly and publicly accessible. There is currently no support for redirection or authentication unless it contains security credentials/tokens expected by your HTTP service as part of the URL. Otherwise, use the following Option 2 to upload the file from a local path. :::: -::::{note} +::::{note} When the file is larger than 5GB, the request may timeout after 2-5 minutes, but streaming will continue on the server. Check the Extensions page in the Cloud UI after 5-10 minutes to make sure that the plugin has been created. A successfully created plugin will contain correct name, type, version, size, and last modified information. :::: @@ -105,7 +105,7 @@ curl -v -X PUT "https://api.elastic-cloud.com/api/v1/deployments/extensions/EXTE -T "/path_to/custom-plugin-8.4.3.zip" ``` -::::{note} +::::{note} When using curl, always use the `-T` option. DO NOT use `-F` (we have seen inconsistency in curl behavior across systems; using `-F` can result in partially uploaded or truncated files). :::: @@ -123,7 +123,7 @@ The above PUT request uploads the file from the local path specified. This reque ``` -## Add an extension to a deployment plan [ec-extension-guide-add-plan] +## Add an extension to a deployment plan [ec-extension-guide-add-plan] Once the extension is created and uploaded, you can add the extension using its `EXTENSION_ID` in an [update deployment API call](https://www.elastic.co/docs/api/doc/cloud/operation/operation-update-deployment). @@ -180,7 +180,7 @@ The previous examples are for plugins. For bundles, use the `user_bundles` const ``` -## Get an extension [ec-extension-guide-get-extension] +## Get an extension [ec-extension-guide-get-extension] You can use the GET call to retrieve information about an extension. @@ -227,7 +227,7 @@ For example, the previous call returns: ``` -## Update the name of an existing extension [ec-extension-guide-update-name] +## Update the name of an existing extension [ec-extension-guide-update-name] To update the name of an existing extension, simply update the name field without uploading a new file. You do not have to specify the `download_url` when only making metadata changes to an extension. @@ -262,12 +262,12 @@ curl -X POST \ Updating the name of an existing extension does not change its `EXTENSION_ID`. -## Update the type of an existing extension [ec-extension-guide-update-type] +## Update the type of an existing extension [ec-extension-guide-update-type] Updating `extension_type` has no effect. You cannot change the extension’s type (`plugin` versus `bundle`) after the initial creation of a plugin. -## Update the version of an existing bundle [ec-extension-guide-update-version-bundle] +## Update the version of an existing bundle [ec-extension-guide-update-version-bundle] For bundles, we recommend setting `version` using wildcard notation that matches the major version of the Elasticsearch deployment. For example, if Elasticsearch is on version 8.4.3, simply set `8.*` as the version. The value `8.*` means that the bundle is compatible with all 7.x versions of Elasticsearch. @@ -304,12 +304,12 @@ curl -X POST \ Updating the name of an existing extension does not change its `EXTENSION_ID`. -## Update the version of an existing plugin [ec-extension-guide-update-version-plugin] +## Update the version of an existing plugin [ec-extension-guide-update-version-plugin] -For plugins, `version` must match (exactly) the `elasticsearch.version` field defined in the plugin’s `plugin-descriptor.properties` file. Check [Help for plugin authors](asciidocalypse://docs/elasticsearch/docs/extend/index.md#plugin-authors) for details. If you change the version, the associated plugin file *must* also be updated accordingly. +For plugins, `version` must match (exactly) the `elasticsearch.version` field defined in the plugin’s `plugin-descriptor.properties` file. Check [Help for plugin authors](elasticsearch://extend/index.md) for details. If you change the version, the associated plugin file *must* also be updated accordingly. -## Update the file associated to an existing extension [ec-extension-guide-update-file] +## Update the file associated to an existing extension [ec-extension-guide-update-file] You may want to update an uploaded file for an existing extension without performing an Elasticsearch upgrade. If you are updating the extension to prepare for an Elasticsearch upgrade, check the [Upgrade Elasticsearch](#ec-extension-guide-upgrade-elasticsearch) scenario later on this page. @@ -340,7 +340,7 @@ curl -v -X PUT "https://api.elastic-cloud.com/api/v1/deployments/extensions/EXTE -T "/path_to/custom-plugin-8.4.3-10212022.zip" ``` -::::{important} +::::{important} If you are not making any other plan changes and simply updating an extension file, you need to issue a no-op plan so that Elasticsearch will make use of this new file. A *no-op* (no operation) plan triggers a rolling restart on the deployment, applying the same (unchanged) plan as the current plan. :::: @@ -348,7 +348,7 @@ If you are not making any other plan changes and simply updating an extension fi Updating the file of an existing extension or bundle does not change its `EXTENSION_ID`. -## Upgrade Elasticsearch [ec-extension-guide-upgrade-elasticsearch] +## Upgrade Elasticsearch [ec-extension-guide-upgrade-elasticsearch] When you upgrade Elasticsearch in a deployment, you must ensure that: @@ -473,7 +473,7 @@ Unlike bundles, plugins *must* match the Elasticsearch version down to the patch -## Delete an extension [ec-extension-guide-delete] +## Delete an extension [ec-extension-guide-delete] You can delete an extension simply by calling a DELETE against the EXTENSION_ID of interest: diff --git a/deploy-manage/deploy/elastic-cloud/restrictions-known-problems.md b/deploy-manage/deploy/elastic-cloud/restrictions-known-problems.md index c1c472980..305abb8d1 100644 --- a/deploy-manage/deploy/elastic-cloud/restrictions-known-problems.md +++ b/deploy-manage/deploy/elastic-cloud/restrictions-known-problems.md @@ -50,7 +50,7 @@ The following restrictions apply when using APIs in {{ecloud}}: $$$ec-restrictions-apis-elasticsearch$$$ Elasticsearch APIs -: The Elasticsearch APIs do not natively enforce rate limiting. However, all requests to the Elasticsearch cluster are subject to Elasticsearch configuration settings, such as the [network HTTP setting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md#http-settings) `http:max_content_length` which restricts the maximum size of an HTTP request body. This setting has a default value of 100MB, hence restricting API request payloads to that size. This setting is not currently configurable in {{ecloud}}. For a list of which Elasticsearch settings are supported on Cloud, check [Add Elasticsearch user settings](edit-stack-settings.md). To learn about using the Elasticsearch APIs in {{ecloud}}, check [Access the Elasticsearch API console](asciidocalypse://docs/cloud/docs/reference/cloud-hosted/ec-api-console.md). And, for full details about the Elasticsearch APIs and their endpoints, check the [Elasticsearch API reference documentation](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/index.md). +: The Elasticsearch APIs do not natively enforce rate limiting. However, all requests to the Elasticsearch cluster are subject to Elasticsearch configuration settings, such as the [network HTTP setting](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#http-settings) `http:max_content_length` which restricts the maximum size of an HTTP request body. This setting has a default value of 100MB, hence restricting API request payloads to that size. This setting is not currently configurable in {{ecloud}}. For a list of which Elasticsearch settings are supported on Cloud, check [Add Elasticsearch user settings](edit-stack-settings.md). To learn about using the Elasticsearch APIs in {{ecloud}}, check [Access the Elasticsearch API console](asciidocalypse://docs/cloud/docs/reference/cloud-hosted/ec-api-console.md). And, for full details about the Elasticsearch APIs and their endpoints, check the [Elasticsearch API reference documentation](elasticsearch://reference/elasticsearch/rest-apis/index.md). $$$ec-restrictions-apis-kibana$$$ @@ -99,7 +99,7 @@ Currently you can’t use SSO to login directly from {{ecloud}} into Kibana endp ## Kibana [ec-restrictions-kibana] * The maximum size of a single {{kib}} instance is 8GB. This means, {{kib}} instances can be scaled up to 8GB before they are scaled out. For example, when creating a deployment with a {{kib}} instance of size 16GB, then 2x8GB instances are created. If you face performance issues with {{kib}} PNG or PDF reports, the recommendations are to create multiple, smaller dashboards to export the data, or to use a third party browser extension for exporting the dashboard in the format you need. -* Running an external Kibana in parallel to {{ecloud}}’s Kibana instances may cause errors, for example [`Unable to decrypt attribute`](../../../explore-analyze/alerts-cases/alerts/alerting-common-issues.md#rule-cannot-decrypt-api-key), due to a mismatched [`xpack.encryptedSavedObjects.encryptionKey`](asciidocalypse://docs/kibana/docs/reference/configuration-reference/security-settings.md#security-encrypted-saved-objects-settings) as {{ecloud}} does not [allow users to set](edit-stack-settings.md) nor expose this value. While workarounds are possible, this is not officially supported nor generally recommended. +* Running an external Kibana in parallel to {{ecloud}}’s Kibana instances may cause errors, for example [`Unable to decrypt attribute`](../../../explore-analyze/alerts-cases/alerts/alerting-common-issues.md#rule-cannot-decrypt-api-key), due to a mismatched [`xpack.encryptedSavedObjects.encryptionKey`](kibana://reference/configuration-reference/security-settings.md#security-encrypted-saved-objects-settings) as {{ecloud}} does not [allow users to set](edit-stack-settings.md) nor expose this value. While workarounds are possible, this is not officially supported nor generally recommended. % ## APM Agent central configuration with PrivateLink or traffic filters [ec-restrictions-apm-traffic-filters] @@ -134,7 +134,7 @@ There are situations where you may need or want to move your installed {{agents} In {{ecloud}}, you can migrate your {{agents}} by taking a snapshot of your source deployment, and restoring it on a target deployment. -To make a seamless migration, after restoring from a snapshot there are some additional steps required, such as updating settings and resetting the agent policy. Check [Migrate Elastic Agents](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/migrate-elastic-agent.md) for details. +To make a seamless migration, after restoring from a snapshot there are some additional steps required, such as updating settings and resetting the agent policy. Check [Migrate Elastic Agents](/reference/ingestion-tools/fleet/migrate-elastic-agent.md) for details. ## Regions and Availability Zones [ec-regions-and-availability-zone] diff --git a/deploy-manage/deploy/elastic-cloud/switch-from-apm-to-integrations-server-payload.md b/deploy-manage/deploy/elastic-cloud/switch-from-apm-to-integrations-server-payload.md index 60a0c7d62..ff58f95d2 100644 --- a/deploy-manage/deploy/elastic-cloud/switch-from-apm-to-integrations-server-payload.md +++ b/deploy-manage/deploy/elastic-cloud/switch-from-apm-to-integrations-server-payload.md @@ -379,7 +379,7 @@ Beginning with Elastic Stack version 8.0, [Integrations Server](manage-integrati :::: -You have the option to add a combined [Application Performance Monitoring (APM) Server](/solutions/observability/apps/application-performance-monitoring-apm.md) and [Fleet Server](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/index.md) to your deployment. APM allows you to monitor software services and applications in real time, turning that data into documents stored in the Elasticsearch cluster. Fleet allows you to centrally manage Elastic Agents on many hosts. +You have the option to add a combined [Application Performance Monitoring (APM) Server](/solutions/observability/apps/application-performance-monitoring-apm.md) and [Fleet Server](/reference/ingestion-tools/fleet/index.md) to your deployment. APM allows you to monitor software services and applications in real time, turning that data into documents stored in the Elasticsearch cluster. Fleet allows you to centrally manage Elastic Agents on many hosts. As part of provisioning, the APM Server and Fleet Server are already configured to work with Elasticsearch and Kibana. At the end of provisioning, you are shown the secret token to configure communication between the APM Server and the backend [APM Agents](https://www.elastic.co/guide/en/apm/agent/index.html). The APM Agents get deployed within your services and applications. diff --git a/deploy-manage/deploy/elastic-cloud/tools-apis.md b/deploy-manage/deploy/elastic-cloud/tools-apis.md index 705c19c46..cc78d2786 100644 --- a/deploy-manage/deploy/elastic-cloud/tools-apis.md +++ b/deploy-manage/deploy/elastic-cloud/tools-apis.md @@ -60,17 +60,17 @@ Note that some [restrictions](/deploy-manage/deploy/elastic-cloud/restrictions-k The following APIs are available for {{es-serverless}} users. These links will take you to the autogenerated API reference documentation. - [Elasticsearch Serverless APIs](https://www.elastic.co/docs/api/doc/elasticsearch-serverless): Use these APIs to index, manage, search, and analyze your data in {{es-serverless}}. Learn how to [connect to your {{es-serverless}} endpoint](/solutions/search/get-started.md). - - ::::{tip} + + ::::{tip} Learn how to [connect to your {{es-serverless}} endpoint](/solutions/search/get-started.md). :::: - + - [Kibana Serverless APIs](https://www.elastic.co/docs/api/doc/serverless): Use these APIs to manage resources such as connectors, data views, and saved objects for your {{serverless-full}} project. **Additional API information** -- [{{es}} API conventions](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md): Reference information about headers and request body conventions for {{es-serverless}} REST APIs. +- [{{es}} API conventions](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md): Reference information about headers and request body conventions for {{es-serverless}} REST APIs. :::: ::::{tab-item} {{ech}} @@ -92,7 +92,7 @@ serverless: unavailable For each deployment, an **API Console** page is available from the {{ecloud}} Console for you to execute queries using the available APIs. You can find this console when selecting a specific deployment to manage. From there, the API Console is available under the **{{es}}** page. :::{note} -This API Console is different from the [Dev Tools Console](/explore-analyze/query-filter/tools/console.md) available in each deployment, from which you can call {{es}} and {{kib}} APIs. On the {{ecloud}} API Console, you cannot run Kibana APIs. +This API Console is different from the [Dev Tools Console](/explore-analyze/query-filter/tools/console.md) available in each deployment, from which you can call {{es}} and {{kib}} APIs. On the {{ecloud}} API Console, you cannot run Kibana APIs. ::: ## ECCTL - Command-line interface for {{ecloud}} diff --git a/deploy-manage/deploy/self-managed/bootstrap-checks-heap-size.md b/deploy-manage/deploy/self-managed/bootstrap-checks-heap-size.md index 36abba2ff..fc6e239fb 100644 --- a/deploy-manage/deploy/self-managed/bootstrap-checks-heap-size.md +++ b/deploy-manage/deploy/self-managed/bootstrap-checks-heap-size.md @@ -5,5 +5,5 @@ mapped_pages: # Heap size check [bootstrap-checks-heap-size] -By default, {{es}} automatically sizes JVM heap based on a node’s [roles](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md#node-roles) and total memory. If you manually override the default sizing and start the JVM with different initial and max heap sizes, the JVM may pause as it resizes the heap during system usage. If you enable [`bootstrap.memory_lock`](setup-configuration-memory.md#bootstrap-memory_lock), the JVM locks the initial heap size on startup. If the initial heap size is not equal to the maximum heap size, some JVM heap may not be locked after a resize. To avoid these issues, start the JVM with an initial heap size equal to the maximum heap size. +By default, {{es}} automatically sizes JVM heap based on a node’s [roles](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md#node-roles) and total memory. If you manually override the default sizing and start the JVM with different initial and max heap sizes, the JVM may pause as it resizes the heap during system usage. If you enable [`bootstrap.memory_lock`](setup-configuration-memory.md#bootstrap-memory_lock), the JVM locks the initial heap size on startup. If the initial heap size is not equal to the maximum heap size, some JVM heap may not be locked after a resize. To avoid these issues, start the JVM with an initial heap size equal to the maximum heap size. diff --git a/deploy-manage/deploy/self-managed/bootstrap-checks-max-map-count.md b/deploy-manage/deploy/self-managed/bootstrap-checks-max-map-count.md index 46c4cac7b..0c94b5d8a 100644 --- a/deploy-manage/deploy/self-managed/bootstrap-checks-max-map-count.md +++ b/deploy-manage/deploy/self-managed/bootstrap-checks-max-map-count.md @@ -7,5 +7,5 @@ mapped_pages: Continuing from the previous [point](max-size-virtual-memory-check.md), to use `mmap` effectively, Elasticsearch also requires the ability to create many memory-mapped areas. The maximum map count check checks that the kernel allows a process to have at least 262,144 memory-mapped areas and is enforced on Linux only. To pass the maximum map count check, you must configure `vm.max_map_count` via `sysctl` to be at least `262144`. -Alternatively, the maximum map count check is only needed if you are using `mmapfs` or `hybridfs` as the [store type](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/store.md) for your indices. If you [do not allow](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/store.md#allow-mmap) the use of `mmap` then this bootstrap check will not be enforced. +Alternatively, the maximum map count check is only needed if you are using `mmapfs` or `hybridfs` as the [store type](elasticsearch://reference/elasticsearch/index-settings/store.md) for your indices. If you [do not allow](elasticsearch://reference/elasticsearch/index-settings/store.md#allow-mmap) the use of `mmap` then this bootstrap check will not be enforced. diff --git a/deploy-manage/deploy/self-managed/configure.md b/deploy-manage/deploy/self-managed/configure.md index 4f4a5b4cb..da6fe6939 100644 --- a/deploy-manage/deploy/self-managed/configure.md +++ b/deploy-manage/deploy/self-managed/configure.md @@ -185,7 +185,7 @@ $$$elasticsearch-service-account-token$$$ `elasticsearch.serviceAccountToken` : Maximum number of documents loaded by each shard to generate autocomplete suggestions. This value must be a whole number greater than zero. **Default: `"100000"`** ::::{note} - To reload the [logging settings](asciidocalypse://docs/kibana/docs/reference/configuration-reference/logging-settings.md), send a SIGHUP signal to {{kib}}. For more logging configuration options, see the [Configure Logging in {{kib}}](../../monitor/logging-configuration/kibana-logging.md) guide. + To reload the [logging settings](kibana://reference/configuration-reference/logging-settings.md), send a SIGHUP signal to {{kib}}. For more logging configuration options, see the [Configure Logging in {{kib}}](../../monitor/logging-configuration/kibana-logging.md) guide. :::: @@ -262,7 +262,7 @@ $$$tilemap-url$$$ `map.tilemap.url` ![logo cloud](https://doc-icons.s3.us-east-2 : [preview] Indicates which roles to configure the {{kib}} process with, which will effectively run {{kib}} in different modes. Valid options are `background_tasks` and `ui`, or `*` to select all roles. **Default: `*`** `notifications.connectors.default.email` -: Choose the default email connector for user notifications. As of `8.6.0`, {{kib}} is shipping with a new notification mechanism that will send email notifications for various user actions, e.g. assigning a *Case* to a user. To enable notifications, an email connector must be [preconfigured](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/pre-configured-connectors.md) in the system via `kibana.yml`, and the notifications plugin must be configured to point to the ID of that connector. +: Choose the default email connector for user notifications. As of `8.6.0`, {{kib}} is shipping with a new notification mechanism that will send email notifications for various user actions, e.g. assigning a *Case* to a user. To enable notifications, an email connector must be [preconfigured](kibana://reference/connectors-kibana/pre-configured-connectors.md) in the system via `kibana.yml`, and the notifications plugin must be configured to point to the ID of that connector. $$$path-data$$$ `path.data` : The path where {{kib}} stores persistent data not saved in {{es}}. **Default: `data`** @@ -472,12 +472,12 @@ $$$settings-xsrf-disableProtection$$$ `server.xsrf.disableProtection` : If authentication is enabled, setting this to `true` enables unauthenticated users to access the {{kib}} server status API and status page. **Default: `false`** $$$telemetry-allowChangingOptInStatus$$$ `telemetry.allowChangingOptInStatus` -: When `false`, users cannot change the opt-in status through [Advanced Settings](asciidocalypse://docs/kibana/docs/reference/advanced-settings.md), and {{kib}} only looks at the value of [`telemetry.optIn`](#settings-telemetry-optIn) to determine whether to send telemetry data or not. **Default: `true`**. +: When `false`, users cannot change the opt-in status through [Advanced Settings](kibana://reference/advanced-settings.md), and {{kib}} only looks at the value of [`telemetry.optIn`](#settings-telemetry-optIn) to determine whether to send telemetry data or not. **Default: `true`**. $$$settings-telemetry-optIn$$$ `telemetry.optIn` : Set to `false` to stop sending any telemetry data to Elastic. Reporting your cluster statistics helps us improve your user experience. When `false`, the telemetry data is never sent to Elastic.
- This setting can be changed at any time in [Advanced Settings](asciidocalypse://docs/kibana/docs/reference/advanced-settings.md). To prevent users from changing it, set [`telemetry.allowChangingOptInStatus`](#telemetry-allowChangingOptInStatus) to `false`. **Default: `true`** + This setting can be changed at any time in [Advanced Settings](kibana://reference/advanced-settings.md). To prevent users from changing it, set [`telemetry.allowChangingOptInStatus`](#telemetry-allowChangingOptInStatus) to `false`. **Default: `true`** `vis_type_vega.enableExternalUrls` ![logo cloud](https://doc-icons.s3.us-east-2.amazonaws.com/logo_cloud.svg "Supported on {{ech}}") diff --git a/deploy-manage/deploy/self-managed/executable-jna-tmpdir.md b/deploy-manage/deploy/self-managed/executable-jna-tmpdir.md index c3d9cef3e..3cf01fadb 100644 --- a/deploy-manage/deploy/self-managed/executable-jna-tmpdir.md +++ b/deploy-manage/deploy/self-managed/executable-jna-tmpdir.md @@ -5,7 +5,7 @@ mapped_pages: # Ensure JNA temporary directory permits executables [executable-jna-tmpdir] -::::{note} +::::{note} This is only relevant for Linux. :::: @@ -30,9 +30,9 @@ To resolve these problems, either remove the `noexec` option from your `/tmp` fi ``` -If you need finer control over the location of these temporary files, you can also configure the path that JNA uses with the [JVM flag](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/jvm-settings.md#set-jvm-options) `-Djna.tmpdir=` and you can configure the path that `libffi` uses for its temporary files by setting the `LIBFFI_TMPDIR` environment variable. Future versions of {{es}} may need additional configuration, so you should prefer to set `ES_TMPDIR` wherever possible. +If you need finer control over the location of these temporary files, you can also configure the path that JNA uses with the [JVM flag](elasticsearch://reference/elasticsearch/jvm-settings.md#set-jvm-options) `-Djna.tmpdir=` and you can configure the path that `libffi` uses for its temporary files by setting the `LIBFFI_TMPDIR` environment variable. Future versions of {{es}} may need additional configuration, so you should prefer to set `ES_TMPDIR` wherever possible. -::::{note} +::::{note} {{es}} does not remove its temporary directory. You should remove leftover temporary directories while {{es}} is not running. It is best to do this automatically, for instance on each reboot. If you are running on Linux, you can achieve this by using the [tmpfs](https://www.kernel.org/doc/html/latest/filesystems/tmpfs.html) file system. :::: diff --git a/deploy-manage/deploy/self-managed/important-settings-configuration.md b/deploy-manage/deploy/self-managed/important-settings-configuration.md index 8b6b28890..cfc6b0286 100644 --- a/deploy-manage/deploy/self-managed/important-settings-configuration.md +++ b/deploy-manage/deploy/self-managed/important-settings-configuration.md @@ -8,15 +8,15 @@ mapped_pages: {{es}} requires very little configuration to get started, but there are a number of items which **must** be considered before using your cluster in production: * [Path settings](#path-settings) -* [Cluster name setting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/miscellaneous-cluster-settings.md#cluster-name) +* [Cluster name setting](elasticsearch://reference/elasticsearch/configuration-reference/miscellaneous-cluster-settings.md#cluster-name) * [Node name setting](#node-name) * [Network host settings](#network.host) * [Discovery settings](#discovery-settings) * [Heap size settings](#heap-size-settings) * [JVM heap dump path setting](#heap-dump-path) -* [GC logging settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/jvm-settings.md#gc-logging) +* [GC logging settings](elasticsearch://reference/elasticsearch/jvm-settings.md#gc-logging) * [Temporary directory settings](#es-tmpdir) -* [JVM fatal error log setting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/jvm-settings.md#error-file-path) +* [JVM fatal error log setting](elasticsearch://reference/elasticsearch/jvm-settings.md#error-file-path) * [Cluster backups](#important-settings-backups) Our [{{ecloud}}](https://cloud.elastic.co/registration?page=docs&placement=docs-body) service configures these items automatically, making your cluster production-ready by default. @@ -60,7 +60,7 @@ Don’t modify anything within the data directory or run processes that might in :::: -Elasticsearch offers a deprecated setting that allows you to specify multiple paths in `path.data`. To learn about this setting, and how to migrate away from it, refer to [Multiple data paths](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/path.md#multiple-data-paths). +Elasticsearch offers a deprecated setting that allows you to specify multiple paths in `path.data`. To learn about this setting, and how to migrate away from it, refer to [Multiple data paths](elasticsearch://reference/elasticsearch/index-settings/path.md#multiple-data-paths). ## Cluster name setting [_cluster_name_setting] @@ -93,7 +93,7 @@ node.name: prod-data-2 ## Network host setting [network.host] -By default, {{es}} only binds to loopback addresses such as `127.0.0.1` and `[::1]`. This is sufficient to run a cluster of one or more nodes on a single server for development and testing, but a [resilient production cluster](../../production-guidance/availability-and-resilience.md) must involve nodes on other servers. There are many [network settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md) but usually all you need to configure is `network.host`: +By default, {{es}} only binds to loopback addresses such as `127.0.0.1` and `[::1]`. This is sufficient to run a cluster of one or more nodes on a single server for development and testing, but a [resilient production cluster](../../production-guidance/availability-and-resilience.md) must involve nodes on other servers. There are many [network settings](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md) but usually all you need to configure is `network.host`: ```yaml network.host: 192.168.1.10 @@ -158,21 +158,21 @@ cluster.initial_master_nodes: <1> 1. Identify the initial master nodes by their [`node.name`](#node-name), which defaults to their hostname. Ensure that the value in `cluster.initial_master_nodes` matches the `node.name` exactly. If you use a fully-qualified domain name (FQDN) such as `master-node-a.example.com` for your node names, then you must use the FQDN in this list. Conversely, if `node.name` is a bare hostname without any trailing qualifiers, you must also omit the trailing qualifiers in `cluster.initial_master_nodes`. -See [bootstrapping a cluster](../../distributed-architecture/discovery-cluster-formation/modules-discovery-bootstrap-cluster.md) and [discovery and cluster formation settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/discovery-cluster-formation-settings.md). +See [bootstrapping a cluster](../../distributed-architecture/discovery-cluster-formation/modules-discovery-bootstrap-cluster.md) and [discovery and cluster formation settings](elasticsearch://reference/elasticsearch/configuration-reference/discovery-cluster-formation-settings.md). ## Heap size settings [heap-size-settings] -By default, {{es}} automatically sets the JVM heap size based on a node’s [roles](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md#node-roles) and total memory. We recommend the default sizing for most production environments. +By default, {{es}} automatically sets the JVM heap size based on a node’s [roles](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md#node-roles) and total memory. We recommend the default sizing for most production environments. -If needed, you can override the default sizing by manually [setting the JVM heap size](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/jvm-settings.md#set-jvm-heap-size). +If needed, you can override the default sizing by manually [setting the JVM heap size](elasticsearch://reference/elasticsearch/jvm-settings.md#set-jvm-heap-size). ## JVM heap dump path setting [heap-dump-path] By default, {{es}} configures the JVM to dump the heap on out of memory exceptions to the default data directory. On [RPM](install-elasticsearch-with-rpm.md) and [Debian](install-elasticsearch-with-debian-package.md) packages, the data directory is `/var/lib/elasticsearch`. On [Linux and MacOS](install-elasticsearch-from-archive-on-linux-macos.md) and [Windows](install-elasticsearch-with-zip-on-windows.md) distributions, the `data` directory is located under the root of the {{es}} installation. -If this path is not suitable for receiving heap dumps, modify the `-XX:HeapDumpPath=...` entry in [`jvm.options`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/jvm-settings.md#set-jvm-options): +If this path is not suitable for receiving heap dumps, modify the `-XX:HeapDumpPath=...` entry in [`jvm.options`](elasticsearch://reference/elasticsearch/jvm-settings.md#set-jvm-options): * If you specify a directory, the JVM will generate a filename for the heap dump based on the PID of the running instance. * If you specify a fixed filename instead of a directory, the file must not exist when the JVM needs to perform a heap dump on an out of memory exception. Otherwise, the heap dump will fail. @@ -180,7 +180,7 @@ If this path is not suitable for receiving heap dumps, modify the `-XX:HeapDumpP ## GC logging settings [_gc_logging_settings] -By default, {{es}} enables garbage collection (GC) logs. These are configured in [`jvm.options`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/jvm-settings.md#set-jvm-options) and output to the same default location as the {{es}} logs. The default configuration rotates the logs every 64 MB and can consume up to 2 GB of disk space. +By default, {{es}} enables garbage collection (GC) logs. These are configured in [`jvm.options`](elasticsearch://reference/elasticsearch/jvm-settings.md#set-jvm-options) and output to the same default location as the {{es}} logs. The default configuration rotates the logs every 64 MB and can consume up to 2 GB of disk space. You can reconfigure JVM logging using the command line options described in [JEP 158: Unified JVM Logging](https://openjdk.java.net/jeps/158). Unless you change the default `jvm.options` file directly, the {{es}} default configuration is applied in addition to your own settings. To disable the default configuration, first disable logging by supplying the `-Xlog:disable` option, then supply your own command line options. This disables *all* JVM logging, so be sure to review the available options and enable everything that you require. @@ -225,7 +225,7 @@ If you intend to run the `.tar.gz` distribution on Linux or MacOS for an extende By default, {{es}} configures the JVM to write fatal error logs to the default logging directory. On [RPM](install-elasticsearch-with-rpm.md) and [Debian](install-elasticsearch-with-debian-package.md) packages, this directory is `/var/log/elasticsearch`. On [Linux and MacOS](install-elasticsearch-from-archive-on-linux-macos.md) and [Windows](install-elasticsearch-with-zip-on-windows.md) distributions, the `logs` directory is located under the root of the {{es}} installation. -These are logs produced by the JVM when it encounters a fatal error, such as a segmentation fault. If this path is not suitable for receiving logs, modify the `-XX:ErrorFile=...` entry in [`jvm.options`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/jvm-settings.md#set-jvm-options). +These are logs produced by the JVM when it encounters a fatal error, such as a segmentation fault. If this path is not suitable for receiving logs, modify the `-XX:ErrorFile=...` entry in [`jvm.options`](elasticsearch://reference/elasticsearch/jvm-settings.md#set-jvm-options). ## Cluster backups [important-settings-backups] diff --git a/deploy-manage/deploy/self-managed/install-elasticsearch-from-archive-on-linux-macos.md b/deploy-manage/deploy/self-managed/install-elasticsearch-from-archive-on-linux-macos.md index 55c1c4f0f..bd23611b1 100644 --- a/deploy-manage/deploy/self-managed/install-elasticsearch-from-archive-on-linux-macos.md +++ b/deploy-manage/deploy/self-managed/install-elasticsearch-from-archive-on-linux-macos.md @@ -7,7 +7,7 @@ mapped_pages: {{es}} is available as a `.tar.gz` archive for Linux and MacOS. -This package contains both free and subscription features. [Start a 30-day trial](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/license-settings.md) to try out all of the features. +This package contains both free and subscription features. [Start a 30-day trial](elasticsearch://reference/elasticsearch/configuration-reference/license-settings.md) to try out all of the features. The latest stable version of {{es}} can be found on the [Download {{es}}](https://elastic.co/downloads/elasticsearch) page. Other versions can be found on the [Past Releases page](https://elastic.co/downloads/past-releases). @@ -130,11 +130,11 @@ When {{es}} starts for the first time, the security auto-configuration process b Before enrolling a new node, additional actions such as binding to an address other than `localhost` or satisfying bootstrap checks are typically necessary in production clusters. During that time, an auto-generated enrollment token could expire, which is why enrollment tokens aren’t generated automatically. -Additionally, only nodes on the same host can join the cluster without additional configuration. If you want nodes from another host to join your cluster, you need to set `transport.host` to a [supported value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md#network-interface-values) (such as uncommenting the suggested value of `0.0.0.0`), or an IP address that’s bound to an interface where other hosts can reach it. Refer to [transport settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md#transport-settings) for more information. +Additionally, only nodes on the same host can join the cluster without additional configuration. If you want nodes from another host to join your cluster, you need to set `transport.host` to a [supported value](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#network-interface-values) (such as uncommenting the suggested value of `0.0.0.0`), or an IP address that’s bound to an interface where other hosts can reach it. Refer to [transport settings](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#transport-settings) for more information. To enroll new nodes in your cluster, create an enrollment token with the `elasticsearch-create-enrollment-token` tool on any existing node in your cluster. You can then start a new node with the `--enrollment-token` parameter so that it joins an existing cluster. -1. In a separate terminal from where {{es}} is running, navigate to the directory where you installed {{es}} and run the [`elasticsearch-create-enrollment-token`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/create-enrollment-token.md) tool to generate an enrollment token for your new nodes. +1. In a separate terminal from where {{es}} is running, navigate to the directory where you installed {{es}} and run the [`elasticsearch-create-enrollment-token`](elasticsearch://reference/elasticsearch/command-line-tools/create-enrollment-token.md) tool to generate an enrollment token for your new nodes. ```sh bin/elasticsearch-create-enrollment-token -s node @@ -307,7 +307,7 @@ When you install {{es}}, the following certificates and keys are generated in th `transport.p12` : Keystore that contains the key and certificate for the transport layer for all the nodes in your cluster. -`http.p12` and `transport.p12` are password-protected PKCS#12 keystores. {{es}} stores the passwords for these keystores as [secure settings](../../security/secure-settings.md). To retrieve the passwords so that you can inspect or change the keystore contents, use the [`bin/elasticsearch-keystore`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/elasticsearch-keystore.md) tool. +`http.p12` and `transport.p12` are password-protected PKCS#12 keystores. {{es}} stores the passwords for these keystores as [secure settings](../../security/secure-settings.md). To retrieve the passwords so that you can inspect or change the keystore contents, use the [`bin/elasticsearch-keystore`](elasticsearch://reference/elasticsearch/command-line-tools/elasticsearch-keystore.md) tool. Use the following command to retrieve the password for `http.p12`: diff --git a/deploy-manage/deploy/self-managed/install-elasticsearch-with-debian-package.md b/deploy-manage/deploy/self-managed/install-elasticsearch-with-debian-package.md index f91553579..ba6e48d29 100644 --- a/deploy-manage/deploy/self-managed/install-elasticsearch-with-debian-package.md +++ b/deploy-manage/deploy/self-managed/install-elasticsearch-with-debian-package.md @@ -7,7 +7,7 @@ mapped_pages: The Debian package for Elasticsearch can be [downloaded from our website](#install-deb) or from our [APT repository](#deb-repo). It can be used to install Elasticsearch on any Debian-based system such as Debian and Ubuntu. -This package contains both free and subscription features. [Start a 30-day trial](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/license-settings.md) to try out all of the features. +This package contains both free and subscription features. [Start a 30-day trial](elasticsearch://reference/elasticsearch/configuration-reference/license-settings.md) to try out all of the features. The latest stable version of Elasticsearch can be found on the [Download Elasticsearch](https://elastic.co/downloads/elasticsearch) page. Other versions can be found on the [Past Releases page](https://elastic.co/downloads/past-releases). @@ -78,7 +78,7 @@ When installing {{es}}, security features are enabled and configured by default. * Authentication and authorization are enabled, and a password is generated for the `elastic` built-in superuser. * Certificates and keys for TLS are generated for the transport and HTTP layer, and TLS is enabled and configured with these keys and certificates. -The password and certificate and keys are output to your terminal. You can reset the password for the `elastic` user with the [`elasticsearch-reset-password`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/reset-password.md) command. +The password and certificate and keys are output to your terminal. You can reset the password for the `elastic` user with the [`elasticsearch-reset-password`](elasticsearch://reference/elasticsearch/command-line-tools/reset-password.md) command. We recommend storing the `elastic` password as an environment variable in your shell. For example: @@ -340,7 +340,7 @@ When you install {{es}}, the following certificates and keys are generated in th `transport.p12` : Keystore that contains the key and certificate for the transport layer for all the nodes in your cluster. -`http.p12` and `transport.p12` are password-protected PKCS#12 keystores. {{es}} stores the passwords for these keystores as [secure settings](../../security/secure-settings.md). To retrieve the passwords so that you can inspect or change the keystore contents, use the [`bin/elasticsearch-keystore`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/elasticsearch-keystore.md) tool. +`http.p12` and `transport.p12` are password-protected PKCS#12 keystores. {{es}} stores the passwords for these keystores as [secure settings](../../security/secure-settings.md). To retrieve the passwords so that you can inspect or change the keystore contents, use the [`bin/elasticsearch-keystore`](elasticsearch://reference/elasticsearch/command-line-tools/elasticsearch-keystore.md) tool. Use the following command to retrieve the password for `http.p12`: diff --git a/deploy-manage/deploy/self-managed/install-elasticsearch-with-docker.md b/deploy-manage/deploy/self-managed/install-elasticsearch-with-docker.md index f3ca14670..3a234709d 100644 --- a/deploy-manage/deploy/self-managed/install-elasticsearch-with-docker.md +++ b/deploy-manage/deploy/self-managed/install-elasticsearch-with-docker.md @@ -7,7 +7,7 @@ mapped_pages: Docker images for {{es}} are available from the Elastic Docker registry. A list of all published Docker images and tags is available at [www.docker.elastic.co](https://www.docker.elastic.co). The source code is in [GitHub](https://github.com/elastic/elasticsearch/blob/master/distribution/docker). -This package contains both free and subscription features. [Start a 30-day trial](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/license-settings.md) to try out all of the features. +This package contains both free and subscription features. [Start a 30-day trial](elasticsearch://reference/elasticsearch/configuration-reference/license-settings.md) to try out all of the features. ::::{tip} If you just want to test {{es}} in local development, refer to [Run {{es}} locally](../../../solutions/search/get-started.md). Please note that this setup is not suitable for production environments. @@ -452,9 +452,9 @@ The image [exposes](https://docs.docker.com/engine/reference/builder/#/expose) T ### Manually set the heap size [docker-set-heap-size] -By default, {{es}} automatically sizes JVM heap based on a nodes’s [roles](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md#node-roles) and the total memory available to the node’s container. We recommend this default sizing for most production environments. If needed, you can override default sizing by manually setting JVM heap size. +By default, {{es}} automatically sizes JVM heap based on a nodes’s [roles](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md#node-roles) and the total memory available to the node’s container. We recommend this default sizing for most production environments. If needed, you can override default sizing by manually setting JVM heap size. -To manually set the heap size in production, bind mount a [JVM options](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/jvm-settings.md#set-jvm-options) file under `/usr/share/elasticsearch/config/jvm.options.d` that includes your desired [heap size](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/jvm-settings.md#set-jvm-heap-size) settings. +To manually set the heap size in production, bind mount a [JVM options](elasticsearch://reference/elasticsearch/jvm-settings.md#set-jvm-options) file under `/usr/share/elasticsearch/config/jvm.options.d` that includes your desired [heap size](elasticsearch://reference/elasticsearch/jvm-settings.md#set-jvm-heap-size) settings. For testing, you can also manually set the heap size using the `ES_JAVA_OPTS` environment variable. For example, to use 1GB, use the following command. @@ -595,7 +595,7 @@ Some plugins require additional security permissions. You must explicitly accept * Attaching a `tty` when you run the Docker image and allowing the permissions when prompted. * Inspecting the security permissions and accepting them (if appropriate) by adding the `--batch` flag to the plugin install command. -See [Plugin management](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch-plugins/_other_command_line_parameters.md) for more information. +See [Plugin management](elasticsearch://reference/elasticsearch-plugins/_other_command_line_parameters.md) for more information. ### Troubleshoot Docker errors for {{es}} [troubleshoot-docker-errors] diff --git a/deploy-manage/deploy/self-managed/install-elasticsearch-with-rpm.md b/deploy-manage/deploy/self-managed/install-elasticsearch-with-rpm.md index 9ebea21cf..be47ceb5c 100644 --- a/deploy-manage/deploy/self-managed/install-elasticsearch-with-rpm.md +++ b/deploy-manage/deploy/self-managed/install-elasticsearch-with-rpm.md @@ -12,7 +12,7 @@ RPM install is not supported on distributions with old versions of RPM, such as :::: -This package contains both free and subscription features. [Start a 30-day trial](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/license-settings.md) to try out all of the features. +This package contains both free and subscription features. [Start a 30-day trial](elasticsearch://reference/elasticsearch/configuration-reference/license-settings.md) to try out all of the features. The latest stable version of Elasticsearch can be found on the [Download Elasticsearch](https://elastic.co/downloads/elasticsearch) page. Other versions can be found on the [Past Releases page](https://elastic.co/downloads/past-releases). @@ -82,7 +82,7 @@ When installing {{es}}, security features are enabled and configured by default. * Authentication and authorization are enabled, and a password is generated for the `elastic` built-in superuser. * Certificates and keys for TLS are generated for the transport and HTTP layer, and TLS is enabled and configured with these keys and certificates. -The password and certificate and keys are output to your terminal. You can reset the password for the `elastic` user with the [`elasticsearch-reset-password`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/reset-password.md) command. +The password and certificate and keys are output to your terminal. You can reset the password for the `elastic` user with the [`elasticsearch-reset-password`](elasticsearch://reference/elasticsearch/command-line-tools/reset-password.md) command. We recommend storing the `elastic` password as an environment variable in your shell. For example: @@ -344,7 +344,7 @@ When you install {{es}}, the following certificates and keys are generated in th `transport.p12` : Keystore that contains the key and certificate for the transport layer for all the nodes in your cluster. -`http.p12` and `transport.p12` are password-protected PKCS#12 keystores. {{es}} stores the passwords for these keystores as [secure settings](../../security/secure-settings.md). To retrieve the passwords so that you can inspect or change the keystore contents, use the [`bin/elasticsearch-keystore`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/elasticsearch-keystore.md) tool. +`http.p12` and `transport.p12` are password-protected PKCS#12 keystores. {{es}} stores the passwords for these keystores as [secure settings](../../security/secure-settings.md). To retrieve the passwords so that you can inspect or change the keystore contents, use the [`bin/elasticsearch-keystore`](elasticsearch://reference/elasticsearch/command-line-tools/elasticsearch-keystore.md) tool. Use the following command to retrieve the password for `http.p12`: diff --git a/deploy-manage/deploy/self-managed/install-elasticsearch-with-zip-on-windows.md b/deploy-manage/deploy/self-managed/install-elasticsearch-with-zip-on-windows.md index 4cdbe53c5..ca5b5a363 100644 --- a/deploy-manage/deploy/self-managed/install-elasticsearch-with-zip-on-windows.md +++ b/deploy-manage/deploy/self-managed/install-elasticsearch-with-zip-on-windows.md @@ -7,7 +7,7 @@ mapped_pages: {{es}} can be installed on Windows using the Windows `.zip` archive. This comes with a `elasticsearch-service.bat` command which will setup {{es}} to run as a service. -This package contains both free and subscription features. [Start a 30-day trial](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/license-settings.md) to try out all of the features. +This package contains both free and subscription features. [Start a 30-day trial](elasticsearch://reference/elasticsearch/configuration-reference/license-settings.md) to try out all of the features. ::::{note} On Windows the {{es}} {{ml}} feature requires the Microsoft Universal C Runtime library. This is built into Windows 10, Windows Server 2016 and more recent versions of Windows. For older versions of Windows it can be installed via Windows Update, or from a [separate download](https://support.microsoft.com/en-us/help/2999226/update-for-universal-c-runtime-in-windows). If you cannot install the Microsoft Universal C Runtime library you can still use the rest of {{es}} if you disable the {{ml}} feature. @@ -87,11 +87,11 @@ When {{es}} starts for the first time, the security auto-configuration process b Before enrolling a new node, additional actions such as binding to an address other than `localhost` or satisfying bootstrap checks are typically necessary in production clusters. During that time, an auto-generated enrollment token could expire, which is why enrollment tokens aren’t generated automatically. -Additionally, only nodes on the same host can join the cluster without additional configuration. If you want nodes from another host to join your cluster, you need to set `transport.host` to a [supported value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md#network-interface-values) (such as uncommenting the suggested value of `0.0.0.0`), or an IP address that’s bound to an interface where other hosts can reach it. Refer to [transport settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md#transport-settings) for more information. +Additionally, only nodes on the same host can join the cluster without additional configuration. If you want nodes from another host to join your cluster, you need to set `transport.host` to a [supported value](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#network-interface-values) (such as uncommenting the suggested value of `0.0.0.0`), or an IP address that’s bound to an interface where other hosts can reach it. Refer to [transport settings](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#transport-settings) for more information. To enroll new nodes in your cluster, create an enrollment token with the `elasticsearch-create-enrollment-token` tool on any existing node in your cluster. You can then start a new node with the `--enrollment-token` parameter so that it joins an existing cluster. -1. In a separate terminal from where {{es}} is running, navigate to the directory where you installed {{es}} and run the [`elasticsearch-create-enrollment-token`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/create-enrollment-token.md) tool to generate an enrollment token for your new nodes. +1. In a separate terminal from where {{es}} is running, navigate to the directory where you installed {{es}} and run the [`elasticsearch-create-enrollment-token`](elasticsearch://reference/elasticsearch/command-line-tools/create-enrollment-token.md) tool to generate an enrollment token for your new nodes. ```sh bin\elasticsearch-create-enrollment-token -s node @@ -194,7 +194,7 @@ You can install {{es}} as a service that runs in the background or starts automa TLS is not enabled or configured when you start {{es}} as a service. :::: -3. Generate a password for the `elastic` user with the [`elasticsearch-reset-password`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/reset-password.md) tool. The password is output to the command line. +3. Generate a password for the `elastic` user with the [`elasticsearch-reset-password`](elasticsearch://reference/elasticsearch/command-line-tools/reset-password.md) tool. The password is output to the command line. ```sh C:\Program Files\elasticsearch-9.0.0-beta1\bin>\bin\elasticsearch-reset-password -u elastic @@ -284,9 +284,9 @@ At its core, `elasticsearch-service.bat` relies on [Apache Commons Daemon](https ::::{note} -By default, {{es}} automatically sizes JVM heap based on a node’s [roles](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md#node-roles) and total memory. We recommend this default sizing for most production environments. If needed, you can override default sizing by manually setting the heap size. +By default, {{es}} automatically sizes JVM heap based on a node’s [roles](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md#node-roles) and total memory. We recommend this default sizing for most production environments. If needed, you can override default sizing by manually setting the heap size. -When installing {{es}} on Windows as a service for the first time or running {{es}} from the command line, you can manually [Set the JVM heap size](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/jvm-settings.md#set-jvm-heap-size). To resize the heap for an already installed service, use the service manager: `bin\elasticsearch-service.bat manager`. +When installing {{es}} on Windows as a service for the first time or running {{es}} from the command line, you can manually [Set the JVM heap size](elasticsearch://reference/elasticsearch/jvm-settings.md#set-jvm-heap-size). To resize the heap for an already installed service, use the service manager: `bin\elasticsearch-service.bat manager`. :::: diff --git a/deploy-manage/deploy/self-managed/install-from-archive-on-linux-macos.md b/deploy-manage/deploy/self-managed/install-from-archive-on-linux-macos.md index dc8991e99..748435391 100644 --- a/deploy-manage/deploy/self-managed/install-from-archive-on-linux-macos.md +++ b/deploy-manage/deploy/self-managed/install-from-archive-on-linux-macos.md @@ -77,7 +77,7 @@ If this is the first time you’re starting {{kib}}, this command generates a un 3. Log in to {{kib}} as the `elastic` user with the password that was generated when you started {{es}}. ::::{note} -If you need to reset the password for the `elastic` user or other built-in users, run the [`elasticsearch-reset-password`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/reset-password.md) tool. To generate new enrollment tokens for {{kib}} or {{es}} nodes, run the [`elasticsearch-create-enrollment-token`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/create-enrollment-token.md) tool. These tools are available in the {{es}} `bin` directory. +If you need to reset the password for the `elastic` user or other built-in users, run the [`elasticsearch-reset-password`](elasticsearch://reference/elasticsearch/command-line-tools/reset-password.md) tool. To generate new enrollment tokens for {{kib}} or {{es}} nodes, run the [`elasticsearch-create-enrollment-token`](elasticsearch://reference/elasticsearch/command-line-tools/create-enrollment-token.md) tool. These tools are available in the {{es}} `bin` directory. :::: diff --git a/deploy-manage/deploy/self-managed/install-on-windows.md b/deploy-manage/deploy/self-managed/install-on-windows.md index cd2e52e71..294f3045c 100644 --- a/deploy-manage/deploy/self-managed/install-on-windows.md +++ b/deploy-manage/deploy/self-managed/install-on-windows.md @@ -50,7 +50,7 @@ If this is the first time you’re starting {{kib}}, this command generates a un 3. Log in to {{kib}} as the `elastic` user with the password that was generated when you started {{es}}. ::::{note} -If you need to reset the password for the `elastic` user or other built-in users, run the [`elasticsearch-reset-password`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/reset-password.md) tool. To generate new enrollment tokens for {{kib}} or {{es}} nodes, run the [`elasticsearch-create-enrollment-token`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/create-enrollment-token.md) tool. These tools are available in the {{es}} `bin` directory. +If you need to reset the password for the `elastic` user or other built-in users, run the [`elasticsearch-reset-password`](elasticsearch://reference/elasticsearch/command-line-tools/reset-password.md) tool. To generate new enrollment tokens for {{kib}} or {{es}} nodes, run the [`elasticsearch-create-enrollment-token`](elasticsearch://reference/elasticsearch/command-line-tools/create-enrollment-token.md) tool. These tools are available in the {{es}} `bin` directory. :::: diff --git a/deploy-manage/deploy/self-managed/install-with-debian-package.md b/deploy-manage/deploy/self-managed/install-with-debian-package.md index 0316337fa..388be1cb5 100644 --- a/deploy-manage/deploy/self-managed/install-with-debian-package.md +++ b/deploy-manage/deploy/self-managed/install-with-debian-package.md @@ -49,7 +49,7 @@ When you start {{es}} for the first time, the following security configuration o The password and certificate and keys are output to your terminal. -You can then generate an enrollment token for {{kib}} with the [`elasticsearch-create-enrollment-token`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/create-enrollment-token.md) tool: +You can then generate an enrollment token for {{kib}} with the [`elasticsearch-create-enrollment-token`](elasticsearch://reference/elasticsearch/command-line-tools/create-enrollment-token.md) tool: ```sh bin/elasticsearch-create-enrollment-token -s kibana diff --git a/deploy-manage/deploy/self-managed/install-with-rpm.md b/deploy-manage/deploy/self-managed/install-with-rpm.md index ce62ded7e..8750dc530 100644 --- a/deploy-manage/deploy/self-managed/install-with-rpm.md +++ b/deploy-manage/deploy/self-managed/install-with-rpm.md @@ -59,7 +59,7 @@ When you start {{es}} for the first time, the following security configuration o The password and certificate and keys are output to your terminal. -You can then generate an enrollment token for {{kib}} with the [`elasticsearch-create-enrollment-token`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/create-enrollment-token.md) tool: +You can then generate an enrollment token for {{kib}} with the [`elasticsearch-create-enrollment-token`](elasticsearch://reference/elasticsearch/command-line-tools/create-enrollment-token.md) tool: ```sh bin/elasticsearch-create-enrollment-token -s kibana diff --git a/deploy-manage/deploy/self-managed/max-number-threads-check.md b/deploy-manage/deploy/self-managed/max-number-threads-check.md index 121251822..43e75f063 100644 --- a/deploy-manage/deploy/self-managed/max-number-threads-check.md +++ b/deploy-manage/deploy/self-managed/max-number-threads-check.md @@ -5,5 +5,5 @@ mapped_pages: # Maximum number of threads check [max-number-threads-check] -Elasticsearch executes requests by breaking the request down into stages and handing those stages off to different thread pool executors. There are different [thread pool executors](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/thread-pool-settings.md) for a variety of tasks within Elasticsearch. Thus, Elasticsearch needs the ability to create a lot of threads. The maximum number of threads check ensures that the Elasticsearch process has the rights to create enough threads under normal use. This check is enforced only on Linux. If you are on Linux, to pass the maximum number of threads check, you must configure your system to allow the Elasticsearch process the ability to create at least 4096 threads. This can be done via `/etc/security/limits.conf` using the `nproc` setting (note that you might have to increase the limits for the `root` user too). +Elasticsearch executes requests by breaking the request down into stages and handing those stages off to different thread pool executors. There are different [thread pool executors](elasticsearch://reference/elasticsearch/configuration-reference/thread-pool-settings.md) for a variety of tasks within Elasticsearch. Thus, Elasticsearch needs the ability to create a lot of threads. The maximum number of threads check ensures that the Elasticsearch process has the rights to create enough threads under normal use. This check is enforced only on Linux. If you are on Linux, to pass the maximum number of threads check, you must configure your system to allow the Elasticsearch process the ability to create at least 4096 threads. This can be done via `/etc/security/limits.conf` using the `nproc` setting (note that you might have to increase the limits for the `root` user too). diff --git a/deploy-manage/deploy/self-managed/networkaddress-cache-ttl.md b/deploy-manage/deploy/self-managed/networkaddress-cache-ttl.md index c554d7a34..294304f53 100644 --- a/deploy-manage/deploy/self-managed/networkaddress-cache-ttl.md +++ b/deploy-manage/deploy/self-managed/networkaddress-cache-ttl.md @@ -5,5 +5,5 @@ mapped_pages: # DNS cache settings [networkaddress-cache-ttl] -Elasticsearch runs with a security manager in place. With a security manager in place, the JVM defaults to caching positive hostname resolutions indefinitely and defaults to caching negative hostname resolutions for ten seconds. Elasticsearch overrides this behavior with default values to cache positive lookups for sixty seconds, and to cache negative lookups for ten seconds. These values should be suitable for most environments, including environments where DNS resolutions vary with time. If not, you can edit the values `es.networkaddress.cache.ttl` and `es.networkaddress.cache.negative.ttl` in the [JVM options](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/jvm-settings.md#set-jvm-options). Note that the values [`networkaddress.cache.ttl=`](https://docs.oracle.com/javase/8/docs/technotes/guides/net/properties.md) and [`networkaddress.cache.negative.ttl=`](https://docs.oracle.com/javase/8/docs/technotes/guides/net/properties.md) in the [Java security policy](https://docs.oracle.com/javase/8/docs/technotes/guides/security/PolicyFiles.md) are ignored by Elasticsearch unless you remove the settings for `es.networkaddress.cache.ttl` and `es.networkaddress.cache.negative.ttl`. +Elasticsearch runs with a security manager in place. With a security manager in place, the JVM defaults to caching positive hostname resolutions indefinitely and defaults to caching negative hostname resolutions for ten seconds. Elasticsearch overrides this behavior with default values to cache positive lookups for sixty seconds, and to cache negative lookups for ten seconds. These values should be suitable for most environments, including environments where DNS resolutions vary with time. If not, you can edit the values `es.networkaddress.cache.ttl` and `es.networkaddress.cache.negative.ttl` in the [JVM options](elasticsearch://reference/elasticsearch/jvm-settings.md#set-jvm-options). Note that the values [`networkaddress.cache.ttl=`](https://docs.oracle.com/javase/8/docs/technotes/guides/net/properties.md) and [`networkaddress.cache.negative.ttl=`](https://docs.oracle.com/javase/8/docs/technotes/guides/net/properties.md) in the [Java security policy](https://docs.oracle.com/javase/8/docs/technotes/guides/security/PolicyFiles.md) are ignored by Elasticsearch unless you remove the settings for `es.networkaddress.cache.ttl` and `es.networkaddress.cache.negative.ttl`. diff --git a/deploy-manage/deploy/self-managed/plugins.md b/deploy-manage/deploy/self-managed/plugins.md index 1bb9edc9b..7361a65f9 100644 --- a/deploy-manage/deploy/self-managed/plugins.md +++ b/deploy-manage/deploy/self-managed/plugins.md @@ -7,7 +7,7 @@ mapped_pages: Plugins are a way to enhance the basic Elasticsearch functionality in a custom manner. They range from adding custom mapping types, custom analyzers (in a more built in fashion), custom script engines, custom discovery and more. -For information about selecting and installing plugins, see [{{es}} Plugins and Integrations](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch-plugins/index.md). +For information about selecting and installing plugins, see [{{es}} Plugins and Integrations](elasticsearch://reference/elasticsearch-plugins/index.md). -For information about developing your own plugin, see [Help for plugin authors](asciidocalypse://docs/elasticsearch/docs/extend/index.md). +For information about developing your own plugin, see [Help for plugin authors](elasticsearch://extend/index.md). diff --git a/deploy-manage/deploy/self-managed/system-config-tcpretries.md b/deploy-manage/deploy/self-managed/system-config-tcpretries.md index 91a3e83e4..8204b24b1 100644 --- a/deploy-manage/deploy/self-managed/system-config-tcpretries.md +++ b/deploy-manage/deploy/self-managed/system-config-tcpretries.md @@ -5,7 +5,7 @@ mapped_pages: # TCP retransmission timeout [system-config-tcpretries] -Each pair of {{es}} nodes communicates via a number of TCP connections which [remain open](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md#long-lived-connections) until one of the nodes shuts down or communication between the nodes is disrupted by a failure in the underlying infrastructure. +Each pair of {{es}} nodes communicates via a number of TCP connections which [remain open](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#long-lived-connections) until one of the nodes shuts down or communication between the nodes is disrupted by a failure in the underlying infrastructure. TCP provides reliable communication over occasionally unreliable networks by hiding temporary network disruptions from the communicating applications. Your operating system will retransmit any lost messages a number of times before informing the sender of any problem. {{es}} must wait while the retransmissions are happening and can only react once the operating system decides to give up. Users must therefore also wait for a sequence of retransmissions to complete. @@ -30,6 +30,6 @@ This setting applies to all TCP connections and will affect the reliability of c {{es}} also implements its own internal health checks with timeouts that are much shorter than the default retransmission timeout on Linux. Since these are application-level health checks their timeouts must allow for application-level effects such as garbage collection pauses. You should not reduce any timeouts related to these application-level health checks. -You must also ensure your network infrastructure does not interfere with the long-lived connections between nodes, [even if those connections appear to be idle](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md#long-lived-connections). Devices which drop connections when they reach a certain age are a common source of problems to {{es}} clusters, and must not be used. +You must also ensure your network infrastructure does not interfere with the long-lived connections between nodes, [even if those connections appear to be idle](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#long-lived-connections). Devices which drop connections when they reach a certain age are a common source of problems to {{es}} clusters, and must not be used. diff --git a/deploy-manage/deploy/self-managed/vm-max-map-count.md b/deploy-manage/deploy/self-managed/vm-max-map-count.md index 808376a7d..2e3f48241 100644 --- a/deploy-manage/deploy/self-managed/vm-max-map-count.md +++ b/deploy-manage/deploy/self-managed/vm-max-map-count.md @@ -5,7 +5,7 @@ mapped_pages: # Virtual memory [vm-max-map-count] -Elasticsearch uses a [`mmapfs`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/store.md#mmapfs) directory by default to store its indices. The default operating system limits on mmap counts is likely to be too low, which may result in out of memory exceptions. +Elasticsearch uses a [`mmapfs`](elasticsearch://reference/elasticsearch/index-settings/store.md#mmapfs) directory by default to store its indices. The default operating system limits on mmap counts is likely to be too low, which may result in out of memory exceptions. On Linux, you can increase the limits by running the following command as `root`: diff --git a/deploy-manage/distributed-architecture.md b/deploy-manage/distributed-architecture.md index a26c9a6de..a1a3cf9ed 100644 --- a/deploy-manage/distributed-architecture.md +++ b/deploy-manage/distributed-architecture.md @@ -16,5 +16,5 @@ The topics in this section provides information about the architecture of {{es}} * [Shard allocation awareness](distributed-architecture/shard-allocation-relocation-recovery/shard-allocation-awareness.md): Learn how to use custom node attributes to distribute shards across different racks or availability zones. -* [Shard request cache](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/shard-request-cache-settings.md): Learn how {{es}} caches search requests to improve performance. +* [Shard request cache](elasticsearch://reference/elasticsearch/configuration-reference/shard-request-cache-settings.md): Learn how {{es}} caches search requests to improve performance. diff --git a/deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles.md b/deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles.md index 3b601a7d7..c3a2178a9 100644 --- a/deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles.md +++ b/deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles.md @@ -5,7 +5,7 @@ mapped_pages: # Node roles [node-roles-overview] -Any time that you start an instance of {{es}}, you are starting a *node*. A collection of connected nodes is called a [cluster](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md). If you are running a single node of {{es}}, then you have a cluster of one node. All nodes know about all the other nodes in the cluster and can forward client requests to the appropriate node. +Any time that you start an instance of {{es}}, you are starting a *node*. A collection of connected nodes is called a [cluster](elasticsearch://reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md). If you are running a single node of {{es}}, then you have a cluster of one node. All nodes know about all the other nodes in the cluster and can forward client requests to the appropriate node. Each node performs one or more roles. Roles control the behavior of the node in the cluster. @@ -61,14 +61,14 @@ Similarly, each master-eligible node maintains the following data on disk: * the index metadata for every index in the cluster, and * the cluster-wide metadata, such as settings and index templates. -Each node checks the contents of its data path at startup. If it discovers unexpected data then it will refuse to start. This is to avoid importing unwanted [dangling indices](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/local-gateway.md#dangling-indices) which can lead to a red cluster health. To be more precise, nodes without the `data` role will refuse to start if they find any shard data on disk at startup, and nodes without both the `master` and `data` roles will refuse to start if they have any index metadata on disk at startup. +Each node checks the contents of its data path at startup. If it discovers unexpected data then it will refuse to start. This is to avoid importing unwanted [dangling indices](elasticsearch://reference/elasticsearch/configuration-reference/local-gateway.md#dangling-indices) which can lead to a red cluster health. To be more precise, nodes without the `data` role will refuse to start if they find any shard data on disk at startup, and nodes without both the `master` and `data` roles will refuse to start if they have any index metadata on disk at startup. It is possible to change the roles of a node by adjusting its `elasticsearch.yml` file and restarting it. This is known as *repurposing* a node. In order to satisfy the checks for unexpected data described above, you must perform some extra steps to prepare a node for repurposing when starting the node without the `data` or `master` roles. -* If you want to repurpose a data node by removing the `data` role then you should first use an [allocation filter](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#cluster-shard-allocation-filtering) to safely migrate all the shard data onto other nodes in the cluster. -* If you want to repurpose a node to have neither the `data` nor `master` roles then it is simplest to start a brand-new node with an empty data path and the desired roles. You may find it safest to use an [allocation filter](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#cluster-shard-allocation-filtering) to migrate the shard data elsewhere in the cluster first. +* If you want to repurpose a data node by removing the `data` role then you should first use an [allocation filter](elasticsearch://reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#cluster-shard-allocation-filtering) to safely migrate all the shard data onto other nodes in the cluster. +* If you want to repurpose a node to have neither the `data` nor `master` roles then it is simplest to start a brand-new node with an empty data path and the desired roles. You may find it safest to use an [allocation filter](elasticsearch://reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#cluster-shard-allocation-filtering) to migrate the shard data elsewhere in the cluster first. -If it is not possible to follow these extra steps then you may be able to use the [`elasticsearch-node repurpose`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/node-tool.md#node-tool-repurpose) tool to delete any excess data that prevents a node from starting. +If it is not possible to follow these extra steps then you may be able to use the [`elasticsearch-node repurpose`](elasticsearch://reference/elasticsearch/command-line-tools/node-tool.md#node-tool-repurpose) tool to delete any excess data that prevents a node from starting. ## Available node roles [node-roles-list] @@ -222,7 +222,7 @@ node.roles: [ data_warm ] Cold data nodes are part of the cold tier. When you no longer need to search time series data regularly, it can move from the warm tier to the cold tier. While still searchable, this tier is typically optimized for lower storage costs rather than search speed. -For better storage savings, you can keep [fully mounted indices](../../tools/snapshot-and-restore/searchable-snapshots.md#fully-mounted) of [{{search-snaps}}](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md) on the cold tier. Unlike regular indices, these fully mounted indices don’t require replicas for reliability. In the event of a failure, they can recover data from the underlying snapshot instead. This potentially halves the local storage needed for the data. A snapshot repository is required to use fully mounted indices in the cold tier. Fully mounted indices are read-only. +For better storage savings, you can keep [fully mounted indices](../../tools/snapshot-and-restore/searchable-snapshots.md#fully-mounted) of [{{search-snaps}}](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md) on the cold tier. Unlike regular indices, these fully mounted indices don’t require replicas for reliability. In the event of a failure, they can recover data from the underlying snapshot instead. This potentially halves the local storage needed for the data. A snapshot repository is required to use fully mounted indices in the cold tier. Fully mounted indices are read-only. Alternatively, you can use the cold tier to store regular indices with replicas instead of using {{search-snaps}}. This lets you store older data on less expensive hardware but doesn’t reduce required disk space compared to the warm tier. diff --git a/deploy-manage/distributed-architecture/discovery-cluster-formation.md b/deploy-manage/distributed-architecture/discovery-cluster-formation.md index ef0a74999..0a1f9bcc2 100644 --- a/deploy-manage/distributed-architecture/discovery-cluster-formation.md +++ b/deploy-manage/distributed-architecture/discovery-cluster-formation.md @@ -30,7 +30,7 @@ The following processes and settings are part of discovery and cluster formation [Cluster fault detection](discovery-cluster-formation/cluster-fault-detection.md) : {{es}} performs health checks to detect and remove faulty nodes. -[Settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/discovery-cluster-formation-settings.md) +[Settings](elasticsearch://reference/elasticsearch/configuration-reference/discovery-cluster-formation-settings.md) : There are settings that enable users to influence the discovery, cluster formation, master election and fault detection processes. diff --git a/deploy-manage/distributed-architecture/discovery-cluster-formation/cluster-fault-detection.md b/deploy-manage/distributed-architecture/discovery-cluster-formation/cluster-fault-detection.md index 4428525e6..6fb315168 100644 --- a/deploy-manage/distributed-architecture/discovery-cluster-formation/cluster-fault-detection.md +++ b/deploy-manage/distributed-architecture/discovery-cluster-formation/cluster-fault-detection.md @@ -7,12 +7,12 @@ mapped_pages: The elected master periodically checks each of the nodes in the cluster to ensure that they are still connected and healthy. Each node in the cluster also periodically checks the health of the elected master. These checks are known respectively as *follower checks* and *leader checks*. -Elasticsearch allows these checks to occasionally fail or timeout without taking any action. It considers a node to be faulty only after a number of consecutive checks have failed. You can control fault detection behavior with [`cluster.fault_detection.*` settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/discovery-cluster-formation-settings.md). +Elasticsearch allows these checks to occasionally fail or timeout without taking any action. It considers a node to be faulty only after a number of consecutive checks have failed. You can control fault detection behavior with [`cluster.fault_detection.*` settings](elasticsearch://reference/elasticsearch/configuration-reference/discovery-cluster-formation-settings.md). If the elected master detects that a node has disconnected, however, this situation is treated as an immediate failure. The master bypasses the timeout and retry setting values and attempts to remove the node from the cluster. Similarly, if a node detects that the elected master has disconnected, this situation is treated as an immediate failure. The node bypasses the timeout and retry settings and restarts its discovery phase to try and find or elect a new master. $$$cluster-fault-detection-filesystem-health$$$ -Additionally, each node periodically verifies that its data path is healthy by writing a small file to disk and then deleting it again. If a node discovers its data path is unhealthy then it is removed from the cluster until the data path recovers. You can control this behavior with the [`monitor.fs.health` settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/discovery-cluster-formation-settings.md). +Additionally, each node periodically verifies that its data path is healthy by writing a small file to disk and then deleting it again. If a node discovers its data path is unhealthy then it is removed from the cluster until the data path recovers. You can control this behavior with the [`monitor.fs.health` settings](elasticsearch://reference/elasticsearch/configuration-reference/discovery-cluster-formation-settings.md). $$$cluster-fault-detection-cluster-state-publishing$$$ The elected master node will also remove nodes from the cluster if nodes are unable to apply an updated cluster state within a reasonable time. The timeout defaults to 2 minutes starting from the beginning of the cluster state update. Refer to [Publishing the cluster state](cluster-state-overview.md#cluster-state-publishing) for a more detailed description. @@ -22,27 +22,27 @@ The elected master node will also remove nodes from the cluster if nodes are una See [*Troubleshooting an unstable cluster*](../../../troubleshoot/elasticsearch/troubleshooting-unstable-cluster.md). -#### Diagnosing `disconnected` nodes [_diagnosing_disconnected_nodes] +#### Diagnosing `disconnected` nodes [_diagnosing_disconnected_nodes] See [Diagnosing `disconnected` nodes](../../../troubleshoot/elasticsearch/troubleshooting-unstable-cluster.md#troubleshooting-unstable-cluster-disconnected). -#### Diagnosing `lagging` nodes [_diagnosing_lagging_nodes] +#### Diagnosing `lagging` nodes [_diagnosing_lagging_nodes] See [Diagnosing `lagging` nodes](../../../troubleshoot/elasticsearch/troubleshooting-unstable-cluster.md#troubleshooting-unstable-cluster-lagging). -#### Diagnosing `follower check retry count exceeded` nodes [_diagnosing_follower_check_retry_count_exceeded_nodes] +#### Diagnosing `follower check retry count exceeded` nodes [_diagnosing_follower_check_retry_count_exceeded_nodes] See [Diagnosing `follower check retry count exceeded` nodes](../../../troubleshoot/elasticsearch/troubleshooting-unstable-cluster.md#troubleshooting-unstable-cluster-follower-check). -#### Diagnosing `ShardLockObtainFailedException` failures [_diagnosing_shardlockobtainfailedexception_failures] +#### Diagnosing `ShardLockObtainFailedException` failures [_diagnosing_shardlockobtainfailedexception_failures] See [Diagnosing `ShardLockObtainFailedException` failures](../../../troubleshoot/elasticsearch/troubleshooting-unstable-cluster.md#troubleshooting-unstable-cluster-shardlockobtainfailedexception). -#### Diagnosing other network disconnections [_diagnosing_other_network_disconnections] +#### Diagnosing other network disconnections [_diagnosing_other_network_disconnections] See [Diagnosing other network disconnections](../../../troubleshoot/elasticsearch/troubleshooting-unstable-cluster.md#troubleshooting-unstable-cluster-network). diff --git a/deploy-manage/distributed-architecture/discovery-cluster-formation/discovery-hosts-providers.md b/deploy-manage/distributed-architecture/discovery-cluster-formation/discovery-hosts-providers.md index 2316afa77..c032a857a 100644 --- a/deploy-manage/distributed-architecture/discovery-cluster-formation/discovery-hosts-providers.md +++ b/deploy-manage/distributed-architecture/discovery-cluster-formation/discovery-hosts-providers.md @@ -19,12 +19,12 @@ Refer to [Troubleshooting discovery](../../../troubleshoot/elasticsearch/discove ## Seed hosts providers [built-in-hosts-providers] -By default the cluster formation module offers two seed hosts providers to configure the list of seed nodes: a *settings*-based and a *file*-based seed hosts provider. It can be extended to support cloud environments and other forms of seed hosts providers via [discovery plugins](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch-plugins/discovery-plugins.md). Seed hosts providers are configured using the `discovery.seed_providers` setting, which defaults to the *settings*-based hosts provider. This setting accepts a list of different providers, allowing you to make use of multiple ways to find the seed hosts for your cluster. +By default the cluster formation module offers two seed hosts providers to configure the list of seed nodes: a *settings*-based and a *file*-based seed hosts provider. It can be extended to support cloud environments and other forms of seed hosts providers via [discovery plugins](elasticsearch://reference/elasticsearch-plugins/discovery-plugins.md). Seed hosts providers are configured using the `discovery.seed_providers` setting, which defaults to the *settings*-based hosts provider. This setting accepts a list of different providers, allowing you to make use of multiple ways to find the seed hosts for your cluster. Each seed hosts provider yields the IP addresses or hostnames of the seed nodes. If it returns any hostnames then these are resolved to IP addresses using a DNS lookup. If a hostname resolves to multiple IP addresses then {{es}} tries to find a seed node at all of these addresses. If the hosts provider does not explicitly give the TCP port of the node by then, it will implicitly use the first port in the port range given by `transport.profiles.default.port`, or by `transport.port` if `transport.profiles.default.port` is not set. The number of concurrent lookups is controlled by `discovery.seed_resolver.max_concurrent_resolvers` which defaults to `10`, and the timeout for each lookup is controlled by `discovery.seed_resolver.timeout` which defaults to `5s`. Note that DNS lookups are subject to [JVM DNS caching](../../deploy/self-managed/networkaddress-cache-ttl.md). -#### Settings-based seed hosts provider [settings-based-hosts-provider] +#### Settings-based seed hosts provider [settings-based-hosts-provider] The settings-based seed hosts provider uses a node setting to configure a static list of the addresses of the seed nodes. These addresses can be given as hostnames or IP addresses; hosts specified as hostnames are resolved to IP addresses during each round of discovery. @@ -42,7 +42,7 @@ discovery.seed_hosts: -#### File-based seed hosts provider [file-based-hosts-provider] +#### File-based seed hosts provider [file-based-hosts-provider] The file-based seed hosts provider configures a list of hosts via an external file. {{es}} reloads this file when it changes, so that the list of seed nodes can change dynamically without needing to restart each node. For example, this gives a convenient mechanism for an {{es}} instance that is run in a Docker container to be dynamically supplied with a list of IP addresses to connect to when those IP addresses may not be known at node startup. @@ -73,18 +73,18 @@ Host names are allowed instead of IP addresses and are resolved by DNS as descri You can also add comments to this file. All comments must appear on their lines starting with `#` (i.e. comments cannot start in the middle of a line). -#### EC2 hosts provider [ec2-hosts-provider] +#### EC2 hosts provider [ec2-hosts-provider] -The [EC2 discovery plugin](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch-plugins/discovery-ec2.md) adds a hosts provider that uses the [AWS API](https://github.com/aws/aws-sdk-java) to find a list of seed nodes. +The [EC2 discovery plugin](elasticsearch://reference/elasticsearch-plugins/discovery-ec2.md) adds a hosts provider that uses the [AWS API](https://github.com/aws/aws-sdk-java) to find a list of seed nodes. -#### Azure Classic hosts provider [azure-classic-hosts-provider] +#### Azure Classic hosts provider [azure-classic-hosts-provider] -The [Azure Classic discovery plugin](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch-plugins/discovery-azure-classic.md) adds a hosts provider that uses the Azure Classic API find a list of seed nodes. +The [Azure Classic discovery plugin](elasticsearch://reference/elasticsearch-plugins/discovery-azure-classic.md) adds a hosts provider that uses the Azure Classic API find a list of seed nodes. -#### Google Compute Engine hosts provider [gce-hosts-provider] +#### Google Compute Engine hosts provider [gce-hosts-provider] -The [GCE discovery plugin](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch-plugins/discovery-gce.md) adds a hosts provider that uses the GCE API find a list of seed nodes. +The [GCE discovery plugin](elasticsearch://reference/elasticsearch-plugins/discovery-gce.md) adds a hosts provider that uses the GCE API find a list of seed nodes. diff --git a/deploy-manage/distributed-architecture/discovery-cluster-formation/modules-discovery-bootstrap-cluster.md b/deploy-manage/distributed-architecture/discovery-cluster-formation/modules-discovery-bootstrap-cluster.md index dd579557b..7e4e85093 100644 --- a/deploy-manage/distributed-architecture/discovery-cluster-formation/modules-discovery-bootstrap-cluster.md +++ b/deploy-manage/distributed-architecture/discovery-cluster-formation/modules-discovery-bootstrap-cluster.md @@ -11,12 +11,12 @@ The initial set of master-eligible nodes is defined in the [`cluster.initial_mas * The [node name](../../deploy/self-managed/important-settings-configuration.md#node-name) of the node. * The node’s hostname if `node.name` is not set, because `node.name` defaults to the node’s hostname. You must use either the fully-qualified hostname or the bare hostname [depending on your system configuration](#modules-discovery-bootstrap-cluster-fqdns). -* The IP address of the node’s [transport publish address](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md#modules-network-binding-publishing), if it is not possible to use the `node.name` of the node. This is normally the IP address to which [`network.host`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md#common-network-settings) resolves but [this can be overridden](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md#advanced-network-settings). +* The IP address of the node’s [transport publish address](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#modules-network-binding-publishing), if it is not possible to use the `node.name` of the node. This is normally the IP address to which [`network.host`](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#common-network-settings) resolves but [this can be overridden](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#advanced-network-settings). * The IP address and port of the node’s publish address, in the form `IP:PORT`, if it is not possible to use the `node.name` of the node and there are multiple nodes sharing a single IP address. Do not set `cluster.initial_master_nodes` on master-ineligible nodes. -::::{important} +::::{important} After the cluster has formed, remove the `cluster.initial_master_nodes` setting from each node’s configuration and never set it again for this cluster. Do not configure this setting on nodes joining an existing cluster. Do not configure this setting on nodes which are restarting. Do not configure this setting when performing a full-cluster restart. If you leave `cluster.initial_master_nodes` in place once the cluster has formed then there is a risk that a future misconfiguration may result in bootstrapping a new cluster alongside your existing cluster. It may not be possible to recover from this situation without losing data. @@ -39,7 +39,7 @@ cluster.initial_master_nodes: - master-c ``` -::::{important} +::::{important} You must set `cluster.initial_master_nodes` to the same list of nodes on each node on which it is set in order to be sure that only a single cluster forms during bootstrapping. If `cluster.initial_master_nodes` varies across the nodes on which it is set then you may bootstrap multiple clusters. It is usually not possible to recover from this situation without losing data. :::: @@ -63,7 +63,7 @@ This message shows the node names `master-a.example.com` and `master-b.example.c ## Choosing a cluster name [bootstrap-cluster-name] -The [`cluster.name`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/miscellaneous-cluster-settings.md#cluster-name) setting enables you to create multiple clusters which are separated from each other. Nodes verify that they agree on their cluster name when they first connect to each other, and Elasticsearch will only form a cluster from nodes that all have the same cluster name. The default value for the cluster name is `elasticsearch`, but it is recommended to change this to reflect the logical name of the cluster. +The [`cluster.name`](elasticsearch://reference/elasticsearch/configuration-reference/miscellaneous-cluster-settings.md#cluster-name) setting enables you to create multiple clusters which are separated from each other. Nodes verify that they agree on their cluster name when they first connect to each other, and Elasticsearch will only form a cluster from nodes that all have the same cluster name. The default value for the cluster name is `elasticsearch`, but it is recommended to change this to reflect the logical name of the cluster. ## Auto-bootstrapping in development mode [bootstrap-auto-bootstrap] @@ -84,14 +84,14 @@ Once an {{es}} node has joined an existing cluster, or bootstrapped a new cluste If you intended to add a node into an existing cluster but instead bootstrapped a separate single-node cluster then you must start again: 1. Shut down the node. -2. Completely wipe the node by deleting the contents of its [data folder](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md#data-path). +2. Completely wipe the node by deleting the contents of its [data folder](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md#data-path). 3. Configure `discovery.seed_hosts` or `discovery.seed_providers` and other relevant discovery settings. Ensure `cluster.initial_master_nodes` is not set on any node. 4. Restart the node and verify that it joins the existing cluster rather than forming its own one-node cluster. If you intended to form a new multi-node cluster but instead bootstrapped a collection of single-node clusters then you must start again: 1. Shut down all the nodes. -2. Completely wipe each node by deleting the contents of their [data folders](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md#data-path). +2. Completely wipe each node by deleting the contents of their [data folders](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md#data-path). 3. Configure `cluster.initial_master_nodes` as described above. 4. Configure `discovery.seed_hosts` or `discovery.seed_providers` and other relevant discovery settings. 5. Restart all the nodes and verify that they have formed a single cluster. diff --git a/deploy-manage/distributed-architecture/discovery-cluster-formation/modules-discovery-quorums.md b/deploy-manage/distributed-architecture/discovery-cluster-formation/modules-discovery-quorums.md index 4481774ef..eba853009 100644 --- a/deploy-manage/distributed-architecture/discovery-cluster-formation/modules-discovery-quorums.md +++ b/deploy-manage/distributed-architecture/discovery-cluster-formation/modules-discovery-quorums.md @@ -24,7 +24,7 @@ After a master-eligible node has joined or left the cluster the elected master m ## Master elections [_master_elections] -Elasticsearch uses an election process to agree on an elected master node, both at startup and if the existing elected master fails. Any master-eligible node can start an election, and normally the first election that takes place will succeed. Elections only usually fail when two nodes both happen to start their elections at about the same time, so elections are scheduled randomly on each node to reduce the probability of this happening. Nodes will retry elections until a master is elected, backing off on failure, so that eventually an election will succeed (with arbitrarily high probability). The scheduling of master elections are controlled by the [master election settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/discovery-cluster-formation-settings.md#master-election-settings). +Elasticsearch uses an election process to agree on an elected master node, both at startup and if the existing elected master fails. Any master-eligible node can start an election, and normally the first election that takes place will succeed. Elections only usually fail when two nodes both happen to start their elections at about the same time, so elections are scheduled randomly on each node to reduce the probability of this happening. Nodes will retry elections until a master is elected, backing off on failure, so that eventually an election will succeed (with arbitrarily high probability). The scheduling of master elections are controlled by the [master election settings](elasticsearch://reference/elasticsearch/configuration-reference/discovery-cluster-formation-settings.md#master-election-settings). ## Cluster maintenance, rolling restarts and migrations [_cluster_maintenance_rolling_restarts_and_migrations] diff --git a/deploy-manage/distributed-architecture/discovery-cluster-formation/modules-discovery-voting.md b/deploy-manage/distributed-architecture/discovery-cluster-formation/modules-discovery-voting.md index 9c0c9229e..4de16d881 100644 --- a/deploy-manage/distributed-architecture/discovery-cluster-formation/modules-discovery-voting.md +++ b/deploy-manage/distributed-architecture/discovery-cluster-formation/modules-discovery-voting.md @@ -32,7 +32,7 @@ The current voting configuration is not necessarily the same as the set of all a Larger voting configurations are usually more resilient, so Elasticsearch normally prefers to add master-eligible nodes to the voting configuration after they join the cluster. Similarly, if a node in the voting configuration leaves the cluster and there is another master-eligible node in the cluster that is not in the voting configuration then it is preferable to swap these two nodes over. The size of the voting configuration is thus unchanged but its resilience increases. -It is not so straightforward to automatically remove nodes from the voting configuration after they have left the cluster. Different strategies have different benefits and drawbacks, so the right choice depends on how the cluster will be used. You can control whether the voting configuration automatically shrinks by using the [`cluster.auto_shrink_voting_configuration` setting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/discovery-cluster-formation-settings.md). +It is not so straightforward to automatically remove nodes from the voting configuration after they have left the cluster. Different strategies have different benefits and drawbacks, so the right choice depends on how the cluster will be used. You can control whether the voting configuration automatically shrinks by using the [`cluster.auto_shrink_voting_configuration` setting](elasticsearch://reference/elasticsearch/configuration-reference/discovery-cluster-formation-settings.md). ::::{note} If `cluster.auto_shrink_voting_configuration` is set to `true` (which is the default and recommended value) and there are at least three master-eligible nodes in the cluster, Elasticsearch remains capable of processing cluster state updates as long as all but one of its master-eligible nodes are healthy. diff --git a/deploy-manage/distributed-architecture/kibana-tasks-management.md b/deploy-manage/distributed-architecture/kibana-tasks-management.md index 624348b24..ac8a95fa2 100644 --- a/deploy-manage/distributed-architecture/kibana-tasks-management.md +++ b/deploy-manage/distributed-architecture/kibana-tasks-management.md @@ -11,7 +11,7 @@ mapped_pages: * **Scaling**: Multiple {{kib}} instances can read from and update the same task queue in {{es}}, allowing the work load to be distributed across instances. If a {{kib}} instance no longer has capacity to run tasks, you can increase capacity by adding additional {{kib}} instances. * **Load Balancing**: Task Manager is equipped with a reactive self-healing mechanism, which allows it to reduce the amount of work it executes in reaction to an increased load related error rate in {{es}}. Additionally, when Task Manager experiences an increase in recurring tasks, it attempts to space out the work to better balance the load. -::::{important} +::::{important} Task definitions for alerts and actions are stored in the index called `.kibana_task_manager`. You must have at least one replica of this index for production deployments. @@ -22,11 +22,11 @@ If you lose this index, all scheduled alerts and actions are lost. -## Running background tasks [task-manager-background-tasks] +## Running background tasks [task-manager-background-tasks] {{kib}} background tasks are managed as follows: -* An {{es}} task index is polled for overdue tasks at 3-second intervals. You can change this interval using the [`xpack.task_manager.poll_interval`](asciidocalypse://docs/kibana/docs/reference/configuration-reference/task-manager-settings.md#task-manager-settings) setting. +* An {{es}} task index is polled for overdue tasks at 3-second intervals. You can change this interval using the [`xpack.task_manager.poll_interval`](kibana://reference/configuration-reference/task-manager-settings.md#task-manager-settings) setting. * Tasks are claimed by updating them in the {{es}} index, using optimistic concurrency control to prevent conflicts. Each {{kib}} instance can run a maximum of 10 concurrent tasks, so a maximum of 10 tasks are claimed each interval. * Tasks are run on the {{kib}} server. * Task Manager ensures that tasks: @@ -36,14 +36,14 @@ If you lose this index, all scheduled alerts and actions are lost. * Are rescheduled to run again at a future point in time (if configured to do so) -::::{important} +::::{important} It is possible for tasks to run late or at an inconsistent schedule. This is usually a symptom of the specific usage or scaling strategy of the cluster in question. To address these issues, tweak the {{kib}} Task Manager settings or the cluster scaling strategy to better suit the unique use case. -For details on the settings that can influence the performance and throughput of Task Manager, see [Task Manager Settings](asciidocalypse://docs/kibana/docs/reference/configuration-reference/task-manager-settings.md). +For details on the settings that can influence the performance and throughput of Task Manager, see [Task Manager Settings](kibana://reference/configuration-reference/task-manager-settings.md). For detailed troubleshooting guidance, see [Troubleshooting](../../troubleshoot/kibana/task-manager.md). @@ -51,19 +51,19 @@ For detailed troubleshooting guidance, see [Troubleshooting](../../troubleshoot/ -## Deployment considerations [_deployment_considerations] +## Deployment considerations [_deployment_considerations] {{es}} and {{kib}} instances use the system clock to determine the current time. To ensure schedules are triggered when expected, synchronize the clocks of all nodes in the cluster using a time service such as [Network Time Protocol](http://www.ntp.org/). -## Scaling guidance [task-manager-scaling-guidance] +## Scaling guidance [task-manager-scaling-guidance] How you deploy {{kib}} largely depends on your use case. Predicting the throughout a deployment might require to support Task Management is difficult because features can schedule an unpredictable number of tasks at a variety of scheduled cadences. However, there is a relatively straight forward method you can follow to produce a rough estimate based on your expected usage. -### Default scale [task-manager-default-scaling] +### Default scale [task-manager-default-scaling] By default, {{kib}} polls for tasks at a rate of 10 tasks every 3 seconds. This means that you can expect a single {{kib}} instance to support up to 200 *tasks per minute* (`200/tpm`). @@ -74,21 +74,21 @@ By [estimating a rough throughput requirement](#task-manager-rough-throughput-es For details on monitoring the health of {{kib}} Task Manager, follow the guidance in [Health monitoring](../monitor/kibana-task-manager-health-monitoring.md). -### Scaling horizontally [task-manager-scaling-horizontally] +### Scaling horizontally [task-manager-scaling-horizontally] At times, the sustainable approach might be to expand the throughput of your cluster by provisioning additional {{kib}} instances. By default, each additional {{kib}} instance will add an additional 10 tasks that your cluster can run concurrently, but you can also scale each {{kib}} instance vertically, if your diagnosis indicates that they can handle the additional workload. -### Scaling vertically [task-manager-scaling-vertically] +### Scaling vertically [task-manager-scaling-vertically] Other times it, might be preferable to increase the throughput of individual {{kib}} instances. -Tweak the capacity with the [`xpack.task_manager.capacity`](asciidocalypse://docs/kibana/docs/reference/configuration-reference/task-manager-settings.md#task-manager-settings) setting, which enables each {{kib}} instance to pull a higher number of tasks per interval. This setting can impact the performance of each instance as the workload will be higher. +Tweak the capacity with the [`xpack.task_manager.capacity`](kibana://reference/configuration-reference/task-manager-settings.md#task-manager-settings) setting, which enables each {{kib}} instance to pull a higher number of tasks per interval. This setting can impact the performance of each instance as the workload will be higher. -Tweak the poll interval with the [`xpack.task_manager.poll_interval`](asciidocalypse://docs/kibana/docs/reference/configuration-reference/task-manager-settings.md#task-manager-settings) setting, which enables each {{kib}} instance to pull scheduled tasks at a higher rate. This setting can impact the performance of the {{es}} cluster as the workload will be higher. +Tweak the poll interval with the [`xpack.task_manager.poll_interval`](kibana://reference/configuration-reference/task-manager-settings.md#task-manager-settings) setting, which enables each {{kib}} instance to pull scheduled tasks at a higher rate. This setting can impact the performance of the {{es}} cluster as the workload will be higher. -### Choosing a scaling strategy [task-manager-choosing-scaling-strategy] +### Choosing a scaling strategy [task-manager-choosing-scaling-strategy] Each scaling strategy comes with its own considerations, and the appropriate strategy largely depends on your use case. @@ -106,7 +106,7 @@ Task Manager, like the rest of the Elastic Stack, is designed to scale horizonta Scaling horizontally requires a higher degree of coordination between {{kib}} instances. One way Task Manager coordinates with other instances is by delaying its polling schedule to avoid conflicts with other instances. By using [health monitoring](../monitor/kibana-task-manager-health-monitoring.md) to evaluate the [date of the `last_polling_delay`](../../troubleshoot/kibana/task-manager.md#task-manager-health-evaluate-the-runtime) across a deployment, you can estimate the frequency at which Task Manager resets its delay mechanism. A higher frequency suggests {{kib}} instances conflict at a high rate, which you can address by scaling vertically rather than horizontally, reducing the required coordination. -### Rough throughput estimation [task-manager-rough-throughput-estimation] +### Rough throughput estimation [task-manager-rough-throughput-estimation] Predicting the required throughput a deployment might need to support Task Management is difficult, as features can schedule an unpredictable number of tasks at a variety of scheduled cadences. However, a rough lower bound can be estimated, which is then used as a guide. @@ -115,9 +115,9 @@ Throughput is best thought of as a measurements in tasks per minute. A default {{kib}} instance can support up to `200/tpm`. -#### Automatic estimation [_automatic_estimation] +#### Automatic estimation [_automatic_estimation] -::::{warning} +::::{warning} This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. :::: @@ -130,7 +130,7 @@ We recommend provisioning enough {{kib}} instances to ensure a buffer between th We recommend provisioning at least as many {{kib}} instances as proposed by `proposed.provisioned_kibana`, but keep in mind that this number is based on the estimated required throughput, which is based on average historical performance, and cannot accurately predict future requirements. -::::{warning} +::::{warning} Automatic capacity estimation is performed by each {{kib}} instance independently. This estimation is performed by observing the task throughput in that instance, the number of {{kib}} instances executing tasks at that moment in time, and the recurring workload in {{es}}. If a {{kib}} instance is idle at the moment of capacity estimation, the number of active {{kib}} instances might be miscounted and the available throughput miscalculated. @@ -141,7 +141,7 @@ When evaluating the proposed {{kib}} instance number under `proposed.provisioned -#### Manual estimation [_manual_estimation] +#### Manual estimation [_manual_estimation] By [evaluating the workload](../../troubleshoot/kibana/task-manager.md#task-manager-health-evaluate-the-workload), you can make a rough estimate as to the required throughput as a *tasks per minute* measurement. diff --git a/deploy-manage/distributed-architecture/reading-and-writing-documents.md b/deploy-manage/distributed-architecture/reading-and-writing-documents.md index 559d0290e..ad038228e 100644 --- a/deploy-manage/distributed-architecture/reading-and-writing-documents.md +++ b/deploy-manage/distributed-architecture/reading-and-writing-documents.md @@ -41,7 +41,7 @@ These indexing stages (coordinating, primary, and replica) are sequential. To en Many things can go wrong during indexing — disks can get corrupted, nodes can be disconnected from each other, or some configuration mistake could cause an operation to fail on a replica despite it being successful on the primary. These are infrequent but the primary has to respond to them. -In the case that the primary itself fails, the node hosting the primary will send a message to the master about it. The indexing operation will wait (up to 1 minute, by [default](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md)) for the master to promote one of the replicas to be a new primary. The operation will then be forwarded to the new primary for processing. Note that the master also monitors the health of the nodes and may decide to proactively demote a primary. This typically happens when the node holding the primary is isolated from the cluster by a networking issue. See [here](#demoted-primary) for more details. +In the case that the primary itself fails, the node hosting the primary will send a message to the master about it. The indexing operation will wait (up to 1 minute, by [default](elasticsearch://reference/elasticsearch/index-settings/index-modules.md)) for the master to promote one of the replicas to be a new primary. The operation will then be forwarded to the new primary for processing. Note that the master also monitors the health of the nodes and may decide to proactively demote a primary. This typically happens when the node holding the primary is isolated from the cluster by a networking issue. See [here](#demoted-primary) for more details. Once the operation has been successfully performed on the primary, the primary has to deal with potential failures when executing it on the replica shards. This may be caused by an actual failure on the replica or due to a network issue preventing the operation from reaching the replica (or preventing the replica from responding). All of these share the same end result: a replica which is part of the in-sync replica set misses an operation that is about to be acknowledged. In order to avoid violating the invariant, the primary sends a message to the master requesting that the problematic shard be removed from the in-sync replica set. Only once removal of the shard has been acknowledged by the master does the primary acknowledge the operation. Note that the master will also instruct another node to start building a new shard copy in order to restore the system to a healthy state. @@ -62,7 +62,7 @@ Reads in Elasticsearch can be very lightweight lookups by ID or a heavy search r When a read request is received by a node, that node is responsible for forwarding it to the nodes that hold the relevant shards, collating the responses, and responding to the client. We call that node the *coordinating node* for that request. The basic flow is as follows: 1. Resolve the read requests to the relevant shards. Note that since most searches will be sent to one or more indices, they typically need to read from multiple shards, each representing a different subset of the data. -2. Select an active copy of each relevant shard, from the shard replication group. This can be either the primary or a replica. By default, {{es}} uses [adaptive replica selection](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/search-shard-routing.md#search-adaptive-replica) to select the shard copies. +2. Select an active copy of each relevant shard, from the shard replication group. This can be either the primary or a replica. By default, {{es}} uses [adaptive replica selection](elasticsearch://reference/elasticsearch/rest-apis/search-shard-routing.md#search-adaptive-replica) to select the shard copies. 3. Send shard level read requests to the selected copies. 4. Combine the results and respond. Note that in the case of get by ID look up, only one shard is relevant and this step can be skipped. diff --git a/deploy-manage/distributed-architecture/shard-allocation-relocation-recovery.md b/deploy-manage/distributed-architecture/shard-allocation-relocation-recovery.md index 6fe8da97b..653dec505 100644 --- a/deploy-manage/distributed-architecture/shard-allocation-relocation-recovery.md +++ b/deploy-manage/distributed-architecture/shard-allocation-relocation-recovery.md @@ -9,7 +9,7 @@ Each [index](../../manage-data/data-store/index-basics.md) in Elasticsearch is d A cluster can contain multiple copies of a shard. Each shard has one distinguished shard copy called the *primary*, and zero or more non-primary copies called *replicas*. The primary shard copy serves as the main entry point for all indexing operations. The operations on the primary shard copy are then forwarded to its replicas. -Replicas maintain redundant copies of your data across the [nodes](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md) in your cluster, protecting against hardware failure and increasing capacity to serve read requests like searching or retrieving a document. If the primary shard copy fails, then a replica is promoted to primary and takes over the primary’s responsibilities. +Replicas maintain redundant copies of your data across the [nodes](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md) in your cluster, protecting against hardware failure and increasing capacity to serve read requests like searching or retrieving a document. If the primary shard copy fails, then a replica is promoted to primary and takes over the primary’s responsibilities. Over the course of normal operation, Elasticsearch allocates shard copies to nodes, relocates shard copies across nodes to balance the cluster or satisfy new allocation constraints, and recovers shards to initialize new copies. In this topic, you’ll learn how these operations work and how you can control them. @@ -30,7 +30,7 @@ By default, the primary and replica shard copies for an index can be allocated t You can control how shard copies are allocated using the following settings: -* [Cluster-level shard allocation settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md): Use these settings to control how shard copies are allocated and balanced across the entire cluster. For example, you might want to [allocate nodes availability zones](shard-allocation-relocation-recovery/shard-allocation-awareness.md), or prevent certain nodes from being used so you can perform maintenance. +* [Cluster-level shard allocation settings](elasticsearch://reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md): Use these settings to control how shard copies are allocated and balanced across the entire cluster. For example, you might want to [allocate nodes availability zones](shard-allocation-relocation-recovery/shard-allocation-awareness.md), or prevent certain nodes from being used so you can perform maintenance. * [Index-level shard allocation settings](shard-allocation-relocation-recovery/index-level-shard-allocation.md): Use these settings to control how the shard copies for a specific index are allocated. For example, you might want to allocate an index to a node in a specific data tier, or to an node with specific attributes. @@ -67,8 +67,8 @@ You can determine the cause of a shard recovery using the [recovery](https://www To control how shards are recovered, for example the resources that can be used by recovery operations, and which indices should be prioritized for recovery, you can adjust the following settings: -* [Index recovery settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/index-recovery-settings.md) -* [Cluster-level shard allocation settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md) +* [Index recovery settings](elasticsearch://reference/elasticsearch/configuration-reference/index-recovery-settings.md) +* [Cluster-level shard allocation settings](elasticsearch://reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md) * [Index-level shard allocation settings](shard-allocation-relocation-recovery/index-level-shard-allocation.md), including [delayed allocation](shard-allocation-relocation-recovery/delaying-allocation-when-node-leaves.md) and [index recovery prioritization](shard-allocation-relocation-recovery/index-level-shard-allocation.md) Shard recovery operations also respect general shard allocation settings. @@ -91,7 +91,7 @@ When a shard copy is relocated, it is created as a new shard copy on the target ### Adjust shard relocation settings [_adjust_shard_relocation_settings] -You can control how and when shard copies are relocated. For example, you can adjust the rebalancing settings that control when shard copies are relocated to balance the cluster, or the high watermark for disk-based shard allocation that can trigger relocation. These settings are part of the [cluster-level shard allocation settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md). +You can control how and when shard copies are relocated. For example, you can adjust the rebalancing settings that control when shard copies are relocated to balance the cluster, or the high watermark for disk-based shard allocation that can trigger relocation. These settings are part of the [cluster-level shard allocation settings](elasticsearch://reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md). Shard relocation operations also respect shard allocation and recovery settings. diff --git a/deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/delaying-allocation-when-node-leaves.md b/deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/delaying-allocation-when-node-leaves.md index 29e21ced5..3bffc7e50 100644 --- a/deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/delaying-allocation-when-node-leaves.md +++ b/deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/delaying-allocation-when-node-leaves.md @@ -13,7 +13,7 @@ When a node leaves the cluster for whatever reason, intentional or otherwise, th These actions are intended to protect the cluster against data loss by ensuring that every shard is fully replicated as soon as possible. -Even though we throttle concurrent recoveries both at the [node level](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/index-recovery-settings.md) and at the [cluster level](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#cluster-shard-allocation-settings), this shard-shuffle can still put a lot of extra load on the cluster which may not be necessary if the missing node is likely to return soon. Imagine this scenario: +Even though we throttle concurrent recoveries both at the [node level](elasticsearch://reference/elasticsearch/configuration-reference/index-recovery-settings.md) and at the [cluster level](elasticsearch://reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#cluster-shard-allocation-settings), this shard-shuffle can still put a lot of extra load on the cluster which may not be necessary if the missing node is likely to return soon. Imagine this scenario: * Node 5 loses network connectivity. * The master promotes a replica shard to primary for each primary that was on Node 5. @@ -47,7 +47,7 @@ With delayed allocation enabled, the above scenario changes to look like this: * Node 5 returns after a few minutes, before the `timeout` expires. * The missing replicas are re-allocated to Node 5 (and sync-flushed shards recover almost immediately). -::::{note} +::::{note} This setting will not affect the promotion of replicas to primaries, nor will it affect the assignment of replicas that have not been assigned previously. In particular, delayed allocation does not come into effect after a full cluster restart. Also, in case of a master failover situation, elapsed delay time is forgotten (i.e. reset to the full initial delay). :::: diff --git a/deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/shard-allocation-awareness.md b/deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/shard-allocation-awareness.md index 495199f72..d91216a9b 100644 --- a/deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/shard-allocation-awareness.md +++ b/deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/shard-allocation-awareness.md @@ -23,7 +23,7 @@ Learn more about [designing resilient clusters](../../production-guidance/availa To enable shard allocation awareness: -1. Specify the location of each node with a [custom node attribute](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md#custom-node-attributes). For example, if you want Elasticsearch to distribute shards across different racks, you might use an awareness attribute called `rack_id`. +1. Specify the location of each node with a [custom node attribute](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md#custom-node-attributes). For example, if you want Elasticsearch to distribute shards across different racks, you might use an awareness attribute called `rack_id`. You can set custom attributes in two ways: diff --git a/deploy-manage/maintenance/add-and-remove-elasticsearch-nodes.md b/deploy-manage/maintenance/add-and-remove-elasticsearch-nodes.md index e71a68179..175a087ea 100644 --- a/deploy-manage/maintenance/add-and-remove-elasticsearch-nodes.md +++ b/deploy-manage/maintenance/add-and-remove-elasticsearch-nodes.md @@ -16,7 +16,7 @@ If you are running a single instance of {{es}}, you have a cluster of one node. :alt: A cluster with one node and three primary shards ::: -You add nodes to a cluster to increase its capacity and reliability. By default, a node is both a data node and eligible to be elected as the master node that controls the cluster. You can also configure a new node for a specific purpose, such as handling ingest requests. For more information, see [Nodes](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md). +You add nodes to a cluster to increase its capacity and reliability. By default, a node is both a data node and eligible to be elected as the master node that controls the cluster. You can also configure a new node for a specific purpose, such as handling ingest requests. For more information, see [Nodes](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md). When you add more nodes to a cluster, it automatically allocates replica shards. When all primary and replica shards are active, the cluster state changes to green. @@ -47,11 +47,11 @@ When {{es}} starts for the first time, the security auto-configuration process b Before enrolling a new node, additional actions such as binding to an address other than `localhost` or satisfying bootstrap checks are typically necessary in production clusters. During that time, an auto-generated enrollment token could expire, which is why enrollment tokens aren’t generated automatically. -Additionally, only nodes on the same host can join the cluster without additional configuration. If you want nodes from another host to join your cluster, you need to set `transport.host` to a [supported value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md#network-interface-values) (such as uncommenting the suggested value of `0.0.0.0`), or an IP address that’s bound to an interface where other hosts can reach it. Refer to [transport settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md#transport-settings) for more information. +Additionally, only nodes on the same host can join the cluster without additional configuration. If you want nodes from another host to join your cluster, you need to set `transport.host` to a [supported value](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#network-interface-values) (such as uncommenting the suggested value of `0.0.0.0`), or an IP address that’s bound to an interface where other hosts can reach it. Refer to [transport settings](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#transport-settings) for more information. To enroll new nodes in your cluster, create an enrollment token with the `elasticsearch-create-enrollment-token` tool on any existing node in your cluster. You can then start a new node with the `--enrollment-token` parameter so that it joins an existing cluster. -1. In a separate terminal from where {{es}} is running, navigate to the directory where you installed {{es}} and run the [`elasticsearch-create-enrollment-token`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/create-enrollment-token.md) tool to generate an enrollment token for your new nodes. +1. In a separate terminal from where {{es}} is running, navigate to the directory where you installed {{es}} and run the [`elasticsearch-create-enrollment-token`](elasticsearch://reference/elasticsearch/command-line-tools/create-enrollment-token.md) tool to generate an enrollment token for your new nodes. ```sh bin\elasticsearch-create-enrollment-token -s node @@ -73,7 +73,7 @@ To enroll new nodes in your cluster, create an enrollment token with the `elasti 3. Repeat the previous step for any new nodes that you want to enroll. -For more information about discovery and shard allocation, refer to [*Discovery and cluster formation*](../distributed-architecture/discovery-cluster-formation.md) and [Cluster-level shard allocation and routing settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md). +For more information about discovery and shard allocation, refer to [*Discovery and cluster formation*](../distributed-architecture/discovery-cluster-formation.md) and [Cluster-level shard allocation and routing settings](elasticsearch://reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md). ## Master-eligible nodes [add-elasticsearch-nodes-master-eligible] @@ -122,7 +122,7 @@ Adding an exclusion for a node creates an entry for that node in the voting conf GET /_cluster/state?filter_path=metadata.cluster_coordination.voting_config_exclusions ``` -This list is limited in size by the `cluster.max_voting_config_exclusions` setting, which defaults to `10`. See [Discovery and cluster formation settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/discovery-cluster-formation-settings.md). Since voting configuration exclusions are persistent and limited in number, they must be cleaned up. Normally an exclusion is added when performing some maintenance on the cluster, and the exclusions should be cleaned up when the maintenance is complete. Clusters should have no voting configuration exclusions in normal operation. +This list is limited in size by the `cluster.max_voting_config_exclusions` setting, which defaults to `10`. See [Discovery and cluster formation settings](elasticsearch://reference/elasticsearch/configuration-reference/discovery-cluster-formation-settings.md). Since voting configuration exclusions are persistent and limited in number, they must be cleaned up. Normally an exclusion is added when performing some maintenance on the cluster, and the exclusions should be cleaned up when the maintenance is complete. Clusters should have no voting configuration exclusions in normal operation. If a node is excluded from the voting configuration because it is to be shut down permanently, its exclusion can be removed after it is shut down and removed from the cluster. Exclusions can also be cleared if they were created in error or were only required temporarily by specifying `?wait_for_removal=false`. diff --git a/deploy-manage/maintenance/ece/start-stop-routing-requests.md b/deploy-manage/maintenance/ece/start-stop-routing-requests.md index c4c0367d4..9130380f3 100644 --- a/deploy-manage/maintenance/ece/start-stop-routing-requests.md +++ b/deploy-manage/maintenance/ece/start-stop-routing-requests.md @@ -16,7 +16,7 @@ The {{ecloud}} proxy routes HTTP requests to its deployment’s individual produ It might be helpful to temporarily block upstream requests in order to protect some or all instances or products within your deployment. For example, you might stop request routing in the following cases: * If another team within your company starts streaming new data into your production {{integrations-server}} without previous load testing, both it and {{es}} might experience performance issues. You might consider stopping routing requests on all {{integrations-server}} instances in order to protect your downstream {{es}} instance. -* If {{es}} is being overwhelmed by upstream requests, it might experience increased response times or even become unresponsive. This might impact your ability to resize components in your deployment and increase the duration of pending plans or increase the chance of plan changes failing. Because every {{es}} node is an [implicit coordinating node](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md), you should stop routing requests across all {{es}} nodes to completely block upstream traffic. +* If {{es}} is being overwhelmed by upstream requests, it might experience increased response times or even become unresponsive. This might impact your ability to resize components in your deployment and increase the duration of pending plans or increase the chance of plan changes failing. Because every {{es}} node is an [implicit coordinating node](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md), you should stop routing requests across all {{es}} nodes to completely block upstream traffic. ## Considerations [request-routing-considerations] diff --git a/deploy-manage/maintenance/start-stop-services/full-cluster-restart-rolling-restart-procedures.md b/deploy-manage/maintenance/start-stop-services/full-cluster-restart-rolling-restart-procedures.md index a4674e858..31bd0123c 100644 --- a/deploy-manage/maintenance/start-stop-services/full-cluster-restart-rolling-restart-procedures.md +++ b/deploy-manage/maintenance/start-stop-services/full-cluster-restart-rolling-restart-procedures.md @@ -11,14 +11,14 @@ applies_to: There may be [situations where you want to perform a full-cluster restart](../../security/secure-cluster-communications.md) or a rolling restart. In the case of [full-cluster restart](#restart-cluster-full), you shut down and restart all the nodes in the cluster while in the case of [rolling restart](#restart-cluster-rolling), you shut down only one node at a time, so the service remains uninterrupted. ::::{warning} -Nodes exceeding the low watermark threshold will be slow to restart. Reduce the disk usage below the [low watermark](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#cluster-routing-watermark-low) before restarting nodes. +Nodes exceeding the low watermark threshold will be slow to restart. Reduce the disk usage below the [low watermark](elasticsearch://reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#cluster-routing-watermark-low) before restarting nodes. :::: ## Full-cluster restart [restart-cluster-full] 1. **Disable shard allocation.** - When you shut down a data node, the allocation process waits for `index.unassigned.node_left.delayed_timeout` (by default, one minute) before starting to replicate the shards on that node to other nodes in the cluster, which can involve a lot of I/O. Since the node is shortly going to be restarted, this I/O is unnecessary. You can avoid racing the clock by [disabling allocation](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#cluster-routing-allocation-enable) of replicas before shutting down [data nodes](../../distributed-architecture/clusters-nodes-shards/node-roles.md#data-node-role): + When you shut down a data node, the allocation process waits for `index.unassigned.node_left.delayed_timeout` (by default, one minute) before starting to replicate the shards on that node to other nodes in the cluster, which can involve a lot of I/O. Since the node is shortly going to be restarted, this I/O is unnecessary. You can avoid racing the clock by [disabling allocation](elasticsearch://reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#cluster-routing-allocation-enable) of replicas before shutting down [data nodes](../../distributed-architecture/clusters-nodes-shards/node-roles.md#data-node-role): ```console PUT _cluster/settings @@ -29,7 +29,7 @@ Nodes exceeding the low watermark threshold will be slow to restart. Reduce the } ``` - You can also consider [gateway settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/local-gateway.md) when restarting large clusters to reduce initial strain while nodes are processing [through discovery](../../distributed-architecture/discovery-cluster-formation.md). + You can also consider [gateway settings](elasticsearch://reference/elasticsearch/configuration-reference/local-gateway.md) when restarting large clusters to reduce initial strain while nodes are processing [through discovery](../../distributed-architecture/discovery-cluster-formation.md). 2. **Stop indexing and perform a flush.** Performing a [flush](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-flush) speeds up shard recovery. @@ -124,7 +124,7 @@ Nodes exceeding the low watermark threshold will be slow to restart. Reduce the ## Rolling restart [restart-cluster-rolling] 1. **Disable shard allocation.** - When you shut down a data node, the allocation process waits for `index.unassigned.node_left.delayed_timeout` (by default, one minute) before starting to replicate the shards on that node to other nodes in the cluster, which can involve a lot of I/O. Since the node is shortly going to be restarted, this I/O is unnecessary. You can avoid racing the clock by [disabling allocation](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#cluster-routing-allocation-enable) of replicas before shutting down [data nodes](../../distributed-architecture/clusters-nodes-shards/node-roles.md#data-node-role): + When you shut down a data node, the allocation process waits for `index.unassigned.node_left.delayed_timeout` (by default, one minute) before starting to replicate the shards on that node to other nodes in the cluster, which can involve a lot of I/O. Since the node is shortly going to be restarted, this I/O is unnecessary. You can avoid racing the clock by [disabling allocation](elasticsearch://reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#cluster-routing-allocation-enable) of replicas before shutting down [data nodes](../../distributed-architecture/clusters-nodes-shards/node-roles.md#data-node-role): ```console PUT _cluster/settings @@ -135,7 +135,7 @@ Nodes exceeding the low watermark threshold will be slow to restart. Reduce the } ``` - You can also consider [gateway settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/local-gateway.md) when restarting large clusters to reduce initial strain while nodes are processing [through discovery](../../distributed-architecture/discovery-cluster-formation.md). + You can also consider [gateway settings](elasticsearch://reference/elasticsearch/configuration-reference/local-gateway.md) when restarting large clusters to reduce initial strain while nodes are processing [through discovery](../../distributed-architecture/discovery-cluster-formation.md). 2. **Stop non-essential indexing and perform a flush.** (Optional) While you can continue indexing during the rolling restart, shard recovery can be faster if you temporarily stop non-essential indexing and perform a [flush](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-flush). diff --git a/deploy-manage/maintenance/start-stop-services/start-stop-kibana.md b/deploy-manage/maintenance/start-stop-services/start-stop-kibana.md index b92b57f5f..9390bc9c0 100644 --- a/deploy-manage/maintenance/start-stop-services/start-stop-kibana.md +++ b/deploy-manage/maintenance/start-stop-services/start-stop-kibana.md @@ -30,8 +30,8 @@ If this is the first time you’re starting {{kib}}, this command generates a un 2. In your browser, paste the enrollment token that was generated in the terminal when you started {{es}}, and then click the button to connect your {{kib}} instance with {{es}}. 3. Log in to {{kib}} as the `elastic` user with the password that was generated when you started {{es}}. -::::{note} -If you need to reset the password for the `elastic` user or other built-in users, run the [`elasticsearch-reset-password`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/reset-password.md) tool. To generate new enrollment tokens for {{kib}} or {{es}} nodes, run the [`elasticsearch-create-enrollment-token`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/create-enrollment-token.md) tool. These tools are available in the {{es}} `bin` directory. +::::{note} +If you need to reset the password for the `elastic` user or other built-in users, run the [`elasticsearch-reset-password`](elasticsearch://reference/elasticsearch/command-line-tools/reset-password.md) tool. To generate new enrollment tokens for {{kib}} or {{es}} nodes, run the [`elasticsearch-create-enrollment-token`](elasticsearch://reference/elasticsearch/command-line-tools/create-enrollment-token.md) tool. These tools are available in the {{es}} `bin` directory. :::: @@ -55,8 +55,8 @@ If this is the first time you’re starting {{kib}}, this command generates a un 2. In your browser, paste the enrollment token that was generated in the terminal when you started {{es}}, and then click the button to connect your {{kib}} instance with {{es}}. 3. Log in to {{kib}} as the `elastic` user with the password that was generated when you started {{es}}. -::::{note} -If you need to reset the password for the `elastic` user or other built-in users, run the [`elasticsearch-reset-password`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/reset-password.md) tool. To generate new enrollment tokens for {{kib}} or {{es}} nodes, run the [`elasticsearch-create-enrollment-token`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/create-enrollment-token.md) tool. These tools are available in the {{es}} `bin` directory. +::::{note} +If you need to reset the password for the `elastic` user or other built-in users, run the [`elasticsearch-reset-password`](elasticsearch://reference/elasticsearch/command-line-tools/reset-password.md) tool. To generate new enrollment tokens for {{kib}} or {{es}} nodes, run the [`elasticsearch-create-enrollment-token`](elasticsearch://reference/elasticsearch/command-line-tools/create-enrollment-token.md) tool. These tools are available in the {{es}} `bin` directory. :::: diff --git a/deploy-manage/manage-connectors.md b/deploy-manage/manage-connectors.md index 085c1f30a..bd8efe9a0 100644 --- a/deploy-manage/manage-connectors.md +++ b/deploy-manage/manage-connectors.md @@ -12,7 +12,7 @@ applies_to: Connectors serve as a central place to store connection information for both Elastic and third-party systems. They enable the linking of actions to rules, which execute as background tasks on the {{kib}} server when rule conditions are met. This allows rules to route actions to various destinations such as log files, ticketing systems, and messaging tools. Different {{kib}} apps may have their own rule types, but they typically share connectors. The **{{stack-manage-app}} > {{connectors-ui}}** provides a central location to view and manage all connectors in the current space. ::::{note} -This page is about {{kib}} connectors that integrate with services like generative AI model providers. If you’re looking for Search connectors that synchronize third-party data into {{es}}, refer to [Connector clients](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/search-connectors/index.md). +This page is about {{kib}} connectors that integrate with services like generative AI model providers. If you’re looking for Search connectors that synchronize third-party data into {{es}}, refer to [Connector clients](elasticsearch://reference/ingestion-tools/search-connectors/index.md). :::: @@ -26,7 +26,7 @@ Access to connectors is granted based on your privileges to alerting-enabled fea stack: ``` -If you're using {{stack}}, use the [action configuration settings](asciidocalypse://docs/kibana/docs/reference/configuration-reference/alerting-settings.md#action-settings) to customize connector networking configurations, such as proxies, certificates, or TLS settings. You can set configurations that apply to all your connectors or use `xpack.actions.customHostSettings` to set per-host configurations. +If you're using {{stack}}, use the [action configuration settings](kibana://reference/configuration-reference/alerting-settings.md#action-settings) to customize connector networking configurations, such as proxies, certificates, or TLS settings. You can set configurations that apply to all your connectors or use `xpack.actions.customHostSettings` to set per-host configurations. ## Connector list [connectors-list] @@ -51,7 +51,7 @@ You can delete a connector even if there are still actions referencing it. When ## Creating a new connector [creating-new-connector] -New connectors can be created with the **Create connector** button, which guides you to select the type of connector and configure its properties. For a full list of available connectors, see [Available connectors](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/connectors-kibana.md). +New connectors can be created with the **Create connector** button, which guides you to select the type of connector and configure its properties. For a full list of available connectors, see [Available connectors](kibana://reference/connectors-kibana.md). ::::{note} Some connector types are paid commercial features, while others are free. For a comparison of the Elastic subscription levels, go to [the subscription page](https://www.elastic.co/subscriptions). @@ -66,7 +66,7 @@ Some connector types are paid commercial features, while others are free. For a After you create a connector, it is available for use any time you set up an action in the current space. ::::{tip} -For out-of-the-box and standardized connectors, refer to [preconfigured connectors](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/pre-configured-connectors.md). +For out-of-the-box and standardized connectors, refer to [preconfigured connectors](kibana://reference/connectors-kibana/pre-configured-connectors.md). You can also manage connectors as resources with the [Elasticstack provider](https://registry.terraform.io/providers/elastic/elasticstack/latest) for Terraform. For more details, refer to the [elasticstack_kibana_action_connector](https://registry.terraform.io/providers/elastic/elasticstack/latest/docs/resources/kibana_action_connector) resource. diff --git a/deploy-manage/manage-spaces.md b/deploy-manage/manage-spaces.md index 1cb6b9ef6..a4a826b56 100644 --- a/deploy-manage/manage-spaces.md +++ b/deploy-manage/manage-spaces.md @@ -30,7 +30,7 @@ $$$spaces-managing$$$ **Spaces** let you organize your content and users according to your needs. -- Each space has its own saved objects. +- Each space has its own saved objects. - Users can only access the spaces that they have been granted access to. This access is based on user roles, and a given role can have different permissions per space. - In {{stack}} deployments on version 8.16 and later, each space has its own navigation, called solution view. @@ -46,18 +46,18 @@ To go to **Spaces**, find **Stack Management** in the navigation menu or use the ## Required permissions [_required_privileges_3] -* **Serverless projects:** `Admin` role or equivalent +* **Serverless projects:** `Admin` role or equivalent * **{{stack}} deployments:** `kibana_admin` or equivalent ## Create a space [spaces-managing] -The maximum number of spaces that you can have differs by deployment type: +The maximum number of spaces that you can have differs by deployment type: * **Serverless projects:** Maximum of 100 spaces. -* **{{stack}} deployments:** Controlled by the `xpack.spaces.maxSpaces` setting. Default is 1000. View the full list of Space settings in [this document](asciidocalypse://docs/kibana/docs/reference/configuration-reference/spaces-settings.md). +* **{{stack}} deployments:** Controlled by the `xpack.spaces.maxSpaces` setting. Default is 1000. View the full list of Space settings in [this document](kibana://reference/configuration-reference/spaces-settings.md). -To create a space: +To create a space: ::::{tab-set} :group: stack-serverless @@ -100,7 +100,7 @@ To create a space: You can edit all of the space settings you just defined at any time, except for the URL identifier. -Elastic also allows you to manage spaces using APIs: +Elastic also allows you to manage spaces using APIs: * **Serverless projects:** [Spaces API](https://www.elastic.co/docs/api/doc/serverless/operation/operation-get-spaces-space) * **{{stack}} deployments:** [Spaces API](https://www.elastic.co/docs/api/doc/kibana/operation/operation-post-spaces-copy-saved-objects) @@ -123,7 +123,7 @@ Users can access spaces based on the roles that they have. - For {{stack}} deployments, check [Creating or editing a role](/deploy-manage/users-roles/cluster-or-deployment-auth/defining-roles.md). -If you're managing an {{stack}} deployment, then you can also assign roles and define permissions for a space from the **Permissions** tab of the space settings. +If you're managing an {{stack}} deployment, then you can also assign roles and define permissions for a space from the **Permissions** tab of the space settings. When a role is assigned to *All Spaces*, you can’t remove its access from the space settings. You must instead edit the role to give it more granular access to individual spaces. @@ -147,7 +147,7 @@ serverless: unavailable You can create a custom experience for users by configuring the {{kib}} landing page on a per-space basis. The landing page can route users to a specific dashboard, application, or saved object as they enter each space. -To configure the landing page, use the default route setting in [Stack Management > {{kib}} > Advanced settings](asciidocalypse://docs/kibana/docs/reference/advanced-settings.md#kibana-general-settings). For example, you might set the default route to `/app/dashboards`. +To configure the landing page, use the default route setting in [Stack Management > {{kib}} > Advanced settings](kibana://reference/advanced-settings.md#kibana-general-settings). For example, you might set the default route to `/app/dashboards`. :::{image} ../images/kibana-spaces-configure-landing-page.png :alt: Configure space-level landing page diff --git a/deploy-manage/monitor/kibana-task-manager-health-monitoring.md b/deploy-manage/monitor/kibana-task-manager-health-monitoring.md index 4124727a0..3b237c5be 100644 --- a/deploy-manage/monitor/kibana-task-manager-health-monitoring.md +++ b/deploy-manage/monitor/kibana-task-manager-health-monitoring.md @@ -35,7 +35,7 @@ Monitoring the `_health` endpoint of each {{kib}} instance in the cluster is the The health monitoring API monitors the performance of Task Manager out of the box. However, certain performance considerations are deployment specific and you can configure them. -A health threshold is the threshold for failed task executions. Once a task exceeds this threshold, a status of `warn` or `error` is set on the task type execution. To configure a health threshold, use the [`xpack.task_manager.monitored_task_execution_thresholds`](asciidocalypse://docs/kibana/docs/reference/configuration-reference/task-manager-settings.md#task-manager-health-settings) setting. You can apply this this setting to all task types in the system, or to a custom task type. +A health threshold is the threshold for failed task executions. Once a task exceeds this threshold, a status of `warn` or `error` is set on the task type execution. To configure a health threshold, use the [`xpack.task_manager.monitored_task_execution_thresholds`](kibana://reference/configuration-reference/task-manager-settings.md#task-manager-health-settings) setting. You can apply this this setting to all task types in the system, or to a custom task type. By default, this setting marks the health of every task type as `warning` when it exceeds 80% failed executions, and as `error` at 90%. Set this value to a number between 0 to 100. The threshold is hit when the value **exceeds** this number. To avoid a status of `error`, set the threshold at 100. To hit `error` the moment any task fails, set the threshold to 0. @@ -75,11 +75,11 @@ logging: level: debug ``` -These stats are logged based on the number of milliseconds set in your [`xpack.task_manager.poll_interval`](asciidocalypse://docs/kibana/docs/reference/configuration-reference/task-manager-settings.md#task-manager-settings) setting, which could add substantial noise to your logs. Only enable this level of logging temporarily. +These stats are logged based on the number of milliseconds set in your [`xpack.task_manager.poll_interval`](kibana://reference/configuration-reference/task-manager-settings.md#task-manager-settings) setting, which could add substantial noise to your logs. Only enable this level of logging temporarily. **Automatic logging** -By default, the health API runs at a regular cadence, and each time it runs, it attempts to self evaluate its performance. If this self evaluation yields a potential problem, a message will log to the {{kib}} server log. In addition, the health API will look at how long tasks have waited to start (from when they were scheduled to start). If this number exceeds a configurable threshold ([`xpack.task_manager.monitored_stats_health_verbose_log.warn_delayed_task_start_in_seconds`](asciidocalypse://docs/kibana/docs/reference/configuration-reference/task-manager-settings.md#task-manager-settings)), the same message as above will log to the {{kib}} server log. +By default, the health API runs at a regular cadence, and each time it runs, it attempts to self evaluate its performance. If this self evaluation yields a potential problem, a message will log to the {{kib}} server log. In addition, the health API will look at how long tasks have waited to start (from when they were scheduled to start). If this number exceeds a configurable threshold ([`xpack.task_manager.monitored_stats_health_verbose_log.warn_delayed_task_start_in_seconds`](kibana://reference/configuration-reference/task-manager-settings.md#task-manager-settings)), the same message as above will log to the {{kib}} server log. This message looks like: @@ -87,7 +87,7 @@ This message looks like: Detected potential performance issue with Task Manager. Set 'xpack.task_manager.monitored_stats_health_verbose_log.enabled: true' in your Kibana.yml to enable debug logging` ``` -If this message appears, set [`xpack.task_manager.monitored_stats_health_verbose_log.enabled`](asciidocalypse://docs/kibana/docs/reference/configuration-reference/task-manager-settings.md#task-manager-settings) to `true` in your `kibana.yml`. This will start logging the health metrics at either a `warn` or `error` log level, depending on the detected severity of the potential problem. +If this message appears, set [`xpack.task_manager.monitored_stats_health_verbose_log.enabled`](kibana://reference/configuration-reference/task-manager-settings.md#task-manager-settings) to `true` in your `kibana.yml`. This will start logging the health metrics at either a `warn` or `error` log level, depending on the detected severity of the potential problem. ## Making sense of Task Manager health stats [making-sense-of-task-manager-health-stats] diff --git a/deploy-manage/monitor/logging-configuration/auditing-search-queries.md b/deploy-manage/monitor/logging-configuration/auditing-search-queries.md index 6afdf75fc..6df7c1471 100644 --- a/deploy-manage/monitor/logging-configuration/auditing-search-queries.md +++ b/deploy-manage/monitor/logging-configuration/auditing-search-queries.md @@ -12,7 +12,7 @@ applies_to: # Audit Elasticsearch search queries [auditing-search-queries] -There is no [audit event type](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/elasticsearch-audit-events.md) specifically dedicated to search queries. Search queries are analyzed and then processed; the processing triggers authorization actions that are audited. However, the original raw query, as submitted by the client, is not accessible downstream when authorization auditing occurs. +There is no [audit event type](elasticsearch://reference/elasticsearch/elasticsearch-audit-events.md) specifically dedicated to search queries. Search queries are analyzed and then processed; the processing triggers authorization actions that are audited. However, the original raw query, as submitted by the client, is not accessible downstream when authorization auditing occurs. Search queries are contained inside HTTP request bodies, however, and some audit events that are generated by the REST layer, on the coordinating node, can be toggled to output the request body to the audit log. Therefore, one must audit request bodies in order to audit search queries. @@ -24,26 +24,26 @@ xpack.security.audit.logfile.events.emit_request_body: true You can apply this setting through [cluster update settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings), as described in [](./configuring-audit-logs.md). Alternatively, you can modify `elasticsearch.yml` in all nodes and restart for the changes to take effect. -::::{important} +::::{important} No filtering is performed when auditing, so sensitive data might be audited in plain text when audit events include the request body. Also, the request body can contain malicious content that can break a parser consuming the audit logs. :::: The request body is printed as an escaped JSON string value (RFC 4627) to the `request.body` event attribute. Not all events contain the `request.body` attribute, even when the above setting is toggled. The ones that do are: - + * `authentication_success` * `authentication_failed` * `realm_authentication_failed` * `tampered_request` -* `run_as_denied` +* `run_as_denied` * `anonymous_access_denied` -The `request.body` attribute is printed on the coordinating node only (the node that handles the REST request). Most of these event types are [not included by default](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/auding-settings.md#xpack-sa-lf-events-include). +The `request.body` attribute is printed on the coordinating node only (the node that handles the REST request). Most of these event types are [not included by default](elasticsearch://reference/elasticsearch/configuration-reference/auding-settings.md#xpack-sa-lf-events-include). A good practical piece of advice is to add `authentication_success` to the event types that are audited (add it to the list in the `xpack.security.audit.logfile.events.include`), as this event type is not audited by default. -::::{note} +::::{note} Typically, the include list contains other event types as well, such as `access_granted` or `access_denied`. :::: diff --git a/deploy-manage/monitor/logging-configuration/configuring-audit-logs.md b/deploy-manage/monitor/logging-configuration/configuring-audit-logs.md index 4d6a21874..d4e13dfe5 100644 --- a/deploy-manage/monitor/logging-configuration/configuring-audit-logs.md +++ b/deploy-manage/monitor/logging-configuration/configuring-audit-logs.md @@ -16,34 +16,34 @@ When auditing security events, a single client request might generate multiple a {{es}} configuration options include: - * [{{es}} audited events settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/auding-settings.md#event-audit-settings): Use include and exclude filters to control the types of events that get logged. - * [{{es}} node information settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/auding-settings.md#node-audit-settings): Control whether to add or hide node information such as hostname or IP address in the audited events. - * [{{es}} ignore policies settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/auding-settings.md#audit-event-ignore-policies): Use ignore policies for fine-grained control over which audit events are printed to the log file. + * [{{es}} audited events settings](elasticsearch://reference/elasticsearch/configuration-reference/auding-settings.md#event-audit-settings): Use include and exclude filters to control the types of events that get logged. + * [{{es}} node information settings](elasticsearch://reference/elasticsearch/configuration-reference/auding-settings.md#node-audit-settings): Control whether to add or hide node information such as hostname or IP address in the audited events. + * [{{es}} ignore policies settings](elasticsearch://reference/elasticsearch/configuration-reference/auding-settings.md#audit-event-ignore-policies): Use ignore policies for fine-grained control over which audit events are printed to the log file. ::::{tip} - In {{es}}, all auditing settings except `xpack.security.audit.enabled` are dynamic. This means you can configure them using the [cluster update settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings), allowing changes to take effect immediately without requiring a restart. This approach is faster and more convenient than modifying `elasticsearch.yml`. + In {{es}}, all auditing settings except `xpack.security.audit.enabled` are dynamic. This means you can configure them using the [cluster update settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings), allowing changes to take effect immediately without requiring a restart. This approach is faster and more convenient than modifying `elasticsearch.yml`. :::: For a complete description of event details and format, refer to the following resources: - * [{{es}} audit events details and schema](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/elasticsearch-audit-events.md) + * [{{es}} audit events details and schema](elasticsearch://reference/elasticsearch/elasticsearch-audit-events.md) * [{{es}} log entry output format](/deploy-manage/monitor/logging-configuration/logfile-audit-output.md#audit-log-entry-format) ### Kibana auditing configuration -To control the logs that are outputted by Kibana, you can use [{{kib}} ignore filters](asciidocalypse://docs/kibana/docs/reference/configuration-reference/security-settings.md#audit-logging-ignore-filters). These are a list of filters that determine which events should be excluded from the audit log. +To control the logs that are outputted by Kibana, you can use [{{kib}} ignore filters](kibana://reference/configuration-reference/security-settings.md#audit-logging-ignore-filters). These are a list of filters that determine which events should be excluded from the audit log. -In self-managed systems, you can optionally configure audit logs location, and file/rolling file using [{{kib}} audit logging settings](asciidocalypse://docs/kibana/docs/reference/configuration-reference/security-settings.md#audit-logging-settings). +In self-managed systems, you can optionally configure audit logs location, and file/rolling file using [{{kib}} audit logging settings](kibana://reference/configuration-reference/security-settings.md#audit-logging-settings). ::::{tip} To configure {{kib}} settings, follow the same [procedure](./enabling-audit-logs.md#enable-audit-logging-procedure) as when enabling {{kib}} audit logs, but apply the relevant settings instead. :::: -For a complete description of auditing event details, such as `category`, `type`, or `action`, refer to [{{kib}} audit events](asciidocalypse://docs/kibana/docs/reference/kibana-audit-events.md). +For a complete description of auditing event details, such as `category`, `type`, or `action`, refer to [{{kib}} audit events](kibana://reference/kibana-audit-events.md). ### General recommendations -* Consider starting with {{es}} [`xpack.security.audit.logfile.events.include`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/auding-settings.md#xpack-sa-lf-events-include) and [{{kib}} ignore filters](asciidocalypse://docs/kibana/docs/reference/configuration-reference/security-settings.md#audit-logging-ignore-filters) settings to specify the type of events you want to include or exclude in the auditing output. +* Consider starting with {{es}} [`xpack.security.audit.logfile.events.include`](elasticsearch://reference/elasticsearch/configuration-reference/auding-settings.md#xpack-sa-lf-events-include) and [{{kib}} ignore filters](kibana://reference/configuration-reference/security-settings.md#audit-logging-ignore-filters) settings to specify the type of events you want to include or exclude in the auditing output. * If you need a more granular control, refer to [{{es}} audit events ignore policies](./logfile-audit-events-ignore-policies.md) for a better understanding how ignore policies work and when they are beneficial. diff --git a/deploy-manage/monitor/logging-configuration/correlating-kibana-elasticsearch-audit-logs.md b/deploy-manage/monitor/logging-configuration/correlating-kibana-elasticsearch-audit-logs.md index a7fc41546..b9df2544b 100644 --- a/deploy-manage/monitor/logging-configuration/correlating-kibana-elasticsearch-audit-logs.md +++ b/deploy-manage/monitor/logging-configuration/correlating-kibana-elasticsearch-audit-logs.md @@ -25,7 +25,7 @@ When an {{es}} request generates multiple audit events across multiple nodes, yo This identifier allows you to trace the flow of a request across the {{es}} cluster and reconstruct the full context of an operation. -Refer to [Audit events](asciidocalypse://elasticsearch/docs/reference/elasticsearch/elasticsearch-audit-events) for a complete reference of event types and attributes. +Refer to [Audit events](elasticsearch://reference/elasticsearch/elasticsearch-audit-events.md) for a complete reference of event types and attributes. ## `trace.id` field in {{kib}} audit events diff --git a/deploy-manage/monitor/logging-configuration/logfile-audit-events-ignore-policies.md b/deploy-manage/monitor/logging-configuration/logfile-audit-events-ignore-policies.md index 50777cef3..8512eddda 100644 --- a/deploy-manage/monitor/logging-configuration/logfile-audit-events-ignore-policies.md +++ b/deploy-manage/monitor/logging-configuration/logfile-audit-events-ignore-policies.md @@ -15,15 +15,15 @@ applies_to: The comprehensive audit trail is necessary to ensure accountability. It offers tremendous value during incident response and can even be required for demonstrating compliance. -The drawback of an audited system is represented by the inevitable performance penalty incurred. In all truth, the audit trail spends *I/O ops* that are not available anymore for the user’s queries. Sometimes the verbosity of the audit trail may become a problem that the event type restrictions, [defined by `include` and `exclude`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/auding-settings.md#event-audit-settings), will not alleviate. +The drawback of an audited system is represented by the inevitable performance penalty incurred. In all truth, the audit trail spends *I/O ops* that are not available anymore for the user’s queries. Sometimes the verbosity of the audit trail may become a problem that the event type restrictions, [defined by `include` and `exclude`](elasticsearch://reference/elasticsearch/configuration-reference/auding-settings.md#event-audit-settings), will not alleviate. **Audit events ignore policies** are a finer way to tune the verbosity of the audit trail. These policies define rules that match audit events which will be *ignored* (read as: not printed). Rules match on the values of attributes of audit events and complement the `include` or `exclude` method. Imagine the corpus of audit events and the policies chopping off unwanted events. With a sole exception, all audit events are subject to the ignore policies. The exception are events of type `security_config_change`, which cannot be filtered out, unless excluded altogether. -::::{important} +::::{important} When utilizing audit events ignore policies you are acknowledging potential accountability gaps that could render illegitimate actions undetectable. Please take time to review these policies whenever your system architecture changes. :::: -A policy is a named set of filter rules. Each filter rule applies to a single event attribute, one of the `users`, `realms`, `actions`, `roles` or `indices` attributes. The filter rule defines a list of [Lucene regexp](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/regexp-syntax.md), **any** of which has to match the value of the audit event attribute for the rule to match. A policy matches an event if **all** the rules comprising it match the event. An audit event is ignored, therefore not printed, if it matches **any** policy. All other non-matching events are printed as usual. +A policy is a named set of filter rules. Each filter rule applies to a single event attribute, one of the `users`, `realms`, `actions`, `roles` or `indices` attributes. The filter rule defines a list of [Lucene regexp](elasticsearch://reference/query-languages/regexp-syntax.md), **any** of which has to match the value of the audit event attribute for the rule to match. A policy matches an event if **all** the rules comprising it match the event. An audit event is ignored, therefore not printed, if it matches **any** policy. All other non-matching events are printed as usual. All policies are defined under the `xpack.security.audit.logfile.events.ignore_filters` settings namespace. For example, the following policy named *example1* matches events from the *kibana_system* or *admin_user* principals that operate over indices of the wildcard form *app-logs**: @@ -36,7 +36,7 @@ xpack.security.audit.logfile.events.ignore_filters: An audit event generated by the *kibana_system* user and operating over multiple indices , some of which do not match the indices wildcard, will not match. As expected, operations generated by all other users (even operating only on indices that match the *indices* filter) will not match this policy either. -Audit events of different types may have [different attributes](asciidocalypse://elasticsearch/docs/reference/elasticsearch/elasticsearch-audit-events/#audit-event-attributes). If an event does not contain an attribute for which some policy defines filters, the event will not match the policy. For example, the following policy will never match `authentication_success` or `authentication_failed` events, irrespective of the user’s roles, because these event schemas do not contain the `role` attribute: +Audit events of different types may have [different attributes](elasticsearch://reference/elasticsearch/elasticsearch-audit-events.md#audit-event-attributes). If an event does not contain an attribute for which some policy defines filters, the event will not match the policy. For example, the following policy will never match `authentication_success` or `authentication_failed` events, irrespective of the user’s roles, because these event schemas do not contain the `role` attribute: ```yaml xpack.security.audit.logfile.events.ignore_filters: diff --git a/deploy-manage/monitor/logging-configuration/logfile-audit-output.md b/deploy-manage/monitor/logging-configuration/logfile-audit-output.md index 718242d0a..87e56c79b 100644 --- a/deploy-manage/monitor/logging-configuration/logfile-audit-output.md +++ b/deploy-manage/monitor/logging-configuration/logfile-audit-output.md @@ -19,7 +19,7 @@ In self-managed clusters, you can configure how the `logfile` is written in the Orchestrated deployments (ECH, ECE, and ECK) do not support changes in `log4j2.properties` files of the {{es}} instances. -::::{note} +::::{note} If you overwrite the `log4j2.properties` and do not specify appenders for any of the audit trails, audit events are forwarded to the root appender, which by default points to the `elasticsearch.log` file. :::: @@ -33,4 +33,4 @@ There are however a few attributes that are exceptions to the above format. The When the `request.body` attribute is present (see [Auditing search queries](auditing-search-queries.md)), it contains a string value containing the full HTTP request body, escaped as per the JSON RFC 4677. -Refer to [audit event types](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/elasticsearch-audit-events.md) for a complete list of fields, as well as examples, for each entry type. +Refer to [audit event types](elasticsearch://reference/elasticsearch/elasticsearch-audit-events.md) for a complete list of fields, as well as examples, for each entry type. diff --git a/deploy-manage/monitor/logging-configuration/security-event-audit-logging.md b/deploy-manage/monitor/logging-configuration/security-event-audit-logging.md index 4c5e7c60a..564ad3a95 100644 --- a/deploy-manage/monitor/logging-configuration/security-event-audit-logging.md +++ b/deploy-manage/monitor/logging-configuration/security-event-audit-logging.md @@ -25,7 +25,7 @@ In this section, you'll learn how to: * [](./configuring-audit-logs.md): Filter and control what security events get logged in the audit log output. -* [Audit {{es}} search queries](./auditing-search-queries.md): Audit and log search request bodies. +* [Audit {{es}} search queries](./auditing-search-queries.md): Audit and log search request bodies. * [Correlate audit events](./correlating-kibana-elasticsearch-audit-logs.md): Explore audit logs and understand how events from the same request are correlated. @@ -33,5 +33,5 @@ By following these guidelines, you can effectively audit system activity, enhanc For a complete description of audit event details and format, refer to: -* [Elasticsearch audit events](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/elasticsearch-audit-events.md) -* [Kibana audit events](asciidocalypse://docs/kibana/docs/reference/kibana-audit-events.md) +* [Elasticsearch audit events](elasticsearch://reference/elasticsearch/elasticsearch-audit-events.md) +* [Kibana audit events](kibana://reference/kibana-audit-events.md) diff --git a/deploy-manage/monitor/monitoring-data/configure-stack-monitoring-alerts.md b/deploy-manage/monitor/monitoring-data/configure-stack-monitoring-alerts.md index cc50b4df2..992eb61f1 100644 --- a/deploy-manage/monitor/monitoring-data/configure-stack-monitoring-alerts.md +++ b/deploy-manage/monitor/monitoring-data/configure-stack-monitoring-alerts.md @@ -14,7 +14,7 @@ applies_to: You can configure Stack monitoring alerts to be sent to you by email when health related events occur in your deployments. To set up email notifications: 1. [Enable logging and monitoring](../stack-monitoring/elastic-cloud-stack-monitoring.md) on deployments for which you want to receive notifications. You need to enable only metrics data being shipped for the notifications to work. -2. In Kibana, configure the email connector to [send email from Elastic Cloud](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/email-action-type.md#elasticcloud). If you want to use the preconfigured `Elastic-Cloud-SMTP` connector in Elastic Cloud, then you can skip this step. +2. In Kibana, configure the email connector to [send email from Elastic Cloud](kibana://reference/connectors-kibana/email-action-type.md#elasticcloud). If you want to use the preconfigured `Elastic-Cloud-SMTP` connector in Elastic Cloud, then you can skip this step. 3. From the Kibana main menu, go to **Stack Monitoring**. On this page you can find a summary of monitoring metrics for your deployment as well as any alerts. 4. Select **Enter setup mode**. 5. On any card showing available alerts, select the **alerts** indicator. Use the menu to select the type of alert for which you’d like to be notified. There are many alert types, including: diff --git a/deploy-manage/monitor/monitoring-data/ec-memory-pressure.md b/deploy-manage/monitor/monitoring-data/ec-memory-pressure.md index 41fd8ae84..5cd900b4f 100644 --- a/deploy-manage/monitor/monitoring-data/ec-memory-pressure.md +++ b/deploy-manage/monitor/monitoring-data/ec-memory-pressure.md @@ -23,7 +23,7 @@ The percentage number used in the JVM memory pressure indicator is actually the When the JVM memory pressure reaches 75%, the indicator turns red. At this level, garbage collection becomes more frequent as the memory usage increases, potentially impacting the performance of your cluster. As long as the cluster performance suits your needs, JVM memory pressure above 75% is not a problem in itself, but there is not much spare memory capacity. Review the [common causes of high JVM memory usage](#ec-memory-pressure-causes) to determine your best course of action. -When the JVM memory pressure indicator rises above 95%, {{es}}'s [real memory circuit breaker](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/circuit-breaker-settings.md#parent-circuit-breaker) triggers to prevent your instance from running out of memory. This situation can reduce the stability of your cluster and the integrity of your data. Unless you expect the load to drop soon, we recommend that you resize to a larger cluster before you reach this level of memory pressure. Even if you’re planning to optimize your memory usage, it is best to resize the cluster first. Resizing the cluster to increase capacity can give you more time to apply other changes, and also provides the cluster with more resource for when those changes are applied. +When the JVM memory pressure indicator rises above 95%, {{es}}'s [real memory circuit breaker](elasticsearch://reference/elasticsearch/configuration-reference/circuit-breaker-settings.md#parent-circuit-breaker) triggers to prevent your instance from running out of memory. This situation can reduce the stability of your cluster and the integrity of your data. Unless you expect the load to drop soon, we recommend that you resize to a larger cluster before you reach this level of memory pressure. Even if you’re planning to optimize your memory usage, it is best to resize the cluster first. Resizing the cluster to increase capacity can give you more time to apply other changes, and also provides the cluster with more resource for when those changes are applied. ## Common causes of high JVM memory usage [ec-memory-pressure-causes] diff --git a/deploy-manage/monitor/monitoring-data/ec-vcpu-boost-instance.md b/deploy-manage/monitor/monitoring-data/ec-vcpu-boost-instance.md index aa41afaed..61036b1fe 100644 --- a/deploy-manage/monitor/monitoring-data/ec-vcpu-boost-instance.md +++ b/deploy-manage/monitor/monitoring-data/ec-vcpu-boost-instance.md @@ -22,7 +22,7 @@ Based on the instance size, the vCPU resources assigned to your instance can be ## What are vCPU credits? [ec_what_are_vcpu_credits] -[vCPU](asciidocalypse://docs/docs-content/docs/reference/glossary/index.md#glossary-vcpu) credits enable a smaller instance to perform as if it were assigned the vCPU resources of a larger instance, but only for a limited time. vCPU credits are available only on smaller instances up to and including 8 GB of RAM. +[vCPU](/reference/glossary/index.md#glossary-vcpu) credits enable a smaller instance to perform as if it were assigned the vCPU resources of a larger instance, but only for a limited time. vCPU credits are available only on smaller instances up to and including 8 GB of RAM. vCPU credits persist through cluster restarts, but they are tied to your existing instance nodes. Operations that create new instance nodes will lose existing vCPU credits. This happens when you resize your instance, or if Elastic performs system maintenance on your nodes. diff --git a/deploy-manage/monitor/monitoring-data/elasticsearch-metrics.md b/deploy-manage/monitor/monitoring-data/elasticsearch-metrics.md index b8777c394..37b1e43a4 100644 --- a/deploy-manage/monitor/monitoring-data/elasticsearch-metrics.md +++ b/deploy-manage/monitor/monitoring-data/elasticsearch-metrics.md @@ -120,5 +120,5 @@ If you use {{filebeat}} to collect log data from your cluster, you can see its r If you click **Logs**, you can see the most recent logs for the cluster. ::::{tip} -By default, up to 10 log entries are shown. You can show up to 50 log entries by changing the [`monitoring.ui.elasticsearch.logFetchCount` setting](asciidocalypse://docs/kibana/docs/reference/configuration-reference/monitoring-settings.md#monitoring-ui-settings). If you changed the default name of filebeat indices, you also need to update `monitoring.ui.logs.index` accordingly. +By default, up to 10 log entries are shown. You can show up to 50 log entries by changing the [`monitoring.ui.elasticsearch.logFetchCount` setting](kibana://reference/configuration-reference/monitoring-settings.md#monitoring-ui-settings). If you changed the default name of filebeat indices, you also need to update `monitoring.ui.logs.index` accordingly. :::: diff --git a/deploy-manage/monitor/monitoring-data/logstash-page.md b/deploy-manage/monitor/monitoring-data/logstash-page.md index 646851b97..59f16201e 100644 --- a/deploy-manage/monitor/monitoring-data/logstash-page.md +++ b/deploy-manage/monitor/monitoring-data/logstash-page.md @@ -25,4 +25,4 @@ If you are monitoring Logstash nodes, click **Overview** in the Logstash section 1. To view Logstash node metrics, click **Nodes**. The Nodes section shows the status of each Logstash node. 2. Click the name of a node to view its statistics over time. -For more information, refer to [Monitoring Logstash](asciidocalypse://docs/logstash/docs/reference/monitoring-logstash-legacy.md). +For more information, refer to [Monitoring Logstash](logstash://reference/monitoring-logstash-legacy.md). diff --git a/deploy-manage/monitor/monitoring-data/monitor-troubleshooting.md b/deploy-manage/monitor/monitoring-data/monitor-troubleshooting.md index 9a2676395..f90a0a6e9 100644 --- a/deploy-manage/monitor/monitoring-data/monitor-troubleshooting.md +++ b/deploy-manage/monitor/monitoring-data/monitor-troubleshooting.md @@ -16,7 +16,7 @@ applies_to: Use the information in this section to troubleshoot common problems and find answers for frequently asked questions related to the {{kib}} {{monitor-features}}. -## Cannot view the cluster because the license information is invalid [_cannot_view_the_cluster_because_the_license_information_is_invalid] +## Cannot view the cluster because the license information is invalid [_cannot_view_the_cluster_because_the_license_information_is_invalid] **Symptoms:** @@ -27,7 +27,7 @@ The following error appears in a banner at the top of the screen in {{kib}} on t You cannot monitor a version 6.3 or later cluster from {{kib}} version 6.2 or earlier. To resolve this issue, upgrade {{kib}} to 6.3 or later. See [Upgrading the {{stack}}](../../upgrade/deployment-or-cluster.md). -## {{filebeat}} index is corrupt [_filebeat_index_is_corrupt] +## {{filebeat}} index is corrupt [_filebeat_index_is_corrupt] **Symptoms:** @@ -40,7 +40,7 @@ The **Stack Monitoring** application displays a Monitoring Request error indicat 3. Restart all your {{filebeat}} instances. -## No monitoring data is visible in {{kib}} [_no_monitoring_data_is_visible_in_kib] +## No monitoring data is visible in {{kib}} [_no_monitoring_data_is_visible_in_kib] **Symptoms:** @@ -48,7 +48,7 @@ The **Stack Monitoring** page in {{kib}} is empty. **Resolution:** -1. Confirm that {{kib}} is seeking monitoring data from the appropriate {{es}} URL. By default, data is retrieved from the cluster specified in the `elasticsearch.hosts` setting in the `kibana.yml` file. If you want to retrieve it from a different monitoring cluster, set `monitoring.ui.elasticsearch.hosts`. See [Monitoring settings](asciidocalypse://docs/kibana/docs/reference/configuration-reference/monitoring-settings.md). +1. Confirm that {{kib}} is seeking monitoring data from the appropriate {{es}} URL. By default, data is retrieved from the cluster specified in the `elasticsearch.hosts` setting in the `kibana.yml` file. If you want to retrieve it from a different monitoring cluster, set `monitoring.ui.elasticsearch.hosts`. See [Monitoring settings](kibana://reference/configuration-reference/monitoring-settings.md). 2. Confirm that there is monitoring data available at that URL. It is stored in indices such as `.monitoring-kibana-*` and `.monitoring-es-*` or `metrics-kibana.stack_monitoring.*`, depending on which method is used to collect monitoring data. At a minimum, you must have monitoring data for the {{es}} production cluster. Once that data exists, {{kib}} can display monitoring data for other products in the cluster. 3. Set the time filter to “Last 1 hour”. When monitoring data appears in your cluster, the page automatically refreshes with the monitoring summary. 4. If using {{agent}}, ensure that all integration assets have been installed in the monitoring cluster. diff --git a/deploy-manage/monitor/monitoring-data/visualizing-monitoring-data.md b/deploy-manage/monitor/monitoring-data/visualizing-monitoring-data.md index 801ccb81e..e9393c805 100644 --- a/deploy-manage/monitor/monitoring-data/visualizing-monitoring-data.md +++ b/deploy-manage/monitor/monitoring-data/visualizing-monitoring-data.md @@ -20,5 +20,5 @@ If you enable monitoring across the {{stack}}, each monitored component is consi For more information, see [Configure monitoring](../stack-monitoring/kibana-monitoring-self-managed.md) and [Monitor a cluster](../../monitor.md). -Want to monitor your fleet of {{agent}}s, too? Use {{fleet}} instead of the Stack Monitoring UI. To learn more, refer to [Monitor {{agent}}s](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/monitor-elastic-agent.md). +Want to monitor your fleet of {{agent}}s, too? Use {{fleet}} instead of the Stack Monitoring UI. To learn more, refer to [Monitor {{agent}}s](/reference/ingestion-tools/fleet/monitor-elastic-agent.md). diff --git a/deploy-manage/monitor/stack-monitoring/collecting-log-data-with-filebeat.md b/deploy-manage/monitor/stack-monitoring/collecting-log-data-with-filebeat.md index b51415693..7b1e19549 100644 --- a/deploy-manage/monitor/stack-monitoring/collecting-log-data-with-filebeat.md +++ b/deploy-manage/monitor/stack-monitoring/collecting-log-data-with-filebeat.md @@ -14,22 +14,22 @@ applies_to: You can use {{filebeat}} to monitor the {{es}} log files, collect log events, and ship them to the monitoring cluster. Your recent logs are visible on the **Monitoring** page in {{kib}}. -::::{important} +::::{important} If you’re using {{agent}}, do not deploy {{filebeat}} for log collection. Instead, configure the {{es}} integration to collect logs. :::: 1. Verify that {{es}} is running and that the monitoring cluster is ready to receive data from {{filebeat}}. - ::::{tip} + ::::{tip} In production environments, we strongly recommend using a separate cluster (referred to as the *monitoring cluster*) to store the data. Using a separate monitoring cluster prevents production cluster outages from impacting your ability to access your monitoring data. It also prevents monitoring activities from impacting the performance of your production cluster. See [*Monitoring in a production environment*](elasticsearch-monitoring-self-managed.md). :::: 2. Identify which logs you want to monitor. - The {{filebeat}} {{es}} module can handle [audit logs](../logging-configuration/logfile-audit-output.md), [deprecation logs](../logging-configuration/elasticsearch-log4j-configuration-self-managed.md#deprecation-logging), [gc logs](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/jvm-settings.md#gc-logging), [server logs](../logging-configuration/elasticsearch-log4j-configuration-self-managed.md), and [slow logs](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/slow-log.md). For more information about the location of your {{es}} logs, see the [path.logs](../../deploy/self-managed/important-settings-configuration.md#path-settings) setting. + The {{filebeat}} {{es}} module can handle [audit logs](../logging-configuration/logfile-audit-output.md), [deprecation logs](../logging-configuration/elasticsearch-log4j-configuration-self-managed.md#deprecation-logging), [gc logs](elasticsearch://reference/elasticsearch/jvm-settings.md#gc-logging), [server logs](../logging-configuration/elasticsearch-log4j-configuration-self-managed.md), and [slow logs](elasticsearch://reference/elasticsearch/index-settings/slow-log.md). For more information about the location of your {{es}} logs, see the [path.logs](../../deploy/self-managed/important-settings-configuration.md#path-settings) setting. - ::::{important} + ::::{important} If there are both structured (`*.json`) and unstructured (plain text) versions of the logs, you must use the structured logs. Otherwise, they might not appear in the appropriate context in {{kib}}. :::: @@ -54,7 +54,7 @@ If you’re using {{agent}}, do not deploy {{filebeat}} for log collection. Inst If you configured the monitoring cluster to use encrypted communications, you must access it via HTTPS. For example, use a `hosts` setting like `https://es-mon-1:9200`. - ::::{important} + ::::{important} The {{es}} {{monitor-features}} use ingest pipelines, therefore the cluster that stores the monitoring data must have at least one [ingest node](../../../manage-data/ingest/transform-enrich/ingest-pipelines.md). :::: @@ -74,7 +74,7 @@ If you’re using {{agent}}, do not deploy {{filebeat}} for log collection. Inst #password: "YOUR_PASSWORD" ``` - ::::{tip} + ::::{tip} In production environments, we strongly recommend using a dedicated {{kib}} instance for your monitoring cluster. :::: @@ -101,13 +101,13 @@ If you’re using {{agent}}, do not deploy {{filebeat}} for log collection. Inst If the logs that you want to monitor aren’t in the default location, set the appropriate path variables in the `modules.d/elasticsearch.yml` file. See [Configure the {{es}} module](asciidocalypse://docs/beats/docs/reference/filebeat/filebeat-module-elasticsearch.md#configuring-elasticsearch-module). - ::::{important} + ::::{important} If there are JSON logs, configure the `var.paths` settings to point to them instead of the plain text logs. :::: 8. [Start {{filebeat}}](asciidocalypse://docs/beats/docs/reference/filebeat/filebeat-starting.md) on each node. - ::::{note} + ::::{note} Depending on how you’ve installed {{filebeat}}, you might see errors related to file ownership or permissions when you try to run {{filebeat}} modules. See [Config file ownership and permissions](asciidocalypse://docs/beats/docs/reference/libbeat/config-file-permissions.md). :::: @@ -115,7 +115,7 @@ If you’re using {{agent}}, do not deploy {{filebeat}} for log collection. Inst For example, use the [cat indices](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cat-indices) command to verify that there are new `filebeat-*` indices. - ::::{tip} + ::::{tip} If you want to use the **Monitoring** UI in {{kib}}, there must also be `.monitoring-*` indices. Those indices are generated when you collect metrics about {{stack}} products. For example, see [Collecting monitoring data with {{metricbeat}}](collecting-monitoring-data-with-metricbeat.md). :::: diff --git a/deploy-manage/monitor/stack-monitoring/ece-stack-monitoring.md b/deploy-manage/monitor/stack-monitoring/ece-stack-monitoring.md index 964a0bf76..247ce65af 100644 --- a/deploy-manage/monitor/stack-monitoring/ece-stack-monitoring.md +++ b/deploy-manage/monitor/stack-monitoring/ece-stack-monitoring.md @@ -22,12 +22,12 @@ Monitoring consists of two components: The steps in this section cover only the enablement of the monitoring and logging features in Elastic Cloud Enterprise. For more information on how to use the monitoring features, refer to [Monitor a cluster](../../monitor.md). -### Before you begin [ece-logging-and-monitoring-limitations] +### Before you begin [ece-logging-and-monitoring-limitations] Some limitations apply when you use monitoring on Elastic Cloud Enterprise. To learn more, check the monitoring [restrictions and limitations](ece-restrictions-monitoring.md). -### Monitoring for production use [ece-logging-and-monitoring-production] +### Monitoring for production use [ece-logging-and-monitoring-production] For production use, you should send your deployment logs and metrics to a dedicated monitoring deployment. Monitoring indexes logs and metrics into {{es}} and these indexes consume storage, memory, and CPU cycles like any other index. By using a separate monitoring deployment, you avoid affecting your other production deployments and can view the logs and metrics even when a production deployment is unavailable. @@ -44,15 +44,15 @@ How many monitoring deployments you use depends on your requirements: Logs and metrics that get sent to a dedicated monitoring {{es}} deployment [may not be cleaned up automatically](#ece-logging-and-monitoring-retention) and might require some additional steps to remove excess data periodically. -### Retention of monitoring daily indices [ece-logging-and-monitoring-retention] +### Retention of monitoring daily indices [ece-logging-and-monitoring-retention] -#### Stack versions 8.0 and above [ece-logging-and-monitoring-retention-8] +#### Stack versions 8.0 and above [ece-logging-and-monitoring-retention-8] When you enable monitoring in Elastic Cloud Enterprise, your monitoring indices are retained for a certain period by default. After the retention period has passed, the monitoring indices are deleted automatically. The retention period is configured in the `.monitoring-8-ilm-policy` index lifecycle policy. To view or edit the policy open {{kib}} **Stack management > Data > Index Lifecycle Policies**. -### Sending monitoring data to itself (self monitoring) [ece-logging-and-monitoring-retention-self-monitoring] +### Sending monitoring data to itself (self monitoring) [ece-logging-and-monitoring-retention-self-monitoring] $$$ece-logging-and-monitoring-retention-7$$$ When you enable self-monitoring in Elastic Cloud Enterprise, your monitoring indices are retained for a certain period by default. After the retention period has passed, the monitoring indices are deleted automatically. Monitoring data is retained for three days by default or as specified by the [`xpack.monitoring.history.duration` user setting](https://www.elastic.co/guide/en/cloud-enterprise/current/ece-change-user-settings-examples.html#xpack-monitoring-history-duration). @@ -74,7 +74,7 @@ PUT /_cluster/settings ``` -### Sending monitoring data to a dedicated monitoring deployment [ece-logging-and-monitoring-retention-dedicated-monitoring] +### Sending monitoring data to a dedicated monitoring deployment [ece-logging-and-monitoring-retention-dedicated-monitoring] When [monitoring for production use](#ece-logging-and-monitoring-production), where you configure your deployments **to send monitoring data to a dedicated monitoring deployment** for indexing, this retention period does not apply. Monitoring indices on a dedicated monitoring deployment are retained until you remove them. There are three options open to you: @@ -107,17 +107,17 @@ When [monitoring for production use](#ece-logging-and-monitoring-production), wh * To retain monitoring indices on a dedicated monitoring deployment as is without deleting them automatically, no additional steps are required other than making sure that you do not enable the monitoring deployment to send monitoring data to itself. You should also monitor the deployment for disk space usage and upgrade your deployment periodically, if necessary. -### Retention of logging indices [ece-logging-and-monitoring-log-retention] +### Retention of logging indices [ece-logging-and-monitoring-log-retention] An ILM policy is pre-configured to manage log retention. The policy can be adjusted according to your requirements. -### Index management [ece-logging-and-monitoring-index-management-ilm] +### Index management [ece-logging-and-monitoring-index-management-ilm] When sending monitoring data to a deployment, you can configure [Index Lifecycle Management (ILM)](/manage-data/lifecycle/index-lifecycle-management.md) to manage retention of your monitoring and logging indices. When sending logs to a deployment, an ILM policy is pre-configured to manage log retention and the policy can be customized to your needs. -### Enable logging and monitoring [ece-enable-logging-and-monitoring-steps] +### Enable logging and monitoring [ece-enable-logging-and-monitoring-steps] Elastic Cloud Enterprise manages the installation and configuration of the monitoring agent for you. When you enable monitoring on a deployment, you are configuring where the monitoring agent for your current deployment should send its logs and metrics. @@ -134,23 +134,23 @@ To enable monitoring on your deployment: If a deployment is not listed, make sure that it is running a compatible version. The monitoring deployment and production deployment must be on the same major version, cloud provider, and region. - ::::{tip} + ::::{tip} Remember to send logs and metrics for production deployments to a dedicated monitoring deployment, so that your production deployments are not impacted by the overhead of indexing and storing monitoring data. A dedicated monitoring deployment also gives you more control over the retention period for monitoring data. :::: -::::{note} +::::{note} Enabling logs and monitoring may trigger a plan change on your deployment. You can monitor the plan change progress from the deployment’s **Activity** page. :::: -::::{note} +::::{note} Enabling logs and monitoring requires some extra resource on a deployment. For production systems, we recommend sizing deployments with logs and monitoring enabled to at least 4 GB of RAM. :::: -### Access the monitoring application in Kibana [ece-access-kibana-monitoring] +### Access the monitoring application in Kibana [ece-access-kibana-monitoring] With monitoring enabled for your deployment, you can access the [logs](https://www.elastic.co/guide/en/kibana/current/observability.html) and [stack monitoring](../monitoring-data/visualizing-monitoring-data.md) through Kibana. @@ -174,28 +174,28 @@ Alternatively, you can access logs and metrics directly on the Kibana **Logs** a | `service.version` | The version of the stack resource that generated the log | `8.13.1` | -### Logging features [ece-extra-logging-features] +### Logging features [ece-extra-logging-features] When shipping logs to a monitoring deployment there are more logging features available to you. These features include: -#### For {{es}}: [ece-extra-logging-features-elasticsearch] +#### For {{es}}: [ece-extra-logging-features-elasticsearch] * [Audit logging](../logging-configuration/enabling-audit-logs.md) - logs security-related events on your deployment -* [Slow query and index logging](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/slow-log.md) - helps find and debug slow queries and indexing +* [Slow query and index logging](elasticsearch://reference/elasticsearch/index-settings/slow-log.md) - helps find and debug slow queries and indexing * Verbose logging - helps debug stack issues by increasing component logs After you’ve enabled log delivery on your deployment, you can [add the Elasticsearch user settings](../../deploy/cloud-enterprise/edit-stack-settings.md) to enable these features. -#### For Kibana: [ece-extra-logging-features-kibana] +#### For Kibana: [ece-extra-logging-features-kibana] * [Audit logging](../logging-configuration/enabling-audit-logs.md) - logs security-related events on your deployment After you’ve enabled log delivery on your deployment, you can [add the Kibana user settings](../../deploy/cloud-enterprise/edit-stack-settings.md) to enable this feature. -### Other components [ece-extra-logging-features-enterprise-search] +### Other components [ece-extra-logging-features-enterprise-search] Enabling log collection also supports collecting and indexing the following types of logs from other components in your deployments: @@ -213,12 +213,12 @@ The ˆ*ˆ indicates that we also index the archived files of each type of log. Check the respective product documentation for more information about the logging capabilities of each product. -## Metrics features [ece-extra-metrics-features] +## Metrics features [ece-extra-metrics-features] With logging and monitoring enabled for a deployment, metrics are collected for Elasticsearch, Kibana, and APM with Fleet Server. -#### Enabling Elasticsearch/Kibana audit logs on your deployment [ece-enable-audit-logs] +#### Enabling Elasticsearch/Kibana audit logs on your deployment [ece-enable-audit-logs] Audit logs are useful for tracking security events on your {{es}} and/or {{kib}} clusters. To enable {{es}} audit logs on your deployment: diff --git a/deploy-manage/monitor/stack-monitoring/es-http-exporter.md b/deploy-manage/monitor/stack-monitoring/es-http-exporter.md index 4d3b617e8..69a036a9e 100644 --- a/deploy-manage/monitor/stack-monitoring/es-http-exporter.md +++ b/deploy-manage/monitor/stack-monitoring/es-http-exporter.md @@ -9,7 +9,7 @@ applies_to: # HTTP exporters [http-exporter] -::::{important} +::::{important} {{agent}} and {{metricbeat}} are the recommended methods for collecting and shipping monitoring data to a monitoring cluster. If you have previously configured legacy collection methods, you should migrate to using [{{agent}}](collecting-monitoring-data-with-elastic-agent.md) or [{{metricbeat}}](collecting-monitoring-data-with-metricbeat.md) collection. Do not use legacy collection alongside other collection methods. @@ -19,9 +19,9 @@ If you have previously configured legacy collection methods, you should migrate The `http` exporter is the preferred exporter in the {{es}} {{monitor-features}} because it enables the use of a separate monitoring cluster. As a secondary benefit, it avoids using a production cluster node as a coordinating node for indexing monitoring data because all requests are HTTP requests to the monitoring cluster. -The `http` exporter uses the low-level {{es}} REST Client, which enables it to send its data to any {{es}} cluster it can access through the network. Its requests make use of the [`filter_path`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/common-options.md#common-options-response-filtering) parameter to reduce bandwidth whenever possible, which helps to ensure that communications between the production and monitoring clusters are as lightweight as possible. +The `http` exporter uses the low-level {{es}} REST Client, which enables it to send its data to any {{es}} cluster it can access through the network. Its requests make use of the [`filter_path`](elasticsearch://reference/elasticsearch/rest-apis/common-options.md#common-options-response-filtering) parameter to reduce bandwidth whenever possible, which helps to ensure that communications between the production and monitoring clusters are as lightweight as possible. -The `http` exporter supports a number of settings that control how it communicates over HTTP to remote clusters. In most cases, it is not necessary to explicitly configure these settings. For detailed descriptions, see [Monitoring settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/monitoring-settings.md). +The `http` exporter supports a number of settings that control how it communicates over HTTP to remote clusters. In most cases, it is not necessary to explicitly configure these settings. For detailed descriptions, see [Monitoring settings](elasticsearch://reference/elasticsearch/configuration-reference/monitoring-settings.md). ```yaml xpack.monitoring.exporters: @@ -49,13 +49,13 @@ xpack.monitoring.exporters: 2. An `http` exporter defined whose arbitrary name is `my_remote`. This name uniquely defines the exporter but is otherwise unused. 3. `host` is a required setting for `http` exporters. It must specify the HTTP port rather than the transport port. The default port value is `9200`. 4. User authentication for those using {{stack}} {{security-features}} or some other form of user authentication protecting the cluster. -5. See [HTTP exporter settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/monitoring-settings.md#http-exporter-settings) for all TLS/SSL settings. If not supplied, the default node-level TLS/SSL settings are used. +5. See [HTTP exporter settings](elasticsearch://reference/elasticsearch/configuration-reference/monitoring-settings.md#http-exporter-settings) for all TLS/SSL settings. If not supplied, the default node-level TLS/SSL settings are used. 6. Optional base path to prefix any outgoing request with in order to work with proxies. 7. Arbitrary key/value pairs to define as headers to send with every request. The array-based key/value format sends one header per value. 8. A mechanism for changing the date suffix used by default. -::::{note} +::::{note} The `http` exporter accepts an array of `hosts` and it will round robin through the list. It is a good idea to take advantage of that feature when the monitoring cluster contains more than one node. :::: @@ -69,17 +69,17 @@ Unlike the `local` exporter, *every* node that uses the `http` exporter attempts The easiest way to trigger a check is to disable, then re-enable the exporter. -::::{warning} +::::{warning} This resource management behavior can create a hole for users that delete monitoring resources. Since the `http` exporter does not re-check its resources unless one of the triggers occurs, this can result in malformed index mappings. :::: Unlike the `local` exporter, the `http` exporter is inherently routing requests outside of the cluster. This situation means that the exporter must provide a username and password when the monitoring cluster requires one (or other appropriate security configurations, such as TLS/SSL settings). -::::{important} +::::{important} When discussing security relative to the `http` exporter, it is critical to remember that all users are managed on the monitoring cluster. This is particularly important to remember when you move from development environments to production environments, where you often have dedicated monitoring clusters. :::: -For more information about the configuration options for the `http` exporter, see [HTTP exporter settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/monitoring-settings.md#http-exporter-settings). +For more information about the configuration options for the `http` exporter, see [HTTP exporter settings](elasticsearch://reference/elasticsearch/configuration-reference/monitoring-settings.md#http-exporter-settings). diff --git a/deploy-manage/monitor/stack-monitoring/es-legacy-collection-methods.md b/deploy-manage/monitor/stack-monitoring/es-legacy-collection-methods.md index 1d8afb703..546caa3e6 100644 --- a/deploy-manage/monitor/stack-monitoring/es-legacy-collection-methods.md +++ b/deploy-manage/monitor/stack-monitoring/es-legacy-collection-methods.md @@ -30,16 +30,16 @@ To learn about monitoring in general, see [Monitor a cluster](../../monitor.md). 1. Verify that the `xpack.monitoring.elasticsearch.collection.enabled` setting is `true`, which is its default value, on each node in the cluster. - ::::{note} + ::::{note} You can specify this setting in either the `elasticsearch.yml` on each node or across the cluster as a dynamic cluster setting. If {{es}} {{security-features}} are enabled, you must have `monitor` cluster privileges to view the cluster settings and `manage` cluster privileges to change them. :::: - For more information, see [Monitoring settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/monitoring-settings.md) and [Cluster update settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings). + For more information, see [Monitoring settings](elasticsearch://reference/elasticsearch/configuration-reference/monitoring-settings.md) and [Cluster update settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings). 2. Set the `xpack.monitoring.collection.enabled` setting to `true` on each node in the cluster. By default, it is disabled (`false`). - ::::{note} + ::::{note} You can specify this setting in either the `elasticsearch.yml` on each node or across the cluster as a dynamic cluster setting. If {{es}} {{security-features}} are enabled, you must have `monitor` cluster privileges to view the cluster settings and `manage` cluster privileges to change them. :::: @@ -61,7 +61,7 @@ To learn about monitoring in general, see [Monitor a cluster](../../monitor.md). Alternatively, you can enable this setting in {{kib}}. In the side navigation, click **Monitoring**. If data collection is disabled, you are prompted to turn it on. - For more information, see [Monitoring settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/monitoring-settings.md) and [Cluster update settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings). + For more information, see [Monitoring settings](elasticsearch://reference/elasticsearch/configuration-reference/monitoring-settings.md) and [Cluster update settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings). 3. Optional: Specify which indices you want to monitor. @@ -73,13 +73,13 @@ To learn about monitoring in general, see [Monitor a cluster](../../monitor.md). You can prepend `-` to explicitly exclude index names or patterns. For example, to include all indices that start with `test` except `test3`, you could specify `test*,-test3`. To include system indices such as .security and .kibana, add `.*` to the list of included names. For example `.*,test*,-test3` - 4. Optional: Specify how often to collect monitoring data. The default value for the `xpack.monitoring.collection.interval` setting 10 seconds. See [Monitoring settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/monitoring-settings.md). + 4. Optional: Specify how often to collect monitoring data. The default value for the `xpack.monitoring.collection.interval` setting 10 seconds. See [Monitoring settings](elasticsearch://reference/elasticsearch/configuration-reference/monitoring-settings.md). 2. Identify where to store monitoring data. By default, the data is stored on the same cluster by using a [`local` exporter](es-local-exporter.md). Alternatively, you can use an [`http` exporter](es-http-exporter.md) to send data to a separate *monitoring cluster*. - ::::{important} + ::::{important} The {{es}} {{monitor-features}} use ingest pipelines, therefore the cluster that stores the monitoring data must have at least one [ingest node](../../../manage-data/ingest/transform-enrich/ingest-pipelines.md). :::: @@ -147,8 +147,8 @@ To learn about monitoring in general, see [Monitor a cluster](../../monitor.md). 4. Configure your cluster to route monitoring data from sources such as {{kib}}, Beats, and {{ls}} to the monitoring cluster. For information about configuring each product to collect and send monitoring data, see [Monitor a cluster](../../monitor.md). 5. If you updated settings in the `elasticsearch.yml` files on your production cluster, restart {{es}}. See [*Stopping Elasticsearch*](../../maintenance/start-stop-services/start-stop-elasticsearch.md) and [*Starting Elasticsearch*](../../maintenance/start-stop-services/start-stop-elasticsearch.md). - ::::{tip} - You may want to temporarily [disable shard allocation](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md) before you restart your nodes to avoid unnecessary shard reallocation during the install process. + ::::{tip} + You may want to temporarily [disable shard allocation](elasticsearch://reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md) before you restart your nodes to avoid unnecessary shard reallocation during the install process. :::: 6. Optional: [Configure the indices that store the monitoring data](../monitoring-data/configuring-data-streamsindices-for-monitoring.md). diff --git a/deploy-manage/monitor/stack-monitoring/es-local-exporter.md b/deploy-manage/monitor/stack-monitoring/es-local-exporter.md index a4e40e0cb..a392e3fe3 100644 --- a/deploy-manage/monitor/stack-monitoring/es-local-exporter.md +++ b/deploy-manage/monitor/stack-monitoring/es-local-exporter.md @@ -8,7 +8,7 @@ applies_to: # Local exporters [local-exporter] -::::{important} +::::{important} {{agent}} and {{metricbeat}} are the recommended methods for collecting and shipping monitoring data to a monitoring cluster. If you have previously configured legacy collection methods, you should migrate to using [{{agent}}](collecting-monitoring-data-with-elastic-agent.md) or [{{metricbeat}}](collecting-monitoring-data-with-metricbeat.md) collection. Do not use legacy collection alongside other collection methods. @@ -39,7 +39,7 @@ The elected master node is the only node to set up resources for the `local` exp One benefit of the `local` exporter is that it lives within the cluster and therefore no extra configuration is required when the cluster is secured with {{stack}} {{security-features}}. All operations, including indexing operations, that occur from a `local` exporter make use of the internal transport mechanisms within {{es}}. This behavior enables the exporter to be used without providing any user credentials when {{security-features}} are enabled. -For more information about the configuration options for the `local` exporter, see [Local exporter settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/monitoring-settings.md#local-exporter-settings). +For more information about the configuration options for the `local` exporter, see [Local exporter settings](elasticsearch://reference/elasticsearch/configuration-reference/monitoring-settings.md#local-exporter-settings). ## Cleaner service [local-exporter-cleaner] diff --git a/deploy-manage/monitor/stack-monitoring/es-monitoring-collectors.md b/deploy-manage/monitor/stack-monitoring/es-monitoring-collectors.md index fbc39ac14..9bfe49790 100644 --- a/deploy-manage/monitor/stack-monitoring/es-monitoring-collectors.md +++ b/deploy-manage/monitor/stack-monitoring/es-monitoring-collectors.md @@ -9,7 +9,7 @@ applies_to: # Collectors [es-monitoring-collectors] -::::{important} +::::{important} {{agent}} and {{metricbeat}} are the recommended methods for collecting and shipping monitoring data to a monitoring cluster. If you have previously configured legacy collection methods, you should migrate to using [{{agent}}](collecting-monitoring-data-with-elastic-agent.md) or [{{metricbeat}}](collecting-monitoring-data-with-metricbeat.md) collection. Do not use legacy collection alongside other collection methods. @@ -40,23 +40,23 @@ Once collection has completed, all of the monitoring data is passed to the expor If gaps exist in the monitoring charts in {{kib}}, it is typically because either a collector failed or the monitoring cluster did not receive the data (for example, it was being restarted). In the event that a collector fails, a logged error should exist on the node that attempted to perform the collection. -::::{note} +::::{note} Collection is currently done serially, rather than in parallel, to avoid extra overhead on the elected master node. The downside to this approach is that collectors might observe a different version of the cluster state within the same collection period. In practice, this does not make a significant difference and running the collectors in parallel would not prevent such a possibility. :::: -For more information about the configuration options for the collectors, see [Monitoring collection settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/monitoring-settings.md#monitoring-collection-settings). +For more information about the configuration options for the collectors, see [Monitoring collection settings](elasticsearch://reference/elasticsearch/configuration-reference/monitoring-settings.md#monitoring-collection-settings). -## Collecting data from across the Elastic Stack [es-monitoring-stack] +## Collecting data from across the Elastic Stack [es-monitoring-stack] {{es}} {{monitor-features}} also receive monitoring data from other parts of the Elastic Stack. In this way, it serves as an unscheduled monitoring data collector for the stack. -By default, data collection is disabled. {{es}} monitoring data is not collected and all monitoring data from other sources such as {{kib}}, Beats, and Logstash is ignored. You must set `xpack.monitoring.collection.enabled` to `true` to enable the collection of monitoring data. See [Monitoring settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/monitoring-settings.md). +By default, data collection is disabled. {{es}} monitoring data is not collected and all monitoring data from other sources such as {{kib}}, Beats, and Logstash is ignored. You must set `xpack.monitoring.collection.enabled` to `true` to enable the collection of monitoring data. See [Monitoring settings](elasticsearch://reference/elasticsearch/configuration-reference/monitoring-settings.md). Once data is received, it is forwarded to the exporters to be routed to the monitoring cluster like all monitoring data. -::::{warning} +::::{warning} Because this stack-level "collector" lives outside of the collection interval of {{es}} {{monitor-features}}, it is not impacted by the `xpack.monitoring.collection.interval` setting. Therefore, data is passed to the exporters whenever it is received. This behavior can result in indices for {{kib}}, Logstash, or Beats being created somewhat unexpectedly. :::: diff --git a/deploy-manage/monitor/stack-monitoring/es-monitoring-exporters.md b/deploy-manage/monitor/stack-monitoring/es-monitoring-exporters.md index 25975f00f..d15f010a9 100644 --- a/deploy-manage/monitor/stack-monitoring/es-monitoring-exporters.md +++ b/deploy-manage/monitor/stack-monitoring/es-monitoring-exporters.md @@ -9,7 +9,7 @@ applies_to: # Exporters [es-monitoring-exporters] -::::{important} +::::{important} {{agent}} and {{metricbeat}} are the recommended methods for collecting and shipping monitoring data to a monitoring cluster. If you have previously configured legacy collection methods, you should migrate to using [{{agent}}](collecting-monitoring-data-with-elastic-agent.md) or [{{metricbeat}}](collecting-monitoring-data-with-metricbeat.md) collection. Do not use legacy collection alongside other collection methods. @@ -31,7 +31,7 @@ Both exporters serve the same purpose: to set up the monitoring cluster and rout Exporters are configurable at both the node and cluster level. Cluster-wide settings, which are updated with the [`_cluster/settings` API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings), take precedence over settings in the `elasticsearch.yml` file on each node. When you update an exporter, it is completely replaced by the updated version of the exporter. -::::{important} +::::{important} It is critical that all nodes share the same setup. Otherwise, monitoring data might be routed in different ways or to different places. :::: @@ -40,15 +40,15 @@ When the exporters route monitoring data into the monitoring cluster, they use ` Routing monitoring data involves indexing it into the appropriate monitoring indices. Once the data is indexed, it exists in a monitoring index that, by default, is named with a daily index pattern. For {{es}} monitoring data, this is an index that matches `.monitoring-es-6-*`. From there, the data lives inside the monitoring cluster and must be curated or cleaned up as necessary. If you do not curate the monitoring data, it eventually fills up the nodes and the cluster might fail due to lack of disk space. -::::{tip} -You are strongly recommended to manage the curation of indices and particularly the monitoring indices. To do so, you can take advantage of the [cleaner service](es-local-exporter.md#local-exporter-cleaner) or [Elastic Curator](asciidocalypse://docs/curator/docs/reference/index.md). +::::{tip} +You are strongly recommended to manage the curation of indices and particularly the monitoring indices. To do so, you can take advantage of the [cleaner service](es-local-exporter.md#local-exporter-cleaner) or [Elastic Curator](curator://reference/index.md). :::: -There is also a disk watermark (known as the flood stage watermark), which protects clusters from running out of disk space. When this feature is triggered, it makes all indices (including monitoring indices) read-only until the issue is fixed and a user manually makes the index writeable again. While an active monitoring index is read-only, it will naturally fail to write (index) new data and will continuously log errors that indicate the write failure. For more information, see [Disk-based shard allocation settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#disk-based-shard-allocation). +There is also a disk watermark (known as the flood stage watermark), which protects clusters from running out of disk space. When this feature is triggered, it makes all indices (including monitoring indices) read-only until the issue is fixed and a user manually makes the index writeable again. While an active monitoring index is read-only, it will naturally fail to write (index) new data and will continuously log errors that indicate the write failure. For more information, see [Disk-based shard allocation settings](elasticsearch://reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#disk-based-shard-allocation). -## Default exporters [es-monitoring-default-exporter] +## Default exporters [es-monitoring-default-exporter] If a node or cluster does not explicitly define an exporter, the following default exporter is used: @@ -63,7 +63,7 @@ xpack.monitoring.exporters.default_local: <1> If another exporter is already defined, the default exporter is *not* created. When you define a new exporter, if the default exporter exists, it is automatically removed. -## Exporter templates and ingest pipelines [es-monitoring-templates] +## Exporter templates and ingest pipelines [es-monitoring-templates] Before exporters can route monitoring data, they must set up certain {{es}} resources. These resources include templates and ingest pipelines. The following table lists the templates that are required before an exporter can route monitoring data: @@ -77,9 +77,9 @@ Before exporters can route monitoring data, they must set up certain {{es}} reso The templates are ordinary {{es}} templates that control the default settings and mappings for the monitoring indices. -By default, monitoring indices are created daily (for example, `.monitoring-es-6-2017.08.26`). You can change the default date suffix for monitoring indices with the `index.name.time_format` setting. You can use this setting to control how frequently monitoring indices are created by a specific `http` exporter. You cannot use this setting with `local` exporters. For more information, see [HTTP exporter settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/monitoring-settings.md#http-exporter-settings). +By default, monitoring indices are created daily (for example, `.monitoring-es-6-2017.08.26`). You can change the default date suffix for monitoring indices with the `index.name.time_format` setting. You can use this setting to control how frequently monitoring indices are created by a specific `http` exporter. You cannot use this setting with `local` exporters. For more information, see [HTTP exporter settings](elasticsearch://reference/elasticsearch/configuration-reference/monitoring-settings.md#http-exporter-settings). -::::{warning} +::::{warning} Some users create their own templates that match *all* index patterns, which therefore impact the monitoring indices that get created. It is critical that you do not disable `_source` storage for the monitoring indices. If you do, {{kib}} {{monitor-features}} do not work and you cannot visualize monitoring data for your cluster. :::: @@ -93,14 +93,14 @@ The following table lists the ingest pipelines that are required before an expor Exporters handle the setup of these resources before ever sending data. If resource setup fails (for example, due to security permissions), no data is sent and warnings are logged. -::::{note} +::::{note} Empty pipelines are evaluated on the coordinating node during indexing and they are ignored without any extra effort. This inherently makes them a safe, no-op operation. :::: For monitoring clusters that have disabled `node.ingest` on all nodes, it is possible to disable the use of the ingest pipeline feature. However, doing so blocks its purpose, which is to upgrade older monitoring data as our mappings improve over time. Beginning in 6.0, the ingest pipeline feature is a requirement on the monitoring cluster; you must have `node.ingest` enabled on at least one node. -::::{warning} +::::{warning} Once any node running 5.5 or later has set up the templates and ingest pipeline on a monitoring cluster, you must use {{kib}} 5.5 or later to view all subsequent data on the monitoring cluster. The easiest way to determine whether this update has occurred is by checking for the presence of indices matching `.monitoring-es-6-*` (or more concretely the existence of the new pipeline). Versions prior to 5.5 used `.monitoring-es-2-*`. :::: diff --git a/deploy-manage/monitor/stack-monitoring/kibana-monitoring-data.md b/deploy-manage/monitor/stack-monitoring/kibana-monitoring-data.md index 2bc817565..a0639141c 100644 --- a/deploy-manage/monitor/stack-monitoring/kibana-monitoring-data.md +++ b/deploy-manage/monitor/stack-monitoring/kibana-monitoring-data.md @@ -33,7 +33,7 @@ If you use a separate monitoring cluster to store the monitoring data, it is str To learn more about typical monitoring architectures, see [How monitoring works](../stack-monitoring.md) and [Monitoring in a production environment](elasticsearch-monitoring-self-managed.md). -2. Verify that `monitoring.ui.enabled` is set to `true`, which is the default value, in the `kibana.yml` file. For more information, see [Monitoring settings](asciidocalypse://docs/kibana/docs/reference/configuration-reference/monitoring-settings.md). +2. Verify that `monitoring.ui.enabled` is set to `true`, which is the default value, in the `kibana.yml` file. For more information, see [Monitoring settings](kibana://reference/configuration-reference/monitoring-settings.md). 3. If the Elastic {{security-features}} are enabled on the monitoring cluster, you must provide a user ID and password so {{kib}} can retrieve the data. 1. Create a user that has the `monitoring_user` [built-in role](../../users-roles/cluster-or-deployment-auth/built-in-roles.md) on the monitoring cluster. diff --git a/deploy-manage/monitor/stack-monitoring/kibana-monitoring-legacy.md b/deploy-manage/monitor/stack-monitoring/kibana-monitoring-legacy.md index d54ffdfb2..99881adc9 100644 --- a/deploy-manage/monitor/stack-monitoring/kibana-monitoring-legacy.md +++ b/deploy-manage/monitor/stack-monitoring/kibana-monitoring-legacy.md @@ -61,9 +61,9 @@ To learn about monitoring in general, see [Monitor a cluster](../../monitor.md). } ``` - For more information, see [Monitoring settings in {{es}}](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/monitoring-settings.md) and [Cluster update settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings). + For more information, see [Monitoring settings in {{es}}](elasticsearch://reference/elasticsearch/configuration-reference/monitoring-settings.md) and [Cluster update settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings). -2. Verify that `monitoring.enabled` and `monitoring.kibana.collection.enabled` are set to `true` in the `kibana.yml` file. These are the default values. For more information, see [Monitoring settings in {{kib}}](asciidocalypse://docs/kibana/docs/reference/configuration-reference/monitoring-settings.md). +2. Verify that `monitoring.enabled` and `monitoring.kibana.collection.enabled` are set to `true` in the `kibana.yml` file. These are the default values. For more information, see [Monitoring settings in {{kib}}](kibana://reference/configuration-reference/monitoring-settings.md). 3. Identify where to send monitoring data. {{kib}} automatically sends metrics to the {{es}} cluster specified in the `elasticsearch.hosts` setting in the `kibana.yml` file. This property has a default value of `http://localhost:9200`.
::::{tip} diff --git a/deploy-manage/monitor/stack-monitoring/kibana-monitoring-metricbeat.md b/deploy-manage/monitor/stack-monitoring/kibana-monitoring-metricbeat.md index 0cc7fef06..69e5c0769 100644 --- a/deploy-manage/monitor/stack-monitoring/kibana-monitoring-metricbeat.md +++ b/deploy-manage/monitor/stack-monitoring/kibana-monitoring-metricbeat.md @@ -28,7 +28,7 @@ To learn about monitoring in general, see [Monitor a cluster](../../monitor.md). monitoring.kibana.collection.enabled: false ``` - Leave the `monitoring.enabled` set to its default value (`true`). For more information, see [Monitoring settings in {{kib}}](asciidocalypse://docs/kibana/docs/reference/configuration-reference/monitoring-settings.md). + Leave the `monitoring.enabled` set to its default value (`true`). For more information, see [Monitoring settings in {{kib}}](kibana://reference/configuration-reference/monitoring-settings.md). 2. [Start {{kib}}](../../maintenance/start-stop-services/start-stop-kibana.md). 3. Set the `xpack.monitoring.collection.enabled` setting to `true` on each node in the production cluster. By default, it is disabled (`false`). @@ -63,7 +63,7 @@ To learn about monitoring in general, see [Monitor a cluster](../../monitor.md). } ``` - For more information, see [Monitoring settings in {{es}}](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/monitoring-settings.md) and [Cluster update settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings). + For more information, see [Monitoring settings in {{es}}](elasticsearch://reference/elasticsearch/configuration-reference/monitoring-settings.md) and [Cluster update settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings). 4. [Install {{metricbeat}}](asciidocalypse://docs/beats/docs/reference/metricbeat/metricbeat-installation-configuration.md) on the same server as {{kib}}. 5. Enable the {{kib}} {{xpack}} module in {{metricbeat}}.
diff --git a/deploy-manage/production-guidance/availability-and-resilience.md b/deploy-manage/production-guidance/availability-and-resilience.md index 278b17e1d..af4cdbd9d 100644 --- a/deploy-manage/production-guidance/availability-and-resilience.md +++ b/deploy-manage/production-guidance/availability-and-resilience.md @@ -10,7 +10,7 @@ Distributed systems like {{es}} are designed to keep working even if some of the There is a limit to how small a resilient cluster can be. All {{es}} clusters require the following components to function: * One [elected master node](../distributed-architecture/discovery-cluster-formation/modules-discovery-quorums.md) -* At least one node for each [role](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md) +* At least one node for each [role](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md) * At least one copy of every [shard](../../deploy-manage/index.md) A resilient cluster requires redundancy for every required cluster component. This means a resilient cluster must have the following components: diff --git a/deploy-manage/production-guidance/availability-and-resilience/resilience-in-larger-clusters.md b/deploy-manage/production-guidance/availability-and-resilience/resilience-in-larger-clusters.md index 529d3f5dd..91ce725c6 100644 --- a/deploy-manage/production-guidance/availability-and-resilience/resilience-in-larger-clusters.md +++ b/deploy-manage/production-guidance/availability-and-resilience/resilience-in-larger-clusters.md @@ -9,7 +9,7 @@ It’s not unusual for nodes to share common infrastructure, such as network int {{es}} expects node-to-node connections to be reliable, have low latency, and have adequate bandwidth. Many {{es}} tasks require multiple round-trips between nodes. A slow or unreliable interconnect may have a significant effect on the performance and stability of your cluster. -For example, a few milliseconds of latency added to each round-trip can quickly accumulate into a noticeable performance penalty. An unreliable network may have frequent network partitions. {{es}} will automatically recover from a network partition as quickly as it can but your cluster may be partly unavailable during a partition and will need to spend time and resources to [resynchronize any missing data](../../distributed-architecture/shard-allocation-relocation-recovery.md#shard-recovery) and [rebalance](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#shards-rebalancing-settings) itself once the partition heals. Recovering from a failure may involve copying a large amount of data between nodes so the recovery time is often determined by the available bandwidth. +For example, a few milliseconds of latency added to each round-trip can quickly accumulate into a noticeable performance penalty. An unreliable network may have frequent network partitions. {{es}} will automatically recover from a network partition as quickly as it can but your cluster may be partly unavailable during a partition and will need to spend time and resources to [resynchronize any missing data](../../distributed-architecture/shard-allocation-relocation-recovery.md#shard-recovery) and [rebalance](elasticsearch://reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#shards-rebalancing-settings) itself once the partition heals. Recovering from a failure may involve copying a large amount of data between nodes so the recovery time is often determined by the available bandwidth. If you’ve divided your cluster into zones, the network connections within each zone are typically of higher quality than the connections between the zones. Ensure the network connections between zones are of sufficiently high quality. You will see the best results by locating all your zones within a single data center with each zone having its own independent power supply and other supporting infrastructure. You can also *stretch* your cluster across nearby data centers as long as the network interconnection between each pair of data centers is good enough. diff --git a/deploy-manage/production-guidance/availability-and-resilience/resilience-in-small-clusters.md b/deploy-manage/production-guidance/availability-and-resilience/resilience-in-small-clusters.md index 8cbb4ca47..d5ffea269 100644 --- a/deploy-manage/production-guidance/availability-and-resilience/resilience-in-small-clusters.md +++ b/deploy-manage/production-guidance/availability-and-resilience/resilience-in-small-clusters.md @@ -11,7 +11,7 @@ In smaller clusters, it is most important to be resilient to single-node failure If your cluster consists of one node, that single node must do everything. To accommodate this, {{es}} assigns nodes every role by default. -A single node cluster is not resilient. If the node fails, the cluster will stop working. Because there are no replicas in a one-node cluster, you cannot store your data redundantly. However, by default at least one replica is required for a [`green` cluster health status](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-health). To ensure your cluster can report a `green` status, override the default by setting [`index.number_of_replicas`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md) to `0` on every index. +A single node cluster is not resilient. If the node fails, the cluster will stop working. Because there are no replicas in a one-node cluster, you cannot store your data redundantly. However, by default at least one replica is required for a [`green` cluster health status](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-health). To ensure your cluster can report a `green` status, override the default by setting [`index.number_of_replicas`](elasticsearch://reference/elasticsearch/index-settings/index-modules.md) to `0` on every index. If the node fails, you may need to restore an older copy of any lost indices from a [snapshot](../../tools/snapshot-and-restore.md). @@ -20,7 +20,7 @@ Because they are not resilient to any failures, we do not recommend using one-no ## Two-node clusters [high-availability-cluster-design-two-nodes] -If you have two nodes, we recommend they both be data nodes. You should also ensure every shard is stored redundantly on both nodes by setting [`index.number_of_replicas`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md) to `1` on every index that is not a [searchable snapshot index](../../tools/snapshot-and-restore/searchable-snapshots.md). This is the default behaviour but may be overridden by an [index template](../../../manage-data/data-store/templates.md). [Auto-expand replicas](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md) can also achieve the same thing, but it’s not necessary to use this feature in such a small cluster. +If you have two nodes, we recommend they both be data nodes. You should also ensure every shard is stored redundantly on both nodes by setting [`index.number_of_replicas`](elasticsearch://reference/elasticsearch/index-settings/index-modules.md) to `1` on every index that is not a [searchable snapshot index](../../tools/snapshot-and-restore/searchable-snapshots.md). This is the default behaviour but may be overridden by an [index template](../../../manage-data/data-store/templates.md). [Auto-expand replicas](elasticsearch://reference/elasticsearch/index-settings/index-modules.md) can also achieve the same thing, but it’s not necessary to use this feature in such a small cluster. We recommend you set only one of your two nodes to be [master-eligible](../../distributed-architecture/clusters-nodes-shards/node-roles.md#master-node-role). This means you can be certain which of your nodes is the elected master of the cluster. The cluster can tolerate the loss of the other master-ineligible node. If you set both nodes to master-eligible, two nodes are required for a master election. Since the election will fail if either node is unavailable, your cluster cannot reliably tolerate the loss of either node. diff --git a/deploy-manage/production-guidance/general-recommendations.md b/deploy-manage/production-guidance/general-recommendations.md index d5f148aa5..57537a71b 100644 --- a/deploy-manage/production-guidance/general-recommendations.md +++ b/deploy-manage/production-guidance/general-recommendations.md @@ -6,16 +6,16 @@ mapped_pages: # General recommendations [general-recommendations] -## Don’t return large result sets [large-size] +## Don’t return large result sets [large-size] -Elasticsearch is designed as a search engine, which makes it very good at getting back the top documents that match a query. However, it is not as good for workloads that fall into the database domain, such as retrieving all documents that match a particular query. If you need to do this, make sure to use the [Scroll](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/paginate-search-results.md#scroll-search-results) API. +Elasticsearch is designed as a search engine, which makes it very good at getting back the top documents that match a query. However, it is not as good for workloads that fall into the database domain, such as retrieving all documents that match a particular query. If you need to do this, make sure to use the [Scroll](elasticsearch://reference/elasticsearch/rest-apis/paginate-search-results.md#scroll-search-results) API. -## Avoid large documents [maximum-document-size] +## Avoid large documents [maximum-document-size] -Given that the default [`http.max_content_length`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md#http-settings) is set to 100MB, Elasticsearch will refuse to index any document that is larger than that. You might decide to increase that particular setting, but Lucene still has a limit of about 2GB. +Given that the default [`http.max_content_length`](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#http-settings) is set to 100MB, Elasticsearch will refuse to index any document that is larger than that. You might decide to increase that particular setting, but Lucene still has a limit of about 2GB. -Even without considering hard limits, large documents are usually not practical. Large documents put more stress on network, memory usage and disk, even for search requests that do not request the `_source` since Elasticsearch needs to fetch the `_id` of the document in all cases, and the cost of getting this field is bigger for large documents due to how the filesystem cache works. Indexing this document can use an amount of memory that is a multiplier of the original size of the document. Proximity search (phrase queries for instance) and [highlighting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/highlighting.md) also become more expensive since their cost directly depends on the size of the original document. +Even without considering hard limits, large documents are usually not practical. Large documents put more stress on network, memory usage and disk, even for search requests that do not request the `_source` since Elasticsearch needs to fetch the `_id` of the document in all cases, and the cost of getting this field is bigger for large documents due to how the filesystem cache works. Indexing this document can use an amount of memory that is a multiplier of the original size of the document. Proximity search (phrase queries for instance) and [highlighting](elasticsearch://reference/elasticsearch/rest-apis/highlighting.md) also become more expensive since their cost directly depends on the size of the original document. It is sometimes useful to reconsider what the unit of information should be. For instance, the fact you want to make books searchable doesn’t necessarily mean that a document should consist of a whole book. It might be a better idea to use chapters or even paragraphs as documents, and then have a property in these documents that identifies which book they belong to. This does not only avoid the issues with large documents, it also makes the search experience better. For instance if a user searches for two words `foo` and `bar`, a match across different chapters is probably very poor, while a match within the same paragraph is likely good. diff --git a/deploy-manage/production-guidance/kibana-alerting-production-considerations.md b/deploy-manage/production-guidance/kibana-alerting-production-considerations.md index bece9d8e8..4d872e0eb 100644 --- a/deploy-manage/production-guidance/kibana-alerting-production-considerations.md +++ b/deploy-manage/production-guidance/kibana-alerting-production-considerations.md @@ -14,7 +14,7 @@ Alerting runs both rule checks and actions as persistent background tasks manage When relying on rules and actions as mission critical services, make sure you follow the [production considerations](../distributed-architecture/kibana-tasks-management.md) for Task Manager. -## Running background rule checks and actions [alerting-background-tasks] +## Running background rule checks and actions [alerting-background-tasks] {{kib}} uses background tasks to run rules and actions, distributed across all {{kib}} instances in the cluster. @@ -24,10 +24,10 @@ Rules are recurring background tasks which are rescheduled according to the chec For more details on Task Manager, see [Running background tasks](../distributed-architecture/kibana-tasks-management.md#task-manager-background-tasks). -::::{important} +::::{important} Rule and action tasks can run late or at an inconsistent schedule. This is typically a symptom of the specific usage of the cluster in question. -You can address such issues by tweaking the [Task Manager settings](asciidocalypse://docs/kibana/docs/reference/configuration-reference/task-manager-settings.md#task-manager-settings) or scaling the deployment to better suit your use case. +You can address such issues by tweaking the [Task Manager settings](kibana://reference/configuration-reference/task-manager-settings.md#task-manager-settings) or scaling the deployment to better suit your use case. For detailed guidance, see [Alerting Troubleshooting](../../explore-analyze/alerts-cases/alerts/alerting-troubleshooting.md). @@ -35,7 +35,7 @@ For detailed guidance, see [Alerting Troubleshooting](../../explore-analyze/aler -## Scaling guidance [alerting-scaling-guidance] +## Scaling guidance [alerting-scaling-guidance] As rules and actions leverage background tasks to perform the majority of work, scaling Alerting is possible by following the [Task Manager Scaling Guidance](../distributed-architecture/kibana-tasks-management.md#task-manager-scaling-guidance). @@ -49,9 +49,9 @@ It is difficult to predict how much throughput is needed to ensure all rules and Predicting the buffer required to account for actions depends heavily on the rule types you use, the amount of alerts they might detect, and the number of actions you might choose to assign to action groups. With that in mind, regularly [monitor the health](../monitor/kibana-task-manager-health-monitoring.md) of your Task Manager instances. -## Event log index lifecycle management [event-log-ilm] +## Event log index lifecycle management [event-log-ilm] -::::{warning} +::::{warning} This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. :::: @@ -61,19 +61,19 @@ Alerts and actions log activity in a set of "event log" data streams, one per Ki For more information on data stream lifecycle management, see: [Data stream lifecycle](../../manage-data/lifecycle/data-stream.md). -## Circuit breakers [alerting-circuit-breakers] +## Circuit breakers [alerting-circuit-breakers] -There are several scenarios where running alerting rules and actions can start to negatively impact the overall health of a {{kib}} instance either by clogging up Task Manager throughput or by consuming so much CPU/memory that other operations cannot complete in a reasonable amount of time. There are several [configurable](asciidocalypse://docs/kibana/docs/reference/configuration-reference/alerting-settings.md#alert-settings) circuit breakers to help minimize these effects. +There are several scenarios where running alerting rules and actions can start to negatively impact the overall health of a {{kib}} instance either by clogging up Task Manager throughput or by consuming so much CPU/memory that other operations cannot complete in a reasonable amount of time. There are several [configurable](kibana://reference/configuration-reference/alerting-settings.md#alert-settings) circuit breakers to help minimize these effects. -### Rules with very short intervals [_rules_with_very_short_intervals] +### Rules with very short intervals [_rules_with_very_short_intervals] Running large numbers of rules at very short intervals can quickly clog up Task Manager throughput, leading to higher schedule drift. Use `xpack.alerting.rules.minimumScheduleInterval.value` to set a minimum schedule interval for rules. The default (and recommended) value for this configuration is `1m`. Use `xpack.alerting.rules.minimumScheduleInterval.enforce` to specify whether to strictly enforce this minimum. While the default value for this setting is `false` to maintain backwards compatibility with existing rules, set this to `true` to prevent new and updated rules from running at an interval below the minimum. Another related setting is `xpack.alerting.rules.maxScheduledPerMinute`, which limits the number of rules that can run per minute. For example if it’s set to `400`, you can have 400 rules with one minute check intervals or 2,000 rules with 5 minute check intervals. You cannot create or edit a rule if its check interval would cause this setting to be exceeded. To stay within this limit, delete or disable some rules or update the check intervals so that your rules run less frequently. -### Rules that run for a long time [_rules_that_run_for_a_long_time] +### Rules that run for a long time [_rules_that_run_for_a_long_time] Rules that run for a long time typically do so because they are issuing resource-intensive {{es}} queries or performing CPU-intensive processing. This can block the event loop, making {{kib}} inaccessible while the rule runs. By default, rule processing is cancelled after `5m` but this can be overriden using the `xpack.alerting.rules.run.timeout` configuration. This value can also be configured per rule type using `xpack.alerting.rules.run.ruleTypeOverrides`. For example, the following configuration sets the global timeout value to `1m` while allowing **Index Threshold** rules to run for `10m` before being cancelled. @@ -88,7 +88,7 @@ xpack.alerting.rules.run: When a rule run is cancelled, any alerts and actions that were generated during the run are discarded. This behavior is controlled by the `xpack.alerting.cancelAlertsOnRuleTimeout` configuration, which defaults to `true`. Set this to `false` to receive alerts and actions after the timeout, although be aware that these may be incomplete and possibly inaccurate. -### Rules that spawn too many actions [_rules_that_spawn_too_many_actions] +### Rules that spawn too many actions [_rules_that_spawn_too_many_actions] Rules that spawn too many actions can quickly clog up Task Manager throughput. This can occur if: diff --git a/deploy-manage/production-guidance/kibana-task-manager-scaling-considerations.md b/deploy-manage/production-guidance/kibana-task-manager-scaling-considerations.md index 8ad4370e6..487b7ecce 100644 --- a/deploy-manage/production-guidance/kibana-task-manager-scaling-considerations.md +++ b/deploy-manage/production-guidance/kibana-task-manager-scaling-considerations.md @@ -11,7 +11,7 @@ mapped_pages: * **Scaling**: Multiple {{kib}} instances can read from and update the same task queue in {{es}}, allowing the work load to be distributed across instances. If a {{kib}} instance no longer has capacity to run tasks, you can increase capacity by adding additional {{kib}} instances. * **Load Balancing**: Task Manager is equipped with a reactive self-healing mechanism, which allows it to reduce the amount of work it executes in reaction to an increased load related error rate in {{es}}. Additionally, when Task Manager experiences an increase in recurring tasks, it attempts to space out the work to better balance the load. -::::{important} +::::{important} Task definitions for alerts and actions are stored in the index called `.kibana_task_manager`. You must have at least one replica of this index for production deployments. @@ -22,11 +22,11 @@ If you lose this index, all scheduled alerts and actions are lost. -## Running background tasks [task-manager-background-tasks] +## Running background tasks [task-manager-background-tasks] {{kib}} background tasks are managed as follows: -* An {{es}} task index is polled for overdue tasks at 3-second intervals. You can change this interval using the [`xpack.task_manager.poll_interval`](asciidocalypse://docs/kibana/docs/reference/configuration-reference/task-manager-settings.md#task-manager-settings) setting. +* An {{es}} task index is polled for overdue tasks at 3-second intervals. You can change this interval using the [`xpack.task_manager.poll_interval`](kibana://reference/configuration-reference/task-manager-settings.md#task-manager-settings) setting. * Tasks are claimed by updating them in the {{es}} index, using optimistic concurrency control to prevent conflicts. Each {{kib}} instance can run a maximum of 10 concurrent tasks, so a maximum of 10 tasks are claimed each interval. * Tasks are run on the {{kib}} server. * Task Manager ensures that tasks: @@ -36,14 +36,14 @@ If you lose this index, all scheduled alerts and actions are lost. * Are rescheduled to run again at a future point in time (if configured to do so) -::::{important} +::::{important} It is possible for tasks to run late or at an inconsistent schedule. This is usually a symptom of the specific usage or scaling strategy of the cluster in question. To address these issues, tweak the {{kib}} Task Manager settings or the cluster scaling strategy to better suit the unique use case. -For details on the settings that can influence the performance and throughput of Task Manager, see [Task Manager Settings](asciidocalypse://docs/kibana/docs/reference/configuration-reference/task-manager-settings.md). +For details on the settings that can influence the performance and throughput of Task Manager, see [Task Manager Settings](kibana://reference/configuration-reference/task-manager-settings.md). For detailed troubleshooting guidance, see [Troubleshooting](../../troubleshoot/kibana/task-manager.md). @@ -51,19 +51,19 @@ For detailed troubleshooting guidance, see [Troubleshooting](../../troubleshoot/ -## Deployment considerations [_deployment_considerations] +## Deployment considerations [_deployment_considerations] {{es}} and {{kib}} instances use the system clock to determine the current time. To ensure schedules are triggered when expected, synchronize the clocks of all nodes in the cluster using a time service such as [Network Time Protocol](http://www.ntp.org/). -## Scaling guidance [task-manager-scaling-guidance] +## Scaling guidance [task-manager-scaling-guidance] How you deploy {{kib}} largely depends on your use case. Predicting the throughout a deployment might require to support Task Management is difficult because features can schedule an unpredictable number of tasks at a variety of scheduled cadences. However, there is a relatively straight forward method you can follow to produce a rough estimate based on your expected usage. -### Default scale [task-manager-default-scaling] +### Default scale [task-manager-default-scaling] By default, {{kib}} polls for tasks at a rate of 10 tasks every 3 seconds. This means that you can expect a single {{kib}} instance to support up to 200 *tasks per minute* (`200/tpm`). @@ -74,21 +74,21 @@ By [estimating a rough throughput requirement](../distributed-architecture/kiban For details on monitoring the health of {{kib}} Task Manager, follow the guidance in [Health monitoring](../monitor/kibana-task-manager-health-monitoring.md). -### Scaling horizontally [task-manager-scaling-horizontally] +### Scaling horizontally [task-manager-scaling-horizontally] At times, the sustainable approach might be to expand the throughput of your cluster by provisioning additional {{kib}} instances. By default, each additional {{kib}} instance will add an additional 10 tasks that your cluster can run concurrently, but you can also scale each {{kib}} instance vertically, if your diagnosis indicates that they can handle the additional workload. -### Scaling vertically [task-manager-scaling-vertically] +### Scaling vertically [task-manager-scaling-vertically] Other times it, might be preferable to increase the throughput of individual {{kib}} instances. -Tweak the capacity with the [`xpack.task_manager.capacity`](asciidocalypse://docs/kibana/docs/reference/configuration-reference/task-manager-settings.md#task-manager-settings) setting, which enables each {{kib}} instance to pull a higher number of tasks per interval. This setting can impact the performance of each instance as the workload will be higher. +Tweak the capacity with the [`xpack.task_manager.capacity`](kibana://reference/configuration-reference/task-manager-settings.md#task-manager-settings) setting, which enables each {{kib}} instance to pull a higher number of tasks per interval. This setting can impact the performance of each instance as the workload will be higher. -Tweak the poll interval with the [`xpack.task_manager.poll_interval`](asciidocalypse://docs/kibana/docs/reference/configuration-reference/task-manager-settings.md#task-manager-settings) setting, which enables each {{kib}} instance to pull scheduled tasks at a higher rate. This setting can impact the performance of the {{es}} cluster as the workload will be higher. +Tweak the poll interval with the [`xpack.task_manager.poll_interval`](kibana://reference/configuration-reference/task-manager-settings.md#task-manager-settings) setting, which enables each {{kib}} instance to pull scheduled tasks at a higher rate. This setting can impact the performance of the {{es}} cluster as the workload will be higher. -### Choosing a scaling strategy [task-manager-choosing-scaling-strategy] +### Choosing a scaling strategy [task-manager-choosing-scaling-strategy] Each scaling strategy comes with its own considerations, and the appropriate strategy largely depends on your use case. @@ -106,7 +106,7 @@ Task Manager, like the rest of the Elastic Stack, is designed to scale horizonta Scaling horizontally requires a higher degree of coordination between {{kib}} instances. One way Task Manager coordinates with other instances is by delaying its polling schedule to avoid conflicts with other instances. By using [health monitoring](../monitor/kibana-task-manager-health-monitoring.md) to evaluate the [date of the `last_polling_delay`](../../troubleshoot/kibana/task-manager.md#task-manager-health-evaluate-the-runtime) across a deployment, you can estimate the frequency at which Task Manager resets its delay mechanism. A higher frequency suggests {{kib}} instances conflict at a high rate, which you can address by scaling vertically rather than horizontally, reducing the required coordination. -### Rough throughput estimation [task-manager-rough-throughput-estimation] +### Rough throughput estimation [task-manager-rough-throughput-estimation] Predicting the required throughput a deployment might need to support Task Management is difficult, as features can schedule an unpredictable number of tasks at a variety of scheduled cadences. However, a rough lower bound can be estimated, which is then used as a guide. @@ -115,9 +115,9 @@ Throughput is best thought of as a measurements in tasks per minute. A default {{kib}} instance can support up to `200/tpm`. -#### Automatic estimation [_automatic_estimation] +#### Automatic estimation [_automatic_estimation] -::::{warning} +::::{warning} This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. :::: @@ -130,7 +130,7 @@ We recommend provisioning enough {{kib}} instances to ensure a buffer between th We recommend provisioning at least as many {{kib}} instances as proposed by `proposed.provisioned_kibana`, but keep in mind that this number is based on the estimated required throughput, which is based on average historical performance, and cannot accurately predict future requirements. -::::{warning} +::::{warning} Automatic capacity estimation is performed by each {{kib}} instance independently. This estimation is performed by observing the task throughput in that instance, the number of {{kib}} instances executing tasks at that moment in time, and the recurring workload in {{es}}. If a {{kib}} instance is idle at the moment of capacity estimation, the number of active {{kib}} instances might be miscounted and the available throughput miscalculated. @@ -141,7 +141,7 @@ When evaluating the proposed {{kib}} instance number under `proposed.provisioned -#### Manual estimation [_manual_estimation] +#### Manual estimation [_manual_estimation] By [evaluating the workload](../../troubleshoot/kibana/task-manager.md#task-manager-health-evaluate-the-workload), you can make a rough estimate as to the required throughput as a *tasks per minute* measurement. diff --git a/deploy-manage/production-guidance/optimize-performance/approximate-knn-search.md b/deploy-manage/production-guidance/optimize-performance/approximate-knn-search.md index 858e36783..5a9ab36b7 100644 --- a/deploy-manage/production-guidance/optimize-performance/approximate-knn-search.md +++ b/deploy-manage/production-guidance/optimize-performance/approximate-knn-search.md @@ -10,33 +10,33 @@ mapped_pages: Many of these recommendations help improve search speed. With approximate kNN, the indexing algorithm runs searches under the hood to create the vector index structures. So these same recommendations also help with indexing speed. -## Reduce vector memory foot-print [_reduce_vector_memory_foot_print] +## Reduce vector memory foot-print [_reduce_vector_memory_foot_print] -The default [`element_type`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/dense-vector.md#dense-vector-element-type) is `float`. But this can be automatically quantized during index time through [`quantization`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/dense-vector.md#dense-vector-quantization). Quantization will reduce the required memory by 4x, 8x, or as much as 32x, but it will also reduce the precision of the vectors and increase disk usage for the field (by up to 25%, 12.5%, or 3.125%, respectively). Increased disk usage is a result of {{es}} storing both the quantized and the unquantized vectors. For example, when int8 quantizing 40GB of floating point vectors an extra 10GB of data will be stored for the quantized vectors. The total disk usage amounts to 50GB, but the memory usage for fast search will be reduced to 10GB. +The default [`element_type`](elasticsearch://reference/elasticsearch/mapping-reference/dense-vector.md#dense-vector-element-type) is `float`. But this can be automatically quantized during index time through [`quantization`](elasticsearch://reference/elasticsearch/mapping-reference/dense-vector.md#dense-vector-quantization). Quantization will reduce the required memory by 4x, 8x, or as much as 32x, but it will also reduce the precision of the vectors and increase disk usage for the field (by up to 25%, 12.5%, or 3.125%, respectively). Increased disk usage is a result of {{es}} storing both the quantized and the unquantized vectors. For example, when int8 quantizing 40GB of floating point vectors an extra 10GB of data will be stored for the quantized vectors. The total disk usage amounts to 50GB, but the memory usage for fast search will be reduced to 10GB. -For `float` vectors with `dim` greater than or equal to `384`, using a [`quantized`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/dense-vector.md#dense-vector-quantization) index is highly recommended. +For `float` vectors with `dim` greater than or equal to `384`, using a [`quantized`](elasticsearch://reference/elasticsearch/mapping-reference/dense-vector.md#dense-vector-quantization) index is highly recommended. -## Reduce vector dimensionality [_reduce_vector_dimensionality] +## Reduce vector dimensionality [_reduce_vector_dimensionality] The speed of kNN search scales linearly with the number of vector dimensions, because each similarity computation considers each element in the two vectors. Whenever possible, it’s better to use vectors with a lower dimension. Some embedding models come in different "sizes", with both lower and higher dimensional options available. You could also experiment with dimensionality reduction techniques like PCA. When experimenting with different approaches, it’s important to measure the impact on relevance to ensure the search quality is still acceptable. -## Exclude vector fields from `_source` [_exclude_vector_fields_from_source] +## Exclude vector fields from `_source` [_exclude_vector_fields_from_source] -{{es}} stores the original JSON document that was passed at index time in the [`_source` field](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md). By default, each hit in the search results contains the full document `_source`. When the documents contain high-dimensional `dense_vector` fields, the `_source` can be quite large and expensive to load. This could significantly slow down the speed of kNN search. +{{es}} stores the original JSON document that was passed at index time in the [`_source` field](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md). By default, each hit in the search results contains the full document `_source`. When the documents contain high-dimensional `dense_vector` fields, the `_source` can be quite large and expensive to load. This could significantly slow down the speed of kNN search. -::::{note} +::::{note} [reindex](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex), [update](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-update), and [update by query](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-update-by-query) operations generally require the `_source` field. Disabling `_source` for a field might result in unexpected behavior for these operations. For example, reindex might not actually contain the `dense_vector` field in the new index. :::: -You can disable storing `dense_vector` fields in the `_source` through the [`excludes`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md#include-exclude) mapping parameter. This prevents loading and returning large vectors during search, and also cuts down on the index size. Vectors that have been omitted from `_source` can still be used in kNN search, since it relies on separate data structures to perform the search. Before using the [`excludes`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md#include-exclude) parameter, make sure to review the downsides of omitting fields from `_source`. +You can disable storing `dense_vector` fields in the `_source` through the [`excludes`](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md#include-exclude) mapping parameter. This prevents loading and returning large vectors during search, and also cuts down on the index size. Vectors that have been omitted from `_source` can still be used in kNN search, since it relies on separate data structures to perform the search. Before using the [`excludes`](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md#include-exclude) parameter, make sure to review the downsides of omitting fields from `_source`. -Another option is to use [synthetic `_source`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source). +Another option is to use [synthetic `_source`](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source). -## Ensure data nodes have enough memory [_ensure_data_nodes_have_enough_memory] +## Ensure data nodes have enough memory [_ensure_data_nodes_have_enough_memory] {{es}} uses the [HNSW](https://arxiv.org/abs/1603.09320) algorithm for approximate kNN search. HNSW is a graph-based algorithm which only works efficiently when most vector data is held in memory. You should ensure that data nodes have at least enough RAM to hold the vector data and index structures. To check the size of the vector data, you can use the [Analyze index disk usage](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-disk-usage) API. @@ -56,11 +56,11 @@ Note that the required RAM is for the filesystem cache, which is separate from t The data nodes should also leave a buffer for other ways that RAM is needed. For example your index might also include text fields and numerics, which also benefit from using filesystem cache. It’s recommended to run benchmarks with your specific dataset to ensure there’s a sufficient amount of memory to give good search performance. You can find [here](https://elasticsearch-benchmarks.elastic.co/#tracks/so_vector) and [here](https://elasticsearch-benchmarks.elastic.co/#tracks/dense_vector) some examples of datasets and configurations that we use for our nightly benchmarks. -## Warm up the filesystem cache [dense-vector-preloading] +## Warm up the filesystem cache [dense-vector-preloading] -If the machine running Elasticsearch is restarted, the filesystem cache will be empty, so it will take some time before the operating system loads hot regions of the index into memory so that search operations are fast. You can explicitly tell the operating system which files should be loaded into memory eagerly depending on the file extension using the [`index.store.preload`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/preloading-data-into-file-system-cache.md) setting. +If the machine running Elasticsearch is restarted, the filesystem cache will be empty, so it will take some time before the operating system loads hot regions of the index into memory so that search operations are fast. You can explicitly tell the operating system which files should be loaded into memory eagerly depending on the file extension using the [`index.store.preload`](elasticsearch://reference/elasticsearch/index-settings/preloading-data-into-file-system-cache.md) setting. -::::{warning} +::::{warning} Loading data into the filesystem cache eagerly on too many indices or too many files will make search *slower* if the filesystem cache is not large enough to hold all the data. Use with caution. :::: @@ -69,47 +69,47 @@ The following file extensions are used for the approximate kNN search: Each exte * `vex` for the HNSW graph * `vec` for all non-quantized vector values. This includes all element types: `float`, `byte`, and `bit`. -* `veq` for quantized vectors indexed with [`quantization`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/dense-vector.md#dense-vector-quantization): `int4` or `int8` -* `veb` for binary vectors indexed with [`quantization`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/dense-vector.md#dense-vector-quantization): `bbq` +* `veq` for quantized vectors indexed with [`quantization`](elasticsearch://reference/elasticsearch/mapping-reference/dense-vector.md#dense-vector-quantization): `int4` or `int8` +* `veb` for binary vectors indexed with [`quantization`](elasticsearch://reference/elasticsearch/mapping-reference/dense-vector.md#dense-vector-quantization): `bbq` * `vem`, `vemf`, `vemq`, and `vemb` for metadata, usually small and not a concern for preloading Generally, if you are using a quantized index, you should only preload the relevant quantized values and the HNSW graph. Preloading the raw vectors is not necessary and might be counterproductive. -## Reduce the number of index segments [_reduce_the_number_of_index_segments] +## Reduce the number of index segments [_reduce_the_number_of_index_segments] -{{es}} shards are composed of segments, which are internal storage elements in the index. For approximate kNN search, {{es}} stores the vector values of each segment as a separate HNSW graph, so kNN search must check each segment. The recent parallelization of kNN search made it much faster to search across multiple segments, but still kNN search can be up to several times faster if there are fewer segments. By default, {{es}} periodically merges smaller segments into larger ones through a background [merge process](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/merge.md). If this isn’t sufficient, you can take explicit steps to reduce the number of index segments. +{{es}} shards are composed of segments, which are internal storage elements in the index. For approximate kNN search, {{es}} stores the vector values of each segment as a separate HNSW graph, so kNN search must check each segment. The recent parallelization of kNN search made it much faster to search across multiple segments, but still kNN search can be up to several times faster if there are fewer segments. By default, {{es}} periodically merges smaller segments into larger ones through a background [merge process](elasticsearch://reference/elasticsearch/index-settings/merge.md). If this isn’t sufficient, you can take explicit steps to reduce the number of index segments. -### Increase maximum segment size [_increase_maximum_segment_size] +### Increase maximum segment size [_increase_maximum_segment_size] {{es}} provides many tunable settings for controlling the merge process. One important setting is `index.merge.policy.max_merged_segment`. This controls the maximum size of the segments that are created during the merge process. By increasing the value, you can reduce the number of segments in the index. The default value is `5GB`, but that might be too small for larger dimensional vectors. Consider increasing this value to `10GB` or `20GB` can help reduce the number of segments. -### Create large segments during bulk indexing [_create_large_segments_during_bulk_indexing] +### Create large segments during bulk indexing [_create_large_segments_during_bulk_indexing] A common pattern is to first perform an initial bulk upload, then make an index available for searches. Instead of force merging, you can adjust the index settings to encourage {{es}} to create larger initial segments: -* Ensure there are no searches during the bulk upload and disable [`index.refresh_interval`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-refresh-interval-setting) by setting it to `-1`. This prevents refresh operations and avoids creating extra segments. -* Give {{es}} a large indexing buffer so it can accept more documents before flushing. By default, the [`indices.memory.index_buffer_size`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/indexing-buffer-settings.md) is set to 10% of the heap size. With a substantial heap size like 32GB, this is often enough. To allow the full indexing buffer to be used, you should also increase the limit [`index.translog.flush_threshold_size`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/translog.md). +* Ensure there are no searches during the bulk upload and disable [`index.refresh_interval`](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#index-refresh-interval-setting) by setting it to `-1`. This prevents refresh operations and avoids creating extra segments. +* Give {{es}} a large indexing buffer so it can accept more documents before flushing. By default, the [`indices.memory.index_buffer_size`](elasticsearch://reference/elasticsearch/configuration-reference/indexing-buffer-settings.md) is set to 10% of the heap size. With a substantial heap size like 32GB, this is often enough. To allow the full indexing buffer to be used, you should also increase the limit [`index.translog.flush_threshold_size`](elasticsearch://reference/elasticsearch/index-settings/translog.md). -## Avoid heavy indexing during searches [_avoid_heavy_indexing_during_searches] +## Avoid heavy indexing during searches [_avoid_heavy_indexing_during_searches] Actively indexing documents can have a negative impact on approximate kNN search performance, since indexing threads steal compute resources from search. When indexing and searching at the same time, {{es}} also refreshes frequently, which creates several small segments. This also hurts search performance, since approximate kNN search is slower when there are more segments. When possible, it’s best to avoid heavy indexing during approximate kNN search. If you need to reindex all the data, perhaps because the vector embedding model changed, then it’s better to reindex the new documents into a separate index rather than update them in-place. This helps avoid the slowdown mentioned above, and prevents expensive merge operations due to frequent document updates. -## Avoid page cache thrashing by using modest readahead values on Linux [_avoid_page_cache_thrashing_by_using_modest_readahead_values_on_linux_2] +## Avoid page cache thrashing by using modest readahead values on Linux [_avoid_page_cache_thrashing_by_using_modest_readahead_values_on_linux_2] -Search can cause a lot of randomized read I/O. When the underlying block device has a high readahead value, there may be a lot of unnecessary read I/O done, especially when files are accessed using memory mapping (see [storage types](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/store.md#file-system)). +Search can cause a lot of randomized read I/O. When the underlying block device has a high readahead value, there may be a lot of unnecessary read I/O done, especially when files are accessed using memory mapping (see [storage types](elasticsearch://reference/elasticsearch/index-settings/store.md#file-system)). Most Linux distributions use a sensible readahead value of `128KiB` for a single plain device, however, when using software raid, LVM or dm-crypt the resulting block device (backing Elasticsearch [path.data](../../deploy/self-managed/important-settings-configuration.md#path-settings)) may end up having a very large readahead value (in the range of several MiB). This usually results in severe page (filesystem) cache thrashing adversely affecting search (or [update](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-document)) performance. You can check the current value in `KiB` using `lsblk -o NAME,RA,MOUNTPOINT,TYPE,SIZE`. Consult the documentation of your distribution on how to alter this value (for example with a `udev` rule to persist across reboots, or via [blockdev --setra](https://man7.org/linux/man-pages/man8/blockdev.8.html) as a transient setting). We recommend a value of `128KiB` for readahead. -::::{warning} +::::{warning} `blockdev` expects values in 512 byte sectors whereas `lsblk` reports values in `KiB`. As an example, to temporarily set readahead to `128KiB` for `/dev/nvme0n1`, specify `blockdev --setra 256 /dev/nvme0n1`. :::: diff --git a/deploy-manage/production-guidance/optimize-performance/disk-usage.md b/deploy-manage/production-guidance/optimize-performance/disk-usage.md index 7d29c1cfd..1db3ae1b3 100644 --- a/deploy-manage/production-guidance/optimize-performance/disk-usage.md +++ b/deploy-manage/production-guidance/optimize-performance/disk-usage.md @@ -24,12 +24,12 @@ PUT index } ``` -[`text`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/text.md) fields store normalization factors in the index to facilitate document scoring. If you only need matching capabilities on a `text` field but do not care about the produced scores, you can use the [`match_only_text`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/text.md#match-only-text-field-type) type instead. This field type saves significant space by dropping scoring and positional information. +[`text`](elasticsearch://reference/elasticsearch/mapping-reference/text.md) fields store normalization factors in the index to facilitate document scoring. If you only need matching capabilities on a `text` field but do not care about the produced scores, you can use the [`match_only_text`](elasticsearch://reference/elasticsearch/mapping-reference/text.md#match-only-text-field-type) type instead. This field type saves significant space by dropping scoring and positional information. ## Don’t use default dynamic string mappings [default-dynamic-string-mapping] -The default [dynamic string mappings](../../../manage-data/data-store/mapping/dynamic-mapping.md) will index string fields both as [`text`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/text.md) and [`keyword`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/keyword.md). This is wasteful if you only need one of them. Typically an `id` field will only need to be indexed as a `keyword` while a `body` field will only need to be indexed as a `text` field. +The default [dynamic string mappings](../../../manage-data/data-store/mapping/dynamic-mapping.md) will index string fields both as [`text`](elasticsearch://reference/elasticsearch/mapping-reference/text.md) and [`keyword`](elasticsearch://reference/elasticsearch/mapping-reference/keyword.md). This is wasteful if you only need one of them. Typically an `id` field will only need to be indexed as a `keyword` while a `body` field will only need to be indexed as a `text` field. This can be disabled by either configuring explicit mappings on string fields or setting up dynamic templates that will map string fields as either `text` or `keyword`. @@ -63,12 +63,12 @@ Keep in mind that large shard sizes come with drawbacks, such as long full recov ## Disable `_source` [disable-source] -The [`_source`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md) field stores the original JSON body of the document. If you don’t need access to it you can disable it. However, APIs that needs access to `_source` such as update, highlight and reindex won’t work. +The [`_source`](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md) field stores the original JSON body of the document. If you don’t need access to it you can disable it. However, APIs that needs access to `_source` such as update, highlight and reindex won’t work. ## Use `best_compression` [best-compression] -The `_source` and stored fields can easily take a non negligible amount of disk space. They can be compressed more aggressively by using the `best_compression` [codec](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-codec). +The `_source` and stored fields can easily take a non negligible amount of disk space. They can be compressed more aggressively by using the `best_compression` [codec](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#index-codec). ## Force merge [_force_merge] @@ -90,14 +90,14 @@ The [shrink API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/ope ## Use the smallest numeric type that is sufficient [_use_the_smallest_numeric_type_that_is_sufficient] -The type that you pick for [numeric data](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/number.md) can have a significant impact on disk usage. In particular, integers should be stored using an integer type (`byte`, `short`, `integer` or `long`) and floating points should either be stored in a `scaled_float` if appropriate or in the smallest type that fits the use-case: using `float` over `double`, or `half_float` over `float` will help save storage. +The type that you pick for [numeric data](elasticsearch://reference/elasticsearch/mapping-reference/number.md) can have a significant impact on disk usage. In particular, integers should be stored using an integer type (`byte`, `short`, `integer` or `long`) and floating points should either be stored in a `scaled_float` if appropriate or in the smallest type that fits the use-case: using `float` over `double`, or `half_float` over `float` will help save storage. ## Use index sorting to colocate similar documents [_use_index_sorting_to_colocate_similar_documents] When Elasticsearch stores `_source`, it compresses multiple documents at once in order to improve the overall compression ratio. For instance it is very common that documents share the same field names, and quite common that they share some field values, especially on fields that have a low cardinality or a [zipfian](https://en.wikipedia.org/wiki/Zipf%27s_law) distribution. -By default documents are compressed together in the order that they are added to the index. If you enabled [index sorting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/sorting.md) then instead they are compressed in sorted order. Sorting documents with similar structure, fields, and values together should improve the compression ratio. +By default documents are compressed together in the order that they are added to the index. If you enabled [index sorting](elasticsearch://reference/elasticsearch/index-settings/sorting.md) then instead they are compressed in sorted order. Sorting documents with similar structure, fields, and values together should improve the compression ratio. ## Put fields in the same order in documents [_put_fields_in_the_same_order_in_documents] diff --git a/deploy-manage/production-guidance/optimize-performance/indexing-speed.md b/deploy-manage/production-guidance/optimize-performance/indexing-speed.md index 36757e4f4..667b1db2a 100644 --- a/deploy-manage/production-guidance/optimize-performance/indexing-speed.md +++ b/deploy-manage/production-guidance/optimize-performance/indexing-speed.md @@ -6,12 +6,12 @@ mapped_pages: # Indexing speed [tune-for-indexing-speed] -## Use bulk requests [_use_bulk_requests] +## Use bulk requests [_use_bulk_requests] Bulk requests will yield much better performance than single-document index requests. In order to know the optimal size of a bulk request, you should run a benchmark on a single node with a single shard. First try to index 100 documents at once, then 200, then 400, etc. doubling the number of documents in a bulk request in every benchmark run. When the indexing speed starts to plateau then you know you reached the optimal size of a bulk request for your data. In case of tie, it is better to err in the direction of too few rather than too many documents. Beware that too large bulk requests might put the cluster under memory pressure when many of them are sent concurrently, so it is advisable to avoid going beyond a couple tens of megabytes per request even if larger requests seem to perform better. -## Use multiple workers/threads to send data to Elasticsearch [multiple-workers-threads] +## Use multiple workers/threads to send data to Elasticsearch [multiple-workers-threads] A single thread sending bulk requests is unlikely to be able to max out the indexing capacity of an Elasticsearch cluster. In order to use all resources of the cluster, you should send data from multiple threads or processes. In addition to making better use of the resources of the cluster, this should help reduce the cost of each fsync. @@ -20,7 +20,7 @@ Make sure to watch for `TOO_MANY_REQUESTS (429)` response codes (`EsRejectedExec Similarly to sizing bulk requests, only testing can tell what the optimal number of workers is. This can be tested by progressively increasing the number of workers until either I/O or CPU is saturated on the cluster. -## Unset or increase the refresh interval [_unset_or_increase_the_refresh_interval] +## Unset or increase the refresh interval [_unset_or_increase_the_refresh_interval] The operation that consists of making changes visible to search - called a [refresh](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-refresh) - is costly, and calling it often while there is ongoing indexing activity can hurt indexing speed. @@ -28,63 +28,63 @@ By default, Elasticsearch periodically refreshes indices every second, but only This is the optimal configuration if you have no or very little search traffic (e.g. less than one search request every 5 minutes) and want to optimize for indexing speed. This behavior aims to automatically optimize bulk indexing in the default case when no searches are performed. In order to opt out of this behavior set the refresh interval explicitly. -On the other hand, if your index experiences regular search requests, this default behavior means that Elasticsearch will refresh your index every 1 second. If you can afford to increase the amount of time between when a document gets indexed and when it becomes visible, increasing the [`index.refresh_interval`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-refresh-interval-setting) to a larger value, e.g. `30s`, might help improve indexing speed. +On the other hand, if your index experiences regular search requests, this default behavior means that Elasticsearch will refresh your index every 1 second. If you can afford to increase the amount of time between when a document gets indexed and when it becomes visible, increasing the [`index.refresh_interval`](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#index-refresh-interval-setting) to a larger value, e.g. `30s`, might help improve indexing speed. -## Disable replicas for initial loads [_disable_replicas_for_initial_loads] +## Disable replicas for initial loads [_disable_replicas_for_initial_loads] If you have a large amount of data that you want to load all at once into Elasticsearch, it may be beneficial to set `index.number_of_replicas` to `0` in order to speed up indexing. Having no replicas means that losing a single node may incur data loss, so it is important that the data lives elsewhere so that this initial load can be retried in case of an issue. Once the initial load is finished, you can set `index.number_of_replicas` back to its original value. If `index.refresh_interval` is configured in the index settings, it may further help to unset it during this initial load and setting it back to its original value once the initial load is finished. -## Disable swapping [_disable_swapping_2] +## Disable swapping [_disable_swapping_2] You should make sure that the operating system is not swapping out the java process by [disabling swapping](../../deploy/self-managed/setup-configuration-memory.md). -## Give memory to the filesystem cache [_give_memory_to_the_filesystem_cache] +## Give memory to the filesystem cache [_give_memory_to_the_filesystem_cache] The filesystem cache will be used in order to buffer I/O operations. You should make sure to give at least half the memory of the machine running Elasticsearch to the filesystem cache. -## Use auto-generated ids [_use_auto_generated_ids] +## Use auto-generated ids [_use_auto_generated_ids] When indexing a document that has an explicit id, Elasticsearch needs to check whether a document with the same id already exists within the same shard, which is a costly operation and gets even more costly as the index grows. By using auto-generated ids, Elasticsearch can skip this check, which makes indexing faster. -## Use faster hardware [indexing-use-faster-hardware] +## Use faster hardware [indexing-use-faster-hardware] If indexing is I/O-bound, consider increasing the size of the filesystem cache (see above) or using faster storage. Elasticsearch generally creates individual files with sequential writes. However, indexing involves writing multiple files concurrently, and a mix of random and sequential reads too, so SSD drives tend to perform better than spinning disks. Stripe your index across multiple SSDs by configuring a RAID 0 array. Remember that it will increase the risk of failure since the failure of any one SSD destroys the index. However this is typically the right tradeoff to make: optimize single shards for maximum performance, and then add replicas across different nodes so there’s redundancy for any node failures. You can also use [snapshot and restore](../../tools/snapshot-and-restore.md) to backup the index for further insurance. -### Local vs. remote storage [_local_vs_remote_storage] +### Local vs. remote storage [_local_vs_remote_storage] Directly-attached (local) storage generally performs better than remote storage because it is simpler to configure well and avoids communications overheads. Some remote storage performs very poorly, especially under the kind of load that {{es}} imposes. However, with careful tuning, it is sometimes possible to achieve acceptable performance using remote storage too. Before committing to a particular storage architecture, benchmark your system with a realistic workload to determine the effects of any tuning parameters. If you cannot achieve the performance you expect, work with the vendor of your storage system to identify the problem. -## Indexing buffer size [_indexing_buffer_size] +## Indexing buffer size [_indexing_buffer_size] -If your node is doing only heavy indexing, be sure [`indices.memory.index_buffer_size`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/indexing-buffer-settings.md) is large enough to give at most 512 MB indexing buffer per shard doing heavy indexing (beyond that indexing performance does not typically improve). Elasticsearch takes that setting (a percentage of the java heap or an absolute byte-size), and uses it as a shared buffer across all active shards. Very active shards will naturally use this buffer more than shards that are performing lightweight indexing. +If your node is doing only heavy indexing, be sure [`indices.memory.index_buffer_size`](elasticsearch://reference/elasticsearch/configuration-reference/indexing-buffer-settings.md) is large enough to give at most 512 MB indexing buffer per shard doing heavy indexing (beyond that indexing performance does not typically improve). Elasticsearch takes that setting (a percentage of the java heap or an absolute byte-size), and uses it as a shared buffer across all active shards. Very active shards will naturally use this buffer more than shards that are performing lightweight indexing. The default is `10%` which is often plenty: for example, if you give the JVM 10GB of memory, it will give 1GB to the index buffer, which is enough to host two shards that are heavily indexing. -## Use {{ccr}} to prevent searching from stealing resources from indexing [_use_ccr_to_prevent_searching_from_stealing_resources_from_indexing] +## Use {{ccr}} to prevent searching from stealing resources from indexing [_use_ccr_to_prevent_searching_from_stealing_resources_from_indexing] Within a single cluster, indexing and searching can compete for resources. By setting up two clusters, configuring [{{ccr}}](../../tools/cross-cluster-replication.md) to replicate data from one cluster to the other one, and routing all searches to the cluster that has the follower indices, search activity will no longer steal resources from indexing on the cluster that hosts the leader indices. -## Avoid hot spotting [_avoid_hot_spotting] +## Avoid hot spotting [_avoid_hot_spotting] [Hot Spotting](../../../troubleshoot/elasticsearch/hotspotting.md) can occur when node resources, shards, or requests are not evenly distributed. {{es}} maintains cluster state by syncing it across nodes, so continually hot spotted nodes can cause overall cluster performance degredation. -## Additional optimizations [_additional_optimizations] +## Additional optimizations [_additional_optimizations] Many of the strategies outlined in [*Tune for disk usage*](disk-usage.md) also provide an improvement in the speed of indexing. diff --git a/deploy-manage/production-guidance/optimize-performance/search-speed.md b/deploy-manage/production-guidance/optimize-performance/search-speed.md index a732bed11..483e56019 100644 --- a/deploy-manage/production-guidance/optimize-performance/search-speed.md +++ b/deploy-manage/production-guidance/optimize-performance/search-speed.md @@ -6,49 +6,49 @@ mapped_pages: # Search speed [tune-for-search-speed] -## Give memory to the filesystem cache [_give_memory_to_the_filesystem_cache_2] +## Give memory to the filesystem cache [_give_memory_to_the_filesystem_cache_2] Elasticsearch heavily relies on the filesystem cache in order to make search fast. In general, you should make sure that at least half the available memory goes to the filesystem cache so that Elasticsearch can keep hot regions of the index in physical memory. -## Avoid page cache thrashing by using modest readahead values on Linux [_avoid_page_cache_thrashing_by_using_modest_readahead_values_on_linux] +## Avoid page cache thrashing by using modest readahead values on Linux [_avoid_page_cache_thrashing_by_using_modest_readahead_values_on_linux] -Search can cause a lot of randomized read I/O. When the underlying block device has a high readahead value, there may be a lot of unnecessary read I/O done, especially when files are accessed using memory mapping (see [storage types](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/store.md#file-system)). +Search can cause a lot of randomized read I/O. When the underlying block device has a high readahead value, there may be a lot of unnecessary read I/O done, especially when files are accessed using memory mapping (see [storage types](elasticsearch://reference/elasticsearch/index-settings/store.md#file-system)). Most Linux distributions use a sensible readahead value of `128KiB` for a single plain device, however, when using software raid, LVM or dm-crypt the resulting block device (backing Elasticsearch [path.data](../../deploy/self-managed/important-settings-configuration.md#path-settings)) may end up having a very large readahead value (in the range of several MiB). This usually results in severe page (filesystem) cache thrashing adversely affecting search (or [update](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-document)) performance. You can check the current value in `KiB` using `lsblk -o NAME,RA,MOUNTPOINT,TYPE,SIZE`. Consult the documentation of your distribution on how to alter this value (for example with a `udev` rule to persist across reboots, or via [blockdev --setra](https://man7.org/linux/man-pages/man8/blockdev.8.html) as a transient setting). We recommend a value of `128KiB` for readahead. -::::{warning} +::::{warning} `blockdev` expects values in 512 byte sectors whereas `lsblk` reports values in `KiB`. As an example, to temporarily set readahead to `128KiB` for `/dev/nvme0n1`, specify `blockdev --setra 256 /dev/nvme0n1`. :::: -## Use faster hardware [search-use-faster-hardware] +## Use faster hardware [search-use-faster-hardware] If your searches are I/O-bound, consider increasing the size of the filesystem cache (see above) or using faster storage. Each search involves a mix of sequential and random reads across multiple files, and there may be many searches running concurrently on each shard, so SSD drives tend to perform better than spinning disks. If your searches are CPU-bound, consider using a larger number of faster CPUs. -### Local vs. remote storage [_local_vs_remote_storage_2] +### Local vs. remote storage [_local_vs_remote_storage_2] Directly-attached (local) storage generally performs better than remote storage because it is simpler to configure well and avoids communications overheads. Some remote storage performs very poorly, especially under the kind of load that {{es}} imposes. However, with careful tuning, it is sometimes possible to achieve acceptable performance using remote storage too. Before committing to a particular storage architecture, benchmark your system with a realistic workload to determine the effects of any tuning parameters. If you cannot achieve the performance you expect, work with the vendor of your storage system to identify the problem. -## Document modeling [_document_modeling] +## Document modeling [_document_modeling] Documents should be modeled so that search-time operations are as cheap as possible. -In particular, joins should be avoided. [`nested`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/nested.md) can make queries several times slower and [parent-child](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/parent-join.md) relations can make queries hundreds of times slower. So if the same questions can be answered without joins by denormalizing documents, significant speedups can be expected. +In particular, joins should be avoided. [`nested`](elasticsearch://reference/elasticsearch/mapping-reference/nested.md) can make queries several times slower and [parent-child](elasticsearch://reference/elasticsearch/mapping-reference/parent-join.md) relations can make queries hundreds of times slower. So if the same questions can be answered without joins by denormalizing documents, significant speedups can be expected. -## Search as few fields as possible [search-as-few-fields-as-possible] +## Search as few fields as possible [search-as-few-fields-as-possible] -The more fields a [`query_string`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-query-string-query.md) or [`multi_match`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-multi-match-query.md) query targets, the slower it is. A common technique to improve search speed over multiple fields is to copy their values into a single field at index time, and then use this field at search time. This can be automated with the [`copy-to`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/copy-to.md) directive of mappings without having to change the source of documents. Here is an example of an index containing movies that optimizes queries that search over both the name and the plot of the movie by indexing both values into the `name_and_plot` field. +The more fields a [`query_string`](elasticsearch://reference/query-languages/query-dsl-query-string-query.md) or [`multi_match`](elasticsearch://reference/query-languages/query-dsl-multi-match-query.md) query targets, the slower it is. A common technique to improve search speed over multiple fields is to copy their values into a single field at index time, and then use this field at search time. This can be automated with the [`copy-to`](elasticsearch://reference/elasticsearch/mapping-reference/copy-to.md) directive of mappings without having to change the source of documents. Here is an example of an index containing movies that optimizes queries that search over both the name and the plot of the movie by indexing both values into the `name_and_plot` field. ```console PUT movies @@ -72,9 +72,9 @@ PUT movies ``` -## Pre-index data [_pre_index_data] +## Pre-index data [_pre_index_data] -You should leverage patterns in your queries to optimize the way data is indexed. For instance, if all your documents have a `price` field and most queries run [`range`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-range-aggregation.md) aggregations on a fixed list of ranges, you could make this aggregation faster by pre-indexing the ranges into the index and using a [`terms`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-terms-aggregation.md) aggregations. +You should leverage patterns in your queries to optimize the way data is indexed. For instance, if all your documents have a `price` field and most queries run [`range`](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-range-aggregation.md) aggregations on a fixed list of ranges, you could make this aggregation faster by pre-indexing the ranges into the index and using a [`terms`](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-terms-aggregation.md) aggregations. For instance, if documents look like: @@ -106,7 +106,7 @@ GET index/_search } ``` -Then documents could be enriched by a `price_range` field at index time, which should be mapped as a [`keyword`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/keyword.md): +Then documents could be enriched by a `price_range` field at index time, which should be mapped as a [`keyword`](elasticsearch://reference/elasticsearch/mapping-reference/keyword.md): ```console PUT index @@ -144,26 +144,26 @@ GET index/_search ``` -## Consider mapping identifiers as `keyword` [map-ids-as-keyword] +## Consider mapping identifiers as `keyword` [map-ids-as-keyword] -Not all numeric data should be mapped as a [numeric](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/number.md) field data type. {{es}} optimizes numeric fields, such as `integer` or `long`, for [`range`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-range-query.md) queries. However, [`keyword`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/keyword.md) fields are better for [`term`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-term-query.md) and other [term-level](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/term-level-queries.md) queries. +Not all numeric data should be mapped as a [numeric](elasticsearch://reference/elasticsearch/mapping-reference/number.md) field data type. {{es}} optimizes numeric fields, such as `integer` or `long`, for [`range`](elasticsearch://reference/query-languages/query-dsl-range-query.md) queries. However, [`keyword`](elasticsearch://reference/elasticsearch/mapping-reference/keyword.md) fields are better for [`term`](elasticsearch://reference/query-languages/query-dsl-term-query.md) and other [term-level](elasticsearch://reference/query-languages/term-level-queries.md) queries. Identifiers, such as an ISBN or a product ID, are rarely used in `range` queries. However, they are often retrieved using term-level queries. Consider mapping a numeric identifier as a `keyword` if: -* You don’t plan to search for the identifier data using [`range`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-range-query.md) queries. +* You don’t plan to search for the identifier data using [`range`](elasticsearch://reference/query-languages/query-dsl-range-query.md) queries. * Fast retrieval is important. `term` query searches on `keyword` fields are often faster than `term` searches on numeric fields. -If you’re unsure which to use, you can use a [multi-field](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/multi-fields.md) to map the data as both a `keyword` *and* a numeric data type. +If you’re unsure which to use, you can use a [multi-field](elasticsearch://reference/elasticsearch/mapping-reference/multi-fields.md) to map the data as both a `keyword` *and* a numeric data type. -## Avoid scripts [_avoid_scripts] +## Avoid scripts [_avoid_scripts] -If possible, avoid using [script](../../../explore-analyze/scripting.md)-based sorting, scripts in aggregations, and the [`script_score`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-script-score-query.md) query. See [Scripts, caching, and search speed](../../../explore-analyze/scripting/scripts-search-speed.md). +If possible, avoid using [script](../../../explore-analyze/scripting.md)-based sorting, scripts in aggregations, and the [`script_score`](elasticsearch://reference/query-languages/query-dsl-script-score-query.md) query. See [Scripts, caching, and search speed](../../../explore-analyze/scripting/scripts-search-speed.md). -## Search rounded dates [_search_rounded_dates] +## Search rounded dates [_search_rounded_dates] Queries on date fields that use `now` are typically not cacheable since the range that is being matched changes all the time. However switching to a rounded date is often acceptable in terms of user experience, and has the benefit of making better use of the query cache. @@ -214,7 +214,7 @@ GET index/_search In that case we rounded to the minute, so if the current time is `16:31:29`, the range query will match everything whose value of the `my_date` field is between `15:31:00` and `16:31:59`. And if several users run a query that contains this range in the same minute, the query cache could help speed things up a bit. The longer the interval that is used for rounding, the more the query cache can help, but beware that too aggressive rounding might also hurt user experience. -::::{note} +::::{note} It might be tempting to split ranges into a large cacheable part and smaller not cacheable parts in order to be able to leverage the query cache, as shown below: :::: @@ -262,19 +262,19 @@ GET index/_search However such practice might make the query run slower in some cases since the overhead introduced by the `bool` query may defeat the savings from better leveraging the query cache. -## Force-merge read-only indices [_force_merge_read_only_indices] +## Force-merge read-only indices [_force_merge_read_only_indices] Indices that are read-only may benefit from being [merged down to a single segment](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-forcemerge). This is typically the case with time-based indices: only the index for the current time frame is getting new documents while older indices are read-only. Shards that have been force-merged into a single segment can use simpler and more efficient data structures to perform searches. -::::{important} +::::{important} Do not force-merge indices to which you are still writing, or to which you will write again in the future. Instead, rely on the automatic background merge process to perform merges as needed to keep the index running smoothly. If you continue to write to a force-merged index then its performance may become much worse. :::: -## Warm up global ordinals [_warm_up_global_ordinals] +## Warm up global ordinals [_warm_up_global_ordinals] -[Global ordinals](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/eager-global-ordinals.md) are a data structure that is used to optimize the performance of aggregations. They are calculated lazily and stored in the JVM heap as part of the [field data cache](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/field-data-cache-settings.md). For fields that are heavily used for bucketing aggregations, you can tell {{es}} to construct and cache the global ordinals before requests are received. This should be done carefully because it will increase heap usage and can make [refreshes](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-refresh) take longer. The option can be updated dynamically on an existing mapping by setting the [eager global ordinals](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/eager-global-ordinals.md) mapping parameter: +[Global ordinals](elasticsearch://reference/elasticsearch/mapping-reference/eager-global-ordinals.md) are a data structure that is used to optimize the performance of aggregations. They are calculated lazily and stored in the JVM heap as part of the [field data cache](elasticsearch://reference/elasticsearch/configuration-reference/field-data-cache-settings.md). For fields that are heavily used for bucketing aggregations, you can tell {{es}} to construct and cache the global ordinals before requests are received. This should be done carefully because it will increase heap usage and can make [refreshes](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-refresh) take longer. The option can be updated dynamically on an existing mapping by setting the [eager global ordinals](elasticsearch://reference/elasticsearch/mapping-reference/eager-global-ordinals.md) mapping parameter: ```console PUT index @@ -291,29 +291,29 @@ PUT index ``` -## Warm up the filesystem cache [_warm_up_the_filesystem_cache] +## Warm up the filesystem cache [_warm_up_the_filesystem_cache] -If the machine running Elasticsearch is restarted, the filesystem cache will be empty, so it will take some time before the operating system loads hot regions of the index into memory so that search operations are fast. You can explicitly tell the operating system which files should be loaded into memory eagerly depending on the file extension using the [`index.store.preload`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/preloading-data-into-file-system-cache.md) setting. +If the machine running Elasticsearch is restarted, the filesystem cache will be empty, so it will take some time before the operating system loads hot regions of the index into memory so that search operations are fast. You can explicitly tell the operating system which files should be loaded into memory eagerly depending on the file extension using the [`index.store.preload`](elasticsearch://reference/elasticsearch/index-settings/preloading-data-into-file-system-cache.md) setting. -::::{warning} +::::{warning} Loading data into the filesystem cache eagerly on too many indices or too many files will make search *slower* if the filesystem cache is not large enough to hold all the data. Use with caution. :::: -## Use index sorting to speed up conjunctions [_use_index_sorting_to_speed_up_conjunctions] +## Use index sorting to speed up conjunctions [_use_index_sorting_to_speed_up_conjunctions] -[Index sorting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/sorting.md) can be useful in order to make conjunctions faster at the cost of slightly slower indexing. Read more about it in the [index sorting documentation](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/sorting-conjunctions.md). +[Index sorting](elasticsearch://reference/elasticsearch/index-settings/sorting.md) can be useful in order to make conjunctions faster at the cost of slightly slower indexing. Read more about it in the [index sorting documentation](elasticsearch://reference/elasticsearch/index-settings/sorting-conjunctions.md). -## Use `preference` to optimize cache utilization [preference-cache-optimization] +## Use `preference` to optimize cache utilization [preference-cache-optimization] -There are multiple caches that can help with search performance, such as the [filesystem cache](https://en.wikipedia.org/wiki/Page_cache), the [request cache](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/shard-request-cache-settings.md) or the [query cache](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-query-cache-settings.md). Yet all these caches are maintained at the node level, meaning that if you run the same request twice in a row, have 1 replica or more and use [round-robin](https://en.wikipedia.org/wiki/Round-robin_DNS), the default routing algorithm, then those two requests will go to different shard copies, preventing node-level caches from helping. +There are multiple caches that can help with search performance, such as the [filesystem cache](https://en.wikipedia.org/wiki/Page_cache), the [request cache](elasticsearch://reference/elasticsearch/configuration-reference/shard-request-cache-settings.md) or the [query cache](elasticsearch://reference/elasticsearch/configuration-reference/node-query-cache-settings.md). Yet all these caches are maintained at the node level, meaning that if you run the same request twice in a row, have 1 replica or more and use [round-robin](https://en.wikipedia.org/wiki/Round-robin_DNS), the default routing algorithm, then those two requests will go to different shard copies, preventing node-level caches from helping. Since it is common for users of a search application to run similar requests one after another, for instance in order to analyze a narrower subset of the index, using a preference value that identifies the current user or session could help optimize usage of the caches. -## Replicas might help with throughput, but not always [_replicas_might_help_with_throughput_but_not_always] +## Replicas might help with throughput, but not always [_replicas_might_help_with_throughput_but_not_always] In addition to improving resiliency, replicas can help improve throughput. For instance if you have a single-shard index and three nodes, you will need to set the number of replicas to 2 in order to have 3 copies of your shard in total so that all nodes are utilized. @@ -322,7 +322,7 @@ Now imagine that you have a 2-shards index and two nodes. In one case, the numbe So what is the right number of replicas? If you have a cluster that has `num_nodes` nodes, `num_primaries` primary shards *in total* and if you want to be able to cope with `max_failures` node failures at once at most, then the right number of replicas for you is `max(max_failures, ceil(num_nodes / num_primaries) - 1)`. -## Tune your queries with the Search Profiler [_tune_your_queries_with_the_search_profiler] +## Tune your queries with the Search Profiler [_tune_your_queries_with_the_search_profiler] The [Profile API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-profile.html) provides detailed information about how each component of your queries and aggregations impacts the time it takes to process the request. @@ -331,17 +331,17 @@ The [Search Profiler](../../../explore-analyze/query-filter/tools/search-profile Because the Profile API itself adds significant overhead to the query, this information is best used to understand the relative cost of the various query components. It does not provide a reliable measure of actual processing time. -## Faster phrase queries with `index_phrases` [faster-phrase-queries] +## Faster phrase queries with `index_phrases` [faster-phrase-queries] -The [`text`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/text.md) field has an [`index_phrases`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/index-phrases.md) option that indexes 2-shingles and is automatically leveraged by query parsers to run phrase queries that don’t have a slop. If your use-case involves running lots of phrase queries, this can speed up queries significantly. +The [`text`](elasticsearch://reference/elasticsearch/mapping-reference/text.md) field has an [`index_phrases`](elasticsearch://reference/elasticsearch/mapping-reference/index-phrases.md) option that indexes 2-shingles and is automatically leveraged by query parsers to run phrase queries that don’t have a slop. If your use-case involves running lots of phrase queries, this can speed up queries significantly. -## Faster prefix queries with `index_prefixes` [faster-prefix-queries] +## Faster prefix queries with `index_prefixes` [faster-prefix-queries] -The [`text`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/text.md) field has an [`index_prefixes`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/index-prefixes.md) option that indexes prefixes of all terms and is automatically leveraged by query parsers to run prefix queries. If your use-case involves running lots of prefix queries, this can speed up queries significantly. +The [`text`](elasticsearch://reference/elasticsearch/mapping-reference/text.md) field has an [`index_prefixes`](elasticsearch://reference/elasticsearch/mapping-reference/index-prefixes.md) option that indexes prefixes of all terms and is automatically leveraged by query parsers to run prefix queries. If your use-case involves running lots of prefix queries, this can speed up queries significantly. -## Use `constant_keyword` to speed up filtering [faster-filtering-with-constant-keyword] +## Use `constant_keyword` to speed up filtering [faster-filtering-with-constant-keyword] There is a general rule that the cost of a filter is mostly a function of the number of matched documents. Imagine that you have an index containing cycles. There are a large number of bicycles and many searches perform a filter on `cycle_type: bicycle`. This very common filter is unfortunately also very costly since it matches most documents. There is a simple way to avoid running this filter: move bicycles to their own index and filter bicycles by searching this index instead of adding a filter to the query. @@ -422,7 +422,7 @@ This is a powerful way of making queries cheaper by putting common values in a d The `constant_keyword` is not strictly required for this optimization: it is also possible to update the client-side logic in order to route queries to the relevant indices based on filters. However `constant_keyword` makes it transparently and allows to decouple search requests from the index topology in exchange of very little overhead. -## Default search timeout [_default_search_timeout] +## Default search timeout [_default_search_timeout] By default, search requests don’t time out. You can set a timeout using the [`search.default_search_timeout`](../../../solutions/search/the-search-api.md#search-timeout) setting. diff --git a/deploy-manage/production-guidance/optimize-performance/size-shards.md b/deploy-manage/production-guidance/optimize-performance/size-shards.md index 5d559037b..aa7b654ec 100644 --- a/deploy-manage/production-guidance/optimize-performance/size-shards.md +++ b/deploy-manage/production-guidance/optimize-performance/size-shards.md @@ -28,14 +28,14 @@ Keep the following things in mind when building your sharding strategy. ### Searches run on a single thread per shard [single-thread-per-shard] -Most searches hit multiple shards. Each shard runs the search on a single CPU thread. While a shard can run multiple concurrent searches, searches across a large number of shards can deplete a node’s [search thread pool](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/thread-pool-settings.md). This can result in low throughput and slow search speeds. +Most searches hit multiple shards. Each shard runs the search on a single CPU thread. While a shard can run multiple concurrent searches, searches across a large number of shards can deplete a node’s [search thread pool](elasticsearch://reference/elasticsearch/configuration-reference/thread-pool-settings.md). This can result in low throughput and slow search speeds. ### Each index, shard, segment and field has overhead [each-shard-has-overhead] Every index and every shard requires some memory and CPU resources. In most cases, a small set of large shards uses fewer resources than many small shards. -Segments play a big role in a shard’s resource usage. Most shards contain several segments, which store its index data. {{es}} keeps some segment metadata in heap memory so it can be quickly retrieved for searches. As a shard grows, its segments are [merged](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/merge.md) into fewer, larger segments. This decreases the number of segments, which means less metadata is kept in heap memory. +Segments play a big role in a shard’s resource usage. Most shards contain several segments, which store its index data. {{es}} keeps some segment metadata in heap memory so it can be quickly retrieved for searches. As a shard grows, its segments are [merged](elasticsearch://reference/elasticsearch/index-settings/merge.md) into fewer, larger segments. This decreases the number of segments, which means less metadata is kept in heap memory. Every mapped field also carries some overhead in terms of memory usage and disk space. By default {{es}} will automatically create a mapping for every field in every document it indexes, but you can switch off this behaviour to [take control of your mappings](../../../manage-data/data-store/mapping/explicit-mapping.md). @@ -54,7 +54,7 @@ Where applicable, use the following best practices as starting points for your s ### Delete indices, not documents [delete-indices-not-documents] -Deleted documents aren’t immediately removed from {{es}}'s file system. Instead, {{es}} marks the document as deleted on each related shard. The marked document will continue to use resources until it’s removed during a periodic [segment merge](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/merge.md). +Deleted documents aren’t immediately removed from {{es}}'s file system. Instead, {{es}} marks the document as deleted on each related shard. The marked document will continue to use resources until it’s removed during a periodic [segment merge](elasticsearch://reference/elasticsearch/index-settings/merge.md). When possible, delete entire indices instead. {{es}} can immediately remove deleted indices directly from the file system and free up resources. @@ -67,8 +67,8 @@ One advantage of this setup is [automatic rollover](../../../manage-data/lifecyc {{ilm-init}} also makes it easy to change your sharding strategy over time: -* **Want to decrease the shard count for new indices?**
Change the [`index.number_of_shards`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-number-of-shards) setting in the data stream’s [matching index template](../../../manage-data/data-store/data-streams/modify-data-stream.md#data-streams-change-mappings-and-settings). -* **Want larger shards or fewer backing indices?**
Increase your {{ilm-init}} policy’s [rollover threshold](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-rollover.md). +* **Want to decrease the shard count for new indices?**
Change the [`index.number_of_shards`](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#index-number-of-shards) setting in the data stream’s [matching index template](../../../manage-data/data-store/data-streams/modify-data-stream.md#data-streams-change-mappings-and-settings). +* **Want larger shards or fewer backing indices?**
Increase your {{ilm-init}} policy’s [rollover threshold](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-rollover.md). * **Need indices that span shorter intervals?**
Offset the increased shard count by deleting older indices sooner. You can do this by lowering the `min_age` threshold for your policy’s [delete phase](../../../manage-data/lifecycle/index-lifecycle-management/index-lifecycle.md). Every new backing index is an opportunity to further tune your strategy. @@ -82,7 +82,7 @@ There is no hard limit on the physical size of a shard, and each shard can in th You may be able to use larger shards depending on your network and use case, and smaller shards may be appropriate for certain use cases. -If you use {{ilm-init}}, set the [rollover action](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-rollover.md)'s `max_primary_shard_size` threshold to `50gb` to avoid shards larger than 50GB and `min_primary_shard_size` threshold to `10gb` to avoid shards smaller than 10GB. +If you use {{ilm-init}}, set the [rollover action](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-rollover.md)'s `max_primary_shard_size` threshold to `50gb` to avoid shards larger than 50GB and `min_primary_shard_size` threshold to `10gb` to avoid shards smaller than 10GB. To see the current size of your shards, use the [cat shards API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cat-shards). @@ -133,7 +133,7 @@ GET _cat/shards?v=true ### Add enough nodes to stay within the cluster shard limits [shard-count-per-node-recommendation] -[Cluster shard limits](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/miscellaneous-cluster-settings.md#cluster-shard-limit) prevent creation of more than 1000 non-frozen shards per node, and 3000 frozen shards per dedicated frozen node. Make sure you have enough nodes of each type in your cluster to handle the number of shards you need. +[Cluster shard limits](elasticsearch://reference/elasticsearch/configuration-reference/miscellaneous-cluster-settings.md#cluster-shard-limit) prevent creation of more than 1000 non-frozen shards per node, and 3000 frozen shards per dedicated frozen node. Make sure you have enough nodes of each type in your cluster to handle the number of shards you need. ### Allow enough heap for field mappers and overheads [field-count-recommendation] @@ -223,7 +223,7 @@ Note that the above rules do not necessarily guarantee the performance of search If too many shards are allocated to a specific node, the node can become a hotspot. For example, if a single node contains too many shards for an index with a high indexing volume, the node is likely to have issues. -To prevent hotspots, use the [`index.routing.allocation.total_shards_per_node`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/total-shards-per-node.md#total-shards-per-node) index setting to explicitly limit the number of shards on a single node. You can configure `index.routing.allocation.total_shards_per_node` using the [update index settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings). +To prevent hotspots, use the [`index.routing.allocation.total_shards_per_node`](elasticsearch://reference/elasticsearch/index-settings/total-shards-per-node.md#total-shards-per-node) index setting to explicitly limit the number of shards on a single node. You can configure `index.routing.allocation.total_shards_per_node` using the [update index settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings). ```console PUT my-index-000001/_settings @@ -237,7 +237,7 @@ PUT my-index-000001/_settings ### Avoid unnecessary mapped fields [avoid-unnecessary-fields] -By default {{es}} [automatically creates a mapping](../../../manage-data/data-store/mapping/dynamic-mapping.md) for every field in every document it indexes. Every mapped field corresponds to some data structures on disk which are needed for efficient search, retrieval, and aggregations on this field. Details about each mapped field are also held in memory. In many cases this overhead is unnecessary because a field is not used in any searches or aggregations. Use [*Explicit mapping*](../../../manage-data/data-store/mapping/explicit-mapping.md) instead of dynamic mapping to avoid creating fields that are never used. If a collection of fields are typically used together, consider using [`copy_to`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/copy-to.md) to consolidate them at index time. If a field is only rarely used, it may be better to make it a [Runtime field](../../../manage-data/data-store/mapping/runtime-fields.md) instead. +By default {{es}} [automatically creates a mapping](../../../manage-data/data-store/mapping/dynamic-mapping.md) for every field in every document it indexes. Every mapped field corresponds to some data structures on disk which are needed for efficient search, retrieval, and aggregations on this field. Details about each mapped field are also held in memory. In many cases this overhead is unnecessary because a field is not used in any searches or aggregations. Use [*Explicit mapping*](../../../manage-data/data-store/mapping/explicit-mapping.md) instead of dynamic mapping to avoid creating fields that are never used. If a collection of fields are typically used together, consider using [`copy_to`](elasticsearch://reference/elasticsearch/mapping-reference/copy-to.md) to consolidate them at index time. If a field is only rarely used, it may be better to make it a [Runtime field](../../../manage-data/data-store/mapping/runtime-fields.md) instead. You can get information about which fields are being used with the [Field usage stats](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-field-usage-stats) API, and you can analyze the disk usage of mapped fields using the [Analyze index disk usage](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-disk-usage) API. Note however that unnecessary mapped fields also carry some memory overhead as well as their disk usage. @@ -273,7 +273,7 @@ DELETE my-index-000001 ### Force merge during off-peak hours [force-merge-during-off-peak-hours] -If you no longer write to an index, you can use the [force merge API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-forcemerge) to [merge](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/merge.md) smaller segments into larger ones. This can reduce shard overhead and improve search speeds. However, force merges are resource-intensive. If possible, run the force merge during off-peak hours. +If you no longer write to an index, you can use the [force merge API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-forcemerge) to [merge](elasticsearch://reference/elasticsearch/index-settings/merge.md) smaller segments into larger ones. This can reduce shard overhead and improve search speeds. However, force merges are resource-intensive. If possible, run the force merge during off-peak hours. ```console POST my-index-000001/_forcemerge @@ -284,7 +284,7 @@ POST my-index-000001/_forcemerge If you no longer write to an index, you can use the [shrink index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-shrink) to reduce its shard count. -{{ilm-init}} also has a [shrink action](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-shrink.md) for indices in the warm phase. +{{ilm-init}} also has a [shrink action](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-shrink.md) for indices in the warm phase. ### Combine smaller indices [combine-smaller-indices] @@ -311,7 +311,7 @@ Here’s how to resolve common shard-related errors. ### this action would add [x] total shards, but this cluster currently has [y]/[z] maximum shards open; [troubleshooting-max-shards-open] -The [`cluster.max_shards_per_node`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/miscellaneous-cluster-settings.md#cluster-max-shards-per-node) cluster setting limits the maximum number of open shards for a cluster. This error indicates an action would exceed this limit. +The [`cluster.max_shards_per_node`](elasticsearch://reference/elasticsearch/configuration-reference/miscellaneous-cluster-settings.md#cluster-max-shards-per-node) cluster setting limits the maximum number of open shards for a cluster. This error indicates an action would exceed this limit. If you’re confident your changes won’t destabilize the cluster, you can temporarily increase the limit using the [cluster update settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings) and retry the action. diff --git a/deploy-manage/remote-clusters/ec-remote-cluster-self-managed.md b/deploy-manage/remote-clusters/ec-remote-cluster-self-managed.md index 3d9557128..c4d6cba4a 100644 --- a/deploy-manage/remote-clusters/ec-remote-cluster-self-managed.md +++ b/deploy-manage/remote-clusters/ec-remote-cluster-self-managed.md @@ -137,7 +137,7 @@ A deployment can be configured to trust all or specific deployments in any envir 7. Configure the self-managed cluster to trust this deployment, so that both deployments are configured to trust each other: * Download the Certificate Authority used to sign the certificates of your deployment nodes (it can be found in the Security page of your deployment) - * Trust this CA either using the [setting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md) `xpack.security.transport.ssl.certificate_authorities` in `elasticsearch.yml` or by [adding it to the trust store](../security/different-ca.md). + * Trust this CA either using the [setting](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md) `xpack.security.transport.ssl.certificate_authorities` in `elasticsearch.yml` or by [adding it to the trust store](../security/different-ca.md). 8. Generate certificates with an `otherName` attribute using the {{es}} certutil. Create a file called `instances.yaml` with all the details of the nodes in your on-premise cluster like below. The `dns` and `ip` settings are optional, but `cn` is mandatory for use with the `trust_restrictions` path setting in the next step. Next, run `./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 -in instances.yaml` to create new certificates for all the nodes at once. You can then copy the resulting files into each node. diff --git a/deploy-manage/remote-clusters/ece-remote-cluster-self-managed.md b/deploy-manage/remote-clusters/ece-remote-cluster-self-managed.md index 820a7f18a..0a5299b3e 100644 --- a/deploy-manage/remote-clusters/ece-remote-cluster-self-managed.md +++ b/deploy-manage/remote-clusters/ece-remote-cluster-self-managed.md @@ -136,7 +136,7 @@ A deployment can be configured to trust all or specific deployments in any envir 7. Configure the self-managed cluster to trust this deployment, so that both deployments are configured to trust each other: * Download the Certificate Authority used to sign the certificates of your deployment nodes (it can be found in the Security page of your deployment) - * Trust this CA either using the [setting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md) `xpack.security.transport.ssl.certificate_authorities` in `elasticsearch.yml` or by [adding it to the trust store](../security/different-ca.md). + * Trust this CA either using the [setting](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md) `xpack.security.transport.ssl.certificate_authorities` in `elasticsearch.yml` or by [adding it to the trust store](../security/different-ca.md). 8. Generate certificates with an `otherName` attribute using the {{es}} certutil. Create a file called `instances.yaml` with all the details of the nodes in your on-premise cluster like below. The `dns` and `ip` settings are optional, but `cn` is mandatory for use with the `trust_restrictions` path setting in the next step. Next, run `./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 -in instances.yaml` to create new certificates for all the nodes at once. You can then copy the resulting files into each node. diff --git a/deploy-manage/remote-clusters/eck-remote-clusters.md b/deploy-manage/remote-clusters/eck-remote-clusters.md index 10c4e0f98..8a002da7c 100644 --- a/deploy-manage/remote-clusters/eck-remote-clusters.md +++ b/deploy-manage/remote-clusters/eck-remote-clusters.md @@ -148,7 +148,7 @@ kubectl get secret cluster-one-es-transport-certs-public \ -o go-template='{{index .data "ca.crt" | base64decode}}' > remote.ca.crt ``` -You then need to configure the CA as one of the trusted CAs in `cluster-two`. If that cluster is hosted outside of Kubernetes, take the CA certificate that you have just extracted and add it to the list of CAs in [`xpack.security.transport.ssl.certificate_authorities`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#_pem_encoded_files_3). +You then need to configure the CA as one of the trusted CAs in `cluster-two`. If that cluster is hosted outside of Kubernetes, take the CA certificate that you have just extracted and add it to the list of CAs in [`xpack.security.transport.ssl.certificate_authorities`](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#_pem_encoded_files_3). ::::{note} Beware of copying the source Secret as-is into a different namespace. Check [Common Problems: Owner References](../../troubleshoot/deployments/cloud-on-k8s/common-problems.md#k8s-common-problems-owner-refs) for more information. diff --git a/deploy-manage/remote-clusters/remote-clusters-api-key.md b/deploy-manage/remote-clusters/remote-clusters-api-key.md index 699798b3b..73174d25e 100644 --- a/deploy-manage/remote-clusters/remote-clusters-api-key.md +++ b/deploy-manage/remote-clusters/remote-clusters-api-key.md @@ -14,7 +14,7 @@ All cross-cluster requests from the local cluster are bound by the API key’s p On the local cluster side, not every local user needs to access every piece of data allowed by the API key. An administrator of the local cluster can further configure additional permission constraints on local users so each user only gets access to the necessary remote data. Note it is only possible to further reduce the permissions allowed by the API key for individual local users. It is impossible to increase the permissions to go beyond what is allowed by the API key. -In this model, cross-cluster operations use [a dedicated server port](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md#remote_cluster.port) (remote cluster interface) for communication between clusters. A remote cluster must enable this port for local clusters to connect. Configure Transport Layer Security (TLS) for this port to maximize security (as explained in [Establish trust with a remote cluster](#remote-clusters-security-api-key)). +In this model, cross-cluster operations use [a dedicated server port](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#remote_cluster.port) (remote cluster interface) for communication between clusters. A remote cluster must enable this port for local clusters to connect. Configure Transport Layer Security (TLS) for this port to maximize security (as explained in [Establish trust with a remote cluster](#remote-clusters-security-api-key)). The local cluster must trust the remote cluster on the remote cluster interface. This means that the local cluster trusts the remote cluster’s certificate authority (CA) that signs the server certificate used by the remote cluster interface. When establishing a connection, all nodes from the local cluster that participate in cross-cluster communication verify certificates from nodes on the other side, based on the TLS trust configuration. @@ -29,7 +29,7 @@ If you run into any issues, refer to [Troubleshooting](/troubleshoot/elasticsear ## Prerequisites [remote-clusters-prerequisites-api-key] -* The {{es}} security features need to be enabled on both clusters, on every node. Security is enabled by default. If it’s disabled, set `xpack.security.enabled` to `true` in `elasticsearch.yml`. Refer to [General security settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#general-security-settings). +* The {{es}} security features need to be enabled on both clusters, on every node. Security is enabled by default. If it’s disabled, set `xpack.security.enabled` to `true` in `elasticsearch.yml`. Refer to [General security settings](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#general-security-settings). * The nodes of the local and remote clusters must be on {{stack}} 8.14 or later. * The local and remote clusters must have an appropriate license. For more information, refer to [https://www.elastic.co/subscriptions](https://www.elastic.co/subscriptions). @@ -45,9 +45,9 @@ If a remote cluster is part of an {{ech}} deployment, it has a valid certificate 1. Enable the remote cluster server on every node of the remote cluster. In `elasticsearch.yml`: - 1. Set [`remote_cluster_server.enabled`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md#remote-cluster-network-settings) to `true`. - 2. Configure the bind and publish address for remote cluster server traffic, for example using [`remote_cluster.host`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md#remote-cluster-network-settings). Without configuring the address, remote cluster traffic may be bound to the local interface, and remote clusters running on other machines can’t connect. - 3. Optionally, configure the remote server port using [`remote_cluster.port`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md#remote_cluster.port) (defaults to `9443`). + 1. Set [`remote_cluster_server.enabled`](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#remote-cluster-network-settings) to `true`. + 2. Configure the bind and publish address for remote cluster server traffic, for example using [`remote_cluster.host`](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#remote-cluster-network-settings). Without configuring the address, remote cluster traffic may be bound to the local interface, and remote clusters running on other machines can’t connect. + 3. Optionally, configure the remote server port using [`remote_cluster.port`](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#remote_cluster.port) (defaults to `9443`). 2. Next, generate a certificate authority (CA) and a server certificate/key pair. On one of the nodes of the remote cluster, from the directory where {{es}} has been installed: @@ -81,7 +81,7 @@ If a remote cluster is part of an {{ech}} deployment, it has a valid certificate 4. If the remote cluster has multiple nodes, you can either: * create a single wildcard certificate for all nodes; - * or, create separate certificates for each node either manually or in batch with the [silent mode](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/certutil.md#certutil-silent). + * or, create separate certificates for each node either manually or in batch with the [silent mode](elasticsearch://reference/elasticsearch/command-line-tools/certutil.md#certutil-silent). 3. On every node of the remote cluster: @@ -139,7 +139,7 @@ You must have the `manage` cluster privilege to connect remote clusters. :::: -The local cluster uses the [remote cluster interface](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md) to establish communication with remote clusters. The coordinating nodes in the local cluster establish [long-lived](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md#long-lived-connections) TCP connections with specific nodes in the remote cluster. {{es}} requires these connections to remain open, even if the connections are idle for an extended period. +The local cluster uses the [remote cluster interface](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md) to establish communication with remote clusters. The coordinating nodes in the local cluster establish [long-lived](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#long-lived-connections) TCP connections with specific nodes in the remote cluster. {{es}} requires these connections to remain open, even if the connections are idle for an extended period. To add a remote cluster from Stack Management in {{kib}}: @@ -206,7 +206,7 @@ The API response indicates that the local cluster is connected to the remote clu Use the [cluster update settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings) to dynamically configure remote settings on every node in the cluster. The following request adds three remote clusters: `cluster_one`, `cluster_two`, and `cluster_three`. -The `seeds` parameter specifies the hostname and [remote cluster port](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md) (default `9443`) of a seed node in the remote cluster. +The `seeds` parameter specifies the hostname and [remote cluster port](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md) (default `9443`) of a seed node in the remote cluster. The `mode` parameter determines the configured connection mode, which defaults to [`sniff`](/deploy-manage/remote-clusters/remote-clusters-self-managed.md#sniff-mode). Because `cluster_one` doesn’t specify a `mode`, it uses the default. Both `cluster_two` and `cluster_three` explicitly use different modes. diff --git a/deploy-manage/remote-clusters/remote-clusters-cert.md b/deploy-manage/remote-clusters/remote-clusters-cert.md index 54514c3cf..ba751510a 100644 --- a/deploy-manage/remote-clusters/remote-clusters-cert.md +++ b/deploy-manage/remote-clusters/remote-clusters-cert.md @@ -26,7 +26,7 @@ If you run into any issues, refer to [Troubleshooting](/troubleshoot/elasticsear ## Prerequisites [remote-clusters-prerequisites-cert] -1. The {{es}} security features need to be enabled on both clusters, on every node. Security is enabled by default. If it’s disabled, set `xpack.security.enabled` to `true` in `elasticsearch.yml`. Refer to [General security settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#general-security-settings). +1. The {{es}} security features need to be enabled on both clusters, on every node. Security is enabled by default. If it’s disabled, set `xpack.security.enabled` to `true` in `elasticsearch.yml`. Refer to [General security settings](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#general-security-settings). 2. The local and remote clusters versions must be compatible. :::{include} _snippets/remote-cluster-certificate-compatibility.md @@ -57,7 +57,7 @@ You must have the `manage` cluster privilege to connect remote clusters. :::: -The local cluster uses the [transport interface](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md) to establish communication with remote clusters. The coordinating nodes in the local cluster establish [long-lived](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md#long-lived-connections) TCP connections with specific nodes in the remote cluster. {{es}} requires these connections to remain open, even if the connections are idle for an extended period. +The local cluster uses the [transport interface](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md) to establish communication with remote clusters. The coordinating nodes in the local cluster establish [long-lived](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#long-lived-connections) TCP connections with specific nodes in the remote cluster. {{es}} requires these connections to remain open, even if the connections are idle for an extended period. To add a remote cluster from Stack Management in {{kib}}: @@ -122,7 +122,7 @@ The API response indicates that the local cluster is connected to the remote clu Use the [cluster update settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings) to dynamically configure remote settings on every node in the cluster. The following request adds three remote clusters: `cluster_one`, `cluster_two`, and `cluster_three`. -The `seeds` parameter specifies the hostname and [transport port](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md) (default `9300`) of a seed node in the remote cluster. +The `seeds` parameter specifies the hostname and [transport port](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md) (default `9300`) of a seed node in the remote cluster. The `mode` parameter determines the configured connection mode, which defaults to [`sniff`](/deploy-manage/remote-clusters/remote-clusters-self-managed.md#sniff-mode). Because `cluster_one` doesn’t specify a `mode`, it uses the default. Both `cluster_two` and `cluster_three` explicitly use different modes. diff --git a/deploy-manage/remote-clusters/remote-clusters-migrate.md b/deploy-manage/remote-clusters/remote-clusters-migrate.md index 59dc1e72e..c5dfd2390 100644 --- a/deploy-manage/remote-clusters/remote-clusters-migrate.md +++ b/deploy-manage/remote-clusters/remote-clusters-migrate.md @@ -42,9 +42,9 @@ On the remote cluster: 1. Enable the remote cluster server on every node of the remote cluster. In `elasticsearch.yml`: - 1. Set [`remote_cluster_server.enabled`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md#remote-cluster-network-settings) to `true`. - 2. Configure the bind and publish address for remote cluster server traffic, for example using [`remote_cluster.host`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md#remote-cluster-network-settings). Without configuring the address, remote cluster traffic may be bound to the local interface, and remote clusters running on other machines can’t connect. - 3. Optionally, configure the remote server port using [`remote_cluster.port`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md#remote_cluster.port) (defaults to `9443`). + 1. Set [`remote_cluster_server.enabled`](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#remote-cluster-network-settings) to `true`. + 2. Configure the bind and publish address for remote cluster server traffic, for example using [`remote_cluster.host`](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#remote-cluster-network-settings). Without configuring the address, remote cluster traffic may be bound to the local interface, and remote clusters running on other machines can’t connect. + 3. Optionally, configure the remote server port using [`remote_cluster.port`](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#remote_cluster.port) (defaults to `9443`). 2. Next, generate a certificate authority (CA) and a server certificate/key pair. On one of the nodes of the remote cluster, from the directory where {{es}} has been installed: @@ -78,7 +78,7 @@ On the remote cluster: 4. If the remote cluster has multiple nodes, you can either: * create a single wildcard certificate for all nodes; - * or, create separate certificates for each node either manually or in batch with the [silent mode](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/certutil.md#certutil-silent). + * or, create separate certificates for each node either manually or in batch with the [silent mode](elasticsearch://reference/elasticsearch/command-line-tools/certutil.md#certutil-silent). 3. On every node of the remote cluster: @@ -224,7 +224,7 @@ Resume any persistent tasks that you stopped earlier. Tasks should be restarted ## Disable certificate based authentication and authorization [remote-clusters-migration-disable-cert] -::::{note} +::::{note} Only proceed with this step if the migration has been proved successful on the local cluster. If the migration is unsuccessful, either [find out what the problem is and attempt to fix it](/troubleshoot/elasticsearch/remote-clusters.md) or [roll back](#remote-clusters-migration-rollback). :::: diff --git a/deploy-manage/remote-clusters/remote-clusters-settings.md b/deploy-manage/remote-clusters/remote-clusters-settings.md index acf9c3610..20bdabbf2 100644 --- a/deploy-manage/remote-clusters/remote-clusters-settings.md +++ b/deploy-manage/remote-clusters/remote-clusters-settings.md @@ -16,8 +16,8 @@ The following settings apply to both [sniff mode](/deploy-manage/remote-clusters `cluster.remote.initial_connect_timeout` : The time to wait for remote connections to be established when the node starts. The default is `30s`. -`remote_cluster_client` [role](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md#node-roles) -: By default, any node in the cluster can act as a cross-cluster client and connect to remote clusters. To prevent a node from connecting to remote clusters, specify the [node.roles](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md#node-roles) setting in `elasticsearch.yml` and exclude `remote_cluster_client` from the listed roles. Search requests targeting remote clusters must be sent to a node that is allowed to act as a cross-cluster client. Other features such as {{ml}} [data feeds](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/machine-learning-settings.md#general-ml-settings), [transforms](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/transforms-settings.md#general-transform-settings), and [{{ccr}}](../tools/cross-cluster-replication/set-up-cross-cluster-replication.md) require the `remote_cluster_client` role. +`remote_cluster_client` [role](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md#node-roles) +: By default, any node in the cluster can act as a cross-cluster client and connect to remote clusters. To prevent a node from connecting to remote clusters, specify the [node.roles](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md#node-roles) setting in `elasticsearch.yml` and exclude `remote_cluster_client` from the listed roles. Search requests targeting remote clusters must be sent to a node that is allowed to act as a cross-cluster client. Other features such as {{ml}} [data feeds](elasticsearch://reference/elasticsearch/configuration-reference/machine-learning-settings.md#general-ml-settings), [transforms](elasticsearch://reference/elasticsearch/configuration-reference/transforms-settings.md#general-transform-settings), and [{{ccr}}](../tools/cross-cluster-replication/set-up-cross-cluster-replication.md) require the `remote_cluster_client` role. `cluster.remote..skip_unavailable` : Per cluster boolean setting that allows to skip specific clusters when no nodes belonging to them are available and they are the target of a remote cluster request. @@ -28,13 +28,13 @@ In {{es}} 8.15, the default value for `skip_unavailable` was changed from `false `cluster.remote..transport.ping_schedule` -: Sets the time interval between regular application-level ping messages that are sent to try and keep remote cluster connections alive. If set to `-1`, application-level ping messages to this remote cluster are not sent. If unset, application-level ping messages are sent according to the global `transport.ping_schedule` setting, which defaults to `-1` meaning that pings are not sent. It is preferable to correctly configure TCP keep-alives instead of configuring a `ping_schedule`, because TCP keep-alives are handled by the operating system and not by {{es}}. By default {{es}} enables TCP keep-alives on remote cluster connections. Remote cluster connections are transport connections so the `transport.tcp.*` [advanced settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md#transport-settings) regarding TCP keep-alives apply to them. +: Sets the time interval between regular application-level ping messages that are sent to try and keep remote cluster connections alive. If set to `-1`, application-level ping messages to this remote cluster are not sent. If unset, application-level ping messages are sent according to the global `transport.ping_schedule` setting, which defaults to `-1` meaning that pings are not sent. It is preferable to correctly configure TCP keep-alives instead of configuring a `ping_schedule`, because TCP keep-alives are handled by the operating system and not by {{es}}. By default {{es}} enables TCP keep-alives on remote cluster connections. Remote cluster connections are transport connections so the `transport.tcp.*` [advanced settings](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#transport-settings) regarding TCP keep-alives apply to them. `cluster.remote..transport.compress` -: Per-cluster setting that enables you to configure compression for requests to a specific remote cluster. The handling cluster will automatically compress responses to compressed requests. The setting options are `true`, `indexing_data`, and `false`. If unset, defaults to the behaviour specified by the node-wide `transport.compress` setting. See the [documentation for the `transport.compress` setting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md#transport-settings-compress) for further information. +: Per-cluster setting that enables you to configure compression for requests to a specific remote cluster. The handling cluster will automatically compress responses to compressed requests. The setting options are `true`, `indexing_data`, and `false`. If unset, defaults to the behaviour specified by the node-wide `transport.compress` setting. See the [documentation for the `transport.compress` setting](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#transport-settings-compress) for further information. `cluster.remote..transport.compression_scheme` -: Per-cluster setting that enables you to configure the compression scheme for requests to a specific cluster if those requests are selected to be compressed by to the `cluster.remote..transport.compress` setting. The handling cluster will automatically use the same compression scheme for responses as for the corresponding requests. The setting options are `deflate` and `lz4`. If unset, defaults to the behaviour specified by the node-wide `transport.compression_scheme` setting. See the [documentation for the `transport.compression_scheme` setting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md#transport-settings-compression-scheme) for further information. +: Per-cluster setting that enables you to configure the compression scheme for requests to a specific cluster if those requests are selected to be compressed by to the `cluster.remote..transport.compress` setting. The handling cluster will automatically use the same compression scheme for responses as for the corresponding requests. The setting options are `deflate` and `lz4`. If unset, defaults to the behaviour specified by the node-wide `transport.compression_scheme` setting. See the [documentation for the `transport.compression_scheme` setting](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#transport-settings-compression-scheme) for further information. $$$remote-cluster-credentials-setting$$$ diff --git a/deploy-manage/security/different-ca.md b/deploy-manage/security/different-ca.md index 5e50eeaac..1b935d4a2 100644 --- a/deploy-manage/security/different-ca.md +++ b/deploy-manage/security/different-ca.md @@ -15,7 +15,7 @@ If you have to trust a new CA from your organization, or you need to generate a Create a new CA certificate, or get the CA certificate of your organization, and add it to your existing CA truststore. After you finish updating your certificates for all nodes, you can remove the old CA certificate from your truststore (but not before!). -::::{note} +::::{note} The following examples use PKCS#12 files, but the same steps apply to JKS keystores. :::: @@ -24,7 +24,7 @@ The following examples use PKCS#12 files, but the same steps apply to JKS keysto In this example, the keystore and truststore are using different files. Your configuration might use the same file for both the keystore and the truststore. - ::::{note} + ::::{note} These instructions assume that the provided certificate is signed by a trusted CA and the verification mode is set to `certificate`. This setting ensures that nodes to not attempt to perform hostname verification. :::: @@ -54,7 +54,7 @@ The following examples use PKCS#12 files, but the same steps apply to JKS keysto 1. Enter a name for the compressed output file that will contain your certificate and key, or accept the default name of `elastic-stack-ca.zip`. 2. Unzip the output file. The resulting directory contains a CA certificate (`ca.crt`) and a private key (`ca.key`). - ::::{important} + ::::{important} Keep these file in a secure location as they contain the private key for your CA. :::: @@ -92,12 +92,12 @@ The following examples use PKCS#12 files, but the same steps apply to JKS keysto -### Generate a new certificate for each node in your cluster [node-certs-different-nodes] +### Generate a new certificate for each node in your cluster [node-certs-different-nodes] Now that your CA truststore is updated, use your new CA certificate to sign a certificate for your nodes. -::::{note} -If your organization has its own CA, you’ll need to [generate Certificate Signing Requests (CSRs)](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/certutil.md#certutil-csr). CSRs contain information that your CA uses to generate and sign a security certificate. +::::{note} +If your organization has its own CA, you’ll need to [generate Certificate Signing Requests (CSRs)](elasticsearch://reference/elasticsearch/command-line-tools/certutil.md#certutil-csr). CSRs contain information that your CA uses to generate and sign a security certificate. :::: @@ -126,7 +126,7 @@ If your organization has its own CA, you’ll need to [generate Certificate Sign 3. Replace your existing keystore with the new keystore, ensuring that the file names match. For example, `elastic-certificates.p12`. - ::::{important} + ::::{important} If your [keystore password is changing](same-ca.md#cert-password-updates), then save the keystore with a new filename so that {{es}} doesn’t attempt to reload the file before you update the password. :::: @@ -167,7 +167,7 @@ If your organization has its own CA, you’ll need to [generate Certificate Sign -### What’s next? [transport-layer-newca-whatsnext] +### What’s next? [transport-layer-newca-whatsnext] Well done! You’ve updated the keystore for the transport layer. You can also [update the keystore for the HTTP layer](#node-certs-different-http) if necessary. If you’re not updating the keystore for the HTTP layer, then you’re all set. @@ -176,8 +176,8 @@ Well done! You’ve updated the keystore for the transport layer. You can also [ You can generate certificates for the HTTP layer using your new CA certificate and private key. Other components such as {{kib}} or any of the Elastic language clients verify this certificate when they connect to {{es}}. -::::{note} -If your organization has its own CA, you’ll need to [generate Certificate Signing Requests (CSRs)](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/certutil.md#certutil-csr). CSRs contain information that your CA uses to generate and sign a security certificate instead of using self-signed certificates that the `elasticsearch-certutil` tool generates. +::::{note} +If your organization has its own CA, you’ll need to [generate Certificate Signing Requests (CSRs)](elasticsearch://reference/elasticsearch/command-line-tools/certutil.md#certutil-csr). CSRs contain information that your CA uses to generate and sign a security certificate instead of using self-signed certificates that the `elasticsearch-certutil` tool generates. :::: @@ -245,7 +245,7 @@ This process is different for each client, so refer to your client’s documenta 6. Replace your existing keystore with the new keystore, ensuring that the file names match. For example, `node1-http.p12`. - ::::{important} + ::::{important} If your [keystore password is changing](same-ca.md#cert-password-updates), then save the keystore with a new filename so that {{es}} doesn’t attempt to reload the file before you update the password. :::: @@ -280,7 +280,7 @@ This process is different for each client, so refer to your client’s documenta 12. Complete the remaining steps for a [rolling restart](../maintenance/start-stop-services/full-cluster-restart-rolling-restart-procedures.md#restart-cluster-rolling), beginning with the step to **Reenable shard allocation**. -### What’s next? [http-kibana-newca-whatsnext] +### What’s next? [http-kibana-newca-whatsnext] Well done! You’ve updated the keystore for the HTTP layer. You can now [update encryption between {{kib}} and {{es}}](#node-certs-different-kibana). @@ -291,7 +291,7 @@ When you ran the `elasticsearch-certutil` tool with the `http` option, it create 1. Copy the `elasticsearch-ca.pem` file to the {{kib}} configuration directory, as defined by the `KBN_PATH_CONF` path. - ::::{note} + ::::{note} `KBN_PATH_CONF` contains the path for the {{kib}} configuration files. If you installed {{kib}} using archive distributions (`zip` or `tar.gz`), the path defaults to `KBN_HOME/config`. If you used package distributions (Debian or RPM), the path defaults to `/etc/kibana`. :::: diff --git a/deploy-manage/security/enabling-cipher-suites-for-stronger-encryption.md b/deploy-manage/security/enabling-cipher-suites-for-stronger-encryption.md index 25ee8bd12..6a2555777 100644 --- a/deploy-manage/security/enabling-cipher-suites-for-stronger-encryption.md +++ b/deploy-manage/security/enabling-cipher-suites-for-stronger-encryption.md @@ -9,9 +9,9 @@ The TLS and SSL protocols use a cipher suite that determines the strength of enc The *Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files* enable the use of additional cipher suites for Java in a separate JAR file that you need to add to your Java installation. You can download this JAR file from Oracle’s [download page](http://www.oracle.com/technetwork/java/javase/downloads/index.html). The *JCE Unlimited Strength Jurisdiction Policy Files`* are required for encryption with key lengths greater than 128 bits, such as 256-bit AES encryption. -After installation, all cipher suites in the JCE are available for use but requires configuration in order to use them. To enable the use of stronger cipher suites with {{es}} {{security-features}}, configure the [`cipher_suites` parameter](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#ssl-tls-settings). +After installation, all cipher suites in the JCE are available for use but requires configuration in order to use them. To enable the use of stronger cipher suites with {{es}} {{security-features}}, configure the [`cipher_suites` parameter](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#ssl-tls-settings). -::::{note} +::::{note} The *JCE Unlimited Strength Jurisdiction Policy Files* must be installed on all nodes in the cluster to establish an improved level of encryption strength. :::: diff --git a/deploy-manage/security/httprest-clients-security.md b/deploy-manage/security/httprest-clients-security.md index 3e42f9962..edcd4c302 100644 --- a/deploy-manage/security/httprest-clients-security.md +++ b/deploy-manage/security/httprest-clients-security.md @@ -17,7 +17,7 @@ Authorization: Basic <1> Alternatively, you can use [token-based authentication services](../users-roles/cluster-or-deployment-auth/token-based-authentication-services.md). -## Client examples [http-clients-examples] +## Client examples [http-clients-examples] This example uses `curl` without basic auth to create an index: @@ -45,7 +45,7 @@ curl --user rdeniro:taxidriver -XPUT 'localhost:9200/idx' ``` -## Secondary authorization [http-clients-secondary-authorization] +## Secondary authorization [http-clients-secondary-authorization] Some APIs support secondary authorization headers for situations where you want tasks to run with a different set of credentials. For example, you can send the following header in addition to the basic authentication header: @@ -66,15 +66,15 @@ es-secondary-authorization: ApiKey <1> -## Client libraries over HTTP [http-clients-libraries] +## Client libraries over HTTP [http-clients-libraries] For more information about using {{security-features}} with the language specific clients, refer to: -* [Java](asciidocalypse://docs/elasticsearch-java/docs/reference/_basic_authentication.md) -* [JavaScript](asciidocalypse://docs/elasticsearch-js/docs/reference/connecting.md) -* [.NET](asciidocalypse://docs/elasticsearch-net/docs/reference/configuration.md) +* [Java](elasticsearch-java://reference/_basic_authentication.md) +* [JavaScript](elasticsearch-js://reference/connecting.md) +* [.NET](elasticsearch-net://reference/configuration.md) * [Perl](https://metacpan.org/pod/Search::Elasticsearch::Cxn::HTTPTiny#CONFIGURATION) -* [PHP](asciidocalypse://docs/elasticsearch-php/docs/reference/connecting.md) +* [PHP](elasticsearch-php://reference/connecting.md) * [Python](https://elasticsearch-py.readthedocs.io/en/master/#ssl-and-authentication) * [Ruby](https://github.com/elasticsearch/elasticsearch-ruby/tree/master/elasticsearch-transport#authentication) diff --git a/deploy-manage/security/kibana-session-management.md b/deploy-manage/security/kibana-session-management.md index 7e63a2cb5..577b7b16b 100644 --- a/deploy-manage/security/kibana-session-management.md +++ b/deploy-manage/security/kibana-session-management.md @@ -9,7 +9,7 @@ When you log in, {{kib}} creates a session that is used to authenticate subseque When your session expires, or you log out, {{kib}} will invalidate your cookie and remove session information from the index. {{kib}} also periodically invalidates and removes any expired sessions that weren’t explicitly invalidated. -To manage user sessions programmatically, {{kib}} exposes [session management APIs](https://www.elastic.co/guide/en/kibana/current/session-management-api.html). For details, check out [Session and cookie security settings](asciidocalypse://docs/kibana/docs/reference/configuration-reference/security-settings.md#security-session-and-cookie-settings). +To manage user sessions programmatically, {{kib}} exposes [session management APIs](https://www.elastic.co/guide/en/kibana/current/session-management-api.html). For details, check out [Session and cookie security settings](kibana://reference/configuration-reference/security-settings.md#security-session-and-cookie-settings). ## Session idle timeout [session-idle-timeout] @@ -31,7 +31,7 @@ xpack.security.session.lifespan: "7d" ## Session cleanup interval [session-cleanup-interval] -::::{important} +::::{important} If you disable session idle timeout and lifespan, then Kibana will not automatically remove session information from the index unless you explicitly log out. This might lead to an infinitely growing session index. As long as either idle timeout or lifespan is configured, Kibana sessions will be cleaned up even if you don’t explicitly log out. :::: @@ -48,7 +48,7 @@ xpack.security.session.cleanupInterval: "1d" By default, there is no limit to the maximum number of concurrent sessions each user can have in {{kib}}. To add a limit, use the `xpack.security.session.сoncurrentSessions.maxSessions` configuration option. If set, the value of this option should be an integer between `1` and `1000`. When the limit is exceeded, the oldest session is automatically invalidated. -::::{note} +::::{note} Due to the rate at which session information is refreshed, there might be a few seconds where the concurrent session limit is not enforced. This is something to consider for use cases where it is common to create multiple sessions simultaneously. :::: diff --git a/deploy-manage/security/same-ca.md b/deploy-manage/security/same-ca.md index e8ae052da..dc1379320 100644 --- a/deploy-manage/security/same-ca.md +++ b/deploy-manage/security/same-ca.md @@ -17,14 +17,14 @@ You don’t have to restart each node, but doing so forces new TLS connections a The following steps provide instructions for generating new node certificates and keys for both the transport layer and the HTTP layer. You might only need to replace one of these layer’s certificates depending on which of your certificates are expiring. -::::{important} +::::{important} :name: cert-password-updates If your keystore is password protected, the password is stored in the {{es}} secure settings, *and* the password needs to change, then you must perform a [rolling restart](../maintenance/start-stop-services/full-cluster-restart-rolling-restart-procedures.md#restart-cluster-rolling) on your cluster. You must also use a different file name for the keystore so that {{es}} doesn’t reload the file before the node is restarted. :::: -::::{tip} +::::{tip} If your CA has changed, complete the steps in [update security certificates with a different CA](different-ca.md). :::: @@ -37,7 +37,7 @@ The following examples use PKCS#12 files, but the same steps apply to JKS keysto In this example, the keystore and truststore are pointing to different files. Your configuration might use the same file that contains the certificate and CA. In this case, include the path to that file for both the keystore and truststore. - ::::{note} + ::::{note} These instructions assume that the provided certificate is signed by a trusted CA and the verification mode is set to `certificate`. This setting ensures that nodes to not attempt to perform hostname verification. :::: @@ -79,7 +79,7 @@ The following examples use PKCS#12 files, but the same steps apply to JKS keysto 5. $$$replace-keystores$$$Replace your existing keystore with the new keystore, ensuring that the file names match. For example, `elastic-certificates.p12`. - ::::{important} + ::::{important} If your [keystore password is changing](#cert-password-updates), then save the keystore with a new filename so that {{es}} doesn’t attempt to reload the file before you update the password. :::: @@ -105,7 +105,7 @@ The following examples use PKCS#12 files, but the same steps apply to JKS keysto -### What’s next? [transport-layer-sameca-whatsnext] +### What’s next? [transport-layer-sameca-whatsnext] Well done! You’ve updated the keystore for the transport layer. You can also [update the keystore for the HTTP layer](#node-certs-same-http) if necessary. If you’re not updating the keystore for the HTTP layer, then you’re all set. @@ -114,8 +114,8 @@ Well done! You’ve updated the keystore for the transport layer. You can also [ Other components such as {{kib}} or any of the Elastic language clients verify this certificate when they connect to {{es}}. -::::{note} -If your organization has its own CA, you’ll need to [generate Certificate Signing Requests (CSRs)](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/certutil.md#certutil-csr). CSRs contain information that your CA uses to generate and sign a certificate. +::::{note} +If your organization has its own CA, you’ll need to [generate Certificate Signing Requests (CSRs)](elasticsearch://reference/elasticsearch/command-line-tools/certutil.md#certutil-csr). CSRs contain information that your CA uses to generate and sign a certificate. :::: @@ -175,7 +175,7 @@ If your organization has its own CA, you’ll need to [generate Certificate Sign 6. Replace your existing keystore with the new keystore, ensuring that the file names match. For example, `node1-http.p12`. - ::::{important} + ::::{important} If your [keystore password is changing](#cert-password-updates), then save the keystore with a new filename so that {{es}} doesn’t attempt to reload the file before you update the password. :::: diff --git a/deploy-manage/security/secure-clients-integrations.md b/deploy-manage/security/secure-clients-integrations.md index 7423fff72..e63862420 100644 --- a/deploy-manage/security/secure-clients-integrations.md +++ b/deploy-manage/security/secure-clients-integrations.md @@ -9,13 +9,13 @@ You will need to update the configuration for several [clients](httprest-clients The {{es}} {{security-features}} enable you to secure your {{es}} cluster. But {{es}} itself is only one product within the {{stack}}. It is often the case that other products in the {{stack}} are connected to the cluster and therefore need to be secured as well, or at least communicate with the cluster in a secured way: -* [Apache Hadoop](asciidocalypse://docs/elasticsearch-hadoop/docs/reference/security.md) +* [Apache Hadoop](elasticsearch-hadoop://reference/security.md) * [Auditbeat](asciidocalypse://docs/beats/docs/reference/auditbeat/securing-auditbeat.md) * [Filebeat](asciidocalypse://docs/beats/docs/reference/filebeat/securing-filebeat.md) -* [{{fleet}} & {{agent}}](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/secure.md) +* [{{fleet}} & {{agent}}](/reference/ingestion-tools/fleet/secure.md) * [Heartbeat](asciidocalypse://docs/beats/docs/reference/heartbeat/securing-heartbeat.md) * [{{kib}}](../security.md) -* [Logstash](asciidocalypse://docs/logstash/docs/reference/secure-connection.md) +* [Logstash](logstash://reference/secure-connection.md) * [Metricbeat](asciidocalypse://docs/beats/docs/reference/metricbeat/securing-metricbeat.md) * [Monitoring and security](../monitor.md) * [Packetbeat](asciidocalypse://docs/beats/docs/reference/packetbeat/securing-packetbeat.md) diff --git a/deploy-manage/security/secure-endpoints.md b/deploy-manage/security/secure-endpoints.md index b8171bc14..6e89c619d 100644 --- a/deploy-manage/security/secure-endpoints.md +++ b/deploy-manage/security/secure-endpoints.md @@ -8,24 +8,24 @@ mapped_pages: Protecting your {{es}} cluster and the data it contains is of utmost importance. Implementing a defense in depth strategy provides multiple layers of security to help safeguard your system. The following principles provide a foundation for running {{es}} in a secure manner that helps to mitigate attacks on your system at multiple levels. -## Run {{es}} with security enabled [security-run-with-security] +## Run {{es}} with security enabled [security-run-with-security] Never run an {{es}} cluster without security enabled. This principle cannot be overstated. Running {{es}} without security leaves your cluster exposed to anyone who can send network traffic to {{es}}, permitting these individuals to download, modify, or delete any data in your cluster. [Start the {{stack}} with security enabled](../deploy/self-managed/installing-elasticsearch.md) or [manually configure security](manually-configure-security-in-self-managed-cluster.md) to prevent unauthorized access to your clusters and ensure that internode communication is secure. -## Run {{es}} with a dedicated non-root user [security-not-root-user] +## Run {{es}} with a dedicated non-root user [security-not-root-user] Never try to run {{es}} as the `root` user, which would invalidate any defense strategy and permit a malicious user to do **anything** on your server. You must create a dedicated, unprivileged user to run {{es}}. By default, the `rpm`, `deb`, `docker`, and Windows packages of {{es}} contain an `elasticsearch` user with this scope. -## Protect {{es}} from public internet traffic [security-protect-cluster-traffic] +## Protect {{es}} from public internet traffic [security-protect-cluster-traffic] Even with security enabled, never expose {{es}} to public internet traffic. Using an application to sanitize requests to {{es}} still poses risks, such as a malicious user writing [`_search`](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-search) requests that could overwhelm an {{es}} cluster and bring it down. Keep {{es}} as isolated as possible, preferably behind a firewall and a VPN. Any internet-facing applications should run pre-canned aggregations, or not run aggregations at all. -While you absolutely shouldn’t expose {{es}} directly to the internet, you also shouldn’t expose {{es}} directly to users. Instead, use an intermediary application to make requests on behalf of users. This implementation allows you to track user behaviors, such as can submit requests, and to which specific nodes in the cluster. For example, you can implement an application that accepts a search term from a user and funnels it through a [`simple_query_string`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-simple-query-string-query.md) query. +While you absolutely shouldn’t expose {{es}} directly to the internet, you also shouldn’t expose {{es}} directly to users. Instead, use an intermediary application to make requests on behalf of users. This implementation allows you to track user behaviors, such as can submit requests, and to which specific nodes in the cluster. For example, you can implement an application that accepts a search term from a user and funnels it through a [`simple_query_string`](elasticsearch://reference/query-languages/query-dsl-simple-query-string-query.md) query. -## Implement role based access control [security-create-appropriate-users] +## Implement role based access control [security-create-appropriate-users] [Define roles](../users-roles/cluster-or-deployment-auth/defining-roles.md) for your users and [assign appropriate privileges](/deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md) to ensure that users have access only to the resources that they need. This process determines whether the user behind an incoming request is allowed to run that request. diff --git a/deploy-manage/security/secure-saved-objects.md b/deploy-manage/security/secure-saved-objects.md index 7cf075884..cdb9461a0 100644 --- a/deploy-manage/security/secure-saved-objects.md +++ b/deploy-manage/security/secure-saved-objects.md @@ -16,14 +16,14 @@ xpack.encryptedSavedObjects: encryptionKey: "min-32-byte-long-strong-encryption-key" ``` -::::{important} +::::{important} If you don’t specify an encryption key, {{kib}} might disable features that rely on encrypted saved objects. :::: -::::{tip} -For help generating the encryption key, refer to the [`kibana-encryption-keys`](asciidocalypse://docs/kibana/docs/reference/commands/kibana-encryption-keys.md) script. +::::{tip} +For help generating the encryption key, refer to the [`kibana-encryption-keys`](kibana://reference/commands/kibana-encryption-keys.md) script. :::: @@ -47,7 +47,7 @@ xpack.encryptedSavedObjects: 2. A list of encryption keys {{kib}} will try to use to decrypt existing saved objects if decryption with the primary encryption key isn’t possible. These keys are known as the decryption-only or secondary encryption keys. -::::{note} +::::{note} You might also leverage this functionality if multiple {{kib}} instances connected to the same {{es}} cluster use different encryption keys. In this case, you might have a mix of saved objects encrypted with different keys, and every {{kib}} instance can only deal with a specific subset of objects. To fix this, you must choose a single primary encryption key for `xpack.encryptedSavedObjects.encryptionKey`, move all other encryption keys to `xpack.encryptedSavedObjects.keyRotation.decryptionOnlyKeys`, and sync this configuration across all {{kib}} instances. :::: diff --git a/deploy-manage/security/security-certificates-keys.md b/deploy-manage/security/security-certificates-keys.md index 26a6512e7..6d41517ff 100644 --- a/deploy-manage/security/security-certificates-keys.md +++ b/deploy-manage/security/security-certificates-keys.md @@ -39,7 +39,7 @@ There are [some cases](../deploy/self-managed/installing-elasticsearch.md#stack- 2. Copy the generated `elastic` password and enrollment token. These credentials are only shown when you start {{es}} for the first time. ::::{note} - If you need to reset the password for the `elastic` user or other built-in users, run the [`elasticsearch-reset-password`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/reset-password.md) tool. To generate new enrollment tokens for {{kib}} or {{es}} nodes, run the [`elasticsearch-create-enrollment-token`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/create-enrollment-token.md) tool. These tools are available in the {{es}} `bin` directory. + If you need to reset the password for the `elastic` user or other built-in users, run the [`elasticsearch-reset-password`](elasticsearch://reference/elasticsearch/command-line-tools/reset-password.md) tool. To generate new enrollment tokens for {{kib}} or {{es}} nodes, run the [`elasticsearch-create-enrollment-token`](elasticsearch://reference/elasticsearch/command-line-tools/create-enrollment-token.md) tool. These tools are available in the {{es}} `bin` directory. :::: @@ -90,11 +90,11 @@ When {{es}} starts for the first time, the security auto-configuration process b Before enrolling a new node, additional actions such as binding to an address other than `localhost` or satisfying bootstrap checks are typically necessary in production clusters. During that time, an auto-generated enrollment token could expire, which is why enrollment tokens aren’t generated automatically. -Additionally, only nodes on the same host can join the cluster without additional configuration. If you want nodes from another host to join your cluster, you need to set `transport.host` to a [supported value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md#network-interface-values) (such as uncommenting the suggested value of `0.0.0.0`), or an IP address that’s bound to an interface where other hosts can reach it. Refer to [transport settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md#transport-settings) for more information. +Additionally, only nodes on the same host can join the cluster without additional configuration. If you want nodes from another host to join your cluster, you need to set `transport.host` to a [supported value](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#network-interface-values) (such as uncommenting the suggested value of `0.0.0.0`), or an IP address that’s bound to an interface where other hosts can reach it. Refer to [transport settings](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#transport-settings) for more information. To enroll new nodes in your cluster, create an enrollment token with the `elasticsearch-create-enrollment-token` tool on any existing node in your cluster. You can then start a new node with the `--enrollment-token` parameter so that it joins an existing cluster. -1. In a separate terminal from where {{es}} is running, navigate to the directory where you installed {{es}} and run the [`elasticsearch-create-enrollment-token`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/create-enrollment-token.md) tool to generate an enrollment token for your new nodes. +1. In a separate terminal from where {{es}} is running, navigate to the directory where you installed {{es}} and run the [`elasticsearch-create-enrollment-token`](elasticsearch://reference/elasticsearch/command-line-tools/create-enrollment-token.md) tool to generate an enrollment token for your new nodes. ```sh bin/elasticsearch-create-enrollment-token -s node @@ -177,7 +177,7 @@ When you install {{es}}, the following certificates and keys are generated in th `transport.p12` : Keystore that contains the key and certificate for the transport layer for all the nodes in your cluster. -`http.p12` and `transport.p12` are password-protected PKCS#12 keystores. {{es}} stores the passwords for these keystores as [secure settings](secure-settings.md). To retrieve the passwords so that you can inspect or change the keystore contents, use the [`bin/elasticsearch-keystore`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/elasticsearch-keystore.md) tool. +`http.p12` and `transport.p12` are password-protected PKCS#12 keystores. {{es}} stores the passwords for these keystores as [secure settings](secure-settings.md). To retrieve the passwords so that you can inspect or change the keystore contents, use the [`bin/elasticsearch-keystore`](elasticsearch://reference/elasticsearch/command-line-tools/elasticsearch-keystore.md) tool. Use the following command to retrieve the password for `http.p12`: @@ -228,11 +228,11 @@ The {{es}} configuration directory isn’t writable The following settings are incompatible with security auto configuration. If any of these settings exist, the node startup process skips configuring security automatically and the node starts normally. -* [`node.roles`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md#node-roles) is set to a value where the node can’t be elected as `master`, or if the node can’t hold data -* [`xpack.security.autoconfiguration.enabled`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#general-security-settings) is set to `false` -* [`xpack.security.enabled`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#general-security-settings) has a value set -* Any of the [`xpack.security.transport.ssl.*`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#transport-tls-ssl-settings) or [`xpack.security.http.ssl.*`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#http-tls-ssl-settings) settings have a value set in the `elasticsearch.yml` configuration file or in the `elasticsearch.keystore` -* Any of the `discovery.type`, `discovery.seed_hosts`, or `cluster.initial_master_nodes` [discovery and cluster formation settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/discovery-cluster-formation-settings.md) have a value set +* [`node.roles`](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md#node-roles) is set to a value where the node can’t be elected as `master`, or if the node can’t hold data +* [`xpack.security.autoconfiguration.enabled`](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#general-security-settings) is set to `false` +* [`xpack.security.enabled`](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#general-security-settings) has a value set +* Any of the [`xpack.security.transport.ssl.*`](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#transport-tls-ssl-settings) or [`xpack.security.http.ssl.*`](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#http-tls-ssl-settings) settings have a value set in the `elasticsearch.yml` configuration file or in the `elasticsearch.keystore` +* Any of the `discovery.type`, `discovery.seed_hosts`, or `cluster.initial_master_nodes` [discovery and cluster formation settings](elasticsearch://reference/elasticsearch/configuration-reference/discovery-cluster-formation-settings.md) have a value set ::::{note} Exceptions are when `discovery.type` is set to `single-node`, or when `cluster.initial_master_nodes` exists but contains only the name of the current node. diff --git a/deploy-manage/security/set-up-basic-security-plus-https.md b/deploy-manage/security/set-up-basic-security-plus-https.md index 3a11acfd3..b77f32be2 100644 --- a/deploy-manage/security/set-up-basic-security-plus-https.md +++ b/deploy-manage/security/set-up-basic-security-plus-https.md @@ -269,7 +269,7 @@ To send monitoring data securely, create a monitoring user and grant it the nece You can use the built-in `beats_system` user, if it’s available in your environment. Because the built-in users are not available in {{ecloud}}, these instructions create a user that is explicitly used for monitoring {{metricbeat}}. -1. If you’re using the built-in `beats_system` user, on any node in your cluster, run the [`elasticsearch-reset-password`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/reset-password.md) utility to set the password for that user: +1. If you’re using the built-in `beats_system` user, on any node in your cluster, run the [`elasticsearch-reset-password`](elasticsearch://reference/elasticsearch/command-line-tools/reset-password.md) utility to set the password for that user: This command resets the password for the `beats_system` user to an auto-generated value. diff --git a/deploy-manage/security/set-up-basic-security.md b/deploy-manage/security/set-up-basic-security.md index e60f3aa30..173d367d9 100644 --- a/deploy-manage/security/set-up-basic-security.md +++ b/deploy-manage/security/set-up-basic-security.md @@ -11,7 +11,7 @@ mapped_pages: When you start {{es}} for the first time, passwords are generated for the `elastic` user and TLS is automatically configured for you. If you configure security manually *before* starting your {{es}} nodes, the auto-configuration process will respect your security configuration. You can adjust your TLS configuration at any time, such as [updating node certificates](updating-certificates.md). -::::{important} +::::{important} If your cluster has multiple nodes, then you must configure TLS between nodes. [Production mode](../deploy/self-managed/bootstrap-checks.md#dev-vs-prod-mode) clusters will not start if you do not enable TLS. :::: @@ -72,7 +72,7 @@ The transport networking layer is used for internal communication between nodes Now that you’ve generated a certificate authority and certificates, you’ll update your cluster to use these files. -::::{note} +::::{note} {{es}} monitors all files such as certificates, keys, keystores, or truststores that are configured as values of TLS-related node settings. If you update any of these files, such as when your hostnames change or your certificates are due to expire, {{es}} reloads them. The files are polled for changes at a frequency determined by the global {{es}} `resource.reload.interval.high` setting, which defaults to 5 seconds. :::: @@ -81,7 +81,7 @@ Complete the following steps **for each node in your cluster**. To join the same 1. Open the `$ES_PATH_CONF/elasticsearch.yml` file and make the following changes: - 1. Add the [`cluster-name`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/miscellaneous-cluster-settings.md#cluster-name) setting and enter a name for your cluster: + 1. Add the [`cluster-name`](elasticsearch://reference/elasticsearch/configuration-reference/miscellaneous-cluster-settings.md#cluster-name) setting and enter a name for your cluster: ```yaml cluster.name: my-cluster @@ -105,7 +105,7 @@ Complete the following steps **for each node in your cluster**. To join the same xpack.security.transport.ssl.truststore.path: elastic-certificates.p12 ``` - 1. If you want to use hostname verification, set the verification mode to `full`. You should generate a different certificate for each host that matches the DNS or IP address. See the `xpack.security.transport.ssl.verification_mode` parameter in [TLS settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#transport-tls-ssl-settings). + 1. If you want to use hostname verification, set the verification mode to `full`. You should generate a different certificate for each host that matches the DNS or IP address. See the `xpack.security.transport.ssl.verification_mode` parameter in [TLS settings](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#transport-tls-ssl-settings). 2. If you entered a password when creating the node certificate, run the following commands to store the password in the {{es}} keystore: @@ -122,7 +122,7 @@ Complete the following steps **for each node in your cluster**. To join the same For example, if you installed {{es}} with an archive distribution (`tar.gz` or `.zip`), you can enter `Ctrl+C` on the command line to stop {{es}}. - ::::{warning} + ::::{warning} You must perform a full cluster restart. Nodes that are configured to use TLS for transport cannot communicate with nodes that use unencrypted transport connection (and vice-versa). :::: diff --git a/deploy-manage/security/set-up-minimal-security.md b/deploy-manage/security/set-up-minimal-security.md index 60c8c9461..a3dd85c8a 100644 --- a/deploy-manage/security/set-up-minimal-security.md +++ b/deploy-manage/security/set-up-minimal-security.md @@ -9,7 +9,7 @@ mapped_pages: # Set up minimal security [security-minimal-setup] -::::{important} +::::{important} You only need to complete the following steps if you’re running an existing, unsecured cluster and want to enable the {{es}} {{security-features}}. :::: @@ -18,7 +18,7 @@ In {{es}} 8.0 and later, security is [enabled automatically](../deploy/self-mana If you’re running an existing {{es}} cluster where security is disabled, you can manually enable the {{es}} {{security-features}} and then create passwords for built-in users. You can add more users later, but using the built-in users simplifies the process of enabling security for your cluster. -::::{important} +::::{important} The minimal security scenario is not sufficient for [production mode](../deploy/self-managed/bootstrap-checks.md#dev-vs-prod-mode) clusters. If your cluster has multiple nodes, you must enable minimal security and then [configure Transport Layer Security (TLS)](secure-cluster-communications.md) between nodes. :::: @@ -34,7 +34,7 @@ Enabling the {{es}} security features provides basic authentication so that you xpack.security.enabled: true ``` - ::::{note} + ::::{note} The `$ES_PATH_CONF` variable is the path for the {{es}} configuration files. If you installed {{es}} using archive distributions (`zip` or `tar.gz`), the variable defaults to `$ES_HOME/config`. If you used package distributions (Debian or RPM), the variable defaults to `/etc/elasticsearch`. :::: @@ -50,7 +50,7 @@ Enabling the {{es}} security features provides basic authentication so that you To communicate with your cluster, you must configure a password for the `elastic` and `kibana_system` built-in users. Unless you enable anonymous access (not recommended), all requests that don’t include credentials are rejected. -::::{note} +::::{note} You only need to set passwords for the `elastic` and `kibana_system` users when enabling minimal or basic security. :::: @@ -61,7 +61,7 @@ You only need to set passwords for the `elastic` and `kibana_system` users when ./bin/elasticsearch ``` -2. On any node in your cluster, open another terminal window and set the password for the `elastic` built-in user by running the [`elasticsearch-reset-password`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/reset-password.md) utility. This command resets the password to an auto-generated value. +2. On any node in your cluster, open another terminal window and set the password for the `elastic` built-in user by running the [`elasticsearch-reset-password`](elasticsearch://reference/elasticsearch/command-line-tools/reset-password.md) utility. This command resets the password to an auto-generated value. ```shell ./bin/elasticsearch-reset-password -u elastic @@ -98,7 +98,7 @@ This account is not meant for individual users and does not have permission to l elasticsearch.username: "kibana_system" ``` - ::::{note} + ::::{note} The `KBN_PATH_CONF` variable is the path for the {{kib}} configuration files. If you installed {{kib}} using archive distributions (`zip` or `tar.gz`), the variable defaults to `KIB_HOME/config`. If you used package distributions (Debian or RPM), the variable defaults to `/etc/kibana`. :::: diff --git a/deploy-manage/security/supported-ssltls-versions-by-jdk-version.md b/deploy-manage/security/supported-ssltls-versions-by-jdk-version.md index e96d8f6ca..4f024d3d3 100644 --- a/deploy-manage/security/supported-ssltls-versions-by-jdk-version.md +++ b/deploy-manage/security/supported-ssltls-versions-by-jdk-version.md @@ -93,9 +93,9 @@ jdk.tls.disabledAlgorithms=SSLv3, TLSv1, RC4, DES, MD5withRSA, \ ### Enable your custom security configuration [_enable_your_custom_security_configuration] -To enable your custom security policy, add a file in the [`jvm.options.d`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/jvm-settings.md#set-jvm-options) directory within your {{es}} configuration directory. +To enable your custom security policy, add a file in the [`jvm.options.d`](elasticsearch://reference/elasticsearch/jvm-settings.md#set-jvm-options) directory within your {{es}} configuration directory. -To enable your custom security policy, create a file named `java.security.options` within the [jvm.options.d](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/jvm-settings.md#set-jvm-options) directory of your {{es}} configuration directory, with this content: +To enable your custom security policy, create a file named `java.security.options` within the [jvm.options.d](elasticsearch://reference/elasticsearch/jvm-settings.md#set-jvm-options) directory of your {{es}} configuration directory, with this content: ```text -Djava.security.properties=/path/to/your/es.java.security @@ -105,7 +105,7 @@ To enable your custom security policy, create a file named `java.security.option ## Enabling TLS versions in {{es}} [_enabling_tls_versions_in_es] -SSL/TLS versions can be enabled and disabled within {{es}} via the [`ssl.supported_protocols` settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#ssl-tls-settings). +SSL/TLS versions can be enabled and disabled within {{es}} via the [`ssl.supported_protocols` settings](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#ssl-tls-settings). {{es}} will only support the TLS versions that are enabled by the underlying JDK. If you configure `ssl.supported_procotols` to include a TLS version that is not enabled in your JDK, then it will be silently ignored. diff --git a/deploy-manage/tools/cross-cluster-replication.md b/deploy-manage/tools/cross-cluster-replication.md index bfdf72589..8a7e696aa 100644 --- a/deploy-manage/tools/cross-cluster-replication.md +++ b/deploy-manage/tools/cross-cluster-replication.md @@ -186,7 +186,7 @@ When you create a follower index, you cannot use it until it is fully initialize Remote recovery is a network intensive process that transfers all of the Lucene segment files from the leader cluster to the follower cluster. The follower requests that a recovery session be initiated on the primary shard in the leader cluster. The follower then requests file chunks concurrently from the leader. By default, the process concurrently requests five 1MB file chunks. This default behavior is designed to support leader and follower clusters with high network latency between them. ::::{tip} -You can modify dynamic [remote recovery settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cross-cluster-replication-settings.md#ccr-recovery-settings) to rate-limit the transmitted data and manage the resources consumed by remote recoveries. +You can modify dynamic [remote recovery settings](elasticsearch://reference/elasticsearch/configuration-reference/cross-cluster-replication-settings.md#ccr-recovery-settings) to rate-limit the transmitted data and manage the resources consumed by remote recoveries. :::: @@ -195,11 +195,11 @@ Use the [recovery API](https://www.elastic.co/docs/api/doc/elasticsearch/operati ## Replicating a leader requires soft deletes [ccr-leader-requirements] -{{ccr-cap}} works by replaying the history of individual write operations that were performed on the shards of the leader index. {{es}} needs to retain the [history of these operations](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/history-retention.md) on the leader shards so that they can be pulled by the follower shard tasks. The underlying mechanism used to retain these operations is *soft deletes*. +{{ccr-cap}} works by replaying the history of individual write operations that were performed on the shards of the leader index. {{es}} needs to retain the [history of these operations](elasticsearch://reference/elasticsearch/index-settings/history-retention.md) on the leader shards so that they can be pulled by the follower shard tasks. The underlying mechanism used to retain these operations is *soft deletes*. A soft delete occurs whenever an existing document is deleted or updated. By retaining these soft deletes up to configurable limits, the history of operations can be retained on the leader shards and made available to the follower shard tasks as it replays the history of operations. -The [`index.soft_deletes.retention_lease.period`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#ccr-index-soft-deletes-retention-period) setting defines the maximum time to retain a shard history retention lease before it is considered expired. This setting determines how long the cluster containing your follower index can be offline, which is 12 hours by default. If a shard copy recovers after its retention lease expires, but the missing operations are still available on the leader index, then {{es}} will establish a new lease and copy the missing operations. However {{es}} does not guarantee to retain unleased operations, so it is also possible that some of the missing operations have been discarded by the leader and are now completely unavailable. If this happens then the follower cannot recover automatically so you must [recreate it](cross-cluster-replication/ccr-recreate-follower-index.md). +The [`index.soft_deletes.retention_lease.period`](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#ccr-index-soft-deletes-retention-period) setting defines the maximum time to retain a shard history retention lease before it is considered expired. This setting determines how long the cluster containing your follower index can be offline, which is 12 hours by default. If a shard copy recovers after its retention lease expires, but the missing operations are still available on the leader index, then {{es}} will establish a new lease and copy the missing operations. However {{es}} does not guarantee to retain unleased operations, so it is also possible that some of the missing operations have been discarded by the leader and are now completely unavailable. If this happens then the follower cannot recover automatically so you must [recreate it](cross-cluster-replication/ccr-recreate-follower-index.md). Soft deletes must be enabled for indices that you want to use as leader indices. Soft deletes are enabled by default on new indices created on or after {{es}} 7.0.0. @@ -223,13 +223,13 @@ This following sections provide more information about how to configure and use {{ccr-cap}} is designed to replicate user-generated indices only, and doesn’t currently replicate any of the following: -* [System indices](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#system-indices) +* [System indices](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#system-indices) * [Machine learning jobs](../../explore-analyze/machine-learning.md) * [index templates](../../manage-data/data-store/templates.md) * [{{ilm-cap}}](../../manage-data/lifecycle/index-lifecycle-management.md) and [{{slm}}](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-slm) polices * [User permissions and role mappings](../users-roles/cluster-or-deployment-auth/mapping-users-groups-to-roles.md) * [Snapshot repository settings](snapshot-and-restore/self-managed.md) -* [Cluster settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md) +* [Cluster settings](elasticsearch://reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md) * [Searchable snapshot](snapshot-and-restore/searchable-snapshots.md) If you want to replicate any of this data, you must replicate it to a remote cluster manually. diff --git a/deploy-manage/tools/cross-cluster-replication/_failback_when_clustera_comes_back.md b/deploy-manage/tools/cross-cluster-replication/_failback_when_clustera_comes_back.md index 13e71bb2a..fe8022fd0 100644 --- a/deploy-manage/tools/cross-cluster-replication/_failback_when_clustera_comes_back.md +++ b/deploy-manage/tools/cross-cluster-replication/_failback_when_clustera_comes_back.md @@ -4,10 +4,10 @@ mapped_pages: applies_to: deployment: - eck: - ess: - ece: - self: + eck: + ess: + ece: + self: --- # Failback when clusterA comes back [_failback_when_clustera_comes_back] @@ -61,8 +61,8 @@ When `clusterA` comes back, `clusterB` becomes the new leader and `clusterA` bec GET kibana_sample_data_ecommerce/_search?q=kimchy ``` - ::::{tip} - If a soft delete is merged away before it can be replicated to a follower the following process will fail due to incomplete history on the leader, see [index.soft_deletes.retention_lease.period](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#ccr-index-soft-deletes-retention-period) for more details. + ::::{tip} + If a soft delete is merged away before it can be replicated to a follower the following process will fail due to incomplete history on the leader, see [index.soft_deletes.retention_lease.period](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#ccr-index-soft-deletes-retention-period) for more details. :::: diff --git a/deploy-manage/tools/cross-cluster-replication/_perform_update_or_delete_by_query.md b/deploy-manage/tools/cross-cluster-replication/_perform_update_or_delete_by_query.md index f9fe7389a..4b7defb7c 100644 --- a/deploy-manage/tools/cross-cluster-replication/_perform_update_or_delete_by_query.md +++ b/deploy-manage/tools/cross-cluster-replication/_perform_update_or_delete_by_query.md @@ -4,10 +4,10 @@ mapped_pages: applies_to: deployment: - eck: - ess: - ece: - self: + eck: + ess: + ece: + self: --- # Perform update or delete by query [_perform_update_or_delete_by_query] @@ -53,8 +53,8 @@ It is possible to update or delete the documents but you can only perform these } ``` - ::::{tip} - If a soft delete is merged away before it can be replicated to a follower the following process will fail due to incomplete history on the leader, see [index.soft_deletes.retention_lease.period](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#ccr-index-soft-deletes-retention-period) for more details. + ::::{tip} + If a soft delete is merged away before it can be replicated to a follower the following process will fail due to incomplete history on the leader, see [index.soft_deletes.retention_lease.period](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#ccr-index-soft-deletes-retention-period) for more details. :::: diff --git a/deploy-manage/tools/cross-cluster-replication/bi-directional-disaster-recovery.md b/deploy-manage/tools/cross-cluster-replication/bi-directional-disaster-recovery.md index 8897af6df..f8bcd6804 100644 --- a/deploy-manage/tools/cross-cluster-replication/bi-directional-disaster-recovery.md +++ b/deploy-manage/tools/cross-cluster-replication/bi-directional-disaster-recovery.md @@ -5,10 +5,10 @@ mapped_pages: applies_to: deployment: - eck: - ess: - ece: - self: + eck: + ess: + ece: + self: --- @@ -18,7 +18,7 @@ applies_to: Learn how to set up disaster recovery between two clusters based on bi-directional {{ccr}}. The following tutorial is designed for data streams which support [update by query](../../../manage-data/data-store/data-streams/use-data-stream.md#update-docs-in-a-data-stream-by-query) and [delete by query](../../../manage-data/data-store/data-streams/use-data-stream.md#delete-docs-in-a-data-stream-by-query). You can only perform these actions on the leader index. -This tutorial works with {{ls}} as the source of ingestion. It takes advantage of a {{ls}} feature where [the {{ls}} output to {{es}}](asciidocalypse://docs/logstash/docs/reference/plugins-outputs-elasticsearch.md) can be load balanced across an array of hosts specified. {{beats}} and {{agents}} currently do not support multiple outputs. It should also be possible to set up a proxy (load balancer) to redirect traffic without {{ls}} in this tutorial. +This tutorial works with {{ls}} as the source of ingestion. It takes advantage of a {{ls}} feature where [the {{ls}} output to {{es}}](logstash://reference/plugins-outputs-elasticsearch.md) can be load balanced across an array of hosts specified. {{beats}} and {{agents}} currently do not support multiple outputs. It should also be possible to set up a proxy (load balancer) to redirect traffic without {{ls}} in this tutorial. * Setting up a remote cluster on `clusterA` and `clusterB`. * Setting up bi-directional cross-cluster replication with exclusion patterns. diff --git a/deploy-manage/tools/cross-cluster-replication/ccr-getting-started-prerequisites.md b/deploy-manage/tools/cross-cluster-replication/ccr-getting-started-prerequisites.md index 27a0a5d49..31606ae5a 100644 --- a/deploy-manage/tools/cross-cluster-replication/ccr-getting-started-prerequisites.md +++ b/deploy-manage/tools/cross-cluster-replication/ccr-getting-started-prerequisites.md @@ -1,13 +1,13 @@ --- mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/ccr-getting-started-prerequisites.html - + applies_to: deployment: - eck: - ess: - ece: - self: + eck: + ess: + ece: + self: --- # Prerequisites [ccr-getting-started-prerequisites] @@ -17,5 +17,5 @@ To complete this tutorial, you need: * The `manage` cluster privilege on the local cluster. * A license on both clusters that includes {{ccr}}. [Activate a free 30-day trial](../../license/manage-your-license-in-self-managed-cluster.md). * An index on the remote cluster that contains the data you want to replicate. This tutorial uses the sample eCommerce orders data set. [Load sample data](../../../explore-analyze/index.md#gs-get-data-into-kibana). -* In the local cluster, all nodes with the `master` [node role](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md#node-roles) must also have the [`remote_cluster_client`](../../distributed-architecture/clusters-nodes-shards/node-roles.md#remote-node) role. The local cluster must also have at least one node with both a data role and the [`remote_cluster_client`](../../distributed-architecture/clusters-nodes-shards/node-roles.md#remote-node) role. Individual tasks for coordinating replication scale based on the number of data nodes with the `remote_cluster_client` role in the local cluster. +* In the local cluster, all nodes with the `master` [node role](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md#node-roles) must also have the [`remote_cluster_client`](../../distributed-architecture/clusters-nodes-shards/node-roles.md#remote-node) role. The local cluster must also have at least one node with both a data role and the [`remote_cluster_client`](../../distributed-architecture/clusters-nodes-shards/node-roles.md#remote-node) role. Individual tasks for coordinating replication scale based on the number of data nodes with the `remote_cluster_client` role in the local cluster. diff --git a/deploy-manage/tools/cross-cluster-replication/ccr-recreate-follower-index.md b/deploy-manage/tools/cross-cluster-replication/ccr-recreate-follower-index.md index 77cdf96fb..92f9ae5e0 100644 --- a/deploy-manage/tools/cross-cluster-replication/ccr-recreate-follower-index.md +++ b/deploy-manage/tools/cross-cluster-replication/ccr-recreate-follower-index.md @@ -4,21 +4,21 @@ mapped_pages: applies_to: deployment: - eck: - ess: - ece: - self: + eck: + ess: + ece: + self: --- # Recreate a follower index [ccr-recreate-follower-index] -When a document is updated or deleted, the underlying operation is retained in the Lucene index for a period of time defined by the [`index.soft_deletes.retention_lease.period`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#ccr-index-soft-deletes-retention-period) parameter. You configure this setting on the [leader index](../cross-cluster-replication.md#ccr-leader-requirements). +When a document is updated or deleted, the underlying operation is retained in the Lucene index for a period of time defined by the [`index.soft_deletes.retention_lease.period`](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#ccr-index-soft-deletes-retention-period) parameter. You configure this setting on the [leader index](../cross-cluster-replication.md#ccr-leader-requirements). When a follower index starts, it acquires a retention lease from the leader index. This lease informs the leader that it should not allow a soft delete to be pruned until either the follower indicates that it has received the operation, or until the lease expires. If a follower index falls sufficiently behind a leader and cannot replicate operations, {{es}} reports an `indices[].fatal_exception` error. To resolve the issue, recreate the follower index. When the new follow index starts, the [remote recovery](../cross-cluster-replication.md#ccr-remote-recovery) process recopies the Lucene segment files from the leader. -::::{important} +::::{important} Recreating the follower index is a destructive action. All existing Lucene segment files are deleted on the cluster containing the follower index. :::: diff --git a/deploy-manage/tools/cross-cluster-replication/manage-auto-follow-patterns.md b/deploy-manage/tools/cross-cluster-replication/manage-auto-follow-patterns.md index 1c8816cb1..71c49f8ea 100644 --- a/deploy-manage/tools/cross-cluster-replication/manage-auto-follow-patterns.md +++ b/deploy-manage/tools/cross-cluster-replication/manage-auto-follow-patterns.md @@ -4,18 +4,18 @@ mapped_pages: applies_to: deployment: - eck: - ess: - ece: - self: + eck: + ess: + ece: + self: --- # Manage auto-follow patterns [ccr-auto-follow] To replicate time series indices, you configure an auto-follow pattern so that each new index in the series is replicated automatically. Whenever the name of a new index on the remote cluster matches the auto-follow pattern, a corresponding follower index is added to the local cluster. -::::{note} -Auto-follow patterns only match open indices on the remote cluster that have all primary shards started. Auto-follow patterns do not match indices that can’t be used for {{ccr-init}} such as [closed indices](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-open) or [{{search-snaps}}](../snapshot-and-restore/searchable-snapshots.md). Avoid using an auto-follow pattern that matches indices with a [read or write block](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-block.md). These blocks prevent follower indices from replicating such indices. +::::{note} +Auto-follow patterns only match open indices on the remote cluster that have all primary shards started. Auto-follow patterns do not match indices that can’t be used for {{ccr-init}} such as [closed indices](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-open) or [{{search-snaps}}](../snapshot-and-restore/searchable-snapshots.md). Avoid using an auto-follow pattern that matches indices with a [read or write block](elasticsearch://reference/elasticsearch/index-settings/index-block.md). These blocks prevent follower indices from replicating such indices. :::: diff --git a/deploy-manage/tools/snapshot-and-restore.md b/deploy-manage/tools/snapshot-and-restore.md index 598578cf9..c1b50cba5 100644 --- a/deploy-manage/tools/snapshot-and-restore.md +++ b/deploy-manage/tools/snapshot-and-restore.md @@ -8,10 +8,10 @@ mapped_urls: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-snapshots.html applies_to: deployment: - eck: - ess: - ece: - self: + eck: + ess: + ece: + self: --- # Snapshot and restore @@ -67,7 +67,7 @@ Use **Kibana** to manage your snapshots. In Kibana, you can: In **Elastic Cloud Enterprise**, you can also [restore snapshots](snapshot-and-restore/restore-snapshot.md) across clusters. :::: - + ::::{dropdown} Elastic Cloud on Kubernetes (ECK) On Elastic Cloud on Kubernetes, you must manually configure snapshot repositories. The system does not create **Snapshot Lifecycle Management (SLM) policies** or **automatic snapshots** by default. @@ -82,7 +82,7 @@ Snapshots back up only open indices. If you close an index, it is not included i By default, a snapshot of a cluster contains the cluster state, all regular data streams, and all regular indices. The cluster state includes: -- [Persistent cluster settings](/deploy-manage/deploy/self-managed/configure-elasticsearch.md#cluster-setting-types) +- [Persistent cluster settings](/deploy-manage/deploy/self-managed/configure-elasticsearch.md#cluster-setting-types) - [Index templates](/manage-data/data-store/templates.md) - [Legacy index templates](https://www.elastic.co/guide/en/elasticsearch/reference/8.17/indices-templates-v1.html) - [Ingest pipelines](/manage-data/ingest/transform-enrich/ingest-pipelines.md) @@ -107,7 +107,7 @@ A **feature state** contains the indices and data streams used to store configur To retrieve a list of feature states, use the [Features API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-features-get-features). :::: -A feature state typically includes one or more [system indices or system data streams](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#system-indices). It may also include regular indices and data streams used by the feature. For example, a feature state may include a regular index that contains the feature’s execution history. Storing this history in a regular index lets you more easily search it. +A feature state typically includes one or more [system indices or system data streams](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#system-indices). It may also include regular indices and data streams used by the feature. For example, a feature state may include a regular index that contains the feature’s execution history. Storing this history in a regular index lets you more easily search it. In Elasticsearch 8.0 and later versions, feature states are the only way to back up and restore system indices and system data streams. @@ -154,7 +154,7 @@ You can’t restore an index to an earlier version of Elasticsearch. For example A compatible snapshot can contain indices created in an older incompatible version. For example, a snapshot of a 7.17 cluster can contain an index created in 6.8. Restoring the 6.8 index to an 8.17 cluster fails unless you can use the [archive functionality](/deploy-manage/upgrade/deployment-or-cluster/reading-indices-from-older-elasticsearch-versions.md). Keep this in mind if you take a snapshot before upgrading a cluster. -As a workaround, you can first restore the index to another cluster running the latest version of Elasticsearch that’s compatible with both the index and your current cluster. You can then use [reindex-from-remote](https://www.elastic.co/guide/en/elasticsearch/reference/8.17/docs-reindex.html#reindex-from-remote) to rebuild the index on your current cluster. Reindex from remote is only possible if the index’s [`_source`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md) is enabled. +As a workaround, you can first restore the index to another cluster running the latest version of Elasticsearch that’s compatible with both the index and your current cluster. You can then use [reindex-from-remote](https://www.elastic.co/guide/en/elasticsearch/reference/8.17/docs-reindex.html#reindex-from-remote) to rebuild the index on your current cluster. Reindex from remote is only possible if the index’s [`_source`](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md) is enabled. Reindexing from remote can take significantly longer than restoring a snapshot. Before you start, test the reindex from remote process with a subset of the data to estimate your time requirements. diff --git a/deploy-manage/tools/snapshot-and-restore/azure-repository.md b/deploy-manage/tools/snapshot-and-restore/azure-repository.md index 6dd3fe6d1..f193a105f 100644 --- a/deploy-manage/tools/snapshot-and-restore/azure-repository.md +++ b/deploy-manage/tools/snapshot-and-restore/azure-repository.md @@ -3,7 +3,7 @@ mapped_urls: - https://www.elastic.co/guide/en/elasticsearch/reference/current/repository-azure.html applies_to: deployment: - self: + self: --- # Azure repository [repository-azure] @@ -103,7 +103,7 @@ The following list describes the available client settings. Those that must be s : A shared access signatures (SAS) token, which the repository’s internal Azure client uses for authentication. The SAS token must have read (r), write (w), list (l), and delete (d) permissions for the repository base path and all its contents. These permissions must be granted for the blob service (b) and apply to resource types service (s), container (c), and object (o). Alternatively, use `key`. `azure.client.CLIENT_NAME.timeout` -: The client side timeout for any single request to Azure, as a [time unit](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#time-units). For example, a value of `5s` specifies a 5 second timeout. There is no default value, which means that {{es}} uses the [default value](https://azure.github.io/azure-storage-java/com/microsoft/azure/storage/RequestOptions.html#setTimeoutIntervalInMs(java.lang.Integer)) set by the Azure client. +: The client side timeout for any single request to Azure, as a [time unit](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#time-units). For example, a value of `5s` specifies a 5 second timeout. There is no default value, which means that {{es}} uses the [default value](https://azure.github.io/azure-storage-java/com/microsoft/azure/storage/RequestOptions.html#setTimeoutIntervalInMs(java.lang.Integer)) set by the Azure client. `azure.client.CLIENT_NAME.endpoint` : The Azure endpoint to connect to. It must include the protocol used to connect to Azure. @@ -159,16 +159,16 @@ PUT _snapshot/my_backup `chunk_size` -: Big files can be broken down into multiple smaller blobs in the blob store during snapshotting. It is not recommended to change this value from its default unless there is an explicit reason for limiting the size of blobs in the repository. Setting a value lower than the default can result in an increased number of API calls to the Azure blob store during snapshot create as well as restore operations compared to using the default value and thus make both operations slower as well as more costly. Specify the chunk size as a [byte unit](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#byte-units), for example: `10MB`, `5KB`, `500B`. Defaults to the maximum size of a blob in the Azure blob store which is `5TB`. +: Big files can be broken down into multiple smaller blobs in the blob store during snapshotting. It is not recommended to change this value from its default unless there is an explicit reason for limiting the size of blobs in the repository. Setting a value lower than the default can result in an increased number of API calls to the Azure blob store during snapshot create as well as restore operations compared to using the default value and thus make both operations slower as well as more costly. Specify the chunk size as a [byte unit](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#byte-units), for example: `10MB`, `5KB`, `500B`. Defaults to the maximum size of a blob in the Azure blob store which is `5TB`. `compress` : When set to `true` metadata files are stored in compressed format. This setting doesn’t affect index files that are already compressed by default. Defaults to `true`. `max_restore_bytes_per_sec` -: (Optional, [byte value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Maximum snapshot restore rate per node. Defaults to unlimited. Note that restores are also throttled through [recovery settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/index-recovery-settings.md). +: (Optional, [byte value](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Maximum snapshot restore rate per node. Defaults to unlimited. Note that restores are also throttled through [recovery settings](elasticsearch://reference/elasticsearch/configuration-reference/index-recovery-settings.md). `max_snapshot_bytes_per_sec` -: (Optional, [byte value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Maximum snapshot creation rate per node. Defaults to `40mb` per second. Note that if the [recovery settings for managed services](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/index-recovery-settings.md#recovery-settings-for-managed-services) are set, then it defaults to unlimited, and the rate is additionally throttled through [recovery settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/index-recovery-settings.md). +: (Optional, [byte value](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Maximum snapshot creation rate per node. Defaults to `40mb` per second. Note that if the [recovery settings for managed services](elasticsearch://reference/elasticsearch/configuration-reference/index-recovery-settings.md#recovery-settings-for-managed-services) are set, then it defaults to unlimited, and the rate is additionally throttled through [recovery settings](elasticsearch://reference/elasticsearch/configuration-reference/index-recovery-settings.md). `readonly` : (Optional, Boolean) If `true`, the repository is read-only. The cluster can retrieve and restore snapshots from the repository but not write to the repository or create snapshots in it. diff --git a/deploy-manage/tools/snapshot-and-restore/create-snapshots.md b/deploy-manage/tools/snapshot-and-restore/create-snapshots.md index 793d23271..0a1de5d64 100644 --- a/deploy-manage/tools/snapshot-and-restore/create-snapshots.md +++ b/deploy-manage/tools/snapshot-and-restore/create-snapshots.md @@ -33,7 +33,7 @@ The guide also provides tips for creating dedicated cluster state snapshots and * You can only take a snapshot from a running cluster with an elected [master node](../../distributed-architecture/clusters-nodes-shards/node-roles.md#master-node-role). * A snapshot repository must be [registered](self-managed.md) and available to the cluster. -* The cluster’s global metadata must be readable. To include an index in a snapshot, the index and its metadata must also be readable. Ensure there aren’t any [cluster blocks](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/miscellaneous-cluster-settings.md#cluster-read-only) or [index blocks](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-block.md) that prevent read access. +* The cluster’s global metadata must be readable. To include an index in a snapshot, the index and its metadata must also be readable. Ensure there aren’t any [cluster blocks](elasticsearch://reference/elasticsearch/configuration-reference/miscellaneous-cluster-settings.md#cluster-read-only) or [index blocks](elasticsearch://reference/elasticsearch/index-settings/index-block.md) that prevent read access. ## Considerations [create-snapshot-considerations] @@ -43,7 +43,7 @@ The guide also provides tips for creating dedicated cluster state snapshots and * Each snapshot is logically independent. You can delete a snapshot without affecting other snapshots. * Taking a snapshot can temporarily pause shard allocations. See [Snapshots and shard allocation](../snapshot-and-restore.md#snapshots-shard-allocation). * Taking a snapshot doesn’t block indexing or other requests. However, the snapshot won’t include changes made after the snapshot process starts. -* You can take multiple snapshots at the same time. The [`snapshot.max_concurrent_operations`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/snapshot-restore-settings.md#snapshot-max-concurrent-ops) cluster setting limits the maximum number of concurrent snapshot operations. +* You can take multiple snapshots at the same time. The [`snapshot.max_concurrent_operations`](elasticsearch://reference/elasticsearch/configuration-reference/snapshot-restore-settings.md#snapshot-max-concurrent-ops) cluster setting limits the maximum number of concurrent snapshot operations. * If you include a data stream in a snapshot, the snapshot also includes the stream’s backing indices and metadata. You can also include only specific backing indices in a snapshot. However, the snapshot won’t include the data stream’s metadata or its other backing indices. @@ -136,7 +136,7 @@ PUT _slm/policy/nightly-snapshots ``` 1. When to take snapshots, written in [Cron syntax](/explore-analyze/alerts-cases/watcher/schedule-types.md#schedule-cron). -2. Snapshot name. Supports [date math](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#api-date-math-index-names). To prevent naming conflicts, the policy also appends a UUID to each snapshot name. +2. Snapshot name. Supports [date math](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#api-date-math-index-names). To prevent naming conflicts, the policy also appends a UUID to each snapshot name. 3. [Registered snapshot repository](self-managed.md) used to store the policy’s snapshots. 4. Data streams and indices to include in the policy’s snapshots. 5. If `true`, the policy’s snapshots include the cluster state. This also includes all feature states by default. To only include specific feature states, see [Back up a specific feature state](#back-up-specific-feature-state). @@ -159,7 +159,7 @@ The snapshot process runs in the background. To monitor its progress, see [Monit ### {{slm-init}} retention [slm-retention-task] -{{slm-init}} snapshot retention is a cluster-level task that runs separately from a policy’s snapshot schedule. To control when the {{slm-init}} retention task runs, configure the [`slm.retention_schedule`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/snapshot-restore-settings.md#slm-retention-schedule) cluster setting. +{{slm-init}} snapshot retention is a cluster-level task that runs separately from a policy’s snapshot schedule. To control when the {{slm-init}} retention task runs, configure the [`slm.retention_schedule`](elasticsearch://reference/elasticsearch/configuration-reference/snapshot-restore-settings.md#slm-retention-schedule) cluster setting. ```console PUT _cluster/settings @@ -188,7 +188,7 @@ A snapshot repository can safely scale to thousands of snapshots. However, to ma ## Manually create a snapshot [manually-create-snapshot] -To take a snapshot without an {{slm-init}} policy, use the [create snapshot API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-snapshot-create). The snapshot name supports [date math](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#api-date-math-index-names). +To take a snapshot without an {{slm-init}} policy, use the [create snapshot API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-snapshot-create). The snapshot name supports [date math](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#api-date-math-index-names). ```console # PUT _snapshot/my_repository/ diff --git a/deploy-manage/tools/snapshot-and-restore/google-cloud-storage-repository.md b/deploy-manage/tools/snapshot-and-restore/google-cloud-storage-repository.md index ef463441c..29b42f60f 100644 --- a/deploy-manage/tools/snapshot-and-restore/google-cloud-storage-repository.md +++ b/deploy-manage/tools/snapshot-and-restore/google-cloud-storage-repository.md @@ -3,7 +3,7 @@ mapped_urls: - https://www.elastic.co/guide/en/elasticsearch/reference/current/repository-gcs.html applies_to: deployment: - self: + self: --- # Google Cloud Storage repository [repository-gcs] @@ -194,10 +194,10 @@ The following settings are supported: : When set to `true` metadata files are stored in compressed format. This setting doesn’t affect index files that are already compressed by default. Defaults to `true`. `max_restore_bytes_per_sec` -: (Optional, [byte value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Maximum snapshot restore rate per node. Defaults to unlimited. Note that restores are also throttled through [recovery settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/index-recovery-settings.md). +: (Optional, [byte value](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Maximum snapshot restore rate per node. Defaults to unlimited. Note that restores are also throttled through [recovery settings](elasticsearch://reference/elasticsearch/configuration-reference/index-recovery-settings.md). `max_snapshot_bytes_per_sec` -: (Optional, [byte value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Maximum snapshot creation rate per node. Defaults to `40mb` per second. Note that if the [recovery settings for managed services](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/index-recovery-settings.md#recovery-settings-for-managed-services) are set, then it defaults to unlimited, and the rate is additionally throttled through [recovery settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/index-recovery-settings.md). +: (Optional, [byte value](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Maximum snapshot creation rate per node. Defaults to `40mb` per second. Note that if the [recovery settings for managed services](elasticsearch://reference/elasticsearch/configuration-reference/index-recovery-settings.md#recovery-settings-for-managed-services) are set, then it defaults to unlimited, and the rate is additionally throttled through [recovery settings](elasticsearch://reference/elasticsearch/configuration-reference/index-recovery-settings.md). `readonly` : (Optional, Boolean) If `true`, the repository is read-only. The cluster can retrieve and restore snapshots from the repository but not write to the repository or create snapshots in it. diff --git a/deploy-manage/tools/snapshot-and-restore/manage-snapshot-repositories.md b/deploy-manage/tools/snapshot-and-restore/manage-snapshot-repositories.md index 1bd5d6887..e62b8f6de 100644 --- a/deploy-manage/tools/snapshot-and-restore/manage-snapshot-repositories.md +++ b/deploy-manage/tools/snapshot-and-restore/manage-snapshot-repositories.md @@ -1,10 +1,10 @@ --- applies_to: deployment: - eck: - ess: - ece: - self: + eck: + ess: + ece: + self: --- # Manage snapshot repositories @@ -26,13 +26,13 @@ If you manage your own Elasticsearch cluster, you can use the following built-in Other repository types are available through official plugins: -* [Hadoop Distributed File System (HDFS)](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch-plugins/repository-hdfs.md) +* [Hadoop Distributed File System (HDFS)](elasticsearch://reference/elasticsearch-plugins/repository-hdfs.md) ### Elastic Cloud Hosted {{ech}} deployments automatically register a repository named `found-snapshots` in {{es}} clusters. These repositories are used together with the `cloud-snapshot-policy` SLM policy to take periodic snapshots of your {{es}} clusters. You can also use the `found-snapshots` repository for your own [SLM policies](/deploy-manage/tools/snapshot-and-restore/create-snapshots.md#automate-snapshots-slm) or to store searchable snapshots. -The `found-snapshots` repository is specific to each deployment. However, you can restore snapshots from another deployment’s found-snapshots repository if the deployments are under the same account and in the same region. +The `found-snapshots` repository is specific to each deployment. However, you can restore snapshots from another deployment’s found-snapshots repository if the deployments are under the same account and in the same region. Elastic Cloud Hosted deployments also support the following repository types: @@ -57,7 +57,7 @@ Elastic Cloud Enterprise installations support the following Elasticsearch snaps * [Minio](/deploy-manage/tools/snapshot-and-restore/minio-on-premise-repository.md) :::{note} -No repository types other than those listed are supported in the Elastic Cloud Enterprise platform, even if they are supported by Elasticsearch. +No repository types other than those listed are supported in the Elastic Cloud Enterprise platform, even if they are supported by Elasticsearch. ::: For more details, refer to [Managing snapshot repositories in Elastic Cloud Enterprise](/deploy-manage/tools/snapshot-and-restore/cloud-enterprise.md). diff --git a/deploy-manage/tools/snapshot-and-restore/minio-on-premise-repository.md b/deploy-manage/tools/snapshot-and-restore/minio-on-premise-repository.md index 4bc5836a7..69465a429 100644 --- a/deploy-manage/tools/snapshot-and-restore/minio-on-premise-repository.md +++ b/deploy-manage/tools/snapshot-and-restore/minio-on-premise-repository.md @@ -3,7 +3,7 @@ mapped_urls: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configuring-minio.html applies_to: deployment: - ece: + ece: --- # Minio on-premise repository [ece-configuring-minio] @@ -65,7 +65,7 @@ How you create the AWS S3 bucket depends on what version of Elasticsearch you ar * For version 7.x: 1. Using the Minio browser or an S3 client application, create an S3 bucket to store your snapshots. - 2. [Log into the Cloud UI](../../deploy/cloud-enterprise/log-into-cloud-ui.md) and [add the S3 repository plugin](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch-plugins/cloud-enterprise/ece-add-plugins.md) to your cluster. + 2. [Log into the Cloud UI](../../deploy/cloud-enterprise/log-into-cloud-ui.md) and [add the S3 repository plugin](elasticsearch://reference/elasticsearch-plugins/cloud-enterprise/ece-add-plugins.md) to your cluster. * For versions 8.0 and later, {{es}} has built-in support for AWS S3 repositories; no repository plugin is needed. Use the Minio browser or an S3 client application to create an S3 bucket to store your snapshots. diff --git a/deploy-manage/tools/snapshot-and-restore/read-only-url-repository.md b/deploy-manage/tools/snapshot-and-restore/read-only-url-repository.md index fe68a22d8..7eb5d438e 100644 --- a/deploy-manage/tools/snapshot-and-restore/read-only-url-repository.md +++ b/deploy-manage/tools/snapshot-and-restore/read-only-url-repository.md @@ -3,12 +3,12 @@ mapped_urls: - https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-read-only-repository.html applies_to: deployment: - self: + self: --- # Read-only URL repository [snapshots-read-only-repository] -::::{note} +::::{note} This repository type is only available if you run {{es}} on your own hardware. If you use {{ech}}, see [{{ech}} repository types](self-managed.md). :::: @@ -30,13 +30,13 @@ PUT _snapshot/my_read_only_url_repository ## Repository settings [read-only-url-repository-settings] `chunk_size` -: (Optional, [byte value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Maximum size of files in snapshots. In snapshots, files larger than this are broken down into chunks of this size or smaller. Defaults to `null` (unlimited file size). +: (Optional, [byte value](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Maximum size of files in snapshots. In snapshots, files larger than this are broken down into chunks of this size or smaller. Defaults to `null` (unlimited file size). `http_max_retries` : (Optional, integer) Maximum number of retries for `http` and `https` URLs. Defaults to `5`. `http_socket_timeout` -: (Optional, [time value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#time-units)) Maximum wait time for data transfers over a connection. Defaults to `50s`. +: (Optional, [time value](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#time-units)) Maximum wait time for data transfers over a connection. Defaults to `50s`. `compress` : (Optional, Boolean) If `true`, metadata files, such as index mappings and settings, are compressed in snapshots. Data files are not compressed. Defaults to `true`. @@ -45,10 +45,10 @@ PUT _snapshot/my_read_only_url_repository : (Optional, integer) Maximum number of snapshots the repository can contain. Defaults to `Integer.MAX_VALUE`, which is `2^31-1` or `2147483647`. `max_restore_bytes_per_sec` -: (Optional, [byte value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Maximum snapshot restore rate per node. Defaults to unlimited. Note that restores are also throttled through [recovery settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/index-recovery-settings.md). +: (Optional, [byte value](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Maximum snapshot restore rate per node. Defaults to unlimited. Note that restores are also throttled through [recovery settings](elasticsearch://reference/elasticsearch/configuration-reference/index-recovery-settings.md). `max_snapshot_bytes_per_sec` -: (Optional, [byte value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Maximum snapshot creation rate per node. Defaults to `40mb` per second. Note that if the [recovery settings for managed services](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/index-recovery-settings.md#recovery-settings-for-managed-services) are set, then it defaults to unlimited, and the rate is additionally throttled through [recovery settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/index-recovery-settings.md). +: (Optional, [byte value](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Maximum snapshot creation rate per node. Defaults to `40mb` per second. Note that if the [recovery settings for managed services](elasticsearch://reference/elasticsearch/configuration-reference/index-recovery-settings.md#recovery-settings-for-managed-services) are set, then it defaults to unlimited, and the rate is additionally throttled through [recovery settings](elasticsearch://reference/elasticsearch/configuration-reference/index-recovery-settings.md). `url` : (Required, string) URL location of the root of the shared filesystem repository. The following protocols are supported: @@ -59,7 +59,7 @@ PUT _snapshot/my_read_only_url_repository * `https` * `jar` -URLs using the `http`, `https`, or `ftp` protocols must be explicitly allowed with the [`repositories.url.allowed_urls`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/snapshot-restore-settings.md#repositories-url-allowed) cluster setting. This setting supports wildcards in the place of a host, path, query, or fragment in the URL. +URLs using the `http`, `https`, or `ftp` protocols must be explicitly allowed with the [`repositories.url.allowed_urls`](elasticsearch://reference/elasticsearch/configuration-reference/snapshot-restore-settings.md#repositories-url-allowed) cluster setting. This setting supports wildcards in the place of a host, path, query, or fragment in the URL. URLs using the `file` protocol must point to the location of a shared filesystem accessible to all master and data nodes in the cluster. This location must be registered in the `path.repo` setting. You don’t need to register URLs using the `ftp`, `http`, `https`, or `jar` protocols in the `path.repo` setting. diff --git a/deploy-manage/tools/snapshot-and-restore/s3-repository.md b/deploy-manage/tools/snapshot-and-restore/s3-repository.md index 49bbc9f4e..920eebbf6 100644 --- a/deploy-manage/tools/snapshot-and-restore/s3-repository.md +++ b/deploy-manage/tools/snapshot-and-restore/s3-repository.md @@ -108,7 +108,7 @@ The following list contains the available client settings. Those that must be st : The password to connect to the `proxy.host` with. `read_timeout` -: ([time value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#time-units)) The maximum time {{es}} will wait to receive the next byte of data over an established, open connection to the repository before it closes the connection. The default value is 50 seconds. +: ([time value](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#time-units)) The maximum time {{es}} will wait to receive the next byte of data over an established, open connection to the repository before it closes the connection. The default value is 50 seconds. `max_connections` : The maximum number of concurrent connections to S3. The default value is `50`. @@ -174,16 +174,16 @@ The following settings are supported: `chunk_size` -: ([byte value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) The maximum size of object that {{es}} will write to the repository when creating a snapshot. Files which are larger than `chunk_size` will be chunked into several smaller objects. {{es}} may also split a file across multiple objects to satisfy other constraints such as the `max_multipart_parts` limit. Defaults to `5TB` which is the [maximum size of an object in AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html). +: ([byte value](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) The maximum size of object that {{es}} will write to the repository when creating a snapshot. Files which are larger than `chunk_size` will be chunked into several smaller objects. {{es}} may also split a file across multiple objects to satisfy other constraints such as the `max_multipart_parts` limit. Defaults to `5TB` which is the [maximum size of an object in AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html). `compress` : When set to `true` metadata files are stored in compressed format. This setting doesn’t affect index files that are already compressed by default. Defaults to `true`. `max_restore_bytes_per_sec` -: (Optional, [byte value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Maximum snapshot restore rate per node. Defaults to unlimited. Note that restores are also throttled through [recovery settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/index-recovery-settings.md). +: (Optional, [byte value](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Maximum snapshot restore rate per node. Defaults to unlimited. Note that restores are also throttled through [recovery settings](elasticsearch://reference/elasticsearch/configuration-reference/index-recovery-settings.md). `max_snapshot_bytes_per_sec` -: (Optional, [byte value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Maximum snapshot creation rate per node. Defaults to `40mb` per second. Note that if the [recovery settings for managed services](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/index-recovery-settings.md#recovery-settings-for-managed-services) are set, then it defaults to unlimited, and the rate is additionally throttled through [recovery settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/index-recovery-settings.md). +: (Optional, [byte value](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Maximum snapshot creation rate per node. Defaults to `40mb` per second. Note that if the [recovery settings for managed services](elasticsearch://reference/elasticsearch/configuration-reference/index-recovery-settings.md#recovery-settings-for-managed-services) are set, then it defaults to unlimited, and the rate is additionally throttled through [recovery settings](elasticsearch://reference/elasticsearch/configuration-reference/index-recovery-settings.md). `readonly` : (Optional, Boolean) If `true`, the repository is read-only. The cluster can retrieve and restore snapshots from the repository but not write to the repository or create snapshots in it. @@ -202,7 +202,7 @@ The following settings are supported: : When set to `true` files are encrypted on server side using AES256 algorithm. Defaults to `false`. `buffer_size` -: ([byte value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Minimum threshold below which the chunk is uploaded using a single request. Beyond this threshold, the S3 repository will use the [AWS Multipart Upload API](https://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html) to split the chunk into several parts, each of `buffer_size` length, and to upload each part in its own request. Note that setting a buffer size lower than `5mb` is not allowed since it will prevent the use of the Multipart API and may result in upload errors. It is also not possible to set a buffer size greater than `5gb` as it is the maximum upload size allowed by S3. Defaults to `100mb` or `5%` of JVM heap, whichever is smaller. +: ([byte value](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Minimum threshold below which the chunk is uploaded using a single request. Beyond this threshold, the S3 repository will use the [AWS Multipart Upload API](https://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html) to split the chunk into several parts, each of `buffer_size` length, and to upload each part in its own request. Note that setting a buffer size lower than `5mb` is not allowed since it will prevent the use of the Multipart API and may result in upload errors. It is also not possible to set a buffer size greater than `5gb` as it is the maximum upload size allowed by S3. Defaults to `100mb` or `5%` of JVM heap, whichever is smaller. `max_multipart_parts` : (integer) The maximum number of parts that {{es}} will write during a multipart upload of a single object. Files which are larger than `buffer_size × max_multipart_parts` will be chunked into several smaller objects. {{es}} may also split a file across multiple objects to satisfy other constraints such as the `chunk_size` limit. Defaults to `10000` which is the [maximum number of parts in a multipart upload in AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html). @@ -220,16 +220,16 @@ The following settings are supported: : (integer) Sets the maximum number of possibly-dangling multipart uploads to clean up in each batch of snapshot deletions. Defaults to `1000` which is the maximum number supported by the [AWS ListMultipartUploads API](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListMultipartUploads.html). If set to `0`, {{es}} will not attempt to clean up dangling multipart uploads. `throttled_delete_retry.delay_increment` -: ([time value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#time-units)) This value is used as the delay before the first retry and the amount the delay is incremented by on each subsequent retry. Default is 50ms, minimum is 0ms. +: ([time value](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#time-units)) This value is used as the delay before the first retry and the amount the delay is incremented by on each subsequent retry. Default is 50ms, minimum is 0ms. `throttled_delete_retry.maximum_delay` -: ([time value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#time-units)) This is the upper bound on how long the delays between retries will grow to. Default is 5s, minimum is 0ms. +: ([time value](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#time-units)) This is the upper bound on how long the delays between retries will grow to. Default is 5s, minimum is 0ms. `throttled_delete_retry.maximum_number_of_retries` : (integer) Sets the number times to retry a throttled snapshot deletion. Defaults to `10`, minimum value is `0` which will disable retries altogether. Note that if retries are enabled in the Azure client, each of these retries comprises that many client-level retries. `get_register_retry_delay` -: ([time value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#time-units)) Sets the time to wait before trying again if an attempt to read a [linearizable register](#repository-s3-linearizable-registers) fails. Defaults to `5s`. +: ([time value](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#time-units)) Sets the time to wait before trying again if an attempt to read a [linearizable register](#repository-s3-linearizable-registers) fails. Defaults to `5s`. ::::{note} The option of defining client settings in the repository settings as documented below is considered deprecated, and will be removed in a future version. @@ -390,7 +390,7 @@ You can perform some basic checks of the suitability of your storage system usin Most storage systems can be configured to log the details of their interaction with {{es}}. If you are investigating a suspected incompatibility with AWS S3, it is usually simplest to collect these logs and provide them to the supplier of your storage system for further analysis. If the incompatibility is not clear from the logs emitted by the storage system, configure {{es}} to log every request it makes to the S3 API by [setting the logging level](../../monitor/logging-configuration/elasticsearch-log4j-configuration-self-managed.md#configuring-logging-levels) of the `com.amazonaws.request` logger to `DEBUG`. -To prevent leaking sensitive information such as credentials and keys in logs, {{es}} rejects configuring this logger at high verbosity unless [insecure network trace logging](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md#http-rest-request-tracer) is enabled. To do so, you must explicitly enable it on each node by setting the system property `es.insecure_network_trace_enabled` to `true`. +To prevent leaking sensitive information such as credentials and keys in logs, {{es}} rejects configuring this logger at high verbosity unless [insecure network trace logging](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md#http-rest-request-tracer) is enabled. To do so, you must explicitly enable it on each node by setting the system property `es.insecure_network_trace_enabled` to `true`. Once enabled, you can configure the `com.amazonaws.request` logger: @@ -403,7 +403,7 @@ PUT /_cluster/settings } ``` -Collect the Elasticsearch logs covering the time period of the failed analysis from all nodes in your cluster and share them with the supplier of your storage system along with the analysis response so they can use them to determine the problem. See the [AWS Java SDK](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-logging.html) documentation for further information, including details about other loggers that can be used to obtain even more verbose logs. When you have finished collecting the logs needed by your supplier, set the logger settings back to `null` to return to the default logging configuration and disable insecure network trace logging again. See [Logger](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/miscellaneous-cluster-settings.md#cluster-logger) and [Cluster update settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings) for more information. +Collect the Elasticsearch logs covering the time period of the failed analysis from all nodes in your cluster and share them with the supplier of your storage system along with the analysis response so they can use them to determine the problem. See the [AWS Java SDK](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-logging.html) documentation for further information, including details about other loggers that can be used to obtain even more verbose logs. When you have finished collecting the logs needed by your supplier, set the logger settings back to `null` to return to the default logging configuration and disable insecure network trace logging again. See [Logger](elasticsearch://reference/elasticsearch/configuration-reference/miscellaneous-cluster-settings.md#cluster-logger) and [Cluster update settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings) for more information. ## Linearizable register implementation [repository-s3-linearizable-registers] diff --git a/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md b/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md index 500b77ecd..cda64f7fa 100644 --- a/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md +++ b/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md @@ -24,13 +24,13 @@ By default, {{search-snap}} indices have no replicas. The underlying snapshot pr If a node fails and {{search-snap}} shards need to be recovered elsewhere, there is a brief window of time while {{es}} allocates the shards to other nodes where the cluster health will not be `green`. Searches that hit these shards may fail or return partial results until the shards are reallocated to healthy nodes. -You typically manage {{search-snaps}} through {{ilm-init}}. The [searchable snapshots](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md) action automatically converts a regular index into a {{search-snap}} index when it reaches the `cold` or `frozen` phase. You can also make indices in existing snapshots searchable by manually mounting them using the [mount snapshot](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-searchable-snapshots-mount) API. +You typically manage {{search-snaps}} through {{ilm-init}}. The [searchable snapshots](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md) action automatically converts a regular index into a {{search-snap}} index when it reaches the `cold` or `frozen` phase. You can also make indices in existing snapshots searchable by manually mounting them using the [mount snapshot](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-searchable-snapshots-mount) API. To mount an index from a snapshot that contains multiple indices, we recommend creating a [clone](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-snapshot-clone) of the snapshot that contains only the index you want to search, and mounting the clone. You should not delete a snapshot if it has any mounted indices, so creating a clone enables you to manage the lifecycle of the backup snapshot independently of any {{search-snaps}}. If you use {{ilm-init}} to manage your {{search-snaps}} then it will automatically look after cloning the snapshot as needed. You can control the allocation of the shards of {{search-snap}} indices using the same mechanisms as for regular indices. For example, you could use [Index-level shard allocation filtering](../../distributed-architecture/shard-allocation-relocation-recovery/index-level-shard-allocation.md) to restrict {{search-snap}} shards to a subset of your nodes. -The speed of recovery of a {{search-snap}} index is limited by the repository setting `max_restore_bytes_per_sec` and the node setting `indices.recovery.max_bytes_per_sec` just like a normal restore operation. By default `max_restore_bytes_per_sec` is unlimited, but the default for `indices.recovery.max_bytes_per_sec` depends on the configuration of the node. See [Recovery settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/index-recovery-settings.md#recovery-settings). +The speed of recovery of a {{search-snap}} index is limited by the repository setting `max_restore_bytes_per_sec` and the node setting `indices.recovery.max_bytes_per_sec` just like a normal restore operation. By default `max_restore_bytes_per_sec` is unlimited, but the default for `indices.recovery.max_bytes_per_sec` depends on the configuration of the node. See [Recovery settings](elasticsearch://reference/elasticsearch/configuration-reference/index-recovery-settings.md#recovery-settings). We recommend that you [force-merge](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-forcemerge) indices to a single segment per shard before taking a snapshot that will be mounted as a {{search-snap}} index. Each read from a snapshot repository takes time and costs money, and the fewer segments there are the fewer reads are needed to restore the snapshot or to respond to a search. @@ -48,7 +48,7 @@ Use any of the following repository types with searchable snapshots: * [AWS S3](s3-repository.md) * [Google Cloud Storage](google-cloud-storage-repository.md) * [Azure Blob Storage](azure-repository.md) -* [Hadoop Distributed File Store (HDFS)](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch-plugins/repository-hdfs.md) +* [Hadoop Distributed File Store (HDFS)](elasticsearch://reference/elasticsearch-plugins/repository-hdfs.md) * [Shared filesystems](shared-file-system-repository.md) such as NFS * [Read-only HTTP and HTTPS repositories](read-only-url-repository.md) @@ -101,7 +101,7 @@ Manually mounting snapshots captured by an Index Lifecycle Management ({{ilm-ini For optimal results, allow {{ilm-init}} to manage snapshots automatically. -[Learn more about {{ilm-init}} snapshot management](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md). +[Learn more about {{ilm-init}} snapshot management](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md). :::: @@ -109,10 +109,10 @@ For optimal results, allow {{ilm-init}} to manage snapshots automatically. $$$searchable-snapshots-shared-cache$$$ `xpack.searchable.snapshot.shared_cache.size` -: ([Static](../../deploy/self-managed/configure-elasticsearch.md#static-cluster-setting)) Disk space reserved for the shared cache of partially mounted indices. Accepts a percentage of total disk space or an absolute [byte value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#byte-units). Defaults to `90%` of total disk space for dedicated frozen data tier nodes. Otherwise defaults to `0b`. +: ([Static](../../deploy/self-managed/configure-elasticsearch.md#static-cluster-setting)) Disk space reserved for the shared cache of partially mounted indices. Accepts a percentage of total disk space or an absolute [byte value](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#byte-units). Defaults to `90%` of total disk space for dedicated frozen data tier nodes. Otherwise defaults to `0b`. `xpack.searchable.snapshot.shared_cache.size.max_headroom` -: ([Static](../../deploy/self-managed/configure-elasticsearch.md#static-cluster-setting), [byte value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) For dedicated frozen tier nodes, the max headroom to maintain. If `xpack.searchable.snapshot.shared_cache.size` is not explicitly set, this setting defaults to `100GB`. Otherwise it defaults to `-1` (not set). You can only configure this setting if `xpack.searchable.snapshot.shared_cache.size` is set as a percentage. +: ([Static](../../deploy/self-managed/configure-elasticsearch.md#static-cluster-setting), [byte value](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) For dedicated frozen tier nodes, the max headroom to maintain. If `xpack.searchable.snapshot.shared_cache.size` is not explicitly set, this setting defaults to `100GB`. Otherwise it defaults to `-1` (not set). You can only configure this setting if `xpack.searchable.snapshot.shared_cache.size` is set as a percentage. To illustrate how these settings work in concert let us look at two examples when using the default values of the settings on a dedicated frozen node: diff --git a/deploy-manage/tools/snapshot-and-restore/self-managed.md b/deploy-manage/tools/snapshot-and-restore/self-managed.md index c7b3a0392..04bc31ca5 100644 --- a/deploy-manage/tools/snapshot-and-restore/self-managed.md +++ b/deploy-manage/tools/snapshot-and-restore/self-managed.md @@ -4,7 +4,7 @@ mapped_urls: navigation_title: "Self-managed" applies_to: deployment: - self: + self: --- # Manage snapshot repositories in self-managed deployments [snapshots-register-repository] @@ -24,7 +24,7 @@ In this guide, you’ll learn how to: * [Cluster privileges](/deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-cluster): `monitor`, `manage_slm`, `cluster:admin/snapshot`, and `cluster:admin/repository` * [Index privilege](/deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-indices): `all` on the `monitor` index -* To register a snapshot repository, the cluster’s global metadata must be writeable. Ensure there aren’t any [cluster blocks](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/miscellaneous-cluster-settings.md#cluster-read-only) that prevent write access. +* To register a snapshot repository, the cluster’s global metadata must be writeable. Ensure there aren’t any [cluster blocks](elasticsearch://reference/elasticsearch/configuration-reference/miscellaneous-cluster-settings.md#cluster-read-only) that prevent write access. ## Considerations [snapshot-repo-considerations] @@ -60,7 +60,7 @@ If you manage your own {{es}} cluster, you can use the following built-in snapsh $$$snapshots-repository-plugins$$$ Other repository types are available through official plugins: -* [Hadoop Distributed File System (HDFS)](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch-plugins/repository-hdfs.md) +* [Hadoop Distributed File System (HDFS)](elasticsearch://reference/elasticsearch-plugins/repository-hdfs.md) You can also use alternative storage implementations with these repository types, as long as the alternative implementation is fully compatible. For instance, [MinIO](https://minio.io) provides an alternative implementation of the AWS S3 API and you can use MinIO with the [`s3` repository type](s3-repository.md). diff --git a/deploy-manage/tools/snapshot-and-restore/shared-file-system-repository.md b/deploy-manage/tools/snapshot-and-restore/shared-file-system-repository.md index b12908ede..c15e3b26f 100644 --- a/deploy-manage/tools/snapshot-and-restore/shared-file-system-repository.md +++ b/deploy-manage/tools/snapshot-and-restore/shared-file-system-repository.md @@ -1,14 +1,14 @@ ---- +--- mapped_urls: - https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-filesystem-repository.html applies_to: deployment: - self: + self: --- # Shared file system repository [snapshots-filesystem-repository] -::::{note} +::::{note} This repository type is only available if you run {{es}} on your own hardware. See [Manage snapshot repositories](/deploy-manage/tools/snapshot-and-restore/manage-snapshot-repositories.md) for other deployment methods. :::: @@ -139,7 +139,7 @@ PUT _snapshot/my_fs_backup ## Repository settings [filesystem-repository-settings] `chunk_size` -: (Optional, [byte value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Maximum size of files in snapshots. In snapshots, files larger than this are broken down into chunks of this size or smaller. Defaults to `null` (unlimited file size). +: (Optional, [byte value](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Maximum size of files in snapshots. In snapshots, files larger than this are broken down into chunks of this size or smaller. Defaults to `null` (unlimited file size). `compress` : (Optional, Boolean) If `true`, metadata files, such as index mappings and settings, are compressed in snapshots. Data files are not compressed. Defaults to `true`. @@ -151,10 +151,10 @@ PUT _snapshot/my_fs_backup : (Optional, integer) Maximum number of snapshots the repository can contain. Defaults to `Integer.MAX_VALUE`, which is `2^31-1` or `2147483647`. `max_restore_bytes_per_sec` -: (Optional, [byte value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Maximum snapshot restore rate per node. Defaults to unlimited. Note that restores are also throttled through [recovery settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/index-recovery-settings.md). +: (Optional, [byte value](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Maximum snapshot restore rate per node. Defaults to unlimited. Note that restores are also throttled through [recovery settings](elasticsearch://reference/elasticsearch/configuration-reference/index-recovery-settings.md). `max_snapshot_bytes_per_sec` -: (Optional, [byte value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Maximum snapshot creation rate per node. Defaults to `40mb` per second. Note that if the [recovery settings for managed services](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/index-recovery-settings.md#recovery-settings-for-managed-services) are set, then it defaults to unlimited, and the rate is additionally throttled through [recovery settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/index-recovery-settings.md). +: (Optional, [byte value](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Maximum snapshot creation rate per node. Defaults to `40mb` per second. Note that if the [recovery settings for managed services](elasticsearch://reference/elasticsearch/configuration-reference/index-recovery-settings.md#recovery-settings-for-managed-services) are set, then it defaults to unlimited, and the rate is additionally throttled through [recovery settings](elasticsearch://reference/elasticsearch/configuration-reference/index-recovery-settings.md). `readonly` : (Optional, Boolean) If `true`, the repository is read-only. The cluster can retrieve and restore snapshots from the repository but not write to the repository or create snapshots in it. @@ -163,7 +163,7 @@ PUT _snapshot/my_fs_backup If `false`, the cluster can write to the repository and create snapshots in it. Defaults to `false`. - ::::{important} + ::::{important} If you register the same snapshot repository with multiple clusters, only one cluster should have write access to the repository. Having multiple clusters write to the repository at the same time risks corrupting the contents of the repository. :::: @@ -182,7 +182,7 @@ If the verify repository or repository analysis APIs fail with an error indicati The verify repository and repository analysis APIs will also fail if the operating system returns any other kind of I/O error when accessing the repository. If this happens, address the cause of the I/O error reported by the operating system. -::::{tip} +::::{tip} Many NFS implementations match accounts across nodes using their *numeric* user IDs (UIDs) and group IDs (GIDs) rather than their names. It is possible for {{es}} to run under an account with the same name (often `elasticsearch`) on each node, but for these accounts to have different numeric user or group IDs. If your shared file system uses NFS then ensure that every node is running with the same numeric UID and GID, or else update your NFS configuration to account for the variance in numeric IDs across nodes. :::: diff --git a/deploy-manage/tools/snapshot-and-restore/source-only-repository.md b/deploy-manage/tools/snapshot-and-restore/source-only-repository.md index 51babae2a..523d63bad 100644 --- a/deploy-manage/tools/snapshot-and-restore/source-only-repository.md +++ b/deploy-manage/tools/snapshot-and-restore/source-only-repository.md @@ -3,7 +3,7 @@ mapped_urls: - https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-source-only-repository.html applies_to: deployment: - self: + self: --- # Source-only repository [snapshots-source-only-repository] @@ -14,7 +14,7 @@ Unlike other repository types, a source-only repository doesn’t directly store When you take a snapshot using a source-only repository, {{es}} creates a source-only snapshot in the delegated storage repository. This snapshot only contains stored fields and metadata. It doesn’t include index or doc values structures and isn’t immediately searchable when restored. To search the restored data, you first have to [reindex](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex) it into a new data stream or index. -::::{important} +::::{important} Source-only snapshots are only supported if the `_source` field is enabled and no source-filtering is applied. As a result, indices adopting synthetic source cannot be restored. When you restore a source-only snapshot: * The restored index is read-only and can only serve `match_all` search or scroll requests to enable reindexing. @@ -41,7 +41,7 @@ PUT _snapshot/my_src_only_repository ## Repository settings [source-only-repository-settings] `chunk_size` -: (Optional, [byte value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Maximum size of files in snapshots. In snapshots, files larger than this are broken down into chunks of this size or smaller. Defaults to `null` (unlimited file size). +: (Optional, [byte value](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Maximum size of files in snapshots. In snapshots, files larger than this are broken down into chunks of this size or smaller. Defaults to `null` (unlimited file size). `compress` : (Optional, Boolean) If `true`, metadata files, such as index mappings and settings, are compressed in snapshots. Data files are not compressed. Defaults to `true`. @@ -56,10 +56,10 @@ PUT _snapshot/my_src_only_repository : (Optional, integer) Maximum number of snapshots the repository can contain. Defaults to `Integer.MAX_VALUE`, which is `2^31-1` or `2147483647`. `max_restore_bytes_per_sec` -: (Optional, [byte value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Maximum snapshot restore rate per node. Defaults to unlimited. Note that restores are also throttled through [recovery settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/index-recovery-settings.md). +: (Optional, [byte value](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Maximum snapshot restore rate per node. Defaults to unlimited. Note that restores are also throttled through [recovery settings](elasticsearch://reference/elasticsearch/configuration-reference/index-recovery-settings.md). `max_snapshot_bytes_per_sec` -: (Optional, [byte value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Maximum snapshot creation rate per node. Defaults to `40mb` per second. Note that if the [recovery settings for managed services](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/index-recovery-settings.md#recovery-settings-for-managed-services) are set, then it defaults to unlimited, and the rate is additionally throttled through [recovery settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/index-recovery-settings.md). +: (Optional, [byte value](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#byte-units)) Maximum snapshot creation rate per node. Defaults to `40mb` per second. Note that if the [recovery settings for managed services](elasticsearch://reference/elasticsearch/configuration-reference/index-recovery-settings.md#recovery-settings-for-managed-services) are set, then it defaults to unlimited, and the rate is additionally throttled through [recovery settings](elasticsearch://reference/elasticsearch/configuration-reference/index-recovery-settings.md). `readonly` : (Optional, Boolean) If `true`, the repository is read-only. The cluster can retrieve and restore snapshots from the repository but not write to the repository or create snapshots in it. @@ -68,7 +68,7 @@ PUT _snapshot/my_src_only_repository If `false`, the cluster can write to the repository and create snapshots in it. Defaults to `false`. - ::::{important} + ::::{important} If you register the same snapshot repository with multiple clusters, only one cluster should have write access to the repository. Having multiple clusters write to the repository at the same time risks corrupting the contents of the repository. :::: diff --git a/deploy-manage/upgrade/deployment-or-cluster/reading-indices-from-older-elasticsearch-versions.md b/deploy-manage/upgrade/deployment-or-cluster/reading-indices-from-older-elasticsearch-versions.md index 4f5fa6dd5..f434b982f 100644 --- a/deploy-manage/upgrade/deployment-or-cluster/reading-indices-from-older-elasticsearch-versions.md +++ b/deploy-manage/upgrade/deployment-or-cluster/reading-indices-from-older-elasticsearch-versions.md @@ -12,30 +12,30 @@ The archive functionality provides slower read-only access to older {{es}} data, For this, {{es}} has the ability to access older snapshot repositories (going back to version 5). The legacy indices in the [snapshot repository](../../tools/snapshot-and-restore.md) can either be [restored](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-snapshot-restore), or can be directly accessed via [searchable snapshots](../../tools/snapshot-and-restore/searchable-snapshots.md) so that the archived data won’t even need to fully reside on local disks for access. -## Supported field types [archive-indices-supported-field-types] +## Supported field types [archive-indices-supported-field-types] Old mappings are imported as much "as-is" as possible into {{es}} 8, but only provide regular query capabilities on a select subset of fields: -* [Numeric types](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/number.md) -* [`boolean` type](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/boolean.md) -* [`ip` type](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/ip.md) -* [`geo_point` type](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/geo-point.md) -* [`date` types](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/date.md): the date `format` setting on date fields is supported as long as it behaves similarly across these versions. In case it is not, for example [when using custom date formats](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/migrate-to-java-time.html), this field can be updated on legacy indices so that it can be changed by a user if need be. -* [`keyword` type](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/keyword.md#keyword-field-type): the `normalizer` setting on keyword fields is supported as long as it behaves similarly across these versions. In case it is not, this field can be updated on legacy indices if need be. -* [`text` type](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/text.md#text-field-type): scoring capabilities are limited, and all queries return constant scores that are equal to 1.0. The `analyzer` settings on text fields are supported as long as they behave similarly across these versions. In case they do not, they can be updated on legacy indices if need be. -* [Multi-fields](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/multi-fields.md) -* [Field aliases](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/field-alias.md) -* [`object`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/object.md) fields +* [Numeric types](elasticsearch://reference/elasticsearch/mapping-reference/number.md) +* [`boolean` type](elasticsearch://reference/elasticsearch/mapping-reference/boolean.md) +* [`ip` type](elasticsearch://reference/elasticsearch/mapping-reference/ip.md) +* [`geo_point` type](elasticsearch://reference/elasticsearch/mapping-reference/geo-point.md) +* [`date` types](elasticsearch://reference/elasticsearch/mapping-reference/date.md): the date `format` setting on date fields is supported as long as it behaves similarly across these versions. In case it is not, for example [when using custom date formats](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/migrate-to-java-time.html), this field can be updated on legacy indices so that it can be changed by a user if need be. +* [`keyword` type](elasticsearch://reference/elasticsearch/mapping-reference/keyword.md#keyword-field-type): the `normalizer` setting on keyword fields is supported as long as it behaves similarly across these versions. In case it is not, this field can be updated on legacy indices if need be. +* [`text` type](elasticsearch://reference/elasticsearch/mapping-reference/text.md#text-field-type): scoring capabilities are limited, and all queries return constant scores that are equal to 1.0. The `analyzer` settings on text fields are supported as long as they behave similarly across these versions. In case they do not, they can be updated on legacy indices if need be. +* [Multi-fields](elasticsearch://reference/elasticsearch/mapping-reference/multi-fields.md) +* [Field aliases](elasticsearch://reference/elasticsearch/mapping-reference/field-alias.md) +* [`object`](elasticsearch://reference/elasticsearch/mapping-reference/object.md) fields * some basic metadata fields, e.g. `_type` for querying {{es}} 5 indices * [runtime fields](../../../manage-data/data-store/mapping/map-runtime-field.md) -* [`_source` field](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md) +* [`_source` field](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md) {{es}} 5 indices with mappings that have [multiple mapping types](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/removal-of-types.html) are collapsed together on a best-effort basis before they are imported. -In case the auto-import of mappings does not work, or the new {{es}} version can’t make sense of the mapping, it falls back to importing the index without the mapping, but stores the original mapping in the [_meta](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-meta-field.md) section of the imported index. The legacy mapping can then be introspected using the [GET mapping](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-mapping) API and an updated mapping can be manually put in place using the [update mapping](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-mapping) API, copying and adapting relevant sections of the legacy mapping to work with the current {{es}} version. While auto-import is expected to work in most cases, failures of doing so should be [raised](https://github.com/elastic/elasticsearch/issues/new/choose) with the Elastic team for future improvements. +In case the auto-import of mappings does not work, or the new {{es}} version can’t make sense of the mapping, it falls back to importing the index without the mapping, but stores the original mapping in the [_meta](elasticsearch://reference/elasticsearch/mapping-reference/mapping-meta-field.md) section of the imported index. The legacy mapping can then be introspected using the [GET mapping](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-mapping) API and an updated mapping can be manually put in place using the [update mapping](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-mapping) API, copying and adapting relevant sections of the legacy mapping to work with the current {{es}} version. While auto-import is expected to work in most cases, failures of doing so should be [raised](https://github.com/elastic/elasticsearch/issues/new/choose) with the Elastic team for future improvements. -## Supported APIs [_supported_apis] +## Supported APIs [_supported_apis] Archive indices are read-only, and provide data access via the [search](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search) and [field capabilities](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-field-caps) APIs. They do not support the [Get API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-get) nor any write APIs. @@ -44,7 +44,7 @@ Archive indices allow running queries as well as aggregations in so far as they Due to `_source` access the data can also be [reindexed](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex) to a new index that has full compatibility with the current {{es}} version. -## How to upgrade older {{es}} 5 or 6 clusters? [_how_to_upgrade_older_es_5_or_6_clusters] +## How to upgrade older {{es}} 5 or 6 clusters? [_how_to_upgrade_older_es_5_or_6_clusters] Take a snapshot of the indices in the old cluster, delete indices that are not directly supported by ES 8 (i.e. indices older than 7.0), upgrade the cluster without the old indices, and then [restore](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-snapshot-restore) the legacy indices from the snapshot or [mount](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-searchable-snapshots-mount) them via searchable snapshots. diff --git a/deploy-manage/upgrade/prepare-to-upgrade/index-compatibility.md b/deploy-manage/upgrade/prepare-to-upgrade/index-compatibility.md index 02ec9af77..682782ada 100644 --- a/deploy-manage/upgrade/prepare-to-upgrade/index-compatibility.md +++ b/deploy-manage/upgrade/prepare-to-upgrade/index-compatibility.md @@ -29,7 +29,7 @@ To upgrade to 9.0.0-beta1 from 7.16 or an earlier version, **you must first upgr ## REST API compatibility [upgrade-rest-api-compatibility] -[REST API compatibility](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/compatibility.md) is a per-request opt-in feature that can help REST clients mitigate non-compatible (breaking) changes to the REST API. +[REST API compatibility](elasticsearch://reference/elasticsearch/rest-apis/compatibility.md) is a per-request opt-in feature that can help REST clients mitigate non-compatible (breaking) changes to the REST API. ## FIPS Compliance and Java 17 [upgrade-fips-java17] diff --git a/deploy-manage/users-roles/cluster-or-deployment-auth/authorization-plugins.md b/deploy-manage/users-roles/cluster-or-deployment-auth/authorization-plugins.md index 02489805b..d4c41fd44 100644 --- a/deploy-manage/users-roles/cluster-or-deployment-auth/authorization-plugins.md +++ b/deploy-manage/users-roles/cluster-or-deployment-auth/authorization-plugins.md @@ -66,7 +66,7 @@ To register the security extension for your custom roles provider or authorizati 1. Implement a plugin class that extends `org.elasticsearch.plugins.Plugin` 2. Create a build configuration file for the plugin; Gradle is our recommendation. -3. Create a `plugin-descriptor.properties` file as described in [Help for plugin authors](asciidocalypse://docs/elasticsearch/docs/extend/index.md). +3. Create a `plugin-descriptor.properties` file as described in [Help for plugin authors](elasticsearch://extend/index.md). 4. Create a `META-INF/services/org.elasticsearch.xpack.core.security.SecurityExtension` descriptor file for the extension that contains the fully qualified class name of your `org.elasticsearch.xpack.core.security.SecurityExtension` implementation 5. Bundle all in a single zip file. diff --git a/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md b/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md index 9f9b2c338..f60ceb156 100644 --- a/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md +++ b/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md @@ -88,7 +88,7 @@ $$$built-in-roles-logstash-admin$$$ `logstash_admin` : Grants access to the `.logstash*` indices for managing configurations, and grants necessary access for logstash-specific APIs exposed by the logstash x-pack plugin. $$$built-in-roles-logstash-system$$$ `logstash_system` -: Grants access necessary for the Logstash system user to send system-level data (such as monitoring) to {{es}}. For more information, see [Configuring Security in Logstash](asciidocalypse://docs/logstash/docs/reference/secure-connection.md). +: Grants access necessary for the Logstash system user to send system-level data (such as monitoring) to {{es}}. For more information, see [Configuring Security in Logstash](logstash://reference/secure-connection.md). ::::{note} * This role should not be assigned to users as the granted permissions may change between releases. diff --git a/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-sm.md b/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-sm.md index 9c1f1b72d..4dcfabf8c 100644 --- a/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-sm.md +++ b/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-sm.md @@ -11,11 +11,11 @@ navigation_title: Change passwords After you implement security, you might need or want to change passwords for different users. If you want to reset a password for a [built-in user](/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-users.md) such as the `elastic` or `kibana_system` users, or a user in the [native realm](/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-users.md), you can use the following tools: * The **Manage users** UI in {{kib}} -* The [`elasticsearch-reset-password`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/reset-password.md) tool +* The [`elasticsearch-reset-password`](elasticsearch://reference/elasticsearch/command-line-tools/reset-password.md) tool * The [change passwords API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-change-password) :::{{tip}} -This topic describes resetting passwords after the initial bootstrap password is reset. To learn about the users that are used to communicate between {{stack}} components, and about managing bootstrap passwords for built-in users, refer to [](/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-users.md). +This topic describes resetting passwords after the initial bootstrap password is reset. To learn about the users that are used to communicate between {{stack}} components, and about managing bootstrap passwords for built-in users, refer to [](/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-users.md). ::: ## Using {{kib}} @@ -47,7 +47,7 @@ POST /_security/user/user1/_password ## Using the `user` API [native-users-api] -You can manage users through the Elasticsearch `user` API. +You can manage users through the Elasticsearch `user` API. For example, you can change a user's password: diff --git a/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-users.md b/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-users.md index b9d1ce34c..3206b0481 100644 --- a/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-users.md +++ b/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-users.md @@ -16,7 +16,7 @@ The built-in users serve specific purposes and are not intended for general use. :::: -::::{note} +::::{note} On {{ecloud}}, [operator privileges](/deploy-manage/users-roles/cluster-or-deployment-auth/operator-privileges.md) are enabled. These privileges restrict some infrastructure functionality, even if a role would otherwise permit a user to complete an administrative task. :::: @@ -45,14 +45,14 @@ The following built-in users are available: : The user {{metricbeat}} uses when collecting and storing monitoring information in {{es}}. It has the `remote_monitoring_agent` and `remote_monitoring_collector` built-in roles. -## How the built-in users work [built-in-user-explanation] +## How the built-in users work [built-in-user-explanation] These built-in users are stored in a special `.security` index, which is managed by {{es}}. If a built-in user is disabled or its password changes, the change is automatically reflected on each node in the cluster. If your `.security` index is deleted or restored from a snapshot, however, any changes you have applied are lost. Although they share the same API, the built-in users are separate and distinct from users managed by the [native realm](/deploy-manage/users-roles/cluster-or-deployment-auth/native.md). Disabling the native realm will not have any effect on the built-in users. The built-in users can be disabled individually, using the [disable users API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-disable-user). -## The Elastic bootstrap password [bootstrap-elastic-passwords] +## The Elastic bootstrap password [bootstrap-elastic-passwords] ```{{applies_to}} deployment: self: @@ -66,11 +66,11 @@ When you install {{es}}, if the `elastic` user does not already have a password, By default, the bootstrap password is derived from a randomized `keystore.seed` setting, which is added to the keystore during installation. You do not need to know or change this bootstrap password. If you have defined a `bootstrap.password` setting in the keystore, however, that value is used instead. For more information about interacting with the keystore, see [Secure settings](/deploy-manage/security/secure-settings.md). -::::{note} +::::{note} After you [set passwords for the built-in users](/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-users.md#set-built-in-user-passwords), in particular for the `elastic` user, there is no further use for the bootstrap password. :::: -## Setting initial built-in user passwords [set-built-in-user-passwords] +## Setting initial built-in user passwords [set-built-in-user-passwords] ```{{applies_to}} deployment: self: @@ -78,7 +78,7 @@ deployment: You must set the passwords for all built-in users. You can set or reset passwords using several methods. -* Using `elasticsearch-setup-passwords` +* Using `elasticsearch-setup-passwords` * Using {{kib}} user management * Using the change password API @@ -96,15 +96,15 @@ The `elasticsearch-setup-passwords` tool is the simplest method to set the built bin/elasticsearch-setup-passwords interactive ``` -For more information about the command options, see [elasticsearch-setup-passwords](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/setup-passwords.md). +For more information about the command options, see [elasticsearch-setup-passwords](elasticsearch://reference/elasticsearch/command-line-tools/setup-passwords.md). -::::{important} +::::{important} After you set a password for the `elastic` user, the bootstrap password is no longer valid; you cannot run the `elasticsearch-setup-passwords` command a second time. :::: ### Using {{kib}} user management or the change password API -You can set the initial passwords for the built-in users by using the **Management > Users** page in {{kib}} or the [change password API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-change-password). +You can set the initial passwords for the built-in users by using the **Management > Users** page in {{kib}} or the [change password API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-change-password). To use these methods, you must supply the `elastic` user and its bootstrap password to log in to {{kib}} or run the API. This requirement means that you can't use the default bootstrap password that is derived from the `keystore.seed` setting. Instead, you must explicitly set a `bootstrap.password` setting in the keystore before you start {{es}}. For example, the following command prompts you to enter a new bootstrap password: @@ -112,13 +112,13 @@ To use these methods, you must supply the `elastic` user and its bootstrap passw bin/elasticsearch-keystore add "bootstrap.password" ``` -You can then start {{es}} and {{kib}} and use the `elastic` user and bootstrap password to log in to {{kib}} and change the passwords. +You can then start {{es}} and {{kib}} and use the `elastic` user and bootstrap password to log in to {{kib}} and change the passwords. ### Using the Change Password API Alternatively, you can submit Change Password API requests for each built-in user. These methods are better suited for changing your passwords after the initial setup is complete, since at that point the bootstrap password is no longer required. -## Adding built-in user passwords to {{kib}} [add-built-in-user-kibana] +## Adding built-in user passwords to {{kib}} [add-built-in-user-kibana] After the `kibana_system` user password is set, you need to update the {{kib}} server with the new password by setting `elasticsearch.password` in the `kibana.yml` configuration file: @@ -129,7 +129,7 @@ elasticsearch.password: kibanapassword See [Configuring security in {{kib}}](/deploy-manage/security.md). -## Adding built-in user passwords to {{ls}} [add-built-in-user-logstash] +## Adding built-in user passwords to {{ls}} [add-built-in-user-logstash] The `logstash_system` user is used internally within Logstash when monitoring is enabled for Logstash. @@ -145,10 +145,10 @@ If you have upgraded from an older version of {{es}}, the `logstash_system` user PUT _security/user/logstash_system/_enable ``` -See [Configuring credentials for {{ls}} monitoring](asciidocalypse://docs/logstash/docs/reference/ingestion-tools/logstash/secure-connection.md#ls-monitoring-user). +See [Configuring credentials for {{ls}} monitoring](logstash://reference/secure-connection.md#ls-monitoring-user). -## Adding built-in user passwords to Beats [add-built-in-user-beats] +## Adding built-in user passwords to Beats [add-built-in-user-beats] The `beats_system` user is used internally within Beats when monitoring is enabled for Beats. @@ -166,7 +166,7 @@ The `remote_monitoring_user` is used when {{metricbeat}} collects and stores mon If you have upgraded from an older version of {{es}}, then you may not have set a password for the `beats_system` or `remote_monitoring_user` users. If this is the case, then you should use the **Management > Users** page in {{kib}} or the [change password API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-change-password) to set a password for these users. -## Adding built-in user passwords to APM [add-built-in-user-apm] +## Adding built-in user passwords to APM [add-built-in-user-apm] The `apm_system` user is used internally within APM when monitoring is enabled. @@ -180,9 +180,9 @@ xpack.monitoring.elasticsearch.password: apmserverpassword If you have upgraded from an older version of {{es}}, then you may not have set a password for the `apm_system` user. If this is the case, then you should use the **Management > Users** page in {{kib}} or the [change password API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-change-password) to set a password for these users. -## Disabling default password functionality [disabling-default-password] +## Disabling default password functionality [disabling-default-password] -::::{important} +::::{important} This setting is deprecated. The elastic user no longer has a default password. The password must be set before the user can be used. See [The Elastic bootstrap password](/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-users.md#bootstrap-elastic-passwords). :::: diff --git a/deploy-manage/users-roles/cluster-or-deployment-auth/controlling-access-at-document-field-level.md b/deploy-manage/users-roles/cluster-or-deployment-auth/controlling-access-at-document-field-level.md index cd8a9f39c..5c52dbfa1 100644 --- a/deploy-manage/users-roles/cluster-or-deployment-auth/controlling-access-at-document-field-level.md +++ b/deploy-manage/users-roles/cluster-or-deployment-auth/controlling-access-at-document-field-level.md @@ -20,15 +20,15 @@ You can control access to data within a data stream or index by adding field and **Document level security** restricts the documents that users have read access to. In particular, it restricts which documents can be accessed from document-based read APIs. -::::{note} +::::{note} Document and field level security is currently meant to operate with read-only privileged accounts. Users with document and field level security enabled for a data stream or index should not perform write operations. :::: A role can define both field and document level permissions on a per-index basis. A role that doesn’t specify field level permissions grants access to ALL fields. Similarly, a role that doesn’t specify document level permissions grants access to ALL documents in the index. -On this page, you'll learn how to implement [document level security](#document-level-security) and [field level security](#field-level-security). +On this page, you'll learn how to implement [document level security](#document-level-security) and [field level security](#field-level-security). -You'll also learn the following: +You'll also learn the following: * How [multiple roles with document and field level security](#multiple-roles-dls-fls) interact * Considerations for using document and field level security when [searching across clusters using cross-cluster API keys](#ccx-apikeys-dls-fls). @@ -36,7 +36,7 @@ The examples on this page use the [Role management API](https://www.elastic.co/d :::{{admonition}} Document and field level security in {{serverless-full}} -This topic explains how to apply document and field level security in {{stack}}. You can also apply document and field level security in {{serverless-full}} projects. +This topic explains how to apply document and field level security in {{stack}}. You can also apply document and field level security in {{serverless-full}} projects. In {{serverless-full}}, you can only manage document and field level security using the {{ecloud}} console. However, document level security is still managed using queries, and you can use the queries on this page as a guideline. @@ -54,7 +54,7 @@ The specified document query: * Accepts queries written as either string values or nested JSON * Supports the majority of the {{es}} [Query DSL](/explore-analyze/query-filter/languages/querydsl.md), with [some limitations](../../../deploy-manage/security.md#field-document-limitations) for field and document level security -::::{important} +::::{important} Omitting the `query` parameter entirely disables document level security for the respective indices permission entry. :::: @@ -188,15 +188,15 @@ POST /_security/role/example3 ### Pre-processing documents to add security details [set-security-user-processor] -To guarantee that a user reads only their own documents, it makes sense to set up document level security. In this scenario, each document must have the username or role name associated with it, so that this information can be used by the role query for document level security. This is a situation where the [set security user processor](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/ingest-node-set-security-user-processor.md) ingest processor can help. +To guarantee that a user reads only their own documents, it makes sense to set up document level security. In this scenario, each document must have the username or role name associated with it, so that this information can be used by the role query for document level security. This is a situation where the [set security user processor](elasticsearch://reference/ingestion-tools/enrich-processor/ingest-node-set-security-user-processor.md) ingest processor can help. -::::{note} +::::{note} Document level security doesn’t apply to write APIs. You must use unique ids for each user that uses the same data stream or index, otherwise they might overwrite other users' documents. The ingest processor just adds properties for the current authenticated user to the documents that are being indexed. :::: -The [set security user processor](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/ingest-node-set-security-user-processor.md) attaches user-related details (such as `username`, `roles`, `email`, `full_name` and `metadata` ) from the current authenticated user to the current document by pre-processing the ingest. When you index data with an ingest pipeline, user details are automatically attached to the document. If the authenticating credential is an API key, the API key `id`, `name` and `metadata` (if it exists and is non-empty) are also attached to the document. +The [set security user processor](elasticsearch://reference/ingestion-tools/enrich-processor/ingest-node-set-security-user-processor.md) attaches user-related details (such as `username`, `roles`, `email`, `full_name` and `metadata` ) from the current authenticated user to the current document by pre-processing the ingest. When you index data with an ingest pipeline, user details are automatically attached to the document. If the authenticating credential is an API key, the API key `id`, `name` and `metadata` (if it exists and is non-empty) are also attached to the document. -For more information, see [Ingest pipelines](/manage-data/ingest/transform-enrich/ingest-pipelines.md) and [Set security user](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/ingest-node-set-security-user-processor.md). +For more information, see [Ingest pipelines](/manage-data/ingest/transform-enrich/ingest-pipelines.md) and [Set security user](elasticsearch://reference/ingestion-tools/enrich-processor/ingest-node-set-security-user-processor.md). ## Field level security [field-level-security] @@ -222,7 +222,7 @@ POST /_security/role/test_role1 Access to the following metadata fields is always allowed: `_id`, `_type`, `_parent`, `_routing`, `_timestamp`, `_ttl`, `_size` and `_index`. If you specify an empty list of fields, only these metadata fields are accessible. -::::{note} +::::{note} Omitting the fields entry entirely disables field level security. :::: @@ -387,13 +387,13 @@ The resulting permission is equal to: } ``` -::::{note} -Field-level security should not be set on [`alias`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/field-alias.md) fields. To secure a concrete field, its field name must be used directly. +::::{note} +Field-level security should not be set on [`alias`](elasticsearch://reference/elasticsearch/mapping-reference/field-alias.md) fields. To secure a concrete field, its field name must be used directly. :::: ## Multiple roles with document and field level security [multiple-roles-dls-fls] -A user can have many roles and each role can define different permissions on the same data stream or index. When assigning users multiple roles, be careful that you don’t inadvertently grant wider access than intended. +A user can have many roles and each role can define different permissions on the same data stream or index. When assigning users multiple roles, be careful that you don’t inadvertently grant wider access than intended. **Document level security** takes into account each role held by the user and combines each document level security query for a given data stream or index with an "OR". This means that only one of the role queries must match for a document to be returned. For example, if a role grants access to an index without document level security and another grants access with document level security, document level security is not applied; the user with both roles has access to all of the documents in the index. @@ -401,7 +401,7 @@ A user can have many roles and each role can define different permissions on the For example, let’s say `role_a` grants access to only the `address` field of the documents in `index1`; it doesn’t specify any document restrictions. Conversely, `role_b` limits access to a subset of the documents in `index1`; it doesn’t specify any field restrictions. If you assign a user both roles, `role_a` gives the user access to all documents and `role_b` gives the user access to all fields. -::::{important} +::::{important} If you need to restrict access to both documents and fields, consider splitting documents by index instead. :::: diff --git a/deploy-manage/users-roles/cluster-or-deployment-auth/controlling-user-cache.md b/deploy-manage/users-roles/cluster-or-deployment-auth/controlling-user-cache.md index 37955b46e..6f7029fe9 100644 --- a/deploy-manage/users-roles/cluster-or-deployment-auth/controlling-user-cache.md +++ b/deploy-manage/users-roles/cluster-or-deployment-auth/controlling-user-cache.md @@ -3,8 +3,8 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/controlling-user-cache.html applies_to: deployment: - ess: - ece: + ess: + ece: eck: self: --- @@ -13,17 +13,17 @@ applies_to: User credentials are cached in memory on each node to avoid connecting to a remote authentication service or hitting the disk for every incoming request. You can configure characteristics of the user cache with the `cache.ttl`, `cache.max_users`, and `cache.hash_algo` realm settings. -::::{note} +::::{note} JWT realms use `jwt.cache.ttl` and `jwt.cache.size` realm settings. :::: -::::{note} +::::{note} PKI and JWT realms do not cache user credentials, but do cache the resolved user object to avoid unnecessarily needing to perform role mapping on each request. :::: -The cached user credentials are hashed in memory. By default, the {{es}} {{security-features}} use a salted `sha-256` hash algorithm. You can use a different hashing algorithm by setting the `cache.hash_algo` realm settings. See [User cache and password hash algorithms](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#hashing-settings). +The cached user credentials are hashed in memory. By default, the {{es}} {{security-features}} use a salted `sha-256` hash algorithm. You can use a different hashing algorithm by setting the `cache.hash_algo` realm settings. See [User cache and password hash algorithms](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#hashing-settings). ## Evicting users from the cache [cache-eviction-api] diff --git a/deploy-manage/users-roles/cluster-or-deployment-auth/jwt.md b/deploy-manage/users-roles/cluster-or-deployment-auth/jwt.md index 0c2ebd3e3..3202b6888 100644 --- a/deploy-manage/users-roles/cluster-or-deployment-auth/jwt.md +++ b/deploy-manage/users-roles/cluster-or-deployment-auth/jwt.md @@ -72,13 +72,13 @@ Not every access token is formatted as a JSON Web Token (JWT). For it to be comp To use JWT authentication, create the realm in the `elasticsearch.yml` file to configure it within the {{es}} authentication chain. -The JWT realm has a few mandatory settings, plus optional settings that are described in [JWT realm settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#ref-jwt-settings). +The JWT realm has a few mandatory settings, plus optional settings that are described in [JWT realm settings](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#ref-jwt-settings). ::::{note} Client authentication is enabled by default for the JWT realms. Disabling client authentication is possible, but strongly discouraged. :::: -1. Configure the realm using your preferred token type: +1. Configure the realm using your preferred token type: :::::{tab-set} @@ -153,7 +153,7 @@ Client authentication is enabled by default for the JWT realms. Disabling client : Specifies a list of JWT subjects that the realm will allow. These values are typically URLs, UUIDs, or other case-sensitive string values. `allowed_subject_patterns` - : Analogous to `allowed_subjects` but it accepts a list of [Lucene regexp](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/regexp-syntax.md) and wildcards for the allowed JWT subjects. Wildcards use the `*` and `?` special characters (which are escaped by `\`) to mean "any string" and "any single character" respectively, for example "a?\**", matches "a1*" and "ab*whatever", but not "a", "abc", or "abc*" (in Java strings `\` must itself be escaped by another `\`). [Lucene regexp](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/regexp-syntax.md) must be enclosed between `/`, for example "/https?://[^/]+/?/" matches any http or https URL with no path component (matches "https://elastic.co/" but not "https://elastic.co/guide"). + : Analogous to `allowed_subjects` but it accepts a list of [Lucene regexp](elasticsearch://reference/query-languages/regexp-syntax.md) and wildcards for the allowed JWT subjects. Wildcards use the `*` and `?` special characters (which are escaped by `\`) to mean "any string" and "any single character" respectively, for example "a?\**", matches "a1*" and "ab*whatever", but not "a", "abc", or "abc*" (in Java strings `\` must itself be escaped by another `\`). [Lucene regexp](elasticsearch://reference/query-languages/regexp-syntax.md) must be enclosed between `/`, for example "/https?://[^/]+/?/" matches any http or https URL with no path component (matches "https://elastic.co/" but not "https://elastic.co/guide"). At least one of the `allowed_subjects` or `allowed_subject_patterns` settings must be specified (and be non-empty) when `token_type` is `access_token`. @@ -172,13 +172,13 @@ Client authentication is enabled by default for the JWT realms. Disabling client 2. Add secure settings [to the {{es}} keystore](/deploy-manage/security/secure-settings.md): - * The `shared_secret` value for `client_authentication.type` - + * The `shared_secret` value for `client_authentication.type` + (`xpack.security.authc.realms.jwt.jwt1.client_authentication.shared_secret1`) - * The HMAC keys for `allowed_signature_algorithms` - + * The HMAC keys for `allowed_signature_algorithms` + (`xpack.security.authc.realms.jwt.jwt1.hmac_jwkset`) - + This setting can be a path to a JWKS, which is a resource for a set of JSON-encoded secret keys. The file can be removed after you load the contents into the {{es}} keystore. @@ -208,7 +208,7 @@ Signature: UnnFmsoFKfNmKMsVoDQmKI_3-j95PCaKdgqqau3jPMY This example illustrates a partial decoding of a JWT. The validity period is from 2000 to 2099 (inclusive), as defined by the issue time (`iat`) and expiration time (`exp`). JWTs typically have a validity period shorter than 100 years, such as 1-2 hours or 1-7 days, not an entire human life. -The signature in this example is deterministic because the header, claims, and HMAC key are fixed. JWTs typically have a `nonce` claim to make the signature non-deterministic. The supported JWT encoding is JSON Web Signature (JWS), and the JWS `Header` and `Signature` are validated using OpenID Connect ID Token validation rules. Some validation is customizable through [JWT realm settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#ref-jwt-settings). +The signature in this example is deterministic because the header, claims, and HMAC key are fixed. JWTs typically have a `nonce` claim to make the signature non-deterministic. The supported JWT encoding is JSON Web Signature (JWS), and the JWS `Header` and `Signature` are validated using OpenID Connect ID Token validation rules. Some validation is customizable through [JWT realm settings](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#ref-jwt-settings). ### Header claims [jwt-validation-header] @@ -280,12 +280,12 @@ You can relax validation of any of the time-based claims by setting `allowed_clo ## Role mapping [jwt-authorization] -You can map LDAP groups to roles in the following ways: +You can map LDAP groups to roles in the following ways: * Using the role mappings page in {{kib}}. -* Using the [role mapping API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-put-role-mapping). +* Using the [role mapping API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-put-role-mapping). * By delegating authorization [to another realm](#jwt-authorization-delegation). - + For more information, see [Mapping users and groups to roles](/deploy-manage/users-roles/cluster-or-deployment-auth/mapping-users-groups-to-roles.md). ::::{important} @@ -445,7 +445,7 @@ Separate reload requests cannot be combined if JWT signature failures trigger: * PKC JWKS reloads in the same {{es}} node at different times ::::{important} -Enabling client authentication (`client_authentication.type`) is strongly recommended. Only trusted client applications and realm-specific JWT users can trigger PKC reload attempts. Additionally, configuring the following [JWT security settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#ref-jwt-settings) is recommended: +Enabling client authentication (`client_authentication.type`) is strongly recommended. Only trusted client applications and realm-specific JWT users can trigger PKC reload attempts. Additionally, configuring the following [JWT security settings](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#ref-jwt-settings) is recommended: * `allowed_audiences` * `allowed_clock_skew` @@ -497,7 +497,7 @@ xpack.security.authc.realms.jwt.jwt8.client_authentication.type: shared_secret ### JWT realm secure settings [_jwt_realm_secure_settings] -After defining the realm settings, use the [`elasticsearch-keystore`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/elasticsearch-keystore.md) tool to add the following secure settings to the {{es}} keystore. In {{ecloud}}, you define settings for the {{es}} keystore under **Security** in your deployment. +After defining the realm settings, use the [`elasticsearch-keystore`](elasticsearch://reference/elasticsearch/command-line-tools/elasticsearch-keystore.md) tool to add the following secure settings to the {{es}} keystore. In {{ecloud}}, you define settings for the {{es}} keystore under **Security** in your deployment. ```yaml xpack.security.authc.realms.jwt.jwt8.hmac_key: hmac-oidc-key-string-for-hs256-algorithm diff --git a/deploy-manage/users-roles/cluster-or-deployment-auth/kerberos.md b/deploy-manage/users-roles/cluster-or-deployment-auth/kerberos.md index c47599d25..18421981a 100644 --- a/deploy-manage/users-roles/cluster-or-deployment-auth/kerberos.md +++ b/deploy-manage/users-roles/cluster-or-deployment-auth/kerberos.md @@ -17,12 +17,12 @@ applies_to: You can configure the {{stack}} {{security-features}} to support Kerberos V5 authentication, an industry standard protocol to authenticate users in {{es}} and {{kib}}. -::::{note} +::::{note} You can't use the Kerberos realm to authenticate on the transport network layer. :::: -To authenticate users with Kerberos, you need to configure a Kerberos realm and map users to roles. For more information on realm settings, see [Kerberos realm settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#ref-kerberos-settings). +To authenticate users with Kerberos, you need to configure a Kerberos realm and map users to roles. For more information on realm settings, see [Kerberos realm settings](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#ref-kerberos-settings). ## Key concepts [kerberos-terms] @@ -47,7 +47,7 @@ realm keytab : A file that stores pairs of principals and encryption keys. - ::::{important} + ::::{important} Anyone with read permissions to this file can use the credentials in the network to access other services so it is important to protect it with proper file permissions. :::: @@ -69,13 +69,13 @@ In Kerberos, users authenticate with an authentication service and later with a Before you set up a Kerberos realm, you must have the Kerberos infrastructure set up in your environment. -::::{note} +::::{note} Kerberos requires a lot of external services to function properly, such as time synchronization between all machines and working forward and reverse DNS mappings in your domain. Refer to your Kerberos documentation for more details. :::: These instructions do not cover setting up and configuring your Kerberos deployment. Where examples are provided, they pertain to an MIT Kerberos V5 deployment. For more information, see [MIT Kerberos documentation](http://web.mit.edu/kerberos/www/index.html) -If you're using a self-managed cluster, then perform the following additional steps: +If you're using a self-managed cluster, then perform the following additional steps: * Enable TLS for HTTP. @@ -83,7 +83,7 @@ If you're using a self-managed cluster, then perform the following additional st This step is necessary to support Kerberos authentication through {{kib}}. It is not required for Kerberos authentication directly against the {{es}} Rest API. - If you started {{es}} [with security enabled](/deploy-manage/deploy/self-managed/installing-elasticsearch.md), then TLS is already enabled for HTTP. + If you started {{es}} [with security enabled](/deploy-manage/deploy/self-managed/installing-elasticsearch.md), then TLS is already enabled for HTTP. {{ech}}, {{ece}}, and {{eck}} have TLS enabled by default. @@ -110,7 +110,7 @@ To configure a Kerberos realm in {{es}}: * `krb5.conf`: The Kerberos configuration file (`krb5.conf`) provides information such as the default realm, the Key Distribution Center (KDC), and other configuration details required for Kerberos authentication. For more information, see [krb5.conf](https://web.mit.edu/kerberos/krb5-latest/doc/admin/conf_files/krb5_conf.html). * `keytab`: A keytab is a file that stores pairs of principals and encryption keys. {{es}} uses the keys from the keytab to decrypt the tickets presented by the user. You must create a keytab for {{es}} by using the tools provided by your Kerberos implementation. For example, some tools that create keytabs are `ktpass.exe` on Windows and `kadmin` for MIT Kerberos. - + The configuration requirements depend on your Kerberos setup. Refer to your Kerberos documentation to configure the `krb5.conf` file. For more information on Java GSS, see [Java GSS Kerberos requirements](https://docs.oracle.com/javase/10/security/kerberos-requirements1.htm). @@ -119,34 +119,34 @@ For more information on Java GSS, see [Java GSS Kerberos requirements](https://d The way that you provide Kerberos config files to {{es}} depends on your deployment method. -For detailed information of available realm settings, see [Kerberos realm settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#ref-kerberos-settings). +For detailed information of available realm settings, see [Kerberos realm settings](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#ref-kerberos-settings). :::::{tab-set} ::::{tab-item} Self-managed 1. Configure the JVM to find the Kerberos configuration file. - - {{es}} uses Java GSS and JAAS Krb5LoginModule to support Kerberos authentication using a Simple and Protected GSSAPI Negotiation (SPNEGO) mechanism. When the JVM needs some configuration properties, it tries to find those values by locating and loading the `krb5.conf` file. The JVM system property to configure the file path is `java.security.krb5.conf`. To configure JVM system properties see [Set JVM options](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/jvm-settings.md#set-jvm-options). If this system property is not specified, Java tries to locate the file based on the conventions. - - :::{tip} + + {{es}} uses Java GSS and JAAS Krb5LoginModule to support Kerberos authentication using a Simple and Protected GSSAPI Negotiation (SPNEGO) mechanism. When the JVM needs some configuration properties, it tries to find those values by locating and loading the `krb5.conf` file. The JVM system property to configure the file path is `java.security.krb5.conf`. To configure JVM system properties see [Set JVM options](elasticsearch://reference/elasticsearch/jvm-settings.md#set-jvm-options). If this system property is not specified, Java tries to locate the file based on the conventions. + + :::{tip} It is recommended that this system property be configured for {{es}}. The method for setting this property depends on your Kerberos infrastructure. Refer to your Kerberos documentation for more details. ::: For more information, see [krb5.conf](https://web.mit.edu/kerberos/krb5-latest/doc/admin/conf_files/krb5_conf.html). 2. Put the keytab file in the {{es}} configuration directory. - + Make sure that this keytab file has read permissions. This file contains credentials, therefore you must take appropriate measures to protect it. - - :::{important} + + :::{important} {{es}} uses Kerberos on the HTTP network layer, therefore there must be a keytab file for the HTTP service principal on every {{es}} node. The service principal name must have the format `HTTP/es.domain.local@ES.DOMAIN.LOCAL`. The keytab files are unique for each node since they include the hostname. An {{es}} node can act as any principal a client requests as long as that principal and its credentials are found in the configured keytab. ::: 3. Create a Kerberos realm. - + To enable Kerberos authentication in {{es}}, you must add a Kerberos realm in the realm chain. - + :::{note} You can configure only one Kerberos realm on {{es}} nodes. ::: @@ -154,7 +154,7 @@ For detailed information of available realm settings, see [Kerberos realm settin To configure a Kerberos realm, there are a few mandatory realm settings and other optional settings that you need to configure in the `elasticsearch.yml` configuration file. Add a realm configuration under the `xpack.security.authc.realms.kerberos` namespace. The most common configuration for a Kerberos realm is as follows: - + ```yaml xpack.security.authc.realms.kerberos.kerb1: order: 3 @@ -166,7 +166,7 @@ For detailed information of available realm settings, see [Kerberos realm settin ::::{tab-item} ECH and ECE -1. Create a [custom bundle](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch-plugins/cloud-enterprise/ece-add-plugins.md) that contains your `krb5.conf` and `keytab` files, and add it to your cluster. +1. Create a [custom bundle](elasticsearch://reference/elasticsearch-plugins/cloud-enterprise/ece-add-plugins.md) that contains your `krb5.conf` and `keytab` files, and add it to your cluster. :::{tip} You should use these exact filenames for {{ecloud}} to recognize the file in the bundle. @@ -191,13 +191,13 @@ For detailed information of available realm settings, see [Kerberos realm settin 1. Install your `krb5.conf` and `keytab` files as a [custom configuration files](/deploy-manage/deploy/cloud-on-k8s/custom-configuration-files-plugins.md#use-a-volume-and-volume-mount-together-with-a-configmap-or-secret). Mount them in a sub-directory of the main config directory, for example `/usr/share/elasticsearch/config/kerberos`, and use a `Secret` instead of a `ConfigMap` to store the information. 2. Configure the JVM to find the Kerberos configuration file. - + {{es}} uses Java GSS and JAAS Krb5LoginModule to support Kerberos authentication using a Simple and Protected GSSAPI Negotiation (SPNEGO) mechanism. When the JVM needs some configuration properties, it tries to find those values by locating and loading the `krb5.conf` file. The JVM system property to configure the file path is `java.security.krb5.conf`. If this system property is not specified, Java tries to locate the file based on the conventions. To provide JVM setting overrides to your cluster: 1. Create a new ConfigMap with a valid JVM options file as the key. The source file should be a JVM `.options` file containing the JVM system property `-Djava.security.krb5.conf=/usr/share/elasticsearch/config/kerberos/krb5.conf`, assuming the `krb5.conf` file was mounted on `/usr/share/elasticsearch/config/kerberos` in the previous step. - + ``` # create a configmap with a key named override.options and the content of your local file kubectl create configmap jvm-options --from-file=override.options= @@ -209,7 +209,7 @@ For detailed information of available realm settings, see [Kerberos realm settin apiVersion: elasticsearch.k8s.elastic.co/v1 kind: Elasticsearch metadata: - name: test-cluster + name: test-cluster spec: version: 8.17.0 nodeSets: @@ -253,12 +253,12 @@ The `username` is extracted from the ticket presented by the user and usually ha ## Map Kerberos users to roles -The `kerberos` realm enables you to map Kerberos users to roles. +The `kerberos` realm enables you to map Kerberos users to roles. -You can map these users to roles in multiple ways: +You can map these users to roles in multiple ways: * Using the role mappings page in {{kib}}. -* Using the [role mapping API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-put-role-mapping). +* Using the [role mapping API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-put-role-mapping). You identify users by their `username` field. @@ -275,14 +275,14 @@ POST /_security/role_mapping/kerbrolemapping } ``` -In case you want to support Kerberos cross realm authentication, you may need to map roles based on the Kerberos realm name. For such scenarios, the following additional user metadata can be used for role mapping: +In case you want to support Kerberos cross realm authentication, you may need to map roles based on the Kerberos realm name. For such scenarios, the following additional user metadata can be used for role mapping: -- `kerberos_realm`: The Kerberos realm name. +- `kerberos_realm`: The Kerberos realm name. - `kerberos_user_principal_name`: The user principal name from the Kerberos ticket. For more information, see [Mapping users and groups to roles](/deploy-manage/users-roles/cluster-or-deployment-auth/mapping-users-groups-to-roles.md). -::::{note} +::::{note} The Kerberos realm supports [authorization realms](/deploy-manage/users-roles/cluster-or-deployment-auth/realm-chains.md#authorization_realms) as an alternative to role mapping. :::: diff --git a/deploy-manage/users-roles/cluster-or-deployment-auth/kibana-authentication.md b/deploy-manage/users-roles/cluster-or-deployment-auth/kibana-authentication.md index ad06aa4e5..2247f7683 100644 --- a/deploy-manage/users-roles/cluster-or-deployment-auth/kibana-authentication.md +++ b/deploy-manage/users-roles/cluster-or-deployment-auth/kibana-authentication.md @@ -2,10 +2,10 @@ navigation_title: "Kibana authentication" applies_to: deployment: - ess: - ece: - eck: - self: + ess: + ece: + eck: + self: --- # Authentication in {{kib}} [kibana-authentication] @@ -71,7 +71,7 @@ xpack.security.authc.providers: :class: screenshot ::: -For more information, refer to [authentication security settings](asciidocalypse://docs/kibana/docs/reference/configuration-reference/security-settings.md#authentication-security-settings). +For more information, refer to [authentication security settings](kibana://reference/configuration-reference/security-settings.md#authentication-security-settings). ::::{tip} If you have multiple authentication providers configured, you can use the `auth_provider_hint` URL query parameter to create a deep link to any provider and bypass the Login Selector UI. Using the `kibana.yml` above as an example, you can add `?auth_provider_hint=basic1` to the login page URL, which will take you directly to the basic login page. @@ -261,7 +261,7 @@ The following sections apply both to [SAML single sign-on](#saml) and [OpenID Co #### Access and refresh tokens [_access_and_refresh_tokens] -Once the user logs in to {{kib}} with SSO, either using SAML or OpenID Connect, {{es}} issues access and refresh tokens that {{kib}} encrypts and stores as a part of its own session. This way, the user isn’t redirected to the Identity Provider for every request that requires authentication. It also means that the {{kib}} session depends on the [`xpack.security.session.idleTimeout` and `xpack.security.session.lifespan`](asciidocalypse://docs/kibana/docs/reference/configuration-reference/security-settings.md#security-session-and-cookie-settings) settings, and the user is automatically logged out if the session expires. An access token that is stored in the session can expire, in which case {{kib}} will automatically renew it with a one-time-use refresh token and store it in the same session. +Once the user logs in to {{kib}} with SSO, either using SAML or OpenID Connect, {{es}} issues access and refresh tokens that {{kib}} encrypts and stores as a part of its own session. This way, the user isn’t redirected to the Identity Provider for every request that requires authentication. It also means that the {{kib}} session depends on the [`xpack.security.session.idleTimeout` and `xpack.security.session.lifespan`](kibana://reference/configuration-reference/security-settings.md#security-session-and-cookie-settings) settings, and the user is automatically logged out if the session expires. An access token that is stored in the session can expire, in which case {{kib}} will automatically renew it with a one-time-use refresh token and store it in the same session. {{kib}} can only determine if an access token has expired if it receives a request that requires authentication. If both access and refresh tokens have already expired (for example, after 24 hours of inactivity), {{kib}} initiates a new "handshake" and redirects the user to the external authentication provider (SAML Identity Provider or OpenID Connect Provider) Depending on {{es}} and the external authentication provider configuration, the user might be asked to re-enter credentials. @@ -364,7 +364,7 @@ For information on how to embed, refer to [Embed {{kib}} content in a web page]( {{kib}} maintains a separate [session](/deploy-manage/security/kibana-session-management.md) for every anonymous user, as it does for all other authentication mechanisms. -You can configure [session idle timeout](/deploy-manage/security/kibana-session-management.md#session-idle-timeout) and [session lifespan](/deploy-manage/security/kibana-session-management.md#session-lifespan) for anonymous sessions the same as you do for any other session with the exception that idle timeout is explicitly disabled for anonymous sessions by default. The global [`xpack.security.session.idleTimeout`](asciidocalypse://docs/kibana/docs/reference/configuration-reference/security-settings.md#security-session-and-cookie-settings) setting doesn’t affect anonymous sessions. To change the idle timeout for anonymous sessions, you must configure the provider-level [`xpack.security.authc.providers.anonymous..session.idleTimeout`](asciidocalypse://docs/kibana/docs/reference/configuration-reference/security-settings.md#anonymous-authentication-provider-settings) setting. +You can configure [session idle timeout](/deploy-manage/security/kibana-session-management.md#session-idle-timeout) and [session lifespan](/deploy-manage/security/kibana-session-management.md#session-lifespan) for anonymous sessions the same as you do for any other session with the exception that idle timeout is explicitly disabled for anonymous sessions by default. The global [`xpack.security.session.idleTimeout`](kibana://reference/configuration-reference/security-settings.md#security-session-and-cookie-settings) setting doesn’t affect anonymous sessions. To change the idle timeout for anonymous sessions, you must configure the provider-level [`xpack.security.authc.providers.anonymous..session.idleTimeout`](kibana://reference/configuration-reference/security-settings.md#anonymous-authentication-provider-settings) setting. ## HTTP authentication [http-authentication] @@ -404,13 +404,13 @@ With this configuration, you can send requests to {{kib}} with the `Authorizatio Once you create a dashboard or a visualization, you might want to share it with your colleagues or friends. The easiest way to do this is to share a direct link to your dashboard or visualization. However, some users might not have access to your {{kib}}. With the {{kib}} embedding functionality, you can display the content you created in {{kib}} to an internal company website or a personal web page. $$$embedding-cookies$$$ -To minimize security risk, embedding with iframes requires careful consideration. By default, modern web browsers enforce the [same-origin policy](https://developer.mozilla.org/en-US/docs/Web/Security/Same-origin_policy) to restrict the behavior of framed pages. When {{stack-security-features}} are enabled on your cluster, make sure the browsers can transmit session cookies to a {{kib}} server. The setting you need to be aware of is [`xpack.security.sameSiteCookies`](asciidocalypse://docs/kibana/docs/reference/configuration-reference/security-settings.md#xpack-security-sameSiteCookies). To support modern browsers, you must set it to `None`: +To minimize security risk, embedding with iframes requires careful consideration. By default, modern web browsers enforce the [same-origin policy](https://developer.mozilla.org/en-US/docs/Web/Security/Same-origin_policy) to restrict the behavior of framed pages. When {{stack-security-features}} are enabled on your cluster, make sure the browsers can transmit session cookies to a {{kib}} server. The setting you need to be aware of is [`xpack.security.sameSiteCookies`](kibana://reference/configuration-reference/security-settings.md#xpack-security-samesitecookies). To support modern browsers, you must set it to `None`: ```yaml xpack.security.sameSiteCookies: "None" ``` -For more information about possible values and implications, refer to [xpack.security.sameSiteCookies](asciidocalypse://docs/kibana/docs/reference/configuration-reference/security-settings.md#xpack-security-sameSiteCookies). For more information about iframe and cookies, refer to [iframe](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe) and [SameSite cookies](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie/SameSite). +For more information about possible values and implications, refer to [xpack.security.sameSiteCookies](kibana://reference/configuration-reference/security-settings.md#xpack-security-samesitecookies). For more information about iframe and cookies, refer to [iframe](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe) and [SameSite cookies](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie/SameSite). If you’re embedding {{kib}} in a website that supports single sign-on (SSO) with SAML, OpenID Connect, Kerberos, or PKI, it’s highly advisable to configure {{kib}} as a part of the SSO setup. Operating in a single and properly configured security domain provides you with the most secure and seamless user experience. diff --git a/deploy-manage/users-roles/cluster-or-deployment-auth/ldap.md b/deploy-manage/users-roles/cluster-or-deployment-auth/ldap.md index 67001f840..b507faca2 100644 --- a/deploy-manage/users-roles/cluster-or-deployment-auth/ldap.md +++ b/deploy-manage/users-roles/cluster-or-deployment-auth/ldap.md @@ -18,7 +18,7 @@ You can configure the {{stack}} {{security-features}} to communicate with a Ligh To integrate with LDAP, you configure an `ldap` realm and map LDAP groups to user roles. :::{{tip}} -This topic describes implementing LDAP at the cluster or deployment level, for the purposes of authenticating with {{es}} and {{kib}}. +This topic describes implementing LDAP at the cluster or deployment level, for the purposes of authenticating with {{es}} and {{kib}}. You can also configure an {{ece}} installation to use an LDAP server to authenticate users. [Learn more](/deploy-manage/users-roles/cloud-enterprise-orchestrator/ldap.md). ::: @@ -31,7 +31,7 @@ The path to an entry is a *Distinguished Name* (DN) that uniquely identifies a u The `ldap` realm supports two modes of operation, a user search mode and a mode with specific templates for user DNs. -::::{important} +::::{important} When you configure realms in `elasticsearch.yml`, only the realms you specify are used for authentication. If you also want to use the `native` or `file` realms, you must include them in the realm chain. :::: @@ -44,13 +44,13 @@ The `ldap` realm supports two modes of operation, a user search mode and a mode * **DN templates**: If your LDAP environment uses a few specific standard naming conditions for users, you can use user DN templates to configure the realm. The advantage of this method is that a search does not have to be performed to find the user DN. However, multiple bind operations might be needed to find the correct user DN. -### Set up LDAP user search mode +### Set up LDAP user search mode To configure an `ldap` realm with user search: -1. Add a realm configuration to `elasticsearch.yml` under the `xpack.security.authc.realms.ldap` namespace. - - At a minimum, you must specify the `url` and `order` of the LDAP server, and set `user_search.base_dn` to the container DN where the users are searched for. See [LDAP realm settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#ref-ldap-settings) for all of the options you can set for an `ldap` realm. +1. Add a realm configuration to `elasticsearch.yml` under the `xpack.security.authc.realms.ldap` namespace. + + At a minimum, you must specify the `url` and `order` of the LDAP server, and set `user_search.base_dn` to the container DN where the users are searched for. See [LDAP realm settings](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#ref-ldap-settings) for all of the options you can set for an `ldap` realm. For example, the following snippet shows an LDAP realm configured with a user search: @@ -90,10 +90,10 @@ To configure an `ldap` realm with user search: 1. (Optional) Configure how the {{security-features}} interact with multiple LDAP servers. - The `load_balance.type` setting can be used at the realm level. The {{es}} {{security-features}} support both failover and load balancing modes of operation. See [LDAP realm settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#ref-ldap-settings). + The `load_balance.type` setting can be used at the realm level. The {{es}} {{security-features}} support both failover and load balancing modes of operation. See [LDAP realm settings](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#ref-ldap-settings). 2. (Optional) To protect passwords, [encrypt communications between {{es}} and the LDAP server](../../../deploy-manage/users-roles/cluster-or-deployment-auth/ldap.md#tls-ldap). - + * **For self-managed clusters and {{eck}} deployments**, clients and nodes that connect using SSL/TLS to the Active Directory server need to have the Active Directory server’s certificate or the server’s root CA certificate installed in their keystore or trust store. * **For {{ece}} and {{ech}} deployments**, if your Domain Controller is configured to use LDAP over TLS and it uses a self-signed certificate or a certificate that is signed by your organization’s CA, you need to enable the deployment to trust this certificate. @@ -103,7 +103,7 @@ To configure an `ldap` realm with user search: To configure an `ldap` realm with user DN templates: -1. Add a realm configuration to `elasticsearch.yml` in the `xpack.security.authc.realms.ldap` namespace. At a minimum, you must specify the `url` and `order` of the LDAP server, and specify at least one template with the `user_dn_templates` option. See [LDAP realm settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#ref-ldap-settings) for all of the options you can set for an `ldap` realm. +1. Add a realm configuration to `elasticsearch.yml` in the `xpack.security.authc.realms.ldap` namespace. At a minimum, you must specify the `url` and `order` of the LDAP server, and specify at least one template with the `user_dn_templates` option. See [LDAP realm settings](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#ref-ldap-settings) for all of the options you can set for an `ldap` realm. For example, the following snippet shows an LDAP realm configured with user DN templates: @@ -136,10 +136,10 @@ To configure an `ldap` realm with user DN templates: 2. (Optional) Configure how the {{security-features}} interact with multiple LDAP servers. - The `load_balance.type` setting can be used at the realm level. The {{es}} {{security-features}} support both failover and load balancing modes of operation. See [LDAP realm settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#ref-ldap-settings). + The `load_balance.type` setting can be used at the realm level. The {{es}} {{security-features}} support both failover and load balancing modes of operation. See [LDAP realm settings](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#ref-ldap-settings). 3. (Optional) To protect passwords, [encrypt communications between {{es}} and the LDAP server](../../../deploy-manage/users-roles/cluster-or-deployment-auth/ldap.md#tls-ldap). - + * **For self-managed clusters and {{eck}} deployments**, clients and nodes that connect using SSL/TLS to the Active Directory server need to have the Active Directory server’s certificate or the server’s root CA certificate installed in their keystore or trust store. * **For {{ece}} and {{ech}} deployments**, if your Domain Controller is configured to use LDAP over TLS and it uses a self-signed certificate or a certificate that is signed by your organization’s CA, you need to enable the deployment to trust this certificate. @@ -151,17 +151,17 @@ An integral part of a realm authentication process is to resolve the roles assoc Because users are managed externally in the LDAP server, the expectation is that their roles are managed there as well. LDAP groups often represent user roles for different systems in the organization. -The `active_directory` realm enables you to map Active Directory users to roles using their Active Directory groups or other metadata. +The `active_directory` realm enables you to map Active Directory users to roles using their Active Directory groups or other metadata. -You can map LDAP groups to roles in the following ways: +You can map LDAP groups to roles in the following ways: * Using the role mappings page in {{kib}}. -* Using the [role mapping API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-put-role-mapping). +* Using the [role mapping API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-put-role-mapping). * Using a role mapping file. For more information, see [Mapping users and groups to roles](/deploy-manage/users-roles/cluster-or-deployment-auth/mapping-users-groups-to-roles.md). -::::{note} +::::{note} The LDAP realm supports [authorization realms](../../../deploy-manage/users-roles/cluster-or-deployment-auth/realm-chains.md#authorization_realms) as an alternative to role mapping. :::: @@ -190,7 +190,7 @@ POST /_security/role_mapping/ldap-superuser <1> ### Example: Using a role mapping file -:::{tip} +:::{tip} If you're using {{ece}} or {{ech}}, then you must [upload this file as a custom bundle](/deploy-manage/deploy/elastic-cloud/upload-custom-plugins-bundles.md) before it can be referenced. If you're using {{eck}}, then install the file as a [custom configuration file](/deploy-manage/deploy/cloud-on-k8s/custom-configuration-files-plugins.md#use-a-volume-and-volume-mount-together-with-a-configmap-or-secret). If you're using a self-managed cluster, then the file must be present on each node. ::: @@ -260,7 +260,7 @@ xpack: The `load_balance.type` setting can be used at the realm level to configure how the {{security-features}} should interact with multiple LDAP servers. The {{security-features}} support both failover and load balancing modes of operation. -See [Load balancing and failover](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#load-balancing). +See [Load balancing and failover](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#load-balancing). ## Encrypting communications between {{es}} and LDAP [tls-ldap] @@ -271,7 +271,7 @@ If you're using {{ech}} or {{ece}}, then you must [upload your certificate as a If you're using {{eck}}, then install the certificate as a [custom configuration file](/deploy-manage/deploy/cloud-on-k8s/custom-configuration-files-plugins.md#use-a-volume-and-volume-mount-together-with-a-configmap-or-secret). -:::{tip} +:::{tip} If you're using {{ece}} or {{ech}}, then these steps are required only if TLS is enabled and the Active Directory controller is using self-signed certificates. ::: @@ -306,7 +306,7 @@ You can also specify the individual server certificates rather than the CA certi For more information about these settings, see [LDAP realm settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-settings.html#ref-ldap-settings). -::::{note} +::::{note} By default, when you configure {{es}} to connect to an LDAP server using SSL/TLS, it attempts to verify the hostname or IP address specified with the `url` attribute in the realm configuration with the values in the certificate. If the values in the certificate and realm configuration do not match, {{es}} does not allow a connection to the LDAP server. This is done to protect against man-in-the-middle attacks. If necessary, you can disable this behavior by setting the `ssl.verification_mode` property to `certificate`. :::: diff --git a/deploy-manage/users-roles/cluster-or-deployment-auth/looking-up-users-without-authentication.md b/deploy-manage/users-roles/cluster-or-deployment-auth/looking-up-users-without-authentication.md index bbb72bf91..4e4635fb7 100644 --- a/deploy-manage/users-roles/cluster-or-deployment-auth/looking-up-users-without-authentication.md +++ b/deploy-manage/users-roles/cluster-or-deployment-auth/looking-up-users-without-authentication.md @@ -3,8 +3,8 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/user-lookup.html applies_to: deployment: - ess: - ece: + ess: + ece: eck: self: --- @@ -28,18 +28,18 @@ See the [run_as](submitting-requests-on-behalf-of-other-users.md) and [delegated * The reserved, [`native`](native.md) and [`file`](file-based.md) realms always support user lookup. * The [`ldap`](ldap.md) realm supports user lookup when the realm is configured in [*user search* mode](ldap.md#ldap-realm-configuration). User lookup is not support when the realm is configured with `user_dn_templates`. -* User lookup support in the [`active_directory`](active-directory.md) realm requires that the realm be configured with a [`bind_dn`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#ref-ad-settings) and a bind password. +* User lookup support in the [`active_directory`](active-directory.md) realm requires that the realm be configured with a [`bind_dn`](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#ref-ad-settings) and a bind password. The `pki`, `saml`, `oidc`, `kerberos` and `jwt` realms do not support user lookup. -::::{note} -If you want to use a realm only for user lookup and prevent users from authenticating against that realm, you can [configure the realm](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#ref-realm-settings) and set `authentication.enabled` to `false` +::::{note} +If you want to use a realm only for user lookup and prevent users from authenticating against that realm, you can [configure the realm](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#ref-realm-settings) and set `authentication.enabled` to `false` :::: The user lookup feature is an internal capability that is used to implement the `run_as` and delegated authorization features - there are no APIs for user lookup. If you want to test your user lookup configuration, then you can do this with `run_as`. Use the [Authenticate](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-authenticate) API, authenticate as a `superuser` (e.g. the builtin `elastic` user) and specify the [`es-security-runas-user` request header](submitting-requests-on-behalf-of-other-users.md). -::::{note} +::::{note} The [Get users](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-get-user) API and [User profiles](user-profiles.md) feature are alternative ways to retrieve information about a {{stack}} user. Those APIs are not related to the user lookup feature. :::: diff --git a/deploy-manage/users-roles/cluster-or-deployment-auth/manage-authentication-for-multiple-clusters.md b/deploy-manage/users-roles/cluster-or-deployment-auth/manage-authentication-for-multiple-clusters.md index 87e53b094..9ead7ddea 100644 --- a/deploy-manage/users-roles/cluster-or-deployment-auth/manage-authentication-for-multiple-clusters.md +++ b/deploy-manage/users-roles/cluster-or-deployment-auth/manage-authentication-for-multiple-clusters.md @@ -34,7 +34,7 @@ Make sure you check the complete [guide to setting up LDAP with {{es}}](/deploy- ### Configure LDAP using {{stack}} configuration policy with user search[k8s_to_configure_ldap_using_elastic_stack_configuration_policy_with_user_search] -1. Add a realm configuration to the `config` field under `elasticsearch` in the `xpack.security.authc.realms.ldap` namespace. At a minimum, you must specify the URL of the LDAP server and the order of the LDAP realm compared to other configured security realms. You also have to set `user_search.base_dn` to the container DN where the users are searched for. Refer to [LDAP realm settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#ref-ldap-settings) for all of the options you can set for an LDAP realm. For example, the following snippet shows an LDAP realm configured with a user search: +1. Add a realm configuration to the `config` field under `elasticsearch` in the `xpack.security.authc.realms.ldap` namespace. At a minimum, you must specify the URL of the LDAP server and the order of the LDAP realm compared to other configured security realms. You also have to set `user_search.base_dn` to the container DN where the users are searched for. Refer to [LDAP realm settings](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#ref-ldap-settings) for all of the options you can set for an LDAP realm. For example, the following snippet shows an LDAP realm configured with a user search: ```yaml elasticsearch: @@ -127,7 +127,7 @@ spec: ### Configure an LDAP realm with user DN templates[k8s_to_configure_an_ldap_realm_with_user_dn_templates] -Add a realm configuration to `elasticsearch.yml` in the xpack.security.authc.realms.ldap namespace. At a minimum, you must specify the url and order of the LDAP server, and specify at least one template with the user_dn_templates option. Check [LDAP realm settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#ref-ldap-settings) for all of the options you can set for an ldap realm. +Add a realm configuration to `elasticsearch.yml` in the xpack.security.authc.realms.ldap namespace. At a minimum, you must specify the url and order of the LDAP server, and specify at least one template with the user_dn_templates option. Check [LDAP realm settings](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#ref-ldap-settings) for all of the options you can set for an ldap realm. For example, the following snippet shows an LDAP realm configured with user DN templates: diff --git a/deploy-manage/users-roles/cluster-or-deployment-auth/mapping-users-groups-to-roles.md b/deploy-manage/users-roles/cluster-or-deployment-auth/mapping-users-groups-to-roles.md index 6edfa92cd..37b87786a 100644 --- a/deploy-manage/users-roles/cluster-or-deployment-auth/mapping-users-groups-to-roles.md +++ b/deploy-manage/users-roles/cluster-or-deployment-auth/mapping-users-groups-to-roles.md @@ -25,7 +25,7 @@ You can map external users and groups to roles in the following ways: * Using the [Role mapping API](#mapping-roles-api) * Using [role mapping files](#mapping-roles-file) (PKI, LDAP, AD realms only) -::::{note} +::::{note} When [anonymous access](/deploy-manage/users-roles/cluster-or-deployment-auth/anonymous-access.md) is enabled, the roles of the anonymous user are assigned to all the other users as well. :::: @@ -37,14 +37,14 @@ The native and file realms assign roles directly to users. Native realms use [us ## Role sources -Before you use role mappings to assign roles to users, the roles must exist. You can assign [built-in roles](/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md), or [custom roles](/deploy-manage/users-roles/cluster-or-deployment-auth/defining-roles.md) defined through [the UI](/deploy-manage/users-roles/cluster-or-deployment-auth/kibana-role-management.md), [the API](/deploy-manage/users-roles/cluster-or-deployment-auth/defining-roles.md#roles-management-api), or [a roles file](/deploy-manage/users-roles/cluster-or-deployment-auth/defining-roles.md#roles-management-file). - +Before you use role mappings to assign roles to users, the roles must exist. You can assign [built-in roles](/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-roles.md), or [custom roles](/deploy-manage/users-roles/cluster-or-deployment-auth/defining-roles.md) defined through [the UI](/deploy-manage/users-roles/cluster-or-deployment-auth/kibana-role-management.md), [the API](/deploy-manage/users-roles/cluster-or-deployment-auth/defining-roles.md#roles-management-api), or [a roles file](/deploy-manage/users-roles/cluster-or-deployment-auth/defining-roles.md#roles-management-file). + Any role mapping method can use any role management method. For example, when you use the role mapping API, you are able to map users to both API-managed roles (added using the UI or directly using the API) and file-managed roles. The same applies to file-based role-mappings. ## Using multiple role mapping methods -You can use a combination of methods to map roles to users. If you create role mapping rules created through the API (and related UI), and create additional rules using a role mapping file, the rules are combined. - +You can use a combination of methods to map roles to users. If you create role mapping rules created through the API (and related UI), and create additional rules using a role mapping file, the rules are combined. + It’s possible for a single user to have some roles that were mapped through the UI or API, and others assigned based on the role mapping file. ## Using the role mapping API [mapping-roles-api] @@ -104,7 +104,7 @@ While the role mapping API and UI are the preferred way to manage role mappings, However, the `role_mapping.yml` file is provided as a minimal administrative function and is not intended to cover and be used to define roles for all use cases. -::::{important} +::::{important} You can't view, edit, or remove any roles that are defined in the role mapping files by using role mapping APIs or the role mapping UI. :::: @@ -116,17 +116,17 @@ To learn about the properties that you can include in a role mapping resource, r Refer to [Realm-specific details](#_realm_specific_details) for examples of mapping roles using the role mapping file. -:::{tip} +:::{tip} If you're using {{ece}} or {{ech}}, then you must [upload this file as a custom bundle](/deploy-manage/deploy/elastic-cloud/upload-custom-plugins-bundles.md) before it can be referenced. -If you're using {{eck}}, then install the file as a [custom configuration file](/deploy-manage/deploy/cloud-on-k8s/custom-configuration-files-plugins.md#use-a-volume-and-volume-mount-together-with-a-configmap-or-secret). +If you're using {{eck}}, then install the file as a [custom configuration file](/deploy-manage/deploy/cloud-on-k8s/custom-configuration-files-plugins.md#use-a-volume-and-volume-mount-together-with-a-configmap-or-secret). If you're using a self-managed cluster, then the file must be present on each node. Tools like Puppet or Chef can help with this. ::: -By default, role mappings are stored in `ES_PATH_CONF/role_mapping.yml`. In self-managed clusters, `ES_PATH_CONF` is `ES_HOME/config` (zip/tar installations) or `/etc/elasticsearch` (package installations). +By default, role mappings are stored in `ES_PATH_CONF/role_mapping.yml`. In self-managed clusters, `ES_PATH_CONF` is `ES_HOME/config` (zip/tar installations) or `/etc/elasticsearch` (package installations). -To specify a different location, you configure the `files.role_mapping` setting in the [Active Directory](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#ref-ad-settings), [LDAP](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#ref-ldap-settings), and [PKI](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#ref-pki-settings) realm settings in `elasticsearch.yml`. +To specify a different location, you configure the `files.role_mapping` setting in the [Active Directory](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#ref-ad-settings), [LDAP](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#ref-ldap-settings), and [PKI](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#ref-pki-settings) realm settings in `elasticsearch.yml`. Within the role mapping file, the security roles are keys and groups and users are values. The mappings can have a many-to-many relationship. When you map roles to groups, the roles of a user in that group are the combination of the roles assigned to that group and the roles assigned to that user. @@ -136,13 +136,13 @@ By default, {{es}} checks role mapping files for changes every 5 seconds. If you Review the following examples to learn how to set up role mapping using the API and using role mapping files for a few common external realms. -### Active Directory and LDAP realms [ldap-role-mapping] +### Active Directory and LDAP realms [ldap-role-mapping] To specify users and groups in the role mappings, you use their *Distinguished Names* (DNs). A DN is a string that uniquely identifies the user or group, for example `"cn=John Doe,cn=contractors,dc=example,dc=com"`. For example, the following examples map the `admins` group to the `monitoring` role and map the `John Doe` user, the `users` group, and the `admins` group to the `user` role. -::::{note} +::::{note} The {{es}} {{security-features}} support only Active Directory security groups. You can't map distribution groups to roles. :::: @@ -185,7 +185,7 @@ user: ::: :::: -### PKI realms [pki-role-mapping] +### PKI realms [pki-role-mapping] PKI realms support mapping users to roles, but you can't map groups as the PKI realm has no concept of a group. diff --git a/deploy-manage/users-roles/cluster-or-deployment-auth/openid-connect.md b/deploy-manage/users-roles/cluster-or-deployment-auth/openid-connect.md index 14091b99c..1f445a29f 100644 --- a/deploy-manage/users-roles/cluster-or-deployment-auth/openid-connect.md +++ b/deploy-manage/users-roles/cluster-or-deployment-auth/openid-connect.md @@ -26,7 +26,7 @@ Because this feature is designed with {{kib}} in mind, most sections of this gui For a detailed description of how to implement OpenID Connect with various OpenID Connect Providers (OPs), refer to [Set up OpenID Connect with Azure, Google, or Okta](/deploy-manage/users-roles/cluster-or-deployment-auth/oidc-examples.md). -::::{note} +::::{note} OpenID Connect realm support in {{kib}} is designed with the expectation that it will be the primary authentication method for the users of that {{kib}} instance. The [Configuring {{kib}}](/deploy-manage/users-roles/cluster-or-deployment-auth/openid-connect.md#oidc-configure-kibana) section describes what this entails and how you can set it up to support other realms if necessary. :::: @@ -39,9 +39,9 @@ In order for the {{stack}} to be able to use your OpenID Connect Provider for au The process for registering the {{stack}} RP will be different from OP to OP, so you should follow your provider's documentation. The information for the RP that you commonly need to provide for registration are the following: * `Relying Party Name`: An arbitrary identifier for the relying party. There are no constraints on this value, either from the specification or the {{stack}} implementation. -* `Redirect URI`: The URI where the OP will redirect the user’s browser after authentication, sometimes referred to as a `Callback URI`. The appropriate value for this will depend on your setup, and whether or not {{kib}} sits behind a proxy or load balancer. - - It will typically be `${kibana-url}/api/security/oidc/callback` (for the authorization code flow) or `${kibana-url}/api/security/oidc/implicit` (for the implicit flow) where *${kibana-url}* is the base URL for your {{kib}} instance. +* `Redirect URI`: The URI where the OP will redirect the user’s browser after authentication, sometimes referred to as a `Callback URI`. The appropriate value for this will depend on your setup, and whether or not {{kib}} sits behind a proxy or load balancer. + + It will typically be `${kibana-url}/api/security/oidc/callback` (for the authorization code flow) or `${kibana-url}/api/security/oidc/implicit` (for the implicit flow) where *${kibana-url}* is the base URL for your {{kib}} instance. If you're using {{ech}}, then set this value to `/api/security/oidc/callback`. @@ -52,13 +52,13 @@ At the end of the registration process, the OP will assign a Client Identifier a Before you set up an OpenID Connect realm, you must have an OpenID Connect Provider where the {{stack}} Relying Party will be registered. -If you're using a self-managed cluster, then perform the following additional steps: +If you're using a self-managed cluster, then perform the following additional steps: * Enable TLS for HTTP. If your {{es}} cluster is operating in production mode, you must configure the HTTP interface to use SSL/TLS before you can enable OIDC authentication. For more information, see [Encrypt HTTP client communications for {{es}}](../../../deploy-manage/security/set-up-basic-security-plus-https.md#encrypt-http-communication). - If you started {{es}} [with security enabled](/deploy-manage/deploy/self-managed/installing-elasticsearch.md), then TLS is already enabled for HTTP. + If you started {{es}} [with security enabled](/deploy-manage/deploy/self-managed/installing-elasticsearch.md), then TLS is already enabled for HTTP. {{ech}}, {{ece}}, and {{eck}} have TLS enabled by default. @@ -76,31 +76,31 @@ If you're using a self-managed cluster, then perform the following additional st OpenID Connect based authentication is enabled by configuring the appropriate realm within the authentication chain for {{es}}. -This realm has a few mandatory settings, and a number of optional settings. The available settings are described in detail in [OpenID Connect realm settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#ref-oidc-settings). This guide will explore the most common settings. +This realm has a few mandatory settings, and a number of optional settings. The available settings are described in detail in [OpenID Connect realm settings](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#ref-oidc-settings). This guide will explore the most common settings. 1. Create an OpenID Connect (the realm type is `oidc`) realm in your `elasticsearch.yml` file similar to what is shown below. If you're using {{ece}} or {{ech}}, and you're using machine learning or a deployment with hot-warm architecture, you must include this configuration in the user settings section for each node type. - ::::{note} + ::::{note} The values used below are meant to be an example and are not intended to apply to every use case. The details below the configuration snippet provide insights and suggestions to help you pick the proper values, depending on your OP configuration. :::: ```yaml - xpack.security.authc.realms.oidc.oidc1: - order: 2 + xpack.security.authc.realms.oidc.oidc1: + order: 2 rp.client_id: "the_client_id" - rp.response_type: code - rp.redirect_uri: "https://kibana.example.org:5601/api/security/oidc/callback" - op.issuer: "https://op.example.org" - op.authorization_endpoint: "https://op.example.org/oauth2/v1/authorize" - op.token_endpoint: "https://op.example.org/oauth2/v1/token" - op.jwkset_path: oidc/jwkset.json - op.userinfo_endpoint: "https://op.example.org/oauth2/v1/userinfo" - op.endsession_endpoint: "https://op.example.org/oauth2/v1/logout" - rp.post_logout_redirect_uri: "https://kibana.example.org:5601/security/logged_out" - claims.principal: sub - claims.groups: "http://example.info/claims/groups" + rp.response_type: code + rp.redirect_uri: "https://kibana.example.org:5601/api/security/oidc/callback" + op.issuer: "https://op.example.org" + op.authorization_endpoint: "https://op.example.org/oauth2/v1/authorize" + op.token_endpoint: "https://op.example.org/oauth2/v1/token" + op.jwkset_path: oidc/jwkset.json + op.userinfo_endpoint: "https://op.example.org/oauth2/v1/userinfo" + op.endsession_endpoint: "https://op.example.org/oauth2/v1/logout" + rp.post_logout_redirect_uri: "https://kibana.example.org:5601/security/logged_out" + claims.principal: sub + claims.groups: "http://example.info/claims/groups" ``` ::::{dropdown} Common settings @@ -139,9 +139,9 @@ This realm has a few mandatory settings, and a number of optional settings. The If your OpenID Connect Provider doesn’t publish its JWKS at an https URL, or if you want to use a local copy, you can upload the JWKS as a file. - :::{tip} - * In self-managed clusters, the specified path is resolved relative to the {{es}} config directory. {{es}} will automatically monitor this file for changes and will reload the configuration whenever it is updated. - * If you're using {{ece}} or {{ech}}, then you must [upload this file as a custom bundle](/deploy-manage/deploy/elastic-cloud/upload-custom-plugins-bundles.md) before it can be referenced. + :::{tip} + * In self-managed clusters, the specified path is resolved relative to the {{es}} config directory. {{es}} will automatically monitor this file for changes and will reload the configuration whenever it is updated. + * If you're using {{ece}} or {{ech}}, then you must [upload this file as a custom bundle](/deploy-manage/deploy/elastic-cloud/upload-custom-plugins-bundles.md) before it can be referenced. * If you're using {{eck}}, then install the file as a [custom configuration file](/deploy-manage/deploy/cloud-on-k8s/custom-configuration-files-plugins.md#use-a-volume-and-volume-mount-together-with-a-configmap-or-secret). ::: @@ -168,7 +168,7 @@ This realm has a few mandatory settings, and a number of optional settings. The In {{ech}} and {{ece}}, after you configure Client Secret, any attempt to restart the deployment will fail until you complete the rest of the configuration steps. If you want to roll back the Active Directory realm configurations, you need to remove the `xpack.security.authc.realms.oidc.oidc1.rp.client_secret` that was just added. ::: -::::{note} +::::{note} According to the OpenID Connect specification, the OP should also make their configuration available at a well known URL, which is the concatenation of their `Issuer` value with the `.well-known/openid-configuration` string. For example: `https://op.org.com/.well-known/openid-configuration`. That document should contain all the necessary information to configure the OpenID Connect realm in {{es}}. @@ -176,17 +176,17 @@ That document should contain all the necessary information to configure the Open ## Map claims [oidc-claims-mappings] -When authenticating to {{kib}} using OpenID Connect, the OP will provide information about the user in the form of **OpenID Connect Claims**. These claims can be included either in the ID Token, or be retrieved from the UserInfo endpoint of the OP. +When authenticating to {{kib}} using OpenID Connect, the OP will provide information about the user in the form of **OpenID Connect Claims**. These claims can be included either in the ID Token, or be retrieved from the UserInfo endpoint of the OP. -An **OpenID Connect Claim** is a piece of information asserted by the OP for the authenticated user, and consists of a name/value pair that contains information about the user. +An **OpenID Connect Claim** is a piece of information asserted by the OP for the authenticated user, and consists of a name/value pair that contains information about the user. -**OpenID Connect Scopes** are identifiers that are used to request access to specific lists of claims. The standard defines a set of scope identifiers that can be requested. +**OpenID Connect Scopes** are identifiers that are used to request access to specific lists of claims. The standard defines a set of scope identifiers that can be requested. * **Mandatory scopes**: `openid` * **Commonly used scopes**: * `profile`: Requests access to the `name`,`family_name`,`given_name`,`middle_name`,`nickname`, `preferred_username`,`profile`,`picture`,`website`,`gender`,`birthdate`,`zoneinfo`,`locale`, and `updated_at` claims. - * `email`: Requests access to the `email` and `email_verified` claims. + * `email`: Requests access to the `email` and `email_verified` claims. The RP requests specific scopes during the authentication request. If the OP Privacy Policy allows it and the authenticating user consents to it, the related claims are returned to the RP (either in the ID Token or as a UserInfo response). @@ -194,7 +194,7 @@ The list of the supported claims will vary depending on the OP you are using, bu ### How claims appear in user metadata [oidc-user-metadata] -By default, users who authenticate through OpenID Connect have additional metadata fields. These fields include every OpenID claim that is provided in the authentication response, regardless of whether it is mapped to an {{es}} user property. +By default, users who authenticate through OpenID Connect have additional metadata fields. These fields include every OpenID claim that is provided in the authentication response, regardless of whether it is mapped to an {{es}} user property. For example, in the metadata field `oidc(claim_name)`, "claim_name" is the name of the claim as it was contained in the ID Token or in the User Info response. Note that these will include all the [ID Token claims](https://openid.net/specs/openid-connect-core-1_0.html#IDToken) that pertain to the authentication event, rather than the user themselves. @@ -207,29 +207,29 @@ The goal of claims mapping is to configure {{es}} in such a way as to be able to To configure claims mapping: -1. Using your OP configuration, identify the claims that it might support. - +1. Using your OP configuration, identify the claims that it might support. + The list provided in the OP’s metadata or in the configuration page of the OP is a list of potentially supported claims. However, for privacy reasons it might not be a complete one, or not all supported claims will be available for all authenticated users. 2. Review the list of [user properties](#oidc-user-properties) that {{es}} supports, and decide which of them are useful to you, and can be provided by your OP in the form of claims. At a minimum, the `principal` user property is required. -3. Configure your OP to "release" those claims to your {{stack}} Relying Party. This process greatly varies by provider. You can use a static configuration while others will support that the RP requests the scopes that correspond to the claims to be "released" on authentication time. See [`rp.requested_scopes`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#ref-oidc-settings) for details about how to configure the scopes to request. To ensure interoperability and minimize the errors, you should only request scopes that the OP supports, and that you intend to map to {{es}} user properties. +3. Configure your OP to "release" those claims to your {{stack}} Relying Party. This process greatly varies by provider. You can use a static configuration while others will support that the RP requests the scopes that correspond to the claims to be "released" on authentication time. See [`rp.requested_scopes`](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#ref-oidc-settings) for details about how to configure the scopes to request. To ensure interoperability and minimize the errors, you should only request scopes that the OP supports, and that you intend to map to {{es}} user properties. :::{note} You can only map claims with values that are strings, numbers, Boolean values, or an array consisting of strings, numbers, and Boolean values. ::: -4. Configure the OpenID Connect realm in {{es}} to associate the [{{es}} user properties](#oidc-user-properties) to the name of the claims that your OP will release. - +4. Configure the OpenID Connect realm in {{es}} to associate the [{{es}} user properties](#oidc-user-properties) to the name of the claims that your OP will release. + The [sample configuration](#oidc-create-realm) configures the `principal` and `groups` user properties as follows: - * `claims.principal: sub`: Instructs {{es}} to look for the OpenID Connect claim named `sub` in the ID Token that the OP issued for the user (or in the UserInfo response) and assign the value of this claim to the `principal` user property. - + * `claims.principal: sub`: Instructs {{es}} to look for the OpenID Connect claim named `sub` in the ID Token that the OP issued for the user (or in the UserInfo response) and assign the value of this claim to the `principal` user property. + `sub` is a commonly used claim for the principal property as it is an identifier of the user in the OP and it is also a required claim of the ID Token. This means that `sub` is available in most OPs. However, the OP may provide another claim that is a better fit for your needs. - * `claims.groups: "http://example.info/claims/groups"`: Instructs {{es}} to look for the claim with the name `http://example.info/claims/groups`, either in the ID Token or in the UserInfo response, and map the value(s) of it to the user property `groups` in {{es}}. - + * `claims.groups: "http://example.info/claims/groups"`: Instructs {{es}} to look for the claim with the name `http://example.info/claims/groups`, either in the ID Token or in the UserInfo response, and map the value(s) of it to the user property `groups` in {{es}}. + There is no standard claim in the specification that is used for expressing roles or group memberships of the authenticated user in the OP, so the name of the claim that should be mapped here will vary between providers. Consult your OP documentation for more details. :::{tip} - In this example, the value is a URI, treated as a string and not a URL pointing to a location that will be retrieved. + In this example, the value is a URI, treated as a string and not a URL pointing to a location that will be retrieved. ::: @@ -241,7 +241,7 @@ The {{es}} OpenID Connect realm can be configured to map OpenID Connect claims t principal : *(Required)* This is the username that will be applied to a user that authenticates against this realm. The `principal` appears in places such as the {{es}} audit logs. -::::{note} +::::{note} If the principal property fails to be mapped from a claim, the authentication fails. :::: @@ -285,7 +285,7 @@ In this case, the user’s `principal` is mapped from the `email_verified` claim In this example, the email address must belong to the `staff.example.com` domain, and then the local-part (anything before the `@`) is used as the principal. Any users who try to login using a different email domain will fail because the regular expression will not match against their email address, and thus their principal user property - which is mandatory - will not be populated. -::::{important} +::::{important} Small mistakes in these regular expressions can have significant security consequences. For example, if we accidentally left off the trailing `$` from the example above, then we would match any email address where the domain starts with `staff.example.com`, and this would accept an email address such as `admin@staff.example.com.attacker.net`. It is important that you make sure your regular expressions are as precise as possible so that you don't open an avenue for user impersonation attacks. :::: @@ -303,9 +303,9 @@ The OpenID Connect realm in {{es}} supports RP-initiated logout functionality as In this process, the OpenID Connect RP (the {{stack}} in this case) will redirect the user’s browser to predefined URL of the OP after successfully completing a local logout. The OP can then logout the user also, depending on the configuration, and should finally redirect the user back to the RP. -RP-initiated logout is controlled by two settings: +RP-initiated logout is controlled by two settings: -* `op.endsession_endpoint`: The URL in the OP that the browser will be redirected to. +* `op.endsession_endpoint`: The URL in the OP that the browser will be redirected to. * `rp.post_logout_redirect_uri` The URL to redirect the user back to after the OP logs them out. When configuring `rp.post_logout_redirect_uri`, do not point to a URL that will trigger re-authentication of the user. For instance, when using OpenID Connect to support single sign-on to {{kib}}, this could be set to either `${kibana-url}/security/logged_out`, which will show a user-friendly message to the user, or `${kibana-url}/login?msg=LOGGED_OUT` which will take the user to the login selector in {{kib}}. @@ -315,7 +315,7 @@ When configuring `rp.post_logout_redirect_uri`, do not point to a URL that will OpenID Connect depends on TLS to provide security properties such as encryption in transit and endpoint authentication. The RP is required to establish back-channel communication with the OP in order to exchange the code for an ID Token during the Authorization code grant flow and in order to get additional user information from the `UserInfo` endpoint. If you configure `op.jwks_path` as a URL, {{es}} will need to get the OP’s signing keys from the file hosted there. As such, it is important that {{es}} can validate and trust the server certificate that the OP uses for TLS. Because the system trust store is used for the client context of outgoing https connections, if your OP is using a certificate from a trusted CA, no additional configuration is needed. -However, if the issuer of your OP’s certificate is not trusted by the JVM on which {{es}} is running (e.g it uses an organization CA), then you must configure {{es}} to trust that CA. +However, if the issuer of your OP’s certificate is not trusted by the JVM on which {{es}} is running (e.g it uses an organization CA), then you must configure {{es}} to trust that CA. If you're using {{ech}} or {{ece}}, then you must [upload your certificate as a custom bundle](/deploy-manage/deploy/elastic-cloud/upload-custom-plugins-bundles.md) before it can be referenced. @@ -336,15 +336,15 @@ When a user authenticates using OpenID Connect, they are identified to the {{sta Your OpenID Connect users can't do anything until they are assigned roles. You can map roles This can be done through either the [add role mapping API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-put-role-mapping) or with [authorization realms](/deploy-manage/users-roles/cluster-or-deployment-auth/realm-chains.md#authorization_realms). -You can map LDAP groups to roles in the following ways: +You can map LDAP groups to roles in the following ways: * Using the role mappings page in {{kib}}. -* Using the [role mapping API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-put-role-mapping). +* Using the [role mapping API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-put-role-mapping). * By delegating authorization to another realm. - + For more information, see [Mapping users and groups to roles](/deploy-manage/users-roles/cluster-or-deployment-auth/mapping-users-groups-to-roles.md). -::::{note} +::::{note} You can't use [role mapping files](/deploy-manage/users-roles/cluster-or-deployment-auth/mapping-users-groups-to-roles.md#mapping-roles-file) to grant roles to users authenticating using OpenID Connect. :::: @@ -440,14 +440,14 @@ xpack.security.authc.providers: The configuration values used in the example above are: `xpack.security.authc.providers` -: Add an `oidc` provider to instruct {{kib}} to use OpenID Connect single sign-on as the authentication method. This instructs Kibana to attempt to initiate an SSO flow every time a user attempts to access a URL in {{kib}}, if the user is not already authenticated. +: Add an `oidc` provider to instruct {{kib}} to use OpenID Connect single sign-on as the authentication method. This instructs Kibana to attempt to initiate an SSO flow every time a user attempts to access a URL in {{kib}}, if the user is not already authenticated. `xpack.security.authc.providers.oidc..realm` : The name of the OpenID Connect realm in {{es}} that should handle authentication for this Kibana instance. ### Supporting OIDC and basic authentication in {{kib}} -If you also want to allow users to log in with a username and password, you must enable the `basic` authentication provider too. This will allow users that haven’t already authenticated with OpenID Connect to log in using the {{kib}} login form: +If you also want to allow users to log in with a username and password, you must enable the `basic` authentication provider too. This will allow users that haven’t already authenticated with OpenID Connect to log in using the {{kib}} login form: ```yaml xpack.security.authc.providers: @@ -468,7 +468,7 @@ The OpenID Connect realm is designed to allow users to authenticate to {{kib}}. Single sign-on realms such as OpenID Connect and SAML make use of the Token Service in {{es}} and in principle exchange a SAML or OpenID Connect Authentication response for an {{es}} access token and a refresh token. The access token is used as credentials for subsequent calls to {{es}}. The refresh token enables the user to get new {{es}} access tokens after the current one expires. -::::{note} +::::{note} The {{es}} Token Service can be seen as a minimal oAuth2 authorization server and the access token and refresh token mentioned above are tokens that pertain *only* to this authorization server. They are generated and consumed *only* by {{es}} and are in no way related to the tokens ( access token and ID Token ) that the OpenID Connect Provider issues. :::: diff --git a/deploy-manage/users-roles/cluster-or-deployment-auth/operator-only-functionality.md b/deploy-manage/users-roles/cluster-or-deployment-auth/operator-only-functionality.md index 03c6f9c33..56a26cfb9 100644 --- a/deploy-manage/users-roles/cluster-or-deployment-auth/operator-only-functionality.md +++ b/deploy-manage/users-roles/cluster-or-deployment-auth/operator-only-functionality.md @@ -3,9 +3,9 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/operator-only-functionality.html applies_to: deployment: - ess: - ece: - eck: + ess: + ece: + eck: --- # Operator-only functionality [operator-only-functionality] @@ -34,7 +34,7 @@ Operator privileges provide protection for APIs and dynamic cluster settings. An ## Operator-only dynamic cluster settings [operator-only-dynamic-cluster-settings] * All [IP filtering](../../security/ip-traffic-filtering.md) settings -* The following dynamic [machine learning settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/machine-learning-settings.md): +* The following dynamic [machine learning settings](elasticsearch://reference/elasticsearch/configuration-reference/machine-learning-settings.md): * `xpack.ml.node_concurrent_job_allocations` * `xpack.ml.max_machine_memory_percent` @@ -46,8 +46,8 @@ Operator privileges provide protection for APIs and dynamic cluster settings. An * `xpack.ml.enable_config_migration` * `xpack.ml.persist_results_max_retries` -* The [`cluster.routing.allocation.disk.threshold_enabled` setting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#cluster-routing-disk-threshold) -* The following [recovery settings for managed services](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/index-recovery-settings.md#recovery-settings-for-managed-services): +* The [`cluster.routing.allocation.disk.threshold_enabled` setting](elasticsearch://reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#cluster-routing-disk-threshold) +* The following [recovery settings for managed services](elasticsearch://reference/elasticsearch/configuration-reference/index-recovery-settings.md#recovery-settings-for-managed-services): * `node.bandwidth.recovery.operator.factor` * `node.bandwidth.recovery.operator.factor.read` diff --git a/deploy-manage/users-roles/cluster-or-deployment-auth/pki.md b/deploy-manage/users-roles/cluster-or-deployment-auth/pki.md index 5467e38bb..3574f6119 100644 --- a/deploy-manage/users-roles/cluster-or-deployment-auth/pki.md +++ b/deploy-manage/users-roles/cluster-or-deployment-auth/pki.md @@ -19,7 +19,7 @@ You can also use PKI certificates to authenticate to {{kib}}, however this requi To use PKI in {{es}}, you configure a PKI realm, enable client authentication on the desired network layers (transport or http), and map the Distinguished Names (DNs) from the `Subject` field in the user certificates to roles. You create the mappings in a role mapping file or use the role mappings API. -1. Add a realm configuration for a `pki` realm to `elasticsearch.yml` under the `xpack.security.authc.realms.pki` namespace. You must explicitly set the `order` attribute. See [PKI realm settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#ref-pki-settings) for all of the options you can set for a `pki` realm. +1. Add a realm configuration for a `pki` realm to `elasticsearch.yml` under the `xpack.security.authc.realms.pki` namespace. You must explicitly set the `order` attribute. See [PKI realm settings](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#ref-pki-settings) for all of the options you can set for a `pki` realm. For example, the following snippet shows the most basic `pki` realm configuration: @@ -35,11 +35,11 @@ To use PKI in {{es}}, you configure a PKI realm, enable client authentication on With this configuration, any certificate trusted by the {{es}} SSL/TLS layer is accepted for authentication. The username is the common name (CN) extracted from the DN in the Subject field of the end-entity certificate. This configuration is not sufficient to permit PKI authentication to {{kib}}; additional steps are required. - ::::{important} + ::::{important} When you configure realms in `elasticsearch.yml`, only the realms you specify are used for authentication. If you also want to use the `native` or `file` realms, you must include them in the realm chain. :::: -2. Optional: The username is defined by the [username_pattern](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#ref-pki-settings). If you want to use something other than the CN of the Subject DN as the username, you can specify a regex to extract the desired username. The regex is applied on the Subject DN. +2. Optional: The username is defined by the [username_pattern](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#ref-pki-settings). If you want to use something other than the CN of the Subject DN as the username, you can specify a regex to extract the desired username. The regex is applied on the Subject DN. For example, the regex in the following configuration extracts the email address from the Subject DN: @@ -54,16 +54,16 @@ To use PKI in {{es}}, you configure a PKI realm, enable client authentication on username_pattern: "EMAILADDRESS=(.*?)(?:,|$)" ``` - ::::{note} + ::::{note} If the regex is too restrictive and does not match the Subject DN of the client’s certificate, then the realm does not authenticate the certificate. :::: 3. Optional: If you want the same users to also be authenticated using certificates when they connect to {{kib}}, you must configure the {{es}} PKI realm to allow delegation. See [PKI authentication for clients connecting to {{kib}}](#pki-realm-for-proxied-clients). 4. Restart {{es}} because realm configuration is not reloaded automatically. If you’re following through with the next steps, you might wish to hold the restart for last. 5. If you're using a self-managed cluster, then [enable SSL/TLS](../../security/secure-cluster-communications.md#encrypt-internode-communication). -6. If you're using a self-managed cluster or {{eck}}, then enable client authentication on the desired network layers (transport or http). +6. If you're using a self-managed cluster or {{eck}}, then enable client authentication on the desired network layers (transport or http). - ::::{important} + ::::{important} To use PKI when clients connect directly to {{es}}, you must enable SSL/TLS with client authentication by setting `xpack.security.transport.ssl.client_authentication` and `xpack.security.http.ssl.client_authentication` to `optional` or `required`. If the setting value is `optional`, clients without certificates can authenticate with other credentials. :::: @@ -75,10 +75,10 @@ To use PKI in {{es}}, you configure a PKI realm, enable client authentication on * The interface must *trust* the certificate that is presented by the client by configuring either the `truststore` or `certificate_authorities` paths, or by setting `verification_mode` to `none`. * The *protocols* supported by the interface must be compatible with those used by the client. - For an explanation of these settings, see [General TLS settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#ssl-tls-settings). + For an explanation of these settings, see [General TLS settings](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#ssl-tls-settings). 7. Optional: Configure the PKI realm to trust a subset of certificates. - + The relevant network interface (transport or http) must be configured to trust any certificate that is to be used within the PKI realm. However, it is possible to configure the PKI realm to trust only a *subset* of the certificates accepted by the network interface. This is useful when the SSL/TLS layer trusts clients with certificates that are signed by a different CA than the one that signs your users' certificates. 1. To configure the PKI realm with its own trust store, specify the `truststore.path` option. The path must be located within the {{es}} configuration directory (`ES_PATH_CONF`). For example: @@ -95,10 +95,10 @@ To use PKI in {{es}}, you configure a PKI realm, enable client authentication on path: "pki1_truststore.jks" ``` - :::{tip} + :::{tip} If you're using {{ece}} or {{ech}}, then you must [upload this file as a custom bundle](/deploy-manage/deploy/elastic-cloud/upload-custom-plugins-bundles.md) before it can be referenced. - If you're using {{eck}}, then install the file as a [custom configuration file](/deploy-manage/deploy/cloud-on-k8s/custom-configuration-files-plugins.md#use-a-volume-and-volume-mount-together-with-a-configmap-or-secret). + If you're using {{eck}}, then install the file as a [custom configuration file](/deploy-manage/deploy/cloud-on-k8s/custom-configuration-files-plugins.md#use-a-volume-and-volume-mount-together-with-a-configmap-or-secret). If you're using a self-managed cluster, then the file must be present on each node. ::: @@ -142,10 +142,10 @@ To use PKI in {{es}}, you configure a PKI realm, enable client authentication on 1. The name of a role. 2. The distinguished name (DN) of a PKI user. - :::{tip} + :::{tip} If you're using {{ece}} or {{ech}}, then you must [upload this file as a custom bundle](/deploy-manage/deploy/elastic-cloud/upload-custom-plugins-bundles.md) before it can be referenced. - If you're using {{eck}}, then install the file as a [custom configuration file](/deploy-manage/deploy/cloud-on-k8s/custom-configuration-files-plugins.md#use-a-volume-and-volume-mount-together-with-a-configmap-or-secret). + If you're using {{eck}}, then install the file as a [custom configuration file](/deploy-manage/deploy/cloud-on-k8s/custom-configuration-files-plugins.md#use-a-volume-and-volume-mount-together-with-a-configmap-or-secret). If you're using a self-managed cluster, then the file must be present on each node. ::: @@ -158,7 +158,7 @@ To use PKI in {{es}}, you configure a PKI realm, enable client authentication on For more information, see [Mapping users and groups to roles](mapping-users-groups-to-roles.md). - ::::{note} + ::::{note} The PKI realm supports [authorization realms](realm-chains.md#authorization_realms) as an alternative to role mapping. :::: diff --git a/deploy-manage/users-roles/cluster-or-deployment-auth/realm-chains.md b/deploy-manage/users-roles/cluster-or-deployment-auth/realm-chains.md index 8d4e1ee05..0faecf5e3 100644 --- a/deploy-manage/users-roles/cluster-or-deployment-auth/realm-chains.md +++ b/deploy-manage/users-roles/cluster-or-deployment-auth/realm-chains.md @@ -11,19 +11,19 @@ applies_to: # Realm chains [realm-chains] -[Realms](authentication-realms.md) live within a *realm chain*. It is essentially a prioritized list of configured realms typically of various types. Realms are consulted in ascending order: the realm with the lowest `order` value is consulted first. +[Realms](authentication-realms.md) live within a *realm chain*. It is essentially a prioritized list of configured realms typically of various types. Realms are consulted in ascending order: the realm with the lowest `order` value is consulted first. You must make sure each configured realm has a distinct `order` setting. In the event that two or more realms have the same `order`, the node will fail to start. During the authentication process, {{es}} consults and tries to authenticate the request one realm at a time. Once one of the realms successfully authenticates the request, the authentication is considered to be successful. The authenticated user is associated with the request, which then proceeds to the authorization phase. If a realm can't authenticate the request, the next realm in the chain is consulted. If none of the realms in the chain can authenticate the request, the authentication is considered to be unsuccessful and an authentication error is returned (as HTTP status code `401`). -::::{note} +::::{note} Some systems, such as Active Directory, have a temporary lock-out period after several successive failed login attempts. If the same username exists in multiple realms, unintentional account lockouts are possible. For more information, see [Users are frequently locked out of Active Directory](/troubleshoot/elasticsearch/security/trouble-shoot-active-directory.md). :::: ## Configure a realm chain -The default realm chain contains the `file` and `native` realms. To explicitly configure a realm chain, specify the chain in the `elasticsearch.yml` file. +The default realm chain contains the `file` and `native` realms. To explicitly configure a realm chain, specify the chain in the `elasticsearch.yml` file. If your realm chain does not contain `file` or `native` realm or does not disable them explicitly, `file` and `native` realms will be added automatically to the beginning of the realm chain in that order. To opt out from this automatic behavior, you can explicitly configure the `file` and `native` realms with the `order` and `enabled` settings. @@ -63,13 +63,13 @@ For example, you might want to use a PKI realm to authenticate your users with T Any realm that supports retrieving users without needing their credentials can be used as an authorization realm. Refer to [Looking up users without authentication](looking-up-users-without-authentication.md) to learn which realms can be used as authorization realms. -For realms that support this feature, it can be enabled by configuring the `authorization_realms` setting on the authenticating realm. Check the list of [supported settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#realm-settings) for each realm to see if they support the `authorization_realms` setting. +For realms that support this feature, it can be enabled by configuring the `authorization_realms` setting on the authenticating realm. Check the list of [supported settings](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#realm-settings) for each realm to see if they support the `authorization_realms` setting. If delegated authorization is enabled for a realm, it authenticates the user in its standard manner, including relevant caching, then looks for that user in the configured list of authorization realms. It tries each realm in the order they are specified in the `authorization_realms` setting. The user is retrieved by principal - the user must have identical usernames in the authentication and authorization realms. If the user can't be found in any of the authorization realms, then authentication fails. See [Configuring authorization delegation](authorization-delegation.md) for more details. -::::{note} +::::{note} Delegated authorization requires that you have a [subscription](https://www.elastic.co/subscriptions) that includes custom authentication and authorization realms. :::: diff --git a/deploy-manage/users-roles/cluster-or-deployment-auth/role-mapping-resources.md b/deploy-manage/users-roles/cluster-or-deployment-auth/role-mapping-resources.md index 69b8c9272..b4d77b9f1 100644 --- a/deploy-manage/users-roles/cluster-or-deployment-auth/role-mapping-resources.md +++ b/deploy-manage/users-roles/cluster-or-deployment-auth/role-mapping-resources.md @@ -38,7 +38,9 @@ A role mapping resource has the following properties: `except` : (object) A single rule as an object. Only valid as a child of an `all` rule. If its child is `false`, the `except` is `true`. -## Field rules [mapping-roles-rule-field] + + +## Field rules [mapping-roles-rule-field] The `field` rule is the primary building block for a role mapping expression. It takes a single object as its value and that object must contain a single member with key *F* and value *V*. The field rule looks up the value of *F* within the user object and then tests whether the user value *matches* the provided value *V*. @@ -48,12 +50,13 @@ The value specified in the field rule can be one of the following types: | --- | --- | --- | | Simple String | Exactly matches the provided value. | `"esadmin"` | | Wildcard String | Matches the provided value using a wildcard. | `"*,dc=example,dc=com"` | -| Regular Expression | Matches the provided value using a [Lucene regexp](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/regexp-syntax.md). | `"/.*-admin[0-9]*/"` | +| Regular Expression | Matches the provided value using a [Lucene regexp](elasticsearch://reference/query-languages/regexp-syntax.md). | `"/.*-admin[0-9]*/"` | | Number | Matches an equivalent numerical value. | `7` | | Null | Matches a null or missing value. | `null` | | Array | Tests each element in the array in accordance with the above definitions. If *any* of elements match, the match is successful. | `["admin", "operator"]` | -### User fields [_user_fields] + +### User fields [_user_fields] The *user object* against which rules are evaluated has the following fields: diff --git a/deploy-manage/users-roles/cluster-or-deployment-auth/role-structure.md b/deploy-manage/users-roles/cluster-or-deployment-auth/role-structure.md index c15ca9b7d..093204779 100644 --- a/deploy-manage/users-roles/cluster-or-deployment-auth/role-structure.md +++ b/deploy-manage/users-roles/cluster-or-deployment-auth/role-structure.md @@ -26,22 +26,22 @@ A role is defined by the following JSON structure: ``` 1. A list of usernames the owners of this role can [impersonate](/deploy-manage/users-roles/cluster-or-deployment-auth/submitting-requests-on-behalf-of-other-users.md). -2. A list of cluster privileges. These privileges define the cluster level actions users with this role are able to execute. - +2. A list of cluster privileges. These privileges define the cluster level actions users with this role are able to execute. + This field is optional (missing `cluster` privileges effectively mean no cluster level permissions). 3. An object defining global privileges. A global privilege is a form of cluster privilege that is request sensitive. A standard cluster privilege makes authorization decisions based solely on the action being executed. A global privilege also considers the parameters included in the request. Support for global privileges is currently limited to the management of application privileges. This field is optional. 4. A list of indices permissions entries. - + This field is optional (missing `indices` privileges effectively mean no index level permissions). 5. A list of application privilege entries. This field is optional. -6. A list of indices permissions entries for [remote clusters configured with the API key based model](/deploy-manage/remote-clusters/remote-clusters-api-key.md). - +6. A list of indices permissions entries for [remote clusters configured with the API key based model](/deploy-manage/remote-clusters/remote-clusters-api-key.md). + This field is optional (missing `remote_indices` privileges effectively mean no index level permissions for any API key based remote clusters). -7. A list of cluster permissions entries for [remote clusters configured with the API key based model](/deploy-manage/remote-clusters/remote-clusters-api-key.md). - +7. A list of cluster permissions entries for [remote clusters configured with the API key based model](/deploy-manage/remote-clusters/remote-clusters-api-key.md). + This field is optional (missing `remote_cluster` privileges effectively means no additional cluster permissions for any API key based remote clusters). -8. Metadata field associated with the role, such as `metadata.app_tag`. Metadata is internally indexed as a [flattened](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/flattened.md) field type. This means that all sub-fields act like `keyword` fields when querying and sorting. Metadata values can be simple values, but also lists and maps. This field is optional. -9. A string value with the description text of the role. The maximum length of it is `1000` chars. The field is internally indexed as a [text](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/text.md#text-field-type) field type (with default values for all parameters). This field is optional. +8. Metadata field associated with the role, such as `metadata.app_tag`. Metadata is internally indexed as a [flattened](elasticsearch://reference/elasticsearch/mapping-reference/flattened.md) field type. This means that all sub-fields act like `keyword` fields when querying and sorting. Metadata values can be simple values, but also lists and maps. This field is optional. +9. A string value with the description text of the role. The maximum length of it is `1000` chars. The field is internally indexed as a [text](elasticsearch://reference/elasticsearch/mapping-reference/text.md#text-field-type) field type (with default values for all parameters). This field is optional. ::::{note} @@ -160,7 +160,7 @@ The remote indices privileges entry has an extra mandatory `clusters` field comp } ``` -1. A list of remote cluster aliases. It supports literal strings as well as [wildcards](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#api-multi-index) and [regular expressions](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/regexp-syntax.md). This field is required. +1. A list of remote cluster aliases. It supports literal strings as well as [wildcards](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#api-multi-index) and [regular expressions](elasticsearch://reference/query-languages/regexp-syntax.md). This field is required. 2. A list of data streams, indices, and aliases to which the permissions in this entry apply. Supports wildcards (`*`). 3. The index level privileges the owners of the role have on the associated data streams and indices specified in the `names` argument. 4. Specification for document fields the owners of the role have read access to. See [Setting up field and document level security](/deploy-manage/users-roles/cluster-or-deployment-auth/controlling-access-at-document-field-level.md) for details. @@ -186,7 +186,7 @@ The following describes the structure of a remote cluster permissions entry: } ``` -1. A list of remote cluster aliases. It supports literal strings as well as [wildcards](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#api-multi-index) and [regular expressions](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/regexp-syntax.md). This field is required. +1. A list of remote cluster aliases. It supports literal strings as well as [wildcards](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#api-multi-index) and [regular expressions](elasticsearch://reference/query-languages/regexp-syntax.md). This field is required. 2. The cluster level privileges for the remote cluster. The allowed values here are a subset of the [cluster privileges](/deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-cluster). The [builtin privileges API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-get-builtin-privileges) can be used to determine which privileges are allowed here. This field is required. diff --git a/deploy-manage/users-roles/cluster-or-deployment-auth/saml.md b/deploy-manage/users-roles/cluster-or-deployment-auth/saml.md index d12d3f81c..e1314621b 100644 --- a/deploy-manage/users-roles/cluster-or-deployment-auth/saml.md +++ b/deploy-manage/users-roles/cluster-or-deployment-auth/saml.md @@ -21,13 +21,13 @@ applies_to: # SAML authentication [saml-realm] -The {{stack}} supports SAML single-sign-on (SSO) into {{kib}}, using {{es}} as a backend service. +The {{stack}} supports SAML single-sign-on (SSO) into {{kib}}, using {{es}} as a backend service. The {{security-features}} provide this support using the Web Browser SSO profile of the SAML 2.0 protocol. This protocol is specifically designed to support authentication using an interactive web browser, so it does not operate as a standard authentication realm. Instead, there are {{kib}} and {{es}} {{security-features}} that work together to enable interactive SAML sessions. This means that the SAML realm is not suitable for use by standard REST clients. If you configure a SAML realm for use in {{kib}}, you should also configure another realm, such as the [native realm](/deploy-manage/users-roles/cluster-or-deployment-auth/native.md) in your authentication chain. -Because this feature is designed with {{kib}} in mind, most sections of this guide assume {{kib}} is used. To learn how a custom web application could use the OpenID Connect REST APIs to authenticate the users to {{es}} with SAML, refer to [SAML without {{kib}}](#saml-no-kibana). +Because this feature is designed with {{kib}} in mind, most sections of this guide assume {{kib}} is used. To learn how a custom web application could use the OpenID Connect REST APIs to authenticate the users to {{es}} with SAML, refer to [SAML without {{kib}}](#saml-no-kibana). The SAML support in {{kib}} is designed with the expectation that it will be the primary (or sole) authentication method for users of that {{kib}} instance. After you enable SAML authentication in {{kib}}, it will affect all users who try to login. The [Configuring {{kib}}](/deploy-manage/users-roles/cluster-or-deployment-auth/saml.md#saml-configure-kibana) section provides more detail about how this works. @@ -47,7 +47,7 @@ Additional steps outlined in this document are optional. :::: ::::{tip} -This topic describes implementing SAML SSO at the deployment or cluster level, for the purposes of authenticating with a {{kib}} instance. +This topic describes implementing SAML SSO at the deployment or cluster level, for the purposes of authenticating with a {{kib}} instance. Depending on your deployment type, you can also configure SSO for the following use cases: @@ -59,7 +59,7 @@ Depending on your deployment type, you can also configure SSO for the following In SAML terminology, the {{stack}} is operating as a *Service Provider*. -The other component that is needed to enable SAML single-sign-on is the *Identity Provider*, which is a service that handles your credentials and performs that actual authentication of users. +The other component that is needed to enable SAML single-sign-on is the *Identity Provider*, which is a service that handles your credentials and performs that actual authentication of users. If you are interested in configuring SSO into {{kib}}, then you need to provide {{es}} with information about your *Identity Provider*, and you will need to register the {{stack}} as a known *Service Provider* within that Identity Provider. There are also a few configuration changes that are required in {{kib}} to activate the SAML authentication provider. @@ -89,13 +89,13 @@ The {{stack}} requires that all messages from the IdP are signed. For authentica Before you set up SAML single-sign on, you must have a [SAML IdP](#saml-guide-idp) configured. -If you're using a self-managed cluster, then perform the following additional steps: +If you're using a self-managed cluster, then perform the following additional steps: * Enable TLS for HTTP. If your {{es}} cluster is operating in production mode, you must configure the HTTP interface to use SSL/TLS before you can enable SAML authentication. For more information, see [Encrypt HTTP client communications for {{es}}](/deploy-manage/security/set-up-basic-security-plus-https.md#encrypt-http-communication). - If you started {{es}} [with security enabled](/deploy-manage/deploy/self-managed/installing-elasticsearch.md), then TLS is already enabled for HTTP. + If you started {{es}} [with security enabled](/deploy-manage/deploy/self-managed/installing-elasticsearch.md), then TLS is already enabled for HTTP. {{ech}}, {{ece}}, and {{eck}} have TLS enabled by default. @@ -113,11 +113,11 @@ If you're using a self-managed cluster, then perform the following additional st SAML authentication is enabled by configuring a SAML realm within the authentication chain for {{es}}. -This realm has a few mandatory settings, and a number of optional settings. The available settings are described in detail in [Security settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md): -* [SAML realm settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#ref-saml-settings) -* [SAML realm signing settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#ref-saml-signing-settings) -* [SAML realm encryption settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#ref-saml-encryption-settings) -* [SAML realm SSL settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#ref-saml-ssl-settings) +This realm has a few mandatory settings, and a number of optional settings. The available settings are described in detail in [Security settings](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md): +* [SAML realm settings](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#ref-saml-settings) +* [SAML realm signing settings](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#ref-saml-signing-settings) +* [SAML realm encryption settings](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#ref-saml-encryption-settings) +* [SAML realm SSL settings](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#ref-saml-ssl-settings) This guide will walk you through the most common settings. @@ -148,13 +148,13 @@ order If you're using {{eck}}, then make sure not to disable Elasticsearch’s file realm set by ECK, as ECK relies on the file realm for its operation. Set the `order` setting of the SAML realm to a greater value than the `order` value set for the file and native realms, which is by default -100 and -99 respectively. idp.metadata.path -: $$$idp-metadata-path$$$ The path to the metadata file for your Identity Provider. The metadata file path can either be a path, or an HTTPS URL. +: $$$idp-metadata-path$$$ The path to the metadata file for your Identity Provider. The metadata file path can either be a path, or an HTTPS URL. - :::{tip} - If you want to pass a file path, then review the following: - - * File path settings are resolved relative to the {{es}} config directory. {{es}} will automatically monitor this file for changes and will reload the configuration whenever it is updated. - * If you're using {{ece}} or {{ech}}, then you must upload the file [as a custom bundle](/deploy-manage/deploy/elastic-cloud/upload-custom-plugins-bundles.md) before it can be referenced. + :::{tip} + If you want to pass a file path, then review the following: + + * File path settings are resolved relative to the {{es}} config directory. {{es}} will automatically monitor this file for changes and will reload the configuration whenever it is updated. + * If you're using {{ece}} or {{ech}}, then you must upload the file [as a custom bundle](/deploy-manage/deploy/elastic-cloud/upload-custom-plugins-bundles.md) before it can be referenced. * If you're using {{eck}}, then install the file as [custom configuration files](/deploy-manage/deploy/cloud-on-k8s/custom-configuration-files-plugins.md#use-a-volume-and-volume-mount-together-with-a-configmap-or-secret). ::: @@ -212,8 +212,8 @@ The recommended steps for configuring these SAML attributes are as follows: 1. Consult your IdP to see what user attributes it can provide. This varies greatly between providers, but you should be able to obtain a list from the documentation, or from your local admin. 2. Review the list of [user properties](#saml-es-user-properties) that {{es}} supports, and decide which of them are useful to you, and can be provided by your IdP. At a *minimum*, the `principal` attribute is required. -3. Configure your IdP to "release" those attributes to your {{kib}} SAML service provider. This process varies by provider: some will provide a user interface for this, while others may require that you edit configuration files. - +3. Configure your IdP to "release" those attributes to your {{kib}} SAML service provider. This process varies by provider: some will provide a user interface for this, while others may require that you edit configuration files. + Because {{es}} does not require that any specific URIs are used, you can use any URIs to use for each attribute as they are recommended by the IDP or your local administrator. 4. Configure the SAML realm in {{es}} to associate the [{{es}} user properties](#saml-es-user-properties) to the URIs that you configured in your IdP. The [sample configuration](#saml-create-realm) configures the `principal` and `groups` attributes. @@ -228,7 +228,7 @@ In general, {{es}} expects that the configured value for an attribute will be a : This uses the SAML `NameID` value (all leading and trailing whitespace removed), but only if the NameID format is `urn:oasis:names:tc:SAML:2.0:nameid-format:persistent`. A SAML `NameID` element has an optional `Format` attribute that indicates the semantics of the provided name. It is common for IdPs to be configured with "transient" NameIDs that present a new identifier for each session. Since it is rarely useful to use a transient NameID as part of an attribute mapping, the "nameid:persistent" attribute name can be used as a safety mechanism that will cause an error if you attempt to map from a `NameID` that does not have a persistent value. :::{note} -Identity Providers can be either statically configured to release a `NameID` with a specific format, or they can be configured to try to conform with the requirements of the SP. The SP declares its requirements as part of the Authentication Request, using an element which is called the `NameIDPolicy`. If this is needed, you can set the relevant [settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#ref-saml-settings) named `nameid_format` in order to request that the IdP releases a `NameID` with a specific format. +Identity Providers can be either statically configured to release a `NameID` with a specific format, or they can be configured to try to conform with the requirements of the SP. The SP declares its requirements as part of the Authentication Request, using an element which is called the `NameIDPolicy`. If this is needed, you can set the relevant [settings](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#ref-saml-settings) named `nameid_format` in order to request that the IdP releases a `NameID` with a specific format. ::: *friendlyName* @@ -304,7 +304,7 @@ It is sometimes necessary for a SAML SP to be able to impose specific restrictio The SAML SP defines a set of Authentication Context Class Reference values, which describe the restrictions to be imposed on the IdP, and sends these in the Authentication Request. The IdP attempts to grant these restrictions. If it cannot grant them, the authentication attempt fails. If the user is successfully authenticated, the Authentication Statement of the SAML Response contains an indication of the restrictions that were satisfied. -You can define the Authentication Context Class Reference values by using the `req_authn_context_class_ref` option in the SAML realm configuration. See [SAML realm settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#ref-saml-settings). +You can define the Authentication Context Class Reference values by using the `req_authn_context_class_ref` option in the SAML realm configuration. See [SAML realm settings](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#ref-saml-settings). {{es}} supports only the `exact` comparison method for the Authentication Context. When it receives the Authentication Response from the IdP, {{es}} examines the value of the Authentication Context Class Reference that is part of the Authentication Statement of the SAML Assertion. If it matches one of the requested values, the authentication is considered successful. Otherwise, the authentication attempt fails. @@ -364,7 +364,7 @@ You can configure {{es}} for signing, encryption or both, using a single key or The {{stack}} uses X.509 certificates with RSA private keys for SAML cryptography. These keys can be generated using any standard SSL tool, including the `elasticsearch-certutil` tool. -Your IdP may require that the {{stack}} have a cryptographic key for signing SAML messages, and that you provide the corresponding signing certificate within the Service Provider configuration (either within the {{stack}} SAML metadata file, or manually configured within the IdP administration interface). +Your IdP may require that the {{stack}} have a cryptographic key for signing SAML messages, and that you provide the corresponding signing certificate within the Service Provider configuration (either within the {{stack}} SAML metadata file, or manually configured within the IdP administration interface). While most IdPs do not expect authentication requests to be signed, it is commonly the case that signatures are required for logout requests. Your IdP will validate these signatures against the signing certificate that has been configured for the {{stack}} Service Provider. @@ -372,7 +372,7 @@ Encryption certificates are rarely needed, but the {{stack}} supports them for c ### Generate certificates and keys [_generating_certificates_and_keys] -{{es}} supports certificates and keys in either PEM, PKCS#12 or JKS format. Some Identity Providers are more restrictive in the formats they support, and will require you to provide the certificates as a file in a particular format. You should consult the documentation for your IdP to determine what formats they support. +{{es}} supports certificates and keys in either PEM, PKCS#12 or JKS format. Some Identity Providers are more restrictive in the formats they support, and will require you to provide the certificates as a file in a particular format. You should consult the documentation for your IdP to determine what formats they support. #### Example: Using `openssl` @@ -387,7 +387,7 @@ deployment: self: ``` -Using the [`elasticsearch-certutil` tool](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/certutil.md), you can generate a signing certificate with the following command. Because PEM format is the most commonly supported format, the example generates a certificate in that format. +Using the [`elasticsearch-certutil` tool](elasticsearch://reference/elasticsearch/command-line-tools/certutil.md), you can generate a signing certificate with the following command. Because PEM format is the most commonly supported format, the example generates a certificate in that format. ```sh bin/elasticsearch-certutil cert --self-signed --pem --days 1100 --name saml-sign --out saml-sign.zip @@ -414,8 +414,8 @@ Encryption certificates can be generated with the same process. By default, {{es}} will sign *all* outgoing SAML messages if a signing key has been configured. -:::{tip} -* In self-managed clusters, file path settings is resolved relative to the {{es}} config directory. {{es}} will automatically monitor this file for changes and will reload the configuration whenever it is updated. +:::{tip} +* In self-managed clusters, file path settings is resolved relative to the {{es}} config directory. {{es}} will automatically monitor this file for changes and will reload the configuration whenever it is updated. * If you're using {{ece}} or {{ech}}, then you must upload any certificate or keystore files [as a custom bundle](/deploy-manage/deploy/elastic-cloud/upload-custom-plugins-bundles.md) before it can be referenced. You can add this file to your existing SAML bundle. * If you're using {{eck}}, then install the files as [custom configuration files](/deploy-manage/deploy/cloud-on-k8s/custom-configuration-files-plugins.md#use-a-volume-and-volume-mount-together-with-a-configmap-or-secret). ::: @@ -478,8 +478,8 @@ The {{es}} {{security-features}} support a single key for message decryption. If If an `Assertion` contains both encrypted and plain-text attributes, then failure to decrypt the encrypted attributes will not cause an automatic rejection. Rather, {{es}} processes the available plain-text attributes (and any `EncryptedAttributes` that could be decrypted). -:::{tip} -* In self-managed clusters, file path settings is resolved relative to the {{es}} config directory. {{es}} will automatically monitor this file for changes and will reload the configuration whenever it is updated. +:::{tip} +* In self-managed clusters, file path settings is resolved relative to the {{es}} config directory. {{es}} will automatically monitor this file for changes and will reload the configuration whenever it is updated. * If you're using {{ece}} or {{ech}}, then you must upload any certificate or keystore files [as a custom bundle](/deploy-manage/deploy/elastic-cloud/upload-custom-plugins-bundles.md) before it can be referenced. You can add this file to your existing SAML bundle. * If you're using {{eck}}, then install the files as [custom configuration files](/deploy-manage/deploy/cloud-on-k8s/custom-configuration-files-plugins.md#use-a-volume-and-volume-mount-together-with-a-configmap-or-secret). ::: @@ -519,7 +519,7 @@ If you want to use **PKCS#12 formatted** files or a **Java Keystore** for SAML e Some Identity Providers support importing a metadata file from the Service Provider. This will automatically configure many of the integration options between the IdP and the SP. -The {{stack}} supports generating such a metadata file using the [SAML service provider metadata API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-saml-service-provider-metadata) or the [`bin/elasticsearch-saml-metadata` command](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/saml-metadata.md). +The {{stack}} supports generating such a metadata file using the [SAML service provider metadata API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-saml-service-provider-metadata) or the [`bin/elasticsearch-saml-metadata` command](elasticsearch://reference/elasticsearch/command-line-tools/saml-metadata.md). ### Using the SAML service provider metadata API @@ -531,7 +531,7 @@ curl -u user_name:password -X GET http://localhost:9200/_security/saml/metadata ### Using the `elasticsearch-saml-metadata` command -You can generate the SAML metadata by running the [`bin/elasticsearch-saml-metadata` command](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/saml-metadata.md). +You can generate the SAML metadata by running the [`bin/elasticsearch-saml-metadata` command](elasticsearch://reference/elasticsearch/command-line-tools/saml-metadata.md). ```{applies_to} deployment: @@ -566,17 +566,17 @@ When a user authenticates using SAML, they are identified to the {{stack}}, but Your SAML users cannot do anything until they are assigned roles. This can be done through either the [add role mapping API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-put-role-mapping) or with [authorization realms](/deploy-manage/users-roles/cluster-or-deployment-auth/realm-chains.md#authorization_realms). -You can map SAML users to roles in the following ways: +You can map SAML users to roles in the following ways: * Using the role mappings page in {{kib}}. -* Using the [role mapping API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-put-role-mapping). +* Using the [role mapping API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-put-role-mapping). * By delegating authorization to another realm. ::::{note} You can't use [role mapping files](/deploy-manage/users-roles/cluster-or-deployment-auth/mapping-users-groups-to-roles.md#mapping-roles-file) to grant roles to users authenticating using SAML. :::: -### Example: Using the role mapping API +### Example: Using the role mapping API This is an example of a simple role mapping that grants the `example_role` role to any user who authenticates against the `saml1` realm: @@ -628,10 +628,10 @@ PUT /_security/role_mapping/saml-finance If your users also exist in a repository that can be directly accessed by {{es}} (such as an LDAP directory) then you can use [authorization realms](/deploy-manage/users-roles/cluster-or-deployment-auth/realm-chains.md#authorization_realms) instead of role mappings. -In this case, you perform the following steps: +In this case, you perform the following steps: -1. In your SAML realm, assigned a SAML attribute to act as the lookup userid, by configuring the `attributes.principal` setting. -2. Create a new realm that can lookup users from your local repository (e.g. an `ldap` realm) +1. In your SAML realm, assigned a SAML attribute to act as the lookup userid, by configuring the `attributes.principal` setting. +2. Create a new realm that can lookup users from your local repository (e.g. an `ldap` realm) 3. In your SAML realm, set `authorization_realms` to the name of the realm you created in step 2. ## Configure {{kib}} [saml-configure-kibana] diff --git a/deploy-manage/users-roles/cluster-or-deployment-auth/service-accounts.md b/deploy-manage/users-roles/cluster-or-deployment-auth/service-accounts.md index 941a67771..5e0bbdf3e 100644 --- a/deploy-manage/users-roles/cluster-or-deployment-auth/service-accounts.md +++ b/deploy-manage/users-roles/cluster-or-deployment-auth/service-accounts.md @@ -3,10 +3,10 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/service-accounts.html applies_to: deployment: - ess: - ece: - eck: - self: + ess: + ece: + eck: + self: --- # Service accounts [service-accounts] @@ -26,7 +26,7 @@ Service accounts provide flexibility over [built-in users](built-in-users.md) be Service accounts are not included in the response of the [get users API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-get-user). To retrieve a service account, use the [get service accounts API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-get-service-accounts). Use the [get service account credentials API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-get-service-credentials) to retrieve all service credentials for a service account. -## Service accounts usage [service-accounts-explanation] +## Service accounts usage [service-accounts-explanation] Service accounts have a [unique principal](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-get-service-accounts#security-api-get-service-accounts-path-params) that takes the format of `/`, where the `namespace` is a top-level grouping of service accounts, and `service` is the name of the service and must be unique within its namespace. @@ -38,13 +38,13 @@ Service accounts are predefined in code. The following service accounts are avai `elastic/kibana` : The service account used by {{kib}} to communicate with {{es}}. -::::{important} +::::{important} Do not attempt to use service accounts for authenticating individual users. Service accounts can only be authenticated with service tokens, which are not applicable to regular users. :::: -## Service account tokens [service-accounts-tokens] +## Service account tokens [service-accounts-tokens] A service account token, or service token, is a unique string that a service uses to authenticate with {{es}}. For a given service account, each token must have a unique name. Because tokens include access credentials, they should always be kept secret by whichever client is using them. @@ -53,7 +53,7 @@ Service tokens can be backed by either the `.security` index (recommended) or th You must create a service token to use a service account. You can create a service token using either: * The [create service account token API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-service-token), which saves the new service token in the `.security` index and returns the bearer token in the HTTP response. -* Self-managed and {{eck}} deployments only: The [elasticsearch-service-tokens](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/service-tokens-command.md) CLI tool, which saves the new service token in the `$ES_HOME/config/service_tokens` file and outputs the bearer token to your terminal +* Self-managed and {{eck}} deployments only: The [elasticsearch-service-tokens](elasticsearch://reference/elasticsearch/command-line-tools/service-tokens-command.md) CLI tool, which saves the new service token in the `$ES_HOME/config/service_tokens` file and outputs the bearer token to your terminal We recommend that you create service tokens via the REST API rather than the CLI. The API stores service tokens within the `.security` index which means that the tokens are available for authentication on all nodes, and will be backed up within cluster snapshots. The use of the CLI is intended for cases where there is an external orchestration process (such as [{{ece}}](https://www.elastic.co/guide/en/cloud-enterprise/current) or [{{eck}}](https://www.elastic.co/guide/en/cloud-on-k8s/current)) that will manage the creation and distribution of the `service_tokens` file. @@ -62,9 +62,9 @@ Both of these methods (API and CLI) create a service token with a guaranteed sec Service tokens never expire. You must actively [delete](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-delete-service-token) them if they are no longer needed. -## Authenticate with service tokens [authenticate-with-service-account-token] +## Authenticate with service tokens [authenticate-with-service-account-token] -::::{note} +::::{note} Service accounts currently do not support basic authentication. :::: diff --git a/deploy-manage/users-roles/cluster-or-deployment-auth/token-based-authentication-services.md b/deploy-manage/users-roles/cluster-or-deployment-auth/token-based-authentication-services.md index dfe380194..67db6261b 100644 --- a/deploy-manage/users-roles/cluster-or-deployment-auth/token-based-authentication-services.md +++ b/deploy-manage/users-roles/cluster-or-deployment-auth/token-based-authentication-services.md @@ -3,10 +3,10 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/token-authentication-services.html applies_to: deployment: - ess: - ece: - eck: - self: + ess: + ece: + eck: + self: --- # Token-based authentication services [token-authentication-services] @@ -16,7 +16,7 @@ The {{stack-security-features}} authenticate users by using realms and one or mo The {{security-features}} provide the following built-in token-based authentication services, which are listed in the order they are consulted: *service-accounts* -: The [service accounts](service-accounts.md) use either the [create service account token API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-service-token) or the [elasticsearch-service-tokens](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/service-tokens-command.md) CLI tool to generate service account tokens. +: The [service accounts](service-accounts.md) use either the [create service account token API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-service-token) or the [elasticsearch-service-tokens](elasticsearch://reference/elasticsearch/command-line-tools/service-tokens-command.md) CLI tool to generate service account tokens. To use a service account token, include the generated token value in a request with an `Authorization: Bearer` header: @@ -24,7 +24,7 @@ To use a service account token, include the generated token value in a request w curl -H "Authorization: Bearer AAEAAWVsYXN0aWMvZ...mXQtc2VydmMTpyNXdkYmRib1FTZTl2R09Ld2FKR0F3" http://localhost:9200/_cluster/health ``` -::::{important} +::::{important} Do not attempt to use service accounts for authenticating individual users. Service accounts can only be authenticated with service tokens, which are not applicable to regular users. :::: @@ -52,7 +52,7 @@ $$$token-authentication-api-key$$$ Depending on your use case, you may want to decide on the lifetime of the tokens generated by these services. You can then use this information to decide which service to use to generate and manage the tokens. Non-expiring API keys may seem like the easy option but you must consider the security implications that come with non-expiring keys. Both the *token-service* and *api-key-service* permit you to invalidate the tokens. See [invalidate token API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-invalidate-token) and [invalidate API key API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-invalidate-api-key). -::::{important} +::::{important} Authentication support for JWT bearer tokens was introduced in {{es}} 8.2 through the [JWT authentication](jwt.md), which cannot be enabled through token-authentication services. Realms offer flexible order and configurations of zero, one, or multiple JWT realms. :::: diff --git a/deploy-manage/users-roles/serverless-custom-roles.md b/deploy-manage/users-roles/serverless-custom-roles.md index 45e45dbb3..94382592c 100644 --- a/deploy-manage/users-roles/serverless-custom-roles.md +++ b/deploy-manage/users-roles/serverless-custom-roles.md @@ -54,14 +54,14 @@ Each role can grant access to multiple data indices, and each index can have a d :class: screenshot ::: -Refer to [index privileges](asciidocalypse://reference/elasticsearch/security-privileges.md#privileges-list-indices) for a complete description of available options. +Refer to [index privileges](elasticsearch://reference/elasticsearch/security-privileges.md#privileges-list-indices) for a complete description of available options. ### Document level and field level security -Document-level and field-level security affords you even more granularity when it comes to granting access to your data: +Document-level and field-level security affords you even more granularity when it comes to granting access to your data: + +* With document-level security (DLS), you can write an {{es}} query to describe which documents this role grants access to. Add your query in the **Granted documents query** field using [QueryDSL](/explore-analyze/query-filter/languages/querydsl.md) syntax. -* With document-level security (DLS), you can write an {{es}} query to describe which documents this role grants access to. Add your query in the **Granted documents query** field using [QueryDSL](/explore-analyze/query-filter/languages/querydsl.md) syntax. - For example, the following query grants read access only to the documents whose `department_id` equals `12`: ```json @@ -106,6 +106,6 @@ As new features are added to {{serverless-full}}, roles that use the custom opti :::: -## Assign custom roles +## Assign custom roles After your roles are set up, the next step to securing access is to assign roles to your users. Click the **Assign roles** link to go to the **Members** tab of the **Organization** page. Learn more in [](/deploy-manage/users-roles/cloud-organization/user-roles.md). diff --git a/docset.yml b/docset.yml index ce2ff4cdc..3eca9c92c 100644 --- a/docset.yml +++ b/docset.yml @@ -8,25 +8,45 @@ exclude: - 'README.md' cross_links: + - apm-agent-android + - apm-agent-dotnet + - apm-agent-go + - apm-agent-ios + - apm-aws-lambda + - apm-k8s-attacher - asciidocalypse - - kibana - - integration-docs - - integrations - - logstash - - elasticsearch - - cloud - beats - - go-elasticsearch + - cloud + - cloud-on-k8s + - curator + - ecctl + - ecs + - ecs-dotnet + - ecs-logging + - ecs-logging-go-zap + - ecs-logging-java + - ecs-logging-nodejs + - ecs-logging-php + - ecs-logging-python + - ecs-logging-ruby + - eland + - elastic-serverless-forwarder + - elasticsearch + - elasticsearch-hadoop - elasticsearch-java + - elasticsearch-js - elasticsearch-net - elasticsearch-php - elasticsearch-py + - elasticsearch-rs - elasticsearch-ruby - - elasticsearch-js - - ecs - - ecs-logging + - go-elasticsearch + - integration-docs + - integrations + - kibana + - logstash - search-ui - - cloud-on-k8s + - security-docs toc: - file: index.md diff --git a/explore-analyze/alerts-cases/alerts/alerting-common-issues.md b/explore-analyze/alerts-cases/alerts/alerting-common-issues.md index 086cc70dd..07df0771e 100644 --- a/explore-analyze/alerts-cases/alerts/alerting-common-issues.md +++ b/explore-analyze/alerts-cases/alerts/alerting-common-issues.md @@ -18,9 +18,9 @@ Rules with a small check interval, such as every two seconds, run later than sch **Solution** -Rules run as background tasks at a cadence defined by their **check interval**. When a Rule **check interval** is smaller than the Task Manager [`poll_interval`](asciidocalypse://docs/kibana/docs/reference/configuration-reference/task-manager-settings.md#task-manager-settings), the rule will run late. +Rules run as background tasks at a cadence defined by their **check interval**. When a Rule **check interval** is smaller than the Task Manager [`poll_interval`](kibana://reference/configuration-reference/task-manager-settings.md#task-manager-settings), the rule will run late. -Either tweak the [{{kib}} Task Manager settings](asciidocalypse://docs/kibana/docs/reference/configuration-reference/task-manager-settings.md#task-manager-settings) or increase the **check interval** of the rules in question. +Either tweak the [{{kib}} Task Manager settings](kibana://reference/configuration-reference/task-manager-settings.md#task-manager-settings) or increase the **check interval** of the rules in question. For more details, see [Tasks with small schedule intervals run late](../../../troubleshoot/kibana/task-manager.md#task-manager-health-scheduled-tasks-small-schedule-interval-run-late). @@ -36,7 +36,7 @@ Actions run long after the status of a rule changes, sending a notification of t Rules and actions run as background tasks by each {{kib}} instance at a default rate of ten tasks every three seconds. When diagnosing issues related to alerting, focus on the tasks that begin with `alerting:` and `actions:`. -Alerting tasks always begin with `alerting:`. For example, the `alerting:.index-threshold` tasks back the [index threshold stack rule](rule-type-index-threshold.md). Action tasks always begin with `actions:`. For example, the `actions:.index` tasks back the [index action](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/index-action-type.md). +Alerting tasks always begin with `alerting:`. For example, the `alerting:.index-threshold` tasks back the [index threshold stack rule](rule-type-index-threshold.md). Action tasks always begin with `actions:`. For example, the `actions:.index` tasks back the [index action](kibana://reference/connectors-kibana/index-action-type.md). For more details on monitoring and diagnosing tasks in Task Manager, refer to [Health monitoring](../../../deploy-manage/monitor/kibana-task-manager-health-monitoring.md). @@ -48,7 +48,7 @@ A connector gets a TLS socket error when connecting to the server to run an acti **Solution** -Configuration options are available to specialize connections to TLS servers, including ignoring server certificate validation and providing certificate authority data to verify servers using custom certificates. For more details, see [Action settings](asciidocalypse://docs/kibana/docs/reference/configuration-reference/alerting-settings.md#action-settings). +Configuration options are available to specialize connections to TLS servers, including ignoring server certificate validation and providing certificate authority data to verify servers using custom certificates. For more details, see [Action settings](kibana://reference/configuration-reference/alerting-settings.md#action-settings). ## Rules take a long time to run [rules-long-run-time] @@ -76,7 +76,7 @@ and in the [details page](create-manage-rules.md#rule-details): :class: screenshot ::: -If you want your rules to run longer, update the `xpack.alerting.rules.run.timeout` configuration in your [Alerting settings](asciidocalypse://docs/kibana/docs/reference/configuration-reference/alerting-settings.md#alert-settings). You can also target a specific rule type by using `xpack.alerting.rules.run.ruleTypeOverrides`. +If you want your rules to run longer, update the `xpack.alerting.rules.run.timeout` configuration in your [Alerting settings](kibana://reference/configuration-reference/alerting-settings.md#alert-settings). You can also target a specific rule type by using `xpack.alerting.rules.run.ruleTypeOverrides`. Rules that consistently run longer than their [check interval](create-manage-rules.md#create-edit-rules) may produce unexpected results. If the average run duration, visible on the [details page](create-manage-rules.md#rule-details), is greater than the check interval, consider increasing the check interval. @@ -243,8 +243,8 @@ This error happens when the `xpack.encryptedSavedObjects.encryptionKey` value us | | | | --- | --- | -| If the value in `xpack.encryptedSavedObjects.encryptionKey` was manually changed, and the previous encryption key is still known. | Ensure any previous encryption key is included in the keys used for [decryption only](asciidocalypse://docs/kibana/docs/reference/configuration-reference/security-settings.md#xpack-encryptedSavedObjects-keyRotation-decryptionOnlyKeys). | -| If another {{kib}} instance with a different encryption key connects to the cluster. | The other {{kib}} instance might be trying to run the rule using a different encryption key than what the rule was created with. Ensure the encryption keys among all the {{kib}} instances are the same, and setting [decryption only keys](asciidocalypse://docs/kibana/docs/reference/configuration-reference/security-settings.md#xpack-encryptedSavedObjects-keyRotation-decryptionOnlyKeys) for previously used encryption keys. | +| If the value in `xpack.encryptedSavedObjects.encryptionKey` was manually changed, and the previous encryption key is still known. | Ensure any previous encryption key is included in the keys used for [decryption only](kibana://reference/configuration-reference/security-settings.md#xpack-encryptedsavedobjects-keyrotation-decryptiononlykeys). | +| If another {{kib}} instance with a different encryption key connects to the cluster. | The other {{kib}} instance might be trying to run the rule using a different encryption key than what the rule was created with. Ensure the encryption keys among all the {{kib}} instances are the same, and setting [decryption only keys](kibana://reference/configuration-reference/security-settings.md#xpack-encryptedsavedobjects-keyrotation-decryptiononlykeys) for previously used encryption keys. | | If other scenarios don’t apply. | Generate a new API key for the rule. For example, in **{{stack-manage-app}} > {{rules-ui}}**, select **Update API key** from the action menu. | ## Rules stop running after upgrade [known-issue-upgrade-rule] diff --git a/explore-analyze/alerts-cases/alerts/alerting-getting-started.md b/explore-analyze/alerts-cases/alerts/alerting-getting-started.md index fcfe99f0e..da34cdf43 100644 --- a/explore-analyze/alerts-cases/alerts/alerting-getting-started.md +++ b/explore-analyze/alerts-cases/alerts/alerting-getting-started.md @@ -7,7 +7,7 @@ navigation_title: Getting started with alerts # Getting started with alerting [alerting-getting-started] -Alerting enables you to define *rules*, which detect complex conditions within different {{kib}} apps and trigger actions when those conditions are met. Alerting is integrated with [**{{observability}}**](../../../solutions/observability/incident-management/alerting.md), [**Security**](asciidocalypse://docs/docs-content/docs/reference/security/prebuilt-rules.md), [**Maps**](../../../explore-analyze/alerts-cases/alerts/geo-alerting.md) and [**{{ml-app}}**](../../../explore-analyze/machine-learning/anomaly-detection/ml-configuring-alerts.md). It can be centrally managed from **{{stack-manage-app}}** and provides a set of built-in [connectors](../../../deploy-manage/manage-connectors.md) and [rules](../../../explore-analyze/alerts-cases/alerts/rule-types.md#stack-rules) for you to use. +Alerting enables you to define *rules*, which detect complex conditions within different {{kib}} apps and trigger actions when those conditions are met. Alerting is integrated with [**{{observability}}**](../../../solutions/observability/incident-management/alerting.md), [**Security**](security-docs://reference/prebuilt-rules/index.md), [**Maps**](../../../explore-analyze/alerts-cases/alerts/geo-alerting.md) and [**{{ml-app}}**](../../../explore-analyze/machine-learning/anomaly-detection/ml-configuring-alerts.md). It can be centrally managed from **{{stack-manage-app}}** and provides a set of built-in [connectors](../../../deploy-manage/manage-connectors.md) and [rules](../../../explore-analyze/alerts-cases/alerts/rule-types.md#stack-rules) for you to use. :::{image} ../../../images/kibana-alerting-overview.png :alt: {{rules-ui}} UI diff --git a/explore-analyze/alerts-cases/alerts/alerting-setup.md b/explore-analyze/alerts-cases/alerts/alerting-setup.md index f8c547b56..bf378a643 100644 --- a/explore-analyze/alerts-cases/alerts/alerting-setup.md +++ b/explore-analyze/alerts-cases/alerts/alerting-setup.md @@ -15,14 +15,14 @@ mapped_pages: If you are using an **on-premises** {{stack}} deployment: -* In the `kibana.yml` configuration file, add the [`xpack.encryptedSavedObjects.encryptionKey`](asciidocalypse://docs/kibana/docs/reference/configuration-reference/alerting-settings.md#general-alert-action-settings) setting. +* In the `kibana.yml` configuration file, add the [`xpack.encryptedSavedObjects.encryptionKey`](kibana://reference/configuration-reference/alerting-settings.md#general-alert-action-settings) setting. * For emails to have a footer with a link back to {{kib}}, set the [`server.publicBaseUrl`](../../../deploy-manage/deploy/self-managed/configure.md#server-publicBaseUrl) configuration setting. If you are using an **on-premises** {{stack}} deployment with [**security**](../../../deploy-manage/security.md): -* If you are unable to access {{kib}} {{alert-features}}, ensure that you have not [explicitly disabled API keys](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/security-settings.md#api-key-service-settings). +* If you are unable to access {{kib}} {{alert-features}}, ensure that you have not [explicitly disabled API keys](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md#api-key-service-settings). -The alerting framework uses queries that require the `search.allow_expensive_queries` setting to be `true`. See the scripts [documentation](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-script-query.md#_allow_expensive_queries_4). +The alerting framework uses queries that require the `search.allow_expensive_queries` setting to be `true`. See the scripts [documentation](elasticsearch://reference/query-languages/query-dsl-script-query.md#_allow_expensive_queries_4). ## Production considerations and scaling guidance [alerting-setup-production] @@ -47,7 +47,7 @@ The **{{connectors-feature}}** feature privilege is required to manage connector Likewise, you can customize the **Rules Settings** sub-feature privileges related to flapping detection settings. -To create a rule that uses the [Cases connector](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/cases-action-type.md), you must also have `All` privileges for the **Cases** feature. +To create a rule that uses the [Cases connector](kibana://reference/connectors-kibana/cases-action-type.md), you must also have `All` privileges for the **Cases** feature. The rule type also affects the privileges that are required. For example, to create or edit {{ml}} rules, you must have `all` privileges for the **Analytics > {{ml-app}}** feature. For {{stack-monitor-app}} rules, you must have the `monitoring_user` role. For {{observability}} rules, you must have `all` privileges for the appropriate {{observability}} features. For Security rules, refer to [Detections prerequisites and requirements](../../../solutions/security/detect-and-alert/detections-requirements.md). @@ -104,7 +104,7 @@ If a rule requires certain privileges, such as index privileges, to run and a us ### Restrict actions [alerting-restricting-actions] -For security reasons you may wish to limit the extent to which {{kib}} can connect to external services. You can use [Action settings](asciidocalypse://docs/kibana/docs/reference/configuration-reference/alerting-settings.md#action-settings) to disable certain [*Connectors*](../../../deploy-manage/manage-connectors.md) and allowlist the hostnames that {{kib}} can connect with. +For security reasons you may wish to limit the extent to which {{kib}} can connect to external services. You can use [Action settings](kibana://reference/configuration-reference/alerting-settings.md#action-settings) to disable certain [*Connectors*](../../../deploy-manage/manage-connectors.md) and allowlist the hostnames that {{kib}} can connect with. ## Space isolation [alerting-spaces] diff --git a/explore-analyze/alerts-cases/alerts/alerting-troubleshooting.md b/explore-analyze/alerts-cases/alerts/alerting-troubleshooting.md index 52b451745..e258af277 100644 --- a/explore-analyze/alerts-cases/alerts/alerting-troubleshooting.md +++ b/explore-analyze/alerts-cases/alerts/alerting-troubleshooting.md @@ -177,7 +177,7 @@ In addition to the above methods, refer to the following approaches and common i ### Temporarily throttle all tasks [alerting-kibana-throttle] -If cluster performance becomes degraded from excessive or expensive rules and {{kib}} is sluggish or unresponsive, you can temporarily reduce load to the Task Manager by updating its [settings](asciidocalypse://docs/kibana/docs/reference/configuration-reference/task-manager-settings.md): +If cluster performance becomes degraded from excessive or expensive rules and {{kib}} is sluggish or unresponsive, you can temporarily reduce load to the Task Manager by updating its [settings](kibana://reference/configuration-reference/task-manager-settings.md): ```txt xpack.task_manager.capacity: 5 diff --git a/explore-analyze/alerts-cases/alerts/create-manage-rules.md b/explore-analyze/alerts-cases/alerts/create-manage-rules.md index 6cb7fa5f3..61880ab49 100644 --- a/explore-analyze/alerts-cases/alerts/create-manage-rules.md +++ b/explore-analyze/alerts-cases/alerts/create-manage-rules.md @@ -8,7 +8,7 @@ mapped_pages: # Create and manage rules [create-and-manage-rules] -The **{{stack-manage-app}}** > **{{rules-ui}}** UI provides a cross-app view of alerting. Different {{kib}} apps like [**{{observability}}**](../../../solutions/observability/incident-management/alerting.md), [**Security**](asciidocalypse://docs/docs-content/docs/reference/security/prebuilt-rules.md), [**Maps**](geo-alerting.md) and [**{{ml-app}}**](../../machine-learning/machine-learning-in-kibana.md) can offer their own rules. +The **{{stack-manage-app}}** > **{{rules-ui}}** UI provides a cross-app view of alerting. Different {{kib}} apps like [**{{observability}}**](../../../solutions/observability/incident-management/alerting.md), [**Security**](security-docs://reference/prebuilt-rules/index.md), [**Maps**](geo-alerting.md) and [**{{ml-app}}**](../../machine-learning/machine-learning-in-kibana.md) can offer their own rules. You can find **Rules** in **Stack Management** > **Alerts and insights** > **Rules** in {{kib}} or by using the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). @@ -58,7 +58,7 @@ You can add one or more actions to your rule to generate notifications when its Each action uses a connector, which provides connection information for a {{kib}} service or third party integration, depending on where you want to send the notifications. -[preview] Some connectors that perform actions within {{kib}}, such as the [Cases connector](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/cases-action-type.md), require less configuration. For example, you do not need to set the action frequency or variables. +[preview] Some connectors that perform actions within {{kib}}, such as the [Cases connector](kibana://reference/connectors-kibana/cases-action-type.md), require less configuration. For example, you do not need to set the action frequency or variables. After you select a connector, set the action frequency. You can choose to create a summary of alerts on each check interval or on a custom interval. Alternatively, you an choose to run actions for each alert (at each check interval, only when the alert status changes, or at a custom interval). @@ -150,7 +150,7 @@ Click the rule name to access a rule details page: In this example, the rule detects when a site serves more than a threshold number of bytes in a 24 hour period. Four sites are above the threshold. These are called alerts - occurrences of the condition being detected - and the alert name, status, time of detection, and duration of the condition are shown in this view. Alerts come and go from the list depending on whether the rule conditions are met. For more information about alerts, go to [*View alerts*](view-alerts.md). -If there are rule actions that failed to run successfully, you can see the details on the **History** tab. In the **Message** column, click the warning or expand icon ![double arrow icon to open a flyout with the document details](../../../images/kibana-expand-icon-2.png "") or click the number in the **Errored actions** column to open the **Errored Actions** panel. In this example, the action failed because the [`xpack.actions.email.domain_allowlist`](asciidocalypse://docs/kibana/docs/reference/configuration-reference/alerting-settings.md#action-config-email-domain-allowlist) setting was updated and the action’s email recipient is no longer included in the allowlist: +If there are rule actions that failed to run successfully, you can see the details on the **History** tab. In the **Message** column, click the warning or expand icon ![double arrow icon to open a flyout with the document details](../../../images/kibana-expand-icon-2.png "") or click the number in the **Errored actions** column to open the **Errored Actions** panel. In this example, the action failed because the [`xpack.actions.email.domain_allowlist`](kibana://reference/configuration-reference/alerting-settings.md#action-config-email-domain-allowlist) setting was updated and the action’s email recipient is no longer included in the allowlist: :::{image} ../../../images/kibana-rule-details-errored-actions.png :alt: Rule histor page with alerts that have errored actions diff --git a/explore-analyze/alerts-cases/alerts/notifications-domain-allowlist.md b/explore-analyze/alerts-cases/alerts/notifications-domain-allowlist.md index 94e0e85e4..e0d790fad 100644 --- a/explore-analyze/alerts-cases/alerts/notifications-domain-allowlist.md +++ b/explore-analyze/alerts-cases/alerts/notifications-domain-allowlist.md @@ -33,7 +33,7 @@ This updates the notifications settings for {{es}} and {{kib}} to reflect what i ### Use the {{ecloud}} Control CLI [use-the-ecloud-control-cli] -Updating multiple deployments through the UI can take a lot of time. Instead, you can use the [{{ecloud}} Control](asciidocalypse://docs/ecctl/docs/reference/index.md) command-line interface (`ecctl`) to automate the deployment update. +Updating multiple deployments through the UI can take a lot of time. Instead, you can use the [{{ecloud}} Control](ecctl://reference/index.md) command-line interface (`ecctl`) to automate the deployment update. The following example script shows how to update all deployments of an organization: diff --git a/explore-analyze/alerts-cases/alerts/rule-action-variables.md b/explore-analyze/alerts-cases/alerts/rule-action-variables.md index 099fa7a98..c01d7adfc 100644 --- a/explore-analyze/alerts-cases/alerts/rule-action-variables.md +++ b/explore-analyze/alerts-cases/alerts/rule-action-variables.md @@ -20,9 +20,9 @@ The available variables differ by rule type, however there are some common varia Some cases exist where the variable values will be "escaped" when used in a context where escaping is needed. For example: -* For the [email connector](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/email-action-type.md), the `message` action configuration property escapes any characters that would be interpreted as Markdown. -* For the [Slack connector](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/slack-action-type.md), the `message` action configuration property escapes any characters that would be interpreted as Slack Markdown. -* For the [Webhook connector](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/webhook-action-type.md), the `body` action configuration property escapes any characters that are invalid in JSON string values. +* For the [email connector](kibana://reference/connectors-kibana/email-action-type.md), the `message` action configuration property escapes any characters that would be interpreted as Markdown. +* For the [Slack connector](kibana://reference/connectors-kibana/slack-action-type.md), the `message` action configuration property escapes any characters that would be interpreted as Slack Markdown. +* For the [Webhook connector](kibana://reference/connectors-kibana/webhook-action-type.md), the `body` action configuration property escapes any characters that are invalid in JSON string values. Mustache also supports "triple braces" of the form `{{{variable name}}}`, which indicates no escaping should be done at all. Use this form with caution, since it could end up rendering the variable content such that the resulting parameter is invalid or formatted incorrectly. @@ -221,7 +221,7 @@ You can enhance the values contained in Mustache variables when the Mustache tem ### Rendering objects as JSON [_rendering_objects_as_json] -Some connectors (such as the [Webhook connector](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/webhook-action-type.md)) expect JSON values to be passed as parameters when the connector is invoked. The following capabilities are available: +Some connectors (such as the [Webhook connector](kibana://reference/connectors-kibana/webhook-action-type.md)) expect JSON values to be passed as parameters when the connector is invoked. The following capabilities are available: * Array values referenced in braces have a predefined rendering by Mustache as string versions of the array elements, joined with a comma (`,`). To render array values as JSON, access the `asJSON` property of the array, instead of the array directly. For example, given a Mustache variable `context.values` that has the value `[1, 4, 9]` the Mustache template `{{context.values}}` will render as `1,4,9`, and the Mustache template `{{context.values.asJSON}}` will render as `[1,4,9]`. * The [ParseHjson lambda](#parse-hjson-lambda) Mustache lambda makes it easier to create JSON in your templates by using [Hjson](https://hjson.github.io/), a syntax extension to JSON, rather than strict JSON. diff --git a/explore-analyze/alerts-cases/alerts/rule-type-es-query.md b/explore-analyze/alerts-cases/alerts/rule-type-es-query.md index 9001556d6..289532836 100644 --- a/explore-analyze/alerts-cases/alerts/rule-type-es-query.md +++ b/explore-analyze/alerts-cases/alerts/rule-type-es-query.md @@ -52,7 +52,7 @@ When you create an {{es}} query rule, your choice of query type affects the info : Specify how to calculate the value that is compared to the threshold. The value is calculated by aggregating a numeric field within the time window. The aggregation options are: `count`, `average`, `sum`, `min`, and `max`. When using `count` the document count is used and an aggregation field is not necessary. Over or Grouped Over - : Specify whether the aggregation is applied over all documents or split into groups using up to four grouping fields. If you choose to use grouping, it’s a [terms](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-terms-aggregation.md) or [multi terms aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-multi-terms-aggregation.md); an alert will be created for each unique set of values when it meets the condition. To limit the number of alerts on high cardinality fields, you must specify the number of groups to check against the threshold. Only the top groups are checked. + : Specify whether the aggregation is applied over all documents or split into groups using up to four grouping fields. If you choose to use grouping, it’s a [terms](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-terms-aggregation.md) or [multi terms aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-multi-terms-aggregation.md); an alert will be created for each unique set of values when it meets the condition. To limit the number of alerts on high cardinality fields, you must specify the number of groups to check against the threshold. Only the top groups are checked. Threshold : Defines a threshold value and a comparison operator (`is above`, `is above or equals`, `is below`, `is below or equals`, or `is between`). The value calculated by the aggregation is compared to this threshold. @@ -150,7 +150,7 @@ The following variables are specific to the {{es}} query rule: {{/context.hits}} ``` - The documents returned by `context.hits` include the [`_source`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md) field. If the {{es}} query search API’s [`fields`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/retrieve-selected-fields.md#search-fields-param) parameter is used, documents will also return the `fields` field, which can be used to access any runtime fields defined by the [`runtime_mappings`](../../../manage-data/data-store/mapping/define-runtime-fields-in-search-request.md) parameter. For example: + The documents returned by `context.hits` include the [`_source`](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md) field. If the {{es}} query search API’s [`fields`](elasticsearch://reference/elasticsearch/rest-apis/retrieve-selected-fields.md#search-fields-param) parameter is used, documents will also return the `fields` field, which can be used to access any runtime fields defined by the [`runtime_mappings`](../../../manage-data/data-store/mapping/define-runtime-fields-in-search-request.md) parameter. For example: ```handlebars {{#context.hits}} @@ -162,7 +162,7 @@ The following variables are specific to the {{es}} query rule: 1. The `fields` parameter here is used to access the `day_of_week` runtime field. - As the [`fields`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/retrieve-selected-fields.md#search-fields-response) response always returns an array of values for each field, the [Mustache](https://mustache.github.io/) template array syntax is used to iterate over these values in your actions. For example: + As the [`fields`](elasticsearch://reference/elasticsearch/rest-apis/retrieve-selected-fields.md#search-fields-response) response always returns an array of values for each field, the [Mustache](https://mustache.github.io/) template array syntax is used to iterate over these values in your actions. For example: ```handlebars {{#context.hits}} diff --git a/explore-analyze/alerts-cases/alerts/rule-types.md b/explore-analyze/alerts-cases/alerts/rule-types.md index 4315079e7..aaf793d85 100644 --- a/explore-analyze/alerts-cases/alerts/rule-types.md +++ b/explore-analyze/alerts-cases/alerts/rule-types.md @@ -30,7 +30,7 @@ Some rule types are subscription features, while others are free features. For a {{observability}} rules detect complex conditions in your observability data and create alerts when a rule’s conditions are met. For example, you can create a rule that detects when the value of a metric exceeds a specified threshold or when an anomaly occurs on a system or service you are monitoring. For more information, refer to [Alerting](../../../solutions/observability/incident-management/alerting.md). -::::{note} +::::{note} If you create a rule in the {{observability}} app, its alerts are not visible in **{{stack-manage-app}} > {{rules-ui}}**. They are visible only in the {{observability}} app. :::: @@ -41,7 +41,7 @@ If you create a rule in the {{observability}} app, its alerts are not visible in ## Security rules [security-rules] -Security rules detect suspicious source events with pre-built or custom rules and create alerts when a rule’s conditions are met. For more information, refer to [Security rules](asciidocalypse://docs/docs-content/docs/reference/security/prebuilt-rules.md). +Security rules detect suspicious source events with pre-built or custom rules and create alerts when a rule’s conditions are met. For more information, refer to [Security rules](security-docs://reference/prebuilt-rules/index.md). ::::{note} Alerts associated with security rules are visible only in the {{security-app}}; they are not visible in **{{stack-manage-app}} > {{rules-ui}}**. diff --git a/explore-analyze/alerts-cases/cases/manage-cases-settings.md b/explore-analyze/alerts-cases/cases/manage-cases-settings.md index e1a6b2032..259b39622 100644 --- a/explore-analyze/alerts-cases/cases/manage-cases-settings.md +++ b/explore-analyze/alerts-cases/cases/manage-cases-settings.md @@ -43,7 +43,7 @@ You can create connectors in **{{stack-manage-app}} > {{connectors-ui}}**, as de 1. From the **Incident management system** list, select **Add new connector**. 2. Select an external incident management system. -3. Enter your required settings. Refer to [{{ibm-r}}](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/resilient-action-type.md), [Jira](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/jira-action-type.md), [{{sn-itsm}}](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/servicenow-action-type.md), [{{sn-sir}}](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/servicenow-sir-action-type.md), [Swimlane](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/swimlane-action-type.md), [{{hive}}](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/thehive-action-type.md), or [{{webhook-cm}}](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/cases-webhook-action-type.md) for connector configuration details. +3. Enter your required settings. Refer to [{{ibm-r}}](kibana://reference/connectors-kibana/resilient-action-type.md), [Jira](kibana://reference/connectors-kibana/jira-action-type.md), [{{sn-itsm}}](kibana://reference/connectors-kibana/servicenow-action-type.md), [{{sn-sir}}](kibana://reference/connectors-kibana/servicenow-sir-action-type.md), [Swimlane](kibana://reference/connectors-kibana/swimlane-action-type.md), [{{hive}}](kibana://reference/connectors-kibana/thehive-action-type.md), or [{{webhook-cm}}](kibana://reference/connectors-kibana/cases-webhook-action-type.md) for connector configuration details. You can subsequently choose the connector when you create cases and use it in case templates. To change the default connector for new cases, select the connector from the **Incident management system** list. diff --git a/explore-analyze/alerts-cases/cases/manage-cases.md b/explore-analyze/alerts-cases/cases/manage-cases.md index 198c1882c..1d9b8573e 100644 --- a/explore-analyze/alerts-cases/cases/manage-cases.md +++ b/explore-analyze/alerts-cases/cases/manage-cases.md @@ -31,7 +31,7 @@ Open a new case to keep track of issues and share their details with colleagues. 6. For the **External incident management system**, select a connector. For more information, refer to [External incident management systems](manage-cases-settings.md#case-connectors). 7. After you’ve completed all of the required fields, click **Create case**. -[preview] Alternatively, you can configure your rules to automatically create cases by using [case actions](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/cases-action-type.md). By default, the rule adds all of the alerts within a specified time window to a single case. You can optionally choose a field to group the alerts and create separate cases for each group. You can also choose whether you want the rule to reopen cases or open new ones when the time window elapses. +[preview] Alternatively, you can configure your rules to automatically create cases by using [case actions](kibana://reference/connectors-kibana/cases-action-type.md). By default, the rule adds all of the alerts within a specified time window to a single case. You can optionally choose a field to group the alerts and create separate cases for each group. You can also choose whether you want the rule to reopen cases or open new ones when the time window elapses. ## Add email notifications [add-case-notifications] @@ -47,14 +47,14 @@ For self-managed {{kib}}: 1. Create a preconfigured email connector. ::::{note} - At this time, email notifications support only preconfigured connectors, which are defined in the `kibana.yml` file. For examples, refer to [Email connectors](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/pre-configured-connectors.md#preconfigured-email-configuration) and [Configure email accounts for well-known services](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/email-action-type.md#configuring-email). + At this time, email notifications support only preconfigured connectors, which are defined in the `kibana.yml` file. For examples, refer to [Email connectors](kibana://reference/connectors-kibana/pre-configured-connectors.md#preconfigured-email-configuration) and [Configure email accounts for well-known services](kibana://reference/connectors-kibana/email-action-type.md#configuring-email). :::: 2. Set the `notifications.connectors.default.email` {{kib}} setting in kibana.yml to the name of your email connector. ```js notifications.connectors.default.email: ‘mail-dev’ - + xpack.actions.preconfigured: mail-dev: name: preconfigured-email-notification-maildev diff --git a/explore-analyze/alerts-cases/cases/setup-cases.md b/explore-analyze/alerts-cases/cases/setup-cases.md index e16c38c8d..6b58a5162 100644 --- a/explore-analyze/alerts-cases/cases/setup-cases.md +++ b/explore-analyze/alerts-cases/cases/setup-cases.md @@ -33,7 +33,7 @@ By default, `All` for the **Cases** feature includes authority to delete cases a ::::{note} Before a user can be assigned to a case, they must log into {{kib}} at least once, which creates a user profile. -This privilege is also required to add [case actions](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/cases-action-type.md) to rules. +This privilege is also required to add [case actions](kibana://reference/connectors-kibana/cases-action-type.md) to rules. :::: diff --git a/explore-analyze/alerts-cases/watcher/actions-email.md b/explore-analyze/alerts-cases/watcher/actions-email.md index 3e18a4b96..ab7a52eb9 100644 --- a/explore-analyze/alerts-cases/watcher/actions-email.md +++ b/explore-analyze/alerts-cases/watcher/actions-email.md @@ -282,7 +282,7 @@ bin/elasticsearch-keystore add xpack.notification.email.account.exchange_account The `email` action supports sending messages with an HTML body. However, for security reasons, {{watcher}} [sanitizes](https://en.wikipedia.org/wiki/HTML_sanitization) the HTML. -You can control which HTML features are allowed or disallowed by configuring the `xpack.notification.email.html.sanitization.allow` and `xpack.notification.email.html.sanitization.disallow` settings in `elasticsearch.yml`. You can specify individual HTML elements and [HTML feature groups](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/watcher-settings.md#html-feature-groups). By default, {{watcher}} allows the following features: `body`, `head`, `_tables`, `_links`, `_blocks`, `_formatting` and `img:embedded`. +You can control which HTML features are allowed or disallowed by configuring the `xpack.notification.email.html.sanitization.allow` and `xpack.notification.email.html.sanitization.disallow` settings in `elasticsearch.yml`. You can specify individual HTML elements and [HTML feature groups](elasticsearch://reference/elasticsearch/configuration-reference/watcher-settings.md#html-feature-groups). By default, {{watcher}} allows the following features: `body`, `head`, `_tables`, `_links`, `_blocks`, `_formatting` and `img:embedded`. For example, the following settings allow the HTML to contain tables and block elements, but disallow `

`, `

` and `
` tags. diff --git a/explore-analyze/alerts-cases/watcher/actions-index.md b/explore-analyze/alerts-cases/watcher/actions-index.md index d28a50d1d..14d52df6a 100644 --- a/explore-analyze/alerts-cases/watcher/actions-index.md +++ b/explore-analyze/alerts-cases/watcher/actions-index.md @@ -43,7 +43,7 @@ The following snippet shows a simple `index` action definition: | `op_type` | no | `index` | The [op_type](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-create) for the index operation. Must be one of either `index` or `create`. Must be `create` if `index` is a data stream. | | `execution_time_field` | no | - | The field that will store/index the watch execution time. | | `timeout` | no | 60s | The timeout for waiting for the index api call to return. If no response is returned within this time, the index action times out and fails. This setting overrides the default timeouts. | -| `refresh` | no | - | Optional setting of the [refresh policy](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/refresh-parameter.md) for the write request | +| `refresh` | no | - | Optional setting of the [refresh policy](elasticsearch://reference/elasticsearch/rest-apis/refresh-parameter.md) for the write request | ## Multi-document support [anatomy-actions-index-multi-doc-support] diff --git a/explore-analyze/alerts-cases/watcher/actions-jira.md b/explore-analyze/alerts-cases/watcher/actions-jira.md index 7ff0ec9b4..dd395e5d7 100644 --- a/explore-analyze/alerts-cases/watcher/actions-jira.md +++ b/explore-analyze/alerts-cases/watcher/actions-jira.md @@ -108,7 +108,7 @@ xpack.notification.jira: It is strongly advised to use Basic Authentication with secured HTTPS protocol only. :::: -You can also specify defaults for the [Jira issues](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/watcher-settings.md#jira-account-attributes): +You can also specify defaults for the [Jira issues](elasticsearch://reference/elasticsearch/configuration-reference/watcher-settings.md#jira-account-attributes): ```yaml xpack.notification.jira: diff --git a/explore-analyze/alerts-cases/watcher/actions-slack.md b/explore-analyze/alerts-cases/watcher/actions-slack.md index 2398e9fb8..0770c0dd7 100644 --- a/explore-analyze/alerts-cases/watcher/actions-slack.md +++ b/explore-analyze/alerts-cases/watcher/actions-slack.md @@ -153,7 +153,7 @@ You can no longer configure Slack accounts using `elasticsearch.yml` settings. P :::: -You can specify defaults for the [Slack notification attributes](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/watcher-settings.md#slack-account-attributes): +You can specify defaults for the [Slack notification attributes](elasticsearch://reference/elasticsearch/configuration-reference/watcher-settings.md#slack-account-attributes): ```yaml xpack.notification.slack: diff --git a/explore-analyze/alerts-cases/watcher/actions-webhook.md b/explore-analyze/alerts-cases/watcher/actions-webhook.md index dcf6dabbf..3c6674f24 100644 --- a/explore-analyze/alerts-cases/watcher/actions-webhook.md +++ b/explore-analyze/alerts-cases/watcher/actions-webhook.md @@ -71,7 +71,7 @@ You can use basic authentication when sending a request to a secured webservice. By default, both the username and the password are stored in the `.watches` index in plain text. When the {{es}} {{security-features}} are enabled, {{watcher}} can encrypt the password before storing it. :::: -You can also use PKI-based authentication when submitting requests to a cluster that has {{es}} {{security-features}} enabled. When you use PKI-based authentication instead of HTTP basic auth, you don’t need to store any authentication information in the watch itself. To use PKI-based authentication, you [configure the SSL key settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/watcher-settings.md#ssl-notification-settings) for {{watcher}} in `elasticsearch.yml`. +You can also use PKI-based authentication when submitting requests to a cluster that has {{es}} {{security-features}} enabled. When you use PKI-based authentication instead of HTTP basic auth, you don’t need to store any authentication information in the watch itself. To use PKI-based authentication, you [configure the SSL key settings](elasticsearch://reference/elasticsearch/configuration-reference/watcher-settings.md#ssl-notification-settings) for {{watcher}} in `elasticsearch.yml`. ## Query parameters [webhook-query-parameters] diff --git a/explore-analyze/alerts-cases/watcher/enable-watcher.md b/explore-analyze/alerts-cases/watcher/enable-watcher.md index d2550a260..52d4b4bf5 100644 --- a/explore-analyze/alerts-cases/watcher/enable-watcher.md +++ b/explore-analyze/alerts-cases/watcher/enable-watcher.md @@ -164,4 +164,4 @@ An example on how to configure a new account from the Elastic cloud console: 6. The new email account is now set up. It will now be used by default for watcher email actions. -For a full reference of all available settings, see the [Elasticsearch documentation](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/watcher-settings.md#email-notification-settings). +For a full reference of all available settings, see the [Elasticsearch documentation](elasticsearch://reference/elasticsearch/configuration-reference/watcher-settings.md#email-notification-settings). diff --git a/explore-analyze/alerts-cases/watcher/encrypting-data.md b/explore-analyze/alerts-cases/watcher/encrypting-data.md index a74ef574a..81233df32 100644 --- a/explore-analyze/alerts-cases/watcher/encrypting-data.md +++ b/explore-analyze/alerts-cases/watcher/encrypting-data.md @@ -14,19 +14,19 @@ Every `password` field that is used in your watch within an HTTP basic authentic To encrypt sensitive data in {{watcher}}: -1. Use the [elasticsearch-syskeygen](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/syskeygen.md) command to create a system key file. +1. Use the [elasticsearch-syskeygen](elasticsearch://reference/elasticsearch/command-line-tools/syskeygen.md) command to create a system key file. 2. Copy the `system_key` file to all of the nodes in your cluster. ::::{important} The system key is a symmetric key, so the same key must be used on every node in the cluster. :::: -3. Set the [`xpack.watcher.encrypt_sensitive_data` setting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/watcher-settings.md): +3. Set the [`xpack.watcher.encrypt_sensitive_data` setting](elasticsearch://reference/elasticsearch/configuration-reference/watcher-settings.md): ```sh xpack.watcher.encrypt_sensitive_data: true ``` -4. Set the [`xpack.watcher.encryption_key` setting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/watcher-settings.md) in the [{{es}} keystore](../../../deploy-manage/security/secure-settings.md) on each node in the cluster. +4. Set the [`xpack.watcher.encryption_key` setting](elasticsearch://reference/elasticsearch/configuration-reference/watcher-settings.md) in the [{{es}} keystore](../../../deploy-manage/security/secure-settings.md) on each node in the cluster. For example, run the following command to import the `system_key` file on each node: diff --git a/explore-analyze/alerts-cases/watcher/input-search.md b/explore-analyze/alerts-cases/watcher/input-search.md index 6b08ae491..277e4fb03 100644 --- a/explore-analyze/alerts-cases/watcher/input-search.md +++ b/explore-analyze/alerts-cases/watcher/input-search.md @@ -141,9 +141,9 @@ The total number of hits in the search response is returned as an object in the | `request.indices` | no | - | The indices to search. If omitted, all indices are searched, which is the default behaviour in Elasticsearch. | | `request.body` | no | - | The body of the request. The [request body](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search) follows the same structure you normally send in the body of a REST `_search` request. The body can be static text or include `mustache` [templates](how-watcher-works.md#templates). | | `request.template` | no | - | The body of the search template. See [configure templates](how-watcher-works.md#templates) for more information. | -| `request.indices_options.expand_wildcards` | no | `open` | How to expand wildcards. Valid values are: `all`, `open`, `closed`, and `none` See [`expand_wildcards`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#api-multi-index) for more information. | -| `request.indices_options.ignore_unavailable` | no | `true` | Whether the search should ignore unavailable indices. See [`ignore_unavailable`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#api-multi-index) for more information. | -| `request.indices_options.allow_no_indices` | no | `true` | Whether to allow a search where a wildcard indices expression results in no concrete indices. See [allow_no_indices](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#api-multi-index) for more information. | +| `request.indices_options.expand_wildcards` | no | `open` | How to expand wildcards. Valid values are: `all`, `open`, `closed`, and `none` See [`expand_wildcards`](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#api-multi-index) for more information. | +| `request.indices_options.ignore_unavailable` | no | `true` | Whether the search should ignore unavailable indices. See [`ignore_unavailable`](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#api-multi-index) for more information. | +| `request.indices_options.allow_no_indices` | no | `true` | Whether to allow a search where a wildcard indices expression results in no concrete indices. See [allow_no_indices](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#api-multi-index) for more information. | | `extract` | no | - | A array of JSON keys to extract from the search response and load as the payload. When a search generates a large response, you can use `extract` to select the relevant fields instead of loading the entire response. | | `timeout` | no | 1m | The timeout for waiting for the search api call to return. If no response is returned within this time, the search input times out and fails. This setting overrides the default search operations timeouts. | diff --git a/explore-analyze/alerts-cases/watcher/schedule-types.md b/explore-analyze/alerts-cases/watcher/schedule-types.md index ba2e6e638..5b164ca40 100644 --- a/explore-analyze/alerts-cases/watcher/schedule-types.md +++ b/explore-analyze/alerts-cases/watcher/schedule-types.md @@ -421,10 +421,10 @@ By default, the `yearly` schedule is evaluated in the UTC time zone. To use a di ## {{watcher}} cron schedule [schedule-cron] -Defines a [`schedule`](trigger-schedule.md) using a [cron expression](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#api-cron-expressions) that specifiues when to execute a watch. +Defines a [`schedule`](trigger-schedule.md) using a [cron expression](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#api-cron-expressions) that specifiues when to execute a watch. ::::{tip} -While cron expressions are powerful, a regularly occurring schedule is easier to configure with the other schedule types. If you must use a cron schedule, make sure you verify it with [`elasticsearch-croneval`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/elasticsearch-croneval.md) . +While cron expressions are powerful, a regularly occurring schedule is easier to configure with the other schedule types. If you must use a cron schedule, make sure you verify it with [`elasticsearch-croneval`](elasticsearch://reference/elasticsearch/command-line-tools/elasticsearch-croneval.md) . :::: @@ -487,7 +487,7 @@ By default, cron expressions are evaluated in the UTC time zone. To use a differ ### Use croneval to validate cron expressions [croneval] -{{es}} provides a [`elasticsearch-croneval`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/elasticsearch-croneval.md) command line tool in the `$ES_HOME/bin` directory that you can use to check that your cron expressions are valid and produce the expected results. +{{es}} provides a [`elasticsearch-croneval`](elasticsearch://reference/elasticsearch/command-line-tools/elasticsearch-croneval.md) command line tool in the `$ES_HOME/bin` directory that you can use to check that your cron expressions are valid and produce the expected results. To validate a cron expression, pass it in as a parameter to `elasticsearch-croneval`: diff --git a/explore-analyze/alerts-cases/watcher/transform-search.md b/explore-analyze/alerts-cases/watcher/transform-search.md index e86a1cbbd..1dd0d73bf 100644 --- a/explore-analyze/alerts-cases/watcher/transform-search.md +++ b/explore-analyze/alerts-cases/watcher/transform-search.md @@ -52,9 +52,9 @@ The following table lists all available settings for the search {{watcher-transf | `request.search_type` | no | query_then_fetch | The search [type](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search). | | `request.indices` | no | all indices | One or more indices to search on. | | `request.body` | no | `match_all` query | The body of the request. The [request body](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search) follows the same structure you normally send in the body of a REST `_search` request. The body can be static text or include `mustache` [templates](how-watcher-works.md#templates). | -| `request.indices_options.expand_wildcards` | no | `open` | Determines how to expand indices wildcards. An array consisting of a combination of `open`, `closed`, and `hidden`. Alternatively a value of `none` or `all`. (see [multi-target syntax](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#api-multi-index)) | -| `request.indices_options.ignore_unavailable` | no | `true` | A boolean value that determines whether the search should leniently ignore unavailable indices (see [multi-target syntax](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#api-multi-index)) | -| `request.indices_options.allow_no_indices` | no | `true` | A boolean value that determines whether the search should leniently return no results when no indices are resolved (see [multi-target syntax](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#api-multi-index)) | +| `request.indices_options.expand_wildcards` | no | `open` | Determines how to expand indices wildcards. An array consisting of a combination of `open`, `closed`, and `hidden`. Alternatively a value of `none` or `all`. (see [multi-target syntax](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#api-multi-index)) | +| `request.indices_options.ignore_unavailable` | no | `true` | A boolean value that determines whether the search should leniently ignore unavailable indices (see [multi-target syntax](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#api-multi-index)) | +| `request.indices_options.allow_no_indices` | no | `true` | A boolean value that determines whether the search should leniently return no results when no indices are resolved (see [multi-target syntax](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#api-multi-index)) | | `request.template` | no | - | The body of the search template. See [configure templates](how-watcher-works.md#templates) for more information. | | `timeout` | no | 30s | The timeout for waiting for the search api call to return. If no response is returned within this time, the search {{watcher-transform}} times out and fails. This setting overrides the default timeouts. | diff --git a/explore-analyze/alerts-cases/watcher/watch-cluster-status.md b/explore-analyze/alerts-cases/watcher/watch-cluster-status.md index 5f1317c31..70edbdab5 100644 --- a/explore-analyze/alerts-cases/watcher/watch-cluster-status.md +++ b/explore-analyze/alerts-cases/watcher/watch-cluster-status.md @@ -85,7 +85,7 @@ PUT _watcher/watch/cluster_health_watch It would be a good idea to create a user with the minimum privileges required for use with such a watch configuration. -Depending on how your cluster is configured, there may be additional settings required before the watch can access your cluster such as keystores, truststores, or certificates. For more information, see [{{watcher}} settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/watcher-settings.md). +Depending on how your cluster is configured, there may be additional settings required before the watch can access your cluster such as keystores, truststores, or certificates. For more information, see [{{watcher}} settings](elasticsearch://reference/elasticsearch/configuration-reference/watcher-settings.md). If you check the watch history, you’ll see that the cluster status is recorded as part of the `watch_record` each time the watch executes. diff --git a/explore-analyze/alerts-cases/watcher/watcher-ui.md b/explore-analyze/alerts-cases/watcher/watcher-ui.md index d8eb217ff..9eb0b55dc 100644 --- a/explore-analyze/alerts-cases/watcher/watcher-ui.md +++ b/explore-analyze/alerts-cases/watcher/watcher-ui.md @@ -135,7 +135,7 @@ On the Watch overview page, click **Create** and choose **Create advanced watch* The **Simulate** tab allows you to override parts of the watch, and then run a simulation. Be aware of these implementation details on overrides: -* Trigger overrides use [date math](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/common-options.md#date-math). +* Trigger overrides use [date math](elasticsearch://reference/elasticsearch/rest-apis/common-options.md#date-math). * Input overrides accepts a JSON blob. * Condition overrides indicates if you want to force the condition to always be `true`. * Action overrides support [multiple options](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-watcher-execute-watch). diff --git a/explore-analyze/dashboards/create-dashboard-of-panels-with-ecommerce-data.md b/explore-analyze/dashboards/create-dashboard-of-panels-with-ecommerce-data.md index 9e7b5c96c..4ba886823 100644 --- a/explore-analyze/dashboards/create-dashboard-of-panels-with-ecommerce-data.md +++ b/explore-analyze/dashboards/create-dashboard-of-panels-with-ecommerce-data.md @@ -37,7 +37,7 @@ Open the visualization editor, then make sure the correct fields appear. ## Create visualizations with custom time intervals [custom-time-interval] -When you create visualizations with time series data, you can use the default time interval or increase and decrease the interval. For performance reasons, the visualization editor allows you to choose the minimum time interval, but not the exact time interval. The interval limit is controlled by the [`histogram:maxBars`](asciidocalypse://docs/kibana/docs/reference/advanced-settings.md#histogram-maxbars) setting and [time range](../query-filter/filtering.md). +When you create visualizations with time series data, you can use the default time interval or increase and decrease the interval. For performance reasons, the visualization editor allows you to choose the minimum time interval, but not the exact time interval. The interval limit is controlled by the [`histogram:maxBars`](kibana://reference/advanced-settings.md#histogram-maxbars) setting and [time range](../query-filter/filtering.md). To analyze the data with a custom time interval, create a bar chart that shows you how many orders were made at your store every hour: diff --git a/explore-analyze/dashboards/create-dashboard-of-panels-with-web-server-data.md b/explore-analyze/dashboards/create-dashboard-of-panels-with-web-server-data.md index d9bf82c1e..2b8f87be1 100644 --- a/explore-analyze/dashboards/create-dashboard-of-panels-with-web-server-data.md +++ b/explore-analyze/dashboards/create-dashboard-of-panels-with-web-server-data.md @@ -118,7 +118,7 @@ To increase the minimum time interval: 1. In the layer pane, click **timestamp**. 2. Change the **Minimum interval** to **1d**, then click **Close**. - You can increase and decrease the minimum interval, but you are unable to decrease the interval below the configured [**Advanced Settings**](asciidocalypse://docs/kibana/docs/reference/advanced-settings.md). + You can increase and decrease the minimum interval, but you are unable to decrease the interval below the configured [**Advanced Settings**](kibana://reference/advanced-settings.md). To save space on the dashboard, hide the axis labels. diff --git a/explore-analyze/discover/discover-get-started.md b/explore-analyze/discover/discover-get-started.md index 411cc2e3c..a469f8876 100644 --- a/explore-analyze/discover/discover-get-started.md +++ b/explore-analyze/discover/discover-get-started.md @@ -29,7 +29,7 @@ Select the data you want to explore, and then specify the time range in which to 2. Select the data view that contains the data you want to explore. ::::{tip} By default, {{kib}} requires a [{{data-source}}](../find-and-organize/data-views.md) to access your Elasticsearch data. A {{data-source}} can point to one or more indices, [data streams](../../manage-data/data-store/data-streams.md), or [index aliases](https://www.elastic.co/guide/en/elasticsearch/reference/current/alias.html). When adding data to {{es}} using one of the many integrations available, sometimes data views are created automatically, but you can also create your own. - + You can also [try {{esql}}](try-esql.md), that let's you query any data you have in {{es}} without specifying a {{data-source}} first. :::: If you’re using sample data, data views are automatically created and are ready to use. @@ -283,5 +283,5 @@ This section references common questions and issues encountered when using Disco This can happen in several cases: -* With runtime fields and regular keyword fields, when the string exceeds the value set for the [ignore_above](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/ignore-above.md) setting used when indexing the data into {{es}}. +* With runtime fields and regular keyword fields, when the string exceeds the value set for the [ignore_above](elasticsearch://reference/elasticsearch/mapping-reference/ignore-above.md) setting used when indexing the data into {{es}}. * Due to the structure of nested fields, a leaf field added to the table as a column will not contain values in any of its cells. Instead, add the root field as a column to view a JSON representation of its values. Learn more in [this blog post](https://www.elastic.co/de/blog/discover-uses-fields-api-in-7-12). diff --git a/explore-analyze/discover/document-explorer.md b/explore-analyze/discover/document-explorer.md index 710464792..61aaec357 100644 --- a/explore-analyze/discover/document-explorer.md +++ b/explore-analyze/discover/document-explorer.md @@ -53,7 +53,7 @@ You can define different settings for the header row and body rows. ### Limit the sample size [document-explorer-sample-size] -When the number of results returned by your search query (displayed at the top of the **Documents** or **Results** tab) is greater than the value of [`discover:sampleSize`](asciidocalypse://docs/kibana/docs/reference/advanced-settings.md#kibana-discover-settings), the number of results displayed in the table is limited to the configured value by default. You can adjust the initial sample size for searches to any number between 10 and `discover:sampleSize` from the **Display options** located in the table toolbar. +When the number of results returned by your search query (displayed at the top of the **Documents** or **Results** tab) is greater than the value of [`discover:sampleSize`](kibana://reference/advanced-settings.md#kibana-discover-settings), the number of results displayed in the table is limited to the configured value by default. You can adjust the initial sample size for searches to any number between 10 and `discover:sampleSize` from the **Display options** located in the table toolbar. On the last page of the table, a message indicates that you’ve reached the end of the loaded search results. From that message, you can choose to load more results to continue exploring. diff --git a/explore-analyze/discover/search-sessions.md b/explore-analyze/discover/search-sessions.md index c1c8fab0e..a6c80c8bb 100644 --- a/explore-analyze/discover/search-sessions.md +++ b/explore-analyze/discover/search-sessions.md @@ -17,7 +17,7 @@ Search Sessions are deprecated and will be removed in a future version. Sometimes you might need to search through large amounts of data, no matter how long the search takes. Consider a threat hunting scenario, where you need to search through years of data. You can save a long-running search, so {{kib}} processes your request in the background, and you can continue your work. -Save your search session from **Discover** or **Dashboard**, and when your session is complete, view and manage it in **Stack Management**. Search sessions are [enabled by default](asciidocalypse://docs/kibana/docs/reference/configuration-reference/search-sessions-settings.md). +Save your search session from **Discover** or **Dashboard**, and when your session is complete, view and manage it in **Stack Management**. Search sessions are [enabled by default](kibana://reference/configuration-reference/search-sessions-settings.md). :::{image} ../../images/kibana-search-session.png :alt: Search Session indicator displaying the current state of the search diff --git a/explore-analyze/find-and-organize/data-views.md b/explore-analyze/find-and-organize/data-views.md index a7dc8db7c..07591d7b1 100644 --- a/explore-analyze/find-and-organize/data-views.md +++ b/explore-analyze/find-and-organize/data-views.md @@ -476,7 +476,7 @@ Built-in validation is unsupported for scripted fields. When your scripts contai 5. Select **Set format**, then enter the **Format** for the field. ::::{note} -For numeric fields the default field formatters are based on the `meta.unit` field. The unit is associated with a [time unit](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#time-units), percent, or byte. The convention for percents is to use value 1 to mean 100%. +For numeric fields the default field formatters are based on the `meta.unit` field. The unit is associated with a [time unit](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#time-units), percent, or byte. The convention for percents is to use value 1 to mean 100%. :::: @@ -613,7 +613,7 @@ Numeric fields support **Bytes**, **Color**, **Duration**, **Histogram**, **Numb The **Bytes**, **Number**, and **Percentage** formatters enable you to choose the display formats of numbers in the field using the [Elastic numeral pattern](../../explore-analyze/numeral-formatting.md) syntax that {{kib}} maintains. -The **Histogram** formatter is used only for the [histogram field type](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/histogram.md). When you use the **Histogram** formatter, you can apply the **Bytes**, **Number**, or **Percentage** format to aggregated data. +The **Histogram** formatter is used only for the [histogram field type](elasticsearch://reference/elasticsearch/mapping-reference/histogram.md). When you use the **Histogram** formatter, you can apply the **Bytes**, **Number**, or **Percentage** format to aggregated data. You can specify the following types to the `Url` field formatter: diff --git a/explore-analyze/find-and-organize/saved-objects.md b/explore-analyze/find-and-organize/saved-objects.md index ce93453cf..951f5f4b6 100644 --- a/explore-analyze/find-and-organize/saved-objects.md +++ b/explore-analyze/find-and-organize/saved-objects.md @@ -149,7 +149,7 @@ After you upgrade, or if you set up a new {{kib}} instance using version 8.x or #### Accessing saved objects using old URLs [saved-object-ids-impact-when-using-legacy-urls] -When you upgrade {{kib}} and saved object IDs change, the "deep link" URLs to access those saved objects will also change. To reduce the impact, each existing URL is preserved with a special [legacy URL alias](asciidocalypse://docs/kibana/docs/extend/legacy-url-aliases.md). This means that if you use a bookmark for a saved object ID that was changed, you’ll be redirected to the new URL for that saved object. +When you upgrade {{kib}} and saved object IDs change, the "deep link" URLs to access those saved objects will also change. To reduce the impact, each existing URL is preserved with a special [legacy URL alias](kibana://extend/legacy-url-aliases.md). This means that if you use a bookmark for a saved object ID that was changed, you’ll be redirected to the new URL for that saved object. #### Importing and copying saved objects [saved-object-ids-impact-when-using-import-and-copy] diff --git a/explore-analyze/geospatial-analysis.md b/explore-analyze/geospatial-analysis.md index 155fb30d1..b9a90ba13 100644 --- a/explore-analyze/geospatial-analysis.md +++ b/explore-analyze/geospatial-analysis.md @@ -15,7 +15,7 @@ Not sure where to get started with {{es}} and geo? Then, you have come to the ri ## Geospatial mapping [geospatial-mapping] -{{es}} supports two types of geo data: [geo_point](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/geo-point.md) fields which support lat/lon pairs, and [geo_shape](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/geo-shape.md) fields, which support points, lines, circles, polygons, multi-polygons, and so on. Use [explicit mapping](../manage-data/data-store/mapping/explicit-mapping.md) to index geo data fields. +{{es}} supports two types of geo data: [geo_point](elasticsearch://reference/elasticsearch/mapping-reference/geo-point.md) fields which support lat/lon pairs, and [geo_shape](elasticsearch://reference/elasticsearch/mapping-reference/geo-shape.md) fields, which support points, lines, circles, polygons, multi-polygons, and so on. Use [explicit mapping](../manage-data/data-store/mapping/explicit-mapping.md) to index geo data fields. Have an index with lat/lon pairs but no geo_point mapping? Use [runtime fields](../manage-data/data-store/mapping/map-runtime-field.md) to make a geo_point field without reindexing. @@ -24,46 +24,46 @@ Have an index with lat/lon pairs but no geo_point mapping? Use [runtime fields]( Data is often messy and incomplete. [Ingest pipelines](../manage-data/ingest/transform-enrich/ingest-pipelines.md) lets you clean, transform, and augment your data before indexing. -* Use [CSV](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/csv-processor.md) together with [explicit mapping](../manage-data/data-store/mapping/explicit-mapping.md) to index CSV files with geo data. Kibana’s [Import CSV](visualize/maps/import-geospatial-data.md) feature can help with this. -* Use [GeoIP](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/geoip-processor.md) to add geographical location of an IPv4 or IPv6 address. -* Use [geo-grid processor](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/ingest-geo-grid-processor.md) to convert grid tiles or hexagonal cell ids to bounding boxes or polygons which describe their shape. +* Use [CSV](elasticsearch://reference/ingestion-tools/enrich-processor/csv-processor.md) together with [explicit mapping](../manage-data/data-store/mapping/explicit-mapping.md) to index CSV files with geo data. Kibana’s [Import CSV](visualize/maps/import-geospatial-data.md) feature can help with this. +* Use [GeoIP](elasticsearch://reference/ingestion-tools/enrich-processor/geoip-processor.md) to add geographical location of an IPv4 or IPv6 address. +* Use [geo-grid processor](elasticsearch://reference/ingestion-tools/enrich-processor/ingest-geo-grid-processor.md) to convert grid tiles or hexagonal cell ids to bounding boxes or polygons which describe their shape. * Use [geo_match enrich policy](../manage-data/ingest/transform-enrich/example-enrich-data-based-on-geolocation.md) for reverse geocoding. For example, use [reverse geocoding](visualize/maps/reverse-geocoding-tutorial.md) to visualize metropolitan areas by web traffic. ## Query [geospatial-query] -[Geo queries](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/geo-queries.md) answer location-driven questions. Find documents that intersect with, are within, are contained by, or do not intersect your query geometry. Combine geospatial queries with full text search queries for unparalleled searching experience. For example, "Show me all subscribers that live within 5 miles of our new gym location, that joined in the last year and have running mentioned in their profile". +[Geo queries](elasticsearch://reference/query-languages/geo-queries.md) answer location-driven questions. Find documents that intersect with, are within, are contained by, or do not intersect your query geometry. Combine geospatial queries with full text search queries for unparalleled searching experience. For example, "Show me all subscribers that live within 5 miles of our new gym location, that joined in the last year and have running mentioned in their profile". ## ES|QL [esql-query] -[ES|QL](query-filter/languages/esql.md) has support for [Geospatial Search](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-functions-operators.md#esql-spatial-functions) functions, enabling efficient index searching for documents that intersect with, are within, are contained by, or are disjoint from a query geometry. In addition, the `ST_DISTANCE` function calculates the distance between two points. +[ES|QL](query-filter/languages/esql.md) has support for [Geospatial Search](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-spatial-functions) functions, enabling efficient index searching for documents that intersect with, are within, are contained by, or are disjoint from a query geometry. In addition, the `ST_DISTANCE` function calculates the distance between two points. -* [`ST_INTERSECTS`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-functions-operators.md#esql-st_intersects) -* [`ST_DISJOINT`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-functions-operators.md#esql-st_disjoint) -* [`ST_CONTAINS`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-functions-operators.md#esql-st_contains) -* [`ST_WITHIN`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-functions-operators.md#esql-st_within) -* [`ST_DISTANCE`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-functions-operators.md#esql-st_distance) +* [`ST_INTERSECTS`](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-st_intersects) +* [`ST_DISJOINT`](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-st_disjoint) +* [`ST_CONTAINS`](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-st_contains) +* [`ST_WITHIN`](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-st_within) +* [`ST_DISTANCE`](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-st_distance) ## Aggregate [geospatial-aggregate] -[Aggregations](query-filter/aggregations.md) summarizes your data as metrics, statistics, or other analytics. Use [bucket aggregations](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/bucket.md) to group documents into buckets, also called bins, based on field values, ranges, or other criteria. Then, use [metric aggregations](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/metrics.md) to calculate metrics, such as a sum or average, from field values in each bucket. Compare metrics across buckets to gain insights from your data. +[Aggregations](query-filter/aggregations.md) summarizes your data as metrics, statistics, or other analytics. Use [bucket aggregations](elasticsearch://reference/data-analysis/aggregations/bucket.md) to group documents into buckets, also called bins, based on field values, ranges, or other criteria. Then, use [metric aggregations](elasticsearch://reference/data-analysis/aggregations/metrics.md) to calculate metrics, such as a sum or average, from field values in each bucket. Compare metrics across buckets to gain insights from your data. Geospatial bucket aggregations: -* [Geo-distance aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-geodistance-aggregation.md) evaluates the distance of each geo_point location from an origin point and determines the buckets it belongs to based on the ranges (a document belongs to a bucket if the distance between the document and the origin falls within the distance range of the bucket). -* [Geohash grid aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-geohashgrid-aggregation.md) groups geo_point and geo_shape values into buckets that represent a grid. -* [Geohex grid aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-geohexgrid-aggregation.md) groups geo_point and geo_shape values into buckets that represent an H3 hexagonal cell. -* [Geotile grid aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-geotilegrid-aggregation.md) groups geo_point and geo_shape values into buckets that represent a grid. Each cell corresponds to a [map tile](https://en.wikipedia.org/wiki/Tiled_web_map) as used by many online map sites. +* [Geo-distance aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-geodistance-aggregation.md) evaluates the distance of each geo_point location from an origin point and determines the buckets it belongs to based on the ranges (a document belongs to a bucket if the distance between the document and the origin falls within the distance range of the bucket). +* [Geohash grid aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-geohashgrid-aggregation.md) groups geo_point and geo_shape values into buckets that represent a grid. +* [Geohex grid aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-geohexgrid-aggregation.md) groups geo_point and geo_shape values into buckets that represent an H3 hexagonal cell. +* [Geotile grid aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-geotilegrid-aggregation.md) groups geo_point and geo_shape values into buckets that represent a grid. Each cell corresponds to a [map tile](https://en.wikipedia.org/wiki/Tiled_web_map) as used by many online map sites. Geospatial metric aggregations: -* [Geo-bounds aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-geobounds-aggregation.md) computes the geographic bounding box containing all values for a Geopoint or Geoshape field. -* [Geo-centroid aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-geocentroid-aggregation.md) computes the weighted centroid from all coordinate values for geo fields. -* [Geo-line aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-geo-line.md) aggregates all geo_point values within a bucket into a LineString ordered by the chosen sort field. Use geo_line aggregation to create [vehicle tracks](visualize/maps/asset-tracking-tutorial.md). +* [Geo-bounds aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-geobounds-aggregation.md) computes the geographic bounding box containing all values for a Geopoint or Geoshape field. +* [Geo-centroid aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-geocentroid-aggregation.md) computes the weighted centroid from all coordinate values for geo fields. +* [Geo-line aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-geo-line.md) aggregates all geo_point values within a bucket into a LineString ordered by the chosen sort field. Use geo_line aggregation to create [vehicle tracks](visualize/maps/asset-tracking-tutorial.md). -Combine aggregations to perform complex geospatial analysis. For example, to calculate the most recent GPS tracks per flight, use a [terms aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-terms-aggregation.md) to group documents into buckets per aircraft. Then use geo-line aggregation to compute a track for each aircraft. In another example, use geotile grid aggregation to group documents into a grid. Then use geo-centroid aggregation to find the weighted centroid of each grid cell. +Combine aggregations to perform complex geospatial analysis. For example, to calculate the most recent GPS tracks per flight, use a [terms aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-terms-aggregation.md) to group documents into buckets per aircraft. Then use geo-line aggregation to compute a track for each aircraft. In another example, use geotile grid aggregation to group documents into a grid. Then use geo-centroid aggregation to find the weighted centroid of each grid cell. ## Integrate [geospatial-integrate] diff --git a/explore-analyze/machine-learning/anomaly-detection/geographic-anomalies.md b/explore-analyze/machine-learning/anomaly-detection/geographic-anomalies.md index 8a3b99cce..3da797b5c 100644 --- a/explore-analyze/machine-learning/anomaly-detection/geographic-anomalies.md +++ b/explore-analyze/machine-learning/anomaly-detection/geographic-anomalies.md @@ -15,9 +15,9 @@ If your data includes geographic fields, you can use {{ml-features}} to detect a To run this type of {{anomaly-job}}, you must have [{{ml-features}} set up](../setting-up-machine-learning.md). You must also have time series data that contains spatial data types. In particular, you must have: * two comma-separated numbers of the form `latitude,longitude`, -* a [`geo_point`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/geo-point.md) field, -* a [`geo_shape`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/geo-shape.md) field that contains point values, or -* a [`geo_centroid`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-geocentroid-aggregation.md) aggregation +* a [`geo_point`](elasticsearch://reference/elasticsearch/mapping-reference/geo-point.md) field, +* a [`geo_shape`](elasticsearch://reference/elasticsearch/mapping-reference/geo-shape.md) field that contains point values, or +* a [`geo_centroid`](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-geocentroid-aggregation.md) aggregation The latitude and longitude must be in the range -180 to 180 and represent a point on the surface of the Earth. @@ -39,7 +39,7 @@ There are a few limitations to consider before you create this type of job: 1. You cannot create forecasts for {{anomaly-jobs}} that contain geographic functions. 2. You cannot add [custom rules with conditions](/explore-analyze/machine-learning/anomaly-detection/ml-ad-run-jobs.md#ml-ad-rules) to detectors that use geographic functions. -If those limitations are acceptable, try creating an {{anomaly-job}} that uses the [`lat_long` function](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ml-geo-functions.md#ml-lat-long) to analyze your own data or the sample data sets. +If those limitations are acceptable, try creating an {{anomaly-job}} that uses the [`lat_long` function](/reference/data-analysis/machine-learning/ml-geo-functions.md#ml-lat-long) to analyze your own data or the sample data sets. To create an {{anomaly-job}} that uses the `lat_long` function, in {{kib}} you must click **Create job** on the **{{ml-cap}} > {{anomaly-detect-cap}} > Jobs** page and select the advanced job wizard. Alternatively, use the [create {{anomaly-jobs}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-job). diff --git a/explore-analyze/machine-learning/anomaly-detection/ml-ad-run-jobs.md b/explore-analyze/machine-learning/anomaly-detection/ml-ad-run-jobs.md index 31588c65b..a9baa16b0 100644 --- a/explore-analyze/machine-learning/anomaly-detection/ml-ad-run-jobs.md +++ b/explore-analyze/machine-learning/anomaly-detection/ml-ad-run-jobs.md @@ -45,7 +45,7 @@ The {{ml-features}} use the concept of a *bucket* to divide the time series into The *bucket span* is part of the configuration information for an {{anomaly-job}}. It defines the time interval that is used to summarize and model the data. This is typically between 5 minutes to 1 hour and it depends on your data characteristics. When you set the bucket span, take into account the granularity at which you want to analyze, the frequency of the input data, the typical duration of the anomalies, and the frequency at which alerting is required. -The bucket span must contain a valid [time interval](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#time-units). When you create an {{anomaly-job}} in {{kib}}, you can choose to estimate a bucket span value based on your data characteristics. If you choose a value that is larger than one day or is significantly different than the estimated value, you receive an informational message. +The bucket span must contain a valid [time interval](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#time-units). When you create an {{anomaly-job}} in {{kib}}, you can choose to estimate a bucket span value based on your data characteristics. If you choose a value that is larger than one day or is significantly different than the estimated value, you receive an informational message. ### Detectors [ml-ad-detectors] diff --git a/explore-analyze/machine-learning/anomaly-detection/ml-configuring-aggregation.md b/explore-analyze/machine-learning/anomaly-detection/ml-configuring-aggregation.md index 01a6ddee2..6cca858c1 100644 --- a/explore-analyze/machine-learning/anomaly-detection/ml-configuring-aggregation.md +++ b/explore-analyze/machine-learning/anomaly-detection/ml-configuring-aggregation.md @@ -34,13 +34,13 @@ There are a number of requirements for using aggregations in {{dfeeds}}. * If your [{{dfeed}} uses aggregations with nested `terms` aggs](#aggs-dfeeds) and model plot is not enabled for the {{anomaly-job}}, neither the **Single Metric Viewer** nor the **Anomaly Explorer** can plot and display an anomaly chart. In these cases, an explanatory message is shown instead of the chart. * Your {{dfeed}} can contain multiple aggregations, but only the ones with names that match values in the job configuration are fed to the job. -* Using [scripted metric](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-scripted-metric-aggregation.md) aggregations is not supported in {{dfeeds}}. +* Using [scripted metric](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-scripted-metric-aggregation.md) aggregations is not supported in {{dfeeds}}. ## Recommendations [aggs-recommendations-dfeeds] -* When your detectors use [metric](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ml-metric-functions.md) or [sum](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ml-sum-functions.md) analytical functions, it’s recommended to set the `date_histogram` or `composite` aggregation interval to a tenth of the bucket span. This creates finer, more granular time buckets, which are ideal for this type of analysis. -* When your detectors use [count](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ml-count-functions.md) or [rare](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ml-rare-functions.md) functions, set the interval to the same value as the bucket span. -* If you have multiple influencers or partition fields or if your field cardinality is more than 1000, use [composite aggregations](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-composite-aggregation.md). +* When your detectors use [metric](/reference/data-analysis/machine-learning/ml-metric-functions.md) or [sum](/reference/data-analysis/machine-learning/ml-sum-functions.md) analytical functions, it’s recommended to set the `date_histogram` or `composite` aggregation interval to a tenth of the bucket span. This creates finer, more granular time buckets, which are ideal for this type of analysis. +* When your detectors use [count](/reference/data-analysis/machine-learning/ml-count-functions.md) or [rare](/reference/data-analysis/machine-learning/ml-rare-functions.md) functions, set the interval to the same value as the bucket span. +* If you have multiple influencers or partition fields or if your field cardinality is more than 1000, use [composite aggregations](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-composite-aggregation.md). To determine the cardinality of your data, you can run searches such as: @@ -254,10 +254,10 @@ Use the following format to define a composite aggregation in your {{dfeed}}: You can also use complex nested aggregations in {{dfeeds}}. -The next example uses the [`derivative` pipeline aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-derivative-aggregation.md) to find the first order derivative of the counter `system.network.out.bytes` for each value of the field `beat.name`. +The next example uses the [`derivative` pipeline aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-pipeline-derivative-aggregation.md) to find the first order derivative of the counter `system.network.out.bytes` for each value of the field `beat.name`. ::::{note} -`derivative` or other pipeline aggregations may not work within `composite` aggregations. See [composite aggregations and pipeline aggregations](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-composite-aggregation.md#search-aggregations-bucket-composite-aggregation-pipeline-aggregations). +`derivative` or other pipeline aggregations may not work within `composite` aggregations. See [composite aggregations and pipeline aggregations](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-composite-aggregation.md#search-aggregations-bucket-composite-aggregation-pipeline-aggregations). :::: ```js @@ -346,7 +346,7 @@ You can also use single bucket aggregations in {{dfeeds}}. The following example It is not currently possible to use `aggregate_metric_double` type fields in {{dfeeds}} without aggregations. :::: -You can use fields with the [`aggregate_metric_double`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/aggregate-metric-double.md) field type in a {{dfeed}} with aggregations. It is required to retrieve the `value_count` of the `aggregate_metric_double` filed in an aggregation and then use it as the `summary_count_field_name` to provide the correct count that represents the aggregation value. +You can use fields with the [`aggregate_metric_double`](elasticsearch://reference/elasticsearch/mapping-reference/aggregate-metric-double.md) field type in a {{dfeed}} with aggregations. It is required to retrieve the `value_count` of the `aggregate_metric_double` filed in an aggregation and then use it as the `summary_count_field_name` to provide the correct count that represents the aggregation value. In the following example, `presum` is an `aggregate_metric_double` type field that has all the possible metrics: `[ min, max, sum, value_count ]`. To use an `avg` aggregation on this field, you need to perform a `value_count` aggregation on `presum` and then set the field that contains the aggregated values `my_count` as the `summary_count_field_name`: diff --git a/explore-analyze/machine-learning/anomaly-detection/ml-configuring-categories.md b/explore-analyze/machine-learning/anomaly-detection/ml-configuring-categories.md index d6c898cde..28d8e9c9c 100644 --- a/explore-analyze/machine-learning/anomaly-detection/ml-configuring-categories.md +++ b/explore-analyze/machine-learning/anomaly-detection/ml-configuring-categories.md @@ -8,7 +8,7 @@ mapped_pages: # Detecting anomalous categories of data [ml-configuring-categories] -Categorization is a {{ml}} process that tokenizes a text field, clusters similar data together, and classifies it into categories. It works best on machine-written messages and application output that typically consist of repeated elements. [Categorization jobs](ml-anomaly-detection-job-types.md#categorization-jobs) enable you to find anomalous behavior in your categorized data. Categorization is not natural language processing (NLP). When you create a categorization {{anomaly-job}}, the {{ml}} model learns what volume and pattern is normal for each category over time. You can then detect anomalies and surface rare events or unusual types of messages by using [count](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ml-count-functions.md) or [rare](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ml-rare-functions.md) functions. Categorization works well on finite set of possible messages, for example: +Categorization is a {{ml}} process that tokenizes a text field, clusters similar data together, and classifies it into categories. It works best on machine-written messages and application output that typically consist of repeated elements. [Categorization jobs](ml-anomaly-detection-job-types.md#categorization-jobs) enable you to find anomalous behavior in your categorized data. Categorization is not natural language processing (NLP). When you create a categorization {{anomaly-job}}, the {{ml}} model learns what volume and pattern is normal for each category over time. You can then detect anomalies and surface rare events or unusual types of messages by using [count](/reference/data-analysis/machine-learning/ml-count-functions.md) or [rare](/reference/data-analysis/machine-learning/ml-rare-functions.md) functions. Categorization works well on finite set of possible messages, for example: ```js {"@timestamp":1549596476000, @@ -101,7 +101,7 @@ If you use the categorization wizard in {{kib}}, you can see which categorizatio :class: screenshot ::: -The categorization analyzer can refer to a built-in {{es}} analyzer or a combination of zero or more character filters, a tokenizer, and zero or more token filters. In this example, adding a [`pattern_replace` character filter](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-pattern-replace-charfilter.md) achieves the same behavior as the `categorization_filters` job configuration option described earlier. For more details about these properties, refer to the [`categorization_analyzer` API object](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-job#ml-put-job-request-body). +The categorization analyzer can refer to a built-in {{es}} analyzer or a combination of zero or more character filters, a tokenizer, and zero or more token filters. In this example, adding a [`pattern_replace` character filter](elasticsearch://reference/data-analysis/text-analysis/analysis-pattern-replace-charfilter.md) achieves the same behavior as the `categorization_filters` job configuration option described earlier. For more details about these properties, refer to the [`categorization_analyzer` API object](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-job#ml-put-job-request-body). If you use the default categorization analyzer in {{kib}} or omit the `categorization_analyzer` property from the API, the following default values are used: @@ -137,7 +137,7 @@ POST _ml/anomaly_detectors/_validate If you specify any part of the `categorization_analyzer`, however, any omitted sub-properties are *not* set to default values. -The `ml_standard` tokenizer and the day and month stopword filter are almost equivalent to the following analyzer, which is defined using only built-in {{es}} [tokenizers](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/tokenizer-reference.md) and [token filters](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/token-filter-reference.md): +The `ml_standard` tokenizer and the day and month stopword filter are almost equivalent to the following analyzer, which is defined using only built-in {{es}} [tokenizers](elasticsearch://reference/data-analysis/text-analysis/tokenizer-reference.md) and [token filters](elasticsearch://reference/data-analysis/text-analysis/token-filter-reference.md): ```console PUT _ml/anomaly_detectors/it_ops_new_logs diff --git a/explore-analyze/machine-learning/anomaly-detection/ml-configuring-transform.md b/explore-analyze/machine-learning/anomaly-detection/ml-configuring-transform.md index bfc2b4ffb..2c407d9b1 100644 --- a/explore-analyze/machine-learning/anomaly-detection/ml-configuring-transform.md +++ b/explore-analyze/machine-learning/anomaly-detection/ml-configuring-transform.md @@ -74,7 +74,7 @@ PUT /my-index-000001/_doc/1 } ``` -1. In this example, string fields are mapped as `keyword` fields to support aggregation. If you want both a full text (`text`) and a keyword (`keyword`) version of the same field, use multi-fields. For more information, see [fields](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/multi-fields.md). +1. In this example, string fields are mapped as `keyword` fields to support aggregation. If you want both a full text (`text`) and a keyword (`keyword`) version of the same field, use multi-fields. For more information, see [fields](elasticsearch://reference/elasticsearch/mapping-reference/multi-fields.md). $$$ml-configuring-transform1$$$ @@ -380,7 +380,7 @@ PUT _ml/anomaly_detectors/test3 GET _ml/datafeeds/datafeed-test3/_preview ``` -In {{es}}, location data can be stored in `geo_point` fields but this data type is not supported natively in {{ml}} analytics. This example of a runtime field transforms the data into an appropriate format. For more information, see [Geographic functions](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ml-geo-functions.md). +In {{es}}, location data can be stored in `geo_point` fields but this data type is not supported natively in {{ml}} analytics. This example of a runtime field transforms the data into an appropriate format. For more information, see [Geographic functions](/reference/data-analysis/machine-learning/ml-geo-functions.md). The preview {{dfeed}} API returns the following results, which show that `41.44` and `90.5` have been combined into "41.44,90.5": diff --git a/explore-analyze/machine-learning/anomaly-detection/ml-functions.md b/explore-analyze/machine-learning/anomaly-detection/ml-functions.md index 5a309db8c..d13240059 100644 --- a/explore-analyze/machine-learning/anomaly-detection/ml-functions.md +++ b/explore-analyze/machine-learning/anomaly-detection/ml-functions.md @@ -18,10 +18,10 @@ You can specify a `summary_count_field_name` with any function except `metric`. If your data is sparse, there may be gaps in the data which means you might have empty buckets. You might want to treat these as anomalies or you might want these gaps to be ignored. Your decision depends on your use case and what is important to you. It also depends on which functions you use. The `sum` and `count` functions are strongly affected by empty buckets. For this reason, there are `non_null_sum` and `non_zero_count` functions, which are tolerant to sparse data. These functions effectively ignore empty buckets. -* [Count functions](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ml-count-functions.md) -* [Geographic functions](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ml-geo-functions.md) -* [Information content functions](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ml-info-functions.md) -* [Metric functions](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ml-metric-functions.md) -* [Rare functions](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ml-rare-functions.md) -* [Sum functions](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ml-sum-functions.md) -* [Time functions](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ml-time-functions.md) +* [Count functions](/reference/data-analysis/machine-learning/ml-count-functions.md) +* [Geographic functions](/reference/data-analysis/machine-learning/ml-geo-functions.md) +* [Information content functions](/reference/data-analysis/machine-learning/ml-info-functions.md) +* [Metric functions](/reference/data-analysis/machine-learning/ml-metric-functions.md) +* [Rare functions](/reference/data-analysis/machine-learning/ml-rare-functions.md) +* [Sum functions](/reference/data-analysis/machine-learning/ml-sum-functions.md) +* [Time functions](/reference/data-analysis/machine-learning/ml-time-functions.md) diff --git a/explore-analyze/machine-learning/anomaly-detection/ml-getting-started.md b/explore-analyze/machine-learning/anomaly-detection/ml-getting-started.md index eb46af80b..2f02272b0 100644 --- a/explore-analyze/machine-learning/anomaly-detection/ml-getting-started.md +++ b/explore-analyze/machine-learning/anomaly-detection/ml-getting-started.md @@ -50,7 +50,7 @@ To get the best results from {{ml}} analytics, you must understand your data. Yo 6. Optional: You can change the random sampling behavior, which affects the number of documents per shard that are used in the {{data-viz}}. You can use automatic random sampling that balances accuracy and speed, manual sampling where you can chose a value for the sampling percentage, or you can turn the feaure off to use the full data set. There is a relatively small number of documents in the {{kib}} sample data, so you can turn random sampling off. For larger data sets, keep in mind that using a large sample size increases query run times and increases the load on the cluster. 7. Explore the fields in the {{data-viz}}. - You can filter the list by field names or [field types](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/field-data-types.md). The {{data-viz}} indicates how many of the documents in the sample for the selected time period contain each field. + You can filter the list by field names or [field types](elasticsearch://reference/elasticsearch/mapping-reference/field-data-types.md). The {{data-viz}} indicates how many of the documents in the sample for the selected time period contain each field. In particular, look at the `clientip`, `response.keyword`, and `url.keyword` fields, since we’ll use them in our {{anomaly-jobs}}. For these fields, the {{data-viz}} provides the number of distinct values, a list of the top values, and the number and percentage of documents that contain the field. For example: :::{image} ../../../images/machine-learning-ml-gs-data-keyword.jpg @@ -271,7 +271,7 @@ To create a forecast in {{kib}}: :class: screenshot ::: -3. Specify a duration for your forecast. This value indicates how far to extrapolate beyond the last record that was processed. You must use [time units](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#time-units). In this example, the duration is one week (`1w`): +3. Specify a duration for your forecast. This value indicates how far to extrapolate beyond the last record that was processed. You must use [time units](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#time-units). In this example, the duration is one week (`1w`): :::{image} ../../../images/machine-learning-ml-gs-duration.png :alt: Specify a duration of 1w :class: screenshot @@ -313,6 +313,6 @@ If you’re now thinking about where {{anomaly-detect}} can be most impactful fo In general, it is a good idea to start with single metric {{anomaly-jobs}} for your key performance indicators. After you examine these simple analysis results, you will have a better idea of what the influencers might be. You can create multi-metric jobs and split the data or create more complex analysis functions as necessary. For examples of more complicated configuration options, see [Examples](/explore-analyze/machine-learning/anomaly-detection/anomaly-how-tos.md). -If you want to find more sample jobs, see [Supplied configurations](ootb-ml-jobs.md). In particular, there are sample jobs for [Apache](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ootb-ml-jobs-apache.md) and [Nginx](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ootb-ml-jobs-nginx.md) that are quite similar to the examples in this tutorial. +If you want to find more sample jobs, see [Supplied configurations](ootb-ml-jobs.md). In particular, there are sample jobs for [Apache](/reference/data-analysis/machine-learning/ootb-ml-jobs-apache.md) and [Nginx](/reference/data-analysis/machine-learning/ootb-ml-jobs-nginx.md) that are quite similar to the examples in this tutorial. If you encounter problems, we’re here to help. If you are an existing Elastic customer with a support contract, please create a ticket in the [Elastic Support portal](http://support.elastic.co). Or post in the [Elastic forum](https://discuss.elastic.co/). diff --git a/explore-analyze/machine-learning/anomaly-detection/ml-limitations.md b/explore-analyze/machine-learning/anomaly-detection/ml-limitations.md index 1c548dcc5..8bb60daa0 100644 --- a/explore-analyze/machine-learning/anomaly-detection/ml-limitations.md +++ b/explore-analyze/machine-learning/anomaly-detection/ml-limitations.md @@ -20,7 +20,7 @@ The following limitations and known problems apply to the 9.0.0-beta1 release of ### CPUs must support SSE4.2 [ml-limitations-sse] -{{ml-cap}} uses Streaming SIMD Extensions (SSE) 4.2 instructions, so it works only on machines whose CPUs [support](https://en.wikipedia.org/wiki/SSE4#Supporting_CPUs) SSE4.2. If you run {{es}} on older hardware you must disable {{ml}} by setting `xpack.ml.enabled` to `false`. See [{{ml-cap}} settings in {{es}}](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/machine-learning-settings.md). +{{ml-cap}} uses Streaming SIMD Extensions (SSE) 4.2 instructions, so it works only on machines whose CPUs [support](https://en.wikipedia.org/wiki/SSE4#Supporting_CPUs) SSE4.2. If you run {{es}} on older hardware you must disable {{ml}} by setting `xpack.ml.enabled` to `false`. See [{{ml-cap}} settings in {{es}}](elasticsearch://reference/elasticsearch/configuration-reference/machine-learning-settings.md). ### CPU scheduling improvements apply to Linux and MacOS only [ml-scheduling-priority] @@ -40,7 +40,7 @@ If you send pre-aggregated data to a job for analysis, you must ensure that the ### Scripted metric aggregations are not supported [_scripted_metric_aggregations_are_not_supported] -Using [scripted metric aggregations](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-scripted-metric-aggregation.md) in {{dfeeds}} is not supported. Refer to the [Aggregating data for faster performance](ml-configuring-aggregation.md) page to learn more about aggregations in {{dfeeds}}. +Using [scripted metric aggregations](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-scripted-metric-aggregation.md) in {{dfeeds}} is not supported. Refer to the [Aggregating data for faster performance](ml-configuring-aggregation.md) page to learn more about aggregations in {{dfeeds}}. ### Fields named "by", "count", or "over" cannot be used to split data [_fields_named_by_count_or_over_cannot_be_used_to_split_data] @@ -124,7 +124,7 @@ In {{kib}}, **Anomaly Explorer** and **Single Metric Viewer** charts are not dis * for anomalies that were due to categorization (if model plot is not enabled), * if the {{dfeed}} uses scripted fields and model plot is not enabled (except for scripts that define metric fields), -* if the {{dfeed}} uses [composite aggregations](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-composite-aggregation.md) that have composite sources other than `terms` and `date_histogram`, +* if the {{dfeed}} uses [composite aggregations](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-composite-aggregation.md) that have composite sources other than `terms` and `date_histogram`, * if your [{{dfeed}} uses aggregations with nested `terms` aggs](ml-configuring-aggregation.md#aggs-dfeeds) and model plot is not enabled, * `freq_rare` functions, * `info_content`, `high_info_content`, `low_info_content` functions, @@ -138,22 +138,22 @@ The charts can also look odd in circumstances where there is very little data to | Detector functions | Function description | Supported | | --- | --- | --- | -| count, high_count, low_count, non_zero_count, low_non_zero_count | [Count functions](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ml-count-functions.md) | yes | -| count, high_count, low_count, non_zero_count, low_non_zero_count with summary_count_field_name that is not doc_count (model plot not enabled) | [Count functions](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ml-count-functions.md) | yes | -| non_zero_count with summary_count_field that is not doc_count using cardinality aggregation in datafeed config (model plot not enabled) | [Count functions](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ml-count-functions.md) | yes | -| distinct_count, high_distinct_count, low_distinct_count | [Count functions](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ml-count-functions.md) | yes | -| mean, high_mean, low_mean | [Mean, high_mean, low_mean](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ml-metric-functions.md#ml-metric-mean) | yes | -| min | [Min](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ml-metric-functions.md#ml-metric-min) | yes | -| max | [Max](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ml-metric-functions.md#ml-metric-max) | yes | -| metric | [Metric](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ml-metric-functions.md#ml-metric-metric) | yes | -| median, high_median, low_median | [Median, high_median, low_median](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ml-metric-functions.md#ml-metric-median) | yes | -| sum, high_sum ,low_sum, non_null_sum, high_non_null_sum, low_non_null_sum | [Sum functions](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ml-sum-functions.md) | yes | -| varp, high_varp, low_varp | [Varp, high_varp, low_varp](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ml-metric-functions.md#ml-metric-varp) | yes (only if model plot is enabled) | -| lat_long | [Lat_long](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ml-geo-functions.md#ml-lat-long) | no (but map is displayed in the Anomaly Explorer) | -| info_content, high_info_content, low_info_content | [Info_content, High_info_content, Low_info_content](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ml-info-functions.md#ml-info-content) | yes (only if model plot is enabled) | -| rare | [Rare](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ml-rare-functions.md#ml-rare) | yes | -| freq_rare | [Freq_rare](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ml-rare-functions.md#ml-freq-rare) | no | -| time_of_day, time_of_week | [Time functions](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ml-time-functions.md) | no | +| count, high_count, low_count, non_zero_count, low_non_zero_count | [Count functions](/reference/data-analysis/machine-learning/ml-count-functions.md) | yes | +| count, high_count, low_count, non_zero_count, low_non_zero_count with summary_count_field_name that is not doc_count (model plot not enabled) | [Count functions](/reference/data-analysis/machine-learning/ml-count-functions.md) | yes | +| non_zero_count with summary_count_field that is not doc_count using cardinality aggregation in datafeed config (model plot not enabled) | [Count functions](/reference/data-analysis/machine-learning/ml-count-functions.md) | yes | +| distinct_count, high_distinct_count, low_distinct_count | [Count functions](/reference/data-analysis/machine-learning/ml-count-functions.md) | yes | +| mean, high_mean, low_mean | [Mean, high_mean, low_mean](/reference/data-analysis/machine-learning/ml-metric-functions.md#ml-metric-mean) | yes | +| min | [Min](/reference/data-analysis/machine-learning/ml-metric-functions.md#ml-metric-min) | yes | +| max | [Max](/reference/data-analysis/machine-learning/ml-metric-functions.md#ml-metric-max) | yes | +| metric | [Metric](/reference/data-analysis/machine-learning/ml-metric-functions.md#ml-metric-metric) | yes | +| median, high_median, low_median | [Median, high_median, low_median](/reference/data-analysis/machine-learning/ml-metric-functions.md#ml-metric-median) | yes | +| sum, high_sum ,low_sum, non_null_sum, high_non_null_sum, low_non_null_sum | [Sum functions](/reference/data-analysis/machine-learning/ml-sum-functions.md) | yes | +| varp, high_varp, low_varp | [Varp, high_varp, low_varp](/reference/data-analysis/machine-learning/ml-metric-functions.md#ml-metric-varp) | yes (only if model plot is enabled) | +| lat_long | [Lat_long](/reference/data-analysis/machine-learning/ml-geo-functions.md#ml-lat-long) | no (but map is displayed in the Anomaly Explorer) | +| info_content, high_info_content, low_info_content | [Info_content, High_info_content, Low_info_content](/reference/data-analysis/machine-learning/ml-info-functions.md#ml-info-content) | yes (only if model plot is enabled) | +| rare | [Rare](/reference/data-analysis/machine-learning/ml-rare-functions.md#ml-rare) | yes | +| freq_rare | [Freq_rare](/reference/data-analysis/machine-learning/ml-rare-functions.md#ml-freq-rare) | no | +| time_of_day, time_of_week | [Time functions](/reference/data-analysis/machine-learning/ml-time-functions.md) | no | ### Jobs created in {{kib}} must use {{dfeeds}} [_jobs_created_in_kib_must_use_dfeeds] diff --git a/explore-analyze/machine-learning/anomaly-detection/ootb-ml-jobs.md b/explore-analyze/machine-learning/anomaly-detection/ootb-ml-jobs.md index 80fa96870..45f53c8d9 100644 --- a/explore-analyze/machine-learning/anomaly-detection/ootb-ml-jobs.md +++ b/explore-analyze/machine-learning/anomaly-detection/ootb-ml-jobs.md @@ -11,15 +11,15 @@ mapped_pages: {{anomaly-jobs-cap}} contain the configuration information and metadata necessary to perform an analytics task. {{kib}} can recognize certain types of data and provide specialized wizards for that context. This page lists the categories of the {{anomaly-jobs}} that are ready to use via {{kib}} in **Machine learning**. Refer to [Create {{anomaly-jobs}}](/explore-analyze/machine-learning/anomaly-detection/ml-ad-run-jobs.md#ml-ad-create-job) to learn more about creating a job by using supplied configurations. Logs and Metrics supplied configurations are available and can be created via the related solution UI in {{kib}}. -* [Apache](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ootb-ml-jobs-apache.md) -* [APM](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ootb-ml-jobs-apm.md) -* [{{auditbeat}}](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ootb-ml-jobs-auditbeat.md) -* [Logs](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ootb-ml-jobs-logs-ui.md) -* [{{metricbeat}}](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ootb-ml-jobs-metricbeat.md) -* [Metrics](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ootb-ml-jobs-metrics-ui.md) -* [Nginx](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ootb-ml-jobs-nginx.md) -* [Security](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ootb-ml-jobs-siem.md) -* [Uptime](asciidocalypse://docs/docs-content/docs/reference/data-analysis/machine-learning/ootb-ml-jobs-uptime.md) +* [Apache](/reference/data-analysis/machine-learning/ootb-ml-jobs-apache.md) +* [APM](/reference/data-analysis/machine-learning/ootb-ml-jobs-apm.md) +* [{{auditbeat}}](/reference/data-analysis/machine-learning/ootb-ml-jobs-auditbeat.md) +* [Logs](/reference/data-analysis/machine-learning/ootb-ml-jobs-logs-ui.md) +* [{{metricbeat}}](/reference/data-analysis/machine-learning/ootb-ml-jobs-metricbeat.md) +* [Metrics](/reference/data-analysis/machine-learning/ootb-ml-jobs-metrics-ui.md) +* [Nginx](/reference/data-analysis/machine-learning/ootb-ml-jobs-nginx.md) +* [Security](/reference/data-analysis/machine-learning/ootb-ml-jobs-siem.md) +* [Uptime](/reference/data-analysis/machine-learning/ootb-ml-jobs-uptime.md) ::::{note} The configurations are only available if data exists that matches the queries specified in the manifest files. These recognizer queries are linked in the descriptions of the individual configurations. diff --git a/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-classification.md b/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-classification.md index fc850eae4..f45c2c825 100644 --- a/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-classification.md +++ b/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-classification.md @@ -193,13 +193,13 @@ For instance, suppose you have an online service and you would like to predict w {{infer-cap}} can be used as a processor specified in an [ingest pipeline](../../../manage-data/ingest/transform-enrich/ingest-pipelines.md). It uses a trained model to infer against the data that is being ingested in the pipeline. The model is used on the ingest node. {{infer-cap}} pre-processes the data by using the model and provides a prediction. After the process, the pipeline continues executing (if there is any other processor in the pipeline), finally the new data together with the results are indexed into the destination index. -Check the [{{infer}} processor](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/inference-processor.md) and [the {{ml}} {{dfanalytics}} API documentation](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-ml-data-frame) to learn more. +Check the [{{infer}} processor](elasticsearch://reference/ingestion-tools/enrich-processor/inference-processor.md) and [the {{ml}} {{dfanalytics}} API documentation](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-ml-data-frame) to learn more. #### {{infer-cap}} aggregation [ml-inference-aggregation-class] {{infer-cap}} can also be used as a pipeline aggregation. You can reference a trained model in the aggregation to infer on the result field of the parent bucket aggregation. The {{infer}} aggregation uses the model on the results to provide a prediction. This aggregation enables you to run {{classification}} or {{reganalysis}} at search time. If you want to perform the analysis on a small set of data, this aggregation enables you to generate predictions without the need to set up a processor in the ingest pipeline. -Check the [{{infer}} bucket aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-inference-bucket-aggregation.md) and [the {{ml}} {{dfanalytics}} API documentation](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-ml-data-frame) to learn more. +Check the [{{infer}} bucket aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-pipeline-inference-bucket-aggregation.md) and [the {{ml}} {{dfanalytics}} API documentation](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-ml-data-frame) to learn more. ::::{note} If you use trained model aliases to reference your trained model in an {{infer}} processor or {{infer}} aggregation, you can replace your trained model with a new one without the need of updating the processor or the aggregation. Reassign the alias you used to a new trained model ID by using the [Create or update trained model aliases API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-trained-model-alias). The new trained model needs to use the same type of {{dfanalytics}} as the old one. diff --git a/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-limitations.md b/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-limitations.md index 5a717eac5..1663cbd74 100644 --- a/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-limitations.md +++ b/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-limitations.md @@ -37,7 +37,7 @@ You cannot update {{dfanalytics}} configurations. Instead, delete the {{dfanalyt ### {{dfanalytics-cap}} memory limitation [dfa-dataframe-size-limitations] -{{dfanalytics-cap}} can only perform analyses that fit into the memory available for {{ml}}. Overspill to disk is not currently possible. For general {{ml}} settings, see [{{ml-cap}} settings in {{es}}](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/machine-learning-settings.md). +{{dfanalytics-cap}} can only perform analyses that fit into the memory available for {{ml}}. Overspill to disk is not currently possible. For general {{ml}} settings, see [{{ml-cap}} settings in {{es}}](elasticsearch://reference/elasticsearch/configuration-reference/machine-learning-settings.md). When you create a {{dfanalytics-job}} and the inference step of the process fails due to the model is too large to fit into JVM, follow the steps in [this GitHub issue](https://github.com/elastic/elasticsearch/issues/76093) for a workaround. diff --git a/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-regression.md b/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-regression.md index 528ef948f..8b98eba0d 100644 --- a/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-regression.md +++ b/explore-analyze/machine-learning/data-frame-analytics/ml-dfa-regression.md @@ -139,13 +139,13 @@ For instance, suppose you have an online service and you would like to predict w {{infer-cap}} can be used as a processor specified in an [ingest pipeline](../../../manage-data/ingest/transform-enrich/ingest-pipelines.md). It uses a trained model to infer against the data that is being ingested in the pipeline. The model is used on the ingest node. {{infer-cap}} pre-processes the data by using the model and provides a prediction. After the process, the pipeline continues executing (if there is any other processor in the pipeline), finally the new data together with the results are indexed into the destination index. -Check the [{{infer}} processor](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/inference-processor.md) and [the {{ml}} {{dfanalytics}} API documentation](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-ml-data-frame) to learn more. +Check the [{{infer}} processor](elasticsearch://reference/ingestion-tools/enrich-processor/inference-processor.md) and [the {{ml}} {{dfanalytics}} API documentation](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-ml-data-frame) to learn more. #### {{infer-cap}} aggregation [ml-inference-aggregation-reg] {{infer-cap}} can also be used as a pipeline aggregation. You can reference a trained model in the aggregation to infer on the result field of the parent bucket aggregation. The {{infer}} aggregation uses the model on the results to provide a prediction. This aggregation enables you to run {{classification}} or {{reganalysis}} at search time. If you want to perform the analysis on a small set of data, this aggregation enables you to generate predictions without the need to set up a processor in the ingest pipeline. -Check the [{{infer}} bucket aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-inference-bucket-aggregation.md) and [the {{ml}} {{dfanalytics}} API documentation](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-ml-data-frame) to learn more. +Check the [{{infer}} bucket aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-pipeline-inference-bucket-aggregation.md) and [the {{ml}} {{dfanalytics}} API documentation](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-ml-data-frame) to learn more. ::::{note} If you use trained model aliases to reference your trained model in an {{infer}} processor or {{infer}} aggregation, you can replace your trained model with a new one without the need of updating the processor or the aggregation. Reassign the alias you used to a new trained model ID by using the [Create or update trained model aliases API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-trained-model-alias). The new trained model needs to use the same type of {{dfanalytics}} as the old one. diff --git a/explore-analyze/machine-learning/data-frame-analytics/ml-trained-models.md b/explore-analyze/machine-learning/data-frame-analytics/ml-trained-models.md index 41200aab0..a5cf64528 100644 --- a/explore-analyze/machine-learning/data-frame-analytics/ml-trained-models.md +++ b/explore-analyze/machine-learning/data-frame-analytics/ml-trained-models.md @@ -106,7 +106,7 @@ A few observations: ::::{note} -* Models exported from the [get trained models API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-trained-models) are limited in size by the [http.max_content_length](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md) global configuration value in {{es}}. The default value is `100mb` and may need to be increased depending on the size of model being exported. +* Models exported from the [get trained models API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-get-trained-models) are limited in size by the [http.max_content_length](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md) global configuration value in {{es}}. The default value is `100mb` and may need to be increased depending on the size of model being exported. * Connection timeouts can occur, for example, when model sizes are very large or your cluster is under load. If needed, you can increase [timeout configurations](https://ec.haxx.se/usingcurl/usingcurl-timeouts) for `curl` (for example, `curl --max-time 600`) or your client of choice. :::: @@ -115,4 +115,4 @@ If you also want to copy the {{dfanalytics-job}} to the new cluster, you can exp ## Importing an external model to the {{stack}} [import-external-model-to-es] -It is possible to import a model to your {{es}} cluster even if the model is not trained by Elastic {{dfanalytics}}. Eland supports [importing models](asciidocalypse://docs/eland/docs/reference/machine-learning.md) directly through its APIs. Please refer to the latest [Eland documentation](https://eland.readthedocs.io/en/latest/index.html) for more information on supported model types and other details of using Eland to import models with. +It is possible to import a model to your {{es}} cluster even if the model is not trained by Elastic {{dfanalytics}}. Eland supports [importing models](eland://reference/machine-learning.md) directly through its APIs. Please refer to the latest [Eland documentation](https://eland.readthedocs.io/en/latest/index.html) for more information on supported model types and other details of using Eland to import models with. diff --git a/explore-analyze/machine-learning/machine-learning-in-kibana.md b/explore-analyze/machine-learning/machine-learning-in-kibana.md index 45b3333fd..76321b4e0 100644 --- a/explore-analyze/machine-learning/machine-learning-in-kibana.md +++ b/explore-analyze/machine-learning/machine-learning-in-kibana.md @@ -35,7 +35,7 @@ File formats supported up to 60 MB: * Rich Text (RTF) * Open Document Format (ODF) -The **{{data-viz}}** identifies the file format and field mappings, and you can import the data into an {{es}} index. To change the default file size limit, see [`fileUpload:maxFileSize`](asciidocalypse://docs/kibana/docs/reference/advanced-settings.md#kibana-general-settings) in advanced settings. +The **{{data-viz}}** identifies the file format and field mappings, and you can import the data into an {{es}} index. To change the default file size limit, see [`fileUpload:maxFileSize`](kibana://reference/advanced-settings.md#kibana-general-settings) in advanced settings. If {{stack-security-features}} are enabled, users must have the necessary privileges to use {{ml-features}}. Refer to [Set up {{ml-features}}](setting-up-machine-learning.md#setup-privileges). diff --git a/explore-analyze/machine-learning/machine-learning-in-kibana/inference-processing.md b/explore-analyze/machine-learning/machine-learning-in-kibana/inference-processing.md index 08ff1ed48..e8041f1f4 100644 --- a/explore-analyze/machine-learning/machine-learning-in-kibana/inference-processing.md +++ b/explore-analyze/machine-learning/machine-learning-in-kibana/inference-processing.md @@ -35,7 +35,7 @@ Most commonly used to detect entities such as People, Places, and Organization i ### Text embedding [ingest-pipeline-search-inference-text-embedding] -Analyzing a text field using a [Text embedding](../nlp/ml-nlp-search-compare.md#ml-nlp-text-embedding) model will generate a [dense vector](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/dense-vector.md) representation of the text. This array of numeric values encodes the semantic *meaning* of the text. Using the same model with a user’s search query will produce a vector that can then be used to search, ranking results based on vector similarity - semantic similarity - as opposed to traditional word or text similarity. +Analyzing a text field using a [Text embedding](../nlp/ml-nlp-search-compare.md#ml-nlp-text-embedding) model will generate a [dense vector](elasticsearch://reference/elasticsearch/mapping-reference/dense-vector.md) representation of the text. This array of numeric values encodes the semantic *meaning* of the text. Using the same model with a user’s search query will produce a vector that can then be used to search, ranking results based on vector similarity - semantic similarity - as opposed to traditional word or text similarity. A common use case is a user searching FAQs, or a support agent searching a knowledge base, where semantically similar content may be indexed with little similarity in phrasing. diff --git a/explore-analyze/machine-learning/machine-learning-in-kibana/xpack-ml-aiops.md b/explore-analyze/machine-learning/machine-learning-in-kibana/xpack-ml-aiops.md index 558e21ac0..a76b5d9af 100644 --- a/explore-analyze/machine-learning/machine-learning-in-kibana/xpack-ml-aiops.md +++ b/explore-analyze/machine-learning/machine-learning-in-kibana/xpack-ml-aiops.md @@ -52,7 +52,7 @@ Select a field for categorization and optionally apply any filters that you want This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. :::: -Change point detection uses the [change point aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-change-point-aggregation.md) to detect distribution changes, trend changes, and other statistically significant change points in a metric of your time series data. +Change point detection uses the [change point aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-change-point-aggregation.md) to detect distribution changes, trend changes, and other statistically significant change points in a metric of your time series data. You can find change point detection under **{{ml-app}}** > **AIOps Labs** or by using the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). Here, you can select the {{data-source}} or saved Discover session that you want to analyze. diff --git a/explore-analyze/machine-learning/nlp/ml-nlp-deploy-model.md b/explore-analyze/machine-learning/nlp/ml-nlp-deploy-model.md index 59657108b..c3736bb54 100644 --- a/explore-analyze/machine-learning/nlp/ml-nlp-deploy-model.md +++ b/explore-analyze/machine-learning/nlp/ml-nlp-deploy-model.md @@ -37,4 +37,4 @@ For the resource levels when adaptive resources are enabled, refer to <[*Trained Each allocation of a model deployment has a dedicated queue to buffer {{infer}} requests. The size of this queue is determined by the `queue_capacity` parameter in the [start trained model deployment API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-start-trained-model-deployment). When the queue reaches its maximum capacity, new requests are declined until some of the queued requests are processed, creating available capacity once again. When multiple ingest pipelines reference the same deployment, the queue can fill up, resulting in rejected requests. Consider using dedicated deployments to prevent this situation. -{{infer-cap}} requests originating from search, such as the [`text_expansion` query](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-text-expansion-query.md), have a higher priority compared to non-search requests. The {{infer}} ingest processor generates normal priority requests. If both a search query and an ingest processor use the same deployment, the search requests with higher priority skip ahead in the queue for processing before the lower priority ingest requests. This prioritization accelerates search responses while potentially slowing down ingest where response time is less critical. +{{infer-cap}} requests originating from search, such as the [`text_expansion` query](elasticsearch://reference/query-languages/query-dsl-text-expansion-query.md), have a higher priority compared to non-search requests. The {{infer}} ingest processor generates normal priority requests. If both a search query and an ingest processor use the same deployment, the search requests with higher priority skip ahead in the queue for processing before the lower priority ingest requests. This prioritization accelerates search responses while potentially slowing down ingest where response time is less critical. diff --git a/explore-analyze/machine-learning/nlp/ml-nlp-elser.md b/explore-analyze/machine-learning/nlp/ml-nlp-elser.md index 488679f0c..08874b9bd 100644 --- a/explore-analyze/machine-learning/nlp/ml-nlp-elser.md +++ b/explore-analyze/machine-learning/nlp/ml-nlp-elser.md @@ -21,7 +21,7 @@ While ELSER V2 is generally available, ELSER V1 is in [preview] and will remain ## Tokens - not synonyms [elser-tokens] -ELSER expands the indexed and searched passages into collections of terms that are learned to co-occur frequently within a diverse set of training data. The terms that the text is expanded into by the model *are not* synonyms for the search terms; they are learned associations capturing relevance. These expanded terms are weighted as some of them are more significant than others. Then the {{es}} [sparse vector](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/sparse-vector.md) (or [rank features](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/rank-features.md)) field type is used to store the terms and weights at index time, and to search against later. +ELSER expands the indexed and searched passages into collections of terms that are learned to co-occur frequently within a diverse set of training data. The terms that the text is expanded into by the model *are not* synonyms for the search terms; they are learned associations capturing relevance. These expanded terms are weighted as some of them are more significant than others. Then the {{es}} [sparse vector](elasticsearch://reference/elasticsearch/mapping-reference/sparse-vector.md) (or [rank features](elasticsearch://reference/elasticsearch/mapping-reference/rank-features.md)) field type is used to store the terms and weights at index time, and to search against later. This approach provides a more understandable search experience compared to vector embeddings. However, attempting to directly interpret the tokens and weights can be misleading, as the expansion essentially results in a vector in a very high-dimensional space. Consequently, certain tokens, especially those with low weight, contain information that is intertwined with other low-weight tokens in the representation. In this regard, they function similarly to a dense vector representation, making it challenging to separate their individual contributions. This complexity can potentially lead to misinterpretations if not carefully considered during analysis. @@ -172,7 +172,7 @@ POST _ml/trained_models/.elser_model_2/deployment/_start?deployment_id=for_searc If you want to deploy ELSER in a restricted or closed network, you have two options: * create your own HTTP/HTTPS endpoint with the model artifacts on it, -* put the model artifacts into a directory inside the config directory on all [master-eligible nodes](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md#master-node). +* put the model artifacts into a directory inside the config directory on all [master-eligible nodes](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md#master-node). ### Model artifact files [elser-model-artifacts] @@ -284,7 +284,7 @@ To learn more about ELSER performance, refer to the [Benchmark information](#els ## Pre-cleaning input text [pre-cleaning] -The quality of the input text significantly affects the quality of the embeddings. To achieve the best results, it’s recommended to clean the input text before generating embeddings. The exact preprocessing you may need to do heavily depends on your text. For example, if your text contains HTML tags, use the [HTML strip processor](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/htmlstrip-processor.md) in an ingest pipeline to remove unnecessary elements. Always review and clean your input text before ingestion to eliminate any irrelevant entities that might affect the results. +The quality of the input text significantly affects the quality of the embeddings. To achieve the best results, it’s recommended to clean the input text before generating embeddings. The exact preprocessing you may need to do heavily depends on your text. For example, if your text contains HTML tags, use the [HTML strip processor](elasticsearch://reference/ingestion-tools/enrich-processor/htmlstrip-processor.md) in an ingest pipeline to remove unnecessary elements. Always review and clean your input text before ingestion to eliminate any irrelevant entities that might affect the results. ## Recommendations for using ELSER [elser-recommendations] diff --git a/explore-analyze/machine-learning/nlp/ml-nlp-import-model.md b/explore-analyze/machine-learning/nlp/ml-nlp-import-model.md index 23ceccf46..56123f99e 100644 --- a/explore-analyze/machine-learning/nlp/ml-nlp-import-model.md +++ b/explore-analyze/machine-learning/nlp/ml-nlp-import-model.md @@ -9,7 +9,7 @@ mapped_pages: # Import the trained model and vocabulary [ml-nlp-import-model] ::::{important} -If you want to install a trained model in a restricted or closed network, refer to [these instructions](asciidocalypse://docs/eland/docs/reference/machine-learning.md#ml-nlp-pytorch-air-gapped). +If you want to install a trained model in a restricted or closed network, refer to [these instructions](eland://reference/machine-learning.md#ml-nlp-pytorch-air-gapped). :::: After you choose a model, you must import it and its tokenizer vocabulary to your cluster. When you import the model, it must be chunked and imported one chunk at a time for storage in parts due to its size. @@ -22,7 +22,7 @@ Trained models must be in a TorchScript representation for use with {{stack-ml-f ## Import with the Eland client installed [ml-nlp-import-script] -1. Install the [Eland Python client](asciidocalypse://docs/eland/docs/reference/installation.md) with PyTorch extra dependencies. +1. Install the [Eland Python client](eland://reference/installation.md) with PyTorch extra dependencies. ```shell python -m pip install 'eland[pytorch]' @@ -43,7 +43,7 @@ Trained models must be in a TorchScript representation for use with {{stack-ml-f 3. Specify the identifier for the model in the Hugging Face model hub. 4. Specify the type of NLP task. Supported values are `fill_mask`, `ner`, `question_answering`, `text_classification`, `text_embedding`, `text_expansion`, `text_similarity`, and `zero_shot_classification`. -For more details, refer to [asciidocalypse://docs/eland/docs/reference/elasticsearch/elasticsearch-client-eland/machine-learning.md#ml-nlp-pytorch](asciidocalypse://docs/eland/docs/reference/machine-learning.md#ml-nlp-pytorch). +For more details, refer to [asciidocalypse://docs/eland/docs/reference/elasticsearch/elasticsearch-client-eland/machine-learning.md#ml-nlp-pytorch](eland://reference/machine-learning.md#ml-nlp-pytorch). ## Import with Docker [ml-nlp-import-docker] @@ -70,7 +70,7 @@ Replace the `$ELASTICSEARCH_URL` with the URL for your {{es}} cluster. Refer to The following authentication options are available when using the import script: * username/password authentication (specified with the `-u` and `-p` options): - + ```bash eland_import_hub_model --url https://: -u -p ... ``` diff --git a/explore-analyze/machine-learning/nlp/ml-nlp-inference.md b/explore-analyze/machine-learning/nlp/ml-nlp-inference.md index 82539d37b..3f76b8d5b 100644 --- a/explore-analyze/machine-learning/nlp/ml-nlp-inference.md +++ b/explore-analyze/machine-learning/nlp/ml-nlp-inference.md @@ -25,7 +25,7 @@ In {{kib}}, you can create and edit pipelines in **{{stack-manage-app}}** > **In ::: 1. Click **Create pipeline** or edit an existing pipeline. -2. Add an [{{infer}} processor](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/inference-processor.md) to your pipeline: +2. Add an [{{infer}} processor](elasticsearch://reference/ingestion-tools/enrich-processor/inference-processor.md) to your pipeline: 1. Click **Add a processor** and select the **{{infer-cap}}** processor type. 2. Set **Model ID** to the name of your trained model, for example `elastic__distilbert-base-cased-finetuned-conll03-english` or `lang_ident_model_1`. @@ -39,7 +39,7 @@ In {{kib}}, you can create and edit pipelines in **{{stack-manage-app}}** > **In } ``` - 2. You can also optionally add [classification configuration options](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/inference-processor.md#inference-processor-classification-opt) in the **{{infer-cap}} configuration** section. For example, to include the top five language predictions: + 2. You can also optionally add [classification configuration options](elasticsearch://reference/ingestion-tools/enrich-processor/inference-processor.md#inference-processor-classification-opt) in the **{{infer-cap}} configuration** section. For example, to include the top five language predictions: ```js { @@ -51,7 +51,7 @@ In {{kib}}, you can create and edit pipelines in **{{stack-manage-app}}** > **In 4. Click **Add** to save the processor. -3. Optional: Add a [set processor](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/set-processor.md) to index the ingest timestamp. +3. Optional: Add a [set processor](elasticsearch://reference/ingestion-tools/enrich-processor/set-processor.md) to index the ingest timestamp. 1. Click **Add a processor** and select the **Set** processor type. 2. Choose a name for the field (such as `event.ingested`) and set its value to `{{{_ingest.timestamp}}}`. For more details, refer to [Access ingest metadata in a processor](../../../manage-data/ingest/transform-enrich/ingest-pipelines.md#access-ingest-metadata). @@ -117,7 +117,7 @@ PUT ner-test ``` ::::{tip} -To use the `annotated_text` data type in this example, you must install the [mapper annotated text plugin](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch-plugins/mapper-annotated-text.md). For more installation details, refer to [Add plugins provided with {{ech}}](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch-plugins/cloud/ec-adding-elastic-plugins.md). +To use the `annotated_text` data type in this example, you must install the [mapper annotated text plugin](elasticsearch://reference/elasticsearch-plugins/mapper-annotated-text.md). For more installation details, refer to [Add plugins provided with {{ech}}](elasticsearch://reference/elasticsearch-plugins/cloud/ec-adding-elastic-plugins.md). :::: You can then use the new pipeline to index some documents. For example, use a bulk indexing request with the `pipeline` query parameter for your NER pipeline: diff --git a/explore-analyze/machine-learning/nlp/ml-nlp-limitations.md b/explore-analyze/machine-learning/nlp/ml-nlp-limitations.md index db6372216..60b4ba8f5 100644 --- a/explore-analyze/machine-learning/nlp/ml-nlp-limitations.md +++ b/explore-analyze/machine-learning/nlp/ml-nlp-limitations.md @@ -12,7 +12,7 @@ The following limitations and known problems apply to the 9.0.0-beta1 release of ## Document size limitations when using `semantic_text` fields [ml-nlp-large-documents-limit-10k-10mb] -When using semantic text to ingest documents, chunking takes place automatically. The number of chunks is limited by the [`index.mapping.nested_objects.limit`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/mapping-limit.md) cluster setting, which defaults to 10k. Documents that are too large will cause errors during ingestion. To avoid this issue, please split your documents into roughly 1MB parts before ingestion. +When using semantic text to ingest documents, chunking takes place automatically. The number of chunks is limited by the [`index.mapping.nested_objects.limit`](elasticsearch://reference/elasticsearch/index-settings/mapping-limit.md) cluster setting, which defaults to 10k. Documents that are too large will cause errors during ingestion. To avoid this issue, please split your documents into roughly 1MB parts before ingestion. ## ELSER semantic search is limited to 512 tokens per field that inference is applied to [ml-nlp-elser-v1-limit-512] diff --git a/explore-analyze/machine-learning/nlp/ml-nlp-model-ref.md b/explore-analyze/machine-learning/nlp/ml-nlp-model-ref.md index d30917239..d3b9af07d 100644 --- a/explore-analyze/machine-learning/nlp/ml-nlp-model-ref.md +++ b/explore-analyze/machine-learning/nlp/ml-nlp-model-ref.md @@ -70,7 +70,7 @@ Sparse embedding models should be configured with the `text_expansion` task type Text Embedding models are designed to work with specific scoring functions for calculating the similarity between the embeddings they produce. Examples of typical scoring functions are: `cosine`, `dot product` and `euclidean distance` (also known as `l2_norm`). -The embeddings produced by these models should be indexed in {{es}} using the [dense vector field type](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/dense-vector.md) with an appropriate [similarity function](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/dense-vector.md#dense-vector-params) chosen for the model. +The embeddings produced by these models should be indexed in {{es}} using the [dense vector field type](elasticsearch://reference/elasticsearch/mapping-reference/dense-vector.md) with an appropriate [similarity function](elasticsearch://reference/elasticsearch/mapping-reference/dense-vector.md#dense-vector-params) chosen for the model. To find similar embeddings in {{es}} use the efficient [Approximate k-nearest neighbor (kNN)](../../../solutions/search/vector/knn.md#approximate-knn) search API with a text embedding as the query vector. Approximate kNN search uses the similarity function defined in the dense vector field mapping is used to calculate the relevance. For the best results the function must be one of the suitable similarity functions for the model. diff --git a/explore-analyze/machine-learning/nlp/ml-nlp-ner-example.md b/explore-analyze/machine-learning/nlp/ml-nlp-ner-example.md index b1c2d3305..af867517b 100644 --- a/explore-analyze/machine-learning/nlp/ml-nlp-ner-example.md +++ b/explore-analyze/machine-learning/nlp/ml-nlp-ner-example.md @@ -113,7 +113,7 @@ Using the example text "Elastic is headquartered in Mountain View, California.", ## Add the NER model to an {{infer}} ingest pipeline [ex-ner-ingest] -You can perform bulk {{infer}} on documents as they are ingested by using an [{{infer}} processor](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/inference-processor.md) in your ingest pipeline. The novel *Les Misérables* by Victor Hugo is used as an example for {{infer}} in the following example. [Download](https://github.com/elastic/stack-docs/blob/8.5/docs/en/stack/ml/nlp/data/les-miserables-nd.json) the novel text split by paragraph as a JSON file, then upload it by using the [Data Visualizer](../../../manage-data/ingest/upload-data-files.md). Give the new index the name `les-miserables` when uploading the file. +You can perform bulk {{infer}} on documents as they are ingested by using an [{{infer}} processor](elasticsearch://reference/ingestion-tools/enrich-processor/inference-processor.md) in your ingest pipeline. The novel *Les Misérables* by Victor Hugo is used as an example for {{infer}} in the following example. [Download](https://github.com/elastic/stack-docs/blob/8.5/docs/en/stack/ml/nlp/data/les-miserables-nd.json) the novel text split by paragraph as a JSON file, then upload it by using the [Data Visualizer](../../../manage-data/ingest/upload-data-files.md). Give the new index the name `les-miserables` when uploading the file. Now create an ingest pipeline either in the [Stack management UI](ml-nlp-inference.md#ml-nlp-inference-processor) or by using the API: diff --git a/explore-analyze/machine-learning/nlp/ml-nlp-overview.md b/explore-analyze/machine-learning/nlp/ml-nlp-overview.md index a43acfc17..1912906d9 100644 --- a/explore-analyze/machine-learning/nlp/ml-nlp-overview.md +++ b/explore-analyze/machine-learning/nlp/ml-nlp-overview.md @@ -20,7 +20,7 @@ The [{{infer}} API](https://www.elastic.co/docs/api/doc/elasticsearch/group/endp You can **upload and manage NLP models** using the Eland client and the [{{stack}}](ml-nlp-deploy-models.md). Find the [list of recommended and compatible models here](ml-nlp-model-ref.md). Refer to [*Examples*](ml-nlp-examples.md) to learn more about how to use {{ml}} models deployed in your cluster. -You can **store embeddings in your {{es}} vector database** if you generate [dense vector](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/dense-vector.md) or [sparse vector](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/sparse-vector.md) model embeddings outside of {{es}}. +You can **store embeddings in your {{es}} vector database** if you generate [dense vector](elasticsearch://reference/elasticsearch/mapping-reference/dense-vector.md) or [sparse vector](elasticsearch://reference/elasticsearch/mapping-reference/sparse-vector.md) model embeddings outside of {{es}}. ## What is NLP? [what-is-nlp] diff --git a/explore-analyze/machine-learning/nlp/ml-nlp-rerank.md b/explore-analyze/machine-learning/nlp/ml-nlp-rerank.md index f53c75baf..c2e51af5a 100644 --- a/explore-analyze/machine-learning/nlp/ml-nlp-rerank.md +++ b/explore-analyze/machine-learning/nlp/ml-nlp-rerank.md @@ -46,7 +46,7 @@ Elastic Rerank is available in Elastic Stack version 8.17+: To download and deploy Elastic Rerank, use the [create inference API](../../elastic-inference/inference-api/elasticsearch-inference-integration.md) to create an {{es}} service `rerank` endpoint. -::::{tip} +::::{tip} Refer to this [Python notebook](https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/12-semantic-reranking-elastic-rerank.ipynb) for an end-to-end example using Elastic Rerank. :::: @@ -166,7 +166,7 @@ For a file-based access, follow these steps: * English language only * Maximum context window of 512 tokens - When using the [`semantic_text` field type](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/semantic-text.md), text is divided into chunks. By default, each chunk contains 250 words (approximately 400 tokens). Be cautious when increasing the chunk size - if the combined length of your query and chunk text exceeds 512 tokens, the model won’t have access to the full content. + When using the [`semantic_text` field type](elasticsearch://reference/elasticsearch/mapping-reference/semantic-text.md), text is divided into chunks. By default, each chunk contains 250 words (approximately 400 tokens). Be cautious when increasing the chunk size - if the combined length of your query and chunk text exceeds 512 tokens, the model won’t have access to the full content. When the combined inputs exceed the 512 token limit, a balanced truncation strategy is used. If both the query and input text are longer than 255 tokens each then both are truncated, otherwise the longest is truncated. diff --git a/explore-analyze/machine-learning/nlp/ml-nlp-search-compare.md b/explore-analyze/machine-learning/nlp/ml-nlp-search-compare.md index 984e9c668..3f899165b 100644 --- a/explore-analyze/machine-learning/nlp/ml-nlp-search-compare.md +++ b/explore-analyze/machine-learning/nlp/ml-nlp-search-compare.md @@ -13,11 +13,11 @@ The {{stack-ml-features}} can generate embeddings, which you can use to search i * [Text embedding](#ml-nlp-text-embedding) * [Text similarity](#ml-nlp-text-similarity) -## Text embedding [ml-nlp-text-embedding] +## Text embedding [ml-nlp-text-embedding] Text embedding is a task which produces a mathematical representation of text called an embedding. The {{ml}} model turns the text into an array of numerical values (also known as a *vector*). Pieces of content with similar meaning have similar representations. This means it is possible to determine whether different pieces of text are either semantically similar, different, or even opposite by using a mathematical similarity function. -This task is responsible for producing only the embedding. When the embedding is created, it can be stored in a [dense_vector](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/dense-vector.md) field and used at search time. For example, you can use these vectors in a [k-nearest neighbor (kNN) search](../../../solutions/search/vector/knn.md) to achieve semantic search capabilities. +This task is responsible for producing only the embedding. When the embedding is created, it can be stored in a [dense_vector](elasticsearch://reference/elasticsearch/mapping-reference/dense-vector.md) field and used at search time. For example, you can use these vectors in a [k-nearest neighbor (kNN) search](../../../solutions/search/vector/knn.md) to achieve semantic search capabilities. The following is an example of producing a text embedding: @@ -39,7 +39,7 @@ The task returns the following result: ... ``` -## Text similarity [ml-nlp-text-similarity] +## Text similarity [ml-nlp-text-similarity] The text similarity task estimates how similar two pieces of text are to each other and expresses the similarity in a numeric value. This is commonly referred to as cross-encoding. This task is useful for ranking document text when comparing it to another provided text input. diff --git a/explore-analyze/machine-learning/nlp/ml-nlp-text-emb-vector-search-example.md b/explore-analyze/machine-learning/nlp/ml-nlp-text-emb-vector-search-example.md index ca3609c6c..5e897abb9 100644 --- a/explore-analyze/machine-learning/nlp/ml-nlp-text-emb-vector-search-example.md +++ b/explore-analyze/machine-learning/nlp/ml-nlp-text-emb-vector-search-example.md @@ -112,7 +112,7 @@ Upload the file by using the [Data Visualizer](../../../manage-data/ingest/uploa ## Add the text embedding model to an {{infer}} ingest pipeline [ex-text-emb-ingest] -Process the initial data with an [{{infer}} processor](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/inference-processor.md). It adds an embedding for each passage. For this, create a text embedding ingest pipeline and then reindex the initial data with this pipeline. +Process the initial data with an [{{infer}} processor](elasticsearch://reference/ingestion-tools/enrich-processor/inference-processor.md). It adds an embedding for each passage. For this, create a text embedding ingest pipeline and then reindex the initial data with this pipeline. Now create an ingest pipeline either in the [{{stack-manage-app}} UI](ml-nlp-inference.md#ml-nlp-inference-processor) or by using the API: diff --git a/explore-analyze/machine-learning/setting-up-machine-learning.md b/explore-analyze/machine-learning/setting-up-machine-learning.md index 732a9708f..62819a592 100644 --- a/explore-analyze/machine-learning/setting-up-machine-learning.md +++ b/explore-analyze/machine-learning/setting-up-machine-learning.md @@ -14,8 +14,8 @@ mapped_pages: To use the {{stack}} {{ml-features}}, you must have: * the [appropriate subscription](https://www.elastic.co/subscriptions) level or the free trial period activated -* `xpack.ml.enabled` set to its default value of `true` on every node in the cluster (refer to [{{ml-cap}} settings in {{es}}](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/machine-learning-settings.md)) -* `ml` value defined in the list of `node.roles` on the [{{ml}} nodes](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md#ml-node) +* `xpack.ml.enabled` set to its default value of `true` on every node in the cluster (refer to [{{ml-cap}} settings in {{es}}](elasticsearch://reference/elasticsearch/configuration-reference/machine-learning-settings.md)) +* `ml` value defined in the list of `node.roles` on the [{{ml}} nodes](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md#ml-node) * {{ml}} features visible in the {{kib}} space * security privileges assigned to the user that: diff --git a/explore-analyze/numeral-formatting.md b/explore-analyze/numeral-formatting.md index aceb44c64..c5db85d27 100644 --- a/explore-analyze/numeral-formatting.md +++ b/explore-analyze/numeral-formatting.md @@ -12,7 +12,7 @@ Numeral formatting in {{kib}} is done through a pattern-based syntax. These patt Numeral formatting patterns are used in multiple places in {{kib}}, including: -* [Advanced settings](asciidocalypse://docs/kibana/docs/reference/advanced-settings.md) +* [Advanced settings](kibana://reference/advanced-settings.md) * [Data view formatters](find-and-organize/data-views.md#field-formatters-numeric) * [**TSVB**](visualize/legacy-editors/tsvb.md) * [**Canvas**](visualize/canvas.md) @@ -28,7 +28,7 @@ Thousands separator Accounting notation : Putting parentheses around your format like `(0.00)` will use accounting notation to show negative numbers. -The display of these patterns is affected by the [advanced setting](asciidocalypse://docs/kibana/docs/reference/advanced-settings.md#kibana-general-settings) `format:number:defaultLocale`. The default locale is `en`, but some examples will specify that they are using an alternate locale. +The display of these patterns is affected by the [advanced setting](kibana://reference/advanced-settings.md#kibana-general-settings) `format:number:defaultLocale`. The default locale is `en`, but some examples will specify that they are using an alternate locale. Most basic examples: diff --git a/explore-analyze/query-filter/aggregations.md b/explore-analyze/query-filter/aggregations.md index 41e013a39..4789c0b0c 100644 --- a/explore-analyze/query-filter/aggregations.md +++ b/explore-analyze/query-filter/aggregations.md @@ -17,13 +17,13 @@ An aggregation summarizes your data as metrics, statistics, or other analytics. {{es}} organizes aggregations into three categories: -* [Metric](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/metrics.md) aggregations that calculate metrics, such as a sum or average, from field values. -* [Bucket](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/bucket.md) aggregations that group documents into buckets, also called bins, based on field values, ranges, or other criteria. -* [Pipeline](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/pipeline.md) aggregations that take input from other aggregations instead of documents or fields. +* [Metric](elasticsearch://reference/data-analysis/aggregations/metrics.md) aggregations that calculate metrics, such as a sum or average, from field values. +* [Bucket](elasticsearch://reference/data-analysis/aggregations/bucket.md) aggregations that group documents into buckets, also called bins, based on field values, ranges, or other criteria. +* [Pipeline](elasticsearch://reference/data-analysis/aggregations/pipeline.md) aggregations that take input from other aggregations instead of documents or fields. ## Run an aggregation [run-an-agg] -You can run aggregations as part of a [search](../../solutions/search/querying-for-search.md) by specifying the [search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search)'s `aggs` parameter. The following search runs a [terms aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-terms-aggregation.md) on `my-field`: +You can run aggregations as part of a [search](../../solutions/search/querying-for-search.md) by specifying the [search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search)'s `aggs` parameter. The following search runs a [terms aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-terms-aggregation.md) on `my-field`: ```console GET /my-index-000001/_search @@ -137,7 +137,7 @@ GET /my-index-000001/_search ## Run sub-aggregations [run-sub-aggs] -Bucket aggregations support bucket or metric sub-aggregations. For example, a terms aggregation with an [avg](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-avg-aggregation.md) sub-aggregation calculates an average value for each bucket of documents. There is no level or depth limit for nesting sub-aggregations. +Bucket aggregations support bucket or metric sub-aggregations. For example, a terms aggregation with an [avg](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-avg-aggregation.md) sub-aggregation calculates an average value for each bucket of documents. There is no level or depth limit for nesting sub-aggregations. ```console GET /my-index-000001/_search @@ -244,7 +244,7 @@ GET /my-index-000001/_search?typed_keys The response returns the aggregation type as a prefix to the aggregation’s name. ::::{important} -Some aggregations return a different aggregation type from the type in the request. For example, the terms, [significant terms](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-significantterms-aggregation.md), and [percentiles](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-percentile-aggregation.md) aggregations return different aggregations types depending on the data type of the aggregated field. +Some aggregations return a different aggregation type from the type in the request. For example, the terms, [significant terms](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-significantterms-aggregation.md), and [percentiles](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-percentile-aggregation.md) aggregations return different aggregations types depending on the data type of the aggregated field. :::: ```console-result @@ -284,14 +284,14 @@ GET /my-index-000001/_search?size=0 } ``` -Scripts calculate field values dynamically, which adds a little overhead to the aggregation. In addition to the time spent calculating, some aggregations like [`terms`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-terms-aggregation.md) and [`filters`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-filters-aggregation.md) can’t use some of their optimizations with runtime fields. In total, performance costs for using a runtime field varies from aggregation to aggregation. +Scripts calculate field values dynamically, which adds a little overhead to the aggregation. In addition to the time spent calculating, some aggregations like [`terms`](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-terms-aggregation.md) and [`filters`](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-filters-aggregation.md) can’t use some of their optimizations with runtime fields. In total, performance costs for using a runtime field varies from aggregation to aggregation. ## Aggregation caches [agg-caches] -For faster responses, {{es}} caches the results of frequently run aggregations in the [shard request cache](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/shard-request-cache-settings.md). To get cached results, use the same [`preference` string](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/search-shard-routing.md#shard-and-node-preference) for each search. If you don’t need search hits, [set `size` to `0`](#return-only-agg-results) to avoid filling the cache. +For faster responses, {{es}} caches the results of frequently run aggregations in the [shard request cache](elasticsearch://reference/elasticsearch/configuration-reference/shard-request-cache-settings.md). To get cached results, use the same [`preference` string](elasticsearch://reference/elasticsearch/rest-apis/search-shard-routing.md#shard-and-node-preference) for each search. If you don’t need search hits, [set `size` to `0`](#return-only-agg-results) to avoid filling the cache. {{es}} routes searches with the same preference string to the same shards. If the shards' data doesn’t change between searches, the shards return cached aggregation results. ## Limits for `long` values [limits-for-long-values] -When running aggregations, {{es}} uses [`double`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/number.md) values to hold and represent numeric data. As a result, aggregations on [`long`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/number.md) numbers greater than `253` are approximate. +When running aggregations, {{es}} uses [`double`](elasticsearch://reference/elasticsearch/mapping-reference/number.md) values to hold and represent numeric data. As a result, aggregations on [`long`](elasticsearch://reference/elasticsearch/mapping-reference/number.md) numbers greater than `253` are approximate. diff --git a/explore-analyze/query-filter/aggregations/tutorial-analyze-ecommerce-data-with-aggregations-using-query-dsl.md b/explore-analyze/query-filter/aggregations/tutorial-analyze-ecommerce-data-with-aggregations-using-query-dsl.md index d7517bf71..def3d4f77 100644 --- a/explore-analyze/query-filter/aggregations/tutorial-analyze-ecommerce-data-with-aggregations-using-query-dsl.md +++ b/explore-analyze/query-filter/aggregations/tutorial-analyze-ecommerce-data-with-aggregations-using-query-dsl.md @@ -268,19 +268,19 @@ The response shows the field mappings for the `kibana_sample_data_ecommerce` ind :::: -The sample data includes the following [field data types](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/field-data-types.md): +The sample data includes the following [field data types](elasticsearch://reference/elasticsearch/mapping-reference/field-data-types.md): -* [`text`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/text.md) and [`keyword`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/keyword.md) for text fields - * Most `text` fields have a `.keyword` subfield for exact matching using [multi-fields](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/multi-fields.md) +* [`text`](elasticsearch://reference/elasticsearch/mapping-reference/text.md) and [`keyword`](elasticsearch://reference/elasticsearch/mapping-reference/keyword.md) for text fields + * Most `text` fields have a `.keyword` subfield for exact matching using [multi-fields](elasticsearch://reference/elasticsearch/mapping-reference/multi-fields.md) -* [`date`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/date.md) for date fields -* 3 [numeric](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/number.md) types: +* [`date`](elasticsearch://reference/elasticsearch/mapping-reference/date.md) for date fields +* 3 [numeric](elasticsearch://reference/elasticsearch/mapping-reference/number.md) types: * `integer` for whole numbers * `long` for large whole numbers * `half_float` for floating-point numbers -* [`geo_point`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/geo-point.md) for geographic coordinates -* [`object`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/object.md) for nested structures such as `products`, `geoip`, `event` +* [`geo_point`](elasticsearch://reference/elasticsearch/mapping-reference/geo-point.md) for geographic coordinates +* [`object`](elasticsearch://reference/elasticsearch/mapping-reference/object.md) for nested structures such as `products`, `geoip`, `event` Now that we understand the structure of our sample data, let’s start analyzing it. @@ -290,7 +290,7 @@ Let’s start by calculating important metrics about orders and customers. ### Get average order size [aggregations-tutorial-order-value] -Calculate the average order value across all orders in the dataset using the [`avg`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-avg-aggregation.md) aggregation. +Calculate the average order value across all orders in the dataset using the [`avg`](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-avg-aggregation.md) aggregation. ```console GET kibana_sample_data_ecommerce/_search @@ -347,7 +347,7 @@ GET kibana_sample_data_ecommerce/_search ### Get multiple order statistics at once [aggregations-tutorial-order-stats] -Calculate multiple statistics about orders in one request using the [`stats`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-stats-aggregation.md) aggregation. +Calculate multiple statistics about orders in one request using the [`stats`](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-stats-aggregation.md) aggregation. ```console GET kibana_sample_data_ecommerce/_search @@ -391,7 +391,7 @@ GET kibana_sample_data_ecommerce/_search :::: ::::{tip} -The [stats aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-stats-aggregation.md) is more efficient than running individual min, max, avg, and sum aggregations. +The [stats aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-stats-aggregation.md) is more efficient than running individual min, max, avg, and sum aggregations. :::: @@ -401,7 +401,7 @@ Let’s group orders in different ways to understand sales patterns. ### Break down sales by category [aggregations-tutorial-category-breakdown] -Group orders by category to see which product categories are most popular, using the [`terms`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-terms-aggregation.md) aggregation. +Group orders by category to see which product categories are most popular, using the [`terms`](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-terms-aggregation.md) aggregation. ```console GET kibana_sample_data_ecommerce/_search @@ -421,7 +421,7 @@ GET kibana_sample_data_ecommerce/_search 1. Name reflecting the business purpose of this breakdown 2. `terms` aggregation groups documents by field values -3. Use [`.keyword`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/keyword.md) field for exact matching on text fields +3. Use [`.keyword`](elasticsearch://reference/elasticsearch/mapping-reference/keyword.md) field for exact matching on text fields 4. Limit to top 5 categories 5. Order by number of orders (descending) @@ -476,7 +476,7 @@ GET kibana_sample_data_ecommerce/_search } ``` -1. Due to Elasticsearch’s distributed architecture, when [terms aggregations](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-terms-aggregation.md) run across multiple shards, the doc counts may have a small margin of error. This value indicates the maximum possible error in the counts. +1. Due to Elasticsearch’s distributed architecture, when [terms aggregations](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-terms-aggregation.md) run across multiple shards, the doc counts may have a small margin of error. This value indicates the maximum possible error in the counts. 2. Count of documents in categories beyond the requested size. 3. Array of category buckets, ordered by count. 4. Category name. @@ -486,7 +486,7 @@ GET kibana_sample_data_ecommerce/_search ### Track daily sales patterns [aggregations-tutorial-daily-sales] -Group orders by day to track daily sales patterns using the [`date_histogram`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-datehistogram-aggregation.md) aggregation. +Group orders by day to track daily sales patterns using the [`date_histogram`](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-datehistogram-aggregation.md) aggregation. ```console GET kibana_sample_data_ecommerce/_search @@ -507,8 +507,8 @@ GET kibana_sample_data_ecommerce/_search 1. Descriptive name for the time-series aggregation results. 2. The `date_histogram` aggregation groups documents into time-based buckets, similar to terms aggregation but for dates. -3. Uses [calendar and fixed time intervals](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-datehistogram-aggregation.md#calendar_and_fixed_intervals) to handle months with different lengths. `"day"` ensures consistent daily grouping regardless of timezone. -4. Formats dates in response using [date patterns](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-date-format.md) (e.g. "yyyy-MM-dd"). Refer to [date math expressions](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/common-options.md#date-math) for additional options. +3. Uses [calendar and fixed time intervals](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-datehistogram-aggregation.md#calendar_and_fixed_intervals) to handle months with different lengths. `"day"` ensures consistent daily grouping regardless of timezone. +4. Formats dates in response using [date patterns](elasticsearch://reference/elasticsearch/mapping-reference/mapping-date-format.md) (e.g. "yyyy-MM-dd"). Refer to [date math expressions](elasticsearch://reference/elasticsearch/rest-apis/common-options.md#date-math) for additional options. 5. When `min_doc_count` is 0, returns buckets for days with no orders, useful for continuous time series visualization. ::::{dropdown} Example response @@ -705,7 +705,7 @@ GET kibana_sample_data_ecommerce/_search ## Combine metrics with groupings [aggregations-tutorial-combined-analysis] -Now let’s calculate [metrics](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/metrics.md) within each group to get deeper insights. +Now let’s calculate [metrics](elasticsearch://reference/data-analysis/aggregations/metrics.md) within each group to get deeper insights. ### Compare category performance [aggregations-tutorial-category-metrics] @@ -827,7 +827,7 @@ GET kibana_sample_data_ecommerce/_search ``` 1. Daily revenue -2. Uses the [`cardinality`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-cardinality-aggregation.md) aggregation to count unique customers per day +2. Uses the [`cardinality`](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-cardinality-aggregation.md) aggregation to count unique customers per day 3. Average number of items per order ::::{dropdown} Example response @@ -1297,11 +1297,11 @@ GET kibana_sample_data_ecommerce/_search ## Track trends and patterns [aggregations-tutorial-trends] -You can use [pipeline aggregations](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/pipeline.md) on the results of other aggregations. Let’s analyze how metrics change over time. +You can use [pipeline aggregations](elasticsearch://reference/data-analysis/aggregations/pipeline.md) on the results of other aggregations. Let’s analyze how metrics change over time. ### Smooth out daily fluctuations [aggregations-tutorial-moving-average] -Moving averages help identify trends by reducing day-to-day noise in the data. Let’s observe sales trends more clearly by smoothing daily revenue variations, using the [Moving Function](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-movfn-aggregation.md) aggregation. +Moving averages help identify trends by reducing day-to-day noise in the data. Let’s observe sales trends more clearly by smoothing daily revenue variations, using the [Moving Function](elasticsearch://reference/data-analysis/aggregations/search-aggregations-pipeline-movfn-aggregation.md) aggregation. ```console GET kibana_sample_data_ecommerce/_search @@ -1724,7 +1724,7 @@ Notice how the smoothed values lag behind the actual values - this is because th ### Track running totals [aggregations-tutorial-cumulative] -Track running totals over time using the [`cumulative_sum`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-cumulative-sum-aggregation.md) aggregation. +Track running totals over time using the [`cumulative_sum`](elasticsearch://reference/data-analysis/aggregations/search-aggregations-pipeline-cumulative-sum-aggregation.md) aggregation. ```console GET kibana_sample_data_ecommerce/_search diff --git a/explore-analyze/query-filter/filtering.md b/explore-analyze/query-filter/filtering.md index 71fb40fb8..93c1ea0ab 100644 --- a/explore-analyze/query-filter/filtering.md +++ b/explore-analyze/query-filter/filtering.md @@ -26,7 +26,7 @@ Some apps provide more options, such as [Dashboards](../dashboards.md). ## Time filter [set-time-filter] -Display data within a specified time range when your index contains time-based events, and a time-field is configured for the selected [{{data-source}}](../find-and-organize/data-views.md). The default time range is 15 minutes, but you can customize it in [Advanced Settings](asciidocalypse://docs/kibana/docs/reference/advanced-settings.md). +Display data within a specified time range when your index contains time-based events, and a time-field is configured for the selected [{{data-source}}](../find-and-organize/data-views.md). The default time range is 15 minutes, but you can customize it in [Advanced Settings](kibana://reference/advanced-settings.md). 1. Click ![calendar icon](../../images/kibana-time-filter-icon.png). 2. Choose one of the following: diff --git a/explore-analyze/query-filter/languages/eql.md b/explore-analyze/query-filter/languages/eql.md index 0903047e3..42646e5c6 100644 --- a/explore-analyze/query-filter/languages/eql.md +++ b/explore-analyze/query-filter/languages/eql.md @@ -18,7 +18,7 @@ Event Query Language (EQL) is a query language for event-based time series data, ## Advantages of EQL [eql-advantages] * **EQL lets you express relationships between events.**
Many query languages allow you to match single events. EQL lets you match a sequence of events across different event categories and time spans. -* **EQL has a low learning curve.**
[EQL syntax](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/eql-syntax.md) looks like other common query languages, such as SQL. EQL lets you write and read queries intuitively, which makes for quick, iterative searching. +* **EQL has a low learning curve.**
[EQL syntax](elasticsearch://reference/query-languages/eql-syntax.md) looks like other common query languages, such as SQL. EQL lets you write and read queries intuitively, which makes for quick, iterative searching. * **EQL is designed for security use cases.**
While you can use it for any event-based data, we created EQL for threat hunting. EQL not only supports indicator of compromise (IOC) searches but can describe activity that goes beyond IOCs. @@ -26,7 +26,7 @@ Event Query Language (EQL) is a query language for event-based time series data, With the exception of sample queries, EQL searches require that the searched data stream or index contains a *timestamp* field. By default, EQL uses the `@timestamp` field from the [Elastic Common Schema (ECS)](https://www.elastic.co/guide/en/ecs/current). -EQL searches also require an *event category* field, unless you use the [`any` keyword](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/eql-syntax.md#eql-syntax-match-any-event-category) to search for documents without an event category field. By default, EQL uses the ECS `event.category` field. +EQL searches also require an *event category* field, unless you use the [`any` keyword](elasticsearch://reference/query-languages/eql-syntax.md#eql-syntax-match-any-event-category) to search for documents without an event category field. By default, EQL uses the ECS `event.category` field. To use a different timestamp or event category field, see [Specify a timestamp or event category field](#specify-a-timestamp-or-event-category-field). @@ -38,7 +38,7 @@ While no schema is required to use EQL, we recommend using the [ECS](https://www ## Run an EQL search [run-an-eql-search] -Use the [EQL search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-eql-search) to run a [basic EQL query](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/eql-syntax.md#eql-basic-syntax). +Use the [EQL search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-eql-search) to run a [basic EQL query](elasticsearch://reference/query-languages/eql-syntax.md#eql-basic-syntax). ```console GET /my-data-stream/_eql/search @@ -119,7 +119,7 @@ GET /my-data-stream/_eql/search ## Search for a sequence of events [eql-search-sequence] -Use EQL’s [sequence syntax](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/eql-syntax.md#eql-sequences) to search for a series of ordered events. List the event items in ascending chronological order, with the most recent event listed last: +Use EQL’s [sequence syntax](elasticsearch://reference/query-languages/eql-syntax.md#eql-sequences) to search for a series of ordered events. List the event items in ascending chronological order, with the most recent event listed last: ```console GET /my-data-stream/_eql/search @@ -188,7 +188,7 @@ The response’s `hits.sequences` property contains the 10 most recent matching } ``` -Use [`with maxspan`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/eql-syntax.md#eql-with-maxspan-keywords) to constrain matching sequences to a timespan: +Use [`with maxspan`](elasticsearch://reference/query-languages/eql-syntax.md#eql-with-maxspan-keywords) to constrain matching sequences to a timespan: ```console GET /my-data-stream/_eql/search @@ -201,7 +201,7 @@ GET /my-data-stream/_eql/search } ``` -Use `!` to match [missing events](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/eql-syntax.md#eql-missing-events): events in a sequence that do not meet a condition within a given timespan: +Use `!` to match [missing events](elasticsearch://reference/query-languages/eql-syntax.md#eql-missing-events): events in a sequence that do not meet a condition within a given timespan: ```console GET /my-data-stream/_eql/search @@ -276,7 +276,7 @@ Missing events are indicated in the response as `missing": true`: } ``` -Use the [`by` keyword](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/eql-syntax.md#eql-by-keyword) to match events that share the same field values: +Use the [`by` keyword](elasticsearch://reference/query-languages/eql-syntax.md#eql-by-keyword) to match events that share the same field values: ```console GET /my-data-stream/_eql/search @@ -320,7 +320,7 @@ The `hits.sequences.join_keys` property contains the shared field values. } ``` -Use the [`until` keyword](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/eql-syntax.md#eql-until-keyword) to specify an expiration event for sequences. Matching sequences must end before this event. +Use the [`until` keyword](elasticsearch://reference/query-languages/eql-syntax.md#eql-until-keyword) to specify an expiration event for sequences. Matching sequences must end before this event. ```console GET /my-data-stream/_eql/search @@ -337,7 +337,7 @@ GET /my-data-stream/_eql/search ## Sample chronologically unordered events [eql-search-sample] -Use EQL’s [sample syntax](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/eql-syntax.md#eql-samples) to search for events that match one or more join keys and a set of filters. Samples are similar to sequences, but do not return events in chronological order. In fact, sample queries can run on data without a timestamp. Sample queries can be useful to find correlations in events that don’t always occur in the same sequence, or that occur across long time spans. +Use EQL’s [sample syntax](elasticsearch://reference/query-languages/eql-syntax.md#eql-samples) to search for events that match one or more join keys and a set of filters. Samples are similar to sequences, but do not return events in chronological order. In fact, sample queries can run on data without a timestamp. Sample queries can be useful to find correlations in events that don’t always occur in the same sequence, or that occur across long time spans. ::::{dropdown} Click to show the sample data used in the examples below ```console @@ -553,7 +553,7 @@ POST /my-index-000003/_bulk?refresh :::: -A sample query specifies at least one join key, using the [`by` keyword](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/eql-syntax.md#eql-by-keyword), and up to five filters: +A sample query specifies at least one join key, using the [`by` keyword](elasticsearch://reference/query-languages/eql-syntax.md#eql-by-keyword), and up to five filters: ```console GET /my-index*/_eql/search @@ -871,7 +871,7 @@ GET /my-index*/_eql/search By default, each hit in the search response includes the document `_source`, which is the entire JSON object that was provided when indexing the document. -You can use the [`filter_path`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/common-options.md#common-options-response-filtering) query parameter to filter the API response. For example, the following search returns only the timestamp and PID from the `_source` of each matching event. +You can use the [`filter_path`](elasticsearch://reference/elasticsearch/rest-apis/common-options.md#common-options-response-filtering) query parameter to filter the API response. For example, the following search returns only the timestamp and PID from the `_source` of each matching event. ```console GET /my-data-stream/_eql/search?filter_path=hits.events._source.@timestamp,hits.events._source.process.pid @@ -909,12 +909,12 @@ The API returns the following response. } ``` -You can also use the `fields` parameter to retrieve and format specific fields in the response. This field is identical to the search API’s [`fields` parameter](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/retrieve-selected-fields.md). +You can also use the `fields` parameter to retrieve and format specific fields in the response. This field is identical to the search API’s [`fields` parameter](elasticsearch://reference/elasticsearch/rest-apis/retrieve-selected-fields.md). Because it consults the index mappings, the `fields` parameter provides several advantages over referencing the `_source` directly. Specifically, the `fields` parameter: * Returns each value in a standardized way that matches its mapping type -* Accepts [multi-fields](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/multi-fields.md) and [field aliases](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/field-alias.md) +* Accepts [multi-fields](elasticsearch://reference/elasticsearch/mapping-reference/multi-fields.md) and [field aliases](elasticsearch://reference/elasticsearch/mapping-reference/field-alias.md) * Formats dates and spatial data types * Retrieves [runtime field values](../../../manage-data/data-store/mapping/retrieve-runtime-field.md) * Returns fields calculated by a script at index time @@ -1055,7 +1055,7 @@ GET /my-data-stream/_eql/search } ``` -The event category field must be mapped as a [`keyword`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/keyword.md) family field type. The timestamp field should be mapped as a [`date`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/date.md) field type. [`date_nanos`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/date_nanos.md) timestamp fields are not supported. You cannot use a [`nested`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/nested.md) field or the sub-fields of a `nested` field as the timestamp or event category field. +The event category field must be mapped as a [`keyword`](elasticsearch://reference/elasticsearch/mapping-reference/keyword.md) family field type. The timestamp field should be mapped as a [`date`](elasticsearch://reference/elasticsearch/mapping-reference/date.md) field type. [`date_nanos`](elasticsearch://reference/elasticsearch/mapping-reference/date_nanos.md) timestamp fields are not supported. You cannot use a [`nested`](elasticsearch://reference/elasticsearch/mapping-reference/nested.md) field or the sub-fields of a `nested` field as the timestamp or event category field. ## Specify a sort tiebreaker [eql-search-specify-a-sort-tiebreaker] @@ -1286,5 +1286,5 @@ GET /cluster_one:my-data-stream,cluster_two:my-data-stream/_eql/search ## EQL circuit breaker settings [eql-circuit-breaker] -The relevant circuit breaker settings can be found in the [Circuit Breakers page](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/circuit-breaker-settings.md#circuit-breakers-page-eql). +The relevant circuit breaker settings can be found in the [Circuit Breakers page](elasticsearch://reference/elasticsearch/configuration-reference/circuit-breaker-settings.md#circuit-breakers-page-eql). diff --git a/explore-analyze/query-filter/languages/esql-cross-clusters.md b/explore-analyze/query-filter/languages/esql-cross-clusters.md index af921965d..4023c3ef3 100644 --- a/explore-analyze/query-filter/languages/esql-cross-clusters.md +++ b/explore-analyze/query-filter/languages/esql-cross-clusters.md @@ -361,7 +361,7 @@ Which returns: ## Enrich across clusters [ccq-enrich] -Enrich in {{esql}} across clusters operates similarly to [local enrich](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md#esql-enrich). If the enrich policy and its enrich indices are consistent across all clusters, simply write the enrich command as you would without remote clusters. In this default mode, {{esql}} can execute the enrich command on either the local cluster or the remote clusters, aiming to minimize computation or inter-cluster data transfer. Ensuring that the policy exists with consistent data on both the local cluster and the remote clusters is critical for ES|QL to produce a consistent query result. +Enrich in {{esql}} across clusters operates similarly to [local enrich](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-enrich). If the enrich policy and its enrich indices are consistent across all clusters, simply write the enrich command as you would without remote clusters. In this default mode, {{esql}} can execute the enrich command on either the local cluster or the remote clusters, aiming to minimize computation or inter-cluster data transfer. Ensuring that the policy exists with consistent data on both the local cluster and the remote clusters is critical for ES|QL to produce a consistent query result. ::::{tip} Enrich in {{esql}} across clusters using the API key based security model was introduced in version **8.15.0**. Cross cluster API keys created in versions prior to 8.15.0 will need to replaced or updated to use the new required permissions. Refer to the example in the [API key authentication](#esql-ccs-security-model-api-key) section. @@ -417,7 +417,7 @@ FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 | LIMIT 10 ``` -A `_remote` enrich cannot be executed after a [stats](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md#esql-stats-by) command. The following example would result in an error: +A `_remote` enrich cannot be executed after a [stats](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-stats-by) command. The following example would result in an error: ```esql FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 diff --git a/explore-analyze/query-filter/languages/esql-getting-started.md b/explore-analyze/query-filter/languages/esql-getting-started.md index 8f7d780db..ccb253d5d 100644 --- a/explore-analyze/query-filter/languages/esql-getting-started.md +++ b/explore-analyze/query-filter/languages/esql-getting-started.md @@ -117,13 +117,13 @@ You can adjust the editor’s height by dragging its bottom border to your likin ## Your first {{esql}} query [esql-getting-started-first-query] -Each {{esql}} query starts with a [source command](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md#esql-source-commands). A source command produces a table, typically with data from {{es}}. +Each {{esql}} query starts with a [source command](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-source-commands). A source command produces a table, typically with data from {{es}}. :::{image} ../../../images/elasticsearch-reference-source-command.svg :alt: A source command producing a table from {{es}} ::: -The [`FROM`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md#esql-from) source command returns a table with documents from a data stream, index, or alias. Each row in the resulting table represents a document. This query returns up to 1000 documents from the `sample_data` index: +The [`FROM`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-from) source command returns a table with documents from a data stream, index, or alias. Each row in the resulting table represents a document. This query returns up to 1000 documents from the `sample_data` index: ```esql FROM sample_data @@ -144,13 +144,13 @@ from sample_data ## Processing commands [esql-getting-started-limit] -A source command can be followed by one or more [processing commands](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md#esql-processing-commands), separated by a pipe character: `|`. Processing commands change an input table by adding, removing, or changing rows and columns. Processing commands can perform filtering, projection, aggregation, and more. +A source command can be followed by one or more [processing commands](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-processing-commands), separated by a pipe character: `|`. Processing commands change an input table by adding, removing, or changing rows and columns. Processing commands can perform filtering, projection, aggregation, and more. :::{image} ../../../images/elasticsearch-reference-esql-limit.png :alt: A processing command changing an input table ::: -For example, you can use the [`LIMIT`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md#esql-limit) command to limit the number of rows that are returned, up to a maximum of 10,000 rows: +For example, you can use the [`LIMIT`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-limit) command to limit the number of rows that are returned, up to a maximum of 10,000 rows: ```esql FROM sample_data @@ -174,7 +174,7 @@ FROM sample_data | LIMIT 3 :alt: A processing command sorting an input table ::: -Another processing command is the [`SORT`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md#esql-sort) command. By default, the rows returned by `FROM` don’t have a defined sort order. Use the `SORT` command to sort rows on one or more columns: +Another processing command is the [`SORT`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-sort) command. By default, the rows returned by `FROM` don’t have a defined sort order. Use the `SORT` command to sort rows on one or more columns: ```esql FROM sample_data @@ -184,14 +184,14 @@ FROM sample_data ### Query the data [esql-getting-started-where] -Use the [`WHERE`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md#esql-where) command to query the data. For example, to find all events with a duration longer than 5ms: +Use the [`WHERE`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-where) command to query the data. For example, to find all events with a duration longer than 5ms: ```esql FROM sample_data | WHERE event_duration > 5000000 ``` -`WHERE` supports several [operators](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-functions-operators.md#esql-operators). For example, you can use [`LIKE`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-functions-operators.md#esql-like-operator) to run a wildcard query against the `message` column: +`WHERE` supports several [operators](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-operators). For example, you can use [`LIKE`](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-like) to run a wildcard query against the `message` column: ```esql FROM sample_data @@ -201,7 +201,7 @@ FROM sample_data ### More processing commands [esql-getting-started-more-commands] -There are many other processing commands, like [`KEEP`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md#esql-keep) and [`DROP`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md#esql-drop) to keep or drop columns, [`ENRICH`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md#esql-enrich) to enrich a table with data from indices in {{es}}, and [`DISSECT`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md#esql-dissect) and [`GROK`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md#esql-grok) to process data. Refer to [Processing commands](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md#esql-processing-commands) for an overview of all processing commands. +There are many other processing commands, like [`KEEP`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-keep) and [`DROP`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-drop) to keep or drop columns, [`ENRICH`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-enrich) to enrich a table with data from indices in {{es}}, and [`DISSECT`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-dissect) and [`GROK`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-grok) to process data. Refer to [Processing commands](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-processing-commands) for an overview of all processing commands. ## Chain processing commands [esql-getting-started-chaining] @@ -228,14 +228,14 @@ The order of processing commands is important. First limiting the result set to ## Compute values [esql-getting-started-eval] -Use the [`EVAL`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md#esql-eval) command to append columns to a table, with calculated values. For example, the following query appends a `duration_ms` column. The values in the column are computed by dividing `event_duration` by 1,000,000. In other words: `event_duration` converted from nanoseconds to milliseconds. +Use the [`EVAL`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-eval) command to append columns to a table, with calculated values. For example, the following query appends a `duration_ms` column. The values in the column are computed by dividing `event_duration` by 1,000,000. In other words: `event_duration` converted from nanoseconds to milliseconds. ```esql FROM sample_data | EVAL duration_ms = event_duration/1000000.0 ``` -`EVAL` supports several [functions](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-functions-operators.md#esql-functions). For example, to round a number to the closest number with the specified number of digits, use the [`ROUND`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-functions-operators.md#esql-round) function: +`EVAL` supports several [functions](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-functions). For example, to round a number to the closest number with the specified number of digits, use the [`ROUND`](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-round) function: ```esql FROM sample_data @@ -245,7 +245,7 @@ FROM sample_data ## Calculate statistics [esql-getting-started-stats] -{{esql}} can not only be used to query your data, you can also use it to aggregate your data. Use the [`STATS`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md#esql-stats-by) command to calculate statistics. For example, the median duration: +{{esql}} can not only be used to query your data, you can also use it to aggregate your data. Use the [`STATS`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-stats-by) command to calculate statistics. For example, the median duration: ```esql FROM sample_data @@ -269,7 +269,7 @@ FROM sample_data ## Access columns [esql-getting-started-access-columns] -You can access columns by their name. If a name contains special characters, [it needs to be quoted](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-syntax.md#esql-identifiers) with backticks (```). +You can access columns by their name. If a name contains special characters, [it needs to be quoted](elasticsearch://reference/query-languages/esql/esql-syntax.md#esql-identifiers) with backticks (```). Assigning an explicit name to a column created by `EVAL` or `STATS` is optional. If you don’t provide a name, the new column name is equal to the function expression. For example: @@ -289,9 +289,9 @@ FROM sample_data ## Create a histogram [esql-getting-started-histogram] -To track statistics over time, {{esql}} enables you to create histograms using the [`BUCKET`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-functions-operators.md#esql-bucket) function. `BUCKET` creates human-friendly bucket sizes and returns a value for each row that corresponds to the resulting bucket the row falls into. +To track statistics over time, {{esql}} enables you to create histograms using the [`BUCKET`](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-bucket) function. `BUCKET` creates human-friendly bucket sizes and returns a value for each row that corresponds to the resulting bucket the row falls into. -Combine `BUCKET` with [`STATS`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md#esql-stats-by) to create a histogram. For example, to count the number of events per hour: +Combine `BUCKET` with [`STATS`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-stats-by) to create a histogram. For example, to count the number of events per hour: ```esql FROM sample_data @@ -309,13 +309,13 @@ FROM sample_data ## Enrich data [esql-getting-started-enrich] -{{esql}} enables you to [enrich](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-enrich-data.md) a table with data from indices in {{es}}, using the [`ENRICH`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md#esql-enrich) command. +{{esql}} enables you to [enrich](elasticsearch://reference/query-languages/esql/esql-enrich-data.md) a table with data from indices in {{es}}, using the [`ENRICH`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-enrich) command. :::{image} ../../../images/elasticsearch-reference-esql-enrich.png :alt: esql enrich ::: -Before you can use `ENRICH`, you first need to [create](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-enrich-data.md#esql-create-enrich-policy) and [execute](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-enrich-data.md#esql-execute-enrich-policy) an [enrich policy](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-enrich-data.md#esql-enrich-policy). +Before you can use `ENRICH`, you first need to [create](elasticsearch://reference/query-languages/esql/esql-enrich-data.md#esql-create-enrich-policy) and [execute](elasticsearch://reference/query-languages/esql/esql-enrich-data.md#esql-execute-enrich-policy) an [enrich policy](elasticsearch://reference/query-languages/esql/esql-enrich-data.md#esql-enrich-policy). :::::::{tab-set} @@ -386,12 +386,12 @@ FROM sample_data | STATS median_duration = MEDIAN(event_duration) BY env ``` -For more about data enrichment with {{esql}}, refer to [Data enrichment](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-enrich-data.md). +For more about data enrichment with {{esql}}, refer to [Data enrichment](elasticsearch://reference/query-languages/esql/esql-enrich-data.md). ## Process data [esql-getting-started-process-data] -Your data may contain unstructured strings that you want to [structure](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-process-data-with-dissect-grok.md) to make it easier to analyze the data. For example, the sample data contains log messages like: +Your data may contain unstructured strings that you want to [structure](elasticsearch://reference/query-languages/esql/esql-process-data-with-dissect-grok.md) to make it easier to analyze the data. For example, the sample data contains log messages like: ```txt "Connected to 10.1.0.3" @@ -399,7 +399,7 @@ Your data may contain unstructured strings that you want to [structure](asciidoc By extracting the IP address from these messages, you can determine which IP has accepted the most client connections. -To structure unstructured strings at query time, you can use the {{esql}} [`DISSECT`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md#esql-dissect) and [`GROK`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md#esql-grok) commands. `DISSECT` works by breaking up a string using a delimiter-based pattern. `GROK` works similarly, but uses regular expressions. This makes `GROK` more powerful, but generally also slower. +To structure unstructured strings at query time, you can use the {{esql}} [`DISSECT`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-dissect) and [`GROK`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-grok) commands. `DISSECT` works by breaking up a string using a delimiter-based pattern. `GROK` works similarly, but uses regular expressions. This makes `GROK` more powerful, but generally also slower. In this case, no regular expressions are needed, as the `message` is straightforward: "Connected to ", followed by the server IP. To match this string, you can use the following `DISSECT` command: @@ -419,10 +419,10 @@ FROM sample_data | STATS COUNT(*) BY server_ip ``` -For more about data processing with {{esql}}, refer to [Data processing with DISSECT and GROK](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-process-data-with-dissect-grok.md). +For more about data processing with {{esql}}, refer to [Data processing with DISSECT and GROK](elasticsearch://reference/query-languages/esql/esql-process-data-with-dissect-grok.md). ## Learn more [esql-getting-learn-more] -To learn more about {{esql}}, refer to [{{esql}} reference](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql.md). +To learn more about {{esql}}, refer to [{{esql}} reference](elasticsearch://reference/query-languages/esql.md). diff --git a/explore-analyze/query-filter/languages/esql-kibana.md b/explore-analyze/query-filter/languages/esql-kibana.md index dd0071f63..43ba26987 100644 --- a/explore-analyze/query-filter/languages/esql-kibana.md +++ b/explore-analyze/query-filter/languages/esql-kibana.md @@ -21,7 +21,7 @@ This guide shows you how to use {{esql}} in Kibana. To follow along with the que ## Enable or disable {{esql}} [esql-kibana-enable] -{{esql}} is enabled by default in {{kib}}. It can be disabled using the `enableESQL` setting from the [Advanced Settings](asciidocalypse://docs/kibana/docs/reference/advanced-settings.md). +{{esql}} is enabled by default in {{kib}}. It can be disabled using the `enableESQL` setting from the [Advanced Settings](kibana://reference/advanced-settings.md). This will hide the {{esql}} user interface from various applications. However, users will be able to access existing {{esql}} artifacts like saved searches and visualizations. @@ -39,9 +39,9 @@ After switching to {{esql}} mode, the query bar shows a sample query. For exampl from kibana_sample_data_logs | limit 10 ``` -Every query starts with a [source command](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md). In this query, the source command is [`FROM`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md#esql-from). `FROM` retrieves data from data streams, indices, or aliases. In this example, the data is retrieved from `kibana_sample_data_logs`. +Every query starts with a [source command](elasticsearch://reference/query-languages/esql/esql-commands.md). In this query, the source command is [`FROM`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-from). `FROM` retrieves data from data streams, indices, or aliases. In this example, the data is retrieved from `kibana_sample_data_logs`. -A source command can be followed by one or more [processing commands](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md). In this query, the processing command is [`LIMIT`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md#esql-limit). `LIMIT` limits the number of rows that are retrieved. +A source command can be followed by one or more [processing commands](elasticsearch://reference/query-languages/esql/esql-commands.md). In this query, the processing command is [`LIMIT`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-limit). `LIMIT` limits the number of rows that are retrieved. ::::{tip} Click the **ES|QL help** button to open the in-product reference documentation for all commands and functions or to get recommended queries that will help you get started. @@ -129,7 +129,7 @@ the 10,000 row limit only applies to the number of rows that are retrieved by th :::: -Each row shows two columns for the example query: a column with the `@timestamp` field and a column with the full document. To display specific fields from the documents, use the [`KEEP`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md#esql-keep) command: +Each row shows two columns for the example query: a column with the `@timestamp` field and a column with the full document. To display specific fields from the documents, use the [`KEEP`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-keep) command: ```esql FROM kibana_sample_data_logs @@ -151,7 +151,7 @@ The maximum number of columns in Discover is 50. If a query returns more than 50 ### Sorting [_sorting] -To sort on one of the columns, click the column name you want to sort on and select the sort order. Note that this performs client-side sorting. It only sorts the rows that were retrieved by the query, which may not be the full dataset because of the (implicit) limit. To sort the full data set, use the [`SORT`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md#esql-sort) command: +To sort on one of the columns, click the column name you want to sort on and select the sort order. Note that this performs client-side sorting. It only sorts the rows that were retrieved by the query, which may not be the full dataset because of the (implicit) limit. To sort the full data set, use the [`SORT`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-sort) command: ```esql FROM kibana_sample_data_logs @@ -179,7 +179,7 @@ FROM my_index | WHERE custom_timestamp >= ?_tstart AND custom_timestamp < ?_tend ``` -You can also use the `?_tstart` and `?_tend` parameters with the [`BUCKET`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-functions-operators.md#esql-bucket) function to create auto-incrementing time buckets in {{esql}} [visualizations](#esql-kibana-visualizations). For example: +You can also use the `?_tstart` and `?_tend` parameters with the [`BUCKET`](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-bucket) function to create auto-incrementing time buckets in {{esql}} [visualizations](#esql-kibana-visualizations). For example: ```esql FROM kibana_sample_data_logs @@ -191,7 +191,7 @@ This example uses `50` buckets, which is the maximum number of buckets. #### WHERE command [_where_command] -You can also limit the time range using the [`WHERE`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md#esql-where) command and the [`NOW`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-functions-operators.md#esql-now) function. For example, if the timestamp field is called `timestamp`, to query the last 15 minutes of data: +You can also limit the time range using the [`WHERE`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-where) command and the [`NOW`](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-now) function. For example, if the timestamp field is called `timestamp`, to query the last 15 minutes of data: ```esql FROM kibana_sample_data_logs @@ -254,7 +254,7 @@ You can also edit the {{esql}} visualization from here. Click the options button ## Create an enrich policy [esql-kibana-enrich] -The {{esql}} [`ENRICH`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md#esql-enrich) command enables you to [enrich](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-enrich-data.md) your query dataset with fields from another dataset. Before you can use `ENRICH`, you need to [create and execute an enrich policy](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-enrich-data.md#esql-set-up-enrich-policy). If a policy exists, it will be suggested by auto-complete. If not, click **Click to create** to create one. +The {{esql}} [`ENRICH`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-enrich) command enables you to [enrich](elasticsearch://reference/query-languages/esql/esql-enrich-data.md) your query dataset with fields from another dataset. Before you can use `ENRICH`, you need to [create and execute an enrich policy](elasticsearch://reference/query-languages/esql/esql-enrich-data.md#esql-set-up-enrich-policy). If a policy exists, it will be suggested by auto-complete. If not, click **Click to create** to create one. :::{image} ../../../images/elasticsearch-reference-esql-kibana-enrich-autocomplete.png :alt: esql kibana enrich autocomplete @@ -296,8 +296,8 @@ You can use {{esql}} queries to create alerts. From Discover, click **Alerts** a ## Limitations [esql-kibana-limitations] -* The user interface to filter data is not enabled when Discover is in {{esql}} mode. To filter data, write a query that uses the [`WHERE`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md#esql-where) command instead. +* The user interface to filter data is not enabled when Discover is in {{esql}} mode. To filter data, write a query that uses the [`WHERE`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-where) command instead. * Discover shows no more than 10,000 rows. This limit only applies to the number of rows that are retrieved by the query and displayed in Discover. Queries and aggregations run on the full data set. * Discover shows no more than 50 columns. If a query returns more than 50 columns, Discover only shows the first 50. * CSV export from Discover shows no more than 10,000 rows. This limit only applies to the number of rows that are retrieved by the query and displayed in Discover. Queries and aggregations run on the full data set. -* Querying many indices at once without any filters can cause an error in kibana which looks like `[esql] > Unexpected error from Elasticsearch: The content length (536885793) is bigger than the maximum allowed string (536870888)`. The response from {{esql}} is too long. Use [`DROP`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md#esql-drop) or [`KEEP`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md#esql-keep) to limit the number of fields returned. +* Querying many indices at once without any filters can cause an error in kibana which looks like `[esql] > Unexpected error from Elasticsearch: The content length (536885793) is bigger than the maximum allowed string (536870888)`. The response from {{esql}} is too long. Use [`DROP`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-drop) or [`KEEP`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-keep) to limit the number of fields returned. diff --git a/explore-analyze/query-filter/languages/esql-multi-index.md b/explore-analyze/query-filter/languages/esql-multi-index.md index 87859125d..9222e484a 100644 --- a/explore-analyze/query-filter/languages/esql-multi-index.md +++ b/explore-analyze/query-filter/languages/esql-multi-index.md @@ -25,7 +25,7 @@ FROM cluster_one:employees-00001,cluster_two:other-employees-* ``` -## Field type mismatches [esql-multi-index-invalid-mapping] +## Field type mismatches [esql-multi-index-invalid-mapping] When querying multiple indices, data streams, or aliases, you might find that the same field is mapped to multiple different types. For example, consider the two indices with the following field mappings: @@ -106,16 +106,16 @@ Cannot use field [client_ip] due to ambiguities being mapped as ``` -## Union types [esql-multi-index-union-types] +## Union types [esql-multi-index-union-types] -::::{warning} +::::{warning} This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. :::: -{{esql}} has a way to handle [field type mismatches](#esql-multi-index-invalid-mapping). When the same field is mapped to multiple types in multiple indices, the type of the field is understood to be a *union* of the various types in the index mappings. As seen in the preceding examples, this *union type* cannot be used in the results, and cannot be referred to by the query — except in `KEEP`, `DROP` or when it’s passed to a type conversion function that accepts all the types in the *union* and converts the field to a single type. {{esql}} offers a suite of [type conversion functions](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-functions-operators.md#esql-type-conversion-functions) to achieve this. +{{esql}} has a way to handle [field type mismatches](#esql-multi-index-invalid-mapping). When the same field is mapped to multiple types in multiple indices, the type of the field is understood to be a *union* of the various types in the index mappings. As seen in the preceding examples, this *union type* cannot be used in the results, and cannot be referred to by the query — except in `KEEP`, `DROP` or when it’s passed to a type conversion function that accepts all the types in the *union* and converts the field to a single type. {{esql}} offers a suite of [type conversion functions](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-type-conversion-functions) to achieve this. -In the above examples, the query can use a command like `EVAL client_ip = TO_IP(client_ip)` to resolve the union of `ip` and `keyword` to just `ip`. You can also use the type-conversion syntax `EVAL client_ip = client_ip::IP`. Alternatively, the query could use [`TO_STRING`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-functions-operators.md#esql-to_string) to convert all supported types into `KEYWORD`. +In the above examples, the query can use a command like `EVAL client_ip = TO_IP(client_ip)` to resolve the union of `ip` and `keyword` to just `ip`. You can also use the type-conversion syntax `EVAL client_ip = client_ip::IP`. Alternatively, the query could use [`TO_STRING`](elasticsearch://reference/query-languages/esql/esql-functions-operators.md#esql-to_string) to convert all supported types into `KEYWORD`. For example, the [query](#query-unsupported) that returned `client_ip:unsupported` with `null` values can be improved using the `TO_IP` function or the equivalent `field::ip` syntax. These changes also resolve the error message. As long as the only reference to the original field is to pass it to a conversion function that resolves the type ambiguity, no error results. @@ -137,9 +137,9 @@ FROM events_* | 2023-10-23T12:15:03.360Z | 172.21.2.162 | 3450233 | Connected to 10.1.0.3 | -## Index metadata [esql-multi-index-index-metadata] +## Index metadata [esql-multi-index-index-metadata] -It can be helpful to know the particular index from which each row is sourced. To get this information, use the [`METADATA`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-metadata-fields.md) option on the [`FROM`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md#esql-from) command. +It can be helpful to know the particular index from which each row is sourced. To get this information, use the [`METADATA`](elasticsearch://reference/query-languages/esql/esql-metadata-fields.md) option on the [`FROM`](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-from) command. ```esql FROM events_* METADATA _index diff --git a/explore-analyze/query-filter/languages/esql.md b/explore-analyze/query-filter/languages/esql.md index 2ea7e3e91..18607a061 100644 --- a/explore-analyze/query-filter/languages/esql.md +++ b/explore-analyze/query-filter/languages/esql.md @@ -26,9 +26,9 @@ mapped_urls: ## What's {{esql}}? [_the_esql_compute_engine] -**Elasticsearch Query Language ({{esql}})** is a piped query language for filtering, transforming, and analyzing data. +**Elasticsearch Query Language ({{esql}})** is a piped query language for filtering, transforming, and analyzing data. -You can author {{esql}} queries to find specific events, perform statistical analysis, and generate visualizations. It supports a wide range of [commands, functions and operators](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-functions-operators.md) to perform various data operations, such as filtering, aggregation, time-series analysis, and more. Today, it supports a subset of the features available in Query DSL, but it is rapidly evolving. +You can author {{esql}} queries to find specific events, perform statistical analysis, and generate visualizations. It supports a wide range of [commands, functions and operators](elasticsearch://reference/query-languages/esql/esql-functions-operators.md) to perform various data operations, such as filtering, aggregation, time-series analysis, and more. Today, it supports a subset of the features available in Query DSL, but it is rapidly evolving. ::::{note} **{{esql}}'s compute architecture** @@ -52,10 +52,10 @@ You can use it: ## Next steps Find more details about {{esql}} in the following documentation pages: -- [{{esql}} reference](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql.md): - - Reference documentation for the [{{esql}} syntax](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-syntax.md), [commands](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md), and [functions and operators](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-functions-operators.md). - - Information about working with [metadata fields](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-metadata-fields.md) and [multivalued fields](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-multivalued-fields.md). - - Guidance for [data processing with DISSECT and GROK](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-process-data-with-dissect-grok.md) and [data enrichment with ENRICH](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-enrich-data.md). +- [{{esql}} reference](elasticsearch://reference/query-languages/esql.md): + - Reference documentation for the [{{esql}} syntax](elasticsearch://reference/query-languages/esql/esql-syntax.md), [commands](elasticsearch://reference/query-languages/esql/esql-commands.md), and [functions and operators](elasticsearch://reference/query-languages/esql/esql-functions-operators.md). + - Information about working with [metadata fields](elasticsearch://reference/query-languages/esql/esql-metadata-fields.md) and [multivalued fields](elasticsearch://reference/query-languages/esql/esql-multivalued-fields.md). + - Guidance for [data processing with DISSECT and GROK](elasticsearch://reference/query-languages/esql/esql-process-data-with-dissect-grok.md) and [data enrichment with ENRICH](elasticsearch://reference/query-languages/esql/esql-enrich-data.md). - Using {{esql}}: - An overview of using the [`_query` API endpoint](/explore-analyze/query-filter/languages/esql-rest.md). @@ -64,7 +64,7 @@ Find more details about {{esql}} in the following documentation pages: - [Using {{esql}} across clusters](/explore-analyze/query-filter/languages/esql-cross-clusters.md). - [Task management](/explore-analyze/query-filter/languages/esql-task-management.md). -- [Limitations](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/limitations.md): The current limitations of {{esql}}. +- [Limitations](elasticsearch://reference/query-languages/esql/limitations.md): The current limitations of {{esql}}. - [Examples](/explore-analyze/query-filter/languages/esql.md): A few examples of what you can do with {{esql}}. diff --git a/explore-analyze/query-filter/languages/example-detect-threats-with-eql.md b/explore-analyze/query-filter/languages/example-detect-threats-with-eql.md index 40f6af614..902cb08fb 100644 --- a/explore-analyze/query-filter/languages/example-detect-threats-with-eql.md +++ b/explore-analyze/query-filter/languages/example-detect-threats-with-eql.md @@ -19,7 +19,7 @@ One common variant of regsvr32 misuse is a [Squiblydoo attack](https://attack.mi ``` -## Setup [eql-ex-threat-detection-setup] +## Setup [eql-ex-threat-detection-setup] This tutorial uses a test dataset from [Atomic Red Team](https://github.com/redcanaryco/atomic-red-team) that includes events imitating a Squiblydoo attack. The data has been mapped to [Elastic Common Schema (ECS)](https://www.elastic.co/guide/en/ecs/current) fields. @@ -58,7 +58,7 @@ To get started: -## Get a count of regsvr32 events [eql-ex-get-a-count-of-regsvr32-events] +## Get a count of regsvr32 events [eql-ex-get-a-count-of-regsvr32-events] First, get a count of events associated with a `regsvr32.exe` process: @@ -95,7 +95,7 @@ The response returns 143 related events. ``` -## Check for command line artifacts [eql-ex-check-for-command-line-artifacts] +## Check for command line artifacts [eql-ex-check-for-command-line-artifacts] `regsvr32.exe` processes were associated with 143 events. But how was `regsvr32.exe` first called? And who called it? `regsvr32.exe` is a command-line utility. Narrow your results to processes where the command line was used: @@ -155,7 +155,7 @@ The query matches one event with an `event.type` of `creation`, indicating the s ``` -## Check for malicious script loads [eql-ex-check-for-malicious-script-loads] +## Check for malicious script loads [eql-ex-check-for-malicious-script-loads] Check if `regsvr32.exe` later loads the `scrobj.dll` library: @@ -205,9 +205,9 @@ The query matches an event, confirming `scrobj.dll` was loaded. ``` -## Determine the likelihood of success [eql-ex-detemine-likelihood-of-success] +## Determine the likelihood of success [eql-ex-detemine-likelihood-of-success] -In many cases, attackers use malicious scripts to connect to remote servers or download other files. Use an [EQL sequence query](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/eql-syntax.md#eql-sequences) to check for the following series of events: +In many cases, attackers use malicious scripts to connect to remote servers or download other files. Use an [EQL sequence query](elasticsearch://reference/query-languages/eql-syntax.md#eql-sequences) to check for the following series of events: 1. A `regsvr32.exe` process 2. A load of the `scrobj.dll` library by the same process diff --git a/explore-analyze/query-filter/languages/kql.md b/explore-analyze/query-filter/languages/kql.md index 474b88c01..fbfcfbdd5 100644 --- a/explore-analyze/query-filter/languages/kql.md +++ b/explore-analyze/query-filter/languages/kql.md @@ -30,7 +30,7 @@ Combine free text search with field-based search using KQL. Type a search term t -## Filter for documents where a field exists [_filter_for_documents_where_a_field_exists] +## Filter for documents where a field exists [_filter_for_documents_where_a_field_exists] To filter documents for which an indexed value exists for a given field, use the `*` operator. For example, to filter for documents where the `http.request.method` field exists, use the following syntax: @@ -41,7 +41,7 @@ http.request.method: * This checks for any indexed value, including an empty string. -## Filter for documents that match a value [_filter_for_documents_that_match_a_value] +## Filter for documents that match a value [_filter_for_documents_that_match_a_value] Use KQL to filter for documents that match a specific number, text, date, or boolean value. For example, to filter for documents where the `http.request.method` is GET, use the following query: @@ -81,7 +81,7 @@ You must escape following characters: ``` -## Filter for documents within a range [_filter_for_documents_within_a_range] +## Filter for documents within a range [_filter_for_documents_within_a_range] To search documents that contain terms within a provided range, use KQL’s range syntax. For example, to search for all documents for which `http.response.bytes` is less than 10000, use the following syntax: @@ -101,10 +101,10 @@ You can also use range syntax for string values, IP addresses, and timestamps. F @timestamp < now-2w ``` -For more examples on acceptable date formats, refer to [Date Math](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/common-options.md#date-math). +For more examples on acceptable date formats, refer to [Date Math](elasticsearch://reference/elasticsearch/rest-apis/common-options.md#date-math). -## Filter for documents using wildcards [_filter_for_documents_using_wildcards] +## Filter for documents using wildcards [_filter_for_documents_using_wildcards] To search for documents matching a pattern, use the wildcard syntax. For example, to find documents where `http.response.status_code` begins with a 4, use the following syntax: @@ -112,15 +112,15 @@ To search for documents matching a pattern, use the wildcard syntax. For example http.response.status_code: 4* ``` -By default, leading wildcards are not allowed for performance reasons. You can modify this with the [`query:allowLeadingWildcards`](asciidocalypse://docs/kibana/docs/reference/advanced-settings.md#query-allowleadingwildcards) advanced setting. +By default, leading wildcards are not allowed for performance reasons. You can modify this with the [`query:allowLeadingWildcards`](kibana://reference/advanced-settings.md#query-allowleadingwildcards) advanced setting. -::::{note} +::::{note} Only `*` is currently supported. This matches zero or more characters. :::: -## Negating a query [_negating_a_query] +## Negating a query [_negating_a_query] To negate or exclude a set of documents, use the `not` keyword (not case-sensitive). For example, to filter documents where the `http.request.method` is **not** GET, use the following query: @@ -129,7 +129,7 @@ NOT http.request.method: GET ``` -## Combining multiple queries [_combining_multiple_queries] +## Combining multiple queries [_combining_multiple_queries] To combine multiple queries, use the `and`/`or` keywords (not case-sensitive). For example, to find documents where the `http.request.method` is GET **or** the `http.response.status_code` is 400, use the following query: @@ -157,7 +157,7 @@ http.request.method: (GET OR POST OR DELETE) ``` -## Matching multiple fields [_matching_multiple_fields] +## Matching multiple fields [_matching_multiple_fields] Wildcards can also be used to query multiple fields. For example, to search for documents where any sub-field of `datastream` contains “logs”, use the following: @@ -165,15 +165,15 @@ Wildcards can also be used to query multiple fields. For example, to search for datastream.*: logs ``` -::::{note} +::::{note} When using wildcards to query multiple fields, errors might occur if the fields are of different types. For example, if `datastream.*` matches both numeric and string fields, the above query will result in an error because numeric fields cannot be queried for string values. :::: -## Querying nested fields [_querying_nested_fields] +## Querying nested fields [_querying_nested_fields] -Querying [nested fields](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/nested.md) requires a special syntax. Consider the following document, where `user` is a nested field: +Querying [nested fields](elasticsearch://reference/elasticsearch/mapping-reference/nested.md) requires a special syntax. Consider the following document, where `user` is a nested field: ```yaml { diff --git a/explore-analyze/query-filter/languages/lucene-query-syntax.md b/explore-analyze/query-filter/languages/lucene-query-syntax.md index 3be04055e..96f325b5f 100644 --- a/explore-analyze/query-filter/languages/lucene-query-syntax.md +++ b/explore-analyze/query-filter/languages/lucene-query-syntax.md @@ -8,7 +8,7 @@ mapped_pages: # Lucene query syntax [lucene-query] -Lucene query syntax is available to {{kib}} users who opt out of the [{{kib}} Query Language](kql.md). Full documentation for this syntax is available as part of {{es}} [query string syntax](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-query-string-query.md#query-string-syntax). +Lucene query syntax is available to {{kib}} users who opt out of the [{{kib}} Query Language](kql.md). Full documentation for this syntax is available as part of {{es}} [query string syntax](elasticsearch://reference/query-languages/query-dsl-query-string-query.md#query-string-syntax). The main reason to use the Lucene query syntax in {{kib}} is for advanced Lucene features, such as regular expressions or fuzzy term matching. However, Lucene syntax is not able to search nested objects or scripted fields. diff --git a/explore-analyze/query-filter/languages/querydsl.md b/explore-analyze/query-filter/languages/querydsl.md index f3376a021..d8fdac4b2 100644 --- a/explore-analyze/query-filter/languages/querydsl.md +++ b/explore-analyze/query-filter/languages/querydsl.md @@ -39,10 +39,10 @@ The [`_search` endpoint](../../../solutions/search/querying-for-search.md) accep Query DSL support a wide range of search techniques, including the following: * [**Full-text search**](/solutions/search/full-text.md): Search text that has been analyzed and indexed to support phrase or proximity queries, fuzzy matches, and more. -* [**Keyword search**](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/keyword.md): Search for exact matches using `keyword` fields. +* [**Keyword search**](elasticsearch://reference/elasticsearch/mapping-reference/keyword.md): Search for exact matches using `keyword` fields. * [**Semantic search**](/solutions/search/semantic-search/semantic-search-semantic-text.md): Search `semantic_text` fields using dense or sparse vector search on embeddings generated in your {{es}} cluster. * [**Vector search**](/solutions/search/vector/knn.md): Search for similar dense vectors using the kNN algorithm for embeddings generated outside of {{es}}. -* [**Geospatial search**](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/geo-queries.md): Search for locations and calculate spatial relationships using geospatial queries. +* [**Geospatial search**](elasticsearch://reference/query-languages/geo-queries.md): Search for locations and calculate spatial relationships using geospatial queries. You can also filter data using Query DSL. Filters enable you to include or exclude documents by retrieving documents that match specific field-level criteria. A query that uses the `filter` parameter indicates [filter context](#filter-context). @@ -54,9 +54,9 @@ Because aggregations leverage the same data structures used for search, they are The following aggregation types are available: -* [Metric](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/metrics.md): Calculate metrics, such as a sum or average, from field values. -* [Bucket](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/bucket.md): Group documents into buckets based on field values, ranges, or other criteria. -* [Pipeline](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/pipeline.md): Run aggregations on the results of other aggregations. +* [Metric](elasticsearch://reference/data-analysis/aggregations/metrics.md): Calculate metrics, such as a sum or average, from field values. +* [Bucket](elasticsearch://reference/data-analysis/aggregations/bucket.md): Group documents into buckets based on field values, ranges, or other criteria. +* [Pipeline](elasticsearch://reference/data-analysis/aggregations/pipeline.md): Run aggregations on the results of other aggregations. Run aggregations by specifying the [search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search)'s `aggs` parameter. Learn more in [Run an aggregation](/explore-analyze/query-filter/aggregations.md#run-an-agg). @@ -65,9 +65,9 @@ Run aggregations by specifying the [search API](https://www.elastic.co/docs/api/ Think of the Query DSL as an AST (Abstract Syntax Tree) of queries, consisting of two types of clauses: -**Leaf query clauses**: Leaf query clauses look for a particular value in a particular field, such as the [`match`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-match-query.md), [`term`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-term-query.md) or [`range`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-range-query.md) queries. These queries can be used by themselves. +**Leaf query clauses**: Leaf query clauses look for a particular value in a particular field, such as the [`match`](elasticsearch://reference/query-languages/query-dsl-match-query.md), [`term`](elasticsearch://reference/query-languages/query-dsl-term-query.md) or [`range`](elasticsearch://reference/query-languages/query-dsl-range-query.md) queries. These queries can be used by themselves. -**Compound query clauses**: Compound query clauses wrap other leaf **or** compound queries and are used to combine multiple queries in a logical fashion (such as the [`bool`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-bool-query.md) or [`dis_max`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-dis-max-query.md) query), or to alter their behavior (such as the [`constant_score`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-constant-score-query.md) query). +**Compound query clauses**: Compound query clauses wrap other leaf **or** compound queries and are used to combine multiple queries in a logical fashion (such as the [`bool`](elasticsearch://reference/query-languages/query-dsl-bool-query.md) or [`dis_max`](elasticsearch://reference/query-languages/query-dsl-dis-max-query.md) query), or to alter their behavior (such as the [`constant_score`](elasticsearch://reference/query-languages/query-dsl-constant-score-query.md) query). Query clauses behave differently depending on whether they are used in [query context or filter context](#query-filter-context). @@ -77,22 +77,22 @@ $$$query-dsl-allow-expensive-queries$$$ - Queries that need to do linear scans to identify matches: - - [`script` queries](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-script-query.md) - - queries on [numeric](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/number.md), [date](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/date.md), [boolean](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/boolean.md), [ip](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/ip.md), [geo_point](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/geo-point.md) or [keyword](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/keyword.md) fields that are not indexed but have [doc values](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/doc-values.md) enabled + - [`script` queries](elasticsearch://reference/query-languages/query-dsl-script-query.md) + - queries on [numeric](elasticsearch://reference/elasticsearch/mapping-reference/number.md), [date](elasticsearch://reference/elasticsearch/mapping-reference/date.md), [boolean](elasticsearch://reference/elasticsearch/mapping-reference/boolean.md), [ip](elasticsearch://reference/elasticsearch/mapping-reference/ip.md), [geo_point](elasticsearch://reference/elasticsearch/mapping-reference/geo-point.md) or [keyword](elasticsearch://reference/elasticsearch/mapping-reference/keyword.md) fields that are not indexed but have [doc values](elasticsearch://reference/elasticsearch/mapping-reference/doc-values.md) enabled - Queries that have a high up-front cost: - - [`fuzzy` queries](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-fuzzy-query.md) (except on [`wildcard`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/keyword.md#wildcard-field-type) fields) - - [`regexp` queries](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-regexp-query.md) (except on [`wildcard`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/keyword.md#wildcard-field-type) fields) - - [`prefix` queries](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-prefix-query.md) (except on [`wildcard`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/keyword.md#wildcard-field-type) fields or those without [`index_prefixes`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/index-prefixes.md)) - - [`wildcard` queries](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-wildcard-query.md) (except on [`wildcard`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/keyword.md#wildcard-field-type) fields) - - [`range` queries](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-range-query.md) on [`text`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/text.md) and [`keyword`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/keyword.md) fields + - [`fuzzy` queries](elasticsearch://reference/query-languages/query-dsl-fuzzy-query.md) (except on [`wildcard`](elasticsearch://reference/elasticsearch/mapping-reference/keyword.md#wildcard-field-type) fields) + - [`regexp` queries](elasticsearch://reference/query-languages/query-dsl-regexp-query.md) (except on [`wildcard`](elasticsearch://reference/elasticsearch/mapping-reference/keyword.md#wildcard-field-type) fields) + - [`prefix` queries](elasticsearch://reference/query-languages/query-dsl-prefix-query.md) (except on [`wildcard`](elasticsearch://reference/elasticsearch/mapping-reference/keyword.md#wildcard-field-type) fields or those without [`index_prefixes`](elasticsearch://reference/elasticsearch/mapping-reference/index-prefixes.md)) + - [`wildcard` queries](elasticsearch://reference/query-languages/query-dsl-wildcard-query.md) (except on [`wildcard`](elasticsearch://reference/elasticsearch/mapping-reference/keyword.md#wildcard-field-type) fields) + - [`range` queries](elasticsearch://reference/query-languages/query-dsl-range-query.md) on [`text`](elasticsearch://reference/elasticsearch/mapping-reference/text.md) and [`keyword`](elasticsearch://reference/elasticsearch/mapping-reference/keyword.md) fields - - [Joining queries](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/joining-queries.md) + - [Joining queries](elasticsearch://reference/query-languages/joining-queries.md) - Queries that may have a high per-document cost: - - [`script_score` queries](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-script-score-query.md) - - [`percolate` queries](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-percolate-query.md) + - [`script_score` queries](elasticsearch://reference/query-languages/query-dsl-script-score-query.md) + - [`percolate` queries](elasticsearch://reference/query-languages/query-dsl-percolate-query.md) The execution of such queries can be prevented by setting the value of the `search.allow_expensive_queries` setting to `false` (defaults to `true`). @@ -142,9 +142,9 @@ Common filter applications include: Filter context applies when a query clause is passed to a `filter` parameter, such as: -* `filter` or `must_not` parameters in [`bool`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-bool-query.md) queries -* `filter` parameter in [`constant_score`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-constant-score-query.md) queries -* [`filter`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-filter-aggregation.md) aggregations +* `filter` or `must_not` parameters in [`bool`](elasticsearch://reference/query-languages/query-dsl-bool-query.md) queries +* `filter` parameter in [`constant_score`](elasticsearch://reference/query-languages/query-dsl-constant-score-query.md) queries +* [`filter`](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-filter-aggregation.md) aggregations Filters optimize query performance and efficiency, especially for structured data queries and when combined with full-text searches. diff --git a/explore-analyze/query-filter/languages/sql-cli.md b/explore-analyze/query-filter/languages/sql-cli.md index ac55a787f..d837bc985 100644 --- a/explore-analyze/query-filter/languages/sql-cli.md +++ b/explore-analyze/query-filter/languages/sql-cli.md @@ -54,7 +54,7 @@ $ ./java -cp [PATH_TO_CLI_JAR]/elasticsearch-sql-cli-[VERSION].jar org.elasticse The jar name will be different for each Elasticsearch version (for example `elasticsearch-sql-cli-7.3.2.jar`), thus the generic `VERSION` specified in the example above. Furthermore, if not running the command from the folder where the SQL CLI jar resides, you’d have to provide the full path, as well. -## CLI commands [cli-commands] +## CLI commands [cli-commands] Apart from SQL queries, CLI can also execute some specific commands: @@ -83,7 +83,7 @@ fetch separator set to "---------------------" ``` `lenient = ` (default `false`) -: If `false`, Elasticsearch SQL returns an error for fields containing [array values](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/array.md). If `true`, Elasticsearch SQL returns the first value from the array with no guarantee of consistent results. +: If `false`, Elasticsearch SQL returns an error for fields containing [array values](elasticsearch://reference/elasticsearch/mapping-reference/array.md). If `true`, Elasticsearch SQL returns the first value from the array with no guarantee of consistent results. ```sql sql> lenient = true; diff --git a/explore-analyze/query-filter/languages/sql-data-types.md b/explore-analyze/query-filter/languages/sql-data-types.md index 5c5988408..1f3d5deff 100644 --- a/explore-analyze/query-filter/languages/sql-data-types.md +++ b/explore-analyze/query-filter/languages/sql-data-types.md @@ -12,31 +12,31 @@ mapped_pages: | --- | --- | --- | --- | | **{{es}} type** | **Elasticsearch SQL type** | **SQL type** | **SQL precision** | | Core types | -| [`null`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/null-value.md) | `null` | NULL | 0 | -| [`boolean`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/boolean.md) | `boolean` | BOOLEAN | 1 | -| [`byte`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/number.md) | `byte` | TINYINT | 3 | -| [`short`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/number.md) | `short` | SMALLINT | 5 | -| [`integer`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/number.md) | `integer` | INTEGER | 10 | -| [`long`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/number.md) | `long` | BIGINT | 19 | -| [`unsigned_long`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/number.md) | `[preview] unsigned_long` | BIGINT | 20 | -| [`double`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/number.md) | `double` | DOUBLE | 15 | -| [`float`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/number.md) | `float` | REAL | 7 | -| [`half_float`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/number.md) | `half_float` | FLOAT | 3 | -| [`scaled_float`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/number.md) | `scaled_float` | DOUBLE | 15 | -| [keyword type family](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/keyword.md) | `keyword` | VARCHAR | 32,766 | -| [`text`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/text.md) | `text` | VARCHAR | 2,147,483,647 | -| [`binary`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/binary.md) | `binary` | VARBINARY | 2,147,483,647 | -| [`date`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/date.md) | `datetime` | TIMESTAMP | 29 | -| [`ip`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/ip.md) | `ip` | VARCHAR | 39 | -| [`version`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/version.md) | `version` | VARCHAR | 32,766 | +| [`null`](elasticsearch://reference/elasticsearch/mapping-reference/null-value.md) | `null` | NULL | 0 | +| [`boolean`](elasticsearch://reference/elasticsearch/mapping-reference/boolean.md) | `boolean` | BOOLEAN | 1 | +| [`byte`](elasticsearch://reference/elasticsearch/mapping-reference/number.md) | `byte` | TINYINT | 3 | +| [`short`](elasticsearch://reference/elasticsearch/mapping-reference/number.md) | `short` | SMALLINT | 5 | +| [`integer`](elasticsearch://reference/elasticsearch/mapping-reference/number.md) | `integer` | INTEGER | 10 | +| [`long`](elasticsearch://reference/elasticsearch/mapping-reference/number.md) | `long` | BIGINT | 19 | +| [`unsigned_long`](elasticsearch://reference/elasticsearch/mapping-reference/number.md) | `[preview] unsigned_long` | BIGINT | 20 | +| [`double`](elasticsearch://reference/elasticsearch/mapping-reference/number.md) | `double` | DOUBLE | 15 | +| [`float`](elasticsearch://reference/elasticsearch/mapping-reference/number.md) | `float` | REAL | 7 | +| [`half_float`](elasticsearch://reference/elasticsearch/mapping-reference/number.md) | `half_float` | FLOAT | 3 | +| [`scaled_float`](elasticsearch://reference/elasticsearch/mapping-reference/number.md) | `scaled_float` | DOUBLE | 15 | +| [keyword type family](elasticsearch://reference/elasticsearch/mapping-reference/keyword.md) | `keyword` | VARCHAR | 32,766 | +| [`text`](elasticsearch://reference/elasticsearch/mapping-reference/text.md) | `text` | VARCHAR | 2,147,483,647 | +| [`binary`](elasticsearch://reference/elasticsearch/mapping-reference/binary.md) | `binary` | VARBINARY | 2,147,483,647 | +| [`date`](elasticsearch://reference/elasticsearch/mapping-reference/date.md) | `datetime` | TIMESTAMP | 29 | +| [`ip`](elasticsearch://reference/elasticsearch/mapping-reference/ip.md) | `ip` | VARCHAR | 39 | +| [`version`](elasticsearch://reference/elasticsearch/mapping-reference/version.md) | `version` | VARCHAR | 32,766 | | Complex types | -| [`object`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/object.md) | `object` | STRUCT | 0 | -| [`nested`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/nested.md) | `nested` | STRUCT | 0 | +| [`object`](elasticsearch://reference/elasticsearch/mapping-reference/object.md) | `object` | STRUCT | 0 | +| [`nested`](elasticsearch://reference/elasticsearch/mapping-reference/nested.md) | `nested` | STRUCT | 0 | | Unsupported types | | *types not mentioned above* | `unsupported` | OTHER | 0 | ::::{note} -Most of {{es}} [data types](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/field-data-types.md) are available in Elasticsearch SQL, as indicated above. As one can see, all of {{es}} [data types](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/field-data-types.md) are mapped to the data type with the same name in Elasticsearch SQL, with the exception of **date** data type which is mapped to **datetime** in Elasticsearch SQL. This is to avoid confusion with the ANSI SQL types **DATE** (date only) and **TIME** (time only), which are also supported by Elasticsearch SQL in queries (with the use of [`CAST`](sql-functions-type-conversion.md#sql-functions-type-conversion-cast)/[`CONVERT`](sql-functions-type-conversion.md#sql-functions-type-conversion-convert)), but don’t correspond to an actual mapping in {{es}} (see the [`table`](#es-sql-only-types) below). +Most of {{es}} [data types](elasticsearch://reference/elasticsearch/mapping-reference/field-data-types.md) are available in Elasticsearch SQL, as indicated above. As one can see, all of {{es}} [data types](elasticsearch://reference/elasticsearch/mapping-reference/field-data-types.md) are mapped to the data type with the same name in Elasticsearch SQL, with the exception of **date** data type which is mapped to **datetime** in Elasticsearch SQL. This is to avoid confusion with the ANSI SQL types **DATE** (date only) and **TIME** (time only), which are also supported by Elasticsearch SQL in queries (with the use of [`CAST`](sql-functions-type-conversion.md#sql-functions-type-conversion-cast)/[`CONVERT`](sql-functions-type-conversion.md#sql-functions-type-conversion-convert)), but don’t correspond to an actual mapping in {{es}} (see the [`table`](#es-sql-only-types) below). :::: @@ -72,9 +72,9 @@ The table below indicates these types: ## SQL and multi-fields [sql-multi-field] -A core concept in {{es}} is that of an `analyzed` field, that is a full-text value that is interpreted in order to be effectively indexed. These fields are of type [`text`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/text.md) and are not used for sorting or aggregations as their actual value depends on the [`analyzer`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/analyzer.md) used hence why {{es}} also offers the [`keyword`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/keyword.md) type for storing the *exact* value. +A core concept in {{es}} is that of an `analyzed` field, that is a full-text value that is interpreted in order to be effectively indexed. These fields are of type [`text`](elasticsearch://reference/elasticsearch/mapping-reference/text.md) and are not used for sorting or aggregations as their actual value depends on the [`analyzer`](elasticsearch://reference/elasticsearch/mapping-reference/analyzer.md) used hence why {{es}} also offers the [`keyword`](elasticsearch://reference/elasticsearch/mapping-reference/keyword.md) type for storing the *exact* value. -In most case, and the default actually, is to use both types for strings which {{es}} supports through [multi-fields](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/multi-fields.md), that is the ability to index the same string in multiple ways; for example index it both as `text` for search but also as `keyword` for sorting and aggregations. +In most case, and the default actually, is to use both types for strings which {{es}} supports through [multi-fields](elasticsearch://reference/elasticsearch/mapping-reference/multi-fields.md), that is the ability to index the same string in multiple ways; for example index it both as `text` for search but also as `keyword` for sorting and aggregations. As SQL requires exact values, when encountering a `text` field Elasticsearch SQL will search for an exact multi-field that it can use for comparisons, sorting and aggregations. To do that, it will search for the first `keyword` that it can find that is *not* normalized and use that as the original field *exact* value. diff --git a/explore-analyze/query-filter/languages/sql-functions-aggs.md b/explore-analyze/query-filter/languages/sql-functions-aggs.md index 7c425154d..3ccd61256 100644 --- a/explore-analyze/query-filter/languages/sql-functions-aggs.md +++ b/explore-analyze/query-filter/languages/sql-functions-aggs.md @@ -11,7 +11,7 @@ mapped_pages: Functions for computing a *single* result from a set of input values. Elasticsearch SQL supports aggregate functions only alongside [grouping](sql-syntax-select.md#sql-syntax-group-by) (implicit or explicit). -## General Purpose [sql-functions-aggs-general] +## General Purpose [sql-functions-aggs-general] ## `AVG` [sql-functions-aggs-avg] @@ -243,13 +243,13 @@ F |umant M |emzi ``` -::::{note} +::::{note} `FIRST` cannot be used in a HAVING clause. :::: -::::{note} -`FIRST` cannot be used with columns of type [`text`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/text.md) unless the field is also [saved as a keyword](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/text.md#before-enabling-fielddata). +::::{note} +`FIRST` cannot be used with columns of type [`text`](elasticsearch://reference/elasticsearch/mapping-reference/text.md) unless the field is also [saved as a keyword](elasticsearch://reference/elasticsearch/mapping-reference/text.md#before-enabling-fielddata). :::: @@ -364,13 +364,13 @@ F |ldiodio M |lari ``` -::::{note} +::::{note} `LAST` cannot be used in `HAVING` clause. :::: -::::{note} -`LAST` cannot be used with columns of type [`text`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/text.md) unless the field is also [`saved as a keyword`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/text.md#before-enabling-fielddata). +::::{note} +`LAST` cannot be used with columns of type [`text`](elasticsearch://reference/elasticsearch/mapping-reference/text.md) unless the field is also [`saved as a keyword`](elasticsearch://reference/elasticsearch/mapping-reference/text.md#before-enabling-fielddata). :::: @@ -406,8 +406,8 @@ SELECT MAX(ABS(salary / -12.0)) AS max FROM emp; 6249.916666666667 ``` -::::{note} -`MAX` on a field of type [`text`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/text.md) or [`keyword`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/keyword.md) is translated into [`LAST/LAST_VALUE`](#sql-functions-aggs-last) and therefore, it cannot be used in `HAVING` clause. +::::{note} +`MAX` on a field of type [`text`](elasticsearch://reference/elasticsearch/mapping-reference/text.md) or [`keyword`](elasticsearch://reference/elasticsearch/mapping-reference/keyword.md) is translated into [`LAST/LAST_VALUE`](#sql-functions-aggs-last) and therefore, it cannot be used in `HAVING` clause. :::: @@ -435,8 +435,8 @@ SELECT MIN(salary) AS min FROM emp; 25324 ``` -::::{note} -`MIN` on a field of type [`text`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/text.md) or [`keyword`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/keyword.md) is translated into [`FIRST/FIRST_VALUE`](#sql-functions-aggs-first) and therefore, it cannot be used in `HAVING` clause. +::::{note} +`MIN` on a field of type [`text`](elasticsearch://reference/elasticsearch/mapping-reference/text.md) or [`keyword`](elasticsearch://reference/elasticsearch/mapping-reference/keyword.md) is translated into [`FIRST/FIRST_VALUE`](#sql-functions-aggs-first) and therefore, it cannot be used in `HAVING` clause. :::: @@ -473,7 +473,7 @@ SELECT ROUND(SUM(salary / 12.0), 1) AS sum FROM emp; ``` -## Statistics [sql-functions-aggs-statistics] +## Statistics [sql-functions-aggs-statistics] ## `KURTOSIS` [sql-functions-aggs-kurtosis] @@ -501,7 +501,7 @@ SELECT MIN(salary) AS min, MAX(salary) AS max, KURTOSIS(salary) AS k FROM emp; 25324 |74999 |2.0444718929142986 ``` -::::{note} +::::{note} `KURTOSIS` cannot be used on top of scalar functions or operators but only directly on a field. So, for example, the following is not allowed and an error is returned: ```sql @@ -560,8 +560,8 @@ PERCENTILE( 1. a numeric field. If this field contains only `null` values, the function returns `null`. Otherwise, the function ignores `null` values in this field. 2. a numeric expression (must be a constant and not based on a field). If `null`, the function returns `null`. -3. optional string literal for the [percentile algorithm](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-percentile-aggregation.md#search-aggregations-metrics-percentile-aggregation-approximation). Possible values: `tdigest` or `hdr`. Defaults to `tdigest`. -4. optional numeric literal that configures the [percentile algorithm](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-percentile-aggregation.md#search-aggregations-metrics-percentile-aggregation-approximation). Configures `compression` for `tdigest` or `number_of_significant_value_digits` for `hdr`. The default is the same as that of the backing algorithm. +3. optional string literal for the [percentile algorithm](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-percentile-aggregation.md#search-aggregations-metrics-percentile-aggregation-approximation). Possible values: `tdigest` or `hdr`. Defaults to `tdigest`. +4. optional numeric literal that configures the [percentile algorithm](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-percentile-aggregation.md#search-aggregations-metrics-percentile-aggregation-approximation). Configures `compression` for `tdigest` or `number_of_significant_value_digits` for `hdr`. The default is the same as that of the backing algorithm. **Output**: `double` numeric value @@ -631,8 +631,8 @@ PERCENTILE_RANK( 1. a numeric field. If this field contains only `null` values, the function returns `null`. Otherwise, the function ignores `null` values in this field. 2. a numeric expression (must be a constant and not based on a field). If `null`, the function returns `null`. -3. optional string literal for the [percentile algorithm](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-percentile-aggregation.md#search-aggregations-metrics-percentile-aggregation-approximation). Possible values: `tdigest` or `hdr`. Defaults to `tdigest`. -4. optional numeric literal that configures the [percentile algorithm](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-percentile-aggregation.md#search-aggregations-metrics-percentile-aggregation-approximation). Configures `compression` for `tdigest` or `number_of_significant_value_digits` for `hdr`. The default is the same as that of the backing algorithm. +3. optional string literal for the [percentile algorithm](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-percentile-aggregation.md#search-aggregations-metrics-percentile-aggregation-approximation). Possible values: `tdigest` or `hdr`. Defaults to `tdigest`. +4. optional numeric literal that configures the [percentile algorithm](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-percentile-aggregation.md#search-aggregations-metrics-percentile-aggregation-approximation). Configures `compression` for `tdigest` or `number_of_significant_value_digits` for `hdr`. The default is the same as that of the backing algorithm. **Output**: `double` numeric value @@ -711,7 +711,7 @@ SELECT MIN(salary) AS min, MAX(salary) AS max, SKEWNESS(salary) AS s FROM emp; 25324 |74999 |0.2707722118423227 ``` -::::{note} +::::{note} `SKEWNESS` cannot be used on top of scalar functions but only directly on a field. So, for example, the following is not allowed and an error is returned: ```sql diff --git a/explore-analyze/query-filter/languages/sql-functions-datetime.md b/explore-analyze/query-filter/languages/sql-functions-datetime.md index f742c8973..b4aea8008 100644 --- a/explore-analyze/query-filter/languages/sql-functions-datetime.md +++ b/explore-analyze/query-filter/languages/sql-functions-datetime.md @@ -14,7 +14,7 @@ Elasticsearch SQL offers a wide range of facilities for performing date/time man A common requirement when dealing with date/time in general revolves around the notion of `interval`, a topic that is worth exploring in the context of {{es}} and Elasticsearch SQL. -{{es}} has comprehensive support for [date math](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/common-options.md#date-math) both inside [index names](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#api-date-math-index-names) and [queries](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-date-format.md). Inside Elasticsearch SQL the former is supported as is by passing the expression in the table name, while the latter is supported through the standard SQL `INTERVAL`. +{{es}} has comprehensive support for [date math](elasticsearch://reference/elasticsearch/rest-apis/common-options.md#date-math) both inside [index names](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#api-date-math-index-names) and [queries](elasticsearch://reference/elasticsearch/mapping-reference/mapping-date-format.md). Inside Elasticsearch SQL the former is supported as is by passing the expression in the table name, while the latter is supported through the standard SQL `INTERVAL`. The table below shows the mapping between {{es}} and Elasticsearch SQL: @@ -34,7 +34,7 @@ The table below shows the mapping between {{es}} and Elasticsearch SQL: `INTERVAL` allows either `YEAR` and `MONTH` to be mixed together *or* `DAY`, `HOUR`, `MINUTE` and `SECOND`. -::::{tip} +::::{tip} Elasticsearch SQL accepts also the plural for each time unit (e.g. both `YEAR` and `YEARS` are valid). :::: @@ -56,7 +56,7 @@ Example of the possible combinations below: ## Comparison [_comparison] -Date/time fields can be compared to [date math](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/common-options.md#date-math) expressions with the equality (`=`) and `IN` operators: +Date/time fields can be compared to [date math](elasticsearch://reference/elasticsearch/rest-apis/common-options.md#date-math) expressions with the equality (`=`) and `IN` operators: ```sql SELECT hire_date FROM emp WHERE hire_date = '1987-03-01||+4y/y'; @@ -153,7 +153,7 @@ CURDATE() **Description**: Returns the date (no time part) when the current query reached the server. It can be used both as a keyword: `CURRENT_DATE` or as a function with no arguments: `CURRENT_DATE()`. -::::{note} +::::{note} Unlike CURRENT_DATE, `CURDATE()` can only be used as a function with no arguments and not as a keyword. :::: @@ -264,7 +264,7 @@ Anoosh Arumugam ``` -::::{important} +::::{important} Currently, using a *precision* greater than 6 doesn’t make any difference to the output of the function as the maximum number of second fractional digits returned is 6. :::: @@ -326,7 +326,7 @@ Anoosh Arumugam ``` -::::{important} +::::{important} Currently, using a *precision* greater than 6 doesn’t make any difference to the output of the function as the maximum number of second fractional digits returned is 6. :::: @@ -352,7 +352,7 @@ DATE_ADD( **Description**: Add the given number of date/time units to a date/datetime. If the number of units is negative then it’s subtracted from the date/datetime. -::::{warning} +::::{warning} If the second argument is a long there is possibility of truncation since an integer value will be extracted and used from that long. :::: @@ -484,7 +484,7 @@ SELECT DATE_DIFF('qq', '2019-09-04'::date, '2025-04-25'::date) AS "diffInQuarter 23 ``` -::::{note} +::::{note} For `hour` and `minute`, `DATEDIFF` doesn’t do any rounding, but instead first truncates the more detailed time fields on the 2 dates to zero and then calculates the subtraction. :::: @@ -532,7 +532,7 @@ DATE_FORMAT( **Description**: Returns the date/datetime/time as a string using the format specified in the 2nd argument. The formatting pattern is one of the specifiers used in the [MySQL DATE_FORMAT() function](https://dev.mysql.com/doc/refman/8.0/en/date-and-time-functions.html#function_date-format). -::::{note} +::::{note} If the 1st argument is of type `time`, then pattern specified by the 2nd argument cannot contain date related units (e.g. *dd*, *MM*, *yyyy*, etc.). If it contains such units an error is returned. Ranges for month and day specifiers (%c, %D, %d, %e, %m) start at one, unlike MySQL, where they start at zero, due to the fact that MySQL permits the storing of incomplete dates such as *2014-00-00*. Elasticsearch in this case returns an error. :::: @@ -580,7 +580,7 @@ DATE_PARSE( **Description**: Returns a date by parsing the 1st argument using the format specified in the 2nd argument. The parsing format pattern used is the one from [`java.time.format.DateTimeFormatter`](https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/time/format/DateTimeFormatter.html). -::::{note} +::::{note} If the parsing pattern does not contain all valid date units (e.g. *HH:mm:ss*, *dd-MM HH:mm:ss*, etc.) an error is returned as the function needs to return a value of `date` type which will contain date part. :::: @@ -593,7 +593,7 @@ SELECT DATE_PARSE('07/04/2020', 'dd/MM/yyyy') AS "date"; 2020-04-07 ``` -::::{note} +::::{note} The resulting `date` will have the time zone specified by the user through the [`time_zone`](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-sql-query)/[`timezone`](sql-jdbc.md#jdbc-cfg-timezone) REST/driver parameters with no conversion applied. ```sql @@ -629,7 +629,7 @@ DATETIME_FORMAT( **Description**: Returns the date/datetime/time as a string using the format specified in the 2nd argument. The formatting pattern used is the one from [`java.time.format.DateTimeFormatter`](https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/time/format/DateTimeFormatter.html). -::::{note} +::::{note} If the 1st argument is of type `time`, then pattern specified by the 2nd argument cannot contain date related units (e.g. *dd*, *MM*, *yyyy*, etc.). If it contains such units an error is returned. :::: @@ -677,7 +677,7 @@ DATETIME_PARSE( **Description**: Returns a datetime by parsing the 1st argument using the format specified in the 2nd argument. The parsing format pattern used is the one from [`java.time.format.DateTimeFormatter`](https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/time/format/DateTimeFormatter.html). -::::{note} +::::{note} If the parsing pattern contains only date or only time units (e.g. *dd/MM/yyyy*, *HH:mm:ss*, etc.) an error is returned as the function needs to return a value of `datetime` type which must contain both. :::: @@ -698,7 +698,7 @@ SELECT DATETIME_PARSE('10:20:30 07/04/2020 Europe/Berlin', 'HH:mm:ss dd/MM/yyyy 2020-04-07T08:20:30.000Z ``` -::::{note} +::::{note} If timezone is not specified in the datetime string expression and the parsing pattern, the resulting `datetime` will have the time zone specified by the user through the [`time_zone`](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-sql-query)/[`timezone`](sql-jdbc.md#jdbc-cfg-timezone) REST/driver parameters with no conversion applied. ```sql @@ -734,7 +734,7 @@ TIME_PARSE( **Description**: Returns a time by parsing the 1st argument using the format specified in the 2nd argument. The parsing format pattern used is the one from [`java.time.format.DateTimeFormatter`](https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/time/format/DateTimeFormatter.html). -::::{note} +::::{note} If the parsing pattern contains only date units (e.g. *dd/MM/yyyy*) an error is returned as the function needs to return a value of `time` type which will contain only time. :::: @@ -755,7 +755,7 @@ SELECT TIME_PARSE('10:20:30-01:00', 'HH:mm:ssXXX') AS "time"; 11:20:30.000Z ``` -::::{note} +::::{note} If timezone is not specified in the time string expression and the parsing pattern, the resulting `time` will have the offset of the time zone specified by the user through the [`time_zone`](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-sql-query)/[`timezone`](sql-jdbc.md#jdbc-cfg-timezone) REST/driver parameters at the Unix epoch date (`1970-01-01`) with no conversion applied. ```sql @@ -841,7 +841,7 @@ SELECT DATE_PART('month', CAST('2019-09-24' AS DATE)) AS month; 9 ``` -::::{note} +::::{note} For `week` and `weekday` the unit is extracted using the non-ISO calculation, which means that a given week is considered to start from Sunday, not Monday. :::: @@ -854,7 +854,7 @@ SELECT DATE_PART('week', '2019-09-22T11:22:33.123Z'::datetime) AS week; 39 ``` -::::{note} +::::{note} The `tzoffset` returns the total number of minutes (signed) that represent the time zone’s offset. :::: @@ -995,7 +995,7 @@ FORMAT( **Description**: Returns the date/datetime/time as a string using the [format](https://docs.microsoft.com/en-us/sql/t-sql/functions/format-transact-sql#arguments) specified in the 2nd argument. The formatting pattern used is the one from [Microsoft SQL Server Format Specification](https://docs.microsoft.com/en-us/dotnet/standard/base-types/custom-date-and-time-format-strings). -::::{note} +::::{note} If the 1st argument is of type `time`, then pattern specified by the 2nd argument cannot contain date related units (e.g. *dd*, *MM*, *yyyy*, etc.). If it contains such units an error is returned.
Format specifier `F` will be working similar to format specifier `f`. It will return the fractional part of seconds, and the number of digits will be same as of the number of `Fs` provided as input (up to 9 digits). Result will contain `0` appended in the end to match with number of `F` provided. e.g.: for a time part `10:20:30.1234` and pattern `HH:mm:ss.FFFFFF`, the output string of the function would be: `10:20:30.123400`.
Format specifier `y` will return year-of-era instead of one/two low-order digits. eg.: For year `2009`, `y` will be returning `2009` instead of `9`. For year `43`, `y` format specifier will return `43`. - Special characters like `"` , `\` and `%` will be returned as it is without any change. eg.: formatting date `17-sep-2020` with `%M` will return `%9` :::: @@ -1043,7 +1043,7 @@ TO_CHAR( **Description**: Returns the date/datetime/time as a string using the format specified in the 2nd argument. The formatting pattern conforms to [PostgreSQL Template Patterns for Date/Time Formatting](https://www.postgresql.org/docs/13/functions-formatting.html). -::::{note} +::::{note} If the 1st argument is of type `time`, then the pattern specified by the 2nd argument cannot contain date related units (e.g. *dd*, *MM*, *YYYY*, etc.). If it contains such units an error is returned.
The result of the patterns `TZ` and `tz` (time zone abbreviations) in some cases differ from the results returned by the `TO_CHAR` in PostgreSQL. The reason is that the time zone abbreviations specified by the JDK are different from the ones specified by PostgreSQL. This function might show an actual time zone abbreviation instead of the generic `LMT` or empty string or offset returned by the PostgreSQL implementation. The summer/daylight markers might also differ between the two implementations (e.g. will show `HT` instead of `HST` for Hawaii).
The `FX`, `TM`, `SP` pattern modifiers are not supported and will show up as `FX`, `TM`, `SP` literals in the output. :::: diff --git a/explore-analyze/query-filter/languages/sql-functions-geo.md b/explore-analyze/query-filter/languages/sql-functions-geo.md index 1df38bd39..4d3dda725 100644 --- a/explore-analyze/query-filter/languages/sql-functions-geo.md +++ b/explore-analyze/query-filter/languages/sql-functions-geo.md @@ -8,7 +8,7 @@ mapped_pages: # Geo Functions [sql-functions-geo] -::::{warning} +::::{warning} This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features. :::: @@ -17,7 +17,7 @@ The geo functions work with geometries stored in `geo_point`, `geo_shape` and `s ## Limitations [_limitations_4] -[`geo_point`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/geo-point.md), [`geo_shape`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/geo-shape.md) and [`shape`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/shape.md) and types are represented in SQL as geometry and can be used interchangeably with the following exceptions: +[`geo_point`](elasticsearch://reference/elasticsearch/mapping-reference/geo-point.md), [`geo_shape`](elasticsearch://reference/elasticsearch/mapping-reference/geo-shape.md) and [`shape`](elasticsearch://reference/elasticsearch/mapping-reference/shape.md) and types are represented in SQL as geometry and can be used interchangeably with the following exceptions: * `geo_shape` and `shape` fields don’t have doc values, therefore these fields cannot be used for filtering, grouping or sorting. * `geo_points` fields are indexed and have doc values by default, however only latitude and longitude are stored and indexed with some loss of precision from the original values (4.190951585769653E-8 for the latitude and 8.381903171539307E-8 for longitude). The altitude component is accepted but not stored in doc values nor indexed. Therefore calling `ST_Z` function in the filtering, grouping or sorting will return `null`. diff --git a/explore-analyze/query-filter/languages/sql-functions-grouping.md b/explore-analyze/query-filter/languages/sql-functions-grouping.md index 7f5e0d137..3a8723a6d 100644 --- a/explore-analyze/query-filter/languages/sql-functions-grouping.md +++ b/explore-analyze/query-filter/languages/sql-functions-grouping.md @@ -39,7 +39,7 @@ bucket_key = Math.floor(value / interval) * interval ``` ::::{note} -The histogram in SQL does **NOT** return empty buckets for missing intervals as the traditional [histogram](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-histogram-aggregation.md) and [date histogram](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-datehistogram-aggregation.md). Such behavior does not fit conceptually in SQL which treats all missing values as `null`; as such the histogram places all missing values in the `null` group. +The histogram in SQL does **NOT** return empty buckets for missing intervals as the traditional [histogram](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-histogram-aggregation.md) and [date histogram](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-datehistogram-aggregation.md). Such behavior does not fit conceptually in SQL which treats all missing values as `null`; as such the histogram places all missing values in the `null` group. :::: @@ -137,7 +137,7 @@ When the histogram in SQL is applied on **DATE** type instead of **DATETIME**, t ::::{important} -All intervals specified for a date/time HISTOGRAM will use a [fixed interval](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-datehistogram-aggregation.md) in their `date_histogram` aggregation definition, with the notable exceptions of `INTERVAL '1' YEAR`, `INTERVAL '1' MONTH` and `INTERVAL '1' DAY` where a calendar interval is used. The choice for a calendar interval was made for having a more intuitive result for YEAR, MONTH and DAY groupings. In the case of YEAR, for example, the calendar intervals consider a one year bucket as the one starting on January 1st that specific year, whereas a fixed interval one-year-bucket considers one year as a number of milliseconds (for example, `31536000000ms` corresponding to 365 days, 24 hours per day, 60 minutes per hour etc.). With fixed intervals, the day of February 5th, 2019 for example, belongs to a bucket that starts on December 20th, 2018 and {{es}} (and implicitly Elasticsearch SQL) would have returned the year 2018 for a date that’s actually in 2019. With calendar interval this behavior is more intuitive, having the day of February 5th, 2019 actually belonging to the 2019 year bucket. +All intervals specified for a date/time HISTOGRAM will use a [fixed interval](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-datehistogram-aggregation.md) in their `date_histogram` aggregation definition, with the notable exceptions of `INTERVAL '1' YEAR`, `INTERVAL '1' MONTH` and `INTERVAL '1' DAY` where a calendar interval is used. The choice for a calendar interval was made for having a more intuitive result for YEAR, MONTH and DAY groupings. In the case of YEAR, for example, the calendar intervals consider a one year bucket as the one starting on January 1st that specific year, whereas a fixed interval one-year-bucket considers one year as a number of milliseconds (for example, `31536000000ms` corresponding to 365 days, 24 hours per day, 60 minutes per hour etc.). With fixed intervals, the day of February 5th, 2019 for example, belongs to a bucket that starts on December 20th, 2018 and {{es}} (and implicitly Elasticsearch SQL) would have returned the year 2018 for a date that’s actually in 2019. With calendar interval this behavior is more intuitive, having the day of February 5th, 2019 actually belonging to the 2019 year bucket. :::: diff --git a/explore-analyze/query-filter/languages/sql-functions-search.md b/explore-analyze/query-filter/languages/sql-functions-search.md index cdfb738d8..87252476b 100644 --- a/explore-analyze/query-filter/languages/sql-functions-search.md +++ b/explore-analyze/query-filter/languages/sql-functions-search.md @@ -10,7 +10,7 @@ mapped_pages: Search functions should be used when performing full-text search, namely when the `MATCH` or `QUERY` predicates are being used. Outside a, so-called, search context, these functions will return default values such as `0` or `NULL`. -Elasticsearch SQL optimizes all queries executed against {{es}} depending on the scoring needs. Using [`track_scores`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/sort-search-results.md#_track_scores) on the search request or [`_doc` sorting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/sort-search-results.md) that disables scores calculation, Elasticsearch SQL instructs {{es}} not to compute scores when these are not needed. For example, every time a `SCORE()` function is encountered in the SQL query, the scores are computed. +Elasticsearch SQL optimizes all queries executed against {{es}} depending on the scoring needs. Using [`track_scores`](elasticsearch://reference/elasticsearch/rest-apis/sort-search-results.md#_track_scores) on the search request or [`_doc` sorting](elasticsearch://reference/elasticsearch/rest-apis/sort-search-results.md) that disables scores calculation, Elasticsearch SQL instructs {{es}} not to compute scores when these are not needed. For example, every time a `SCORE()` function is encountered in the SQL query, the scores are computed. ## `MATCH` [sql-functions-search-match] @@ -28,7 +28,7 @@ MATCH( 3. additional parameters; optional -**Description**: A full-text search option, in the form of a predicate, available in Elasticsearch SQL that gives the user control over powerful [match](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-match-query.md) and [multi_match](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-multi-match-query.md) {{es}} queries. +**Description**: A full-text search option, in the form of a predicate, available in Elasticsearch SQL that gives the user control over powerful [match](elasticsearch://reference/query-languages/query-dsl-match-query.md) and [multi_match](elasticsearch://reference/query-languages/query-dsl-multi-match-query.md) {{es}} queries. The first parameter is the field or fields to match against. In case it receives one value only, Elasticsearch SQL will use a `match` query to perform the search: @@ -56,8 +56,8 @@ Frank Herbert |Children of Dune |8.043278 Frank Herbert |God Emperor of Dune|7.0029488 ``` -::::{note} -The `multi_match` query in {{es}} has the option of [per-field boosting](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-multi-match-query.md) that gives preferential weight (in terms of scoring) to fields being searched in, using the `^` character. In the example above, the `name` field has a greater weight in the final score than the `author` field when searching for `frank dune` text in both of them. +::::{note} +The `multi_match` query in {{es}} has the option of [per-field boosting](elasticsearch://reference/query-languages/query-dsl-multi-match-query.md) that gives preferential weight (in terms of scoring) to fields being searched in, using the `^` character. In the example above, the `name` field has a greater weight in the final score than the `author` field when searching for `frank dune` text in both of them. :::: @@ -73,12 +73,12 @@ Douglas Adams |The Hitchhiker's Guide to the Galaxy|3.1756816 Peter F. Hamilton|Pandora's Star |3.0997515 ``` -::::{note} +::::{note} The allowed optional parameters for a single-field `MATCH()` variant (for the `match` {{es}} query) are: `analyzer`, `auto_generate_synonyms_phrase_query`, `lenient`, `fuzziness`, `fuzzy_transpositions`, `fuzzy_rewrite`, `minimum_should_match`, `operator`, `max_expansions`, `prefix_length`. :::: -::::{note} +::::{note} The allowed optional parameters for a multi-field `MATCH()` variant (for the `multi_match` {{es}} query) are: `analyzer`, `auto_generate_synonyms_phrase_query`, `lenient`, `fuzziness`, `fuzzy_transpositions`, `fuzzy_rewrite`, `minimum_should_match`, `operator`, `max_expansions`, `prefix_length`, `slop`, `tie_breaker`, `type`. :::: @@ -98,7 +98,7 @@ QUERY( 2. additional parameters; optional -**Description**: Just like `MATCH`, `QUERY` is a full-text search predicate that gives the user control over the [query_string](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-query-string-query.md) query in {{es}}. +**Description**: Just like `MATCH`, `QUERY` is a full-text search predicate that gives the user control over the [query_string](elasticsearch://reference/query-languages/query-dsl-query-string-query.md) query in {{es}}. The first parameter is basically the input that will be passed as is to the `query_string` query, which means that anything that `query_string` accepts in its `query` field can be used here as well: @@ -140,7 +140,7 @@ SELECT author, name, SCORE() FROM library WHERE QUERY('dune god', 'default_opera Frank Herbert |God Emperor of Dune|3.6984892 ``` -::::{note} +::::{note} The allowed optional parameters for `QUERY()` are: `allow_leading_wildcard`, `analyze_wildcard`, `analyzer`, `auto_generate_synonyms_phrase_query`, `default_field`, `default_operator`, `enable_position_increments`, `escape`, `fuzziness`, `fuzzy_max_expansions`, `fuzzy_prefix_length`, `fuzzy_rewrite`, `fuzzy_transpositions`, `lenient`, `max_determinized_states`, `minimum_should_match`, `phrase_slop`, `rewrite`, `quote_analyzer`, `quote_field_suffix`, `tie_breaker`, `time_zone`, `type`. :::: @@ -158,8 +158,8 @@ SCORE() **Description**: Returns the [relevance](https://www.elastic.co/guide/en/elasticsearch/guide/2.x/relevance-intro.html) of a given input to the executed query. The higher score, the more relevant the data. -::::{note} -When doing multiple text queries in the `WHERE` clause then, their scores will be combined using the same rules as {{es}}'s [bool query](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-bool-query.md). +::::{note} +When doing multiple text queries in the `WHERE` clause then, their scores will be combined using the same rules as {{es}}'s [bool query](elasticsearch://reference/query-languages/query-dsl-bool-query.md). :::: diff --git a/explore-analyze/query-filter/languages/sql-index-patterns.md b/explore-analyze/query-filter/languages/sql-index-patterns.md index 7e946f50e..772e15d5d 100644 --- a/explore-analyze/query-filter/languages/sql-index-patterns.md +++ b/explore-analyze/query-filter/languages/sql-index-patterns.md @@ -11,9 +11,9 @@ mapped_pages: Elasticsearch SQL supports two types of patterns for matching multiple indices or tables: -## {{es}} multi-target syntax [sql-index-patterns-multi] +## {{es}} multi-target syntax [sql-index-patterns-multi] -The {{es}} notation for enumerating, including or excluding [multi-target syntax](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#api-multi-index) is supported *as long* as it is quoted or escaped as a table identifier. +The {{es}} notation for enumerating, including or excluding [multi-target syntax](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#api-multi-index) is supported *as long* as it is quoted or escaped as a table identifier. For example: @@ -40,7 +40,7 @@ SELECT emp_no FROM "e*p" LIMIT 1; 10001 ``` -::::{note} +::::{note} There is the restriction that all resolved concrete tables have the exact same mapping. :::: @@ -58,7 +58,7 @@ SELECT emp_no FROM "my*cluster:*emp" LIMIT 1; ``` -## SQL `LIKE` notation [sql-index-patterns-like] +## SQL `LIKE` notation [sql-index-patterns-like] The common `LIKE` statement (including escaping if needed) to match a wildcard pattern, based on one `_` or multiple `%` characters. @@ -101,7 +101,7 @@ In a nutshell, the differences between the two type of patterns are: Which one to use, is up to you however try to stick to the same one across your queries for consistency. -::::{note} +::::{note} As the query type of quoting between the two patterns is fairly similar (`"` vs `'`), Elasticsearch SQL *always* requires the keyword `LIKE` for SQL `LIKE` pattern. :::: diff --git a/explore-analyze/query-filter/languages/sql-lexical-structure.md b/explore-analyze/query-filter/languages/sql-lexical-structure.md index 4616ba2eb..12da08300 100644 --- a/explore-analyze/query-filter/languages/sql-lexical-structure.md +++ b/explore-analyze/query-filter/languages/sql-lexical-structure.md @@ -43,7 +43,7 @@ Identifiers can be of two types: *quoted* and *unquoted*: SELECT ip_address FROM "hosts-*" ``` -This query has two identifiers, `ip_address` and `hosts-*` (an [index pattern](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#api-multi-index)). As `ip_address` does not clash with any key words it can be used verbatim, `hosts-*` on the other hand cannot as it clashes with `-` (minus operation) and `*` hence the double quotes. +This query has two identifiers, `ip_address` and `hosts-*` (an [index pattern](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#api-multi-index)). As `ip_address` does not clash with any key words it can be used verbatim, `hosts-*` on the other hand cannot as it clashes with `-` (minus operation) and `*` hence the double quotes. Another example: @@ -51,7 +51,7 @@ Another example: SELECT "from" FROM "" ``` -The first identifier from needs to quoted as otherwise it clashes with the `FROM` key word (which is case insensitive as thus can be written as `from`) while the second identifier using {{es}} [Date math support in index and index alias names](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#api-date-math-index-names) would have otherwise confuse the parser. +The first identifier from needs to quoted as otherwise it clashes with the `FROM` key word (which is case insensitive as thus can be written as `from`) while the second identifier using {{es}} [Date math support in index and index alias names](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#api-date-math-index-names) would have otherwise confuse the parser. Hence why in general, **especially** when dealing with user input it is **highly** recommended to use quotes for identifiers. It adds minimal increase to your queries and in return offers clarity and disambiguation. @@ -61,17 +61,17 @@ Hence why in general, **especially** when dealing with user input it is **highly Elasticsearch SQL supports two kind of *implicitly-typed* literals: strings and numbers. -#### String Literals [sql-syntax-string-literals] +#### String Literals [sql-syntax-string-literals] A string literal is an arbitrary number of characters bounded by single quotes `'`: `'Giant Robot'`. To include a single quote in the string, escape it using another single quote: `'Captain EO''s Voyage'`. -::::{note} +::::{note} An escaped single quote is **not** a double quote (`"`), but a single quote `'` *repeated* (`''`). :::: -#### Numeric Literals [_numeric_literals] +#### Numeric Literals [_numeric_literals] Numeric literals are accepted both in decimal and scientific notation with exponent marker (`e` or `E`), starting either with a digit or decimal point `.`: @@ -86,7 +86,7 @@ Numeric literals are accepted both in decimal and scientific notation with expon Numeric literals that contain a decimal point are always interpreted as being of type `double`. Those without are considered `integer` if they fit otherwise their type is `long` (or `BIGINT` in ANSI SQL types). -#### Generic Literals [sql-syntax-generic-literals] +#### Generic Literals [sql-syntax-generic-literals] When dealing with arbitrary type literal, one creates the object by casting, typically, the string representation to the desired type. This can be achieved through the dedicated [cast operator](sql-operators-cast.md) and [functions](sql-functions-type-conversion.md): @@ -116,7 +116,7 @@ SELECT "first_name" <1> 2. Single quotes `'` used for a string literal -::::{note} +::::{note} To escape single or double quotes, one needs to use that specific quote one more time. For example, the literal `John's` can be escaped like `SELECT 'John''s' AS name`. The same goes for double quotes escaping - `SELECT 123 AS "test""number"` will display as a result a column with the name `test"number`. :::: diff --git a/explore-analyze/query-filter/languages/sql-like-rlike-operators.md b/explore-analyze/query-filter/languages/sql-like-rlike-operators.md index bf5a96bdc..7b5d86e84 100644 --- a/explore-analyze/query-filter/languages/sql-like-rlike-operators.md +++ b/explore-analyze/query-filter/languages/sql-like-rlike-operators.md @@ -10,8 +10,8 @@ mapped_pages: `LIKE` and `RLIKE` operators are commonly used to filter data based on string patterns. They usually act on a field placed on the left-hand side of the operator, but can also act on a constant (literal) expression. The right-hand side of the operator represents the pattern. Both can be used in the `WHERE` clause of the `SELECT` statement, but `LIKE` can also be used in other places, such as defining an [index pattern](sql-index-patterns.md) or across various [SHOW commands](sql-commands.md). This section covers only the `SELECT ... WHERE ...` usage. -::::{note} -One significant difference between `LIKE`/`RLIKE` and the [full-text search predicates](sql-functions-search.md) is that the former act on [exact fields](sql-data-types.md#sql-multi-field) while the latter also work on [analyzed](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/text.md) fields. If the field used with `LIKE`/`RLIKE` doesn’t have an exact not-normalized sub-field (of [keyword](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/keyword.md) type) Elasticsearch SQL will not be able to run the query. If the field is either exact or has an exact sub-field, it will use it as is, or it will automatically use the exact sub-field even if it wasn’t explicitly specified in the statement. +::::{note} +One significant difference between `LIKE`/`RLIKE` and the [full-text search predicates](sql-functions-search.md) is that the former act on [exact fields](sql-data-types.md#sql-multi-field) while the latter also work on [analyzed](elasticsearch://reference/elasticsearch/mapping-reference/text.md) fields. If the field used with `LIKE`/`RLIKE` doesn’t have an exact not-normalized sub-field (of [keyword](elasticsearch://reference/elasticsearch/mapping-reference/keyword.md) type) Elasticsearch SQL will not be able to run the query. If the field is either exact or has an exact sub-field, it will use it as is, or it will automatically use the exact sub-field even if it wasn’t explicitly specified in the statement. :::: @@ -33,7 +33,7 @@ LIKE constant_exp <2> The percent sign represents zero, one or multiple characters. The underscore represents a single number or character. These symbols can be used in combinations. -::::{note} +::::{note} No other characters have special meaning or act as wildcard. Characters often used as wildcards in other languages (`*` or `?`) are treated as normal characters. :::: @@ -54,7 +54,7 @@ SELECT name, author FROM library WHERE name LIKE 'Dune/%' ESCAPE '/'; ``` In the example above `/` is defined as an escape character which needs to be placed before the `%` or `_` characters if one needs to match those characters in the pattern specifically. By default, there is no escape character defined. -::::{important} +::::{important} Even though `LIKE` is a valid option when searching or filtering in Elasticsearch SQL, full-text search predicates `MATCH` and `QUERY` are [faster and much more powerful and are the preferred alternative](#sql-like-prefer-full-text). :::: @@ -73,7 +73,7 @@ RLIKE constant_exp <2> **Description**: This operator is similar to `LIKE`, but the user is not limited to search for a string based on a fixed pattern with the percent sign (`%`) and underscore (`_`); the pattern in this case is a regular expression which allows the construction of more flexible patterns. -For supported syntax, see [*Regular expression syntax*](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/regexp-syntax.md). +For supported syntax, see [*Regular expression syntax*](elasticsearch://reference/query-languages/regexp-syntax.md). ```sql SELECT author, name FROM library WHERE name RLIKE 'Child.* Dune'; @@ -83,7 +83,7 @@ SELECT author, name FROM library WHERE name RLIKE 'Child.* Dune'; Frank Herbert |Children of Dune ``` -::::{important} +::::{important} Even though `RLIKE` is a valid option when searching or filtering in Elasticsearch SQL, full-text search predicates `MATCH` and `QUERY` are [faster and much more powerful and are the preferred alternative](#sql-like-prefer-full-text). :::: diff --git a/explore-analyze/query-filter/languages/sql-limitations.md b/explore-analyze/query-filter/languages/sql-limitations.md index e86fae499..86352db5e 100644 --- a/explore-analyze/query-filter/languages/sql-limitations.md +++ b/explore-analyze/query-filter/languages/sql-limitations.md @@ -134,7 +134,7 @@ But, if the sub-select would include a `GROUP BY` or `HAVING` or the enclosing ` ## Using [`FIRST`](sql-functions-aggs.md#sql-functions-aggs-first)/[`LAST`](sql-functions-aggs.md#sql-functions-aggs-last) aggregation functions in `HAVING` clause [first-last-agg-functions-having-clause] -Using `FIRST` and `LAST` in the `HAVING` clause is not supported. The same applies to [`MIN`](sql-functions-aggs.md#sql-functions-aggs-min) and [`MAX`](sql-functions-aggs.md#sql-functions-aggs-max) when their target column is of type [`keyword`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/keyword.md) or [`unsigned_long`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/number.md) as they are internally translated to `FIRST` and `LAST`. +Using `FIRST` and `LAST` in the `HAVING` clause is not supported. The same applies to [`MIN`](sql-functions-aggs.md#sql-functions-aggs-min) and [`MAX`](sql-functions-aggs.md#sql-functions-aggs-max) when their target column is of type [`keyword`](elasticsearch://reference/elasticsearch/mapping-reference/keyword.md) or [`unsigned_long`](elasticsearch://reference/elasticsearch/mapping-reference/number.md) as they are internally translated to `FIRST` and `LAST`. ## Using TIME data type in GROUP BY or [`HISTOGRAM`](sql-functions-grouping.md#sql-functions-grouping-histogram) [group-by-time] @@ -167,7 +167,7 @@ By default,`geo_points` fields are indexed and have doc values. However only lat ## Retrieving using the `fields` search parameter [using-fields-api] -Elasticsearch SQL retrieves column values using the [search API’s `fields` parameter](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/retrieve-selected-fields.md#search-fields-param). Any limitations on the `fields` parameter also apply to Elasticsearch SQL queries. For example, if `_source` is disabled for any of the returned fields or at index level, the values cannot be retrieved. +Elasticsearch SQL retrieves column values using the [search API’s `fields` parameter](elasticsearch://reference/elasticsearch/rest-apis/retrieve-selected-fields.md#search-fields-param). Any limitations on the `fields` parameter also apply to Elasticsearch SQL queries. For example, if `_source` is disabled for any of the returned fields or at index level, the values cannot be retrieved. ## Aggregations in the [`PIVOT`](sql-syntax-select.md#sql-syntax-pivot) clause [aggs-in-pivot] diff --git a/explore-analyze/query-filter/languages/sql-pagination.md b/explore-analyze/query-filter/languages/sql-pagination.md index 3362e4db5..fc7567c95 100644 --- a/explore-analyze/query-filter/languages/sql-pagination.md +++ b/explore-analyze/query-filter/languages/sql-pagination.md @@ -34,7 +34,7 @@ Which looks like: Note that the `columns` object is only part of the first page. -You’ve reached the last page when there is no `cursor` returned in the results. Like Elasticsearch’s [scroll](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/paginate-search-results.md#scroll-search-results), SQL may keep state in Elasticsearch to support the cursor. Unlike scroll, receiving the last page is enough to guarantee that the Elasticsearch state is cleared. +You’ve reached the last page when there is no `cursor` returned in the results. Like Elasticsearch’s [scroll](elasticsearch://reference/elasticsearch/rest-apis/paginate-search-results.md#scroll-search-results), SQL may keep state in Elasticsearch to support the cursor. Unlike scroll, receiving the last page is enough to guarantee that the Elasticsearch state is cleared. To clear the state earlier, use the [clear cursor API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-sql-clear-cursor): diff --git a/explore-analyze/query-filter/languages/sql-rest-filtering.md b/explore-analyze/query-filter/languages/sql-rest-filtering.md index 898a83665..3136e2587 100644 --- a/explore-analyze/query-filter/languages/sql-rest-filtering.md +++ b/explore-analyze/query-filter/languages/sql-rest-filtering.md @@ -34,8 +34,8 @@ Which returns: Douglas Adams |The Hitchhiker's Guide to the Galaxy|180 |1979-10-12T00:00:00.000Z ``` -::::{tip} -A useful and less obvious usage for standard Query DSL filtering is to search documents by a specific [routing key](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/search-shard-routing.md#search-routing). Because Elasticsearch SQL does not support a `routing` parameter, one can specify a [`terms` filter for the `_routing` field](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-routing-field.md) instead: +::::{tip} +A useful and less obvious usage for standard Query DSL filtering is to search documents by a specific [routing key](elasticsearch://reference/elasticsearch/rest-apis/search-shard-routing.md#search-routing). Because Elasticsearch SQL does not support a `routing` parameter, one can specify a [`terms` filter for the `_routing` field](elasticsearch://reference/elasticsearch/mapping-reference/mapping-routing-field.md) instead: ```console POST /_sql?format=txt diff --git a/explore-analyze/query-filter/languages/sql-syntax-select.md b/explore-analyze/query-filter/languages/sql-syntax-select.md index 7b6ba4b1e..39212d726 100644 --- a/explore-analyze/query-filter/languages/sql-syntax-select.md +++ b/explore-analyze/query-filter/languages/sql-syntax-select.md @@ -133,7 +133,7 @@ SELECT * FROM "emp" LIMIT 1; 1953-09-02T00:00:00Z|10001 |Georgi |M |1986-06-26T00:00:00.000Z|2 |Facello |Georgi Facello |57305 ``` -The name can be a [pattern](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#api-multi-index) pointing to multiple indices (likely requiring quoting as mentioned above) with the restriction that **all** resolved concrete tables have **exact mapping**. +The name can be a [pattern](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#api-multi-index) pointing to multiple indices (likely requiring quoting as mentioned above) with the restriction that **all** resolved concrete tables have **exact mapping**. ```sql SELECT emp_no FROM "e*p" LIMIT 1; @@ -507,7 +507,7 @@ Ordering by aggregation is possible for up to **10000** entries for memory consu When doing full-text queries in the `WHERE` clause, results can be returned based on their [score](https://www.elastic.co/guide/en/elasticsearch/guide/2.x/relevance-intro.html) or *relevance* to the given query. ::::{note} -When doing multiple text queries in the `WHERE` clause then, their scores will be combined using the same rules as {{es}}'s [bool query](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-bool-query.md). +When doing multiple text queries in the `WHERE` clause then, their scores will be combined using the same rules as {{es}}'s [bool query](elasticsearch://reference/query-languages/query-dsl-bool-query.md). :::: diff --git a/explore-analyze/query-filter/languages/sql-syntax-show-tables.md b/explore-analyze/query-filter/languages/sql-syntax-show-tables.md index 65cd00dd9..65173078b 100644 --- a/explore-analyze/query-filter/languages/sql-syntax-show-tables.md +++ b/explore-analyze/query-filter/languages/sql-syntax-show-tables.md @@ -38,7 +38,7 @@ javaRestTest |employees |VIEW |ALIAS javaRestTest |library |TABLE |INDEX ``` -Match multiple indices by using {{es}} [multi-target syntax](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#api-multi-index) notation: +Match multiple indices by using {{es}} [multi-target syntax](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#api-multi-index) notation: ```sql SHOW TABLES "*,-l*"; diff --git a/explore-analyze/query-filter/languages/sql-translate.md b/explore-analyze/query-filter/languages/sql-translate.md index 42ccf9cee..dbd6dac7f 100644 --- a/explore-analyze/query-filter/languages/sql-translate.md +++ b/explore-analyze/query-filter/languages/sql-translate.md @@ -52,7 +52,7 @@ Which returns: } ``` -Which is the request that SQL will run to provide the results. In this case, SQL will use the [scroll](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/paginate-search-results.md#scroll-search-results) API. If the result contained an aggregation then SQL would use the normal [search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search). +Which is the request that SQL will run to provide the results. In this case, SQL will use the [scroll](elasticsearch://reference/elasticsearch/rest-apis/paginate-search-results.md#scroll-search-results) API. If the result contained an aggregation then SQL would use the normal [search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search). The request body accepts the same [parameters](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-sql-query) as the [SQL search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-sql-query), excluding `cursor`. diff --git a/explore-analyze/query-filter/tools/console.md b/explore-analyze/query-filter/tools/console.md index 3de92d1f8..143168877 100644 --- a/explore-analyze/query-filter/tools/console.md +++ b/explore-analyze/query-filter/tools/console.md @@ -26,7 +26,7 @@ $$$configuring-console$$$ $$$import-export-console-requests$$$ -**Console** is an interactive UI for sending requests to [{{es}} APIs](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/index.md) and [{{kib}} APIs](https://www.elastic.co/docs/api) and viewing their responses. +**Console** is an interactive UI for sending requests to [{{es}} APIs](elasticsearch://reference/elasticsearch/rest-apis/index.md) and [{{kib}} APIs](https://www.elastic.co/docs/api) and viewing their responses. :::{image} ../../../images/kibana-console.png :alt: Console diff --git a/explore-analyze/query-filter/tools/grok-debugger.md b/explore-analyze/query-filter/tools/grok-debugger.md index 7e9e9f135..e15fbe9d1 100644 --- a/explore-analyze/query-filter/tools/grok-debugger.md +++ b/explore-analyze/query-filter/tools/grok-debugger.md @@ -10,7 +10,7 @@ mapped_pages: You can build and debug grok patterns in the {{kib}} **Grok Debugger** before you use them in your data processing pipelines. Grok is a pattern matching syntax that you can use to parse arbitrary text and structure it. Grok is good for parsing syslog, apache, and other webserver logs, mysql logs, and in general, any log format that is written for human consumption. -Grok patterns are supported in {{es}} [runtime fields](../../../manage-data/data-store/mapping/runtime-fields.md), the {{es}} [grok ingest processor](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/grok-processor.md), and the {{ls}} [grok filter](asciidocalypse://docs/logstash/docs/reference/plugins-filters-grok.md). For syntax, see [Grokking grok](../../scripting/grok.md). +Grok patterns are supported in {{es}} [runtime fields](../../../manage-data/data-store/mapping/runtime-fields.md), the {{es}} [grok ingest processor](elasticsearch://reference/ingestion-tools/enrich-processor/grok-processor.md), and the {{ls}} [grok filter](logstash://reference/plugins-filters-grok.md). For syntax, see [Grokking grok](../../scripting/grok.md). The {{stack}} ships with more than 120 reusable grok patterns. For a complete list of patterns, see [{{es}} grok patterns](https://github.com/elastic/elasticsearch/tree/master/libs/grok/src/main/resources/patterns) and [{{ls}} grok patterns](https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns). diff --git a/explore-analyze/report-and-share.md b/explore-analyze/report-and-share.md index 602a2a86a..20945e0b3 100644 --- a/explore-analyze/report-and-share.md +++ b/explore-analyze/report-and-share.md @@ -132,7 +132,7 @@ To work around the limitations, use filters to create multiple smaller reports, For more information on using Elasticsearch APIs directly, see [Scroll API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-scroll), [Point in time API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-open-point-in-time), [ES|QL](/explore-analyze/query-filter/languages/esql-rest.md) or [SQL](/explore-analyze/query-filter/languages/sql-rest-format.md#_csv) with CSV response data format. We recommend that you use an official Elastic language client: details for each programming language library that Elastic provides are in the [{{es}} Client documentation](https://www.elastic.co/guide/en/elasticsearch/client/index.html). -[Reporting parameters](asciidocalypse://docs/kibana/docs/reference/configuration-reference/reporting-settings.md) can be adjusted to overcome some of these limiting scenarios. Results are dependent on data size, availability, and latency factors and are not guaranteed. +[Reporting parameters](kibana://reference/configuration-reference/reporting-settings.md) can be adjusted to overcome some of these limiting scenarios. Results are dependent on data size, availability, and latency factors and are not guaranteed. ### PNG/PDF report limitations [pdf-limitations] diff --git a/explore-analyze/report-and-share/reporting-troubleshooting-csv.md b/explore-analyze/report-and-share/reporting-troubleshooting-csv.md index 8e1ceaed4..3735744c6 100644 --- a/explore-analyze/report-and-share/reporting-troubleshooting-csv.md +++ b/explore-analyze/report-and-share/reporting-troubleshooting-csv.md @@ -26,7 +26,7 @@ To work around the limitations, use filters to create multiple smaller reports, For more information on using Elasticsearch APIs directly, see [Scroll API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-scroll), [Point in time API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-open-point-in-time), [ES|QL](../query-filter/languages/esql-rest.md) or [SQL](../query-filter/languages/sql-rest-format.md#_csv) with CSV response data format. We recommend that you use an official Elastic language client: details for each programming language library that Elastic provides are in the [{{es}} Client documentation](https://www.elastic.co/guide/en/elasticsearch/client/index.html). -[Reporting parameters](asciidocalypse://docs/kibana/docs/reference/configuration-reference/reporting-settings.md) can be adjusted to overcome some of these limiting scenarios. Results are dependent on data size, availability, and latency factors and are not guaranteed. +[Reporting parameters](kibana://reference/configuration-reference/reporting-settings.md) can be adjusted to overcome some of these limiting scenarios. Results are dependent on data size, availability, and latency factors and are not guaranteed. :::: @@ -43,7 +43,7 @@ The Kibana CSV export feature collects all of the data from Elasticsearch by usi 1. Permissions to read data aliases alone will not work: the permissions are needed on the underlying indices or data streams. 2. In cases where data shards are unavailable or time out, the export will be empty rather than returning partial data. -Some users may benefit from using the [scroll API](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/paginate-search-results.md#scroll-search-results), an alternative to paging through the data. The behavior of this API does not have the limitations of point in time API, however it has its own limitations: +Some users may benefit from using the [scroll API](elasticsearch://reference/elasticsearch/rest-apis/paginate-search-results.md#scroll-search-results), an alternative to paging through the data. The behavior of this API does not have the limitations of point in time API, however it has its own limitations: 1. Search is limited to 500 shards at the very most. 2. In cases where the data shards are unavailable or time out, the export may return partial data. @@ -54,7 +54,7 @@ If you prefer the internal implementation of CSV export to use the scroll API, y xpack.reporting.csv.scroll.strategy: scroll ``` -For more details about CSV export settings, go to [CSV settings](asciidocalypse://docs/kibana/docs/reference/configuration-reference/reporting-settings.md#reporting-csv-settings). +For more details about CSV export settings, go to [CSV settings](kibana://reference/configuration-reference/reporting-settings.md#reporting-csv-settings). ## Socket hangups [reporting-troubleshooting-csv-socket-hangup] diff --git a/explore-analyze/report-and-share/reporting-troubleshooting.md b/explore-analyze/report-and-share/reporting-troubleshooting.md index 1f0c46f35..723857cca 100644 --- a/explore-analyze/report-and-share/reporting-troubleshooting.md +++ b/explore-analyze/report-and-share/reporting-troubleshooting.md @@ -17,12 +17,12 @@ Kibana excels as a data visualization tool. The {{report-features}} exist to exp If you have trouble creating simple reports, there are some general solutions to common problems you might encounter while using {{report-features}}. For tips related to specific types of reports, refer to [CSV](reporting-troubleshooting-csv.md) and [PDF/PNG](reporting-troubleshooting-pdf.md). -## Error messages [reporting-troubleshooting-error-messages] +## Error messages [reporting-troubleshooting-error-messages] There are some common solutions for error messages that you might encounter in {{report-features}}. -### Version conflict engine exceptions [reporting-troubleshooting-version-conflict-exception] +### Version conflict engine exceptions [reporting-troubleshooting-version-conflict-exception] If you are running multiple instances of {{kib}} in a cluster, the instances share the work of running report jobs to evenly distribute the workload. Each instance searches the reporting index for "pending" jobs that the user has requested. It is possible for multiple instances to find the same job in these searches. Only the instance that successfully updated the job status to "processing" will actually run the report job. The other instances that unsuccessfully tried to make the same update will log something similar to this: @@ -44,17 +44,17 @@ StatusCodeError: [version_conflict_engine_exception] [...]: version conflict, re These messages alone don’t indicate a problem. They show normal events that happen in a healthy system. -### Max attempts reached [_max_attempts_reached] +### Max attempts reached [_max_attempts_reached] There are two primary causes for a "Max attempts reached" error: * You’re creating a PDF of a visualization or dashboard that spans a large amount of data and Kibana is hitting the `xpack.reporting.queue.timeout` -* Kibana is hosted behind a reverse-proxy, and the [Kibana server settings](asciidocalypse://docs/kibana/docs/reference/configuration-reference/reporting-settings.md#reporting-kibana-server-settings) are not configured correctly +* Kibana is hosted behind a reverse-proxy, and the [Kibana server settings](kibana://reference/configuration-reference/reporting-settings.md#reporting-kibana-server-settings) are not configured correctly -Create a Markdown visualization and then create a PDF report. If this succeeds, increase the `xpack.reporting.queue.timeout` setting. If the PDF report fails with "Max attempts reached," check your [Kibana server settings](asciidocalypse://docs/kibana/docs/reference/configuration-reference/reporting-settings.md#reporting-kibana-server-settings). +Create a Markdown visualization and then create a PDF report. If this succeeds, increase the `xpack.reporting.queue.timeout` setting. If the PDF report fails with "Max attempts reached," check your [Kibana server settings](kibana://reference/configuration-reference/reporting-settings.md#reporting-kibana-server-settings). -## Verbose logging [reporting-troubleshooting-verbose-logs] +## Verbose logging [reporting-troubleshooting-verbose-logs] {{kib}} server logs have a lot of useful information for troubleshooting and understanding how things work. The full logs from {{report-features}} are a good place to look when you encounter problems. In `kibana.yml`: diff --git a/explore-analyze/scripting/dissect.md b/explore-analyze/scripting/dissect.md index 48592568e..4630a5fd4 100644 --- a/explore-analyze/scripting/dissect.md +++ b/explore-analyze/scripting/dissect.md @@ -55,7 +55,7 @@ Now that you have a dissect pattern, how do you test and use it? ## Test dissect patterns with Painless [dissect-patterns-test] -You can incorporate dissect patterns into Painless scripts to extract data. To test your script, use either the [field contexts](asciidocalypse://docs/elasticsearch/docs/reference/scripting-languages/painless/painless-api-examples.md#painless-execute-runtime-field-context) of the Painless execute API or create a runtime field that includes the script. Runtime fields offer greater flexibility and accept multiple documents, but the Painless execute API is a great option if you don’t have write access on a cluster where you’re testing a script. +You can incorporate dissect patterns into Painless scripts to extract data. To test your script, use either the [field contexts](elasticsearch://reference/scripting-languages/painless/painless-api-examples.md#painless-execute-runtime-field-context) of the Painless execute API or create a runtime field that includes the script. Runtime fields offer greater flexibility and accept multiple documents, but the Painless execute API is a great option if you don’t have write access on a cluster where you’re testing a script. For example, test your dissect pattern with the Painless execute API by including your Painless script and a single document that matches your data. Start by indexing the `message` field as a `wildcard` data type: diff --git a/explore-analyze/scripting/grok.md b/explore-analyze/scripting/grok.md index 370afb18b..2b5e389f8 100644 --- a/explore-analyze/scripting/grok.md +++ b/explore-analyze/scripting/grok.md @@ -46,14 +46,14 @@ The first value is a number, followed by what appears to be an IP address. You c To ease migration to the [Elastic Common Schema (ECS)](https://www.elastic.co/guide/en/ecs/current), a new set of ECS-compliant patterns is available in addition to the existing patterns. The new ECS pattern definitions capture event field names that are compliant with the schema. -The ECS pattern set has all of the pattern definitions from the legacy set, and is a drop-in replacement. Use the [`ecs-compatability`](asciidocalypse://docs/logstash/docs/reference/plugins-filters-grok.md#plugins-filters-grok-ecs_compatibility) setting to switch modes. +The ECS pattern set has all of the pattern definitions from the legacy set, and is a drop-in replacement. Use the [`ecs-compatability`](logstash://reference/plugins-filters-grok.md#plugins-filters-grok-ecs_compatibility) setting to switch modes. New features and enhancements will be added to the ECS-compliant files. The legacy patterns may still receive bug fixes which are backwards compatible. ## Use grok patterns in Painless scripts [grok-patterns] -You can incorporate predefined grok patterns into Painless scripts to extract data. To test your script, use either the [field contexts](asciidocalypse://docs/elasticsearch/docs/reference/scripting-languages/painless/painless-api-examples.md#painless-execute-runtime-field-context) of the Painless execute API or create a runtime field that includes the script. Runtime fields offer greater flexibility and accept multiple documents, but the Painless execute API is a great option if you don’t have write access on a cluster where you’re testing a script. +You can incorporate predefined grok patterns into Painless scripts to extract data. To test your script, use either the [field contexts](elasticsearch://reference/scripting-languages/painless/painless-api-examples.md#painless-execute-runtime-field-context) of the Painless execute API or create a runtime field that includes the script. Runtime fields offer greater flexibility and accept multiple documents, but the Painless execute API is a great option if you don’t have write access on a cluster where you’re testing a script. ::::{tip} If you need help building grok patterns to match your data, use the [Grok Debugger](../query-filter/tools/grok-debugger.md) tool in {{kib}}. @@ -67,7 +67,7 @@ For example, if you’re working with Apache log data, you can use the `%{{COMMO [30/Apr/2020:14:30:17 -0500] \"GET /images/hm_bg.jpg HTTP/1.0\" 200 24736" ``` -To extract the IP address from the `message` field, you can write a Painless script that incorporates the `%{{COMMONAPACHELOG}}` syntax. You can test this script using the [`ip` field context](asciidocalypse://docs/elasticsearch/docs/reference/scripting-languages/painless/painless-api-examples.md#painless-runtime-ip) of the Painless execute API, but let’s use a runtime field instead. +To extract the IP address from the `message` field, you can write a Painless script that incorporates the `%{{COMMONAPACHELOG}}` syntax. You can test this script using the [`ip` field context](elasticsearch://reference/scripting-languages/painless/painless-api-examples.md#painless-runtime-ip) of the Painless execute API, but let’s use a runtime field instead. Based on the sample document, index the `@timestamp` and `message` fields. To remain flexible, use `wildcard` as the field type for `message`: @@ -154,7 +154,7 @@ GET my-index/_search ## Return calculated results [grok-pattern-results] -Using the `http.clientip` runtime field, you can define a simple query to run a search for a specific IP address and return all related fields. The [`fields`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/retrieve-selected-fields.md) parameter on the `_search` API works for all fields, even those that weren’t sent as part of the original `_source`: +Using the `http.clientip` runtime field, you can define a simple query to run a search for a specific IP address and return all related fields. The [`fields`](elasticsearch://reference/elasticsearch/rest-apis/retrieve-selected-fields.md) parameter on the `_search` API works for all fields, even those that weren’t sent as part of the original `_source`: ```console GET my-index/_search diff --git a/explore-analyze/scripting/modules-scripting-engine.md b/explore-analyze/scripting/modules-scripting-engine.md index 06e0f1d25..52093e8bd 100644 --- a/explore-analyze/scripting/modules-scripting-engine.md +++ b/explore-analyze/scripting/modules-scripting-engine.md @@ -10,7 +10,7 @@ mapped_pages: A `ScriptEngine` is a backend for implementing a scripting language. It may also be used to write scripts that need to use advanced internals of scripting. For example, a script that wants to use term frequencies while scoring. -The plugin [documentation](asciidocalypse://docs/elasticsearch/docs/extend/index.md) has more information on how to write a plugin so that Elasticsearch will properly load it. To register the `ScriptEngine`, your plugin should implement the `ScriptPlugin` interface and override the `getScriptEngine(Settings settings)` method. +The plugin [documentation](elasticsearch://extend/index.md) has more information on how to write a plugin so that Elasticsearch will properly load it. To register the `ScriptEngine`, your plugin should implement the `ScriptPlugin` interface and override the `getScriptEngine(Settings settings)` method. The following is an example of a custom `ScriptEngine` which uses the language name `expert_scripts`. It implements a single script called `pure_df` which may be used as a search script to override each document’s score as the document frequency of a provided term. diff --git a/explore-analyze/scripting/modules-scripting-fields.md b/explore-analyze/scripting/modules-scripting-fields.md index 88c591a86..5f891ea4c 100644 --- a/explore-analyze/scripting/modules-scripting-fields.md +++ b/explore-analyze/scripting/modules-scripting-fields.md @@ -11,34 +11,34 @@ mapped_pages: Depending on where a script is used, it will have access to certain special variables and document fields. -## Update scripts [_update_scripts] +## Update scripts [_update_scripts] A script used in the [update](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-update), [update-by-query](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-update-by-query), or [reindex](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex) API will have access to the `ctx` variable which exposes: `ctx._source` -: Access to the document [`_source` field](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md). +: Access to the document [`_source` field](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md). `ctx.op` : The operation that should be applied to the document: `index` or `delete`. `ctx._index` etc -: Access to [document metadata fields](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/document-metadata-fields.md), some of which may be read-only. +: Access to [document metadata fields](elasticsearch://reference/elasticsearch/mapping-reference/document-metadata-fields.md), some of which may be read-only. These scripts do not have access to the `doc` variable and have to use `ctx` to access the documents they operate on. -## Search and aggregation scripts [_search_and_aggregation_scripts] +## Search and aggregation scripts [_search_and_aggregation_scripts] -With the exception of [script fields](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/retrieve-selected-fields.md#script-fields) which are executed once per search hit, scripts used in search and aggregations will be executed once for every document which might match a query or an aggregation. Depending on how many documents you have, this could mean millions or billions of executions: these scripts need to be fast! +With the exception of [script fields](elasticsearch://reference/elasticsearch/rest-apis/retrieve-selected-fields.md#script-fields) which are executed once per search hit, scripts used in search and aggregations will be executed once for every document which might match a query or an aggregation. Depending on how many documents you have, this could mean millions or billions of executions: these scripts need to be fast! Field values can be accessed from a script using [doc-values](#modules-scripting-doc-vals), [the `_source` field](#modules-scripting-source), or [stored fields](#modules-scripting-stored), each of which is explained below. -### Accessing the score of a document within a script [scripting-score] +### Accessing the score of a document within a script [scripting-score] -Scripts used in the [`function_score` query](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-function-score-query.md), in [script-based sorting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/sort-search-results.md), or in [aggregations](../query-filter/aggregations.md) have access to the `_score` variable which represents the current relevance score of a document. +Scripts used in the [`function_score` query](elasticsearch://reference/query-languages/query-dsl-function-score-query.md), in [script-based sorting](elasticsearch://reference/elasticsearch/rest-apis/sort-search-results.md), or in [aggregations](../query-filter/aggregations.md) have access to the `_score` variable which represents the current relevance score of a document. -Here’s an example of using a script in a [`function_score` query](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-function-score-query.md) to alter the relevance `_score` of each document: +Here’s an example of using a script in a [`function_score` query](elasticsearch://reference/query-languages/query-dsl-function-score-query.md) to alter the relevance `_score` of each document: ```console PUT my-index-000001/_doc/1?refresh @@ -74,11 +74,11 @@ GET my-index-000001/_search ``` -### Accessing term statistics of a document within a script [scripting-term-statistics] +### Accessing term statistics of a document within a script [scripting-term-statistics] -Scripts used in a [`script_score`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-script-score-query.md) query have access to the `_termStats` variable which provides statistical information about the terms in the child query. +Scripts used in a [`script_score`](elasticsearch://reference/query-languages/query-dsl-script-score-query.md) query have access to the `_termStats` variable which provides statistical information about the terms in the child query. -In the following example, `_termStats` is used within a [`script_score`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-script-score-query.md) query to retrieve the average term frequency for the terms `quick`, `brown`, and `fox` in the `text` field: +In the following example, `_termStats` is used within a [`script_score`](elasticsearch://reference/query-languages/query-dsl-script-score-query.md) query to retrieve the average term frequency for the terms `quick`, `brown`, and `fox` in the `text` field: ```console PUT my-index-000001/_doc/1?refresh @@ -141,9 +141,9 @@ The `_termStats` variable is only available when using the [Painless](modules-sc -### Doc values [modules-scripting-doc-vals] +### Doc values [modules-scripting-doc-vals] -By far the fastest most efficient way to access a field value from a script is to use the `doc['field_name']` syntax, which retrieves the field value from [doc values](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/doc-values.md). Doc values are a columnar field value store, enabled by default on all fields except for [analyzed `text` fields](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/text.md). +By far the fastest most efficient way to access a field value from a script is to use the `doc['field_name']` syntax, which retrieves the field value from [doc values](elasticsearch://reference/elasticsearch/mapping-reference/doc-values.md). Doc values are a columnar field value store, enabled by default on all fields except for [analyzed `text` fields](elasticsearch://reference/elasticsearch/mapping-reference/text.md). ```console PUT my-index-000001/_doc/1?refresh @@ -180,22 +180,22 @@ The `doc['field']` will throw an error if `field` is missing from the mappings. ::::{admonition} Doc values and `text` fields :class: note -The `doc['field']` syntax can also be used for [analyzed `text` fields](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/text.md) if [`fielddata`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/text.md#fielddata-mapping-param) is enabled, but **BEWARE**: enabling fielddata on a `text` field requires loading all of the terms into the JVM heap, which can be very expensive both in terms of memory and CPU. It seldom makes sense to access `text` fields from scripts. +The `doc['field']` syntax can also be used for [analyzed `text` fields](elasticsearch://reference/elasticsearch/mapping-reference/text.md) if [`fielddata`](elasticsearch://reference/elasticsearch/mapping-reference/text.md#fielddata-mapping-param) is enabled, but **BEWARE**: enabling fielddata on a `text` field requires loading all of the terms into the JVM heap, which can be very expensive both in terms of memory and CPU. It seldom makes sense to access `text` fields from scripts. :::: -### The document `_source` [modules-scripting-source] +### The document `_source` [modules-scripting-source] -The document [`_source`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md) can be accessed using the `_source.field_name` syntax. The `_source` is loaded as a map-of-maps, so properties within object fields can be accessed as, for example, `_source.name.first`. +The document [`_source`](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md) can be accessed using the `_source.field_name` syntax. The `_source` is loaded as a map-of-maps, so properties within object fields can be accessed as, for example, `_source.name.first`. ::::{admonition} Prefer doc-values to _source :class: important Accessing the `_source` field is much slower than using doc-values. The _source field is optimised for returning several fields per result, while doc values are optimised for accessing the value of a specific field in many documents. -It makes sense to use `_source` when generating a [script field](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/retrieve-selected-fields.md#script-fields) for the top ten hits from a search result but, for other search and aggregation use cases, always prefer using doc values. +It makes sense to use `_source` when generating a [script field](elasticsearch://reference/elasticsearch/rest-apis/retrieve-selected-fields.md#script-fields) for the top ten hits from a search result but, for other search and aggregation use cases, always prefer using doc values. :::: @@ -237,9 +237,9 @@ GET my-index-000001/_search ``` -### Stored fields [modules-scripting-stored] +### Stored fields [modules-scripting-stored] -*Stored fields* — fields explicitly marked as [`"store": true`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-store.md) in the mapping — can be accessed using the `_fields['field_name'].value` or `_fields['field_name']` syntax: +*Stored fields* — fields explicitly marked as [`"store": true`](elasticsearch://reference/elasticsearch/mapping-reference/mapping-store.md) in the mapping — can be accessed using the `_fields['field_name'].value` or `_fields['field_name']` syntax: ```console PUT my-index-000001 diff --git a/explore-analyze/scripting/modules-scripting-painless.md b/explore-analyze/scripting/modules-scripting-painless.md index a668b26ee..99bbef45c 100644 --- a/explore-analyze/scripting/modules-scripting-painless.md +++ b/explore-analyze/scripting/modules-scripting-painless.md @@ -22,4 +22,4 @@ Painless provides numerous capabilities that center around the following core pr Ready to start scripting with Painless? Learn how to [write your first script](modules-scripting-using.md). -If you’re already familiar with Painless, see the [Painless Language Specification](asciidocalypse://docs/elasticsearch/docs/reference/scripting-languages/painless/painless-language-specification.md) for a detailed description of the Painless syntax and language features. +If you’re already familiar with Painless, see the [Painless Language Specification](elasticsearch://reference/scripting-languages/painless/painless-language-specification.md) for a detailed description of the Painless syntax and language features. diff --git a/explore-analyze/scripting/modules-scripting-security.md b/explore-analyze/scripting/modules-scripting-security.md index 410e2c5cd..79ef2fd30 100644 --- a/explore-analyze/scripting/modules-scripting-security.md +++ b/explore-analyze/scripting/modules-scripting-security.md @@ -16,16 +16,16 @@ The second layer of security is the [Java Security Manager](https://www.oracle.c {{es}} uses [seccomp](https://en.wikipedia.org/wiki/Seccomp) in Linux, [Seatbelt](https://www.chromium.org/developers/design-documents/sandbox/osx-sandboxing-design) in macOS, and [ActiveProcessLimit](https://msdn.microsoft.com/en-us/library/windows/desktop/ms684147) on Windows as additional security layers to prevent {{es}} from forking or running other processes. -Finally, scripts used in [scripted metrics aggregations](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-scripted-metric-aggregation.md) can be restricted to a defined list of scripts, or forbidden altogether. This can prevent users from running particularly slow or resource intensive aggregation queries. +Finally, scripts used in [scripted metrics aggregations](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-scripted-metric-aggregation.md) can be restricted to a defined list of scripts, or forbidden altogether. This can prevent users from running particularly slow or resource intensive aggregation queries. -You can modify the following script settings to restrict the type of scripts that are allowed to run, and control the available [contexts](asciidocalypse://docs/elasticsearch/docs/reference/scripting-languages/painless/painless-contexts.md) that scripts can run in. To implement additional layers in your defense in depth strategy, follow the [{{es}} security principles](../../deploy-manage/security.md). +You can modify the following script settings to restrict the type of scripts that are allowed to run, and control the available [contexts](elasticsearch://reference/scripting-languages/painless/painless-contexts.md) that scripts can run in. To implement additional layers in your defense in depth strategy, follow the [{{es}} security principles](../../deploy-manage/security.md). -## Allowed script types setting [allowed-script-types-setting] +## Allowed script types setting [allowed-script-types-setting] {{es}} supports two script types: `inline` and `stored`. By default, {{es}} is configured to run both types of scripts. To limit what type of scripts are run, set `script.allowed_types` to `inline` or `stored`. To prevent any scripts from running, set `script.allowed_types` to `none`. -::::{important} +::::{important} If you use {{kib}}, set `script.allowed_types` to both or just `inline`. Some {{kib}} features rely on inline scripts and do not function as expected if {{es}} does not allow inline scripts. :::: @@ -37,7 +37,7 @@ script.allowed_types: inline ``` -## Allowed script contexts setting [allowed-script-contexts-setting] +## Allowed script contexts setting [allowed-script-contexts-setting] By default, all script contexts are permitted. Use the `script.allowed_contexts` setting to specify the contexts that are allowed. To specify that no contexts are allowed, set `script.allowed_contexts` to `none`. @@ -48,9 +48,9 @@ script.allowed_contexts: score, update ``` -## Allowed scripts in scripted metrics aggregations [allowed-script-in-aggs-settings] +## Allowed scripts in scripted metrics aggregations [allowed-script-in-aggs-settings] -By default, all scripts are permitted in [scripted metrics aggregations](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-scripted-metric-aggregation.md). To restrict the set of allowed scripts, set [`search.aggs.only_allowed_metric_scripts`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/search-settings.md#search-settings-only-allowed-scripts) to `true` and provide the allowed scripts using [`search.aggs.allowed_inline_metric_scripts`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/search-settings.md#search-settings-allowed-inline-scripts) and/or [`search.aggs.allowed_stored_metric_scripts`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/search-settings.md#search-settings-allowed-stored-scripts). +By default, all scripts are permitted in [scripted metrics aggregations](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-scripted-metric-aggregation.md). To restrict the set of allowed scripts, set [`search.aggs.only_allowed_metric_scripts`](elasticsearch://reference/elasticsearch/configuration-reference/search-settings.md#search-settings-only-allowed-scripts) to `true` and provide the allowed scripts using [`search.aggs.allowed_inline_metric_scripts`](elasticsearch://reference/elasticsearch/configuration-reference/search-settings.md#search-settings-allowed-inline-scripts) and/or [`search.aggs.allowed_stored_metric_scripts`](elasticsearch://reference/elasticsearch/configuration-reference/search-settings.md#search-settings-allowed-stored-scripts). To disallow certain script types, omit the corresponding script list (`search.aggs.allowed_inline_metric_scripts` or `search.aggs.allowed_stored_metric_scripts`) or set it to an empty array. When both script lists are not empty, the given stored scripts and the given inline scripts will be allowed. diff --git a/explore-analyze/scripting/modules-scripting-using.md b/explore-analyze/scripting/modules-scripting-using.md index e37b0a284..085f7e036 100644 --- a/explore-analyze/scripting/modules-scripting-using.md +++ b/explore-analyze/scripting/modules-scripting-using.md @@ -28,13 +28,13 @@ Wherever scripting is supported in the {{es}} APIs, the syntax follows the same : Specifies any named parameters that are passed into the script as variables. [Use parameters](#prefer-params) instead of hard-coded values to decrease compile time. -## Write your first script [hello-world-script] +## Write your first script [hello-world-script] [Painless](modules-scripting-painless.md) is the default scripting language for {{es}}. It is secure, performant, and provides a natural syntax for anyone with a little coding experience. A Painless script is structured as one or more statements and optionally has one or more user-defined functions at the beginning. A script must always have at least one statement. -The [Painless execute API](asciidocalypse://docs/elasticsearch/docs/reference/scripting-languages/painless/painless-api-examples.md) provides the ability to test a script with simple user-defined parameters and receive a result. Let’s start with a complete script and review its constituent parts. +The [Painless execute API](elasticsearch://reference/scripting-languages/painless/painless-api-examples.md) provides the ability to test a script with simple user-defined parameters and receive a result. Let’s start with a complete script and review its constituent parts. First, index a document with a single field so that we have some data to work with: @@ -45,7 +45,7 @@ PUT my-index-000001/_doc/1 } ``` -We can then construct a script that operates on that field and run evaluate the script as part of a query. The following query uses the [`script_fields`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/retrieve-selected-fields.md#script-fields) parameter of the search API to retrieve a script valuation. There’s a lot happening here, but we’ll break it down the components to understand them individually. For now, you only need to understand that this script takes `my_field` and operates on it. +We can then construct a script that operates on that field and run evaluate the script as part of a query. The following query uses the [`script_fields`](elasticsearch://reference/elasticsearch/rest-apis/retrieve-selected-fields.md#script-fields) parameter of the search API to retrieve a script valuation. There’s a lot happening here, but we’ll break it down the components to understand them individually. For now, you only need to understand that this script takes `my_field` and operates on it. ```console GET my-index-000001/_search @@ -70,7 +70,7 @@ GET my-index-000001/_search The `script` is a standard JSON object that defines scripts under most APIs in {{es}}. This object requires `source` to define the script itself. The script doesn’t specify a language, so it defaults to Painless. -## Use parameters in your script [prefer-params] +## Use parameters in your script [prefer-params] The first time {{es}} sees a new script, it compiles the script and stores the compiled version in a cache. Compilation can be a heavy process. Rather than hard-coding values in your script, pass them as named `params` instead. @@ -97,13 +97,13 @@ You can compile up to 150 scripts per 5 minutes by default. For ingest contexts, script.context.field.max_compilations_rate=100/10m ``` -::::{important} +::::{important} If you compile too many unique scripts within a short time, {{es}} rejects the new dynamic scripts with a `circuit_breaking_exception` error. :::: -## Shorten your script [script-shorten-syntax] +## Shorten your script [script-shorten-syntax] Using syntactic abilities that are native to Painless, you can reduce verbosity in your scripts and make them shorter. Here’s a simple script that we can make shorter: @@ -152,13 +152,13 @@ This version of the script removes several components and simplifies the syntax Use this abbreviated syntax anywhere that {{es}} supports scripts, such as when you’re creating [runtime fields](../../manage-data/data-store/mapping/map-runtime-field.md). -## Store and retrieve scripts [script-stored-scripts] +## Store and retrieve scripts [script-stored-scripts] You can store and retrieve scripts from the cluster state using the [stored script APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-script). Stored scripts allow you to reference shared scripts for operations like scoring, aggregating, filtering, and reindexing. Instead of embedding scripts inline in each query, you can reference these shared operations. Stored scripts can also reduce request payload size. Depending on script size and request frequency, this can help lower latency and data transfer costs. -::::{note} +::::{note} Unlike regular scripts, stored scripts require that you specify a script language using the `lang` parameter. :::: @@ -214,7 +214,7 @@ DELETE _scripts/calculate-score ``` -## Update documents with scripts [scripts-update-scripts] +## Update documents with scripts [scripts-update-scripts] You can use the [update API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-update) to update documents with a specified script. The script can update, delete, or skip modifying the document. The update API also supports passing a partial document, which is merged into the existing document. diff --git a/explore-analyze/scripting/scripting-field-extraction.md b/explore-analyze/scripting/scripting-field-extraction.md index 3f4ca58d7..ad3490e90 100644 --- a/explore-analyze/scripting/scripting-field-extraction.md +++ b/explore-analyze/scripting/scripting-field-extraction.md @@ -246,7 +246,7 @@ The following pattern tells dissect to return the term `used`, a blank space, th emit("used" + ' ' + gc.usize + ', ' + "capacity" + ' ' + gc.csize + ', ' + "committed" + ' ' + gc.comsize) ``` -Putting it all together, you can create a runtime field named `gc_size` in a search request. Using the [`fields` option](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/retrieve-selected-fields.md#search-fields-param), you can retrieve all values for the `gc_size` runtime field. This query also includes a bucket aggregation to group your data. +Putting it all together, you can create a runtime field named `gc_size` in a search request. Using the [`fields` option](elasticsearch://reference/elasticsearch/rest-apis/retrieve-selected-fields.md#search-fields-param), you can retrieve all values for the `gc_size` runtime field. This query also includes a bucket aggregation to group your data. ```console GET my-index/_search diff --git a/explore-analyze/scripting/scripts-search-speed.md b/explore-analyze/scripting/scripts-search-speed.md index db12416d8..ba2bbd18f 100644 --- a/explore-analyze/scripting/scripts-search-speed.md +++ b/explore-analyze/scripting/scripts-search-speed.md @@ -16,13 +16,13 @@ If you see a large number of script cache evictions and a rising number of compi All scripts are cached by default so that they only need to be recompiled when updates occur. By default, scripts do not have a time-based expiration. You can change this behavior by using the `script.cache.expire` setting. Use the `script.cache.max_size` setting to configure the size of the cache. -::::{note} +::::{note} The size of scripts is limited to 65,535 bytes. Set the value of `script.max_size_in_bytes` to increase that soft limit. If your scripts are really large, then consider using a [native script engine](modules-scripting-engine.md). :::: -## Improving search speed [_improving_search_speed] +## Improving search speed [_improving_search_speed] Scripts are incredibly useful, but can’t use {{es}}'s index structures or related optimizations. This relationship can sometimes result in slower search speeds. @@ -72,7 +72,7 @@ PUT /my_test_scores/_mapping } ``` -Next, use an [ingest pipeline](../../manage-data/ingest/transform-enrich/ingest-pipelines.md) containing the [script processor](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/script-processor.md) to calculate the sum of `math_score` and `verbal_score` and index it in the `total_score` field. +Next, use an [ingest pipeline](../../manage-data/ingest/transform-enrich/ingest-pipelines.md) containing the [script processor](elasticsearch://reference/ingestion-tools/enrich-processor/script-processor.md) to calculate the sum of `math_score` and `verbal_score` and index it in the `total_score` field. ```console PUT _ingest/pipeline/my_test_scores_pipeline diff --git a/explore-analyze/transforms/ecommerce-transforms.md b/explore-analyze/transforms/ecommerce-transforms.md index 3e938b3cb..fb82e8af3 100644 --- a/explore-analyze/transforms/ecommerce-transforms.md +++ b/explore-analyze/transforms/ecommerce-transforms.md @@ -27,7 +27,7 @@ mapped_pages: :class: screenshot ::: - Group the data by customer ID and add one or more aggregations to learn more about each customer’s orders. For example, let’s calculate the sum of products they purchased, the total price of their purchases, the maximum number of products that they purchased in a single order, and their total number of orders. We’ll accomplish this by using the [`sum` aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-sum-aggregation.md) on the `total_quantity` and `taxless_total_price` fields, the [`max` aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-max-aggregation.md) on the `total_quantity` field, and the [`cardinality` aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-cardinality-aggregation.md) on the `order_id` field: + Group the data by customer ID and add one or more aggregations to learn more about each customer’s orders. For example, let’s calculate the sum of products they purchased, the total price of their purchases, the maximum number of products that they purchased in a single order, and their total number of orders. We’ll accomplish this by using the [`sum` aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-sum-aggregation.md) on the `total_quantity` and `taxless_total_price` fields, the [`max` aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-max-aggregation.md) on the `total_quantity` field, and the [`cardinality` aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-cardinality-aggregation.md) on the `order_id` field: :::{image} ../../images/elasticsearch-reference-ecommerce-pivot2.png :alt: Adding multiple aggregations to a {{transform}} in {{kib}} @@ -171,7 +171,7 @@ mapped_pages: :::: 5. Optional: Create the destination index. - If the destination index does not exist, it is created the first time you start your {{transform}}. A pivot transform deduces the mappings for the destination index from the source indices and the transform aggregations. If there are fields in the destination index that are derived from scripts (for example, if you use [`scripted_metrics`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-scripted-metric-aggregation.md) or [`bucket_scripts`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-bucket-script-aggregation.md) aggregations), they’re created with [dynamic mappings](../../manage-data/data-store/mapping/dynamic-mapping.md). You can use the preview {{transform}} API to preview the mappings it will use for the destination index. In {{kib}}, if you copied the API request to your clipboard, paste it into the console, then refer to the `generated_dest_index` object in the API response. + If the destination index does not exist, it is created the first time you start your {{transform}}. A pivot transform deduces the mappings for the destination index from the source indices and the transform aggregations. If there are fields in the destination index that are derived from scripts (for example, if you use [`scripted_metrics`](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-scripted-metric-aggregation.md) or [`bucket_scripts`](elasticsearch://reference/data-analysis/aggregations/search-aggregations-pipeline-bucket-script-aggregation.md) aggregations), they’re created with [dynamic mappings](../../manage-data/data-store/mapping/dynamic-mapping.md). You can use the preview {{transform}} API to preview the mappings it will use for the destination index. In {{kib}}, if you copied the API request to your clipboard, paste it into the console, then refer to the `generated_dest_index` object in the API response. ::::{note} {{transforms-cap}} might have more configuration options provided by the APIs than the options available in {{kib}}. For example, you can set an ingest pipeline for `dest` by calling the [Create {{transform}}](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-put-transform). For all the {{transform}} configuration options, refer to the [documentation](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-transform). :::: diff --git a/explore-analyze/transforms/transform-checkpoints.md b/explore-analyze/transforms/transform-checkpoints.md index 09699505e..db09d7eb7 100644 --- a/explore-analyze/transforms/transform-checkpoints.md +++ b/explore-analyze/transforms/transform-checkpoints.md @@ -41,7 +41,7 @@ If the cluster experiences unsuitable performance degradation due to the {{trans In most cases, it is strongly recommended to use the ingest timestamp of the source indices for syncing the {{transform}}. This is the most optimal way for {{transforms}} to be able to identify new changes. If your data source follows the [ECS standard](asciidocalypse://docs/ecs/docs/reference/index.md), you might already have an [`event.ingested`](asciidocalypse://docs/ecs/docs/reference/ecs-event.md#field-event-ingested) field. In this case, use `event.ingested` as the `sync`.`time`.`field` property of your {{transform}}. -If you don’t have a `event.ingested` field or it isn’t populated, you can set it by using an ingest pipeline. Create an ingest pipeline either using the [ingest pipeline API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-put-pipeline) (like the example below) or via {{kib}} under **Stack Management > Ingest Pipelines**. Use a [`set` processor](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/set-processor.md) to set the field and associate it with the value of the ingest timestamp. +If you don’t have a `event.ingested` field or it isn’t populated, you can set it by using an ingest pipeline. Create an ingest pipeline either using the [ingest pipeline API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-put-pipeline) (like the example below) or via {{kib}} under **Stack Management > Ingest Pipelines**. Use a [`set` processor](elasticsearch://reference/ingestion-tools/enrich-processor/set-processor.md) to set the field and associate it with the value of the ingest timestamp. ```console PUT _ingest/pipeline/set_ingest_time diff --git a/explore-analyze/transforms/transform-examples.md b/explore-analyze/transforms/transform-examples.md index 9d112bb7b..5feffc7ba 100644 --- a/explore-analyze/transforms/transform-examples.md +++ b/explore-analyze/transforms/transform-examples.md @@ -94,7 +94,7 @@ It’s possible to answer these questions using aggregations alone, however {{tr ## Finding air carriers with the most delays [example-airline] -This example uses the Flights sample data set to find out which air carrier had the most delays. First, filter the source data such that it excludes all the cancelled flights by using a query filter. Then transform the data to contain the distinct number of flights, the sum of delayed minutes, and the sum of the flight minutes by air carrier. Finally, use a [`bucket_script`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-bucket-script-aggregation.md) to determine what percentage of the flight time was actually delay. +This example uses the Flights sample data set to find out which air carrier had the most delays. First, filter the source data such that it excludes all the cancelled flights by using a query filter. Then transform the data to contain the distinct number of flights, the sum of delayed minutes, and the sum of the flight minutes by air carrier. Finally, use a [`bucket_script`](elasticsearch://reference/data-analysis/aggregations/search-aggregations-pipeline-bucket-script-aggregation.md) to determine what percentage of the flight time was actually delay. ```console POST _transform/_preview @@ -415,9 +415,9 @@ This {{transform}} makes it easier to answer questions such as: ## Finding client IPs that sent the most bytes to the server [example-bytes] -This example uses the web log sample data set to find the client IP that sent the most bytes to the server in every hour. The example uses a `pivot` {{transform}} with a [`top_metrics`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-top-metrics.md) aggregation. +This example uses the web log sample data set to find the client IP that sent the most bytes to the server in every hour. The example uses a `pivot` {{transform}} with a [`top_metrics`](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-top-metrics.md) aggregation. -Group the data by a [date histogram](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-composite-aggregation.md#_date_histogram) on the time field with an interval of one hour. Use a [max aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-max-aggregation.md) on the `bytes` field to get the maximum amount of data that is sent to the server. Without the `max` aggregation, the API call still returns the client IP that sent the most bytes, however, the amount of bytes that it sent is not returned. In the `top_metrics` property, specify `clientip` and `geo.src`, then sort them by the `bytes` field in descending order. The {{transform}} returns the client IP that sent the biggest amount of data and the 2-letter ISO code of the corresponding location. +Group the data by a [date histogram](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-composite-aggregation.md#_date_histogram) on the time field with an interval of one hour. Use a [max aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-max-aggregation.md) on the `bytes` field to get the maximum amount of data that is sent to the server. Without the `max` aggregation, the API call still returns the client IP that sent the most bytes, however, the amount of bytes that it sent is not returned. In the `top_metrics` property, specify `clientip` and `geo.src`, then sort them by the `bytes` field in descending order. The {{transform}} returns the client IP that sent the biggest amount of data and the 2-letter ISO code of the corresponding location. ```console POST _transform/_preview diff --git a/explore-analyze/transforms/transform-limitations.md b/explore-analyze/transforms/transform-limitations.md index febf4c0d1..226f02477 100644 --- a/explore-analyze/transforms/transform-limitations.md +++ b/explore-analyze/transforms/transform-limitations.md @@ -49,7 +49,7 @@ A {{ctransform}} periodically checks for changes to source data. The functionali ### Aggregation responses may be incompatible with destination index mappings [transform-aggresponse-limitations] -When a pivot {{transform}} is first started, it will deduce the mappings required for the destination index. This process is based on the field types of the source index and the aggregations used. If the fields are derived from [`scripted_metrics`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-scripted-metric-aggregation.md) or [`bucket_scripts`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-bucket-script-aggregation.md), [dynamic mappings](../../manage-data/data-store/mapping/dynamic-mapping.md) will be used. In some instances the deduced mappings may be incompatible with the actual data. For example, numeric overflows might occur or dynamically mapped fields might contain both numbers and strings. Please check {{es}} logs if you think this may have occurred. +When a pivot {{transform}} is first started, it will deduce the mappings required for the destination index. This process is based on the field types of the source index and the aggregations used. If the fields are derived from [`scripted_metrics`](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-scripted-metric-aggregation.md) or [`bucket_scripts`](elasticsearch://reference/data-analysis/aggregations/search-aggregations-pipeline-bucket-script-aggregation.md), [dynamic mappings](../../manage-data/data-store/mapping/dynamic-mapping.md) will be used. In some instances the deduced mappings may be incompatible with the actual data. For example, numeric overflows might occur or dynamically mapped fields might contain both numbers and strings. Please check {{es}} logs if you think this may have occurred. You can view the deduced mappings by using the [preview transform API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-preview-transform). See the `generated_dest_index` object in the API response. @@ -57,7 +57,7 @@ If it’s required, you may define custom mappings prior to starting the {{trans ### Batch {{transforms}} may not account for changed documents [transform-batch-limitations] -A batch {{transform}} uses a [composite aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-composite-aggregation.md) which allows efficient pagination through all buckets. Composite aggregations do not yet support a search context, therefore if the source data is changed (deleted, updated, added) while the batch {{dataframe}} is in progress, then the results may not include these changes. +A batch {{transform}} uses a [composite aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-composite-aggregation.md) which allows efficient pagination through all buckets. Composite aggregations do not yet support a search context, therefore if the source data is changed (deleted, updated, added) while the batch {{dataframe}} is in progress, then the results may not include these changes. ### {{ctransform-cap}} consistency does not account for deleted or updated documents [transform-consistency-limitations] @@ -77,7 +77,7 @@ When deleting a {{transform}} using `DELETE _transform/index` neither the destin During the development of {{transforms}}, control was favoured over performance. In the design considerations, it is preferred for the {{transform}} to take longer to complete quietly in the background rather than to finish quickly and take precedence in resource consumption. -Composite aggregations are well suited for high cardinality data enabling pagination through results. If a [circuit breaker](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/circuit-breaker-settings.md) memory exception occurs when performing the composite aggregated search then we try again reducing the number of buckets requested. This circuit breaker is calculated based upon all activity within the cluster, not just activity from {{transforms}}, so it therefore may only be a temporary resource availability issue. +Composite aggregations are well suited for high cardinality data enabling pagination through results. If a [circuit breaker](elasticsearch://reference/elasticsearch/configuration-reference/circuit-breaker-settings.md) memory exception occurs when performing the composite aggregated search then we try again reducing the number of buckets requested. This circuit breaker is calculated based upon all activity within the cluster, not just activity from {{transforms}}, so it therefore may only be a temporary resource availability issue. For a batch {{transform}}, the number of buckets requested is only ever adjusted downwards. The lowering of value may result in a longer duration for the {{transform}} checkpoint to complete. For {{ctransforms}}, the number of buckets requested is reset back to its default at the start of every checkpoint and it is possible for circuit breaker exceptions to occur repeatedly in the {{es}} logs. @@ -85,11 +85,11 @@ The {{transform}} retrieves data in batches which means it calculates several bu ### Handling dynamic adjustments for many terms [transform-dynamic-adjustments-limitations] -For each checkpoint, entities are identified that have changed since the last time the check was performed. This list of changed entities is supplied as a [terms query](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-terms-query.md) to the {{transform}} composite aggregation, one page at a time. Then updates are applied to the destination index for each page of entities. +For each checkpoint, entities are identified that have changed since the last time the check was performed. This list of changed entities is supplied as a [terms query](elasticsearch://reference/query-languages/query-dsl-terms-query.md) to the {{transform}} composite aggregation, one page at a time. Then updates are applied to the destination index for each page of entities. The page `size` is defined by `max_page_search_size` which is also used to define the number of buckets returned by the composite aggregation search. The default value is 500, the minimum is 10. -The index setting [`index.max_terms_count`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#dynamic-index-settings) defines the maximum number of terms that can be used in a terms query. The default value is 65536. If `max_page_search_size` exceeds `index.max_terms_count` the {{transform}} will fail. +The index setting [`index.max_terms_count`](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#dynamic-index-settings) defines the maximum number of terms that can be used in a terms query. The default value is 65536. If `max_page_search_size` exceeds `index.max_terms_count` the {{transform}} will fail. Using smaller values for `max_page_search_size` may result in a longer duration for the {{transform}} checkpoint to complete. @@ -109,7 +109,7 @@ If using a `sync.time.field` that represents the data ingest time and using a ze ### Support for date nanoseconds data type [transform-date-nanos] -If your data uses the [date nanosecond data type](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/date_nanos.md), aggregations are nonetheless on millisecond resolution. This limitation also affects the aggregations in your {{transforms}}. +If your data uses the [date nanosecond data type](elasticsearch://reference/elasticsearch/mapping-reference/date_nanos.md), aggregations are nonetheless on millisecond resolution. This limitation also affects the aggregations in your {{transforms}}. ### Data streams as destination indices are not supported [transform-data-streams-destination] @@ -119,7 +119,7 @@ If your data uses the [date nanosecond data type](asciidocalypse://docs/elastics [ILM](../../manage-data/lifecycle/index-lifecycle-management.md) is not recommended to use as a {{transform}} destination index. {{transforms-cap}} update documents in the current destination, and cannot delete documents in the indices previously used by ILM. This may lead to duplicated documents when you use {{transforms}} combined with ILM in case of a rollover. -If you use ILM to have time-based indices, please consider using the [Date index name](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/date-index-name-processor.md) instead. The processor works without duplicated documents if your {{transform}} contains a `group_by` based on `date_histogram`. +If you use ILM to have time-based indices, please consider using the [Date index name](elasticsearch://reference/ingestion-tools/enrich-processor/date-index-name-processor.md) instead. The processor works without duplicated documents if your {{transform}} contains a `group_by` based on `date_histogram`. ## Limitations in {{kib}} [transform-ui-limitations] diff --git a/explore-analyze/transforms/transform-painless-examples.md b/explore-analyze/transforms/transform-painless-examples.md index acc5061f3..962bff1a5 100644 --- a/explore-analyze/transforms/transform-painless-examples.md +++ b/explore-analyze/transforms/transform-painless-examples.md @@ -9,11 +9,11 @@ mapped_pages: # Painless examples [transform-painless-examples] -::::{important} +::::{important} The examples that use the `scripted_metric` aggregation are not supported on {{es}} Serverless. :::: -These examples demonstrate how to use Painless in {{transforms}}. You can learn more about the Painless scripting language in the [Painless guide](asciidocalypse://docs/elasticsearch/docs/reference/scripting-languages/painless/painless.md). +These examples demonstrate how to use Painless in {{transforms}}. You can learn more about the Painless scripting language in the [Painless guide](elasticsearch://reference/scripting-languages/painless/painless.md). * [Getting top hits by using scripted metric aggregation](#painless-top-hits) * [Getting time features by using aggregations](#painless-time-features) @@ -31,7 +31,7 @@ These examples demonstrate how to use Painless in {{transforms}}. You can learn ## Getting top hits by using scripted metric aggregation [painless-top-hits] -This snippet shows how to find the latest document, in other words the document with the latest timestamp. From a technical perspective, it helps to achieve the function of a [Top hits](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-top-hits-aggregation.md) by using scripted metric aggregation in a {{transform}}, which provides a metric output. +This snippet shows how to find the latest document, in other words the document with the latest timestamp. From a technical perspective, it helps to achieve the function of a [Top hits](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-top-hits-aggregation.md) by using scripted metric aggregation in a {{transform}}, which provides a metric output. ::::{important} This example uses a `scripted_metric` aggregation which is not supported on {{es}} Serverless. @@ -66,7 +66,7 @@ This example uses a `scripted_metric` aggregation which is not supported on {{es 3. The `combine_script` returns `state` from each shard. 4. The `reduce_script` iterates through the value of `s.timestamp_latest` returned by each shard and returns the document with the latest timestamp (`last_doc`). In the response, the top hit (in other words, the `latest_doc`) is nested below the `latest_doc` field. -Check the [scope of scripts](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-scripted-metric-aggregation.md#scripted-metric-aggregation-scope) for detailed explanation on the respective scripts. +Check the [scope of scripts](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-scripted-metric-aggregation.md#scripted-metric-aggregation-scope) for detailed explanation on the respective scripts. You can retrieve the last value in a similar way: @@ -215,7 +215,7 @@ This snippet shows how to extract time based features by using Painless in a {{t ## Getting duration by using bucket script [painless-bucket-script] -This example shows you how to get the duration of a session by client IP from a data log by using [bucket script](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-bucket-script-aggregation.md). The example uses the {{kib}} sample web logs dataset. +This example shows you how to get the duration of a session by client IP from a data log by using [bucket script](elasticsearch://reference/data-analysis/aggregations/search-aggregations-pipeline-bucket-script-aggregation.md). The example uses the {{kib}} sample web logs dataset. ```console PUT _transform/data_log diff --git a/explore-analyze/transforms/transform-scale.md b/explore-analyze/transforms/transform-scale.md index 1c7525674..7b829b557 100644 --- a/explore-analyze/transforms/transform-scale.md +++ b/explore-analyze/transforms/transform-scale.md @@ -55,7 +55,7 @@ Imagine your {{ctransform}} is configured to group by `IP` and calculate the sum To limit which historical indices are accessed, exclude certain tiers (for example `"must_not": { "terms": { "_tier": [ "data_frozen", "data_cold" ] } }` and/or use an absolute time value as a date range filter in your source query (for example, greater than 2024-01-01T00:00:00). If you use a relative time value (for example, gte now-30d/d) then ensure date rounding is applied to take advantage of query caching and ensure that the relative time is much larger than the largest of `frequency` or `time.sync.delay` or the date histogram bucket, otherwise data may be missed. Do not use date filters which are less than a date value (for example, `lt`: less than or `lte`: less than or equal to) as this conflicts with the logic applied at each checkpoint execution and data may be missed. -Consider using [date math](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#api-date-math-index-names) in your index names to reduce the number of indices to resolve in your queries. Add a date pattern - for example, `yyyy-MM-dd` - to your index names and use it to limit your query to a specific date. The example below queries indices only from yesterday and today: +Consider using [date math](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#api-date-math-index-names) in your index names to reduce the number of indices to resolve in your queries. Add a date pattern - for example, `yyyy-MM-dd` - to your index names and use it to limit your query to a specific date. The example below queries indices only from yesterday and today: ```js "source": { @@ -88,10 +88,10 @@ Index sorting enables you to store documents on disk in a specific order which c ## 9. Disable the `_source` field on the destination index (storage) [disable-source-dest] -The [`_source` field](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md) contains the original JSON document body that was passed at index time. The `_source` field itself is not indexed (and thus is not searchable), but it is still stored in the index and incurs a storage overhead. Consider disabling `_source` to save storage space if you have a large destination index. Disabling `_source` is only possible during index creation. +The [`_source` field](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md) contains the original JSON document body that was passed at index time. The `_source` field itself is not indexed (and thus is not searchable), but it is still stored in the index and incurs a storage overhead. Consider disabling `_source` to save storage space if you have a large destination index. Disabling `_source` is only possible during index creation. ::::{note} -When the `_source` field is disabled, a number of features are not supported. Consult [Disabling the `_source` field](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md#disable-source-field) to understand the consequences before disabling it. +When the `_source` field is disabled, a number of features are not supported. Consult [Disabling the `_source` field](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md#disable-source-field) to understand the consequences before disabling it. :::: ## Further reading [_further_reading] diff --git a/explore-analyze/transforms/transform-usage.md b/explore-analyze/transforms/transform-usage.md index 470dea6ee..cb94d5dee 100644 --- a/explore-analyze/transforms/transform-usage.md +++ b/explore-analyze/transforms/transform-usage.md @@ -18,11 +18,11 @@ You might want to consider using {{transforms}} instead of aggregations when: In {{ml}}, you often need a complete set of behavioral features rather just the top-N. For example, if you are predicting customer churn, you might look at features such as the number of website visits in the last week, the total number of sales, or the number of emails sent. The {{stack}} {{ml-features}} create models based on this multi-dimensional feature space, so they benefit from the full feature indices that are created by {{transforms}}. - This scenario also applies when you are trying to search across the results of an aggregation or multiple aggregations. Aggregation results can be ordered or filtered, but there are [limitations to ordering](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-terms-aggregation.md#search-aggregations-bucket-terms-aggregation-order) and [filtering by bucket selector](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-pipeline-bucket-selector-aggregation.md) is constrained by the maximum number of buckets returned. If you want to search all aggregation results, you need to create the complete {{dataframe}}. If you need to sort or filter the aggregation results by multiple fields, {{transforms}} are particularly useful. + This scenario also applies when you are trying to search across the results of an aggregation or multiple aggregations. Aggregation results can be ordered or filtered, but there are [limitations to ordering](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-terms-aggregation.md#search-aggregations-bucket-terms-aggregation-order) and [filtering by bucket selector](elasticsearch://reference/data-analysis/aggregations/search-aggregations-pipeline-bucket-selector-aggregation.md) is constrained by the maximum number of buckets returned. If you want to search all aggregation results, you need to create the complete {{dataframe}}. If you need to sort or filter the aggregation results by multiple fields, {{transforms}} are particularly useful. * You need to sort aggregation results by a pipeline aggregation. - [Pipeline aggregations](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/pipeline.md) cannot be used for sorting. Technically, this is because pipeline aggregations are run during the reduce phase after all other aggregations have already completed. If you create a {{transform}}, you can effectively perform multiple passes over the data. + [Pipeline aggregations](elasticsearch://reference/data-analysis/aggregations/pipeline.md) cannot be used for sorting. Technically, this is because pipeline aggregations are run during the reduce phase after all other aggregations have already completed. If you create a {{transform}}, you can effectively perform multiple passes over the data. * You want to create summary tables to optimize queries. diff --git a/explore-analyze/visualize/canvas/canvas-function-reference.md b/explore-analyze/visualize/canvas/canvas-function-reference.md index 02a76444c..bc643ad7f 100644 --- a/explore-analyze/visualize/canvas/canvas-function-reference.md +++ b/explore-analyze/visualize/canvas/canvas-function-reference.md @@ -1302,7 +1302,7 @@ Interprets a `TinyMath` math expression using a `number` or `datatable` as *cont | Argument | Type | Description | | --- | --- | --- | -| *Unnamed*
Alias: `expression` | `string` | An evaluated `TinyMath` expression. See [canvas-tinymath-functions.md](asciidocalypse://docs/docs-content/docs/reference/data-analysis/kibana/tinymath-functions.md). | +| *Unnamed*
Alias: `expression` | `string` | An evaluated `TinyMath` expression. See [canvas-tinymath-functions.md](/reference/data-analysis/kibana/tinymath-functions.md). | | `onError` | `string` | In case the `TinyMath` evaluation fails or returns NaN, the return value is specified by onError. When `'throw'`, it will throw an exception, terminating expression execution (default). | **Returns:** Depends on your input and arguments @@ -1317,7 +1317,7 @@ Adds a column by evaluating `TinyMath` on each row. This function is optimized f | Argument | Type | Description | | --- | --- | --- | | *Unnamed* *****
Aliases: `column`, `name` | `string` | The name of the resulting column. Names are not required to be unique. | -| *Unnamed*
Alias: `expression` | `string` | An evaluated `TinyMath` expression. See [canvas-tinymath-functions.md](asciidocalypse://docs/docs-content/docs/reference/data-analysis/kibana/tinymath-functions.md). | +| *Unnamed*
Alias: `expression` | `string` | An evaluated `TinyMath` expression. See [canvas-tinymath-functions.md](/reference/data-analysis/kibana/tinymath-functions.md). | | `castColumns` † | `string` | The column ids that are cast to numbers before the formula is applied. | | `copyMetaFrom` | `string`, `null` | If set, the meta object from the specified column id is copied over to the specified target column. If the column doesn’t exist it silently fails.
Default: `null` | | `id` ***** | `string` | id of the resulting column. Must be unique. | @@ -1437,7 +1437,7 @@ Subdivides a `datatable` by the unique values of the specified columns, and pass ### `pointseries` [pointseries_fn] -Turn a `datatable` into a point series model. Currently we differentiate measure from dimensions by looking for a `TinyMath` expression. See [canvas-tinymath-functions.md](asciidocalypse://docs/docs-content/docs/reference/data-analysis/kibana/tinymath-functions.md). If you enter a `TinyMath` expression in your argument, we treat that argument as a measure, otherwise it is a dimension. Dimensions are combined to create unique keys. Measures are then deduplicated by those keys using the specified `TinyMath` function +Turn a `datatable` into a point series model. Currently we differentiate measure from dimensions by looking for a `TinyMath` expression. See [canvas-tinymath-functions.md](/reference/data-analysis/kibana/tinymath-functions.md). If you enter a `TinyMath` expression in your argument, we treat that argument as a measure, otherwise it is a dimension. Dimensions are combined to create unique keys. Measures are then deduplicated by those keys using the specified `TinyMath` function **Accepts:** `datatable` diff --git a/explore-analyze/visualize/custom-visualizations-with-vega.md b/explore-analyze/visualize/custom-visualizations-with-vega.md index 7c90ffa4c..55567608a 100644 --- a/explore-analyze/visualize/custom-visualizations-with-vega.md +++ b/explore-analyze/visualize/custom-visualizations-with-vega.md @@ -116,7 +116,7 @@ POST kibana_sample_data_ecommerce/_search } ``` -Add the [terms aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-terms-aggregation.md), then click **Click to send request**: +Add the [terms aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-terms-aggregation.md), then click **Click to send request**: ```js POST kibana_sample_data_ecommerce/_search diff --git a/explore-analyze/visualize/esorql.md b/explore-analyze/visualize/esorql.md index 940bded3d..f0801a41d 100644 --- a/explore-analyze/visualize/esorql.md +++ b/explore-analyze/visualize/esorql.md @@ -24,7 +24,7 @@ You can then **Save** and add it to an existing or a new dashboard using the sav 2. Choose **ES|QL** under **Visualizations**. An ES|QL editor appears and lets you configure your query and its associated visualization. The **Suggestions** panel can help you find alternative ways to configure the visualization. ::::{tip} - Check the [ES|QL reference](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql.md) to get familiar with the syntax and optimize your query. + Check the [ES|QL reference](elasticsearch://reference/query-languages/esql.md) to get familiar with the syntax and optimize your query. :::: 3. When editing your query or its configuration, run the query to update the preview of the visualization. diff --git a/explore-analyze/visualize/field-statistics.md b/explore-analyze/visualize/field-statistics.md index 3fd743e36..f90df3ab8 100644 --- a/explore-analyze/visualize/field-statistics.md +++ b/explore-analyze/visualize/field-statistics.md @@ -14,7 +14,7 @@ mapped_pages: 2. Choose **Field statistics** under **Visualizations**. An ES|QL editor appears and lets you configure your query with the fields and information that you want to show. ::::{tip} - Check the [ES|QL reference](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql.md) to get familiar with the syntax and optimize your query. + Check the [ES|QL reference](elasticsearch://reference/query-languages/esql.md) to get familiar with the syntax and optimize your query. :::: 3. When editing your query or its configuration, run the query to update the preview of the visualization. diff --git a/explore-analyze/visualize/graph/graph-troubleshooting.md b/explore-analyze/visualize/graph/graph-troubleshooting.md index c688fb391..e4340fe05 100644 --- a/explore-analyze/visualize/graph/graph-troubleshooting.md +++ b/explore-analyze/visualize/graph/graph-troubleshooting.md @@ -13,7 +13,7 @@ mapped_pages: -## Why are results missing? [_why_are_results_missing] +## Why are results missing? [_why_are_results_missing] The default settings in Graph API requests are configured to tune out noisy results by using the following strategies: @@ -28,20 +28,20 @@ These are useful defaults for getting the "big picture" signals from noisy data, * Set the `min_doc_count` for your vertices to 1 to ensure only one document is required to assert a relationship. -## What can I do to improve performance? [_what_can_i_do_to_improve_performance] +## What can I do to improve performance? [_what_can_i_do_to_improve_performance] With the default setting of `use_significance` set to `true`, the Graph API performs a background frequency check of the terms it discovers as part of exploration. Each unique term has to have its frequency looked up in the index, which costs at least one disk seek. Disk seeks are expensive. If you don’t need to perform this noise-filtering, setting `use_significance` to `false` eliminates all of these expensive checks (at the expense of not performing any quality-filtering on the terms). If your data is noisy and you need to filter based on significance, you can reduce the number of frequency checks by: * Reducing the `sample_size`. Considering fewer documents can actually be better when the quality of matches is quite variable. -* Avoiding noisy documents that have a large number of terms. You can do this by either allowing ranking to naturally favor shorter documents in the top-results sample (see [enabling norms](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/norms.md)) or by explicitly excluding large documents with your seed and guiding queries. +* Avoiding noisy documents that have a large number of terms. You can do this by either allowing ranking to naturally favor shorter documents in the top-results sample (see [enabling norms](elasticsearch://reference/elasticsearch/mapping-reference/norms.md)) or by explicitly excluding large documents with your seed and guiding queries. * Increasing the frequency threshold. Many many terms occur very infrequently so even increasing the frequency threshold by one can massively reduce the number of candidate terms whose background frequencies are checked. Keep in mind that all of these options reduce the scope of information analyzed and can increase the potential to miss what could be interesting details. However, the information that’s lost tends to be associated with lower-quality documents with lower-frequency terms, which can be an acceptable trade-off. -## Limited support for multiple indices [_limited_support_for_multiple_indices] +## Limited support for multiple indices [_limited_support_for_multiple_indices] The graph API can explore multiple indices, types, or aliases in a single API request, but the assumption is that each "hop" it performs is querying the same set of indices. Currently, it is not possible to take a term found in a field from one index and use that value to explore connections in *a different field* held in another type or index. diff --git a/explore-analyze/visualize/legacy-editors/tsvb.md b/explore-analyze/visualize/legacy-editors/tsvb.md index 3b6abbea8..39e046abe 100644 --- a/explore-analyze/visualize/legacy-editors/tsvb.md +++ b/explore-analyze/visualize/legacy-editors/tsvb.md @@ -3,7 +3,7 @@ applies_to: stack: ga serverless: ga --- - + # TSVB [tsvb-panel] **TSVB** is a set of visualization types that you configure and display on dashboards. @@ -37,7 +37,7 @@ When you use only {{data-sources}}, you are able to: ::::{important} :name: tsvb-index-patterns-mode -Creating **TSVB** visualizations with an {{es}} index string is deprecated and will be removed in a future release. By default, you create **TSVB** visualizations with only {{data-sources}}. To use an {{es}} index string, contact your administrator, or go to [Advanced Settings](asciidocalypse://docs/kibana/docs/reference/advanced-settings.md) and set `metrics:allowStringIndices` to `true`. +Creating **TSVB** visualizations with an {{es}} index string is deprecated and will be removed in a future release. By default, you create **TSVB** visualizations with only {{data-sources}}. To use an {{es}} index string, contact your administrator, or go to [Advanced Settings](kibana://reference/advanced-settings.md) and set `metrics:allowStringIndices` to `true`. :::: diff --git a/explore-analyze/visualize/link-panels.md b/explore-analyze/visualize/link-panels.md index deefd4db9..a0424a296 100644 --- a/explore-analyze/visualize/link-panels.md +++ b/explore-analyze/visualize/link-panels.md @@ -8,7 +8,7 @@ mapped_pages: # Link panels [dashboard-links] -You can use **Links** panels to create links to other dashboards or external websites. When creating links to other dashboards, you have the option to carry the time range, query, and filters to apply over to the linked dashboard. Links to external websites follow the [`externalUrl.policy`](asciidocalypse://docs/kibana/docs/reference/configuration-reference/url-drilldown-settings.md#external-URL-policy) settings. **Links** panels support vertical and horizontal layouts and may be saved to the **Library** for use in other dashboards. +You can use **Links** panels to create links to other dashboards or external websites. When creating links to other dashboards, you have the option to carry the time range, query, and filters to apply over to the linked dashboard. Links to external websites follow the [`externalUrl.policy`](kibana://reference/configuration-reference/url-drilldown-settings.md#external-url-policy) settings. **Links** panels support vertical and horizontal layouts and may be saved to the **Library** for use in other dashboards. :::{image} ../../images/kibana-dashboard_links_panel.png :alt: A screenshot displaying the new links panel diff --git a/explore-analyze/visualize/maps/asset-tracking-tutorial.md b/explore-analyze/visualize/maps/asset-tracking-tutorial.md index c2f6162b1..79efb3623 100644 --- a/explore-analyze/visualize/maps/asset-tracking-tutorial.md +++ b/explore-analyze/visualize/maps/asset-tracking-tutorial.md @@ -32,7 +32,7 @@ When you complete this tutorial, you’ll have a map that looks like this: * If you don’t already have {{kib}}, sign up for [a free Elastic Cloud trial](https://www.elastic.co/cloud/elasticsearch-service/signup?baymax=docs-body&elektra=docs) and create a hosted deployment. When creating it, download the deployment credentials. * Obtain an API key for [TriMet web services](https://developer.trimet.org/) at [https://developer.trimet.org/appid/registration/](https://developer.trimet.org/appid/registration/). -* [Fleet](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/index.md) is enabled on your cluster, and one or more [{{agent}}s](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/install-elastic-agents.md) is enrolled. +* [Fleet](/reference/ingestion-tools/fleet/index.md) is enabled on your cluster, and one or more [{{agent}}s](/reference/ingestion-tools/fleet/install-elastic-agents.md) is enrolled. ## Part 1: Ingest the Portland public transport data [_part_1_ingest_the_portland_public_transport_data] @@ -725,7 +725,7 @@ For this example, you will set the rule to check every minute. However, when run 16. Click **Save**. -The **TriMet Alerts connector** is added to the **{{connectors-ui}}** page. For more information on common connectors, refer to the [Slack](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/slack-action-type.md) and [Email](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/email-action-type.md) connectors. +The **TriMet Alerts connector** is added to the **{{connectors-ui}}** page. For more information on common connectors, refer to the [Slack](kibana://reference/connectors-kibana/slack-action-type.md) and [Email](kibana://reference/connectors-kibana/email-action-type.md) connectors. ### Step 3. View alerts in real time [_step_3_view_alerts_in_real_time] diff --git a/explore-analyze/visualize/maps/heatmap-layer.md b/explore-analyze/visualize/maps/heatmap-layer.md index 62f4ab3fc..405533e5a 100644 --- a/explore-analyze/visualize/maps/heatmap-layer.md +++ b/explore-analyze/visualize/maps/heatmap-layer.md @@ -15,7 +15,7 @@ Heat map layers cluster point data to show locations with higher densities. :class: screenshot ::: -To add a heat map layer to your map, click **Add layer**, then select **Heat map**. The index must contain at least one field mapped as [geo_point](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/geo-point.md) or [geo_shape](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/geo-shape.md). +To add a heat map layer to your map, click **Add layer**, then select **Heat map**. The index must contain at least one field mapped as [geo_point](elasticsearch://reference/elasticsearch/mapping-reference/geo-point.md) or [geo_shape](elasticsearch://reference/elasticsearch/mapping-reference/geo-shape.md). ::::{note} Only count, sum, unique count metric aggregations are available with the grid aggregation source and heat map layers. Average, min, and max are turned off because the heat map will blend nearby values. Blending two average values would make the cluster more prominent, even though it just might literally mean that these nearby areas are average. diff --git a/explore-analyze/visualize/maps/import-geospatial-data.md b/explore-analyze/visualize/maps/import-geospatial-data.md index a92370a38..84e119d6c 100644 --- a/explore-analyze/visualize/maps/import-geospatial-data.md +++ b/explore-analyze/visualize/maps/import-geospatial-data.md @@ -8,7 +8,7 @@ mapped_pages: # Import geospatial data [import-geospatial-data] -To import geospatical data into the Elastic Stack, the data must be indexed as [geo_point](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/geo-point.md) or [geo_shape](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/geo-shape.md). Geospatial data comes in many formats. Choose an import tool based on the format of your geospatial data. +To import geospatical data into the Elastic Stack, the data must be indexed as [geo_point](elasticsearch://reference/elasticsearch/mapping-reference/geo-point.md) or [geo_shape](elasticsearch://reference/elasticsearch/mapping-reference/geo-shape.md). Geospatial data comes in many formats. Choose an import tool based on the format of your geospatial data. ## Security privileges [import-geospatial-privileges] @@ -114,7 +114,7 @@ To draw features: ## Upload data with IP addresses [_upload_data_with_ip_addresses] -The GeoIP processor adds information about the geographical location of IP addresses. See [GeoIP processor](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/geoip-processor.md) for details. For private IP addresses, see [Enriching data with GeoIPs from internal, private IP addresses](https://www.elastic.co/blog/enriching-elasticsearch-data-geo-ips-internal-private-ip-addresses). +The GeoIP processor adds information about the geographical location of IP addresses. See [GeoIP processor](elasticsearch://reference/ingestion-tools/enrich-processor/geoip-processor.md) for details. For private IP addresses, see [Enriching data with GeoIPs from internal, private IP addresses](https://www.elastic.co/blog/enriching-elasticsearch-data-geo-ips-internal-private-ip-addresses). ## Upload data with GDAL [_upload_data_with_gdal] diff --git a/explore-analyze/visualize/maps/indexing-geojson-data-tutorial.md b/explore-analyze/visualize/maps/indexing-geojson-data-tutorial.md index 945a357ca..b7a7187da 100644 --- a/explore-analyze/visualize/maps/indexing-geojson-data-tutorial.md +++ b/explore-analyze/visualize/maps/indexing-geojson-data-tutorial.md @@ -50,7 +50,7 @@ For each GeoJSON file you downloaded, complete the following steps: 2. From the list of layer types, click **Upload file**. 3. Using the File Picker, upload the GeoJSON file. - Depending on the geometry type of your features, this will auto-populate **Index type** with either [geo_point](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/geo-point.md) or [geo_shape](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/geo-shape.md) and **Index name** with ``. + Depending on the geometry type of your features, this will auto-populate **Index type** with either [geo_point](elasticsearch://reference/elasticsearch/mapping-reference/geo-point.md) or [geo_shape](elasticsearch://reference/elasticsearch/mapping-reference/geo-shape.md) and **Index name** with ``. 4. Click **Import file**. @@ -72,12 +72,12 @@ For each GeoJSON file you downloaded, complete the following steps: ## Add a heatmap aggregation layer [_add_a_heatmap_aggregation_layer] -Looking at the `Lightning detected` layer, it’s clear where lightning has struck. What’s less clear, is if there have been more lightning strikes in some areas than others, in other words, where the lightning hot spots are. An advantage of having indexed [geo_point](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/geo-point.md) data for the lightning strikes is that you can perform aggregations on the data. +Looking at the `Lightning detected` layer, it’s clear where lightning has struck. What’s less clear, is if there have been more lightning strikes in some areas than others, in other words, where the lightning hot spots are. An advantage of having indexed [geo_point](elasticsearch://reference/elasticsearch/mapping-reference/geo-point.md) data for the lightning strikes is that you can perform aggregations on the data. 1. Click **Add layer**. 2. From the list of layer types, click **Heat map**. - Because you indexed `lightning_detected.geojson` using the index name and pattern `lightning_detected`, that data is available as a [geo_point](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/geo-point.md) aggregation. + Because you indexed `lightning_detected.geojson` using the index name and pattern `lightning_detected`, that data is available as a [geo_point](elasticsearch://reference/elasticsearch/mapping-reference/geo-point.md) aggregation. 3. Select `lightning_detected`. 4. Click **Add layer** to add the heat map layer "Lightning intensity". diff --git a/explore-analyze/visualize/maps/maps-create-filter-from-map.md b/explore-analyze/visualize/maps/maps-create-filter-from-map.md index 94ba53584..3b2ea3dc9 100644 --- a/explore-analyze/visualize/maps/maps-create-filter-from-map.md +++ b/explore-analyze/visualize/maps/maps-create-filter-from-map.md @@ -35,7 +35,7 @@ A spatial filter narrows search results to documents that either intersect with, Spatial filters have the following properties: * **Geometry label** enables you to provide a meaningful name for your spatial filter. -* **Spatial relation** determines the [spatial relation operator](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-geo-shape-query.md#geo-shape-spatial-relations) to use at search time. +* **Spatial relation** determines the [spatial relation operator](elasticsearch://reference/query-languages/query-dsl-geo-shape-query.md#geo-shape-spatial-relations) to use at search time. * **Action** specifies whether to apply the filter to the current view or to a drilldown action. ::::{note} diff --git a/explore-analyze/visualize/maps/maps-grid-aggregation.md b/explore-analyze/visualize/maps/maps-grid-aggregation.md index a414974b8..35683a847 100644 --- a/explore-analyze/visualize/maps/maps-grid-aggregation.md +++ b/explore-analyze/visualize/maps/maps-grid-aggregation.md @@ -8,21 +8,21 @@ mapped_pages: # Clusters [maps-grid-aggregation] -Clusters use [Geotile grid aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-geotilegrid-aggregation.md) or [Geohex grid aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-geohexgrid-aggregation.md) to group your documents into grids. You can calculate metrics for each gridded cell. +Clusters use [Geotile grid aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-geotilegrid-aggregation.md) or [Geohex grid aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-geohexgrid-aggregation.md) to group your documents into grids. You can calculate metrics for each gridded cell. Symbolize cluster metrics as: **Clusters** -: Uses [Geotile grid aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-geotilegrid-aggregation.md) to group your documents into grids. Creates a [vector layer](vector-layer.md) with a cluster symbol for each gridded cell. The cluster location is the weighted centroid for all documents in the gridded cell. +: Uses [Geotile grid aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-geotilegrid-aggregation.md) to group your documents into grids. Creates a [vector layer](vector-layer.md) with a cluster symbol for each gridded cell. The cluster location is the weighted centroid for all documents in the gridded cell. **Grids** -: Uses [Geotile grid aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-geotilegrid-aggregation.md) to group your documents into grids. Creates a [vector layer](vector-layer.md) with a bounding box polygon for each gridded cell. +: Uses [Geotile grid aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-geotilegrid-aggregation.md) to group your documents into grids. Creates a [vector layer](vector-layer.md) with a bounding box polygon for each gridded cell. **Heat map** -: Uses [Geotile grid aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-geotilegrid-aggregation.md) to group your documents into grids. Creates a [heat map layer](heatmap-layer.md) that clusters the weighted centroids for each gridded cell. +: Uses [Geotile grid aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-geotilegrid-aggregation.md) to group your documents into grids. Creates a [heat map layer](heatmap-layer.md) that clusters the weighted centroids for each gridded cell. **Hexbins** -: Uses [Geohex grid aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-geohexgrid-aggregation.md) to group your documents into H3 hexagon grids. Creates a [vector layer](vector-layer.md) with a hexagon polygon for each gridded cell. +: Uses [Geohex grid aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-geohexgrid-aggregation.md) to group your documents into H3 hexagon grids. Creates a [vector layer](vector-layer.md) with a hexagon polygon for each gridded cell. To enable a clusters layer: diff --git a/explore-analyze/visualize/maps/maps-search-across-multiple-indices.md b/explore-analyze/visualize/maps/maps-search-across-multiple-indices.md index 491c0798b..3fbf51134 100644 --- a/explore-analyze/visualize/maps/maps-search-across-multiple-indices.md +++ b/explore-analyze/visualize/maps/maps-search-across-multiple-indices.md @@ -20,7 +20,7 @@ One strategy for eliminating unintentional empty layers from a cross index searc ## Use _index in a search [maps-add-index-search] -Add [_index](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-index-field.md) to your search to include documents from indices that do not contain a search field. +Add [_index](elasticsearch://reference/elasticsearch/mapping-reference/mapping-index-field.md) to your search to include documents from indices that do not contain a search field. For example, suppose you have a vector layer showing the `kibana_sample_data_logs` documents and another vector layer with `kibana_sample_data_flights` documents. (See [adding sample data](/explore-analyze/index.md) to install the `kibana_sample_data_logs` and `kibana_sample_data_flights` indices.) diff --git a/explore-analyze/visualize/maps/maps-top-hits-aggregation.md b/explore-analyze/visualize/maps/maps-top-hits-aggregation.md index d727b035a..9abeaebe9 100644 --- a/explore-analyze/visualize/maps/maps-top-hits-aggregation.md +++ b/explore-analyze/visualize/maps/maps-top-hits-aggregation.md @@ -8,7 +8,7 @@ mapped_pages: # Display the most relevant documents per entity [maps-top-hits-aggregation] -Use **Top hits per entity** to display the most relevant documents per entity, for example, the most recent GPS tracks per flight route. To get this data, {{es}} first groups your data using a [terms aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-terms-aggregation.md), then accumulates the most relevant documents based on sort order for each entry using a [top hits metric aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-top-hits-aggregation.md). +Use **Top hits per entity** to display the most relevant documents per entity, for example, the most recent GPS tracks per flight route. To get this data, {{es}} first groups your data using a [terms aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-terms-aggregation.md), then accumulates the most relevant documents based on sort order for each entry using a [top hits metric aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-top-hits-aggregation.md). To enable top hits: diff --git a/explore-analyze/visualize/maps/maps-troubleshooting.md b/explore-analyze/visualize/maps/maps-troubleshooting.md index 3c439bc31..5711c7973 100644 --- a/explore-analyze/visualize/maps/maps-troubleshooting.md +++ b/explore-analyze/visualize/maps/maps-troubleshooting.md @@ -35,7 +35,7 @@ Maps uses the [{{es}} vector tile search API](https://www.elastic.co/docs/api/do ### Data view not listed when adding layer [_data_view_not_listed_when_adding_layer] -* Verify your geospatial data is correctly mapped as [geo_point](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/geo-point.md) or [geo_shape](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/geo-shape.md). +* Verify your geospatial data is correctly mapped as [geo_point](elasticsearch://reference/elasticsearch/mapping-reference/geo-point.md) or [geo_shape](elasticsearch://reference/elasticsearch/mapping-reference/geo-shape.md). * Run `GET myIndexName/_field_caps?fields=myGeoFieldName` in [Console](../../query-filter/tools/console.md), replacing `myIndexName` and `myGeoFieldName` with your index and geospatial field name. * Ensure response specifies `type` as `geo_point` or `geo_shape`. diff --git a/explore-analyze/visualize/maps/point-to-point.md b/explore-analyze/visualize/maps/point-to-point.md index 3b5dc1929..7806015b2 100644 --- a/explore-analyze/visualize/maps/point-to-point.md +++ b/explore-analyze/visualize/maps/point-to-point.md @@ -10,7 +10,7 @@ mapped_pages: A point-to-point connection plots aggregated data paths between the source and the destination. Thicker, darker lines symbolize more connections between a source and destination, and thinner, lighter lines symbolize less connections. -Point to point uses an {{es}} [terms aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-terms-aggregation.md) to group your documents by destination. Then, a nested [GeoTile grid aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-geotilegrid-aggregation.md) groups sources for each destination into grids. A line connects each source grid centroid to each destination. +Point to point uses an {{es}} [terms aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-terms-aggregation.md) to group your documents by destination. Then, a nested [GeoTile grid aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-geotilegrid-aggregation.md) groups sources for each destination into grids. A line connects each source grid centroid to each destination. Point-to-point layers are used in several common use cases: diff --git a/explore-analyze/visualize/maps/reverse-geocoding-tutorial.md b/explore-analyze/visualize/maps/reverse-geocoding-tutorial.md index ab54eae6a..71ad146f2 100644 --- a/explore-analyze/visualize/maps/reverse-geocoding-tutorial.md +++ b/explore-analyze/visualize/maps/reverse-geocoding-tutorial.md @@ -17,7 +17,7 @@ In this tutorial, you’ll use reverse geocoding to visualize United States Cens You’ll learn to: * Upload custom regions. -* Reverse geocode with the {{es}} [enrich processor](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/enrich-processor.md). +* Reverse geocode with the {{es}} [enrich processor](elasticsearch://reference/ingestion-tools/enrich-processor/enrich-processor.md). * Create a map and visualize CSA regions by web traffic. When you complete this tutorial, you’ll have a map that looks like this: @@ -32,7 +32,7 @@ When you complete this tutorial, you’ll have a map that looks like this: GeoIP is a common way of transforming an IP address to a longitude and latitude. GeoIP is roughly accurate on the city level globally and neighborhood level in selected countries. It’s not as good as an actual GPS location from your phone, but it’s much more precise than just a country, state, or province. -You’ll use the [web logs sample data set](../../index.md#gs-get-data-into-kibana) that comes with Kibana for this tutorial. Web logs sample data set has longitude and latitude. If your web log data does not contain longitude and latitude, use [GeoIP processor](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/geoip-processor.md) to transform an IP address into a [geo_point](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/geo-point.md) field. +You’ll use the [web logs sample data set](../../index.md#gs-get-data-into-kibana) that comes with Kibana for this tutorial. Web logs sample data set has longitude and latitude. If your web log data does not contain longitude and latitude, use [GeoIP processor](elasticsearch://reference/ingestion-tools/enrich-processor/geoip-processor.md) to transform an IP address into a [geo_point](elasticsearch://reference/elasticsearch/mapping-reference/geo-point.md) field. ## Step 2: Index Combined Statistical Area (CSA) regions [_step_2_index_combined_statistical_area_csa_regions] @@ -75,7 +75,7 @@ Looking at the map, you get a sense of what constitutes a metro area in the eyes ## Step 3: Reverse geocoding [_step_3_reverse_geocoding] -To visualize CSA regions by web log traffic, the web log traffic must contain a CSA region identifier. You’ll use {{es}} [enrich processor](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/enrich-processor.md) to add CSA region identifiers to the web logs sample data set. You can skip this step if your source data already contains region identifiers. +To visualize CSA regions by web log traffic, the web log traffic must contain a CSA region identifier. You’ll use {{es}} [enrich processor](elasticsearch://reference/ingestion-tools/enrich-processor/enrich-processor.md) to add CSA region identifiers to the web logs sample data set. You can skip this step if your source data already contains region identifiers. 1. Go to **Developer tools** using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). 2. In **Console**, create a [geo_match enrichment policy](../../../manage-data/ingest/transform-enrich/example-enrich-data-based-on-geolocation.md): diff --git a/explore-analyze/visualize/maps/terms-join.md b/explore-analyze/visualize/maps/terms-join.md index 717a8daca..77fea4082 100644 --- a/explore-analyze/visualize/maps/terms-join.md +++ b/explore-analyze/visualize/maps/terms-join.md @@ -62,7 +62,7 @@ In the following example, **iso2** property defines the shared key for the left The right source uses the Kibana sample data set "Sample web logs". In this data set, the **geo.src** field contains the ISO 3166-1 alpha-2 code of the country of origin. -A [terms aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-terms-aggregation.md) groups the sample web log documents by **geo.src** and calculates metrics for each term. +A [terms aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-terms-aggregation.md) groups the sample web log documents by **geo.src** and calculates metrics for each term. The METRICS configuration defines two metric aggregations: diff --git a/explore-analyze/visualize/maps/vector-layer.md b/explore-analyze/visualize/maps/vector-layer.md index acbc03449..bc3b94eeb 100644 --- a/explore-analyze/visualize/maps/vector-layer.md +++ b/explore-analyze/visualize/maps/vector-layer.md @@ -21,18 +21,18 @@ To add a vector layer to your map, click **Add layer**, then select one of the f : Shaded areas to compare statistics across boundaries. **Clusters** -: Geospatial data grouped in grids with metrics for each gridded cell. The index must contain at least one field mapped as [geo_point](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/geo-point.md) or [geo_shape](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/geo-shape.md). +: Geospatial data grouped in grids with metrics for each gridded cell. The index must contain at least one field mapped as [geo_point](elasticsearch://reference/elasticsearch/mapping-reference/geo-point.md) or [geo_shape](elasticsearch://reference/elasticsearch/mapping-reference/geo-shape.md). **Create index** : Draw shapes on the map and index in Elasticsearch. **Documents** -: Points, lines, and polyons from Elasticsearch. The index must contain at least one field mapped as [geo_point](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/geo-point.md) or [geo_shape](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/geo-shape.md). +: Points, lines, and polyons from Elasticsearch. The index must contain at least one field mapped as [geo_point](elasticsearch://reference/elasticsearch/mapping-reference/geo-point.md) or [geo_shape](elasticsearch://reference/elasticsearch/mapping-reference/geo-shape.md). Results are limited to the `index.max_result_window` index setting, which defaults to 10000. Select the appropriate **Scaling** option for your use case. * **Limit results to 10,000** The layer displays features from the first `index.max_result_window` documents. Results exceeding `index.max_result_window` are not displayed. - * **Show clusters when results exceed 10,000** When results exceed `index.max_result_window`, the layer uses [GeoTile grid aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-geotilegrid-aggregation.md) to group your documents into clusters and displays metrics for each cluster. When results are less then `index.max_result_window`, the layer displays features from individual documents. + * **Show clusters when results exceed 10,000** When results exceed `index.max_result_window`, the layer uses [GeoTile grid aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-geotilegrid-aggregation.md) to group your documents into clusters and displays metrics for each cluster. When results are less then `index.max_result_window`, the layer displays features from individual documents. * **Use vector tiles.** Vector tiles partition your map into tiles. Each tile request is limited to the `index.max_result_window` index setting. When a tile exceeds `index.max_result_window`, results exceeding `index.max_result_window` are not contained in the tile and a dashed rectangle outlining the bounding box containing all geo values within the tile is displayed. @@ -43,13 +43,13 @@ To add a vector layer to your map, click **Add layer**, then select one of the f : Points and lines associated with anomalies. The {{anomaly-job}} must use a `lat_long` function. Go to [Detecting anomalous locations in geographic data](../../machine-learning/anomaly-detection/geographic-anomalies.md) for an example. **Point to point** -: Aggregated data paths between the source and destination. The index must contain at least 2 fields mapped as [geo_point](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/geo-point.md), source and destination. +: Aggregated data paths between the source and destination. The index must contain at least 2 fields mapped as [geo_point](elasticsearch://reference/elasticsearch/mapping-reference/geo-point.md), source and destination. **Top hits per entity** -: The layer displays the [most relevant documents per entity](maps-top-hits-aggregation.md). The index must contain at least one field mapped as [geo_point](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/geo-point.md) or [geo_shape](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/geo-shape.md). +: The layer displays the [most relevant documents per entity](maps-top-hits-aggregation.md). The index must contain at least one field mapped as [geo_point](elasticsearch://reference/elasticsearch/mapping-reference/geo-point.md) or [geo_shape](elasticsearch://reference/elasticsearch/mapping-reference/geo-shape.md). **Tracks** -: Create lines from points. The index must contain at least one field mapped as [geo_point](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/geo-point.md). +: Create lines from points. The index must contain at least one field mapped as [geo_point](elasticsearch://reference/elasticsearch/mapping-reference/geo-point.md). **Upload Geojson** : Index GeoJSON data in Elasticsearch. diff --git a/explore-analyze/visualize/supported-chart-types.md b/explore-analyze/visualize/supported-chart-types.md index c8f5a28f0..2b76bc5e0 100644 --- a/explore-analyze/visualize/supported-chart-types.md +++ b/explore-analyze/visualize/supported-chart-types.md @@ -25,7 +25,7 @@ $$$aggregation-reference$$$ | Tag cloud | ✓ | | ✓ | ✓ | | -## Bar, line, and area chart features [xy-features] +## Bar, line, and area chart features [xy-features] | Feature | **Lens** | **TSVB** | **Aggregation-based** | **Vega** | **Timelion** | | --- | --- | --- | --- | --- | --- | @@ -37,7 +37,7 @@ $$$aggregation-reference$$$ | Synchronized tooltips | ✓ | ✓ | | | | -## Advanced features [other-features] +## Advanced features [other-features] | Feature | **Lens** | **TSVB** | **Vega** | **Timelion** | | --- | --- | --- | --- | --- | @@ -51,7 +51,7 @@ $$$aggregation-reference$$$ | Annotations | ✓ | ✓ | | | -## Table features [table-features] +## Table features [table-features] | Feature | **Lens** | **TSVB** | **Aggregation-based** | | --- | --- | --- | --- | @@ -61,7 +61,7 @@ $$$aggregation-reference$$$ | Color by value | ✓ | ✓ | | -## Functions [custom-functions] +## Functions [custom-functions] | Function | **Lens** | **TSVB** | | --- | --- | --- | @@ -72,7 +72,7 @@ $$$aggregation-reference$$$ | Static value | ✓ | ✓ | -## Metrics aggregations [metrics-aggregations] +## Metrics aggregations [metrics-aggregations] Metric aggregations are calculated from the values in the aggregated documents. The values are extracted from the document fields. @@ -89,10 +89,10 @@ Metric aggregations are calculated from the values in the aggregated documents. | Value count | ✓ | | ✓ | ✓ | | Variance | ✓ | ✓ | | ✓ | -For information about {{es}} metrics aggregations, refer to [Metrics aggregations](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/metrics.md). +For information about {{es}} metrics aggregations, refer to [Metrics aggregations](elasticsearch://reference/data-analysis/aggregations/metrics.md). -## Bucket aggregations [bucket-aggregations] +## Bucket aggregations [bucket-aggregations] Bucket aggregations group, or bucket, documents based on the aggregation type. To define the document buckets, bucket aggregations compute and return the number of documents for each bucket. @@ -110,10 +110,10 @@ Bucket aggregations group, or bucket, documents based on the aggregation type. T | Terms | ✓ | ✓ | ✓ | ✓ | | Significant terms | ✓ | | ✓ | ✓ | -For information about {{es}} bucket aggregations, refer to [Bucket aggregations](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/bucket.md). +For information about {{es}} bucket aggregations, refer to [Bucket aggregations](elasticsearch://reference/data-analysis/aggregations/bucket.md). -## Pipeline aggregations [pipeline-aggregations] +## Pipeline aggregations [pipeline-aggregations] Pipeline aggregations are dependent on the outputs calculated from other aggregations. Parent pipeline aggregations are provided with the output of the parent aggregation, and compute new buckets or aggregations that are added to existing buckets. Sibling pipeline aggregations are provided with the output of a sibling aggregation, and compute new aggregations for the same level as the sibling aggregation. @@ -130,5 +130,5 @@ Pipeline aggregations are dependent on the outputs calculated from other aggrega | Bucket selector | | | | ✓ | | Serial differencing | | ✓ | ✓ | ✓ | -For information about {{es}} pipeline aggregations, refer to [Pipeline aggregations](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/pipeline.md). +For information about {{es}} pipeline aggregations, refer to [Pipeline aggregations](elasticsearch://reference/data-analysis/aggregations/pipeline.md). diff --git a/extend/index.md b/extend/index.md index 9a118b7f3..c7baeaae5 100644 --- a/extend/index.md +++ b/extend/index.md @@ -6,15 +6,15 @@ This section contains information on how to extend or contribute to our various You can contribute to various projects, including: -- [Kibana](asciidocalypse://docs/extend/index.md): Enhance our data visualization platform by contributing to Kibana. -- [Logstash](asciidocalypse://docs/extend/index.md): Help us improve the data processing pipeline with your contributions to Logstash. -- [Beats](asciidocalypse://docs/extend/index.md): Add new features or beats to our lightweight data shippers. +- [Kibana](kibana://extend/index.md): Enhance our data visualization platform by contributing to Kibana. +- [Logstash](logstash://extend/index.md): Help us improve the data processing pipeline with your contributions to Logstash. +- [Beats](asciidocalypse://docs/beats/docs/extend/index.md): Add new features or beats to our lightweight data shippers. ## Creating Integrations -Extend the capabilities of Elastic by creating integrations that connect Elastic products with other tools and systems. Visit our [Integrations Guide](asciidocalypse://docs/extend/index.md) to get started. +Extend the capabilities of Elastic by creating integrations that connect Elastic products with other tools and systems. Visit our [Integrations Guide](integrations://extend/index.md) to get started. ## Elasticsearch Plugins -Develop custom plugins to add new functionalities to Elasticsearch. Check out our [Elasticsearch Plugins Development Guide](asciidocalypse://docs/extend/index.md) for detailed instructions and best practices. +Develop custom plugins to add new functionalities to Elasticsearch. Check out our [Elasticsearch Plugins Development Guide](elasticsearch://extend/index.md) for detailed instructions and best practices. diff --git a/get-started/the-stack.md b/get-started/the-stack.md index 12d944d97..9116a666e 100644 --- a/get-started/the-stack.md +++ b/get-started/the-stack.md @@ -35,7 +35,7 @@ $$$stack-components-agent$$$ {{fleet}} enables you to centrally manage {{agents}} and their policies. Use {{fleet}} to monitor the state of all your {{agents}}, manage agent policies, and upgrade {{agent}} binaries or integrations. - [Learn more about {{fleet}} and {{agent}}](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/index.md). + [Learn more about {{fleet}} and {{agent}}](/reference/ingestion-tools/fleet/index.md). $$$stack-components-apm$$$ @@ -56,7 +56,7 @@ $$$stack-components-ingest-pipelines$$$ $$$stack-components-logstash$$$ {{ls}} -: {{ls}} is a data collection engine with real-time pipelining capabilities. It can dynamically unify data from disparate sources and normalize the data into destinations of your choice. {{ls}} supports a broad array of input, filter, and output plugins, with many native codecs further simplifying the ingestion process. [Learn more about {{ls}}](asciidocalypse://docs/logstash/docs/reference/index.md). +: {{ls}} is a data collection engine with real-time pipelining capabilities. It can dynamically unify data from disparate sources and normalize the data into destinations of your choice. {{ls}} supports a broad array of input, filter, and output plugins, with many native codecs further simplifying the ingestion process. [Learn more about {{ls}}](logstash://reference/index.md). ### Store [_store] diff --git a/manage-data/data-store/aliases.md b/manage-data/data-store/aliases.md index 49a8962fb..043961a36 100644 --- a/manage-data/data-store/aliases.md +++ b/manage-data/data-store/aliases.md @@ -316,7 +316,7 @@ Filters are only applied when using the [Query DSL](../../explore-analyze/query- ## Routing [alias-routing] -Use the `routing` option to [route](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-routing-field.md) requests for an alias to a specific shard. This lets you take advantage of [shard caches](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/shard-request-cache-settings.md) to speed up searches. Data stream aliases do not support routing options. +Use the `routing` option to [route](elasticsearch://reference/elasticsearch/mapping-reference/mapping-routing-field.md) requests for an alias to a specific shard. This lets you take advantage of [shard caches](elasticsearch://reference/elasticsearch/configuration-reference/shard-request-cache-settings.md) to speed up searches. Data stream aliases do not support routing options. ```console POST _aliases diff --git a/manage-data/data-store/data-streams.md b/manage-data/data-store/data-streams.md index eb0965268..f551f9d29 100644 --- a/manage-data/data-store/data-streams.md +++ b/manage-data/data-store/data-streams.md @@ -30,7 +30,7 @@ Keep in mind that some features such as [Time Series Data Streams (TSDS)](../dat ## Backing indices [backing-indices] -A data stream consists of one or more [hidden](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-hidden), auto-generated backing indices. +A data stream consists of one or more [hidden](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#index-hidden), auto-generated backing indices. :::{image} ../../images/elasticsearch-reference-data-streams-diagram.svg :alt: data streams diagram @@ -38,7 +38,7 @@ A data stream consists of one or more [hidden](asciidocalypse://docs/elasticsear A data stream requires a matching [index template](templates.md). The template contains the mappings and settings used to configure the stream’s backing indices. -Every document indexed to a data stream must contain a `@timestamp` field, mapped as a [`date`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/date.md) or [`date_nanos`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/date_nanos.md) field type. If the index template doesn’t specify a mapping for the `@timestamp` field, {{es}} maps `@timestamp` as a `date` field with default options. +Every document indexed to a data stream must contain a `@timestamp` field, mapped as a [`date`](elasticsearch://reference/elasticsearch/mapping-reference/date.md) or [`date_nanos`](elasticsearch://reference/elasticsearch/mapping-reference/date_nanos.md) field type. If the index template doesn’t specify a mapping for the `@timestamp` field, {{es}} maps `@timestamp` as a `date` field with default options. The same index template can be used for multiple data streams. You cannot delete an index template in use by a data stream. diff --git a/manage-data/data-store/data-streams/downsampling-time-series-data-stream.md b/manage-data/data-store/data-streams/downsampling-time-series-data-stream.md index 81b6763a2..8e1608540 100644 --- a/manage-data/data-store/data-streams/downsampling-time-series-data-stream.md +++ b/manage-data/data-store/data-streams/downsampling-time-series-data-stream.md @@ -86,7 +86,7 @@ POST /my-time-series-index/_downsample/my-downsampled-time-series-index } ``` -To downsample time series data as part of ILM, include a [Downsample action](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-downsample.md) in your ILM policy and set `fixed_interval` to the level of granularity that you’d like: +To downsample time series data as part of ILM, include a [Downsample action](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-downsample.md) in your ILM policy and set `fixed_interval` to the level of granularity that you’d like: ```console PUT _ilm/policy/my_policy @@ -118,12 +118,12 @@ The result of a time based histogram aggregation is in a uniform bucket size and There are a few things to note about querying downsampled indices: * When you run queries in {{kib}} and through Elastic solutions, a normal response is returned without notification that some of the queried indices are downsampled. -* For [date histogram aggregations](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-datehistogram-aggregation.md), only `fixed_intervals` (and not calendar-aware intervals) are supported. +* For [date histogram aggregations](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-datehistogram-aggregation.md), only `fixed_intervals` (and not calendar-aware intervals) are supported. * Timezone support comes with caveats: * Date histograms at intervals that are multiples of an hour are based on values generated at UTC. This works well for timezones that are on the hour, e.g. +5:00 or -3:00, but requires offsetting the reported time buckets, e.g. `2020-01-01T10:30:00.000` instead of `2020-03-07T10:00:00.000` for timezone +5:30 (India), if downsampling aggregates values per hour. In this case, the results include the field `downsampled_results_offset: true`, to indicate that the time buckets are shifted. This can be avoided if a downsampling interval of 15 minutes is used, as it allows properly calculating hourly values for the shifted buckets. * Date histograms at intervals that are multiples of a day are similarly affected, in case downsampling aggregates values per day. In this case, the beginning of each day is always calculated at UTC when generated the downsampled values, so the time buckets need to be shifted, e.g. reported as `2020-03-07T19:00:00.000` instead of `2020-03-07T00:00:00.000` for timezone `America/New_York`. The field `downsampled_results_offset: true` is added in this case too. - * Daylight savings and similar peculiarities around timezones affect reported results, as [documented](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-datehistogram-aggregation.md#datehistogram-aggregation-time-zone) for date histogram aggregation. Besides, downsampling at daily interval hinders tracking any information related to daylight savings changes. + * Daylight savings and similar peculiarities around timezones affect reported results, as [documented](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-datehistogram-aggregation.md#datehistogram-aggregation-time-zone) for date histogram aggregation. Besides, downsampling at daily interval hinders tracking any information related to daylight savings changes. @@ -136,9 +136,9 @@ The following restrictions and limitations apply for downsampling: * Within a data stream, a downsampled index replaces the original index and the original index is deleted. Only one index can exist for a given time period. * A source index must be in read-only mode for the downsampling process to succeed. Check the [Run downsampling manually](./run-downsampling-manually.md) example for details. * Downsampling data for the same period many times (downsampling of a downsampled index) is supported. The downsampling interval must be a multiple of the interval of the downsampled index. -* Downsampling is provided as an ILM action. See [Downsample](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-downsample.md). +* Downsampling is provided as an ILM action. See [Downsample](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-downsample.md). * The new, downsampled index is created on the data tier of the original index and it inherits its settings (for example, the number of shards and replicas). -* The numeric `gauge` and `counter` [metric types](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-field-meta.md) are supported. +* The numeric `gauge` and `counter` [metric types](elasticsearch://reference/elasticsearch/mapping-reference/mapping-field-meta.md) are supported. * The downsampling configuration is extracted from the time series data stream [index mapping](./set-up-tsds.md#create-tsds-index-template). The only additional required setting is the downsampling `fixed_interval`. diff --git a/manage-data/data-store/data-streams/logs-data-stream.md b/manage-data/data-store/data-streams/logs-data-stream.md index 8de970a00..cccf0513f 100644 --- a/manage-data/data-store/data-streams/logs-data-stream.md +++ b/manage-data/data-store/data-streams/logs-data-stream.md @@ -47,13 +47,13 @@ You can also set the index mode and adjust other template settings in [the Elast ## Synthetic source [logsdb-synthetic-source] -If you have the required [subscription](https://www.elastic.co/subscriptions), `logsdb` index mode uses [synthetic `_source`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source), which omits storing the original `_source` field. Instead, the document source is synthesized from doc values or stored fields upon document retrieval. +If you have the required [subscription](https://www.elastic.co/subscriptions), `logsdb` index mode uses [synthetic `_source`](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source), which omits storing the original `_source` field. Instead, the document source is synthesized from doc values or stored fields upon document retrieval. If you don’t have the required [subscription](https://www.elastic.co/subscriptions), `logsdb` mode uses the original `_source` field. -Before using synthetic source, make sure to review the [restrictions](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source-restrictions). +Before using synthetic source, make sure to review the [restrictions](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source-restrictions). -When working with multi-value fields, the `index.mapping.synthetic_source_keep` setting controls how field values are preserved for [synthetic source](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source) reconstruction. In `logsdb`, the default value is `arrays`, which retains both duplicate values and the order of entries. However, the exact structure of array elements and objects is not necessarily retained. Preserving duplicates and ordering can be critical for some log fields, such as DNS A records, HTTP headers, and log entries that represent sequential or repeated events. +When working with multi-value fields, the `index.mapping.synthetic_source_keep` setting controls how field values are preserved for [synthetic source](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source) reconstruction. In `logsdb`, the default value is `arrays`, which retains both duplicate values and the order of entries. However, the exact structure of array elements and objects is not necessarily retained. Preserving duplicates and ordering can be critical for some log fields, such as DNS A records, HTTP headers, and log entries that represent sequential or repeated events. ## Index sort settings [logsdb-sort-settings] @@ -68,7 +68,7 @@ In `logsdb` index mode, indices are sorted by the fields `host.name` and `@times * To prioritize the latest data, `host.name` is sorted in ascending order and `@timestamp` is sorted in descending order. -You can override the default sort settings by manually configuring `index.sort.field` and `index.sort.order`. For more details, see [*Index Sorting*](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/sorting.md). +You can override the default sort settings by manually configuring `index.sort.field` and `index.sort.order`. For more details, see [*Index Sorting*](elasticsearch://reference/elasticsearch/index-settings/sorting.md). To modify the sort configuration of an existing data stream, update the data stream’s component templates, and then perform or wait for a [rollover](../data-streams.md#data-streams-rollover). @@ -86,7 +86,7 @@ To avoid mapping conflicts, consider these options: * **Adjust mappings:** Check your existing mappings to ensure that `host.name` is mapped as a keyword. * **Change sorting:** If needed, you can remove `host.name` from the sort settings and use a different set of fields. Sorting by `@timestamp` can be a good fallback. -* **Switch to a different [index mode](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting)**: If resolving `host.name` mapping conflicts is not feasible, you can choose not to use `logsdb` mode. +* **Switch to a different [index mode](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#index-mode-setting)**: If resolving `host.name` mapping conflicts is not feasible, you can choose not to use `logsdb` mode. ::::{important} On existing data streams, `logsdb` mode is applied on [rollover](../data-streams.md#data-streams-rollover) (automatic or manual). @@ -103,15 +103,15 @@ In benchmarks, routing optimizations reduced storage requirements by 20% compare To configure a routing optimization: * Include the index setting `[index.logsdb.route_on_sort_fields:true]` in the data stream configuration. -* [Configure index sorting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/sorting.md) with two or more fields, in addition to `@timestamp`. -* Make sure the [`_id`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-id-field.md) field is not populated in ingested documents. It should be auto-generated instead. +* [Configure index sorting](elasticsearch://reference/elasticsearch/index-settings/sorting.md) with two or more fields, in addition to `@timestamp`. +* Make sure the [`_id`](elasticsearch://reference/elasticsearch/mapping-reference/mapping-id-field.md) field is not populated in ingested documents. It should be auto-generated instead. A custom sort configuration is required, to improve storage efficiency and to minimize hotspots from logging spikes that may route documents to a single shard. For best results, use a few sort fields that have a relatively low cardinality and don’t co-vary (for example, `host.name` and `host.id` are not optimal). ## Specialized codecs [logsdb-specialized-codecs] -By default, `logsdb` index mode uses the `best_compression` [codec](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-codec), which applies [ZSTD](https://en.wikipedia.org/wiki/Zstd) compression to stored fields. You can switch to the `default` codec for faster compression with a slightly larger storage footprint. +By default, `logsdb` index mode uses the `best_compression` [codec](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#index-codec), which applies [ZSTD](https://en.wikipedia.org/wiki/Zstd) compression to stored fields. You can switch to the `default` codec for faster compression with a slightly larger storage footprint. The `logsdb` index mode also automatically applies specialized codecs for numeric doc values, in order to optimize storage usage. Numeric fields are encoded using the following sequence of codecs: @@ -158,7 +158,7 @@ When automatically injected, `host.name` and `@timestamp` count toward the limit ## Fields without `doc_values` [logsdb-nodocvalue-fields] -When the `logsdb` index mode uses synthetic `_source` and `doc_values` are disabled for a field in the mapping, {{es}} might set the `store` setting to `true` for that field. This ensures that the field’s data remains accessible for reconstructing the document’s source when using [synthetic source](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source). +When the `logsdb` index mode uses synthetic `_source` and `doc_values` are disabled for a field in the mapping, {{es}} might set the `store` setting to `true` for that field. This ensures that the field’s data remains accessible for reconstructing the document’s source when using [synthetic source](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source). For example, this adjustment occurs with text fields when `store` is `false` and no suitable multi-field is available for reconstructing the original value. diff --git a/manage-data/data-store/data-streams/modify-data-stream.md b/manage-data/data-store/data-streams/modify-data-stream.md index 5a565e6cd..82d70cac5 100644 --- a/manage-data/data-store/data-streams/modify-data-stream.md +++ b/manage-data/data-store/data-streams/modify-data-stream.md @@ -23,7 +23,7 @@ If you later need to change the mappings or settings for a data stream, you have * [Change a static index setting for a data stream](../data-streams/modify-data-stream.md#change-static-index-setting-for-a-data-stream) ::::{tip} -If your changes include modifications to existing field mappings or [static index settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md), a reindex is often required to apply the changes to a data stream’s backing indices. If you are already performing a reindex, you can use the same process to add new field mappings and change [dynamic index settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md). See [Use reindex to change mappings or settings](../data-streams/modify-data-stream.md#data-streams-use-reindex-to-change-mappings-settings). +If your changes include modifications to existing field mappings or [static index settings](elasticsearch://reference/elasticsearch/index-settings/index.md), a reindex is often required to apply the changes to a data stream’s backing indices. If you are already performing a reindex, you can use the same process to add new field mappings and change [dynamic index settings](elasticsearch://reference/elasticsearch/index-settings/index.md). See [Use reindex to change mappings or settings](../data-streams/modify-data-stream.md#data-streams-use-reindex-to-change-mappings-settings). :::: @@ -92,13 +92,13 @@ To add a mapping for a new field to a data stream, following these steps: ### Change an existing field mapping in a data stream [change-existing-field-mapping-in-a-data-stream] -The documentation for each [mapping parameter](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-parameters.md) indicates whether you can update it for an existing field using the [update mapping API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-mapping). To update these parameters for an existing field, follow these steps: +The documentation for each [mapping parameter](elasticsearch://reference/elasticsearch/mapping-reference/mapping-parameters.md) indicates whether you can update it for an existing field using the [update mapping API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-mapping). To update these parameters for an existing field, follow these steps: 1. Update the index template used by the data stream. This ensures the updated field mapping is added to future backing indices created for the stream. For example, `my-data-stream-template` is an existing index template used by `my-data-stream`. - The following [create or update index template](../templates.md) request changes the argument for the `host.ip` field’s [`ignore_malformed`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/ignore-malformed.md) mapping parameter to `true`. + The following [create or update index template](../templates.md) request changes the argument for the `host.ip` field’s [`ignore_malformed`](elasticsearch://reference/elasticsearch/mapping-reference/ignore-malformed.md) mapping parameter to `true`. ```console PUT /_index_template/my-data-stream-template @@ -173,7 +173,7 @@ If you need to change the mapping of an existing field, create a new data stream ### Change a dynamic index setting for a data stream [change-dynamic-index-setting-for-a-data-stream] -To change a [dynamic index setting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md) for a data stream, follow these steps: +To change a [dynamic index setting](elasticsearch://reference/elasticsearch/index-settings/index.md) for a data stream, follow these steps: 1. Update the index template used by the data stream. This ensures the setting is applied to future backing indices created for the stream. @@ -219,7 +219,7 @@ To change the `index.lifecycle.name` setting, first use the [remove policy API]( ### Change a static index setting for a data stream [change-static-index-setting-for-a-data-stream] -[Static index settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md) can only be set when a backing index is created. You cannot update static index settings using the [update index settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings). +[Static index settings](elasticsearch://reference/elasticsearch/index-settings/index.md) can only be set when a backing index is created. You cannot update static index settings using the [update index settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings). To apply a new static setting to future backing indices, update the index template used by the data stream. The setting is automatically applied to any backing index created after the update. @@ -426,7 +426,7 @@ Follow these steps: You can also use a query to reindex only a subset of documents with each request. - The following [reindex API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex) request copies documents from `my-data-stream` to `new-data-stream`. The request uses a [`range` query](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-range-query.md) to only reindex documents with a timestamp within the last week. Note the request’s `op_type` is `create`. + The following [reindex API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex) request copies documents from `my-data-stream` to `new-data-stream`. The request uses a [`range` query](elasticsearch://reference/query-languages/query-dsl-range-query.md) to only reindex documents with a timestamp within the last week. Note the request’s `op_type` is `create`. ```console POST /_reindex diff --git a/manage-data/data-store/data-streams/reindex-tsds.md b/manage-data/data-store/data-streams/reindex-tsds.md index 0a6eff68b..69384cee5 100644 --- a/manage-data/data-store/data-streams/reindex-tsds.md +++ b/manage-data/data-store/data-streams/reindex-tsds.md @@ -13,7 +13,7 @@ applies_to: -## Introduction [tsds-reindex-intro] +## Introduction [tsds-reindex-intro] With reindexing, you can copy documents from an old [time-series data stream (TSDS)](../data-streams/time-series-data-stream-tsds.md) to a new one. Data streams support reindexing in general, with a few [restrictions](use-data-stream.md#reindex-with-a-data-stream). Still, time-series data streams introduce additional challenges due to tight control on the accepted timestamp range for each backing index they contain. Direct use of the reindex API would likely error out due to attempting to insert documents with timestamps that are outside the current acceptance window. @@ -30,7 +30,7 @@ To avoid these limitations, use the process that is outlined below: 4. Revert the overriden index settings in the destination index template. 5. Invoke the `rollover` api to create a new backing index that can receive new documents. -::::{note} +::::{note} This process only applies to time-series data streams without [downsampling](./downsampling-time-series-data-stream.md) configuration. Data streams with downsampling can only be re-indexed by re-indexing their backing indexes individually and adding them to an empty destination data stream. :::: @@ -38,7 +38,7 @@ This process only applies to time-series data streams without [downsampling](./d In what follows, we elaborate on each step of the process with examples. -## Create a TSDS template to accept old documents [tsds-reindex-create-template] +## Create a TSDS template to accept old documents [tsds-reindex-create-template] Consider a TSDS with the following template: @@ -201,7 +201,7 @@ POST /_index_template/2 ``` -## Reindex [tsds-reindex-op] +## Reindex [tsds-reindex-op] Invoke the reindex api, for instance: @@ -219,7 +219,7 @@ POST /_reindex ``` -## Restore the destination index template [tsds-reindex-restore] +## Restore the destination index template [tsds-reindex-restore] Once the reindexing operation completes, restore the index template for the destination TSDS as follows: @@ -267,5 +267,5 @@ POST /k9s/_rollover/ This creates a new backing index with the updated index settings. The destination data stream is now ready to accept new documents. -Note that the initial backing index can still accept documents within the range of timestamps derived from the source data stream. If this is not desired, mark it as [read-only](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-block.md#index-blocks-read-only) explicitly. +Note that the initial backing index can still accept documents within the range of timestamps derived from the source data stream. If this is not desired, mark it as [read-only](elasticsearch://reference/elasticsearch/index-settings/index-block.md#index-blocks-read-only) explicitly. diff --git a/manage-data/data-store/data-streams/run-downsampling-using-data-stream-lifecycle.md b/manage-data/data-store/data-streams/run-downsampling-using-data-stream-lifecycle.md index 879c281c0..a21929429 100644 --- a/manage-data/data-store/data-streams/run-downsampling-using-data-stream-lifecycle.md +++ b/manage-data/data-store/data-streams/run-downsampling-using-data-stream-lifecycle.md @@ -317,7 +317,7 @@ POST /datastream/_rollover/ ## View downsampling results [downsampling-dsl-view-results] -By default, data stream lifecycle actions are executed every five minutes. Downsampling takes place after the index is rolled over and the [index time series end time](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/time-series.md#index-time-series-end-time) has lapsed as the source index is still expected to receive major writes until then. Index is now rolled over after previous step but its time series range end is likely still in the future. Once index time series range is in the past, re-run the `GET _data_stream` request. +By default, data stream lifecycle actions are executed every five minutes. Downsampling takes place after the index is rolled over and the [index time series end time](elasticsearch://reference/elasticsearch/index-settings/time-series.md#index-time-series-end-time) has lapsed as the source index is still expected to receive major writes until then. Index is now rolled over after previous step but its time series range end is likely still in the future. Once index time series range is in the past, re-run the `GET _data_stream` request. ```console GET _data_stream diff --git a/manage-data/data-store/data-streams/run-downsampling-with-ilm.md b/manage-data/data-store/data-streams/run-downsampling-with-ilm.md index 8546f72dd..505127951 100644 --- a/manage-data/data-store/data-streams/run-downsampling-with-ilm.md +++ b/manage-data/data-store/data-streams/run-downsampling-with-ilm.md @@ -32,9 +32,9 @@ Before running this example you may want to try the [Run downsampling manually]( Create an ILM policy for your time series data. While not required, an ILM policy is recommended to automate the management of your time series data stream indices. -To enable downsampling, add a [Downsample action](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-downsample.md) and set [`fixed_interval`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-downsample.md#ilm-downsample-options) to the downsampling interval at which you want to aggregate the original time series data. +To enable downsampling, add a [Downsample action](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-downsample.md) and set [`fixed_interval`](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-downsample.md#ilm-downsample-options) to the downsampling interval at which you want to aggregate the original time series data. -In this example, an ILM policy is configured for the `hot` phase. The downsample takes place after the index is rolled over and the [index time series end time](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/time-series.md#index-time-series-end-time) has lapsed as the source index is still expected to receive major writes until then. {{ilm-cap}} will not proceed with any action that expects the index to not receive writes anymore until the [index’s end time](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/time-series.md#index-time-series-end-time) has passed. The {{ilm-cap}} actions that wait on the end time before proceeding are: - [Delete](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-delete.md) - [Downsample](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-downsample.md) - [Force merge](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-forcemerge.md) - [Read only](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-readonly.md) - [Searchable snapshot](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md) - [Shrink](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-shrink.md) +In this example, an ILM policy is configured for the `hot` phase. The downsample takes place after the index is rolled over and the [index time series end time](elasticsearch://reference/elasticsearch/index-settings/time-series.md#index-time-series-end-time) has lapsed as the source index is still expected to receive major writes until then. {{ilm-cap}} will not proceed with any action that expects the index to not receive writes anymore until the [index’s end time](elasticsearch://reference/elasticsearch/index-settings/time-series.md#index-time-series-end-time) has passed. The {{ilm-cap}} actions that wait on the end time before proceeding are: - [Delete](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-delete.md) - [Downsample](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-downsample.md) - [Force merge](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-forcemerge.md) - [Read only](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-readonly.md) - [Searchable snapshot](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md) - [Shrink](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-shrink.md) ```console PUT _ilm/policy/datastream_policy diff --git a/manage-data/data-store/data-streams/set-up-data-stream.md b/manage-data/data-store/data-streams/set-up-data-stream.md index 581ee8196..dc1c0be0a 100644 --- a/manage-data/data-store/data-streams/set-up-data-stream.md +++ b/manage-data/data-store/data-streams/set-up-data-stream.md @@ -21,7 +21,7 @@ You can also [convert an index alias to a data stream](#convert-index-alias-to-d ::::{important} If you use {{fleet}}, {{agent}}, or {{ls}}, skip this tutorial. They all set up data streams for you. -For {{fleet}} and {{agent}}, check out this [data streams documentation](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/data-streams.md). For {{ls}}, check out the [data streams settings](asciidocalypse://docs/logstash/docs/reference/plugins-outputs-elasticsearch.md#plugins-outputs-elasticsearch-data_stream) for the `elasticsearch output` plugin. +For {{fleet}} and {{agent}}, check out this [data streams documentation](/reference/ingestion-tools/fleet/data-streams.md). For {{ls}}, check out the [data streams settings](logstash://reference/plugins-outputs-elasticsearch.md#plugins-outputs-elasticsearch-data_stream) for the `elasticsearch output` plugin. :::: @@ -92,13 +92,13 @@ A data stream requires a matching index template. In most cases, you compose thi When creating your component templates, include: -* A [`date`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/date.md) or [`date_nanos`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/date_nanos.md) mapping for the `@timestamp` field. If you don’t specify a mapping, {{es}} maps `@timestamp` as a `date` field with default options. +* A [`date`](elasticsearch://reference/elasticsearch/mapping-reference/date.md) or [`date_nanos`](elasticsearch://reference/elasticsearch/mapping-reference/date_nanos.md) mapping for the `@timestamp` field. If you don’t specify a mapping, {{es}} maps `@timestamp` as a `date` field with default options. * Your lifecycle policy in the `index.lifecycle.name` index setting. ::::{tip} Use the [Elastic Common Schema (ECS)](https://www.elastic.co/guide/en/ecs/current) when mapping your fields. ECS fields integrate with several {{stack}} features by default. -If you’re unsure how to map your fields, use [runtime fields](../mapping/define-runtime-fields-in-search-request.md) to extract fields from [unstructured content](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/keyword.md#mapping-unstructured-content) at search time. For example, you can index a log message to a `wildcard` field and later extract IP addresses and other data from this field during a search. +If you’re unsure how to map your fields, use [runtime fields](../mapping/define-runtime-fields-in-search-request.md) to extract fields from [unstructured content](elasticsearch://reference/elasticsearch/mapping-reference/keyword.md#mapping-unstructured-content) at search time. For example, you can index a log message to a `wildcard` field and later extract IP addresses and other data from this field during a search. :::: @@ -150,7 +150,7 @@ PUT _component_template/my-settings Use your component templates to create an index template. Specify: -* One or more index patterns that match the data stream’s name. We recommend using our [data stream naming scheme](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/data-streams.md#data-streams-naming-scheme). +* One or more index patterns that match the data stream’s name. We recommend using our [data stream naming scheme](/reference/ingestion-tools/fleet/data-streams.md#data-streams-naming-scheme). * That the template is data stream enabled. * Any component templates that contain your mappings and index settings. * A priority higher than `200` to avoid collisions with built-in templates. See [Avoid index pattern collisions](../templates.md#avoid-index-pattern-collisions). diff --git a/manage-data/data-store/data-streams/set-up-tsds.md b/manage-data/data-store/data-streams/set-up-tsds.md index 5aebe1cf7..6035713f6 100644 --- a/manage-data/data-store/data-streams/set-up-tsds.md +++ b/manage-data/data-store/data-streams/set-up-tsds.md @@ -93,11 +93,11 @@ PUT _ilm/policy/my-weather-sensor-lifecycle-policy To setup a TSDS create an index template with the following details: -* One or more index patterns that match the TSDS’s name. We recommend using our [data stream naming scheme](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/data-streams.md#data-streams-naming-scheme). +* One or more index patterns that match the TSDS’s name. We recommend using our [data stream naming scheme](/reference/ingestion-tools/fleet/data-streams.md#data-streams-naming-scheme). * Enable data streams. * Specify a mapping that defines your dimensions and metrics: - * One or more [dimension fields](time-series-data-stream-tsds.md#time-series-dimension) with a `time_series_dimension` value of `true`. Alternatively, one or more [pass-through](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/passthrough.md#passthrough-dimensions) fields configured as dimension containers, provided that they will contain at least one sub-field (mapped statically or dynamically). + * One or more [dimension fields](time-series-data-stream-tsds.md#time-series-dimension) with a `time_series_dimension` value of `true`. Alternatively, one or more [pass-through](elasticsearch://reference/elasticsearch/mapping-reference/passthrough.md#passthrough-dimensions) fields configured as dimension containers, provided that they will contain at least one sub-field (mapped statically or dynamically). * One or more [metric fields](time-series-data-stream-tsds.md#time-series-metric), marked using the `time_series_metric` mapping parameter. * Optional: A `date` or `date_nanos` mapping for the `@timestamp` field. If you don’t specify a mapping, Elasticsearch maps `@timestamp` as a `date` field with default options. @@ -105,7 +105,7 @@ To setup a TSDS create an index template with the following details: * Set `index.mode` setting to `time_series`. * Your lifecycle policy in the `index.lifecycle.name` index setting. - * Optional: Other index settings, such as [`index.number_of_replicas`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#dynamic-index-number-of-replicas), for your TSDS’s backing indices. + * Optional: Other index settings, such as [`index.number_of_replicas`](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#dynamic-index-number-of-replicas), for your TSDS’s backing indices. * A priority higher than `200` to avoid collisions with built-in templates. See [Avoid index pattern collisions](../templates.md#avoid-index-pattern-collisions). * Optional: Component templates containing your mappings and other index settings. diff --git a/manage-data/data-store/data-streams/time-series-data-stream-tsds.md b/manage-data/data-store/data-streams/time-series-data-stream-tsds.md index 24d2ea729..d333e21a1 100644 --- a/manage-data/data-store/data-streams/time-series-data-stream-tsds.md +++ b/manage-data/data-store/data-streams/time-series-data-stream-tsds.md @@ -32,9 +32,9 @@ A TSDS works like a regular data stream with some key differences: * {{es}} generates a hidden [`_tsid`](#tsid) metadata field for each document in a TSDS. * A TSDS uses [time-bound backing indices](#time-bound-indices) to store data from the same time period in the same backing index. * The matching index template for a TSDS must contain the `index.routing_path` index setting. A TSDS uses this setting to perform [dimension-based routing](#dimension-based-routing). -* A TSDS uses internal [index sorting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/sorting.md) to order shard segments by `_tsid` and `@timestamp`. +* A TSDS uses internal [index sorting](elasticsearch://reference/elasticsearch/index-settings/sorting.md) to order shard segments by `_tsid` and `@timestamp`. * TSDS documents only support auto-generated document `_id` values. For TSDS documents, the document `_id` is a hash of the document’s dimensions and `@timestamp`. A TSDS doesn’t support custom document `_id` values. -* A TSDS uses [synthetic `_source`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source), and as a result is subject to some [restrictions](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source-restrictions) and [modifications](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source-modifications) applied to the `_source` field. +* A TSDS uses [synthetic `_source`](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source), and as a result is subject to some [restrictions](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source-restrictions) and [modifications](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source-modifications) applied to the `_source` field. ::::{note} A time series index can contain fields other than dimensions or metrics. @@ -66,18 +66,18 @@ A TSDS document is uniquely identified by its time series and timestamp, both of You mark a field as a dimension using the boolean `time_series_dimension` mapping parameter. The following field types support the `time_series_dimension` parameter: -* [`keyword`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/keyword.md#keyword-field-type) -* [`ip`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/ip.md) -* [`byte`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/number.md) -* [`short`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/number.md) -* [`integer`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/number.md) -* [`long`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/number.md) -* [`unsigned_long`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/number.md) -* [`boolean`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/boolean.md) +* [`keyword`](elasticsearch://reference/elasticsearch/mapping-reference/keyword.md#keyword-field-type) +* [`ip`](elasticsearch://reference/elasticsearch/mapping-reference/ip.md) +* [`byte`](elasticsearch://reference/elasticsearch/mapping-reference/number.md) +* [`short`](elasticsearch://reference/elasticsearch/mapping-reference/number.md) +* [`integer`](elasticsearch://reference/elasticsearch/mapping-reference/number.md) +* [`long`](elasticsearch://reference/elasticsearch/mapping-reference/number.md) +* [`unsigned_long`](elasticsearch://reference/elasticsearch/mapping-reference/number.md) +* [`boolean`](elasticsearch://reference/elasticsearch/mapping-reference/boolean.md) -For a flattened field, use the `time_series_dimensions` parameter to configure an array of fields as dimensions. For details refer to [`flattened`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/flattened.md#flattened-params). +For a flattened field, use the `time_series_dimensions` parameter to configure an array of fields as dimensions. For details refer to [`flattened`](elasticsearch://reference/elasticsearch/mapping-reference/flattened.md#flattened-params). -Dimension definitions can be simplified through [pass-through](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/passthrough.md#passthrough-dimensions) fields. +Dimension definitions can be simplified through [pass-through](elasticsearch://reference/elasticsearch/mapping-reference/passthrough.md#passthrough-dimensions) fields. ### Metrics [time-series-metric] @@ -88,9 +88,9 @@ Metrics differ from dimensions in that while dimensions generally remain constan To mark a field as a metric, you must specify a metric type using the `time_series_metric` mapping parameter. The following field types support the `time_series_metric` parameter: -* [`aggregate_metric_double`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/aggregate-metric-double.md) -* [`histogram`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/histogram.md) -* All [numeric field types](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/number.md) +* [`aggregate_metric_double`](elasticsearch://reference/elasticsearch/mapping-reference/aggregate-metric-double.md) +* [`histogram`](elasticsearch://reference/elasticsearch/mapping-reference/histogram.md) +* All [numeric field types](elasticsearch://reference/elasticsearch/mapping-reference/number.md) Accepted metric types vary based on the field type: @@ -123,7 +123,7 @@ Due to the cumulative nature of counter fields, the following aggregations are s ## Time series mode [time-series-mode] -The matching index template for a TSDS must contain a `data_stream` object with the `index_mode: time_series` option. This option ensures the TSDS creates backing indices with an [`index.mode`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/time-series.md#index-mode) setting of `time_series`. This setting enables most TSDS-related functionality in the backing indices. +The matching index template for a TSDS must contain a `data_stream` object with the `index_mode: time_series` option. This option ensures the TSDS creates backing indices with an [`index.mode`](elasticsearch://reference/elasticsearch/index-settings/time-series.md#index-mode) setting of `time_series`. This setting enables most TSDS-related functionality in the backing indices. If you convert an existing data stream to a TSDS, only backing indices created after the conversion have an `index.mode` of `time_series`. You can’t change the `index.mode` of an existing backing index. @@ -132,7 +132,7 @@ If you convert an existing data stream to a TSDS, only backing indices created a When you add a document to a TSDS, {{es}} automatically generates a `_tsid` metadata field for the document. The `_tsid` is an object containing the document’s dimensions. Documents in the same TSDS with the same `_tsid` are part of the same time series. -The `_tsid` field is not queryable or updatable. You also can’t retrieve a document’s `_tsid` using a [get document](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-get) request. However, you can use the `_tsid` field in aggregations and retrieve the `_tsid` value in searches using the [`fields` parameter](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/retrieve-selected-fields.md#search-fields-param). +The `_tsid` field is not queryable or updatable. You also can’t retrieve a document’s `_tsid` using a [get document](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-get) request. However, you can use the `_tsid` field in aggregations and retrieve the `_tsid` value in searches using the [`fields` parameter](elasticsearch://reference/elasticsearch/rest-apis/retrieve-selected-fields.md#search-fields-param). ::::{warning} The format of the `_tsid` field shouldn’t be relied upon. It may change from version to version. @@ -142,7 +142,7 @@ The format of the `_tsid` field shouldn’t be relied upon. It may change from v ### Time-bound indices [time-bound-indices] -In a TSDS, each backing index, including the most recent backing index, has a range of accepted `@timestamp` values. This range is defined by the [`index.time_series.start_time`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/time-series.md#index-time-series-start-time) and [`index.time_series.end_time`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/time-series.md#index-time-series-end-time) index settings. +In a TSDS, each backing index, including the most recent backing index, has a range of accepted `@timestamp` values. This range is defined by the [`index.time_series.start_time`](elasticsearch://reference/elasticsearch/index-settings/time-series.md#index-time-series-start-time) and [`index.time_series.end_time`](elasticsearch://reference/elasticsearch/index-settings/time-series.md#index-time-series-end-time) index settings. When you add a document to a TSDS, {{es}} adds the document to the appropriate backing index based on its `@timestamp` value. As a result, a TSDS can add documents to any TSDS backing index that can receive writes. This applies even if the index isn’t the most recent backing index. @@ -151,7 +151,7 @@ When you add a document to a TSDS, {{es}} adds the document to the appropriate b ::: ::::{tip} -Some {{ilm-init}} actions mark the source index as read-only, or expect the index to not be actively written anymore in order to provide good performance. These actions are: - [Delete](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-delete.md) - [Downsample](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-downsample.md) - [Force merge](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-forcemerge.md) - [Read only](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-readonly.md) - [Searchable snapshot](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md) - [Shrink](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-shrink.md) {{ilm-cap}} will **not** proceed with executing these actions until the upper time-bound for accepting writes, represented by the [`index.time_series.end_time`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/time-series.md#index-time-series-end-time) index setting, has lapsed. +Some {{ilm-init}} actions mark the source index as read-only, or expect the index to not be actively written anymore in order to provide good performance. These actions are: - [Delete](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-delete.md) - [Downsample](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-downsample.md) - [Force merge](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-forcemerge.md) - [Read only](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-readonly.md) - [Searchable snapshot](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md) - [Shrink](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-shrink.md) {{ilm-cap}} will **not** proceed with executing these actions until the upper time-bound for accepting writes, represented by the [`index.time_series.end_time`](elasticsearch://reference/elasticsearch/index-settings/time-series.md#index-time-series-end-time) index setting, has lapsed. :::: @@ -162,7 +162,7 @@ If no backing index can accept a document’s `@timestamp` value, {{es}} rejects ### Look-ahead time [tsds-look-ahead-time] -Use the [`index.look_ahead_time`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/time-series.md#index-look-ahead-time) index setting to configure how far into the future you can add documents to an index. When you create a new write index for a TSDS, {{es}} calculates the index’s `index.time_series.end_time` value as: +Use the [`index.look_ahead_time`](elasticsearch://reference/elasticsearch/index-settings/time-series.md#index-look-ahead-time) index setting to configure how far into the future you can add documents to an index. When you create a new write index for a TSDS, {{es}} calculates the index’s `index.time_series.end_time` value as: `now + index.look_ahead_time` @@ -175,7 +175,7 @@ This process continues until the write index rolls over. When the index rolls ov ### Look-back time [tsds-look-back-time] -Use the [`index.look_back_time`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/time-series.md#index-look-back-time) index setting to configure how far in the past you can add documents to an index. When you create a data stream for a TSDS, {{es}} calculates the index’s `index.time_series.start_time` value as: +Use the [`index.look_back_time`](elasticsearch://reference/elasticsearch/index-settings/time-series.md#index-look-back-time) index setting to configure how far in the past you can add documents to an index. When you create a data stream for a TSDS, {{es}} calculates the index’s `index.time_series.start_time` value as: `now - index.look_back_time` @@ -196,24 +196,24 @@ You can use the [get data stream API](https://www.elastic.co/docs/api/doc/elasti ### Dimension-based routing [dimension-based-routing] -Within each TSDS backing index, {{es}} uses the [`index.routing_path`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/time-series.md#index-routing-path) index setting to route documents with the same dimensions to the same shards. +Within each TSDS backing index, {{es}} uses the [`index.routing_path`](elasticsearch://reference/elasticsearch/index-settings/time-series.md#index-routing-path) index setting to route documents with the same dimensions to the same shards. When you create the matching index template for a TSDS, you must specify one or more dimensions in the `index.routing_path` setting. Each document in a TSDS must contain one or more dimensions that match the `index.routing_path` setting. The `index.routing_path` setting accepts wildcard patterns (for example `dim.*`) and can dynamically match new fields. However, {{es}} will reject any mapping updates that add scripted, runtime, or non-dimension fields that match the `index.routing_path` value. -[Pass-through](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/passthrough.md#passthrough-dimensions) fields may be configured as dimension containers. In this case, their sub-fields get included to the routing path automatically. +[Pass-through](elasticsearch://reference/elasticsearch/mapping-reference/passthrough.md#passthrough-dimensions) fields may be configured as dimension containers. In this case, their sub-fields get included to the routing path automatically. TSDS documents don’t support a custom `_routing` value. Similarly, you can’t require a `_routing` value in mappings for a TSDS. ### Index sorting [tsds-index-sorting] -{{es}} uses [compression algorithms](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-codec) to compress repeated values. This compression works best when repeated values are stored near each other — in the same index, on the same shard, and side-by-side in the same shard segment. +{{es}} uses [compression algorithms](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#index-codec) to compress repeated values. This compression works best when repeated values are stored near each other — in the same index, on the same shard, and side-by-side in the same shard segment. Most time series data contains repeated values. Dimensions are repeated across documents in the same time series. The metric values of a time series may also change slowly over time. -Internally, each TSDS backing index uses [index sorting](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/sorting.md) to order its shard segments by `_tsid` and `@timestamp`. This makes it more likely that these repeated values are stored near each other for better compression. A TSDS doesn’t support any [`index.sort.*`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/sorting.md) index settings. +Internally, each TSDS backing index uses [index sorting](elasticsearch://reference/elasticsearch/index-settings/sorting.md) to order its shard segments by `_tsid` and `@timestamp`. This makes it more likely that these repeated values are stored near each other for better compression. A TSDS doesn’t support any [`index.sort.*`](elasticsearch://reference/elasticsearch/index-settings/sorting.md) index settings. ## What’s next? [tsds-whats-next] diff --git a/manage-data/data-store/data-streams/use-data-stream.md b/manage-data/data-store/data-streams/use-data-stream.md index 7d547f646..1b17f34f4 100644 --- a/manage-data/data-store/data-streams/use-data-stream.md +++ b/manage-data/data-store/data-streams/use-data-stream.md @@ -21,7 +21,7 @@ After you [set up a data stream](set-up-data-stream.md), you can do the followin * [Update or delete documents in a backing index](#update-delete-docs-in-a-backing-index) -## Add documents to a data stream [add-documents-to-a-data-stream] +## Add documents to a data stream [add-documents-to-a-data-stream] To add an individual document, use the [index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-create). [Ingest pipelines](../../ingest/transform-enrich/ingest-pipelines.md) are supported. @@ -51,7 +51,7 @@ PUT /my-data-stream/_bulk?refresh ``` -## Search a data stream [search-a-data-stream] +## Search a data stream [search-a-data-stream] The following search APIs support data streams: @@ -62,7 +62,7 @@ The following search APIs support data streams: * [EQL search](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-eql-search) -## Get statistics for a data stream [get-stats-for-a-data-stream] +## Get statistics for a data stream [get-stats-for-a-data-stream] Use the [data stream stats API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-data-streams-stats-1) to get statistics for one or more data streams: @@ -71,7 +71,7 @@ GET /_data_stream/my-data-stream/_stats?human=true ``` -## Manually roll over a data stream [manually-roll-over-a-data-stream] +## Manually roll over a data stream [manually-roll-over-a-data-stream] Use the [rollover API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-rollover) to manually [roll over](../data-streams.md#data-streams-rollover) a data stream. You have two options when manually rolling over: @@ -91,7 +91,7 @@ Use the [rollover API](https://www.elastic.co/docs/api/doc/elasticsearch/operati -## Open closed backing indices [open-closed-backing-indices] +## Open closed backing indices [open-closed-backing-indices] You cannot search a [closed](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-close) backing index, even by searching its data stream. You also cannot [update](#update-docs-in-a-data-stream-by-query) or [delete](#delete-docs-in-a-data-stream-by-query) documents in a closed index. @@ -108,7 +108,7 @@ POST /my-data-stream/_open/ ``` -## Reindex with a data stream [reindex-with-a-data-stream] +## Reindex with a data stream [reindex-with-a-data-stream] Use the [reindex API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex) to copy documents from an existing index, alias, or data stream to a data stream. Because data streams are [append-only](../data-streams.md#data-streams-append-only), a reindex into a data stream must use an `op_type` of `create`. A reindex cannot update existing documents in a data stream. @@ -126,7 +126,7 @@ POST /_reindex ``` -## Update documents in a data stream by query [update-docs-in-a-data-stream-by-query] +## Update documents in a data stream by query [update-docs-in-a-data-stream-by-query] Use the [update by query API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-update-by-query) to update documents in a data stream that match a provided query: @@ -148,7 +148,7 @@ POST /my-data-stream/_update_by_query ``` -## Delete documents in a data stream by query [delete-docs-in-a-data-stream-by-query] +## Delete documents in a data stream by query [delete-docs-in-a-data-stream-by-query] Use the [delete by query API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-delete-by-query) to delete documents in a data stream that match a provided query: @@ -164,13 +164,13 @@ POST /my-data-stream/_delete_by_query ``` -## Update or delete documents in a backing index [update-delete-docs-in-a-backing-index] +## Update or delete documents in a backing index [update-delete-docs-in-a-backing-index] If needed, you can update or delete documents in a data stream by sending requests to the backing index containing the document. You’ll need: -* The [document ID](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-id-field.md) +* The [document ID](elasticsearch://reference/elasticsearch/mapping-reference/mapping-id-field.md) * The name of the backing index containing the document -* If updating the document, its [sequence number and primary term](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/optimistic-concurrency-control.md) +* If updating the document, its [sequence number and primary term](elasticsearch://reference/elasticsearch/rest-apis/optimistic-concurrency-control.md) To get this information, use a [search request](#search-a-data-stream): diff --git a/manage-data/data-store/index-basics.md b/manage-data/data-store/index-basics.md index 0590615dd..2dbeefb86 100644 --- a/manage-data/data-store/index-basics.md +++ b/manage-data/data-store/index-basics.md @@ -14,7 +14,7 @@ This content applies to: [![Elasticsearch](/images/serverless-es-badge.svg "")]( An index is a fundamental unit of storage in {{es}}. It is a collection of documents uniquely identified by a name or an [alias](/manage-data/data-store/aliases.md). This unique name is important because it’s used to target the index in search queries and other operations. -::::{tip} +::::{tip} A closely related concept is a [data stream](/manage-data/data-store/data-streams.md). This index abstraction is optimized for append-only timestamped data, and is made up of hidden, auto-generated backing indices. If you’re working with timestamped data, we recommend the [Elastic Observability](https://www.elastic.co/guide/en/observability/current) solution for additional tools and optimized content. :::: @@ -22,7 +22,7 @@ A closely related concept is a [data stream](/manage-data/data-store/data-stream An index is made up of the following components. -### Documents [elasticsearch-intro-documents-fields] +### Documents [elasticsearch-intro-documents-fields] {{es}} serializes and stores data in the form of JSON documents. A document is a set of fields, which are key-value pairs that contain your data. Each document has a unique ID, which you can create or have {{es}} auto-generate. @@ -53,17 +53,17 @@ A simple {{es}} document might look like this: } ``` -### Metadata fields [elasticsearch-intro-documents-fields-data-metadata] +### Metadata fields [elasticsearch-intro-documents-fields-data-metadata] -An indexed document contains data and metadata. [Metadata fields](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/document-metadata-fields.md) are system fields that store information about the documents. In {{es}}, metadata fields are prefixed with an underscore. For example, the following fields are metadata fields: +An indexed document contains data and metadata. [Metadata fields](elasticsearch://reference/elasticsearch/mapping-reference/document-metadata-fields.md) are system fields that store information about the documents. In {{es}}, metadata fields are prefixed with an underscore. For example, the following fields are metadata fields: * `_index`: The name of the index where the document is stored. * `_id`: The document’s ID. IDs must be unique per index. -### Mappings and data types [elasticsearch-intro-documents-fields-mappings] +### Mappings and data types [elasticsearch-intro-documents-fields-mappings] -Each index has a [mapping](/manage-data/data-store/mapping.md) or schema for how the fields in your documents are indexed. A mapping defines the [data type](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/field-data-types.md) for each field, how the field should be indexed, and how it should be stored. +Each index has a [mapping](/manage-data/data-store/mapping.md) or schema for how the fields in your documents are indexed. A mapping defines the [data type](elasticsearch://reference/elasticsearch/mapping-reference/field-data-types.md) for each field, how the field should be indexed, and how it should be stored. ## Index management @@ -82,12 +82,12 @@ Investigate your indices and perform operations from the **Indices** view. * To show details and perform operations, click the index name. To perform operations on multiple indices, select their checkboxes and then open the **Manage** menu. For more information on managing indices, refer to [Index APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-indices). * To filter the list of indices, use the search bar or click a badge. Badges indicate if an index is a [follower index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ccr-follow), a [rollup index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-rollup-get-rollup-index-caps), or [frozen](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-unfreeze). -* To drill down into the index [mappings](/manage-data/data-store/mapping.md), [settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md), and statistics, click an index name. From this view, you can navigate to **Discover** to further explore the documents in the index. +* To drill down into the index [mappings](/manage-data/data-store/mapping.md), [settings](elasticsearch://reference/elasticsearch/index-settings/index.md), and statistics, click an index name. From this view, you can navigate to **Discover** to further explore the documents in the index. * To create new indices, use the **Create index** wizard. ### Manage data streams -A [data stream](/manage-data/data-store/data-streams.md) lets you store append-only time series data across multiple indices while giving you a single named resource for requests. +A [data stream](/manage-data/data-store/data-streams.md) lets you store append-only time series data across multiple indices while giving you a single named resource for requests. Investigate your data streams and address lifecycle management needs in the **Data Streams** view. @@ -96,16 +96,16 @@ Investigate your data streams and address lifecycle management needs in the **Da :class: screenshot ::: -In {{es-serverless}}, indices matching the `logs-*-*` pattern use the logsDB index mode by default. The logsDB index mode creates a [logs data stream](https://www.elastic.co/guide/en/elasticsearch/reference/master/logs-data-stream.html). +In {{es-serverless}}, indices matching the `logs-*-*` pattern use the logsDB index mode by default. The logsDB index mode creates a [logs data stream](https://www.elastic.co/guide/en/elasticsearch/reference/master/logs-data-stream.html). * To view information about the stream's backing indices, click the number in the **Indices** column. -* A value in the **Data retention** column indicates that the data stream is managed by a data stream lifecycle policy. This value is the time period for which your data is guaranteed to be stored. Data older than this period can be deleted by {{es}} at a later time. +* A value in the **Data retention** column indicates that the data stream is managed by a data stream lifecycle policy. This value is the time period for which your data is guaranteed to be stored. Data older than this period can be deleted by {{es}} at a later time. * To modify the data retention value, select an index, open the **Manage** menu, and click **Edit data retention**. * To view more information about a data stream, such as its generation or its current index lifecycle policy, click the stream's name. From this view, you can navigate to **Discover** to further explore data within the data stream. ### Manage index templates [index-management-manage-index-templates] -An [index template](/manage-data/data-store/templates.md) is a way to tell {{es}} how to configure an index when it is created. +An [index template](/manage-data/data-store/templates.md) is a way to tell {{es}} how to configure an index when it is created. Create, edit, clone, and delete your index templates in the **Index Templates** view. Changes made to an index template do not affect existing indices. @@ -115,7 +115,7 @@ Create, edit, clone, and delete your index templates in the **Index Templates** ::: * To show details and perform operations, click the template name. -* To view more information about the component templates within an index template, click the value in the **Component templates** column. +* To view more information about the component templates within an index template, click the value in the **Component templates** column. * Values in the **Content** column indicate whether a template contains index mappings, settings, and aliases. * To create new index templates, use the **Create template** wizard. diff --git a/manage-data/data-store/mapping.md b/manage-data/data-store/mapping.md index 115cdf833..79e7b0773 100644 --- a/manage-data/data-store/mapping.md +++ b/manage-data/data-store/mapping.md @@ -35,7 +35,7 @@ $$$mapping-explicit$$$ Mapping is the process of defining how a document and the fields it contains are stored and indexed. -Each document is a collection of fields, which each have their own [data type](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/field-data-types.md). When mapping your data, you create a mapping definition, which contains a list of fields that are pertinent to the document. A mapping definition also includes [metadata fields](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/document-metadata-fields.md), like the `_source` field, which customize how a document’s associated metadata is handled. +Each document is a collection of fields, which each have their own [data type](elasticsearch://reference/elasticsearch/mapping-reference/field-data-types.md). When mapping your data, you create a mapping definition, which contains a list of fields that are pertinent to the document. A mapping definition also includes [metadata fields](elasticsearch://reference/elasticsearch/mapping-reference/document-metadata-fields.md), like the `_source` field, which customize how a document’s associated metadata is handled. Depending on where you are in your data journey, use *dynamic mapping* and *explicit mapping* to define your data. For example, you can explicitly map fields where you don’t want to use the defaults, or to gain greater control over which fields are created. Then you can allow {{es}} to dynamically map other fields. Using a combination of dynamic and explicit mapping on the same index is especially useful when you have a mix of known and unknown fields in your data. @@ -43,13 +43,13 @@ Depending on where you are in your data journey, use *dynamic mapping* and *expl Before 7.0.0, the mapping definition included a type name. {{es}} 7.0.0 and later no longer accept a default mapping. [Removal of mapping types](/manage-data/data-store/mapping/removal-of-mapping-types.md) provides more information. :::: -## Dynamic mapping [mapping-dynamic] +## Dynamic mapping [mapping-dynamic] -When you use [dynamic mapping](/manage-data/data-store/mapping/dynamic-mapping.md), {{es}} automatically detects the data types of fields in your documents and creates mappings for you. If you index additional documents with new fields, {{es}} will add these fields automatically. You can add fields to the top-level mapping, and to inner [`object`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/object.md) and [`nested`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/nested.md) fields. Dynamic mapping helps you get started quickly, but might yield suboptimal results for your specific use case due to automatic field type inference. +When you use [dynamic mapping](/manage-data/data-store/mapping/dynamic-mapping.md), {{es}} automatically detects the data types of fields in your documents and creates mappings for you. If you index additional documents with new fields, {{es}} will add these fields automatically. You can add fields to the top-level mapping, and to inner [`object`](elasticsearch://reference/elasticsearch/mapping-reference/object.md) and [`nested`](elasticsearch://reference/elasticsearch/mapping-reference/nested.md) fields. Dynamic mapping helps you get started quickly, but might yield suboptimal results for your specific use case due to automatic field type inference. -Use [dynamic templates](/manage-data/data-store/mapping/dynamic-templates.md) to define custom mappings that are applied to dynamically added fields based on the matching condition. +Use [dynamic templates](/manage-data/data-store/mapping/dynamic-templates.md) to define custom mappings that are applied to dynamically added fields based on the matching condition. -## Explicit mapping [mapping-explicit] +## Explicit mapping [mapping-explicit] Use [explicit mapping](/manage-data/data-store/mapping/explicit-mapping.md) to define mappings by specifying data types for each field. This is recommended for production use cases, because you have full control over how your data is indexed to suit your specific use case. @@ -58,7 +58,7 @@ Defining your own mappings enables you to: * Define which string fields should be treated as full-text fields. * Define which fields contain numbers, dates, or geolocations. * Use data types that cannot be automatically detected (such as `geo_point` and `geo_shape`.) -* Choose date value [formats](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-date-format.md), including custom date formats. +* Choose date value [formats](elasticsearch://reference/elasticsearch/mapping-reference/mapping-date-format.md), including custom date formats. * Create custom rules to control the mapping for [dynamically added fields](/manage-data/data-store/mapping/dynamic-mapping.md). * Optimize fields for partial matching. * Perform language-specific text analysis. @@ -87,11 +87,11 @@ In most cases, you can’t change mappings for fields that are already mapped. T However, you can update mappings under certain conditions: * You can add new fields to an existing mapping at any time, dynamically or explicitly. -* You can add new [multi-fields](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/multi-fields.md) for existing fields. +* You can add new [multi-fields](elasticsearch://reference/elasticsearch/mapping-reference/multi-fields.md) for existing fields. * Documents indexed before the mapping update will not have values for the new multi-fields until they are updated or reindexed. Documents indexed after the mapping change will automatically have values for the new multi-fields. -* Some [mapping parameters](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-parameters.md) can be updated for existing fields of certain [data types](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/field-data-types.md). +* Some [mapping parameters](elasticsearch://reference/elasticsearch/mapping-reference/mapping-parameters.md) can be updated for existing fields of certain [data types](elasticsearch://reference/elasticsearch/mapping-reference/field-data-types.md). ## Prevent mapping explosions [mapping-limit-settings] @@ -100,4 +100,4 @@ Defining too many fields in an index can lead to a mapping explosion, which can Consider a situation where every new document inserted introduces new fields, such as with [dynamic mapping](/manage-data/data-store/mapping/dynamic-mapping.md). Each new field is added to the index mapping, which can become a problem as the mapping grows. -Use the [mapping limit settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/mapping-limit.md) to limit the number of field mappings (created manually or dynamically) and prevent documents from causing a mapping explosion. \ No newline at end of file +Use the [mapping limit settings](elasticsearch://reference/elasticsearch/index-settings/mapping-limit.md) to limit the number of field mappings (created manually or dynamically) and prevent documents from causing a mapping explosion. \ No newline at end of file diff --git a/manage-data/data-store/mapping/define-runtime-fields-in-search-request.md b/manage-data/data-store/mapping/define-runtime-fields-in-search-request.md index ca434b28a..d2c13c490 100644 --- a/manage-data/data-store/mapping/define-runtime-fields-in-search-request.md +++ b/manage-data/data-store/mapping/define-runtime-fields-in-search-request.md @@ -36,7 +36,7 @@ GET my-index-000001/_search ``` -## Create runtime fields that use other runtime fields [runtime-search-request-examples] +## Create runtime fields that use other runtime fields [runtime-search-request-examples] You can even define runtime fields in a search request that return values from other runtime fields. For example, let’s say you bulk index some sensor data: @@ -74,7 +74,7 @@ PUT my-index-000001/_mapping Runtime fields take precedence over fields defined with the same name in the index mappings. This flexibility allows you to shadow existing fields and calculate a different value, without modifying the field itself. If you made a mistake in your index mapping, you can use runtime fields to calculate values that [override values](override-field-values-at-query-time.md) in the mapping during the search request. -Now, you can easily run an [average aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-avg-aggregation.md) on the `measures.start` and `measures.end` fields: +Now, you can easily run an [average aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-avg-aggregation.md) on the `measures.start` and `measures.end` fields: ```console GET my-index-000001/_search @@ -109,7 +109,7 @@ The response includes the aggregation results without changing the values for th } ``` -Further, you can define a runtime field as part of a search query that calculates a value, and then run a [stats aggregation](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-stats-aggregation.md) on that field *in the same query*. +Further, you can define a runtime field as part of a search query that calculates a value, and then run a [stats aggregation](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-stats-aggregation.md) on that field *in the same query*. The `duration` runtime field doesn’t exist in the index mapping, but we can still search and aggregate on that field. The following query returns the calculated value for the `duration` field and runs a stats aggregation to compute statistics over numeric values extracted from the aggregated documents. diff --git a/manage-data/data-store/mapping/dynamic-field-mapping.md b/manage-data/data-store/mapping/dynamic-field-mapping.md index 996e863c2..19fc993c3 100644 --- a/manage-data/data-store/mapping/dynamic-field-mapping.md +++ b/manage-data/data-store/mapping/dynamic-field-mapping.md @@ -8,12 +8,12 @@ applies_to: # Dynamic field mapping [dynamic-field-mapping] -When {{es}} detects a new field in a document, it *dynamically* adds the field to the type mapping by default. The [`dynamic`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/dynamic.md) parameter controls this behavior. +When {{es}} detects a new field in a document, it *dynamically* adds the field to the type mapping by default. The [`dynamic`](elasticsearch://reference/elasticsearch/mapping-reference/dynamic.md) parameter controls this behavior. You can explicitly instruct {{es}} to dynamically create fields based on incoming documents by setting the `dynamic` parameter to `true` or `runtime`. When dynamic field mapping is enabled, {{es}} uses the rules in the following table to determine how to map data types for each field. -::::{note} -The field data types in the following table are the only [field data types](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/field-data-types.md) that {{es}} detects dynamically. You must explicitly map all other data types. +::::{note} +The field data types in the following table are the only [field data types](elasticsearch://reference/elasticsearch/mapping-reference/field-data-types.md) that {{es}} detects dynamically. You must explicitly map all other data types. :::: @@ -33,9 +33,9 @@ $$$dynamic-field-mapping-types$$$ | `string` that passes [numeric detection](#numeric-detection) | `float` or `long` | `double` or `long` | | `string` that doesn’t pass `date` detection or `numeric` detection | `text` with a `.keyword` sub-field | `keyword` | -You can disable dynamic mapping, both at the document and at the [`object`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/object.md) level. Setting the `dynamic` parameter to `false` ignores new fields, and `strict` rejects the document if {{es}} encounters an unknown field. +You can disable dynamic mapping, both at the document and at the [`object`](elasticsearch://reference/elasticsearch/mapping-reference/object.md) level. Setting the `dynamic` parameter to `false` ignores new fields, and `strict` rejects the document if {{es}} encounters an unknown field. -::::{tip} +::::{tip} Use the [update mapping API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-mapping) to update the `dynamic` setting on existing fields. :::: @@ -44,11 +44,11 @@ You can customize dynamic field mapping rules for [date detection](#date-detecti ## Date detection [date-detection] -If `date_detection` is enabled (default), then new string fields are checked to see whether their contents match any of the date patterns specified in `dynamic_date_formats`. If a match is found, a new [`date`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/date.md) field is added with the corresponding format. +If `date_detection` is enabled (default), then new string fields are checked to see whether their contents match any of the date patterns specified in `dynamic_date_formats`. If a match is found, a new [`date`](elasticsearch://reference/elasticsearch/mapping-reference/date.md) field is added with the corresponding format. The default value for `dynamic_date_formats` is: -[ [`"strict_date_optional_time"`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-date-format.md#strict-date-time),`"yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"`] +[ [`"strict_date_optional_time"`](elasticsearch://reference/elasticsearch/mapping-reference/mapping-date-format.md#strict-date-time),`"yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"`] For example: @@ -61,7 +61,7 @@ PUT my-index-000001/_doc/1 GET my-index-000001/_mapping <1> ``` -1. The `create_date` field has been added as a [`date`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/date.md) field with the [`format`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-date-format.md):
`"yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"`. +1. The `create_date` field has been added as a [`date`](elasticsearch://reference/elasticsearch/mapping-reference/date.md) field with the [`format`](elasticsearch://reference/elasticsearch/mapping-reference/mapping-date-format.md):
`"yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"`. ### Disabling date detection [_disabling_date_detection] @@ -82,13 +82,13 @@ PUT my-index-000001/_doc/1 <1> } ``` -1. The `create_date` field has been added as a [`text`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/text.md) field. +1. The `create_date` field has been added as a [`text`](elasticsearch://reference/elasticsearch/mapping-reference/text.md) field. ### Customizing detected date formats [_customizing_detected_date_formats] -Alternatively, the `dynamic_date_formats` can be customized to support your own [date formats](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-date-format.md): +Alternatively, the `dynamic_date_formats` can be customized to support your own [date formats](elasticsearch://reference/elasticsearch/mapping-reference/mapping-date-format.md): ```console PUT my-index-000001 @@ -104,7 +104,7 @@ PUT my-index-000001/_doc/1 } ``` -::::{note} +::::{note} There is a difference between configuring an array of date patterns and configuring multiple patterns in a single string separated by `||`. When you configure an array of date patterns, the pattern that matches the date in the first document with an unmapped date field will determine the mapping of that field: ```console @@ -181,7 +181,7 @@ The resulting mapping will be: :::: -::::{note} +::::{note} Epoch formats (`epoch_millis` and `epoch_second`) are not supported as dynamic date formats. :::: @@ -208,8 +208,8 @@ PUT my-index-000001/_doc/1 } ``` -1. The `my_float` field is added as a [`float`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/number.md) field. -2. The `my_integer` field is added as a [`long`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/number.md) field. +1. The `my_float` field is added as a [`float`](elasticsearch://reference/elasticsearch/mapping-reference/number.md) field. +2. The `my_integer` field is added as a [`long`](elasticsearch://reference/elasticsearch/mapping-reference/number.md) field. diff --git a/manage-data/data-store/mapping/dynamic-templates.md b/manage-data/data-store/mapping/dynamic-templates.md index bcc6f2f80..6d6adebd7 100644 --- a/manage-data/data-store/mapping/dynamic-templates.md +++ b/manage-data/data-store/mapping/dynamic-templates.md @@ -17,7 +17,7 @@ Dynamic templates allow you greater control over how {{es}} maps your data beyon Use the `{{name}}` and `{{dynamic_type}}` [template variables](#template-variables) in the mapping specification as placeholders. -::::{important} +::::{important} Dynamic field mappings are only added when a field contains a concrete value. {{es}} doesn’t add a dynamic field mapping when the field contains `null` or an empty array. If the `null_value` option is used in a `dynamic_template`, it will only be applied after the first document with a concrete value for the field has been indexed. :::: @@ -89,7 +89,7 @@ The `match_mapping_type` parameter matches fields by the data type detected by t Because JSON doesn’t distinguish a `long` from an `integer` or a `double` from a `float`, any parsed floating point number is considered a `double` JSON data type, while any parsed `integer` number is considered a `long`. -::::{note} +::::{note} With dynamic mappings, {{es}} will always choose the wider data type. The one exception is `float`, which requires less storage space than `double` and is precise enough for most applications. Runtime fields do not support `float`, which is why `"dynamic":"runtime"` uses `double`. :::: @@ -174,7 +174,7 @@ PUT my-index-000001/_doc/1 ``` 1. The `my_integer` field is mapped as an `integer`. -2. The `my_string` field is mapped as a `text`, with a `keyword` [multi-field](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/multi-fields.md). +2. The `my_string` field is mapped as a `text`, with a `keyword` [multi-field](elasticsearch://reference/elasticsearch/mapping-reference/multi-fields.md). 3. The `my_boolean` field is mapped as a `keyword`. 4. The `field.count` field is mapped as a `long`. @@ -359,7 +359,7 @@ PUT my-index-000001/_doc/2 ## Template variables [template-variables] -The `{{name}}` and `{{dynamic_type}}` placeholders are replaced in the `mapping` with the field name and detected dynamic type. The following example sets all string fields to use an [`analyzer`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/analyzer.md) with the same name as the field, and disables [`doc_values`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/doc-values.md) for all non-string fields: +The `{{name}}` and `{{dynamic_type}}` placeholders are replaced in the `mapping` with the field name and detected dynamic type. The following example sets all string fields to use an [`analyzer`](elasticsearch://reference/elasticsearch/mapping-reference/analyzer.md) with the same name as the field, and disables [`doc_values`](elasticsearch://reference/elasticsearch/mapping-reference/doc-values.md) for all non-string fields: ```console PUT my-index-000001 diff --git a/manage-data/data-store/mapping/explicit-mapping.md b/manage-data/data-store/mapping/explicit-mapping.md index f2881f960..f49cbfb52 100644 --- a/manage-data/data-store/mapping/explicit-mapping.md +++ b/manage-data/data-store/mapping/explicit-mapping.md @@ -13,7 +13,7 @@ You know more about your data than {{es}} can guess, so while dynamic mapping ca You can create field mappings when you [create an index](#create-mapping) and [add fields to an existing index](#add-field-mapping). -## Create an index with an explicit mapping [create-mapping] +## Create an index with an explicit mapping [create-mapping] You can use the [create index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-create) API to create a new index with an explicit mapping. @@ -30,17 +30,17 @@ PUT /my-index-000001 } ``` -1. Creates `age`, an [`integer`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/number.md) field -2. Creates `email`, a [`keyword`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/keyword.md) field -3. Creates `name`, a [`text`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/text.md) field +1. Creates `age`, an [`integer`](elasticsearch://reference/elasticsearch/mapping-reference/number.md) field +2. Creates `email`, a [`keyword`](elasticsearch://reference/elasticsearch/mapping-reference/keyword.md) field +3. Creates `name`, a [`text`](elasticsearch://reference/elasticsearch/mapping-reference/text.md) field -## Add a field to an existing mapping [add-field-mapping] +## Add a field to an existing mapping [add-field-mapping] You can use the [update mapping](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-mapping) API to add one or more new fields to an existing index. -The following example adds `employee-id`, a `keyword` field with an [`index`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-index.md) mapping parameter value of `false`. This means values for the `employee-id` field are stored but not indexed or available for search. +The following example adds `employee-id`, a `keyword` field with an [`index`](elasticsearch://reference/elasticsearch/mapping-reference/mapping-index.md) mapping parameter value of `false`. This means values for the `employee-id` field are stored but not indexed or available for search. ```console PUT /my-index-000001/_mapping @@ -55,18 +55,18 @@ PUT /my-index-000001/_mapping ``` -## Update the mapping of a field [update-mapping] +## Update the mapping of a field [update-mapping] -Except for supported [mapping parameters](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-parameters.md), you can’t change the mapping or field type of an existing field. Changing an existing field could invalidate data that’s already indexed. +Except for supported [mapping parameters](elasticsearch://reference/elasticsearch/mapping-reference/mapping-parameters.md), you can’t change the mapping or field type of an existing field. Changing an existing field could invalidate data that’s already indexed. If you need to change the mapping of a field in a data stream’s backing indices, see [Change mappings and settings for a data stream](../data-streams/modify-data-stream.md#data-streams-change-mappings-and-settings). If you need to change the mapping of a field in other indices, create a new index with the correct mapping and [reindex](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex) your data into that index. -Renaming a field would invalidate data already indexed under the old field name. Instead, add an [`alias`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/field-alias.md) field to create an alternate field name. +Renaming a field would invalidate data already indexed under the old field name. Instead, add an [`alias`](elasticsearch://reference/elasticsearch/mapping-reference/field-alias.md) field to create an alternate field name. -## View the mapping of an index [view-mapping] +## View the mapping of an index [view-mapping] You can use the [get mapping](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-mapping) API to view the mapping of an existing index. @@ -101,7 +101,7 @@ The API returns the following response: ``` -## View the mapping of specific fields [view-field-mapping] +## View the mapping of specific fields [view-field-mapping] If you only want to view the mapping of one or more specific fields, you can use the [get field mapping](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-mapping) API. diff --git a/manage-data/data-store/mapping/explore-data-with-runtime-fields.md b/manage-data/data-store/mapping/explore-data-with-runtime-fields.md index f58913092..4fe70b312 100644 --- a/manage-data/data-store/mapping/explore-data-with-runtime-fields.md +++ b/manage-data/data-store/mapping/explore-data-with-runtime-fields.md @@ -238,7 +238,7 @@ If the script didn’t include this condition, the query would fail on any shard ### Search for documents in a specific range [runtime-examples-grok-range] -You can also run a [range query](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-range-query.md) that operates on the `timestamp` field. The following query returns any documents where the `timestamp` is greater than or equal to `2020-04-30T14:31:27-05:00`: +You can also run a [range query](elasticsearch://reference/query-languages/query-dsl-range-query.md) that operates on the `timestamp` field. The following query returns any documents where the `timestamp` is greater than or equal to `2020-04-30T14:31:27-05:00`: ```console GET my-index-000001/_search @@ -292,7 +292,7 @@ The response includes the document where the log format doesn’t match, but the ## Define a runtime field with a dissect pattern [runtime-examples-dissect] -If you don’t need the power of regular expressions, you can use [dissect patterns](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/dissect-processor.md) instead of grok patterns. Dissect patterns match on fixed delimiters but are typically faster than grok. +If you don’t need the power of regular expressions, you can use [dissect patterns](elasticsearch://reference/ingestion-tools/enrich-processor/dissect-processor.md) instead of grok patterns. Dissect patterns match on fixed delimiters but are typically faster than grok. You can use dissect to achieve the same results as parsing the Apache logs with a [grok pattern](#runtime-examples-grok). Instead of matching on a log pattern, you include the parts of the string that you want to discard. Paying special attention to the parts of the string you want to discard will help build successful dissect patterns. diff --git a/manage-data/data-store/mapping/index-runtime-field.md b/manage-data/data-store/mapping/index-runtime-field.md index f764f3843..c38d1528f 100644 --- a/manage-data/data-store/mapping/index-runtime-field.md +++ b/manage-data/data-store/mapping/index-runtime-field.md @@ -10,14 +10,14 @@ applies_to: Runtime fields are defined by the context where they run. For example, you can define runtime fields in the [context of a search query](define-runtime-fields-in-search-request.md) or within the [`runtime` section](map-runtime-field.md) of an index mapping. If you decide to index a runtime field for greater performance, just move the full runtime field definition (including the script) to the context of an index mapping. {{es}} automatically uses these indexed fields to drive queries, resulting in a fast response time. This capability means you can write a script only once, and apply it to any context that supports runtime fields. -::::{note} +::::{note} Indexing a `composite` runtime field is currently not supported. :::: You can then use runtime fields to limit the number of fields that {{es}} needs to calculate values for. Using indexed fields in tandem with runtime fields provides flexibility in the data that you index and how you define queries for other fields. -::::{important} +::::{important} After indexing a runtime field, you cannot update the included script. If you need to change the script, create a new field with the updated script. :::: @@ -85,7 +85,7 @@ PUT my-index-000001/_mapping } ``` -You retrieve the calculated values using the [`fields`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/retrieve-selected-fields.md) parameter on the `_search` API: +You retrieve the calculated values using the [`fields`](elasticsearch://reference/elasticsearch/rest-apis/retrieve-selected-fields.md) parameter on the `_search` API: ```console GET my-index-000001/_search @@ -157,7 +157,7 @@ POST my-index-000001/_bulk?refresh=true { "timestamp": 1516297294000, "temperature": 202, "voltage": 4.0, "node": "c"} ``` -You can now retrieve calculated values in a search query, and find documents based on precise values. The following range query returns all documents where the calculated `voltage_corrected` is greater than or equal to `16`, but less than or equal to `20`. Again, use the [`fields`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/retrieve-selected-fields.md) parameter on the `_search` API to retrieve the fields you want: +You can now retrieve calculated values in a search query, and find documents based on precise values. The following range query returns all documents where the calculated `voltage_corrected` is greater than or equal to `16`, but less than or equal to `20`. Again, use the [`fields`](elasticsearch://reference/elasticsearch/rest-apis/retrieve-selected-fields.md) parameter on the `_search` API to retrieve the fields you want: ```console POST my-index-000001/_search diff --git a/manage-data/data-store/mapping/map-runtime-field.md b/manage-data/data-store/mapping/map-runtime-field.md index 68bf74781..9d56e5418 100644 --- a/manage-data/data-store/mapping/map-runtime-field.md +++ b/manage-data/data-store/mapping/map-runtime-field.md @@ -11,7 +11,7 @@ applies_to: You map runtime fields by adding a `runtime` section under the mapping definition and defining [a Painless script](../../../explore-analyze/scripting/modules-scripting-using.md). This script has access to the entire context of a document, including the original `_source` via `params._source` and any mapped fields plus their values. At query time, the script runs and generates values for each scripted field that is required for the query. ::::{admonition} Emitting runtime field values -When defining a Painless script to use with runtime fields, you must include the [`emit` method](asciidocalypse://docs/elasticsearch/docs/reference/scripting-languages/painless/painless-runtime-fields-context.md) to emit calculated values. +When defining a Painless script to use with runtime fields, you must include the [`emit` method](elasticsearch://reference/scripting-languages/painless/painless-runtime-fields-context.md) to emit calculated values. :::: @@ -49,7 +49,7 @@ The `runtime` section can be any of these data types: * `long` * [`lookup`](retrieve-runtime-field.md#lookup-runtime-fields) -Runtime fields with a `type` of `date` can accept the [`format`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-date-format.md) parameter exactly as the `date` field type. +Runtime fields with a `type` of `date` can accept the [`format`](elasticsearch://reference/elasticsearch/mapping-reference/mapping-date-format.md) parameter exactly as the `date` field type. Runtime fields with a `type` of `lookup` allow retrieving fields from related indices. See [`retrieve fields from related indices`](retrieve-runtime-field.md#lookup-runtime-fields). @@ -88,11 +88,11 @@ PUT my-index-000001/ When no script is provided, {{es}} implicitly looks in `_source` at query time for a field with the same name as the runtime field, and returns a value if one exists. If a field with the same name doesn’t exist, the response doesn’t include any values for that runtime field. -In most cases, retrieve field values through [`doc_values`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/doc-values.md) whenever possible. Accessing `doc_values` with a runtime field is faster than retrieving values from `_source` because of how data is loaded from Lucene. +In most cases, retrieve field values through [`doc_values`](elasticsearch://reference/elasticsearch/mapping-reference/doc-values.md) whenever possible. Accessing `doc_values` with a runtime field is faster than retrieving values from `_source` because of how data is loaded from Lucene. However, there are cases where retrieving fields from `_source` is necessary. For example, `text` fields do not have `doc_values` available by default, so you have to retrieve values from `_source`. In other instances, you might choose to disable `doc_values` on a specific field. -::::{note} +::::{note} You can alternatively prefix the field you want to retrieve values for with `params._source` (such as `params._source.day_of_week`). For simplicity, defining a runtime field in the mapping definition without a script is the recommended option, whenever possible. :::: @@ -119,7 +119,7 @@ PUT my-index-000001/_mapping :::::{admonition} Downstream impacts Updating or removing a runtime field while a dependent query is running can return inconsistent results. Each shard might have access to different versions of the script, depending on when the mapping change takes effect. -::::{warning} +::::{warning} Existing queries or visualizations in {{kib}} that rely on runtime fields can fail if you remove or update the field. For example, a bar chart visualization that uses a runtime field of type `ip` will fail if the type is changed to `boolean`, or if the runtime field is removed. :::: diff --git a/manage-data/data-store/mapping/override-field-values-at-query-time.md b/manage-data/data-store/mapping/override-field-values-at-query-time.md index 1b434ba67..f703548aa 100644 --- a/manage-data/data-store/mapping/override-field-values-at-query-time.md +++ b/manage-data/data-store/mapping/override-field-values-at-query-time.md @@ -86,7 +86,7 @@ The response includes indexed values for documents matching model number `HG537P The following request defines a runtime field where the script evaluates the `model_number` field where the value is `HG537PU`. For each match, the script multiplies the value for the `voltage` field by `1.7`. -Using the [`fields`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/retrieve-selected-fields.md) parameter on the `_search` API, you can retrieve the value that the script calculates for the `measures.voltage` field for documents matching the search request: +Using the [`fields`](elasticsearch://reference/elasticsearch/rest-apis/retrieve-selected-fields.md) parameter on the `_search` API, you can retrieve the value that the script calculates for the `measures.voltage` field for documents matching the search request: ```console POST my-index-000001/_search diff --git a/manage-data/data-store/mapping/retrieve-runtime-field.md b/manage-data/data-store/mapping/retrieve-runtime-field.md index 2df1c243c..a6da515e0 100644 --- a/manage-data/data-store/mapping/retrieve-runtime-field.md +++ b/manage-data/data-store/mapping/retrieve-runtime-field.md @@ -8,7 +8,7 @@ applies_to: # Retrieve a runtime field [runtime-retrieving-fields] -Use the [`fields`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/retrieve-selected-fields.md) parameter on the `_search` API to retrieve the values of runtime fields. Runtime fields won’t display in `_source`, but the `fields` API works for all fields, even those that were not sent as part of the original `_source`. +Use the [`fields`](elasticsearch://reference/elasticsearch/rest-apis/retrieve-selected-fields.md) parameter on the `_search` API to retrieve the values of runtime fields. Runtime fields won’t display in `_source`, but the `fields` API works for all fields, even those that were not sent as part of the original `_source`. ## Define a runtime field to calculate the day of week [runtime-define-field-dayofweek] @@ -155,7 +155,7 @@ This time, the response includes only two hits. The value for `day_of_week` (`Su ## Retrieve fields from related indices [lookup-runtime-fields] -The [`fields`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/retrieve-selected-fields.md) parameter on the `_search` API can also be used to retrieve fields from the related indices via runtime fields with a type of `lookup`. +The [`fields`](elasticsearch://reference/elasticsearch/rest-apis/retrieve-selected-fields.md) parameter on the `_search` API can also be used to retrieve fields from the related indices via runtime fields with a type of `lookup`. ::::{note} Fields that are retrieved by runtime fields of type `lookup` can be used to enrich the hits in a search response. It’s not possible to query or aggregate on these fields. @@ -202,11 +202,11 @@ POST logs/_search } ``` -1. Define a runtime field in the main search request with a type of `lookup` that retrieves fields from the target index using the [`term`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-term-query.md) queries. +1. Define a runtime field in the main search request with a type of `lookup` that retrieves fields from the target index using the [`term`](elasticsearch://reference/query-languages/query-dsl-term-query.md) queries. 2. The target index where the lookup query executes against 3. A field on the main index whose values are used as the input values of the lookup term query 4. A field on the lookup index which the lookup query searches against -5. A list of fields to retrieve from the lookup index. See the [`fields`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/retrieve-selected-fields.md) parameter of a search request. +5. A list of fields to retrieve from the lookup index. See the [`fields`](elasticsearch://reference/elasticsearch/rest-apis/retrieve-selected-fields.md) parameter of a search request. The above search returns the country and city from the `ip_location` index for each ip address of the returned search hits. diff --git a/manage-data/data-store/mapping/runtime-fields.md b/manage-data/data-store/mapping/runtime-fields.md index e0e24661e..7e86a09b0 100644 --- a/manage-data/data-store/mapping/runtime-fields.md +++ b/manage-data/data-store/mapping/runtime-fields.md @@ -17,12 +17,12 @@ A *runtime field* is a field that is evaluated at query time. Runtime fields ena You access runtime fields from the search API like any other field, and {{es}} sees runtime fields no differently. You can define runtime fields in the [index mapping](map-runtime-field.md) or in the [search request](define-runtime-fields-in-search-request.md). Your choice, which is part of the inherent flexibility of runtime fields. -Use the [`fields`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/retrieve-selected-fields.md) parameter on the `_search` API to [retrieve the values of runtime fields](retrieve-runtime-field.md). Runtime fields won’t display in `_source`, but the `fields` API works for all fields, even those that were not sent as part of the original `_source`. +Use the [`fields`](elasticsearch://reference/elasticsearch/rest-apis/retrieve-selected-fields.md) parameter on the `_search` API to [retrieve the values of runtime fields](retrieve-runtime-field.md). Runtime fields won’t display in `_source`, but the `fields` API works for all fields, even those that were not sent as part of the original `_source`. Runtime fields are useful when working with log data (see [examples](explore-data-with-runtime-fields.md)), especially when you’re unsure about the data structure. Your search speed decreases, but your index size is much smaller and you can more quickly process logs without having to index them. -## Benefits [runtime-benefits] +## Benefits [runtime-benefits] Because runtime fields aren’t indexed, adding a runtime field doesn’t increase the index size. You define runtime fields directly in the index mapping, saving storage costs and increasing ingestion speed. You can more quickly ingest data into the Elastic Stack and access it right away. When you define a runtime field, you can immediately use it in search requests, aggregations, filtering, and sorting. @@ -31,20 +31,20 @@ If you change a runtime field into an indexed field, you don’t need to modify At its core, the most important benefit of runtime fields is the ability to add fields to documents after you’ve ingested them. This capability simplifies mapping decisions because you don’t have to decide how to parse your data up front, and can use runtime fields to amend the mapping at any time. Using runtime fields allows for a smaller index and faster ingest time, which combined use less resources and reduce your operating costs. -## Incentives [runtime-incentives] +## Incentives [runtime-incentives] Runtime fields can replace many of the ways you can use scripting with the `_search` API. How you use a runtime field is impacted by the number of documents that the included script runs against. For example, if you’re using the `fields` parameter on the `_search` API to [retrieve the values of a runtime field](retrieve-runtime-field.md), the script runs only against the top hits just like script fields do. -You can use [script fields](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/retrieve-selected-fields.md#script-fields) to access values in `_source` and return calculated values based on a script valuation. Runtime fields have the same capabilities, but provide greater flexibility because you can query and aggregate on runtime fields in a search request. Script fields can only fetch values. +You can use [script fields](elasticsearch://reference/elasticsearch/rest-apis/retrieve-selected-fields.md#script-fields) to access values in `_source` and return calculated values based on a script valuation. Runtime fields have the same capabilities, but provide greater flexibility because you can query and aggregate on runtime fields in a search request. Script fields can only fetch values. -Similarly, you could write a [script query](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-script-query.md) that filters documents in a search request based on a script. Runtime fields provide a very similar feature that is more flexible. You write a script to create field values and they are available everywhere, such as [`fields`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/retrieve-selected-fields.md), [all queries](../../../explore-analyze/query-filter/languages/querydsl.md), and [aggregations](../../../explore-analyze/query-filter/aggregations.md). +Similarly, you could write a [script query](elasticsearch://reference/query-languages/query-dsl-script-query.md) that filters documents in a search request based on a script. Runtime fields provide a very similar feature that is more flexible. You write a script to create field values and they are available everywhere, such as [`fields`](elasticsearch://reference/elasticsearch/rest-apis/retrieve-selected-fields.md), [all queries](../../../explore-analyze/query-filter/languages/querydsl.md), and [aggregations](../../../explore-analyze/query-filter/aggregations.md). -You can also use scripts to [sort search results](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/sort-search-results.md#script-based-sorting), but that same script works exactly the same in a runtime field. +You can also use scripts to [sort search results](elasticsearch://reference/elasticsearch/rest-apis/sort-search-results.md#script-based-sorting), but that same script works exactly the same in a runtime field. If you move a script from any of these sections in a search request to a runtime field that is computing values from the same number of documents, the performance should be about the same. The performance for these features is largely dependent upon the calculations that the included script is running and how many documents the script runs against. -## Compromises [runtime-compromises] +## Compromises [runtime-compromises] Runtime fields use less disk space and provide flexibility in how you access your data, but can impact search performance based on the computation defined in the runtime script. @@ -52,7 +52,7 @@ To balance search performance and flexibility, index fields that you’ll freque Use the [asynchronous search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-async-search-submit) to run searches that include runtime fields. This method of search helps to offset the performance impacts of computing values for runtime fields in each document containing that field. If the query can’t return the result set synchronously, you’ll get results asynchronously as they become available. -::::{important} +::::{important} Queries against runtime fields are considered expensive. If [`search.allow_expensive_queries`](../../../explore-analyze/query-filter/languages/querydsl.md#query-dsl-allow-expensive-queries) is set to `false`, expensive queries are not allowed and {{es}} will reject any queries against runtime fields. :::: diff --git a/manage-data/data-store/near-real-time-search.md b/manage-data/data-store/near-real-time-search.md index 57b3fc8bd..f57309e9e 100644 --- a/manage-data/data-store/near-real-time-search.md +++ b/manage-data/data-store/near-real-time-search.md @@ -30,7 +30,7 @@ Lucene allows new segments to be written and opened, making the documents they c In {{es}}, this process of writing and opening a new segment is called a *refresh*. A refresh makes all operations performed on an index since the last refresh available for search. You can control refreshes through the following means: * Waiting for the refresh interval -* Setting the [?refresh](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/refresh-parameter.md) option +* Setting the [?refresh](elasticsearch://reference/elasticsearch/rest-apis/refresh-parameter.md) option * Using the [Refresh API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-refresh) to explicitly complete a refresh (`POST _refresh`) By default, {{es}} periodically refreshes indices every second, but only on indices that have received one search request or more in the last 30 seconds. This is why we say that {{es}} has *near* real-time search: document changes are not visible to search immediately, but will become visible within this timeframe. diff --git a/manage-data/data-store/templates.md b/manage-data/data-store/templates.md index 109334157..1f57cf775 100644 --- a/manage-data/data-store/templates.md +++ b/manage-data/data-store/templates.md @@ -37,11 +37,11 @@ The following conditions apply to index templates: * `profiling-*` * `security_solution-*-*` -[{{agent}}](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/index.md) uses these templates to create data streams. Index templates created by {{fleet}} integrations use similar overlapping index patterns and have a priority up to `200`. +[{{agent}}](/reference/ingestion-tools/fleet/index.md) uses these templates to create data streams. Index templates created by {{fleet}} integrations use similar overlapping index patterns and have a priority up to `200`. If you use {{fleet}} or {{agent}}, assign your index templates a priority lower than `100` to avoid overriding these templates. Otherwise, to avoid accidentally applying the templates, do one or more of the following: -* To disable all built-in index and component templates, set [`stack.templates.enabled`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/index-management-settings.md#stack-templates-enabled) to `false` using the [cluster update settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings). Note, however, that this is not recommended, see the [setting documentation](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/index-management-settings.md#stack-templates-enabled) for more information. +* To disable all built-in index and component templates, set [`stack.templates.enabled`](elasticsearch://reference/elasticsearch/configuration-reference/index-management-settings.md#stack-templates-enabled) to `false` using the [cluster update settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings). Note, however, that this is not recommended, see the [setting documentation](elasticsearch://reference/elasticsearch/configuration-reference/index-management-settings.md#stack-templates-enabled) for more information. * Use a non-overlapping index pattern. * Assign templates with an overlapping pattern a `priority` higher than `500`. For example, if you don’t use {{fleet}} or {{agent}} and want to create a template for the `logs-*` index pattern, assign your template a priority of `500`. This ensures your template is applied instead of the built-in template for `logs-*-*`. * To avoid naming collisions with built-in and Fleet-managed index templates, avoid using `@` as part of the name of your own index templates. diff --git a/manage-data/data-store/templates/index-template-management.md b/manage-data/data-store/templates/index-template-management.md index e599b9376..666239ed4 100644 --- a/manage-data/data-store/templates/index-template-management.md +++ b/manage-data/data-store/templates/index-template-management.md @@ -51,7 +51,7 @@ In this tutorial, you’ll create an index template and use it to configure two ::: 2. Define index settings. These are optional. For this tutorial, leave this section blank. -3. Define a mapping that contains an [object](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/object.md) field named `geo` with a child [`geo_point`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/geo-point.md) field named `coordinates`: +3. Define a mapping that contains an [object](elasticsearch://reference/elasticsearch/mapping-reference/object.md) field named `geo` with a child [`geo_point`](elasticsearch://reference/elasticsearch/mapping-reference/geo-point.md) field named `coordinates`: :::{image} ../../../images/elasticsearch-reference-management-index-templates-mappings.png :alt: Mapped fields page diff --git a/manage-data/data-store/text-analysis.md b/manage-data/data-store/text-analysis.md index 968264ab4..d93b3a986 100644 --- a/manage-data/data-store/text-analysis.md +++ b/manage-data/data-store/text-analysis.md @@ -14,7 +14,7 @@ _Text analysis_ is the process of converting unstructured text, like the body of Text analysis enables {{es}} to perform full-text search, where the search returns all *relevant* results rather than just exact matches. For example, if you search for `Quick fox jumps`, you probably want the document that contains `A quick brown fox jumps over the lazy dog`, and you might also want documents that contain related words like `fast fox` or `foxes leap`. -{{es}} performs text analysis when indexing or searching [`text`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/text.md) fields. If your index does _not_ contain `text` fields, no further setup is needed; you can skip the pages in this section. If you _do_ use `text` fields or your text searches aren’t returning results as expected, configuring text analysis can often help. You should also look into analysis configuration if you’re using {{es}} to: +{{es}} performs text analysis when indexing or searching [`text`](elasticsearch://reference/elasticsearch/mapping-reference/text.md) fields. If your index does _not_ contain `text` fields, no further setup is needed; you can skip the pages in this section. If you _do_ use `text` fields or your text searches aren’t returning results as expected, configuring text analysis can often help. You should also look into analysis configuration if you’re using {{es}} to: * Build a search engine * Mine unstructured data @@ -47,9 +47,9 @@ To ensure search terms match these words as intended, you can apply the same tok Text analysis is performed by an [*analyzer*](/manage-data/data-store/text-analysis/anatomy-of-an-analyzer.md), a set of rules that govern the entire process. -{{es}} includes a default analyzer, called the [standard analyzer](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-standard-analyzer.md), which works well for most use cases right out of the box. +{{es}} includes a default analyzer, called the [standard analyzer](elasticsearch://reference/data-analysis/text-analysis/analysis-standard-analyzer.md), which works well for most use cases right out of the box. -If you want to tailor your search experience, you can choose a different [built-in analyzer](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analyzer-reference.md) or even [configure a custom one](/manage-data/data-store/text-analysis/create-custom-analyzer.md). A custom analyzer gives you control over each step of the analysis process, including: +If you want to tailor your search experience, you can choose a different [built-in analyzer](elasticsearch://reference/data-analysis/text-analysis/analyzer-reference.md) or even [configure a custom one](/manage-data/data-store/text-analysis/create-custom-analyzer.md). A custom analyzer gives you control over each step of the analysis process, including: * Changes to the text *before* tokenization * How text is converted to tokens diff --git a/manage-data/data-store/text-analysis/anatomy-of-an-analyzer.md b/manage-data/data-store/text-analysis/anatomy-of-an-analyzer.md index c16327d3d..968175199 100644 --- a/manage-data/data-store/text-analysis/anatomy-of-an-analyzer.md +++ b/manage-data/data-store/text-analysis/anatomy-of-an-analyzer.md @@ -10,30 +10,30 @@ applies_to: An *analyzer*  — whether built-in or custom — is just a package which contains three lower-level building blocks: *character filters*, *tokenizers*, and *token filters*. -The built-in [analyzers](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analyzer-reference.md) pre-package these building blocks into analyzers suitable for different languages and types of text. Elasticsearch also exposes the individual building blocks so that they can be combined to define new [`custom`](create-custom-analyzer.md) analyzers. +The built-in [analyzers](elasticsearch://reference/data-analysis/text-analysis/analyzer-reference.md) pre-package these building blocks into analyzers suitable for different languages and types of text. Elasticsearch also exposes the individual building blocks so that they can be combined to define new [`custom`](create-custom-analyzer.md) analyzers. ## Character filters [analyzer-anatomy-character-filters] A *character filter* receives the original text as a stream of characters and can transform the stream by adding, removing, or changing characters. For instance, a character filter could be used to convert Hindu-Arabic numerals (٠‎١٢٣٤٥٦٧٨‎٩‎) into their Arabic-Latin equivalents (0123456789), or to strip HTML elements like `` from the stream. -An analyzer may have **zero or more** [character filters](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/character-filter-reference.md), which are applied in order. +An analyzer may have **zero or more** [character filters](elasticsearch://reference/data-analysis/text-analysis/character-filter-reference.md), which are applied in order. ## Tokenizer [analyzer-anatomy-tokenizer] -A *tokenizer* receives a stream of characters, breaks it up into individual *tokens* (usually individual words), and outputs a stream of *tokens*. For instance, a [`whitespace`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-whitespace-tokenizer.md) tokenizer breaks text into tokens whenever it sees any whitespace. It would convert the text `"Quick brown fox!"` into the terms `[Quick, brown, fox!]`. +A *tokenizer* receives a stream of characters, breaks it up into individual *tokens* (usually individual words), and outputs a stream of *tokens*. For instance, a [`whitespace`](elasticsearch://reference/data-analysis/text-analysis/analysis-whitespace-tokenizer.md) tokenizer breaks text into tokens whenever it sees any whitespace. It would convert the text `"Quick brown fox!"` into the terms `[Quick, brown, fox!]`. The tokenizer is also responsible for recording the order or *position* of each term and the start and end *character offsets* of the original word which the term represents. -An analyzer must have **exactly one** [tokenizer](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/tokenizer-reference.md). +An analyzer must have **exactly one** [tokenizer](elasticsearch://reference/data-analysis/text-analysis/tokenizer-reference.md). ## Token filters [analyzer-anatomy-token-filters] -A *token filter* receives the token stream and may add, remove, or change tokens. For example, a [`lowercase`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-lowercase-tokenfilter.md) token filter converts all tokens to lowercase, a [`stop`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-stop-tokenfilter.md) token filter removes common words (*stop words*) like `the` from the token stream, and a [`synonym`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-synonym-tokenfilter.md) token filter introduces synonyms into the token stream. +A *token filter* receives the token stream and may add, remove, or change tokens. For example, a [`lowercase`](elasticsearch://reference/data-analysis/text-analysis/analysis-lowercase-tokenfilter.md) token filter converts all tokens to lowercase, a [`stop`](elasticsearch://reference/data-analysis/text-analysis/analysis-stop-tokenfilter.md) token filter removes common words (*stop words*) like `the` from the token stream, and a [`synonym`](elasticsearch://reference/data-analysis/text-analysis/analysis-synonym-tokenfilter.md) token filter introduces synonyms into the token stream. Token filters are not allowed to change the position or character offsets of each token. -An analyzer may have **zero or more** [token filters](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/token-filter-reference.md), which are applied in order. +An analyzer may have **zero or more** [token filters](elasticsearch://reference/data-analysis/text-analysis/token-filter-reference.md), which are applied in order. diff --git a/manage-data/data-store/text-analysis/configure-text-analysis.md b/manage-data/data-store/text-analysis/configure-text-analysis.md index 6537fdf6d..c2672633c 100644 --- a/manage-data/data-store/text-analysis/configure-text-analysis.md +++ b/manage-data/data-store/text-analysis/configure-text-analysis.md @@ -8,9 +8,9 @@ applies_to: # Configure text analysis [configure-text-analysis] -By default, {{es}} uses the [`standard` analyzer](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-standard-analyzer.md) for all text analysis. The `standard` analyzer gives you out-of-the-box support for most natural languages and use cases. If you chose to use the `standard` analyzer as-is, no further configuration is needed. +By default, {{es}} uses the [`standard` analyzer](elasticsearch://reference/data-analysis/text-analysis/analysis-standard-analyzer.md) for all text analysis. The `standard` analyzer gives you out-of-the-box support for most natural languages and use cases. If you chose to use the `standard` analyzer as-is, no further configuration is needed. -If the standard analyzer does not fit your needs, review and test {{es}}'s other built-in [built-in analyzers](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analyzer-reference.md). Built-in analyzers don’t require configuration, but some support options that can be used to adjust their behavior. For example, you can configure the `standard` analyzer with a list of custom stop words to remove. +If the standard analyzer does not fit your needs, review and test {{es}}'s other built-in [built-in analyzers](elasticsearch://reference/data-analysis/text-analysis/analyzer-reference.md). Built-in analyzers don’t require configuration, but some support options that can be used to adjust their behavior. For example, you can configure the `standard` analyzer with a list of custom stop words to remove. If no built-in analyzer fits your needs, you can test and create a custom analyzer. Custom analyzers involve selecting and combining different [analyzer components](anatomy-of-an-analyzer.md), giving you greater control over the process. diff --git a/manage-data/data-store/text-analysis/configuring-built-in-analyzers.md b/manage-data/data-store/text-analysis/configuring-built-in-analyzers.md index de6cb3bea..a93a9177b 100644 --- a/manage-data/data-store/text-analysis/configuring-built-in-analyzers.md +++ b/manage-data/data-store/text-analysis/configuring-built-in-analyzers.md @@ -8,7 +8,7 @@ applies_to: # Configuring built-in analyzers [configuring-analyzers] -The built-in analyzers can be used directly without any configuration. Some of them, however, support configuration options to alter their behaviour. For instance, the [`standard` analyzer](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-standard-analyzer.md) can be configured to support a list of stop words: +The built-in analyzers can be used directly without any configuration. Some of them, however, support configuration options to alter their behaviour. For instance, the [`standard` analyzer](elasticsearch://reference/data-analysis/text-analysis/analysis-standard-analyzer.md) can be configured to support a list of stop words: ```console PUT my-index-000001 diff --git a/manage-data/data-store/text-analysis/create-custom-analyzer.md b/manage-data/data-store/text-analysis/create-custom-analyzer.md index 9be4da578..0fd42f661 100644 --- a/manage-data/data-store/text-analysis/create-custom-analyzer.md +++ b/manage-data/data-store/text-analysis/create-custom-analyzer.md @@ -10,46 +10,46 @@ applies_to: When the built-in analyzers do not fulfill your needs, you can create a `custom` analyzer which uses the appropriate combination of: -* zero or more [character filters](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/character-filter-reference.md) -* a [tokenizer](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/tokenizer-reference.md) -* zero or more [token filters](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/token-filter-reference.md). +* zero or more [character filters](elasticsearch://reference/data-analysis/text-analysis/character-filter-reference.md) +* a [tokenizer](elasticsearch://reference/data-analysis/text-analysis/tokenizer-reference.md) +* zero or more [token filters](elasticsearch://reference/data-analysis/text-analysis/token-filter-reference.md). -## Configuration [_configuration] +## Configuration [_configuration] The `custom` analyzer accepts the following parameters: `type` -: Analyzer type. Accepts [built-in analyzer types](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analyzer-reference.md). For custom analyzers, use `custom` or omit this parameter. +: Analyzer type. Accepts [built-in analyzer types](elasticsearch://reference/data-analysis/text-analysis/analyzer-reference.md). For custom analyzers, use `custom` or omit this parameter. `tokenizer` -: A built-in or customised [tokenizer](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/tokenizer-reference.md). (Required) +: A built-in or customised [tokenizer](elasticsearch://reference/data-analysis/text-analysis/tokenizer-reference.md). (Required) `char_filter` -: An optional array of built-in or customised [character filters](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/character-filter-reference.md). +: An optional array of built-in or customised [character filters](elasticsearch://reference/data-analysis/text-analysis/character-filter-reference.md). `filter` -: An optional array of built-in or customised [token filters](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/token-filter-reference.md). +: An optional array of built-in or customised [token filters](elasticsearch://reference/data-analysis/text-analysis/token-filter-reference.md). `position_increment_gap` -: When indexing an array of text values, Elasticsearch inserts a fake "gap" between the last term of one value and the first term of the next value to ensure that a phrase query doesn’t match two terms from different array elements. Defaults to `100`. See [`position_increment_gap`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/position-increment-gap.md) for more. +: When indexing an array of text values, Elasticsearch inserts a fake "gap" between the last term of one value and the first term of the next value to ensure that a phrase query doesn’t match two terms from different array elements. Defaults to `100`. See [`position_increment_gap`](elasticsearch://reference/elasticsearch/mapping-reference/position-increment-gap.md) for more. -## Example configuration [_example_configuration] +## Example configuration [_example_configuration] Here is an example that combines the following: Character Filter -: * [HTML Strip Character Filter](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-htmlstrip-charfilter.md) +: * [HTML Strip Character Filter](elasticsearch://reference/data-analysis/text-analysis/analysis-htmlstrip-charfilter.md) Tokenizer -: * [Standard Tokenizer](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-standard-tokenizer.md) +: * [Standard Tokenizer](elasticsearch://reference/data-analysis/text-analysis/analysis-standard-tokenizer.md) Token Filters -: * [Lowercase Token Filter](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-lowercase-tokenfilter.md) -* [ASCII-Folding Token Filter](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-asciifolding-tokenfilter.md) +: * [Lowercase Token Filter](elasticsearch://reference/data-analysis/text-analysis/analysis-lowercase-tokenfilter.md) +* [ASCII-Folding Token Filter](elasticsearch://reference/data-analysis/text-analysis/analysis-asciifolding-tokenfilter.md) ```console @@ -95,16 +95,16 @@ The previous example used tokenizer, token filters, and character filters with t Here is a more complicated example that combines the following: Character Filter -: * [Mapping Character Filter](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-mapping-charfilter.md), configured to replace `:)` with `_happy_` and `:(` with `_sad_` +: * [Mapping Character Filter](elasticsearch://reference/data-analysis/text-analysis/analysis-mapping-charfilter.md), configured to replace `:)` with `_happy_` and `:(` with `_sad_` Tokenizer -: * [Pattern Tokenizer](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-pattern-tokenizer.md), configured to split on punctuation characters +: * [Pattern Tokenizer](elasticsearch://reference/data-analysis/text-analysis/analysis-pattern-tokenizer.md), configured to split on punctuation characters Token Filters -: * [Lowercase Token Filter](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-lowercase-tokenfilter.md) -* [Stop Token Filter](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-stop-tokenfilter.md), configured to use the pre-defined list of English stop words +: * [Lowercase Token Filter](elasticsearch://reference/data-analysis/text-analysis/analysis-lowercase-tokenfilter.md) +* [Stop Token Filter](elasticsearch://reference/data-analysis/text-analysis/analysis-stop-tokenfilter.md), configured to use the pre-defined list of English stop words Here is an example: diff --git a/manage-data/data-store/text-analysis/index-search-analysis.md b/manage-data/data-store/text-analysis/index-search-analysis.md index 9c023e5e8..e7834e1d0 100644 --- a/manage-data/data-store/text-analysis/index-search-analysis.md +++ b/manage-data/data-store/text-analysis/index-search-analysis.md @@ -11,10 +11,10 @@ applies_to: Text analysis occurs at two times: Index time -: When a document is indexed, any [`text`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/text.md) field values are analyzed. +: When a document is indexed, any [`text`](elasticsearch://reference/elasticsearch/mapping-reference/text.md) field values are analyzed. Search time -: When running a [full-text search](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/full-text-queries.md) on a `text` field, the query string (the text the user is searching for) is analyzed. Search time is also called *query time*. +: When running a [full-text search](elasticsearch://reference/query-languages/full-text-queries.md) on a `text` field, the query string (the text the user is searching for) is analyzed. Search time is also called *query time*. For more details on text analysis at search time, refer to [Text analysis during search](/solutions/search/full-text/text-analysis-during-search.md). diff --git a/manage-data/data-store/text-analysis/specify-an-analyzer.md b/manage-data/data-store/text-analysis/specify-an-analyzer.md index aa9a08375..44553bf8d 100644 --- a/manage-data/data-store/text-analysis/specify-an-analyzer.md +++ b/manage-data/data-store/text-analysis/specify-an-analyzer.md @@ -31,15 +31,15 @@ If you don’t typically create mappings for your indices, you can use [index te {{es}} determines which index analyzer to use by checking the following parameters in order: -1. The [`analyzer`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/analyzer.md) mapping parameter for the field. See [Specify the analyzer for a field](#specify-index-field-analyzer). +1. The [`analyzer`](elasticsearch://reference/elasticsearch/mapping-reference/analyzer.md) mapping parameter for the field. See [Specify the analyzer for a field](#specify-index-field-analyzer). 2. The `analysis.analyzer.default` index setting. See [Specify the default analyzer for an index](#specify-index-time-default-analyzer). -If none of these parameters are specified, the [`standard` analyzer](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-standard-analyzer.md) is used. +If none of these parameters are specified, the [`standard` analyzer](elasticsearch://reference/data-analysis/text-analysis/analysis-standard-analyzer.md) is used. ## Specify the analyzer for a field [specify-index-field-analyzer] -When mapping an index, you can use the [`analyzer`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/analyzer.md) mapping parameter to specify an analyzer for each `text` field. +When mapping an index, you can use the [`analyzer`](elasticsearch://reference/elasticsearch/mapping-reference/analyzer.md) mapping parameter to specify an analyzer for each `text` field. The following [create index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-create) request sets the `whitespace` analyzer as the analyzer for the `title` field. @@ -92,19 +92,19 @@ If you choose to specify a separate search analyzer, we recommend you thoroughly At search time, {{es}} determines which analyzer to use by checking the following parameters in order: -1. The [`analyzer`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/analyzer.md) parameter in the search query. See [Specify the search analyzer for a query](#specify-search-query-analyzer). -2. The [`search_analyzer`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/search-analyzer.md) mapping parameter for the field. See [Specify the search analyzer for a field](#specify-search-field-analyzer). +1. The [`analyzer`](elasticsearch://reference/elasticsearch/mapping-reference/analyzer.md) parameter in the search query. See [Specify the search analyzer for a query](#specify-search-query-analyzer). +2. The [`search_analyzer`](elasticsearch://reference/elasticsearch/mapping-reference/search-analyzer.md) mapping parameter for the field. See [Specify the search analyzer for a field](#specify-search-field-analyzer). 3. The `analysis.analyzer.default_search` index setting. See [Specify the default search analyzer for an index](#specify-search-default-analyzer). -4. The [`analyzer`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/analyzer.md) mapping parameter for the field. See [Specify the analyzer for a field](#specify-index-field-analyzer). +4. The [`analyzer`](elasticsearch://reference/elasticsearch/mapping-reference/analyzer.md) mapping parameter for the field. See [Specify the analyzer for a field](#specify-index-field-analyzer). -If none of these parameters are specified, the [`standard` analyzer](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-standard-analyzer.md) is used. +If none of these parameters are specified, the [`standard` analyzer](elasticsearch://reference/data-analysis/text-analysis/analysis-standard-analyzer.md) is used. ## Specify the search analyzer for a query [specify-search-query-analyzer] -When writing a [full-text query](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/full-text-queries.md), you can use the `analyzer` parameter to specify a search analyzer. If provided, this overrides any other search analyzers. +When writing a [full-text query](elasticsearch://reference/query-languages/full-text-queries.md), you can use the `analyzer` parameter to specify a search analyzer. If provided, this overrides any other search analyzers. -The following [search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search) request sets the `stop` analyzer as the search analyzer for a [`match`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-match-query.md) query. +The following [search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search) request sets the `stop` analyzer as the search analyzer for a [`match`](elasticsearch://reference/query-languages/query-dsl-match-query.md) query. ```console GET my-index-000001/_search @@ -123,7 +123,7 @@ GET my-index-000001/_search ## Specify the search analyzer for a field [specify-search-field-analyzer] -When mapping an index, you can use the [`search_analyzer`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/analyzer.md) mapping parameter to specify a search analyzer for each `text` field. +When mapping an index, you can use the [`search_analyzer`](elasticsearch://reference/elasticsearch/mapping-reference/analyzer.md) mapping parameter to specify a search analyzer for each `text` field. If a search analyzer is provided, the index analyzer must also be specified using the `analyzer` parameter. diff --git a/manage-data/data-store/text-analysis/stemming.md b/manage-data/data-store/text-analysis/stemming.md index 88ab114ca..d00720aed 100644 --- a/manage-data/data-store/text-analysis/stemming.md +++ b/manage-data/data-store/text-analysis/stemming.md @@ -44,10 +44,10 @@ However, most algorithmic stemmers only alter the existing text of a word. This The following token filters use algorithmic stemming: -* [`stemmer`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-stemmer-tokenfilter.md), which provides algorithmic stemming for several languages, some with additional variants. -* [`kstem`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-kstem-tokenfilter.md), a stemmer for English that combines algorithmic stemming with a built-in dictionary. -* [`porter_stem`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-porterstem-tokenfilter.md), our recommended algorithmic stemmer for English. -* [`snowball`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-snowball-tokenfilter.md), which uses [Snowball](https://snowballstem.org/)-based stemming rules for several languages. +* [`stemmer`](elasticsearch://reference/data-analysis/text-analysis/analysis-stemmer-tokenfilter.md), which provides algorithmic stemming for several languages, some with additional variants. +* [`kstem`](elasticsearch://reference/data-analysis/text-analysis/analysis-kstem-tokenfilter.md), a stemmer for English that combines algorithmic stemming with a built-in dictionary. +* [`porter_stem`](elasticsearch://reference/data-analysis/text-analysis/analysis-porterstem-tokenfilter.md), our recommended algorithmic stemmer for English. +* [`snowball`](elasticsearch://reference/data-analysis/text-analysis/analysis-snowball-tokenfilter.md), which uses [Snowball](https://snowballstem.org/)-based stemming rules for several languages. ## Dictionary stemmers [dictionary-stemmers] @@ -68,10 +68,10 @@ In practice, algorithmic stemmers typically outperform dictionary stemmers. This * **Dictionary quality**
A dictionary stemmer is only as good as its dictionary. To work well, these dictionaries must include a significant number of words, be updated regularly, and change with language trends. Often, by the time a dictionary has been made available, it’s incomplete and some of its entries are already outdated. * **Size and performance**
Dictionary stemmers must load all words, prefixes, and suffixes from its dictionary into memory. This can use a significant amount of RAM. Low-quality dictionaries may also be less efficient with prefix and suffix removal, which can slow the stemming process significantly. -You can use the [`hunspell`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-hunspell-tokenfilter.md) token filter to perform dictionary stemming. +You can use the [`hunspell`](elasticsearch://reference/data-analysis/text-analysis/analysis-hunspell-tokenfilter.md) token filter to perform dictionary stemming. -::::{tip} -If available, we recommend trying an algorithmic stemmer for your language before using the [`hunspell`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-hunspell-tokenfilter.md) token filter. +::::{tip} +If available, we recommend trying an algorithmic stemmer for your language before using the [`hunspell`](elasticsearch://reference/data-analysis/text-analysis/analysis-hunspell-tokenfilter.md) token filter. :::: @@ -83,10 +83,10 @@ Sometimes stemming can produce shared root words that are spelled similarly but To prevent this and better control stemming, you can use the following token filters: -* [`stemmer_override`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-stemmer-override-tokenfilter.md), which lets you define rules for stemming specific tokens. -* [`keyword_marker`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-keyword-marker-tokenfilter.md), which marks specified tokens as keywords. Keyword tokens are not stemmed by subsequent stemmer token filters. -* [`conditional`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-condition-tokenfilter.md), which can be used to mark tokens as keywords, similar to the `keyword_marker` filter. +* [`stemmer_override`](elasticsearch://reference/data-analysis/text-analysis/analysis-stemmer-override-tokenfilter.md), which lets you define rules for stemming specific tokens. +* [`keyword_marker`](elasticsearch://reference/data-analysis/text-analysis/analysis-keyword-marker-tokenfilter.md), which marks specified tokens as keywords. Keyword tokens are not stemmed by subsequent stemmer token filters. +* [`conditional`](elasticsearch://reference/data-analysis/text-analysis/analysis-condition-tokenfilter.md), which can be used to mark tokens as keywords, similar to the `keyword_marker` filter. -For built-in [language analyzers](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-lang-analyzer.md), you also can use the [`stem_exclusion`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-lang-analyzer.md#_excluding_words_from_stemming) parameter to specify a list of words that won’t be stemmed. +For built-in [language analyzers](elasticsearch://reference/data-analysis/text-analysis/analysis-lang-analyzer.md), you also can use the [`stem_exclusion`](elasticsearch://reference/data-analysis/text-analysis/analysis-lang-analyzer.md#_excluding_words_from_stemming) parameter to specify a list of words that won’t be stemmed. diff --git a/manage-data/data-store/text-analysis/token-graphs.md b/manage-data/data-store/text-analysis/token-graphs.md index 9212a0fdc..02d140f08 100644 --- a/manage-data/data-store/text-analysis/token-graphs.md +++ b/manage-data/data-store/text-analysis/token-graphs.md @@ -36,10 +36,10 @@ Some token filters can add tokens that span multiple positions. These can includ However, only some token filters, known as *graph token filters*, accurately record the `positionLength` for multi-position tokens. These filters include: -* [`synonym_graph`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-synonym-graph-tokenfilter.md) -* [`word_delimiter_graph`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-word-delimiter-graph-tokenfilter.md) +* [`synonym_graph`](elasticsearch://reference/data-analysis/text-analysis/analysis-synonym-graph-tokenfilter.md) +* [`word_delimiter_graph`](elasticsearch://reference/data-analysis/text-analysis/analysis-word-delimiter-graph-tokenfilter.md) -Some tokenizers, such as the [`nori_tokenizer`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch-plugins/analysis-nori-tokenizer.md), also accurately decompose compound tokens into multi-position tokens. +Some tokenizers, such as the [`nori_tokenizer`](elasticsearch://reference/elasticsearch-plugins/analysis-nori-tokenizer.md), also accurately decompose compound tokens into multi-position tokens. In the following graph, `domain name system` and its synonym, `dns`, both have a position of `0`. However, `dns` has a `positionLength` of `3`. Other tokens in the graph have a default `positionLength` of `1`. @@ -51,7 +51,7 @@ In the following graph, `domain name system` and its synonym, `dns`, both have a [Indexing](index-search-analysis.md) ignores the `positionLength` attribute and does not support token graphs containing multi-position tokens. -However, queries, such as the [`match`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-match-query.md) or [`match_phrase`](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-match-query-phrase.md) query, can use these graphs to generate multiple sub-queries from a single query string. +However, queries, such as the [`match`](elasticsearch://reference/query-languages/query-dsl-match-query.md) or [`match_phrase`](elasticsearch://reference/query-languages/query-dsl-match-query-phrase.md) query, can use these graphs to generate multiple sub-queries from a single query string. :::::{dropdown} Example A user runs a search for the following phrase using the `match_phrase` query: @@ -81,8 +81,8 @@ This means the query matches documents containing either `dns is fragile` *or* ` The following token filters can add tokens that span multiple positions but only record a default `positionLength` of `1`: -* [`synonym`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-synonym-tokenfilter.md) -* [`word_delimiter`](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-word-delimiter-tokenfilter.md) +* [`synonym`](elasticsearch://reference/data-analysis/text-analysis/analysis-synonym-tokenfilter.md) +* [`word_delimiter`](elasticsearch://reference/data-analysis/text-analysis/analysis-word-delimiter-tokenfilter.md) This means these filters will produce invalid token graphs for streams containing such tokens. diff --git a/manage-data/ingest.md b/manage-data/ingest.md index 983944963..28e2408f8 100644 --- a/manage-data/ingest.md +++ b/manage-data/ingest.md @@ -30,7 +30,7 @@ Elastic offer tools designed to ingest specific types of general content. The co * To index **documents** directly into {{es}}, use the {{es}} [document APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-document). * To send **application data** directly to {{es}}, use an [{{es}} language client](https://www.elastic.co/guide/en/elasticsearch/client/index.html). * To index **web page content**, use the Elastic [web crawler](https://www.elastic.co/web-crawler). -* To sync **data from third-party sources**, use [connectors](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/search-connectors/index.md). A connector syncs content from an original data source to an {{es}} index. Using connectors you can create *searchable*, read-only replicas of your data sources. +* To sync **data from third-party sources**, use [connectors](elasticsearch://reference/ingestion-tools/search-connectors/index.md). A connector syncs content from an original data source to an {{es}} index. Using connectors you can create *searchable*, read-only replicas of your data sources. * To index **single files** for testing in a non-production environment, use the {{kib}} [file uploader](ingest/upload-data-files.md). If you would like to try things out before you add your own data, try using our [sample data](ingest/sample-data.md). diff --git a/manage-data/ingest/ingest-reference-architectures.md b/manage-data/ingest/ingest-reference-architectures.md index 47bebdb00..2596ba2e2 100644 --- a/manage-data/ingest/ingest-reference-architectures.md +++ b/manage-data/ingest/ingest-reference-architectures.md @@ -24,6 +24,6 @@ You can host {{es}} on your own hardware or send your data to {{es}} on {{ecloud | [*{{agent}} to {{ls}} to Elasticsearch*](./ingest-reference-architectures/agent-ls.md)

![Image showing {{agent}} to {{ls}} to {{es}}](../../images/ingest-ea-ls-es.png "") | You need additional capabilities offered by {{ls}}:

* [**enrichment**](./ingest-reference-architectures/ls-enrich.md) between {{agent}} and {{es}}
* [**persistent queue (PQ) buffering**](./ingest-reference-architectures/lspq.md) to accommodate network issues and downstream unavailability
* [**proxying**](./ingest-reference-architectures/ls-networkbridge.md) in cases where {{agent}}s have network restrictions for connecting outside of the {{agent}} network
* data needs to be [**routed to multiple**](./ingest-reference-architectures/ls-multi.md) {{es}} clusters and other destinations depending on the content
| | [*{{agent}} to proxy to Elasticsearch*](./ingest-reference-architectures/agent-proxy.md)

![Image showing connections between {{agent}} and {{es}} using a proxy](../../images/ingest-ea-proxy-es.png "") | Agents have [network restrictions](./ingest-reference-architectures/agent-proxy.md) that prevent connecting outside of the {{agent}} network Note that [{{ls}} as proxy](./ingest-reference-architectures/ls-networkbridge.md) is one option.
| | [*{{agent}} to {{es}} with Kafka as middleware message queue*](./ingest-reference-architectures/agent-kafka-es.md)

![Image showing {{agent}} collecting data and using Kafka as a message queue enroute to {{es}}](../../images/ingest-ea-kafka.png "") | Kafka is your [middleware message queue](./ingest-reference-architectures/agent-kafka-es.md):

* [Kafka ES sink connector](./ingest-reference-architectures/agent-kafka-essink.md) to write from Kafka to {{es}}
* [{{ls}} to read from Kafka and route to {{es}}](./ingest-reference-architectures/agent-kafka-ls.md)
| -| [*{{ls}} to Elasticsearch*](./ingest-reference-architectures/ls-for-input.md)

![Image showing {{ls}} collecting data and sending to {{es}}](../../images/ingest-ls-es.png "") | You need to collect data from a source that {{agent}} can’t read (such as databases, AWS Kinesis). Check out the [{{ls}} input plugins](asciidocalypse://docs/logstash/docs/reference/input-plugins.md).
| +| [*{{ls}} to Elasticsearch*](./ingest-reference-architectures/ls-for-input.md)

![Image showing {{ls}} collecting data and sending to {{es}}](../../images/ingest-ls-es.png "") | You need to collect data from a source that {{agent}} can’t read (such as databases, AWS Kinesis). Check out the [{{ls}} input plugins](logstash://reference/input-plugins.md).
| | [*Elastic air-gapped architectures*](./ingest-reference-architectures/airgapped-env.md)

![Image showing {{stack}} in an air-gapped environment](../../images/ingest-ea-airgapped.png "") | You want to deploy {{agent}} and {{stack}} in an air-gapped environment (no access to outside networks)
| diff --git a/manage-data/ingest/ingest-reference-architectures/agent-es-airgapped.md b/manage-data/ingest/ingest-reference-architectures/agent-es-airgapped.md index 898f30b02..d4108d5a6 100644 --- a/manage-data/ingest/ingest-reference-architectures/agent-es-airgapped.md +++ b/manage-data/ingest/ingest-reference-architectures/agent-es-airgapped.md @@ -25,5 +25,5 @@ Use when Info for air-gapped environments: * [Installing the {{stack}} in an air-gapped environment](../../../deploy-manage/deploy/cloud-enterprise/air-gapped-install.md) -* [Using a proxy server with Elastic Agent and Fleet](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/fleet-agent-proxy-support.md) +* [Using a proxy server with Elastic Agent and Fleet](/reference/ingestion-tools/fleet/fleet-agent-proxy-support.md) diff --git a/manage-data/ingest/ingest-reference-architectures/agent-kafka-essink.md b/manage-data/ingest/ingest-reference-architectures/agent-kafka-essink.md index 61d96a3c9..45142ac28 100644 --- a/manage-data/ingest/ingest-reference-architectures/agent-kafka-essink.md +++ b/manage-data/ingest/ingest-reference-architectures/agent-kafka-essink.md @@ -32,8 +32,8 @@ Info on {{agent}} and agent integrations: Info on {{ls}} and {{ls}} plugins: * [{{ls}} Reference](https://www.elastic.co/guide/en/logstash/current) -* [{{ls}} {{agent}} input](asciidocalypse://docs/logstash/docs/reference/plugins-inputs-elastic_agent.md) -* [{{ls}} Kafka output](asciidocalypse://docs/logstash/docs/reference/plugins-outputs-kafka.md) +* [{{ls}} {{agent}} input](logstash://reference/plugins-inputs-elastic_agent.md) +* [{{ls}} Kafka output](logstash://reference/plugins-outputs-kafka.md) Info on {{es}}: diff --git a/manage-data/ingest/ingest-reference-architectures/agent-kafka-ls.md b/manage-data/ingest/ingest-reference-architectures/agent-kafka-ls.md index 81c8f3ded..60d230ea4 100644 --- a/manage-data/ingest/ingest-reference-architectures/agent-kafka-ls.md +++ b/manage-data/ingest/ingest-reference-architectures/agent-kafka-ls.md @@ -32,10 +32,10 @@ Info on {{agent}} and agent integrations: Info on {{ls}} and {{ls}} Kafka plugins: * [{{ls}} Reference](https://www.elastic.co/guide/en/logstash/current) -* [{{ls}} {{agent}} input](asciidocalypse://docs/logstash/docs/reference/plugins-inputs-elastic_agent.md) -* [{{ls}} Kafka input](asciidocalypse://docs/logstash/docs/reference/plugins-inputs-kafka.md) -* [{{ls}} Kafka output](asciidocalypse://docs/logstash/docs/reference/plugins-outputs-kafka.md) -* [{{ls}} Elasticsearch output](asciidocalypse://docs/logstash/docs/reference/plugins-outputs-elasticsearch.md) +* [{{ls}} {{agent}} input](logstash://reference/plugins-inputs-elastic_agent.md) +* [{{ls}} Kafka input](logstash://reference/plugins-inputs-kafka.md) +* [{{ls}} Kafka output](logstash://reference/plugins-outputs-kafka.md) +* [{{ls}} Elasticsearch output](logstash://reference/plugins-outputs-elasticsearch.md) Info on {{es}}: diff --git a/manage-data/ingest/ingest-reference-architectures/agent-ls-airgapped.md b/manage-data/ingest/ingest-reference-architectures/agent-ls-airgapped.md index e000c2d82..0a4123706 100644 --- a/manage-data/ingest/ingest-reference-architectures/agent-ls-airgapped.md +++ b/manage-data/ingest/ingest-reference-architectures/agent-ls-airgapped.md @@ -25,10 +25,10 @@ Use when Info for air-gapped environments: * [Installing the {{stack}} in an air-gapped environment](../../../deploy-manage/deploy/cloud-enterprise/air-gapped-install.md) -* [Using a proxy server with Elastic Agent and Fleet](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/fleet-agent-proxy-support.md) +* [Using a proxy server with Elastic Agent and Fleet](/reference/ingestion-tools/fleet/fleet-agent-proxy-support.md) ## Geoip database management in air-gapped environments [ls-geoip] -The [{{ls}} geoip filter](asciidocalypse://docs/logstash/docs/reference/plugins-filters-geoip.md) requires regular database updates to remain up-to-date with the latest information. If you are using the {{ls}} geoip filter plugin in an air-gapped environment, you can manage updates through a proxy, a custom endpoint, or manually. Check out [Manage your own database updates](asciidocalypse://docs/logstash/docs/reference/plugins-filters-geoip.md#plugins-filters-geoip-manage_update) for more info. +The [{{ls}} geoip filter](logstash://reference/plugins-filters-geoip.md) requires regular database updates to remain up-to-date with the latest information. If you are using the {{ls}} geoip filter plugin in an air-gapped environment, you can manage updates through a proxy, a custom endpoint, or manually. Check out [Manage your own database updates](logstash://reference/plugins-filters-geoip.md#plugins-filters-geoip-manage_update) for more info. diff --git a/manage-data/ingest/ingest-reference-architectures/agent-proxy.md b/manage-data/ingest/ingest-reference-architectures/agent-proxy.md index 8e713fe28..436c68982 100644 --- a/manage-data/ingest/ingest-reference-architectures/agent-proxy.md +++ b/manage-data/ingest/ingest-reference-architectures/agent-proxy.md @@ -39,7 +39,7 @@ Info on {{agent}} and agent integrations: Info on using a proxy server: -* [Using a proxy server with {{agent}} and {{fleet}}](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/fleet-agent-proxy-support.md) +* [Using a proxy server with {{agent}} and {{fleet}}](/reference/ingestion-tools/fleet/fleet-agent-proxy-support.md) Info on {{es}}: diff --git a/manage-data/ingest/ingest-reference-architectures/ls-enrich.md b/manage-data/ingest/ingest-reference-architectures/ls-enrich.md index 7a99bcaa4..f66d41141 100644 --- a/manage-data/ingest/ingest-reference-architectures/ls-enrich.md +++ b/manage-data/ingest/ingest-reference-architectures/ls-enrich.md @@ -31,14 +31,14 @@ Examples Info on configuring {{agent}}: * [Fleet and Elastic Agent Guide](https://www.elastic.co/guide/en/fleet/current) -* [Configuring outputs for {{agent}}](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/elastic-agent-output-configuration.md) +* [Configuring outputs for {{agent}}](/reference/ingestion-tools/fleet/elastic-agent-output-configuration.md) For info on {{ls}} for enriching data, check out these sections in the [Logstash Reference](https://www.elastic.co/guide/en/logstash/current): -* [{{ls}} {{agent}} input](asciidocalypse://docs/logstash/docs/reference/plugins-inputs-elastic_agent.md) -* [{{ls}} plugins for enriching data](asciidocalypse://docs/logstash/docs/reference/lookup-enrichment.md) -* [Logstash filter plugins](asciidocalypse://docs/logstash/docs/reference/filter-plugins.md) -* [{{ls}} {{es}} output](asciidocalypse://docs/logstash/docs/reference/plugins-outputs-elasticsearch.md) +* [{{ls}} {{agent}} input](logstash://reference/plugins-inputs-elastic_agent.md) +* [{{ls}} plugins for enriching data](logstash://reference/lookup-enrichment.md) +* [Logstash filter plugins](logstash://reference/filter-plugins.md) +* [{{ls}} {{es}} output](logstash://reference/plugins-outputs-elasticsearch.md) Info on {{es}}: diff --git a/manage-data/ingest/ingest-reference-architectures/ls-for-input.md b/manage-data/ingest/ingest-reference-architectures/ls-for-input.md index 1ce70c2da..ed5c66022 100644 --- a/manage-data/ingest/ingest-reference-architectures/ls-for-input.md +++ b/manage-data/ingest/ingest-reference-architectures/ls-for-input.md @@ -29,8 +29,8 @@ Info on {{ls}} and {{ls}} input and output plugins: * [{{ls}} plugin support matrix](https://www.elastic.co/support/matrix#logstash_plugins) * [{{ls}} Reference](https://www.elastic.co/guide/en/logstash/current) -* [{{ls}} input plugins](asciidocalypse://docs/logstash/docs/reference/input-plugins.md) -* [{{es}} output plugin](asciidocalypse://docs/logstash/docs/reference/plugins-outputs-elasticsearch.md) +* [{{ls}} input plugins](logstash://reference/input-plugins.md) +* [{{es}} output plugin](logstash://reference/plugins-outputs-elasticsearch.md) Info on {{es}} and ingest pipelines: diff --git a/manage-data/ingest/ingest-reference-architectures/ls-multi.md b/manage-data/ingest/ingest-reference-architectures/ls-multi.md index 6507f7022..7015094e7 100644 --- a/manage-data/ingest/ingest-reference-architectures/ls-multi.md +++ b/manage-data/ingest/ingest-reference-architectures/ls-multi.md @@ -57,13 +57,13 @@ output { Info on configuring {{agent}}: * [Fleet and Elastic Agent Guide](https://www.elastic.co/guide/en/fleet/current) -* [Configuring outputs for {{agent}}](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/elastic-agent-output-configuration.md) +* [Configuring outputs for {{agent}}](/reference/ingestion-tools/fleet/elastic-agent-output-configuration.md) Info on {{ls}} and {{ls}} outputs: * [{{ls}} Reference](https://www.elastic.co/guide/en/logstash/current) -* [{{ls}} {{es}} output plugin](asciidocalypse://docs/logstash/docs/reference/plugins-outputs-elasticsearch.md) -* [{{ls}} output plugins](asciidocalypse://docs/logstash/docs/reference/output-plugins.md) +* [{{ls}} {{es}} output plugin](logstash://reference/plugins-outputs-elasticsearch.md) +* [{{ls}} output plugins](logstash://reference/output-plugins.md) Info on {{es}}: diff --git a/manage-data/ingest/ingest-reference-architectures/ls-networkbridge.md b/manage-data/ingest/ingest-reference-architectures/ls-networkbridge.md index 0f7bc72ea..36a903461 100644 --- a/manage-data/ingest/ingest-reference-architectures/ls-networkbridge.md +++ b/manage-data/ingest/ingest-reference-architectures/ls-networkbridge.md @@ -24,12 +24,12 @@ Example Info on configuring {{agent}}: * [Fleet and Elastic Agent Guide](https://www.elastic.co/guide/en/fleet/current) -* [Configuring outputs for {{agent}}](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/elastic-agent-output-configuration.md) +* [Configuring outputs for {{agent}}](/reference/ingestion-tools/fleet/elastic-agent-output-configuration.md) Info on {{ls}} and {{ls}} plugins: * [{{ls}} Reference](https://www.elastic.co/guide/en/logstash/current) -* [{{es}} output plugin](asciidocalypse://docs/logstash/docs/reference/plugins-outputs-elasticsearch.md) +* [{{es}} output plugin](logstash://reference/plugins-outputs-elasticsearch.md) Info on {{es}}: diff --git a/manage-data/ingest/ingest-reference-architectures/lspq.md b/manage-data/ingest/ingest-reference-architectures/lspq.md index c19a5992b..7042c84c0 100644 --- a/manage-data/ingest/ingest-reference-architectures/lspq.md +++ b/manage-data/ingest/ingest-reference-architectures/lspq.md @@ -21,16 +21,16 @@ Use when Info on configuring {{agent}}: * [Fleet and Elastic Agent Guide](https://www.elastic.co/guide/en/fleet/current) -* [Configuring outputs for {{agent}}](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/elastic-agent-output-configuration.md) +* [Configuring outputs for {{agent}}](/reference/ingestion-tools/fleet/elastic-agent-output-configuration.md) For info on {{ls}} plugins: -* [{{agent}} input](asciidocalypse://docs/logstash/docs/reference/plugins-inputs-elastic_agent.md) -* [{{es}} output plugin](asciidocalypse://docs/logstash/docs/reference/plugins-outputs-elasticsearch.md) +* [{{agent}} input](logstash://reference/plugins-inputs-elastic_agent.md) +* [{{es}} output plugin](logstash://reference/plugins-outputs-elasticsearch.md) For info on using {{ls}} for buffering and data resiliency, check out this section in the [Logstash Reference](https://www.elastic.co/guide/en/logstash/current): -* [{{ls}} Persistent Queues (PQ)](asciidocalypse://docs/logstash/docs/reference/persistent-queues.md) +* [{{ls}} Persistent Queues (PQ)](logstash://reference/persistent-queues.md) Info on {{es}}: diff --git a/manage-data/ingest/ingesting-data-for-elastic-solutions.md b/manage-data/ingest/ingesting-data-for-elastic-solutions.md index 3fc3627c3..96a77720c 100644 --- a/manage-data/ingest/ingesting-data-for-elastic-solutions.md +++ b/manage-data/ingest/ingesting-data-for-elastic-solutions.md @@ -17,7 +17,7 @@ To use [Elastic Agent](https://www.elastic.co/guide/en/fleet/current) and [Elast 1. Create an [{{ecloud}}](https://www.elastic.co/cloud) deployment for your solution. If you don’t have an {{ecloud}} account, you can sign up for a [free trial](https://cloud.elastic.co/registration) to get started. 2. Add the [Elastic integration](https://docs.elastic.co/en/integrations) for your data source to the deployment. -3. [Install {{agent}}](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/install-elastic-agents.md) on the systems whose data you want to collect. +3. [Install {{agent}}](/reference/ingestion-tools/fleet/install-elastic-agents.md) on the systems whose data you want to collect. :::: @@ -34,14 +34,14 @@ To use [Elastic Agent](https://www.elastic.co/guide/en/fleet/current) and [Elast **Resources** -* [Install {{agent}}](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/install-elastic-agents.md) +* [Install {{agent}}](/reference/ingestion-tools/fleet/install-elastic-agents.md) * [Elastic Search for integrations](https://www.elastic.co/integrations/data-integrations?solution=search) * [{{es}} Guide](https://www.elastic.co/guide/en/elasticsearch/reference/current) * [{{es}} document APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-document) * [{{es}} language clients](https://www.elastic.co/guide/en/elasticsearch/client/index.html) * [Elastic web crawler](https://www.elastic.co/web-crawler) - * [Elastic connectors](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/search-connectors/index.md) + * [Elastic connectors](elasticsearch://reference/ingestion-tools/search-connectors/index.md) @@ -67,7 +67,7 @@ With [Elastic Observability](https://www.elastic.co/observability), you can moni **Resources** -* [Install {{agent}}](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/install-elastic-agents.md) +* [Install {{agent}}](/reference/ingestion-tools/fleet/install-elastic-agents.md) * [Elastic Observability integrations](https://www.elastic.co/integrations/data-integrations?solution=observability) @@ -82,7 +82,7 @@ You can detect and respond to threats when you use [Elastic Security](https://ww **Resources** -* [Install {{agent}}](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/install-elastic-agents.md) +* [Install {{agent}}](/reference/ingestion-tools/fleet/install-elastic-agents.md) * [Elastic Security integrations](https://www.elastic.co/integrations/data-integrations?solution=search) * [Elastic Security documentation](/solutions/security.md) @@ -95,12 +95,12 @@ Bring your ideas and use {{es}} and the {{stack}} to store, search, and visualiz **Resources** -* [Install {{agent}}](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/install-elastic-agents.md) +* [Install {{agent}}](/reference/ingestion-tools/fleet/install-elastic-agents.md) * [{{es}} Guide](https://www.elastic.co/guide/en/elasticsearch/reference/current) * [{{es}} document APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-document) * [{{es}} language clients](https://www.elastic.co/guide/en/elasticsearch/client/index.html) * [Elastic web crawler](https://www.elastic.co/web-crawler) - * [Elastic connectors](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/search-connectors/index.md) + * [Elastic connectors](elasticsearch://reference/ingestion-tools/search-connectors/index.md) * [Tutorial: Get started with vector search and generative AI](https://www.elastic.co/guide/en/starting-with-the-elasticsearch-platform-and-its-solutions/current/getting-started-general-purpose.html) diff --git a/manage-data/ingest/ingesting-data-from-applications/ingest-data-from-beats-to-elasticsearch-service-with-logstash-as-proxy.md b/manage-data/ingest/ingesting-data-from-applications/ingest-data-from-beats-to-elasticsearch-service-with-logstash-as-proxy.md index f533cf708..34bbb3326 100644 --- a/manage-data/ingest/ingesting-data-from-applications/ingest-data-from-beats-to-elasticsearch-service-with-logstash-as-proxy.md +++ b/manage-data/ingest/ingesting-data-from-applications/ingest-data-from-beats-to-elasticsearch-service-with-logstash-as-proxy.md @@ -295,7 +295,7 @@ Now the Filebeat and Metricbeat are set up, let’s configure a {{ls}} pipeline 1. {{ls}} listens for Beats input on the default port of 5044. Only one line is needed to do this. {{ls}} can handle input from many Beats of the same and also of varying types (Metricbeat, Filebeat, and others). 2. This sends output to the standard output, which displays through your command line interface. This plugin enables you to verify the data before you send it to {{es}}, in a later step. -3. Save the new *beats.conf* file in your Logstash folder. To learn more about the file format and options, check [{{ls}} Configuration Examples](asciidocalypse://docs/logstash/docs/reference/config-examples.md). +3. Save the new *beats.conf* file in your Logstash folder. To learn more about the file format and options, check [{{ls}} Configuration Examples](logstash://reference/config-examples.md). ## Output {{ls}} data to stdout [ec-beats-logstash-stdout] @@ -529,7 +529,7 @@ In this section, you configure {{ls}} to send the Metricbeat and Filebeat data t ::::{note} In this guide, you manually launch each of the Elastic stack applications through the command line interface. In production, you may prefer to configure {{ls}}, Metricbeat, and Filebeat to run as System Services. Check the following pages for the steps to configure each application to run as a service: -* [Running {{ls}} as a service on Debian or RPM](asciidocalypse://docs/logstash/docs/reference/running-logstash.md) +* [Running {{ls}} as a service on Debian or RPM](logstash://reference/running-logstash.md) * [Metricbeat and systemd](asciidocalypse://docs/beats/docs/reference/metricbeat/running-with-systemd.md) * [Start filebeat](asciidocalypse://docs/beats/docs/reference/filebeat/filebeat-starting.md) diff --git a/manage-data/ingest/ingesting-data-from-applications/ingest-data-from-relational-database-into-elasticsearch-service.md b/manage-data/ingest/ingesting-data-from-applications/ingest-data-from-relational-database-into-elasticsearch-service.md index 081af2985..0dd8b2080 100644 --- a/manage-data/ingest/ingesting-data-from-applications/ingest-data-from-relational-database-into-elasticsearch-service.md +++ b/manage-data/ingest/ingesting-data-from-applications/ingest-data-from-relational-database-into-elasticsearch-service.md @@ -40,7 +40,7 @@ $$$ece-db-logstash-pipeline$$$ $$$ece-db-logstash-prerequisites$$$ -This guide explains how to ingest data from a relational database into {{ecloud}} through [{{ls}}](asciidocalypse://docs/logstash/docs/reference/index.md), using the Logstash [JDBC input plugin](asciidocalypse://docs/logstash/docs/reference/plugins-inputs-jdbc.md). It demonstrates how Logstash can be used to efficiently copy records and to receive updates from a relational database, and then send them into {{es}} in an {{ech}} or {{ece}} deployment. +This guide explains how to ingest data from a relational database into {{ecloud}} through [{{ls}}](logstash://reference/index.md), using the Logstash [JDBC input plugin](logstash://reference/plugins-inputs-jdbc.md). It demonstrates how Logstash can be used to efficiently copy records and to receive updates from a relational database, and then send them into {{es}} in an {{ech}} or {{ece}} deployment. The code and methods presented here have been tested with MySQL. They should work with other relational databases. @@ -234,13 +234,13 @@ Let’s set up a sample Logstash input pipeline to ingest data from your new JDB : The Logstash JDBC plugin does not come packaged with JDBC driver libraries. The JDBC driver library must be passed explicitly into the plugin using the `jdbc_driver_library` configuration option. tracking_column - : This parameter specifies the field `unix_ts_in_secs` that tracks the last document read by Logstash from MySQL, stored on disk in [logstash_jdbc_last_run](asciidocalypse://docs/logstash/docs/reference/plugins-inputs-jdbc.md#plugins-inputs-jdbc-last_run_metadata_path). The parameter determines the starting value for documents that Logstash requests in the next iteration of its polling loop. The value stored in `logstash_jdbc_last_run` can be accessed in a SELECT statement as `sql_last_value`. + : This parameter specifies the field `unix_ts_in_secs` that tracks the last document read by Logstash from MySQL, stored on disk in [logstash_jdbc_last_run](logstash://reference/plugins-inputs-jdbc.md#plugins-inputs-jdbc-last_run_metadata_path). The parameter determines the starting value for documents that Logstash requests in the next iteration of its polling loop. The value stored in `logstash_jdbc_last_run` can be accessed in a SELECT statement as `sql_last_value`. unix_ts_in_secs : The field generated by the SELECT statement, which contains the `modification_time` as a standard [Unix timestamp](https://en.wikipedia.org/wiki/Unix_time) (seconds since the epoch). The field is referenced by the `tracking column`. A Unix timestamp is used for tracking progress rather than a normal timestamp, as a normal timestamp may cause errors due to the complexity of correctly converting back and forth between UMT and the local timezone. sql_last_value - : This is a [built-in parameter](asciidocalypse://docs/logstash/docs/reference/plugins-inputs-jdbc.md#_predefined_parameters) containing the starting point of the current iteration of the Logstash polling loop, and it is referenced in the SELECT statement line of the JDBC input configuration. This parameter is set to the most recent value of `unix_ts_in_secs`, which is read from `.logstash_jdbc_last_run`. This value is the starting point for documents returned by the MySQL query that is executed in the Logstash polling loop. Including this variable in the query guarantees that we’re not resending data that is already stored in Elasticsearch. + : This is a [built-in parameter](logstash://reference/plugins-inputs-jdbc.md#_predefined_parameters) containing the starting point of the current iteration of the Logstash polling loop, and it is referenced in the SELECT statement line of the JDBC input configuration. This parameter is set to the most recent value of `unix_ts_in_secs`, which is read from `.logstash_jdbc_last_run`. This value is the starting point for documents returned by the MySQL query that is executed in the Logstash polling loop. Including this variable in the query guarantees that we’re not resending data that is already stored in Elasticsearch. schedule : This uses cron syntax to specify how often Logstash should poll MySQL for changes. The specification `*/5 * * * * *` tells Logstash to contact MySQL every 5 seconds. Input from this plugin can be scheduled to run periodically according to a specific schedule. This scheduling syntax is powered by [rufus-scheduler](https://github.com/jmettraux/rufus-scheduler). The syntax is cron-like with some extensions specific to Rufus (for example, timezone support). @@ -330,7 +330,7 @@ In this section, we configure Logstash to send the MySQL data to Elasticsearch. ``` 1. Use the Cloud ID of your {{ech}} or {{ece}} deployment. You can include or omit the `:` prefix at the beginning of the Cloud ID. Both versions work fine. Find your Cloud ID by going to the {{kib}} main menu and selecting Management > Integrations, and then selecting View deployment details. - 2. the default username is `elastic`. It is not recommended to use the `elastic` account for ingesting data as this is a superuser. We recommend using a user with reduced permissions, or an API Key with permissions specific to the indices or data streams that will be written to. Check [Configuring security in Logstash](asciidocalypse://docs/logstash/docs/reference/secure-connection.md) for information on roles and API Keys. Use the password provided when you created the deployment if using the `elastic` user, or the password used when creating a new ingest user with the roles specified in the [Configuring security in Logstash](asciidocalypse://docs/logstash/docs/reference/secure-connection.md) documentation. + 2. the default username is `elastic`. It is not recommended to use the `elastic` account for ingesting data as this is a superuser. We recommend using a user with reduced permissions, or an API Key with permissions specific to the indices or data streams that will be written to. Check [Configuring security in Logstash](logstash://reference/secure-connection.md) for information on roles and API Keys. Use the password provided when you created the deployment if using the `elastic` user, or the password used when creating a new ingest user with the roles specified in the [Configuring security in Logstash](logstash://reference/secure-connection.md) documentation. Following are some additional details about the configuration file settings: diff --git a/manage-data/ingest/ingesting-data-from-applications/ingest-data-with-nodejs-on-elasticsearch-service.md b/manage-data/ingest/ingesting-data-from-applications/ingest-data-with-nodejs-on-elasticsearch-service.md index 967823fa4..6516553f5 100644 --- a/manage-data/ingest/ingesting-data-from-applications/ingest-data-with-nodejs-on-elasticsearch-service.md +++ b/manage-data/ingest/ingesting-data-from-applications/ingest-data-with-nodejs-on-elasticsearch-service.md @@ -168,7 +168,7 @@ async function run() { run().catch(console.log) ``` -When using the [client.index](asciidocalypse://docs/elasticsearch-js/docs/reference/api-reference.md#_index) API, the request automatically creates the `game-of-thrones` index if it doesn’t already exist, as well as document IDs for each indexed document if they are not explicitly specified. +When using the [client.index](elasticsearch-js://reference/api-reference.md#_index) API, the request automatically creates the `game-of-thrones` index if it doesn’t already exist, as well as document IDs for each indexed document if they are not explicitly specified. ## Search and modify data [ec_search_and_modify_data] @@ -215,7 +215,7 @@ async function update() { update().catch(console.log) ``` -This [more comprehensive list of API examples](asciidocalypse://docs/elasticsearch-js/docs/reference/examples.md) includes bulk operations, checking the existence of documents, updating by query, deleting, scrolling, and SQL queries. To learn more, check the complete [API reference](asciidocalypse://docs/elasticsearch-js/docs/reference/api-reference.md). +This [more comprehensive list of API examples](elasticsearch-js://reference/examples.md) includes bulk operations, checking the existence of documents, updating by query, deleting, scrolling, and SQL queries. To learn more, check the complete [API reference](elasticsearch-js://reference/api-reference.md). ## Switch to API key authentication [ec_switch_to_api_key_authentication] @@ -302,11 +302,11 @@ Security Connections ({{ech}} only) -: If your application connecting to {{ech}} runs under the Java security manager, you should at least disable the caching of positive hostname resolutions. To learn more, check the [Java API Client documentation](asciidocalypse://docs/elasticsearch-java/docs/reference/_others.md). +: If your application connecting to {{ech}} runs under the Java security manager, you should at least disable the caching of positive hostname resolutions. To learn more, check the [Java API Client documentation](elasticsearch-java://reference/_others.md). Schema : When the example code was run an index mapping was created automatically. The field types were selected by {{es}} based on the content seen when the first record was ingested, and updated as new fields appeared in the data. It would be more efficient to specify the fields and field types in advance to optimize performance. Refer to the Elastic Common Schema documentation and Field Type documentation when you are designing the schema for your production use cases. Ingest -: For more advanced scenarios, this [bulk ingestion](asciidocalypse://docs/elasticsearch-js/docs/reference/bulk_examples.md) reference gives an example of the `bulk` API that makes it possible to perform multiple operations in a single call. This bulk example also explicitly specifies document IDs. If you have a lot of documents to index, using bulk to batch document operations is significantly faster than submitting requests individually. +: For more advanced scenarios, this [bulk ingestion](elasticsearch-js://reference/bulk_examples.md) reference gives an example of the `bulk` API that makes it possible to perform multiple operations in a single call. This bulk example also explicitly specifies document IDs. If you have a lot of documents to index, using bulk to batch document operations is significantly faster than submitting requests individually. diff --git a/manage-data/ingest/ingesting-data-from-applications/ingest-data-with-python-on-elasticsearch-service.md b/manage-data/ingest/ingesting-data-from-applications/ingest-data-with-python-on-elasticsearch-service.md index 89996e033..6d9024495 100644 --- a/manage-data/ingest/ingesting-data-from-applications/ingest-data-with-python-on-elasticsearch-service.md +++ b/manage-data/ingest/ingesting-data-from-applications/ingest-data-with-python-on-elasticsearch-service.md @@ -293,7 +293,7 @@ es.get(index='lord-of-the-rings', id='2EkAzngB_pyHD3p65UMt') 'birthplace': 'The Shire'}} ``` -For frequently used API calls with the Python client, check [Examples](asciidocalypse://docs/elasticsearch-py/docs/reference/examples.md). +For frequently used API calls with the Python client, check [Examples](elasticsearch-py://reference/examples.md). ## Switch to API key authentication [ec_switch_to_api_key_authentication_2] @@ -353,7 +353,7 @@ es = Elasticsearch( Check [Create API key API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-api-key) to learn more about API Keys and [Security privileges](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md) to understand which privileges are needed. If you are not sure what the right combination of privileges for your custom application is, you can enable [audit logging](../../../deploy-manage/monitor/logging-configuration/enabling-audit-logs.md) on {{es}} to find out what privileges are being used. To learn more about how logging works on {{ech}} or {{ece}}, check [Monitoring Elastic Cloud deployment logs and metrics](https://www.elastic.co/blog/monitoring-elastic-cloud-deployment-logs-and-metrics). -For more information on refreshing an index, searching, updating, and deleting, check the [elasticsearch-py examples](asciidocalypse://docs/elasticsearch-py/docs/reference/examples.md). +For more information on refreshing an index, searching, updating, and deleting, check the [elasticsearch-py examples](elasticsearch-py://reference/examples.md). ### Best practices [ec_best_practices_2] @@ -368,5 +368,5 @@ Schema : When the example code is run, an index mapping is created automatically. The field types are selected by {{es}} based on the content seen when the first record was ingested, and updated as new fields appeared in the data. It would be more efficient to specify the fields and field types in advance to optimize performance. Refer to the Elastic Common Schema documentation and Field Type documentation when you design the schema for your production use cases. Ingest -: For more advanced scenarios, [Bulk helpers](asciidocalypse://docs/elasticsearch-py/docs/reference/client-helpers.md#bulk-helpers) gives examples for the `bulk` API that makes it possible to perform multiple operations in a single call. If you have a lot of documents to index, using bulk to batch document operations is significantly faster than submitting requests individually. +: For more advanced scenarios, [Bulk helpers](elasticsearch-py://reference/client-helpers.md#bulk-helpers) gives examples for the `bulk` API that makes it possible to perform multiple operations in a single call. If you have a lot of documents to index, using bulk to batch document operations is significantly faster than submitting requests individually. diff --git a/manage-data/ingest/ingesting-data-from-applications/ingest-logs-from-nodejs-web-application-using-filebeat.md b/manage-data/ingest/ingesting-data-from-applications/ingest-logs-from-nodejs-web-application-using-filebeat.md index 93eba8809..c8d617eb2 100644 --- a/manage-data/ingest/ingesting-data-from-applications/ingest-logs-from-nodejs-web-application-using-filebeat.md +++ b/manage-data/ingest/ingesting-data-from-applications/ingest-logs-from-nodejs-web-application-using-filebeat.md @@ -49,7 +49,7 @@ $$$ece-node-logs-send-ess$$$ $$$ece-node-logs-view-kibana$$$ -This guide demonstrates how to ingest logs from a Node.js web application and deliver them securely into an {{ech}} or {{ece}} deployment. You’ll set up Filebeat to monitor a JSON-structured log file that has standard Elastic Common Schema (ECS) formatted fields, and you’ll then view real-time visualizations of the log events in Kibana as requests are made to the Node.js server. While Node.js is used for this example, this approach to monitoring log output is applicable across many client types. Check the list of [available ECS logging plugins](asciidocalypse://docs/ecs-logging/docs/reference/intro.md#_get_started). +This guide demonstrates how to ingest logs from a Node.js web application and deliver them securely into an {{ech}} or {{ece}} deployment. You’ll set up Filebeat to monitor a JSON-structured log file that has standard Elastic Common Schema (ECS) formatted fields, and you’ll then view real-time visualizations of the log events in Kibana as requests are made to the Node.js server. While Node.js is used for this example, this approach to monitoring log output is applicable across many client types. Check the list of [available ECS logging plugins](ecs-logging://reference/intro.md#_get_started). *Time required: 1.5 hours* @@ -71,7 +71,7 @@ For the three following packages, you can create a working directory to install npm install winston ``` -* The [Elastic Common Schema (ECS) formatter](asciidocalypse://docs/ecs-logging-nodejs/docs/reference/winston.md) for the Node.js winston logger - This plugin formats your Node.js logs into an ECS structured JSON format ideally suited for ingestion into Elasticsearch. To install the ECS winston logger, run the following command in your working directory so that the package is installed in the same location as the winston package: +* The [Elastic Common Schema (ECS) formatter](ecs-logging-nodejs://reference/winston.md) for the Node.js winston logger - This plugin formats your Node.js logs into an ECS structured JSON format ideally suited for ingestion into Elasticsearch. To install the ECS winston logger, run the following command in your working directory so that the package is installed in the same location as the winston package: ```sh npm install @elastic/ecs-winston-format diff --git a/manage-data/ingest/ingesting-data-from-applications/ingest-logs-from-python-application-using-filebeat.md b/manage-data/ingest/ingesting-data-from-applications/ingest-logs-from-python-application-using-filebeat.md index 8f622c092..052c39501 100644 --- a/manage-data/ingest/ingesting-data-from-applications/ingest-logs-from-python-application-using-filebeat.md +++ b/manage-data/ingest/ingesting-data-from-applications/ingest-logs-from-python-application-using-filebeat.md @@ -33,13 +33,13 @@ $$$ece-python-logs-send-ess$$$ $$$ece-python-logs-view-kibana$$$ -This guide demonstrates how to ingest logs from a Python application and deliver them securely into an {{ech}} deployment. You’ll set up Filebeat to monitor a JSON-structured log file that has standard Elastic Common Schema (ECS) formatted fields, and you’ll then view real-time visualizations of the log events in {{kib}} as they occur. While Python is used for this example, this approach to monitoring log output is applicable across many client types. Check the list of [available ECS logging plugins](asciidocalypse://docs/ecs-logging/docs/reference/intro.md). +This guide demonstrates how to ingest logs from a Python application and deliver them securely into an {{ech}} deployment. You’ll set up Filebeat to monitor a JSON-structured log file that has standard Elastic Common Schema (ECS) formatted fields, and you’ll then view real-time visualizations of the log events in {{kib}} as they occur. While Python is used for this example, this approach to monitoring log output is applicable across many client types. Check the list of [available ECS logging plugins](ecs-logging://reference/intro.md). *Time required: 1 hour* ## Prerequisites [ec_prerequisites_2] -To complete these steps you need to have [Python](https://www.python.org/) installed on your system as well as the [Elastic Common Schema (ECS) logger](asciidocalypse://docs/ecs-logging-python/docs/reference/installation.md) for the Python logging library. +To complete these steps you need to have [Python](https://www.python.org/) installed on your system as well as the [Elastic Common Schema (ECS) logger](ecs-logging-python://reference/installation.md) for the Python logging library. To install *ecs-logging-python*, run: diff --git a/manage-data/ingest/ingesting-timeseries-data.md b/manage-data/ingest/ingesting-timeseries-data.md index 81b6ace3c..6bbf36b66 100644 --- a/manage-data/ingest/ingesting-timeseries-data.md +++ b/manage-data/ingest/ingesting-timeseries-data.md @@ -20,13 +20,13 @@ In this section, we’ll help you determine which option is best for you. ## {{agent}} and Elastic integrations [ingest-ea] -A single [{{agent}}](https://www.elastic.co/guide/en/fleet/current) can collect multiple types of data when it is [installed](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/install-elastic-agents.md) on a host computer. You can use standalone {{agent}}s and manage them locally on the systems where they are installed, or you can manage all of your agents and policies with the [Fleet UI in {{kib}}](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/manage-elastic-agents-in-fleet.md). +A single [{{agent}}](https://www.elastic.co/guide/en/fleet/current) can collect multiple types of data when it is [installed](/reference/ingestion-tools/fleet/install-elastic-agents.md) on a host computer. You can use standalone {{agent}}s and manage them locally on the systems where they are installed, or you can manage all of your agents and policies with the [Fleet UI in {{kib}}](/reference/ingestion-tools/fleet/manage-elastic-agents-in-fleet.md). Use {{agent}} with one of hundreds of [Elastic integrations](https://docs.elastic.co/en/integrations) to simplify collecting, transforming, and visualizing data. Integrations include default ingestion rules, dashboards, and visualizations to help you start analyzing your data right away. Check out the [Integration quick reference](https://docs.elastic.co/en/integrations/all_integrations) to search for available integrations that can reduce your time to value. -{{agent}} is the best option for collecting timestamped data for most data sources and use cases. If your data requires additional processing before going to {{es}}, you can use [{{agent}} processors](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/agent-processors.md), [{{ls}}](https://www.elastic.co/guide/en/logstash/current), or additional processing features in {{es}}. Check out [additional processing](/manage-data/ingest/transform-enrich.md) to see options. +{{agent}} is the best option for collecting timestamped data for most data sources and use cases. If your data requires additional processing before going to {{es}}, you can use [{{agent}} processors](/reference/ingestion-tools/fleet/agent-processors.md), [{{ls}}](https://www.elastic.co/guide/en/logstash/current), or additional processing features in {{es}}. Check out [additional processing](/manage-data/ingest/transform-enrich.md) to see options. -Ready to try [{{agent}}](https://www.elastic.co/guide/en/fleet/current)? Check out the [installation instructions](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/install-elastic-agents.md). +Ready to try [{{agent}}](https://www.elastic.co/guide/en/fleet/current)? Check out the [installation instructions](/reference/ingestion-tools/fleet/install-elastic-agents.md). ## {{beats}} [ingest-beats] @@ -35,7 +35,7 @@ Ready to try [{{agent}}](https://www.elastic.co/guide/en/fleet/current)? Check o Beats require that you install a separate Beat for each type of data you want to collect. A single Elastic Agent installed on a host can collect and transport multiple types of data. -**Best practice:** Use [{{agent}}](https://www.elastic.co/guide/en/fleet/current) whenever possible. If your data source is not yet supported by {{agent}}, use {{beats}}. Check out the {{beats}} and {{agent}} [comparison](/manage-data/ingest/tools.md#additional-capabilities-beats-and-agent) for more info. When you are ready to upgrade, check out [Migrate from {{beats}} to {{agent}}](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/migrate-from-beats-to-elastic-agent.md). +**Best practice:** Use [{{agent}}](https://www.elastic.co/guide/en/fleet/current) whenever possible. If your data source is not yet supported by {{agent}}, use {{beats}}. Check out the {{beats}} and {{agent}} [comparison](/manage-data/ingest/tools.md#additional-capabilities-beats-and-agent) for more info. When you are ready to upgrade, check out [Migrate from {{beats}} to {{agent}}](/reference/ingestion-tools/fleet/migrate-from-beats-to-elastic-agent.md). ## OpenTelemetry (OTel) collectors [ingest-otel] @@ -47,10 +47,10 @@ In addition to supporting upstream OTel development, Elastic provides [Elastic D ## Logstash [ingest-logstash] -[{{ls}}](https://www.elastic.co/guide/en/logstash/current) is a versatile open source data ETL (extract, transform, load) engine that can expand your ingest capabilities. {{ls}} can *collect data* from a wide variety of data sources with {{ls}} [input plugins](asciidocalypse://docs/logstash/docs/reference/input-plugins.md), *enrich and transform* the data with {{ls}} [filter plugins](asciidocalypse://docs/logstash/docs/reference/filter-plugins.md), and *output* the data to {{es}} and other destinations with the {{ls}} [output plugins](asciidocalypse://docs/logstash/docs/reference/output-plugins.md). +[{{ls}}](https://www.elastic.co/guide/en/logstash/current) is a versatile open source data ETL (extract, transform, load) engine that can expand your ingest capabilities. {{ls}} can *collect data* from a wide variety of data sources with {{ls}} [input plugins](logstash://reference/input-plugins.md), *enrich and transform* the data with {{ls}} [filter plugins](logstash://reference/filter-plugins.md), and *output* the data to {{es}} and other destinations with the {{ls}} [output plugins](logstash://reference/output-plugins.md). Many users never need to use {{ls}}, but it’s available if you need it for: -* **Data collection** (if an Elastic integration isn’t available). {{agent}} and Elastic [integrations](https://docs.elastic.co/en/integrations/all_integrations) provide many features out-of-the-box, so be sure to search or browse integrations for your data source. If you don’t find an Elastic integration for your data source, check {{ls}} for an [input plugin](asciidocalypse://docs/logstash/docs/reference/input-plugins.md) for your data source. -* **Additional processing.** One of the most common {{ls}} use cases is [extending Elastic integrations](asciidocalypse://docs/logstash/docs/reference/using-logstash-with-elastic-integrations.md). You can take advantage of the extensive, built-in capabilities of Elastic Agent and Elastic Integrations, and then use {{ls}} for additional data processing before sending the data on to {{es}}. +* **Data collection** (if an Elastic integration isn’t available). {{agent}} and Elastic [integrations](https://docs.elastic.co/en/integrations/all_integrations) provide many features out-of-the-box, so be sure to search or browse integrations for your data source. If you don’t find an Elastic integration for your data source, check {{ls}} for an [input plugin](logstash://reference/input-plugins.md) for your data source. +* **Additional processing.** One of the most common {{ls}} use cases is [extending Elastic integrations](logstash://reference/using-logstash-with-elastic-integrations.md). You can take advantage of the extensive, built-in capabilities of Elastic Agent and Elastic Integrations, and then use {{ls}} for additional data processing before sending the data on to {{es}}. * **Advanced use cases.** {{ls}} can help with advanced use cases, such as when you need [persistence or buffering](/manage-data/ingest/ingest-reference-architectures/lspq.md), additional [data enrichment](/manage-data/ingest/ingest-reference-architectures/ls-enrich.md), [proxying](/manage-data/ingest/ingest-reference-architectures/ls-networkbridge.md) as a way to bridge network connections, or the ability to route data to [multiple destinations](/manage-data/ingest/ingest-reference-architectures/ls-multi.md). diff --git a/manage-data/ingest/tools.md b/manage-data/ingest/tools.md index c9e09b3e5..5dc9a9f37 100644 --- a/manage-data/ingest/tools.md +++ b/manage-data/ingest/tools.md @@ -45,13 +45,13 @@ Depending on the type of data you want to ingest, you have a number of methods a | File upload | Upload data from a file and inspect it before importing it into {{es}}. | [Upload data files](/manage-data/ingest/upload-data-files.md) | | APIs | Ingest data through code by using the APIs of one of the language clients or the {{es}} HTTP APIs. | [Document APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-document) | | OpenTelemetry | Collect and send your telemetry data to Elastic Observability | [Elastic Distributions of OpenTelemetry](https://github.com/elastic/opentelemetry?tab=readme-ov-file#elastic-distributions-of-opentelemetry) | -| Fleet and Elastic Agent | Add monitoring for logs, metrics, and other types of data to a host using Elastic Agent, and centrally manage it using Fleet. | [Fleet and {{agent}} overview](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/index.md)
[{{fleet}} and {{agent}} restrictions (Serverless)](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/fleet-agent-serverless-restrictions.md)
[{{beats}} and {{agent}} capabilities](/manage-data/ingest/tools.md)|| +| Fleet and Elastic Agent | Add monitoring for logs, metrics, and other types of data to a host using Elastic Agent, and centrally manage it using Fleet. | [Fleet and {{agent}} overview](/reference/ingestion-tools/fleet/index.md)
[{{fleet}} and {{agent}} restrictions (Serverless)](/reference/ingestion-tools/fleet/fleet-agent-serverless-restrictions.md)
[{{beats}} and {{agent}} capabilities](/manage-data/ingest/tools.md)|| | {{elastic-defend}} | {{elastic-defend}} provides organizations with prevention, detection, and response capabilities with deep visibility for EPP, EDR, SIEM, and Security Analytics use cases across Windows, macOS, and Linux operating systems running on both traditional endpoints and public cloud environments. | [Configure endpoint protection with {{elastic-defend}}](/solutions/security/configure-elastic-defend.md) | -| {{ls}} | Dynamically unify data from a wide variety of data sources and normalize it into destinations of your choice with {{ls}}. | [Logstash (Serverless)](asciidocalypse://docs/logstash/docs/reference/index.md)
[Logstash pipelines](/manage-data/ingest/transform-enrich/logstash-pipelines.md) | +| {{ls}} | Dynamically unify data from a wide variety of data sources and normalize it into destinations of your choice with {{ls}}. | [Logstash (Serverless)](logstash://reference/index.md)
[Logstash pipelines](/manage-data/ingest/transform-enrich/logstash-pipelines.md) | | {{beats}} | Use {{beats}} data shippers to send operational data to Elasticsearch directly or through Logstash. | [{{beats}} (Serverless)](asciidocalypse://docs/beats/docs/reference/index.md)
[What are {{beats}}?](asciidocalypse://docs/beats/docs/reference/index.md)
[{{beats}} and {{agent}} capabilities](/manage-data/ingest/tools.md)| | APM | Collect detailed performance information on response time for incoming requests, database queries, calls to caches, external HTTP requests, and more. | [Application performance monitoring (APM)](/solutions/observability/apps/application-performance-monitoring-apm.md) | | Application logs | Ingest application logs using Filebeat, {{agent}}, or the APM agent, or reformat application logs into Elastic Common Schema (ECS) logs and then ingest them using Filebeat or {{agent}}. | [Stream application logs](/solutions/observability/logs/stream-application-logs.md)
[ECS formatted application logs](/solutions/observability/logs/ecs-formatted-application-logs.md) | -| Elastic Serverless forwarder for AWS | Ship logs from your AWS environment to cloud-hosted, self-managed Elastic environments, or {{ls}}. | [Elastic Serverless Forwarder](asciidocalypse://docs/elastic-serverless-forwarder/docs/reference/index.md) | +| Elastic Serverless forwarder for AWS | Ship logs from your AWS environment to cloud-hosted, self-managed Elastic environments, or {{ls}}. | [Elastic Serverless Forwarder](elastic-serverless-forwarder://reference/index.md) | | Connectors | Use connectors to extract data from an original data source and sync it to an {{es}} index. | [Ingest content with Elastic connectors -](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/search-connectors/index.md)
[Connector clients](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/search-connectors/index.md) | +](elasticsearch://reference/ingestion-tools/search-connectors/index.md)
[Connector clients](elasticsearch://reference/ingestion-tools/search-connectors/index.md) | | Web crawler | Discover, extract, and index searchable content from websites and knowledge bases using the web crawler. | [Elastic Open Web Crawler](https://github.com/elastic/crawler#readme) | \ No newline at end of file diff --git a/manage-data/ingest/transform-enrich.md b/manage-data/ingest/transform-enrich.md index 9245f1233..29d04b06e 100644 --- a/manage-data/ingest/transform-enrich.md +++ b/manage-data/ingest/transform-enrich.md @@ -19,7 +19,7 @@ According to your use case, you may want to control the structure of your ingest Finally, to help ensure optimal query results, you may want to customize how text is analyzed and how text fields are defined inside {{es}}. {{agent}} processors -: You can use [{{agent}} processors](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/agent-processors.md) to sanitize or enrich raw data at the source. Use {{agent}} processors if you need to control what data is sent across the wire, or if you need to enrich the raw data with information available on the host. +: You can use [{{agent}} processors](/reference/ingestion-tools/fleet/agent-processors.md) to sanitize or enrich raw data at the source. Use {{agent}} processors if you need to control what data is sent across the wire, or if you need to enrich the raw data with information available on the host. {{es}} ingest pipelines : You can use [{{es}} ingest pipelines](transform-enrich/ingest-pipelines.md) to enrich incoming data or normalize field data before the data is indexed. {{es}} ingest pipelines enable you to manipulate the data as it comes in. This approach helps you avoid adding processing overhead to the hosts from which you’re collecting data. @@ -27,7 +27,7 @@ Finally, to help ensure optimal query results, you may want to customize how tex : When you define a pipeline, you can configure one or more processors to operate on the incoming data. A typical use case is to transform specific strings to lowercase, or to sort the elements of incoming arrays into a given order. This section describes: * How to create, view, edit, and delete an ingest pipeline * How to set up processors to transform the data -* How to test a pipeline before putting it into production. +* How to test a pipeline before putting it into production. : You can try out the [Parse logs](transform-enrich/example-parse-logs.md) example which shows you how to set up in ingest pipeline to transform incoming server logs into a standard format. @@ -36,10 +36,10 @@ Finally, to help ensure optimal query results, you may want to customize how tex {{ls}} and the {{ls}} `elastic_integration filter` : If you're using {{ls}} as your primary ingest tool, you can take advantage of its built-in pipeline capabilities to transform your data. You configure a pipeline by stringing together a series of input, output, filtering, and optional codec plugins to manipulate all incoming data. -: If you're ingesting using {{agent}} with Elastic {{integrations}}, you can use the {{ls}} [`elastic_integration filter`](https://www.elastic.co/guide/en/logstash/current/) and other [{{ls}} filters](asciidocalypse://docs/logstash/docs/reference/filter-plugins.md) to [extend Elastic integrations](asciidocalypse://docs/logstash/docs/reference/using-logstash-with-elastic-integrations.md) by transforming data before it goes to {{es}}. +: If you're ingesting using {{agent}} with Elastic {{integrations}}, you can use the {{ls}} [`elastic_integration filter`](https://www.elastic.co/guide/en/logstash/current/) and other [{{ls}} filters](logstash://reference/filter-plugins.md) to [extend Elastic integrations](logstash://reference/using-logstash-with-elastic-integrations.md) by transforming data before it goes to {{es}}. Index mapping -: Index mapping lets you control the structure that incoming data has within an {{es}} index. You can define all of the fields that are included in the index and their respective data types. For example, you can set fields for dates, numbers, or geolocations, and define the fields to have specific formats. +: Index mapping lets you control the structure that incoming data has within an {{es}} index. You can define all of the fields that are included in the index and their respective data types. For example, you can set fields for dates, numbers, or geolocations, and define the fields to have specific formats. : Ingested data can be mapped dynamically, where {{es}} adds all fields automatically based on the detected data types, or explicitly, where {{es}} maps the incoming data to fields based on your custom rules. @@ -50,4 +50,4 @@ Index mapping Text analysis : Like index mapping, text analysis is another form of data transformation that runs on data as it's being ingested. This process analyzes incoming, unstructured text and organizes it in a way to ensure that all relevant documents are matched for a given text query, and not just exact string matches. -: Refer to the [Text analysis](../data-store/text-analysis.md) pages to learn how to configure an analyzer to run on incoming text. You can opt to use one of several built-in analyzers, or create a custom analyzer for specific use cases. +: Refer to the [Text analysis](../data-store/text-analysis.md) pages to learn how to configure an analyzer to run on incoming text. You can opt to use one of several built-in analyzers, or create a custom analyzer for specific use cases. diff --git a/manage-data/ingest/transform-enrich/data-enrichment.md b/manage-data/ingest/transform-enrich/data-enrichment.md index 3b56249d1..5c277f11d 100644 --- a/manage-data/ingest/transform-enrich/data-enrichment.md +++ b/manage-data/ingest/transform-enrich/data-enrichment.md @@ -9,7 +9,7 @@ applies_to: # Data enrichment -You can use the [enrich processor](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/enrich-processor.md) to add data from your existing indices to incoming documents during ingest. +You can use the [enrich processor](elasticsearch://reference/ingestion-tools/enrich-processor/enrich-processor.md) to add data from your existing indices to incoming documents during ingest. For example, you can use the enrich processor to: @@ -75,7 +75,7 @@ Use the **Enrich Policies** view to add data from your existing indices to incom * The source indices that store enrich data as documents * The fields from the source indices used to match incoming documents * The enrich fields containing enrich data from the source indices that you want to add to incoming documents -* An optional [query](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-match-all-query.md). +* An optional [query](elasticsearch://reference/query-languages/query-dsl-match-all-query.md). :::{image} ../../../images/elasticsearch-reference-management-enrich-policies.png :alt: Enrich policies diff --git a/manage-data/ingest/transform-enrich/example-enrich-data-based-on-exact-values.md b/manage-data/ingest/transform-enrich/example-enrich-data-based-on-exact-values.md index b66fb7e52..b10658a74 100644 --- a/manage-data/ingest/transform-enrich/example-enrich-data-based-on-exact-values.md +++ b/manage-data/ingest/transform-enrich/example-enrich-data-based-on-exact-values.md @@ -8,7 +8,7 @@ applies_to: # Example: Enrich your data based on exact values [match-enrich-policy-type] -`match` [enrich policies](data-enrichment.md#enrich-policy) match enrich data to incoming documents based on an exact value, such as a email address or ID, using a [`term` query](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-term-query.md). +`match` [enrich policies](data-enrichment.md#enrich-policy) match enrich data to incoming documents based on an exact value, such as a email address or ID, using a [`term` query](elasticsearch://reference/query-languages/query-dsl-term-query.md). The following example creates a `match` enrich policy that adds user name and contact information to incoming documents based on an email address. It then adds the `match` enrich policy to a processor in an ingest pipeline. @@ -53,7 +53,7 @@ Use the [execute enrich policy API](https://www.elastic.co/docs/api/doc/elastics POST /_enrich/policy/users-policy/_execute?wait_for_completion=false ``` -Use the [create or update pipeline API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-put-pipeline) to create an ingest pipeline. In the pipeline, add an [enrich processor](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/enrich-processor.md) that includes: +Use the [create or update pipeline API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-put-pipeline) to create an ingest pipeline. In the pipeline, add an [enrich processor](elasticsearch://reference/ingestion-tools/enrich-processor/enrich-processor.md) that includes: * Your enrich policy. * The `field` of incoming documents used to match documents from the enrich index. diff --git a/manage-data/ingest/transform-enrich/example-enrich-data-based-on-geolocation.md b/manage-data/ingest/transform-enrich/example-enrich-data-based-on-geolocation.md index 19cf704cd..d386a2673 100644 --- a/manage-data/ingest/transform-enrich/example-enrich-data-based-on-geolocation.md +++ b/manage-data/ingest/transform-enrich/example-enrich-data-based-on-geolocation.md @@ -8,7 +8,7 @@ applies_to: # Example: Enrich your data based on geolocation [geo-match-enrich-policy-type] -`geo_match` [enrich policies](data-enrichment.md#enrich-policy) match enrich data to incoming documents based on a geographic location, using a [`geo_shape` query](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-geo-shape-query.md). +`geo_match` [enrich policies](data-enrichment.md#enrich-policy) match enrich data to incoming documents based on a geographic location, using a [`geo_shape` query](elasticsearch://reference/query-languages/query-dsl-geo-shape-query.md). The following example creates a `geo_match` enrich policy that adds postal codes to incoming documents based on a set of coordinates. It then adds the `geo_match` enrich policy to a processor in an ingest pipeline. @@ -66,12 +66,12 @@ Use the [execute enrich policy API](https://www.elastic.co/docs/api/doc/elastics POST /_enrich/policy/postal_policy/_execute?wait_for_completion=false ``` -Use the [create or update pipeline API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-put-pipeline) to create an ingest pipeline. In the pipeline, add an [enrich processor](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/enrich-processor.md) that includes: +Use the [create or update pipeline API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-put-pipeline) to create an ingest pipeline. In the pipeline, add an [enrich processor](elasticsearch://reference/ingestion-tools/enrich-processor/enrich-processor.md) that includes: * Your enrich policy. * The `field` of incoming documents used to match the geoshape of documents from the enrich index. * The `target_field` used to store appended enrich data for incoming documents. This field contains the `match_field` and `enrich_fields` specified in your enrich policy. -* The `shape_relation`, which indicates how the processor matches geoshapes in incoming documents to geoshapes in documents from the enrich index. See [Spatial Relations](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-shape-query.md#_spatial_relations) for valid options and more information. +* The `shape_relation`, which indicates how the processor matches geoshapes in incoming documents to geoshapes in documents from the enrich index. See [Spatial Relations](elasticsearch://reference/query-languages/query-dsl-shape-query.md#_spatial_relations) for valid options and more information. ```console PUT /_ingest/pipeline/postal_lookup diff --git a/manage-data/ingest/transform-enrich/example-enrich-data-by-matching-value-to-range.md b/manage-data/ingest/transform-enrich/example-enrich-data-by-matching-value-to-range.md index 03fdd90da..1098ba802 100644 --- a/manage-data/ingest/transform-enrich/example-enrich-data-by-matching-value-to-range.md +++ b/manage-data/ingest/transform-enrich/example-enrich-data-by-matching-value-to-range.md @@ -8,7 +8,7 @@ applies_to: # Example: Enrich your data by matching a value to a range [range-enrich-policy-type] -A `range` [enrich policy](data-enrichment.md#enrich-policy) uses a [`term` query](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-term-query.md) to match a number, date, or IP address in incoming documents to a range of the same type in the enrich index. Matching a range to a range is not supported. +A `range` [enrich policy](data-enrichment.md#enrich-policy) uses a [`term` query](elasticsearch://reference/query-languages/query-dsl-term-query.md) to match a number, date, or IP address in incoming documents to a range of the same type in the enrich index. Matching a range to a range is not supported. The following example creates a `range` enrich policy that adds a descriptive network name and responsible department to incoming documents based on an IP address. It then adds the enrich policy to a processor in an ingest pipeline. @@ -63,7 +63,7 @@ Use the [execute enrich policy API](https://www.elastic.co/docs/api/doc/elastics POST /_enrich/policy/networks-policy/_execute?wait_for_completion=false ``` -Use the [create or update pipeline API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-put-pipeline) to create an ingest pipeline. In the pipeline, add an [enrich processor](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/enrich-processor.md) that includes: +Use the [create or update pipeline API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-put-pipeline) to create an ingest pipeline. In the pipeline, add an [enrich processor](elasticsearch://reference/ingestion-tools/enrich-processor/enrich-processor.md) that includes: * Your enrich policy. * The `field` of incoming documents used to match documents from the enrich index. diff --git a/manage-data/ingest/transform-enrich/example-parse-logs.md b/manage-data/ingest/transform-enrich/example-parse-logs.md index 513ef9464..d38317bce 100644 --- a/manage-data/ingest/transform-enrich/example-parse-logs.md +++ b/manage-data/ingest/transform-enrich/example-parse-logs.md @@ -31,7 +31,7 @@ These logs contain a timestamp, IP address, and user agent. You want to give the 2. Click **Create pipeline > New pipeline**. 3. Set **Name** to `my-pipeline` and optionally add a description for the pipeline. -4. Add a [grok processor](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/grok-processor.md) to parse the log message: +4. Add a [grok processor](elasticsearch://reference/ingestion-tools/enrich-processor/grok-processor.md) to parse the log message: 1. Click **Add a processor** and select the **Grok** processor type. 2. Set **Field** to `message` and **Patterns** to the following [grok pattern](../../../explore-analyze/scripting/grok.md): @@ -47,9 +47,9 @@ These logs contain a timestamp, IP address, and user agent. You want to give the | Processor type | Field | Additional options | Description | | --- | --- | --- | --- | - | [**Date**](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/date-processor.md) | `@timestamp` | **Formats**: `dd/MMM/yyyy:HH:mm:ss Z` | `Format '@timestamp' as 'dd/MMM/yyyy:HH:mm:ss Z'` | - | [**GeoIP**](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/geoip-processor.md) | `source.ip` | **Target field**: `source.geo` | `Add 'source.geo' GeoIP data for 'source.ip'` | - | [**User agent**](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/user-agent-processor.md) | `user_agent` | | `Extract fields from 'user_agent'` | + | [**Date**](elasticsearch://reference/ingestion-tools/enrich-processor/date-processor.md) | `@timestamp` | **Formats**: `dd/MMM/yyyy:HH:mm:ss Z` | `Format '@timestamp' as 'dd/MMM/yyyy:HH:mm:ss Z'` | + | [**GeoIP**](elasticsearch://reference/ingestion-tools/enrich-processor/geoip-processor.md) | `source.ip` | **Target field**: `source.geo` | `Add 'source.geo' GeoIP data for 'source.ip'` | + | [**User agent**](elasticsearch://reference/ingestion-tools/enrich-processor/user-agent-processor.md) | `user_agent` | | `Extract fields from 'user_agent'` | Your form should look similar to this: @@ -135,7 +135,7 @@ These logs contain a timestamp, IP address, and user agent. You want to give the } ``` -12. To verify, search the data stream to retrieve the document. The following search uses [`filter_path`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/common-options.md#common-options-response-filtering) to return only the [document source](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md). +12. To verify, search the data stream to retrieve the document. The following search uses [`filter_path`](elasticsearch://reference/elasticsearch/rest-apis/common-options.md#common-options-response-filtering) to return only the [document source](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md). ```console GET my-data-stream/_search?filter_path=hits.hits._source diff --git a/manage-data/ingest/transform-enrich/ingest-pipelines.md b/manage-data/ingest/transform-enrich/ingest-pipelines.md index 67d1dc244..ed94272ef 100644 --- a/manage-data/ingest/transform-enrich/ingest-pipelines.md +++ b/manage-data/ingest/transform-enrich/ingest-pipelines.md @@ -10,7 +10,7 @@ applies_to: {{es}} ingest pipelines let you perform common transformations on your data before indexing. For example, you can use pipelines to remove fields, extract values from text, and enrich your data. -A pipeline consists of a series of configurable tasks called [processors](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/index.md). Each processor runs sequentially, making specific changes to incoming documents. After the processors have run, {{es}} adds the transformed documents to your data stream or index. +A pipeline consists of a series of configurable tasks called [processors](elasticsearch://reference/ingestion-tools/enrich-processor/index.md). Each processor runs sequentially, making specific changes to incoming documents. After the processors have run, {{es}} adds the transformed documents to your data stream or index. :::{image} ../../../images/elasticsearch-reference-ingest-process.svg :alt: Ingest pipeline diagram @@ -49,7 +49,7 @@ The **New pipeline from CSV** option lets you use a CSV to create an ingest pipe :::: -You can also use the [ingest APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-ingest) to create and manage pipelines. The following [create pipeline API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-put-pipeline) request creates a pipeline containing two [`set`](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/set-processor.md) processors followed by a [`lowercase`](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/lowercase-processor.md) processor. The processors run sequentially in the order specified. +You can also use the [ingest APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-ingest) to create and manage pipelines. The following [create pipeline API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-put-pipeline) request creates a pipeline containing two [`set`](elasticsearch://reference/ingestion-tools/enrich-processor/set-processor.md) processors followed by a [`lowercase`](elasticsearch://reference/ingestion-tools/enrich-processor/lowercase-processor.md) processor. The processors run sequentially in the order specified. ```console PUT _ingest/pipeline/my-pipeline @@ -228,12 +228,12 @@ POST _reindex ## Set a default pipeline [set-default-pipeline] -Use the [`index.default_pipeline`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-default-pipeline) index setting to set a default pipeline. {{es}} applies this pipeline to indexing requests if no `pipeline` parameter is specified. +Use the [`index.default_pipeline`](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#index-default-pipeline) index setting to set a default pipeline. {{es}} applies this pipeline to indexing requests if no `pipeline` parameter is specified. ## Set a final pipeline [set-final-pipeline] -Use the [`index.final_pipeline`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-final-pipeline) index setting to set a final pipeline. {{es}} applies this pipeline after the request or default pipeline, even if neither is specified. +Use the [`index.final_pipeline`](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#index-final-pipeline) index setting to set a final pipeline. {{es}} applies this pipeline after the request or default pipeline, even if neither is specified. ## Pipelines for {{beats}} [pipelines-for-beats] @@ -249,9 +249,9 @@ output.elasticsearch: ## Pipelines for {{fleet}} and {{agent}} [pipelines-for-fleet-elastic-agent] -{{agent}} integrations ship with default ingest pipelines that preprocess and enrich data before indexing. [{{fleet}}](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/index.md) applies these pipelines using [index templates](../../data-store/templates.md) that include [pipeline index settings](ingest-pipelines.md#set-default-pipeline). {{es}} matches these templates to your {{fleet}} data streams based on the [stream’s naming scheme](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/data-streams.md#data-streams-naming-scheme). +{{agent}} integrations ship with default ingest pipelines that preprocess and enrich data before indexing. [{{fleet}}](/reference/ingestion-tools/fleet/index.md) applies these pipelines using [index templates](../../data-store/templates.md) that include [pipeline index settings](ingest-pipelines.md#set-default-pipeline). {{es}} matches these templates to your {{fleet}} data streams based on the [stream’s naming scheme](/reference/ingestion-tools/fleet/data-streams.md#data-streams-naming-scheme). -Each default integration pipeline calls a nonexistent, unversioned `*@custom` ingest pipeline. If unaltered, this pipeline call has no effect on your data. However, you can modify this call to create custom pipelines for integrations that persist across upgrades. Refer to [Tutorial: Transform data with custom ingest pipelines](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/data-streams-pipeline-tutorial.md) to learn more. +Each default integration pipeline calls a nonexistent, unversioned `*@custom` ingest pipeline. If unaltered, this pipeline call has no effect on your data. However, you can modify this call to create custom pipelines for integrations that persist across upgrades. Refer to [Tutorial: Transform data with custom ingest pipelines](/reference/ingestion-tools/fleet/data-streams-pipeline-tutorial.md) to learn more. {{fleet}} doesn’t provide a default ingest pipeline for the **Custom logs** integration, but you can specify a pipeline for this integration using an [index template](ingest-pipelines.md#pipeline-custom-logs-index-template) or a [custom configuration](ingest-pipelines.md#pipeline-custom-logs-configuration). @@ -270,7 +270,7 @@ $$$pipeline-custom-logs-index-template$$$ } ``` -2. Create an [index template](../../data-store/templates.md) that includes your pipeline in the [`index.default_pipeline`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-default-pipeline) or [`index.final_pipeline`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-final-pipeline) index setting. Ensure the template is [data stream enabled](../../data-store/data-streams/set-up-data-stream.md#create-index-template). The template’s index pattern should match `logs--*`. +2. Create an [index template](../../data-store/templates.md) that includes your pipeline in the [`index.default_pipeline`](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#index-default-pipeline) or [`index.final_pipeline`](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#index-final-pipeline) index setting. Ensure the template is [data stream enabled](../../data-store/data-streams/set-up-data-stream.md#create-index-template). The template’s index pattern should match `logs--*`. You can create this template using {{kib}}'s [**Index Management**](../../lifecycle/index-lifecycle-management/index-management-in-kibana.md#manage-index-templates) feature or the [create index template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template). @@ -345,12 +345,12 @@ $$$pipeline-custom-logs-configuration$$$ **{{agent}} standalone** -If you run {{agent}} standalone, you can apply pipelines using an [index template](../../data-store/templates.md) that includes the [`index.default_pipeline`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-default-pipeline) or [`index.final_pipeline`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#index-final-pipeline) index setting. Alternatively, you can specify the `pipeline` policy setting in your `elastic-agent.yml` configuration. See [Install standalone {{agent}}s](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/install-standalone-elastic-agent.md). +If you run {{agent}} standalone, you can apply pipelines using an [index template](../../data-store/templates.md) that includes the [`index.default_pipeline`](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#index-default-pipeline) or [`index.final_pipeline`](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#index-final-pipeline) index setting. Alternatively, you can specify the `pipeline` policy setting in your `elastic-agent.yml` configuration. See [Install standalone {{agent}}s](/reference/ingestion-tools/fleet/install-standalone-elastic-agent.md). ## Pipelines for search indices [pipelines-in-enterprise-search] -When you create Elasticsearch indices for search use cases, for example, using the [web crawler^](https://www.elastic.co/guide/en/enterprise-search/current/crawler.html) or [connectors](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/search-connectors/index.md), these indices are automatically set up with specific ingest pipelines. These processors help optimize your content for search. See [*Ingest pipelines in Search*](../../../solutions/search/ingest-for-search.md) for more information. +When you create Elasticsearch indices for search use cases, for example, using the [web crawler^](https://www.elastic.co/guide/en/enterprise-search/current/crawler.html) or [connectors](elasticsearch://reference/ingestion-tools/search-connectors/index.md), these indices are automatically set up with specific ingest pipelines. These processors help optimize your content for search. See [*Ingest pipelines in Search*](../../../solutions/search/ingest-for-search.md) for more information. ## Access source fields in a processor [access-source-fields] @@ -390,7 +390,7 @@ PUT _ingest/pipeline/my-pipeline Use dot notation to access object fields. ::::{important} -If your document contains flattened objects, use the [`dot_expander`](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/dot-expand-processor.md) processor to expand them first. Other ingest processors cannot access flattened objects. +If your document contains flattened objects, use the [`dot_expander`](elasticsearch://reference/ingestion-tools/enrich-processor/dot-expand-processor.md) processor to expand them first. Other ingest processors cannot access flattened objects. :::: @@ -636,10 +636,10 @@ PUT _ingest/pipeline/my-pipeline ## Conditionally run a processor [conditionally-run-processor] -Each processor supports an optional `if` condition, written as a [Painless script](asciidocalypse://docs/elasticsearch/docs/reference/scripting-languages/painless/painless.md). If provided, the processor only runs when the `if` condition is `true`. +Each processor supports an optional `if` condition, written as a [Painless script](elasticsearch://reference/scripting-languages/painless/painless.md). If provided, the processor only runs when the `if` condition is `true`. ::::{important} -`if` condition scripts run in Painless’s [ingest processor context](asciidocalypse://docs/elasticsearch/docs/reference/scripting-languages/painless/painless-ingest-processor-context.md). In `if` conditions, `ctx` values are read-only. +`if` condition scripts run in Painless’s [ingest processor context](elasticsearch://reference/scripting-languages/painless/painless-ingest-processor-context.md). In `if` conditions, `ctx` values are read-only. :::: @@ -657,7 +657,7 @@ PUT _ingest/pipeline/my-pipeline } ``` -If the [`script.painless.regex.enabled`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/circuit-breaker-settings.md#script-painless-regex-enabled) cluster setting is enabled, you can use regular expressions in your `if` condition scripts. For supported syntax, see [Painless regular expressions](asciidocalypse://docs/elasticsearch/docs/reference/scripting-languages/painless/painless-regexes.md). +If the [`script.painless.regex.enabled`](elasticsearch://reference/elasticsearch/configuration-reference/circuit-breaker-settings.md#script-painless-regex-enabled) cluster setting is enabled, you can use regular expressions in your `if` condition scripts. For supported syntax, see [Painless regular expressions](elasticsearch://reference/scripting-languages/painless/painless-regexes.md). ::::{tip} If possible, avoid using regular expressions. Expensive regular expressions can slow indexing speeds. @@ -745,7 +745,7 @@ PUT _ingest/pipeline/my-pipeline } ``` -Incoming documents often contain object fields. If a processor script attempts to access a field whose parent object does not exist, {{es}} returns a `NullPointerException`. To avoid these exceptions, use [null safe operators](asciidocalypse://docs/elasticsearch/docs/reference/scripting-languages/painless/painless-operators-reference.md#null-safe-operator), such as `?.`, and write your scripts to be null safe. +Incoming documents often contain object fields. If a processor script attempts to access a field whose parent object does not exist, {{es}} returns a `NullPointerException`. To avoid these exceptions, use [null safe operators](elasticsearch://reference/scripting-languages/painless/painless-operators-reference.md#null-safe-operator), such as `?.`, and write your scripts to be null safe. For example, `ctx.network?.name.equalsIgnoreCase('Guest')` is not null safe. `ctx.network?.name` can return null. Rewrite the script as `'Guest'.equalsIgnoreCase(ctx.network?.name)`, which is null safe because `Guest` is always non-null. @@ -768,7 +768,7 @@ PUT _ingest/pipeline/my-pipeline ## Conditionally apply pipelines [conditionally-apply-pipelines] -Combine an `if` condition with the [`pipeline`](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/pipeline-processor.md) processor to apply other pipelines to documents based on your criteria. You can use this pipeline as the [default pipeline](ingest-pipelines.md#set-default-pipeline) in an [index template](../../data-store/templates.md) used to configure multiple data streams or indices. +Combine an `if` condition with the [`pipeline`](elasticsearch://reference/ingestion-tools/enrich-processor/pipeline-processor.md) processor to apply other pipelines to documents based on your criteria. You can use this pipeline as the [default pipeline](ingest-pipelines.md#set-default-pipeline) in an [index template](../../data-store/templates.md) used to configure multiple data streams or indices. ```console PUT _ingest/pipeline/one-pipeline-to-rule-them-all diff --git a/manage-data/ingest/transform-enrich/logstash-pipelines.md b/manage-data/ingest/transform-enrich/logstash-pipelines.md index 747becfef..8d295472a 100644 --- a/manage-data/ingest/transform-enrich/logstash-pipelines.md +++ b/manage-data/ingest/transform-enrich/logstash-pipelines.md @@ -28,7 +28,7 @@ After you configure {{ls}} to use centralized pipeline management, you can no lo ## Manage pipelines [logstash-pipelines-manage-pipelines] -1. [Configure centralized pipeline management](asciidocalypse://docs/logstash/docs/reference/configuring-centralized-pipelines.md). +1. [Configure centralized pipeline management](logstash://reference/configuring-centralized-pipelines.md). 2. To add a new pipeline, go to **{{project-settings}} → {{manage-app}} → {{ls-pipelines-app}}** and click **Create pipeline**. Provide the following details, then click **Create and deploy**. Pipeline ID @@ -61,4 +61,4 @@ After you configure {{ls}} to use centralized pipeline management, you can no lo To delete one or more pipelines, select their checkboxes then click **Delete**. -For more information about pipeline behavior, go to [Centralized Pipeline Management](asciidocalypse://docs/logstash/docs/reference/logstash-centralized-pipeline-management.md#_pipeline_behavior). +For more information about pipeline behavior, go to [Centralized Pipeline Management](logstash://reference/logstash-centralized-pipeline-management.md#_pipeline_behavior). diff --git a/manage-data/ingest/transform-enrich/set-up-an-enrich-processor.md b/manage-data/ingest/transform-enrich/set-up-an-enrich-processor.md index ddf1c3d16..b6fe860f4 100644 --- a/manage-data/ingest/transform-enrich/set-up-an-enrich-processor.md +++ b/manage-data/ingest/transform-enrich/set-up-an-enrich-processor.md @@ -20,7 +20,7 @@ To set up an enrich processor, follow these steps: Once you have an enrich processor set up, you can [update your enrich data](#update-enrich-data) and [update your enrich policies](#update-enrich-policies). ::::{important} -The enrich processor performs several operations and may impact the speed of your ingest pipeline. We recommend [node roles](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md) co-locating ingest and data roles to minimize remote search operations. +The enrich processor performs several operations and may impact the speed of your ingest pipeline. We recommend [node roles](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md) co-locating ingest and data roles to minimize remote search operations. We strongly recommend testing and benchmarking your enrich processors before deploying them in production. @@ -68,7 +68,7 @@ Once the enrich policy is created, you need to execute it using the [execute enr The *enrich index* contains documents from the policy’s source indices. Enrich indices always begin with `.enrich-*`, are read-only, and are [force merged](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-forcemerge). ::::{warning} -Enrich indices should only be used by the [enrich processor](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/enrich-processor.md) or the [{{esql}} `ENRICH` command](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/esql-commands.md#esql-enrich). Avoid using enrich indices for other purposes. +Enrich indices should only be used by the [enrich processor](elasticsearch://reference/ingestion-tools/enrich-processor/enrich-processor.md) or the [{{esql}} `ENRICH` command](elasticsearch://reference/query-languages/esql/esql-commands.md#esql-enrich). Avoid using enrich indices for other purposes. :::: @@ -82,7 +82,7 @@ Once you have source indices, an enrich policy, and the related enrich index in :alt: enrich processor ::: -Define an [enrich processor](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/enrich-processor.md) and add it to an ingest pipeline using the [create or update pipeline API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-put-pipeline). +Define an [enrich processor](elasticsearch://reference/ingestion-tools/enrich-processor/enrich-processor.md) and add it to an ingest pipeline using the [create or update pipeline API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ingest-put-pipeline). When defining the enrich processor, you must include at least the following: @@ -92,9 +92,9 @@ When defining the enrich processor, you must include at least the following: You also can use the `max_matches` option to set the number of enrich documents an incoming document can match. If set to the default of `1`, data is added to an incoming document’s target field as a JSON object. Otherwise, the data is added as an array. -See [Enrich](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/enrich-processor.md) for a full list of configuration options. +See [Enrich](elasticsearch://reference/ingestion-tools/enrich-processor/enrich-processor.md) for a full list of configuration options. -You also can add other [processors](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/index.md) to your ingest pipeline. +You also can add other [processors](elasticsearch://reference/ingestion-tools/enrich-processor/index.md) to your ingest pipeline. ## Ingest and enrich documents [ingest-enrich-docs] diff --git a/manage-data/lifecycle/curator.md b/manage-data/lifecycle/curator.md index ee168418d..abb578035 100644 --- a/manage-data/lifecycle/curator.md +++ b/manage-data/lifecycle/curator.md @@ -9,4 +9,4 @@ applies_to: Similar to {{ilm-cap}} ({{ilm-init}}), Elasticsearch Curator can help you manage index lifecycles. **If {{ilm-init}} provides the functionality to manage your index lifecycle and you have at least a Basic license, use {{ilm-init}} instead of Curator.** Many {{stack}} components use {{ilm-init}} by default. -If you're looking for additional functionality for managing your index lifecycle, you can read more about how Elasticsearch Curator may help in [Curator index management](asciidocalypse://docs/curator/docs/reference/index.md). +If you're looking for additional functionality for managing your index lifecycle, you can read more about how Elasticsearch Curator may help in [Curator index management](curator://reference/index.md). diff --git a/manage-data/lifecycle/data-stream.md b/manage-data/lifecycle/data-stream.md index 5839ac19c..ec0c2389e 100644 --- a/manage-data/lifecycle/data-stream.md +++ b/manage-data/lifecycle/data-stream.md @@ -21,28 +21,28 @@ To achieve that, it supports: A data stream lifecycle also supports downsampling the data stream backing indices. See [the downsampling example](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-data-lifecycle) for more details. -## How does it work? [data-streams-lifecycle-how-it-works] +## How does it work? [data-streams-lifecycle-how-it-works] -In intervals configured by [`data_streams.lifecycle.poll_interval`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#data-streams-lifecycle-poll-interval), {{es}} goes over each data stream and performs the following steps: +In intervals configured by [`data_streams.lifecycle.poll_interval`](elasticsearch://reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#data-streams-lifecycle-poll-interval), {{es}} goes over each data stream and performs the following steps: 1. Checks if the data stream has a data stream lifecycle configured, skipping any indices not part of a managed data stream. -2. Rolls over the write index of the data stream, if it fulfills the conditions defined by [`cluster.lifecycle.default.rollover`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#cluster-lifecycle-default-rollover). +2. Rolls over the write index of the data stream, if it fulfills the conditions defined by [`cluster.lifecycle.default.rollover`](elasticsearch://reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#cluster-lifecycle-default-rollover). 3. After an index is not the write index anymore (i.e. the data stream has been rolled over), automatically tail merges the index. Data stream lifecycle executes a merge operation that only targets the long tail of small segments instead of the whole shard. As the segments are organised into tiers of exponential sizes, merging the long tail of small segments is only a fraction of the cost of force merging to a single segment. The small segments would usually hold the most recent data so tail merging will focus the merging resources on the higher-value data that is most likely to keep being queried. 4. If [downsampling](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-data-lifecycle) is configured it will execute all the configured downsampling rounds. -5. Applies retention to the remaining backing indices. This means deleting the backing indices whose `generation_time` is longer than the effective retention period (read more about the [effective retention calculation](data-stream/tutorial-data-stream-retention.md#effective-retention-calculation)). The `generation_time` is only applicable to rolled over backing indices and it is either the time since the backing index got rolled over, or the time optionally configured in the [`index.lifecycle.origination_date`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#index-data-stream-lifecycle-origination-date) setting. +5. Applies retention to the remaining backing indices. This means deleting the backing indices whose `generation_time` is longer than the effective retention period (read more about the [effective retention calculation](data-stream/tutorial-data-stream-retention.md#effective-retention-calculation)). The `generation_time` is only applicable to rolled over backing indices and it is either the time since the backing index got rolled over, or the time optionally configured in the [`index.lifecycle.origination_date`](elasticsearch://reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#index-data-stream-lifecycle-origination-date) setting. -::::{important} +::::{important} We use the `generation_time` instead of the creation time because this ensures that all data in the backing index have passed the retention period. As a result, the retention period is not the exact time data gets deleted, but the minimum time data will be stored. :::: -::::{note} -Steps `2-4` apply only to backing indices that are not already managed by {{ilm-init}}, meaning that these indices either do not have an {{ilm-init}} policy defined, or if they do, they have [`index.lifecycle.prefer_ilm`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#index-lifecycle-prefer-ilm) set to `false`. +::::{note} +Steps `2-4` apply only to backing indices that are not already managed by {{ilm-init}}, meaning that these indices either do not have an {{ilm-init}} policy defined, or if they do, they have [`index.lifecycle.prefer_ilm`](elasticsearch://reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#index-lifecycle-prefer-ilm) set to `false`. :::: -## Configuring data stream lifecycle [data-stream-lifecycle-configuration] +## Configuring data stream lifecycle [data-stream-lifecycle-configuration] Since the lifecycle is configured on the data stream level, the process to configure a lifecycle on a new data stream and on an existing one differ. @@ -52,7 +52,7 @@ In the following sections, we will go through the following tutorials: * To update the lifecycle of an existing data stream you need to use the [data stream lifecycle APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-data-stream) to edit the lifecycle on the data stream itself (see [Tutorial: Update existing data stream](data-stream/tutorial-update-existing-data-stream.md)). * Migrate an existing {{ilm-init}} managed data stream to Data stream lifecycle using [Tutorial: Migrate ILM managed data stream to data stream lifecycle](data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle.md). -::::{note} +::::{note} Updating the data stream lifecycle of an existing data stream is different from updating the settings or the mapping, because it is applied on the data stream level and not on the individual backing indices. :::: diff --git a/manage-data/lifecycle/data-stream/tutorial-data-stream-retention.md b/manage-data/lifecycle/data-stream/tutorial-data-stream-retention.md index 1d85d6e4b..8c0487ca3 100644 --- a/manage-data/lifecycle/data-stream/tutorial-data-stream-retention.md +++ b/manage-data/lifecycle/data-stream/tutorial-data-stream-retention.md @@ -53,8 +53,8 @@ Retention does not define the period that the data will be removed, but the mini We define 4 different types of retention: * The data stream retention, or `data_retention`, which is the retention configured on the data stream level. It can be set via an [index template](../../data-store/templates.md) for future data streams or via the [PUT data stream lifecycle API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-data-lifecycle) for an existing data stream. When the data stream retention is not set, it implies that the data need to be kept forever. -* The global default retention, let’s call it `default_retention`, which is a retention configured via the cluster setting [`data_streams.lifecycle.retention.default`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#data-streams-lifecycle-retention-default) and will be applied to all data streams managed by data stream lifecycle that do not have `data_retention` configured. Effectively, it ensures that there will be no data streams keeping their data forever. This can be set via the [update cluster settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings). -* The global max retention, let’s call it `max_retention`, which is a retention configured via the cluster setting [`data_streams.lifecycle.retention.max`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#data-streams-lifecycle-retention-max) and will be applied to all data streams managed by data stream lifecycle. Effectively, it ensures that there will be no data streams whose retention will exceed this time period. This can be set via the [update cluster settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings). +* The global default retention, let’s call it `default_retention`, which is a retention configured via the cluster setting [`data_streams.lifecycle.retention.default`](elasticsearch://reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#data-streams-lifecycle-retention-default) and will be applied to all data streams managed by data stream lifecycle that do not have `data_retention` configured. Effectively, it ensures that there will be no data streams keeping their data forever. This can be set via the [update cluster settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings). +* The global max retention, let’s call it `max_retention`, which is a retention configured via the cluster setting [`data_streams.lifecycle.retention.max`](elasticsearch://reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#data-streams-lifecycle-retention-max) and will be applied to all data streams managed by data stream lifecycle. Effectively, it ensures that there will be no data streams whose retention will exceed this time period. This can be set via the [update cluster settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings). * The effective retention, or `effective_retention`, which is the retention applied at a data stream on a given moment. Effective retention cannot be set, it is derived by taking into account all the configured retention listed above and is calculated as it is described [here](#effective-retention-calculation). ::::{note} @@ -169,7 +169,7 @@ We see that it will remain the same with what the user configured: ## How is the effective retention applied? [effective-retention-application] -Retention is applied to the remaining backing indices of a data stream as the last step of [a data stream lifecycle run](../data-stream.md#data-streams-lifecycle-how-it-works). Data stream lifecycle will retrieve the backing indices whose `generation_time` is longer than the effective retention period and delete them. The `generation_time` is only applicable to rolled over backing indices and it is either the time since the backing index got rolled over, or the time optionally configured in the [`index.lifecycle.origination_date`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#index-data-stream-lifecycle-origination-date) setting. +Retention is applied to the remaining backing indices of a data stream as the last step of [a data stream lifecycle run](../data-stream.md#data-streams-lifecycle-how-it-works). Data stream lifecycle will retrieve the backing indices whose `generation_time` is longer than the effective retention period and delete them. The `generation_time` is only applicable to rolled over backing indices and it is either the time since the backing index got rolled over, or the time optionally configured in the [`index.lifecycle.origination_date`](elasticsearch://reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#index-data-stream-lifecycle-origination-date) setting. ::::{important} We use the `generation_time` instead of the creation time because this ensures that all data in the backing index have passed the retention period. As a result, the retention period is not the exact time data get deleted, but the minimum time data will be stored. diff --git a/manage-data/lifecycle/data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle.md b/manage-data/lifecycle/data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle.md index a5b6b842c..0c64035a6 100644 --- a/manage-data/lifecycle/data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle.md +++ b/manage-data/lifecycle/data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle.md @@ -15,7 +15,7 @@ In this tutorial we’ll look at migrating an existing data stream from [Index L To migrate a data stream from {{ilm-init}} to data stream lifecycle we’ll have to execute two steps: -1. Update the index template that’s backing the data stream to set [prefer_ilm](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#index-lifecycle-prefer-ilm) to `false`, and to configure data stream lifecycle. +1. Update the index template that’s backing the data stream to set [prefer_ilm](elasticsearch://reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#index-lifecycle-prefer-ilm) to `false`, and to configure data stream lifecycle. 2. Configure the data stream lifecycle for the *existing* data stream using the [lifecycle API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-data-lifecycle). For more details see the [migrate to data stream lifecycle](#migrate-from-ilm-to-dsl) section. @@ -127,11 +127,11 @@ Inspecting the response we’ll see that both backing indices are managed by {{i ``` 1. The name of the backing index. -2. For each backing index we display the value of the [prefer_ilm](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#index-lifecycle-prefer-ilm) configuration which will indicate if {{ilm-init}} takes precedence over data stream lifecycle in case both systems are configured for an index. +2. For each backing index we display the value of the [prefer_ilm](elasticsearch://reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#index-lifecycle-prefer-ilm) configuration which will indicate if {{ilm-init}} takes precedence over data stream lifecycle in case both systems are configured for an index. 3. The {{ilm-init}} policy configured for this index. 4. The system that manages this index (possible values are "Index Lifecycle Management", "Data stream lifecycle", or "Unmanaged") 5. The system that will manage the next generation index (the new write index of this data stream, once the data stream is rolled over). The possible values are "Index Lifecycle Management", "Data stream lifecycle", or "Unmanaged". -6. The [prefer_ilm](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#index-lifecycle-prefer-ilm) value configured in the index template that’s backing the data stream. This value will be configured for all the new backing indices. If it’s not configured in the index template the backing indices will receive the `true` default value ({{ilm-init}} takes precedence over data stream lifecycle by default as it’s currently richer in features). +6. The [prefer_ilm](elasticsearch://reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#index-lifecycle-prefer-ilm) value configured in the index template that’s backing the data stream. This value will be configured for all the new backing indices. If it’s not configured in the index template the backing indices will receive the `true` default value ({{ilm-init}} takes precedence over data stream lifecycle by default as it’s currently richer in features). 7. The {{ilm-init}} policy configured in the index template that’s backing this data stream (which will be configured on all the new backing indices, as long as it exists in the index template). @@ -140,7 +140,7 @@ Inspecting the response we’ll see that both backing indices are managed by {{i To migrate the `dsl-data-stream` to data stream lifecycle we’ll have to execute two steps: -1. Update the index template that’s backing the data stream to set [prefer_ilm](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#index-lifecycle-prefer-ilm) to `false`, and to configure data stream lifecycle. +1. Update the index template that’s backing the data stream to set [prefer_ilm](elasticsearch://reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#index-lifecycle-prefer-ilm) to `false`, and to configure data stream lifecycle. 2. Configure the data stream lifecycle for the *existing* `dsl-data-stream` using the [lifecycle API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-data-lifecycle). ::::{important} diff --git a/manage-data/lifecycle/data-tiers.md b/manage-data/lifecycle/data-tiers.md index 060841957..470789937 100644 --- a/manage-data/lifecycle/data-tiers.md +++ b/manage-data/lifecycle/data-tiers.md @@ -10,7 +10,7 @@ applies_to: # Data tiers -A *data tier* is a collection of [nodes](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md) within a cluster that share the same [data node role](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md#node-roles), and a hardware profile that’s appropriately sized for the role. Elastic recommends that nodes in the same tier share the same hardware profile to avoid [hot spotting](/troubleshoot/elasticsearch/hotspotting.md). +A *data tier* is a collection of [nodes](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md) within a cluster that share the same [data node role](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md#node-roles), and a hardware profile that’s appropriately sized for the role. Elastic recommends that nodes in the same tier share the same hardware profile to avoid [hot spotting](/troubleshoot/elasticsearch/hotspotting.md). ## Available data tiers [available-tier] @@ -24,8 +24,8 @@ The data tiers that you use, and the way that you use them, depends on the data * [Hot tier](/manage-data/lifecycle/data-tiers.md#hot-tier) nodes handle the indexing load for time series data, such as logs or metrics. They hold your most recent, most-frequently-accessed data. * [Warm tier](/manage-data/lifecycle/data-tiers.md#warm-tier) nodes hold time series data that is accessed less-frequently and rarely needs to be updated. -* [Cold tier](/manage-data/lifecycle/data-tiers.md#cold-tier) nodes hold time series data that is accessed infrequently and not normally updated. To save space, you can keep [fully mounted indices](/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md#fully-mounted) of [{{search-snaps}}](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md) on the cold tier. These fully mounted indices eliminate the need for replicas, reducing required disk space by approximately 50% compared to the regular indices. -* [Frozen tier](/manage-data/lifecycle/data-tiers.md#frozen-tier) nodes hold time series data that is accessed rarely and never updated. The frozen tier stores [partially mounted indices](/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md#partially-mounted) of [{{search-snaps}}](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md) exclusively. This extends the storage capacity even further — by up to 20 times compared to the warm tier. +* [Cold tier](/manage-data/lifecycle/data-tiers.md#cold-tier) nodes hold time series data that is accessed infrequently and not normally updated. To save space, you can keep [fully mounted indices](/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md#fully-mounted) of [{{search-snaps}}](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md) on the cold tier. These fully mounted indices eliminate the need for replicas, reducing required disk space by approximately 50% compared to the regular indices. +* [Frozen tier](/manage-data/lifecycle/data-tiers.md#frozen-tier) nodes hold time series data that is accessed rarely and never updated. The frozen tier stores [partially mounted indices](/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md#partially-mounted) of [{{search-snaps}}](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md) exclusively. This extends the storage capacity even further — by up to 20 times compared to the warm tier. ::::{tip} The performance of an {{es}} node is often limited by the performance of the underlying storage and hardware profile. For example hardware profiles, refer to Elastic Cloud’s [instance configurations](asciidocalypse://docs/cloud/docs/reference/cloud-hosted/hardware.md). Review our recommendations for optimizing your storage for [indexing](/deploy-manage/production-guidance/optimize-performance/indexing-speed.md#indexing-use-faster-hardware) and [search](/deploy-manage/production-guidance/optimize-performance/search-speed.md#search-use-faster-hardware). @@ -69,7 +69,7 @@ Time series data can move to the warm tier once it is being queried less frequen When you no longer need to search time series data regularly, it can move from the warm tier to the cold tier. While still searchable, this tier is typically optimized for lower storage costs rather than search speed. -For better storage savings, you can keep [fully mounted indices](/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md#fully-mounted) of [{{search-snaps}}](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md) on the cold tier. Unlike regular indices, these fully mounted indices don’t require replicas for reliability. In the event of a failure, they can recover data from the underlying snapshot instead. This potentially halves the local storage needed for the data. A snapshot repository is required to use fully mounted indices in the cold tier. Fully mounted indices are read-only. +For better storage savings, you can keep [fully mounted indices](/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md#fully-mounted) of [{{search-snaps}}](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md) on the cold tier. Unlike regular indices, these fully mounted indices don’t require replicas for reliability. In the event of a failure, they can recover data from the underlying snapshot instead. This potentially halves the local storage needed for the data. A snapshot repository is required to use fully mounted indices in the cold tier. Fully mounted indices are read-only. Alternatively, you can use the cold tier to store regular indices with replicas instead of using {{search-snaps}}. This lets you store older data on less expensive hardware but doesn’t reduce required disk space compared to the warm tier. @@ -452,7 +452,7 @@ We recommend you use [dedicated nodes](/deploy-manage/distributed-architecture/c ## Data tier index allocation [data-tier-allocation] -The [`index.routing.allocation.include._tier_preference`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/data-tier-allocation.md#tier-preference-allocation-filter) setting determines which tier the index should be allocated to. +The [`index.routing.allocation.include._tier_preference`](elasticsearch://reference/elasticsearch/index-settings/data-tier-allocation.md#tier-preference-allocation-filter) setting determines which tier the index should be allocated to. When you create an index, by default {{es}} sets the `_tier_preference` to `data_content` to automatically allocate the index shards to the content tier. @@ -467,7 +467,7 @@ You can override this setting after index creation by [updating the index settin This setting also accepts multiple tiers in order of preference. This prevents indices from remaining unallocated if no nodes are available in the preferred tier. For example, when {{ilm}} migrates an index to the cold phase, it sets the index `_tier_preference` to `data_cold,data_warm,data_hot`. -To remove the data tier preference setting, set the `_tier_preference` value to `null`. This allows the index to allocate to any data node within the cluster. Setting the `_tier_preference` to `null` does not restore the default value. Note that, in the case of managed indices, a [migrate](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-migrate.md) action might apply a new value in its place. +To remove the data tier preference setting, set the `_tier_preference` value to `null`. This allows the index to allocate to any data node within the cluster. Setting the `_tier_preference` to `null` does not restore the default value. Note that, in the case of managed indices, a [migrate](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-migrate.md) action might apply a new value in its place. ### Determine the current data tier preference [data-tier-allocation-value] @@ -486,4 +486,4 @@ This setting will not unallocate a currently allocated shard, but might prevent ### Automatic data tier migration [data-tier-migration] -{{ilm-init}} automatically transitions managed indices through the available data tiers using the [migrate](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-migrate.md) action. By default, this action is automatically injected in every phase. You can explicitly specify the migrate action with `"enabled": false` to [disable automatic migration](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-migrate.md#ilm-disable-migrate-ex), for example, if you’re using the [allocate action](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-allocate.md) to manually specify allocation rules. +{{ilm-init}} automatically transitions managed indices through the available data tiers using the [migrate](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-migrate.md) action. By default, this action is automatically injected in every phase. You can explicitly specify the migrate action with `"enabled": false` to [disable automatic migration](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-migrate.md#ilm-disable-migrate-ex), for example, if you’re using the [allocate action](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-allocate.md) to manually specify allocation rules. diff --git a/manage-data/lifecycle/index-lifecycle-management.md b/manage-data/lifecycle/index-lifecycle-management.md index edd8b8ee3..5d4395c8f 100644 --- a/manage-data/lifecycle/index-lifecycle-management.md +++ b/manage-data/lifecycle/index-lifecycle-management.md @@ -40,7 +40,7 @@ To use {{ilm-init}}, all nodes in a cluster must run the same version. Although * **Shrink**: Reduces the number of primary shards in an index. * **Force merge**: Triggers a [force merge](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-forcemerge) to reduce the number of segments in an index’s shards. * **Delete**: Permanently remove an index, including all of its data and metadata. -* [And more](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/index.md) +* [And more](elasticsearch://reference/elasticsearch/index-lifecycle-actions/index.md) Each action has options you can use to specify index behavior and characteristics like: @@ -58,7 +58,7 @@ For example, if you are indexing metrics data from a fleet of ATMs into Elastics 3. After 7 days, move the index into the cold phase and move it to less expensive hardware. 4. Delete the index once the required 30 day retention period is reached. -**Learn about all available actions in [Index lifecycle actions](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/index.md).** +**Learn about all available actions in [Index lifecycle actions](elasticsearch://reference/elasticsearch/index-lifecycle-actions/index.md).** ## Create and manage {{ilm-init}} policies diff --git a/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md b/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md index c08f0ad0d..2e822e069 100644 --- a/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md +++ b/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md @@ -205,7 +205,7 @@ To switch an index’s lifecycle policy, follow these steps: 2. The remove policy API removes all {{ilm-init}} metadata from the index and doesn’t consider the index’s lifecycle status. This can leave indices in an undesired state. - For example, the [`forcemerge`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-forcemerge.md) action temporarily closes an index before reopening it. Removing an index’s {{ilm-init}} policy during a `forcemerge` can leave the index closed indefinitely. + For example, the [`forcemerge`](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-forcemerge.md) action temporarily closes an index before reopening it. Removing an index’s {{ilm-init}} policy during a `forcemerge` can leave the index closed indefinitely. After policy removal, use the [get index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get) to check an index’s state . Target a data stream or alias to get the state of all its indices. diff --git a/manage-data/lifecycle/index-lifecycle-management/index-lifecycle.md b/manage-data/lifecycle/index-lifecycle-management/index-lifecycle.md index fbcce4ad2..e045b4b12 100644 --- a/manage-data/lifecycle/index-lifecycle-management/index-lifecycle.md +++ b/manage-data/lifecycle/index-lifecycle-management/index-lifecycle.md @@ -34,7 +34,7 @@ If you use {{es}}'s security features, {{ilm-init}} performs operations as the u The minimum age defaults to zero, which causes {{ilm-init}} to move indices to the next phase as soon as all actions in the current phase complete. ::::{note} -If an index has been [rolled over](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-rollover.md), then the `min_age` value is relative to the time the index was rolled over, not the index creation time. [Learn more](../../../troubleshoot/elasticsearch/index-lifecycle-management-errors.md#min-age-calculation). +If an index has been [rolled over](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-rollover.md), then the `min_age` value is relative to the time the index was rolled over, not the index creation time. [Learn more](../../../troubleshoot/elasticsearch/index-lifecycle-management-errors.md#min-age-calculation). :::: @@ -59,42 +59,42 @@ When an index enters a phase, {{ilm-init}} caches the phase definition in the in * Hot - * [Set Priority](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-set-priority.md) - * [Unfollow](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-unfollow.md) - * [Rollover](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-rollover.md) - * [Read-Only](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-readonly.md) - * [Downsample](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-downsample.md) - * [Shrink](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-shrink.md) - * [Force Merge](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-forcemerge.md) - * [Searchable Snapshot](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md) + * [Set Priority](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-set-priority.md) + * [Unfollow](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-unfollow.md) + * [Rollover](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-rollover.md) + * [Read-Only](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-readonly.md) + * [Downsample](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-downsample.md) + * [Shrink](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-shrink.md) + * [Force Merge](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-forcemerge.md) + * [Searchable Snapshot](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md) * Warm - * [Set Priority](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-set-priority.md) - * [Unfollow](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-unfollow.md) - * [Read-Only](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-readonly.md) - * [Downsample](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-downsample.md) - * [Allocate](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-allocate.md) - * [Migrate](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-migrate.md) - * [Shrink](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-shrink.md) - * [Force Merge](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-forcemerge.md) + * [Set Priority](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-set-priority.md) + * [Unfollow](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-unfollow.md) + * [Read-Only](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-readonly.md) + * [Downsample](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-downsample.md) + * [Allocate](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-allocate.md) + * [Migrate](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-migrate.md) + * [Shrink](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-shrink.md) + * [Force Merge](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-forcemerge.md) * Cold - * [Set Priority](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-set-priority.md) - * [Unfollow](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-unfollow.md) - * [Read-Only](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-readonly.md) - * [Downsample](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-downsample.md) - * [Searchable Snapshot](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md) - * [Allocate](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-allocate.md) - * [Migrate](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-migrate.md) + * [Set Priority](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-set-priority.md) + * [Unfollow](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-unfollow.md) + * [Read-Only](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-readonly.md) + * [Downsample](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-downsample.md) + * [Searchable Snapshot](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md) + * [Allocate](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-allocate.md) + * [Migrate](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-migrate.md) * Frozen - * [Unfollow](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-unfollow.md) - * [Searchable Snapshot](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md) + * [Unfollow](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-unfollow.md) + * [Searchable Snapshot](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-searchable-snapshot.md) * Delete - * [Wait For Snapshot](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-wait-for-snapshot.md) - * [Delete](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-delete.md) + * [Wait For Snapshot](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-wait-for-snapshot.md) + * [Delete](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-delete.md) diff --git a/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md b/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md index ae3be01f4..bce55cb0d 100644 --- a/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md +++ b/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md @@ -35,7 +35,7 @@ Investigate your indices and perform operations from the **Indices** view. * To show details and perform operations such as close, forcemerge, and flush, click the index name. To perform operations on multiple indices, select their checkboxes and then open the **Manage** menu. For more information on managing indices, refer to [Index APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-indices). * To filter the list of indices, use the search bar or click a badge. Badges indicate if an index is a [follower index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ccr-follow), a [rollup index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-rollup-get-rollup-index-caps), or [frozen](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-unfreeze). -* To drill down into the index [mappings](../../data-store/mapping.md), [settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md), and statistics, click an index name. From this view, you can navigate to **Discover** to further explore the documents in the index. +* To drill down into the index [mappings](../../data-store/mapping.md), [settings](elasticsearch://reference/elasticsearch/index-settings/index.md), and statistics, click an index name. From this view, you can navigate to **Discover** to further explore the documents in the index. :::{image} ../../../images/elasticsearch-reference-management_index_details.png :alt: Index Management UI @@ -102,7 +102,7 @@ In this tutorial, you’ll create an index template and use it to configure two ::: 2. Define index settings. These are optional. For this tutorial, leave this section blank. -3. Define a mapping that contains an [object](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/object.md) field named `geo` with a child [`geo_point`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/geo-point.md) field named `coordinates`: +3. Define a mapping that contains an [object](elasticsearch://reference/elasticsearch/mapping-reference/object.md) field named `geo` with a child [`geo_point`](elasticsearch://reference/elasticsearch/mapping-reference/geo-point.md) field named `coordinates`: :::{image} ../../../images/elasticsearch-reference-management-index-templates-mappings.png :alt: Mapped fields page @@ -192,7 +192,7 @@ Use the **Enrich Policies** view to add data from your existing indices to incom * The source indices that store enrich data as documents * The fields from the source indices used to match incoming documents * The enrich fields containing enrich data from the source indices that you want to add to incoming documents -* An optional [query](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-match-all-query.md). +* An optional [query](elasticsearch://reference/query-languages/query-dsl-match-all-query.md). :::{image} ../../../images/elasticsearch-reference-management-enrich-policies.png :alt: Enrich policies diff --git a/manage-data/lifecycle/index-lifecycle-management/manage-existing-indices.md b/manage-data/lifecycle/index-lifecycle-management/manage-existing-indices.md index 2648c3902..d6c82e7f7 100644 --- a/manage-data/lifecycle/index-lifecycle-management/manage-existing-indices.md +++ b/manage-data/lifecycle/index-lifecycle-management/manage-existing-indices.md @@ -27,7 +27,7 @@ Define a separate policy for your older indices that omits the rollover action. Keep in mind that policies applied to existing indices compare the `min_age` for each phase to the original creation date of the index, and might proceed through multiple phases immediately. If your policy performs resource-intensive operations like force merge, you don’t want to have a lot of indices performing those operations all at once when you switch over to {{ilm-init}}. -You can specify different `min_age` values in the policy you use for existing indices, or set [`index.lifecycle.origination_date`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/index-lifecycle-management-settings.md#index-lifecycle-origination-date) to control how the index age is calculated. +You can specify different `min_age` values in the policy you use for existing indices, or set [`index.lifecycle.origination_date`](elasticsearch://reference/elasticsearch/configuration-reference/index-lifecycle-management-settings.md#index-lifecycle-origination-date) to control how the index age is calculated. Once all pre-{{ilm-init}} indices have been aged out and removed, you can delete the policy you used to manage them. diff --git a/manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md b/manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md index 8fd0b551a..58ffd8473 100644 --- a/manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md +++ b/manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md @@ -8,9 +8,9 @@ applies_to: # Migrate index allocation filters to node roles [migrate-index-allocation-filters] -If you currently use [custom node attributes](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md#custom-node-attributes) and [attribute-based allocation filters](../../../deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/index-level-shard-allocation.md) to move indices through [data tiers](../data-tiers.md) in a [hot-warm-cold architecture](https://www.elastic.co/blog/implementing-hot-warm-cold-in-elasticsearch-with-index-lifecycle-management), we recommend that you switch to using the built-in node roles and automatic [data tier allocation](../data-tiers.md#data-tier-allocation). Using node roles enables {{ilm-init}} to automatically move indices between data tiers. +If you currently use [custom node attributes](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md#custom-node-attributes) and [attribute-based allocation filters](../../../deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/index-level-shard-allocation.md) to move indices through [data tiers](../data-tiers.md) in a [hot-warm-cold architecture](https://www.elastic.co/blog/implementing-hot-warm-cold-in-elasticsearch-with-index-lifecycle-management), we recommend that you switch to using the built-in node roles and automatic [data tier allocation](../data-tiers.md#data-tier-allocation). Using node roles enables {{ilm-init}} to automatically move indices between data tiers. -::::{note} +::::{note} While we recommend relying on automatic data tier allocation to manage your data in a hot-warm-cold architecture, you can still use attribute-based allocation filters to control shard allocation for other purposes. :::: @@ -18,7 +18,7 @@ While we recommend relying on automatic data tier allocation to manage your data {{ech}} and {{ece}} can perform the migration automatically. For self-managed deployments, you need to manually update your configuration, ILM policies, and indices to switch to node roles. -## Automatically migrate to node roles on {{ech}} or {{ece}} [cloud-migrate-to-node-roles] +## Automatically migrate to node roles on {{ech}} or {{ece}} [cloud-migrate-to-node-roles] If you are using node attributes from the default deployment template in {{ech}} or {{ece}}, you will be prompted to switch to node roles when you: @@ -30,13 +30,13 @@ These actions automatically update your cluster configuration and {{ilm-init}} p If you use custom index templates, check them after the automatic migration completes and remove any [attribute-based allocation filters](../../../deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/index-level-shard-allocation.md). -::::{note} +::::{note} You do not need to take any further action after the automatic migration. The following manual steps are only necessary if you do not allow the automatic migration or have a self-managed deployment. :::: -## Migrate to node roles on self-managed deployments [on-prem-migrate-to-node-roles] +## Migrate to node roles on self-managed deployments [on-prem-migrate-to-node-roles] To switch to using node roles: @@ -46,9 +46,9 @@ To switch to using node roles: 4. Update existing indices to [set a tier preference](#set-tier-preference). -### Assign data nodes to a data tier [assign-data-tier] +### Assign data nodes to a data tier [assign-data-tier] -Configure the appropriate roles for each data node to assign it to one or more data tiers: `data_hot`, `data_content`, `data_warm`, `data_cold`, or `data_frozen`. A node can also have other [roles](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md). By default, new nodes are configured with all roles. +Configure the appropriate roles for each data node to assign it to one or more data tiers: `data_hot`, `data_content`, `data_warm`, `data_cold`, or `data_frozen`. A node can also have other [roles](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md). By default, new nodes are configured with all roles. When you add a data tier to an {{ech}} deployment, one or more nodes are automatically configured with the corresponding role. To explicitly change the role of a node in an {{ech}} deployment, use the [Update deployment API](../../../deploy-manage/deploy/elastic-cloud/manage-deployments-using-elastic-cloud-api.md#ec_update_a_deployment). Replace the node’s `node_type` configuration with the appropriate `node_roles`. For example, the following configuration adds the node to the hot and content tiers, and enables it to act as an ingest node, remote, and transform node. @@ -69,19 +69,19 @@ node.roles [ data_hot, data_content ] ``` -### Remove custom allocation settings from existing {{ilm-init}} policies [remove-custom-allocation-settings] +### Remove custom allocation settings from existing {{ilm-init}} policies [remove-custom-allocation-settings] -Update the allocate action for each lifecycle phase to remove the attribute-based allocation settings. {{ilm-init}} will inject a [migrate](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-migrate.md) action into each phase to automatically transition the indices through the data tiers. +Update the allocate action for each lifecycle phase to remove the attribute-based allocation settings. {{ilm-init}} will inject a [migrate](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-migrate.md) action into each phase to automatically transition the indices through the data tiers. If the allocate action does not set the number of replicas, remove the allocate action entirely. (An empty allocate action is invalid.) -::::{important} +::::{important} The policy must specify the corresponding phase for each data tier in your architecture. Each phase must be present so {{ilm-init}} can inject the migrate action to move indices through the data tiers. If you don’t need to perform any other actions, the phase can be empty. For example, if you enable the warm and cold data tiers for a deployment, your policy must include the hot, warm, and cold phases. :::: -### Stop setting the custom hot attribute on new indices [stop-setting-custom-hot-attribute] +### Stop setting the custom hot attribute on new indices [stop-setting-custom-hot-attribute] When you create a data stream, its first backing index is now automatically assigned to `data_hot` nodes. Similarly, when you directly create an index, it is automatically assigned to `data_content` nodes. @@ -96,14 +96,14 @@ If you’re using a custom index template, update it to remove the [attribute-ba To completely avoid the issues that raise when mixing the tier preference and custom attribute routing setting we also recommend updating all the legacy, composable, and component templates to remove the [attribute-based allocation filters](../../../deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/index-level-shard-allocation.md) from the settings they configure. -### Set a tier preference for existing indices [set-tier-preference] +### Set a tier preference for existing indices [set-tier-preference] -{{ilm-init}} automatically transitions managed indices through the available data tiers by automatically injecting a [migrate action](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/ilm-migrate.md) into each phase. +{{ilm-init}} automatically transitions managed indices through the available data tiers by automatically injecting a [migrate action](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-migrate.md) into each phase. To enable {{ilm-init}} to move an *existing* managed index through the data tiers, update the index settings to: 1. Remove the custom allocation filter by setting it to `null`. -2. Set the [tier preference](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/data-tier-allocation.md#tier-preference-allocation-filter). +2. Set the [tier preference](elasticsearch://reference/elasticsearch/index-settings/data-tier-allocation.md#tier-preference-allocation-filter). For example, if your old template set the `data` attribute to `hot` to allocate shards to the hot tier, set the `data` attribute to `null` and set the `_tier_preference` to `data_hot`. diff --git a/manage-data/lifecycle/index-lifecycle-management/rollover.md b/manage-data/lifecycle/index-lifecycle-management/rollover.md index 55db51d2a..e981bb835 100644 --- a/manage-data/lifecycle/index-lifecycle-management/rollover.md +++ b/manage-data/lifecycle/index-lifecycle-management/rollover.md @@ -20,7 +20,7 @@ We recommend using [data streams](https://www.elastic.co/docs/api/doc/elasticsea Each data stream requires an [index template](../../data-store/templates.md) that contains: * A name or wildcard (`*`) pattern for the data stream. -* The data stream’s timestamp field. This field must be mapped as a [`date`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/date.md) or [`date_nanos`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/date_nanos.md) field data type and must be included in every document indexed to the data stream. +* The data stream’s timestamp field. This field must be mapped as a [`date`](elasticsearch://reference/elasticsearch/mapping-reference/date.md) or [`date_nanos`](elasticsearch://reference/elasticsearch/mapping-reference/date_nanos.md) field data type and must be included in every document indexed to the data stream. * The mappings and settings applied to each backing index when it’s created. Data streams are designed for append-only data, where the data stream name can be used as the operations (read, write, rollover, shrink etc.) target. If your use case requires data to be updated in place, you can instead manage your time series data using [index aliases](../../data-store/aliases.md). However, there are a few more configuration steps and concepts: @@ -29,28 +29,28 @@ Data streams are designed for append-only data, where the data stream name can b * An *index alias* that references the entire set of indices. * A single index designated as the *write index*. This is the active index that handles all write requests. On each rollover, the new index becomes the write index. -::::{note} +::::{note} When an index is rolled over, the previous index’s age is updated to reflect the rollover time. This date, rather than the index’s `creation_date`, is used in {{ilm}} `min_age` phase calculations. [Learn more](../../../troubleshoot/elasticsearch/index-lifecycle-management-errors.md#min-age-calculation). :::: -## Automatic rollover [ilm-automatic-rollover] +## Automatic rollover [ilm-automatic-rollover] {{ilm-init}} and the data stream lifecycle (in [preview]]) enable you to automatically roll over to a new index based on conditions like the index size, document count, or age. When a rollover is triggered, a new index is created, the write alias is updated to point to the new index, and all subsequent updates are written to the new index. -::::{tip} +::::{tip} Rolling over to a new index based on size, document count, or age is preferable to time-based rollovers. Rolling over at an arbitrary time often results in many small indices, which can have a negative impact on performance and resource usage. :::: -::::{important} +::::{important} Empty indices will not be rolled over, even if they have an associated `max_age` that would otherwise result in a roll over occurring. A policy can override this behavior, and explicitly opt in to rolling over empty indices, by adding a `"min_docs": 0` condition. This can also be disabled on a cluster-wide basis by setting `indices.lifecycle.rollover.only_if_has_documents` to `false`. :::: -::::{important} +::::{important} The rollover action implicitly always rolls over a data stream or alias if one or more shards contain 200000000 or more documents. Normally a shard will reach 50GB long before it reaches 200M documents, but this isn’t the case for space efficient data sets. Search performance will very likely suffer if a shard contains more than 200M documents. This is the reason of the builtin limit. :::: diff --git a/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md b/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md index cdc92f763..625837f68 100644 --- a/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md +++ b/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md @@ -18,7 +18,7 @@ When you continuously index timestamped documents into {{es}}, you typically use To automate rollover and management of a data stream with {{ilm-init}}, you: -1. [Create a lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#ilm-gs-create-policy) that defines the appropriate [phases](index-lifecycle.md) and [actions](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-lifecycle-actions/index.md). +1. [Create a lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#ilm-gs-create-policy) that defines the appropriate [phases](index-lifecycle.md) and [actions](elasticsearch://reference/elasticsearch/index-lifecycle-actions/index.md). 2. [Create an index template](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#ilm-gs-apply-policy) to [create the data stream](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#ilm-gs-create-the-data-stream) and apply the ILM policy and the indices settings and mappings configurations for the backing indices. 3. [Verify indices are moving through the lifecycle phases](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md#ilm-gs-check-progress) as expected. diff --git a/manage-data/lifecycle/rollup/understanding-groups.md b/manage-data/lifecycle/rollup/understanding-groups.md index c430a5b10..51dbf639d 100644 --- a/manage-data/lifecycle/rollup/understanding-groups.md +++ b/manage-data/lifecycle/rollup/understanding-groups.md @@ -111,7 +111,7 @@ Ultimately, when configuring `groups` for a job, think in terms of how you might ## Calendar vs fixed time intervals [rollup-understanding-group-intervals] -Each rollup-job must have a date histogram group with a defined interval. {{es}} understands both [calendar and fixed time intervals](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-bucket-datehistogram-aggregation.md#calendar_and_fixed_intervals). Fixed time intervals are fairly easy to understand; `60s` means sixty seconds. But what does `1M` mean? One month of time depends on which month we are talking about, some months are longer or shorter than others. This is an example of calendar time and the duration of that unit depends on context. Calendar units are also affected by leap-seconds, leap-years, etc. +Each rollup-job must have a date histogram group with a defined interval. {{es}} understands both [calendar and fixed time intervals](elasticsearch://reference/data-analysis/aggregations/search-aggregations-bucket-datehistogram-aggregation.md#calendar_and_fixed_intervals). Fixed time intervals are fairly easy to understand; `60s` means sixty seconds. But what does `1M` mean? One month of time depends on which month we are talking about, some months are longer or shorter than others. This is an example of calendar time and the duration of that unit depends on context. Calendar units are also affected by leap-seconds, leap-years, etc. This is important because the buckets generated by rollup are in either calendar or fixed intervals and this limits how you can query them later. See [Requests must be multiples of the config](rollup-search-limitations.md#rollup-search-limitations-intervals). diff --git a/manage-data/use-case-use-elasticsearch-to-manage-time-series-data.md b/manage-data/use-case-use-elasticsearch-to-manage-time-series-data.md index 04836910a..c43b4185c 100644 --- a/manage-data/use-case-use-elasticsearch-to-manage-time-series-data.md +++ b/manage-data/use-case-use-elasticsearch-to-manage-time-series-data.md @@ -34,7 +34,7 @@ The steps for setting up data tiers vary based on your deployment type: :::::: ::::::{tab-item} Self-managed -To assign a node to a data tier, add the respective [node role](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md#node-roles) to the node’s `elasticsearch.yml` file. Changing an existing node’s roles requires a [rolling restart](../deploy-manage/maintenance/start-stop-services/full-cluster-restart-rolling-restart-procedures.md#restart-cluster-rolling). +To assign a node to a data tier, add the respective [node role](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md#node-roles) to the node’s `elasticsearch.yml` file. Changing an existing node’s roles requires a [rolling restart](../deploy-manage/maintenance/start-stop-services/full-cluster-restart-rolling-restart-procedures.md#restart-cluster-rolling). ```yaml # Content tier @@ -94,7 +94,7 @@ Use any of the following repository types with searchable snapshots: * [AWS S3](../deploy-manage/tools/snapshot-and-restore/s3-repository.md) * [Google Cloud Storage](../deploy-manage/tools/snapshot-and-restore/google-cloud-storage-repository.md) * [Azure Blob Storage](../deploy-manage/tools/snapshot-and-restore/azure-repository.md) -* [Hadoop Distributed File Store (HDFS)](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch-plugins/repository-hdfs.md) +* [Hadoop Distributed File Store (HDFS)](elasticsearch://reference/elasticsearch-plugins/repository-hdfs.md) * [Shared filesystems](../deploy-manage/tools/snapshot-and-restore/shared-file-system-repository.md) such as NFS * [Read-only HTTP and HTTPS repositories](../deploy-manage/tools/snapshot-and-restore/read-only-url-repository.md) @@ -249,13 +249,13 @@ If you use a custom application, you need to set up your own data stream. A data When creating your component templates, include: -* A [`date`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/date.md) or [`date_nanos`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/date_nanos.md) mapping for the `@timestamp` field. If you don’t specify a mapping, {{es}} maps `@timestamp` as a `date` field with default options. +* A [`date`](elasticsearch://reference/elasticsearch/mapping-reference/date.md) or [`date_nanos`](elasticsearch://reference/elasticsearch/mapping-reference/date_nanos.md) mapping for the `@timestamp` field. If you don’t specify a mapping, {{es}} maps `@timestamp` as a `date` field with default options. * Your lifecycle policy in the `index.lifecycle.name` index setting. ::::{tip} Use the [Elastic Common Schema (ECS)](https://www.elastic.co/guide/en/ecs/current) when mapping your fields. ECS fields integrate with several {{stack}} features by default. -If you’re unsure how to map your fields, use [runtime fields](data-store/mapping/define-runtime-fields-in-search-request.md) to extract fields from [unstructured content](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/keyword.md#mapping-unstructured-content) at search time. For example, you can index a log message to a `wildcard` field and later extract IP addresses and other data from this field during a search. +If you’re unsure how to map your fields, use [runtime fields](data-store/mapping/define-runtime-fields-in-search-request.md) to extract fields from [unstructured content](elasticsearch://reference/elasticsearch/mapping-reference/keyword.md#mapping-unstructured-content) at search time. For example, you can index a log message to a `wildcard` field and later extract IP addresses and other data from this field during a search. :::: @@ -307,7 +307,7 @@ PUT _component_template/my-settings Use your component templates to create an index template. Specify: -* One or more index patterns that match the data stream’s name. We recommend using our [data stream naming scheme](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/data-streams.md#data-streams-naming-scheme). +* One or more index patterns that match the data stream’s name. We recommend using our [data stream naming scheme](/reference/ingestion-tools/fleet/data-streams.md#data-streams-naming-scheme). * That the template is data stream enabled. * Any component templates that contain your mappings and index settings. * A priority higher than `200` to avoid collisions with built-in templates. See [Avoid index pattern collisions](data-store/templates.md#avoid-index-pattern-collisions). diff --git a/raw-migrated-files/apm-agent-android/apm-agent-android/release-notes.md b/raw-migrated-files/apm-agent-android/apm-agent-android/release-notes.md index ca48fcb62..13525e270 100644 --- a/raw-migrated-files/apm-agent-android/apm-agent-android/release-notes.md +++ b/raw-migrated-files/apm-agent-android/apm-agent-android/release-notes.md @@ -5,6 +5,6 @@ This functionality is in technical preview and may be changed or removed in a fu :::: -* [Android agent version 0.x](asciidocalypse://docs/apm-agent-android/docs/release-notes/index.md) +* [Android agent version 0.x](apm-agent-android://release-notes/index.md) diff --git a/raw-migrated-files/cloud-on-k8s/cloud-on-k8s/k8s-orchestration.md b/raw-migrated-files/cloud-on-k8s/cloud-on-k8s/k8s-orchestration.md index 5aca84f13..6ab3f1e40 100644 --- a/raw-migrated-files/cloud-on-k8s/cloud-on-k8s/k8s-orchestration.md +++ b/raw-migrated-files/cloud-on-k8s/cloud-on-k8s/k8s-orchestration.md @@ -169,7 +169,7 @@ Advanced users may force an upgrade by manually deleting Pods themselves. The de Operations that reduce the number of nodes in the cluster cannot make progress without user intervention, if the Elasticsearch index replica settings are incompatible with the intended downscale. Specifically, if the Elasticsearch index settings demand a higher number of shard copies than data nodes in the cluster after the downscale operation, ECK cannot migrate the data away from the node about to be removed. You can address this in the following ways: * Adjust the Elasticsearch [index settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings) to a number of replicas that allow the desired node removal. -* Use [`auto_expand_replicas`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index-modules.md#dynamic-index-settings) to automatically adjust the replicas to the number of data nodes in the cluster. +* Use [`auto_expand_replicas`](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#dynamic-index-settings) to automatically adjust the replicas to the number of data nodes in the cluster. ## Advanced control during rolling upgrades [k8s-advanced-upgrade-control] diff --git a/raw-migrated-files/cloud/cloud-enterprise/ece-add-custom-bundle-plugin.md b/raw-migrated-files/cloud/cloud-enterprise/ece-add-custom-bundle-plugin.md index 460d85a66..4d26f8dc8 100644 --- a/raw-migrated-files/cloud/cloud-enterprise/ece-add-custom-bundle-plugin.md +++ b/raw-migrated-files/cloud/cloud-enterprise/ece-add-custom-bundle-plugin.md @@ -353,7 +353,7 @@ You do not need to do this step if you are using default filename and password ( } ``` -4. To use this bundle, you can refer it in the [GeoIP processor](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/geoip-processor.md) of an ingest pipeline as `MyGeoLite2-City.mmdb` under `database_file` such as: +4. To use this bundle, you can refer it in the [GeoIP processor](elasticsearch://reference/ingestion-tools/enrich-processor/geoip-processor.md) of an ingest pipeline as `MyGeoLite2-City.mmdb` under `database_file` such as: ```sh ... diff --git a/raw-migrated-files/cloud/cloud-enterprise/ece-getting-started-node-js.md b/raw-migrated-files/cloud/cloud-enterprise/ece-getting-started-node-js.md index 41e59a552..246dba52e 100644 --- a/raw-migrated-files/cloud/cloud-enterprise/ece-getting-started-node-js.md +++ b/raw-migrated-files/cloud/cloud-enterprise/ece-getting-started-node-js.md @@ -158,7 +158,7 @@ async function run() { run().catch(console.log) ``` -When using the [client.index](asciidocalypse://docs/elasticsearch-js/docs/reference/api-reference.md#_index) API, the request automatically creates the `game-of-thrones` index if it doesn’t already exist, as well as document IDs for each indexed document if they are not explicitly specified. +When using the [client.index](elasticsearch-js://reference/api-reference.md#_index) API, the request automatically creates the `game-of-thrones` index if it doesn’t already exist, as well as document IDs for each indexed document if they are not explicitly specified. ## Search and modify data [ece_search_and_modify_data] @@ -205,7 +205,7 @@ async function update() { update().catch(console.log) ``` -This [more comprehensive list of API examples](asciidocalypse://docs/elasticsearch-js/docs/reference/examples.md) includes bulk operations, checking the existence of documents, updating by query, deleting, scrolling, and SQL queries. To learn more, check the complete [API reference](asciidocalypse://docs/elasticsearch-js/docs/reference/api-reference.md). +This [more comprehensive list of API examples](elasticsearch-js://reference/examples.md) includes bulk operations, checking the existence of documents, updating by query, deleting, scrolling, and SQL queries. To learn more, check the complete [API reference](elasticsearch-js://reference/api-reference.md). ## Switch to API key authentication [ece_switch_to_api_key_authentication] @@ -294,5 +294,5 @@ Schema : When the example code was run an index mapping was created automatically. The field types were selected by {{es}} based on the content seen when the first record was ingested, and updated as new fields appeared in the data. It would be more efficient to specify the fields and field types in advance to optimize performance. Refer to the Elastic Common Schema documentation and Field Type documentation when you are designing the schema for your production use cases. Ingest -: For more advanced scenarios, this [bulk ingestion](asciidocalypse://docs/elasticsearch-js/docs/reference/bulk_examples.md) reference gives an example of the `bulk` API that makes it possible to perform multiple operations in a single call. This bulk example also explicitly specifies document IDs. If you have a lot of documents to index, using bulk to batch document operations is significantly faster than submitting requests individually. +: For more advanced scenarios, this [bulk ingestion](elasticsearch-js://reference/bulk_examples.md) reference gives an example of the `bulk` API that makes it possible to perform multiple operations in a single call. This bulk example also explicitly specifies document IDs. If you have a lot of documents to index, using bulk to batch document operations is significantly faster than submitting requests individually. diff --git a/raw-migrated-files/cloud/cloud-enterprise/ece-getting-started-python.md b/raw-migrated-files/cloud/cloud-enterprise/ece-getting-started-python.md index ac72d9da9..0b580691b 100644 --- a/raw-migrated-files/cloud/cloud-enterprise/ece-getting-started-python.md +++ b/raw-migrated-files/cloud/cloud-enterprise/ece-getting-started-python.md @@ -282,7 +282,7 @@ es.get(index='lord-of-the-rings', id='2EkAzngB_pyHD3p65UMt') 'birthplace': 'The Shire'}} ``` -For frequently used API calls with the Python client, check [Examples](asciidocalypse://docs/elasticsearch-py/docs/reference/examples.md). +For frequently used API calls with the Python client, check [Examples](elasticsearch-py://reference/examples.md). ## Switch to API key authentication [ece_switch_to_api_key_authentication_2] @@ -342,7 +342,7 @@ es = Elasticsearch( Check [Create API key API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-api-key) to learn more about API Keys and [Security privileges](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md) to understand which privileges are needed. If you are not sure what the right combination of privileges for your custom application is, you can enable [audit logging](../../../deploy-manage/monitor/logging-configuration/enabling-audit-logs.md) on {{es}} to find out what privileges are being used. To learn more about how logging works on {{ece}}, check [Monitoring Elastic Cloud deployment logs and metrics](https://www.elastic.co/blog/monitoring-elastic-cloud-deployment-logs-and-metrics). -For more information on refreshing an index, searching, updating, and deleting, check the [elasticsearch-py examples](asciidocalypse://docs/elasticsearch-py/docs/reference/examples.md). +For more information on refreshing an index, searching, updating, and deleting, check the [elasticsearch-py examples](elasticsearch-py://reference/examples.md). ### Best practices [ece_best_practices_2] @@ -357,5 +357,5 @@ Schema : When the example code is run, an index mapping is created automatically. The field types are selected by {{es}} based on the content seen when the first record was ingested, and updated as new fields appeared in the data. It would be more efficient to specify the fields and field types in advance to optimize performance. Refer to the Elastic Common Schema documentation and Field Type documentation when you design the schema for your production use cases. Ingest -: For more advanced scenarios, [Bulk helpers](asciidocalypse://docs/elasticsearch-py/docs/reference/client-helpers.md#bulk-helpers) gives examples for the `bulk` API that makes it possible to perform multiple operations in a single call. If you have a lot of documents to index, using bulk to batch document operations is significantly faster than submitting requests individually. +: For more advanced scenarios, [Bulk helpers](elasticsearch-py://reference/client-helpers.md#bulk-helpers) gives examples for the `bulk` API that makes it possible to perform multiple operations in a single call. If you have a lot of documents to index, using bulk to batch document operations is significantly faster than submitting requests individually. diff --git a/raw-migrated-files/cloud/cloud-enterprise/ece-getting-started-search-use-cases-beats-logstash.md b/raw-migrated-files/cloud/cloud-enterprise/ece-getting-started-search-use-cases-beats-logstash.md index ce0c668c9..fa585aa75 100644 --- a/raw-migrated-files/cloud/cloud-enterprise/ece-getting-started-search-use-cases-beats-logstash.md +++ b/raw-migrated-files/cloud/cloud-enterprise/ece-getting-started-search-use-cases-beats-logstash.md @@ -245,7 +245,7 @@ Now the Filebeat and Metricbeat are set up, let’s configure a {{ls}} pipeline 1. {{ls}} listens for Beats input on the default port of 5044. Only one line is needed to do this. {{ls}} can handle input from many Beats of the same and also of varying types (Metricbeat, Filebeat, and others). 2. This sends output to the standard output, which displays through your command line interface. This plugin enables you to verify the data before you send it to {{es}}, in a later step. -3. Save the new *beats.conf* file in your Logstash folder. To learn more about the file format and options, check [{{ls}} Configuration Examples](asciidocalypse://docs/logstash/docs/reference/config-examples.md). +3. Save the new *beats.conf* file in your Logstash folder. To learn more about the file format and options, check [{{ls}} Configuration Examples](logstash://reference/config-examples.md). ## Output {{ls}} data to stdout [ece-beats-logstash-stdout] @@ -481,7 +481,7 @@ In this section, you configure {{ls}} to send the Metricbeat and Filebeat data t ::::{note} In this guide, you manually launch each of the Elastic stack applications through the command line interface. In production, you may prefer to configure {{ls}}, Metricbeat, and Filebeat to run as System Services. Check the following pages for the steps to configure each application to run as a service: -* [Running {{ls}} as a service on Debian or RPM](asciidocalypse://docs/logstash/docs/reference/running-logstash.md) +* [Running {{ls}} as a service on Debian or RPM](logstash://reference/running-logstash.md) * [Metricbeat and systemd](asciidocalypse://docs/beats/docs/reference/metricbeat/running-with-systemd.md) * [Start filebeat](asciidocalypse://docs/beats/docs/reference/filebeat/filebeat-starting.md) diff --git a/raw-migrated-files/cloud/cloud-enterprise/ece-getting-started-search-use-cases-db-logstash.md b/raw-migrated-files/cloud/cloud-enterprise/ece-getting-started-search-use-cases-db-logstash.md index 95c1b5816..19eb056ee 100644 --- a/raw-migrated-files/cloud/cloud-enterprise/ece-getting-started-search-use-cases-db-logstash.md +++ b/raw-migrated-files/cloud/cloud-enterprise/ece-getting-started-search-use-cases-db-logstash.md @@ -1,6 +1,6 @@ # Ingest data from a relational database into Elastic Cloud Enterprise [ece-getting-started-search-use-cases-db-logstash] -This guide explains how to ingest data from a relational database into Elastic Cloud Enterprise through [Logstash](asciidocalypse://docs/logstash/docs/reference/index.md), using the Logstash [JDBC input plugin](asciidocalypse://docs/logstash/docs/reference/plugins-inputs-jdbc.md). It demonstrates how Logstash can be used to efficiently copy records and to receive updates from a relational database, and then send them into {{es}} in an Elastic Cloud Enterprise deployment. +This guide explains how to ingest data from a relational database into Elastic Cloud Enterprise through [Logstash](logstash://reference/index.md), using the Logstash [JDBC input plugin](logstash://reference/plugins-inputs-jdbc.md). It demonstrates how Logstash can be used to efficiently copy records and to receive updates from a relational database, and then send them into {{es}} in an Elastic Cloud Enterprise deployment. The code and methods presented here have been tested with MySQL. They should work with other relational databases. @@ -189,13 +189,13 @@ Let’s set up a sample Logstash input pipeline to ingest data from your new JDB : The Logstash JDBC plugin does not come packaged with JDBC driver libraries. The JDBC driver library must be passed explicitly into the plugin using the `jdbc_driver_library` configuration option. tracking_column - : This parameter specifies the field `unix_ts_in_secs` that tracks the last document read by Logstash from MySQL, stored on disk in [logstash_jdbc_last_run](asciidocalypse://docs/logstash/docs/reference/plugins-inputs-jdbc.md#plugins-inputs-jdbc-last_run_metadata_path). The parameter determines the starting value for documents that Logstash requests in the next iteration of its polling loop. The value stored in `logstash_jdbc_last_run` can be accessed in a SELECT statement as `sql_last_value`. + : This parameter specifies the field `unix_ts_in_secs` that tracks the last document read by Logstash from MySQL, stored on disk in [logstash_jdbc_last_run](logstash://reference/plugins-inputs-jdbc.md#plugins-inputs-jdbc-last_run_metadata_path). The parameter determines the starting value for documents that Logstash requests in the next iteration of its polling loop. The value stored in `logstash_jdbc_last_run` can be accessed in a SELECT statement as `sql_last_value`. unix_ts_in_secs : The field generated by the SELECT statement, which contains the `modification_time` as a standard [Unix timestamp](https://en.wikipedia.org/wiki/Unix_time) (seconds since the epoch). The field is referenced by the `tracking column`. A Unix timestamp is used for tracking progress rather than a normal timestamp, as a normal timestamp may cause errors due to the complexity of correctly converting back and forth between UMT and the local timezone. sql_last_value - : This is a [built-in parameter](asciidocalypse://docs/logstash/docs/reference/plugins-inputs-jdbc.md#_predefined_parameters) containing the starting point of the current iteration of the Logstash polling loop, and it is referenced in the SELECT statement line of the JDBC input configuration. This parameter is set to the most recent value of `unix_ts_in_secs`, which is read from `.logstash_jdbc_last_run`. This value is the starting point for documents returned by the MySQL query that is executed in the Logstash polling loop. Including this variable in the query guarantees that we’re not resending data that is already stored in Elasticsearch. + : This is a [built-in parameter](logstash://reference/plugins-inputs-jdbc.md#_predefined_parameters) containing the starting point of the current iteration of the Logstash polling loop, and it is referenced in the SELECT statement line of the JDBC input configuration. This parameter is set to the most recent value of `unix_ts_in_secs`, which is read from `.logstash_jdbc_last_run`. This value is the starting point for documents returned by the MySQL query that is executed in the Logstash polling loop. Including this variable in the query guarantees that we’re not resending data that is already stored in Elasticsearch. schedule : This uses cron syntax to specify how often Logstash should poll MySQL for changes. The specification `*/5 * * * * *` tells Logstash to contact MySQL every 5 seconds. Input from this plugin can be scheduled to run periodically according to a specific schedule. This scheduling syntax is powered by [rufus-scheduler](https://github.com/jmettraux/rufus-scheduler). The syntax is cron-like with some extensions specific to Rufus (for example, timezone support). @@ -286,7 +286,7 @@ In this section, we configure Logstash to send the MySQL data to Elasticsearch. ``` 1. Use the Cloud ID of your Elastic Cloud Enterprise deployment. You can include or omit the `:` prefix at the beginning of the Cloud ID. Both versions work fine. Find your Cloud ID by going to the {{kib}} main menu and selecting Management > Integrations, and then selecting View deployment details. - 2. the default username is `elastic`. It is not recommended to use the `elastic` account for ingesting data as this is a superuser. We recommend using a user with reduced permissions, or an API Key with permissions specific to the indices or data streams that will be written to. Check [Configuring security in Logstash](asciidocalypse://docs/logstash/docs/reference/secure-connection.md) for information on roles and API Keys. Use the password provided when you created the deployment if using the `elastic` user, or the password used when creating a new ingest user with the roles specified in the [Configuring security in Logstash](asciidocalypse://docs/logstash/docs/reference/secure-connection.md) documentation. + 2. the default username is `elastic`. It is not recommended to use the `elastic` account for ingesting data as this is a superuser. We recommend using a user with reduced permissions, or an API Key with permissions specific to the indices or data streams that will be written to. Check [Configuring security in Logstash](logstash://reference/secure-connection.md) for information on roles and API Keys. Use the password provided when you created the deployment if using the `elastic` user, or the password used when creating a new ingest user with the roles specified in the [Configuring security in Logstash](logstash://reference/secure-connection.md) documentation. 3. This line is only used when you have a self signed certificate for your Elastic Cloud Enterprise proxy. If needed, specify the full path to the PEM formatted root certificate (Root CA) used for the Elastic Cloud Enterprise proxy. You can retrieve the certificate chain from your ECE system by following the instructions in [Get existing ECE security certificates](../../../deploy-manage/security/secure-your-elastic-cloud-enterprise-installation/manage-security-certificates.md#ece-existing-security-certificates). Save the final certificate in the chain to a file. In this example, the file is named `elastic-ece-ca-cert.pem`. diff --git a/raw-migrated-files/cloud/cloud-enterprise/ece-getting-started-search-use-cases-node-logs.md b/raw-migrated-files/cloud/cloud-enterprise/ece-getting-started-search-use-cases-node-logs.md index 60a66bd43..42886a9b2 100644 --- a/raw-migrated-files/cloud/cloud-enterprise/ece-getting-started-search-use-cases-node-logs.md +++ b/raw-migrated-files/cloud/cloud-enterprise/ece-getting-started-search-use-cases-node-logs.md @@ -1,6 +1,6 @@ # Ingest logs from a Node.js web application using Filebeat [ece-getting-started-search-use-cases-node-logs] -This guide demonstrates how to ingest logs from a Node.js web application and deliver them securely into an Elastic Cloud Enterprise deployment. You’ll set up Filebeat to monitor a JSON-structured log file that has standard Elastic Common Schema (ECS) formatted fields, and you’ll then view real-time visualizations of the log events in Kibana as requests are made to the Node.js server. While Node.js is used for this example, this approach to monitoring log output is applicable across many client types. Check the list of [available ECS logging plugins](asciidocalypse://docs/ecs-logging/docs/reference/intro.md#_get_started). +This guide demonstrates how to ingest logs from a Node.js web application and deliver them securely into an Elastic Cloud Enterprise deployment. You’ll set up Filebeat to monitor a JSON-structured log file that has standard Elastic Common Schema (ECS) formatted fields, and you’ll then view real-time visualizations of the log events in Kibana as requests are made to the Node.js server. While Node.js is used for this example, this approach to monitoring log output is applicable across many client types. Check the list of [available ECS logging plugins](ecs-logging://reference/intro.md#_get_started). This guide presents: @@ -33,7 +33,7 @@ For the three following packages, you can create a working directory to install npm install winston ``` -* The [Elastic Common Schema (ECS) formatter](asciidocalypse://docs/ecs-logging-nodejs/docs/reference/winston.md) for the Node.js winston logger - This plugin formats your Node.js logs into an ECS structured JSON format ideally suited for ingestion into Elasticsearch. To install the ECS winston logger, run the following command in your working directory so that the package is installed in the same location as the winston package: +* The [Elastic Common Schema (ECS) formatter](ecs-logging-nodejs://reference/winston.md) for the Node.js winston logger - This plugin formats your Node.js logs into an ECS structured JSON format ideally suited for ingestion into Elasticsearch. To install the ECS winston logger, run the following command in your working directory so that the package is installed in the same location as the winston package: ```sh npm install @elastic/ecs-winston-format diff --git a/raw-migrated-files/cloud/cloud-enterprise/ece-getting-started-search-use-cases-python-logs.md b/raw-migrated-files/cloud/cloud-enterprise/ece-getting-started-search-use-cases-python-logs.md index b2f41b287..47296125a 100644 --- a/raw-migrated-files/cloud/cloud-enterprise/ece-getting-started-search-use-cases-python-logs.md +++ b/raw-migrated-files/cloud/cloud-enterprise/ece-getting-started-search-use-cases-python-logs.md @@ -1,6 +1,6 @@ # Ingest logs from a Python application using Filebeat [ece-getting-started-search-use-cases-python-logs] -This guide demonstrates how to ingest logs from a Python application and deliver them securely into an Elastic Cloud Enterprise deployment. You’ll set up Filebeat to monitor a JSON-structured log file that has standard Elastic Common Schema (ECS) formatted fields, and you’ll then view real-time visualizations of the log events in {{kib}} as they occur. While Python is used for this example, this approach to monitoring log output is applicable across many client types. Check the list of [available ECS logging plugins](asciidocalypse://docs/ecs-logging/docs/reference/intro.md). +This guide demonstrates how to ingest logs from a Python application and deliver them securely into an Elastic Cloud Enterprise deployment. You’ll set up Filebeat to monitor a JSON-structured log file that has standard Elastic Common Schema (ECS) formatted fields, and you’ll then view real-time visualizations of the log events in {{kib}} as they occur. While Python is used for this example, this approach to monitoring log output is applicable across many client types. Check the list of [available ECS logging plugins](ecs-logging://reference/intro.md). You are going to learn how to: @@ -14,7 +14,7 @@ You are going to learn how to: ## Prerequisites [ece_prerequisites_2] -To complete these steps you need to have [Python](https://www.python.org/) installed on your system as well as the [Elastic Common Schema (ECS) logger](asciidocalypse://docs/ecs-logging-python/docs/reference/installation.md) for the Python logging library. +To complete these steps you need to have [Python](https://www.python.org/) installed on your system as well as the [Elastic Common Schema (ECS) logger](ecs-logging-python://reference/installation.md) for the Python logging library. To install *ecs-logging-python*, run: diff --git a/raw-migrated-files/cloud/cloud-enterprise/ece-manage-apm-settings.md b/raw-migrated-files/cloud/cloud-enterprise/ece-manage-apm-settings.md index 3e62e7fcc..6cc885008 100644 --- a/raw-migrated-files/cloud/cloud-enterprise/ece-manage-apm-settings.md +++ b/raw-migrated-files/cloud/cloud-enterprise/ece-manage-apm-settings.md @@ -24,7 +24,7 @@ Users running {{stack}} versions 7.16 or 7.17 need to manually configure TLS. Th Pick one of the following options: -1. Upload and configure a publicly signed {{es}} TLS certificates. Check [Encrypt traffic in clusters with a self-managed Fleet Server](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/secure-connections.md) for details. +1. Upload and configure a publicly signed {{es}} TLS certificates. Check [Encrypt traffic in clusters with a self-managed Fleet Server](/reference/ingestion-tools/fleet/secure-connections.md) for details. 2. Change the {{es}} hosts where {{agent}}s send data from the default public URL, to the internal URL. In {{kib}}, navigate to **Fleet** and select the **Elastic Cloud agent policy**. Click **Fleet settings** and update the {{es}} hosts URL. For example, if the current URL is `https://123abc.us-central1.gcp.foundit.no:9244`, change it to `http://123abc.containerhost:9244`. diff --git a/raw-migrated-files/cloud/cloud-enterprise/ece-manage-integrations-server.md b/raw-migrated-files/cloud/cloud-enterprise/ece-manage-integrations-server.md index dcb7819e3..c1d5ff17b 100644 --- a/raw-migrated-files/cloud/cloud-enterprise/ece-manage-integrations-server.md +++ b/raw-migrated-files/cloud/cloud-enterprise/ece-manage-integrations-server.md @@ -1,6 +1,6 @@ # Manage your Integrations Server [ece-manage-integrations-server] -For deployments that are version 8.0 and later, you have the option to add a combined [Application Performance Monitoring (APM) Server](/solutions/observability/apps/application-performance-monitoring-apm.md) and [Fleet Server](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/index.md) to your deployment. APM allows you to monitor software services and applications in real time, turning that data into documents stored in the Elasticsearch cluster. Fleet allows you to centrally manage Elastic Agents on many hosts. +For deployments that are version 8.0 and later, you have the option to add a combined [Application Performance Monitoring (APM) Server](/solutions/observability/apps/application-performance-monitoring-apm.md) and [Fleet Server](/reference/ingestion-tools/fleet/index.md) to your deployment. APM allows you to monitor software services and applications in real time, turning that data into documents stored in the Elasticsearch cluster. Fleet allows you to centrally manage Elastic Agents on many hosts. As part of provisioning, the APM Server and Fleet Server are already configured to work with Elasticsearch and Kibana. At the end of provisioning, you are shown the secret token to configure communication between the APM Server and the backend [APM Agents](https://www.elastic.co/guide/en/apm/agent/index.html). The APM Agents get deployed within your services and applications. diff --git a/raw-migrated-files/cloud/cloud-enterprise/ece-manage-kibana-settings.md b/raw-migrated-files/cloud/cloud-enterprise/ece-manage-kibana-settings.md index 2ac1b3c62..32ae76d99 100644 --- a/raw-migrated-files/cloud/cloud-enterprise/ece-manage-kibana-settings.md +++ b/raw-migrated-files/cloud/cloud-enterprise/ece-manage-kibana-settings.md @@ -3,7 +3,7 @@ Elastic Cloud Enterprise supports most of the standard Kibana and X-Pack settings. Through a YAML editor in the console, you can append Kibana properties to the `kibana.yml` file. Your changes to the configuration file are read on startup. ::::{important} -Be aware that some settings that could break your cluster if set incorrectly and that the syntax might change between major versions. Before upgrading, be sure to review the full list of the [latest Kibana settings and syntax](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md). +Be aware that some settings that could break your cluster if set incorrectly and that the syntax might change between major versions. Before upgrading, be sure to review the full list of the [latest Kibana settings and syntax](kibana://reference/configuration-reference/general-settings.md). :::: diff --git a/raw-migrated-files/cloud/cloud-enterprise/ece-upgrade-deployment.md b/raw-migrated-files/cloud/cloud-enterprise/ece-upgrade-deployment.md index 4d57619e6..b3ec18b83 100644 --- a/raw-migrated-files/cloud/cloud-enterprise/ece-upgrade-deployment.md +++ b/raw-migrated-files/cloud/cloud-enterprise/ece-upgrade-deployment.md @@ -43,7 +43,7 @@ To upgrade a cluster in Elastic Cloud Enterprise: 4. Select one of the available software versions. Let the user interface guide you through the steps for upgrading a deployment. When you save your changes, your deployment configuration is updated to the new version. ::::{tip} - You cannot downgrade after upgrading, so plan ahead to make sure that your applications still work after upgrading. For more information on changes that might affect your applications, check [Breaking changes](asciidocalypse://docs/elasticsearch/docs/release-notes/breaking-changes.md). + You cannot downgrade after upgrading, so plan ahead to make sure that your applications still work after upgrading. For more information on changes that might affect your applications, check [Breaking changes](elasticsearch://release-notes/breaking-changes.md). :::: 5. If you are upgrading to version 6.6 and earlier, major upgrades require a full cluster restart to complete the upgrade process. diff --git a/raw-migrated-files/cloud/cloud-enterprise/ece-upgrade.md b/raw-migrated-files/cloud/cloud-enterprise/ece-upgrade.md index dc5dad8a1..78dc45d8d 100644 --- a/raw-migrated-files/cloud/cloud-enterprise/ece-upgrade.md +++ b/raw-migrated-files/cloud/cloud-enterprise/ece-upgrade.md @@ -24,7 +24,7 @@ The following table shows the recommended upgrade paths from older {{ece}} versi ## The upgrade process [ece-upgrade-overview] -Upgrading Elastic Cloud Enterprise works by replacing the [containers](asciidocalypse://docs/docs-content/docs/reference/glossary/index.md#glossary-container) that ECE itself requires to run on each host. Upgrading ECE does not touch any of the containers that run Elasticsearch clusters and Kibana instances. Each container that needs to be upgraded is renamed and stopped, followed by the creation of a new container with an upgraded instance of the ECE software and its dependencies. When the upgrade process has completed successfully, it cleans up after itself and removes the old containers. +Upgrading Elastic Cloud Enterprise works by replacing the [containers](/reference/glossary/index.md#glossary-container) that ECE itself requires to run on each host. Upgrading ECE does not touch any of the containers that run Elasticsearch clusters and Kibana instances. Each container that needs to be upgraded is renamed and stopped, followed by the creation of a new container with an upgraded instance of the ECE software and its dependencies. When the upgrade process has completed successfully, it cleans up after itself and removes the old containers. The upgrade process creates a `frc-upgraders-monitor` container on the host where you initiate the process that performs the following actions: diff --git a/raw-migrated-files/cloud/cloud-heroku/ech-add-user-settings.md b/raw-migrated-files/cloud/cloud-heroku/ech-add-user-settings.md index 11c447e1b..f9ed62a51 100644 --- a/raw-migrated-files/cloud/cloud-heroku/ech-add-user-settings.md +++ b/raw-migrated-files/cloud/cloud-heroku/ech-add-user-settings.md @@ -35,7 +35,7 @@ Elasticsearch Add-On for Heroku supports the following `elasticsearch.yml` setti The following general settings are supported: $$$http-cors-settings$$$`http.cors.*` -: Enables cross-origin resource sharing (CORS) settings for the [HTTP module](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md). +: Enables cross-origin resource sharing (CORS) settings for the [HTTP module](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md). ::::{note} If your use case depends on the ability to receive CORS requests and you have a cluster that was provisioned prior to January 25th 2019, you must manually set `http.cors.enabled` to `true` and allow a specific set of hosts with `http.cors.allow-origin`. Applying these changes in your Elasticsearch configuration allows cross-origin resource sharing requests. @@ -43,13 +43,13 @@ $$$http-cors-settings$$$`http.cors.*` `http.compression` -: Support for [HTTP compression](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md) when possible (with Accept-Encoding). Defaults to `true`. +: Support for [HTTP compression](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md) when possible (with Accept-Encoding). Defaults to `true`. `transport.compress` -: Configures [transport compression](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md) for node-to-node traffic. +: Configures [transport compression](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md) for node-to-node traffic. `transport.compression_scheme` -: Configures [transport compression](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md) for node-to-node traffic. +: Configures [transport compression](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md) for node-to-node traffic. `repositories.url.allowed_urls` : Enables explicit allowing of [read-only URL repositories](../../../deploy-manage/tools/snapshot-and-restore/read-only-url-repository.md). @@ -61,7 +61,7 @@ $$$http-cors-settings$$$`http.cors.*` : To learn more on how to configure reindex SSL user settings, check [configuring reindex SSL parameters](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex). `script.painless.regex.enabled` -: Enables [regular expressions](asciidocalypse://docs/elasticsearch/docs/reference/scripting-languages/painless/brief-painless-walkthrough.md#modules-scripting-painless-regex) for the Painless scripting language. +: Enables [regular expressions](elasticsearch://reference/scripting-languages/painless/brief-painless-walkthrough.md#modules-scripting-painless-regex) for the Painless scripting language. `action.auto_create_index` : [Automatically create index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-create) if it doesn’t already exist. @@ -94,19 +94,19 @@ $$$http-cors-settings$$$`http.cors.*` The following circuit breaker settings are supported: `indices.breaker.total.limit` -: Configures [the parent circuit breaker settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/circuit-breaker-settings.md#parent-circuit-breaker). +: Configures [the parent circuit breaker settings](elasticsearch://reference/elasticsearch/configuration-reference/circuit-breaker-settings.md#parent-circuit-breaker). `indices.breaker.fielddata.limit` -: Configures [the limit for the fielddata breaker](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/circuit-breaker-settings.md#fielddata-circuit-breaker). +: Configures [the limit for the fielddata breaker](elasticsearch://reference/elasticsearch/configuration-reference/circuit-breaker-settings.md#fielddata-circuit-breaker). `indices.breaker.fielddata.overhead` -: Configures [a constant that all field data estimations are multiplied with to determine a final estimation](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/circuit-breaker-settings.md#fielddata-circuit-breaker). +: Configures [a constant that all field data estimations are multiplied with to determine a final estimation](elasticsearch://reference/elasticsearch/configuration-reference/circuit-breaker-settings.md#fielddata-circuit-breaker). `indices.breaker.request.limit` -: Configures [the limit for the request breaker](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/circuit-breaker-settings.md#request-circuit-breaker). +: Configures [the limit for the request breaker](elasticsearch://reference/elasticsearch/configuration-reference/circuit-breaker-settings.md#request-circuit-breaker). `indices.breaker.request.overhead` -: Configures [a constant that all request estimations are multiplied by to determine a final estimation](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/circuit-breaker-settings.md#request-circuit-breaker). +: Configures [a constant that all request estimations are multiplied by to determine a final estimation](elasticsearch://reference/elasticsearch/configuration-reference/circuit-breaker-settings.md#request-circuit-breaker). ### Indexing pressure settings [echindexing_pressure_settings] @@ -114,7 +114,7 @@ The following circuit breaker settings are supported: The following indexing pressure settings are supported: `indexing_pressure.memory.limit` -: Configures [the indexing pressure settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/pressure.md). +: Configures [the indexing pressure settings](elasticsearch://reference/elasticsearch/index-settings/pressure.md). ### X-Pack [echx_pack] @@ -128,7 +128,7 @@ The following indexing pressure settings are supported: #### All supported versions [echall_supported_versions] `xpack.ml.inference_model.time_to_live` -: Sets the duration of time that the trained models are cached. Check [{{ml-cap}} settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/machine-learning-settings.md). +: Sets the duration of time that the trained models are cached. Check [{{ml-cap}} settings](elasticsearch://reference/elasticsearch/configuration-reference/machine-learning-settings.md). `xpack.security.loginAssistanceMessage` : Adds a message to the login screen. Useful for displaying corporate messages. @@ -146,10 +146,10 @@ The following indexing pressure settings are supported: : Defines when the watch should start, based on date and time [Learn more](/explore-analyze/alerts-cases/watcher/trigger-schedule.md). `xpack.notification.email.html.sanitization.*` -: Enables [email notification settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/watcher-settings.md) to sanitize HTML elements in emails that are sent. +: Enables [email notification settings](elasticsearch://reference/elasticsearch/configuration-reference/watcher-settings.md) to sanitize HTML elements in emails that are sent. `xpack.monitoring.collection.interval` -: Controls [how often data samples are collected](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/monitoring-settings.md#monitoring-collection-settings). +: Controls [how often data samples are collected](elasticsearch://reference/elasticsearch/configuration-reference/monitoring-settings.md#monitoring-collection-settings). `xpack.monitoring.collection.min_interval_seconds` : Specifies the minimum number of seconds that a time bucket in a chart can represent. If you modify the `xpack.monitoring.collection.interval`, use the same value in this setting. @@ -158,10 +158,10 @@ The following indexing pressure settings are supported: $$$xpack-monitoring-history-duration$$$`xpack.monitoring.history.duration` -: Sets the [retention duration](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/monitoring-settings.md#monitoring-collection-settings) beyond which the indices created by a monitoring exporter will be automatically deleted. +: Sets the [retention duration](elasticsearch://reference/elasticsearch/configuration-reference/monitoring-settings.md#monitoring-collection-settings) beyond which the indices created by a monitoring exporter will be automatically deleted. `xpack.watcher.history.cleaner_service.enabled` -: Controls [whether old watcher indices are automatically deleted](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/watcher-settings.md#general-notification-settings). +: Controls [whether old watcher indices are automatically deleted](elasticsearch://reference/elasticsearch/configuration-reference/watcher-settings.md#general-notification-settings). `xpack.http.ssl.cipher_suites` : Controls the list of supported cipher suites for all outgoing TLS connections. @@ -197,16 +197,16 @@ The following search settings are supported: The following disk-based allocation settings are supported: `cluster.routing.allocation.disk.threshold_enabled` -: Enable or disable [disk allocation](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#disk-based-shard-allocation) decider and defaults to `true`. +: Enable or disable [disk allocation](elasticsearch://reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#disk-based-shard-allocation) decider and defaults to `true`. `cluster.routing.allocation.disk.watermark.low` -: Configures [disk-based shard allocation’s low watermark](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#disk-based-shard-allocation). +: Configures [disk-based shard allocation’s low watermark](elasticsearch://reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#disk-based-shard-allocation). `cluster.routing.allocation.disk.watermark.high` -: Configures [disk-based shard allocation’s high watermark](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#disk-based-shard-allocation). +: Configures [disk-based shard allocation’s high watermark](elasticsearch://reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#disk-based-shard-allocation). `cluster.routing.allocation.disk.watermark.flood_stage` -: Configures [disk-based shard allocation’s flood_stage](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#disk-based-shard-allocation). +: Configures [disk-based shard allocation’s flood_stage](elasticsearch://reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#disk-based-shard-allocation). ::::{tip} Remember to update user settings for alerts when performing a major version upgrade. diff --git a/raw-migrated-files/cloud/cloud-heroku/ech-enable-logging-and-monitoring.md b/raw-migrated-files/cloud/cloud-heroku/ech-enable-logging-and-monitoring.md index 50c6b171d..6a8df2a48 100644 --- a/raw-migrated-files/cloud/cloud-heroku/ech-enable-logging-and-monitoring.md +++ b/raw-migrated-files/cloud/cloud-heroku/ech-enable-logging-and-monitoring.md @@ -13,12 +13,12 @@ Monitoring consists of two components: The steps in this section cover only the enablement of the monitoring and logging features in Elasticsearch Add-On for Heroku. For more information on how to use the monitoring features, refer to [Monitor a cluster](../../../deploy-manage/monitor.md). -### Before you begin [ech-logging-and-monitoring-limitations] +### Before you begin [ech-logging-and-monitoring-limitations] Some limitations apply when you use monitoring on Elasticsearch Add-On for Heroku. To learn more, check the monitoring [restrictions and limitations](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md). -### Monitoring for production use [ech-logging-and-monitoring-production] +### Monitoring for production use [ech-logging-and-monitoring-production] For production use, you should send your deployment logs and metrics to a dedicated monitoring deployment. Monitoring indexes logs and metrics into {{es}} and these indexes consume storage, memory, and CPU cycles like any other index. By using a separate monitoring deployment, you avoid affecting your other production deployments and can view the logs and metrics even when a production deployment is unavailable. @@ -35,15 +35,15 @@ How many monitoring deployments you use depends on your requirements: Logs and metrics that get sent to a dedicated monitoring {{es}} deployment [may not be cleaned up automatically](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md#ech-logging-and-monitoring-retention) and might require some additional steps to remove excess data periodically. -### Retention of monitoring daily indices [ech-logging-and-monitoring-retention] +### Retention of monitoring daily indices [ech-logging-and-monitoring-retention] -#### Stack versions 8.0 and above [ech-logging-and-monitoring-retention-8] +#### Stack versions 8.0 and above [ech-logging-and-monitoring-retention-8] When you enable monitoring in Elasticsearch Add-On for Heroku, your monitoring indices are retained for a certain period by default. After the retention period has passed, the monitoring indices are deleted automatically. The retention period is configured in the `.monitoring-8-ilm-policy` index lifecycle policy. To view or edit the policy open {{kib}} **Stack management > Data > Index Lifecycle Policies**. -### Sending monitoring data to itself (self monitoring) [ech-logging-and-monitoring-retention-self-monitoring] +### Sending monitoring data to itself (self monitoring) [ech-logging-and-monitoring-retention-self-monitoring] $$$ech-logging-and-monitoring-retention-7$$$ When you enable self-monitoring in Elasticsearch Add-On for Heroku, your monitoring indices are retained for a certain period by default. After the retention period has passed, the monitoring indices are deleted automatically. Monitoring data is retained for three days by default or as specified by the [`xpack.monitoring.history.duration` user setting](../../../deploy-manage/deploy/elastic-cloud/edit-stack-settings.md#xpack-monitoring-history-duration). @@ -65,7 +65,7 @@ PUT /_cluster/settings ``` -### Sending monitoring data to a dedicated monitoring deployment [ech-logging-and-monitoring-retention-dedicated-monitoring] +### Sending monitoring data to a dedicated monitoring deployment [ech-logging-and-monitoring-retention-dedicated-monitoring] When [monitoring for production use](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md#ech-logging-and-monitoring-production), where you configure your deployments **to send monitoring data to a dedicated monitoring deployment** for indexing, this retention period does not apply. Monitoring indices on a dedicated monitoring deployment are retained until you remove them. There are three options open to you: @@ -98,17 +98,17 @@ When [monitoring for production use](../../../deploy-manage/monitor/stack-monito * To retain monitoring indices on a dedicated monitoring deployment as is without deleting them automatically, no additional steps are required other than making sure that you do not enable the monitoring deployment to send monitoring data to itself. You should also monitor the deployment for disk space usage and upgrade your deployment periodically, if necessary. -### Retention of logging indices [ech-logging-and-monitoring-log-retention] +### Retention of logging indices [ech-logging-and-monitoring-log-retention] An ILM policy is pre-configured to manage log retention. The policy can be adjusted according to your requirements. -### Index management [ech-logging-and-monitoring-index-management-ilm] +### Index management [ech-logging-and-monitoring-index-management-ilm] When sending monitoring data to a deployment, you can configure [Index Lifecycle Management (ILM)](../../../manage-data/lifecycle/index-lifecycle-management.md) to manage retention of your monitoring and logging indices. When sending logs to a deployment, an ILM policy is pre-configured to manage log retention and the policy can be customized to your needs. -### Enable logging and monitoring [ech-enable-logging-and-monitoring-steps] +### Enable logging and monitoring [ech-enable-logging-and-monitoring-steps] Elasticsearch Add-On for Heroku manages the installation and configuration of the monitoring agent for you. When you enable monitoring on a deployment, you are configuring where the monitoring agent for your current deployment should send its logs and metrics. @@ -125,23 +125,23 @@ To enable monitoring on your deployment: If a deployment is not listed, make sure that it is running a compatible version. The monitoring deployment and production deployment must be on the same major version, cloud provider, and region. - ::::{tip} + ::::{tip} Remember to send logs and metrics for production deployments to a dedicated monitoring deployment, so that your production deployments are not impacted by the overhead of indexing and storing monitoring data. A dedicated monitoring deployment also gives you more control over the retention period for monitoring data. :::: -::::{note} +::::{note} Enabling logs and monitoring may trigger a plan change on your deployment. You can monitor the plan change progress from the deployment’s **Activity** page. :::: -::::{note} +::::{note} Enabling logs and monitoring requires some extra resource on a deployment. For production systems, we recommend sizing deployments with logs and monitoring enabled to at least 4 GB of RAM. :::: -### Access the monitoring application in Kibana [ech-access-kibana-monitoring] +### Access the monitoring application in Kibana [ech-access-kibana-monitoring] With monitoring enabled for your deployment, you can access the [logs](https://www.elastic.co/guide/en/kibana/current/observability.html) and [stack monitoring](../../../deploy-manage/monitor/monitoring-data/visualizing-monitoring-data.md) through Kibana. @@ -165,28 +165,28 @@ Alternatively, you can access logs and metrics directly on the Kibana **Logs** a | `service.version` | The version of the stack resource that generated the log | `8.13.1` | -### Logging features [ech-extra-logging-features] +### Logging features [ech-extra-logging-features] When shipping logs to a monitoring deployment there are more logging features available to you. These features include: -#### For {{es}}: [ech-extra-logging-features-elasticsearch] +#### For {{es}}: [ech-extra-logging-features-elasticsearch] * [Audit logging](../../../deploy-manage/monitor/logging-configuration/enabling-audit-logs.md) - logs security-related events on your deployment -* [Slow query and index logging](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/slow-log.md) - helps find and debug slow queries and indexing +* [Slow query and index logging](elasticsearch://reference/elasticsearch/index-settings/slow-log.md) - helps find and debug slow queries and indexing * Verbose logging - helps debug stack issues by increasing component logs After you’ve enabled log delivery on your deployment, you can [add the Elasticsearch user settings](../../../deploy-manage/deploy/elastic-cloud/edit-stack-settings.md) to enable these features. -#### For Kibana: [ech-extra-logging-features-kibana] +#### For Kibana: [ech-extra-logging-features-kibana] * [Audit logging](../../../deploy-manage/monitor/logging-configuration/enabling-audit-logs.md) - logs security-related events on your deployment After you’ve enabled log delivery on your deployment, you can [add the Kibana user settings](../../../deploy-manage/deploy/elastic-cloud/edit-stack-settings.md) to enable this feature. -### Other components [ech-extra-logging-features-enterprise-search] +### Other components [ech-extra-logging-features-enterprise-search] Enabling log collection also supports collecting and indexing the following types of logs from other components in your deployments: @@ -204,12 +204,12 @@ The ˆ*ˆ indicates that we also index the archived files of each type of log. Check the respective product documentation for more information about the logging capabilities of each product. -## Metrics features [ech-extra-metrics-features] +## Metrics features [ech-extra-metrics-features] With logging and monitoring enabled for a deployment, metrics are collected for Elasticsearch, Kibana, and APM with Fleet Server. -#### Enabling Elasticsearch/Kibana audit logs on your deployment [ech-enable-audit-logs] +#### Enabling Elasticsearch/Kibana audit logs on your deployment [ech-enable-audit-logs] Audit logs are useful for tracking security events on your {{es}} and/or {{kib}} clusters. To enable {{es}} audit logs on your deployment: diff --git a/raw-migrated-files/cloud/cloud-heroku/ech-getting-started.md b/raw-migrated-files/cloud/cloud-heroku/ech-getting-started.md index b17a27b6d..e39dba7a5 100644 --- a/raw-migrated-files/cloud/cloud-heroku/ech-getting-started.md +++ b/raw-migrated-files/cloud/cloud-heroku/ech-getting-started.md @@ -4,7 +4,7 @@ This documentation applies to Heroku users who want to make use of the Elasticse The add-on runs on {{ecloud}} and provides access to [Elasticsearch](https://www.elastic.co/products/elasticsearch), the open source, distributed, RESTful search engine. Many other features of the Elastic Stack are also readily available to Heroku users through the [Elasticsearch Add-On for Heroku console](https://cloud.elastic.co?page=docs&placement=docs-body) after you install the add-on. For example, you can use Kibana to visualize your Elasticsearch data. -[Elasticsearch Machine Learning](/explore-analyze/machine-learning.md), [Elastic Enterprise Search](https://www.elastic.co/guide/en/enterprise-search/current/index.html), [Elastic APM](/solutions/observability/apps/application-performance-monitoring-apm.md) and [Elastic Fleet Server](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/index.md) are not supported by the Elasticsearch Add-On for Heroku. +[Elasticsearch Machine Learning](/explore-analyze/machine-learning.md), [Elastic Enterprise Search](https://www.elastic.co/guide/en/enterprise-search/current/index.html), [Elastic APM](/solutions/observability/apps/application-performance-monitoring-apm.md) and [Elastic Fleet Server](/reference/ingestion-tools/fleet/index.md) are not supported by the Elasticsearch Add-On for Heroku. To learn more about what plans are available for Heroku users and their cost, check the [Elasticsearch add-on](https://elements.heroku.com/addons/foundelasticsearch) in the Elements Marketplace. diff --git a/raw-migrated-files/cloud/cloud-heroku/ech-manage-kibana-settings.md b/raw-migrated-files/cloud/cloud-heroku/ech-manage-kibana-settings.md index 4a944696f..059c64355 100644 --- a/raw-migrated-files/cloud/cloud-heroku/ech-manage-kibana-settings.md +++ b/raw-migrated-files/cloud/cloud-heroku/ech-manage-kibana-settings.md @@ -3,7 +3,7 @@ Elasticsearch Add-On for Heroku supports most of the standard Kibana and X-Pack settings. Through a YAML editor in the console, you can append Kibana properties to the `kibana.yml` file. Your changes to the configuration file are read on startup. ::::{important} -Be aware that some settings that could break your cluster if set incorrectly and that the syntax might change between major versions. Before upgrading, be sure to review the full list of the [latest Kibana settings and syntax](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md). +Be aware that some settings that could break your cluster if set incorrectly and that the syntax might change between major versions. Before upgrading, be sure to review the full list of the [latest Kibana settings and syntax](kibana://reference/configuration-reference/general-settings.md). :::: @@ -43,7 +43,7 @@ If a setting is not supported by Elasticsearch Add-On for Heroku, you will get a ### Version 8.9.0+ [echversion_8_9_0] `xpack.fleet.createArtifactsBulkBatchSize` -: Allow to configure batch size for creating and updating Fleet user artifacts. Examples include creation of Trusted Applications and Endpoint Exceptions in Security. To learn more, check [Fleet settings in Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/fleet-settings.md). +: Allow to configure batch size for creating and updating Fleet user artifacts. Examples include creation of Trusted Applications and Endpoint Exceptions in Security. To learn more, check [Fleet settings in Kibana](kibana://reference/configuration-reference/fleet-settings.md). `xpack.securitySolution.maxUploadResponseActionFileBytes` : Allow to configure the max file upload size for use with the Upload File Repsonse action available with the Defend Integration. To learn more, check [Endpoint Response actions](/solutions/security/endpoint-response-actions.md). @@ -55,43 +55,43 @@ If a setting is not supported by Elasticsearch Add-On for Heroku, you will get a : Set the maximum number of sessions each user is allowed to have active in {{kib}}. By default, no limit is applied. If set, the value of this option should be an integer between 1 and 1000. When the limit is exceeded, the oldest session is automatically invalidated. To learn more, check [Session management](/deploy-manage/security/kibana-session-management.md#session-max-sessions). `server.securityResponseHeaders.crossOriginOpenerPolicy` -: Controls whether the [`Cross-Origin-Opener-Policy`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cross-Origin-Opener-Policy) header is used in all responses to the client from the Kibana server. To learn more, see [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md#server-securityResponseHeaders-crossOriginOpenerPolicy). +: Controls whether the [`Cross-Origin-Opener-Policy`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cross-Origin-Opener-Policy) header is used in all responses to the client from the Kibana server. To learn more, see [Configure Kibana](kibana://reference/configuration-reference/general-settings.md#server-securityresponseheaders-crossoriginopenerpolicy). ### Version 8.6.0+ [echversion_8_6_0] `server.compression.brotli.enabled` -: Enable brotli compression format for browser-server communications. Default: false. To learn more, check [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md). +: Enable brotli compression format for browser-server communications. Default: false. To learn more, check [Configure Kibana](kibana://reference/configuration-reference/general-settings.md). `xpack.fleet.enableExperimental` -: Allow to configure experimental feature for Fleet. To learn more, check [Fleet settings in Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/fleet-settings.md). +: Allow to configure experimental feature for Fleet. To learn more, check [Fleet settings in Kibana](kibana://reference/configuration-reference/fleet-settings.md). ### Version 8.4.0+ [echversion_8_4_0] `migrations.discardUnknownObjects` -: Discard saved objects with unknown types during a migration. Must be set to the target version, e.g.: `8.4.0`. Default: undefined. To learn more, check [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md). +: Discard saved objects with unknown types during a migration. Must be set to the target version, e.g.: `8.4.0`. Default: undefined. To learn more, check [Configure Kibana](kibana://reference/configuration-reference/general-settings.md). `migrations.discardCorruptObjects` -: Discard corrupt saved objects, as well as those that cause transform errors during a migration. Must be set to the target version, e.g.: `8.4.0`. Default: undefined. To learn more, check [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md). +: Discard corrupt saved objects, as well as those that cause transform errors during a migration. Must be set to the target version, e.g.: `8.4.0`. Default: undefined. To learn more, check [Configure Kibana](kibana://reference/configuration-reference/general-settings.md). ### Version 8.3.0+ [echversion_8_3_0] `elasticsearch.compression` -: Enable compression for communications with Elasticsearch. Default: false. To learn more, check [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md). +: Enable compression for communications with Elasticsearch. Default: false. To learn more, check [Configure Kibana](kibana://reference/configuration-reference/general-settings.md). ### Version 8.2.0+ [echversion_8_2_0] `elasticsearch.maxSockets` -: The maximum number of sockets that can be used for communications with Elasticsearch. Default: Infinity. To learn more, check [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md). +: The maximum number of sockets that can be used for communications with Elasticsearch. Default: Infinity. To learn more, check [Configure Kibana](kibana://reference/configuration-reference/general-settings.md). ### Version 8.1.0+ [echversion_8_1_0] `execution_context.enabled` -: Propagate request-specific metadata to Elasticsearch server by way of the `x-opaque-id` header. To learn more, check [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md). +: Propagate request-specific metadata to Elasticsearch server by way of the `x-opaque-id` header. To learn more, check [Configure Kibana](kibana://reference/configuration-reference/general-settings.md). ### Supported versions before 8.x [echsupported_versions_before_8_x] @@ -106,34 +106,34 @@ If a setting is not supported by Elasticsearch Add-On for Heroku, you will get a ### All supported versions [echall_supported_versions_2] `migrations.maxBatchSizeBytes` -: Defines the maximum payload size for indexing batches of saved objects during upgrade migrations. To learn more, check [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md). +: Defines the maximum payload size for indexing batches of saved objects during upgrade migrations. To learn more, check [Configure Kibana](kibana://reference/configuration-reference/general-settings.md). `server.maxPayload` -: The maximum payload size in bytes for incoming server requests. Default: 1048576. To learn more, check [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md#server-maxPayload). +: The maximum payload size in bytes for incoming server requests. Default: 1048576. To learn more, check [Configure Kibana](kibana://reference/configuration-reference/general-settings.md#server-maxpayload). `server.securityResponseHeaders.strictTransportSecurity` -: Controls whether the [`Strict-Transport-Security`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security) header is used in all responses to the client from the Kibana server. To learn more, check [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md#server-securityResponseHeaders-strictTransportSecurity). +: Controls whether the [`Strict-Transport-Security`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security) header is used in all responses to the client from the Kibana server. To learn more, check [Configure Kibana](kibana://reference/configuration-reference/general-settings.md#server-securityresponseheaders-stricttransportsecurity). `server.securityResponseHeaders.xContentTypeOptions` -: Controls whether the [`X-Content-Type-Options`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options) header is used in all responses to the client from the Kibana server. To learn more, check [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md#server-securityResponseHeaders-xContentTypeOptions). +: Controls whether the [`X-Content-Type-Options`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options) header is used in all responses to the client from the Kibana server. To learn more, check [Configure Kibana](kibana://reference/configuration-reference/general-settings.md#server-securityresponseheaders-xcontenttypeoptions). `server.securityResponseHeaders.referrerPolicy` -: Controls whether the [`Referrer-Policy`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Referrer-Policy) header is used in all responses to the client from the Kibana server. To learn more, see [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md#server-securityResponseHeaders-referrerPolicy). +: Controls whether the [`Referrer-Policy`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Referrer-Policy) header is used in all responses to the client from the Kibana server. To learn more, see [Configure Kibana](kibana://reference/configuration-reference/general-settings.md#server-securityresponseheaders-referrerpolicy). `server.securityResponseHeaders.permissionsPolicy` -: Controls whether the `Permissions-Policy` header is used in all responses to the client from the Kibana server. To learn more, see [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md#server-securityResponseHeaders-permissionsPolicy). +: Controls whether the `Permissions-Policy` header is used in all responses to the client from the Kibana server. To learn more, see [Configure Kibana](kibana://reference/configuration-reference/general-settings.md#server-securityresponseheaders-permissionspolicy). `server.securityResponseHeaders.permissionsPolicyReportOnly` -: Controls whether the `Permissions-Policy-Report-Only` header is used in all responses to the client from the Kibana server. To learn more, see [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md#server-securityResponseHeaders-permissionsPolicy). +: Controls whether the `Permissions-Policy-Report-Only` header is used in all responses to the client from the Kibana server. To learn more, see [Configure Kibana](kibana://reference/configuration-reference/general-settings.md#server-securityresponseheaders-permissionspolicy). `server.securityResponseHeaders.disableEmbedding` -: Controls whether the [`Content-Security-Policy`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy) and [`X-Frame-Options`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options) headers are configured to disable embedding Kibana in other webpages using iframes. To learn more, see [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md#server-securityResponseHeaders-disableEmbedding). +: Controls whether the [`Content-Security-Policy`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy) and [`X-Frame-Options`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options) headers are configured to disable embedding Kibana in other webpages using iframes. To learn more, see [Configure Kibana](kibana://reference/configuration-reference/general-settings.md#server-securityresponseheaders-disableembedding). `data.autocomplete.valueSuggestions.timeout` -: Specifies the time in milliseconds to wait for autocomplete suggestions from Elasticsearch. The default is 1000. Allowed values are between 1 and 1200000. To learn more, check [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md). +: Specifies the time in milliseconds to wait for autocomplete suggestions from Elasticsearch. The default is 1000. Allowed values are between 1 and 1200000. To learn more, check [Configure Kibana](kibana://reference/configuration-reference/general-settings.md). `data.autocomplete.valueSuggestions.terminateAfter` -: Specifies the max number of documents loaded by each shard to generate autocomplete suggestions. The default is 100000. Allowed values are between 1 and 10000000. To learn more, check [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md). +: Specifies the max number of documents loaded by each shard to generate autocomplete suggestions. The default is 100000. Allowed values are between 1 and 10000000. To learn more, check [Configure Kibana](kibana://reference/configuration-reference/general-settings.md). `map.tilemap.options.attribution` : Adds the map attribution string. @@ -154,7 +154,7 @@ If a setting is not supported by Elasticsearch Add-On for Heroku, you will get a : Specifies the locale for all strings, dates, and number formats that can be localized. Defaults to `en` (English). `migrations.batchSize` -: Defines the number of documents migrated at a time during saved object upgrade migrations. To learn more, check [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md). +: Defines the number of documents migrated at a time during saved object upgrade migrations. To learn more, check [Configure Kibana](kibana://reference/configuration-reference/general-settings.md). `server.defaultRoute` : Specifies the default route when opening Kibana. You can use this setting to modify the landing page when opening Kibana. @@ -309,7 +309,7 @@ If you want to allow anonymous authentication in Kibana, these settings are supp #### Supported versions before 8.0.0 [echsupported_versions_before_8_0_0] `xpack.security.sessionTimeout` -: Specifies the session duration in milliseconds. Allows a value between 15000 (15 seconds) and 86400000 (1 day). To learn more, check [Security settings in Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/security-settings.md). Deprecated in versions 7.6+ and removed in versions 8.0+. +: Specifies the session duration in milliseconds. Allows a value between 15000 (15 seconds) and 86400000 (1 day). To learn more, check [Security settings in Kibana](kibana://reference/configuration-reference/security-settings.md). Deprecated in versions 7.6+ and removed in versions 8.0+. #### All supported versions [echall_supported_versions_4] @@ -474,7 +474,7 @@ This setting is not available in versions 8.0.0 through 8.2.0. As such, this set : Sets the size of the ephemeral queue. Defaults to `10`. `xpack.actions.customHostSettings` -: An array of objects, one per host, containing the SSL/TLS settings used when executing connectors which make HTTPS and SMTP connections to the host servers. For details about using this setting, check [Alerting and action settings in Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/alerting-settings.md). +: An array of objects, one per host, containing the SSL/TLS settings used when executing connectors which make HTTPS and SMTP connections to the host servers. For details about using this setting, check [Alerting and action settings in Kibana](kibana://reference/configuration-reference/alerting-settings.md). `xpack.actions.ssl.proxyVerificationMode` : Controls the verification of the proxy server certificate that hosted-ems receives when making an outbound SSL/TLS connection to the host server. Valid values are `full`, `certificate`, and `none`. Use `full` to perform hostname verification, `certificate` to skip hostname verification, and `none` to skip verification. Default: `full`. @@ -564,10 +564,10 @@ This setting is not available in versions 8.0.0 through 8.2.0. As such, this set : Set to `true` to enable logging event log documents from alerting to the Kibana log, in addition to being indexed into the event log index. Default: `false`. `xpack.security.session.idleTimeout` -: Set the session duration. The format is a string of `count` and `unit`, where unit is one of `ms`,`s`,`m`,`h`,`d`,`w`,`M`,`Y`. For example, `70ms`, `5s`, `3d`, `1Y`. To learn more, check [Security settings in Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/security-settings.md). +: Set the session duration. The format is a string of `count` and `unit`, where unit is one of `ms`,`s`,`m`,`h`,`d`,`w`,`M`,`Y`. For example, `70ms`, `5s`, `3d`, `1Y`. To learn more, check [Security settings in Kibana](kibana://reference/configuration-reference/security-settings.md). `xpack.security.session.lifespan` -: Sets the maximum duration, also known as "absolute timeout". After this duration, the session will expire even if it is not idle. To learn more, check [Security settings in Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/security-settings.md). +: Sets the maximum duration, also known as "absolute timeout". After this duration, the session will expire even if it is not idle. To learn more, check [Security settings in Kibana](kibana://reference/configuration-reference/security-settings.md). `xpack.maps.showMapVisualizationTypes` : Set to `true` if you want to create new region map visualizations. @@ -588,7 +588,7 @@ This setting is not available in versions 8.0.0 through 8.2.0. As such, this set : When enabled, specifies the email address to receive cluster alert notifications. `xpack.monitoring.kibana.collection.interval` -: Controls [how often data samples are collected](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/monitoring-settings.md#monitoring-collection-settings). +: Controls [how often data samples are collected](elasticsearch://reference/elasticsearch/configuration-reference/monitoring-settings.md#monitoring-collection-settings). `xpack.monitoring.min_interval_seconds` : Specifies the minimum number of seconds that a time bucket in a chart can represent. If you modify the `xpack.monitoring.kibana.collection.interval`, use the same value in this setting. @@ -599,7 +599,7 @@ This setting is not available in versions 8.0.0 through 8.2.0. As such, this set `xpack.ml.enabled` : Set to true (default) to enable machine learning. - If set to `false` in `kibana.yml`, the machine learning icon is hidden in this Kibana instance. If `xpack.ml.enabled` is set to `true` in `elasticsearch.yml`, however, you can still use the machine learning APIs. To disable machine learning entirely, check the [Elasticsearch Machine Learning Settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/machine-learning-settings.md). + If set to `false` in `kibana.yml`, the machine learning icon is hidden in this Kibana instance. If `xpack.ml.enabled` is set to `true` in `elasticsearch.yml`, however, you can still use the machine learning APIs. To disable machine learning entirely, check the [Elasticsearch Machine Learning Settings](elasticsearch://reference/elasticsearch/configuration-reference/machine-learning-settings.md). #### Content security policy configuration [echcontent_security_policy_configuration] @@ -635,7 +635,7 @@ This setting is not available in versions 8.0.0 through 8.2.0. As such, this set : Add sources for the [Content Security Policy `form-action` directive](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/form-action) in reporting mode. $$$csp-strict$$$ `csp.strict` -: Blocks Kibana access to any browser that does not enforce even rudimentary CSP rules. In practice, this disables support for older, less safe browsers like Internet Explorer. **Default: `true`** To learn more, check [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md)]. +: Blocks Kibana access to any browser that does not enforce even rudimentary CSP rules. In practice, this disables support for older, less safe browsers like Internet Explorer. **Default: `true`** To learn more, check [Configure Kibana](kibana://reference/configuration-reference/general-settings.md)]. `csp.warnLegacyBrowsers` : Shows a warning message after loading Kibana to any browser that does not enforce even rudimentary CSP rules, though Kibana is still accessible. This configuration is effectively ignored when [`csp.strict`](../../../deploy-manage/deploy/elastic-cloud/edit-stack-settings.md#csp-strict) is enabled. **Default: `true`** @@ -650,7 +650,7 @@ $$$csp-strict$$$ `csp.strict` #### Permissions policy configuration [echpermissions_policy_configuration] `permissionsPolicy.report_to` -: Add sources for the permissions policy `report-to` directive. To learn more, see [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md#server-securityResponseHeaders-permissionsPolicy) +: Add sources for the permissions policy `report-to` directive. To learn more, see [Configure Kibana](kibana://reference/configuration-reference/general-settings.md#server-securityresponseheaders-permissionspolicy) #### Banner settings [echbanner_settings] @@ -692,7 +692,7 @@ Each method has its own unique limitations which are important to understand. `xpack.reporting.csv.scroll.duration` -: Amount of [time](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#time-units) allowed before {{kib}} cleans the scroll context during a CSV export. Valid option is either `auto` or [time](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#time-units), Defaults to `30s`. +: Amount of [time](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#time-units) allowed before {{kib}} cleans the scroll context during a CSV export. Valid option is either `auto` or [time](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#time-units), Defaults to `30s`. ::::{note} Support for the The option `auto` was included here, when the config value is set to `auto` the scroll context will be preserved for as long as is possible, before the report task is terminated due to the limits of `xpack.reporting.queue.timeout`. @@ -757,7 +757,7 @@ Support for the The option `auto` was included here, when the config value is se Defaults to `true`. `xpack.reporting.csv.scroll.duration` -: Amount of [time](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#time-units) allowed before {{kib}} cleans the scroll context during a CSV export. +: Amount of [time](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#time-units) allowed before {{kib}} cleans the scroll context during a CSV export. Defaults to `30s` (30 seconds). @@ -929,7 +929,7 @@ The following APM settings are supported in Kibana: `xpack.apm.ui.maxTraceItems` : Maximum number of child items displayed when viewing trace details. - Defaults to `1000`. Any positive value is valid. To learn more, check [APM settings in Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/apm-settings.md). + Defaults to `1000`. Any positive value is valid. To learn more, check [APM settings in Kibana](kibana://reference/configuration-reference/apm-settings.md). `xpack.apm.ui.enabled` diff --git a/raw-migrated-files/cloud/cloud-heroku/ech-monitoring-setup.md b/raw-migrated-files/cloud/cloud-heroku/ech-monitoring-setup.md index 0c318af48..e41fd678f 100644 --- a/raw-migrated-files/cloud/cloud-heroku/ech-monitoring-setup.md +++ b/raw-migrated-files/cloud/cloud-heroku/ech-monitoring-setup.md @@ -27,7 +27,7 @@ After you have created a new deployment, you should enable shipping logs and met 5. Select **Save**. -Optionally, turn on [audit logging](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/auding-settings.md) to capture security-related events, such as authentication failures, refused connections, and data-access events through the proxy. To turn on audit logging, [edit your deployment’s elasticsearch.yml file](../../../deploy-manage/deploy/elastic-cloud/edit-stack-settings.md) to add these lines: +Optionally, turn on [audit logging](elasticsearch://reference/elasticsearch/configuration-reference/auding-settings.md) to capture security-related events, such as authentication failures, refused connections, and data-access events through the proxy. To turn on audit logging, [edit your deployment’s elasticsearch.yml file](../../../deploy-manage/deploy/elastic-cloud/edit-stack-settings.md) to add these lines: ```sh xpack.security.audit.enabled: true @@ -106,12 +106,12 @@ You will get this request reported as a new log. Audit logs do not currently rep You should take advantage of the default [Elastic Stack monitoring alerts](/deploy-manage/monitor/monitoring-data/kibana-alerts.md) that are available out-of-the-box. You don’t have to do anything other than enable shipping logs and metrics to have them made available to you (which you did earlier on). -On top of these default alerts that write to indices you can investigate, you might want to add some custom actions, such as a [connector](asciidocalypse://docs/kibana/docs/reference/connectors-kibana.md) for Slack notifications. To set up these notifications, you first configure a Slack connector and then append it to the default alerts and actions. From Kibana: +On top of these default alerts that write to indices you can investigate, you might want to add some custom actions, such as a [connector](kibana://reference/connectors-kibana.md) for Slack notifications. To set up these notifications, you first configure a Slack connector and then append it to the default alerts and actions. From Kibana: 1. Go to **Stack Management** > **Rules and Connectors** > **Connectors** and create your Slack connector: 1. Select **Slack**. - 2. [Create a Slack Webhook URL](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/slack-action-type.md#configuring-slack) and paste it into the **Webhook URL** field. + 2. [Create a Slack Webhook URL](kibana://reference/connectors-kibana/slack-action-type.md#configuring-slack) and paste it into the **Webhook URL** field. 3. Select **Save**. 2. Go to **Stack Monitoring** and select **Enter setup mode**. diff --git a/raw-migrated-files/cloud/cloud/ec-add-user-settings.md b/raw-migrated-files/cloud/cloud/ec-add-user-settings.md index 9c8dcb5eb..9828a4f41 100644 --- a/raw-migrated-files/cloud/cloud/ec-add-user-settings.md +++ b/raw-migrated-files/cloud/cloud/ec-add-user-settings.md @@ -35,7 +35,7 @@ In some cases, you may get a warning saying "User settings are different across The following general settings are supported: $$$http-cors-settings$$$`http.cors.*` -: Enables cross-origin resource sharing (CORS) settings for the [HTTP module](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md). +: Enables cross-origin resource sharing (CORS) settings for the [HTTP module](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md). ::::{note} If your use case depends on the ability to receive CORS requests and you have a cluster that was provisioned prior to January 25th 2019, you must manually set `http.cors.enabled` to `true` and allow a specific set of hosts with `http.cors.allow-origin`. Applying these changes in your Elasticsearch configuration allows cross-origin resource sharing requests. @@ -43,13 +43,13 @@ $$$http-cors-settings$$$`http.cors.*` `http.compression` -: Support for [HTTP compression](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md) when possible (with Accept-Encoding). Defaults to `true`. +: Support for [HTTP compression](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md) when possible (with Accept-Encoding). Defaults to `true`. `transport.compress` -: Configures [transport compression](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md) for node-to-node traffic. +: Configures [transport compression](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md) for node-to-node traffic. `transport.compression_scheme` -: Configures [transport compression](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md) for node-to-node traffic. +: Configures [transport compression](elasticsearch://reference/elasticsearch/configuration-reference/networking-settings.md) for node-to-node traffic. `repositories.url.allowed_urls` : Enables explicit allowing of [read-only URL repositories](../../../deploy-manage/tools/snapshot-and-restore/read-only-url-repository.md). @@ -61,7 +61,7 @@ $$$http-cors-settings$$$`http.cors.*` : To learn more on how to configure reindex SSL user settings, check [configuring reindex SSL parameters](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex). `script.painless.regex.enabled` -: Enables [regular expressions](asciidocalypse://docs/elasticsearch/docs/reference/scripting-languages/painless/brief-painless-walkthrough.md#modules-scripting-painless-regex) for the Painless scripting language. +: Enables [regular expressions](elasticsearch://reference/scripting-languages/painless/brief-painless-walkthrough.md#modules-scripting-painless-regex) for the Painless scripting language. `action.auto_create_index` : [Automatically create index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-create) if it doesn’t already exist. @@ -94,19 +94,19 @@ $$$http-cors-settings$$$`http.cors.*` The following circuit breaker settings are supported: `indices.breaker.total.limit` -: Configures [the parent circuit breaker settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/circuit-breaker-settings.md#parent-circuit-breaker). +: Configures [the parent circuit breaker settings](elasticsearch://reference/elasticsearch/configuration-reference/circuit-breaker-settings.md#parent-circuit-breaker). `indices.breaker.fielddata.limit` -: Configures [the limit for the fielddata breaker](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/circuit-breaker-settings.md#fielddata-circuit-breaker). +: Configures [the limit for the fielddata breaker](elasticsearch://reference/elasticsearch/configuration-reference/circuit-breaker-settings.md#fielddata-circuit-breaker). `indices.breaker.fielddata.overhead` -: Configures [a constant that all field data estimations are multiplied with to determine a final estimation](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/circuit-breaker-settings.md#fielddata-circuit-breaker). +: Configures [a constant that all field data estimations are multiplied with to determine a final estimation](elasticsearch://reference/elasticsearch/configuration-reference/circuit-breaker-settings.md#fielddata-circuit-breaker). `indices.breaker.request.limit` -: Configures [the limit for the request breaker](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/circuit-breaker-settings.md#request-circuit-breaker). +: Configures [the limit for the request breaker](elasticsearch://reference/elasticsearch/configuration-reference/circuit-breaker-settings.md#request-circuit-breaker). `indices.breaker.request.overhead` -: Configures [a constant that all request estimations are multiplied by to determine a final estimation](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/circuit-breaker-settings.md#request-circuit-breaker). +: Configures [a constant that all request estimations are multiplied by to determine a final estimation](elasticsearch://reference/elasticsearch/configuration-reference/circuit-breaker-settings.md#request-circuit-breaker). ### Indexing pressure settings [ec_indexing_pressure_settings] @@ -114,7 +114,7 @@ The following circuit breaker settings are supported: The following indexing pressure settings are supported: `indexing_pressure.memory.limit` -: Configures [the indexing pressure settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/pressure.md). +: Configures [the indexing pressure settings](elasticsearch://reference/elasticsearch/index-settings/pressure.md). ### X-Pack [ec_x_pack] @@ -128,7 +128,7 @@ The following indexing pressure settings are supported: #### All supported versions [ec_all_supported_versions] `xpack.ml.inference_model.time_to_live` -: Sets the duration of time that the trained models are cached. Check [{{ml-cap}} settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/machine-learning-settings.md). +: Sets the duration of time that the trained models are cached. Check [{{ml-cap}} settings](elasticsearch://reference/elasticsearch/configuration-reference/machine-learning-settings.md). `xpack.security.loginAssistanceMessage` : Adds a message to the login screen. Useful for displaying corporate messages. @@ -146,10 +146,10 @@ The following indexing pressure settings are supported: : Defines when the watch should start, based on date and time [Learn more](/explore-analyze/alerts-cases/watcher/trigger-schedule.md). `xpack.notification.email.html.sanitization.*` -: Enables [email notification settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/watcher-settings.md) to sanitize HTML elements in emails that are sent. +: Enables [email notification settings](elasticsearch://reference/elasticsearch/configuration-reference/watcher-settings.md) to sanitize HTML elements in emails that are sent. `xpack.monitoring.collection.interval` -: Controls [how often data samples are collected](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/monitoring-settings.md#monitoring-collection-settings). +: Controls [how often data samples are collected](elasticsearch://reference/elasticsearch/configuration-reference/monitoring-settings.md#monitoring-collection-settings). `xpack.monitoring.collection.min_interval_seconds` : Specifies the minimum number of seconds that a time bucket in a chart can represent. If you modify the `xpack.monitoring.collection.interval`, use the same value in this setting. @@ -158,10 +158,10 @@ The following indexing pressure settings are supported: $$$xpack-monitoring-history-duration$$$`xpack.monitoring.history.duration` -: Sets the [retention duration](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/monitoring-settings.md#monitoring-collection-settings) beyond which the indices created by a monitoring exporter will be automatically deleted. +: Sets the [retention duration](elasticsearch://reference/elasticsearch/configuration-reference/monitoring-settings.md#monitoring-collection-settings) beyond which the indices created by a monitoring exporter will be automatically deleted. `xpack.watcher.history.cleaner_service.enabled` -: Controls [whether old watcher indices are automatically deleted](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/watcher-settings.md#general-notification-settings). +: Controls [whether old watcher indices are automatically deleted](elasticsearch://reference/elasticsearch/configuration-reference/watcher-settings.md#general-notification-settings). `xpack.http.ssl.cipher_suites` : Controls the list of supported cipher suites for all outgoing TLS connections. @@ -197,16 +197,16 @@ The following search settings are supported: The following disk-based allocation settings are supported: `cluster.routing.allocation.disk.threshold_enabled` -: Enable or disable [disk allocation](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#disk-based-shard-allocation) decider and defaults to `true`. +: Enable or disable [disk allocation](elasticsearch://reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#disk-based-shard-allocation) decider and defaults to `true`. `cluster.routing.allocation.disk.watermark.low` -: Configures [disk-based shard allocation’s low watermark](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#disk-based-shard-allocation). +: Configures [disk-based shard allocation’s low watermark](elasticsearch://reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#disk-based-shard-allocation). `cluster.routing.allocation.disk.watermark.high` -: Configures [disk-based shard allocation’s high watermark](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#disk-based-shard-allocation). +: Configures [disk-based shard allocation’s high watermark](elasticsearch://reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#disk-based-shard-allocation). `cluster.routing.allocation.disk.watermark.flood_stage` -: Configures [disk-based shard allocation’s flood_stage](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#disk-based-shard-allocation). +: Configures [disk-based shard allocation’s flood_stage](elasticsearch://reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#disk-based-shard-allocation). ::::{tip} Remember to update user settings for alerts when performing a major version upgrade. diff --git a/raw-migrated-files/cloud/cloud/ec-cloud-ingest-data.md b/raw-migrated-files/cloud/cloud/ec-cloud-ingest-data.md index 36f8fffcd..84e75c948 100644 --- a/raw-migrated-files/cloud/cloud/ec-cloud-ingest-data.md +++ b/raw-migrated-files/cloud/cloud/ec-cloud-ingest-data.md @@ -5,7 +5,7 @@ You have a number of options for getting data into Elasticsearch, referred to as $$$ec-ingest-methods$$$ General content -: Index content like HTML pages, catalogs and other files. Send data directly to Elasticseach from your application using an Elastic language client. Otherwise use Elastic content [connectors](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/search-connectors/index.md) or the Elastic [web crawler](https://github.com/elastic/crawler). +: Index content like HTML pages, catalogs and other files. Send data directly to Elasticseach from your application using an Elastic language client. Otherwise use Elastic content [connectors](elasticsearch://reference/ingestion-tools/search-connectors/index.md) or the Elastic [web crawler](https://github.com/elastic/crawler). Timestamped data : The preferred way to index timestamped data is to use Elastic Agent. Elastic Agent is a single, unified way to add monitoring for logs, metrics, and other types of data to a host. It can also protect hosts from security threats, query data from operating systems, and forward data from remote services or hardware. Each Elastic Agent based integration includes default ingestion rules, dashboards, and visualizations to start analyzing your data right away. Fleet Management enables you to centrally manage all of your deployed Elastic Agents from Kibana. @@ -143,16 +143,16 @@ One reason for preprocessing your data is to control the structure of the data t ### Data integrity [ec-data-integrity] -Logstash boosts data resiliency for important data that you don’t want to lose. Logstash offers an on-disk [persistent queue (PQ)](asciidocalypse://docs/logstash/docs/reference/persistent-queues.md) that absorbs bursts of events without an external buffering mechanism. It attempts to deliver messages stored in the PQ until delivery succeeds at least once. +Logstash boosts data resiliency for important data that you don’t want to lose. Logstash offers an on-disk [persistent queue (PQ)](logstash://reference/persistent-queues.md) that absorbs bursts of events without an external buffering mechanism. It attempts to deliver messages stored in the PQ until delivery succeeds at least once. -The Logstash [dead letter queue (DLQ)](asciidocalypse://docs/logstash/docs/reference/dead-letter-queues.md) provides on-disk storage for events that Logstash can’t process, giving you a chance to evaluate them. You can use the dead_letter_queue input plugin to easily reprocess DLQ events. +The Logstash [dead letter queue (DLQ)](logstash://reference/dead-letter-queues.md) provides on-disk storage for events that Logstash can’t process, giving you a chance to evaluate them. You can use the dead_letter_queue input plugin to easily reprocess DLQ events. ### Data flow [ec-data-flow] If you need to collect data from multiple Beats or Elastic Agents, consider using Logstash as a proxy. Logstash can receive data from multiple endpoints, even on different networks, and send the data on to Elasticsearch through a single firewall rule. You get more security for less work than if you set up individual rules for each endpoint. -Logstash can send to multiple [outputs](asciidocalypse://docs/logstash/docs/reference/output-plugins.md) from a single pipeline to help you get the most value from your data. +Logstash can send to multiple [outputs](logstash://reference/output-plugins.md) from a single pipeline to help you get the most value from your data. ## Where to go from here [ec-data-ingest-where-to-go] @@ -166,7 +166,7 @@ We have guides and many hands-on tutorials to help get you started with ingestin : Use Elastic Observability to gain deeper insight into the behavior of your applications and systems. Follow our guides to ingest various data types, such as [logs and metrics](/solutions/observability/infra-and-hosts/get-started-with-system-metrics.md), [traces and APM](/solutions/observability/apps/get-started-with-apm.md), and [data from Splunk](/solutions/observability/get-started/add-data-from-splunk.md). There are also several [tutorials](https://www.elastic.co/guide/en/observability/current/observability-tutorials.html) to choose from. [Add data to Elastic Security](/solutions/security/get-started/ingest-data-to-elastic-security.md) -: Use Elastic Security to quickly detect, investigate, and respond to threats and vulnerabilities across your environment. You can use {{agent}} to ingest data into the [{{elastic-defend}} integration](/solutions/security/configure-elastic-defend/install-elastic-defend.md), or with many other [{{integrations}}](https://docs.elastic.co/en/integrations) that work together with {{elastic-sec}}. You can also [ingest data from Splunk](/solutions/observability/get-started/add-data-from-splunk.md) or from various third party collectors that ship [ECS compliant security data](asciidocalypse://docs/docs-content/docs/reference/security/fields-and-object-schemas/siem-field-reference.md). +: Use Elastic Security to quickly detect, investigate, and respond to threats and vulnerabilities across your environment. You can use {{agent}} to ingest data into the [{{elastic-defend}} integration](/solutions/security/configure-elastic-defend/install-elastic-defend.md), or with many other [{{integrations}}](https://docs.elastic.co/en/integrations) that work together with {{elastic-sec}}. You can also [ingest data from Splunk](/solutions/observability/get-started/add-data-from-splunk.md) or from various third party collectors that ship [ECS compliant security data](/reference/security/fields-and-object-schemas/siem-field-reference.md). ### Ingest data with Elastic Agent, Beats, and Logstash [ec-ingest-timestamped] @@ -179,10 +179,10 @@ For users who want to build their own solution, we can help you get started inge [Beats and Elastic Agent comparison](../../../manage-data/ingest/tools.md) : {{beats}} and {{agent}} can both send data to {{es}} either directly or via {{ls}}. You can use this guide to determine which of these primary ingest tools best matches your use case. -[Introduction to Fleet management](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/index.md) +[Introduction to Fleet management](/reference/ingestion-tools/fleet/index.md) : {{fleet}} provides a web-based UI in Kibana for centrally managing Elastic Agents and their policies. -[{{ls}} introduction](asciidocalypse://docs/logstash/docs/reference/index.md) +[{{ls}} introduction](logstash://reference/index.md) : Use {{ls}} to dynamically unify data from disparate sources and normalize the data into destinations of your choice. @@ -191,7 +191,7 @@ For users who want to build their own solution, we can help you get started inge [Add data with the web crawler](https://github.com/elastic/crawler) : Use the web crawler to programmatically discover, extract, and index searchable content from websites and knowledge bases. -[Add data with connectors](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/search-connectors/index.md) +[Add data with connectors](elasticsearch://reference/ingestion-tools/search-connectors/index.md) : Sync data from an original data source to an {{es}} index. Connectors enable you to create searchable, read-only replicas of your data sources. @@ -209,10 +209,10 @@ For users who want to build their own solution, we can help you get started inge [Ingest pipelines](/manage-data/ingest/transform-enrich/ingest-pipelines.md) : {{es}} ingest pipelines let you perform common transformations on your data before indexing. -[{{agent}} processors](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/agent-processors.md) +[{{agent}} processors](/reference/ingestion-tools/fleet/agent-processors.md) : Use the {{agent}} lightweight processors to parse, filter, transform, and enrich data at the source. -[Creating a {{ls}} pipeline](asciidocalypse://docs/logstash/docs/reference/creating-logstash-pipeline.md) +[Creating a {{ls}} pipeline](logstash://reference/creating-logstash-pipeline.md) : Create a {{ls}} pipeline by stringing together plugins—​inputs, outputs, filters, and sometimes codecs—​in order to process your data during ingestion. diff --git a/raw-migrated-files/cloud/cloud/ec-custom-bundles.md b/raw-migrated-files/cloud/cloud/ec-custom-bundles.md index 61e50495b..b78577cb2 100644 --- a/raw-migrated-files/cloud/cloud/ec-custom-bundles.md +++ b/raw-migrated-files/cloud/cloud/ec-custom-bundles.md @@ -16,7 +16,7 @@ The selected plugins/bundles are downloaded and provided when a node starts. Cha With great power comes great responsibility: your plugins can extend your deployment with new functionality, but also break it. Be careful. We obviously cannot guarantee that your custom code works. -::::{important} +::::{important} You cannot edit or delete a custom extension after it has been used in a deployment. To remove it from your deployment, you can disable the extension and update your deployment configuration. :::: @@ -39,7 +39,7 @@ Plugins {{es}} assumes that the uploaded ZIP file contains binaries. If it finds any source code, it fails with an error message, causing provisioning to fail. Make sure you upload binaries, and not source code. - ::::{note} + ::::{note} Plugins larger than 5GB should have the plugin descriptor file at the top of the archive. This order can be achieved by specifying at time of creating the ZIP file: ```sh @@ -76,7 +76,7 @@ Bundles The dictionary `synonyms.txt` can be used as `synonyms.txt` or using the full path `/app/config/synonyms.txt` in the `synonyms_path` of the `synonym-filter`. - To learn more about analyzing with synonyms, check [Synonym token filter](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/text-analysis/analysis-synonym-tokenfilter.md) and [Formatting Synonyms](https://www.elastic.co/guide/en/elasticsearch/guide/2.x/synonym-formats.html). + To learn more about analyzing with synonyms, check [Synonym token filter](elasticsearch://reference/data-analysis/text-analysis/analysis-synonym-tokenfilter.md) and [Formatting Synonyms](https://www.elastic.co/guide/en/elasticsearch/guide/2.x/synonym-formats.html). **GeoIP database bundle** @@ -110,7 +110,7 @@ You must upload your files before you can apply them to your cluster configurati After creating your extension, you can [enable them for existing {{es}} deployments](../../../deploy-manage/deploy/elastic-cloud/upload-custom-plugins-bundles.md#ec-update-bundles) or enable them when creating new deployments. -::::{note} +::::{note} Creating extensions larger than 200MB should be done through the extensions API. Refer to [Managing plugins and extensions through the API](../../../deploy-manage/deploy/elastic-cloud/manage-plugins-extensions-through-api.md) for more details. @@ -169,7 +169,7 @@ To update an extension with a new file version, ## How to use the extensions API [ec-extension-api-usage-guide] -::::{note} +::::{note} For a full set of examples, check [Managing plugins and extensions through the API](../../../deploy-manage/deploy/elastic-cloud/manage-plugins-extensions-through-api.md). :::: diff --git a/raw-migrated-files/cloud/cloud/ec-enable-logging-and-monitoring.md b/raw-migrated-files/cloud/cloud/ec-enable-logging-and-monitoring.md index 27c12b67b..fb308686a 100644 --- a/raw-migrated-files/cloud/cloud/ec-enable-logging-and-monitoring.md +++ b/raw-migrated-files/cloud/cloud/ec-enable-logging-and-monitoring.md @@ -13,12 +13,12 @@ Monitoring consists of two components: The steps in this section cover only the enablement of the monitoring and logging features in {{ech}}. For more information on how to use the monitoring features, refer to [Monitor a cluster](../../../deploy-manage/monitor.md). -### Before you begin [ec-logging-and-monitoring-limitations] +### Before you begin [ec-logging-and-monitoring-limitations] Some limitations apply when you use monitoring on {{ech}}. To learn more, check the monitoring [restrictions and limitations](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md#ec-restrictions-monitoring). -### Monitoring for production use [ec-logging-and-monitoring-production] +### Monitoring for production use [ec-logging-and-monitoring-production] For production use, you should send your deployment logs and metrics to a dedicated monitoring deployment. Monitoring indexes logs and metrics into {{es}} and these indexes consume storage, memory, and CPU cycles like any other index. By using a separate monitoring deployment, you avoid affecting your other production deployments and can view the logs and metrics even when a production deployment is unavailable. @@ -35,15 +35,15 @@ How many monitoring deployments you use depends on your requirements: Logs and metrics that get sent to a dedicated monitoring {{es}} deployment [may not be cleaned up automatically](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md#ec-logging-and-monitoring-retention) and might require some additional steps to remove excess data periodically. -### Retention of monitoring daily indices [ec-logging-and-monitoring-retention] +### Retention of monitoring daily indices [ec-logging-and-monitoring-retention] -#### Stack versions 8.0 and above [ec-logging-and-monitoring-retention-8] +#### Stack versions 8.0 and above [ec-logging-and-monitoring-retention-8] When you enable monitoring in {{ech}}, your monitoring indices are retained for a certain period by default. After the retention period has passed, the monitoring indices are deleted automatically. The retention period is configured in the `.monitoring-8-ilm-policy` index lifecycle policy. To view or edit the policy open {{kib}} **Stack management > Data > Index Lifecycle Policies**. -### Sending monitoring data to itself (self monitoring) [ec-logging-and-monitoring-retention-self-monitoring] +### Sending monitoring data to itself (self monitoring) [ec-logging-and-monitoring-retention-self-monitoring] $$$ec-logging-and-monitoring-retention-7$$$ When you enable self-monitoring in {{ech}}, your monitoring indices are retained for a certain period by default. After the retention period has passed, the monitoring indices are deleted automatically. Monitoring data is retained for three days by default or as specified by the [`xpack.monitoring.history.duration` user setting](../../../deploy-manage/deploy/elastic-cloud/edit-stack-settings.md#xpack-monitoring-history-duration). @@ -65,7 +65,7 @@ PUT /_cluster/settings ``` -### Sending monitoring data to a dedicated monitoring deployment [ec-logging-and-monitoring-retention-dedicated-monitoring] +### Sending monitoring data to a dedicated monitoring deployment [ec-logging-and-monitoring-retention-dedicated-monitoring] When [monitoring for production use](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md#ec-logging-and-monitoring-production), where you configure your deployments **to send monitoring data to a dedicated monitoring deployment** for indexing, this retention period does not apply. Monitoring indices on a dedicated monitoring deployment are retained until you remove them. There are three options open to you: @@ -98,17 +98,17 @@ When [monitoring for production use](../../../deploy-manage/monitor/stack-monito * To retain monitoring indices on a dedicated monitoring deployment as is without deleting them automatically, no additional steps are required other than making sure that you do not enable the monitoring deployment to send monitoring data to itself. You should also monitor the deployment for disk space usage and upgrade your deployment periodically, if necessary. -### Retention of logging indices [ec-logging-and-monitoring-log-retention] +### Retention of logging indices [ec-logging-and-monitoring-log-retention] An ILM policy is pre-configured to manage log retention. The policy can be adjusted according to your requirements. -### Index management [ec-logging-and-monitoring-index-management-ilm] +### Index management [ec-logging-and-monitoring-index-management-ilm] When sending monitoring data to a deployment, you can configure [Index Lifecycle Management (ILM)](../../../manage-data/lifecycle/index-lifecycle-management.md) to manage retention of your monitoring and logging indices. When sending logs to a deployment, an ILM policy is pre-configured to manage log retention and the policy can be customized to your needs. -### Enable logging and monitoring [ec-enable-logging-and-monitoring-steps] +### Enable logging and monitoring [ec-enable-logging-and-monitoring-steps] {{ech}} manages the installation and configuration of the monitoring agent for you. When you enable monitoring on a deployment, you are configuring where the monitoring agent for your current deployment should send its logs and metrics. @@ -125,23 +125,23 @@ To enable monitoring on your deployment: If a deployment is not listed, make sure that it is running a compatible version. The monitoring deployment and production deployment must be on the same major version, cloud provider, and region. - ::::{tip} + ::::{tip} Remember to send logs and metrics for production deployments to a dedicated monitoring deployment, so that your production deployments are not impacted by the overhead of indexing and storing monitoring data. A dedicated monitoring deployment also gives you more control over the retention period for monitoring data. :::: -::::{note} +::::{note} Enabling logs and monitoring may trigger a plan change on your deployment. You can monitor the plan change progress from the deployment’s **Activity** page. :::: -::::{note} +::::{note} Enabling logs and monitoring requires some extra resource on a deployment. For production systems, we recommend sizing deployments with logs and monitoring enabled to at least 4 GB of RAM. :::: -### Access the monitoring application in Kibana [ec-access-kibana-monitoring] +### Access the monitoring application in Kibana [ec-access-kibana-monitoring] With monitoring enabled for your deployment, you can access the [logs](https://www.elastic.co/guide/en/kibana/current/observability.html) and [stack monitoring](../../../deploy-manage/monitor/monitoring-data/visualizing-monitoring-data.md) through Kibana. @@ -165,28 +165,28 @@ Alternatively, you can access logs and metrics directly on the Kibana **Logs** a | `service.version` | The version of the stack resource that generated the log | `8.13.1` | -### Logging features [ec-extra-logging-features] +### Logging features [ec-extra-logging-features] When shipping logs to a monitoring deployment there are more logging features available to you. These features include: -#### For {{es}}: [ec-extra-logging-features-elasticsearch] +#### For {{es}}: [ec-extra-logging-features-elasticsearch] * [Audit logging](../../../deploy-manage/monitor/logging-configuration/enabling-audit-logs.md) - logs security-related events on your deployment -* [Slow query and index logging](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/slow-log.md) - helps find and debug slow queries and indexing +* [Slow query and index logging](elasticsearch://reference/elasticsearch/index-settings/slow-log.md) - helps find and debug slow queries and indexing * Verbose logging - helps debug stack issues by increasing component logs After you’ve enabled log delivery on your deployment, you can [add the Elasticsearch user settings](../../../deploy-manage/deploy/elastic-cloud/edit-stack-settings.md) to enable these features. -#### For Kibana: [ec-extra-logging-features-kibana] +#### For Kibana: [ec-extra-logging-features-kibana] * [Audit logging](../../../deploy-manage/monitor/logging-configuration/enabling-audit-logs.md) - logs security-related events on your deployment After you’ve enabled log delivery on your deployment, you can [add the Kibana user settings](../../../deploy-manage/deploy/elastic-cloud/edit-stack-settings.md) to enable this feature. -### Other components [ec-extra-logging-features-enterprise-search] +### Other components [ec-extra-logging-features-enterprise-search] Enabling log collection also supports collecting and indexing the following types of logs from other components in your deployments: @@ -204,12 +204,12 @@ The ˆ*ˆ indicates that we also index the archived files of each type of log. Check the respective product documentation for more information about the logging capabilities of each product. -## Metrics features [ec-extra-metrics-features] +## Metrics features [ec-extra-metrics-features] With logging and monitoring enabled for a deployment, metrics are collected for Elasticsearch, Kibana, and APM with Fleet Server. -#### Enabling Elasticsearch/Kibana audit logs on your deployment [ec-enable-audit-logs] +#### Enabling Elasticsearch/Kibana audit logs on your deployment [ec-enable-audit-logs] % Added by eedugon to audit logging in deploy and manage -> monitoring -> logging section Audit logs are useful for tracking security events on your {{es}} and/or {{kib}} clusters. To enable {{es}} audit logs on your deployment: diff --git a/raw-migrated-files/cloud/cloud/ec-getting-started-node-js.md b/raw-migrated-files/cloud/cloud/ec-getting-started-node-js.md index 4e9b0fb23..528cc9e8a 100644 --- a/raw-migrated-files/cloud/cloud/ec-getting-started-node-js.md +++ b/raw-migrated-files/cloud/cloud/ec-getting-started-node-js.md @@ -150,7 +150,7 @@ async function run() { run().catch(console.log) ``` -When using the [client.index](asciidocalypse://docs/elasticsearch-js/docs/reference/api-reference.md#_index) API, the request automatically creates the `game-of-thrones` index if it doesn’t already exist, as well as document IDs for each indexed document if they are not explicitly specified. +When using the [client.index](elasticsearch-js://reference/api-reference.md#_index) API, the request automatically creates the `game-of-thrones` index if it doesn’t already exist, as well as document IDs for each indexed document if they are not explicitly specified. ## Search and modify data [ec_search_and_modify_data] @@ -197,7 +197,7 @@ async function update() { update().catch(console.log) ``` -This [more comprehensive list of API examples](asciidocalypse://docs/elasticsearch-js/docs/reference/examples.md) includes bulk operations, checking the existence of documents, updating by query, deleting, scrolling, and SQL queries. To learn more, check the complete [API reference](asciidocalypse://docs/elasticsearch-js/docs/reference/api-reference.md). +This [more comprehensive list of API examples](elasticsearch-js://reference/examples.md) includes bulk operations, checking the existence of documents, updating by query, deleting, scrolling, and SQL queries. To learn more, check the complete [API reference](elasticsearch-js://reference/api-reference.md). ## Switch to API key authentication [ec_switch_to_api_key_authentication] @@ -284,11 +284,11 @@ Security Connections -: If your application connecting to {{ech}} runs under the Java security manager, you should at least disable the caching of positive hostname resolutions. To learn more, check the [Java API Client documentation](asciidocalypse://docs/elasticsearch-java/docs/reference/_others.md). +: If your application connecting to {{ech}} runs under the Java security manager, you should at least disable the caching of positive hostname resolutions. To learn more, check the [Java API Client documentation](elasticsearch-java://reference/_others.md). Schema : When the example code was run an index mapping was created automatically. The field types were selected by {{es}} based on the content seen when the first record was ingested, and updated as new fields appeared in the data. It would be more efficient to specify the fields and field types in advance to optimize performance. Refer to the Elastic Common Schema documentation and Field Type documentation when you are designing the schema for your production use cases. Ingest -: For more advanced scenarios, this [bulk ingestion](asciidocalypse://docs/elasticsearch-js/docs/reference/bulk_examples.md) reference gives an example of the `bulk` API that makes it possible to perform multiple operations in a single call. This bulk example also explicitly specifies document IDs. If you have a lot of documents to index, using bulk to batch document operations is significantly faster than submitting requests individually. +: For more advanced scenarios, this [bulk ingestion](elasticsearch-js://reference/bulk_examples.md) reference gives an example of the `bulk` API that makes it possible to perform multiple operations in a single call. This bulk example also explicitly specifies document IDs. If you have a lot of documents to index, using bulk to batch document operations is significantly faster than submitting requests individually. diff --git a/raw-migrated-files/cloud/cloud/ec-getting-started-python.md b/raw-migrated-files/cloud/cloud/ec-getting-started-python.md index 5068b510d..4f6f6967b 100644 --- a/raw-migrated-files/cloud/cloud/ec-getting-started-python.md +++ b/raw-migrated-files/cloud/cloud/ec-getting-started-python.md @@ -275,7 +275,7 @@ es.get(index='lord-of-the-rings', id='2EkAzngB_pyHD3p65UMt') 'birthplace': 'The Shire'}} ``` -For frequently used API calls with the Python client, check [Examples](asciidocalypse://docs/elasticsearch-py/docs/reference/examples.md). +For frequently used API calls with the Python client, check [Examples](elasticsearch-py://reference/examples.md). ## Switch to API key authentication [ec_switch_to_api_key_authentication_2] @@ -335,7 +335,7 @@ es = Elasticsearch( Check [Create API key API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-api-key) to learn more about API Keys and [Security privileges](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md) to understand which privileges are needed. If you are not sure what the right combination of privileges for your custom application is, you can enable [audit logging](../../../deploy-manage/monitor/logging-configuration/enabling-audit-logs.md) on {{es}} to find out what privileges are being used. To learn more about how logging works on {{ech}}, check [Monitoring Elastic Cloud deployment logs and metrics](https://www.elastic.co/blog/monitoring-elastic-cloud-deployment-logs-and-metrics). -For more information on refreshing an index, searching, updating, and deleting, check the [elasticsearch-py examples](asciidocalypse://docs/elasticsearch-py/docs/reference/examples.md). +For more information on refreshing an index, searching, updating, and deleting, check the [elasticsearch-py examples](elasticsearch-py://reference/examples.md). ### Best practices [ec_best_practices_2] @@ -350,5 +350,5 @@ Schema : When the example code is run, an index mapping is created automatically. The field types are selected by {{es}} based on the content seen when the first record was ingested, and updated as new fields appeared in the data. It would be more efficient to specify the fields and field types in advance to optimize performance. Refer to the Elastic Common Schema documentation and Field Type documentation when you design the schema for your production use cases. Ingest -: For more advanced scenarios, [Bulk helpers](asciidocalypse://docs/elasticsearch-py/docs/reference/client-helpers.md#bulk-helpers) gives examples for the `bulk` API that makes it possible to perform multiple operations in a single call. If you have a lot of documents to index, using bulk to batch document operations is significantly faster than submitting requests individually. +: For more advanced scenarios, [Bulk helpers](elasticsearch-py://reference/client-helpers.md#bulk-helpers) gives examples for the `bulk` API that makes it possible to perform multiple operations in a single call. If you have a lot of documents to index, using bulk to batch document operations is significantly faster than submitting requests individually. diff --git a/raw-migrated-files/cloud/cloud/ec-getting-started-search-use-cases-beats-logstash.md b/raw-migrated-files/cloud/cloud/ec-getting-started-search-use-cases-beats-logstash.md index 64e663a34..26add6117 100644 --- a/raw-migrated-files/cloud/cloud/ec-getting-started-search-use-cases-beats-logstash.md +++ b/raw-migrated-files/cloud/cloud/ec-getting-started-search-use-cases-beats-logstash.md @@ -238,7 +238,7 @@ Now the Filebeat and Metricbeat are set up, let’s configure a {{ls}} pipeline 1. {{ls}} listens for Beats input on the default port of 5044. Only one line is needed to do this. {{ls}} can handle input from many Beats of the same and also of varying types (Metricbeat, Filebeat, and others). 2. This sends output to the standard output, which displays through your command line interface. This plugin enables you to verify the data before you send it to {{es}}, in a later step. -3. Save the new *beats.conf* file in your Logstash folder. To learn more about the file format and options, check [{{ls}} Configuration Examples](asciidocalypse://docs/logstash/docs/reference/config-examples.md). +3. Save the new *beats.conf* file in your Logstash folder. To learn more about the file format and options, check [{{ls}} Configuration Examples](logstash://reference/config-examples.md). ## Output {{ls}} data to stdout [ec-beats-logstash-stdout] @@ -472,7 +472,7 @@ In this section, you configure {{ls}} to send the Metricbeat and Filebeat data t ::::{note} In this guide, you manually launch each of the Elastic stack applications through the command line interface. In production, you may prefer to configure {{ls}}, Metricbeat, and Filebeat to run as System Services. Check the following pages for the steps to configure each application to run as a service: -* [Running {{ls}} as a service on Debian or RPM](asciidocalypse://docs/logstash/docs/reference/running-logstash.md) +* [Running {{ls}} as a service on Debian or RPM](logstash://reference/running-logstash.md) * [Metricbeat and systemd](asciidocalypse://docs/beats/docs/reference/metricbeat/running-with-systemd.md) * [Start filebeat](asciidocalypse://docs/beats/docs/reference/filebeat/filebeat-starting.md) diff --git a/raw-migrated-files/cloud/cloud/ec-getting-started-search-use-cases-db-logstash.md b/raw-migrated-files/cloud/cloud/ec-getting-started-search-use-cases-db-logstash.md index b50fba317..fe31e39be 100644 --- a/raw-migrated-files/cloud/cloud/ec-getting-started-search-use-cases-db-logstash.md +++ b/raw-migrated-files/cloud/cloud/ec-getting-started-search-use-cases-db-logstash.md @@ -1,6 +1,6 @@ # Ingest data from a relational database into {{ech}} [ec-getting-started-search-use-cases-db-logstash] -This guide explains how to ingest data from a relational database into {{ech}} through [Logstash](asciidocalypse://docs/logstash/docs/reference/index.md), using the Logstash [JDBC input plugin](asciidocalypse://docs/logstash/docs/reference/plugins-inputs-jdbc.md). It demonstrates how Logstash can be used to efficiently copy records and to receive updates from a relational database, and then send them into {{es}} in an {{ech}} deployment. +This guide explains how to ingest data from a relational database into {{ech}} through [Logstash](logstash://reference/index.md), using the Logstash [JDBC input plugin](logstash://reference/plugins-inputs-jdbc.md). It demonstrates how Logstash can be used to efficiently copy records and to receive updates from a relational database, and then send them into {{es}} in an {{ech}} deployment. The code and methods presented here have been tested with MySQL. They should work with other relational databases. @@ -192,13 +192,13 @@ Let’s set up a sample Logstash input pipeline to ingest data from your new JDB : The Logstash JDBC plugin does not come packaged with JDBC driver libraries. The JDBC driver library must be passed explicitly into the plugin using the `jdbc_driver_library` configuration option. tracking_column - : This parameter specifies the field `unix_ts_in_secs` that tracks the last document read by Logstash from MySQL, stored on disk in [logstash_jdbc_last_run](asciidocalypse://docs/logstash/docs/reference/plugins-inputs-jdbc.md#plugins-inputs-jdbc-last_run_metadata_path). The parameter determines the starting value for documents that Logstash requests in the next iteration of its polling loop. The value stored in `logstash_jdbc_last_run` can be accessed in a SELECT statement as `sql_last_value`. + : This parameter specifies the field `unix_ts_in_secs` that tracks the last document read by Logstash from MySQL, stored on disk in [logstash_jdbc_last_run](logstash://reference/plugins-inputs-jdbc.md#plugins-inputs-jdbc-last_run_metadata_path). The parameter determines the starting value for documents that Logstash requests in the next iteration of its polling loop. The value stored in `logstash_jdbc_last_run` can be accessed in a SELECT statement as `sql_last_value`. unix_ts_in_secs : The field generated by the SELECT statement, which contains the `modification_time` as a standard [Unix timestamp](https://en.wikipedia.org/wiki/Unix_time) (seconds since the epoch). The field is referenced by the `tracking column`. A Unix timestamp is used for tracking progress rather than a normal timestamp, as a normal timestamp may cause errors due to the complexity of correctly converting back and forth between UMT and the local timezone. sql_last_value - : This is a [built-in parameter](asciidocalypse://docs/logstash/docs/reference/plugins-inputs-jdbc.md#_predefined_parameters) containing the starting point of the current iteration of the Logstash polling loop, and it is referenced in the SELECT statement line of the JDBC input configuration. This parameter is set to the most recent value of `unix_ts_in_secs`, which is read from `.logstash_jdbc_last_run`. This value is the starting point for documents returned by the MySQL query that is executed in the Logstash polling loop. Including this variable in the query guarantees that we’re not resending data that is already stored in Elasticsearch. + : This is a [built-in parameter](logstash://reference/plugins-inputs-jdbc.md#_predefined_parameters) containing the starting point of the current iteration of the Logstash polling loop, and it is referenced in the SELECT statement line of the JDBC input configuration. This parameter is set to the most recent value of `unix_ts_in_secs`, which is read from `.logstash_jdbc_last_run`. This value is the starting point for documents returned by the MySQL query that is executed in the Logstash polling loop. Including this variable in the query guarantees that we’re not resending data that is already stored in Elasticsearch. schedule : This uses cron syntax to specify how often Logstash should poll MySQL for changes. The specification `*/5 * * * * *` tells Logstash to contact MySQL every 5 seconds. Input from this plugin can be scheduled to run periodically according to a specific schedule. This scheduling syntax is powered by [rufus-scheduler](https://github.com/jmettraux/rufus-scheduler). The syntax is cron-like with some extensions specific to Rufus (for example, timezone support). @@ -288,7 +288,7 @@ In this section, we configure Logstash to send the MySQL data to Elasticsearch. ``` 1. Use the Cloud ID of your {{ech}} deployment. You can include or omit the `:` prefix at the beginning of the Cloud ID. Both versions work fine. Find your Cloud ID by going to the {{kib}} main menu and selecting Management > Integrations, and then selecting View deployment details. - 2. the default username is `elastic`. It is not recommended to use the `elastic` account for ingesting data as this is a superuser. We recommend using a user with reduced permissions, or an API Key with permissions specific to the indices or data streams that will be written to. Check [Configuring security in Logstash](asciidocalypse://docs/logstash/docs/reference/secure-connection.md) for information on roles and API Keys. Use the password provided when you created the deployment if using the `elastic` user, or the password used when creating a new ingest user with the roles specified in the [Configuring security in Logstash](asciidocalypse://docs/logstash/docs/reference/secure-connection.md) documentation. + 2. the default username is `elastic`. It is not recommended to use the `elastic` account for ingesting data as this is a superuser. We recommend using a user with reduced permissions, or an API Key with permissions specific to the indices or data streams that will be written to. Check [Configuring security in Logstash](logstash://reference/secure-connection.md) for information on roles and API Keys. Use the password provided when you created the deployment if using the `elastic` user, or the password used when creating a new ingest user with the roles specified in the [Configuring security in Logstash](logstash://reference/secure-connection.md) documentation. Following are some additional details about the configuration file settings: diff --git a/raw-migrated-files/cloud/cloud/ec-getting-started-search-use-cases-node-logs.md b/raw-migrated-files/cloud/cloud/ec-getting-started-search-use-cases-node-logs.md index c1263dec1..52e0ce000 100644 --- a/raw-migrated-files/cloud/cloud/ec-getting-started-search-use-cases-node-logs.md +++ b/raw-migrated-files/cloud/cloud/ec-getting-started-search-use-cases-node-logs.md @@ -1,6 +1,6 @@ # Ingest logs from a Node.js web application using Filebeat [ec-getting-started-search-use-cases-node-logs] -This guide demonstrates how to ingest logs from a Node.js web application and deliver them securely into an {{ech}} deployment. You’ll set up Filebeat to monitor a JSON-structured log file that has standard Elastic Common Schema (ECS) formatted fields, and you’ll then view real-time visualizations of the log events in Kibana as requests are made to the Node.js server. While Node.js is used for this example, this approach to monitoring log output is applicable across many client types. Check the list of [available ECS logging plugins](asciidocalypse://docs/ecs-logging/docs/reference/intro.md#_get_started). +This guide demonstrates how to ingest logs from a Node.js web application and deliver them securely into an {{ech}} deployment. You’ll set up Filebeat to monitor a JSON-structured log file that has standard Elastic Common Schema (ECS) formatted fields, and you’ll then view real-time visualizations of the log events in Kibana as requests are made to the Node.js server. While Node.js is used for this example, this approach to monitoring log output is applicable across many client types. Check the list of [available ECS logging plugins](ecs-logging://reference/intro.md#_get_started). This guide presents: @@ -33,7 +33,7 @@ For the three following packages, you can create a working directory to install npm install winston ``` -* The [Elastic Common Schema (ECS) formatter](asciidocalypse://docs/ecs-logging-nodejs/docs/reference/winston.md) for the Node.js winston logger - This plugin formats your Node.js logs into an ECS structured JSON format ideally suited for ingestion into Elasticsearch. To install the ECS winston logger, run the following command in your working directory so that the package is installed in the same location as the winston package: +* The [Elastic Common Schema (ECS) formatter](ecs-logging-nodejs://reference/winston.md) for the Node.js winston logger - This plugin formats your Node.js logs into an ECS structured JSON format ideally suited for ingestion into Elasticsearch. To install the ECS winston logger, run the following command in your working directory so that the package is installed in the same location as the winston package: ```sh npm install @elastic/ecs-winston-format diff --git a/raw-migrated-files/cloud/cloud/ec-getting-started-search-use-cases-python-logs.md b/raw-migrated-files/cloud/cloud/ec-getting-started-search-use-cases-python-logs.md index 5ee8ccf43..48028eabf 100644 --- a/raw-migrated-files/cloud/cloud/ec-getting-started-search-use-cases-python-logs.md +++ b/raw-migrated-files/cloud/cloud/ec-getting-started-search-use-cases-python-logs.md @@ -1,6 +1,6 @@ # Ingest logs from a Python application using Filebeat [ec-getting-started-search-use-cases-python-logs] -This guide demonstrates how to ingest logs from a Python application and deliver them securely into an {{ech}} deployment. You’ll set up Filebeat to monitor a JSON-structured log file that has standard Elastic Common Schema (ECS) formatted fields, and you’ll then view real-time visualizations of the log events in {{kib}} as they occur. While Python is used for this example, this approach to monitoring log output is applicable across many client types. Check the list of [available ECS logging plugins](asciidocalypse://docs/ecs-logging/docs/reference/intro.md). +This guide demonstrates how to ingest logs from a Python application and deliver them securely into an {{ech}} deployment. You’ll set up Filebeat to monitor a JSON-structured log file that has standard Elastic Common Schema (ECS) formatted fields, and you’ll then view real-time visualizations of the log events in {{kib}} as they occur. While Python is used for this example, this approach to monitoring log output is applicable across many client types. Check the list of [available ECS logging plugins](ecs-logging://reference/intro.md). You are going to learn how to: @@ -14,7 +14,7 @@ You are going to learn how to: ## Prerequisites [ec_prerequisites_2] -To complete these steps you need to have [Python](https://www.python.org/) installed on your system as well as the [Elastic Common Schema (ECS) logger](asciidocalypse://docs/ecs-logging-python/docs/reference/installation.md) for the Python logging library. +To complete these steps you need to have [Python](https://www.python.org/) installed on your system as well as the [Elastic Common Schema (ECS) logger](ecs-logging-python://reference/installation.md) for the Python logging library. To install *ecs-logging-python*, run: diff --git a/raw-migrated-files/cloud/cloud/ec-maintenance-mode-routing.md b/raw-migrated-files/cloud/cloud/ec-maintenance-mode-routing.md index 223d5d9a3..badafd1ca 100644 --- a/raw-migrated-files/cloud/cloud/ec-maintenance-mode-routing.md +++ b/raw-migrated-files/cloud/cloud/ec-maintenance-mode-routing.md @@ -7,7 +7,7 @@ The {{ecloud}} proxy routes HTTP requests to its deployment’s individual produ It might be helpful to temporarily block upstream requests in order to protect some or all instances or products within your deployment. For example, you might stop request routing in the following cases: * If another team within your company starts streaming new data into your production {{integrations-server}} without previous load testing, both it and {{es}} might experience performance issues. You might consider stopping routing requests on all {{integrations-server}} instances in order to protect your downstream {{es}} instance. -* If {{es}} is being overwhelmed by upstream requests, it might experience increased response times or even become unresponsive. This might impact your ability to resize components in your deployment and increase the duration of pending plans or increase the chance of plan changes failing. Because every {{es}} node is an [implicit coordinating node](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/node-settings.md), you should stop routing requests across all {{es}} nodes to completely block upstream traffic. +* If {{es}} is being overwhelmed by upstream requests, it might experience increased response times or even become unresponsive. This might impact your ability to resize components in your deployment and increase the duration of pending plans or increase the chance of plan changes failing. Because every {{es}} node is an [implicit coordinating node](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md), you should stop routing requests across all {{es}} nodes to completely block upstream traffic. ## Considerations [ec_considerations] diff --git a/raw-migrated-files/cloud/cloud/ec-manage-kibana-settings.md b/raw-migrated-files/cloud/cloud/ec-manage-kibana-settings.md index 059023584..d20d62cb5 100644 --- a/raw-migrated-files/cloud/cloud/ec-manage-kibana-settings.md +++ b/raw-migrated-files/cloud/cloud/ec-manage-kibana-settings.md @@ -3,7 +3,7 @@ {{ech}} supports most of the standard Kibana and X-Pack settings. Through a YAML editor in the console, you can append Kibana properties to the `kibana.yml` file. Your changes to the configuration file are read on startup. ::::{important} -Be aware that some settings that could break your cluster if set incorrectly and that the syntax might change between major versions. Before upgrading, be sure to review the full list of the [latest Kibana settings and syntax](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md). +Be aware that some settings that could break your cluster if set incorrectly and that the syntax might change between major versions. Before upgrading, be sure to review the full list of the [latest Kibana settings and syntax](kibana://reference/configuration-reference/general-settings.md). :::: @@ -43,7 +43,7 @@ If a setting is not supported by {{ech}}, you will get an error message when you ### Version 8.9.0+ [ec_version_8_9_0] `xpack.fleet.createArtifactsBulkBatchSize` -: Allow to configure batch size for creating and updating Fleet user artifacts. Examples include creation of Trusted Applications and Endpoint Exceptions in Security. To learn more, check [Fleet settings in Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/fleet-settings.md). +: Allow to configure batch size for creating and updating Fleet user artifacts. Examples include creation of Trusted Applications and Endpoint Exceptions in Security. To learn more, check [Fleet settings in Kibana](kibana://reference/configuration-reference/fleet-settings.md). `xpack.securitySolution.maxUploadResponseActionFileBytes` : Allow to configure the max file upload size for use with the Upload File Repsonse action available with the Defend Integration. To learn more, check [Endpoint Response actions](/solutions/security/endpoint-response-actions.md). @@ -55,43 +55,43 @@ If a setting is not supported by {{ech}}, you will get an error message when you : Set the maximum number of sessions each user is allowed to have active in {{kib}}. By default, no limit is applied. If set, the value of this option should be an integer between 1 and 1000. When the limit is exceeded, the oldest session is automatically invalidated. To learn more, check [Session management](/deploy-manage/security/kibana-session-management.md#session-max-sessions). `server.securityResponseHeaders.crossOriginOpenerPolicy` -: Controls whether the [`Cross-Origin-Opener-Policy`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cross-Origin-Opener-Policy) header is used in all responses to the client from the Kibana server. To learn more, see [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md#server-securityResponseHeaders-crossOriginOpenerPolicy). +: Controls whether the [`Cross-Origin-Opener-Policy`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cross-Origin-Opener-Policy) header is used in all responses to the client from the Kibana server. To learn more, see [Configure Kibana](kibana://reference/configuration-reference/general-settings.md#server-securityresponseheaders-crossoriginopenerpolicy). ### Version 8.6.0+ [ec_version_8_6_0] `server.compression.brotli.enabled` -: Enable brotli compression format for browser-server communications. Default: false. To learn more, check [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md). +: Enable brotli compression format for browser-server communications. Default: false. To learn more, check [Configure Kibana](kibana://reference/configuration-reference/general-settings.md). `xpack.fleet.enableExperimental` -: Allow to configure experimental feature for Fleet. To learn more, check [Fleet settings in Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/fleet-settings.md). +: Allow to configure experimental feature for Fleet. To learn more, check [Fleet settings in Kibana](kibana://reference/configuration-reference/fleet-settings.md). ### Version 8.4.0+ [ec_version_8_4_0] `migrations.discardUnknownObjects` -: Discard saved objects with unknown types during a migration. Must be set to the target version, e.g.: `8.4.0`. Default: undefined. To learn more, check [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md). +: Discard saved objects with unknown types during a migration. Must be set to the target version, e.g.: `8.4.0`. Default: undefined. To learn more, check [Configure Kibana](kibana://reference/configuration-reference/general-settings.md). `migrations.discardCorruptObjects` -: Discard corrupt saved objects, as well as those that cause transform errors during a migration. Must be set to the target version, e.g.: `8.4.0`. Default: undefined. To learn more, check [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md). +: Discard corrupt saved objects, as well as those that cause transform errors during a migration. Must be set to the target version, e.g.: `8.4.0`. Default: undefined. To learn more, check [Configure Kibana](kibana://reference/configuration-reference/general-settings.md). ### Version 8.3.0+ [ec_version_8_3_0] `elasticsearch.compression` -: Enable compression for communications with Elasticsearch. Default: false. To learn more, check [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md). +: Enable compression for communications with Elasticsearch. Default: false. To learn more, check [Configure Kibana](kibana://reference/configuration-reference/general-settings.md). ### Version 8.2.0+ [ec_version_8_2_0] `elasticsearch.maxSockets` -: The maximum number of sockets that can be used for communications with Elasticsearch. Default: Infinity. To learn more, check [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md). +: The maximum number of sockets that can be used for communications with Elasticsearch. Default: Infinity. To learn more, check [Configure Kibana](kibana://reference/configuration-reference/general-settings.md). ### Version 8.1.0+ [ec_version_8_1_0] `execution_context.enabled` -: Propagate request-specific metadata to Elasticsearch server by way of the `x-opaque-id` header. To learn more, check [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md). +: Propagate request-specific metadata to Elasticsearch server by way of the `x-opaque-id` header. To learn more, check [Configure Kibana](kibana://reference/configuration-reference/general-settings.md). ### Supported versions before 8.x [ec_supported_versions_before_8_x] @@ -106,34 +106,34 @@ If a setting is not supported by {{ech}}, you will get an error message when you ### All supported versions [ec_all_supported_versions_2] `migrations.maxBatchSizeBytes` -: Defines the maximum payload size for indexing batches of saved objects during upgrade migrations. To learn more, check [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md). +: Defines the maximum payload size for indexing batches of saved objects during upgrade migrations. To learn more, check [Configure Kibana](kibana://reference/configuration-reference/general-settings.md). `server.maxPayload` -: The maximum payload size in bytes for incoming server requests. Default: 1048576. To learn more, check [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md#server-maxPayload). +: The maximum payload size in bytes for incoming server requests. Default: 1048576. To learn more, check [Configure Kibana](kibana://reference/configuration-reference/general-settings.md#server-maxpayload). `server.securityResponseHeaders.strictTransportSecurity` -: Controls whether the [`Strict-Transport-Security`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security) header is used in all responses to the client from the Kibana server. To learn more, check [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md#server-securityResponseHeaders-strictTransportSecurity). +: Controls whether the [`Strict-Transport-Security`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security) header is used in all responses to the client from the Kibana server. To learn more, check [Configure Kibana](kibana://reference/configuration-reference/general-settings.md#server-securityresponseheaders-stricttransportsecurity). `server.securityResponseHeaders.xContentTypeOptions` -: Controls whether the [`X-Content-Type-Options`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options) header is used in all responses to the client from the Kibana server. To learn more, check [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md#server-securityResponseHeaders-xContentTypeOptions). +: Controls whether the [`X-Content-Type-Options`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options) header is used in all responses to the client from the Kibana server. To learn more, check [Configure Kibana](kibana://reference/configuration-reference/general-settings.md#server-securityresponseheaders-xcontenttypeoptions). `server.securityResponseHeaders.referrerPolicy` -: Controls whether the [`Referrer-Policy`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Referrer-Policy) header is used in all responses to the client from the Kibana server. To learn more, see [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md#server-securityResponseHeaders-referrerPolicy). +: Controls whether the [`Referrer-Policy`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Referrer-Policy) header is used in all responses to the client from the Kibana server. To learn more, see [Configure Kibana](kibana://reference/configuration-reference/general-settings.md#server-securityresponseheaders-referrerpolicy). `server.securityResponseHeaders.permissionsPolicy` -: Controls whether the `Permissions-Policy` header is used in all responses to the client from the Kibana server. To learn more, see [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md#server-securityResponseHeaders-permissionsPolicy). +: Controls whether the `Permissions-Policy` header is used in all responses to the client from the Kibana server. To learn more, see [Configure Kibana](kibana://reference/configuration-reference/general-settings.md#server-securityresponseheaders-permissionspolicy). `server.securityResponseHeaders.permissionsPolicyReportOnly` -: Controls whether the `Permissions-Policy-Report-Only` header is used in all responses to the client from the Kibana server. To learn more, see [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md#server-securityResponseHeaders-permissionsPolicy). +: Controls whether the `Permissions-Policy-Report-Only` header is used in all responses to the client from the Kibana server. To learn more, see [Configure Kibana](kibana://reference/configuration-reference/general-settings.md#server-securityresponseheaders-permissionspolicy). `server.securityResponseHeaders.disableEmbedding` -: Controls whether the [`Content-Security-Policy`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy) and [`X-Frame-Options`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options) headers are configured to disable embedding Kibana in other webpages using iframes. To learn more, see [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md#server-securityResponseHeaders-disableEmbedding). +: Controls whether the [`Content-Security-Policy`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy) and [`X-Frame-Options`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options) headers are configured to disable embedding Kibana in other webpages using iframes. To learn more, see [Configure Kibana](kibana://reference/configuration-reference/general-settings.md#server-securityresponseheaders-disableembedding). `data.autocomplete.valueSuggestions.timeout` -: Specifies the time in milliseconds to wait for autocomplete suggestions from Elasticsearch. The default is 1000. Allowed values are between 1 and 1200000. To learn more, check [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md). +: Specifies the time in milliseconds to wait for autocomplete suggestions from Elasticsearch. The default is 1000. Allowed values are between 1 and 1200000. To learn more, check [Configure Kibana](kibana://reference/configuration-reference/general-settings.md). `data.autocomplete.valueSuggestions.terminateAfter` -: Specifies the max number of documents loaded by each shard to generate autocomplete suggestions. The default is 100000. Allowed values are between 1 and 10000000. To learn more, check [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md). +: Specifies the max number of documents loaded by each shard to generate autocomplete suggestions. The default is 100000. Allowed values are between 1 and 10000000. To learn more, check [Configure Kibana](kibana://reference/configuration-reference/general-settings.md). `map.tilemap.options.attribution` : Adds the map attribution string. @@ -154,7 +154,7 @@ If a setting is not supported by {{ech}}, you will get an error message when you : Specifies the locale for all strings, dates, and number formats that can be localized. Defaults to `en` (English). `migrations.batchSize` -: Defines the number of documents migrated at a time during saved object upgrade migrations. To learn more, check [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md). +: Defines the number of documents migrated at a time during saved object upgrade migrations. To learn more, check [Configure Kibana](kibana://reference/configuration-reference/general-settings.md). `server.defaultRoute` : Specifies the default route when opening Kibana. You can use this setting to modify the landing page when opening Kibana. @@ -309,7 +309,7 @@ If you want to allow anonymous authentication in Kibana, these settings are supp #### Supported versions before 8.0.0 [ec_supported_versions_before_8_0_0] `xpack.security.sessionTimeout` -: Specifies the session duration in milliseconds. Allows a value between 15000 (15 seconds) and 86400000 (1 day). To learn more, check [Security settings in Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/security-settings.md). Deprecated in versions 7.6+ and removed in versions 8.0+. +: Specifies the session duration in milliseconds. Allows a value between 15000 (15 seconds) and 86400000 (1 day). To learn more, check [Security settings in Kibana](kibana://reference/configuration-reference/security-settings.md). Deprecated in versions 7.6+ and removed in versions 8.0+. #### All supported versions [ec_all_supported_versions_4] @@ -474,7 +474,7 @@ This setting is not available in versions 8.0.0 through 8.2.0. As such, this set : Sets the size of the ephemeral queue. Defaults to `10`. `xpack.actions.customHostSettings` -: An array of objects, one per host, containing the SSL/TLS settings used when executing connectors which make HTTPS and SMTP connections to the host servers. For details about using this setting, check [Alerting and action settings in Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/alerting-settings.md). +: An array of objects, one per host, containing the SSL/TLS settings used when executing connectors which make HTTPS and SMTP connections to the host servers. For details about using this setting, check [Alerting and action settings in Kibana](kibana://reference/configuration-reference/alerting-settings.md). `xpack.actions.ssl.proxyVerificationMode` : Controls the verification of the proxy server certificate that hosted-ems receives when making an outbound SSL/TLS connection to the host server. Valid values are `full`, `certificate`, and `none`. Use `full` to perform hostname verification, `certificate` to skip hostname verification, and `none` to skip verification. Default: `full`. @@ -564,10 +564,10 @@ This setting is not available in versions 8.0.0 through 8.2.0. As such, this set : Set to `true` to enable logging event log documents from alerting to the Kibana log, in addition to being indexed into the event log index. Default: `false`. `xpack.security.session.idleTimeout` -: Set the session duration. The format is a string of `count` and `unit`, where unit is one of `ms`,`s`,`m`,`h`,`d`,`w`,`M`,`Y`. For example, `70ms`, `5s`, `3d`, `1Y`. To learn more, check [Security settings in Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/security-settings.md). +: Set the session duration. The format is a string of `count` and `unit`, where unit is one of `ms`,`s`,`m`,`h`,`d`,`w`,`M`,`Y`. For example, `70ms`, `5s`, `3d`, `1Y`. To learn more, check [Security settings in Kibana](kibana://reference/configuration-reference/security-settings.md). `xpack.security.session.lifespan` -: Sets the maximum duration, also known as "absolute timeout". After this duration, the session will expire even if it is not idle. To learn more, check [Security settings in Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/security-settings.md). +: Sets the maximum duration, also known as "absolute timeout". After this duration, the session will expire even if it is not idle. To learn more, check [Security settings in Kibana](kibana://reference/configuration-reference/security-settings.md). `xpack.maps.showMapVisualizationTypes` : Set to `true` if you want to create new region map visualizations. @@ -588,7 +588,7 @@ This setting is not available in versions 8.0.0 through 8.2.0. As such, this set : When enabled, specifies the email address to receive cluster alert notifications. `xpack.monitoring.kibana.collection.interval` -: Controls [how often data samples are collected](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/monitoring-settings.md#monitoring-collection-settings). +: Controls [how often data samples are collected](elasticsearch://reference/elasticsearch/configuration-reference/monitoring-settings.md#monitoring-collection-settings). `xpack.monitoring.min_interval_seconds` : Specifies the minimum number of seconds that a time bucket in a chart can represent. If you modify the `xpack.monitoring.kibana.collection.interval`, use the same value in this setting. @@ -599,7 +599,7 @@ This setting is not available in versions 8.0.0 through 8.2.0. As such, this set `xpack.ml.enabled` : Set to true (default) to enable machine learning. - If set to `false` in `kibana.yml`, the machine learning icon is hidden in this Kibana instance. If `xpack.ml.enabled` is set to `true` in `elasticsearch.yml`, however, you can still use the machine learning APIs. To disable machine learning entirely, check the [Elasticsearch Machine Learning Settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/machine-learning-settings.md). + If set to `false` in `kibana.yml`, the machine learning icon is hidden in this Kibana instance. If `xpack.ml.enabled` is set to `true` in `elasticsearch.yml`, however, you can still use the machine learning APIs. To disable machine learning entirely, check the [Elasticsearch Machine Learning Settings](elasticsearch://reference/elasticsearch/configuration-reference/machine-learning-settings.md). #### Content security policy configuration [ec_content_security_policy_configuration] @@ -635,7 +635,7 @@ This setting is not available in versions 8.0.0 through 8.2.0. As such, this set : Add sources for the [Content Security Policy `form-action` directive](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/form-action) in reporting mode. $$$csp-strict$$$ `csp.strict` -: Blocks Kibana access to any browser that does not enforce even rudimentary CSP rules. In practice, this disables support for older, less safe browsers like Internet Explorer. **Default: `true`** To learn more, check [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md)]. +: Blocks Kibana access to any browser that does not enforce even rudimentary CSP rules. In practice, this disables support for older, less safe browsers like Internet Explorer. **Default: `true`** To learn more, check [Configure Kibana](kibana://reference/configuration-reference/general-settings.md)]. `csp.warnLegacyBrowsers` : Shows a warning message after loading Kibana to any browser that does not enforce even rudimentary CSP rules, though Kibana is still accessible. This configuration is effectively ignored when [`csp.strict`](../../../deploy-manage/deploy/elastic-cloud/edit-stack-settings.md#csp-strict) is enabled. **Default: `true`** @@ -650,7 +650,7 @@ $$$csp-strict$$$ `csp.strict` #### Permissions policy configuration [ec_permissions_policy_configuration] `permissionsPolicy.report_to` -: Add sources for the permissions policy `report-to` directive. To learn more, see [Configure Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/general-settings.md#server-securityResponseHeaders-permissionsPolicy) +: Add sources for the permissions policy `report-to` directive. To learn more, see [Configure Kibana](kibana://reference/configuration-reference/general-settings.md#server-securityresponseheaders-permissionspolicy) #### Banner settings [ec_banner_settings] @@ -692,7 +692,7 @@ Each method has its own unique limitations which are important to understand. `xpack.reporting.csv.scroll.duration` -: Amount of [time](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#time-units) allowed before {{kib}} cleans the scroll context during a CSV export. Valid option is either `auto` or [time](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#time-units), Defaults to `30s`. +: Amount of [time](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#time-units) allowed before {{kib}} cleans the scroll context during a CSV export. Valid option is either `auto` or [time](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#time-units), Defaults to `30s`. ::::{note} Support for the The option `auto` was included here, when the config value is set to `auto` the scroll context will be preserved for as long as is possible, before the report task is terminated due to the limits of `xpack.reporting.queue.timeout`. @@ -757,7 +757,7 @@ Support for the The option `auto` was included here, when the config value is se Defaults to `true`. `xpack.reporting.csv.scroll.duration` -: Amount of [time](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/rest-apis/api-conventions.md#time-units) allowed before {{kib}} cleans the scroll context during a CSV export. +: Amount of [time](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#time-units) allowed before {{kib}} cleans the scroll context during a CSV export. Defaults to `30s` (30 seconds). @@ -929,7 +929,7 @@ The following APM settings are supported in Kibana: `xpack.apm.ui.maxTraceItems` : Maximum number of child items displayed when viewing trace details. - Defaults to `1000`. Any positive value is valid. To learn more, check [APM settings in Kibana](asciidocalypse://docs/kibana/docs/reference/configuration-reference/apm-settings.md). + Defaults to `1000`. Any positive value is valid. To learn more, check [APM settings in Kibana](kibana://reference/configuration-reference/apm-settings.md). `xpack.apm.ui.enabled` diff --git a/raw-migrated-files/cloud/cloud/ec-monitoring-setup.md b/raw-migrated-files/cloud/cloud/ec-monitoring-setup.md index 13d58758a..af8f4bf4e 100644 --- a/raw-migrated-files/cloud/cloud/ec-monitoring-setup.md +++ b/raw-migrated-files/cloud/cloud/ec-monitoring-setup.md @@ -27,7 +27,7 @@ After you have created a new deployment, you should enable shipping logs and met 5. Select **Save**. -Optionally, turn on [audit logging](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/auding-settings.md) to capture security-related events, such as authentication failures, refused connections, and data-access events through the proxy. To turn on audit logging, [edit your deployment’s elasticsearch.yml file](../../../deploy-manage/deploy/elastic-cloud/edit-stack-settings.md) to add these lines: +Optionally, turn on [audit logging](elasticsearch://reference/elasticsearch/configuration-reference/auding-settings.md) to capture security-related events, such as authentication failures, refused connections, and data-access events through the proxy. To turn on audit logging, [edit your deployment’s elasticsearch.yml file](../../../deploy-manage/deploy/elastic-cloud/edit-stack-settings.md) to add these lines: ```sh xpack.security.audit.enabled: true @@ -106,12 +106,12 @@ You will get this request reported as a new log. Audit logs do not currently rep You should take advantage of the default [Elastic Stack monitoring alerts](/deploy-manage/monitor/monitoring-data/kibana-alerts.md) that are available out-of-the-box. You don’t have to do anything other than enable shipping logs and metrics to have them made available to you (which you did earlier on). -On top of these default alerts that write to indices you can investigate, you might want to add some custom actions, such as a [connector](asciidocalypse://docs/kibana/docs/reference/connectors-kibana.md) for Slack notifications. To set up these notifications, you first configure a Slack connector and then append it to the default alerts and actions. From Kibana: +On top of these default alerts that write to indices you can investigate, you might want to add some custom actions, such as a [connector](kibana://reference/connectors-kibana.md) for Slack notifications. To set up these notifications, you first configure a Slack connector and then append it to the default alerts and actions. From Kibana: 1. Go to **Stack Management** > **Rules and Connectors** > **Connectors** and create your Slack connector: 1. Select **Slack**. - 2. [Create a Slack Webhook URL](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/slack-action-type.md#configuring-slack) and paste it into the **Webhook URL** field. + 2. [Create a Slack Webhook URL](kibana://reference/connectors-kibana/slack-action-type.md#configuring-slack) and paste it into the **Webhook URL** field. 3. Select **Save**. 2. Go to **Stack Monitoring** and select **Enter setup mode**. diff --git a/raw-migrated-files/docs-content/serverless/ai-assistant-knowledge-base.md b/raw-migrated-files/docs-content/serverless/ai-assistant-knowledge-base.md index 9d2c0b497..0d5fadaf3 100644 --- a/raw-migrated-files/docs-content/serverless/ai-assistant-knowledge-base.md +++ b/raw-migrated-files/docs-content/serverless/ai-assistant-knowledge-base.md @@ -117,7 +117,7 @@ Refer to the following video for an example of adding a document to Knowledge Ba Add an index as a knowledge source when you want new information added to that index to automatically inform AI Assistant’s responses. Common security examples include asset inventories, network configuration information, on-call matrices, threat intelligence reports, and vulnerability scans. ::::{important} -Indices added to Knowledge Base must have at least one field mapped as [semantic text](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/semantic-text.md). +Indices added to Knowledge Base must have at least one field mapped as [semantic text](elasticsearch://reference/elasticsearch/mapping-reference/semantic-text.md). :::: diff --git a/raw-migrated-files/docs-content/serverless/detections-logsdb-index-mode-impact.md b/raw-migrated-files/docs-content/serverless/detections-logsdb-index-mode-impact.md index c5cae52bc..ebb99b4ce 100644 --- a/raw-migrated-files/docs-content/serverless/detections-logsdb-index-mode-impact.md +++ b/raw-migrated-files/docs-content/serverless/detections-logsdb-index-mode-impact.md @@ -2,22 +2,22 @@ Logsdb is enabled by default for {{serverless-full}}. This topic explains the impact of using logsdb index mode with {{sec-serverless}}. -With logsdb index mode, the original `_source` field is not stored in the index but can be reconstructed using [synthetic `_source`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source). +With logsdb index mode, the original `_source` field is not stored in the index but can be reconstructed using [synthetic `_source`](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source). -When the `_source` is reconstructed, [modifications](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source-modifications) are possible. Therefore, there could be a mismatch between users' expectations and how fields are formatted. +When the `_source` is reconstructed, [modifications](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source-modifications) are possible. Therefore, there could be a mismatch between users' expectations and how fields are formatted. Continue reading to find out how this affects specific {{sec-serverless}} components. -## Alerts [logsdb-alerts] +## Alerts [logsdb-alerts] When alerts are generated, the `_source` event is copied into the alert to retain the original data. When the logsdb index mode is applied, the `_source` event stored in the alert is reconstructed using synthetic `_source`. If you’re switching to use logsdb index mode, the `_source` field stored in the alert might look different in certain situations: -* [Arrays can be reconstructed differently or deduplicated](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source-modifications-leaf-arrays) -* [Field names](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source-modifications-field-names) -* `geo_point` data fields (refer to [Representation of ranges](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source-modifications-ranges) and [Reduced precision of `geo_point` values](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source-precision-loss-for-point-types) for more information) +* [Arrays can be reconstructed differently or deduplicated](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source-modifications-leaf-arrays) +* [Field names](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source-modifications-field-names) +* `geo_point` data fields (refer to [Representation of ranges](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source-modifications-ranges) and [Reduced precision of `geo_point` values](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md#synthetic-source-precision-loss-for-point-types) for more information) Alerts generated by the following rule types could be affected: @@ -28,7 +28,7 @@ Alerts generated by the following rule types could be affected: Alerts that are generated by threshold, {{ml}}, and event correlation sequence rules are not affected since they do not contain copies of the original source. -## Rule actions [logsdb-rule-actions] +## Rule actions [logsdb-rule-actions] While we do not recommend using `_source` for actions, in cases where the action relies on the `_source`, the same limitations and changes apply. @@ -37,7 +37,7 @@ If you send alert notifications by enabling [actions](../../../explore-analyze/a We recommend checking and adjusting the rule actions using `_source` before switching to logsdb index mode. -## Runtime fields [logsdb-runtime-fields] +## Runtime fields [logsdb-runtime-fields] Runtime fields that reference `_source` may be affected. Some runtime fields might not work and need to be adjusted. For example, if an event was indexed with the value of `agent.name` in the dot-notation form, it will be returned in the nested form and might not work. diff --git a/raw-migrated-files/docs-content/serverless/elasticsearch-differences.md b/raw-migrated-files/docs-content/serverless/elasticsearch-differences.md index 25035f5cb..db61c8909 100644 --- a/raw-migrated-files/docs-content/serverless/elasticsearch-differences.md +++ b/raw-migrated-files/docs-content/serverless/elasticsearch-differences.md @@ -36,7 +36,7 @@ To ensure optimal performance, follow these recommendations for sizing individua For large datasets that exceed the recommended maximum size for a single index, consider splitting your data across smaller indices and using an alias to search them collectively. -These recommendations do not apply to indices using better binary quantization (BBQ). Refer to [vector quantization](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/dense-vector.md#dense-vector-quantization) in the core {{es}} docs for more information. +These recommendations do not apply to indices using better binary quantization (BBQ). Refer to [vector quantization](elasticsearch://reference/elasticsearch/mapping-reference/dense-vector.md#dense-vector-quantization) in the core {{es}} docs for more information. ## API availability [elasticsearch-differences-serverless-apis-availability] @@ -88,7 +88,7 @@ When attempting to use an unavailable API, you’ll receive a clear error messag ## Settings availability [elasticsearch-differences-serverless-settings-availability] -In {{es-serverless}}, you can only configure [index-level settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md). Cluster-level settings and node-level settings are not required by end users and the `elasticsearch.yml` file is fully managed by Elastic. +In {{es-serverless}}, you can only configure [index-level settings](elasticsearch://reference/elasticsearch/index-settings/index.md). Cluster-level settings and node-level settings are not required by end users and the `elasticsearch.yml` file is fully managed by Elastic. Available settings : **Index-level settings**: Settings that control how {{es}} documents are processed, stored, and searched are available to end users. These include: @@ -147,7 +147,7 @@ The following features are planned for future support in all {{serverless-full}} The following features are not available in {{es-serverless}} and are not planned for future support: * [Custom plugins and bundles](/deploy-manage/deploy/elastic-cloud/upload-custom-plugins-bundles.md) -* [{{es}} for Apache Hadoop](asciidocalypse://docs/elasticsearch-hadoop/docs/reference/elasticsearch-for-apache-hadoop.md) -* [Scripted metric aggregations](asciidocalypse://docs/elasticsearch/docs/reference/data-analysis/aggregations/search-aggregations-metrics-scripted-metric-aggregation.md) +* [{{es}} for Apache Hadoop](elasticsearch-hadoop://reference/index.md) +* [Scripted metric aggregations](elasticsearch://reference/data-analysis/aggregations/search-aggregations-metrics-scripted-metric-aggregation.md) * Managed web crawler: You can use the [self-managed web crawler](https://github.com/elastic/crawler) instead. -* Managed Search connectors: You can use [self-managed Search connectors](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/search-connectors/self-managed-connectors.md) instead. +* Managed Search connectors: You can use [self-managed Search connectors](elasticsearch://reference/ingestion-tools/search-connectors/self-managed-connectors.md) instead. diff --git a/raw-migrated-files/docs-content/serverless/elasticsearch-ingest-data-through-api.md b/raw-migrated-files/docs-content/serverless/elasticsearch-ingest-data-through-api.md index be1fb2220..874cbb1b5 100644 --- a/raw-migrated-files/docs-content/serverless/elasticsearch-ingest-data-through-api.md +++ b/raw-migrated-files/docs-content/serverless/elasticsearch-ingest-data-through-api.md @@ -1,6 +1,6 @@ # Ingest data through API [elasticsearch-ingest-data-through-api] -The {{es}} APIs enable you to ingest data through code. You can use the APIs of one of the [language clients](../../../solutions/search/site-or-app/clients.md) or the {{es}} HTTP APIs. The examples on this page use the HTTP APIs to demonstrate how ingesting works in {{es}} through APIs. If you want to ingest timestamped data or have a more complex ingestion use case, check out [Beats](asciidocalypse://docs/beats/docs/reference/index.md) or [Logstash](asciidocalypse://docs/logstash/docs/reference/index.md). +The {{es}} APIs enable you to ingest data through code. You can use the APIs of one of the [language clients](../../../solutions/search/site-or-app/clients.md) or the {{es}} HTTP APIs. The examples on this page use the HTTP APIs to demonstrate how ingesting works in {{es}} through APIs. If you want to ingest timestamped data or have a more complex ingestion use case, check out [Beats](asciidocalypse://docs/beats/docs/reference/index.md) or [Logstash](logstash://reference/index.md). ## Using the bulk API [elasticsearch-ingest-data-through-api-using-the-bulk-api] diff --git a/raw-migrated-files/docs-content/serverless/observability-ai-assistant.md b/raw-migrated-files/docs-content/serverless/observability-ai-assistant.md index 5c5b613dd..6f6b59d86 100644 --- a/raw-migrated-files/docs-content/serverless/observability-ai-assistant.md +++ b/raw-migrated-files/docs-content/serverless/observability-ai-assistant.md @@ -12,9 +12,9 @@ The AI Assistant uses generative AI to provide: The AI Assistant integrates with your large language model (LLM) provider through our supported Elastic connectors: -* [OpenAI connector](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/openai-action-type.md) for OpenAI or Azure OpenAI Service. -* [Amazon Bedrock connector](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/bedrock-action-type.md) for Amazon Bedrock, specifically for the Claude models. -* [Google Gemini connector](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/gemini-action-type.md) for Google Gemini. +* [OpenAI connector](kibana://reference/connectors-kibana/openai-action-type.md) for OpenAI or Azure OpenAI Service. +* [Amazon Bedrock connector](kibana://reference/connectors-kibana/bedrock-action-type.md) for Amazon Bedrock, specifically for the Claude models. +* [Google Gemini connector](kibana://reference/connectors-kibana/gemini-action-type.md) for Google Gemini. ::::{important} The AI Assistant is powered by an integration with your large language model (LLM) provider. LLMs are known to sometimes present incorrect information as if it’s correct. Elastic supports configuration and connection to the LLM provider and your knowledge base, but is not responsible for the LLM’s responses. @@ -66,9 +66,9 @@ To set up the AI Assistant: 2. From **Project settings** → **Management** → **Connectors**, create a connector for your AI provider: - * [OpenAI](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/openai-action-type.md) - * [Amazon Bedrock](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/bedrock-action-type.md) - * [Google Gemini](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/gemini-action-type.md) + * [OpenAI](kibana://reference/connectors-kibana/openai-action-type.md) + * [Amazon Bedrock](kibana://reference/connectors-kibana/bedrock-action-type.md) + * [Google Gemini](kibana://reference/connectors-kibana/gemini-action-type.md) 3. Authenticate communication between {{obs-serverless}} and the AI provider by providing the following information: @@ -248,7 +248,7 @@ Clicking a prompt generates a message specific to that log entry. You can contin ### Add the AI Assistant connector to alerting workflows [observability-ai-assistant-add-the-ai-assistant-connector-to-alerting-workflows] -You can use the [Observability AI Assistant connector](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/obs-ai-assistant-action-type.md) to add AI-generated insights and custom actions to your alerting workflows. To do this: +You can use the [Observability AI Assistant connector](kibana://reference/connectors-kibana/obs-ai-assistant-action-type.md) to add AI-generated insights and custom actions to your alerting workflows. To do this: 1. [Create (or edit) an alerting rule](../../../solutions/observability/incident-management/create-manage-rules.md) and specify the conditions that must be met for the alert to fire. 2. Under **Actions**, select the **Observability AI Assistant** connector type. @@ -311,4 +311,4 @@ Enabling that feature can be done from the **Settings** tab of the AI Assistant ### Token limits [token-limits] -Most LLMs have a set number of tokens they can manage in single a conversation. When you reach the token limit, the LLM will throw an error, and Elastic will display a "Token limit reached" error. The exact number of tokens that the LLM can support depends on the LLM provider and model you’re using. If you are using an OpenAI connector, you can monitor token usage in **OpenAI Token Usage** dashboard. For more information, refer to the [OpenAI Connector documentation](asciidocalypse://docs/kibana/docs/reference/connectors-kibana/openai-action-type.md#openai-connector-token-dashboard). +Most LLMs have a set number of tokens they can manage in single a conversation. When you reach the token limit, the LLM will throw an error, and Elastic will display a "Token limit reached" error. The exact number of tokens that the LLM can support depends on the LLM provider and model you’re using. If you are using an OpenAI connector, you can monitor token usage in **OpenAI Token Usage** dashboard. For more information, refer to the [OpenAI Connector documentation](kibana://reference/connectors-kibana/openai-action-type.md#openai-connector-token-dashboard). diff --git a/raw-migrated-files/docs-content/serverless/observability-apm-get-started.md b/raw-migrated-files/docs-content/serverless/observability-apm-get-started.md index dae217945..c78944043 100644 --- a/raw-migrated-files/docs-content/serverless/observability-apm-get-started.md +++ b/raw-migrated-files/docs-content/serverless/observability-apm-get-started.md @@ -81,14 +81,14 @@ To send APM data to Elastic, you must install an APM agent and configure it to s Instrumentation is the process of extending your application’s code to report trace data to Elastic APM. Go applications must be instrumented manually at the source code level. To instrument your applications, use one of the following approaches: - * [Built-in instrumentation modules](asciidocalypse://docs/apm-agent-go/docs/reference/builtin-modules.md). - * [Custom instrumentation](asciidocalypse://docs/apm-agent-go/docs/reference/custom-instrumentation.md) and context propagation with the Go Agent API. + * [Built-in instrumentation modules](apm-agent-go://reference/builtin-modules.md). + * [Custom instrumentation](apm-agent-go://reference/custom-instrumentation.md) and context propagation with the Go Agent API. **Learn more in the {{apm-agent}} reference** - * [Supported technologies](asciidocalypse://docs/apm-agent-go/docs/reference/supported-technologies.md) - * [Advanced configuration](asciidocalypse://docs/apm-agent-go/docs/reference/configuration.md) - * [Detailed guide to instrumenting Go source code](asciidocalypse://docs/apm-agent-go/docs/reference/set-up-apm-go-agent.md) + * [Supported technologies](apm-agent-go://reference/supported-technologies.md) + * [Advanced configuration](apm-agent-go://reference/configuration.md) + * [Detailed guide to instrumenting Go source code](apm-agent-go://reference/set-up-apm-go-agent.md)