Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deploy and manage -> Monitoring post migration initial cleaning #314

Merged
merged 14 commits into from
Feb 5, 2025
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ Data transfer out of deployments and between nodes of the cluster is hard to con

The largest contributor to inter-node data transfer is usually shard movement between nodes in a cluster. The only way to prevent shard movement is by having a single node in a single availability zone. This solution is only possible for clusters up to 64GB RAM and is not recommended as it creates a risk of data loss. [Oversharding](https://www.elastic.co/guide/en/elasticsearch/reference/current/avoid-oversharding.html) can cause excessive shard movement. Avoiding oversharding can also help control costs and improve performance. Note that creating snapshots generates inter-node data transfer. The *storage* cost of snapshots is detailed later in this document.

The exact root cause of unusual data transfer is not always something we can identify as it can have many causes, some of which are out of our control and not associated with Cloud configuration changes. It may help to [enable monitoring](../../monitor/stack-monitoring/stack-monitoring-on-elastic-cloud-deployments.md) and examine index and shard activity on your cluster.
The exact root cause of unusual data transfer is not always something we can identify as it can have many causes, some of which are out of our control and not associated with Cloud configuration changes. It may help to [enable monitoring](../../monitor/stack-monitoring/elastic-cloud-stack-monitoring.md) and examine index and shard activity on your cluster.


## Storage [storage]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ Before you start creating your own deployment templates, you should have: [tagge
9. On this page you can [configure index management](ece-configure-templates-index-management.md) by assigning attributes to each of the data nodes in the deployment template. In Kibana, you can configure an index lifecycle management (ILM) policy, based on the node attributes, to control how data moves across the nodes in your deployment.
10. Select **Stack features**.
11. You can select a [snapshot repository](../../tools/snapshot-and-restore/cloud-enterprise.md) to be used by default for deployment backups.
12. You can choose to [enable logging and monitoring](../../monitor/stack-monitoring/enable-stack-monitoring-on-ece-deployments.md) by default, so that deployment logs and metrics are send to a dedicated monitoring deployment, and so that additional log types, retention options, and Kibana visualizations are available on all deployments created using this template.
12. You can choose to [enable logging and monitoring](../../monitor/stack-monitoring/ece-stack-monitoring.md) by default, so that deployment logs and metrics are send to a dedicated monitoring deployment, and so that additional log types, retention options, and Kibana visualizations are available on all deployments created using this template.
13. Select **Extensions**.
14. Select any Elasticsearch extensions that you would like to be available automatically to all deployments created using the template.
15. Select **Save and create template**.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ In the case of the `admin-console-elasticsearch` and `security` system deploymen
The `logging-and-metrics` cluster is different since, as an ECE admin, you likely want to provide users with access to the cluster in order to troubleshoot issues without your assistance, for example. In order to manage access to that cluster, you can configure roles that will provide access to the relevant indices, map those to users, and manage access to Kibana by leveraging the Elastic security integration with your authentication provider, such as LDAP, SAML, or AD. To configure one of those security realms, check [LDAP](../../users-roles/cluster-or-deployment-auth/ldap.md), [Active Directory](../../users-roles/cluster-or-deployment-auth/active-directory.md) or [SAML](../../users-roles/cluster-or-deployment-auth/saml.md).

::::{note}
The `logging-and-metrics` cluster is only intended for troubleshooting ECE deployment issues. If your use case involves modifying or normalizing logs from {{es}} or {{kib}}, use a separate [dedicated monitoring deployment](../../monitor/stack-monitoring/enable-stack-monitoring-on-ece-deployments.md) instead.
The `logging-and-metrics` cluster is only intended for troubleshooting ECE deployment issues. If your use case involves modifying or normalizing logs from {{es}} or {{kib}}, use a separate [dedicated monitoring deployment](../../monitor/stack-monitoring/ece-stack-monitoring.md) instead.
::::


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -329,7 +329,7 @@ To avoid this, explicitly define the requests and limits mandated by your enviro

#### Monitoring Elasticsearch CPU using Stack Monitoring [k8s-monitor-compute-resources-stack-monitoring]

If [Stack Monitoring](../../monitor/stack-monitoring/enable-stack-monitoring-on-eck-deployments.md) is enabled, the pressure applied by the CPU cgroup controller to an Elasticsearch node can be evaluated from the **Stack Monitoring** page in Kibana.
If [Stack Monitoring](../../monitor/stack-monitoring/eck-stack-monitoring.md) is enabled, the pressure applied by the CPU cgroup controller to an Elasticsearch node can be evaluated from the **Stack Monitoring** page in Kibana.

1. On the **Stack Monitoring** page select the Elasticsearch node you want to monitor.
2. Select the **Advanced** tab.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -324,7 +324,7 @@ $$$azure-integration-modify-deployment$$$How can I modify my {{ecloud}} deployme
* [Update {{stack}} user settings](edit-stack-settings.md) in the component YML files.
* [Add or remove custom plugins](add-plugins-extensions.md).
* [Configure IP filtering](../../security/traffic-filtering.md).
* [Monitor your {{ecloud}} deployment](../../monitor/stack-monitoring/stack-monitoring-on-elastic-cloud-deployments.md) to ensure it remains healthy.
* [Monitor your {{ecloud}} deployment](../../monitor/stack-monitoring/elastic-cloud-stack-monitoring.md) to ensure it remains healthy.
* Add or remove API keys to use the [REST API](https://www.elastic.co/guide/en/cloud/current/ec-restful-api.html).
* [And more](cloud-hosted.md)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Autoscaling reduces some of the manual effort required to manage a deployment by

## {{es}} [ec-cluster-size]

Depending upon how much data you have and what queries you plan to run, you need to select a cluster size that fits your needs. There is no silver bullet for deciding how much memory you need other than simply testing it. The [cluster performance metrics](../../monitor/stack-monitoring.md) in the [Elasticsearch Service Console](https://cloud.elastic.co?page=docs&placement=docs-body) can tell you if your cluster is sized appropriately. You can also [enable deployment monitoring](../../monitor/stack-monitoring/stack-monitoring-on-elastic-cloud-deployments.md) for more detailed performance metrics. Fortunately, you can change the amount of memory allocated to the cluster later without any downtime for HA deployments.
Depending upon how much data you have and what queries you plan to run, you need to select a cluster size that fits your needs. There is no silver bullet for deciding how much memory you need other than simply testing it. The [cluster performance metrics](../../monitor/stack-monitoring.md) in the [Elasticsearch Service Console](https://cloud.elastic.co?page=docs&placement=docs-body) can tell you if your cluster is sized appropriately. You can also [enable deployment monitoring](../../monitor/stack-monitoring/elastic-cloud-stack-monitoring.md) for more detailed performance metrics. Fortunately, you can change the amount of memory allocated to the cluster later without any downtime for HA deployments.

To change a cluster’s topology, from deployment management, select **Edit deployment** from the **Actions** dropdown. Next, select a storage and RAM setting from the **Size per zone** drop-down list, and save your changes. When downsizing the cluster, make sure to have enough resources to handle the current load, otherwise your cluster will be under stress.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Autoscaling reduces some of the manual effort required to manage a deployment by

### {{es}} [ech-cluster-size]

Depending upon how much data you have and what queries you plan to run, you need to select a cluster size that fits your needs. There is no silver bullet for deciding how much memory you need other than simply testing it. The [cluster performance metrics](../../monitor/stack-monitoring/stack-monitoring-on-elastic-cloud-deployments.md) in the [Elasticsearch Add-On for Heroku console](https://cloud.elastic.co?page=docs&placement=docs-body) can tell you if your cluster is sized appropriately. You can also [enable deployment monitoring](../../monitor/stack-monitoring/stack-monitoring-on-elastic-cloud-deployments.md) for more detailed performance metrics. Fortunately, you can change the amount of memory allocated to the cluster later without any downtime for HA deployments.
Depending upon how much data you have and what queries you plan to run, you need to select a cluster size that fits your needs. There is no silver bullet for deciding how much memory you need other than simply testing it. The [cluster performance metrics](../../monitor/stack-monitoring/elastic-cloud-stack-monitoring.md) in the [Elasticsearch Add-On for Heroku console](https://cloud.elastic.co?page=docs&placement=docs-body) can tell you if your cluster is sized appropriately. You can also [enable deployment monitoring](../../monitor/stack-monitoring/elastic-cloud-stack-monitoring.md) for more detailed performance metrics. Fortunately, you can change the amount of memory allocated to the cluster later without any downtime for HA deployments.

To change a cluster’s topology, from deployment management, select **Edit deployment** from the **Actions** dropdown. Next, select a storage and RAM setting from the **Size per zone** drop-down list, and save your changes. When downsizing the cluster, make sure to have enough resources to handle the current load, otherwise your cluster will be under stress.

Expand Down
4 changes: 2 additions & 2 deletions deploy-manage/deploy/elastic-cloud/ech-restrictions.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ When using Elasticsearch Add-On for Heroku, there are some limitations you shoul
* [Regions and Availability Zones](#ech-regions-and-availability-zone)
* [Known problems](#ech-known-problems)

For limitations related to logging and monitoring, check the [Restrictions and limitations](../../monitor/stack-monitoring/stack-monitoring-on-elastic-cloud-deployments.md) section of the logging and monitoring page.
For limitations related to logging and monitoring, check the [Restrictions and limitations](../../monitor/stack-monitoring/elastic-cloud-stack-monitoring.md) section of the logging and monitoring page.

Occasionally, we also publish information about [Known problems](#ech-known-problems) with our Elasticsearch Add-On for Heroku or the Elastic Stack.

Expand Down Expand Up @@ -53,7 +53,7 @@ Generally, if a feature is shown as available in the [Elasticsearch Add-On for H
* Elasticsearch plugins, are not enabled by default for security purposes. Please reach out to support if you would like to enable Elasticsearch plugins support on your account.
* Some Elasticsearch plugins do not apply to Elasticsearch Add-On for Heroku. For example, you won’t ever need to change discovery, as Elasticsearch Add-On for Heroku handles how nodes discover one another.
* In Elasticsearch 5.0 and later, site plugins are no longer supported. This change does not affect the site plugins Elasticsearch Add-On for Heroku might provide out of the box, such as Kopf or Head, since these site plugins are serviced by our proxies and not Elasticsearch itself.
* In Elasticsearch 5.0 and later, site plugins such as Kopf and Paramedic are no longer provided. We recommend that you use our [cluster performance metrics](../../monitor/stack-monitoring/stack-monitoring-on-elastic-cloud-deployments.md), [X-Pack monitoring features](../../monitor/stack-monitoring/stack-monitoring-on-elastic-cloud-deployments.md) and Kibana’s (6.3+) [Index Management UI](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-mgmt.html) if you want more detailed information or perform index management actions.
* In Elasticsearch 5.0 and later, site plugins such as Kopf and Paramedic are no longer provided. We recommend that you use our [cluster performance metrics](../../monitor/stack-monitoring/elastic-cloud-stack-monitoring.md), [X-Pack monitoring features](../../monitor/stack-monitoring/elastic-cloud-stack-monitoring.md) and Kibana’s (6.3+) [Index Management UI](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-mgmt.html) if you want more detailed information or perform index management actions.


## Private Link and SSO to Kibana URLs [ech-restrictions-traffic-filters-kibana-sso]
Expand Down
2 changes: 1 addition & 1 deletion deploy-manage/deploy/elastic-cloud/manage-deployments.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ mapped_pages:
Sometimes you might need to make changes to the entire deployment, a specific component, or just a single data tier.

* Make adjustments to specific deployment components, such as an [Integrations Server](manage-integrations-server.md), [APM & Fleet Server](switch-from-apm-to-integrations-server-payload.md#ec-manage-apm-and-fleet), [Enterprise Search](https://www.elastic.co/guide/en/cloud/current/ec-enable-enterprise-search.html), [Watcher](../../../explore-analyze/alerts/watcher.md), or [Kibana](access-kibana.md#ec-enable-kibana2).
* [Enable logging and monitoring](../../monitor/stack-monitoring/stack-monitoring-on-elastic-cloud-deployments.md) of the deployment performance.
* [Enable logging and monitoring](../../monitor/stack-monitoring/elastic-cloud-stack-monitoring.md) of the deployment performance.
* [Disable a data tier](../../../manage-data/lifecycle/index-lifecycle-management.md).
* [Restart](../../maintenance/start-stop-services/restart-cloud-hosted-deployment.md), [stop routing](../../maintenance/ece/start-stop-routing-requests.md), or [delete your deployment](../../uninstall/delete-a-cloud-deployment.md).
* [Upgrade the Elastic Stack version](../../upgrade/deployment-or-cluster.md) for the deployment.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ When using Elasticsearch Service, there are some limitations you should be aware
* [Regions and Availability Zones](#ec-regions-and-availability-zone)
* [Known problems](#ec-known-problems)

For limitations related to logging and monitoring, check the [Restrictions and limitations](../../monitor/stack-monitoring/stack-monitoring-on-elastic-cloud-deployments.md#ec-restrictions-monitoring) section of the logging and monitoring page.
For limitations related to logging and monitoring, check the [Restrictions and limitations](../../monitor/stack-monitoring/elastic-cloud-stack-monitoring.md#ec-restrictions-monitoring) section of the logging and monitoring page.

Occasionally, we also publish information about [Known problems](#ec-known-problems) with our Elasticsearch Service or the Elastic Stack.

Expand Down
2 changes: 1 addition & 1 deletion deploy-manage/deploy/self-managed/access.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ Troubleshoot the `Kibana Server is not Ready yet` error.
These {{kib}}-backing indices must also not have [index settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-settings.html) flagging `read_only_allow_delete` or `write` [index blocks](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-blocks.html).

3. [Shut down all {{kib}} nodes](../../maintenance/start-stop-services/start-stop-kibana.md).
4. Choose any {{kib}} node, then update the config to set the [debug logging](../../monitor/logging-configuration/log-settings-examples.md#change-overall-log-level).
4. Choose any {{kib}} node, then update the config to set the [debug logging](../../monitor/logging-configuration/kibana-log-settings-examples.md#change-overall-log-level).
5. [Start the node](../../maintenance/start-stop-services/start-stop-kibana.md), then check the start-up debug logs for `ERROR` messages or other start-up issues.

For example:
Expand Down
6 changes: 3 additions & 3 deletions deploy-manage/deploy/self-managed/configure.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,10 +196,10 @@ $$$logging-root-appenders$$$ `logging.root.appenders`
: A list of logging appenders to forward the root level logger instance to. By default `root` is configured with the `default` appender that logs to stdout with a `pattern` layout. This is the configuration that all custom loggers will use unless they’re re-configured explicitly. You can override the default behavior by configuring a different [appender](../../monitor/logging-configuration/kibana-logging.md#logging-appenders) to apply to `root`.

$$$logging-root-level$$$ `logging.root.level` ![logo cloud](https://doc-icons.s3.us-east-2.amazonaws.com/logo_cloud.svg "Supported on {{ess}}")
: Level at which a log record should be logged. Supported levels are: *all*, *fatal*, *error*, *warn*, *info*, *debug*, *trace*, *off*. Levels are ordered from *all* (highest) to *off* and a log record will be logged it its level is higher than or equal to the level of its logger, otherwise the log record is ignored. Use this value to [change the overall log level](../../monitor/logging-configuration/log-settings-examples.md#change-overall-log-level). **Default: `info`**.
: Level at which a log record should be logged. Supported levels are: *all*, *fatal*, *error*, *warn*, *info*, *debug*, *trace*, *off*. Levels are ordered from *all* (highest) to *off* and a log record will be logged it its level is higher than or equal to the level of its logger, otherwise the log record is ignored. Use this value to [change the overall log level](../../monitor/logging-configuration/kibana-log-settings-examples.md#change-overall-log-level). **Default: `info`**.

::::{tip}
Set to `all` to log all events, including system usage information and all requests. Set to `off` to silence all logs. You can also use the logging [cli commands](../../monitor/logging-configuration/_cli_configuration.md#logging-cli-migration) to set log level to `verbose` or silence all logs.
Set to `all` to log all events, including system usage information and all requests. Set to `off` to silence all logs. You can also use the logging [cli commands](../../monitor/logging-configuration/kibana-logging-cli-configuration.md#logging-cli-migration) to set log level to `verbose` or silence all logs.
::::


Expand All @@ -220,7 +220,7 @@ $$$logging-root-level$$$ `logging.root.level` ![logo cloud](https://doc-icons.s3


$$$logging-loggers$$$ `logging.loggers[]`
: Allows you to [customize a specific logger instance](../../monitor/logging-configuration/log-settings-examples.md#customize-specific-log-records).
: Allows you to [customize a specific logger instance](../../monitor/logging-configuration/kibana-log-settings-examples.md#customize-specific-log-records).

`logging.appenders[]`
: [Appenders](../../monitor/logging-configuration/kibana-logging.md#logging-appenders) define how and where log messages are displayed (eg. **stdout** or console) and stored (eg. file on the disk).
Expand Down
6 changes: 6 additions & 0 deletions deploy-manage/monitor.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,12 @@
mapped_urls:
- https://www.elastic.co/guide/en/elasticsearch/reference/current/monitor-elasticsearch-cluster.html
- https://www.elastic.co/guide/en/elasticsearch/reference/current/secure-monitoring.html
applies:
serverless: all
hosted: all
ece: all
eck: all
stack: all
---

# Monitoring
Expand Down
2 changes: 2 additions & 0 deletions deploy-manage/monitor/autoops.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
---
mapped_pages:
- https://www.elastic.co/guide/en/cloud/current/ec-autoops.html
applies:
hosted: all
---

# AutoOps [ec-autoops]
Expand Down
2 changes: 2 additions & 0 deletions deploy-manage/monitor/autoops/ec-autoops-deployment-view.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
---
mapped_pages:
- https://www.elastic.co/guide/en/cloud/current/ec-autoops-deployment-view.html
applies:
hosted: all
---

# Deployment [ec-autoops-deployment-view]
Expand Down
2 changes: 2 additions & 0 deletions deploy-manage/monitor/autoops/ec-autoops-dismiss-event.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
---
mapped_pages:
- https://www.elastic.co/guide/en/cloud/current/ec-autoops-dismiss-event.html
applies:
hosted: all
---

# Dismiss Events [ec-autoops-dismiss-event]
Expand Down
2 changes: 2 additions & 0 deletions deploy-manage/monitor/autoops/ec-autoops-event-settings.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
---
mapped_pages:
- https://www.elastic.co/guide/en/cloud/current/ec-autoops-event-settings.html
applies:
hosted: all
---

# Events Settings [ec-autoops-event-settings]
Expand Down
2 changes: 2 additions & 0 deletions deploy-manage/monitor/autoops/ec-autoops-events.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
---
mapped_pages:
- https://www.elastic.co/guide/en/cloud/current/ec-autoops-events.html
applies:
hosted: all
---

# AutoOps events [ec-autoops-events]
Expand Down
2 changes: 2 additions & 0 deletions deploy-manage/monitor/autoops/ec-autoops-faq.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
---
mapped_pages:
- https://www.elastic.co/guide/en/cloud/current/ec-autoops-faq.html
applies:
hosted: all
---

# AutoOps FAQ [ec-autoops-faq]
Expand Down
Loading
Loading