[AWS Firehose] Improve Firehose documentation (#4818)

* update firehose endpoint and data stream details

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

* improve api key and retry duration section

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

* fix wording of storage location

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

* improve further by modifying all related documents

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

* Update docs/en/observability/cloud-monitoring/aws/ingest-aws-firehose.asciidoc

Co-authored-by: Arianna Laudazzi <46651782+alaudazzi@users.noreply.github.com>

---------

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>
Co-authored-by: Arianna Laudazzi <46651782+alaudazzi@users.noreply.github.com>
(cherry picked from commit 9813ea7)

# Conflicts:
#	docs/en/observability/cloud-monitoring/aws/monitor-aws-cloudtrail-firehose.asciidoc
#	docs/en/observability/cloud-monitoring/aws/monitor-aws-cloudwatch-firehose.asciidoc
#	docs/en/observability/cloud-monitoring/aws/monitor-aws-firewall-firehose.asciidoc
#	docs/en/observability/cloud-monitoring/aws/monitor-aws-waf-firehose.asciidoc

docs/en/observability/cloud-monitoring/aws/ingest-aws-firehose.asciidoc

@@ -53,20 +53,21 @@ NOTE: For advanced use cases, source records can be transformed by invoking a cu
 
 . From the *Destination settings* panel, specify the following settings:
 +
-* *Elastic endpoint URL*: Enter the Elastic endpoint URL of your Elasticsearch cluster. To find the Elasticsearch endpoint, go to the Elastic Cloud console and select *Connection details*. Here is an example of how it looks like: `https://my-deployment.es.us-east-1.aws.elastic-cloud.com`.
+* *Elastic endpoint URL*: Enter the {es} endpoint here. To find the {es} endpoint, go to the https://cloud.elastic.co/[Elastic Cloud] console, select your deployment and use *Copy endpoint* next to *Elasticsearch*. Example: `https://<deployment_name>.es.<region>.<csp>.elastic-cloud.com`.
 +
-* *API key*: Enter the encoded Elastic API key. To create an API key, go to the Elastic Cloud console, select *Connection details* and click *Create and manage API keys*. If you are using an API key with *Restrict privileges*, make sure to review the Indices privileges to provide at least "auto_configure" & "write" permissions for the indices you will be using with this delivery stream.
+* *API key*: Enter the encoded Elastic API key. This can be created in Kibana by following the instructions under API Keys. If you are using an API key with Restricted privileges, be sure to review the Indices privileges to provide at least "auto_configure" & "write" permissions for the indices you will be using with this delivery stream.
 +
-* *Content encoding*: For a better network efficiency, leave content encoding set to GZIP.
+* *Content encoding*: To reduce the data transfer costs, use GZIP encoding.
 +
-* *Retry duration*: Determines how long Firehose continues retrying the request in the event of an error. A duration of 60-300s should be suitable for most use cases.
+* *Retry duration*: Determines how long Firehose continues retrying the request in the event of an error. A duration between 60 and 300 seconds should be suitable for most use cases.
 +
 * *Parameters*:
 +
-  ** `es_datastream_name`: This parameter is optional and can be used to set which data stream documents will be stored. If this parameter is not specified, data is sent to the `logs-awsfirehose-default` data stream by default.
+  ** `es_datastream_name`: This parameter is optional and can be used to set which data stream documents will be stored. If not specified, logs are stored in `logs-awsfirehose-default` data stream and metrics are stored in `metrics-aws.cloudwatch-default` data stream.
   ** `include_cw_extracted_fields`: This parameter is optional and can be set when using a CloudWatch logs subscription filter as the Firehose data source. When set to true, extracted fields generated by the filter pattern in the subscription filter will be collected. Setting this parameter can add many fields into each record and may significantly increase data volume in Elasticsearch. As such, use of this parameter should be carefully considered and used only when the extracted fields are required for specific filtering and/or aggregation.
   ** `set_es_document_id`: This parameter is optional and can be set to allow Elasticsearch to assign each document a random ID or use a calculated unique ID for each document. Default is true. When set to false, a random ID will be used for each document which will help indexing performance.
-. In the *Backup settings* panel, it is recommended to configure S3 backup for failed records. It’s then possible to configure workflows to automatically retry failed records, for example by using {esf-ref}/aws-elastic-serverless-forwarder.html[Elastic Serverless Forwarder].
++
+* *Backup settings*: It is recommended to configure S3 backup for failed records. These backups can be used to restore data losses caused by unforeseen service outages.
 
 [discrete]
 [[firehose-step-four]]

docs/en/observability/cloud-monitoring/aws/monitor-aws-cloudtrail-firehose.asciidoc

@@ -85,16 +85,24 @@ For more information on how to set up a Amazon Data Firehose delivery stream to
 
 . Collect {es} endpoint and API key from your deployment on Elastic Cloud.
 +
-- Elasticsearch endpoint URL: Enter the Elasticsearch endpoint URL of your Elasticsearch cluster. To find the Elasticsearch endpoint, go to the Elastic Cloud console and select *Connection details*.
-- API key: Enter the encoded Elastic API key. To create an API key, go to the Elastic Cloud console, select *Connection details* and click *Create and manage API keys*. If you are using an API key with *Restrict privileges*, make sure to review the Indices privileges to provide at least "auto_configure" & "write" permissions for the indices you will be using with this delivery stream.
+- *To find the Elasticsearch endpoint URL*:
+.. Go to the https://cloud.elastic.co/[Elastic Cloud] console
+.. Find your deployment in the *Hosted deployments* card and select *Manage*.
+.. Under *Applications* click *Copy endpoint* next to *Elasticsearch*.
+.. Make sure that your *Elasticsearch endpoint URL* includes `.es.` between the *deployment name* and *region*. Example: `https://<deployment_name>.es.<region>.<csp>.elastic-cloud.com`
+
+- *To create the API key*:
+.. Go to the https://cloud.elastic.co/[Elastic Cloud] console
+.. Select *Open Kibana*.
+.. Expand the left-hand menu, under *Management* select *Stack management > API Keys* and click *Create API key*. If you are using an API key with *Restrict privileges*, make sure to review the Indices privileges to provide at least `auto_configure` and `write` permissions for the indices you will be using with this delivery stream.
 
 . Set up the delivery stream by specifying the following data:
 +
-- Elastic endpoint URL
-- API key
-- Content encoding: gzip
-- Retry duration: 60 (default)
-- Backup settings: failed data only to s3 bucket
+- Elastic endpoint URL: The URL that you copied in the previous step.
+- API key: The API key that you created in the previous step.
+- Content encoding: To reduce the data transfer costs, use GZIP encoding.
+- Retry duration: A duration between 60 and 300 seconds should be suitable for most use cases.
+- Backup settings: It is recommended to configure S3 backup for failed records. These backups can then be used to restore failed data ingestion caused by unforeseen service outages.
 
 You now have an Amazon Data Firehose delivery specified with:
 

docs/en/observability/cloud-monitoring/aws/monitor-aws-cloudwatch-firehose.asciidoc

@@ -100,8 +100,24 @@ image::firehose-cloudwatch-firehose-stream.png[Amazon Firehose Stream]
 +
 NOTE: For advanced use cases, source records can be transformed by invoking a custom Lambda function. When using Elastic integrations, this should not be required.
 
-. In the **Destination settings** section, set the following parameter:
-`es_datastream_name` = `logs-aws.generic-default`
+. From the *Destination settings* panel, specify the following settings:
++
+* *To find the Elasticsearch endpoint URL*:
+.. Go to the https://cloud.elastic.co/[Elastic Cloud] console
+.. Find your deployment in the *Hosted deployments* card and select *Manage*.
+.. Under *Applications* click *Copy endpoint* next to *Elasticsearch*.
+.. Make sure that your *Elasticsearch endpoint URL* includes `.es.` between the *deployment name* and *region*. Example: `https://<deployment_name>.es.<region>.<csp>.elastic-cloud.com`
++
+* *To create the API key*:
+.. Go to the https://cloud.elastic.co/[Elastic Cloud] console
+.. Select *Open Kibana*.
+.. Expand the left-hand menu, under *Management* select *Stack management > API Keys* and click *Create API key*. If you are using an API key with *Restrict privileges*, make sure to review the Indices privileges to provide at least `auto_configure` and `write` permissions for the indices you will be using with this delivery stream.
++
+* *Content encoding*: To reduce the data transfer costs, use GZIP encoding.
++
+* *Retry duration*: Determines how long Firehose continues retrying the request in the event of an error. A duration between 60 and 300 seconds should be suitable for most use cases.
+
+. It is recommended to configure S3 backup for failed records from the *Backup settings* panel. These backups can be used to restore data losses caused by unforeseen service outages.
 
 The Firehose stream is now ready to send logs to your Elastic Cloud deployment.
 

docs/en/observability/cloud-monitoring/aws/monitor-aws-firewall-firehose.asciidoc

@@ -57,18 +57,24 @@ image::firehose-networkfirewall-stream.png[Firehose stream]
 
 . Collect {es} endpoint and API key from your deployment on Elastic Cloud.
 +
-- Elastic endpoint URL: Enter the Elasticsearch endpoint URL of your Elasticsearch cluster. To find the Elasticsearch endpoint, go to the Elastic Cloud console and select *Connection details*.
-+
-- API key: Enter the encoded Elastic API key. To create an API key, go to the Elastic Cloud console, select *Connection details* and click *Create and manage API keys*. If you are using an API key with *Restrict privileges*, make sure to review the Indices privileges to provide at least "auto_configure" and "write" permissions for the indices you will be using with this delivery stream.
+- *To find the Elasticsearch endpoint URL*:
+.. Go to the https://cloud.elastic.co/[Elastic Cloud] console
+.. Find your deployment in the *Hosted deployments* card and select *Manage*.
+.. Under *Applications* click *Copy endpoint* next to *Elasticsearch*.
+.. Make sure that your *Elasticsearch endpoint URL* includes `.es.` between the *deployment name* and *region*. Example: `https://<deployment_name>.es.<region>.<csp>.elastic-cloud.com`
+
+- *To create the API key*:
+.. Go to the https://cloud.elastic.co/[Elastic Cloud] console
+.. Select *Open Kibana*.
+.. Expand the left-hand menu, under *Management* select *Stack management > API Keys* and click *Create API key*. If you are using an API key with *Restrict privileges*, make sure to review the Indices privileges to provide at least `auto_configure` and `write` permissions for the indices you will be using with this delivery stream.
 
 . Set up the delivery stream by specifying the following data:
 +
-- Elastic endpoint URL
-- API key
-- Content encoding: gzip
-- Retry duration: 60 (default)
-- Parameter *es_datastream_name* = `logs-aws.firewall_logs-default`
-- Backup settings: failed data only to s3 bucket
+- Elastic endpoint URL: The URL that you copied in the previous step.
+- API key: The API key that you created in the previous step.
+- Content encoding: To reduce the data transfer costs, use GZIP encoding.
+- Retry duration: A duration between 60 and 300 seconds should be suitable for most use cases.
+- Backup settings: It is recommended to configure S3 backup for failed records. These backups can then be used to restore failed data ingestion caused by unforeseen service outages.
 
 The Firehose stream is ready to send logs to our Elastic Cloud deployment.
 

docs/en/observability/cloud-monitoring/aws/monitor-aws-waf-firehose.asciidoc

@@ -54,15 +54,22 @@ NOTE: For advanced use cases, source records can be transformed by invoking a cu
 
 . From the *Destination settings* panel, specify the following settings:
 +
-* *Elastic endpoint URL*: Enter the Elastic endpoint URL of your Elasticsearch cluster. To find the Elasticsearch endpoint, go to the Elastic Cloud console, navigate to the Integrations page, and select *Connection details*. Here is an example of how it looks like: `https://my-deployment.es.us-east-1.aws.elastic-cloud.com`.
+* *To find the Elasticsearch endpoint URL*:
+.. Go to the https://cloud.elastic.co/[Elastic Cloud] console
+.. Find your deployment in the *Hosted deployments* card and select *Manage*.
+.. Under *Applications* click *Copy endpoint* next to *Elasticsearch*.
+.. Make sure that your *Elasticsearch endpoint URL* includes `.es.` between the *deployment name* and *region*. Example: `https://<deployment_name>.es.<region>.<csp>.elastic-cloud.com`
 +
-* *API key*: Enter the encoded Elastic API key. To create an API key, go to the Elastic Cloud console, navigate to the Integrations page, select *Connection details* and click *Create and manage API keys*. If you are using an API key with *Restrict privileges*, make sure to review the Indices privileges to provide at least "auto_configure" & "write" permissions for the indices you will be using with this delivery stream.
+* *To create the API key*:
+.. Go to the https://cloud.elastic.co/[Elastic Cloud] console
+.. Select *Open Kibana*.
+.. Expand the left-hand menu, under *Management* select *Stack management > API Keys* and click *Create API key*. If you are using an API key with *Restrict privileges*, make sure to review the Indices privileges to provide at least `auto_configure` and `write` permissions for the indices you will be using with this delivery stream.
 +
-* *Content encoding*: For a better network efficiency, leave content encoding set to GZIP.
+* *Content encoding*: To reduce the data transfer costs, use GZIP encoding.
 +
-* *Retry duration*: Determines how long Firehose continues retrying the request in the event of an error. A duration of 60-300s should be suitable for most use cases.
-+
-* *es_datastream_name*: `logs-aws.waf-default`
+* *Retry duration*: Determines how long Firehose continues retrying the request in the event of an error. A duration between 60 and 300 seconds should be suitable for most use cases.
+
+. It is recommended to configure S3 backup for failed records from the *Backup settings* panel. These backups can be used to restore data losses caused by unforeseen service outages.
 
 [discrete]
 [[firehose-waf-step-four]]