Update webhooks all tenants

site/docs/v1/tech/events-webhooks/_tenant-or-application-scoped-event.adoc

@@ -1,3 +1,3 @@
-This is a tenant or application scoped event. It can be sent to all applications in a tenant or to one or more specified applications.
+This is a tenant or application scoped event. It can be sent to all applications in a tenant or to one or more specified applications. It will also be sent to all tenants that are listening for this event. A [field]#tenantId# will be present in the payload to allow for filtering.
 
-The ability to limit the generation of an event for only certain applications is legacy functionality and may be modified in the future. You almost certainly want to enable this event at the tenant level and optionally filter on the `applicationId` when consuming the event.
+The ability to limit the generation of an event for only certain applications is legacy functionality and may be modified in the future. You almost certainly want to enable this event at the tenant level and optionally filter on the [field]#applicationId# when consuming the event.

site/docs/v1/tech/events-webhooks/_tenant-scoped-event.adoc

@@ -1 +1 @@
-This is a tenant scoped event.
+This is a tenant scoped event. This event will be sent to all tenants that are listening, but will contain a [field]#tenantId# to allow for filtering.

site/docs/v1/tech/events-webhooks/writing-a-webhook.adoc

@@ -6,7 +6,7 @@ description: Learn how to write a Webhook to handle and process events sent by F
 
 == Writing a Webhook
 
-In order to appropriately handle requests from the FusionAuth event, you must build a simple HTTP Webhook that listens for requests from the FusionAuth event system. Your Webhook must be designed to receive to simple HTTP `POST` requests with a JSON request body. The HTTP request will be sent using a `Content-Type` header value of `application/json`.
+In order to appropriately handle requests from the FusionAuth event, you must build a simple HTTP Webhook that listens for requests from the FusionAuth event system. Your Webhook must be designed to receive simple HTTP `POST` requests with a JSON request body. The HTTP request will be sent using a `Content-Type` header value of `application/json`.
 
 Additional headers may be added to the request by adding headers to the Webhook configuration.
 
@@ -16,13 +16,40 @@ Your Webhook must handle the RESTful request described above and send back an ap
 
 === Configuration
 
-Once your Webhook is complete and listening for events, you must configure your Webhook URL in FusionAuth. To add a webhook navigate to [breadcrumb]#Settings -> Webhooks#. If you have multiple Webhooks configured for a single Application, the transaction setting for the event or the User Action will dictate if FusionAuth will commit the transaction or not.
+Once your Webhook is complete and listening for events, you must configure your Webhook URL in FusionAuth. To add a webhook navigate to [breadcrumb]#Settings -> Webhooks#. 
+
+Then, configure the Tenant to listen for the event by navigating to [breadcrumb]#Tenants -> Your Tenant -> Webhooks#. 
+
+If you have multiple Webhooks configured for a single Tenant, the transaction setting for the event will dictate if FusionAuth will commit the transaction or not.
+
+If you have multiple tenants listening for the same event, they will all receive that event and can filter on the provided [field]#tenantId# to determine if they should handle the event.
+
+=== Example Configuration 
+
+Here's an example scenario. You have two tenants, Pied Piper and Hooli. You have configured two webhooks listening for `user.create` events. One updates a separate user database, the other records information in an analytics system. Both the Pied Piper and Hooli tenants have the `user.create` event enabled in their webhook configurations. 
+
+In this scenario, each webhook will receive data when a user is created in either tenant, Pied Piper or Hooli.
+
+Transaction settings can be managed at the tenant level, but the webhooks receiving an event are not. Any webhook that is configured to receive the `user.create` event will play a role in the transaction. It is not possible, for example, to require only the analytics webhook to succeed for the Pied Piper tenant and only the user database sync to succeed for the Hooli tenant. If you need this level of granularity, run different FusionAuth instances.
+
+If you are separating your staging and production environments using tenants, webhooks will cross those boundaries. While you can filter on the tenant in the webhook itself, if you register both a production webhook and a staging webhook for the same event, the production webhook will receive staging data and the staging webhook will receive production data. In addition, webhook transactions will depend on both. The workaround is to run separate FusionAuth instances.
+
+Please review this issue for additional information about https://github.com/FusionAuth/fusionauth-issues/issues/1543[future webhook improvements].
+
+=== Retries
+
+If a webhook endpoint times out, this is considered neither a success or a failure. Instead, FusionAuth tries to send the payload again.
+
+//TODO update when https://github.com/FusionAuth/fusionauth-issues/issues/1543 lands
+FusionAuth will retry sendin the webhook payload up to three times. This retry logic means that webhook endpoints may receive a payload multiple times and should be prepared to handle such a situation.
+
+If the endpoint does not respond after the third time, the failure will be logged to the system log.
 
 === Calling FusionAuth APIs In Webhooks
 
 Some events fire on creation of an entity in FusionAuth, such as `user.create`. You may want to modify the created entity, but if your webhook tries to modify the newly created object in a webhook handling the create event, the operation will fail. This is due to the fact that the operation occurs in a transaction and has not yet completed when the webhook runs. 
 
-In fact, the created user will not be visible to any other API request until the transaction is committed. The operation fails because the webhook is trying to modify an object that has not yet been completely created and has not yet been committed to persistant storage. Depending upon your transaction configuration for a particular event, FusionAuth may wait until all webhooks have responded before committing the transaction.
+In fact, the created user will not be visible to any other API request until the transaction is committed. The operation fails because the webhook is trying to modify an object that has not yet been completely created and has not yet been committed to persistent storage. Depending upon your transaction configuration for a particular event, FusionAuth may wait until all webhooks have responded before committing the transaction.
 
 Even if you configure your webhook transaction to not require any webhooks to succeed, it is unlikely your code will operate as intended due to the parallel timing of the requests. The `user.create` event was not designed to allow a webhook to retrieve and modify the user.