Update webhooks all tenants

site/_diagrams/docs/events-webhooks/transaction-any-succeed-three-failures.plantuml

@@ -0,0 +1,19 @@
+@startuml
+  !include ../../_config.plantuml
+
+participant "Original Caller" as c
+participant "FusionAuth" as f
+participant "Webhook Recipient 1" as w1
+participant "Webhook Recipient 2" as w2
+participant "Webhook Recipient 3" as w3
+
+c -> f  : Updates user
+f -> w1  : Sends payload
+f -> w2  : Sends payload
+f -> w3  : Sends payload
+w1 -> f  : Times out
+w2 -> f  : Sends failure response
+w3 -> f  : Sends failure response
+f -> c   : Failure, operation rolls back, error returned
+
+@enduml

site/_diagrams/docs/events-webhooks/transaction-any-succeed-two-failures.plantuml

@@ -0,0 +1,29 @@
+@startuml
+  !include ../../_config.plantuml
+
+participant "Original Caller" as c
+participant "FusionAuth" as f
+participant "Webhook Recipient 1" as w1
+participant "Webhook Recipient 2" as w2
+participant "Webhook Recipient 3" as w3
+
+c -> f  : Updates user
+f -> w1  : Sends payload
+f -> w2  : Sends payload
+f -> w3  : Sends payload
+w1 -> f  : Sends success response
+w2 -> f  : Sends failure response
+w3 -> f  : Sends failure response
+f -> w2  : Sends payload
+f -> w3  : Sends payload
+w2 -> f  : Times out
+w3 -> f  : Times out
+f -> w2  : Sends payload
+f -> w3  : Sends payload
+w2 -> f  : Succeeds
+w3 -> f  : Times out
+f -> w3  : Sends payload
+w3 -> f  : Times out
+f -> c  : Success, operation completes
+
+@enduml

site/_diagrams/docs/events-webhooks/transaction-none.plantuml

@@ -0,0 +1,16 @@
+@startuml
+  !include ../../_config.plantuml
+
+participant "Original Caller" as c
+participant "FusionAuth" as f
+participant "Webhook Recipient 1" as w1
+participant "Webhook Recipient 2" as w2
+participant "Webhook Recipient 3" as w3
+
+c -> f  : Updates user
+f -> w1  : Sends payload
+f -> w2  : Sends payload
+f -> w3  : Sends payload
+f -> c  : Success, operation completes
+
+@enduml

site/docs/v1/tech/events-webhooks/_tenant-or-application-scoped-event.adoc

@@ -1,3 +1,3 @@
-This is a tenant or application scoped event. It can be sent to all applications in a tenant or to one or more specified applications.
+This is a tenant or application scoped event. It can be sent to all applications in a tenant or to one or more specified applications. It will also be sent to all tenants that are listening for this event. A [field]#tenantId# will be present in the payload to allow for filtering.
 
-The ability to limit the generation of an event for only certain applications is legacy functionality and may be modified in the future. You almost certainly want to enable this event at the tenant level and optionally filter on the `applicationId` when consuming the event.
+The ability to limit the generation of an event for only certain applications is legacy functionality and may be modified in the future. You almost certainly want to enable this event at the tenant level and optionally filter on the [field]#applicationId# when consuming the event.

site/docs/v1/tech/events-webhooks/_tenant-scoped-event.adoc

@@ -1 +1 @@
-This is a tenant scoped event.
+This is a tenant scoped event. This event will be sent to all tenants that are listening, but will contain a [field]#tenantId# to allow for filtering.

site/docs/v1/tech/events-webhooks/securing.adoc

@@ -6,7 +6,14 @@ description: Learn how to secure your Webhooks so that events coming from Fusion
 
 == Securing Webhooks
 
-FusionAuth sends JSON events to your configured Webhooks that might include user information or other sensitive data. Therefore, it is important to ensure that your Webhooks are secured properly to prevent data from being leaked. This document covers the standard methods for securing Webhooks.
+FusionAuth sends JSON events to your configured Webhooks that might include user information or other sensitive data. Therefore, it is important to ensure that your Webhooks are secured properly to prevent data from being leaked or stolen.
+
+This document covers the standard methods for securing Webhooks.
+
+* <<TLS v1.2>>
+* <<Headers>>
+* <<Certificates>>
+* <<Firewalls>>
 
 :request_entity: Webhook
 :request_entity_lc: webhook

site/docs/v1/tech/events-webhooks/writing-a-webhook.adoc

@@ -3,26 +3,86 @@ layout: doc
 title: Writing a Webhook
 description: Learn how to write a Webhook to handle and process events sent by FusionAuth
 ---
+:page-liquid:
+
 
 == Writing a Webhook
 
-In order to appropriately handle requests from the FusionAuth event, you must build a simple HTTP Webhook that listens for requests from the FusionAuth event system. Your Webhook must be designed to receive to simple HTTP `POST` requests with a JSON request body. The HTTP request will be sent using a `Content-Type` header value of `application/json`.
+In order to appropriately handle requests from the FusionAuth event, you must build a simple HTTP Webhook that listens for requests from the FusionAuth event system. Your Webhook must be designed to receive simple HTTP `POST` requests with a JSON request body. The HTTP request will be sent using a `Content-Type` header value of `application/json`.
 
 Additional headers may be added to the request by adding headers to the Webhook configuration.
 
-=== Responses
+* <<Writing a Webhook>>
+* <<Responses>>
+* <<Configuration>>
+** <<Example Configuration>>
+* <<Retries>>
+** <<Retry Examples>>
+* <<Calling FusionAuth APIs In Webhooks>>
+* <<Example Code>>
+
+== Responses
 
 Your Webhook must handle the RESTful request described above and send back an appropriate status code. Your Webhook must send back to FusionAuth an HTTP response code that indicates whether or not the event was successfully handled or not. If your Webhook handled the event properly, it must send back an HTTP response status code of `2xx`. If there was any type of error or failure, your Webhook must send back a non `2xx` HTTP response status.
 
-=== Configuration
+== Configuration
+
+Once your Webhook is complete and listening for events, you must configure your Webhook URL in FusionAuth. To add a webhook navigate to [breadcrumb]#Settings -> Webhooks#. 
+
+Then, configure the Tenant to listen for the event by navigating to [breadcrumb]#Tenants -> Your Tenant -> Webhooks#. 
+
+If you have multiple Webhooks configured for a single Tenant, the transaction setting for the event will dictate if FusionAuth will commit the transaction or not.
+
+If you have multiple tenants listening for the same event, they will all receive that event and can filter on the provided [field]#tenantId# to determine if they should handle the event.
+
+=== Example Configuration
+
+Here's an example scenario. You have two tenants, Pied Piper and Hooli. You have configured two webhooks listening for `user.create` events. One updates a separate user database, the other records information in an analytics system. Both the Pied Piper and Hooli tenants have the `user.create` event enabled in their webhook configurations. 
+
+In this scenario, each webhook will receive data when a user is created in either tenant, Pied Piper or Hooli.
+
+Transaction settings can be managed at the tenant level, but the webhooks receiving an event are not. Any webhook that is configured to receive the `user.create` event will play a role in the transaction. It is not possible, for example, to require only the analytics webhook to succeed for the Pied Piper tenant and only the user database sync to succeed for the Hooli tenant. If you need this level of granularity, run different FusionAuth instances.
+
+If you are separating your staging and production environments using tenants, webhooks will cross those boundaries. While you can filter on the tenant in the webhook itself, if you register both a production webhook and a staging webhook for the same event, the production webhook will receive staging data and the staging webhook will receive production data. In addition, webhook transactions will depend on both. The workaround is to run separate FusionAuth instances.
+
+Please review this issue for additional information about https://github.com/FusionAuth/fusionauth-issues/issues/1543[future webhook improvements].
+
+== Retries
+
+If the webhook transaction succeeds, FusionAuth will try to send the payload to any failed webhooks again. For example, if there are three webhooks set up to listen to a `user.update` request, and the transaction level is set to "Any single webhook must succeed" and one webhook succeeds, the two failures will be retried. FusionAuth will retry sending the payload up to three additional times. This retry logic means that webhook endpoints may receive a payload multiple times and should be prepared to handle such a situation.
+//TODO update when https://github.com/FusionAuth/fusionauth-issues/issues/1543 lands
+
+If not enough of the webhooks succeed to satisfy the transaction type initially, the operation will not succeed; for example, the user will not be updated. The originating call will receive an error HTTP status code.
+
+If a webhook endpoint times out, this is considered a failure, the same as if a non `2xx` status code is returned. If the endpoint does not respond after the retries, the failure will be logged in the system log.
+
+
+=== Retry Examples
+
+Below are flow diagrams of example requests. The order of the requests is not guaranteed, but is merely illustrative. In each of these, an API call such as a user update is made, and FusionAuth has been configured to fire off to three different webhooks at that time. The webhook transaction level and the webhook success statuses vary.
+
+Here's a situation with three webhooks and a webhook transaction level of "No webhooks are required to succeed". In this scenario, FusionAuth "fires and forgets":
+++++
+{% plantuml source: _diagrams/docs/events-webhooks/transaction-none.plantuml, alt: "A flow with three webhooks and a 'no webhooks are required to succeed' transaction level." %}
+++++
+
+Next, consider the scenario with three webhooks and a webhook transaction configuration of "Any single webhook must succeed" where "Webhook 1" succeeds. In this case, the other two webhooks are retried up to three additional times. "Webhook 2" succeeds eventually, but "Webhook 3" fails:
+
+++++
+{% plantuml source: _diagrams/docs/events-webhooks/transaction-any-succeed-two-failures.plantuml, alt: "A flow with three webhooks, a 'any single webhook must succeed' transaction level, and one success." %}
+++++
+
+Here's a configuration with three webhooks and a webhook transaction configuration of "Any single webhook must succeed" where all webhooks fail or time out. In this case, there are no retries, since the webhook transaction level was not met.
 
-Once your Webhook is complete and listening for events, you must configure your Webhook URL in FusionAuth. To add a webhook navigate to [breadcrumb]#Settings -> Webhooks#. If you have multiple Webhooks configured for a single Application, the transaction setting for the event or the User Action will dictate if FusionAuth will commit the transaction or not.
+++++
+{% plantuml source: _diagrams/docs/events-webhooks/transaction-any-succeed-three-failures.plantuml, alt: "A flow with three webhooks, a 'any single webhook must succeed' transaction level, and three failures." %}
+++++
 
-=== Calling FusionAuth APIs In Webhooks
+== Calling FusionAuth APIs In Webhooks
 
-Some events fire on creation of an entity in FusionAuth, such as `user.create`. You may want to modify the created entity, but if your webhook tries to modify the newly created object in a webhook handling the create event, the operation will fail. This is due to the fact that the operation occurs in a transaction and has not yet completed when the webhook runs. 
+Some events fire on creation of an entity in FusionAuth, such as `user.create`. You may want to modify the created entity, but if your webhook tries to modify the newly created object in a webhook handling the create event, the operation will fail. This is due to the fact that the operation occurs in a database transaction and has not yet completed when the webhook runs. 
 
-In fact, the created user will not be visible to any other API request until the transaction is committed. The operation fails because the webhook is trying to modify an object that has not yet been completely created and has not yet been committed to persistant storage. Depending upon your transaction configuration for a particular event, FusionAuth may wait until all webhooks have responded before committing the transaction.
+In fact, the created user will not be visible to any other API request until the transaction is committed. The operation fails because the webhook is trying to modify an object that has not yet been completely created and has not yet been committed to persistent storage. Depending upon your transaction configuration for a particular event, FusionAuth may wait until all webhooks have responded before committing the transaction.
 
 Even if you configure your webhook transaction to not require any webhooks to succeed, it is unlikely your code will operate as intended due to the parallel timing of the requests. The `user.create` event was not designed to allow a webhook to retrieve and modify the user.
 
@@ -41,7 +101,7 @@ In this example, you have a few options; which one is best depends on when you n
 
 While this scenario is most obvious when a user or registration is being created, it applies to all webhooks. The final state of the operation which caused the webhook is not persisted to FusionAuth until after the webhook finishes.
 
-=== Example Code
+== Example Code
 
 Here's a simple example of a Webhook written in Node using Express. In this example, if the event is a ``user.delete`` event, this code deletes all of the user's Todos.