Skip to content

Commit

Permalink
Fix retention and backlog quota diagram (#973)
Browse files Browse the repository at this point in the history
* Fix retention and backlog quota diagram

* Address code review

* Add source diagrams file
  • Loading branch information
shibd authored Nov 12, 2024
1 parent b45d14c commit 6e206e6
Show file tree
Hide file tree
Showing 8 changed files with 515 additions and 21 deletions.
Binary file added diagrams/backlog-quota-new.graffle
Binary file not shown.
Binary file added diagrams/retention-new.graffle
Binary file not shown.
10 changes: 5 additions & 5 deletions docs/cookbooks-retention-expiry.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Pulsar's [admin interface](admin-api-overview.md) enables you to manage both ret
By default, when a Pulsar message arrives at a broker, the message is stored until it has been acknowledged on all subscriptions, at which point it is marked for deletion. You can override this behavior and retain messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.

The diagram below illustrates the concept of message retention.
![Concept of message retention in Pulsar.](/assets/retention.svg)
![Concept of message retention in Pulsar.](/assets/retention-new.svg)

Retention policies are useful when you use the Reader interface. The Reader interface does not use acknowledgments, and messages do not exist within backlogs. It is required to configure retention for Reader-only use cases.

Expand Down Expand Up @@ -189,7 +189,7 @@ admin.namespaces().getRetention(namespace);
You can control the allowable size and/or time of backlogs, at the namespace level, using *backlog quotas*. Pulsar uses a quota to enforce a hard limit on the logical size of the backlogs in a topic. Backlog quota triggers an alert policy (for example, producer exception) once the quota limit is reached.

The diagram below illustrates the concept of backlog quota.
![Concept of backlog quota in Pulsar](/assets/backlog-quota.svg)
![Concept of backlog quota in Pulsar](/assets/backlog-quota-new.svg)

Setting a backlog quota involves setting:

Expand All @@ -204,9 +204,9 @@ The following retention policies are available:
| `producer_exception` | The producer throws an exception when trying to send a message |
| `consumer_backlog_eviction` | The broker will begin discarding backlog messages |

> #### Beware the distinction between retention policy types
> As you may have noticed, there are two definitions of the term "retention policy" in Pulsar, one that applies to persistent storage of messages not in backlogs, and one that applies to messages within backlogs.
:::note
The scope of retention includes the backlog, so retention must be greater than the backlog quota. Otherwise, Pulsar will throw an error when both are set.
:::

Backlog quotas are handled at the namespace level. They can be managed via:

Expand Down
262 changes: 262 additions & 0 deletions static/assets/backlog-quota-new.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
233 changes: 233 additions & 0 deletions static/assets/retention-new.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
11 changes: 5 additions & 6 deletions versioned_docs/version-3.0.x/cookbooks-retention-expiry.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ Pulsar's [admin interface](admin-api-overview.md) enables you to manage both ret
By default, when a Pulsar message arrives at a broker, the message is stored until it has been acknowledged on all subscriptions, at which point it is marked for deletion. You can override this behavior and retain messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.

The diagram below illustrates the concept of message retention.
![](/assets/retention.svg)
![](/assets/retention-new.svg)

Retention policies are useful when you use the Reader interface. The Reader interface does not use acknowledgments, and messages do not exist within backlogs. It is required to configure retention for Reader-only use cases.

Expand Down Expand Up @@ -188,7 +188,7 @@ admin.namespaces().getRetention(namespace);
You can control the allowable size and/or time of backlogs, at the namespace level, using *backlog quotas*. Pulsar uses a quota to enforce a hard limit on the logical size of the backlogs in a topic. Backlog quota triggers an alert policy (for example, producer exception) once the quota limit is reached.

The diagram below illustrates the concept of backlog quota.
![](/assets/backlog-quota.svg)
![](/assets/backlog-quota-new.svg)

Setting a backlog quota involves setting:

Expand All @@ -203,10 +203,9 @@ Policy | Action
`producer_exception` | The producer throws an exception when trying to send a message
`consumer_backlog_eviction` | The broker will begin discarding backlog messages


> #### Beware the distinction between retention policy types
> As you may have noticed, there are two definitions of the term "retention policy" in Pulsar, one that applies to persistent storage of messages not in backlogs, and one that applies to messages within backlogs.
:::note
The scope of retention includes the backlog, so retention must be greater than the backlog quota. Otherwise, Pulsar will throw an error when both are set.
:::

Backlog quotas are handled at the namespace level. They can be managed via:

Expand Down
10 changes: 5 additions & 5 deletions versioned_docs/version-3.3.x/cookbooks-retention-expiry.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Pulsar's [admin interface](admin-api-overview.md) enables you to manage both ret
By default, when a Pulsar message arrives at a broker, the message is stored until it has been acknowledged on all subscriptions, at which point it is marked for deletion. You can override this behavior and retain messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.

The diagram below illustrates the concept of message retention.
![Concept of message retention in Pulsar.](/assets/retention.svg)
![Concept of message retention in Pulsar.](/assets/retention-new.svg)

Retention policies are useful when you use the Reader interface. The Reader interface does not use acknowledgments, and messages do not exist within backlogs. It is required to configure retention for Reader-only use cases.

Expand Down Expand Up @@ -189,7 +189,7 @@ admin.namespaces().getRetention(namespace);
You can control the allowable size and/or time of backlogs, at the namespace level, using *backlog quotas*. Pulsar uses a quota to enforce a hard limit on the logical size of the backlogs in a topic. Backlog quota triggers an alert policy (for example, producer exception) once the quota limit is reached.

The diagram below illustrates the concept of backlog quota.
![Concept of backlog quota in Pulsar](/assets/backlog-quota.svg)
![Concept of backlog quota in Pulsar](/assets/backlog-quota-new.svg)

Setting a backlog quota involves setting:

Expand All @@ -204,9 +204,9 @@ The following retention policies are available:
| `producer_exception` | The producer throws an exception when trying to send a message |
| `consumer_backlog_eviction` | The broker will begin discarding backlog messages |

> #### Beware the distinction between retention policy types
> As you may have noticed, there are two definitions of the term "retention policy" in Pulsar, one that applies to persistent storage of messages not in backlogs, and one that applies to messages within backlogs.
:::note
The scope of retention includes the backlog, so retention must be greater than the backlog quota. Otherwise, Pulsar will throw an error when both are set.
:::

Backlog quotas are handled at the namespace level. They can be managed via:

Expand Down
10 changes: 5 additions & 5 deletions versioned_docs/version-4.0.x/cookbooks-retention-expiry.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Pulsar's [admin interface](admin-api-overview.md) enables you to manage both ret
By default, when a Pulsar message arrives at a broker, the message is stored until it has been acknowledged on all subscriptions, at which point it is marked for deletion. You can override this behavior and retain messages that have already been acknowledged on all subscriptions by setting a *retention policy* for all topics in a given namespace. Retention is based on both a *size limit* and a *time limit*.

The diagram below illustrates the concept of message retention.
![Concept of message retention in Pulsar.](/assets/retention.svg)
![Concept of message retention in Pulsar.](/assets/retention-new.svg)

Retention policies are useful when you use the Reader interface. The Reader interface does not use acknowledgments, and messages do not exist within backlogs. It is required to configure retention for Reader-only use cases.

Expand Down Expand Up @@ -189,7 +189,7 @@ admin.namespaces().getRetention(namespace);
You can control the allowable size and/or time of backlogs, at the namespace level, using *backlog quotas*. Pulsar uses a quota to enforce a hard limit on the logical size of the backlogs in a topic. Backlog quota triggers an alert policy (for example, producer exception) once the quota limit is reached.

The diagram below illustrates the concept of backlog quota.
![Concept of backlog quota in Pulsar](/assets/backlog-quota.svg)
![Concept of backlog quota in Pulsar](/assets/backlog-quota-new.svg)

Setting a backlog quota involves setting:

Expand All @@ -204,9 +204,9 @@ The following retention policies are available:
| `producer_exception` | The producer throws an exception when trying to send a message |
| `consumer_backlog_eviction` | The broker will begin discarding backlog messages |

> #### Beware the distinction between retention policy types
> As you may have noticed, there are two definitions of the term "retention policy" in Pulsar, one that applies to persistent storage of messages not in backlogs, and one that applies to messages within backlogs.
:::note
The scope of retention includes the backlog, so retention must be greater than the backlog quota. Otherwise, Pulsar will throw an error when both are set.
:::

Backlog quotas are handled at the namespace level. They can be managed via:

Expand Down

0 comments on commit 6e206e6

Please sign in to comment.