Skip to content

Commit 886e2ed

Browse files
authored
add offset management section in http outbound (#311)
1 parent d070126 commit 886e2ed

File tree

5 files changed

+180
-5
lines changed
  • docs/hub/connectors/outbound
  • versioned_docs
    • version-0.11.11/hub/connectors/outbound
    • version-0.11.12/hub/connectors/outbound
    • version-0.12.0/hub/connectors/outbound
    • version-0.13.0/hub/connectors/outbound

5 files changed

+180
-5
lines changed

docs/hub/connectors/outbound/http.mdx

Lines changed: 36 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -165,10 +165,45 @@ In this case, additional transformation will be performed before records are sen
165165

166166
Read more about [JSON to JSON transformations].
167167

168+
### Offset Management
169+
170+
Fluvio Consumer Offset feature allows for a connector to store the offset in the Fluvio cluster and use it on restart.
171+
To activate it, you need to provide the `consumer` name and set the `strategy: auto`.
172+
See the example below:
173+
174+
```yaml
175+
apiVersion: 0.1.0
176+
meta:
177+
version: x.y.z
178+
name: my-http-sink
179+
type: http-sink
180+
topic: http-sink-topic
181+
consumer:
182+
id: my-http-sink
183+
offset:
184+
strategy: auto
185+
start: beginning
186+
flush-period:
187+
secs: 10
188+
nanos: 0
189+
http:
190+
endpoint: "http://127.0.0.1/post"
191+
```
192+
193+
After the connector processed any records, you can check the last stored offset value via:
194+
```bash
195+
$ fluvio consumer list
196+
CONSUMER TOPIC PARTITION OFFSET LAST SEEN
197+
my-http-sink http-sink-topic 0 0 3s
198+
```
199+
200+
Read more about [Fluvio Offset].
201+
168202
## Contributing
169203

170204
Follow on the conventional `CONTRIBUTING.md` file to setup your environment and
171205
contribute to this project.
172206

173207
[Transformations]: /docs/latest/fluvio/concepts/transformations
174-
[JSON to JSON transformations]: /docs/latest/hub/smartmodules/jolt
208+
[JSON to JSON transformations]: /docs/latest/hub/smartmodules/jolt
209+
[Fluvio Offset]: /docs/latest/fluvio/concepts/offsets

versioned_docs/version-0.11.11/hub/connectors/outbound/http.mdx

Lines changed: 36 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -165,10 +165,45 @@ In this case, additional transformation will be performed before records are sen
165165

166166
Read more about [JSON to JSON transformations].
167167

168+
### Offset Management
169+
170+
Fluvio Consumer Offset feature allows for a connector to store the offset in the Fluvio cluster and use it on restart.
171+
To activate it, you need to provide the `consumer` name and set the `strategy: auto`.
172+
See the example below:
173+
174+
```yaml
175+
apiVersion: 0.1.0
176+
meta:
177+
version: x.y.z
178+
name: my-http-sink
179+
type: http-sink
180+
topic: http-sink-topic
181+
consumer:
182+
id: my-http-sink
183+
offset:
184+
strategy: auto
185+
start: beginning
186+
flush-period:
187+
secs: 10
188+
nanos: 0
189+
http:
190+
endpoint: "http://127.0.0.1/post"
191+
```
192+
193+
After the connector processed any records, you can check the last stored offset value via:
194+
```bash
195+
$ fluvio consumer list
196+
CONSUMER TOPIC PARTITION OFFSET LAST SEEN
197+
my-http-sink http-sink-topic 0 0 3s
198+
```
199+
200+
Read more about [Fluvio Offset].
201+
168202
## Contributing
169203

170204
Follow on the conventional `CONTRIBUTING.md` file to setup your environment and
171205
contribute to this project.
172206

173207
[Transformations]: /docs/latest/fluvio/concepts/transformations
174-
[JSON to JSON transformations]: /docs/latest/hub/smartmodules/jolt
208+
[JSON to JSON transformations]: /docs/latest/hub/smartmodules/jolt
209+
[Fluvio Offset]: /docs/latest/fluvio/concepts/offsets

versioned_docs/version-0.11.12/hub/connectors/outbound/http.mdx

Lines changed: 36 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -165,10 +165,45 @@ In this case, additional transformation will be performed before records are sen
165165

166166
Read more about [JSON to JSON transformations].
167167

168+
### Offset Management
169+
170+
Fluvio Consumer Offset feature allows for a connector to store the offset in the Fluvio cluster and use it on restart.
171+
To activate it, you need to provide the `consumer` name and set the `strategy: auto`.
172+
See the example below:
173+
174+
```yaml
175+
apiVersion: 0.1.0
176+
meta:
177+
version: x.y.z
178+
name: my-http-sink
179+
type: http-sink
180+
topic: http-sink-topic
181+
consumer:
182+
id: my-http-sink
183+
offset:
184+
strategy: auto
185+
start: beginning
186+
flush-period:
187+
secs: 10
188+
nanos: 0
189+
http:
190+
endpoint: "http://127.0.0.1/post"
191+
```
192+
193+
After the connector processed any records, you can check the last stored offset value via:
194+
```bash
195+
$ fluvio consumer list
196+
CONSUMER TOPIC PARTITION OFFSET LAST SEEN
197+
my-http-sink http-sink-topic 0 0 3s
198+
```
199+
200+
Read more about [Fluvio Offset].
201+
168202
## Contributing
169203

170204
Follow on the conventional `CONTRIBUTING.md` file to setup your environment and
171205
contribute to this project.
172206

173207
[Transformations]: /docs/latest/fluvio/concepts/transformations
174-
[JSON to JSON transformations]: /docs/latest/hub/smartmodules/jolt
208+
[JSON to JSON transformations]: /docs/latest/hub/smartmodules/jolt
209+
[Fluvio Offset]: /docs/latest/fluvio/concepts/offsets

versioned_docs/version-0.12.0/hub/connectors/outbound/http.mdx

Lines changed: 36 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -165,10 +165,45 @@ In this case, additional transformation will be performed before records are sen
165165

166166
Read more about [JSON to JSON transformations].
167167

168+
### Offset Management
169+
170+
Fluvio Consumer Offset feature allows for a connector to store the offset in the Fluvio cluster and use it on restart.
171+
To activate it, you need to provide the `consumer` name and set the `strategy: auto`.
172+
See the example below:
173+
174+
```yaml
175+
apiVersion: 0.1.0
176+
meta:
177+
version: x.y.z
178+
name: my-http-sink
179+
type: http-sink
180+
topic: http-sink-topic
181+
consumer:
182+
id: my-http-sink
183+
offset:
184+
strategy: auto
185+
start: beginning
186+
flush-period:
187+
secs: 10
188+
nanos: 0
189+
http:
190+
endpoint: "http://127.0.0.1/post"
191+
```
192+
193+
After the connector processed any records, you can check the last stored offset value via:
194+
```bash
195+
$ fluvio consumer list
196+
CONSUMER TOPIC PARTITION OFFSET LAST SEEN
197+
my-http-sink http-sink-topic 0 0 3s
198+
```
199+
200+
Read more about [Fluvio Offset].
201+
168202
## Contributing
169203

170204
Follow on the conventional `CONTRIBUTING.md` file to setup your environment and
171205
contribute to this project.
172206

173207
[Transformations]: /docs/latest/fluvio/concepts/transformations
174-
[JSON to JSON transformations]: /docs/latest/hub/smartmodules/jolt
208+
[JSON to JSON transformations]: /docs/latest/hub/smartmodules/jolt
209+
[Fluvio Offset]: /docs/latest/fluvio/concepts/offsets

versioned_docs/version-0.13.0/hub/connectors/outbound/http.mdx

Lines changed: 36 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -165,10 +165,45 @@ In this case, additional transformation will be performed before records are sen
165165

166166
Read more about [JSON to JSON transformations].
167167

168+
### Offset Management
169+
170+
Fluvio Consumer Offset feature allows for a connector to store the offset in the Fluvio cluster and use it on restart.
171+
To activate it, you need to provide the `consumer` name and set the `strategy: auto`.
172+
See the example below:
173+
174+
```yaml
175+
apiVersion: 0.1.0
176+
meta:
177+
version: x.y.z
178+
name: my-http-sink
179+
type: http-sink
180+
topic: http-sink-topic
181+
consumer:
182+
id: my-http-sink
183+
offset:
184+
strategy: auto
185+
start: beginning
186+
flush-period:
187+
secs: 10
188+
nanos: 0
189+
http:
190+
endpoint: "http://127.0.0.1/post"
191+
```
192+
193+
After the connector processed any records, you can check the last stored offset value via:
194+
```bash
195+
$ fluvio consumer list
196+
CONSUMER TOPIC PARTITION OFFSET LAST SEEN
197+
my-http-sink http-sink-topic 0 0 3s
198+
```
199+
200+
Read more about [Fluvio Offset].
201+
168202
## Contributing
169203

170204
Follow on the conventional `CONTRIBUTING.md` file to setup your environment and
171205
contribute to this project.
172206

173207
[Transformations]: /docs/latest/fluvio/concepts/transformations
174-
[JSON to JSON transformations]: /docs/latest/hub/smartmodules/jolt
208+
[JSON to JSON transformations]: /docs/latest/hub/smartmodules/jolt
209+
[Fluvio Offset]: /docs/latest/fluvio/concepts/offsets

0 commit comments

Comments
 (0)