Skip to content

Commit 5b33686

Browse files
ajhunyadyajhunyady
authored
updated how to run hub (#297)
1 parent 12803e4 commit 5b33686

5 files changed

+99
-46
lines changed

‎docs/_embeds/cmds/cdk-deploy-start-http-sink-generic.bash

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
cdk deploy start --ipkg infinyon-http-sink-0.2.11.ipkg --config <CONFIG>

‎docs/_embeds/cmds/cdk-hub-download-http-sink.bash

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
cdk hub download infinyon/http-sink@0.2.11

‎docs/_embeds/cmds/hub-conn-list.bash

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
fluvio hub connector list
2+
CONNECTOR Visibility
3+
infinyon-labs/graphite-sink@0.2.0 public
4+
infinyon/duckdb-sink@0.1.0 public
5+
infinyon/http-sink@0.2.11 public
6+
infinyon/http-source@0.4.3 public
7+
infinyon/ic-webhook-source@0.1.4 public
8+
infinyon/inf-webhook-source@0.1.9 public
9+
infinyon/kafka-sink@0.2.9 public
10+
infinyon/kafka-source@0.2.7 public
11+
infinyon/mqtt-source@0.2.9 public
12+
infinyon/sql-sink@0.4.3 public
13+
qdrant/qdrant-sink@0.1.0 public

‎docs/_embeds/cmds/hub-sm-list.bash

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
fluvio hub smartmodule list
2+
SMARTMODULE Visibility
3+
infinyon-labs/array-map-json@0.1.1 public
4+
infinyon-labs/csv-json@0.2.0 public
5+
infinyon-labs/dedup-filter@0.0.2 public
6+
infinyon-labs/json-formatter@0.1.3 public
7+
infinyon-labs/key-gen-json@0.1.1 public
8+
infinyon-labs/regex-map-json@0.1.2 public
9+
infinyon-labs/regex-map@0.1.1 public
10+
infinyon-labs/rss-json@0.1.1 public
11+
infinyon-labs/stars-forks-changes@0.1.4 public
12+
infinyon/csv-json@0.2.0 public
13+
infinyon/jolt@0.4.1 public
14+
infinyon/json-sql@0.2.1 public
15+
infinyon/regex-filter@0.2.0 public

‎docs/hub/use-the-hub.mdx

Lines changed: 69 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -3,68 +3,91 @@ sidebar_position: 2
33
title: "Use the Hub"
44
description: "The Hub is a central repository for Connectors, SmartModules, Dataflows, and more"
55
---
6+
import CodeBlock from '@theme/CodeBlock';
7+
import HubConnList from '!!raw-loader!../_embeds/cmds/hub-conn-list.bash';
8+
import HubSmList from '!!raw-loader!../_embeds/cmds/hub-sm-list.bash';
9+
import HubSmDownloadJolt from '!!raw-loader!../_embeds/cmds/hub-sm-download-jolt.bash';
10+
import CdkDownloadHttpSink from '!!raw-loader!../_embeds/cmds/cdk-hub-download-http-sink.bash';
11+
import CdkDeployHttpSinkGen from '!!raw-loader!../_embeds/cmds/cdk-deploy-start-http-sink-generic.bash';
612

713
# How to use the Hub
814

9-
The InfinyOn Hub serves as a centralized repository for InfinyOn tConnectors, SmartModules, and Dataflows. It facilitates the discovery and deployment of extensions to enhance the data streaming capabilities of Fluvio. The hub provides a platform for developers and data engineers to easily access pre-built solutions or share their own extensions.
15+
The InfinyOn Hub serves as a centralized repository for InfinyOn [Connectors](#connectors), [SmartModules](#smartmodules), and Dataflows. It facilitates the discovery and deployment of extensions to enhance Fluvio's data streaming capabilities. The hub provides developers and data engineers a platform to access pre-built solutions or share their extensions easily.
1016

1117

12-
## List
18+
## Connectors
1319

14-
List the SmartModules available for download.
20+
List the Connectors available for download. The official connectors (prepended with `infinyon/`) can run on InfinyOn Cloud and in your local environment.
1521

16-
```shell copy="fl"
17-
$ fluvio hub smartmodule list
18-
SMARTMODULE Visibility
19-
infinyon-labs/array-map-json@x.y.z public
20-
infinyon-labs/dedup-filter@x.y.z public
21-
infinyon-labs/json-formatter@x.y.z public
22-
infinyon-labs/key-gen-json@x.y.z public
23-
infinyon-labs/regex-map-json@x.y.z public
24-
infinyon-labs/regex-map@x.y.z public
25-
infinyon-labs/stars-forks-changes@x.y.z public
26-
infinyon/jolt@x.y.z public
27-
infinyon/json-sql@x.y.z public
28-
infinyon/regex-filter@x.y.z public
29-
```
22+
### List Connectors
3023

31-
List the Connectors available for download.
24+
Use `fluvio hub connector list` or `cdk hub list` to list the available connectors.
3225

33-
```shell copy="fl"
34-
$ fluvio hub connector list
35-
CONNECTOR Visibility
36-
infinyon-labs/graphite-sink@x.y.z public
37-
infinyon/duckdb-sink@x.y.z public
38-
infinyon/http-sink@x.y.z public
39-
infinyon/http-source@x.y.z public
40-
infinyon/ic-webhook-source@x.y.z public
41-
infinyon/kafka-sink@x.y.z public
42-
infinyon/kafka-source@x.y.z public
43-
infinyon/mqtt-source@x.y.z public
44-
infinyon/sql-sink@x.y.z public
45-
```
26+
<CodeBlock language="bash" copyBehavior="fl">$ {HubConnList}</CodeBlock>
4627

28+
Check out [Connectors for Developers] to learn how to build and publish your own connectors for the Hub.
4729

48-
## Download
30+
### Use Connectors
4931

50-
Downloading a SmartModule to a target cluster defined by your [`profile`]. SmartModules need to be downloaded before they are used in transformations.
32+
If you use a connector in InfinyOn Cloud, the Cloud infrastructure will automatically download and run the connector package. If you use connectors in your local environment, you must use the `cdk` command to download and run the connector package.
5133

52-
```shell copy="fl"
53-
$ fluvio hub sm download infinyon/jolt@x.y.z
54-
downloading infinyon/jolt@x.y.z to infinyon-jolt-x.y.z.ipkg
55-
... downloading complete
56-
... checking package
57-
trying connection to fluvio router.infinyon.cloud:9003
58-
... cluster smartmodule install complete
59-
```
34+
#### Download and Run Locally
35+
36+
Connectors are independent binaries that run outside of Fluvio. Use the `cdk` command to download and run connectors locally:
37+
38+
<CodeBlock language="bash" copyBehavior="fl">$ {CdkDownloadHttpSink}</CodeBlock>
39+
40+
The same connector configuration file can run the connector locally or in InfinyOn Cloud.
41+
42+
<CodeBlock language="bash" copyBehavior="fl">$ {CdkDeployHttpSinkGen}</CodeBlock>
43+
44+
To stop a connector, use the `cdk deploy shutdown` command.
45+
46+
47+
#### Deploy on InfinyOn Cloud
6048

61-
Running `fluvio hub conn download` will download the package containing source code of the Connector.
49+
Connectors are automatically downloaded to InfinyOn Cloud. The management of InfinyOn Cloud is performed using the `fluvio cloud` commands. To run a connector in InfinyOn Cloud, use the command `fluvio cloud connector create`.
6250

6351
```shell copy="fl"
64-
$ fluvio hub conn download infinyon/http-source@x.y.z
65-
downloading infinyon/http-source@x.y.z to infinyon-http-source-x.y.z.ipkg
66-
... downloading complete
52+
$ fluvio cloud connector create --config <CONFIG>
6753
```
6854

55+
**Note** List your profiles (`fluvio profile list`) to ensure your CLI points to the InfinyOn Cloud cluster.
6956

70-
[`profile`]: fluvio/cli/fluvio/profile.mdx
57+
58+
## SmartModules
59+
60+
SmartModules are custom-defined programs that connectors use to transform data records during processing. Similar to connectors, SmartModules can be published to the Hub and made available for download. However, unlike connectors, SmartModules are downloaded to the cluster and are not executed outside of Fluvio.
61+
62+
### List SmartModules
63+
64+
List the SmartModules available for download. The official SmartModules (prepended with `infinyon/`).
65+
66+
<CodeBlock language="bash" copyBehavior="fl">$ {HubSmList}</CodeBlock>
67+
68+
69+
### Download and Run SmartModules
70+
71+
Use `fluvio hub smartmodule download` to download specific SmartModules to a local or InfinyOn Cloud cluster. Use `fluvio profile list` to identify the target cluster.
72+
73+
<CodeBlock language="bash" copyBehavior="fl">$ {HubSmDownloadJolt}</CodeBlock>
74+
75+
SmartModules downloaded to a cluster can be used by the producer CLI or consumer CLI to transform records. They can also be accessed within the Connectors via the configuration file.
76+
77+
To see the SmartModules currently installed on the cluster, use the following command:
78+
79+
```shell copy="fl"
80+
$ fluvio smartmodule list
81+
```
82+
83+
### References
84+
- [Connector Developer Kit (cdk)]
85+
- [Connector Configuration File]
86+
- [SmartModule Developer Kit (smdk)]
87+
88+
[cdk deploy shutdown]: /connectors/developers/overview.mdx#shutdown-a-connector
89+
[`profile`]: fluvio/cli/fluvio/profile.mdx
90+
[Connectors for Developers]: /connectors/developers/overview.mdx
91+
[Connector Configuration File]: connectors/configuration.mdx
92+
[Connector Developer Kit (cdk)]: /connectors/developers/overview.mdx
93+
[SmartModule Developer Kit (smdk)]: smartmodules/developers/overview.mdx

0 commit comments

Comments
 (0)