You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
test Test command shell (requires sdf-package.yaml)
111
-
run Run dataflow (requires: dataflow.yaml)
112
-
setup Setup pre-requisites for Stateful Dataflows
113
-
version Prints binary version
114
-
log Print dataflow logs
115
-
```
116
-
117
-
Let's start with an example. Create a package that split sentences into words and counts the number of characters in each word.
118
-
119
-
##### 1. Create the Package file
120
-
121
-
Create a fresh project directory `split-sentence` with two subdirectories: `packages` and `sentence`:
122
-
123
-
```bash
124
-
$ mkdir -p split-sentence/packages/sentence
125
-
$ cd split-sentence/packages/sentence
126
-
```
127
-
128
-
Inside the `sentence` directory and create the `sdf-package.yaml` and add the following content:
129
-
130
-
```yaml
131
-
#sdf-package.yaml
132
-
apiVersion: 0.5.0
133
-
134
-
meta:
135
-
name: sentence-pkg
136
-
version: 0.1.0
137
-
namespace: example
138
-
139
-
functions:
140
-
141
-
sentence-to-words:
142
-
operator: flat-map
143
-
inputs:
144
-
- name: sentence
145
-
type: string
146
-
output:
147
-
type: string
148
-
149
-
augment-count:
150
-
operator: map
151
-
inputs:
152
-
- name: word
153
-
type: string
154
-
output:
155
-
type: string
156
-
157
-
dev:
158
-
converter: raw
159
-
```
160
-
161
-
##### 2. Generate the Package Project
162
-
163
-
Use SDF generate command to build the project:
164
-
165
-
```bash copy="fl"
166
-
$ sdf generate
167
-
```
168
-
169
-
The generator created several directories and files that we'll edit next.
170
-
171
-
172
-
##### 3. Add the Custom Code
173
-
174
-
First, let's update the first function, `sentence-to-words`. Open `rust/sentence-to-words/src/lib.rs` and update the function body with the following code:
The tests passed, and the package is now ready to use in the dataflow file.
254
-
255
94
## The Dataflow
256
95
257
96
The **dataflow** imports functions, types, and states from one more package. Packages may also import components from others; however, dataflow maintains the final composition.
258
97
259
-
260
98
#### `dataflow.yaml`
261
99
262
100
The [SDF] command line tool uses the dataflow definition file `dataflow.yaml` to assemble the data application, and it has the following hierarchy:
@@ -326,144 +164,16 @@ The sections are as follows:
326
164
327
165
The dataflow file has other variants such as `window` and `partitions`, which are omitted for simplicity. For additional details check the [Dataflow file] section.
328
166
167
+
# Getting started
329
168
330
-
#### Create a Dataflow
331
-
332
-
We'll import the package built in the [previous](#build-and-test-a-package) section to create the `split-sentence` dataflow.
333
-
334
-
##### 1. Create the Dataflow file
335
-
336
-
Create a dataflow file in the directory `split-sentence` directory:
337
-
338
-
```bash
339
-
$ cd split-sentence
340
-
```
341
-
342
-
Create the `dataflow.yaml` and add the following content:
343
-
344
-
```yaml name="dataflow.yaml"
345
-
apiVersion: 0.5.0
346
-
347
-
meta:
348
-
name: split-sentence
349
-
version: 0.1.0
350
-
namespace: example
351
-
352
-
imports:
353
-
- pkg: example/sentence-pkg@0.1.0
354
-
functions:
355
-
- name: sentence-to-words
356
-
- name: augment-count
357
-
358
-
config:
359
-
converter: raw
360
-
361
-
topics:
362
-
sentence:
363
-
schema:
364
-
value:
365
-
type: string
366
-
converter: raw
367
-
words:
368
-
schema:
369
-
value:
370
-
type: string
371
-
converter: raw
372
-
373
-
services:
374
-
sentence-words:
375
-
sources:
376
-
- type: topic
377
-
id: sentence
378
-
379
-
transforms:
380
-
- operator: flat-map
381
-
uses: sentence-to-words
382
-
- operator: map
383
-
uses: augment-count
384
-
385
-
sinks:
386
-
- type: topic
387
-
id: words
388
-
389
-
dev:
390
-
imports:
391
-
- pkg: example/sentence-pkg@0.1.0
392
-
path: ./packages/sentence
393
-
```
394
-
395
-
##### 2. Run the Dataflow
396
-
397
-
Use `sdf` command line tool to run the dataflow:
398
-
399
-
```bash copy="fl"
400
-
$ sdf run --dev
401
-
```
402
-
403
-
Use `--dev` to ask the engine to change the path to the local package. Without this flag, the engine will look for the package in `InfinyOn Hub`.
404
-
405
-
406
-
##### 3. Test the Dataflow
407
-
408
-
1. Produce sentences to in `sentence` topic:
409
-
410
-
```bash copy="fl"
411
-
$ fluvio produce sentence
412
-
```
413
-
414
-
```bash
415
-
Hello world
416
-
Hi there
417
-
```
418
-
419
-
Consume from `words` to retrieve the result:
420
-
421
-
```bash
422
-
$ fluvio consume words -Bd
423
-
```
169
+
See [next section] for a quickstart.
424
170
425
-
```bash
426
-
Hello(5)
427
-
world(5)
428
-
Hi(2)
429
-
there(5)
430
-
```
431
-
432
-
##### 4. Show State
433
-
434
-
The dataflow collects runtime metrics that you can inspect in the runtime terminal.
435
-
436
-
Check the `sentence-to-words` counters:
437
-
438
-
```bash copy="fl"
439
-
>> show state sentence-words/sentence-to-words/metrics --table
440
-
Key Window succeeded failed
441
-
stats * 2 0
442
-
```
443
-
444
-
Check the `augment-count` counters:
445
-
446
-
```bash copy="fl"
447
-
>> show state sentence-words/augment-count/metrics --table
448
-
Key Window succeeded failed
449
-
stats * 4 0
450
-
```
451
-
452
-
Congratulations! You've successfully built and run a composable dataflow! The project is available for download in [github].
0 commit comments