Skip to content

Commit 36c3834

Browse files
authored
chore: start whats'new (#287)
Updates for new release --------- Co-authored-by: Luis Moreno <morenol@users.noreply.github.com>
1 parent 06876b4 commit 36c3834

File tree

9 files changed

+47
-373
lines changed

9 files changed

+47
-373
lines changed

sdf/_embeds/install-sdf.bash

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
fvm install sdf-beta3
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
fn augment_count(word: String) -> Result<String> {
2+
Ok(format!("{}({})", word, word.chars().count()))
3+
}
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
fn sentence_to_words(sentence: String) -> Result<Vec<String>> {
2+
Ok(sentence.split_whitespace().map(String::from).collect())
3+
}

sdf/cli/index.mdx

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,9 @@ description: Stateful Dataflows Command Line Interface.
44
sidebar_position: 10
55
---
66

7+
import CodeBlock from '@theme/CodeBlock';
8+
import InstallFvm from '!!raw-loader!../_embeds/install-sdf.bash';
9+
710
Stateful Dataflows (sdf) is a binary, shipped with fluvio, that helps developers build, test, and deploy packages and dataflows.
811

912
### Install SDF
@@ -16,9 +19,7 @@ $ curl -fsS https://hub.infinyon.cloud/install/install.sh | bash
1619

1720
2. Install the preview release:
1821

19-
```bash copy="fl"
20-
$ fvm install sdf-beta2
21-
```
22+
<CodeBlock language="bash">{InstallFvm}</CodeBlock>
2223

2324
### SDF Commands
2425

sdf/composition/overview.mdx

Lines changed: 3 additions & 293 deletions
Original file line numberDiff line numberDiff line change
@@ -91,172 +91,10 @@ The sections are as follows:
9191
* `functions` - defines functions in the package
9292
* `dev` - defines substitutions used during development and test
9393

94-
95-
#### Build and Test a Package
96-
97-
We'll use the [SDF] command line tool with the package definition file to generate the WebAssembly glue code and the placeholder for the custom logic.
98-
99-
```bash copy="fl"
100-
$ sdf -h
101-
Stateful Dataflow Command Line Interface
102-
103-
Usage: sdf <COMMAND>
104-
105-
Commands:
106-
clean Clean generated artifacts such as state and internal objects
107-
build Build package (requires: sdf-package.yaml)
108-
generate Generate package (requires: sdf-package.yaml)
109-
update Update package (requires: sdf-package.yaml)
110-
test Test command shell (requires sdf-package.yaml)
111-
run Run dataflow (requires: dataflow.yaml)
112-
setup Setup pre-requisites for Stateful Dataflows
113-
version Prints binary version
114-
log Print dataflow logs
115-
```
116-
117-
Let's start with an example. Create a package that split sentences into words and counts the number of characters in each word.
118-
119-
##### 1. Create the Package file
120-
121-
Create a fresh project directory `split-sentence` with two subdirectories: `packages` and `sentence`:
122-
123-
```bash
124-
$ mkdir -p split-sentence/packages/sentence
125-
$ cd split-sentence/packages/sentence
126-
```
127-
128-
Inside the `sentence` directory and create the `sdf-package.yaml` and add the following content:
129-
130-
```yaml
131-
#sdf-package.yaml
132-
apiVersion: 0.5.0
133-
134-
meta:
135-
name: sentence-pkg
136-
version: 0.1.0
137-
namespace: example
138-
139-
functions:
140-
141-
sentence-to-words:
142-
operator: flat-map
143-
inputs:
144-
- name: sentence
145-
type: string
146-
output:
147-
type: string
148-
149-
augment-count:
150-
operator: map
151-
inputs:
152-
- name: word
153-
type: string
154-
output:
155-
type: string
156-
157-
dev:
158-
converter: raw
159-
```
160-
161-
##### 2. Generate the Package Project
162-
163-
Use SDF generate command to build the project:
164-
165-
```bash copy="fl"
166-
$ sdf generate
167-
```
168-
169-
The generator created several directories and files that we'll edit next.
170-
171-
172-
##### 3. Add the Custom Code
173-
174-
First, let's update the first function, `sentence-to-words`. Open `rust/sentence-to-words/src/lib.rs` and update the function body with the following code:
175-
176-
```rust
177-
pub(crate) fn sentence_to_words(sentence: String) -> Result<Vec<String>> {
178-
Ok(sentence.split_whitespace().map(String::from).collect())
179-
}
180-
```
181-
182-
Next update `augment_count`. Open `rust/augment-count/src/lib.rs` and replace the function body:
183-
184-
```rust
185-
pub(crate) fn augment_count(word: String) -> Result<String> {
186-
Ok(format!("{}({})", word, word.chars().count()))
187-
}
188-
```
189-
190-
Let's add some tests as well:
191-
192-
```rust
193-
#[cfg(test)]
194-
mod test {
195-
use super::*;
196-
197-
#[test]
198-
fn test_augment_count() {
199-
let input = "Hello".to_string();
200-
let output = augment_count(input);
201-
assert_eq!(output.unwrap(), "Hello(5)");
202-
}
203-
}
204-
```
205-
206-
We've implemented both functions; it's time to compile and test our work.
207-
208-
209-
##### 4. Build and Test the Package
210-
211-
To build the package, run:
212-
213-
```bash
214-
$ sdf build
215-
```
216-
217-
Use `sdf test` interactive shell to test the code:
218-
219-
```bash
220-
$ sdf test
221-
```
222-
223-
In the test shell, you can view the functions available for testing:
224-
225-
```bash copy="fl"
226-
>> show functions
227-
sentence-to-words
228-
augment-count
229-
```
230-
231-
Let's test `sentence-to-words` first:
232-
233-
```bash copy="fl"
234-
>> test function sentence-to-words --input "Hello World"
235-
Hello
236-
World
237-
```
238-
239-
Next, test `augment-count`:
240-
241-
```bash copy="fl"
242-
>> test function augment-count --input "Hello"
243-
Hello(5)
244-
```
245-
246-
You may also test the `rust` code via Cargo:
247-
248-
```bash
249-
$ cd rust
250-
$ cargo test
251-
```
252-
253-
The tests passed, and the package is now ready to use in the dataflow file.
254-
25594
## The Dataflow
25695

25796
The **dataflow** imports functions, types, and states from one more package. Packages may also import components from others; however, dataflow maintains the final composition.
25897

259-
26098
#### `dataflow.yaml`
26199

262100
The [SDF] command line tool uses the dataflow definition file `dataflow.yaml` to assemble the data application, and it has the following hierarchy:
@@ -326,144 +164,16 @@ The sections are as follows:
326164

327165
The dataflow file has other variants such as `window` and `partitions`, which are omitted for simplicity. For additional details check the [Dataflow file] section.
328166

167+
# Getting started
329168

330-
#### Create a Dataflow
331-
332-
We'll import the package built in the [previous](#build-and-test-a-package) section to create the `split-sentence` dataflow.
333-
334-
##### 1. Create the Dataflow file
335-
336-
Create a dataflow file in the directory `split-sentence` directory:
337-
338-
```bash
339-
$ cd split-sentence
340-
```
341-
342-
Create the `dataflow.yaml` and add the following content:
343-
344-
```yaml name="dataflow.yaml"
345-
apiVersion: 0.5.0
346-
347-
meta:
348-
name: split-sentence
349-
version: 0.1.0
350-
namespace: example
351-
352-
imports:
353-
- pkg: example/sentence-pkg@0.1.0
354-
functions:
355-
- name: sentence-to-words
356-
- name: augment-count
357-
358-
config:
359-
converter: raw
360-
361-
topics:
362-
sentence:
363-
schema:
364-
value:
365-
type: string
366-
converter: raw
367-
words:
368-
schema:
369-
value:
370-
type: string
371-
converter: raw
372-
373-
services:
374-
sentence-words:
375-
sources:
376-
- type: topic
377-
id: sentence
378-
379-
transforms:
380-
- operator: flat-map
381-
uses: sentence-to-words
382-
- operator: map
383-
uses: augment-count
384-
385-
sinks:
386-
- type: topic
387-
id: words
388-
389-
dev:
390-
imports:
391-
- pkg: example/sentence-pkg@0.1.0
392-
path: ./packages/sentence
393-
```
394-
395-
##### 2. Run the Dataflow
396-
397-
Use `sdf` command line tool to run the dataflow:
398-
399-
```bash copy="fl"
400-
$ sdf run --dev
401-
```
402-
403-
Use `--dev` to ask the engine to change the path to the local package. Without this flag, the engine will look for the package in `InfinyOn Hub`.
404-
405-
406-
##### 3. Test the Dataflow
407-
408-
1. Produce sentences to in `sentence` topic:
409-
410-
```bash copy="fl"
411-
$ fluvio produce sentence
412-
```
413-
414-
```bash
415-
Hello world
416-
Hi there
417-
```
418-
419-
Consume from `words` to retrieve the result:
420-
421-
```bash
422-
$ fluvio consume words -Bd
423-
```
169+
See [next section] for a quickstart.
424170

425-
```bash
426-
Hello(5)
427-
world(5)
428-
Hi(2)
429-
there(5)
430-
```
431-
432-
##### 4. Show State
433-
434-
The dataflow collects runtime metrics that you can inspect in the runtime terminal.
435-
436-
Check the `sentence-to-words` counters:
437-
438-
```bash copy="fl"
439-
>> show state sentence-words/sentence-to-words/metrics --table
440-
Key Window succeeded failed
441-
stats * 2 0
442-
```
443-
444-
Check the `augment-count` counters:
445-
446-
```bash copy="fl"
447-
>> show state sentence-words/augment-count/metrics --table
448-
Key Window succeeded failed
449-
stats * 4 0
450-
```
451-
452-
Congratulations! You've successfully built and run a composable dataflow! The project is available for download in [github].
453-
454-
##### 5. Clean-up
455-
456-
Exit the `sdf` terminal and remove the topics:
457-
458-
```bash
459-
fluvio topic delete sentence
460-
fluvio topic delete words
461-
```
462171

463172
### References
464173

465174
* [Example Workflows in github]
466175

176+
[next section]: quickstart.mdx
467177
[Inline Dataflows]: sdf/quickstart.mdx
468178
[Dataflow file]: sdf/concepts/dataflow-yaml.mdx
469179
[SDF]: sdf/cli/index.mdx

0 commit comments

Comments
 (0)