Merge pull request #10 from scramjetorg/fix/make-all-samples-bundle

Make scripts for all samples

Author: MichalCz

.github/workflows/node.js.yml

@@ -0,0 +1,28 @@
+name: Make a refapps release with compiled package archives
+
+on: pull_request
+
+jobs:
+  build-only:
+    name: Build refapps
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Setup Nodejs v16
+        uses: actions/setup-node@v2
+        with:
+          node-version: 16
+
+      - name: Install dependencies
+        run: npm install
+
+      - name: Build sample sequences
+        run: npm run build
+
+      - name: Pack sample sequences
+        run: npm run pack
+
+      - name: List packages
+        run: ls -la dist/

.github/workflows/refapps-build.yml

@@ -19,37 +19,17 @@ jobs:
           node-version: 16
 
       - name: Install dependencies
-        run: yarn install
+        run: npm install
 
       - name: Build reference apps
-        run: yarn build:refapps
-
-      - name: Prefix python packages to prevent name clash
-        run: |
-          cd python;
-          for pkg in *.tar.gz; do
-            mv $pkg python-$pkg;
-          done
-      
-      - name: Prefix typescript packages to prevent name clash
-        run: |
-          cd typescript;
-          for pkg in *.tar.gz; do
-            mv $pkg typescript-$pkg;
-          done
-      
-      - name: Prefix javascript packages to prevent name clash
-        run: |
-          cd javascript;
-          for pkg in *.tar.gz; do
-            mv $pkg javascript-$pkg;
-          done
+        run: npm run build
+
+      - name: Build reference apps
+        run: npm run pack
 
       - name: Create a release and upload artifacts
         run: >
           gh release create ${{ github.ref }}
-          javascript/*.tar.gz
-          typescript/*.tar.gz
-          python/*.tar.gz
+          dist/*.tar.gz
         env:
           GITHUB_TOKEN: ${{ secrets.SCRAMJET_BOT_TOKEN }}

.github/workflows/refapps-release.yml

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2021 scramjet.org
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

LICENSE

@@ -0,0 +1,49 @@
+# Samples
+
+Try out our samples prepared some samples for you. We differentiated them according to the language in which they were written.
+
+Every sample contains a short readme with a guidance describing the procedure for running the example. The execution will be performed from the command line using our CLI, which full description and documentation you will find [here](https://docs.scramjet.org/platform/cli-reference).
+
+## Python samples
+
+Python packages work in a similar way to all the others. They need to pack the dependencies and carry over anything they may need to run the program. They also need a `package.json` file.
+
+- [kafka-transformer](python/kafka-transformer/) - Sequence sending data to any Kafka topic
+- [kafka-consumer](python/kafka-consumer/) - Sequence fetching data from any Kafka topic
+- [voice-recognition](python/voice-recognition) - a sample Sequence that performs voice recognition in real time.
+
+## JavaScript samples
+
+JavaScript packages are pretty straight forward in use. After the dependency installation is performed inside the Sequence package it is ready to be compressed and sent to STH.
+
+- [hello](javascript/hello) - Sequence that modifies incoming stream of strings by saying "Hello".
+- [hello-snowman](javascript/hello-snowman) - Sequence that reads incoming stream, and modifies it by adding a text message according to the incoming data.
+- [simple-counter-js](javascript/simple-counter-js) - Sequence, that counts to 1000, and logs the number in one-second intervals.
+- [test-output](javascript/test-output) - Sequence that simply writes random values to the output stream.
+- [json-url-stdout](javascript/json-url-stdout/) - Make a real-time JSON API scraper (stdout version)
+- [json-url-output](javascript/json-url-output/) - Make a real-time JSON API scraper (output version)
+
+## TypeScript samples
+
+TypeScript compiles to JavaScript. It means that TypeScript packages except dependency installation also need to be compiled. We added a `build` and `postbuild` scripts to every `package.json` in TypeScript packages, which are responsible for compiling files into a `dist` folder and coping `package.json` file into the `dist` folder. In effect, it is the `dist` folder, that becomes the Sequence package, ready to be compressed and sent to STH.
+
+- [crypto-prices](typescript/crypto-prices) - Sequence that keeps printing current crypto prices for a provided pair of currencies every 3s.
+- [slack-write](typescript/slack-write/) - Write incoming data to a slack channel
+- [slack-read](typescript/slack-read/) - Read data from Slack
+- [discord-write](typescript/discord-write/) - Write to Discord
+- [discord-read](typescript/discord-read/) - Read from Discord
+- [mediawiki](typescript/mediawiki) - Sequence that keeps printing mediawiki event stream.
+- [rss](typescript/rss) - Sequence that gets a list of RSS and then retrieves each feed and passes links to scraper.
+- [write-to-database](typescript/write-to-database/) - Write incoming data to a database table
+- [scraping](typescript/scraping) - Sequence that scrapes web pages.
+- [stack-overflow](typescript/stack-overflow) - Sequence that gets a number of changes in Stack Overflow tag count.
+- [transform-string-stream](typescript/transform-string-stream) - Sequence that modifies incoming stream of strings by adding a prefix and a suffix.
+- [read-targz-stats](typescript/read-targz-stats/) - Stream through tar.gz file and output stats
+- [send-to-github](typescript/send-to-github/) - Write incoming data to a file on github
+- [linkedin](typescript/linkedin/) - Enrich csv data with Linkedin via RapidAPI
+- [hexdump](typescript/hexdump/) -Sequence returns input in hex format.
+
+## Guides and complex examples
+
+- [kafka-setup](guides/kafka-setup/) - how to set up a kafka instance and consume data from it with Scramjet Cloud Platform
+- [discord-slack-connection](guides/discord-slack-connection/) - how to interconnect discord and slack with Scramjet Cloud Platform

README.md

@@ -0,0 +1,28 @@
+# Discord to Slack (and vice versa) Connection
+
+___
+
+### The video that illustrates the execution of the sample is on our [YouTube](https://www.youtube.com/channel/UChgTmKeuAsKj8kDnylkmP6Q) channel [How to Connect Discord and Slack in 4 minutes](https://www.youtube.com/watch?v=y2L1f-f-wLM&t=1s).
+
+___
+
+> 💡 **Please note that the sample below requires some previous installations before you start running it, you will find them [here](../../README.md#3-install-scramjet-transform-hub).**
+
+In this project you will find 4 STH Sequences:
+
+* [discord-read](../../typescript/discord-read/)
+* [discord-write](../../typescript/discord-write/)
+* [slack-read](../../typescript/slack-read/)
+* [slack-write](../../typescript/slack-write/)
+
+You must run at least two (discord-read, slack write or discord-write, slack-read) in order to get one directional communication or all four for bi-directional communication.
+
+Each Sequence either reads or writes to specified application respectively and uses `const TOPIC = ...` topic to exchange data between applications. You need two topics for bi-directional communication.
+
+You can add your own topics at invoke time 
+
+## TODO
+
+As there is no way to map message IDs between the two applications, threads are not supported.
+
+In order to get this working, Sequences must keep a track of posted and written IDs.

guides/discord-slack-connection/README.md

@@ -1,13 +1,13 @@
 # Sequences fetching and sending data from any Kafka server
 
-Easy example of usage Kaffka with Scramjet platform. As the part of this PR, three different directories are created for organizational purposes. You need also 3 different terminals to check the whole.
+Easy example of usage Kafka with Scramjet platform. As the part of this PR, three different directories are created for organizational purposes. You need also 3 different terminals to check the whole.
 
 On first terminal:
 
-setup kaffka in docker container
+setup kafka in docker container
 
 ```bash
-cd kaffka-setup
+cd docs/kafka-setup
 docker-compose up -d
 ```
 
@@ -17,34 +17,34 @@ create 'scramjet' topic
 docker-compose exec kafka kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic scramjet
 ```
 
-run watcher for 'scramjet' topic, you will see here logs directly from kaffka
+run watcher for 'scramjet' topic, you will see here logs directly from kafka
 
 ```bash
 docker-compose exec kafka kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic scramjet --from-beginning
 ```
 
-On second terminal run STH
+On a second terminal run STH
 
 ```bash
 DEVELOPMENT=1 sth --runtime-adapter=process
 ```
 
-On last terminal you need to make few things. First you need to build our two samples. You can run command visible below in main directory of platform-samples project to build all samples, or you can run it directly from the kaffka-transformer and kaffka-consumer directories.
+On another terminal you need to make few things. First you need to build our two samples. You can run command visible below in main directory of platform-samples project to build all samples, or you can run it directly from the kafka-transformer and kafka-consumer directories.
 
 ```bash
-yarn build:refapps
+npm run build
 ```
 
 Once you've built these packages, you need to run consumer Sequence first. Please note that we are passing two arguments to Sequence, topic name and server name.
 
 ```bash
-si seq deploy kaffka-consumer/dist --args [{\"topic\":\"scramjet\"},{\"server\":\"0.0.0.0:29092\"}]
+si seq deploy python/kafka-consumer/dist --args [{\"topic\":\"scramjet\"},{\"server\":\"0.0.0.0:29092\"}]
 ```
 
 Now you can deploy transform Sequence, with exactly same arguments:
 
 ```bash
-si seq deploy kaffka-transformer/dist --args [{\"topic\":\"scramjet\"},{\"server\":\"0.0.0.0:29092\"}]
+si seq deploy python/kafka-transformer/dist --args [{\"topic\":\"scramjet\"},{\"server\":\"0.0.0.0:29092\"}]
 ```
 
 You should be able to list both instances with command:
@@ -53,13 +53,13 @@ You should be able to list both instances with command:
 si inst ls`
 ```
 
-Last step is to start sending some data to transformer Sequence, which send it to Kaffka topic.
+Last step is to start sending some data to transformer Sequence, which send it to Kafka topic.
 
 ```bash
 si inst input -
 ```
 
-Type anything you want to blinking terminal, you should then see same data in first terminal, which shows logs from kaffka directly, but also you should be able to see information on STH terminal. It should look similar to:
+Type anything you want to blinking terminal, you should then see same data in first terminal, which shows logs from kafka directly, but also you should be able to see information on STH terminal. It should look similar to:
 
 ```bash
 Topic name=scramjet, Message=b'test message'

python/kaffka-setup/README.md renamed to guides/kafka-setup/README.md

@@ -0,0 +1,97 @@
+# hello-snowman ⛄
+
+Sequence that reads incoming stream (input), and and modifies it by adding a text message according to the incoming data.
+
+Stream is generated in [stream-gen.js](../tools/stream-gen-tool/stream-gen.js) file, where numbers in range of <-50,50> are randomly chosen and sent as Celsius degrees to `hello-snowman` Instance API endpoint `/input`.
+
+Our `hello-snowman` app will read and interpret these Celsius degrees, and will inform us about state of our Snowman:
+
+- if temperature will be 0 or below, Sequence will return message:
+
+```bash
+Snowman ⛄ is freezing 🥶 Winter is coming ❄️ ❄️ ❄️ ❄️ ❄️
+```
+
+- in the other case (temperature above 0 degrees), Sequence will return message:
+
+```bash
+ Snowman ⛄ is melting! 🥵
+```
+
+> 💡 **Please note that the sample below requires some previous installations before you start running it, you will find them [here](../../README.md#3-install-scramjet-transform-hub).**
+
+## Running
+
+Open three terminals and run the following commands:
+
+**The first terminal:**
+
+```bash
+# start sth
+scramjet-transform-hub
+```
+
+**The second terminal**
+
+```bash
+# go to 'hello-snowman' directory
+cd samples/hello-snowman
+
+# install dependencies
+npm install
+
+# go back to samples/ directory
+cd ../
+
+# deploy 'hello-snowman' Sequence
+si seq deploy hello-snowman
+
+# see the Instance output
+si inst output -    # nothing happens until some is sent to input
+```
+
+> 💡**NOTE:** Command `deploy` performs three actions at once: `pack`, `send` and `start` the Sequence. It is the same as if you would run those three commands separately:
+
+```bash
+si seq pack . -o hello-snowman.tar.gz    # compress 'hello-snowman/' directory into file named 'hello-snowman.tar.gz'
+
+si seq send hello-snowman.tar.gz    # send compressed Sequence to STH, this will output Sequence ID
+
+si seq start -    # start the Sequence, this will output Instance ID
+```
+
+**In the third terminal**
+
+```bash
+# Start stream generator tool with Instance ID as parameter
+node ./tools/stream-gen-tool/stream-gen.js <instance_id>
+```
+
+## Output
+
+Now you should see the output in the console:
+
+```js
+$ si inst output -
+----------------------------------------
+Message# 1 | Temperature measure
+INPUT | -16
+OUTPUT| Snowman ⛄ is freezing 🥶 Winter is coming ❄️ ❄️ ❄️ ❄️ ❄️
+
+----------------------------------------
+Message# 2 | Temperature measure
+INPUT | 49
+OUTPUT| Snowman ⛄ is melting! 🥵
+
+----------------------------------------
+Message# 3 | Temperature measure
+INPUT | 16
+OUTPUT| Snowman ⛄ is melting! 🥵
+
+----------------------------------------
+Message# 4 | Temperature measure
+INPUT | -46
+OUTPUT| Snowman ⛄ is freezing 🥶 Winter is coming ❄️ ❄️ ❄️ ❄️ ❄️
+
+----------------------------------------
+```

python/kaffka-setup/docker-compose.yml renamed to guides/kafka-setup/docker-compose.yml

@@ -0,0 +1,20 @@
+const { PassThrough } = require("stream");
+
+module.exports = function(input) {
+    // create output stream
+    const out = new PassThrough({encoding: 'utf-8'});
+
+    // do something for each number in input stream
+    input.on("data", data => {
+        let outMessage = "";
+        if (data > 0) {
+            outMessage = "Snowman ⛄ is melting! 🥵" + "\n";
+        } else {
+            outMessage = "Snowman ⛄ is freezing 🥶 Winter is coming ❄️ ❄️ ❄️ ❄️ ❄️" + "\n";
+        }
+        // write to output stream
+        out.write(outMessage)
+    });
+    // return output stream so it can be consumed (e.g. by CLI client)
+    return out;
+};