Skip to content

Commit 248608e

Browse files
authored
[FSTORE-1231] README for the MageAI tutorial (#233)
* Update README.md
1 parent ed0c662 commit 248608e

16 files changed

+114
-0
lines changed

README.md

+1
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,7 @@ In order to understand the tutorials you need to be familiar with general concep
6666
- [Bytewax](https://github.com/logicalclocks/hopsworks-tutorials/tree/master/integrations/bytewax): Real time feature computation using Bytewax.
6767
- [Apache Beam](https://github.com/logicalclocks/hopsworks-tutorials/tree/master/integrations/java/beam): Real time feature computation using Apache Beam, Google Cloud Dataflow and Hopsworks Feature Store.
6868
- [Apache Flink](https://github.com/logicalclocks/hopsworks-tutorials/tree/master/integrations/java/flink): Real time feature computation using Apache Flink and Hopsworks Feature Store.
69+
- [MageAI](https://github.com/logicalclocks/hopsworks-tutorials/tree/master/integrations/mage_ai): Build and operate a ML system with Mage and Hopsworks.
6970

7071

7172
## 📝 Feedbacks & Comments:

integrations/mage_ai/README.md

+113
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
# 🧙🏻‍♂️ Machine Learning Pipelines with Mage.ai and Hopsworks
2+
3+
Mage.ai is an open-source platform for orchestrating data and machine learning pipelines. Hopsworks is a ML infrastructure platform. You can run your feature, training, and inference pipelines on Mage.ai and get the benefits of visualization, debugging, and so on.
4+
5+
6+
## ⬇️ Installation
7+
8+
9+
Create a new Python environment for this tutorial.
10+
11+
Open the terminal and pass the next command:
12+
13+
```
14+
git clone https://github.com/logicalclocks/hopsworks-tutorials.git
15+
```
16+
17+
Go to the project folder.
18+
19+
```
20+
cd hopsworks-tutorials/integrations/mage_ai
21+
```
22+
23+
Install the mage-ai library:
24+
25+
```
26+
pip install mage-ai
27+
```
28+
29+
Install the requirements.txt
30+
31+
```
32+
pip install -r requirements.txt
33+
```
34+
35+
To launch the Mage tool run the next command:
36+
37+
```
38+
mage start mage_tutorial
39+
```
40+
41+
Open http://localhost:6789 in your browser.
42+
43+
44+
## ⚙️ API Key Setup
45+
46+
47+
Navigate to the Pipelines section and open the hopsworks_mage_quickstart pipeline.
48+
Here you can find a Pipeline Tree which shows you the block flow.
49+
50+
![tree](images/tree.png)
51+
52+
Go to the Edit Pipeline section.
53+
54+
![edit](images/edit.png)
55+
56+
57+
The next step is to add the Hopsworks Api key to the secrets. Open the Secrets section and press the **+New** button.
58+
59+
![secrets](images/secrets.png)
60+
61+
62+
- *Secret Name: HOPSWORKS_API_KEY*
63+
64+
- *Secret Value: {YOUR_HOPSWORKS_API_KEY}*
65+
66+
![secret](images/secret.png)
67+
68+
Press Enter to save the secret.
69+
70+
![ready_secret](images/ready_secret.png)
71+
72+
If you are running your own Hopsworks cluster, you will also have to add the following secrets, where the secret values should be the IP address or FQDN of your Hopsworks cluster and the name of the project in that Hopsworks cluster, respectively:
73+
74+
- HOPSWORKS_HOST
75+
76+
- HOPSWORKS_PROJECT
77+
78+
Now you are ready to run Pipeline blocks.
79+
80+
81+
## 🚀 Launch
82+
83+
If you want to run all blocks at once, press on the last block on the blocks tree. Then press **Execute with all upstream blocks**.
84+
85+
![execute](images/execute.png)
86+
87+
88+
If you want to run every block step by step, press the next button on the required block:
89+
90+
![run](images/run.png)
91+
92+
At the bottom of a block you will see the block output.
93+
94+
![output](images/output.png)
95+
96+
97+
After running **transactions_fg** and **window_aggs_fg** blocks, you will see **transactions** and **transactions_4h_aggs** Feature Groups in the Hopsworks UI.
98+
99+
![fg](images/fg.png)
100+
101+
![fg_data](images/fg_data.png)
102+
103+
The **model_building** block trains and saves the trained model into the Hopsworks Model Registry.
104+
105+
![model_registry](images/model_registry.png)
106+
107+
After running the **model_deployment** block, you can find your deployed model in the Hopsworks Deployments.
108+
109+
![deployment](images/deployment.png)
110+
111+
The **deployment_inference** block shows how to make online inference using your deployed model.
112+
113+
![inference](images/inference.png)
537 KB
Loading
240 KB
Loading

integrations/mage_ai/images/edit.png

326 KB
Loading
537 KB
Loading

integrations/mage_ai/images/fg.png

331 KB
Loading
541 KB
Loading
125 KB
Loading
237 KB
Loading
181 KB
Loading
99.3 KB
Loading

integrations/mage_ai/images/run.png

287 KB
Loading
111 KB
Loading
461 KB
Loading

integrations/mage_ai/images/tree.png

136 KB
Loading

0 commit comments

Comments
 (0)