|
| 1 | +# 🧙🏻♂️ Machine Learning Pipelines with Mage.ai and Hopsworks |
| 2 | + |
| 3 | +Mage.ai is an open-source platform for orchestrating data and machine learning pipelines. Hopsworks is a ML infrastructure platform. You can run your feature, training, and inference pipelines on Mage.ai and get the benefits of visualization, debugging, and so on. |
| 4 | + |
| 5 | + |
| 6 | +## ⬇️ Installation |
| 7 | + |
| 8 | + |
| 9 | +Create a new Python environment for this tutorial. |
| 10 | + |
| 11 | +Open the terminal and pass the next command: |
| 12 | + |
| 13 | +``` |
| 14 | +git clone https://github.com/logicalclocks/hopsworks-tutorials.git |
| 15 | +``` |
| 16 | + |
| 17 | +Go to the project folder. |
| 18 | + |
| 19 | +``` |
| 20 | +cd hopsworks-tutorials/integrations/mage_ai |
| 21 | +``` |
| 22 | + |
| 23 | +Install the mage-ai library: |
| 24 | + |
| 25 | +``` |
| 26 | +pip install mage-ai |
| 27 | +``` |
| 28 | + |
| 29 | +Install the requirements.txt |
| 30 | + |
| 31 | +``` |
| 32 | +pip install -r requirements.txt |
| 33 | +``` |
| 34 | + |
| 35 | +To launch the Mage tool run the next command: |
| 36 | + |
| 37 | +``` |
| 38 | +mage start mage_tutorial |
| 39 | +``` |
| 40 | + |
| 41 | +Open http://localhost:6789 in your browser. |
| 42 | + |
| 43 | + |
| 44 | +## ⚙️ API Key Setup |
| 45 | + |
| 46 | + |
| 47 | +Navigate to the Pipelines section and open the hopsworks_mage_quickstart pipeline. |
| 48 | +Here you can find a Pipeline Tree which shows you the block flow. |
| 49 | + |
| 50 | + |
| 51 | + |
| 52 | +Go to the Edit Pipeline section. |
| 53 | + |
| 54 | + |
| 55 | + |
| 56 | + |
| 57 | +The next step is to add the Hopsworks Api key to the secrets. Open the Secrets section and press the **+New** button. |
| 58 | + |
| 59 | + |
| 60 | + |
| 61 | + |
| 62 | +- *Secret Name: HOPSWORKS_API_KEY* |
| 63 | + |
| 64 | +- *Secret Value: {YOUR_HOPSWORKS_API_KEY}* |
| 65 | + |
| 66 | + |
| 67 | + |
| 68 | +Press Enter to save the secret. |
| 69 | + |
| 70 | + |
| 71 | + |
| 72 | +If you are running your own Hopsworks cluster, you will also have to add the following secrets, where the secret values should be the IP address or FQDN of your Hopsworks cluster and the name of the project in that Hopsworks cluster, respectively: |
| 73 | + |
| 74 | +- HOPSWORKS_HOST |
| 75 | + |
| 76 | +- HOPSWORKS_PROJECT |
| 77 | + |
| 78 | +Now you are ready to run Pipeline blocks. |
| 79 | + |
| 80 | + |
| 81 | +## 🚀 Launch |
| 82 | + |
| 83 | +If you want to run all blocks at once, press on the last block on the blocks tree. Then press **Execute with all upstream blocks**. |
| 84 | + |
| 85 | + |
| 86 | + |
| 87 | + |
| 88 | +If you want to run every block step by step, press the next button on the required block: |
| 89 | + |
| 90 | + |
| 91 | + |
| 92 | +At the bottom of a block you will see the block output. |
| 93 | + |
| 94 | + |
| 95 | + |
| 96 | + |
| 97 | +After running **transactions_fg** and **window_aggs_fg** blocks, you will see **transactions** and **transactions_4h_aggs** Feature Groups in the Hopsworks UI. |
| 98 | + |
| 99 | + |
| 100 | + |
| 101 | + |
| 102 | + |
| 103 | +The **model_building** block trains and saves the trained model into the Hopsworks Model Registry. |
| 104 | + |
| 105 | + |
| 106 | + |
| 107 | +After running the **model_deployment** block, you can find your deployed model in the Hopsworks Deployments. |
| 108 | + |
| 109 | + |
| 110 | + |
| 111 | +The **deployment_inference** block shows how to make online inference using your deployed model. |
| 112 | + |
| 113 | + |
0 commit comments