Skip to content

Commit cea212d

Browse files
committed
update to python guides
1 parent 7318007 commit cea212d

13 files changed

+55
-25
lines changed
40.9 KB
Loading
32.3 KB
Loading
Loading
51.1 KB
Loading
1.18 MB
Loading
Loading
Loading
Loading

docs/user_guides/projects/python/custom_commands.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -23,11 +23,11 @@ To use the UI, navigate to the Python environment in the Project settings. In th
2323
</p>
2424

2525
## Code
26-
You can also run the custom commands using the REST API. From the REST API, you should provide the path, in HOPSFS, to the bash script and the artifacts(comma separated string of paths in HopsFs). The REST API endpoint for running custom commands is: `hopsworks-api/api/project/<projectId>/python/environments/<pythonVersion>/commands/custom` and the body should look like this:
26+
You can also run the custom commands using the REST API. From the REST API, you should provide the path, in HOPSFS, to the bash script and the artifacts(comma separated string of paths in HopsFs). The REST API endpoint for running custom commands is: `hopsworks-api/api/project/<projectId>/python/environments/<environmentName>/commands/custom` and the body should look like this:
2727
```python
2828
{
2929
"commandsFile": "<pathToYourBashScriptInHopsFS>",
30-
"artifacts": "<commaSeperatedListOfPathsToArtifactsInHopsFS>"
30+
"artifacts": "<commaSeparatedListOfPathsToArtifactsInHopsFS>"
3131
}
3232

3333
```

docs/user_guides/projects/python/python_env_clone.md

+17-7
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
### Introduction
44

5-
Cloning an environment in Hopsworks means creating a snapshot of one of our provided base environments. The base environments are immutable, meaning that it is required to clone an environment before you can make any change to it, such as installing your own libraries. This is to ensure that the project maintains a set of stable environments that are tested with the capabilities of the platform and allows users to build pipelines using supported versions of different python packages.
5+
Cloning an environment in Hopsworks means creating a copy of one of the base environments. The base environments are immutable, meaning that it is required to clone an environment before you can make any change to it, such as installing your own libraries. This ensures that the project maintains a set of stable environments that are tested with the capabilities of the platform, meanwhile through cloning, allowing users to further customize an environment without affecting the base environments.
66

77
In this guide, you will learn how to clone an environment.
88

@@ -15,30 +15,40 @@ First select an environment, for example the `python-feature-pipeline`.
1515
<p align="center">
1616
<figure>
1717
<img src="../../../../assets/images/guides/python/clone_env_1.png" alt="">
18-
<figcaption>Remove environment</figcaption>
18+
<figcaption>Select a base environment</figcaption>
1919
</figure>
2020
</p>
2121

2222
## Step 2: Clone environment
2323

24-
The environment can now be cloned by clicking `Clone environment` and entering a name and description. The interface will show `Syncing packages` while creating the environment.
24+
The environment can now be cloned by clicking `Clone env` and entering a name and description. The interface will show `Syncing packages` while creating the environment.
2525

2626
<p align="center">
2727
<figure>
2828
<img src="../../../../assets/images/guides/python/clone_env_2.png" alt="Create environment">
29-
<figcaption>Create environment</figcaption>
29+
<figcaption>Clone a base environment</figcaption>
3030
</figure>
3131
</p>
3232

3333
## Step 3: Created environment
3434

35-
!!! notice "Notice"
36-
Notice that the cloned environment is tagged as `CUSTOM`, meaning that it is a base environment which has been modified.
35+
<p align="center">
36+
<figure>
37+
<img src="../../../../assets/images/guides/python/clone_env_3.png" alt="">
38+
<figcaption>Environment is now cloned</figcaption>
39+
</figure>
40+
</p>
41+
42+
!!! notice "What does the CUSTOM mean?"
43+
Notice that the cloned environment is tagged as `CUSTOM`, it means that it is a base environment which has been cloned.
44+
45+
!!! notice "Base environment also marked"
46+
When you select a `CUSTOM` environment the base environment it was cloned from is also shown.
3747

3848
## Concerning upgrades
3949

4050
!!! warning "Please note"
41-
The base environments are automatically upgraded when Hopsworks is upgraded and application code should keep functioning provided that no breaking changes were made in the upgraded version of the environment. However a `CUSTOM` environment is not automatically upgraded and the user will need to manually apply the required changes if they encounter issues.
51+
The base environments are automatically upgraded when Hopsworks is upgraded and application code should keep functioning provided that no breaking changes were made in the upgraded version of the environment. A `CUSTOM` environment is not automatically upgraded and the users is recommended to reapply the modifications to a base environment if they encounter issues after an upgrade.
4252

4353
## Next steps
4454

docs/user_guides/projects/python/python_env_export.md

+5-1
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,11 @@ In this guide, you will learn how to export a python environment.
1010

1111
Under the `Project settings` section you can find the `Python environment` setting.
1212

13-
## Step 2: Click Export env
13+
## Step 2: Select a CUSTOM environment
14+
15+
Select the environment that you have previously cloned and want to export.
16+
17+
## Step 3: Click Export env
1418

1519
An existing Anaconda environment can be exported as a yml file, clicking the `Export env` will download the `environment.yml` file in your browser.
1620

docs/user_guides/projects/python/python_env_overview.md

+5-5
Original file line numberDiff line numberDiff line change
@@ -39,16 +39,16 @@ The `MODEL TRAINING` environments can be used in [Jupyter notebooks](../jupyter/
3939

4040
* `tensorflow-training-pipeline` to train TensorFlow models
4141
* `torch-training-pipeline` to train PyTorch models
42-
* `pandas-training-pipeline` to train XGBoost, Catboost and SkLearn models
42+
* `pandas-training-pipeline` to train XGBoost, Catboost and Sklearn models
4343

4444
### Model inference
4545

4646
The `MODEL INFERENCE` environments can be used in a deployment using a custom predictor script.
4747

48-
* `tensorflow-inference-pipeline` to train TensorFlow models
49-
* `torch-inference-pipeline` to train PyTorch models
50-
* `pandas-inference-pipeline` to train XGBoost, Catboost and SkLearn models
51-
* `minimal-inference-pipeline` to install your own custom framework
48+
* `tensorflow-inference-pipeline` to load and serve TensorFlow models
49+
* `torch-inference-pipeline` to load and serve PyTorch models
50+
* `pandas-inference-pipeline` to load and serve XGBoost, Catboost and Sklearn models
51+
* `minimal-inference-pipeline` to install your own custom framework, contains a minimal set of dependencies
5252

5353
## Next steps
5454

docs/user_guides/projects/python/python_install.md

+26-10
Original file line numberDiff line numberDiff line change
@@ -8,17 +8,25 @@ In this guide, you will learn how to install Python packages using these differe
88

99
* PyPi, using pip package manager
1010
* A conda channel, using conda package manager
11-
* Packages saved in certain file formats, currently we support .whl or .egg
11+
* Packages contained in .whl format
1212
* A public or private git repository
1313
* A requirements.txt file to install multiple libraries at the same time using pip
14-
* An environment.yml file to install multiple libraries at the same time using conda and pip
15-
16-
Under the `Project settings` section you can find the `Python environment` setting.
1714

1815
!!! notice "Notice"
1916
If your libraries require installing some extra OS-Level packages, refer to the guide custom commands guide on how to install OS-Level packages.
2017

21-
### Name and version
18+
19+
### Step 1: Go to environments page
20+
21+
Under the `Project settings` section select the `Python environment` setting.
22+
23+
### Step 2: Select a CUSTOM environment
24+
25+
Select the environment that you have previously cloned and want to modify.
26+
27+
### Step 3: Installation options
28+
29+
#### Name and version
2230

2331
Enter the name and, optionally, the desired version to install.
2432

@@ -29,7 +37,7 @@ Enter the name and, optionally, the desired version to install.
2937
</figure>
3038
</p>
3139

32-
### Search
40+
#### Search
3341

3442
Enter the search term and select a library and version to install.
3543

@@ -40,7 +48,7 @@ Enter the search term and select a library and version to install.
4048
</figure>
4149
</p>
4250

43-
### Distribution (.whl, .egg..)
51+
#### Distribution (.whl, .egg..)
4452

4553
Install a python package by uploading the corresponding package file and selecting it in the file browser.
4654

@@ -51,12 +59,20 @@ Install a python package by uploading the corresponding package file and selecti
5159
</figure>
5260
</p>
5361

54-
### Git source
62+
#### Git source
63+
64+
The URL you should provide is the same as you would enter on the command line using `pip install git+{repo_url}`, where `repo_url` is the part that you enter in `Git URL`.
65+
66+
For example to install matplotlib 3.7.2, the following are correct inputs:
67+
68+
`matplotlib @ git+https://github.com/matplotlib/matplotlib@v3.7.2`
69+
70+
`git+https://github.com/matplotlib/matplotlib@v3.7.2`
5571

56-
To install from a git repository simply provide the repository URL. The URL you should provide is the same as you would enter on the command line using `pip install git+{repo_url}`.
5772
In the case of a private git repository, also select whether it is a GitHub or GitLab repository and the preconfigured access token for the repository.
5873

59-
**Note**: If you are installing from a git repository which is not GitHub or GitLab simply supply the access token in the URL. Keep in mind that in this case the access token may be visible in logs for other users in the same project to see.
74+
!!! notice "Keep your secrets safe"
75+
If you are installing from a git repository which is not GitHub or GitLab simply supply the access token in the URL. Keep in mind that in this case the access token may be visible in logs for other users in the same project to see.
6076

6177
<p align="center">
6278
<figure>

0 commit comments

Comments
 (0)