[HWORKS-1670] Ray documentation

docs/user_guides/projects/jobs/ray_job.md

@@ -0,0 +1,210 @@
+---
+description: Documentation on how to configure and execute a Ray job on Hopsworks.
+---
+
+# How To Run A Ray Job
+
+## Introduction
+
+All members of a project in Hopsworks can launch the following types of applications through a project's Jobs service:
+
+- Python (*Hopsworks Enterprise only*)
+- Apache Spark
+- Ray 
+
+Launching a job of any type is very similar process, what mostly differs between job types is
+the various configuration parameters each job type comes with. Hopsworks support scheduling to run jobs on a regular basis,
+e.g backfilling a Feature Group by running your feature engineering pipeline nightly. Scheduling can be done both through the UI and the python API,
+checkout [our Scheduling guide](schedule_job.md).
+
+!!!warning "Enable Ray"
+
+    Support for Ray needs to be explicitly enabled by adding the following option in the `values.yaml` file for the deployment:
+
+    ```yaml
+    global:
+      ray:
+        enabled: true
+    ```
+
+## UI
+
+### Step 1: Jobs overview
+
+The image below shows the Jobs overview page in Hopsworks and is accessed by clicking `Jobs` in the sidebar.
+
+<p align="center">
+  <figure>
+    <img src="../../../../assets/images/guides/jobs/jobs_overview.png" alt="Jobs overview">
+    <figcaption>Jobs overview</figcaption>
+  </figure>
+</p>
+
+### Step 2: Create new job dialog
+
+Click `New Job` and the following dialog will appear.
+
+<p align="center">
+  <figure>
+    <img src="../../../../assets/images/guides/jobs/create_new_job.png" alt="Create new job dialog">
+    <figcaption>Create new job dialog</figcaption>
+  </figure>
+</p>
+
+### Step 3: Set the job type
+
+By default, the dialog will create a Spark job. Make sure `RAY` is chosen.
+
+### Step 4: Set the script
+
+Next step is to select the program to run. You can either select `From project`, if the file was previously uploaded to Hopsworks, or `Upload new file` which lets you select a file from your local filesystem as demonstrated below. After that set the name for the job. By default, the job name is the same as the file name, but you can customize it here.
+
+<p align="center">
+  <figure>
+    <img src="../../../../assets/images/guides/jobs/upload_ray_script.png" alt="Configure program">
+    <figcaption>Configure program</figcaption>
+  </figure>
+</p>
+
+### Step 5 (optional): Advanced configuration
+
+Resource allocation for the Driver and Workers can be configured.
+
+!!! notice "Using the resources in the Ray script"
+    The resource configurations describe the cluster that will be provisioned when launching the Ray job. User can still 
+    provide extra configurations in the job script using `ScalingConfig`, i.e. `ScalingConfig(num_workers=4, trainer_resources={"CPU": 1}, use_gpu=True)`.
+
+* `Driver memory`: Memory in MBs to allocate for Driver
+
+* `Driver virtual cores`: Number of cores to allocate for the Driver
+
+* `Worker memory`: Memory in MBs to allocate for each worker
+
+* `Worker cores`: Number of cores to allocate for each worker
+
+* `Min workers`: Minimum number of workers to start with
+
+* `Max workers`: Maximum number of workers to scale up to
+
+
+<p align="center">
+  <figure>
+    <img src="../../../../assets/images/guides/jobs/resource_configuration_ray.png" alt="Resource configuration 
+for Ray Job">
+    <figcaption>Resource configuration for the Ray Job</figcaption>
+  </figure>
+</p>
+
+Runtime environment and Additional files required for the Ray job can also be provided.
+
+* `Runtime Environment (Optional)`:  A [runtime environment](https://docs.ray.io/en/latest/ray-core/handling-dependencies.html#runtime-environments) describes the dependencies required for the Ray job including files, packages, environment variables, and more. This is useful when you need to install specific packages and set environment variables for this particular Ray job. It should be provided as a YAML file. You can select the file from the project or upload a new one.
+
+* `Additional files`: List of other files required for the Ray job. These files will be placed in `/srv/hops/ray/job`.
+
+<p align="center">
+  <figure>
+    <img src="../../../../assets/images/guides/jobs/ray_runtime_env_and_additional_files.png" alt="Runtime 
+environment and additional files">
+    <figcaption>Runtime configuration and additional files for Ray job</figcaption>
+  </figure>
+</p>
+
+### Step 6: Execute the job
+
+Now click the `Run` button to start the execution of the job, and then click on `Executions` to see the list of all executions.
+
+<p align="center">
+  <figure>
+    <img src="../../../../assets/images/guides/jobs/start_ray_job.png" alt="Start job execution">
+    <figcaption>Start job execution</figcaption>
+  </figure>
+</p>
+
+## Ray Dashboard
+When the Ray job is running, you can access the Ray dashboard to monitor the job. The Ray dashboard is accessible from the 
+`Executions` page. Please note that the Ray dashboard is only available when the job execution is running. In the Ray Dashboard, 
+you can monitor the resources used by the job, the number of workers, logs, and the tasks that are running.
+<p align="center">
+  <figure>
+    <img src="../../../../assets/images/guides/jobs/ray_dashboard.png" alt="Access Ray Dashboard">
+    <figcaption>Access Ray Dashboard</figcaption>
+  </figure>
+</p>
+
+### Step 9: Application logs
+Once the execution is finished, you can click on `Logs` to see the full logs for execution.
+
+<p align="center">
+  <figure>
+    <img src="../../../../assets/images/guides/jobs/ray_application_logs.png" alt="Access Ray logs">
+    <figcaption>Access Ray job execution logs</figcaption>
+  </figure>
+</p>
+
+## Code
+
+### Step 1: Upload the Ray script
+
+This snippet assumes the Ray program is in the current working directory and named `ray_job.py`. If the file is already in the project, you can skip this step.
+
+It will upload the jar to the `Resources` dataset in your project.
+
+```python
+
+import hopsworks
+
+project = hopsworks.login()
+
+dataset_api = project.get_dataset_api()
+
+uploaded_file_path = dataset_api.upload("ray_job.py", "Resources")
+
+```
+
+
+### Step 2: Create Ray job
+
+In this snippet we get the `JobsApi` object to get the default job configuration for a `RAY` job, set the python script to run and create the `Job` object.
+
+```python
+
+jobs_api = project.get_jobs_api()
+
+ray_config = jobs_api.get_configuration("RAY")
+
+ray_config['appPath'] = uploaded_file_path
+ray_config['environmentName'] = "ray-training-pipeline"
+ray_config['driverCores'] = 2
+ray_config['driverMemory'] = 2048
+ray_config['workerCores'] = 2
+ray_config['workerMemory'] = 4096
+ray_config['minWorkers'] = 1
+ray_config['maxWorkers'] = 4
+
+job = jobs_api.create_job("ray_job", ray_config)
+
+```
+
+### Step 3: Execute the job
+
+In this snippet we execute the job synchronously, that is wait until it reaches a terminal state, and then download and print the logs.
+
+```python
+
+execution = job.run(await_termination=True)
+
+out, err = execution.download_logs()
+
+f_out = open(out, "r")
+print(f_out.read())
+
+f_err = open(err, "r")
+print(f_err.read())
+
+```
+
+### API Reference
+
+[Jobs](https://docs.hopsworks.ai/hopsworks-api/{{{ hopsworks_version }}}/generated/api/jobs/)
+
+[Executions](https://docs.hopsworks.ai/hopsworks-api/{{{ hopsworks_version }}}/generated/api/executions/)

docs/user_guides/projects/jupyter/ray_notebook.md

@@ -0,0 +1,142 @@
+---
+description: Documentation on how to run Ray applications on Jupyter on Hopsworks.
+---
+# How To Run A Ray Notebook
+
+### Introduction
+
+Jupyter is provided as a service in Hopsworks, providing the same user experience and features as if run on your laptop.
+
+* Supports JupyterLab and the classic Jupyter front-end
+* Configured with Python3, PySpark and Ray kernels
+
+!!!warning "Enable Ray"
+
+    Support for Ray needs to be explicitly enabled by adding the following option in the `values.yaml` file for the deployment:
+
+    ```yaml
+    global:
+      ray:
+        enabled: true
+    ```
+
+## Step 1: Jupyter dashboard
+
+The image below shows the Jupyter service page in Hopsworks and is accessed by clicking `Jupyter` in the sidebar.
+
+<p align="center">
+  <figure>
+    <img src="../../../../assets/images/guides/jupyter/jupyter_overview.png" alt="Jupyter dashboard in Hopsworks">
+    <figcaption>Jupyter dashboard in Hopsworks</figcaption>
+  </figure>
+</p>
+
+From this page, you can configure various options and settings to start Jupyter with as described in the sections below.
+
+## Step 2 (Optional): Configure Ray
+
+Next step is to configure the Ray cluster configuration that will be created when you start Ray session later in 
+Jupyter. Click `edit configuration` to get to the configuration page and select `Ray`.
+
+### Resource and compute
+
+Resource allocation for the Driver and Workers can be configured.
+
+!!! notice "Using the resources in the Ray script"
+    The resource configurations describe the cluster that will be provisioned when launching the Ray job. User can still 
+    provide extra configurations in the job script using `ScalingConfig`, i.e. `ScalingConfig(num_workers=4, trainer_resources={"CPU": 1}, use_gpu=True)`.
+
+* `Driver memory`: Memory in MBs to allocate for Driver
+
+* `Driver virtual cores`: Number of cores to allocate for the Driver
+
+* `Worker memory`: Memory in MBs to allocate for each worker
+
+* `Worker cores`: Number of cores to allocate for each worker
+
+* `Min workers`: Minimum number of workers to start with
+
+* `Max workers`: Maximum number of workers to scale up to
+
+<p align="center">
+  <figure>
+    <img src="../../../../assets/images/guides/jupyter/ray_resource_and_compute.png" alt="Resource configuration for 
+the Ray kernels">
+    <figcaption>Resource configuration for the Ray kernels</figcaption>
+  </figure>
+</p>
+
+Runtime environment and Additional files required for the Ray job can also be provided.
+
+* `Runtime Environment (Optional)`:  A [runtime environment](https://docs.ray.io/en/latest/ray-core/handling-dependencies.html#runtime-environments) describes the dependencies required for the Ray job including files, packages, environment variables, and more. This is useful when you need to install specific packages and set environment variables for this particular Ray job. It should be provided as a YAML file. You can select the file from the project or upload a new one.
+
+
+* `Additional files`: List of other files required for the Ray job. These files will be placed in `/srv/hops/ray/job`.
+
+<p align="center">
+  <figure>
+    <img src="../../../../assets/images/guides/jobs/ray_runtime_env_and_additional_files.png" alt="Runtime 
+environment and additional files">
+    <figcaption>Runtime configuration and additional files for Ray jupyter session</figcaption>
+  </figure>
+</p>
+
+
+Click `Save` to save the new configuration.
+
+## Step 3 (Optional): Configure max runtime and root path
+
+Before starting the server there are two additional configurations that can be set next to the `Run Jupyter` button.
+
+The runtime of the Jupyter instance can be configured, this is useful to ensure that idle instances will not be hanging around and keep allocating resources. If a limited runtime is not desirable, this can be disabled by setting `no limit`.
+
+<p align="center">
+  <figure>
+    <img src="../../../../assets/images/guides/jupyter/configure_shutdown.png" alt="Configure maximum runtime">
+    <figcaption>Configure maximum runtime</figcaption>
+  </figure>
+</p>
+
+The root path from which to start the Jupyter instance can be configured. By default it starts by setting the `/Jupyter` folder as the root.
+
+<p align="center">
+  <figure>
+    <img src="../../../../assets/images/guides/jupyter/start_from_folder.png" alt="Configure root folder">
+    <figcaption>Configure root folder</figcaption>
+  </figure>
+</p>
+
+## Step 4: Select the environment
+Hopsworks provides a variety of environments to run Jupyter notebooks. Select the environment you want to use by clicking on the dropdown menu.
+In order to be able to run a Ray notebook, you need to select the environment that has the Ray kernel installed. 
+Environment with Ray kernel have a `Ray Enabled` label next to them.
+
+## Step 5: Start Jupyter
+
+Start the Jupyter instance by clicking the `Run Jupyter` button.
+
+## Running Ray code in Jupyter
+Once the Jupyter instance is started, you can create a new notebook by clicking on the `New` button and selecting 
+`Ray` kernel. You can now write and run Ray code in the notebook. When you first run a cell with Ray code, a Ray session will be started and you can monitor the resources used by the job in the Ray dashboard.
+
+<p align="center">
+  <figure>
+    <img src="../../../../assets/images/guides/jupyter/ray_kernel.png" alt="Ray Kernel">
+    <figcaption>Ray Kernel</figcaption>
+  </figure>
+</p>
+
+## Step 6: Access Ray Dashboard
+
+When you start a Ray session in Jupyter, a new application will appear in the Jupyter page.
+The notebook name from which the session was started is displayed. You can access the Ray UI by clicking on the `Ray Dashboard` and a new 
+tab will be opened. The Ray dashboard is only available while the Ray kernel is running. 
+You can kill the Ray session to free up resources by shutting down the kernel in Jupyter. 
+In the Ray Dashboard, you can monitor the resources used  by code you are running, the number of workers, logs, and the tasks that are running.
+
+<p align="center">
+  <figure>
+    <img src="../../../../assets/images/guides/jupyter/ray_jupyter_notebook_session.png" alt="Access Ray Dashboard">
+    <figcaption>Access Ray Dashboard for Jupyter Ray session</figcaption>
+  </figure>
+</p>

docs/user_guides/projects/jupyter/spark_notebook.md

@@ -5,7 +5,7 @@
 Jupyter is provided as a service in Hopsworks, providing the same user experience and features as if run on your laptop.
 
 * Supports JupyterLab and the classic Jupyter front-end
-* Configured with Python and PySpark kernels
+* Configured with Python3, PySpark and Ray kernels
 
 
 ## Step 1: Jupyter dashboard
@@ -34,7 +34,8 @@ You can easily refer to the green ticks as to what kernels are available in whic
 
 ## Step 3 (Optional): Configure spark properties
 
-Next step is to configure the Spark properties to be used in Jupyter, Click `edit configuration` to get to the configuration page and select `Spark`.
+Next step is to configure the Ray properties to be used in Jupyter, Click `edit configuration` to get to the 
+configuration page and select `Ray`.
 
 ### Resource and compute
 

mkdocs.yml

@@ -147,11 +147,13 @@ nav:
           - Jupyter:
               - Run Python Notebook: user_guides/projects/jupyter/python_notebook.md
               - Run PySpark Notebook: user_guides/projects/jupyter/spark_notebook.md
+              - Run Ray Notebook: user_guides/projects/jupyter/ray_notebook.md
               - Remote Filesystem Driver: user_guides/projects/jupyter/remote_filesystem_driver.md
           - Jobs:
               - Run PySpark Job: user_guides/projects/jobs/pyspark_job.md
               - Run Spark Job: user_guides/projects/jobs/spark_job.md
               - Run Python Job: user_guides/projects/jobs/python_job.md
+              - Run Ray Job: user_guides/projects/jobs/ray_job.md
               - Run Jupyter Notebook Job: user_guides/projects/jobs/notebook_job.md
               - Scheduling: user_guides/projects/jobs/schedule_job.md
           - Airflow: user_guides/projects/airflow/airflow.md