Skip to content

Commit 2676450

Browse files
committed
Add short info about scheduling in Jobs guide
1 parent ae0ea1d commit 2676450

File tree

4 files changed

+14
-5
lines changed

4 files changed

+14
-5
lines changed

docs/user_guides/projects/jobs/pyspark_job.md

+6-2
Original file line numberDiff line numberDiff line change
@@ -12,9 +12,13 @@ All members of a project in Hopsworks can launch the following types of applicat
1212
- Apache Spark
1313

1414
Launching a job of any type is very similar process, what mostly differs between job types is
15-
the various configuration parameters each job type comes with. After following this guide you will be able to create a PySpark job.
15+
the various configuration parameters each job type comes with. Hopsworks clusters support schedyuling to run jobs on a regular basis,
16+
e.g backfilling a Feature Group by running your feature engineering pipeline nightly. Scheduling can be done both through the UI and the python API,
17+
checkout [our Scheduling guide](schedule_job.md).
1618

17-
The PySpark program can either be a `.py` script or a `.ipynb` file.
19+
20+
PySpark program can either be a `.py` script or a `.ipynb` file, however be mindful of how to access/create
21+
the spark session based on the extension you provide.
1822

1923
!!! notice "Instantiate the SparkSession"
2024
For a `.py` file, remember to instantiate the SparkSession i.e `spark=SparkSession.builder.getOrCreate()`

docs/user_guides/projects/jobs/python_job.md

+3-1
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,9 @@ All members of a project in Hopsworks can launch the following types of applicat
1212
- Apache Spark
1313

1414
Launching a job of any type is very similar process, what mostly differs between job types is
15-
the various configuration parameters each job type comes with. After following this guide you will be able to create a Python job.
15+
the various configuration parameters each job type comes with. Hopsworks clusters support schedyuling to run jobs on a regular basis,
16+
e.g backfilling a Feature Group by running your feature engineering pipeline nightly. Scheduling can be done both through the UI and the python API,
17+
checkout [our Scheduling guide](schedule_job.md).
1618

1719
!!! note "Kubernetes integration required"
1820
Python Jobs are only available if Hopsworks has been integrated with a Kubernetes cluster.

docs/user_guides/projects/jobs/schedule_job.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ description: Documentation on how to schedule a job on Hopsworks.
66

77
## Introduction
88

9-
Hopsworks jobs can be scheduled to run at regular intervals using the scheduling function provided by Hopsworks. Each job can be configured to have a single schedule.
9+
Hopsworks clusters can run jobs on a schedule, allowing you to automate the execution of your jobs. Whether you need to backfill your feature groups on a nightly basis or run a model training pipeline every week, the Hopsworks scheduler can help you automate these tasks. Each job can be configured to have a single schedule. For more complex use cases, Hopsworks integrates with the open-source [Apache Airflow](https://airflow.apache.org/use-cases/), check out our [Airflow Guide](../airflow/airflow.md).
1010

1111
Schedules can be defined using the drop down menus in the UI or a Quartz [cron](https://en.wikipedia.org/wiki/Cron) expression.
1212

docs/user_guides/projects/jobs/spark_job.md

+4-1
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,10 @@ All members of a project in Hopsworks can launch the following types of applicat
1212
- Apache Spark
1313

1414
Launching a job of any type is very similar process, what mostly differs between job types is
15-
the various configuration parameters each job type comes with. After following this guide you will be able to create a Spark job.
15+
the various configuration parameters each job type comes with. Hopsworks clusters support schedyuling to run jobs on a regular basis,
16+
e.g backfilling a Feature Group by running your feature engineering pipeline nightly. Scheduling can be done both through the UI and the python API,
17+
checkout [our Scheduling guide](schedule_job.md).
18+
1619

1720
## UI
1821

0 commit comments

Comments
 (0)