Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rcnnnghm/k8s release docs #410

Merged
merged 24 commits into from
Oct 24, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
c990122
Remove virtual machines from setup/admin page
rcnnnghm Oct 4, 2024
4f3e184
Move content around for kubernetes
rcnnnghm Oct 4, 2024
71fe595
Change VM content to kubernetes in index.md
rcnnnghm Oct 4, 2024
1cf7922
Fix broken assets after admin move.
rcnnnghm Oct 4, 2024
9f26d5e
Remove references to Karamel in kerberos/ldap config.
rcnnnghm Oct 4, 2024
21b17f7
Change Azure instructions to use kubernetes.
rcnnnghm Oct 4, 2024
77d2352
Change AWS instructions to use kubernetes.
rcnnnghm Oct 4, 2024
4d3e735
Fix step sub numbers in azure.
rcnnnghm Oct 4, 2024
abdf4e5
Switch to kubernetes steps for GCP/GKE
rcnnnghm Oct 4, 2024
1ffa695
Remove Managed files and top-level tab.
rcnnnghm Oct 7, 2024
17e519b
Rework On-Prem content to retain contact Hopsworks.
rcnnnghm Oct 7, 2024
f58e193
Fix last step number in each getting started.
rcnnnghm Oct 7, 2024
431fad2
Remove "--devel" from helm install in getting started.
rcnnnghm Oct 7, 2024
d10fd2c
Change "hopsworks-dev" to hopsworks for helm commands.
rcnnnghm Oct 7, 2024
a4fc983
Add try URL to content and update helm command.
rcnnnghm Oct 7, 2024
839da8c
Remove kubectl create secret commands
rcnnnghm Oct 7, 2024
49e812a
Remove reference to kserve in values.yaml file.
rcnnnghm Oct 7, 2024
b1df44e
Fix broken link to on-prem contact Hopsworks.
rcnnnghm Oct 7, 2024
6e776b0
Fix link checks and corresponding content.
rcnnnghm Oct 7, 2024
44deac2
Fix more link check failures.
rcnnnghm Oct 7, 2024
19b93cb
Remove last old link to hopsworks installer
rcnnnghm Oct 7, 2024
8b31521
Remove non-existant reference to aws permissions.
rcnnnghm Oct 8, 2024
c6c7905
Fix assets reference in ha.md
rcnnnghm Oct 8, 2024
005b455
Add 4.0 migration info.
rcnnnghm Oct 24, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -185,7 +185,7 @@ pointer-events: initial;
</a>
</div>
<div id="w-node-_4a479fbb-90c7-9f47-d439-20aa6a224339-46672785" class="infra">
<a href="./setup_installation/on_prem/hopsworks_installer/">
<a href="./setup_installation/on_prem/contact_hopsworks/">
<img src="images/icons8-database.svg" loading="lazy" alt="" class="infra-icon">
<div class="name_item small">On-premise</div>
</a>
Expand Down Expand Up @@ -265,7 +265,7 @@ Hopsworks provides projects as a secure sandbox in which teams can collaborate a
Hopsworks provides a FTI (feature/training/inference) pipeline architecture for ML systems. Each part of the pipeline is defined in a Hopsworks job which corresponds to a Jupyter notebook, a python script or a jar. The production pipelines are then orchestrated with Airflow which is bundled in Hopsworks. Hopsworks provides several python environments that can be used and customized for each part of the FTI pipeline, for example switching between using PyTorch or TensorFlow in the training pipeline. You can train models on as many GPUs as are installed in a Hopsworks cluster and easily share them among users. You can also run Spark, Spark Streaming, or Flink programs on Hopsworks. JupyterLab is also bundled which can be used to run Python and Spark interactively.

## Available on any Platform
Hopsworks is available as a both managed platform in the cloud on AWS, Azure, and GCP, and can be installed on any Linux-based virtual machines (Ubuntu/Redhat compatible), even in air-gapped data centers. Hopsworks is also available as a serverless platform that manages and serves both your features and models.
Hopsworks is available to be installed on a kubernetes cluster in the cloud on AWS, Azure, and GCP, and On-Prem (Ubuntu/Redhat compatible), even in air-gapped data centers. Hopsworks is also available as a serverless platform that manages and serves both your features and models.

## Join the community
- Ask questions and give us feedback in the [Hopsworks Community](https://community.hopsworks.ai/)
Expand Down
4 changes: 2 additions & 2 deletions docs/js/dropdown.js
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
document.getElementsByClassName("md-tabs__link")[7].style.display = "none";
document.getElementsByClassName("md-tabs__link")[9].style.display = "none";
document.getElementsByClassName("md-tabs__link")[6].style.display = "none";
document.getElementsByClassName("md-tabs__link")[8].style.display = "none";

18 changes: 9 additions & 9 deletions docs/admin/alert.md → docs/setup_installation/admin/alert.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Cluster Settings from the dropdown menu. In the Cluster Settings' Alerts tab you
manager to send alerts via email, slack or pagerduty.

<figure>
<img src="../../assets/images/alerts/configure-alerts.png" alt="Configure alerts"/>
<img src="../../../assets/images/alerts/configure-alerts.png" alt="Configure alerts"/>
<figcaption>Configure alerts</figcaption>
</figure>

Expand All @@ -23,7 +23,7 @@ To send alerts via email you need to configure an SMTP server. Click on the _Con
button on the left side of the **email** row and fill out the form that pops up.

<figure>
<img src="../../assets/images/alerts/smtp-config.png" alt="Configure Email Alerts"/>
<img src="../../../assets/images/alerts/smtp-config.png" alt="Configure Email Alerts"/>
<figcaption>Configure Email Alerts</figcaption>
</figure>

Expand All @@ -34,27 +34,27 @@ button on the left side of the **email** row and fill out the form that pops up.
CRAM-MD5, LOGIN or PLAIN.

Optionally cluster wide Email alert receivers can be added in _Default receiver emails_.
These receivers will be available to all users when they create event triggered [alerts](../../user_guides/fs/feature_group/data_validation_best_practices#setup-alerts).
These receivers will be available to all users when they create event triggered [alerts](../../../user_guides/fs/feature_group/data_validation_best_practices#setup-alerts).

### Step 3: Configure Slack Alerts
Alerts can also be sent via Slack messages. To be able to send Slack messages you first need to configure
a Slack webhook. Click on the _Configure_ button on the left side of the **slack** row and past in your
[Slack webhook](https://api.slack.com/messaging/webhooks) in _Webhook_.

<figure>
<img src="../../assets/images/alerts/slack-config.png" alt="Configure slack Alerts"/>
<img src="../../../assets/images/alerts/slack-config.png" alt="Configure slack Alerts"/>
<figcaption>Configure slack Alerts</figcaption>
</figure>

Optionally cluster wide Slack alert receivers can be added in _Slack channel/user_.
These receivers will be available to all users when they create event triggered [alerts](../../user_guides/fs/feature_group/data_validation_best_practices/#setup-alerts).
These receivers will be available to all users when they create event triggered [alerts](../../../user_guides/fs/feature_group/data_validation_best_practices/#setup-alerts).

### Step 4: Configure Pagerduty Alerts
Pagerduty is another way you can send alerts from Hopsworks. Click on the _Configure_ button on the left side of
the **pagerduty** row and fill out the form that pops up.

<figure>
<img src="../../assets/images/alerts/pagerduty-config.png" alt="Configure Pagerduty Alerts"/>
<img src="../../../assets/images/alerts/pagerduty-config.png" alt="Configure Pagerduty Alerts"/>
<figcaption>Configure Pagerduty Alerts</figcaption>
</figure>

Expand All @@ -76,7 +76,7 @@ If you are familiar with Prometheus' [Alert manager](https://prometheus.io/docs/
you can also configure alerts by editing the _yaml/json_ file directly.

<figure>
<img src="../../assets/images/alerts/advanced-config.png" alt="Advanced configuration"/>
<img src="../../../assets/images/alerts/advanced-config.png" alt="Advanced configuration"/>
<figcaption>Advanced configuration</figcaption>
</figure>

Expand All @@ -93,7 +93,7 @@ global:
...
```

To test the alerts by creating triggers from Jobs and Feature group validations see [Alerts](../../user_guides/fs/feature_group/data_validation_best_practices/#setup-alerts).
To test the alerts by creating triggers from Jobs and Feature group validations see [Alerts](../../../user_guides/fs/feature_group/data_validation_best_practices/#setup-alerts).

The yaml syntax in the UI is slightly different in that it does not allow double quotes (it will ignore the values but give no error).
Below is an example configuration, that can be used in the UI, with both email and slack receivers configured for system alerts.
Expand Down Expand Up @@ -144,4 +144,4 @@ receivers:
```

## Conclusion
In this guide you learned how to configure alerts in Hopsworks.
In this guide you learned how to configure alerts in Hopsworks.
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Audit logs can be configured from the _Cluster Settings_ Configuration tab.
You can access the _Configuration_ page of your Hopsworks cluster by clicking on your name, in the top right corner, and choosing _Cluster Settings_ from the dropdown menu.

<figure>
<img src="../../../assets/images/admin/audit/audit-log-vars.png" alt="Audit log configuration" />
<img src="../../../../assets/images/admin/audit/audit-log-vars.png" alt="Audit log configuration" />
<figcaption>Audit log configuration</figcaption>
</figure>

Expand Down
4 changes: 2 additions & 2 deletions docs/admin/auth.md → docs/setup_installation/admin/auth.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,12 +35,12 @@ In the **Cluster Settings** _Authentication_ tab you can configure how users aut
[Configure LDAP](../ldap/configure-ldap) and [Configure Kerberos](../ldap/configure-krb).

<figure>
<img src="../../assets/images/admin/auth-config.png" alt="Authentication config" />
<img src="../../../assets/images/admin/auth-config.png" alt="Authentication config" />
<figcaption>Setup Authentication Methods</figcaption>
</figure>

In the figure above we see a cluster with Two-factor authentication disabled, OAuth enabled with one registered
identity provider and LDAP authentication enabled.

## Conclusion
In this guide you learned how to configure authentication methods in Hopsworks.
In this guide you learned how to configure authentication methods in Hopsworks.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ At a high level a Hopsworks cluster can be divided into 4 groups of nodes. Each
Example deployment:

<figure>
<img width="800px" src="../../../assets/images/admin/ha_dr/example_ha_cluster.svg" alt="Example HA deployment"/>
<img width="800px" src="../../../../assets/images/admin/ha_dr/example_ha_cluster.svg" alt="Example HA deployment"/>
<figcaption>Example High Available deployment</figcaption>
</figure>

Expand Down
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -19,15 +19,15 @@ If LDAP/Kerberos checkbox is not checked, make sure that you configured your app
clicking on the checkbox.

<figure>
<img src="../../../assets/images/admin/ldap/auth-config-krb.png" alt="Authentication config" />
<img src="../../../../assets/images/admin/ldap/auth-config-krb.png" alt="Authentication config" />
<figcaption>Setup Authentication Methods</figcaption>
</figure>

### Step 2: Edit configuration
Finally, click on edit configuration and fill in the attributes.

<figure>
<img src="../../../assets/images/admin/ldap/configure-krb.png" alt="Kerberos config" />
<img src="../../../../assets/images/admin/ldap/configure-krb.png" alt="Kerberos config" />
<figcaption>Configure Kerberos</figcaption>
</figure>

Expand Down Expand Up @@ -59,7 +59,7 @@ All defaults are taken from [OpenLDAP](https://www.openldap.org/).

The login page will now have the choice to use Kerberos for authentication.
<figure>
<img width="400px" src="../../../assets/images/admin/ldap/login-using-krb.png" alt="Log in using Kerberos" />
<img width="400px" src="../../../../assets/images/admin/ldap/login-using-krb.png" alt="Log in using Kerberos" />
<figcaption>Log in using Kerberos</figcaption>
</figure>

Expand All @@ -69,4 +69,4 @@ The login page will now have the choice to use Kerberos for authentication.
Kerberos support must also be configured on the browser to use Kerberos for authentication.

## Conclusion
In this guide you learned how to configure Kerberos for authentication.
In this guide you learned how to configure Kerberos for authentication.
Original file line number Diff line number Diff line change
Expand Up @@ -20,15 +20,15 @@ If LDAP/Kerberos checkbox is not checked make sure that you configured your appl
clicking on the checkbox.

<figure>
<img src="../../../assets/images/admin/auth-config.png" alt="Authentication config" />
<img src="../../../../assets/images/admin/auth-config.png" alt="Authentication config" />
<figcaption>Setup Authentication Methods</figcaption>
</figure>

### Step 2: Edit configuration
Finally, click on edit configuration and fill in the attributes.

<figure>
<img src="../../../assets/images/admin/ldap/configure-ldap.png" alt="LDAP config" />
<img src="../../../../assets/images/admin/ldap/configure-ldap.png" alt="LDAP config" />
<figcaption>Configure LDAP</figcaption>
</figure>

Expand All @@ -53,7 +53,7 @@ All defaults are taken from [OpenLDAP](https://www.openldap.org/).

The login page will now have the choice to use LDAP for authentication.
<figure>
<img width="400px" src="../../../assets/images/admin/ldap/login-ldap.png" alt="Log in using LDAP" />
<img width="400px" src="../../../../assets/images/admin/ldap/login-ldap.png" alt="Log in using LDAP" />
<figcaption>Log in using LDAP</figcaption>
</figure>

Expand All @@ -65,4 +65,4 @@ The login page will now have the choice to use LDAP for authentication.


## Conclusion
In this guide you learned how to configure LDAP for authentication.
In this guide you learned how to configure LDAP for authentication.
Original file line number Diff line number Diff line change
Expand Up @@ -21,16 +21,16 @@ corner of the navigation bar and choosing *Cluster Settings* from the dropdown m
In the _Project mapping_ tab, you can create a new mapping by clicking on _Create new mapping_.

<figure>
<a href="../../../assets/images/admin/project-mapping/project-mapping-empty.png">
<img src="../../../assets/images/admin/project-mapping/project-mapping-empty.png" alt="Project mapping tab" />
<a href="../../../../assets/images/admin/project-mapping/project-mapping-empty.png">
<img src="../../../../assets/images/admin/project-mapping/project-mapping-empty.png" alt="Project mapping tab" />
</a>
<figcaption>Project mapping</figcaption>
</figure>

This will take you to the create mapping page shown below
<figure>
<a href="../../../assets/images/admin/project-mapping/create-mapping.png">
<img src="../../../assets/images/admin/project-mapping/create-mapping.png" alt="Create mapping" />
<a href="../../../../assets/images/admin/project-mapping/create-mapping.png">
<img src="../../../../assets/images/admin/project-mapping/create-mapping.png" alt="Create mapping" />
</a>
<figcaption>Create mapping</figcaption>
</figure>
Expand All @@ -41,8 +41,8 @@ You can also choose the _Project role_ users will be assigned when they are adde
Finally, click on _Create mapping_ and go back to mappings. You should see the newly created mapping(s) as shown below.

<figure>
<a href="../../../assets/images/admin/project-mapping/project-mappings.png">
<img src="../../../assets/images/admin/project-mapping/project-mappings.png" alt="Project mappings" />
<a href="../../../../assets/images/admin/project-mapping/project-mappings.png">
<img src="../../../../assets/images/admin/project-mapping/project-mappings.png" alt="Project mappings" />
</a>
<figcaption>Project mappings</figcaption>
</figure>
Expand All @@ -63,8 +63,8 @@ From the list of mappings click on the edit button (:material-pencil:). This wil
the _remote group_, _project name_, and _project role_ of a mapping.

<figure>
<a href="../../../assets/images/admin/project-mapping/edit-mapping.png">
<img src="../../../assets/images/admin/project-mapping/edit-mapping.png" alt="Edit mapping" />
<a href="../../../../assets/images/admin/project-mapping/edit-mapping.png">
<img src="../../../../assets/images/admin/project-mapping/edit-mapping.png" alt="Edit mapping" />
</a>
<figcaption>Edit mapping</figcaption>
</figure>
Expand All @@ -88,4 +88,4 @@ add or remove users from projects.


## Conclusion
In this guide you learned how to configure LDAP group to project mapping.
In this guide you learned how to configure LDAP group to project mapping.
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Configure Server for LDAP and Kerberos

## Introduction
LDAP and Kerberos integration need some configuration in the [Karamel](https://github.com/logicalclocks/karamel-chef)
LDAP and Kerberos integration need some configuration in the helm charts for your
cluster definition used to deploy your Hopsworks cluster. This tutorial shows an administrator how to configure the application
server for LDAP and Kerberos integration.

Expand Down Expand Up @@ -34,12 +34,11 @@ ldap:
- security_credentials: contains the password of the user that will be used to query LDAP.
- referral: whether to follow or ignore an alternate location in which an LDAP Request may be processed.

### Without Karamel/Chef
An already deployed instance can be configured to connect to LDAP without re-running Karamel/Chef.
An already deployed instance can be configured to connect to LDAP.
Go to the payara admin UI and create a new JNDI external resource. The name of the resource should be __ldap/LdapResource__.

<figure>
<img src="../../../assets/images/admin/ldap/ldap-resource.png" alt="LDAP Resource" />
<img src="../../../../assets/images/admin/ldap/ldap-resource.png" alt="LDAP Resource" />
<figcaption>LDAP Resource</figcaption>
</figure>

Expand Down Expand Up @@ -95,4 +94,4 @@ Both Kerberos and LDAP attributes need to be specified to configure Kerberos. Th
Initiator should be set to false.

## Conclusion
In this guide you learned how to configure the application server for LDAP and Kerberos.
In this guide you learned how to configure the application server for LDAP and Kerberos.
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ You can access the admin page of your Hopsworks cluster by clicking on your name
You can then navigate to the _Monitoring_ tab. The _Monitoring_ tab gives you access to several of the observability tools that are already deployed to help you manage the health of the cluster.

<figure>
<img src="../../../assets/images/admin/monitoring/monitoring_tab.png" alt="monitoring tab" />
<img src="../../../../assets/images/admin/monitoring/monitoring_tab.png" alt="monitoring tab" />
<figcaption>Monitoring tab</figcaption>
</figure>

Expand All @@ -36,7 +36,7 @@ Dashboards are organized into three folders:
- **Kubernetes**: If you have integrated Hopsworks with a Kubernetes cluster, this folder contains the dashboards to monitor the health of the Kubernetes cluster.

<figure>
<img src="../../../assets/images/admin/monitoring/grafana.png" alt="Grafana view" />
<img src="../../../../assets/images/admin/monitoring/grafana.png" alt="Grafana view" />
<figcaption>Grafana view</figcaption>
</figure>

Expand All @@ -50,4 +50,4 @@ The default dashboards are read only and cannot be edited. Additional dashboards

In this guide you learned how to access the Grafana dashboards to monitor the health and performance of the Hopsworks services.

You can find additional documentation on Grafana itself at: [https://grafana.com/docs/](https://grafana.com/docs/)
You can find additional documentation on Grafana itself at: [https://grafana.com/docs/](https://grafana.com/docs/)
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ You can access the admin page of your Hopsworks cluster by clicking on your name
You can then navigate to the _Monitoring_ tab. The _Monitoring_ tab gives you access to several of the observability tools that are already deployed to help you manage the health of the cluster.

<figure>
<img src="../../../assets/images/admin/monitoring/monitoring_tab.png" alt="monitoring tab" />
<img src="../../../../assets/images/admin/monitoring/monitoring_tab.png" alt="monitoring tab" />
<figcaption>Monitoring tab</figcaption>
</figure>

Expand All @@ -32,7 +32,7 @@ You can filter the logs of a specific service by searching for the term `service
Currently only the logs of the following services are collected and indexed: Hopsworks web application (called `domain1` in the log entries), namenodes, resource managers, datanodes, nodemanagers, Kafka brokers, Hive services and RonDB. These are the core component of the platform, additional logs will be added in the future.

<figure>
<img src="../../../assets/images/admin/monitoring/services_logs.png" alt="OpenSearch Dashboards with services logs" />
<img src="../../../../assets/images/admin/monitoring/services_logs.png" alt="OpenSearch Dashboards with services logs" />
<figcaption>OpenSearch Dashboards displaying the logs</figcaption>
</figure>

Expand Down
Loading