Merge branch 'dev'

CHANGELOG.md

@@ -1,9 +1,34 @@
 # Changelog
 
+## Next release (dev branch)
+
+## Ananke 0.6
+
+* New features:
+  * install script for [jupyter_scheduler](https://github.com/jupyter-server/jupyter-scheduler)
+  * [jupyter-resource-usage](https://github.com/jupyter-server/jupyter-resource-usage)
+  * [Jupyter RTC](https://github.com/jupyterlab/jupyter-collaboration) supports [server-side execution](https://github.com/datalayer/jupyter-server-nbmodel) (see [Update from Ananke 0.5 to Ananke 0.6](https://gauss.whz.de/ananke/doc/container-admins.html#update-to-0_6) for important hints before updating your Ananke containers)
+* Documentation improvements:
+  * hints on batch autograding with Nbgrader
+  * alternative method for enlarging container admins' home directories
+  * more realistic test environment for development
+  * connect to LMS via HTTPS with enterprise root CA
+* Bugfixes:
+  * increase PID limit in container templates
+  * avoid `:` in nbgrader feedback paths (timestamps!) to be compatible with jupyter-fs
+  * downgrade jupyter-collaboration to stable release (instead of beta)
+* Update all packages:
+  * JupyterHub 5.2.1
+  * JupyterLab 4.3.5
+  * Notebook 7.3.2
+  * Nbgrader 9.5
+  * Python 3.13.1
+
 ## Ananke 0.5
+
 * New features:
   * **BREAKING CHANGE** - LTI/Nbgrader course and grades management is now accessible through the 'Kore' menu item in the instructor's JupyterLab (see [kore-extension](https://github.com/marcwit/kore-extension)). This replaces the former Kore service GUI. See [Update form Ananke 0.4 to Ananke 0.5](https://gauss.whz.de/ananke/doc/container-admins.html#update-to-0_5) for hints on updating your installation.
-  * **BREAKING CHANGE** - Images and Containers now are managed by one Python script instead of several bash scripts. Directory structure changed, too. See [Update form Ananke 0.4 to Ananke 0.5](https://gauss.whz.de/ananke/doc/container-admins.html#update-to-0_5) for hints on updating your installation.
+  * **BREAKING CHANGE** - Images and Containers now are managed by one Python script instead of several bash scripts. Directory structure changed, too. See [Update from Ananke 0.4 to Ananke 0.5](https://gauss.whz.de/ananke/doc/container-admins.html#update-to-0_5) for hints on updating your installation.
   * [jupyterlab-execute-time extension](https://github.com/deshaw/jupyterlab-execute-time) is activated in all JupyterLabs by default. 
 * Minor additions and corrections to doc.
 * Bugfixes:

ananke

@@ -227,6 +227,7 @@ def ask_int(low: int, high: int, default: int) -> int:
     int
         Chosen integer.
     """
+
     range_text = f'{low if low is not None else "-∞"}...{high if high is not None else "+∞"}'
 
     while True:
@@ -424,12 +425,18 @@ def subcmd_create(args):
                 if not choice:
                     break
                 else:
+                    choice_valid = True
+
                     for gpu in choice.split(','):
                         gpu = gpu.strip()
                         if gpu in gpus:
                             chosen_gpus.append(gpu)
                         else:
                             print(f'GPU name "{gpu}" invalid!')
+                            choice_valid = False
+
+                    if choice_valid:
+                        break
 
             for gpu in chosen_gpus:
                 gpu_devices.append(f'nvidia.com/gpu={gpu}')
@@ -483,7 +490,7 @@ def subcmd_create(args):
         return
 
     # create systemd unit
-    cmd = f'podman generate systemd --restart-policy=always --files --start-timeout=60 --stop-timeout=60'
+    cmd = f'podman generate systemd --restart-policy=always --files --start-timeout=120 --stop-timeout=120 --restart-sec=10'
     if 'requires' in config:
         for unit in config['requires']:
             cmd += f' --after={unit} --requires={unit}'

containers/template-base-forget/config.py

@@ -16,7 +16,9 @@
 config['gpus'] = 'interactive'
 
 # additional arguments to Podman (list of strings)
-config['podman_args'] = []
+config['podman_args'] = [
+    '--pids-limit -1'  # no PID limit (Podman default is 2048, which is too small)
+]
 
 # host directories accessible inside the container (list of (host_path, container_path) tuples)
 # Paths have to be absolute or relative to config file path.

containers/template-base/config.py

@@ -16,7 +16,9 @@
 config['gpus'] = 'interactive'
 
 # additional arguments to Podman (list of strings)
-config['podman_args'] = []
+config['podman_args'] = [
+    '--pids-limit -1'  # no PID limit (Podman default is 2048, which is too small)
+]
 
 # host directories accessible inside the container (list of (host_path, container_path) tuples)
 # Paths have to be absolute or relative to config file path.

containers/template-nbgrader/config.py

@@ -16,7 +16,9 @@
 config['gpus'] = 'interactive'
 
 # additional arguments to Podman (list of strings)
-config['podman_args'] = []
+config['podman_args'] = [
+    '--pids-limit -1'  # no PID limit (Podman default is 2048, which is too small)
+]
 
 # host directories accessible inside the container (list of (host_path, container_path) tuples)
 # Paths have to be absolute or relative to config file path.

doc/src/conf.py

@@ -4,10 +4,10 @@
 # https://www.sphinx-doc.org/en/master/usage/configuration.html
 
 project = 'Ananke'
-copyright = '2024, Ananke Team'
+copyright = '2025, Ananke Team'
 author = 'Ananke Team'
 version = ''
-release = '0.5'
+release = '0.6'
 
 extensions = ['myst_parser']
 myst_heading_anchors = 4

doc/src/container-admins.md

@@ -266,7 +266,7 @@ Note that the value for `c.LTI13Authenticator.client_id` has to be a list of str
 Don't abuse `30_lms.py` for other configuration purposes than the described LTI configuration.
 This may lead to unexpected behavior.
 
-(lit-lms)=
+(lti-lms)=
 #### LMS
 
 For your LMS you need the following configuration information (field names are taken from Moodle here and may be slightly different in other LMS):
@@ -283,6 +283,22 @@ Thus, in Moodle only `Existing window` works.
 Even `new window` is not possible due to it's implementation in Moodle via embedding techniques.
 ```
 
+(container-admins-enterprise-ca)=
+#### HTTPS with enterprise root CA or self-signed cert
+
+If JupyterHub shall connect to your LMS via HTTPS with a cert issued by an enterprise root CA, you have to install the CA's root cert in the Ananke container:
+1. On the host machine copy the cert file to your container's `jupyterhub_config.d` directory.
+2. In the container's root shell run
+   ```
+   mv /opt/conda/envs/jhub/etc/jupyterhub/jupyterhub_config.d/YOUR_CERT_FILE /usr/local/share/ca-certificates/
+   update-ca-certificates
+   ```
+3. Check that the output contains `1 added`.
+
+```{important}
+JupyterHub refuses to connect to servers via HTTPS if the cert is self-signed. Thus, if you want or have to use a self-signed cert for your LMS, you have to create a custom root CA and issue your own certs with that CA. See [documentation for developers](developers.md), where the process of creating a custom CA and issuing certs is described for setting up the development environment.
+```
+
 ### Hub admins
 
 To give a hub user admin privileges inside the hub (see [For hub admins](hub-admins.md)), get the user's username (from URL `.../user/USERNAME/...` when user visits the hub) and write it to `ananke/containers/my-hub/jupyterhub_config.d/20_users.py`:
@@ -413,6 +429,20 @@ Alternatively to in-container updates yoyu may replace your container by a new o
 
 Remember to back up your user's home directories and modifications you made to the container (Python environments, ...).
 
+(update-to-0_6)=
+### Update from Ananke 0.5 to Ananke 0.6
+
+If you use Jupyter RTC, rename your containers RTC config file (usually `80_rtc.py`) to something like `80_rtc.py.disabled` before creating the new container. After running the RTC install script in the new container, reset the file to the original name and restart the hub. Without this procedure some new RTC features (server-side execution) will not work, because corresponding lab extensions won't get enabled.
+
+If your hub has many users (or at least many notebooks opened in parallel) the container may hit its default PID limit. Ananke 0.6 adds
+```
+# additional arguments to Podman (list of strings)
+config['podman_args'] = [
+    '--pids-limit -1'  # no PID limit (Podman default is 2048, which is too small)
+]
+```
+to all container definition templates (in `config.py`) to allow for an unlimited number of processes (might be a security issue depending on your environment). If you reuse you Ananke 0.5 container definition you have to add this option manually.
+
 (update-to-0_5)=
 ### Update from Ananke 0.4 to Ananke 0.5
 
@@ -583,9 +613,17 @@ To install TensorFlow run `/opt/install/tensorflow.sh` in the container's root s
 The install script also runs some TensorFlow commands to test the installation. Carefully check the output for errors.
 
 ```{important}
-TensorFlow 2.17 does not have NumPy 2 support. The install script will downgrade NumPy to 1.26.4!
+TensorFlow 2.18 does not have Python 3.13 support. The install script will downgrade Python to 3.11.9!
+```
+
+```{important}
+TensorFlow 2.18 does not have NumPy 2.2.3 support. The install script will downgrade NumPy to 2.0.2!
 ```
 
 #### Assign GPUs to users
 
 Every hub user has access to all GPUs. How to confine a user's TensorFlow commands to a subset of GPUs is described in [TensorFlow and GPUs](hub-users.md#tensorflow-and-gpus).
+
+### Job scheduling in JupyterLab
+
+To allow users to schedul jobs based on notebooks in JupyterLab install [jupyter_scheduler](https://github.com/jupyter-server/jupyter-scheduler) by running `/opt/install/scheduler.sh` in the container's root shell and restart all user servers.