feat: migrate to plugins.v1 with filters & actions

This is a very large refactoring which aims at making Tutor both more
extendable and more generic. Historically, the Tutor plugin system was
designed as an ad-hoc solution to allow developers to modify their own
Open edX platforms without having to fork Tutor. The plugin API was
simple, but limited, because of its ad-hoc nature. As a consequence,
there were many things that plugin developers could not do, such as
extending different parts of the CLI or adding custom template filters.

Here, we refactor the whole codebase to make use of a generic plugin
system. This system was inspired by the Wordpress plugin API and the
Open edX "hooks and filters" API. The various components are added to a
small core thanks to a set of actions and filters. Actions are callback
functions that can be triggered at different points of the application
lifecycle. Filters are functions that modify some data. Both actions and
filters are collectively named as "hooks". Hooks can optionally be
created within a certain context, which makes it easier to keep track of
which application created which callback.

This new hooks system allows us to provide a Python API that developers
can use to extend their applications. The API reference is added to the
documentation, along with a new plugin development tutorial.

The plugin v0 API remains supported for backward compatibility of
existing plugins.

Done:
- Do not load commands from plugins which are not enabled.
- Load enabled plugins once on start.
- Implement contexts for actions and filters, which allow us to keep track of
  the source of every hook.
- Migrate patches
- Migrate commands
- Migrate plugin detection
- Migrate templates_root
- Migrate config
- Migrate template environment globals and filters
- Migrate hooks to tasks
- Generate hook documentation

Doing:
- Resolve all TODOs

TODO:
- Write a better commit message here.
- Write a plugin development tutorial.

CHANGELOG.md

@@ -4,6 +4,9 @@ Note: Breaking changes between versions are indicated by "💥".
 
 ## Unreleased
 
+<!-- TODO add an entry here... -->
+- [Improvement] Improved the output of `tutor plugins list`.
+
 ## v13.1.8 (2022-03-18)
 
 - [Bugfix] Fix "evalsymlink failure" during `k8s quickstart` (#611).

Makefile

@@ -28,6 +28,8 @@ push-pythonpackage: ## Push python package to pypi
 
 test: test-lint test-unit test-types test-format test-pythonpackage ## Run all tests by decreasing order of priority
 
+test-static: test-lint test-types test-format  ## Run only static tests
+
 test-format: ## Run code formatting tests
 	black --check --diff $(BLACK_OPTS)
 
@@ -38,7 +40,7 @@ test-unit: ## Run unit tests
 	python -m unittest discover tests
 
 test-types: ## Check type definitions
-	mypy --exclude=templates --ignore-missing-imports --strict tutor/ tests/
+	mypy --exclude=templates --ignore-missing-imports --strict ${SRC_DIRS}
 
 test-pythonpackage: build-pythonpackage ## Test that package can be uploaded to pypi
 	twine check dist/tutor-$(shell make version).tar.gz
@@ -49,6 +51,9 @@ test-k8s: ## Validate the k8s format with kubectl. Not part of the standard test
 format: ## Format code automatically
 	black $(BLACK_OPTS)
 
+isort: ##  Sort imports. This target is not mandatory because the output may be incompatible with black formatting. Provided for convenience purposes.
+	isort --skip=templates ${SRC_DIRS}
+
 bootstrap-dev: ## Install dev requirements
 	pip install .
 	pip install -r requirements/dev.txt

bin/main.py

@@ -1,25 +1,16 @@
 #!/usr/bin/env python3
-from tutor.plugins import OfficialPlugin
+from tutor import hooks
 from tutor.commands.cli import main
+from tutor.plugins.v0 import OfficialPlugin
+
+
+@hooks.actions.on(hooks.Actions.INSTALL_PLUGINS)
+def _install_official_plugins() -> None:
+    # Manually install plugins: that's because entrypoint plugins are not properly
+    # detected within the binary bundle.
+    OfficialPlugin.install_all()
 
-# Manually install plugins (this is for creating the bundle)
-for plugin_name in [
-    "android",
-    "discovery",
-    "ecommerce",
-    "forum",
-    "license",
-    "mfe",
-    "minio",
-    "notes",
-    "richie",
-    "webui",
-    "xqueue",
-]:
-    try:
-        OfficialPlugin.load(plugin_name)
-    except ImportError:
-        pass
 
 if __name__ == "__main__":
+    # Call the regular main function, which will not detect any entrypoint plugin
     main()

docs/Makefile

@@ -11,4 +11,7 @@ browse:
 	sensible-browser _build/html/index.html
 
 watch: build browse
-	while true; do inotifywait -e modify *.rst */*.rst */*/*.rst ../*.rst conf.py; $(MAKE) build || true; done
+	while true; do $(MAKE) wait-for-change build || true; done
+
+wait-for-change:
+	inotifywait -e modify $(shell find . -name "*.rst") ../*.rst ../tutor/hooks/*.py conf.py

docs/conf.py

@@ -27,6 +27,10 @@
 exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
 pygments_style = None
 
+# Autodocumentation of modules
+extensions.append("sphinx.ext.autodoc")
+autodoc_typehints = "description"
+
 # -- Sphinx-Click configuration
 # https://sphinx-click.readthedocs.io/
 extensions.append("sphinx_click")
@@ -108,5 +112,5 @@ def youtube(
     ]
 
 
-youtube.content = True
-docutils.parsers.rst.directives.register_directive("youtube", youtube)
+setattr(youtube, "content", True)
+docutils.parsers.rst.directives.register_directive("youtube", youtube)  # type: ignore

docs/index.rst

@@ -59,6 +59,6 @@ This work is licensed under the terms of the `GNU Affero General Public License
 
 The AGPL license covers the Tutor code, including the Dockerfiles, but not the content of the Docker images which can be downloaded from https://hub.docker.com. Software other than Tutor provided with the docker images retain their original license.
 
-The Tutor plugin system is licensed under the terms of the `Apache License, Version 2.0 <https://opensource.org/licenses/Apache-2.0>`__.
+The Tutor plugin and hooks system is licensed under the terms of the `Apache License, Version 2.0 <https://opensource.org/licenses/Apache-2.0>`__.
 
 © 2021 Tutor is a registered trademark of SASU NULI NULI. All Rights Reserved.

docs/intro.rst

@@ -70,6 +70,8 @@ Urls:
 
 The platform is reset every day at 9:00 AM, `Paris (France) time <https://time.is/Paris>`__, so feel free to try and break things as much as you want.
 
+.. _how_does_tutor_work:
+
 How does Tutor work?
 --------------------
 

docs/plugins.rst

@@ -12,10 +12,12 @@ Tutor comes with a plugin system that allows anyone to customise the deployment
     # 3) Reconfigure and restart the platform
     tutor local quickstart
 
-For simple changes, it may be extremely easy to create a Tutor plugin: even non-technical users may get started with :ref:`simple yaml plugins <plugins_yaml>`.
+For simple changes, it may be extremely easy to create a Tutor plugin: even non-technical users may get started with our :ref:`plugin_development_tutorial`.
 
 In the following we learn how to use and create Tutor plugins.
 
+.. TODO move this part of the docs to the tutorial?
+
 Commands
 --------
 
@@ -39,12 +41,14 @@ Existing plugins
 
 Officially-supported plugins are listed on the `Overhang.IO <https://overhang.io/tutor/plugins>`__ website.
 
-Plugin development
-------------------
+Legacy plugin v0 development
+----------------------------
+
+.. TODO put more words here
 
 .. toctree::
    :maxdepth: 2
 
-   plugins/api
-   plugins/gettingstarted
-   plugins/examples
+   plugins/v0/api
+   plugins/v0/gettingstarted
+   plugins/v0/examples

docs/reference.rst

@@ -1,13 +1,9 @@
-CLI Reference
-=============
+Reference
+=========
 
 .. toctree::
    :maxdepth: 2
 
-   reference/cli/tutor
-   reference/cli/config
-   reference/cli/dev
-   reference/cli/images
-   reference/cli/k8s
-   reference/cli/local
-   reference/cli/plugins
+   reference/api
+   reference/cli
+   reference/patches

docs/reference/api.rst

@@ -0,0 +1,14 @@
+===
+API
+===
+
+Hooks
+=====
+
+.. toctree::
+   :maxdepth: 2
+
+   api/hooks/actions
+   api/hooks/filters
+   api/hooks/contexts
+   api/hooks/consts

docs/reference/api/hooks/actions.rst

@@ -0,0 +1,8 @@
+=======
+Actions
+=======
+
+.. autofunction:: tutor.hooks.actions::on
+.. autofunction:: tutor.hooks.actions::do
+.. autofunction:: tutor.hooks.actions::clear
+.. autofunction:: tutor.hooks.actions::clear_all

docs/reference/api/hooks/consts.rst

@@ -0,0 +1,21 @@
+=========
+Constants
+=========
+
+Actions
+=======
+
+.. autoclass:: tutor.hooks.Actions
+    :members:
+
+Filters
+=======
+
+.. autoclass:: tutor.hooks.Filters
+    :members:
+
+Contexts
+========
+
+.. autoclass:: tutor.hooks.Contexts
+    :members:

docs/reference/api/hooks/contexts.rst

@@ -0,0 +1,5 @@
+========
+Contexts
+========
+
+.. autofunction:: tutor.hooks.contexts::enter

docs/reference/api/hooks/filters.rst

@@ -0,0 +1,13 @@
+.. _filters:
+
+=======
+Filters
+=======
+
+.. autofunction:: tutor.hooks.filters::add
+.. autofunction:: tutor.hooks.filters::add_item
+.. autofunction:: tutor.hooks.filters::add_items
+.. autofunction:: tutor.hooks.filters::apply
+.. autofunction:: tutor.hooks.filters::iterate
+.. autofunction:: tutor.hooks.filters::clear
+.. autofunction:: tutor.hooks.filters::clear_all

docs/reference/cli.rst

@@ -0,0 +1,13 @@
+Command line interface (CLI)
+============================
+
+.. toctree::
+   :maxdepth: 2
+
+   cli/tutor
+   cli/config
+   cli/dev
+   cli/images
+   cli/k8s
+   cli/local
+   cli/plugins

docs/reference/patches.rst

@@ -0,0 +1,13 @@
+.. _patches:
+
+================
+Template patches
+================
+
+.. TODO fill me
+
+This is a work-in-progress. For the moment, the list of patches in Tutor templates can be obtained by grepping the source code::
+
+    git clone https://github.com/overhangio/tutor
+    cd tutor
+    git grep "{{ patch" -- tutor/templates

docs/tutorials.rst

@@ -7,6 +7,7 @@ Open edX customization
 .. toctree::
    :maxdepth: 2
 
+   tutorials/plugin
    tutorials/theming
    tutorials/edx-platform-settings
    tutorials/google-smtp

docs/tutorials/plugin.rst

@@ -0,0 +1,93 @@
+.. _plugin_development_tutorial:
+
+=======================
+Creating a Tutor plugin
+=======================
+
+Tutor plugins are the offically recommended way of customizing the behaviour of Tutor. If Tutor does not do things the way you want, then your first reaction should *not* be to fork Tutor, but instead to figure out whether you can create a plugin that will allow you to achieve what you want.
+
+You may be thinking that creating a plugin might be overkill for your use case. It's almost certainly not! The stable plugin API guarantees that your changes will keep working even after you upgrade from one major release to the next, with little to no extra work. Also, it allows you to distribute your changes to other users.
+
+.. TODO reformat everything below this line
+
+::
+
+    touch "$(tutor plugins printroot)/myplugin.py"
+
+::
+
+    tutor plugins list
+
+::
+
+    myplugin    (disabled)    /home/yourusername/.local/share/tutor-plugins/myplugin.py
+
+Enable your plugin::
+
+    tutor plugins enable myplugin
+
+::
+
+    Plugin myplugin enabled
+    Configuration saved to /home/yourusername/.local/share/tutor/config.yml
+    You should now re-generate your environment with `tutor config save`.
+
+At this point you could re-generate your environment with ``tutor config save``, but there would not be any change to your environment... because the plugin does not do anything. So let's get started and make some changes.
+
+Defining patches
+================
+
+We'll start by modifying some of our Open edX settings. It's a frequent requirement to modify the ``FEATURES`` setting from the LMS or the CMS in edx-platform. In the legacy native installation, this was done by modifying the ``lms.env.yml`` and ``cms.env.yml`` files. Here we'll modify the Python setting files that define the edx-platform configuration. To achieve that we'll make use of two concepts from the Tutor API: :ref:`patches` and :ref:`filters`.
+
+If you have not already read :ref:`how_does_tutor_work` now would be a good time :-) Tutor uses templates to generate various files, such as settings, Dockerfiles, etc. These templates include ``{{ patch("patch-name") }}`` statements that allow plugins to insert arbitrary content in there. These patches are located at strategic locations. See :ref:`patches` for more information.
+
+Let's say that we would like to limit access to our brandh new Open edX platform. It is not ready for prime-time yet, so we want to prevent users from registering new accounts. There is a feature flag for that in the LMS: `FEATURES['ALLOW_PUBLIC_ACCOUNT_CREATION'] <https://edx.readthedocs.io/projects/edx-platform-technical/en/latest/featuretoggles.html#featuretoggle-FEATURES['ALLOW_PUBLIC_ACCOUNT_CREATION']>`__. By default this flag is set to a true value, enabling anyone to create an account. In the following we'll set it to false.
+
+Add the following content to the ```myplugin.py`` file that you created earlier::
+
+    from tutor import hooks
+
+    hooks.filters.add_item(
+        hooks.Filters.ENV_PATCHES % "openedx-lms-common-settings",
+        "FEATURES['ALLOW_PUBLIC_ACCOUNT_CREATION'] = False"
+    )
+
+Let's go over these changes one by one::
+
+    from tutor import hooks
+
+This imports the ``hooks`` module from Tutor, which grants us access to ``hooks.filters`` and ``hooks.Filters`` (among other things).
+
+::
+
+    hooks.filters.add_item(
+        <filter>,
+        <item>
+    )
+
+This means "add <item> to the <filter>". In our case, we want to modify the LMS settings, both in production and development. The right patch for that is ``"openedx-lms-common-settings"`` and the corresponding filter is named ``hooks.Filters.ENV_PATCHES % "openedx-lms-common-settings"``. We add one item, which is a single Python-formatted line of code::
+
+    "FEATURES['ALLOW_PUBLIC_ACCOUNT_CREATION'] = False"
+
+Notice how "False" starts with a capital "F"? That's how booleans are created in Python.
+
+Now, re-render your environment with::
+
+    tutor config save
+
+You can check that the feature was added to your environment by running::
+
+    grep -r ALLOW_PUBLIC_ACCOUNT_CREATION "$(tutor config printroot)/env"
+
+This should print::
+
+    /home/yourusername/.local/share/tutor/env/apps/openedx/settings/lms/production.py:FEATURES['ALLOW_PUBLIC_ACCOUNT_CREATION'] = False
+    /home/yourusername/.local/share/tutor/env/apps/openedx/settings/lms/development.py:FEATURES['ALLOW_PUBLIC_ACCOUNT_CREATION'] = False
+
+Your new settings will be taken into account by restarting your platform::
+
+    tutor local restart
+
+Congratulations! You've made your first working plugins. As you can guess, you can add changes to other files by adding other similar patch statements to your plugin.
+
+.. TODO to be continued

requirements/base.txt

@@ -26,7 +26,7 @@ kubernetes==18.20.0
     # via -r requirements/base.in
 markupsafe==2.0.1
     # via jinja2
-mypy==0.931
+mypy==0.941
     # via -r requirements/base.in
 mypy-extensions==0.4.3
     # via mypy

requirements/dev.in

@@ -7,5 +7,6 @@ twine
 coverage
 
 # Types packages
+types-docutils
 types-PyYAML
 types-setuptools