Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

draft: feat: move plugin CLIs under dev/local/k8s; let plugins run jobs #567

Closed
wants to merge 1 commit into from

Conversation

kdmccormick
Copy link
Collaborator

@kdmccormick kdmccormick commented Jan 14, 2022

Description

This is a potentially-overzealous yet backwards-compatible change to the Plugin API and Tutor Plugin CLI. Check out the commit message for the full description.

Why?

In the tutor-coursegraph plugin I'm developing, I want to be able to run a job against a service container from the plugin's custom CLI. As far as I could tell, there was no way to do that without this change.

If there's a way to do what I'm trying to achieve without this tutor PR, do enlighten me!

Related tutor-coursegraph issue: openedx-unsupported/tutor-contrib-coursegraph#5

Status

I'm not trying to get this reviewed and merged imminently; I'd like to go over the implementation with @regisb first.

Among other things, this PR still needs a couple unit tests and some documentation updates.

Other information

Part of openedx/platform-roadmap#11

Testing instructions

Test instructions TBD.

I have manually tested the dev version of this but haven't tested out local or k8s.

Given some plugin "myplugin"...

Formerly, myplugin's CLI was only accessible directly under
the `tutor` command:

  tutor myplugin dosomething

However, this meant that the plugin's custom CLI was unaware
of whether the plugin was running dev, local, or k8s mode.
Because of that, plugins could not easily run jobs (commands against
services) because the click `context.obj` variable did not expose a
`.job_runner` method.

This commit modifies Tutor to allow plugin CLIs to be accessed as
subcommands of `tutor dev/local/k8s`, like so:

  tutor dev myplugin dosomething

Furthermore, the commit refactors `context.obj` to expose a ready-
to-use, configuration-included `job_runner()` method. So,
in the definition of myplugin, the `dosomething` subcommand may now
run jobs from templates:

  @click.command(help="do something against the mypluginservice")
  @click.pass_obj
  def dosomething(context):
      context.obj.job_runner().run_job_from_template(
          "mypluginservice", "myplugin", "apps", "dosomething.sh"
      )

Plugins commands are still invokable directly from the tutor level
(`tutor myplugin ...`) for backwards-compatibility; however, if
the invoked command attempts to run a job, then Tutor will raise an
error explaining that the provided command must be run from within
a context (k8s/local/dev).


@click.group(help="Run Open edX locally with docker-compose")
@click.pass_context
def local(context: click.Context) -> None:
context.obj = LocalContext(context.obj.root)
context.obj = LocalContext(context.obj.root, tutor_config.load(context.obj.root))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line will fail the first time someone runs tutor local quickstart with an empty project root.

@regisb
Copy link
Contributor

regisb commented Jan 19, 2022

Hey Kyle! I believe that I understand the changes in this PR, but I'm not sure what new feature they are supposed to introduce?

Usually, when I need to run a specific job for a plugin, I just run tutor local init --limit=myplugin. This is overzealous, as it will apply all initialisation scripts, and not just a specific one. If we really need to run a small command frequently in one of the services, then there are two options:

  1. Run a container that will run a cron job (or something similar). I'm really not a big fan of cron tasks, as they are frequently anti-patterns, but sometimes there is no way around it.
  2. Add a convenient CLI utility to the plugin image, such that people can run tutor local run myservice myservice-ctl ....

@kdmccormick
Copy link
Collaborator Author

kdmccormick commented Jan 20, 2022

Add a convenient CLI utility to the plugin image, such that people can run tutor local run myservice myservice-ctl ....

@regisb This almost gets me what I'm looking for, but I am still not sure how a plugin could expose a convenient CLI for running commands within a different service.

Taking this back to the specific CourseGraph use case: CourseGraph users need to be able to manually call a CMS management command. Users could type out the long management command by hand:

tutor local run cms ./manage.py cms dump_to_neo4j --courses ... --host localhost --user neo4j --password openedx ....etc

but I think it'd be a better experience if they could just type:

tutor local coursegraph refresh

or, as you suggested:

tutor local run cms refresh-coursegraph

But, to be able to run tutor local run cms refresh-coursegraph, the CourseGraph plugin would need to add the refresh-coursegraph script to the CMS container's filesystem, which I am not sure how to do.

(I guess this wouldn't be necessary if ./manage.py cms dump_to_neo4j had a more convenient interface, would it? Perhaps scenarios like this would best be solved at the app level instead of the plugin level?)

@regisb
Copy link
Contributor

regisb commented Jan 20, 2022

(I guess this wouldn't be necessary if ./manage.py cms dump_to_neo4j had a more convenient interface, would it? Perhaps scenarios like this would best be solved at the app level instead of the plugin level?)

I think that this is the "right" solution. If a CLI tool does not prove convenient for end users, then that tool should be modified.

@kdmccormick
Copy link
Collaborator Author

Makes sense to me.

@kdmccormick kdmccormick deleted the plugin-jobs branch January 20, 2022 13:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants