Monitor failed job runs

[ljwoodley]

Possible solution to issue #100?

Note, this only sends an email alert with the attached log on job failure. I figured since successful runs are logged in the db we wouldn't need to create a log file for that.

[pbchase]

This is super simple and it works. We could use it as-is, but I'd like to see two more features. Could you add these?

1. Add the ability to pass optional arguments to an etl.
2. implement the same functionality in the existing render_report.R and its copy in the study_template

[pbchase]

Could you allow for optional parameters? it's quite rare but we do pass an argument to one of out ETLs in the rcc.ctsit repo

[pbchase]

...and then use the arguments here?

[pbchase]

Here's some example code I wrote to test a different way of specifying the optional arguments to the script called by run_etl.R

library(argparse)

parser <- ArgumentParser()

parser$add_argument("file", nargs = 1, help = "RScript to be run")
parser$add_argument("optional_args", nargs='+', help = "optional arguments to the script to be run")

args <- parser$parse_args()

paste0("The first argument, file, has a value of '", args$file, "'")
paste0("The optional arguments that follow are '", paste(args$optional_args, collapse = " "), "'")

Could you revise run_etl.R to use use this model for optional arguments?

[ljwoodley]

Here's some example code I wrote to test a different way of specifying the optional arguments to the script called by run_etl.R

library(argparse)

parser <- ArgumentParser()

parser$add_argument("file", nargs = 1, help = "RScript to be run")
parser$add_argument("optional_args", nargs='+', help = "optional arguments to the script to be run")

args <- parser$parse_args()

paste0("The first argument, file, has a value of '", args$file, "'")
paste0("The optional arguments that follow are '", paste(args$optional_args, collapse = " "), "'")

Could you revise run_etl.R to use use this model for optional arguments?

I don't understand this change?

parser$add_argument("optional_args", nargs='+', help = "optional arguments to the script to be run") does not allow for an optional argumnet as it requires at least one value when the script runs due to nargs='+'.

Rscript etl/run_etl.R etl/test_failure_alert.R fails with this error

Also, Rscript etl/run_etl.R etl/test_failure_alert.R arg1 arg2 is not parsing the arguments properly.

@pbchase

[pbchase]

I made some suggestions.

We'll make changes to render_report.R later after the rcc.grace PR is finished.

[pbchase]

Suggested change

	parser$add_argument("--optional_args", nargs='*', help="Zero or more optional arguments of any type")
	if (!interactive()) {
	args <- parser$parse_args()
	} else {
	args <- parser$parse_args(
	c(
	"study_template/etl/test_failure_alert.R",
	"--optional_args",
	"test",
	parser$add_argument("optional_args", nargs='*', help="Zero or more optional arguments of any type")
	if (!interactive()) {
	args <- parser$parse_args()
	} else {
	args <- parser$parse_args(
	c(
	"study_template/etl/test_failure_alert.R",
	"test",

[pbchase]

Suggested change

	# Test email alert on script failure
	0 0 1 * * root /usr/bin/docker run --rm --env-file /rcc/study_template/.env rcc.site Rscript etl/run_etl.R etl/test_failure_alert.R --optional_args test "another test"
	# Test email alert on script failure
	0 0 1 * * root /usr/bin/docker run --rm --env-file /rcc/study_template/.env rcc.site Rscript etl/run_etl.R etl/test_failure_alert.R test "another test"

[pbchase]

Suggested change

	parser$add_argument("--optional_args", nargs='*', help="Zero or more optional arguments of any type")
	if (!interactive()) {
	args <- parser$parse_args()
	} else {
	args <- parser$parse_args(
	c(
	"study_template/etl/test_failure_alert.R",
	"--optional_args",
	parser$add_argument("optional_args", nargs='*', help="Zero or more optional arguments of any type")
	if (!interactive()) {
	args <- parser$parse_args()
	} else {
	args <- parser$parse_args(
	c(
	"study_template/etl/test_failure_alert.R",