Redesign workflow engine and provisioner API #352
Conversation
Can you elaborate on this? |
|
The API change will affect what recovery information is stored in the database, so any workflow that attempted to be recovered when this feature is first deployed will fail. |
|
OK, but after that recovery will work? |
|
Yup. Maybe I should include a migration that drops any running workflows. |
plugin-guide.md
Outdated
| The class `BaseJsonInputProvisioner` is a partial implementation that can store | ||
| crash recovery information in a JSON object of the implementor's choosing, | ||
| making recovery easier. | ||
| This plugin type uses the operations API. |
There was a problem hiding this comment.
Can these be a link to the section? if markdown can do that?
apmasell
force-pushed
the
monad_async
branch
from
1f59193 to
00bd19c
Compare
plugin-guide.md
Outdated
| required. This primes the sequence of events by computing a value from the | ||
| state information alone which will be the input for the first step. After this, | ||
| `then` may be called to manipulate this value. Additionally, there are | ||
| convinced methods `map`, to modify the current value, and `reload`, to discard |
There was a problem hiding this comment.
convenience. Fixed.
plugin-guide.md
Outdated
| @@ -174,17 +168,15 @@ which ones are allowed via the `supports` method. | |||
| The workflow engine will be given the complete input to the workflow (with real | |||
| paths provided by the input provisioners) and the workflow itself. Once the | |||
| workflow has completed, it must provide a JSON structure that references the | |||
| output of the workflow. Vidarr will identified the output files generated by | |||
| output of the workflow. Víðarr will identified the output files generated by | |||
| .then( | ||
| log( | ||
| Level.INFO, | ||
| id -> String.format("Started Cromwell workflow %s on %s", id, cromwellUrl))) |
There was a problem hiding this comment.
Is there a reason we're not preserving the logging that indicates this is specifically the provision-out job? I know that's technically just another cromwell workflow but it was clearer before imo
There was a problem hiding this comment.
Mostly due to copy-paste. I'll fix that. Also, the logs get prefixed with context, so it wouldn't be hard to tell.
apmasell
force-pushed
the
monad_async
branch
2 times, most recently
from
1f9e831 to
5f0d855
Compare
|
Also, the build issue seems to be a rebasing problem. |
| * Recover and operation from the database and discard and error conditions and allow it to start | ||
| * again. | ||
| * | ||
| * @param state the |
There was a problem hiding this comment.
Drop the the. It's cleaner.
| @@ -24,9 +26,31 @@ | |||
| /** Defines an engine that knows how to execute workflows and track the results */ | |||
| @JsonTypeIdResolver(WorkflowEngine.WorkflowEngineIdResolver.class) | |||
| @JsonTypeInfo(use = JsonTypeInfo.Id.CUSTOM, include = As.PROPERTY, property = "type") | |||
| public interface WorkflowEngine { | |||
| public interface WorkflowEngine<S extends Record, C extends Record> { | |||
There was a problem hiding this comment.
Renamed for clarity and added JavaDoc.
apmasell
force-pushed
the
monad_async
branch
2 times, most recently
from
1a1fb41 to
22b7d9d
Compare
| @@ -0,0 +1,6 @@ | |||
| DELETE FROM active_operation WHERE workflow_run_id IN (SELECT active_workflow_run.id FROM active_workflow_run); | |||
| DELETE FROM analysis_external_id WHERE analysis_external_id.external_id_id IN (SELECT external_id.id FROM external_id WHERE external_id.workflow_run_id IN (SELECT active_workflow_run.id FROM active_workflow_run)); | |||
| DELETE FROM external_id WHERE external_id.workflow_run_id IN (SELECT active_workflow_run.id FROM active_workflow_run); | |||
There was a problem hiding this comment.
ERROR: update or delete on table "external_id" violates foreign key constraint "external_id_version_external_id_id_fkey" on table "external_id_version"
This replaces the call-back based `WorkMonitor` with the fluent monad `OperationAction` API. This change will prevent recovery of any inflight operations.
This replaces the call-back based `WorkMonitor` with the fluent monad `OperationAction` API. This change will prevent recovery of any inflight operations.
This replaces the call-back based `WorkMonitor` with the fluent monad `OperationAction` API. This change will prevent recovery of any inflight operations.
This replaces the call-back based
WorkMonitorwith the fluent monadOperationActionAPI. This change will prevent recovery of any inflight operations.Jira ticket: