|
| 1 | +# Fastq Sync Service |
| 2 | + |
| 3 | +The fastq sync service is a simple service that allows step functions with task tokens to 'hang' |
| 4 | +until the requirements of either a fastq list row or fastq set have been met. |
| 5 | + |
| 6 | +This is useful for workflow-glue services that have fastq set ids but need to wait for either |
| 7 | + |
| 8 | +1. The fastq set readsets to be created |
| 9 | +2. The fastq set to have been qc'd AND have a fingerprint file and compression information |
| 10 | +3. This is also useful for data sharing services that require the fastqs to be unarchived before they can be shared |
| 11 | + |
| 12 | +The step function will then hang at that step until the task token has been 'unlocked' |
| 13 | + |
| 14 | +## Registering task tokens |
| 15 | + |
| 16 | +Workflow glue services can use the fastq sync service by generating the following event |
| 17 | + |
| 18 | +```json5 |
| 19 | +{ |
| 20 | + "EventBusName": "OrcaBusMain", |
| 21 | + "Source": "doesnt matter", |
| 22 | + "DetailType": "FastqSync", |
| 23 | + "Detail": { |
| 24 | + "taskToken": "uuid", |
| 25 | + "fastqSetId": "fqs.123456", |
| 26 | + // Then one or more of the following |
| 27 | + // Requirements can be left out if not needed |
| 28 | + // Do all fastq list rows in the set contain readsets? |
| 29 | + "hasReadsets": true, |
| 30 | + // Do all fastq list rows in the set contain an ntsm uri? |
| 31 | + "hasFingerprints": true, |
| 32 | + // Do all fastq list rows in the set contain compression information? |
| 33 | + // Useful if the fastq list rows are in ora format. |
| 34 | + // Some pipelines require the gzip file size in bytes in order |
| 35 | + // to stream the gzip file from ora back into s3 |
| 36 | + "hasCompressionInformation": true, |
| 37 | + // Do all fastq list rows in the set contain qc information? |
| 38 | + // We don't use this for anything yet but we may use this in the future |
| 39 | + // to ensure that a fastq set has met the ideal coverage levels |
| 40 | + "hasQc": true, |
| 41 | + } |
| 42 | +} |
| 43 | +``` |
| 44 | + |
| 45 | +The fastq sync service will also trigger the qc, fingerprint or compression information services if they do not exist. |
| 46 | + |
| 47 | +If any of the fastq list rows are in archive, the fastq sync service will also trigger the archive service to unarchive these fastq list rows. |
| 48 | + |
| 49 | +## Unlocking task tokens |
| 50 | + |
| 51 | +The fastq sync service will listen for the following events: |
| 52 | + |
| 53 | +1. FastqListRowUpdated (from the fastq management service) |
| 54 | +2. FastqSetUpdated (from the fastq management service) |
| 55 | +3. UnarchivingCompleted (from the fastq unarchiving service) |
| 56 | + |
| 57 | +The fastq sync service will then check against the requirements of the fastq set or fastq list row for each task token and if so, unlock the task token. |
| 58 | + |
0 commit comments