Skip to content

Commit 00ea9fd

Browse files
committed
Initial commit of readme for Fastq Sync service
Allows other services' step functions to 'hang' on a put event task until the said fastq set has become available.
1 parent e9d6b60 commit 00ea9fd

File tree

1 file changed

+58
-0
lines changed
  • lib/workload/stateless/stacks/fastq-sync

1 file changed

+58
-0
lines changed
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
# Fastq Sync Service
2+
3+
The fastq sync service is a simple service that allows step functions with task tokens to 'hang'
4+
until the requirements of either a fastq list row or fastq set have been met.
5+
6+
This is useful for workflow-glue services that have fastq set ids but need to wait for either
7+
8+
1. The fastq set readsets to be created
9+
2. The fastq set to have been qc'd AND have a fingerprint file and compression information
10+
3. This is also useful for data sharing services that require the fastqs to be unarchived before they can be shared
11+
12+
The step function will then hang at that step until the task token has been 'unlocked'
13+
14+
## Registering task tokens
15+
16+
Workflow glue services can use the fastq sync service by generating the following event
17+
18+
```json5
19+
{
20+
"EventBusName": "OrcaBusMain",
21+
"Source": "doesnt matter",
22+
"DetailType": "FastqSync",
23+
"Detail": {
24+
"taskToken": "uuid",
25+
"fastqSetId": "fqs.123456",
26+
// Then one or more of the following
27+
// Requirements can be left out if not needed
28+
// Do all fastq list rows in the set contain readsets?
29+
"hasReadsets": true,
30+
// Do all fastq list rows in the set contain an ntsm uri?
31+
"hasFingerprints": true,
32+
// Do all fastq list rows in the set contain compression information?
33+
// Useful if the fastq list rows are in ora format.
34+
// Some pipelines require the gzip file size in bytes in order
35+
// to stream the gzip file from ora back into s3
36+
"hasCompressionInformation": true,
37+
// Do all fastq list rows in the set contain qc information?
38+
// We don't use this for anything yet but we may use this in the future
39+
// to ensure that a fastq set has met the ideal coverage levels
40+
"hasQc": true,
41+
}
42+
}
43+
```
44+
45+
The fastq sync service will also trigger the qc, fingerprint or compression information services if they do not exist.
46+
47+
If any of the fastq list rows are in archive, the fastq sync service will also trigger the archive service to unarchive these fastq list rows.
48+
49+
## Unlocking task tokens
50+
51+
The fastq sync service will listen for the following events:
52+
53+
1. FastqListRowUpdated (from the fastq management service)
54+
2. FastqSetUpdated (from the fastq management service)
55+
3. UnarchivingCompleted (from the fastq unarchiving service)
56+
57+
The fastq sync service will then check against the requirements of the fastq set or fastq list row for each task token and if so, unlock the task token.
58+

0 commit comments

Comments
 (0)