B-21322 TPPS ECS Scheduled Task - MAIN

[traskowskycaci]

POA Received to merge into staging and production environment, see Agility ticket comments

Agility ticket

INT PRs:

• Initial int PR
  - There will be some int specific commits in that PR, due to deploying that specific int branch to exp for testing. I tried to remember to throw "int specific commit" in front of those but could've missed some.
• Removed RECEIVED_BY_GEX status enum
• Added loadtest config file
• Updated move history labels
• Added stg/prd env script updates

https://dp3.atlassian.net/wiki/spaces/MT/pages/1469284485/How+to+create+an+ECS+Scheduled+Task

Agility ticket

Summary

The responsibility of this scheduled task is to grab a TPPS payment file from an S3 bucket, check its ClamAV status to ensure it's CLEAN, and if CLEAN, process the file line by line and store it in our database. Each line in the payment file has an invoice number that will match up to a payment request number.

If no matching payment request number is found, or some error with one line is found, we will continue to process the rest of the entries in the file so that one bad row doesn't keep the rest of the good data in that file from being processed.

The TPPS lambda, which is the service responsible for putting the payment files into the S3 bucket for this new task service to process, should run at 1 a.m. central time. Then at 2 a.m. central time, this task will run once, processing the payment file for the previous day's date.

AWS Config

S3 permissions needed in each environment for this task:

- ListBucket
- GetObject
- GetDataAccess

Parameters added for this work to the AWS parameter store:

• Please check that these parameters have been added to the stg and prd environments. The below links are for the exp env

1. /app-exp/process_tpps_custom_date_file: Parameter than can be modified if we want to process a custom dated TPPS payment file as opposed to the scheduled daily run where we process the previous day's payment file

2. /app-exp/tpps_s3_bucket: S3 bucket path for the TPPS payment files
   

3. /app-exp/tpps_s3_folder: S3 subfolder inside bucket path for the TPPS payment files

Next steps after testing in exp

Higher environment to-dos after this works in experimental:

• Make sure S3 bucket is created and pulled in files on the daily run of the lambda
• Check that the retention policy is set to 30 days for the TPPS files (see proof of 30 day retention policy for exp below)
• Add the parameter store values
• Set both of those as environment variables keys in the ecs task
• Add this to .gitlab-ci.yml under deploy_tasks_stg and deploy_tasks_prd (was already added to deploy_tasks_dp3 for exp work):

- echo "Deploying process TPPS task service"
- ./scripts/ecs-deploy-task-container process-tpps "${ECR_REPOSITORY_URI}/app-tasks@${ECR_DIGEST}" "${APP_ENVIRONMENT}"

• add this to config.yml under deploy_tasks_steps (was already added to deploy_dp3_tasks_steps for exp work):

- run:
          name: Deploy process TPPS files service
          command: scripts/do-exclusively --job-name ${CIRCLE_JOB} scripts/ecs-deploy-task-container process-tpps "${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/app-tasks@${ECR_DIGEST}" "${APP_ENVIRONMENT}"
          no_output_timeout: 20m
      - announce_failure

• add this to Dockerfile.tasks ( was already added to Dockerfile.tasks_dp3):

# Mount mutable tmp for process-tpps
# hadolint ignore=DL3007
VOLUME ["/tmp"]

Bright Stars work that enables the work here:

In the terraform-aws-app-environment repo:

• https://github.com/transcom/terraform-aws-app-environment/pull/517

In the transcom-infrasec-gov-nonato repo:

• https://github.com/transcom/transcom-infrasec-gov-nonato/pull/1256
• https://github.com/transcom/transcom-infrasec-gov-nonato/pull/1257

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[cameroncaci]

POA'd, git history clean and matches int commits - kudos to being able to maintain that history over so many branches. What was the plan for this one getting tested overnight?

[traskowskycaci]

POA'd, git history clean and matches int commits - kudos to being able to maintain that history over so many branches. What was the plan for this one getting tested overnight?

I shall coordinate with Mr. @deandreJones 👀

[traskowskycaci]

POA'd, git history clean and matches int commits - kudos to being able to maintain that history over so many branches. What was the plan for this one getting tested overnight?

I shall coordinate with Mr. @deandreJones 👀

And also the Bright Stars folks, I think they've got a few things to check yet

[cameroncaci]

Leaving my approval here, but note to other codeowners, don't merge this one until main is locked and ready for tpps testing in prod (params all set, etc.)

[traskowskycaci]

Parameters added to stg:

1. /app-stg/process_tpps_custom_date_file: Parameter than can be modified if we want to process a custom dated TPPS payment file as opposed to the scheduled daily run where we process the previous day's payment file

Parameters not yet added to stg:
2. /app-stg/tpps_s3_bucket
3. /app-stg/tpps_s3_folder

Can't see for sure but would assume these are also not in prod yet - will update when I know the needed stg and prd env vars have been added

[traskowskycaci]

All parameters needed have been added to stg:

1. /app-stg/process_tpps_custom_date_file
2. /app-stg/tpps_s3_bucket
3. /app-stg/tpps_s3_folder

[traskowskycaci]

All parameters needed have been added to prd

• process_tpps_custom_date_file
• tpps_s3_bucket
• tpps_s3_folder

[traskowskycaci]

Merged into main, removing do not merge label