Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

B-21322 TPPS ECS Scheduled Task - MAIN #14233

Merged
merged 135 commits into from
Feb 27, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
135 commits
Select commit Hold shift + click to select a range
ad2c048
cleanup from older tpps feature work
traskowskycaci Nov 15, 2024
0a6a812
cli works to run but needs more
traskowskycaci Nov 18, 2024
a4920f7
add step to deploy process-tpps task
traskowskycaci Nov 18, 2024
900a116
additional steps for deploying process-tpps task based on https://git…
traskowskycaci Nov 18, 2024
9893072
cleanup and some fine-tuning
traskowskycaci Nov 19, 2024
ce0b61c
rename process-tpps to process_tpps
traskowskycaci Nov 19, 2024
c4299cd
Merge branch 'main' into B-21322-MAIN
traskowskycaci Jan 6, 2025
b8a21f1
environment variable updates for pickup dir
traskowskycaci Jan 6, 2025
136b023
return out of processing if no valid filepath found
traskowskycaci Jan 7, 2025
0066beb
add temp logging for s3 path
traskowskycaci Jan 7, 2025
5828df2
circle config for demo
traskowskycaci Jan 7, 2025
437f3e3
Merge branch 'main' into B-21322-MAIN
traskowskycaci Jan 7, 2025
3046c07
add demo env process tips
traskowskycaci Jan 7, 2025
bc7ac26
release demo env
traskowskycaci Jan 8, 2025
27adca0
deploy to exp
traskowskycaci Jan 8, 2025
0c744e8
release exp
traskowskycaci Jan 8, 2025
9d7e15a
deploy to exp
traskowskycaci Jan 9, 2025
d48b52d
release exp
traskowskycaci Jan 9, 2025
eb37ded
Merge branch 'main' into B-21322-MAIN
traskowskycaci Jan 14, 2025
d4070bb
update filepath and deploy_dp3_tasks
traskowskycaci Jan 15, 2025
3fdad8f
deploy to exp
traskowskycaci Jan 15, 2025
76a9742
release exp
traskowskycaci Jan 15, 2025
87213e7
comment out some things in config for now
traskowskycaci Jan 15, 2025
a8af25b
add some temp logging to ecs deploy script for debugging
traskowskycaci Jan 17, 2025
40e70e1
deploy to exp
traskowskycaci Jan 17, 2025
a329f8d
add process-edis back in to deploy_dp3_tasks_steps
traskowskycaci Jan 17, 2025
44d0238
remove temp logging
traskowskycaci Jan 17, 2025
07f3d70
release exp
traskowskycaci Jan 17, 2025
28e11d1
init process tpps flags before adding command
traskowskycaci Jan 20, 2025
622b671
add pseudocode plan for processing specific filenames
traskowskycaci Jan 20, 2025
3f5e66d
get the filepath to process in process_tpps.go to pass filepath to Pr…
traskowskycaci Jan 20, 2025
65399c1
add logging to process_tpps.go and update SilenceUsage to false for p…
traskowskycaci Jan 20, 2025
55f7530
deploy to exp
traskowskycaci Jan 20, 2025
7e98df4
release exp
traskowskycaci Jan 20, 2025
4c4b08a
add logging for env var troubleshooting
traskowskycaci Jan 22, 2025
9f3b472
deploy to exp
traskowskycaci Jan 22, 2025
8eff3b4
swap order for db config call
traskowskycaci Jan 22, 2025
e00a34c
db init updates
traskowskycaci Jan 22, 2025
e69e4e6
updates
traskowskycaci Jan 22, 2025
eafa86b
logging updates
traskowskycaci Jan 22, 2025
cdec799
init those dang flags again I guess
traskowskycaci Jan 22, 2025
e884c23
update logging
traskowskycaci Jan 22, 2025
0a7883e
add db_region to exp.process-tpps.env
traskowskycaci Jan 23, 2025
9e0e391
add dod_ca_package back in
traskowskycaci Jan 23, 2025
649b480
remove second call to initProcessTPPSFlags in case that is causing re…
traskowskycaci Jan 23, 2025
0913e7e
add logging to ecs deploy script
traskowskycaci Jan 23, 2025
85a4a5b
fix filepath to hard-coded value for test
traskowskycaci Jan 23, 2025
194f713
filepath fixes that may work
traskowskycaci Jan 23, 2025
5858f50
release exp
traskowskycaci Jan 23, 2025
0f6528f
s3 filepath doesn't need cleaned up, remove call to that
traskowskycaci Jan 23, 2025
be90c57
deploy to exp
traskowskycaci Jan 23, 2025
c9ba4bd
make process-tpps.env match process-edis.env to see if task revision …
traskowskycaci Jan 23, 2025
f6c544b
remove some logging from ecs deploy
traskowskycaci Jan 24, 2025
0b3621c
config file changes as a test
traskowskycaci Jan 24, 2025
03f00bd
add a bunch of logging to put target and task def
traskowskycaci Jan 24, 2025
b964eff
Revert "add a bunch of logging to put target and task def"
traskowskycaci Jan 24, 2025
e72fa94
release exp
traskowskycaci Jan 24, 2025
d8f9d86
cleanup unneeded logging statements
traskowskycaci Jan 24, 2025
88e6fd9
filepath date improvements
traskowskycaci Jan 24, 2025
2967aea
Merge branch 'main' into B-21322-MAIN
traskowskycaci Jan 24, 2025
8225d79
deploy to exp
traskowskycaci Jan 24, 2025
b47efb5
release exp
traskowskycaci Jan 24, 2025
3ca0c7f
deploy to exp
traskowskycaci Feb 3, 2025
f4ddf1c
Merge branch 'main' into B-21322-MAIN
traskowskycaci Feb 3, 2025
542a18a
add logging to figure out go time.now issue with weird year
traskowskycaci Feb 3, 2025
81e7935
add clamAV status check and s3 download to tmp file
traskowskycaci Feb 3, 2025
9cded67
add logging to test deploy again
traskowskycaci Feb 4, 2025
2acf786
modify gitlab.yml to deploy to exp
traskowskycaci Feb 4, 2025
9489140
changes for make nonato_deploy_restore
traskowskycaci Feb 4, 2025
2bf777b
update deploy of process tpps to deploy_tasks_dp3 in gitlab-ci.yml
traskowskycaci Feb 4, 2025
aaa7865
hard code stuff for testing purposes for now
traskowskycaci Feb 4, 2025
df4c9dc
add logging of s3 getObject response
traskowskycaci Feb 4, 2025
80ccc3f
more general logging
traskowskycaci Feb 4, 2025
09655ac
convert to utf-8 encoding and log metadata so we can see
traskowskycaci Feb 5, 2025
abde80e
get s3 object tags for av-status, not metadata
traskowskycaci Feb 5, 2025
c0546cb
adding a few new env vars for s3 bucket and s3 folder, logging, cleanup
traskowskycaci Feb 5, 2025
cb12e78
download file only if scan is clean
traskowskycaci Feb 5, 2025
95de363
mutable tmp volume for tasks_dp3
traskowskycaci Feb 5, 2025
25410a5
renaming func, some cleanup
traskowskycaci Feb 5, 2025
dc648fd
try with 0116 file with data in it
traskowskycaci Feb 5, 2025
729c74f
add logging of contents of local file
traskowskycaci Feb 5, 2025
adf6091
still trying to log contents of tmp file
traskowskycaci Feb 6, 2025
1fe9bf4
convert the logged output to utf16
traskowskycaci Feb 6, 2025
d664e0a
Merge branch 'main' into B-21322-MAIN
traskowskycaci Feb 6, 2025
2ae6dc8
undo deploy to exp
traskowskycaci Feb 6, 2025
07b9d73
changing how we process entries to fail gracefully if no matching pay…
traskowskycaci Feb 6, 2025
464abf4
deploy to exp
traskowskycaci Feb 6, 2025
309014b
cleanup some logs and remove hard-coded s3 key
traskowskycaci Feb 7, 2025
42f4802
make the logged output preview more readable
traskowskycaci Feb 7, 2025
dc9b769
more log cleanup
traskowskycaci Feb 7, 2025
d7398d0
log count of successful and unsuccessful rows added to DB and log cou…
traskowskycaci Feb 7, 2025
ac730cb
Merge branch 'main' into B-21322-MAIN
traskowskycaci Feb 7, 2025
246842d
additions and updates for testing
traskowskycaci Feb 10, 2025
31313bb
removing unusued env var
traskowskycaci Feb 10, 2025
46ccede
test updates
traskowskycaci Feb 10, 2025
c942d11
Merge branch 'main' into B-21322-MAIN
traskowskycaci Feb 10, 2025
fca8ab1
update timezone from utc to America/Chicago (central time)
traskowskycaci Feb 10, 2025
3d3be68
more tests
traskowskycaci Feb 10, 2025
9e55d69
temp hard coding of 0208 file
traskowskycaci Feb 10, 2025
2f026a5
Revert "more tests"
traskowskycaci Feb 11, 2025
c11e466
Revert "test updates"
traskowskycaci Feb 11, 2025
ae95a26
refactor parser to use csv reader to handle complex messages
traskowskycaci Feb 11, 2025
71e7806
Reapply "more tests"
traskowskycaci Feb 11, 2025
9c11ebf
Reapply "test updates"
traskowskycaci Feb 11, 2025
4afb196
remove unnecessary logs
traskowskycaci Feb 11, 2025
35489c9
Merge branch 'main' into B-21322-MAIN
traskowskycaci Feb 11, 2025
ec9d2fd
remove unused arg
traskowskycaci Feb 11, 2025
dbc4c17
undo deploy to exp
traskowskycaci Feb 11, 2025
c44796f
Merge branch 'main' into B-21322-MAIN
traskowskycaci Feb 11, 2025
35e4990
test updates
traskowskycaci Feb 12, 2025
afee218
update AVStatusUNKNOWN comment
traskowskycaci Feb 12, 2025
4b87765
lowercase err messages
traskowskycaci Feb 12, 2025
72dfefc
set processTPPS SilenceUsage to true
traskowskycaci Feb 12, 2025
0bac6d8
add comment explaining AVStatusUNKNOWN usage
traskowskycaci Feb 12, 2025
0e5e0da
check tpps flags in config check
traskowskycaci Feb 12, 2025
bbc4bee
migration for dropping received by gex
traskowskycaci Feb 13, 2025
30c998d
update comments
traskowskycaci Feb 14, 2025
017bf03
Merge branch 'main' into B-21322-MAIN-status-cleanup
traskowskycaci Feb 14, 2025
b0d4aed
add init flags check
traskowskycaci Feb 14, 2025
7c62f49
remove changes from config.yml as it's not used anymore
traskowskycaci Feb 17, 2025
6728c55
add loadtest config file
traskowskycaci Feb 18, 2025
1eef81e
remove gex vars
traskowskycaci Feb 18, 2025
78cc1c5
Merge branch 'main' into B-21322-MAIN
traskowskycaci Feb 20, 2025
67d4b91
updating Paid and Received in move history
traskowskycaci Feb 20, 2025
efd8e86
Merge branch 'main' into B-21322-MAIN-status-cleanup
traskowskycaci Feb 20, 2025
bb24ee2
Merge branch 'B-21322-MAIN-status-cleanup' into B-21322-MAIN
traskowskycaci Feb 20, 2025
ad35b78
stg prd updates
traskowskycaci Feb 21, 2025
86e629b
Revert "stg prd updates"
traskowskycaci Feb 21, 2025
d5ed8ab
Merge branch 'main' into B-21322-MAIN
traskowskycaci Feb 21, 2025
bc247e4
Merge branch 'main' into B-21322-MAIN
traskowskycaci Feb 21, 2025
5a56dea
again - stg prd updates
traskowskycaci Feb 25, 2025
dd2559e
Merge branch 'main' into B-21322-MAIN
traskowskycaci Feb 25, 2025
ca78918
Merge branch 'main' into B-21322-MAIN
traskowskycaci Feb 25, 2025
305b80e
Merge branch 'main' into B-21322-MAIN
traskowskycaci Feb 26, 2025
2cee433
Merge branch 'main' into B-21322-MAIN
traskowskycaci Feb 27, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .gitlab-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1457,6 +1457,8 @@ deploy_tasks_dp3:
- ./scripts/ecs-deploy-task-container save-ghc-fuel-price-data "${ECR_REPOSITORY_URI}/app-tasks@${ECR_DIGEST}" "${APP_ENVIRONMENT}"
- echo "Deploying payment reminder email task service"
- ./scripts/ecs-deploy-task-container send-payment-reminder "${ECR_REPOSITORY_URI}/app-tasks@${ECR_DIGEST}" "${APP_ENVIRONMENT}"
- echo "Deploying process TPPS task service"
- ./scripts/ecs-deploy-task-container process-tpps "${ECR_REPOSITORY_URI}/app-tasks@${ECR_DIGEST}" "${APP_ENVIRONMENT}"
after_script:
- *announce_failure
rules:
Expand Down Expand Up @@ -1723,6 +1725,8 @@ deploy_tasks_stg:
- ./scripts/ecs-deploy-task-container save-ghc-fuel-price-data "${ECR_REPOSITORY_URI}/app-tasks@${ECR_DIGEST}" "${APP_ENVIRONMENT}"
- echo "Deploying payment reminder email task service"
- ./scripts/ecs-deploy-task-container send-payment-reminder "${ECR_REPOSITORY_URI}/app-tasks@${ECR_DIGEST}" "${APP_ENVIRONMENT}"
- echo "Deploying process TPPS task service"
- ./scripts/ecs-deploy-task-container process-tpps "${ECR_REPOSITORY_URI}/app-tasks@${ECR_DIGEST}" "${APP_ENVIRONMENT}"
after_script:
- *announce_failure
rules:
Expand Down Expand Up @@ -2008,6 +2012,8 @@ deploy_tasks_prd:
- ./scripts/ecs-deploy-task-container save-ghc-fuel-price-data "${ECR_REPOSITORY_URI}/app-tasks@${ECR_DIGEST}" "${APP_ENVIRONMENT}"
- echo "Deploying payment reminder email task service"
- ./scripts/ecs-deploy-task-container send-payment-reminder "${ECR_REPOSITORY_URI}/app-tasks@${ECR_DIGEST}" "${APP_ENVIRONMENT}"
- echo "Deploying process TPPS task service"
- ./scripts/ecs-deploy-task-container process-tpps "${ECR_REPOSITORY_URI}/app-tasks@${ECR_DIGEST}" "${APP_ENVIRONMENT}"
after_script:
- *announce_failure
rules:
Expand Down
4 changes: 4 additions & 0 deletions Dockerfile.tasks
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,8 @@ COPY bin/rds-ca-rsa4096-g1.pem /bin/rds-ca-rsa4096-g1.pem
COPY bin/rds-ca-2019-root.pem /bin/rds-ca-2019-root.pem
COPY bin/milmove-tasks /bin/milmove-tasks

# Mount mutable tmp for process-tpps
# hadolint ignore=DL3007
VOLUME ["/tmp"]

WORKDIR /bin
4 changes: 4 additions & 0 deletions Dockerfile.tasks_dp3
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,8 @@ COPY bin/rds-ca-rsa4096-g1.pem /bin/rds-ca-rsa4096-g1.pem

COPY bin/milmove-tasks /bin/milmove-tasks

# Mount mutable tmp for process-tpps
# hadolint ignore=DL3007
VOLUME ["/tmp"]

WORKDIR /bin
16 changes: 16 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -831,6 +831,22 @@ tasks_process_edis: tasks_build_linux_docker ## Run process-edis from inside doc
$(TASKS_DOCKER_CONTAINER):latest \
milmove-tasks process-edis

.PHONY: tasks_process_tpps
tasks_process_tpps: tasks_build_linux_docker ## Run process-tpps from inside docker container
@echo "Processing TPPS files with docker command..."
DB_NAME=$(DB_NAME_DEV) DB_DOCKER_CONTAINER=$(DB_DOCKER_CONTAINER_DEV) scripts/wait-for-db-docker
docker run \
-t \
-e DB_HOST="database" \
-e DB_NAME \
-e DB_PORT \
-e DB_USER \
-e DB_PASSWORD \
--link="$(DB_DOCKER_CONTAINER_DEV):database" \
--rm \
$(TASKS_DOCKER_CONTAINER):latest \
milmove-tasks process-tpps

.PHONY: tasks_save_ghc_fuel_price_data
tasks_save_ghc_fuel_price_data: tasks_build_linux_docker ## Run save-ghc-fuel-price-data from inside docker container
@echo "Saving the fuel price data to the ${DB_NAME_DEV} database with docker command..."
Expand Down
1 change: 1 addition & 0 deletions cmd/ecs-deploy/put_target.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ var names = []string{
"connect-to-gex-via-sftp",
"post-file-to-gex",
"process-edis",
"process-tpps",
"save-ghc-fuel-price-data",
"send-payment-reminder",
}
Expand Down
1 change: 1 addition & 0 deletions cmd/ecs-deploy/task_def.go
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ var servicesToEntryPoints = map[string][]string{
fmt.Sprintf("%s connect-to-gex-via-sftp", binMilMoveTasks),
fmt.Sprintf("%s post-file-to-gex", binMilMoveTasks),
fmt.Sprintf("%s process-edis", binMilMoveTasks),
fmt.Sprintf("%s process-tpps", binMilMoveTasks),
fmt.Sprintf("%s save-ghc-fuel-price-data", binMilMoveTasks),
fmt.Sprintf("%s send-payment-reminder", binMilMoveTasks),
},
Expand Down
10 changes: 10 additions & 0 deletions cmd/milmove-tasks/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,16 @@ func main() {
initConnectToGEXViaSFTPFlags(processEDIsCommand.Flags())
root.AddCommand(processEDIsCommand)

processTPPSCommand := &cobra.Command{
Use: "process-tpps",
Short: "process TPPS files asynchrounously",
Long: "process TPPS files asynchrounously",
RunE: processTPPS,
SilenceUsage: true,
}
initProcessTPPSFlags(processTPPSCommand.Flags())
root.AddCommand(processTPPSCommand)

completionCommand := &cobra.Command{
Use: "completion",
Short: "Generates bash completion scripts",
Expand Down
11 changes: 0 additions & 11 deletions cmd/milmove-tasks/process_edis.go
Original file line number Diff line number Diff line change
Expand Up @@ -244,16 +244,5 @@ func processEDIs(_ *cobra.Command, _ []string) error {
logger.Info("Successfully processed EDI824 application advice responses")
}

// Pending completion of B-20560, uncomment the code below
/*
// Process TPPS paid invoice report
pathTPPSPaidInvoiceReport := v.GetString(cli.SFTPTPPSPaidInvoiceReportPickupDirectory)
_, err = syncadaSFTPSession.FetchAndProcessSyncadaFiles(appCtx, pathTPPSPaidInvoiceReport, lastReadTime, invoice.NewTPPSPaidInvoiceReportProcessor())
if err != nil {
logger.Error("Error reading TPPS Paid Invoice Report application advice responses", zap.Error(err))
} else {
logger.Info("Successfully processed TPPS Paid Invoice Report application advice responses")
}
*/
return nil
}
Loading
Loading