add aws as storage backend for plan storage

[larhauga]

Hi, this is my first contribution to digger, so any feedback or if this is something thats wanted is appreciated.

ref eg #817

This pull requests adds support for using AWS S3 as backing store for the plan file.
It allows users to use S3 as the backing store for plans instead of github packages, which has limited access control capabilities.

It is currently implemented with the same feature set as GCP buckets, and to maintain compatability with the GCP bucket env var, a specific env var has been added for s3 as well.

Any input on the naming of the actions inputs and/or env variables?

TODO: It has currently only been tested with emulation of the S3 sdk APIs, but I think that it needs testing in a real environment.
Update: It ran OK in our environment in combination with #1350 (testing release https://github.com/larhauga/digger/releases/tag/0.0.1)

[motatoes]

hey @larhauga ! Thanks for this feature 💯 very valuable. Please bear with me to get it reviewed, I have a backlog of a few issues to review so appologies in advance if I'm late to review it

[motatoes]

(also closing and re-opening to trigger all tests, since we have a glitch in our workflows)

[motatoes]

Hey @larhauga thanks again for this pull request! I have had a review and it looks great to me. We need to do a rebase from develop and a go mod tidy to fix the conflicts if you could do that it should be close to mergabable.

One thing I would love to see is an e2e test for it. We have one for GCP gcp here and it writes/reads from a bucket on our account. I would love to replicate that for aws. Is this something you can quickly add to that file? I'm happy to create a bucket and access on our account so it runs on every merge and pre release. Also I'm happy to do this as a separate PR if you prefer. I'm thinking we can have bucket and access provided as external vars so that it is configurable by us.

Sidenote: This test is currently failing for the current PR "Cli e2e tests" due to GCP creds not available in external PRs (github security feature). I have moved this test to be run only after merge so that it will stop fialing for external contributions.

[larhauga]

Hey @larhauga thanks again for this pull request! I have had a review and it looks great to me. We need to do a rebase from develop and a go mod tidy to fix the conflicts if you could do that it should be close to mergabable.

One thing I would love to see is an e2e test for it. We have one for GCP gcp here and it writes/reads from a bucket on our account. I would love to replicate that for aws. Is this something you can quickly add to that file? I'm happy to create a bucket and access on our account so it runs on every merge and pre release. Also I'm happy to do this as a separate PR if you prefer. I'm thinking we can have bucket and access provided as external vars so that it is configurable by us.

Sidenote: This test is currently failing for the current PR "Cli e2e tests" due to GCP creds not available in external PRs (github security feature). I have moved this test to be run only after merge so that it will stop fialing for external contributions.

Hi, and thanks for the review! 🎉
I have rebased this branch from latest develop now :)

I think an e2e test would be great here, and I can try to get some time to work on it.
Maybe it is best as a seperate PR?

I have also been working on migrating the sdk from v1 to v2 in this PR (#1350) and have some other WIP that depends on the new sdk (adding tests for dynamodb locking and a small bugfix): https://github.com/diggerhq/digger/compare/develop...larhauga:digger:dynamodblock-fix-and-test?expand=1
I would like a e2e test to verify those changes as well, even though we have been running a fork with the changes.

[motatoes]

@larhauga Perfect! That makes sense to me. I've created an issue for tracking e2e test in #1380.

Merging then