#278: EKS Node Group Variable and Airflow Worker IAM Role

[drewm-swe]

Purpose

• This PR enables users of the terraform-eks-cluster module to specify node group configurations (instance types, min/max/desired size) for their cluster at deployment time.
• It also enables Airflow workers to be able to write logs to S3. This was achieved by creating an IAM role for Airflow Workers with AssumeRoleWithWebIdentity using OIDC.

Proposed Changes

• [ADD] Input variable for EKS node group configuration.
• [ADD] IAM role for Airflow Workers with AssumeRoleWithWebIdentity using OIDC

Issues

• [New Feature] Allow choosing EC2 types when deploying an EKS cluster #35

Testing

• An EKS cluster was manually provisioned in unity-venue-dev using terraform-unity-eks_module. Following the creation of the EKS cluster, an Airflow-based U-SPS was deployed onto the cluster. DAGs utilizing Docker-in-Docker containers were manually triggered and successfully ran to completion.

[LucaCinquini]

I followed the instructions and recreated the EKS cluster, redeployed SPS, executed the SBG PreProcess workflow - it all worked. Great job!

Please add the following instruction to the SPS guide to allow developers to work with multiple clusters at once:
terraform workspace new $CLUSTER_NAME

I also wonder whether we should make the policy more flexible - allow access to ALL AWS services from the worker node? Or at least, ALL operations on specific services like S3, SNS, SQS, SSM, ...?

[drewm-swe]

Please add the following instruction to the SPS guide to allow developers to work with multiple clusters at once:

Thanks @LucaCinquini, I'll update the docs to mention using the terraform workspace.

Regarding the IAM policy, AWS recommends applying least-privilege permissions as a cybersecurity best practice. I think the policy is actually over-permissive in our current implementation and I think we should try to pare it down to only what's required.