EventSchedulePeriod does not exist in the template v6.9.0

[mubeta06]

I attempted to use the most recent v6.9.0 release and receive the following error:

Error: waiting for CloudFormation Stack (arn:aws:cloudformation:ap-southeast-2:<redacted>:stack/terraform-v6-9-0/<redacted>) create: failed to create CloudFormation stack, rollback requested (ROLLBACK_COMPLETE): ["The following resource(s) failed to create: [AzRebalancingSuspender, Autoscaling]. Rollback requested by user." "Resource creation cancelled" "Parameters: [EventSchedulePeriod] do not exist in the template"]

Expected behavior
The stack comes to life and is operational for use.

Actual behaviour
The stack was not functional and was rolled back by CloudFormation

Stack parameters

• AWS Region: [e.g. ap-southeast-2]
• Version [e.g. v6.9.0]
  "AgentEnvFileUrl" = ""
  "AgentsPerInstance" = "1"
  "ArtifactsBucket" = ""
  "AssociatePublicIpAddress" = "true"
  "AuthorizedUsersUrl" = ""
  "AvailabilityZones" = ","
  "BootstrapScriptUrl" = ""
  "BuildkiteAdditionalSudoPermissions" = ""
  "BuildkiteAgentEnableGitMirrors" = "false"
  "BuildkiteAgentExperiments" = ""
  "BuildkiteAgentRelease" = "stable"
  "BuildkiteAgentScalerServerlessARN" = "arn:aws:serverlessrepo:us-east-1:172840064832:applications/buildkite-agent-scaler"
  "BuildkiteAgentScalerVersion" = "1.5.0"
  "BuildkiteAgentTags" = ""
  "BuildkiteAgentTimestampLines" = "false"
  "BuildkiteAgentToken" = "****"
  "BuildkiteAgentTokenParameterStoreKMSKey" = ""
  "BuildkiteAgentTokenParameterStorePath" = ""
  "BuildkiteAgentTracingBackend" = ""
  "BuildkiteQueue" = "terraform"
  "BuildkiteTerminateInstanceAfterJob" = "false"
  "BuildkiteWindowsAdministrator" = "true"
  "CostAllocationTagName" = "aws:createdBy"
  "CostAllocationTagValue" = "buildkite-elastic-ci-stack-for-aws"
  "ECRAccessPolicy" = "full"
  "EnableCostAllocationTags" = "false"
  "EnableDetailedMonitoring" = "false"
  "EnableDockerExperimental" = "false"
  "EnableDockerLoginPlugin" = "true"
  "EnableDockerUserNamespaceRemap" = "true"
  "EnableECRPlugin" = "true"
  "EnableInstanceStorage" = "false"
  "EnableSecretsPlugin" = "true"
  "IMDSv2Tokens" = "optional"
  "ImageId" = ""
  "ImageIdParameter" = ""
  "InstanceCreationTimeout" = "PT5M"
  "InstanceName" = ""
  "InstanceOperatingSystem" = "linux"
  "InstanceRoleName" = ""
  "InstanceRolePermissionsBoundaryARN" = ""
  "InstanceTypes" = "t2.large"
  "KeyName" = ""
  "ManagedPolicyARNs" = ""
  "MaxSize" = "10"
  "MinSize" = "0"
  "OnDemandPercentage" = "100"
  "RootVolumeEncrypted" = "false"
  "RootVolumeName" = "/dev/xvda"
  "RootVolumeSize" = "10"
  "RootVolumeType" = "gp2"
  "ScaleInIdlePeriod" = "600"
  "ScaleOutFactor" = "1"
  "ScaleOutForWaitingJobs" = "false"
  "ScalerEventSchedulePeriod" = "1 minute"
  "ScalerMinPollInterval" = "10s"
  "SecretsBucket" = ""
  "SecretsBucketEncryption" = "false"
  "SecretsBucketRegion" = ""
  "SecurityGroupIds" = ""
  "SpotAllocationStrategy" = "capacity-optimized"
  "Subnets" = ","
  "VpcId" = ""

Additional context
As a long time fan and user of the Elastic CI Stack I have wrapped a Terraform module around the CloudFormation template to more easily manage the deployment and upgrading of the various stacks I manage.

I believe the issue is somehow related to #1243

[triarius]

@mubeta06 thanks for letting us know about this, and thanks for listing the parameters too. That should really help with debugging this.

One thing that immediately stands out is that even though you are using Elastic CI Stack v6.9.0, the parameter BuildkiteAgentScalerVersion is 1.5.0, whereas the default is 1.7.0. This suggests that you are upgrading the stack from an older version.

There is a known issue with how CloudFormation handles default parameter values that may explain this. We mentioned this in the changelog for v6.8.0, but perhaps we should make it more prominent.

If you are upgrading en existing stack, what happens if you:

1. Use aws cloudformation update-stack to upgrade your stack instead, or
2. Manually set the BuildkiteAgentScalerVersion parameter to 1.7.0

[mubeta06]

Thanks @triarius you were correct in identifying the BuildkiteAgentScalerVersion as the source of the issue. Although I wasn't upgrading the stack perse I hadn't aligned my Terraform remote module variable defaults to align with the new defaults of the elastic stack CloudFormation template for the BuildkiteAgentScalerVersion (which was still 1.5.0) so it was totally my bad 😮‍💨. After ensuring BuildkiteAgentScalerVersion was set to 1.7.0 as you suggested I was able to successfully deploy the stack.