Filename changes and recreates Lambdas on different platforms (CI/CD vs local)

[jussapaavo]

Description

Okay,

here's a deep dive. I've been trying to debug this persisting issue a couple of times and here are my latest findings on why this module tries to recreate Lambdas everytime we change platforms, essentially between local and CI/CD deployments. Even though there is no code changes and the dependencies are lock as tight as they possibly can, still the module wants to recreate the zip and redeploy the Lambda.

Let me try to describe the issue as clearly as possible.

• ✋ I have searched the open/closed issues and my issue is not listed.
• This issue still persists in the newest module version 7.20.1, even though it has been previously discussed in:
  • lambda.null_resource.archive[0] must be replaced #51
  • Multiple Users Cause lambda.null_resource.archive[0] timestamp Differences #63
  • Multiple Users Cause lambda.null_resource.archive[0] timestamp Differences #323
  • Module always detect change even if src code doesn't change #339

Versions

• Module version [Required]: 7.20.1.

• Terraform version: 1.11.3

• Terragrunt version: 0.76.8

Reproduction Code [Required]

Steps to reproduce the behavior, here is our wrapper module config:

module "this" {
  source  = "terraform-aws-modules/lambda/aws"
  version = "~> 7.20"

  publish                      = true
  store_on_s3                  = var.s3_bucket != null ? true : false
  s3_bucket                    = var.s3_bucket
  s3_prefix                    = "services/"
  recreate_missing_package     = false
  trigger_on_package_timestamp = false

  source_path = [
    {
      path = "${path.module}/../../${var.source_dir_path}/src/"
      commands = [
        ":zip . app",
      ]
      patterns = [
        "!.*/__pycache__.*",
        "!.*/.*terragrunt.*",
      ]
    },
    {
      path = "${path.module}/../../../${var.source_dir_path}"
      commands = [
        "uv export --frozen --no-dev --no-editable -o requirements.txt",
        join(" ", [
          "uv pip install",
          "--no-installer-metadata",
          "--no-compile-bytecode",
          "--python-platform x86_64-manylinux2014",
          "--python ${replace(var.runtime, "python", "")}",
          "--target packages",
          "-r requirements.txt"
        ]),
        "cd packages",
        ":zip"
      ]
    }
  ]

  function_name         = var.function_name
  handler               = "app.${var.handler}"
  runtime               = var.runtime
  timeout               = var.timeout
  memory_size           = var.memory_size
  layers                = var.layers
  environment_variables = var.environment_variables
  event_source_mapping  = var.event_source_mapping

  create_role            = true
  policy_jsons           = var.policy_jsons
  number_of_policy_jsons = length(var.policy_jsons)
  attach_policy_jsons    = length(var.policy_jsons) > 0
  authorization_type     = "AWS_IAM"
  allowed_triggers       = var.allowed_triggers

  cloudwatch_logs_retention_in_days = 90
}

Basically, we are using uv to build the Python dependencies and pushing them inside the zip archive. This is for performance benefits, but also to make sure the dependencies are consistent between platforms (MacOS vs Linux). And we also push the actual lambda code inside the archive, with patterns to ignore Python & Terragrunt artifacts.

This creates a clean zip file with only the necessary contents. Works as intended. When running terragrunt apply locally, everything goes smoothly. Even rerunning the deploy says No changes.

Then running the same code in our CI/CD pipeline, the module sees changes, even though nothing has changed:

13:38:48.459 STDOUT terraform:   # module.lambda_name.module.this.local_file.archive_plan[0] will be created
13:38:48.459 STDOUT terraform:   + resource "local_file" "archive_plan" {
13:38:48.459 STDOUT terraform:       + content              = jsonencode(
13:38:48.459 STDOUT terraform:             {
13:38:48.459 STDOUT terraform:               + artifacts_dir = "builds"
13:38:48.459 STDOUT terraform:               + build_plan    = [
13:38:48.459 STDOUT terraform:                   + [
13:38:48.459 STDOUT terraform:                       + [
13:38:48.459 STDOUT terraform:                           + "set:filter",
13:38:48.459 STDOUT terraform:                           + [
13:38:48.459 STDOUT terraform:                               + "!.*/__pycache__.*",
13:38:48.459 STDOUT terraform:                               + "!.*/.*terragrunt.*",
13:38:48.459 STDOUT terraform:                             ],
13:38:48.459 STDOUT terraform:                         ],
13:38:48.459 STDOUT terraform:                       + [
13:38:48.459 STDOUT terraform:                           + "set:workdir",
13:38:48.459 STDOUT terraform:                           + "../services/lambda_name/src",
13:38:48.459 STDOUT terraform:                         ],
13:38:48.459 STDOUT terraform:                       + [
13:38:48.459 STDOUT terraform:                           + "zip:embedded",
13:38:48.459 STDOUT terraform:                           + ".",
13:38:48.460 STDOUT terraform:                           + "app",
13:38:48.460 STDOUT terraform:                         ],
13:38:48.460 STDOUT terraform:                     ],
13:38:48.460 STDOUT terraform:                   + [
13:38:48.460 STDOUT terraform:                       + [
13:38:48.460 STDOUT terraform:                           + "set:workdir",
13:38:48.460 STDOUT terraform:                           + "../services/lambda_name",
13:38:48.460 STDOUT terraform:                         ],
13:38:48.460 STDOUT terraform:                       + [
13:38:48.460 STDOUT terraform:                           + "sh",
13:38:48.460 STDOUT terraform:                           + <<-EOT
13:38:48.460 STDOUT terraform:                                 uv export --frozen --no-dev --no-editable -o requirements.txt
13:38:48.460 STDOUT terraform:                                 uv pip install --no-installer-metadata --no-compile-bytecode --python-platform x86_64-manylinux2014 --python 3.12 --target packages -r requirements.txt
13:38:48.460 STDOUT terraform:                                 cd packages
13:38:48.460 STDOUT terraform:                             EOT,
13:38:48.460 STDOUT terraform:                         ],
13:38:48.460 STDOUT terraform:                       + [
13:38:48.460 STDOUT terraform:                           + "zip:embedded",
13:38:48.460 STDOUT terraform:                         ],
13:38:48.460 STDOUT terraform:                     ],
13:38:48.460 STDOUT terraform:                 ]
13:38:48.460 STDOUT terraform:               + filename      = "builds/f31c1744302f59f06bdc4ea10df26dfca3a824d9425b6a469a8bc23509983c39.zip"
13:38:48.460 STDOUT terraform:               + runtime       = "python3.12"
13:38:48.460 STDOUT terraform:             }
13:38:48.460 STDOUT terraform:         )
13:38:48.460 STDOUT terraform:       + content_base64sha256 = (known after apply)
13:38:48.460 STDOUT terraform:       + content_base64sha512 = (known after apply)
13:38:48.460 STDOUT terraform:       + content_md5          = (known after apply)
13:38:48.460 STDOUT terraform:       + content_sha1         = (known after apply)
13:38:48.460 STDOUT terraform:       + content_sha256       = (known after apply)
13:38:48.460 STDOUT terraform:       + content_sha512       = (known after apply)
13:38:48.460 STDOUT terraform:       + directory_permission = "0755"
13:38:48.461 STDOUT terraform:       + file_permission      = "0644"
13:38:48.461 STDOUT terraform:       + filename             = "builds/f31c1744302f59f06bdc4ea10df26dfca3a824d9425b6a469a8bc23509983c39.plan.json"
13:38:48.461 STDOUT terraform:       + id                   = (known after apply)
13:38:48.461 STDOUT terraform:     }
13:38:48.461 STDOUT terraform:   # module.lambda_name.module.this.null_resource.archive[0] must be replaced
13:38:48.461 STDOUT terraform: +/- resource "null_resource" "archive" {
13:38:48.461 STDOUT terraform:       ~ id       = "15267922883778[230](https://gitlab.company.com/repo/-/jobs/16461290#L230)18" -> (known after apply)
13:38:48.461 STDOUT terraform:       ~ triggers = { # forces replacement
13:38:48.461 STDOUT terraform:           ~ "filename"  = "builds/0a245ee6947e3290d4249c503d616657537187535d020a9815380e4ce28dc0f5.zip" -> "builds/f31c1744302f59f06bdc4ea10df26dfca3a824d9425b6a469a8bc[235](https://gitlab.company.com/repo/-/jobs/16461290#L235)09983c39.zip"
13:38:48.461 STDOUT terraform:             # (1 unchanged element hidden)
13:38:48.461 STDOUT terraform:         }
13:38:48.461 STDOUT terraform:     }

I have downloaded the zip files from both deployments and compared them locally. They seem to be identical in every metric.

Running:

diff -y \
  <(unzip -l /Users/jussapaavo/Downloads/lambdas/cicd/f31c1744302f59f06bdc4ea10df26dfca3a824d9425b6a469a8bc23509983c39.zip) \
  <(unzip -l /Users/jussapaavo/Downloads/lambdas/local/0a245ee6947e3290d4249c503d616657537187535d020a9815380e4ce28dc0f5.zip)

Returns:

Archive:  /Users/jussapaavo/Downloads/lambdas/cicd/f31c | Archive:  /Users/jussapaavo/Downloads/lambdas/local/0a2
  Length      Date    Time    Name                                Length      Date    Time    Name
---------  ---------- -----   ----                              ---------  ---------- -----   ----
     3171  01-01-1980 00:00   app/main.py                            3171  01-01-1980 00:00   app/main.py
        0  01-01-1980 00:00   .lock                                     0  01-01-1980 00:00   .lock
    34703  01-01-1980 00:00   six.py                                34703  01-01-1980 00:00   six.py
        0  01-01-1980 00:00   dateutil/                                 0  01-01-1980 00:00   dateutil/
      222  01-01-1980 00:00   dateutil/__init__.py                    222  01-01-1980 00:00   dateutil/__init__.py
      932  01-01-1980 00:00   dateutil/_common.py                     932  01-01-1980 00:00   dateutil/_common.py
      116  01-01-1980 00:00   dateutil/_version.py                    116  01-01-1980 00:00   dateutil/_version.py
     2684  01-01-1980 00:00   dateutil/easter.py                     2684  01-01-1980 00:00   dateutil/easter.py
    24418  01-01-1980 00:00   dateutil/relativedelta.py             24418  01-01-1980 00:00   dateutil/relativedelta.py
    64802  01-01-1980 00:00   dateutil/rrule.py                     64802  01-01-1980 00:00   dateutil/rrule.py
       59  01-01-1980 00:00   dateutil/tzwin.py                        59  01-01-1980 00:00   dateutil/tzwin.py
     1963  01-01-1980 00:00   dateutil/utils.py                      1963  01-01-1980 00:00   dateutil/utils.py
        0  01-01-1980 00:00   dateutil/parser/                          0  01-01-1980 00:00   dateutil/parser/
     1727  01-01-1980 00:00   dateutil/parser/__init__.py            1727  01-01-1980 00:00   dateutil/parser/__init__.py
    57607  01-01-1980 00:00   dateutil/parser/_parser.py            57607  01-01-1980 00:00   dateutil/parser/_parser.py
    12902  01-01-1980 00:00   dateutil/parser/isoparser.py          12902  01-01-1980 00:00   dateutil/parser/isoparser.py
        0  01-01-1980 00:00   dateutil/tz/                              0  01-01-1980 00:00   dateutil/tz/
      551  01-01-1980 00:00   dateutil/tz/__init__.py                 551  01-01-1980 00:00   dateutil/tz/__init__.py
    12892  01-01-1980 00:00   dateutil/tz/_common.py                12892  01-01-1980 00:00   dateutil/tz/_common.py
     1434  01-01-1980 00:00   dateutil/tz/_factories.py              1434  01-01-1980 00:00   dateutil/tz/_factories.py
    60472  01-01-1980 00:00   dateutil/tz/tz.py                     60472  01-01-1980 00:00   dateutil/tz/tz.py
    11318  01-01-1980 00:00   dateutil/tz/win.py                    11318  01-01-1980 00:00   dateutil/tz/win.py
        0  01-01-1980 00:00   dateutil/zoneinfo/                        0  01-01-1980 00:00   dateutil/zoneinfo/
     5889  01-01-1980 00:00   dateutil/zoneinfo/__init__.py          5889  01-01-1980 00:00   dateutil/zoneinfo/__init__.py
   154226  01-01-1980 00:00   dateutil/zoneinfo/dateutil-zone      154226  01-01-1980 00:00   dateutil/zoneinfo/dateutil-zone
     1719  01-01-1980 00:00   dateutil/zoneinfo/rebuild.py           1719  01-01-1980 00:00   dateutil/zoneinfo/rebuild.py
        0  01-01-1980 00:00   python_dateutil-2.7.5.dist-info           0  01-01-1980 00:00   python_dateutil-2.7.5.dist-info
     2889  01-01-1980 00:00   python_dateutil-2.7.5.dist-info        2889  01-01-1980 00:00   python_dateutil-2.7.5.dist-info
     7486  01-01-1980 00:00   python_dateutil-2.7.5.dist-info        7486  01-01-1980 00:00   python_dateutil-2.7.5.dist-info
     2048  01-01-1980 00:00   python_dateutil-2.7.5.dist-info        2048  01-01-1980 00:00   python_dateutil-2.7.5.dist-info
      110  01-01-1980 00:00   python_dateutil-2.7.5.dist-info         110  01-01-1980 00:00   python_dateutil-2.7.5.dist-info
        9  01-01-1980 00:00   python_dateutil-2.7.5.dist-info           9  01-01-1980 00:00   python_dateutil-2.7.5.dist-info
        1  01-01-1980 00:00   python_dateutil-2.7.5.dist-info           1  01-01-1980 00:00   python_dateutil-2.7.5.dist-info
        0  01-01-1980 00:00   six-1.17.0.dist-info/                     0  01-01-1980 00:00   six-1.17.0.dist-info/
     1066  01-01-1980 00:00   six-1.17.0.dist-info/LICENSE           1066  01-01-1980 00:00   six-1.17.0.dist-info/LICENSE
     1658  01-01-1980 00:00   six-1.17.0.dist-info/METADATA          1658  01-01-1980 00:00   six-1.17.0.dist-info/METADATA
      435  01-01-1980 00:00   six-1.17.0.dist-info/RECORD             435  01-01-1980 00:00   six-1.17.0.dist-info/RECORD
      109  01-01-1980 00:00   six-1.17.0.dist-info/WHEEL              109  01-01-1980 00:00   six-1.17.0.dist-info/WHEEL
        4  01-01-1980 00:00   six-1.17.0.dist-info/top_level.           4  01-01-1980 00:00   six-1.17.0.dist-info/top_level.
---------                     -------                           ---------                     -------
   469622                     39 files                             469622                     39 files

So file amount, size, bytes and contents are identical.

Then running checksums on both files return same values:

❯ sha256sum /Users/jussapaavo/Downloads/lambdas/local/0a245ee6947e3290d4249c503d616657537187535d020a9815380e4ce28dc0f5.zip
367e760914429f676f0c48186a45dcc6e89f39c028d590edc0986494e02faa4b  /Users/jussapaavo/Downloads/lambdas/local/0a245ee6947e3290d4249c503d616657537187535d020a9815380e4ce28dc0f5.zip

❯ sha256sum /Users/jussapaavo/Downloads/lambdas/cicd/f31c1744302f59f06bdc4ea10df26dfca3a824d9425b6a469a8bc23509983c39.zip
367e760914429f676f0c48186a45dcc6e89f39c028d590edc0986494e02faa4b  /Users/jussapaavo/Downloads/lambdas/cicd/f31c1744302f59f06bdc4ea10df26dfca3a824d9425b6a469a8bc23509983c39.zip

Also when running local and CI/CD pipelines back to back, they recreate the zips and Lambdas. Though, when comparing the builds of these different runs on the same platform, I can seen the results are consistent. So basically when running locally, I always get filename
0a245ee6947e3290d4249c503d616657537187535d020a9815380e4ce28dc0f5.zip and in CI/CD I get f31c1744302f59f06bdc4ea10df26dfca3a824d9425b6a469a8bc23509983c39.zip for the same code builds.
Note that neither of these are the sha256sum (I assumed it would be for it to be consistent).

This leads me to believe that the hashing and file naming process of the packages are inconsistent between the platforms.

Expected behavior

Consistent zip file creation and file naming between different platforms, resulting in no changes in Terraform deployment between local and CI/CD runs.

Actual behavior

New zip files are created, even though there are no changes to code or dependencies. This does not happen when run on the same platform, only when the platform changes (local vs CI/CD pipelines).

Additional context

Note that I'm explicitly running extra commands in the source_path:

  source_path = [
    {
      path = "${path.module}/../../${var.source_dir_path}/src/"
      commands = [
        ":zip . app",
      ]
      patterns = [
        "!.*/__pycache__.*",
        "!.*/.*terragrunt.*",
      ]
    },
    ...
  ]

This is because when running the vanilla path, the process (Python's zipfile?) retains the filesystem date for the files instead of stripping them. When comparing the zip files, I could see the dates changing (inside the zip file) between local and CI/CD deployments. After running :zip . app explicitly, I get 01-01-1980 for all files.

[jussapaavo]

I did further digging.

If I understood correctly, the filename is actually composed during planning, not during apply (obious in hindsight). As per the plan output:

13:38:48.461 STDOUT terraform:   # module.lambda_name.module.this.null_resource.archive[0] must be replaced
13:38:48.461 STDOUT terraform: +/- resource "null_resource" "archive" {
13:38:48.461 STDOUT terraform:       ~ id       = "15267922883778[230](https://gitlab.company.com/repo/-/jobs/16461290#L230)18" -> (known after apply)
13:38:48.461 STDOUT terraform:       ~ triggers = { # forces replacement
13:38:48.461 STDOUT terraform:           ~ "filename"  = "builds/0a245ee6947e3290d4249c503d616657537187535d020a9815380e4ce28dc0f5.zip" -> "builds/f31c1744302f59f06bdc4ea10df26dfca3a824d9425b6a469a8bc[235](https://gitlab.company.com/repo/-/jobs/16461290#L235)09983c39.zip"
13:38:48.461 STDOUT terraform:             # (1 unchanged element hidden)
13:38:48.461 STDOUT terraform:         }
13:38:48.461 STDOUT terraform:     }

But the plan stage doesn't build any zips, so the filename is actually based on the current contents of the source dir, not the built zip available after apply. Usually the source dir might have artifacts like __pycache__, operating filesystem differences, or timestamp differences as I demonstrated in the original post, thus resulting in differences between local and CI/CD pipelines.

So the filename changes, because the source code hash is calculated before the build artifact is created and "normalized". This might be intended as the hash is required for detecting changes in the plan stage, but at the same time it definitely causes these issues with repeated deploys.

I don't know if it is possible to somehow detect the changes in some other manner or like so that it respects the build commands for normalizing the output zip?

[Drewster727]

Seeing this same issue, we have folks on windows and mac. I'm thinking about just bypassing the package creation in the module, do the creation outside of it, then pass it in. Theoretically that will work if I base the zip name and what not on the source hash value (which should not change across platforms theoretically). However, this should still get resolved.

[github-actions]

This issue has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this issue will be closed in 10 days

[jussapaavo]

Not stale.