Bug: git+https requirements fail to install when using python3.12 container

[zhemaituk]

Description:

sam build fails when requirements.txt contains git+https links, python3.12 only

Steps to reproduce:

1. SAM init hello world example
2. Runtime: python3.12; Architectures: - arm64
3. Add to requirements.txt:

gendercomputer@git+https://github.com/tue-mdse/genderComputer.git

4. sam build --use-container --debug

Observed result:

Fetching public.ecr.aws/sam/build-python3.12:latest-arm64 Docker container image......
2024-08-14 13:41:13,759 | Mounting .../sam-app/hello_world as /tmp/samcli/source:ro,delegated, inside runtime container                                                  
Using the request object from command line argument
...
 Running PythonPipBuilder:ResolveDependencies
calling pip download -r /tmp/samcli/source/requirements.txt --dest /tmp/samcli/scratch --exists-action i
pip stdout: b"Collecting gendercomputer@ git+https://github.com/tue-mdse/genderComputer.git (from -r /tmp/samcli/source/requirements.txt (line 1))\n  Cloning https://github.com/tue-mdse/genderComputer.git to /tmp/pip-download-cnqo230n/gendercomputer_83fc8aa3dae241db97874f2c91a268bd\n  Resolved https://github.com/tue-mdse/genderComputer.git to commit f6267615517913e53cb0b882b248f1c2e11b8bbc\n  Installing build dependencies: started\n  Installing build dependencies: finished with status 'done'\n  Getting requirements to build wheel: started\n  Getting requirements to build wheel: finished with status 'done'\n  Preparing metadata (pyproject.toml): started\n  Preparing metadata (pyproject.toml): finished with status 'done'\nCollecting unidecode==1.3.2 (from gendercomputer@ git+https://github.com/tue-mdse/genderComputer.git->-r /tmp/samcli/source/requirements.txt (line 1))\n  Downloading Unidecode-1.3.2-py3-none-any.whl.metadata (13 kB)\nCollecting nameparser==1.0.6 (from gendercomputer@ git+https://github.com/tue-mdse/genderComputer.git->-r /tmp/samcli/source/requirements.txt (line 1))\n  Downloading nameparser-1.0.6-py2.py3-none-any.whl.metadata (5.9 kB)\nDownloading nameparser-1.0.6-py2.py3-none-any.whl (23 kB)\nDownloading Unidecode-1.3.2-py3-none-any.whl (235 kB)\n   \xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81 235.7/235.7 kB 5.9 MB/s eta 0:00:00\nSaved /tmp/samcli/scratch/gendercomputer-0.1.zip\nSaved /tmp/samcli/scratch/nameparser-1.0.6-py2.py3-none-any.whl\nSaved /tmp/samcli/scratch/Unidecode-1.3.2-py3-none-any.whl\nSuccessfully downloaded gendercomputer nameparser unidecode\n"
pip stderr: b"WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.\nPlease see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.\nTo avoid this problem you can invoke Python with '-m pip' instead of running pip directly.\n  Running command git clone --filter=blob:none --quiet https://github.com/tue-mdse/genderComputer.git /tmp/pip-download-cnqo230n/gendercomputer_83fc8aa3dae241db97874f2c91a268bd\n\n[notice] A new release of pip is available: 24.0 -> 24.2\n[notice] To update, run: pip install --upgrade pip\n"
Non zero rc (1) from the setup.py egg_info command: b'Traceback (most recent call last):\n  File "<string>", line 1, in <module>\nModuleNotFoundError: No module named \'setuptools\'\n'
Using fallback location for PKG-INFO file in package directory: /tmp/tmpdamop0ow/gendercomputer
PythonPipBuilder:ResolveDependencies raised unhandled exception
Traceback (most recent call last):
  File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflow.py", line 374, in run
    action.execute()
  File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 71, in execute
    package_builder.build_dependencies(
  File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 157, in build_dependencies
    self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path)
  File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 262, in build_site_packages
    wheels, packages_without_wheels = self._download_dependencies(scratch_directory, requirements_filepath)
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 286, in _download_dependencies
    deps = self._download_all_dependencies(requirements_filename, directory)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 370, in _download_all_dependencies
    Package(directory, filename, self.python_exe)
  File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 522, in __init__
    self._name, self._version = self._calculate_name_and_version()
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 564, in _calculate_name_and_version
    name, version = info_fetcher.get_package_name_and_version(sdist_path)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 706, in get_package_name_and_version
    pkg_info_filepath = self._get_pkg_info_filepath(package_dir)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 618, in _get_pkg_info_filepath
    raise UnsupportedPackageError(self._osutils.basename(package_dir))
aws_lambda_builders.workflows.python_pip.packager.UnsupportedPackageError: Unable to retrieve name/version for package: gendercomputer
Builder workflow failed
Traceback (most recent call last):
  File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflow.py", line 374, in run
    action.execute()
  File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 71, in execute
    package_builder.build_dependencies(
  File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 157, in build_dependencies
    self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path)
  File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 262, in build_site_packages
    wheels, packages_without_wheels = self._download_dependencies(scratch_directory, requirements_filepath)
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 286, in _download_dependencies
    deps = self._download_all_dependencies(requirements_filename, directory)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 370, in _download_all_dependencies
    Package(directory, filename, self.python_exe)
  File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 522, in __init__
    self._name, self._version = self._calculate_name_and_version()
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 564, in _calculate_name_and_version
    name, version = info_fetcher.get_package_name_and_version(sdist_path)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 706, in get_package_name_and_version
    pkg_info_filepath = self._get_pkg_info_filepath(package_dir)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 618, in _get_pkg_info_filepath
    raise UnsupportedPackageError(self._osutils.basename(package_dir))
aws_lambda_builders.workflows.python_pip.packager.UnsupportedPackageError: Unable to retrieve name/version for package: gendercomputer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/__main__.py", line 117, in main
    builder.build(
  File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/builder.py", line 172, in build
    return workflow.run()
           ^^^^^^^^^^^^^^
  File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflow.py", line 116, in wrapper
    func(self, *args, **kwargs)
  File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflow.py", line 385, in run
    raise WorkflowUnknownError(workflow_name=self.NAME, action_name=action.NAME, reason=str(ex))
aws_lambda_builders.exceptions.WorkflowUnknownError: PythonPipBuilder:ResolveDependencies - Unable to retrieve name/version for package: gendercomputer
2024-08-14 13:41:17,306 | Build inside container returned response {"jsonrpc": "2.0", "id": 1, "error": {"code": 400, "message": "PythonPipBuilder:ResolveDependencies - Unable to retrieve name/version for
package: gendercomputer"}}                                                                                                                                                                                  
2024-08-14 13:41:17,634 | Exception raised during the execution                                                                                                                                             

Build Failed

Expected result:

Build successful.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

Notice No module named \'setuptools\' in the logs.
The issue is not reproducible when changing Runtime to python3.11.

{
  "version": "1.121.0",
  "system": {
    "python": "3.12.5",
    "os": "macOS-14.6.1-arm64-arm-64bit"
  },
  "additional_dependencies": {
    "docker_engine": "27.1.1",
    "aws_cdk": "Not available",
    "terraform": "1.9.4"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}

[hawflau]

@zhemaituk thanks for raising the issue. I verified that the same issue happened if I ran sam build --use-container for Python3.12. However, there's no issue if I switch Python3.11 or if I run sam build without --use-container.

Marking it as bug.

[Dizotoff]

I'm experiencing a similar issue when not using the --use-container flag. It fails a certain percentage of the time, but not always

[kru2710shna]

Quick Questions: Do you need to assign the issue to me , or i can hop on directly ?

[johnc44]

I am also seeing the same problem when using "sam build" without "--use-container". This happens every time. We have observed it on different machines (Mac and Windows/WSL) including with fresh versions of our code, sno nothing should be getting cached. It also fails on python 3.13 but works fine on 3.11.

--use-container doesn't get that far as we are using a private repo which needs access to our ssh keys.

For now we are sticking to 3.11.

[nowei]

I'm also running into this issue with --use-container on python3.12.

It also breaks when I try it with python 3.11 when I only have a pyproject.toml, but this seems to work properly on 3.11 when I have a pyproject.toml with a setup.py.