Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug: Sam Build fails when specifying Jar location for Java Function CodeUri #4575

Closed
cortexcompiler opened this issue Jan 13, 2023 · 13 comments
Labels
area/build sam build command area/java blocked/more-info-needed More info is needed from the requester. If no response in 14 days, it will become stale.

Comments

@cortexcompiler
Copy link

Description:

I am using gradle to create an "Uber Jar" file per Lambda Function leveraging a single gradle.build file in the root of my project (this is similar to managing my own esbuild or webpack for node, and that does work). I am trying to point to the already created jar file similar to what I have seen in various samples, such as:
https://github.com/aws-samples/aws-sam-java-rest/blob/master/template.yaml#L22

    Properties:
      CodeUri: target/aws-sam-java-rest-1.0.0.jar
      Handler: com.amazonaws.handler.GetOrderHandler::handleRequest

Mine looks like:

    Properties:
      CodeUri: build/libs/lambda-hello-uber.jar
      Handler: HelloHandler::handleRequest

Running The aws-sample above results in this error (a Maven project):

Building codeuri: <full-path>/aws-sam-java-rest/target/aws-sam-java-rest-1.0.0.jar runtime: java8 metadata: {} architecture: x86_64 functions: GetOrderFunction, GetOrdersFunction, UpdateOrderFunction, DeleteOrderFunction, CreateOrderFunction
Running JavaMavenWorkflow:CopySource

Build Failed
Error: JavaMavenWorkflow:CopySource - [Errno 20] Not a directory: '<full-path>/aws-sam-java-rest/target/aws-sam-java-rest-1.0.0.jar'

My project results in a similar error (a Gradle project):

Building codeuri: <full-path>/build/libs/lambda-hello-uber.jar runtime: java11 metadata: {} architecture: arm64 functions: HelloFunction
Running JavaGradleWorkflow:GradleBuild

Build Failed
Error: JavaGradleWorkflow:GradleBuild - Gradle Failed: Gradle build file not found: <full-path>/build/libs/lambda-hello-uber.jar/build.gradle

Steps to reproduce:

  • Create a Maven or Gradle-based Java project that points to a pre-built Jar in the CodeUri for an AWS::Serverless::Function that uses Runtime: java11.
  • Run sam build

This can be done using the aws sample: https://github.com/aws-samples/aws-sam-java-rest

Observed result:

Relevant deug output:

2023-01-13 00:01:47,488 | 5 resources found in the stack 
2023-01-13 00:01:47,488 | Found Serverless function with name='HelloFunction' and CodeUri='build/libs/lambda-hello-uber.jar'
2023-01-13 00:01:47,488 | Found Serverless function with name='TimeFunction' and CodeUri='build/libs/lambda-time-uber.jar'
2023-01-13 00:01:47,488 | Instantiating build definitions
2023-01-13 00:01:47,488 | No previous build graph found, generating new one
2023-01-13 00:01:47,488 | Unique function build definition found, adding as new (Function Build Definition: BuildDefinition(java11, /Users/username/project/java-lambda-poc/build/libs/lambda-hello-uber.jar, Zip, , a99be4e2-e3fc-42fe-80da-5720fc05a2ec, {}, {}, arm64, []), Function: Function(function_id='HelloFunction', name='HelloFunction', functionname='HelloFunction', runtime='java11', memory=512, timeout=10, handler='HelloHandler::handleRequest', imageuri=None, packagetype='Zip', imageconfig=None, codeuri='/Users/username/project/java-lambda-poc/build/libs/lambda-hello-uber.jar', environment={'Variables': {'ENVIRONMENT': 'Environment', 'LOG_LEVEL': 'info', 'REGION': 'us-west-2', 'AWS_XRAY_CONTEXT_MISSING': 'LOG_ERROR', 'POWERTOOLS_TRACER_CAPTURE_HTTPS_REQUESTS': True, 'POWERTOOLS_TRACER_CAPTURE_RESPONSE': True, 'POWERTOOLS_METRICS_NAMESPACE': 'Testing-POC', 'POWERTOOLS_SERVICE_NAME': 'local-hello'}}, rolearn=None, layers=[], events=None, metadata={'SamResourceId': 'HelloFunction'}, inlinecode=None, codesign_config_arn=None, architectures=['arm64'], function_url_config=None, stack_path=''))
2023-01-13 00:01:47,488 | Unique function build definition found, adding as new (Function Build Definition: BuildDefinition(java11, /Users/username/project/java-lambda-poc/build/libs/lambda-time-uber.jar, Zip, , 845c91a1-9dca-4663-9417-dc21b0bbd44b, {}, {}, arm64, []), Function: Function(function_id='TimeFunction', name='TimeFunction', functionname='TimeFunction', runtime='java11', memory=512, timeout=10, handler='TimeHandler::handleRequest', imageuri=None, packagetype='Zip', imageconfig=None, codeuri='/Users/username/project/java-lambda-poc/build/libs/lambda-time-uber.jar', environment={'Variables': {'ENVIRONMENT': 'Environment', 'LOG_LEVEL': 'info', 'REGION': 'us-west-2', 'AWS_XRAY_CONTEXT_MISSING': 'LOG_ERROR', 'POWERTOOLS_TRACER_CAPTURE_HTTPS_REQUESTS': True, 'POWERTOOLS_TRACER_CAPTURE_RESPONSE': True, 'POWERTOOLS_METRICS_NAMESPACE': 'Testing-POC', 'POWERTOOLS_SERVICE_NAME': 'local-hello'}}, rolearn=None, layers=[], events=None, metadata={'SamResourceId': 'TimeFunction'}, inlinecode=None, codesign_config_arn=None, architectures=['arm64'], function_url_config=None, stack_path=''))
2023-01-13 00:01:47,489 | Building codeuri: /Users/username/project/java-lambda-poc/build/libs/lambda-hello-uber.jar runtime: java11 metadata: {} architecture: arm64 functions: HelloFunction
2023-01-13 00:01:47,489 | Building to following folder /Users/username/project/java-lambda-poc/.aws-sam/build/HelloFunction
2023-01-13 00:01:47,489 | Looking for a supported build workflow in following directories: ['/Users/username/project/java-lambda-poc/build/libs/lambda-hello-uber.jar', '/Users/username/project/java-lambda-poc']
2023-01-13 00:01:47,490 | Loading workflow module 'aws_lambda_builders.workflows'
2023-01-13 00:01:47,495 | Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2023-01-13 00:01:47,496 | Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
2023-01-13 00:01:47,498 | Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
2023-01-13 00:01:47,499 | Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
2023-01-13 00:01:47,502 | Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
2023-01-13 00:01:47,503 | Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
2023-01-13 00:01:47,504 | Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
2023-01-13 00:01:47,506 | Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
2023-01-13 00:01:47,507 | Registering workflow 'NodejsNpmEsbuildBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm-esbuild', application_framework=None)'
2023-01-13 00:01:47,508 | Found workflow 'JavaGradleWorkflow' to support capabilities 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
2023-01-13 00:01:48,509 | Running workflow 'JavaGradleWorkflow'
2023-01-13 00:01:48,509 | Running JavaGradleWorkflow:GradleBuild
2023-01-13 00:01:48,510 | JavaGradleWorkflow:GradleBuild failed
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/aws-sam-cli/1.61.0/libexec/lib/python3.8/site-packages/aws_lambda_builders/workflows/java_gradle/actions.py", line 48, in _build_project
    self.subprocess_gradle.build(
  File "/opt/homebrew/Cellar/aws-sam-cli/1.61.0/libexec/lib/python3.8/site-packages/aws_lambda_builders/workflows/java_gradle/gradle.py", line 34, in build
    raise BuildFileNotFoundError(build_file)
aws_lambda_builders.workflows.java_gradle.gradle.BuildFileNotFoundError: Gradle Failed: Gradle build file not found: /Users/username/project/java-lambda-poc/build/libs/lambda-hello-uber.jar/build.gradle

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/aws-sam-cli/1.61.0/libexec/lib/python3.8/site-packages/aws_lambda_builders/workflow.py", line 301, in run
    action.execute()
  File "/opt/homebrew/Cellar/aws-sam-cli/1.61.0/libexec/lib/python3.8/site-packages/aws_lambda_builders/workflows/java_gradle/actions.py", line 30, in execute
    self._build_project(init_script_file)
  File "/opt/homebrew/Cellar/aws-sam-cli/1.61.0/libexec/lib/python3.8/site-packages/aws_lambda_builders/workflows/java_gradle/actions.py", line 56, in _build_project
    raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: Gradle Failed: Gradle build file not found: /Users/username/project/java-lambda-poc/build/libs/lambda-hello-uber.jar/build.gradle

Build Failed
2023-01-13 00:01:48,634 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2023-01-13 00:01:48,635 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': 'e5823110-fe18-4366-9ca6-c952f887d721', 'installationId': 'ba633f33-0ca6-43f8-bba9-08a46aae8468', 'sessionId': '869ff89c-8fd9-496e-bb18-f69cd0b71f93', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.16', 'samcliVersion': '1.61.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'metricSpecificAttributes': {'projectType': 'CFN', 'gitOrigin': None, 'projectName': '6d41a41f844def384ad978c8293ffd292ffd26f68f1fd2301a477527dc596871', 'initialCommit': None}, 'duration': 1424, 'exitReason': 'WorkflowFailedError', 'exitCode': 1}}]}
2023-01-13 00:01:48,635 | Unable to find Click Context for getting session_id.
2023-01-13 00:01:48,636 | Sending Telemetry: {'metrics': [{'events': {'requestId': 'c0f32b54-c0ce-4dae-bc8e-b7b62fc24a71', 'installationId': 'ba633f33-0ca6-43f8-bba9-08a46aae8468', 'sessionId': '869ff89c-8fd9-496e-bb18-f69cd0b71f93', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.16', 'samcliVersion': '1.61.0', 'metricSpecificAttributes': {'events': [{'event_name': 'BuildWorkflowUsed', 'event_value': 'java-gradle', 'thread_id': 8073607808, 'time_stamp': '2023-01-13 05:01:47.490'}]}}}]}
2023-01-13 00:01:49,014 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
2023-01-13 00:01:49,021 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Error: JavaGradleWorkflow:GradleBuild - Gradle Failed: Gradle build file not found: /Users/username/project/java-lambda-poc/build/libs/lambda-hello-uber.jar/build.gradle

Expected result:

The specified Jar file is moved into the .aws-sam/build/FunctionName directory without attempting to run a gradle build

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

  1. OS: Mac
  2. sam --version: SAM CLI, version 1.61.0
  3. AWS region: us-west-2
@cortexcompiler cortexcompiler added stage/needs-triage Automatically applied to new issues and PRs, indicating they haven't been looked at. type/bug labels Jan 13, 2023
@cortexcompiler
Copy link
Author

Looks like a workaround for now would be to use a makefile BuildMethod, like this:

template.yaml

  HelloFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: .
      Handler: HelloHandler::handleRequest
      Runtime: java11
    Metadata:
      BuildMethod: makefile
  TimeFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: .
      Handler: TimeHandler::handleRequest
      Runtime: java11
    Metadata:
      BuildMethod: makefile

Makefile

build/libs/%.jar:
	./gradlew build

build-HelloFunction: build/libs/lambda-hello-uber.jar
	cp build/libs/lambda-hello-uber.jar $(ARTIFACTS_DIR)

build-TimeFunction: build/libs/lambda-time-uber.jar
	cp build/libs/lambda-time-uber.jar $(ARTIFACTS_DIR)

This looks to work, though unfortunately it looks like each build happens in it's own directory, so the gradle build step is duplicated. This is seen once for each Lambda:

Running CustomMakeBuilder:CopySource
Running CustomMakeBuilder:MakeBuild

I could see that the make commands run out of a different directory for each Lambda:

  • /private/var/folders/th/69sc4h292qnbr4zbm22vjlh80000gp/T/tmpiyzmwiw0
  • /private/var/folders/th/69sc4h292qnbr4zbm22vjlh80000gp/T/tmp6to8kqkc

To prevent the redundant .gradlew build that step can be run before sam build.

@cortexcompiler
Copy link
Author

cortexcompiler commented Jan 13, 2023

This is somewhat related to #3227, as allowing developers to specify a pre-built artifact would enable whatever gradle configuration they prefer. Their CI pipeline would just need to run the appropriate gradle build steps prior to calling sam build.

I feel that the current Java (gradle and maven) and even Node build processes within sam build are a little to opinionated or inflexible and allowing developers to reference pre-built artifacts would help folks customize as needed. It seems like this worked perviously for Java and does work for Node (with minor ignored warnings/errors from SAM)

@mndeveci
Copy link
Contributor

Thanks for reporting this issue @cortexcompiler ,

We are working on build-in-source feature, which will let customers to run their builds inside build directory, with that improvement, Makefile concerns will be resolved. You can read more about it here: #4571

We also have another improvement, where you can skip to build certain functions. Which will be useful if you want to build some of the functions manually, and let others to be build by SAM CLI. For instance, if you want to build TimeFunction manually but let SAM CLI to build HelloFunction, you can configure your template like;

 HelloFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: .
      Handler: HelloHandler::handleRequest
      Runtime: java11
  TimeFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: .
      Handler: TimeHandler::handleRequest
      Runtime: java11
    Metadata:
      SkipBuild: True

I believe this SkipBuild is not documented, so we will be updating our documentation with that.

Please let us know if you have questions.

Thanks!

@mndeveci mndeveci added blocked/more-info-needed More info is needed from the requester. If no response in 14 days, it will become stale. area/build sam build command area/java and removed type/bug stage/needs-triage Automatically applied to new issues and PRs, indicating they haven't been looked at. labels Jan 13, 2023
@cortexcompiler
Copy link
Author

Thanks for the response @mndeveci , I am not sure how "Build in source directory" would help remove the need for the Makefile but perhaps I can wait for more details on the feature. I noticed that if you specify a directory containing a pre-bundled file in a TypeScript Lambda sam build will complain but then copy the source over. I was hoping it would do something like that for the Java:

Building codeuri: <project-dir>/dist/function-name runtime: nodejs16.x metadata: {} architecture: arm64 functions: FunctionName
package.json file not found. Continuing the build without dependencies.
Running NodejsNpmBuilder:CopySource 

I will try out SkipBuild to see what it does.

@cortexcompiler
Copy link
Author

@mndeveci with the Java/Gradle project SkipBuild does exactly what it says and builds nothing for the Lambda, and then it adds "../../" to the CodeUri for the handler. This will not really do what I need.
For the TypeScript Lambda it seems to ignore the SkipBuild and just copies the pre-built code into the .aws-sam/build/FunctionName directory and appropriately updates the CodeUri in the .aws-sam/build/template.yaml. The output is the same as not having the SkipBuild Metadata:

Building codeuri: <project-dir>/dist/function-name runtime: nodejs16.x metadata: {'SkiBuild': True} architecture: arm64 functions: FunctionName
package.json file not found. Continuing the build without dependencies.
Running NodejsNpmBuilder:CopySource

This is essentially what I am looking for with the Java project. I have pre-built a minimized "Uber-Jar" and I would like it (or more specifically, the contents) put into the .aws-sam/build/FunctionName directory and then the .aws-sam/build/template.yaml should set the CodeUri to be the Function Name.

I have this working using the following generic Makefile and a pattern:

# This is here in case the gradle build is not run before sam build, but it is faster to run it first
build/libs/%.jar:
	./gradlew build

# This requires that part of the jar name match the function name (e.g. helloFunction => lambda-hello-uber.jar)
build-%Function: build/libs/lambda-%-uber.jar
	(cd $(ARTIFACTS_DIR) && exec jar -xf ../../../$<)

@mndeveci
Copy link
Contributor

mndeveci commented Jan 18, 2023

@cortexcompiler from what I understand you want to use an uber-jar plugin and want to opt out of regular sam build since you want to run specific target in Maven. So you have 2 options here;

  1. Building with an external tool, and keeping template.yaml with built artifacts.

First of all, if you build all of your functions with a external tool (maven in this case), then you don't need to run sam build at all. Because all the things that sam build needs to do is already done, and you can skip to using next commands (sam local, sam package, sam deploy etc.).

But if you still want to run sam build (maybe some of your functions are manually built, but some can be built by SAM CLI) then you can add SkipBuild: True into Metadata section of each function resource that you want to skip building. I cloned your example, ran the commands in the order you specified and I can see that sam build is failing. But if I add SkipBuild: True, then sam build also succeeds. You mention that template.yaml now has ../../<pre-built-jar-file>, but this is expected since built template is moved into .aws-sam/build folder and it needs to update relative location of the artifacts.

  1. Building with a Makefile

This would work but as you mentioned in your 2nd comment, but this will duplicate the build by the amount of functions that you have in your template. And each build will happen in a scratch temporary directory, so any caching (or incremental build) of the underlying tool might not work. This should be fixed once we support Build in Source for Java functions (which will run Makefile commands in the source folder so that we can utilize caching/incremental build of the underlying package manager). You can read more about Build in Source project here: #4571


For your last comment about NodeJS feature, it is only applicable for interpreted languages (NodeJS and Python) since they don't require compiling of the source code. If they define their lambda function without any dependency, then they don't need to add package.json or requirements.txt file and we will just copy their source files into build directory. This is not something that we can do for complied languages like Java.

@cortexcompiler
Copy link
Author

cortexcompiler commented Jan 18, 2023

Thanks @mndeveci. I don't think sam package will do what I need, as I do not want to upload the code to S3. I am running sam build with the intention of packaging up .aws-sam/ and the samconfig.toml file as a versioned deployable artifact. This artifact is then used with sam deploy --config-env in order to deploy to different accounts for different "environments". (this is working with TypeScript and the Makefile approach for Java)

Using sam build makes it so that I do not need to specify an S3 location and all of the Lambda code is bundled into the .aws-sam/build/<functionName> directories, and that bundled code is referenced from the CodeUri properties in the .aws-sam/template.yaml that sam build generates.

Perhaps I could use the SkipBuild: True and attempt to create the .aws-sam directory myself... I am concerned I might miss something that sam build does that sam deploy expects. The Build in Source could possibly help with this. Is there some other approach recommended for creating versioned deployable artifacts with SAM?

@mndeveci
Copy link
Contributor

OK I think I get what you are trying to do now. For your use case, you can either use Makefile approach (which won't be super performant until build in source is completed) or you can do something like this;

  • Add SkipBuild: True to the functions that you build outside of SAM CLI
  • For built jar files, you can move them into a folder, something like .aws-sam/pre-build
  • Run sam build which will move built template into .aws-sam/build folder and update references to the jar files
  • Then you can package samconfig.toml, .aws-sam/build and .aws-sam/pre-build folders

Let me know if this doesn't work for your use case.

@mndeveci
Copy link
Contributor

@cortexcompiler we have also published a blog-post about how to build Serverless Java Applications using SAM CLI with different ways. I would recommend giving it a read: https://aws.amazon.com/blogs/compute/building-serverless-java-applications-with-the-aws-sam-cli/

I will be resolving this issue for now, please let us know if you have other questions.

@mndeveci mndeveci closed this as not planned Won't fix, can't repro, duplicate, stale Mar 10, 2023
@github-actions
Copy link
Contributor

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

@cj-mantas-gridinas
Copy link

cj-mantas-gridinas commented Sep 11, 2024

@mndeveci In what version is SkipBuild support added? I'm using 1.121, but running sam build --template template.yml lambda on following

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: An AWS Serverless Application Model template describing your function.
Resources:
  lambda:
    Type: AWS::Serverless::Function
    Metadata:
      SkipBuild: True
    Properties:
      FunctionName: "foo"
      CodeUri: ./build/libs/lambda.zip
      Description: lambda with zip containing libraries as dedicated jars
      MemorySize: 2048
      Timeout: 600
      Handler: foo.Main::handleRequest
      Runtime: java21
      Architectures:
        - x86_64
      EphemeralStorage:
        Size: 1024
      EventInvokeConfig:
        MaximumEventAgeInSeconds: 21600
        MaximumRetryAttempts: 2
        DestinationConfig: {}
      PackageType: Zip
      SnapStart:
        ApplyOn: None
      RuntimeManagementConfig:
        UpdateRuntimeOn: Auto
      VpcConfig:
        SecurityGroupIds:
          - sg-1
        SubnetIds:
          - subnet-1
          - subnet-2
        Ipv6AllowedForDualStack: false

The SAM cli tool picks up that i'm using gradle, but fails to find a gradle build file in ./build/libs/lambda.zip/build.gradle? Which is confusing, because I have set skip build flag, and the aws toolkit in intellij still tries to invoke the build command to build the project, and as a result causes it to fail. The only working part in your previous suggestions is to use the makefile.

Even with the makefile build overtaking the actual build, deployment fails, because SAM expects that the resulting binary is a jar instead of a zip that you would deploy to actual lambda. Only the unzipping the contents into ARTIFACT_DIR solves the problems.

@mndeveci
Copy link
Contributor

@cj-mantas-gridinas what happens if you run sam build --template template.yml (without defining a specific lambda function name)? If it works this might be a bug when building single lambda functions, if so would you mind creating a new issue for your use case?

@cj-mantas-gridinas
Copy link

Yeah, that works. Problem is that it's not how the intellij plugin works. As a result, it's a nonsolution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/build sam build command area/java blocked/more-info-needed More info is needed from the requester. If no response in 14 days, it will become stale.
Projects
None yet
Development

No branches or pull requests

3 participants