Allow to run node_modules binaries from makefile builder #2565

HaaLeo · 2021-01-29T08:01:40Z

Description:

When I try to execute scripts from locally npm packages the makefile builder fails

Steps to reproduce:

create a template containing a simple lambda with a makefile builder:

MyLambda:
  Type: AWS::Serverless::Function
  Properties:
    CodeUri: ./
    Handler: index.lambdaHandler
    Runtime: nodejs12.x
    MemorySize: 128
    Timeout: 60
    Description: my lambda.
    Policies:
      - AWSLambdaBasicExecutionRole
  Metadata:
    BuildMethod: makefile

A package.json with dependencies, for example typescript:

{
    "scripts": {
        "build": "tsc -p tsconfig.json"
    },
    "devDependencies": {
        "typescript": "^4.1.3"
    }
}

A Makefile containing:

build-MyLambda:
    npm run build #or node node_modules/.bin/tsc -p tsconfig.json
.PHONY: build
build:
    npm run build

A tsconfig.json file:

{
    "compilerOptions": {
        "module": "commonjs",
        "target": "es2019",
        "noImplicitAny": true,
        "preserveConstEnums": true,
        "outDir": "./dist",
        "sourceMap": true,
        "moduleResolution": "node"
    },
    "include": [
        "src/**/*"
    ],
    "exclude": [
        "node_modules",
        "**/*.spec.ts"
    ]
}

Run make build to confirm the setup is working.
Now run sam build --debug to reproduce the error

Observed result:

sam build --debug errors with the following message:

2021-01-29 08:56:16,376 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2021-01-29 08:56:16,479 | 'build' command is called
2021-01-29 08:56:16,485 | Collected default values for parameters: {}
2021-01-29 08:56:16,507 | 1 resources found in the template
2021-01-29 08:56:16,507 | Found Serverless function with name='MyLambda' and CodeUri='./'
2021-01-29 08:56:16,507 | Collected default values for parameters: {}
2021-01-29 08:56:16,528 | Instantiating build definitions
2021-01-29 08:56:16,530 | Unique function build definition found, adding as new (Function Build Definition: BuildDefinition(nodejs12.x, ../, Zip, , ae285d5a-7ad0-4790-b62a-82c2d9b32503, {'BuildMethod': 'makefile'}, []), Function: Function(name='InvalidateCloudFrontCache', functionname='InvalidateCloudFrontCache', runtime='nodejs12.x', memory=128, timeout=60, handler='index.lambdaHandler', imageuri=None, packagetype='Zip', imageconfig=None, codeuri='../', environment=None, rolearn=None, layers=[], events=None, metadata={'BuildMethod': 'makefile'}, codesign_config_arn=None))
2021-01-29 08:56:16,531 | Building codeuri: ../ runtime: nodejs12.x metadata: {'BuildMethod': 'makefile'} functions: ['InvalidateCloudFrontCache']
2021-01-29 08:56:16,596 | Building to following folder /Somewhere/my-lambda/.aws-sam/build/InvalidateCloudFrontCache
2021-01-29 08:56:16,597 | Loading workflow module 'aws_lambda_builders.workflows'
2021-01-29 08:56:16,600 | Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2021-01-29 08:56:16,602 | Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
2021-01-29 08:56:16,604 | Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
2021-01-29 08:56:16,605 | Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
2021-01-29 08:56:16,607 | Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
2021-01-29 08:56:16,609 | Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
2021-01-29 08:56:16,622 | Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
2021-01-29 08:56:16,624 | Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
2021-01-29 08:56:16,718 | Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
2021-01-29 08:56:16,718 | Found workflow 'CustomMakeBuilder' to support capabilities 'Capability(language='provided', dependency_manager=None, application_framework=None)'
2021-01-29 08:56:16,719 | Running workflow 'CustomMakeBuilder'
2021-01-29 08:56:16,719 | Running CustomMakeBuilder:CopySource
2021-01-29 08:56:20,847 | CustomMakeBuilder:CopySource succeeded
2021-01-29 08:56:20,847 | Running CustomMakeBuilder:MakeBuild
2021-01-29 08:56:20,847 | Current Artifacts Directory : /Somewhere/my-lambda/.aws-sam/build/InvalidateCloudFrontCache
2021-01-29 08:56:20,847 | executing Make: ['make', '--makefile', '/Somewhere/my-lambda/Makefile', 'build-InvalidateCloudFrontCache']
2021-01-29 08:56:21,400 | CustomMakeBuilder:MakeBuild failed
Traceback (most recent call last):
  File "/usr/local/Cellar/aws-sam-cli/1.15.0/libexec/lib/python3.8/site-packages/aws_lambda_builders/workflows/custom_make/actions.py", line 87, in execute
    self.subprocess_make.run(
  File "/usr/local/Cellar/aws-sam-cli/1.15.0/libexec/lib/python3.8/site-packages/aws_lambda_builders/workflows/custom_make/make.py", line 88, in run
    raise MakeExecutionError(message=err.decode("utf8").strip())
aws_lambda_builders.workflows.custom_make.make.MakeExecutionError: Make Failed: internal/modules/cjs/loader.js:818
  throw err;
  ^

Error: Cannot find module '../lib/tsc.js'
Require stack:
- /private/var/folders/_h/33xwt_j11sjd5p50drcl2kfr0000gn/T/tmp_degkr1o/node_modules/.bin/tsc
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:815:15)
    at Function.Module._load (internal/modules/cjs/loader.js:667:27)
    at Module.require (internal/modules/cjs/loader.js:887:19)
    at require (internal/modules/cjs/helpers.js:74:18)
    at Object.<anonymous> (/private/var/folders/_h/33xwt_j11sjd5p50drcl2kfr0000gn/T/tmp_degkr1o/node_modules/.bin/tsc:2:1)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
    at Module.load (internal/modules/cjs/loader.js:863:32)
    at Function.Module._load (internal/modules/cjs/loader.js:708:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:60:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/private/var/folders/_h/33xwt_j11sjd5p50drcl2kfr0000gn/T/tmp_degkr1o/node_modules/.bin/tsc'
  ]
}
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! invalidate-cf-cache@0.0.1 build: `tsc -p tsconfig.json`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the invalidate-cf-cache@0.0.1 build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/z0049e1t/.npm/_logs/2021-01-29T07_56_21_371Z-debug.log
make: *** [build-InvalidateCloudFrontCache] Error 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/aws-sam-cli/1.15.0/libexec/lib/python3.8/site-packages/aws_lambda_builders/workflow.py", line 269, in run
    action.execute()
  File "/usr/local/Cellar/aws-sam-cli/1.15.0/libexec/lib/python3.8/site-packages/aws_lambda_builders/workflows/custom_make/actions.py", line 97, in execute
    raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: Make Failed: internal/modules/cjs/loader.js:818
  throw err;
  ^

Error: Cannot find module '../lib/tsc.js'
Require stack:
- /private/var/folders/_h/33xwt_j11sjd5p50drcl2kfr0000gn/T/tmp_degkr1o/node_modules/.bin/tsc
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:815:15)
    at Function.Module._load (internal/modules/cjs/loader.js:667:27)
    at Module.require (internal/modules/cjs/loader.js:887:19)
    at require (internal/modules/cjs/helpers.js:74:18)
    at Object.<anonymous> (/private/var/folders/_h/33xwt_j11sjd5p50drcl2kfr0000gn/T/tmp_degkr1o/node_modules/.bin/tsc:2:1)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
    at Module.load (internal/modules/cjs/loader.js:863:32)
    at Function.Module._load (internal/modules/cjs/loader.js:708:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:60:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/private/var/folders/_h/33xwt_j11sjd5p50drcl2kfr0000gn/T/tmp_degkr1o/node_modules/.bin/tsc'
  ]
}
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! invalidate-cf-cache@0.0.1 build: `tsc -p tsconfig.json`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the invalidate-cf-cache@0.0.1 build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/z0049e1t/.npm/_logs/2021-01-29T07_56_21_371Z-debug.log
make: *** [build-InvalidateCloudFrontCache] Error 1

Build Failed
2021-01-29 08:56:22,223 | Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 5846, 'exitReason': 'WorkflowFailedError', 'exitCode': 1, 'requestId': 'ed59d879-ed40-4e0e-ad2d-2a0f678c8597', 'installationId': '17adb356-4bd3-44f0-8b2c-2e783726a1f8', 'sessionId': '62216092-cb80-44e7-8292-83b33897c81f', 'executionEnvironment': 'CLI', 'pyversion': '3.8.7', 'samcliVersion': '1.15.0'}}]}
2021-01-29 08:56:22,927 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Error: CustomMakeBuilder:MakeBuild - Make Failed: internal/modules/cjs/loader.js:818
  throw err;
  ^

Error: Cannot find module '../lib/tsc.js'
Require stack:
- /private/var/folders/_h/33xwt_j11sjd5p50drcl2kfr0000gn/T/tmp_degkr1o/node_modules/.bin/tsc
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:815:15)
    at Function.Module._load (internal/modules/cjs/loader.js:667:27)
    at Module.require (internal/modules/cjs/loader.js:887:19)
    at require (internal/modules/cjs/helpers.js:74:18)
    at Object.<anonymous> (/private/var/folders/_h/33xwt_j11sjd5p50drcl2kfr0000gn/T/tmp_degkr1o/node_modules/.bin/tsc:2:1)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
    at Module.load (internal/modules/cjs/loader.js:863:32)
    at Function.Module._load (internal/modules/cjs/loader.js:708:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:60:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/private/var/folders/_h/33xwt_j11sjd5p50drcl2kfr0000gn/T/tmp_degkr1o/node_modules/.bin/tsc'
  ]
}
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! invalidate-cf-cache@0.0.1 build: `tsc -p tsconfig.json`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the invalidate-cf-cache@0.0.1 build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/z0049e1t/.npm/_logs/2021-01-29T07_56_21_371Z-debug.log
make: *** [build-InvalidateCloudFrontCache] Error 1

Expected result:

The build does not error.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

Interestingly the build does not error if you install typescript globally (npm i -g typescript) and call it in the makefile directly.
However, I want all the dependencies of my project installed locally, not globally to build my app.

OS: macOS Catalina 10.15.7
sam --version: 1.15.0

The text was updated successfully, but these errors were encountered:

mgrandis · 2021-02-04T01:32:20Z

Hi @HaaLeo,
Thanks for reporting this.
We have an ongoing PR to package local NPM dependencies: aws/aws-lambda-builders#215
Would this help?

HaaLeo · 2021-02-04T10:00:49Z

The PR you mentioned seems to be at least related. I also found aws/aws-lambda-builders#106 which seems to be closer related to this issue.
However, after a brief look at those PRs I am unsure whether those consider the makefile builder as well.

mgrandis · 2021-02-05T02:17:12Z

Could reproduce using your example (devdependencies needs to be renamed to devDependencies)
However, the make build only worked after I ran npm install.

I tried adding npm install to the Makefile before the npm build but the error persisted.

Based on the last comment here jupyter/nbdime#201 I also tried adding rm -rf node_modules before npm install, this time the execution took longer and ended on a different error.

hung3102 · 2021-07-22T14:53:58Z

It takes much of time on npm install when removing node_modules in each function build.
Remove node_modules/.bin and reinstall typescript works for me:

build-lambda-common:
        npm install
        rm -rf node_modules/.bin
        npm i typescript

MattLish · 2021-11-18T10:40:45Z

For anyone else that comes across this, it seems that with an npm install the node_modules/.bin seems to get broken.
If you replace npm install in your Makefile with npm ci, it seems to solve the problem (although unfortunately, a little slower).

e.g.

build-MyLambda:
    npm ci
    npm run build

heitormon · 2021-12-08T17:57:12Z

use the absolute path in node_modules/.bin

…comment)

jfuss · 2022-08-30T21:18:27Z

Solution is explained in #2565 (comment)

Closing

HaaLeo · 2022-08-31T11:57:38Z

IMHO the proposed solution is insufficient, because npm ci takes way longer than a normal npm install.

Please consider to reopen this issue and to provide a sustainable fix 🙏🏼.

siarhei-klimuts · 2023-11-27T12:05:24Z

I've had the same issue.

Looks like sam build copies entire project folder including node_modules installed locally and then uses it to build a project but fails for some reason. When I manually remove node_modules, sam build installs node_modules itself (if you have npm install in Makefile) and successfully builds the project.

So running rm -rf node_modules/.bin before npm install in Makefile works for me perfectly.

The only question I have - why sam build can't build my project using existing node_modules?

mgrandis added the area/build sam build command label Feb 4, 2021

gojko mentioned this issue Jul 12, 2021

Esbuild nodejs builder aws/aws-lambda-builders#252

Closed

aws-sam-cli-bot added maintainer/need-followup and removed maintainer/need-followup labels Oct 7, 2021

mildaniel mentioned this issue Nov 9, 2021

Enable esbuild in the Node.js builder aws/aws-lambda-builders#307

Merged

aws-sam-cli-bot added maintainer/need-followup and removed maintainer/need-followup labels Nov 18, 2021

aws-sam-cli-bot added maintainer/need-followup and removed maintainer/need-followup labels Dec 9, 2021

JordanKoeller added a commit to JordanKoeller/TrinityMutualAid that referenced this issue Feb 14, 2022

Updating lambda makefile based on this comment aws/aws-sam-cli#2565 (…

be2c8c8

…comment)

jfuss closed this as completed Aug 30, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow to run node_modules binaries from makefile builder #2565

Allow to run node_modules binaries from makefile builder #2565

HaaLeo commented Jan 29, 2021 •

edited

Loading

mgrandis commented Feb 4, 2021

HaaLeo commented Feb 4, 2021

mgrandis commented Feb 5, 2021

hung3102 commented Jul 22, 2021

MattLish commented Nov 18, 2021

heitormon commented Dec 8, 2021

jfuss commented Aug 30, 2022

HaaLeo commented Aug 31, 2022

siarhei-klimuts commented Nov 27, 2023

Allow to run node_modules binaries from makefile builder #2565

Allow to run node_modules binaries from makefile builder #2565

Comments

HaaLeo commented Jan 29, 2021 • edited Loading

Description:

Steps to reproduce:

Observed result:

Expected result:

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

mgrandis commented Feb 4, 2021

HaaLeo commented Feb 4, 2021

mgrandis commented Feb 5, 2021

hung3102 commented Jul 22, 2021

MattLish commented Nov 18, 2021

heitormon commented Dec 8, 2021

jfuss commented Aug 30, 2022

HaaLeo commented Aug 31, 2022

siarhei-klimuts commented Nov 27, 2023

HaaLeo commented Jan 29, 2021 •

edited

Loading