Native extensions aren't being compiled

[kevinreedy]

I'm attempting use grunt-aws-lambda for a project (https://github.com/kevinreedy/chef-asg-cleanup) that uses https://github.com/normanjoyner/chef-api. One of its dependencies is https://github.com/quartzjer/ursa, which has native extensions. When I run an npm install, the extensions are built! When I run grunt lambda_package, they are not. Steps to reproduce below:

$ git clone git@github.com:kevinreedy/chef-asg-cleanup.git
$ cd chef-asg-cleanup/
$ npm install
[output snipped]
$ grunt lambda_package
Running "lambda_package:default" (lambda_package) task
chef-asg-cleanup@0.1.0 ../../../tmp/1443742405683.5662/node_modules/chef-asg-cleanup
Created package at ./dist/chef-asg-cleanup_0-1-0_2015-9-1-23-33-25.zip

Done, without errors.
$ cd dist
$ unzip chef-asg-cleanup_0-1-0_2015-9-1-23-33-25.zip
[output snipped]
$ cd ..
root@8dd621056992:/usr/src/app# ls node_modules/chef-api/node_modules/ursa/build
Makefile  Release  binding.Makefile  config.gypi  ursaNative.target.mk
root@8dd621056992:/usr/src/app# ls dist/node_modules/chef-api/node_modules/ursa/build
ls: cannot access dist/node_modules/chef-api/node_modules/ursa/build: No such file or directory

I imagine there's an option that is needed to pass into npm.commands.install (https://github.com/Tim-B/grunt-aws-lambda/blob/master/tasks/lambda_package.js#L59-L66) to ensure that node-gyp gets called, but it's not obvious from looking at npm's code. Any thoughts to push me in the right direction? Thanks!

[Tim-B]

Thanks for bringing this up.

I can imagine this will be a complicated to resolve if you're developing on an environment that doesn't match the Lambda environment (eg. developing on Windows and running on Lambda/Linux).

Perhaps you could manually build binaries then include them in the correct folder using something like the include_files option.

If you unzip the generated package and run npm build does it work?

[kevinreedy]

I've been using a Docker container to do the build, so that it'll actually build correctly for lambda. As for running npm build . in the root after unzipping the package, it doesn't do anything. But, if I cd node_modules/chef-api/node_modules/ursa && npm build ., it will do the build of the native extensions for that nested requirement.

[kevinreedy]

I was able to get around this using the include_files option for now. It looks like that doesn't include recursively though, so I'm getting a bit verbose. https://github.com/kevinreedy/chef-asg-cleanup/blob/master/Gruntfile.js#L18-L29

[aheissenberger]

@kevinreedy can you share your docker file which simulates the lambda environment - at the moment I use a ec2 instance and this would make it much easier to build the node modules with binary parts (iconv, fibers) - thanks

[kevinreedy]

@aheissenberger take a look at https://github.com/kevinreedy/chef-asg-cleanup for how I'm successfully building now, specifically https://github.com/kevinreedy/chef-asg-cleanup/blob/master/Dockerfile and https://github.com/kevinreedy/chef-asg-cleanup/blob/master/docker-compose.yml

[dcrockwell]

@kevinreedy @aheissenberger @Tim-B,

I've made a library that may help with these kinds of issues: https://github.com/FreeAllMedia/akiro