V3 prerelease - packaging issue

[amitaymolko]

Using #187 ^3.0.0-rc.1 via npm
Packaging seems to be ignoring package settings in serverless.yml (include/exclude)
The output package includes the whole project folder.

package settings:

package:
  include:
    - node_modules/**
    - yarn.lock
  exclude:
    - .git
    - .requirements
    - .ses-mailbox
    - apidoc.json
    - apidoc_shared.js
    - docs
    - cfn_lambda_helpers
    - seed
    - test

I have also set webpackIncludeModules: true

It deploys successfully but there is a lot being zipped that doesn't need to be.

[HyperBrain]

With V3 the module packaging is improved, i.e. webpack decides what needs to be packaged and what not. E.g. if you have 5 modules in your node modules, but your code only uses 1, then only one dependency should be packaged.

Do you use the node-externals package in your webpack config?

As Webpack decides over the packages, the include/exclude settings in the serverless.yml are not relevant anymore (in fact packaging did not work perfectly with previous versions), but exclusions should be declared in the webpack config file.
Additionally, with V3, you can set package: individually: true in your serverless.yml. Then webpack will analyze each function separately and only package the modules needed per function.

BTW: With webpack non-source files will not be packaged at all (however you can use the webpack file-loader oder copyfiles plugins to achieve that explicitly).

Can you please post your webpack.conf.js?

[amitaymolko]

webpack.config.js:

const path = require('path');
const slsw = require('serverless-webpack');
const nodeExternals = require('webpack-node-externals');

module.exports = {
    entry: slsw.lib.entries,
    target: 'node',
    externals: [nodeExternals({
        modulesFromFile: true
    })],
    module: {
        loaders: [
            {
                test: /\.jql$/,
                exclude: /node_modules/,
                loaders: ['jql-loader', 'babel-loader']
            },
            {
                test: /\.jsx?$/,
                exclude: /node_modules/,
                loaders: ['shebang-loader', 'babel-loader']
            },
            {
                test: /\.json$/,
                loader: 'json-loader'
            }
        ]
    },
    output: {
        libraryTarget: 'commonjs',
        path: path.join(__dirname, '.webpack'),
        filename: '[name].js'
    }
};

To better explain the issue:
Files in project folder:

 8300633 -rw-r--r--    1 amitay  admin     869 Aug 13 15:10 .eslintrc.json
  156504 drwxr-xr-x    5 amitay  admin     442 Aug 16 10:57 .git
 8300634 -rw-r--r--    1 amitay  admin     162 Jul  3 16:10 .gitignore
13367421 drwxr-xr-x    3 amitay  admin     272 Aug 16 10:30 .serverless
13334006 drwxr-xr-x    2 amitay  admin     102 Aug 15 18:58 .serverless_plugins
 8300635 drwxr-xr-x    2 amitay  admin     102 Jul  3 16:10 .vscode
11436726 drwxr-xr-x   15 amitay  admin     510 Jul 30 13:00 _optimize
 8594184 drwxr-xr-x    3 amitay  admin     136 Aug 15 18:58 apidoc
13334008 -rw-r--r--    1 amitay  admin     836 Aug 15 18:58 apidoc.json
13334011 -rw-r--r--    1 amitay  admin    2246 Aug 15 18:58 apidoc_shared.js
13334012 -rw-r--r--    1 amitay  admin    1318 Aug 15 18:58 bitbucket-pipelines.yml
13334013 -rw-r--r--    1 amitay  admin    1033 Aug 15 18:58 config.dev.yml
13335563 -rw-r--r--    1 amitay  admin     844 Aug 15 20:45 config.prod.yml
 8649829 drwxr-xr-x    8 amitay  admin     476 Jul  7 10:11 docs
13334015 -rw-r--r--    1 amitay  admin    1561 Aug 15 18:58 env.yml
 8300644 drwxr-xr-x   10 amitay  admin     340 Aug 15 18:58 functions
 8300595 drwxr-xr-x   18 amitay  admin     714 Aug 15 18:58 lib
 7487378 drwxr-xr-x  791 amitay  admin   26928 Aug 13 18:04 node_modules
13334102 -rw-r--r--    1 amitay  admin   21513 Aug 15 18:58 package-lock.json
13334103 -rw-r--r--    1 amitay  admin    1978 Aug 15 18:58 package.json
13334104 -rw-r--r--    1 amitay  admin   19105 Aug 15 20:37 resources.yml
 8300749 drwxr-xr-x    2 amitay  admin     272 Aug 15 18:58 seed
13334109 -rw-r--r--    1 amitay  admin   11711 Aug 15 18:58 serverless.yml
 8300755 drwxr-xr-x    6 amitay  admin     340 Jul  3 16:10 test
13334125 -rw-r--r--    1 amitay  admin     898 Aug 15 18:58 webpack.config.js
13334126 -rw-r--r--    1 amitay  admin  202811 Aug 15 18:58 yarn.lock

Files in zip:

13412535 -rw-r--r--@   1 amitay  admin    6148 Aug 16 10:30 .DS_Store
13382688 -rw-r--r--    1 amitay  admin     869 Aug 16 07:29 .eslintrc.json
13382689 drwxr-xr-x    2 amitay  admin     102 Aug 16 10:30 .serverless_plugins
13382691 drwxr-xr-x    2 amitay  admin     102 Aug 16 10:30 .vscode
13382693 drwxr-xr-x    4 amitay  admin     170 Aug 16 10:30 .webpack
13382726 drwxr-xr-x   15 amitay  admin     510 Aug 16 10:30 _optimize
13382779 drwxr-xr-x    3 amitay  admin     136 Aug 16 10:30 apidoc
13382783 -rw-r--r--    1 amitay  admin    1318 Aug 16 07:29 bitbucket-pipelines.yml
13382784 -rw-r--r--    1 amitay  admin    1033 Aug 16 07:29 config.dev.yml
13382785 -rw-r--r--    1 amitay  admin     844 Aug 16 07:29 config.prod.yml
13382786 drwxr-xr-x    8 amitay  admin     476 Aug 16 10:30 docs
13382897 -rw-r--r--    1 amitay  admin    1561 Aug 16 07:29 env.yml
13382898 drwxr-xr-x   10 amitay  admin     340 Aug 16 10:30 functions
13382958 drwxr-xr-x   18 amitay  admin     714 Aug 16 10:30 lib
13383070 drwxr-xr-x  791 amitay  admin   26928 Aug 16 10:30 node_modules
13383028 -rw-r--r--    1 amitay  admin   21513 Aug 16 07:29 package-lock.json
13383029 -rw-r--r--    1 amitay  admin    1978 Aug 16 07:29 package.json
13383030 -rw-r--r--    1 amitay  admin   19105 Aug 16 07:29 resources.yml
13383031 drwxr-xr-x    2 amitay  admin     272 Aug 16 10:30 seed
13383038 drwxr-xr-x    6 amitay  admin     340 Aug 16 10:30 test
13383068 -rw-r--r--    1 amitay  admin     898 Aug 16 07:29 webpack.config.js
13383069 -rw-r--r--    1 amitay  admin  202811 Aug 16 07:29 yarn.lock

I don't want irrelevant items bundled into the zip: seed, test, etc..

[HyperBrain]

@amitaymolko The contents in the ZIP look indeed as they should not! Something seems to tell webpack to report anything as "external".
From a first glance, the only exceptional thing that I see is the loader definition:
Since webpack 2.x.x the loaders should be defined as:

module:
  rules: [
    {
      test: ...
      use: [
        ... loaders for test pattern here ...
      ]
  ]

Some further questions:
Are you using webpack ^2.0.0 or webpack ^3.0.0 ?
How do you define your function handlers in serverless.yml?
Do you use any other serverless plugins that could trigger the issue?

[amitaymolko]

using webpack: ^3.5.4

we have a single endpoint per resource that routes the different requests:
serverless function handler example(partial):

users:
    handler: functions/users/routes.routes
    events:
      - http:
          method: get
          path: me
          authorizer: ${self:custom.authorizer}
          cors: ${self:custom.cors}
      - http:
          method: get
          path: user/{id}
          authorizer: ${self:custom.authorizer}
          cors: ${self:custom.cors}
    // ... more http endpoints here

serverles.yml:

plugins:
  - serverless-webpack
  - serverless-dynamodb-local
  - serverless-s3-local
  - serverless-offline-scheduler
  - serverless-offline
  - serverless-mocha-plugin
  - api-docs //custom plugin

[amitaymolko]

testing package: individually: true and webpack structure

[amitaymolko]

Ok after fixing webpack structure things seems to be working properly, both individually true and false
Testing deploy to aws now.

Oh another thing, I get:

Serverless: WARNING: Plugin ServerlessWebpack uses deprecated hook before:deploy:createDeploymentArtifacts
Serverless: WARNING: Plugin ServerlessWebpack uses deprecated hook after:deploy:createDeploymentArtifacts

every time I run serverless

[HyperBrain]

Great that it works now 👍 .

The warnings are already addressed with #126 . This is just a warning that it uses the old Serverless hooks. As I implemented the deprecation in Serverless by myself, I can assure that the warning is only annoying but not critical 😄 .

[amitaymolko]

Created new issue for dependency issue I'm having. #190
The packaged structure looks good so I'm closing this issue.