Table of Contents

• SUMMARY
• WHERE TO START
  • Prerequisites
  • Commands
  • Coding Conventions
  • Configurations
    • Loading order
    • Special Cases
    • Server Port
• SYSTEM ARCHITECTURE
  • Model View Controller
  • The Core
  • The Model in MVC
  • The View in MVC
  • The Controller in MVC
  • Services
  • Best Practices
    • 1. Don't use your Model directly in the controller
    • 2. Don't litter the routes
  • Helpers
  • Error Handling
  • Logging
• CUSTOMISATION
  • Behind The Scene
  • Gulp
    • Structure
    • How To Find A Task
• TESTING
• PRODUCTION
• DEBUG

SUMMARY

This NodeJS project uses MVC architecture with Express underspinning. This can be used as a boilerplate to start a new NodeJS project.

WHERE TO START

Prerequisites

• Ruby (optional. There is a configuration flag to turn on/off)
• SASS (optional. There is a configuration flag to turn on/off)
• Node
• NPM
• Bower
• Gulp
• PostgreSQL: This can be changed in the config/env/{{name}}.json. NOTE: Please change the database name and user account before running the application. Alternatively you can disable database acces entirely by using the flag noDatabase to be true.

The reason we have Ruby and SASS in the list is that this project uses gulp-ruby-sass to compile sass files. Hence, ruby and sass should be available on the command line. You can switch off this option to use plain CSS files by using the flag noSass to be true.

How To Fire Up

If you haven't got ruby and sass availble in your console, please follow the steps to install them SASS Installation.

I assume you have cloned the code to your local PC. Please run the following commands to get the project up and running.

1. npm install
2. npm install bower -g
3. bower install
4. NODE_ENV=development DEBUG=express,ProjectJS gulp run

Open browser and go to http://localhost:3000.

Commands

Depending on which OS your computer is running on, you might set NODE_ENV differently. For example, it would be NODE_ENV=development gulp run on MacOS or Linux. On Windows machines, it would be set NODE_ENV=development && gulp run. Also this project is set up to use NPM Debug. If you want to have more logging messages, you can run DEBUG=express,ProjectJS gulp run on Linux or set DEBUG=express,ProjectJS && gulp run on Windows.

• npm install: This installs all the node dependencies. This should be run before the other commands.

• bower install: This installs bower components.

• gulp run: This starts up the server for NODE_ENV=development. You can then open http://localhost:3000. Optionally, you can run with '--debug' argument to turn on debugging with Node Debugger. Alternatively, you can use '--inspect' to run with Node Inspector

• gulp test: This is supposed to run all the test cases.

  • gulp testServer: continuous testing for server. If --debug added at command line, Istanbul will be ignored and you can debug with real source code not the instrumented one. Also it increases the timeout period for mocha test cases so that you can debug your code. However, this flag doesn't put you into debug mode, you need to run node debug $(which gulp) testServer --debug instead.
  • gulp testServerOnce: one off testing for server. The same explanation for --debug flag.
  • gulp testOthers: continuous testing for configuration. The same explanation for --debug flag.
  • gulp testOthersOnce: one off testing for configuration. The same explanation for --debug flag.
  • gulp testProtractor: runs only the protractor test cases
  • gulp testE2e: boots up the server and then runs protractor test cases

• gulp db: This does all the database migration jobs. All available options are below:

  • gulp db --make <name_of_the_migration>
  • gulp db --migrate
  • gulp db --rollback
  • gulp db --version
  • gulp db --seed <name_of_the_seed_file>

• node bin/www: This starts up the server like gulp run but doesn't prepare css, javascript and layout files.

• npm run test-server: Just another way of running test.

Coding Conventions

For this project, I'm following general rules/conventions such as __variable__ is a hidden/private variable. The same is applied for naming file. For example, config/__bootstrapServer__.js and config/__bootstrapProject__.js are not to be used independently but will be called in config/bootstrap.js.

Configurations

All the configuration files are saved under config/env. Depending on which environment you are passing from the command line, it picks up the file with the matching name such as test.json or development.json

Loading order

The following files are used in the same order of setting up the environment.

1. __{env}__.json
2. {env}.json
3. default.json

In other words, for any given key, it first checks in the __{env}__.json file and then the others. The reason of using __{env}__.json file is that developers can put their account username and password there without worrying them being checked into git and shared with others.

Just have a quick look into .gitignore files, you can find this line config/env/__*__.json.

Special Cases

• rootLogFolder is prefixed with Project.ROOT_FOLDER
• logFolder is rootLogFolder + '/' + env
• appLogFolder is logFolder + 'app'. This is used to store application logged messages.
• accessLogFolder is logFolder + 'access'. It mimics the Apache http access log.
• gulp: any properties under this is prefixed with Project.ROOT_FOLDER if its name ends with 'Folder'

Server Port

This can be set in your config\env\your_environment.json

{
	"database": {
	},
	"port": 3000
}

This number can also be changed by using environment variable PORT. For example, you can run PORT=12345 gulp run to start your server with port number 12345.

SYSTEM ARCHITECTURE

Model View Controller

This project follows the MVC architecture with the key directories listed below:

• frontend/views: Where views are defined. The current templating engine being used is Handlebars.
• server/models: the model part
• server/routes: the controller part

The Core

Before going further, let's have a deeper understanding of the core of the project. config/bootstrap.js is the heart of this project. There are a number of ways to require this module. It gives you an instance of Project setting object. The Project setting object has access to all configurations, Models and Services. Given var ProjectJS= require('config/bootstrap.js'), we can have:

• ProjectJS() This way is used when we just want to get an instance Project.
• ProjectJS(require('express')()) This way is used to get an instance of Project and to apply all the setup for server such as where views and routes files are placed, which view engine is used and how logging for http access is done.
• ProjectJS(require('express')(), {project: AnotherProjectInstance }) This is used when we don't want to create another Project instance but rather reusing the AnotherProjectInstance and to apply server settings to the instance require('express')() above.

The Model in MVC

After getting an instance of Project setting object, you can access your model by Project.Models.YourModelName. The model name is the name of the corresponding file under server/models folder. If you put your models under sub-folders of the server folder, you can access them the same way as Project.Models.YourModelName. Hence, duplicate models with the same file names will be overriden. BookShelf framework is being used for this model layer. In short, BookShelf is a Javascript ORM for Nodejs and is built on top of Knex which handles the connnection pools to database. Knex is built for Postgres, MySQL, MariaDB, SQLite3, and Oracle. Hence, you can switch to any of these DBMS as you wish. The following settings need to be changed accordingly.

	"database": {
		"client"	: "postgresql",
		"name"		: "database_name",
		"host"		: "localhost",
		"username"	: "postgres",
		"password"	: "123456"

	}

However, if you choose to develop an app without database access, you can use noDatabase flag to switch off this feature. As a result of the flag, Project.Models will be an empty map object.

	"noDatabase": true

You can have access to the underlying knex instance by Project.Models.__knex__. To create a new model, simply create a file with the model name and place it under server/models folder. An example of the initial file content is as follows:

module.exports = function(Project, bookshelf){
	return bookshelf.Model.extend({
		tableName: 'table_name'
	});
}

BookShelf gives plenty of examples on how to extend model definition such as one-to-one, one-to-many relationships.

The View in MVC

There is a known delimiter conflict between Handlebars and AngularJS. Please change your Angular app's delimiter symbols accordingly. An example has been done in frontend/js/main.js file.

Views files can be found under frontend/views folder. Handlebars is the template engine employed for this project. You can change this by changing the code in config/__bootstrapServer__.js. Look for the line serverSettings.engine('html', require('hbs').__express).

When you start the server, the express server is configured to look for the views and templates from .tmp/frontend/views instead of the frontend/views folder. The reason is that some pre-processing is required for javascript and css files to be injected before hand. Hence, if you only run node bin/www, this pre-processing should have taken place earlier. In order to do that, you can run gulp with a proper task to get this done. More on this will be explained later all.

Some gulp tasks placeholders are used in the frontend/views/layout.html and will be replaced as part of the processing.

	<!-- build:css /static/stylesheets/all.css -->
		<!-- bower:css -->
		<!-- endbower -->

		<!-- app_bower:css -->
		<!-- endbower -->

	<!-- endbuild -->

	<!-- build:js /static/js/all.js -->
		<!-- bower:js -->
		<!-- endbower -->

		<!-- app_bower:js -->
		<!-- endbower -->
	<!-- endbuild -->

The section bower:css is used to inject css files of the bower components defined in bower.json. The section app_bower:css is used to inject project specific css files which are defined in config/app_bower.json. The same is applied for bower:js and app_bower:js for javascript files. The section build:css and build:js are the last piece of the masterpiece. They are used to look up for the files vendor.css, project.css, vendor.js and project.js and put references to them into the layout file(s).

Please be advised the order of javascript for both vendor and project can be defined in config/app_bower.json. An example of the file is below:

{
	"name": "gokien",
  	"main": [
		".tmp/frontend/js/angular/modules/*.js",
		".tmp/frontend/js/angular/services/*.js",
		".tmp/frontend/js/angular/directives/*.js",
		".tmp/frontend/js/angular/controllers/*.js",
		".tmp/frontend/js/**/*.js",
		".tmp/frontend/stylesheets/**/*.css"
  	]
}

All layout.html files which are under frontend/views folder and its sub folders are watched and when changes take place against those files, the whole compilation process for front end will take place.

The Controller in MVC

Controllers can be found under server/routes folder. A controller file is a node module with Project setting object passed in as a parameter and is supposed to return a dictionary with 2 keys router and base. It is easier to look into examples.

module.exports = function(Project){
	var router = require('express').Router();
	router.get('/path', function(req, res){
		res.render('aView', {key: '123'});
	});	
	return {
		base: '/parentPath',
		router: router
	}
};

With the above code, a GET /parentPath/path request will render the view aView.html under frontend/views folder. You can access the services and models with Project.Services and Project.Models respectively. More on how to define router can be found on Express API.

Services

You can define your services under server/services and then access it with Project.Services.YourServiceName. The name of the service will be the corresponding file's name. Following code is to show how a service can be defined.

module.exports = function(Project){
	var Topic = Project.Models.Topic;
	return {
		customMethod: function(fields, cb) {
			Topic.save(fields).
				then(function(topic){
					cb && cb(null, topic);
				}).
				catch(function(error){
					cb && cb(error, null);
				});
		}
	}
}

Data Access Object (DA0)

Since the Models are backed by Knex, they act as DAO objects in this framework. According to Knex, you can add your own methods to enrich the Models. Therefore, there is no folder so called dao under server folder. If you do have a need to have dedicated DAO objects, what you can do is to add them under helpers folder. However, this is only a suggestion, it's still very much up to your design and you have to make this decision.

Best Practices

Following are only suggestions how we can structure projects to promote reusability and code quality.

1. Don't use your Model directly in the controller

Yet for simple code, it's not always necessary to use a service. However, you typically happen to have code that requires access to a few different models before rendering the result to the user. Hence, it makes sense to keep all the logic in your services and reuse them accross the application.

2. Don't litter the routes

After your application reaches a certain size, your routes might end up with more than hundreds lines of code to define routes with this implementation function(req, res){}. What you can actually do is to create sub folders under server/routes to keep the logic. For example, we can have a sub folder server/routes/user to have all the javascript code related to user module and then require it.

var func = require('./users/index.js');

router.get('/path1', func.path1);
router.get('/path2', func.path2);
router.get('/path3', func.path3);
router.get('/path4', func.path4);
router.get('/path4', func.path5);
router.get('/path6', func.path6);

That is even easier if you define your methods as helper functions which will be addressed next.

Helpers

Helpers can be accessed by calling Project.Helpers.{{subFolder}}.{{yourHelperMethod}}. There are different ways of writing helpers. As an example, we will use helpers to define our controller code.

// File: helpers/controllers/index.js
module.exports = function(Project){
	return {
		homepage: function(req, res, next) {
		  res.render('index', { title: 'Express' });
		}
	}
}

and then use it to define the homepage route.

module.exports = function(Project){
	
	/* GET home page. */
	router.get('/', Project.Helpers.controllers.homepage);

	return {
		router 	: router,
		base	: '/'
	};
};

Notice the way we call our homepage method which is Project.Helpers.controllers.homepage not Project.Helpers.controllers.index.homepage. Hence, the filename doesn't matter here. It will only copy all the methods from all the files under server/helpers/controllers folder to the object Project.Helpers.controllers. Since you got a hold of the route function, you can do unit testing for the method.

Error Handling

You can utilise the concept of Helpers to write your custom code for error handling. An example has been given under helpers/errors/index.js

module.exports = function(Project) {
	return {
		handle: function(res, error) {
			if (error.constructor.name === 'TypeError') {
				console.log(error);
			}
			
			res.send('error:' + error);
		}
	};
}

Then you can access the function in your controllers by calling Project.Helpers.errors.handle(response, errors)

Logging

There are 2 types of logging. One is NPM Debug and the other is NPM Winston. The differences are only when they are used in. In this project, NPM Debug is used for logging all the setup including server ports, entering/exiting a function because of its simplicity whereas NPM Winston is used after the project settings have been loaded such as controllers' logging, services' logging. Also NPM Winston has been used for file logging.

CUSTOMISATION

This part explains how the gulp tasks are defined. After this, you can have a better understanding of the Project setting object and then be able to create more interesting code with your application.

Behind The Scene

When you run gulp run, it brings up the server. For this to happen, we actually have 2 instances of Project setting object. One is used by gulp process. The other one is used by our application which is injected for all Models, Services and Routes as explained earlier. Hence, when you shut down the process. You will see 2 instances of database connections powered by Knex are closed at the end. It's also arguable why we need database connections for gulp tasks. It's actually up to your creativity. You can disable this behaviour.

Still why do we need to have an instance of Project setting object for gulp? The answer is that a single place of loading configuration is always a good ideas. With the initialisation of the Project setting objects, we can use the same settings for both gulp tasks and the application code.

Gulp

Structure

There are 2 levels defining gulp tasks.

1. config/gulp/{{env}}.js
2. config/gulp/default.js

Anything defined in the config/gulp/{{env}}.js will override the one in config/gulp/default.js. It's not advisable to define tasks in config/gulp/index.js. To define a new gulp task, consider where you want to put it in the 2 files above and following is what you should do.

//For example, this is default.js

var	gulp 			= require('gulp'),
	Project			= null,
	__tasks__		= {};

module.exports = function(__Project__){
	Project = __Project__;
	return __tasks__;
}

__tasks__.stylesheet = function(){
	return sass(Project.gulp.frontEndStyleSheets).
		//... other code
		pipe(gulp.dest(Project.gulp.tmpStyleSheetFolder));
};

With this, you will have a gulp task named stylesheet. Hence, you can run gulp stylesheet in the console.

How To Find A Task

To have a good controll on gulp tasks, all the definitions of them are kept in config/gulp/index.js and the task itself only has javascript documtation like /* @Inherit */. Let's have a look at the index.js file.

/**
 * Automatically loading all the tasks to gulp
 * ******************TASKS*********************
 *
 * **********DEFAULT***********
 * clean: This cleans out the .tmp folder
 * .....Some other tasks.......
 *
 * **********DEVELOPMENT***********
 * server: This runs/restarts the express server
 * .....Some other tasks.......
 *
 * **********PRODUCTION***********
 *
 * prepare: This does all the pre-processing such as javascript, stylesheets
 *			but it doesn't run the server
 * .....Some other tasks.......
 *
 */

As you can see, the tasks are grouped to different environments. Their implementation can be found in the corresponding file such as config/gulp/development.js or config/gulp/production.js. As a reminder, it's good to follow the same convention so that the code can be more maintainable.

TESTING

Currently MochaJS is being used to do testing. All the server testing can be found under test/test-sever folder. The difference between test-server and test-other is that the Project setting object is created before hand for test-server which can be found in test/test-server/bootstrap.js. Doing that helps all the test cases share the same database connection and the NODE_ENV is set to test for them. For test-other, you can get the Project setting object at will and shut it down when you're finished. For unit testing such as controllers, if you follow the structure describe in Helpers section, you can gain access to the controller function such as homepage by calling Project.Helpers.controllers.homepage which returns a function and you can mock the request and response parameters to pass in.

After running test-server and test-other, coverage reports can be found under coverage folder. If you add --debug at run time, it will not put you into the debug mode. The flag --debug is used to stop istanbul from instrumenting the source code, thereby giving you the real source code and it is used to increase the timeout period of Mocha test.

Protractor

Protractor test cases are located under test/test-e2e folder. The naming convention is *.spec.js. The configuration file can be found at test/test-e2e/protractor.conf.js file. You can use the given commands to run the test cases. Alternatively, you can follow How To Install Protractor. After you run webdriver-manager start, you can use the command below to run the test cases:

./node_modules/protractor/bin/protractor  --specs test/test-e2e/**/*.spec.js test/test-e2e/protractor.conf.js

PRODUCTION

It's desirable to not use gulp to start up the server even though it's possible. If gulp is used, it means there will be 2 processes running at the same time. One is the gulp process and the other is your web application. In order to bring up the server in production, there are 3 steps.

1. Install all prerequisite libraries

• npm install
• bower install

2. Database Migration Make sure that the database schema in production is updated by running NODE_ENV=production gulp db --migrate.
3. Pre-processing static content Javascript, stylesheet and view layout files can be processed and placed under .tmp folder correctly by running NODE_ENV=production gulp prepare.
4. Run the server You can bring up the server by running NODE_ENV=production node bin/www from the root of your project's folder.

DEBUG

Node Debugger

When you turn on debugging, i.e. gulp run --debug; the system will be in debug mode. You can find more information on NodeJS Debug. Here I'm just listing a few interesting points if you are new to debugging in NodeJS.

1. Default Breakpoint Even when you haven't put any debugger; statement in your code, NodeJS will enter the debug mode and put it to the very first line in your code base. In this case, it would be the the first line in www/bin file.
2. Debugger statements You can put debugger; statement in your code to set the breakpoint.
3. Repl mode After it hits the breakpoints, you can type repl. After that, you can type variables' name to query information and their values
4. Reloading If you run gulp run --debug, every time you change a file which is monitored, gulp reloads the build. Therefore, your code will stop at the Default Breakpoint which is mentioned in #1
5. Exitting When you hit Ctrl-C twice to quit the NodeJS Debugger, you have to press Ctrl-C to send the terminate signal to quit the server too.

Node Inspector

You can run gulp run --inspect to run debugging with Node Inspector. Node Inspector also puts you at the first default breakpoint as Node Debugger but it launches Chrome Debug Tool which allows you to visually select breakpoints and to inspect variables.

UncaughtException: spawn

If you happen to encounter this problem when runnin gulp run --debug or gulp run --inspect, the chances are that you have special settings for .bash_profile or the way $PATH variable gets set up. For example, in my case, it runs perfectly ok on MacOS but when running on Windows with Cygwin, it was giving the error. Particularly, I have *.bat file to set up local environment before running Cygwin and hence node and node-debug couldn't be found on $PATH.