04 Writing Robust Code.md

Writing Robust Code

By now you have learned the core features of JavaScript as used by NodeJS and, in this chapter you will learn about some important features of NodeJS and some useful tools to make you a more productive programmer.

There is a slide deck https://goo.gl/XHBbMx to accompay this worksheet. Make sure you are familiar with its contents before proceeding.

In this chapter you will be covering:

1. Project metadata *
2. CommonJS modules *
3. Linters *
4. Debugging
5. Documentation *
6. Git flow
7. Creating packages

Any section above marked with an asterix * should be considered essential knowledge.

1 Project Metadata

Every project has a range of metadata associate with it. This includes basic information such as its name and author but also should include the package dependencies (the packages that should be installed). In NodeJS this information is stored in a special file called package.json which is located in the root directory of the project.

1.1 The Node Package Manager

Rather than manually creating the package.json file, it is created and maintained using the Node Package Manager. To create it you need to run npm init which will guide you through the process of creating the file by prompting you for key information.

1.2 Installing and Tracking Packages

Up to now you have been installing individual package dependencies such as readline-sync using the npm tool. Once you have a package.json file its important that these dependencies are recorded here. This is done by adding a special flag.

Packages are split into two different categories:

• Some packages are used by the running script. An example of this would be the readline-sync package.
• Some packages are used as part of the development process. An example of this would be the jsdoc package which we will use later to generate the documentation for our code.

It is important to keep track of both types of dependency. On our development computer we would want to install them all but on a live server we would only want the modules that are needed by the running script.

1.2.1 Script Dependencies

To install a package needed by the script we use the --save flag when we install it using npm. This will add it to the dependencies object inside package.json.

npm install --save readline-sync

If we examine the package.json file we will see that it now contains a reference to readline-sync.

{
  "name": "shopping",
  "version": "1.0.0",
  // additional data not shown
  "dependencies": {
    "readline-sync": "^1.2.22"
  }
}

To install all the packages listed in the dependencies object we can run.

npm install --production

If you have an environment variable NODE_ENV=production you don't need to use the --production flag and can install using.

npm install

1.2.2 Developer Dependencies

Some packages are not used directly by the scripts but are used as part of the development process. An example of this is the jsdoc package which is used to extract and build documentation that is written in the JSDoc syntax. The installation process uses a different flag.

npm install --save-dev jsdoc

This makes no difference to how the package is installed but adds the reference to a different object in package.json as shown below.

{
  "name": "shopping",
  "version": "1.0.0",
  // additional data not shown
  "devDependencies": {
    "jsdoc": "^3.4.0"
  }
}

Normally, if you need the devDependencies you will also want the dependencies as well. To install all the packages we can run.

npm install

1.3 Global and Local Modules

Modules can either be installed locally or globally so what is the difference?

####1.3.1 Locally Installed Modules

If a module is installed locally it is installed using standard permissions in the node_modules/ directory within the current directory. If there is a binary executable, it can be found in the .bin/ directory. This is the recommended way to install packages and how we have been doing this until now.

You will see this approach used for the first time when we install and run our linter. Let's install the eslint package locally and run its binary from the terminal.

npm install eslint --save-dev
node_modules/.bin/eslint

Remember that the linter is used as a developer tool and not by the script itself so it needs to be added to the devDependencies object in package.json, this is done by specifying the --save-dev flag.

Also notice that the eslint binary is inside the .bin/ directory which is inside the node_modules/ directory. To run it we need to specify the relative path.

1.3.2 Globally Installed Modules

There are some situations where you need to install a module globally, this is normally only done for packages with a binary executable and needs to be done as root. The main benefit is that, once installed, a module can be called from any directory without needing to know the file path. The disadvantage is that these can't be automatically tracked and installed as part of the script dependencies (from package.json). It is not recommended this approach is used unless absolutely necessary.

1.4 Script Aliases

The last object to understand in package.json is the scripts. This is where we can define aliases for our scripts.

"scripts": {
  "app": "node index.js",
  "lint": "node_modules/.bin/eslint modules/"
}

In the example above, the first alias, app is already defined and will execute the index.js script. The second one has been manually added and runs eslint on all the files in the modules/ directory. To execute these you would type.

npm run app
npm run lint

As you can see, by using a script alias you can simplify how you call scripts, especially if there are a lot of flags and options.

1.5 Test Your Knowledge

1. modify the package.json file and change the version to 1.0.1
2. install the request module, making sure it is referenced in the dependencies object in your package.json file.
3. install the jsdoc module, making sure it is referenced in the devDependencies object and add an alias called doc set to node_modules/.bin/jsdoc modules/.
4. add a new script alias called debug and set its value to node debug index.js, you will be using this later in the worksheet.

2 CommonJS Modules

Up to this point you have written your scripts in single files. Whilst this keeps everything in one place there are a number of problems.

• Creates a huge file!
• Makes it difficult to find stuff.
• Makes it impossible to write unit tests (more on this in the next chapter).
• There are problems with public variables and objects.

A much better way is to keep any code related to business logic in separate file(s) and to support this, NodeJS supports the CommonJS module standard which is part of the latest ECMA6 standards. You should aim to create one module for each chunk of functionality.

Here is a simple module:

// shopping.js
var data = new Map()

exports.count = () => {
	return data.size
}

And here is how we import this module into our code:

const list = require('./shopping')

const size = list.count()
console.log(`there are ${size} items`)

Open the shopping/ directory and examine the files and directory structure.

.
├── debug.js
├── index.js
├── modules
│   └── shopping.js
├── node_modules
└── package.json

Notice that there is a modules/ directory. In NodeJS we capture functionality in modules and these are what we will run the tests on. Each module should be self-contained. All the application business logic should be moved into modules.

Open the modules/shopping.js script and read it carefully.

1. Notice that the module imports its own dependencies which will not be visible to other scripts (node-persist in this case).
2. By default, all functions and variables are private and not accessible from outside the module.
3. The exports object contains the public-facing functionality

• each object stores an anonymous function which can be called by other parts of our app.
• each anonymous function is defined using the ECMA6 Arrow Function syntax
• the functions in some of the properties (count, clear, getAll) take no parameters (indicated by a pair of empty brackets in front of the => arrow)
• the other functions take a single parameter, listed before the => arrow

3. Some functionality has been completed but there are some stub functions which represent functionality we have not yet implemented.

3 Using a Linter

JavaScript is a powerful language and includes a number of powerful features but it also includes features that, whilst they may produce working code, will make your code difficult to read and debug! A linter is a program that analyses your scripts for programmatic and stylistic errors and will flag up anything that may cause problems. They define a professional subset of the language and reject any code that doesn't meet that standard.

There are a number of linters available for the JavaScript language. The original one was written by Douglas Crockford (the author of JavaScript: The Good Parts). This enforced his idea of good JavaScript and was notorously not configurable! There are now a number of popular linters available, your choice should be based on what will run in your IDE as well as your personal preferences. The most popular linters are:

1. JSLint, the original, by Douglas Crockford
2. JSHint
3. ESLint which is supported by many IDEs and code editors

EsLint is highly configurable through a hidden config file that needs to be added to the project root directory. When you submit your coursework you must demonstrate that it contains no errors or warnings when run using an identical configuration file.

3.1 The Linter Configuration

ESLint is completely configurable through a configuration file .eslintrc which should be located in the project's root directory. By default any file or directory starting with the period (.) character is hidden although most editors will show them by default. A sample configuration file has been provided in the sample code repository. Open it and take a look at the structure.

The .eslintrc file uses a JSON string to organise its settings which are in three sections.

{
	"env": {
		"es6": true,
		"jasmine": true,
		"node": true
	},
	"parserOptions": {
		"ecmaVersion": 8
	},
	"rules": {
		"no-var": 2,
    "semi": [1, "never"],
		"arrow-spacing": [1, { "before": true, "after": true }]
	}
}

Near the bottom of the file list you should see a file called .eslintrc, the initial dot (.) in the filename caused it to be hidden by default. Open the file now.

1. Notice that the file contents are using the JSON format which should be familiar to you.
2. There are three JSON objects, env, parseOptions and rules.
3. The env object describes the environment you are using. In our example we indicate that we will be using the ECMA6 syntax, are writing NodeJS code, and will be using Jasmine tests.
4. the parserOptions object defines the version of JavaScript it should use. Because we will be using the cutting edge language constructs we choose version 8.
5. The rules object defines the rules we want to apply. As you can see we are requiring tab indentation, single quotes and avoiding the use of semicolons on each line. The full list of rules can be found in the ESLint Documentation.
6. Each rule has a reporting level where 0 means disabled, 1 means warning and 2 means error. This affects how the rule violations are reported.
7. Some rules allow for additional options. If these are specified, both the reporting level and options need to be in an array.

Whilst you need to use the supplied configuration file in your assignment you should take time to understand the range of rules available and adding them to your .eslintrc configuration file. To make this process easier you can use this handy ESLint Rule Generator.

3.2 Running ESLint

Although there are plugins for many popular IDEs you should get used to running the linter from the terminal. ESLint is available as a NodeJS package which allows you to run it from the terminal. Since you may find yourself using an editor with support baked in, why would you want to do this?

1. Some editors don't have ESLint support.
2. Running ESLint in the Terminal gives a summary of the linting errors.to check if all the scripts are fixed.
3. It can be configured to fix many of the most common error such as whitespace and semicolon usage.
4. The linting can be run as part of the Continous Integration and Continous Deployment process (more on this in a later worksheet)
5. Finally, it will be used during the marking of your assignment to make sure your code is formatted correctly!

3.2.1 Test Your Knowledge

lets install, configure and run the console-based linter.

1. Start by opening a terminal window and navigating to the shopping/ directory.
2. Install the NodeJS ESLint package npm install eslint --save-dev. This installs it and adds it to your package.json file in the devDependencies section.
3. Run the executable node_modules/.bin/eslint index.js. This runs the eslint executable located in the hidden .bin/ directory in the node_modules/ directory.
4. You will see a list of all the errors and warnings found together with a summary with the total number of errors and the total number of warnings.
5. Open the index.js file and, using the linter report, fix the issues. You should re-run the linter regularly to check your progress.
6. Run the linter on the module using node_modules/.bin/eslint modules/shopping.js and note the errors it finds.
7. Now run the same command but this time pass the fix flag, node_modules/.bin/eslint --fix modules/shopping.js
8. Open the shopping.js file and notice that most of the issues have gone. The fix tool is not perfect, it may have introduced new errors so use it with caution!
9. Manually correct any errors until the linter reports 0 errors and warnings.
10. Finally run the linter against all the files in the project using node_modules/.bin/eslint . to ensure there are absolutely no linting errors in your code. Well done!

4 Debugging

As our script grow larger, so the odds of introducing a bug. So how can we locate and fix them?

One option would be to insert lots of console.log() statements into our code. This would help us to:

1. Track the program flow.
2. Track the contents of variables and objects.

A much better solution is to use a debugger. A debugger allows you to:

1. Pause execution of the code at any point.
2. Step through the code one line at a time.
3. Monitor the values of variables and objects at any point.

Debuggers offer a number of features. Breakpoints allow the code to be paused on specific lines of code. Code execution can be run a line at a time or resumed. Functions can be stepped in and out. Watchers can be used to monitor variables and objects.

There is a powerful debugger built-in to NodeJS and two alternative ways to work with it.

1. A terminal debugger controlled through the terminal window.
2. A visual debugger that allows you to interact using the mouse. There are two main choices.
   • A TCP-based debugger that uses the Chrome browser.
   • Using the integrated debugger built into modern code editors.

It is important to learn both these.

4.1 Terminal Debugger

NodeJS offers a terminal debugger which allows you to debug code directly in the terminal window. Whilst it is not the most intuitive of tools it does offer an impressive range of features. lets take a look at these in action. Start by opening the simpleDebug.js file in the shopping directory. Notice the debugger statement on line 10, this is now we insert a breakpoint. We can run the script in debug mode using.

node debug simpleDebug.js
< Debugger listening on 127.0.0.1:5858
connecting to 127.0.0.1:5858 ... ok
break in simpleDebug.js:2
  1 
> 2 'use strict'
  3 
  4 const data = new Map()
debug> 

Notice that the debugger has paused at the first line of code but has not executed it yet. You are now presented with a debug> prompt where you can enter a debugger command. These are:

Command	Abbr	Meaning
cont	c	continue execution
next	n	step next
step	s	step in
out	o	step out
quit		exit the debugger
restart		restart the script

It is also possible to attach and remove a watcher to or from a variable or object and list all watched variables.

Command	Meaning
watch('expr')	add expr to watch list
unwatch('expr')	remove expression from watch list
watchers	list all watchers and values

Breakpoints are added directly to the code

Script is run in debug mode

Can add watchers to monitor variables

Commands for stepping through code.

Let's try out some of the debugger features.

1. At the prompt lets add some variables to the watch list.
   • It would be good to see the content of the item variable which should hold the name of the item we are adding. type debug> watch('item') to add this variable to the watch list.
   • The items should get added to the Map() object. Its would be helpful to see what it contains by typing debug> watch('[...data]'), note the use of the spread operator.
2. Now we can run the code. The first breakpoint is just after we have asked for the command so we can continue running the code from where it is currently (the first line), it will stop at the breakpoint. type debug> c, we will see the command prompt and can enter the first item (lets add bread).

Notice that the two objects in the watch list are listed followed by the current line of code plus a few lines before and after.

Watchers:
  0: item = "<error>"
  1: [...data] = []

  8 do {
  9 	input = String(readline.question('enter command: ')).trim()
>10 	debugger
 11 	if (input.indexOf('add ') === 0) {
 12 		const space = input.indexOf(' ')

At the moment the code is paused on the breakpoint. We can now single-step through the code by entering debug> n. You should repeat the command, stepping over each line of code until there is a value assigned to the item variable.

We are about to execute the code that handles the item quantity so it would be a good idea to add the qty variable to the watch list, debug> watch('qty'). Continue single stepping through the code until the new item has been added to the item map. Notice that as soon as the if statement has exited, both the item and qty variables are flagged as <error>, this is because they were defined inside the first if statement and are now out of scope.

4.1.1 Test Your Knowledge

1. If you run the script outside the debugger you will notice that there are some logical errors. Use the debugger to trace the flow of the code and the values of the variables and objects to help you locate the error and fix it.
2. The script should ignore the case of the item you are entering, so Bread, bread and BREAD should all be treated as the same.
   • Use the debugger to see what currently happens when you add these three variations.
   • The script should store a single product Bread and a quantity of 3 units.
   • Use the debugger to help you implement this new feature.
3. Finally you should be able to remove a product using the command remove bread. This should remove the item from the list. Use the debugger to help you implement this feature.

4.2 Visual Debugger

Whilst the terminal debugger offers the key features it is not simple to use. A visual debugger is far more intuitive.

Many modern editors include built-in debuggers and it is worth learning how these work but even it your editor does not have this facility, NodeJS provides an out of process TCP-based debugger that connects to the Chrome web browser. We will explore this. Start by opening the shopping/ directory. The main script is index.js. Open this, notice it imports a module called shopping.js which is located in the modules/ directory.

To start the visual debugger you need to navigate to the shopping/ directory and run the main index.js script but pass two flags.

node --inspect --debug-brk index.js

1. The --inspect flag starts the debugger.
2. The --debug-brk flag pauses the debugger on the first line of the script.

Then open about:inspect in the Chrome browser. You should see some options appear in the browser window. Click on the Open dedicated DevTools for Node link and this will open a new window containing the visual debugger. Let's take a look at the key features and how these relate to the terminal debugger we have already used.

1. The screen is split into 3 panes. We don't need the left one so this can be closed using the button to the left of the simpleDebug.js tab.

   • The middle pane contains the code we are debugging. Rather than adding debugging commands into the source code we can click in the left gutter (where the line numbers are) to add blue breakpoint markers. Add one to first if statement and another to the second one.

2. The right-hand pane contains a number of collapsable sections. In this exercise the only one you need to keep open is the Scope pane. This lists the variables and objects and is grouped into different scopes.

   • The Local scope section contains the variables and object within the current script.
   • The Global scope is not needed.
   • If the script is in a block you will see another section which lists the locally scoped variables.

At the top of this right-hand pane are the script execution buttons, these replicate the debugger commands we used in the previous exercise.

Start by clicking on the Resume script execution button which will run the script up to the next breakpoint. The debugger is now greyed out because it is waiting for input in the terminal. Switch to the terminal and type add bread, this will return control to the debugger.

Now try the next button, Step over next function call. This will step through your code one line at a time. Repeat this until you get to the List.add(item) line.

Now it's time to use the next button, Step into next function call. This will load the shopping.js file and place the program execution on the first line of the add function. If we used the Step over button we would have run this function and returned control to the next line in the index.js script.

Use the Step out of current function button. This returns control to the controlling script back in index.js.

Now use the buttons to loop back to the List.add(item) line but this time use the Step over next function call button. Notice that this time control moves directly to the next line without going into the function definition.

One really useful feature is that the browser-based debugger will automatically reconnect if the dvript is restarted. Press ctrl+c to quit the script then restart with node --inspect --debug-brk index.js and see how the debugger reconnects.

5 Documentation

Whenever we write a program it is important that it is fully documented. A simple solution is to add comments to the code which can then be read by anyone who opens the script. A better solution would be to write up detailed human readable documentation.

A documentation generator is a tool that takes structured comments added to source files and turns them into properly-formatted documentation. We will be using the jsdoc tool which is based on the JavaDoc tool used by Java programmers. JSDoc is a markup language used to annotate JavaScript source code files. Using comments containing JSDoc, you can add documentation describing the application programming interface of your code and turn these into a documentation website.

In this exercise you will learn how to use JSDoc to create detailed professional documentation for your code.

5.1 Adding JSDoc Comments

So what needs to be documented?

1. all function signatures (names, purpose, parameter, return types)
2. any exceptions that may be thrown.
3. an explanation of the purpose of any obscure lines of code

Let's look at a simple example, taken from modules/shopping.js

/**
 * Returns details for the named item.
 * @param   {string} item - The item name to retrieve.
 * @returns {string} The name of the item
 * @throws  {InvalidItem} item not in list.
 */
exports.getItem = item => {
	if (data.get(item) === undefined) {
    // error thrown in the item is not in the Map()
		throw new Error('item not in list')
	}
	return data.get(item)
}

Notice that the JSDoc comments start with /** and end with */. Each line starts with an asterix * and the first line describes the purpose of the function. All parameters are then listed together with their preferred data type and this is followed by the return value and its expected type. Finally any errors/exceptions thrown by the function are shown.

You should also add standard JavaScript comments to explain the code itself. These begin with //. Even though these won't be extracted into the documentation they are important when trying to understand the code logic.

5.2 Generating the Documentation

JSDoc is available as a NodeJS plugin. It has already been added as a Dev Dependency in the package.json file and so it should already be installed. Lets use it to build the documentation for our modules.

ode_modules/.bin/jsdoc -d docs/jsdoc/ modules/

This will create a docs/jsdoc/ directory containing the complete documentation as a website.

.
├── fonts
├── index.html  < this is the documentation home page...
├── module-shopping.html
├── scripts
├── shopping.js.html
└── styles

Right-click on the index.html file and choose preview. This will open the home page. The module list can be found down the right-hand side of the screen. Locate the shopping link and click to see the full documentation.

1. Notice that each variable and method definition has been lifted from the source code.
2. Notice that each method has a description, this was added to the source code file using the special JSDoc markers /** */.
3. The getItem() method documentation includes a lot more detail, this is defined in block tags:

• a list of the parameters and their types
• an explanation of the return value and type
• a list of any exceptions that could be thrown

There are a lot more features that can be added to this, you should take time to read the full documentation to find out what block tags can be used.

5.1 Test Your Knowledge

1. Your challenge is to complete the documentation for the shopping.js module. You should use the getItem() documentation as a guide.
   • Make sure you add the JSDoc documentation block and also add comments to explain the code.
2. To check your results, run the documentation tool and reload the documentation web page.
3. Carefully read through the generated documentation to ensure it is both complete and makes sense.

6 Creating Packages

6.1 Local Packages

Importing modules you have created has a problem

Correct path needs to be provided

Can get messy:

const async = require('../../mod/async')

We want:

const async = require('async')

Local Module Dependencies

You have already imported modules from npmjs

You can also define and import local modules

npm install --save ./mod/async

This will copy your module into node_modules

{
  "name": "bookshop",
  "dependencies": {
    "request": "^2.67.0",
    "async": "file:local_modules/async",
  }
}

What is a Module?

Any directory that has a package.json file

Everything in the directory will be included

Files in the .gitignore or .npmignore file will not be published

Reloading a Module

There are several approaches:

Use the rm-local-modules command

Increment the version number in its package.json then npm install

Delete and reload

rm -rf node_modules/async
npm install

https://www.hacksparrow.com/how-to-write-node-js-modules.html

$ npm pack
  currency-0.0.1.tgz

6.1.2 Installing a Local Package

TODO

$ npm install --save ../currency

6.2 Publishing Packages

Logging into NPMJS

Need an account and to be logged in

npm adduser
npm login

Check account details

npm config ls

https://www.npmjs.com/~

Publishing to NPMJS

Package name must not exist in npmjs

npm publish

TODO