Automated Testing

Building robust code is vitally important which means it should be frequently tested against the agreed spec. You will have already been shown the principles of Test-Driven Development (TDD) in the module 205CDE but in this module you will be taking your skills to an entirely new level and building the process into everything you do.

In this chapter you will be introduced to the principles behind test-driven development (TDD) and shown how to apply unit testing to automate the testing of NodeJS modules.

Unit testing
Code coverage
Testing async code

There are two main types of automated tests.

Unit Testing
- Testing code logic directly (functions)
- Sometimes called ‘white-box’ testing
Acceptance Testing
- Testing the application as a ‘user’
- Sometimes called ‘black-box’ testing

We will be focussing on Unit Testing in this module.

1 Unit Testing

Involves writing code to test your code

This code can be quickly run whenever you add a new feature

Like a robot to do all the boring stuff

Benefits:

Run same tests repeatedly/accurately
Run a large 'suite' of tests
No human time wasted
Quick one-click testing of all the code
Prevents human error creeping in
Ensure bugs have not crept into existing code
Check that there are no bugs in new code
When all the tests pass, your software is complete!

Unit testing is a vital skill if you are planning a career in software development. It is also a key part of your module assessment. In this exercise you will be using a framework called Jasmine. In this exercise you will learn how to run tests.

We will need to install some packages to allow us to run the tests. These are all developer dependencies and so need to be added to the devDependency object in package.json. This was covered in the previous chapter.

$ npm install --save-dev jasmine jasmine-console-reporter

Now the testing packages are installed we can execute the unit tests by running the /spec/runTests script. Open this file and study it.

First we import the jasmine package and this is used to call the Jasmine() constructor function to create a new object.
Next we call the loadConfigFile() method which loads the testing config from the jasmine.json file. This contains a spec_file array listing the test suites to run.
Now we create a reporter which is responsible for displaying the test results. Different reporters output the results in different ways, our chosen one renders this in colour to the terminal. The addReporter() method loads this data into Jasmine.
Finally we call the execute() method to run the tests.

Run the test suite using.

$ node spec/runTests

Watch the results output to the terminal window.

1) Shopping List
   ✔ should add a new item
   ✔ should increment the qty if item exists
   ✔ should return an array containing all items in order of entry
   ✖ should be able to return a shopping item (1 failure)
   ⚠ should throw an error if item not in list
   ⚠ should delete first item
   ⚠ should delete last item
   ⚠ should throw error if item not in list

The start of the output shows that there are 8 tests.

The first 3 tests pass.
The next test fails.
The remaining 4 tests are pending meaning they have been disabled.

Following this you can see details of any failed tests.

Failures:
1) Shopping List should be able to return a shopping item
  Message:
    TypeError: Cannot read property 'title' of undefined
  Stack:
    TypeError: Cannot read property 'title' of undefined
        at Object.it (/.../shopping/spec/shopping-spec.js:41:31)

Reading this you should get an idea of the problem. The test is trying to retrieve a shopping item however there is a problem on line 41, character 31 in the test suite which is trying to retrieve a 'title' property but this has failed. There is more information about the pending tests but this can be ignored.

Now lets take a moment to understand the test suite we have run. Open spec/shopping-spec.js.

Carefully read this test suite file:

notice that we import the module we are testing, specifying the relative path
the test suite is defined using the describe() function, it takes two parameters, the description and an anonymous function containing the specs
each spec is defined in either an it() or xit() function. By placing an x in the function name the spec is set as pending which means it won't run.
each spec contains tests, called expectations that are used to check the state of the code.
the beforeEach() function is executed before each spec is run (once per spec) and is used to configure a known environment
the afterEach() function runs after each spec finishes and is used to tidy up before the next spec is run.

1.1 Test Your Knowledge

Implement the getItem() function in shopping.js so that the new test passes (you will need to use the get() method that forms part of the Map prototype). As soon as this test passed you can move on.
Uncomment the next spec (should throw an error if item not in list) by removing the x from xit and run the test suite. You should see this test fails.
modify the getItem() function in shopping.js. You will need to correctly implement exception handling which was covered in the previous lab. As soon as the test passes you know you have implemented the function properly.
Uncomment the next spec (should delete first item) and implement the removeItem() function in shopping.js. You will need to use the Map.delete() method. Move on once the test passes.
Uncomment the next spec (should delete last item) and make sure the tests pass.
Finally Uncomment the last spec (should throw error if item not in list) and implement error handling so that the test passes. You have now implemented the required functionality and you know it has been done properly because the tests pass.

1.2 Test-Driven Development

start by defining the goal

unit tests are both specification and documentation

we define our functionality through the tests

we write our tests before starting to 'code'

forces us to think about what we are writing

TDD Workflow

Write the tests first (they will fail)
Write enough code to pass the tests
- simplest solution that could possibly work
Commit the working code
Refactor the code to improve
Comments/documentation
Commit (again)

Benefits of TDD

Working code
Enforcing single responsibility
Conscious development
Improved productivity

1.2.1 Fixing Bugs

Write a test to replicate the bug

Test should of course fail every time it runs

Try to fix the bug

Bug is fixed when the test(s) pass

Test name should match bug name/code

1.3 Regression Testing

Are we introducing bugs in previously working code?

All existing code should be supported by exhaustive tests

Even if a test is passed we should always carry it out

All tests run every time code changes

1.4 Test Your Knowledge

TODO exercise in identifying bugs, isolating then in tests and passing the tests. Can we include a fix that breaks earlier code (regression testing)

2 Code Coverage

If we are going to rely on our automated tests to guarantee our code runs as expected we need to check that these tests are comprehensive. There are two aspects we need to check:

Can our script handle both good and bad data.
- bad data might include missing parameters as well as invalid data types or values (trying to access an array index that doesn't exist for example)
Do the automated tests test every line of our code including all conditional branches.
- this is called code coverage and there are automated tools to help us with this.

How much of the code is being tested?

Are all code branches being tested?

How many times are each line tested?

Helps identify any blind-spots

also consider range of test data...

2.1 Running the Code Coverage Test

Start by installing the node Istanbul module has already been installed so we can run our coverage tests. Note that any command-line tools installed by a package can be found in the node_modules/.bin directory.

$ ./node_modules/.bin/istanbul cover -x **spec/** -x **index.js** -x **debug.js** jasmine-node spec

There are a number of important parameters and flags so lets analyse these:

the command we use is called istanbul which was installed using the command on the previous line.
to generate a complete coverage report we need to pass the cover parameter.
There are some files we don't want to check for code coverage:
- the spec/ directory contains our tests and we won't be writing tests for our tests!
- the index.js file is used to run our app interactively, its not part of the application logic model.
- the debug.js script serves a similar purpose.
Finally we specify the command to run the unit tests.

The parameters and flags won't change between runs so we should create an alias to make it easier to trigger the coverage suite. These are stored in the package.json file under the scripts key. If you open this you will see that there is an alias called coverage, so to run our coverage suite we call.

$ npm run coverage
  =============================== Coverage summary =============================
  Statements   : 65.85% ( 27/41 )
  Branches     : 37.5% ( 6/16 )
  Functions    : 100% ( 0/0 )
  Lines        : 65.85% ( 27/41 )
  ==============================================================================

Let's examine the four different criteria.

Statement coverage
- Has each statement in the program been executed?
Branch coverage
- Has each branch of each control structure been executed?
Function coverage
- Has each function in the program been called?
Condition coverage (or predicate coverage)
- Has each Boolean sub-expression evaluated both to true and false?

2.2 Analysing the Code Coverage Report

When the coverage test has finished it generates a report in a coverage/ directory.

.
├── coverage.json
├── lcov-report
│   ├── base.css
│   ├── index.html  < this is the file you need to open...
│   ├── modules
│   ├── prettify.css
│   ├── prettify.js
│   ├── shopping
│   ├── sort-arrow-sprite.png
│   └── sorter.js
└── lcov.info

Open the index.html file (as shown above). If you are using a cloud-based IDE you may need to install a web server (see the notes at the end of this chapter).
As soon as the webpage opens you will immediately spot we have very poor coverage with only 27 out of 41 lines of code being tested!
Click on the modules/ link and drill down to the shopping.js file to see details which appear down the left margin.
- any line of code that is being tested appears in green (the number represents the number of times it was called in the test suite)
- any line in red has never been called by the test suite and so has not been tested.

2.3 Test Your Knowledge

It is immediately clear from the coverage report that there is a large chunk of code that is not being tested! If we are not testing code we have no confidence that it is working. The detailed report flags up the decrement() function.

Write unit tests to test that the decrement() function works correctly.
- Make sure you test all code paths.
Periodically re-run the coverage tool and refresh the report page.
You are aiming for 100% code coverage.

3 Asynchronous Testing

In the previous section all module calls resolved immediately however it is often the case that a call may take a callback parameter to be executed in a different thread. Obviously we need to be able to test this type of code.

In this task we will be testing and debugging a module that uses third-party APIs to calculate taxi fares between named places. Taxi fares are calculated according to an agreed formula. Start by understanding the project files.

.
├── index.js
├── modules
│   └── taxi.js
├── package.json
└── spec
    ├── medium-spec.js
    ├── routedata
    │   ├── cov_uni_cat.json
    │   └── cov_war_uni.json
    └── short-spec.js

The key differences between this and the previous example are the multiple specs and the json documents in the routedata/ directory. Examine the files to understand the purpose of each file and how they interact.

install the module dependencies and read through the documentation for rewire and sync-request to understand how the module works.
run the index.js script and use it to interact with the module, ensure you understand fully how it works.
- compare these tests with the synchronous version, there is a done() parameter that needs to be called to indicate the callback has completed.
The tests have been split into two suites. A suite contains a group of related specs. Each spec contains one or more expectations which will either pass or fail. Launch the test-runner in a new terminal window and run the two suites.
notice that both the suite and spec descriptions appear in the test results. The execution time is displayed against each spec. Notice that at least one of the specs will fail. Why can't we be certain how many tests will pass or fail?
- the module calculations make use of live data from an external API. Sometimes the API will return exactly the same data as before but often the return data will be different. This is not helpful when testing a module!
- the getRouteData property contains the API call so the test will substitute this for a function that always returns the same data, known as a Mock.
- notice we used the rewire package to load the taxi module, this allows us to replace any private property
- open the medium-spec.js file and uncomment the taxi.__set__() method which substitutes a different function, this loads data from a file rather than make the API call.
- check the results of the test suite

3.1 Test Your Knowledge

create a third test suite to simulate a longer journey of more than 150 miles
- use the index.js script to generate a suitable URL for the API and print this to the console
- paste this into a browser tab to capture the standard response
- paste this into a json parser to check it is valid and tidy up the formatting
- paste this into a new json file in the routedata/ directory.
- use the taxi fare calculator to calculate the correct fare.
- add suitable specs to the new test suite
there is an error in the fare values returned, fix the error so that the tests pass.

TEACHING-MATERIALS/12 Automated Testing.md