05 Creating an MQTT Broker.md

# Installing an MQTT Broker

Now we will install (and partially secure) a [Mosquitto MQTT broker](https://mosquitto.org/) so that data can be published to it and the client can subscribe to and receive data. We will be installing this using the `apt` command however the version installed by default is quite old and does not support the latest MQTT features. To resolve this we will add a package repository containing the more recent version. We need the version that matches the version of Debian we have installed. In the example below, the response from the first command indicates we are running `jessie` so the url when we retrieve the list points to the jessie version. If you get a different result, change the URL to match.

```
$ cat /etc/os-release
  VERSION="8 (jessie)"
$ wget http://repo.mosquitto.org/debian/mosquitto-repo.gpg.key
$ sudo apt-key add mosquitto-repo.gpg.key
  OK
$ cd /etc/apt/sources.list.d/
$ sudo wget http://repo.mosquitto.org/debian/mosquitto-jessie.list
cd ~
```

Once this is done we can update the cached package list and install the mosquitto server and the client tools.

```
sudo apt update
sudo apt search mosquitto
sudo apt install mosquitto mosquitto-clients
mosquitto_pub
```

This last command tries to run the mosquitto publish tool with no parameters. This will show the help page. We know we have the latest version if we can see a `--cafile` flag listed. To see if the server is running (on its default port of `1883`) we can run:

### 4.1 Managing the Service

```
$ netstat -at
  Active Internet connections (servers and established)
  Proto Recv-Q Send-Q Local Address           Foreign Address         State
  tcp        0      0 *:ssh                   *:*                     LISTEN
  tcp        0      0 *:1883                  *:*                     LISTEN
  tcp        0    208 elephant:ssh            local-comp:55951        ESTABLISHED
  tcp6       0      0 [::]:ssh                [::]:*                  LISTEN
  tcp6       0      0 [::]:1883               [::]:*                  LISTEN <--
```

Notice the last line, this shows the service is running on port `1883`.

Whenever you update the configuration you need to restart the service. Here are the commands to stop start and restart:

```
sudo /etc/init.d/mosquitto stop
sudo /etc/init.d/mosquitto start
sudo /etc/init.d/mosquitto restart
```

### 4.2 Publishing and Subscribing

Now we have the MQTT Broker configured we need to learn how to publish messages to and subscribe to published messages.

You should be able to use the `mosquitto-clients` tools covered in the previous lab. Remember that the broker is currently not secure and so will run on port `1883`. The hostname will be the one you assigned to the server earlier in the lab and of course you won't need to specify username, password or CA file.

The server is logging its activity to the `/var/log/mosquitto/mosquitto.log` file. Linux allows you to see the last entries in the log file using the `tail` command. To see a live, updating you can add the `-f` (follow) flag.

```
sudo tail -f /var/log/mosquitto/mosquitto.log
```

Run this command on the server and see what gets logged when you publish and subscribe. If you are getting any errors with the mosquitto tools you should check this log to find out the cause.

## 5 Securing the Broker

Although the broker does work it lacks any form of security. In this section we will be correcting this. Under linux, all system-related configuration files are in the `/etc` directory and, if you study its contents, you will find a `mosquitto` subdirectory. This is where we access all the config settings for the broker. In the rest of this section all files we mention are in this directory.

### 5.1 User Accounts

The first step is to add some usernames and passwords. The mosquitto server comes with the `mosquitto_passwd` command to allow us to add multiple user accounts. These are stored in the `passwd` text file. The file is only read-only for non-root users. Use the following command to add a new user, substituting the chosen username and password where shown.

```
sudo touch /etc/mosquitto/passwd
sudo mosquitto_passwd -b /etc/mosquitto/passwd username password
```

You will also need to modify your config file to prevent anonymous use and tell it where to find the password file. Edit the `mosquitto.conf` file and add the following two lines:

```
allow_anonymous false
password_file /etc/mosquitto/passwd
```

Once you have added users you need to restart the broker for the changes to take effect.

After restarting the server you will need to supply a valid username and password when you connect. Try publishing and subscribing to check this is true.

### 5.2 Access Control List

You have probably noticed that even with multiple accounts, all of these can publish and subscribe to any topic. Since we want to introduce some privacy we need to create an _access control list_ which defines which topics can be published to (written) or subscribed to (read). You should create a new file in the `/etc/mosquitto/` directory called `acl`. You will need root permissions to do this.

```
sudo nano /etc/mosquitto/acl
```

You should then define the access permissions for the different accounts. In the example below we have two user accounts called `house` and `garden`. You can assign one of three permissions:

1. `read` means the user can _subscribe_ to the topic.
2. `write` means the user can _publish_ to the topic.
3. `readwrite` means the user can both publish and subscribe to the topic.

The `#` character signifies a wildcard (any combination of topic segments).

```
user house
topic readwrite house/#

user garden
topic readwrite garden/#

user powerx
topic readwrite powerx/#

user 4009user
topic readwrite 4009user/#
topic readwrite owntracks/#
```

Once you have added the appropriate access permissions we need to let mosquitto know the access control list exists. Edit the `/etc/mosquitto/mosquitto.conf` file and add the following line:

```
acl_file /etc/mosquitto/acl
```

Restart the broker for the changes to take effect. Now try to publish and subscribe both to topics in and not in the access control list. Notice that if you try to publish or subscribe off-topic you dont get an error but the action will fail.

### 5.3 Implementing SSL

The final step is to implement SSL to ensure the data is encrypted as it passes between the broker and clients. Let's Encrypt is a new service offering free SSL certificates through an automated API.

```
wget https://github.com/owntracks/tools/raw/master/TLS/generate-CA.sh
```

--------

### 5.3 Implementing SSL (self signed)

The final step is to implement SSL to ensure data is encrypted as it passes between the broker and client.

#### 5.3.1 Installing OpenSSL

**Warning: depending on the speed of the Raspberry Pi, thie process of building OpenSSL can take several hours to complete. It is recommended that you run the installation step overnight.**

OpenSSL is a robust, commercial-grade, and full-featured toolkit for the Transport Layer Security (TLS) and Secure Sockets Layer (SSL) protocols. It is also a general-purpose cryptography library. It is needed to generate the required certificates. We need to ensure we are always using the latest version and so, rather than install from a repository we will build from source.

```
git clone git://git.openssl.org/openssl.git
cd openssl
./config
make
make test
sudo make install
```

#### 5.3.2 Generating the Certificates

On most Linux distros, the certificates are typically stored in the `/etc/ssl/` directory so we need to create this and navigate to it. Note you may need root privileges to create the directory if it does not already exist.

```
mkdir /etc/ssl/
cd /etc/ssl/
```

Now we need to create a 2048-bit key called `mosq-ca.key` and use this to create an X509 certificate. During the second step you will be asked a series of questions, it doesn't matter what you enter.

```
sudo openssl genrsa -out mosq-ca.key 2048
sudo openssl req -new -x509 -days 365 -key mosq-ca.key -out mosq-ca.crt
```

Next we create the private key and use this to create the Certificate Signing Request (CSR). Note you will need to enter the same information a second time.

```
sudo openssl genrsa -out mosq-serv.key 2048
sudo openssl req -new -key mosq-serv.key -out mosq-serv.csr
```

Normally this would be sent to the Certification authority that, after verifying the author identity, returns a certificate but in this example we will use a self-signed certificate and verify the certificate.

```
sudo openssl x509 -req -in mosq-serv.csr -CA mosq-ca.crt -CAkey mosq-ca.key -CAcreateserial -out mosq-serv.crt -days 365 -sha256
sudo openssl x509 -in mosq-serv.crt -noout -text
```

#### 5.3.3 Configuring the Broker

Now the certificates are ready we have to configure MQTT Mosquitto Server so that it can use these certificates by editing the  `mosquitto.conf` file and adding the following to the end.

```
listener 8883
cafile /etc/ssl/mosq-ca.crt
certfile /etc/ssl/mosq-serv.crt
keyfile /etc/ssl/mosq-serv.key
```

Restarting the broker will put the changes into effect, you should see that this is now running over port 8883. If you want to connect to the broker now you will need to download the `mosq-ca.crt` certificate and use this when making connections to the broker.

One way to download the certificate is to log out of the server and use the rsync tool to copy the file to your computer using the ssh connection. For example the following command copies the file to the `~/Documents` directory on the local workstation:

```
rsync -avzhe ssh root@elephant:/etc/ssl/mosq-ca.crt ~/Documents/
```

### 5.4 Configuring Websockets

The server is now configured as an MQTT broker however if you plan to use a browser to connect to it you will need to enable _MQTT over Websocket__. You will need to enable this in your `mosquitto.conf` configuration file by adding the following:

```
listener 9001
protocol websockets
```

Rather than restarting the broker this time we will tell it to use our updated config file. Run the command:

```
$ mosquitto -c /etc/mosquitto/mosquitto.conf
  opening websockets listen socket on port 9001
  opening ipv4 listen socket on port 8883
  opening ipv6 listen socket on port 8883
```

When you restart the broker you should see that it is now running a websocket connection over port 9001 as well as the standard MQTT connection on port 8883.

# Publish Subscribe

Up to this point all the activities have been using the **HTTP Protocol**, which uses a _request-response_ process (the client requests a resource and the server responds with this resource). If this seems unfamiliar you should work through the HTTP Protocol worksheet.

Whilst this approach works fine for delivering content to a web browser it is not a useful approach for certain applications. Imagine a chat room where you had to refresh the page to view new messages.

In this worksheet you will learn how to use a new HTML5 **websocket** protocol that allows a full duplex (2 way) communication over a single TCP connection. We will then explore the **MQTT** protocol which can be run over websockets and is used to implement a _push message_ system, technically called _publish-subscribe_.

## 1 Set Up

Start by installing the [Mosquitto Tools](https://mosquitto.org/download).

### 1.1 MacOS

If you are using MacOS you should install the [Brew Package Manager](https://brew.sh) and use this to install Mosquitto using `brew install mosquitto`.

### 1.2 Ubuntu

If you are using Ubuntu you can install using the following commands:

```
sudo apt-add-repository ppa:mosquitto-dev/mosquitto-ppa
sudo apt update
sudo apt install mosquitto-clients
```

### 1.3 Windows 10

If you are using Windows 10 you can download the 64 bit Binary exe and install. When you try running the commands you may need to include the **full path to the executable**.

If (when) you encounter issues using Windows we recommend you either dual boot your computer or install [Virtual Box](https://www.virtualbox.org/wiki/Downloads) and use this to install [Ubuntu](https://www.ubuntu.com/download/desktop). There are [detailed instructions](https://askubuntu.com/questions/142549/how-to-install-ubuntu-on-virtualbox) online.

## 2 The MQTT Protocol

Now we have the tools installed we can start using the protocol. You have installed 2 tools, `mosquitto_pub` is used to publish messages and `mosquitto_sub` subscribes to a channel. We will use the `test.mosquitto.org` broker.

Before connecting you will need to download the public key certificate that enables encryption between your computer and the broker. You can find this in the `exercises/MQTT` directory. Since the `mqtt.crt` file contains text this will display in the browser when selected rather than being downloaded. To solve this problem you should download the `mqtt.crt.zip` file and then unzip this to extract the key. Place this in a sensible directory on your computer such as documents.

Nowpening _two_ terminal windows and in both of these navigate to the directory you placed the certificate file in:

In the first window we will run the `mosquitto_sub` command  and subscribe to a _topic_ called `302CEM/XXX` where `XXX` is your university username. Make sure you are running the commands in the same directory as the `mqtt.crt`. The correct password will be given out in the lab session. The topic should include your team name and your own username (replace the xxx). Note that on Windows computers you will need to include the full path to the `mosquitto_sub` command.

```shell
$ mosquitto_sub -v -h mqtt.coventry.ac.uk -p 8883 --cafile mqtt.crt -u 302CEM -P xxx -t 302CEM/elephant/xxx
```

1. The `-v` is the verbose flag and this will force the program to display both the topic and message (if this is omitted it will only print the message).
2. The `-h` flag allows us to specify the _host_, in this case `test.mosquitto.org`.
3. The `-p` flag allows us to set the port (`1883` for non-secure connections and `8883` if the connection is encrypted).
4. The `--cafile` flag is where we tell the tool where the _server's public key_ is located.
5. The `-u` and `-P` flags are used to supply the username and password needed to validate the subscription.
6. The `-t` flag allows us to specify the _topic_, in this case `302CEM/elephant/XXX` (remember to substitute your team nane and username)

The terminal will sit there, subscribed to the chosen topic, waiting to be sent some data which we will send using the second terminal window.

In the second terminal we will run the `mosquitto_pub` command to publish messages to our topic.

```shell
$ mosquitto_pub -h mqtt.coventry.ac.uk -p 8883 --cafile mqtt.crt -u 302CEM -P xxx -t 302CEM/elephant/xxx  -m "hello world"
```

1. The `-m` flag allows us to specify the _message_, in this case `hello world`.

If you look at the first terminal window (running `mosquitto_sub`) you should see your message displayed.

The first terminal will continue to wait for messages until the command is exited by pressing ctrl+c.

### 2.1 Test Your Understanding

Working in small groups of between 2 and 4 people:

1. Each member of the team should choose the same _topic name_ `302CEM/elephant`, substituting your team name.
2. Everyone runs the `mosquitto_sub` tool and subscribes to this same topic.
3. Each person launches a new terminal in a new pane (so you can see both terminal windows).
4. use the `mosquitto_pub` tool to send a message to your chosen _topic name_.
5. Look at the output of your `mosquitto_sub` command (in the first terminal window).
6. What happens if you subscribe to the `302CEM/` topic?

What have you produced? Can you think of any application for this...


## References

https://www.thepolyglotdeveloper.com/2016/06/connect-raspberry-pi-zero-usb-cable-ssh/

https://dzone.com/articles/mqtt-security-securing-a-mosquitto-server

http://www.steves-internet-guide.com/mqtt-websockets/