Installing an MQTT Broker

Now we will install (and partially secure) a Mosquitto MQTT broker so that data can be published to it and the client can subscribe to and receive data. We will be installing this using the apt command however the version installed by default is quite old and does not support the latest MQTT features. To resolve this we will add a package repository containing the more recent version. We need the version that matches the version of Debian we have installed. In the example below, the response from the first command indicates we are running jessie so the url when we retrieve the list points to the jessie version. If you get a different result, change the URL to match.

$ cat /etc/os-release
  VERSION="8 (jessie)"
$ wget http://repo.mosquitto.org/debian/mosquitto-repo.gpg.key
$ sudo apt-key add mosquitto-repo.gpg.key
  OK
$ cd /etc/apt/sources.list.d/
$ sudo wget http://repo.mosquitto.org/debian/mosquitto-jessie.list
cd ~

Once this is done we can update the cached package list and install the mosquitto server and the client tools.

sudo apt update
sudo apt search mosquitto
sudo apt install mosquitto mosquitto-clients
mosquitto_pub

This last command tries to run the mosquitto publish tool with no parameters. This will show the help page. We know we have the latest version if we can see a --cafile flag listed. To see if the server is running (on its default port of 1883) we can run:

4.1 Managing the Service

$ netstat -at
  Active Internet connections (servers and established)
  Proto Recv-Q Send-Q Local Address           Foreign Address         State
  tcp        0      0 *:ssh                   *:*                     LISTEN
  tcp        0      0 *:1883                  *:*                     LISTEN
  tcp        0    208 elephant:ssh            local-comp:55951        ESTABLISHED
  tcp6       0      0 [::]:ssh                [::]:*                  LISTEN
  tcp6       0      0 [::]:1883               [::]:*                  LISTEN <--

Notice the last line, this shows the service is running on port 1883.

Whenever you update the configuration you need to restart the service. Here are the commands to stop start and restart:

sudo /etc/init.d/mosquitto stop
sudo /etc/init.d/mosquitto start
sudo /etc/init.d/mosquitto restart

4.2 Publishing and Subscribing

Now we have the MQTT Broker configured we need to learn how to publish messages to and subscribe to published messages.

You should be able to use the mosquitto-clients tools covered in the previous lab. Remember that the broker is currently not secure and so will run on port 1883. The hostname will be the one you assigned to the server earlier in the lab and of course you won't need to specify username, password or CA file.

The server is logging its activity to the /var/log/mosquitto/mosquitto.log file. Linux allows you to see the last entries in the log file using the tail command. To see a live, updating you can add the -f (follow) flag.

sudo tail -f /var/log/mosquitto/mosquitto.log

Run this command on the server and see what gets logged when you publish and subscribe. If you are getting any errors with the mosquitto tools you should check this log to find out the cause.

5 Securing the Broker

Although the broker does work it lacks any form of security. In this section we will be correcting this. Under linux, all system-related configuration files are in the /etc directory and, if you study its contents, you will find a mosquitto subdirectory. This is where we access all the config settings for the broker. In the rest of this section all files we mention are in this directory.

5.1 User Accounts

The first step is to add some usernames and passwords. The mosquitto server comes with the mosquitto_passwd command to allow us to add multiple user accounts. These are stored in the passwd text file. The file is only read-only for non-root users. Use the following command to add a new user, substituting the chosen username and password where shown.

sudo touch /etc/mosquitto/passwd
sudo mosquitto_passwd -b /etc/mosquitto/passwd username password

You will also need to modify your config file to prevent anonymous use and tell it where to find the password file. Edit the mosquitto.conf file and add the following two lines:

allow_anonymous false
password_file /etc/mosquitto/passwd

Once you have added users you need to restart the broker for the changes to take effect.

After restarting the server you will need to supply a valid username and password when you connect. Try publishing and subscribing to check this is true.

5.2 Access Control List

You have probably noticed that even with multiple accounts, all of these can publish and subscribe to any topic. Since we want to introduce some privacy we need to create an access control list which defines which topics can be published to (written) or subscribed to (read). You should create a new file in the /etc/mosquitto/ directory called acl. You will need root permissions to do this.

sudo nano /etc/mosquitto/acl

You should then define the access permissions for the different accounts. In the example below we have two user accounts called house and garden. You can assign one of three permissions:

read means the user can subscribe to the topic.
write means the user can publish to the topic.
readwrite means the user can both publish and subscribe to the topic.

The # character signifies a wildcard (any combination of topic segments).

user house
topic readwrite house/#

user garden
topic readwrite garden/#

user powerx
topic readwrite powerx/#

user 4009user
topic readwrite 4009user/#
topic readwrite owntracks/#

Once you have added the appropriate access permissions we need to let mosquitto know the access control list exists. Edit the /etc/mosquitto/mosquitto.conf file and add the following line:

acl_file /etc/mosquitto/acl

Restart the broker for the changes to take effect. Now try to publish and subscribe both to topics in and not in the access control list. Notice that if you try to publish or subscribe off-topic you dont get an error but the action will fail.

5.3 Implementing SSL

The final step is to implement SSL to ensure the data is encrypted as it passes between the broker and clients. Let's Encrypt is a new service offering free SSL certificates through an automated API.

wget https://github.com/owntracks/tools/raw/master/TLS/generate-CA.sh

5.3 Implementing SSL (self signed)

The final step is to implement SSL to ensure data is encrypted as it passes between the broker and client.

5.3.1 Installing OpenSSL

Warning: depending on the speed of the Raspberry Pi, thie process of building OpenSSL can take several hours to complete. It is recommended that you run the installation step overnight.

OpenSSL is a robust, commercial-grade, and full-featured toolkit for the Transport Layer Security (TLS) and Secure Sockets Layer (SSL) protocols. It is also a general-purpose cryptography library. It is needed to generate the required certificates. We need to ensure we are always using the latest version and so, rather than install from a repository we will build from source.

git clone git://git.openssl.org/openssl.git
cd openssl
./config
make
make test
sudo make install

5.3.2 Generating the Certificates

On most Linux distros, the certificates are typically stored in the /etc/ssl/ directory so we need to create this and navigate to it. Note you may need root privileges to create the directory if it does not already exist.

mkdir /etc/ssl/
cd /etc/ssl/

Now we need to create a 2048-bit key called mosq-ca.key and use this to create an X509 certificate. During the second step you will be asked a series of questions, it doesn't matter what you enter.

sudo openssl genrsa -out mosq-ca.key 2048
sudo openssl req -new -x509 -days 365 -key mosq-ca.key -out mosq-ca.crt

Next we create the private key and use this to create the Certificate Signing Request (CSR). Note you will need to enter the same information a second time.

sudo openssl genrsa -out mosq-serv.key 2048
sudo openssl req -new -key mosq-serv.key -out mosq-serv.csr

Normally this would be sent to the Certification authority that, after verifying the author identity, returns a certificate but in this example we will use a self-signed certificate and verify the certificate.

sudo openssl x509 -req -in mosq-serv.csr -CA mosq-ca.crt -CAkey mosq-ca.key -CAcreateserial -out mosq-serv.crt -days 365 -sha256
sudo openssl x509 -in mosq-serv.crt -noout -text

5.3.3 Configuring the Broker

Now the certificates are ready we have to configure MQTT Mosquitto Server so that it can use these certificates by editing the mosquitto.conf file and adding the following to the end.

listener 8883
cafile /etc/ssl/mosq-ca.crt
certfile /etc/ssl/mosq-serv.crt
keyfile /etc/ssl/mosq-serv.key

Restarting the broker will put the changes into effect, you should see that this is now running over port 8883. If you want to connect to the broker now you will need to download the mosq-ca.crt certificate and use this when making connections to the broker.

One way to download the certificate is to log out of the server and use the rsync tool to copy the file to your computer using the ssh connection. For example the following command copies the file to the ~/Documents directory on the local workstation:

rsync -avzhe ssh root@elephant:/etc/ssl/mosq-ca.crt ~/Documents/

5.4 Configuring Websockets

The server is now configured as an MQTT broker however if you plan to use a browser to connect to it you will need to enable MQTT over Websocket_. You will need to enable this in your mosquitto.conf configuration file by adding the following:

listener 9001
protocol websockets

Rather than restarting the broker this time we will tell it to use our updated config file. Run the command:

$ mosquitto -c /etc/mosquitto/mosquitto.conf
  opening websockets listen socket on port 9001
  opening ipv4 listen socket on port 8883
  opening ipv6 listen socket on port 8883

When you restart the broker you should see that it is now running a websocket connection over port 9001 as well as the standard MQTT connection on port 8883.

Publish Subscribe

Up to this point all the activities have been using the HTTP Protocol, which uses a request-response process (the client requests a resource and the server responds with this resource). If this seems unfamiliar you should work through the HTTP Protocol worksheet.

Whilst this approach works fine for delivering content to a web browser it is not a useful approach for certain applications. Imagine a chat room where you had to refresh the page to view new messages.

In this worksheet you will learn how to use a new HTML5 websocket protocol that allows a full duplex (2 way) communication over a single TCP connection. We will then explore the MQTT protocol which can be run over websockets and is used to implement a push message system, technically called publish-subscribe.

1 Set Up

Start by installing the Mosquitto Tools.

1.1 MacOS

If you are using MacOS you should install the Brew Package Manager and use this to install Mosquitto using brew install mosquitto.

1.2 Ubuntu

If you are using Ubuntu you can install using the following commands:

sudo apt-add-repository ppa:mosquitto-dev/mosquitto-ppa
sudo apt update
sudo apt install mosquitto-clients

1.3 Windows 10

If you are using Windows 10 you can download the 64 bit Binary exe and install. When you try running the commands you may need to include the full path to the executable.

If (when) you encounter issues using Windows we recommend you either dual boot your computer or install Virtual Box and use this to install Ubuntu. There are detailed instructions online.

2 The MQTT Protocol

Now we have the tools installed we can start using the protocol. You have installed 2 tools, mosquitto_pub is used to publish messages and mosquitto_sub subscribes to a channel. We will use the test.mosquitto.org broker.

Before connecting you will need to download the public key certificate that enables encryption between your computer and the broker. You can find this in the exercises/MQTT directory. Since the mqtt.crt file contains text this will display in the browser when selected rather than being downloaded. To solve this problem you should download the mqtt.crt.zip file and then unzip this to extract the key. Place this in a sensible directory on your computer such as documents.

Nowpening two terminal windows and in both of these navigate to the directory you placed the certificate file in:

In the first window we will run the mosquitto_sub command and subscribe to a topic called 302CEM/XXX where XXX is your university username. Make sure you are running the commands in the same directory as the mqtt.crt. The correct password will be given out in the lab session. The topic should include your team name and your own username (replace the xxx). Note that on Windows computers you will need to include the full path to the mosquitto_sub command.

$ mosquitto_sub -v -h mqtt.coventry.ac.uk -p 8883 --cafile mqtt.crt -u 302CEM -P xxx -t 302CEM/elephant/xxx

The -v is the verbose flag and this will force the program to display both the topic and message (if this is omitted it will only print the message).
The -h flag allows us to specify the host, in this case test.mosquitto.org.
The -p flag allows us to set the port (1883 for non-secure connections and 8883 if the connection is encrypted).
The --cafile flag is where we tell the tool where the server's public key is located.
The -u and -P flags are used to supply the username and password needed to validate the subscription.
The -t flag allows us to specify the topic, in this case 302CEM/elephant/XXX (remember to substitute your team nane and username)

The terminal will sit there, subscribed to the chosen topic, waiting to be sent some data which we will send using the second terminal window.

In the second terminal we will run the mosquitto_pub command to publish messages to our topic.

$ mosquitto_pub -h mqtt.coventry.ac.uk -p 8883 --cafile mqtt.crt -u 302CEM -P xxx -t 302CEM/elephant/xxx  -m "hello world"

The -m flag allows us to specify the message, in this case hello world.

If you look at the first terminal window (running mosquitto_sub) you should see your message displayed.

The first terminal will continue to wait for messages until the command is exited by pressing ctrl+c.

2.1 Test Your Understanding

Working in small groups of between 2 and 4 people:

Each member of the team should choose the same topic name 302CEM/elephant, substituting your team name.
Everyone runs the mosquitto_sub tool and subscribes to this same topic.
Each person launches a new terminal in a new pane (so you can see both terminal windows).
use the mosquitto_pub tool to send a message to your chosen topic name.
Look at the output of your mosquitto_sub command (in the first terminal window).
What happens if you subscribe to the 302CEM/ topic?

What have you produced? Can you think of any application for this...

References

https://www.thepolyglotdeveloper.com/2016/06/connect-raspberry-pi-zero-usb-cable-ssh/

https://dzone.com/articles/mqtt-security-securing-a-mosquitto-server

http://www.steves-internet-guide.com/mqtt-websockets/

rpi/05 Creating an MQTT Broker.md