02 MQTT Raspberry Pi.md

# Setting up the Raspberry Pi

In this worksheet you will be shown how to setup and configure a raspberry pi as a central 'hub' for your home automation system. By the end of this you will have:

1. The operating system (minibian) installed on the hub.
2. The I2C interface configured for use (and a small display attached).
3. A working MQTT broker installed (but not fully secured!).
4. The latest versions of Python and Pip.
5. A NodeJS runtime environment for running scripts. We will then use this to publish a simple API.

## 1 Installing the Operating System

In this worksheet we will be using the [Raspbian Stretch Lite distro](https://www.raspberrypi.org/downloads/raspbian/) which is a very cut down version of the [Debian Server](https://www.debian.org/distrib/). To complete this step you will need an SD card of at least 8GB that fits your Raspberry Pi, the size needs to be at least 4GB. Make sure you download the **lite** version (which should be around 370MB)!

The download will be an `.img` file and needs to be burned onto the SD card. There are [detailed instructions](https://www.raspberrypi.org/documentation/installation/installing-images/) for the different OS you might be running on your laptop.

On a Mac you will need the following commands, google them to understand what they do:

```
diskutil list external physical                                        find the drive number.
diskutil randomDisk diskX                                              formats disk, replace X with the drive number.
 sudo dd bs=1m if=2018-11-13-raspbian-stretch-lite.img of=/dev/disk3   burns the specified image to the specified drive.
```

Once you have installed the image, insert it into the RPi, connect a network cable, keyboard and monitor plus a suitable MicroUSB power supply then you can boot up the computer. Once you have booted to the login prompt you can log in, username pi password raspberry.

### 1.1 Connecting Using SSH Over USB

At this stage you would normally need to plug in a keyboard and monitor before booting the device. If you don't have the facilities to do this (or fancy a challenge) you can configure your device to allow you to plug the Pi into your computer using a USB cable and secure shell into it over this cable.

With the USB card still inserted into your laptop, open it and locate the `confg.txt` file, open it in a suitable editor such as VS Code and add the following line to the end:

```
dtoverlay=dwc2
```

The next file we alter is cmdline.txt, but it is a bit different. Parameters in this file are not delimited by new lines or commas, they are delimited by space characters. In this file we want to add the following after the `rootwait` parameter:

```
modules-load=dwc2,g_ether
```

Finally we want to enable SSH. By default this is disabled so, for the first connection we override this by creating an empty text file called `ssh` on the SD card (no file extension).

### 1.2 Resizing the Partition

When you install the image it only uses a small part of the SD card. Your first task is to install the `raspi-config` tool.

```
apt update
apt upgrade -y
apt install -y apt-utils raspi-config

raspi-config
```

This will launch the config program. Choose the **Advanced** option from the menu and the option to extend the partition. Once this is done you need to exit the config program then restart the server.

_PRO TIP: there is a subcommand called `nonint` (non-interactive) which supports commands that correspond with the options you select in the interactive tool, these are now [documented](https://raspberrypi.stackexchange.com/questions/28907/how-could-one-automate-the-raspbian-raspi-config-setup). By using the subcommands you can automate the process by creating a shell script. To extend the partition you would run the command:_

```
raspi-config nonint do_expand_rootfs
```

If you change to partition tables you need to restart the server for the changes to take effect.

```
reboot
```

When the system has rebooted and you have logged in, check that the main partition is as big as possible:

```
df -h
```

### 1.3 Changing the Host Name

Every device on the network is configured with a _host name_. If you run the `hostname` command you can see that this defaults to `minibian`. Since every raspberry pi running the Minibian distro could have the same name this is not very satisfactory! Run the `raspi-config` tool again and note the second menu item, `Hostname`. Select this. The next screen contains detailed instructions, click OK once you have read these.

On the next screen you can delete the existing name and enter your own. For the purposes of this module you should make the hostname your team's animal name in lowercase, for example `elephant`. If you exit the tool you will be asked if you want to reboot, do this.

Once the system has rebooted and you are logged in, run the `hostname` command to check this has updated.

_PRO TIP: Use the `raspi-config nonint` subcommand to set the hostname:_

```
raspi-config nonint do_hostname elephant
```

### 1.4 Enabling SSH

Since we want to run the server headles (no keyboard, mouse and monitor) we need to enable SSH to allow us to connect remotely. This is done using the `raspi-config` tool and accessing the _interfacing options_ menu. Make sure you complete this step!

_PRO TIP: use the raspi-config nonint_ subcommand to enable SSH:_

```
raspi-config nonint do_ssh 0
```

### 1.5 Installing Key Packages

Now we need to install a number of important packages:

```
apt install -y apt-transport-https rsync sudo nano git tree curl software-properties-common mc build-essential libssl-dev libffi-dev i2c-tools
```

### 1.6 Creating a Normal User

Up to now we have been logging in as `root`, this is both insecure and dangerous. In this section we will be creating a user with standard permissions then adding them to the _sudoers_ group so they can execute commands as root if needed. In this example the username is `userx` but you should change this to whatever you want.

1. The `-m` flag tells the computer to create the home directory if it does not exist.
2. The `-d` flag tells the computer to make the `/home/userx` directory the default home directory.
3. The `-s` flag indicates that the default shell should be `bash`.
4. The `-c` flag is a comment attached to the account.
5. Finally we specify the username.

Once the account is created we need to set its password and add it to the __sudoers_ group so that we can execute commands that need root permissions.

```
useradd -m -d /home/userx -s /bin/bash -c "userx account" userx
passwd userx
usermod -aG sudo userx
```

_PRO TIP: changing a user's password required you to type the password into the terminal (using stdin). To automate this process in a shell script you need to pipe a string to the `chpasswd` command. This means you won't be prompted for the password._

```
echo 'userx:mysecurepw'|chpasswd
```

We should also take the opportunity to change the root password to something a little less _default_. You should make this password very difficult to guess but then make sure you keep a record of this. You will rarely ever need to log in using this root account.

```
passwd
```

Now log out of the root account and log in using your new user account.

## 2 Configuring I2C

Next we will configure the I2C interface so that you can attach sensors that make use of it. By the end of this section you should have a screen that displays the computer host name and ip address whenever you power it on. We will also explore how to capture data from an I2C enabled sensor.

Start by running the `raspi-config` command. This time access the _interface options_ and choose `Enable/Disable automatic loading of I2C kernel module` to enable I2C. After exiting the tool you need to reboot. We are now no longer logged in as `root` but rebooting requires root permissions. Since we are part of the _sudoers_ group we can run a command with root privileges by prefixing it with the `sudo` keyword.

_PRO TIP: use the `raspi-config` tool in non-interactive mode to enable I2C:_

```
sudo raspi-config nonint do_i2c 0
```

```
sudo reboot
```

When it has rebooted, log in again using your normal user account.

### 2.1 Fixing I2C Permissions

By default, the I2C bus is only accessible if you use root permissions. The problem is that we don't want to run all our python scripts as root! To fix this you need to edit the `/lib/udev/rules.d/60-i2c-tools.rules` file using the nano text editor.

```
sudo nano /lib/udev/rules.d/60-i2c-tools.rules
```

Notice that we have to edit the file with root privileges. The Raspberry Pi includes two I2C buses, labelled `i2c-0` and `i2c-1`, We will be using bus 1. Replace the existing contents with the following:

```
KERNEL=="i2c-0"     , GROUP="i2c", MODE="0660"
KERNEL=="i2c-[1-9]*", GROUP="i2c", MODE="0666"
```

This splits this so that the permissions are retained for interface 0 but every user can access all the others (bus 1).

### 2.2 Changing the I2C Baud Rate

By default the I2C bus runs at a slow 100Kb/s speed however the Raspberry Pi has a "fast mode" (400Kb/s) driver. To enable this we need to edit the `/boot/config.txt` file using root privileges:

```
sudo nano /boot/config.txt
```

modify the following line:

```
dtparam=i2c_arm=on,i2c_arm_baudrate=400000
```

Having made our changes we need to restart the server.

### 2.3 Connecting an OLED Screen

There is a command called `i2cdetect` which scans an I2C but looking for attached devices. We will be using I2C bus 1 so we run the following command:

```
$ sudo i2cdetect -y 1
       0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
  00:          -- -- -- -- -- -- -- -- -- -- -- -- -- 
  10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
  20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
  30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
  40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
  50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
  60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
  70: -- -- -- -- -- -- -- --
```

As you can see, no devices are detected.

Now lets check the I2C bus is configured correctly by wiring up a small OLED screen. You should find one in your electronics kit. Notice that it has 4 pins labelled `GND`, `VCC`, `SCK`, `SDA`. These need to be connected to the matching pins on the Raspberry Pi header. Below you can see the pins labelled. This labelling applies to both the RPi v2 and v3. If you look carefully at the diagram you can see that the pins connected to the `I2C-1` bus are clearly labelled:

1. Pin 6 is labelled `GND`, this should be connected to the `GND` pin on the screen.
2. Pin 4 is labelled `5V PWR`, this should be connected to the `VCC` pin.
3. Pin 5 is labelled `I2C SCL`, this should be connected to the `SCK` pin.
4. Pin 3 is labelled `I2C1 SDA`, this should be connected to the `SDA` pin.

![Raspberry Pi 2 Pinout](exercises/.images/rp2_pinout.png)

If we run the `i2cdetect` command again we should see that the screen is detected and is using I2C address `0x3c`.

```
$ sudo i2cdetect -y 1
       0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
  00:          -- -- -- -- -- -- -- -- -- -- -- -- --
  10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
  20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
  30: -- -- -- -- -- -- -- -- -- -- -- -- 3c -- -- --
  40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
  50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
  60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
  70: -- -- -- -- -- -- -- --
```

This demonstrates both that the i2c bus is configured correctly and that the screen is working.

### 2.4 Writing to the Screen

Finally we can create a simple python script to test out the screen. We first need to install a couple of packages and then some libraries. The first is the Adafruit SSD1306 library that is needed to control the screen.

```
sudo apt install -y python-pip python-imaging python-dev python-smbus
```

```
git clone https://github.com/adafruit/Adafruit_Python_SSD1306.git
cd Adafruit_Python_SSD1306
sudo python setup.py install
```

The second is the GPIO library that needs to be installed using the pip package tool:

```
sudo pip install RPi.GPIO
```

Create a new script in your home directory on the Raspberry Pi and call it `oled.py`. Open it in the nano editor and paste in the contents of `oled.py` which you can find in the `exercises/01_rpi/` directory, save and exit from nano.

At the moment the script is not executable so, before running it we need to set the execution flag on the file:

```
chmod +x oled.py
./oled.py
```

If you look at the screen you should see a message...

### 2.5 Running a Script at Boot

The final task in this section is to create a script that displays the hostname and ip address on the screen when the raspberry pi first boots. The first task is to create a script in the `/bin/` directory.

```
sudo nano /bin/network.py
```

Copy in the contents of the `network.py` script, save and quit nano. Now you need to make the file executable.

```
sudo chmod +x /bin/network.py
```

If we want the script to run automatically we need to add it as a _cron_ task. We do this by running the `crontab -e` command, this opens the _crontab_ file. Add the following to the end:

```
@reboot python /bin/network.py &
```

This tells the crontab to run the script when the device boots. The `&` character runs the script in the background. Try rebooting the server, you should see the information displayed as soon as the reboot is complete.

## 3 Connecting using Secure Shell

Up to this point we have connected a monitor and keyboard to our server and used this to interact with the system. Most servers run _headless_ and we connect remotely using secure shell (ssh). To do this we need to know the _hostname_ or the _ip address_ of the system. Thanks to the previous step, every time we boot the server both these pieces of information are displayed on the attached OLED screen!

Start by disconnecting the screen and keyboard and reconnecting any cables you removed in the first place...

If you are using a Mac or Linux laptop you connect using the ssh command through the terminal:

```
$ ssh userx@elephant
```

You will be prompted for your password and, the first time you connect, the server will attempt to send you its public encryption key, you will be asked if you want to accept this.

If the credentials are correct you are now connected to the server and all the commands you enter will be run on it. To quit and return to your computer enter the `exit` command.

If you are using Windows you will probably need to download the [Putty](https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html) application. After launch you need to enter the hostname and user credentials in the appropriate boxes.

From now onwards you should always connect using SSH (you will never need to plug in the keyboard and monitor).

## 4 Installing an MQTT Broker

Now we will install (and partially secure) a [Mosquitto MQTT broker](https://mosquitto.org/) so that data can be published to it and the client can subscribe to and receive data. We will be installing this using the `apt` command however the version installed by default is quite old and does not support the latest MQTT features. To resolve this we will add a package repository containing the more recent version. We need the version that matches the version of Debian we have installed. In the example below, the response from the first command indicates we are running `jessie` so the url when we retrieve the list points to the jessie version. If you get a different result, change the URL to match.

```
$ cat /etc/os-release
  VERSION="8 (jessie)"
$ wget http://repo.mosquitto.org/debian/mosquitto-repo.gpg.key
$ sudo apt-key add mosquitto-repo.gpg.key
  OK
$ cd /etc/apt/sources.list.d/
$ sudo wget http://repo.mosquitto.org/debian/mosquitto-jessie.list
cd ~
```

Once this is done we can update the cached package list and install the mosquitto server and the client tools.

```
sudo apt update
sudo apt search mosquitto
sudo apt install mosquitto mosquitto-clients
mosquitto_pub
```

This last command tries to run the mosquitto publish tool with no parameters. This will show the help page. We know we have the latest version if we can see a `--cafile` flag listed. To see if the server is running (on its default port of `1883`) we can run:

### 4.1 Managing the Service

```
$ netstat -at
  Active Internet connections (servers and established)
  Proto Recv-Q Send-Q Local Address           Foreign Address         State
  tcp        0      0 *:ssh                   *:*                     LISTEN
  tcp        0      0 *:1883                  *:*                     LISTEN
  tcp        0    208 elephant:ssh            local-comp:55951        ESTABLISHED
  tcp6       0      0 [::]:ssh                [::]:*                  LISTEN
  tcp6       0      0 [::]:1883               [::]:*                  LISTEN <--
```

Notice the last line, this shows the service is running on port `1883`.

Whenever you update the configuration you need to restart the service. Here are the commands to stop start and restart:

```
sudo /etc/init.d/mosquitto stop
sudo /etc/init.d/mosquitto start
sudo /etc/init.d/mosquitto restart
```

### 4.2 Publishing and Subscribing

Now we have the MQTT Broker configured we need to learn how to publish messages to and subscribe to published messages.

You should be able to use the `mosquitto-clients` tools covered in the previous lab. Remember that the broker is currently not secure and so will run on port `1883`. The hostname will be the one you assigned to the server earlier in the lab and of course you won't need to specify username, password or CA file.

The server is logging its activity to the `/var/log/mosquitto/mosquitto.log` file. Linux allows you to see the last entries in the log file using the `tail` command. To see a live, updating you can add the `-f` (follow) flag.

```
sudo tail -f /var/log/mosquitto/mosquitto.log
```

Run this command on the server and see what gets logged when you publish and subscribe. If you are getting any errors with the mosquitto tools you should check this log to find out the cause.

## 5 Securing the Broker

Although the broker does work it lacks any form of security. In this section we will be correcting this. Under linux, all system-related configuration files are in the `/etc` directory and, if you study its contents, you will find a `mosquitto` subdirectory. This is where we access all the config settings for the broker. In the rest of this section all files we mention are in this directory.

### 5.1 User Accounts

The first step is to add some usernames and passwords. The mosquitto server comes with the `mosquitto_passwd` command to allow us to add multiple user accounts. These are stored in the `passwd` text file. The file is only read-only for non-root users. Use the following command to add a new user, substituting the chosen username and password where shown.

```
sudo touch /etc/mosquitto/passwd
sudo mosquitto_passwd -b /etc/mosquitto/passwd username password
```

You will also need to modify your config file to prevent anonymous use and tell it where to find the password file. Edit the `mosquitto.conf` file and add the following two lines:

```
allow_anonymous false
password_file /etc/mosquitto/passwd
```

Once you have added users you need to restart the broker for the changes to take effect.

After restarting the server you will need to supply a valid username and password when you connect. Try publishing and subscribing to check this is true.

### 5.2 Access Control List

You have probably noticed that even with multiple accounts, all of these can publish and subscribe to any topic. Since we want to introduce some privacy we need to create an _access control list_ which defines which topics can be published to (written) or subscribed to (read). You should create a new file in the `/etc/mosquitto/` directory called `acl`. You will need root permissions to do this.

```
sudo nano /etc/mosquitto/acl
```

You should then define the access permissions for the different accounts. In the example below we have two user accounts called `house` and `garden`. You can assign one of three permissions:

1. `read` means the user can _subscribe_ to the topic.
2. `write` means the user can _publish_ to the topic.
3. `readwrite` means the user can both publish and subscribe to the topic.

The `#` character signifies a wildcard (any combination of topic segments).

```
user house
topic readwrite house/#

user garden
topic readwrite garden/#

user powerx
topic readwrite powerx/#

user 4009user
topic readwrite 4009user/#
topic readwrite owntracks/#
```

Once you have added the appropriate access permissions we need to let mosquitto know the access control list exists. Edit the `/etc/mosquitto/mosquitto.conf` file and add the following line:

```
acl_file /etc/mosquitto/acl
```

Restart the broker for the changes to take effect. Now try to publish and subscribe both to topics in and not in the access control list. Notice that if you try to publish or subscribe off-topic you dont get an error but the action will fail.

### 5.3 Implementing SSL

The final step is to implement SSL to ensure the data is encrypted as it passes between the broker and clients. Let's Encrypt is a new service offering free SSL certificates through an automated API.

```
wget https://github.com/owntracks/tools/raw/master/TLS/generate-CA.sh
```

--------

### 5.3 Implementing SSL (self signed)

The final step is to implement SSL to ensure data is encrypted as it passes between the broker and client.

#### 5.3.1 Installing OpenSSL

**Warning: depending on the speed of the Raspberry Pi, thie process of building OpenSSL can take several hours to complete. It is recommended that you run the installation step overnight.**

OpenSSL is a robust, commercial-grade, and full-featured toolkit for the Transport Layer Security (TLS) and Secure Sockets Layer (SSL) protocols. It is also a general-purpose cryptography library. It is needed to generate the required certificates. We need to ensure we are always using the latest version and so, rather than install from a repository we will build from source.

```
git clone git://git.openssl.org/openssl.git
cd openssl
./config
make
make test
sudo make install
```

#### 5.3.2 Generating the Certificates

On most Linux distros, the certificates are typically stored in the `/etc/ssl/` directory so we need to create this and navigate to it. Note you may need root privileges to create the directory if it does not already exist.

```
mkdir /etc/ssl/
cd /etc/ssl/
```

Now we need to create a 2048-bit key called `mosq-ca.key` and use this to create an X509 certificate. During the second step you will be asked a series of questions, it doesn't matter what you enter.

```
sudo openssl genrsa -out mosq-ca.key 2048
sudo openssl req -new -x509 -days 365 -key mosq-ca.key -out mosq-ca.crt
```

Next we create the private key and use this to create the Certificate Signing Request (CSR). Note you will need to enter the same information a second time.

```
sudo openssl genrsa -out mosq-serv.key 2048
sudo openssl req -new -key mosq-serv.key -out mosq-serv.csr
```

Normally this would be sent to the Certification authority that, after verifying the author identity, returns a certificate but in this example we will use a self-signed certificate and verify the certificate.

```
sudo openssl x509 -req -in mosq-serv.csr -CA mosq-ca.crt -CAkey mosq-ca.key -CAcreateserial -out mosq-serv.crt -days 365 -sha256
sudo openssl x509 -in mosq-serv.crt -noout -text
```

#### 5.3.3 Configuring the Broker

Now the certificates are ready we have to configure MQTT Mosquitto Server so that it can use these certificates by editing the  `mosquitto.conf` file and adding the following to the end.

```
listener 8883
cafile /etc/ssl/mosq-ca.crt
certfile /etc/ssl/mosq-serv.crt
keyfile /etc/ssl/mosq-serv.key
```

Restarting the broker will put the changes into effect, you should see that this is now running over port 8883. If you want to connect to the broker now you will need to download the `mosq-ca.crt` certificate and use this when making connections to the broker.

One way to download the certificate is to log out of the server and use the rsync tool to copy the file to your computer using the ssh connection. For example the following command copies the file to the `~/Documents` directory on the local workstation:

```
rsync -avzhe ssh root@elephant:/etc/ssl/mosq-ca.crt ~/Documents/
```

### 5.4 Configuring Websockets

The server is now configured as an MQTT broker however if you plan to use a browser to connect to it you will need to enable _MQTT over Websocket__. You will need to enable this in your `mosquitto.conf` configuration file by adding the following:

```
listener 9001
protocol websockets
```

Rather than restarting the broker this time we will tell it to use our updated config file. Run the command:

```
$ mosquitto -c /etc/mosquitto/mosquitto.conf
  opening websockets listen socket on port 9001
  opening ipv4 listen socket on port 8883
  opening ipv6 listen socket on port 8883
```

When you restart the broker you should see that it is now running a websocket connection over port 9001 as well as the standard MQTT connection on port 8883.

## References

https://www.thepolyglotdeveloper.com/2016/06/connect-raspberry-pi-zero-usb-cable-ssh/

https://dzone.com/articles/mqtt-security-securing-a-mosquitto-server

http://www.steves-internet-guide.com/mqtt-websockets/