In this worksheet you will be shown how to setup and configure a raspberry pi as a central 'hub' for your home automation system. By the end of this you will have:
- The operating system (minibian) installed on the hub.
- The I2C interface configured for use (and a small display attached).
- A working MQTT broker installed (but not fully secured!).
- The latest versions of Python and Pip.
- A NodeJS runtime environment for running scripts. We will then use this to publish a simple API.
In this worksheet we will be using the Raspbian Stretch Lite distro which is a very cut down version of the Debian Server. To complete this step you will need an SD card of at least 8GB that fits your Raspberry Pi, the size needs to be at least 4GB. Make sure you download the lite version (which should be around 370MB)!
The download will be an .img file and needs to be burned onto the SD card. There are detailed instructions for the different OS you might be running on your laptop.
On a Mac you will need the following commands, google them to understand what they do:
diskutil list external physical find the drive number.
diskutil randomDisk diskX formats disk, replace X with the drive number.
sudo dd bs=1m if=2018-11-13-raspbian-stretch-lite.img of=/dev/disk3 burns the specified image to the specified drive.
Once you have installed the image, insert it into the RPi, connect a network cable, keyboard and monitor plus a suitable MicroUSB power supply then you can boot up the computer. Once you have booted to the login prompt you can log in, username pi password raspberry.
At this stage you would normally need to plug in a keyboard and monitor before booting the device. If you don't have the facilities to do this (or fancy a challenge) you can configure your device to allow you to plug the Pi into your computer using a USB cable and secure shell into it over this cable.
With the USB card still inserted into your laptop, open it and locate the confg.txt file, open it in a suitable editor such as VS Code and add the following line to the end:
dtoverlay=dwc2
The next file we alter is cmdline.txt, but it is a bit different. Parameters in this file are not delimited by new lines or commas, they are delimited by space characters. In this file we want to add the following after the rootwait parameter:
modules-load=dwc2,g_ether
Finally we want to enable SSH. By default this is disabled so, for the first connection we override this by creating an empty text file called ssh on the SD card (no file extension).
When you install the image it only uses a small part of the SD card. Your first task is to install the raspi-config tool.
apt update
apt upgrade -y
apt install -y apt-utils raspi-config
raspi-config
This will launch the config program. Choose the Advanced option from the menu and the option to extend the partition. Once this is done you need to exit the config program then restart the server.
PRO TIP: there is a subcommand called nonint (non-interactive) which supports commands that correspond with the options you select in the interactive tool, these are now documented. By using the subcommands you can automate the process by creating a shell script. To extend the partition you would run the command:
raspi-config nonint do_expand_rootfs
If you change to partition tables you need to restart the server for the changes to take effect.
reboot
When the system has rebooted and you have logged in, check that the main partition is as big as possible:
df -h
Every device on the network is configured with a host name. If you run the hostname command you can see that this defaults to minibian. Since every raspberry pi running the Minibian distro could have the same name this is not very satisfactory! Run the raspi-config tool again and note the second menu item, Hostname. Select this. The next screen contains detailed instructions, click OK once you have read these.
On the next screen you can delete the existing name and enter your own. For the purposes of this module you should make the hostname your team's animal name in lowercase, for example elephant. If you exit the tool you will be asked if you want to reboot, do this.
Once the system has rebooted and you are logged in, run the hostname command to check this has updated.
PRO TIP: Use the raspi-config nonint subcommand to set the hostname:
raspi-config nonint do_hostname elephant
Since we want to run the server headles (no keyboard, mouse and monitor) we need to enable SSH to allow us to connect remotely. This is done using the raspi-config tool and accessing the interfacing options menu. Make sure you complete this step!
PRO TIP: use the raspi-config nonint subcommand to enable SSH:_
raspi-config nonint do_ssh 0
Now we need to install a number of important packages:
apt install -y apt-transport-https rsync sudo nano git tree curl software-properties-common mc build-essential libssl-dev libffi-dev i2c-tools
Up to now we have been logging in as root, this is both insecure and dangerous. In this section we will be creating a user with standard permissions then adding them to the sudoers group so they can execute commands as root if needed. In this example the username is userx but you should change this to whatever you want.
- The
-mflag tells the computer to create the home directory if it does not exist. - The
-dflag tells the computer to make the/home/userxdirectory the default home directory. - The
-sflag indicates that the default shell should bebash. - The
-cflag is a comment attached to the account. - Finally we specify the username.
Once the account is created we need to set its password and add it to the _sudoers group so that we can execute commands that need root permissions.
useradd -m -d /home/userx -s /bin/bash -c "userx account" userx
passwd userx
usermod -aG sudo userx
PRO TIP: changing a user's password required you to type the password into the terminal (using stdin). To automate this process in a shell script you need to pipe a string to the chpasswd command. This means you won't be prompted for the password.
echo 'userx:mysecurepw'|chpasswd
We should also take the opportunity to change the root password to something a little less default. You should make this password very difficult to guess but then make sure you keep a record of this. You will rarely ever need to log in using this root account.
passwd
Now log out of the root account and log in using your new user account.
Next we will configure the I2C interface so that you can attach sensors that make use of it. By the end of this section you should have a screen that displays the computer host name and ip address whenever you power it on. We will also explore how to capture data from an I2C enabled sensor.
Start by running the raspi-config command. This time access the interface options and choose Enable/Disable automatic loading of I2C kernel module to enable I2C. After exiting the tool you need to reboot. We are now no longer logged in as root but rebooting requires root permissions. Since we are part of the sudoers group we can run a command with root privileges by prefixing it with the sudo keyword.
PRO TIP: use the raspi-config tool in non-interactive mode to enable I2C:
sudo raspi-config nonint do_i2c 0
sudo reboot
When it has rebooted, log in again using your normal user account.
By default, the I2C bus is only accessible if you use root permissions. The problem is that we don't want to run all our python scripts as root! To fix this you need to edit the /lib/udev/rules.d/60-i2c-tools.rules file using the nano text editor.
sudo nano /lib/udev/rules.d/60-i2c-tools.rules
Notice that we have to edit the file with root privileges. The Raspberry Pi includes two I2C buses, labelled i2c-0 and i2c-1, We will be using bus 1. Replace the existing contents with the following:
KERNEL=="i2c-0" , GROUP="i2c", MODE="0660"
KERNEL=="i2c-[1-9]*", GROUP="i2c", MODE="0666"
This splits this so that the permissions are retained for interface 0 but every user can access all the others (bus 1).
By default the I2C bus runs at a slow 100Kb/s speed however the Raspberry Pi has a "fast mode" (400Kb/s) driver. To enable this we need to edit the /boot/config.txt file using root privileges:
sudo nano /boot/config.txt
modify the following line:
dtparam=i2c_arm=on,i2c_arm_baudrate=400000
Having made our changes we need to restart the server.
There is a command called i2cdetect which scans an I2C but looking for attached devices. We will be using I2C bus 1 so we run the following command:
$ sudo i2cdetect -y 1
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
70: -- -- -- -- -- -- -- --
As you can see, no devices are detected.
Now lets check the I2C bus is configured correctly by wiring up a small OLED screen. You should find one in your electronics kit. Notice that it has 4 pins labelled GND, VCC, SCK, SDA. These need to be connected to the matching pins on the Raspberry Pi header. Below you can see the pins labelled. This labelling applies to both the RPi v2 and v3. If you look carefully at the diagram you can see that the pins connected to the I2C-1 bus are clearly labelled:
- Pin 6 is labelled
GND, this should be connected to theGNDpin on the screen. - Pin 4 is labelled
5V PWR, this should be connected to theVCCpin. - Pin 5 is labelled
I2C SCL, this should be connected to theSCKpin. - Pin 3 is labelled
I2C1 SDA, this should be connected to theSDApin.
If we run the i2cdetect command again we should see that the screen is detected and is using I2C address 0x3c.
$ sudo i2cdetect -y 1
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- 3c -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
70: -- -- -- -- -- -- -- --
This demonstrates both that the i2c bus is configured correctly and that the screen is working.
Finally we can create a simple python script to test out the screen. We first need to install a couple of packages and then some libraries. The first is the Adafruit SSD1306 library that is needed to control the screen.
sudo apt install -y python-pip python-imaging python-dev python-smbus
git clone https://github.com/adafruit/Adafruit_Python_SSD1306.git
cd Adafruit_Python_SSD1306
sudo python setup.py install
The second is the GPIO library that needs to be installed using the pip package tool:
sudo pip install RPi.GPIO
Create a new script in your home directory on the Raspberry Pi and call it oled.py. Open it in the nano editor and paste in the contents of oled.py which you can find in the exercises/01_rpi/ directory, save and exit from nano.
At the moment the script is not executable so, before running it we need to set the execution flag on the file:
chmod +x oled.py
./oled.py
If you look at the screen you should see a message...
The final task in this section is to create a script that displays the hostname and ip address on the screen when the raspberry pi first boots. The first task is to create a script in the /bin/ directory.
sudo nano /bin/network.py
Copy in the contents of the network.py script, save and quit nano. Now you need to make the file executable.
sudo chmod +x /bin/network.py
If we want the script to run automatically we need to add it as a cron task. We do this by running the crontab -e command, this opens the crontab file. Add the following to the end:
@reboot python /bin/network.py &
This tells the crontab to run the script when the device boots. The & character runs the script in the background. Try rebooting the server, you should see the information displayed as soon as the reboot is complete.
Up to this point we have connected a monitor and keyboard to our server and used this to interact with the system. Most servers run headless and we connect remotely using secure shell (ssh). To do this we need to know the hostname or the ip address of the system. Thanks to the previous step, every time we boot the server both these pieces of information are displayed on the attached OLED screen!
Start by disconnecting the screen and keyboard and reconnecting any cables you removed in the first place...
If you are using a Mac or Linux laptop you connect using the ssh command through the terminal:
$ ssh userx@elephant
You will be prompted for your password and, the first time you connect, the server will attempt to send you its public encryption key, you will be asked if you want to accept this.
If the credentials are correct you are now connected to the server and all the commands you enter will be run on it. To quit and return to your computer enter the exit command.
If you are using Windows you will probably need to download the Putty application. After launch you need to enter the hostname and user credentials in the appropriate boxes.
From now onwards you should always connect using SSH (you will never need to plug in the keyboard and monitor).
Now we will install (and partially secure) a Mosquitto MQTT broker so that data can be published to it and the client can subscribe to and receive data. We will be installing this using the apt command however the version installed by default is quite old and does not support the latest MQTT features. To resolve this we will add a package repository containing the more recent version. We need the version that matches the version of Debian we have installed. In the example below, the response from the first command indicates we are running jessie so the url when we retrieve the list points to the jessie version. If you get a different result, change the URL to match.
$ cat /etc/os-release
VERSION="8 (jessie)"
$ wget http://repo.mosquitto.org/debian/mosquitto-repo.gpg.key
$ sudo apt-key add mosquitto-repo.gpg.key
OK
$ cd /etc/apt/sources.list.d/
$ sudo wget http://repo.mosquitto.org/debian/mosquitto-jessie.list
cd ~
Once this is done we can update the cached package list and install the mosquitto server and the client tools.
sudo apt update
sudo apt search mosquitto
sudo apt install mosquitto mosquitto-clients
mosquitto_pub
This last command tries to run the mosquitto publish tool with no parameters. This will show the help page. We know we have the latest version if we can see a --cafile flag listed. To see if the server is running (on its default port of 1883) we can run:
$ netstat -at
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 *:ssh *:* LISTEN
tcp 0 0 *:1883 *:* LISTEN
tcp 0 208 elephant:ssh local-comp:55951 ESTABLISHED
tcp6 0 0 [::]:ssh [::]:* LISTEN
tcp6 0 0 [::]:1883 [::]:* LISTEN <--
Notice the last line, this shows the service is running on port 1883.
Whenever you update the configuration you need to restart the service. Here are the commands to stop start and restart:
sudo /etc/init.d/mosquitto stop
sudo /etc/init.d/mosquitto start
sudo /etc/init.d/mosquitto restart
Now we have the MQTT Broker configured we need to learn how to publish messages to and subscribe to published messages.
You should be able to use the mosquitto-clients tools covered in the previous lab. Remember that the broker is currently not secure and so will run on port 1883. The hostname will be the one you assigned to the server earlier in the lab and of course you won't need to specify username, password or CA file.
The server is logging its activity to the /var/log/mosquitto/mosquitto.log file. Linux allows you to see the last entries in the log file using the tail command. To see a live, updating you can add the -f (follow) flag.
sudo tail -f /var/log/mosquitto/mosquitto.log
Run this command on the server and see what gets logged when you publish and subscribe. If you are getting any errors with the mosquitto tools you should check this log to find out the cause.
Although the broker does work it lacks any form of security. In this section we will be correcting this. Under linux, all system-related configuration files are in the /etc directory and, if you study its contents, you will find a mosquitto subdirectory. This is where we access all the config settings for the broker. In the rest of this section all files we mention are in this directory.
The first step is to add some usernames and passwords. The mosquitto server comes with the mosquitto_passwd command to allow us to add multiple user accounts. These are stored in the passwd text file. The file is only read-only for non-root users. Use the following command to add a new user, substituting the chosen username and password where shown.
sudo touch /etc/mosquitto/passwd
sudo mosquitto_passwd -b /etc/mosquitto/passwd username password
You will also need to modify your config file to prevent anonymous use and tell it where to find the password file. Edit the mosquitto.conf file and add the following two lines:
allow_anonymous false
password_file /etc/mosquitto/passwd
Once you have added users you need to restart the broker for the changes to take effect.
After restarting the server you will need to supply a valid username and password when you connect. Try publishing and subscribing to check this is true.
You have probably noticed that even with multiple accounts, all of these can publish and subscribe to any topic. Since we want to introduce some privacy we need to create an access control list which defines which topics can be published to (written) or subscribed to (read). You should create a new file in the /etc/mosquitto/ directory called acl. You will need root permissions to do this.
sudo nano /etc/mosquitto/acl
You should then define the access permissions for the different accounts. In the example below we have two user accounts called house and garden. You can assign one of three permissions:
readmeans the user can subscribe to the topic.writemeans the user can publish to the topic.readwritemeans the user can both publish and subscribe to the topic.
The # character signifies a wildcard (any combination of topic segments).
user house
topic readwrite house/#
user garden
topic readwrite garden/#
user powerx
topic readwrite powerx/#
user 4009user
topic readwrite 4009user/#
topic readwrite owntracks/#
Once you have added the appropriate access permissions we need to let mosquitto know the access control list exists. Edit the /etc/mosquitto/mosquitto.conf file and add the following line:
acl_file /etc/mosquitto/acl
Restart the broker for the changes to take effect. Now try to publish and subscribe both to topics in and not in the access control list. Notice that if you try to publish or subscribe off-topic you dont get an error but the action will fail.
The final step is to implement SSL to ensure the data is encrypted as it passes between the broker and clients. Let's Encrypt is a new service offering free SSL certificates through an automated API.
wget https://github.com/owntracks/tools/raw/master/TLS/generate-CA.sh
The final step is to implement SSL to ensure data is encrypted as it passes between the broker and client.
Warning: depending on the speed of the Raspberry Pi, thie process of building OpenSSL can take several hours to complete. It is recommended that you run the installation step overnight.
OpenSSL is a robust, commercial-grade, and full-featured toolkit for the Transport Layer Security (TLS) and Secure Sockets Layer (SSL) protocols. It is also a general-purpose cryptography library. It is needed to generate the required certificates. We need to ensure we are always using the latest version and so, rather than install from a repository we will build from source.
git clone git://git.openssl.org/openssl.git
cd openssl
./config
make
make test
sudo make install
On most Linux distros, the certificates are typically stored in the /etc/ssl/ directory so we need to create this and navigate to it. Note you may need root privileges to create the directory if it does not already exist.
mkdir /etc/ssl/
cd /etc/ssl/
Now we need to create a 2048-bit key called mosq-ca.key and use this to create an X509 certificate. During the second step you will be asked a series of questions, it doesn't matter what you enter.
sudo openssl genrsa -out mosq-ca.key 2048
sudo openssl req -new -x509 -days 365 -key mosq-ca.key -out mosq-ca.crt
Next we create the private key and use this to create the Certificate Signing Request (CSR). Note you will need to enter the same information a second time.
sudo openssl genrsa -out mosq-serv.key 2048
sudo openssl req -new -key mosq-serv.key -out mosq-serv.csr
Normally this would be sent to the Certification authority that, after verifying the author identity, returns a certificate but in this example we will use a self-signed certificate and verify the certificate.
sudo openssl x509 -req -in mosq-serv.csr -CA mosq-ca.crt -CAkey mosq-ca.key -CAcreateserial -out mosq-serv.crt -days 365 -sha256
sudo openssl x509 -in mosq-serv.crt -noout -text
Now the certificates are ready we have to configure MQTT Mosquitto Server so that it can use these certificates by editing the mosquitto.conf file and adding the following to the end.
listener 8883
cafile /etc/ssl/mosq-ca.crt
certfile /etc/ssl/mosq-serv.crt
keyfile /etc/ssl/mosq-serv.key
Restarting the broker will put the changes into effect, you should see that this is now running over port 8883. If you want to connect to the broker now you will need to download the mosq-ca.crt certificate and use this when making connections to the broker.
One way to download the certificate is to log out of the server and use the rsync tool to copy the file to your computer using the ssh connection. For example the following command copies the file to the ~/Documents directory on the local workstation:
rsync -avzhe ssh root@elephant:/etc/ssl/mosq-ca.crt ~/Documents/
The server is now configured as an MQTT broker however if you plan to use a browser to connect to it you will need to enable MQTT over Websocket_. You will need to enable this in your mosquitto.conf configuration file by adding the following:
listener 9001
protocol websockets
Rather than restarting the broker this time we will tell it to use our updated config file. Run the command:
$ mosquitto -c /etc/mosquitto/mosquitto.conf
opening websockets listen socket on port 9001
opening ipv4 listen socket on port 8883
opening ipv6 listen socket on port 8883
When you restart the broker you should see that it is now running a websocket connection over port 9001 as well as the standard MQTT connection on port 8883.
https://www.thepolyglotdeveloper.com/2016/06/connect-raspberry-pi-zero-usb-cable-ssh/
https://dzone.com/articles/mqtt-security-securing-a-mosquitto-server