Docker Guide and QRC

In this section we will have a quick run through of the core docker commands.

Docker has some official docs on this too.

What is Docker

Docker is a containerisation technology. It allows us to create sandboxed processes, (with their own filesystems) on our computers. The processes run on-top of our native filesystem, and make use of its core functionality but give the illusion of being a separate system

These sandboxed processes contain everything that is needed to run the application.

We misuse docker to create lightweight VMs with examples for you to play with. It means that I can distribute tasks in an easy way, as everyone should have exactly the same setup. Additionally, as the images make use of existing OS functionality, they are pretty lightweight. Linux distros, on the other hand are getting fatter (you are looking at around 2GB minimal installs now)

This means you only need to grab a few hundred MB of docker image, rather than a whole new VM each time.

Note

While docker is pretty amazing, there are a few things it cant do.
As we share the same OS, things like kernel-level exploits are hard to do.
Additionally, networking functionality (so stuff like firewalls), is also shared with the host, so becomes difficult.

However, for about 90% of what we need, it's going to be more efficient.

Terminology

Image: A Docker image is the sandbox that contains our filesystem and programs. I like to think of them as the equivalent of an ISO file
Container: A container is a running instance of an image. We can have multiple containers, each based on the same image, which can let us scale applications (although this isn't really applicable here).
Stack: A group of containers running together to serve a purpose. This is a really nifty thing as it can let us plumb together groups of containers and make them appear like a single VM. For example, we can combine images for a Webserver, and database into one seamless system.

Docker-Compose

While we can use low level docker commands (see below) to start and stop containers, or manage the running docker processes. I intend to make use of docker-compose.

Docker-Compose is a system that allows us to specify a "recipe" for running docker containers. For example, specifying which images to start, what port they should listen on and other information. It also allows us to group images together into a stack.

Docker compose makes use of compose files docker-compose.yml that contain the instructions for building the stack.

Multiple Compose Files

If we need to deal with multiple compose files, you have two options

Call them different names and tell compose what one to run using the -f flag
Put them in different directories

I prefer the second approach. Compose will also build its images based on the dir you are working in, so it simplifies image management.

Starting and stopping compose-based services

To start a compose based service docker-compose up

dang@DESKTOP-KJDVQ2J:~/6005/6005-CW-Koala$ docker-compose up
Starting 6005-cw-koala_pyramid_1 ... done
Attaching to 6005-cw-koala_pyramid_1

It's not hung

Docker will continue to run in the background, so it will look like the system has hung. (In the case of some images it may display logging information)

You will need a new window to interact with the system.

To Stop a compose based service you need to do two things.

If docker is running in the foreground, stop it with ctrl-c
docker-compose down

# Stop the Container Running
^CGracefully stopping... (press Ctrl+C again to force)
Stopping webtrainer_database_1 ... done
Stopping webtrainer_web_1      ... done

# Remove from Container List
dang@DESKTOP-KJDVQ2J:~/6005/webtrainer$ docker-compose down
Removing webtrainer_database_1 ... done
Removing webtrainer_web_1      ... done
Removing network webtrainer_main_network

Low-Level Docker access.

If we are not using compose we also have the option to deal with docker directly.

Starting and Stopping containers

Container Management

You may also want to examine the currently running containers

To list containers that are currently running you can use docker ps

dang@DESKTOP-KJDVQ2J:~$ docker ps -a
CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS                     PORTS                               NAMES

To list all containers (including those that are stopped) docker ps -a

dang@DESKTOP-KJDVQ2J:~$ docker ps -a
CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS                     PORTS                               NAMES
e4a8e2d38bdb        6005-cw-koala_pyramid   "pserve production.i…"   2 weeks ago         Exited (137) 2 weeks ago                                       6005-cw-koala_pyramid_1
8cefe7adba95        7024cem/webdb           "docker-entrypoint.s…"   4 weeks ago         Exited (255) 2 weeks ago   0.0.0.0:3306->3306/tcp, 33060/tcp   webtrainer_database_1
e7d7e6005693        7024cem/webtrainer      "docker-php-entrypoi…"   4 weeks ago         Exited (255) 2 weeks ago   0.0.0.0:80->80/tcp                  webtrainer_web_1
6f3bbd7104c7        7024cem/webdb           "docker-entrypoint.s…"   5 weeks ago         Exited (255) 4 weeks ago   0.0.0.0:3306->3306/tcp, 33060/tcp   6005_database_1
5fecfd61782f        7024cem/webtrainer      "docker-php-entrypoi…"   5 weeks ago         Exited (255) 4 weeks ago   0.0.0.0:80->80/tcp                  6005_web_1

To remove a stopped containers use docker rm <name>

dang@DESKTOP-KJDVQ2J:~$ docker rm 6005-cw-koala_pyramid_1
6005-cw-koala_pyramid_1

Image Management

You can also manage the images currently installed on your system

To list the current set of images use docker image list

dang@DESKTOP-KJDVQ2J:~$ docker image list
REPOSITORY                TAG                 IMAGE ID            CREATED             SIZE
6005-cw-koala_pyramid     latest              afc71342b28f        4 weeks ago         946MB
python                    3                   5336a27a9b1f        5 weeks ago         886MB
cueh/ftpanon              latest              e68af1bf98d7        2 months ago        148MB
cueh/scanningweb          latest              4d4adf600a0d        2 months ago        413MB
cueh/typewriter_postfix   latest              7c2fe5b9677a        3 months ago        161MB
7024cem/webtrainer        latest              ee9ee758bfbe        18 months ago       382MB
7024cem/webdb             latest              9b235fdedb19        18 months ago       376MB
cueh/debian_ssh           latest              80f6116013fe        19 months ago       131MB

To remove an image: docker rmi <image name>

dang@DESKTOP-KJDVQ2J:~$ docker rmi 6005-cw-koala_pyramid:latest
Untagged: 6005-cw-koala_pyramid:latest
Deleted: sha256:afc71342b28feb67e1e7a41a07a5450b432926f46daa6147b8eae98cb5771876
Deleted: sha256:0311c5037503db900b04073069146141c186166a24b4861a26ec88bc750750e3
Deleted: sha256:d538d349d16647f86d475cb1f52cf930d804f87e642c49a1bb846335af73ca84

There is also a command that will prune image layers that are not used by anything else

docker image prune

FAQ

You are asking me to install a lot of images, what about my HDD space.

One of the really cool things about docker is the "layering" system. This means that images can reuse elements of other images to reduce the overall size. Unfortunately, the docker image list command reports the total size of the image, rather than taking account of any layers its reusing¹

I have tried to design the images we use to make best use of this layering approach. For example, most of the web-based challenges use the same base image, and the only thing unique to each one is the vulnerable website itself.

This means that you may have 10 web challenges each reporting as being ~200MB each, when in reality it' one "base" image of ~190MB, with a separate 10MB layer for each challenge. So instead of 2GB it's only ~300 MB of space that is actually used.

I keep getting Permission Denied Errors

Docker needs to run as the root user. Try repeating the command with sudo.

$sudo docker-compose up

I get asked if the Daemon is running

If you get a message like

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

In this case it's likely that docker is not started. You can confirm this using whatever init tools you have (for example systemctl / init.d)

#Get Status of docker
$systemctl status docker

#Start Docker
$systemctl start docker

I get a message about network in use

This can sometimes happen if we don't close docker-compose down correctly. If we just stop the containers they are kept as inactive along with any configuration that was in place. You can avoid this by using docker-compose down when you have finished a task

Another method is to use the docker-network prune command. As this will remove old network configurations from the list.

I cant access the system on WSL

Usually, the docker container is running 'on top' of the host system. This means that any services are exposed as they would be on localhost.

However, with WSL, we are running our Linux base inside a virtualised environment, so can't access at localhost.

Instead, you will need to use the address of the WSL machine that is running docker.

My laptop reports that the docker images take up more than twice the space that I actually have available. ↩