Deployment and virtualization, practice session 2

1. Introduction

This second session about Docker will teach you how to write Dockerfiles and build images for a couple of use cases. The next session will give you more freedom about dockerizing applications and you will have to reuse what you will learn during this session.

All actions you need to perform are indicated by the following pictogram:

Make sure you read and understand all the material of this document, and go through all the required actions.

Also, be sure to keep notes about what you are doing, as you will have to answer graded questions about your work during the last 15 minutes of the session.

2. Use case 1: Distribute a Go program

The Go language creators did a great job a providing their community with easy-to-use tools thanks to Docker. Studying how things are done with Go is a good example of how to leverage Docker for software distribution.

The tools provided using Docker cover two major aspects of software development:

1. The creation of redistributable software artifacts: compile, link, package, etc. For this purpose we need a particular software stack which contains compilers, static analysis tools, linkers, cross-build toolchains, and so on…
2. The execution of the piece of software previously created over a runtime software stack. Ideally, such software stack should be well-identified and minimized to reduce as much as possible the surface for attack or dependency issues, and also to facilitate maintenance.

Why not installing and using a toolchain directly from our distribution? Here is a great answer from Docker’s blog:

If you write Go code, or if you have even the slightest interest into the Go language, you certainly have the Go compiler and toolchain installed, so you might be wondering “what’s the point?”; but there are a few scenarios where you want to compile Go without installing Go.

• You still have this old Go 1.2 on your machine (that you can’t or won’t upgrade), and you have to work on this codebase that requires a newer version of the toolchain.
• You want to play with cross compilation features of Go 1.5 (for instance, to make sure that you can create OS X binaries from a Linux system).
• You want to have multiple versions of Go side-by-side, but don’t want to completely litter your system.
• You want to be 100% sure that your project and all its dependencies download, build, and run fine on a clean system.

If any of this is relevant to you, then Docker should be useful!

In what follows, we will

1. Build a simple Go program using different techniques;
2. Distribute this program and the software stack it relies on using a Dockerfile.
3. Optimize a bit the size of our image to facilitate its distribution.

2.1. Build a Go program using tools from a Docker image

The very simple program we are going to build and run, and the associated resources, are packaged in the resources.tar.gz archive. It is composed of a single file named “simple_static_server.go” which should serve the static files under the “resources/static” directory.

We will build and run this program in various ways.

2.1.1. Identify a good base image

Let us first identify the right Docker image to build our program. What we need is a reliable image which contains build tools for the Go language.

2.1.2. Build your program using the image directly

Using the golang image variant you selected, we can call the compilation tools directly on our files thanks a bind mount of the current directory.

docker run --rm -v "$PWD":/usr/src/myapp -w /usr/src/myapp golang:1.8 go build -v

You should obtain a myapp file under your current directory.

2.2. Prepare a runtime image

We will now focus on building an image to distribute our program and run it in isolation.

2.2.1. Turn-key solution

The easiest solution (but not the best one) is to use the image used to build as a base for our new image.

2.2.2. Use a smaller base image

According to Docker’s documentation about best practices (and common sense), it is not desirable to ship large images containing a lot of useless elements.

In our particular case, we do not want to ship a Debian distribution with our tiny server (assuming you used the default “latest” image).

We are now going to use a smaller image.

We do not need all the build tools in any of the golang image variants!

2.2.3. Use a really small base image

The right approach is to use a minimal base image, like an alpine image, to run our program. In our case, because our program does not have complicated dependencies, we can simply copy it to a new linux image.

2.2.4. Use a multistage build

It is possible to merge the two steps of the previous section using a multistage build. This consists in creating several images in the same Dockerfile. In our case, we need only two images:

1. a build image (exactly what we did before);
2. a runtime image (which contains only the bare minimum) like in the previous section.

The official documentation details this tricks which relies on two simple things:

1. naming images with the syntax: FROM image as name;
2. being able to copy from any named image using the following syntax: COPY --from=name /path/in/name/ /path/in/current

2.2.5. Use the smallest possible image

Go is particularly useful to produce minimal binaries thanks to static compilation.

This allows us to build a program which does not depend on anything but the kernel, except for a few files if we use, for instance, HTTPS which requires some certificates.

Go static compilation can be triggered by using:

go build -ldflags "-linkmode external -extldflags -static" -v

3. Use case 2: Create a Conda-like environment

Conda is an open source package management system and environment management system. “Package management” is about solving the dependency problem of package installation: What should I install to make this program work? “Environment management” is about enabling to separate separate independent and potentially conflicting software stacks.

Conda was first designed for Python language but it can now handle binaries and libraries from various languages. It can also run in Linux, OSX and Windows.

Conda addresses the limitations of pure-Python tools for package management (easy_install, pip, etc.) and environment management (virtualenv): those tools fail at properly managing library dependencies for Python packages. It is indeed common to face compilation issues because pip triggered a compilation of some C code which relies on a library for which the development header were not installed. This happens when you want to install NumPy, for instance, in a very small base Docker image: the headers for linear algebra libraries are not installed.

Conda’s solution is to rely on a massive repository of pre-built packages, maintained by the Anaconda company. Each Conda package can contain:

• system-level libraries
• Python or other modules
• executable programs and other components
• some metadata.

This really looks like a Docker image layer!

We are going to simulate the kind of environment Conda produces using Docker containers. This will allow us to share some parts of the environments we will create, and have the full control over our software stack.

Of course, this is a bit more complex than using Conda’s tools, but not that much in practice and it provides a few advantages:

• once an environment is defined using a Dockerfile, it is very easy to replicate it;
• we can limit the resources used by a container.

In what follows, we will setup a Python environment for scientific computing. Before and while writing a Dockerfile, we encourage you to test your commands in a container based on the image you want to use.

3.1. Choose a base image

Like in the previous case, identifying the right base image is a critical choice. The Docker Hub references a lot of base image to build on.

For what follows, we recommend using an Ubuntu base image or the official Python image, as their support is very good and their reasonable. You may, however, use another image you more confident with. What follows assume you use an Ubuntu base, because it contains extra necessary steps.

3.2. Install Python and pip

3.3. Locale setup

python -c 'print("h\xe9h\xe9")'

So not long ago, the ubuntu terminal is in POSIX locale (20th century is calling…), accepting only ASCII characters both for input and output.

If the previous command failed, you need to fix your terminal: set the LANG environment variable to a saner C.UTF-8 default.
You can check the language defaults using the locale command.

# Shell
export LANG=C.UTF-8
# Dockerfile
ENV LANG=C.UTF-8

3.4. Install NumPy, IPython and Jupyter

We are now ready to install Python tools.

3.5. Add a special user (not needed with podman)

This is not needed with podman (or docker in rootless mode). The root user inside podman is automatically mapped to the current user.

We will do a little trick here. To avoid several issues (programs complaining when run as root, messing file permissions, etc.) we will add a new user in the container. We want this user to have the same UID and GID as our current (host) user, so we can share files easily with her, but we want this user to have a different name and a different home directory, to be able to store container-specific configuration, avoid avoid messing our own configuration on the host.

3.6. Jupyter configuration and expose the ports

To avoid authentication issues with Jupyter, we will remove the security token required. This is a security issue, and you should put a custom string instead of a empty one in production.

RUN jupyter notebook --generate-config && \
  echo "c.NotebookApp.token = ''" >> $HOME/.jupyter/jupyter_notebook_config.py

3.7. Setup a volume for specific configuration

To avoid loosing data related to container configuration, we can declare a volume for the home directory of the new container user. This will create an external storage for this part of the filesystem, with the current content of the container at this time.

3.8. Setup the entry point and/or the default command

jupyter notebook --no-browser --ip=0.0.0.0 --port $JUPYTER_PORT

3.9. Build the image

3.10. Run the container

We will now run a container based on our new image.

3.11. Resource limitation

3.12. New tools in new image layers for alternative environments