Docker and Ansible (Again)

So you want to provision docker with ansible, and you want to invoke ansible from the docker host (not from inside the container). Nice, the the number one result searching for ansible/docker just gives you a "request more info" button! Welcome to the dark side of hype.

Of course there are other search results but many are outdated, for old ansible, old docker, or both. Did you maybe find five different blog posts where it looks like each writer is just copy/pasting more or less verbatim what is written elsewhere?

Generally speaking, despite what superficially looks like a lot of information and a lot of people already doing this, it was very frustrating to actually get this working.

There's the docker connection driver which works great for simple things, but inevitably breaks down for large playbooks that are using many modules. The docker module is officially deprecated. The docker_container module is more like something that does all the things you'd do with docker-compose, and doesn't help to provision containers.

My own approach will probably be quickly out of date or maybe even produce mysterious errors for someone else as I write this, but what the hell? They work for me.. at least today. Let's throw more gasoline on the fire !

For what it's worth it terms of future-proofing this writeup, my version information from docker info is here:

Client: Version: 1.11.2 API version: 1.23 Go version: go1.5.4 Git commit: b9f10c9 Built: Wed Jun 1 21:47:50 2016 OS/Arch: linux/amd64

Server: Version: 1.11.2 API version: 1.23 Go version: go1.5.4 Git commit: b9f10c9 Built: Wed Jun 1 21:47:50 2016 OS/Arch: linux/amd64

Docker+SSH: the horror

Yeah, yeah, yeah. Docker+ssh considered harmful. I disagree, at least for a great many use-cases, but can't be bothered to articulate all the arguments. Other people have already been there and done that more expertly than I can manage anyway and at great length.

I really have nothing to add to that argument conversation, but a few things to reiterate. Most importantly, I want my ansible stuff to JustWork™ regardless of whether it's pointed at vagrant, AWS, or docker. I'd prefer to avoid complicating the recipes or having tons of extra dependencies to accomplish this. Dockerfile's get annoying pretty quickly compared with a proper CM language. Golden images have their own problems, but the alternative is often worse.

The approach covered in this write-up uses phusion's baseimage-docker. Phusion's stuff is actually really well documented, but leaves out steps since it already assumes some experience with docker, and it doesn't cover the ansible angle. The docker-ssh approach initially looked very promising but it doesn't work with scp.

Getting started

Dockerfile

Create a new folder to work in, and edit Dockerfile to look something like what you see below.

Useful links: the changelog describes other baseimage versions, and this section describes insecure-ssh-key related things.

# See https://github.com/phusion/baseimage-docker/blob/master/Changelog.md for
# a list of version numbers.
FROM phusion/baseimage:0.9.19

# Use baseimage-docker's init system.
CMD ["/sbin/my_init"]

# Ansible will require python on the container
RUN apt-get update
RUN apt-get install -y python2.7

# Enable ssh with insecure key permanently
RUN rm -f /etc/service/sshd/down
RUN /usr/sbin/enable_insecure_key

# Clean up APT when done.
RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

Boondoggles

From now on, just for the sake of choosing names I bet you don't already have inside your systems, we're going to the word boondoggle everywhere that we can. More specificially boondoggle/iX will be used to refer to an image, and boondoggle-cX to refer to containers derived from the image.

Build image

Inside the folder with your Dockerfile, construct the base image:

$ docker build -t "boondoggle/i1" .

Instantiate container

Instantiate the boondogglei1 image into the boondoggle-c1 container in such a way that it's ready and waiting with the ssh server. This container will continue to run in the foreground just in case you're interested in seeing the logs.

$ docker run --name boondoggle-c1 -t -i boondoggle/i1 /sbin/my_init --enable-insecure-key

Download Key

In another terminal, change into your Dockerfile directory and download the insecure private key from phusion:

$ curl -o insecure_key -fSL https://github.com/phusion/baseimage-docker/raw/master/image/services/sshd/keys/insecure_key
$ chmod 600 insecure_key

Bash Helpers

Disclaimer

At this point, things get very bash'y which I kind of hate. The trouble is there are a lot of long commands that are hard to remember and hard to type. These commands can also be based on variables that are potentially changing (like the container IP address). If you leave something out, you're going to get mysterious errors. You also might need to execute these commands repeatedly as you test things, so we're going to make some bash-functions to use as helpers. You might want use set -x in your console to enable some bash debugging information.

Finding the container IP

First, make a bash function "refresh_cip" to grab the IP address for the named container. This address will be written to the `$CIP env-var for use by other functions, and will be used to update the ansible inventory file (the ansible.hosts file will be written/refreshed in your working directory).

# "CNAME" stores the container name.  IMPORTANT:
# the refresh_cip" function MUST have this variable available
$ CNAME=boondoggle-c1

# Bash function definition for "refresh_cip"
$ refresh_cip(){
  export CIP="`docker inspect -f \"{{.NetworkSettings.IPAddress }}\" $CNAME`";
  echo "docker ansible_host=$CIP ansible_python_interpreter=/usr/bin/python2.7" > ansible.hosts;
  }

# Test it / Example usage
$ refresh_cip
$ echo $CIP
$ cat ansible.hosts

Connecting to the container

Make a bash function "run_ssh" to help with ssh'ing into your container. Also works to execute individual commands.

# creates the "run_ssh" function
# Bash function definition for "run_ssh"
$ run_ssh(){
  refresh_cip; ssh -o StrictHostKeyChecking=no \
    -o UserKnownHostsFile=/dev/null \
    -i ./insecure_key \
    root@$CIP "$@";
  }

# Test it / Example usage:
# display "root" or drop to shell
$ run_ssh whoami
$ run_ssh

Running ansible

Create a trivial ansible playbook ping.yml, just to test with. Phusion's baseimage is ubuntu-based, so using ansible's apt module is appropriate for installing system-level packages. Uncomment the appropriate section below if you want to try it.

# ping.yml
- hosts: docker
  remote_user: root
  gather_facts: no
  tasks:
    - name: test connection
      ping:
      remote_user: root
    - name: Install system package
      apt: pkg=tree state=installed

I assume you already have installed ansible? Great. Now make a bash function "run_ansible" to help with running ansible (). This function requires at least one argument (the playbook to use). You can pass additional ansible arguments if you like as well but ansible will complain if you pass any of the ones already mentioned below.

# Bash function definition for "run_ansible"
# remove or reduce verbosity with the "-vvv" part
$ run_ansible(){
  refresh_cip;
  export ANSIBLE_HOST_KEY_CHECKING=False
  ansible-playbook -vvv \
    --user=root --private-key=./insecure_key \
    --inventory=ansible.hosts $@;
  }

# Test it / Example usage
$ run_ansible ./ping.yml

Saving changes

After you're done testing and you're sure the playbook is working well, you probably want to start over from scratch with a clean image just to be sure. You can run everything again to test it, and save your work if things went smoothly. Hopefully you haven't been modifying the Dockerfile much since now you can change things with ansible, but just in case.

# Set vars for contain/image names
$ export CNAME=boondoggle-c1; export INAME=boondoggle/i1

# Remove the boondoggle container instance, even if it's running
$ docker rm --force $CNAME

# Remove the boondoggle image iself, rebuilding from scratch.
$ docker rmi --force $INAME
$ docker build -t "$INAME" .

# Run the container in the background this time, cuz why not
$ docker run --detach --name $CNAME -t -i $INAME /sbin/my_init --enable-insecure-key

# This will succeed because the container is listening with ssh
$ run_ssh ls /usr

# This should fail because we didn't run ansible yet
$ run_ssh tree /usr

# provision to fix it
$ run_ansible ping.yml
$ run_ssh tree /usr

# Save the ansible'ized stuff to the derivative image "boondoggle/i2"
$ docker commit boondoggle-c1 boondoggle/i2
$ docker stop boondoggle-c1

# Bootstrap the 2nd container from the derivative image.
# Since ansible has already been run there, the tree command is available.
$ export CNAME=boondoggle-c2; export INAME=boondoggle/i2
$ docker run --rm --name $CNAME -t -i $INAME tree /usr

Note that the last "docker run" invocation above differs in some important ways from the previous ones. Before when we were using ansible, used a phusion-baseimage "/sbin/my_init" style call. Because we did NOT use that here, ssh is not available1. Also here, the "--rm" option ensures a cleanup of the container filesystem after exit.

Cleanup

Just in case you don't want your system cluttered with boondoggles.

# Cleanup the original image and container
$ docker rmi --force boondoggle/i1
$ docker rmi --force boondoggle/i2
$ docker rm --force boondoggle-c1
$ docker rm --force boondoggle-c2
  1. ..so the docker filesystem might be fat, but it's not any more insecure than usual