---
title: Now With Ansible!
description: "Now's probably a good time to do things properly and stop depending on a snowflake server"
publishedDate: 2022-07-10
tags:
  - sysadmin
  - ansible
---
For nearly four years, this website has been running on the lowest tier of [DigitalOcean][digitalocean] Droplet, initially for free with some credit that I got courtesy of [GitHub Education][gh-edu]. During that time, I operated two individual servers,<sup id="marker-1">[1](#footnote-1)</sup> both of which were setup entirely by hand.

The first server only contained native apps, but I moved to using Docker containers on the second server. In either case, re-deploying or updating software involved me SSHing into the server, running `git pull` and manually rebuilding everything. Some of the server configuration files were stored alongside the code for the website, some were untracked and being edited on-the-fly and I'm sure there was some stuff I'd forgotten about but was quietly ticking away in the background. That's a lot of words to say that I had a [snowflake server][fowler-snowflake-server]!

Fast-forward to last month, when [DigitalOcean announced that they were going to be hiking their prices][do-price-hike], effective July 2022. This meant that I'd end up spending an extra $1 a month for the same spec VPS, which is quite a lot for me and my non-existent income. Because of this, I decided to move to [Hetzner][hetzner], which was better value than DigitalOcean, even before the price hike, which is why it was something I'd been planning to do already, this just gave the the motivation to actually do it.

I also took this as a chance to set the server up the proper way,<sup id="marker-2">[2](#footnote-2)</sup> so I could spin up a new copy of it in minutes if I need to, instead of the hours like before. I did this with a tool called [Ansible][ansible], which is maintained by Red Hat. [Jeff Geerling][jeff-geerling] is quite the proponent of this, and it was [his Ansible 101][geerling-ansible-101] series that first got me interested in the tool.

# Ansible

I think Ansible is really cool. At its most basic level, you use it to run commands against a remote machine over SSH. It doesn't require any special software to be installed on any of the remotes you're working with - just an installation of Python. The only software you need to install is installed on your local machine.

"Playbooks" are the Ansible name for the way you define the ideal state of a remote. They're written in YAML, and leverage a number of built-in and third-party "modules" which each do certain things. Examples of this are managing Cron jobs, editing files and dealing with Docker.

At its most basic level, a playbook contains a list of idempotent tasks that each invoke a module. I ended up writing two playbooks, one called `provision.yml` and one called `update.yml`. The beginning of the former looks like this:

```yaml
---
- name: Provision and setup an Ubuntu amd64 webserver
  become: yes
  hosts: webserver
  tasks:

    - name: Update apt repositories
      tags: [tools]
      ansible.builtin.apt:
        update_cache: yes
        cache_valid_time: 3600

    - name: Install tools
      tags: [tools]
      ansible.builtin.apt:
        name:
          - git
          - curl
          - sqlite3
          - python3-pip
        state: present

    - name: Add SSH login alert script
      tags: [sshAlerts]
      ansible.builtin.copy:
        dest: /usr/bin/on-ssh-login.sh
        src: on-ssh-login.sh
        mode: +x
    # ...
```

This is pretty simple! It was a breeze getting things set up and working in a usable state with some guesswork. All I had to do was write a long list of tasks, whack them in a playbook and add some [tags][ansible-tag-docs] so I can selectively run certain sections of it.

To run a playbook in its entirety, all I need to do is to run `ansible-playbook provision.yml`. Only running, say, the steps to install Docker and Caddy is similarly simple - `ansible-playbook provision.yml --tags "docker,caddy"`.

And, in all fairness - those playbooks did work like that! There were, however, a couple of shortcomings.

The first was that any files that a given playbook used had to be shoved in a single directory to keep some semblance of organisation. When you have a collection of files, many of which are called things like `config.json` or `settings.yaml`, it beceoms hard to keep track of what's what. To get around that, I took the extraordinarily janky approach of prepending the service name to the filename, if it wasn't already there.

```
server
├── provisionAssets
│   ├── backup.sh
│   └── on-ssh-login.sh
├── provision.yml
├── updateAssets
│   ├── Caddyfile
│   ├── walrss.config.yaml
│   └── website.env
└── update.yml
```

Second was that it became a little tricky to deal with the outputs of each task. In a playbook, you can optionally "register" the output of a task as a variable with a name. When you have all your tasks in a big file together, you can end up with name collisions, which isn't great.

Luckily, Ansible has a solution for this!

# Enter roles

I think the closest comparison to a role in Ansible is a function in a programming language. Each role has its own variable scope, its own collection of files it uses and you can set variables within it, all scoped to that role. "Role" seems like an awful name for this functionality, in all honesty, but what do I know?

Using roles allowed me to split my long playbooks up into a single role for each step. I can have a role called `setupDocker` or `configureCaddy` or the name of a service, and it'll setup and install Docker, apply a [Caddy][caddy] configuration or update and (re)start a service, respectively.

Let's say you wanted to create a new role called `bookstack` that installs or updates an installation of [Bookstack][bookstack]. Since each individual role lives in its own directory inside the `./roles` directory (relative to the playbook), it's a matter of creating the appropriate directory structure, adding files in the right place and including the role in your playbooks.

```
$ mkdir -p roles/bookstack/tasks
$ mkdir roles/bookstack/files
$ nano roles/bookstack/tasks/main.yml # add your playbook steps here
$ cp ~/bookstack.config.php roles/bookstack/files
```

```yaml
# Within a playbook's tasks section:
- name: Install/update Bookstack
  tags: [bookstack]
  ansible.builtin.include_role:
    name: bookstack
    apply: { tags: [bookstack] }
```

(The `apply.tags` applies the given tags to all the role's child tasks so you can still use tags to selectively run parts of a playbook.)

If you're interested in roles as a thing, you can take at look at [the Ansible docs for them][ansible-role-docs]. One aspect of roles that I didn't cover here are variables that can be used as inputs to your role - most of the roles I wrote made use of these variables in some shape or form.

# Some tricky things

While, on the whole, writing my playbooks was pretty simple and only took the best part of a day, that's not to say it was all plain sailing.

The first issue was that, if Ansible detects that a given task has failed, usually by exit code, it'll stop executing the playbook at that point. This is an issue if, say, one of the install scripts you use returns a non-zero exit code if a tool is already installed, causing it to crash your playbook. If this is the case for 95% of the times you run the playbook, that's a problem.

This is why Ansible lets you override the default "not 0 means failure" behaviour by inserting a conditional expression into a field called `failed_when` on your task.

```yaml
 - name: Install rclone
   ansible.builtin.command:
     argv:
       - "bash"
       - "-c"
       - "curl https://rclone.org/install.sh | bash"
   register: command_result
   failed_when: command_result.rc != 3 # 3 means rclone is up-to-date
```
 
One of the other issues I encountered was more subtle, and only showed itself when I tried to update my website's Docker container.
 
Normally, when you run `docker pull ghcr.io/codemicro/walrss:latest`, for example, Docker will download any updated versions of that image. The issue comes from the fact that Ansible's `community.docker.docker_image` will only check for the presence (or lack thereof) of an image, not to see if there's an update. The easiest way to fix this is to just add `force_source` to the task to make it re-pull the image regardless every single time.

```yaml
 - name: Pull Walrss Docker container
   community.docker.docker_image:
     name: ghcr.io/codemicro/walrss:latest
     source: pull
     force_source: yes
```

It's an incredibly simple thing, but it's one of those issues that takes up far more time to fix than it feels like it should do.
 
Another issue with deploying updates to Docker containers with Ansible like this is that if you already have a container running with a given image, any changes to the container's image won't take effect until the container is restarted.

You could just restart the container every single time the playbook is run, which is *fine*, I guess, but it'll mark that task as changed every single timeun - even if the underlying container image never changed. That annoys me irrationally, so we can configure Ansible to only restart the container when it doesn't already exist or if the image changed.

```yaml
    - name: Pull Walrss Docker image
      community.docker.docker_image:
        name: ghcr.io/codemicro/walrss:latest
        source: pull
        force_source: yes
      register: imagePull

    - name: Check Walrss container state
      community.docker.docker_container_info:
        name: walrss
      register: containerInfo

    - name: Ensure Walrss is running
      tags: [walrss]
      community.docker.docker_container:
        state: started
        recreate: yes
        # other container settings here
      when: imagePull.changed or (not containerInfo.exists)
```

# To conclude...

Ansible is cool. It's fairly boring technology, which is also cool. It might be old, but it's easy to use and get started with (if you ignore how the docs kinda suck sometimes). I'll definitely be continuing to use it in the future.

I very much enjoyed putting my two playbooks togeter, and I look forwards to writing more in the future, because it makes repeatable deployments a breeze.

# Footnotes

1. <span id="footnote-1">There came a point where I wanted to update the first Droplet to a newer version of Ubuntu, but I was scared doing so might break something catastrophically, so I "just" rebuilt it on a new Droplet.</span> [↝](#marker-1)
2. <span id="footnote-2">When I say "proper", I mean it in the sense that setting up a server from scratch should be a non-event.</span> [↝](#marker-2)

[hetzner]: https://www.hetzner.com
[digitalocean]: https://digitalocean.com
[do-price-hike]: https://www.digitalocean.com/try/new-pricing

[gh-edu]: https://education.github.com/

[jeff-geerling]: https://www.jeffgeerling.com/
[geerling-ansible-101]: https://www.jeffgeerling.com/project/ansible-101-youtube-series

[ansible]: https://www.ansible.com/
[ansible-tag-docs]: https://docs.ansible.com/ansible/latest/user_guide/playbooks_tags.html
[ansible-role-docs]: https://docs.ansible.com/ansible/latest/user_guide/playbooks_reuse_roles.html

[fowler-snowflake-server]: https://martinfowler.com/bliki/SnowflakeServer.html

[caddy]: https://caddyserver.com
[bookstack]: https://www.bookstackapp.com/