OUTPUT

The blog of Maxime Kjaer

Travis CI deployments to DigitalOcean

I really like using static site generators. I guess the computer scientist in me likes optimized systems, and that’s exactly what I get here: static sites make for a secure, performant and simple setup. It doesn’t get much more basic than serving static files with Nginx. It’s rather hard convincing myself to manage a big PHP framework and an SQL database just to show some blog posts, but I am painfully aware of how much easier to use Wordpress is for the users. Compare Wordpress’ workflow to that of a static site: even though making changes to my Jekyll site may seem rather easy to me, it really isn’t that straightforward. Here’s how I’ve done it up until this point:

  1. Write a post in Markdown
  2. Optimize images manually
  3. Commit changes to GitHub
  4. Build the site on my local machine
  5. Compress generated HTML and CSS files
  6. Manually transfer the changed files to the server

Good luck trying to convince your clients to use a static site if this is what it takes for them to do a simple task, like fixing a typo. Getting your content online requires knowledge about Markdown, compression, git, command line, and file transfer. That’s a very steep learning curve if you aren’t very technically inclined. It’s also a rather tedious process. What if we could reduce it to one or two steps?

Another problem I had with that workflow was that I would often have either undeployed or uncommitted changes. The GitHub repo sometimes wouldn’t be up to date with the content I was serving, or vice versa, because I hadn’t exactly followed all the steps. I want the code on GitHub to actually represent what I’m serving. I mean, what’s the idea of releasing the code if it doesn’t reflect the reality? This hard link between the source code and what I’m serving needed to rely on more than my goodwill.

In my continuous quest to make this site lean and mean, all of the above have led me to set up automatic deployments to my server. This means that my workflow now just consists of creating content and committing it — the rest is done automatically. Travis CI builds the site, compresses assets, runs a few tests, and deploys it all to DigitalOcean, where Nginx serves it.

Server set-up

The first step to achieving this new workflow was to have rigorous rules set up on the server. No more deployments as root, from now on we’re doing things correctly.

For the sake of simplicity, we’ll assume that our server’s live directory is /var/www/kjaer.io/public_html. Additionally, commands run on the server will be denoted by a $, while a λ will precede commands entered on our local machine; if you are on Windows, I would recommend using WSL or cmder, as all following instructions will rely on Linux command-line tools.

Create a new user with restricted access

I’ve created a new user called deploy with limited permissions: it’s only allowed to operate in /var/www/kjaer.io, the directory for this site’s content.

First, we’ll need to log in via SSH to the root account. We’ll create a new user that Travis CI can use to log in, and we’ll give it ownership of our live directory.

$ adduser deploy
$ chown -R deploy:deploy /var/www/kjaer.io

It will ask you to create a password for that user (pick something strong!), and fill out some more optional information. Just press Enter if you wish to skip some of those steps.

Set up public key authentication

For Travis CI to log in to the server, we’ll set up public key authentication. To do this, on your local machine, you should run:

λ ssh-keygen

Which should output something like this:

Generating public/private rsa key pair.
Enter file in which to save the key (~/.ssh/id_rsa):

I suggest calling it deploy-key. It will also ask for a passphrase; leave that blank. This process should leave you with two files in your .ssh folder: deploy-key, which is your private key (this is like a password, so protect it at all times!) and deploy-key.pub, your public key.

The server needs to know your public key so that you can log in using your private key. There are multiple ways of copying it, but since we have an open SSH connection, we can just copy paste it. Not the most elegant solution, but it’ll do the job just fine. To do that, we’ll first need to log in to the deploy user:

$ su - deploy

Then, we’ll create a .ssh directory in the home directory, and an authorized_keys file inside of it:

$ mkdir .ssh
$ chmod 700 .ssh
$ nano .ssh/authorized_keys

At this point, you can paste in the contents of your deploy-key.pub file. Hit Ctrl+X, Y and Enter to save and exit. Before we log out, we’ll just restrict the permissions to the authorized_keys file:

$ chmod 600 .ssh/authorized_keys
$ exit

You now have a way to log in to the server. Now, let’s set it up to receive files.

Create a remote

You most likely already have a local repository on your computer, which is where you work. A remote repository, or just “a remote”, is a place where you can keep everything in sync. GitHub, for instance, is a remote. It’s essentially just a hosted copy of your Git project. In our case, it’ll just mean a repository on our web server to which we can push.

We’ll want to push to a repository in /var/www/kjaer.io/.git, and have the files available in /var/www/kjaer.io/public_html. We’re placing the .git one level above public_html because we don’t want to serve it.

To do this, we’ll start out by creating an empty Git repo in /var/www/kjaer.io:

$ cd /var/www/kjaer.io
$ mkdir .git
$ cd .git
$ git init --bare

That’s all it really takes to have a remote that we can push to! But our repository is in /var/www/kjaer.io, and we want to have the pushed files in /var/www/kjaer.io/public_html. To achieve this, we can add what’s called a post-receive hook, which is a list of commands to run once the remote has received a push.

$ cd hooks
$ nano post-receive

Then, you’ll need to type the following in:

#!/bin/sh
git --work-tree=/var/www/kjaer.io/public_html/ --git-dir=/var/www/kjaer.io/.git checkout -f

Save and exit with Ctrl+X, Y and Enter. The post-receive hook will just need to be executable, so that it actually can do its job:

$ chmod +x post-receive

That’s it for server configuration!

Set Travis CI up

Everything should now be ready for Travis to log in to our server and push content to the repo. However, we have a slight problem: how can we give Travis CI the private key without disclosing it to the general public? If our GitHub repo were private, we could just place the key in the repo and use that directly in Travis (though I’d still recommend against that approach, because if you ever plan on open-sourcing the repo, the private key will be in the commit history), but that won’t be an option here.

Encrypt the private key

It turns out that Travis has a command-line utility to encrypt files. Using this, it’s safe to upload the key publicly to GitHub, and Travis CI can still use it. You’ll need to log in first though, because it will have to add two private environment variables to your Travis account, so that it can decrypt the encrypted private key later on (yes, we’re getting into some pretty heavy crypto setups now, but it’s the best we can do).

If you don’t have a .travis.yml file yet, you can just create one real fast. We’ll talk about what it does later on.

λ touch .travis.yml

You’ll need to copy your private key file (deploy-key) to your local repository. Then, we’ll install the Travis command line utility, log in, and encrypt the file:

λ gem install travis
λ travis login
λ travis encrypt-file deploy-key --add

The last command will create an encrypted copy of your deployment key, deploy-key.enc. It will also add a few lines to your .travis.yml, something that looks a bit like this:

1
2
before_install:
  - openssl aes-256-cbc -K $encrypted_22009518e18d_key -iv $encrypted_22009518e18d_iv -in deploy-key.enc -out deploy-key -d

This is the command that will allow you to decrypt the key on the Travis CI platform. Once Travis runs this, it will be able to access the decrypted key, which is in the deploy-key file.

You should immediately remove the unencrypted private key from your repo, so that you don’t risk uploading it to GitHub. There are bots out there that do nothing but analyze GitHub commits to find login information, and they will compromise your server in a matter of seconds if you accidentally disclose the key.

λ rm deploy-key

In fact, once you have Travis CI deployments working, you should probably delete the unencrypted private key altogether from your computer.

Create your .travis.yml file

All we need to do now is to tell Travis how to build, test and deploy the site. For that purpose, we have a file called .travis.yml, which acts as a list of instructions for Travis.

Right now, my .travis.yml file looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
language: ruby
rvm:
  - 2.4.3
env:
  global:
  - NOKOGIRI_USE_SYSTEM_LIBRARIES=true
addons:
  ssh_known_hosts: kjaer.io
before_install:
  - bash _scripts/install.sh
script:
  - bash _scripts/build.sh
  - bash _scripts/test.sh
after_success:
  - bash _scripts/deploy.sh

It defines that Ruby version 2.4.3 (a stable release compatible with the latest versions of Jekyll) should be installed on Travis, gives the encrypted key, says that kjaer.io is a known host so that I don’t get a warning when I try to connect to it, and references a few bash scripts to run at various stages of the build process. These scripts have been placed in the _scripts directory, which won’t be built by Jekyll since its name starts with an underscore. I’ve made a bash script for each task, because I think it’s more readable this way.

Install

Before we do anything else, we need decrypt and import the private key, and also install a few dependencies. Ubuntu 12.04 (the distribution Travis uses) comes preinstalled with a lot of things, but it may not always be enough. For instance, I need to install Zopfli so that I can compress my HTML, CSS, XML (and JS, if I had any) as much as possible.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#!/bin/bash
set -x # Show the output of the following commands (useful for debugging)
    
# Import the SSH deployment key
openssl aes-256-cbc -K $encrypted_22009518e18d_key -iv $encrypted_22009518e18d_iv -in deploy-key.enc -out deploy-key -d
rm deploy-key.enc # Don't need it anymore
chmod 600 deploy-key
mv deploy-key ~/.ssh/id_rsa
    
# Install zopfli
git clone https://code.google.com/p/zopfli/
cd zopfli
make
chmod +x zopfli
cd ..

The openssl command decrypts the encrypted private key. I haven’t written it by hand, I’ve actually just copy-pasted what travis encrypt-file added to .travis.yml earlier on.

Also, note that I’m moving the key to the .ssh directory under the name id_rsa. This is the default name for the key Git will look for when pushing to the server. It makes our lives a bit easier to place it there, under that name, since we won’t need to specify what key should be used later on.

Build

Building is actually pretty straightforward. I build the site with Jekyll, and compress with Zopfli, so that a super compressed version can be served by Nginx (see gzip_static).

1
2
3
4
5
6
7
#!/bin/bash
set -x
# Build the site with Jekyll
bundle exec jekyll build

# Compress assets with Zopfli
zopfli/zopfli --i1000 _site/**/*.html _site/*.html  _site/**/*.css _site/*.css _site/**/*.js _site/*.js _site/**/*.xml _site/*.xml

This level of Zopfli compression is quite slow, so I’ve actually set up my build step to look at the git diff and only work on changed files. This is much faster, but the build script becomes fairly complex as a result, so I won’t go into too much detail.

Test

Testing is done with HTML-Proofer. Note that I haven’t installed html-proofer manually, since it is in my Gemfile, and Travis is therefore nice enough to automatically install it for us.

1
2
3
4
#!/bin/bash
set -x

timeout 30s bundle exec htmlproofer _site --only-4xx --external_only --check-html --check-favicon --allow-hash-href --http-status-ignore 429

In this one command, I’m validating HTML, checking that no external link returns a 4xx-error (which would mean that I’m linking to something that no longer exists, or that I have a typo in one of my links), and that the favicon is present and referenced on every page. It’s rather important to have the --external_only flag; otherwise, whenever you’re adding a new post, it’ll return an error because the post isn’t online yet.

All of this has to be done within a 30 second deadline, just in case it starts hanging for no apparent reason and blocks the deployment (it’s happened before).

Deploy

If nothing has failed yet, then we should be good to deploy! However, we need to be a bit careful: we don’t want to deploy the code that is on GitHub, we want to deploy what we’ve just built. That’s why we’re going to push from a new repo in our _site folder.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
#!/bin/bash
set -x
if [ $TRAVIS_BRANCH == 'master' ] ; then
    # Initialize a new git repo in _site, and push it to our server.
    cd _site
    git init
        
    git remote add deploy "[email protected]:/var/www/kjaer.io"
    git config user.name "Travis CI"
    git config user.email "[email protected]"
    
    git add .
    git commit -m "Deploy"
    git push --force deploy master
else
    echo "Not deploying, since this branch isn't master."
fi

We’re adding our remote with the deployment username, on the path where our .git directory is. We commit everything and push it using the --force. This is necessary, since this new repo isn’t strictly linked to our remote; it doesn’t have the same history or anything. This argument just tells Git to ignore that fact.

Jekyll + Travis CI + DigitalOcean =

That should be all it takes to get automatic deployments to DigitalOcean! I know that it’s no small task, but hopefully, this guide will make it easier. I’m now proud to say that my workflow now consists of only two steps:

  1. Write content in Markdown
  2. Commit to GitHub

This is getting pretty good, almost good enough to sell to a client! Prose.io provides us with an editor that commits directly to Github, which abstracts the technical difficulties away from the user on both steps. As the developer, you can also revert commits if the user messes up, which is a huge plus. Add Travis CI tests to it, and you can even see if everything still is going fine. It’s close to the ideal setup to manage a static site.

Setting this continuous integration workflow up probably took more time than I’m willing to admit. I couldn’t find much online about deploying from Travis to DigitalOcean. Though Travis’ documentation provided a good starting point, it didn’t really help. I had to piece this setup up from a lot of different resources, which took a lot of time, but also led me to learn a lot about a few different technologies.

If I missed something, please make sure to scold me in the comments, and I’ll update this post accordingly. If anything isn’t clear, feel free to ask or check out my GitHub repo.

« Back