Replacing nginx with Caddy
My final configuration files are at the bottom of the post
A couple days ago I was reading HackerNews and I read about this thing called Caddy. It's a server thing like nginx or Apache except it's new and written in Go. I actually only cared about the fact that it has Let's Encrypt support built-in. Just launch the server and you get https for free. I currently use nginx but last time I checked (whenever ago) the support for nginx was experimental so I never bothered with it. Also my nginx config file is also messy as fuck but that's probably because I never actually bothered trying to learn how to use it properly (or use vim properly). So I decided to try the new hotness that is Caddy.
First I downloaded Caddy from their website - https://caddyserver.com/. They offer a couple different extensions (like deploy your site with git) that I didn't care about. Next thing I did was try and figure out the best way of running it. nginx is pretty simple...I guess. sudo service nginx start. I'm pretty sure I just followed one of the DigitalOcean tutorials on setting up a LEMP stack so it was pretty easy to get up and running (without all the config stuff). Caddy not so much. First I set up a user for Caddy since I guess running your webserver as root isn't a good idea. Then I needed to find a place for the Caddy executable to live. Since nginx lived in /etc I guess Caddy could live there too. Then I wanted Caddy to be as easy to run as sudo service caddy start but that wasn't in the documentation anywhere. After some googling it seems like there are many ways to do it so I just opted for this because I'm terrible at being a sysadmin/devops for my server. After all that was set up just needed to make sure Caddy could bind to ports lower than 1024 so I needed to run sudo setcap cap_net_bind_service=+ep ./caddy.
Next part(y) was configuring Caddy since running ./caddy wasn't going to cut it. First I took a look at the monstrosity that is my nginx config. I have 21 virtual server blocks all running or redirecting to various things. 1 for my main site, 1 pointing to ghost which runs my blog, 2 pointing to ASP.NET vNext/ASP.NET Core 1.0/whatever the hell Microsoft is calling the new cross platform ASP.NET server/DNX/dotnet/corefx stuff, 3 pointing to python flask projects, and 13 302 redirects. Oh and one points to the static home page of attentionpassengers.com since I couldn't figure out how to get flask to serve that page up correctly.
So I read some docs and realized I still had no idea what I was doing so I just started typing. First I defined all my servers blocks and then went through each one translating the nginx syntax to Caddy syntax. My config is pretty simple. Serve up my home page, redirect to some places, and then reverse proxy for ghost and my other servers (uwsgi/flask and asp.net). After getting everything down I though I was ready to just run sudo service caddy start. So I did that. And nothing worked. Cool. Running Caddy directly on the command line and getting error output would probably be a good call so I did that and I get an error about one of my first server blocks. My config had a tls off line at the beginning and then a bunch of virtual server blocks. I guessed that if you have virtual server blocks set up then you can't have global settings outside those blocks. I moved the tls off into my first server block and tried again.
Big mistake. Caddy started launching, tried to get certificates for all my subdomains and then I hit the certificate limit. Let's Encrypt sets a limit of 5 certificates per domain per week. Well that sucks. Wait, but you can have up to 100 subdomains under one certificate. Well, Caddy doesn't support that and it looks like the maintainer doesn't want to include support for that anytime soon, if ever. (Thank god for open source software. Right?!) So I put tls off in all of my blocks and finally Caddy got up and started running.
I hit my main page and it worked. I was actually pretty happy. No https for a week but maybe it would work for the other subdomains that managed to get certificates. So I removed tls off from one of my subdomains and then I got this error 2016/03/12 16:51:50 configuration error: Cannot multiplex golf1052.com:443 (HTTP) and blog.golf1052.com:443 (HTTPS) on same address. Well I don't know what that means. My golf1052.com block had tls off set. Googling gave me some information about the same error with Apache but nothing about Caddy so I tried commenting out my golf1052.com block and tried again. 2016/03/12 17:07:34 configuration error: Cannot multiplex blog.golf1052.com:443 (HTTPS) and attentionpassengers.com:443 (HTTP) on same address. The block after now was throwing an error even with tls off. So I commented out every other block and I could access my blog with https. Cool...I guess.
So the entire reason for switching to Caddy kind of didn't work at this point. Maybe I'm doing something wrong but this wasn't the "easy to configure and use" server I was expecting. But I wasn't ready to give up yet so I kept on trying to set things up. I had to make a few changes to my uwsgi/flask projects because they used the native uwsgi socket protocol to talk to nginx and Caddy doesn't support that so I switched them to use an http socket instead and they worked. My asp.net projects worked without any extra configuration so everything was working pretty great besides the https issues.
The last thing that I needed to do was to block access to certain files under my website. I'm way too lazy to move things to the right place. The internal command says it blocks all resources in the specified directory but I couldn't get it to work. I did some more Googling (actually searching through the Caddy issues list) and I found that I could use the rewrite command to rewrite requests to another place. After struggling with regex for 15 minutes I finally got it working.
Only took me a couple hours of frustration but I got Caddy working. It's currently serving up ghost which serves up this blog. Too bad I have to wait a week to try again on https (and I'll probably fuck it up again). Maybe it would be worth it to get the one certificate for all my subdomains manually then set up Caddy to use that. But that kind of defeats the purpose of switching to Caddy. The homepage of Caddy makes some pretty bold claims about how easy it is to setup. Honestly, it probably is easy to set up if you have one simple website or a static page. It has cool features like serving up websockets or turning Markdown pages into HTML. I feel like my config isn't even that complicated. I guess the https setup is just hard to do right. Let's Encrypt makes it easier but maybe getting https everywhere easily just isn't there yet. I do think the configuration was easier than nginx's configuration. I could digest the entirety of the somewhat confusing docs in an hour. I never bothered looking at actual nginx docs, I figured they would be too confusing with too many options. And maybe that's why my entire server setup is shitty. I don't have the patience to read the fucking manual for hours and hours and try to set things up right. There are a ton of ways to set things up in nginx. I just want to develop random side projects and post mediocre blog posts.
golf1052.com - My old nginx config. I took out some unused blocks but I left some comments in to show the weirdness I had in it.
golf1052.conf - My current Caddy configuration file.
caddy.conf - My Caddy upstart file (for service start/stop). I had a couple of issues after I finalized my config. I was getting /proc/self/fd/9: 2: cd: can't cd to /etc/caddy^M in /var/log/upstart/caddy.log. The problem was that the file endings were CRLF instead of LF because I created the file on Windows and then transferred things over.