I'm joining the Fediverse

Ghost 6.0 launched this month bringing with it a bunch of new features designed to connect and grow blogs. The most interesting to me were improved analytics via a native Tinybird integration and integration into the Social Web. Analytics were a much needed upgrade, with the existing analytics limited to telling you how many members had viewed a post. The Tinybird integration brings table-stakes stats like true page views, referrers, and reader locations. The Social Web brings your Ghost blog into the Fediverse - this means your site can be followed by users on Mastodon, other Ghost blogs and more.

While the newest 6.0 features come free to Ghost Pro users, self hosters like yours truly need to do it manually. I installed Ghost originally using the Ghost CLI which handles installing the database, web server and the Ghost application itself. Normally a simple ghost update on your server will handle the upgrade process, however to get the new analytics and social web features, you need to migrate to the brand new Docker installation method. The devs over at Ghost decided that deploying the services separately as docker images vs installing and maintaining them as a monolith would be more future proof and I tend to agree, however it does complicate the upgrade process. Luckily the team provided a migration guide with a script to automatically move your data and configuration across to the new docker containers.

You don't work as a software engineer for any length of time without learning some hard fought lessons, and one of the lessons I have learned is to never expect an upgrade to go smoothly. Luckily for me, Proxmox provides an incredibly simple backup process via the UI. Just head to the container -> Backup and hit Backup Now.

Backup secured, I followed the instructions in the migration guide and, according to the script at least, everything went swimmingly. However, when trying to access the site I was met with a gateway error. This first problem was easy to fix, the Ghost CLI install is exposed on port 2386 however the Docker install listens on port 80 instead. I opened up my traefik config for ghost and changed the port, sudo systemctl restart traefik and tried again.

http:
  routers:
    ghost-router:
      rule: "Host(`miggl.es`)"
      service: ghost
      tls:
        certResolver: letsencrypt
  services:
    ghost:
      loadBalancer:
        servers:
          - url: "http://192.168.0.107:80"

This time my browser complained of Too Many Redirects. A different error, progress! It was time to look at the source code and understand what was going on.

The docker compose file that Ghost provides sets up a bunch of different containers for everything needed to run the stack: tinybird, activitypub, db, ghost and caddy being the main ones. The important service to us in that list is Caddy. Caddy is a web server with a very similar ethos to Traefik but with even more emphasis on SSL, actually creating certificates by default. Reading through the default Caddyfile it seems that the Ghost docker setup expects web traffic to come straight to Caddy, with Caddy terminating the SSL connection and passing the traffic on to Ghost (along with the rest of the services). But I already have Traefik handling this, so I need to change the Caddy configuration to stop any SSL funny business and make sure the requests are forwarded to Ghost as expected.

Disabling automatic SSL provisioning is pretty easy, global config is put at the top of the Caddyfile like so

{
        auto_https off
}

But even with auto_https disabled I was still getting Too Many Redirects. The debugging process had already taken too long, and petrified that one of my 5 members might try to access my site at 1am I decided to restore my backup and try again in the morning. The backup restore process in Proxmox is amazing, taking seconds to take your container back to the exact state it was in before. Site restored, I went to bed to tackle this in the morning.

Refreshed, I decided to test out another Proxmox feature: Instead of making a backup, I cloned the entire container. Cloning does require you to shut down the container first, but the whole process takes less than a minute and now I had two identical containers running, ghost and ghost-docker. This meant I could hack around on ghost-docker to my heart's content and whenever I needed to test anything I just edited the Traefik config to point to the new container. If it didn't work, I just changed the IP back and tried again.

In trying to solve my problem I had trawled the Ghost forums, reddit, and LLMs without finding the magic bullet. Looking at the Caddy logs I discovered that the problem was not with Caddy as I had expected, in fact the 301 redirect was coming from Ghost itself. Embarrassingly, the answer eventually came to me from the Ghost documentation itself, where they have a page dedicated to running Ghost behind a reverse proxy and explicitly call out the redirect looping problem. Maybe their SEO needs work 🤔. The issue is that Ghost expects the header X-Forwarded-Proto to be set to https otherwise it will automatically try to redirect the browser to the https:// version of the URL. The problem was that the browser was already at that URL, hence the redirect loop. Traefik actually sets X-Forwarded-Proto to https however it is Caddy that gets that request next and because it sees a non SSL request it sends the request on as http. In my Caddyfile I have hardcoded Caddy to send https however I suspect that I should just forward whatever it gets instead. The docs say that trusted_proxies should enable that forwarding, but that wasn't my experience. The relevant portion of my current Caddyfile is shared below, disclaimer I tried multiple fixes and have removed none of them in fear that any of them may be load bearing.

{
        auto_https off
}

:80 {
        import snippets/Logging

        # Traffic Analytics service
        import snippets/TrafficAnalytics

        # ActivityPub Service
        import snippets/ActivityPub

        # Default proxy everything else to Ghost
        handle {
                reverse_proxy ghost:2368 {
                        trusted_proxies private_ranges
                        header_up X-Forwarded-For {remote_host}
                        header_up X-Forwarded-Proto https
                        header_up X-Real-IP {remote_host}
                        header_up Host {host}
                }
        }

        # Optional: Enable gzip compression
        encode gzip

        # Optional: Add security headers
        import snippets/SecurityHeaders
}

So we are live on Ghost 6.0 and almost everything is working. I still can't follow other Ghost sites via ActivityPub, trying to throws an authorisation error in the logs, something about missing keys. When I find a resolution I will update this post.

See you on the social web!