I've been using Philips Hue lightbulbs in my house for a few years. The hardware has been reliable. The software on the other hand is not the best. Being tied to a mobile app as the only source of control can be cumbersome. While my Philips Hue bulbs are also connected to Google Home, I don't have one set up in every room of the house. Plus, the Philips Hue app is lackluster when it comes to making automation rules. Most of the time the GPS-based triggers do not work properly.

I've also been wanting to create more home automation – especially with my lights. Being able to dim or turn off the lights completely when no one is in the room is extremely useful and can save a few dollars on the electricity bill. Buying a Philips Hue motion sensor is an option but they are quite expensive ($45-65USD). There are many other cheaper options available but that would require buying another gateway and maybe even a whole set of new lights since there's a lack of cross-brand support.

The IoT market is very segmented. At the lowest level, there are several competing wireless protocols available; Zigbee, Z-Wave, Lora, and WiFi to name a few. Sticking with one protocol also presents compatibility issues directly resulting from proprietary implementation across different brands. A Philips Hue hub/bridge will work with Hue lightbulbs and a handful of partnered brands. Even with partner brand support, there's a high chance of limited features when mixing. A typical off-the-shelf smart home today means multiple proprietary hardware and cloud services that do not guarantee direct integration with one another. This makes for a very expensive setup and requires looking for workarounds and expensive third-party services to connect them all, like Google Home and Samsung Smarthings. Luckily, Home Assistant falls into the cost effective solution if you're willing to put in the work to set it up.

With an underutilized thin-client acting as a local DNS and Adblock (Pi-hole) server, Home Assistant seems like the perfect solution to all of the above-mentioned issues by eliminating the need to buy into different IoT ecosystems. Home Assistant not only acts as a unifying abstraction layer, but it also has a large community-driven set of tools and plugins to accomplish tasks such as creating automation rules for supported devices.

Getting Started

Bill of Materials

• Zigbee enabled devices (pre-existing or just to test). e.g. Light bulbs, motion sensors, temperature sensors, etc.
• Zigbee coordinator. This could be a USB-dongle like the ConBee II, or SONOFF Zigbee 3.0. For this project, I chose the Sonoff ZBBridge. At the time of writing, the Pro version of the Sonoff ZBBridge was released a few weeks ago (it supports up to 128 devices).
• (Optional) FTDI Programmer. Necessary if you want to use the Sonoff ZBBridge route to load a custom firmware called Tasmota.
• A dedicated server with Docker installed. This can be a Rapsberry Pi or an old computer. The idea is to reduce power consumption, so low-powered is important. The Dell Wyse 5070 thin client I am using has a Pentium Silver J5005 which only consumes roughly 10W max. If you are planning on using a Raspberry Pi, then the Home Assistant Pi image is better suited than a Docker setup (you can still use the configuration files in this guide).
• If you're using an Intel based system, I highly recommend Intel Clear Linux Server. It's highly optimized for Intel CPUs.
• Reference: https://zigbee.blakadder.com/ is a great place to check device support for Zigbee2MQTT or ZHA (alternative Zigbee integration for Home Assistant).

I chose to use the Sonoff ZBBridge instead of the USB Zigbee dongle because my server is sitting in a closet at one end of the house. While Zigbee is a mesh protocol, I wasn't too sure if there would be a strong signal to the first device. Plus, having an untethered coordinator makes it easier to place it almost anywhere in the house.

Sonoff ZBBridge is an ESP MCU based controller that's shipped with Sonoff's firmware for their IoT managed solution. Thankfully there are pin-outs on the board that allows for reflashing. You can either choose ESPHome or Tasmota. I went with Tasmota.

Follow the guides below to flash your Sonoff ZBBridge  before proceeding:

• Sonoff ZBBridge Tasmota flashing guide
• Sonoff ZBBridge Pro Tasmota flashing guide (source via Github)

Tasmota flashing and setup are required before proceeding.

Flashing will be a two step process: 1) Flash the ESP MCU with Tasmota 2) Flash a custom Zigbee module firmware once Tasmota is up and running. Lastly, remember to follow the guide's configuration template to set it up for Zigbee2Tasmota. ZHA (Home Assistant Zigbee plugin) is another alternative that provides a direct connection without requiring an MQTT broker such as Eclipse Mosquitto. However, from my experience, ZHA has less device support and a bit complicated to custom add new devices.

Once you've completed the step above:

• Make note of your Tasmota Sonoff ZBBridge's IP address.
• Update your router's DHCP settings to make the IP address static for the Tasmota Sonoff ZBBridge. Without this configuration, your setup will break when Tasmota's IP address changes.

On your server, clone the project template:

git clone https://github.com/foureight84/ha_zigbee_docker.git && cd ha_zigbee_docker

Docker

Templates repository is located here: https://github.com/foureight84/ha_zigbee_docker

If you are already using Traefik then modifications will need to be made before running the docker-compose.yaml / docker-swarm.yaml. The same applies to importing the templates into Portainer.

The scaffold Docker volumes folders adhere to "iot" as the service name for the docker instances. If you wish to use a different name, make sure to edit the foler name prefix accordingly.

Docker Swarm Setup

Skip to the next section if you're using standalone Docker.

• Create "traefik" overlay network

docker network create --driver=overlay --attachable --subnet=48.84.0.0/16 --gateway=48.84.0.1 traefik

Make sure to update iot_homeassistant\_data\configuration.yaml if you set a different subnet for the "traefik" network. This is necessary to access Home Assistant's web UI.

• Update iot_zigbee2mqtt\_data\configuration.yaml with your Tasmota Sonoff ZBBridge's IP address

serial:
  port: tcp://<<tasmota ip address>>:8888 #update to match your static ip for Tasmota Sonoff ZBBridge
  adapter: ezsp

• Copy configuration to docker volumes storage location:

sudo cp -a iot_* /var/lib/docker/volumes/

• Run the Home Assistant stack:

docker stack deploy -c docker-swarm.yaml iot

Stack name needs to match volumes folders' prefix "iot_". Rename volumes folder if you wish to use a different stack name.

Docker Standalone Setup

Ignore this step if you are using Docker Swarm.

• Update iot_zigbee2mqtt\_data\configuration.yaml with your Tasmota Sonoff ZBBridge's IP address

serial:
  port: tcp://<<tasmota ip address>>:8888 #update to match your static ip for Tasmota Sonoff ZBBridge
  adapter: ezsp

• Copy configuration to docker volumes storage location:

sudo cp -a iot_* /var/lib/docker/volumes/

• Run the Home Assistant stack:

docker-compose -p iot up -d

DNS

You will need to create DNS entries in your router to access the running services. For my instance, 192.168.1.2 is my Docker server and these are my router DNS records:

192.168.1.2 traefik.home
192.168.1.2 home-assistant.home
192.168.1.2 node-red.home
192.168.1.2 zigbee2mqtt.home

Home Assistant Initial Setup

• Browse to http://home-assistant.home and create your account. At the end of the account creation wizard, you should see a screen similar similar to this:

• Click on mqtt and mosquitto as the broker and submit. You should get a "Success" confirmation. Instead of using the Docker IP for the Eclipse Mosquitto container, we are providing it with the hostname instead.

Installing HACS (Home Assistant Community Store)

HACS (Home Assistant Community Store) as the name suggests, this is a repository of community developed plugins for Home Assistant. While there are well maintained plugins, they are unofficially supported. Be aware of this when installing any plugin as it may cause unwanted behaviors with Home Assistant.

We will need HACS in order to complete the Node-RED integration.

A Github account is required to install HACS.

Why?

HACS uses the GitHub API to gather information about all available and downloaded repositories. This API is rate limited to 60 requsets every hour for unauthenticated requests, which is not enough. So HACS needs to make authenticated requests to that API. (source)

Download | HACS

HACS download steps

HACS

• You'll need to go inside the Home Assistant container:

docker exec -it $(docker ps -q -f name=iot_homeassistant) bash

• Once inside the Home Assistant container:

wget -O - https://get.hacs.xyz | bash -

• Restart Home Assistant from the web UI (http://home-assistant.home) by going to Settings > System > Restart.

• After Home Assistant restarts, complete the installation by adding the HACS integration. To do so, go to Settings > Device & Services > + Add Integration and type HACS. You will need to acknowledge all the checkboxes and follow the instruction to add HACS to your Github account.

You will see HACS as one of the left-menu items in the Home Assistant web UI.

Setting up Node-RED for creating Automation Rules

Node-RED is a flow-based development tool for visual programming. We will be installing specific a community developed "Palette" for Home Assistant.

• From the left-menu in the Home Assistant web UI, click on Node Red.

Node Red and Zigbee2MQTT menu items are custom added via the Home Assistant configuration.yaml. iframe Panel documentation
These are essentially iframe links to our Node-RED and Zigbee2MQTT web UI instances.

• Access the "Palette Manager" by pressing alt + shift + p or go to the hamburger menu on the top right of the iframe then choose Manage palette.
• Click on the Install tab and search for home-assistant and look for node-red-contrib-home-assistant-websocket.
• After the palette installs, scroll to the bottom of the Node-RED nodes list on the left and you should see the home assistant section with all of the associated nodes.
• Drag the 'API' node into the flow workspace and double-click to edit its properties.

• Open a new browser tab and head to your Home Assistant web UI (http://home-assistant.home). We will be creating a long-lived token for Node-RED to connect with HA. Click on your profile on the left-menu and scroll to the bottom. Click Create Token and call it Node RED. Copy the entire token string that should look like this: eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJlNmY1Nzk5YjAzNjg0MjY1ODRkYTc5Yjc0YTVhMTI1ZCIsImlhdCI6MTY1NDgwMTM4NiwiZXhwIjoxOTcwMTYxMzg2fQ.vjbD8T378Da7nMzgAKotVbzMd-77pAVGI6c7wfcKf6U
• Go back to the HA tab with Node-RED opened with the API node properties menu. Click to add new server. Fill in http://home-assistant:8123 for Base URL field and paste your access token.

• After the server has been added, the API node can be deleted. Make sure to click Deploy to save your configuration.

You're all set to create automation rules.

Pairing New Zigbee Devices

• In your HA web UI, click on the Zigbee2MQTT. By default via the iot_zigbee2mqtt\_data\configuration.yaml all Zigbee devices in pairing mode will be allowed to join. This can disabled from the Zigbee2MQTT (top-right button).
• Put your device into pairing Mode. Depending on the device, it will either be holding down the reset button on the device or turning a switch on and off in combination for light bulbs. See manufacturer instructions for details.

You will see the device show up in Zigbee2MQTT after a few seconds after entering pairing mode. You'll want to name the device appropriately.

These devices should also appear in Home Assistant under the Settings > Devices & Services > mosquitto MQTT.

Home Assistant also has an Android and iOS companion app that provides additional tracking such as your location. All of this data is fed to your local instance allowing for GPS based automation rules such as turn the lights on when coming home at night, or setting up motion and time based light triggering.

In the next guides, I'll walk through WireGuard VPN setup to maintain a tunnel to your home network to access Home Assistant without exposing it to the public internet.

I've been using Pi-hole as a whole network ad blocker for a while now and it's been great. The only mistake I made the first time was using it as a standalone docker instance. Whenever I perform a manual update to a newer release, there would be a few minutes of internet downtime. Personally, it's not a big deal and I could just force a DHCP-renew. But I share my internet connection with my parents living next door and an internet outage immediately leads to a phone call.

There are various ways to update a running Docker container and minimize downtime, and I've chosen to update my stack to use Docker Swarm to accomplish this goal. Yes, the setup is overkill for a home network, but making things causal is always better down the road when you need to make changes.

I am currently using a J5005 Dell Wyse 5070 thin client as my server, but this article should apply to a Raspberry Pi. You should take a look at the Docker images used and swap them for ARM-compatible.

Disclaimer: This is meant for home use only. DO NOT deploy this in a Production environment. Pi-hole uses SQLite and concurrent write is not supported. While DNS resolution and blocking won't have issues, there will be loss of log data if two or more swarm nodes are writing to DB at the same time. Also, make sure your router is properly configured to not expose your DNS to the public internet. It will crash and burn.

Stack Overview

• Management Stack (mgmt)
  • Traefik - reverse proxy and load balancer
  • Portainer - Web UI Docker management tool
  • Whoami - Tiny Go webserver that prints os information and HTTP request to output
• DNS Stack (dns)
  • Cloudflared DoH Proxy
  • Pi-hole DNS adblocker and DHCP server

A few Details

Traefik is a GoLang reverse proxy and load-balancer with built-in support for Let's Encrypt. In my earlier post about using Terraform to deploy this Ghost blog, I used jwilder/nginx-proxy along with jrcs/letsencrypt-nginx-proxy-companion to serve as a reverse-proxy and automatic SSL certificate handling. However, that setup is on a Docker standalone deployment using Docker Compose. While it is possible to use an Nginx reverse-proxy with Docker Swarm, Traefik is a much better fit with Docker Swarm with fewer configurations involved.

Portainer is a container management tool with a web UI. It's used in this project to manage the DNS swarm, which makes scaling and service updates more casual. I personally prefer this to keep the project simple and doesn't require looking up Docker CLI commands.

Pi-hole, the main reason for this project, is a Linux based DNS server with built-in DHCP support. The main use for Pi-hole is to block advertisement domains, but it also has additional benefits in blocking known malware and phishing domains as well. As it is a DNS server, it can be used to apply to an entire network.

Getting Started

We'll first need to clone the project

git clone https://github.com/foureight84/traefik-pihole-doh.git && cd traefik-pihole-doh

You'll need to enable Docker swarm on your host if not already.

docker swarm init

We will need to create a Docker overlay network called traefik. Additional Docker services you choose to add later that require the use of Traefik will need to be attached to this network.

docker network create --driver=overlay --attachable traefik

Now we spawn the mgmt stack and use Portainer to deploy and maintain swarms from here on out.

docker stack deploy -c mgmt/docker-compose.yaml mgmt

If your server is running Ubuntu >= 18.04, chances are, systemd-resolved is probably occupying port 53. We'll need to turn that off.

Update your hosts file with the hostnames for the running web services. Windows: C:\Windows\System32\drivers\etc\hosts, Linux/Mac: /etc/hosts/. For my setup, my home network has the domain of .home. You will need to change this for your setup. 192.x.x.x denotes the server this docker stack is currently running on.

192.x.x.x	portainer.home
192.x.x.x	traefik.home
192.x.x.x	whoami.home
192.x.x.x	pihole.home

Traekfik dashboard is available via http://traefik.home/local/dashboard/ don't forget the / at the end. You should see entry points for port 80 as well as 53 tcp and udp.

Go to Portainer via http://portainer.home and create your admin account. Once logged in, we will need to create a Docker Secret that will hold the web password for Pi-Hole web ui. The secret needs to be named pihole_webpw.

Make note of the docker_gwbridge network's gateway IPV4. This is used to forward DNS requests to the upstream Cloudflared DoH docker container. We can't use the IP the container gets assigned on the traefik network since that is not static.

Open the docker-compose.yaml in the dns folder with a text editor and update the DNS1 field with your docker_gwbridge's IP address.

environment:
      - TZ=America/Los_Angeles
      - DNS1=172.18.0.1#5054 #replace with docker_gwbridge's gateway ip
      - DNS2=no
      - REV_SERVER=true
      - REV_SERVER_CIDR=192.168.1.0/24 #Update these fields to match your environment
      - REV_SERVER_TARGET=192.168.1.1
      - REV_SERVER_DOMAIN=home
      - WEBPASSWORD_FILE=/run/secrets/pihole_webpw

The REV_* environment variables should also be updated to match your setup as well. REV_SERVER and related fields allow Pi-Hole to perform a reverse DNS lookup against the router. I'm currently using my router's DHCP server (REV_SERVER_TARGET). Static ip and hostnames are kept on the router. Requests for my local NAS for example, will go through Pi-Hole and that will get forwarded to my router.

After making changes to the dns/docker-compose.yaml file, we will need to upload this to Portainer as a docker-swarm template and deploy the swarm.

It should take a few minutes for Pi-Hole deployment to complete. Update the DNS on your computer with your server's IP and test before applying it as the DNS to use on your router. To check if DoH is working properly, head to https://1.1.1.1/help

Update Services

Since we are running in Docker Swarm mode, we can take advantage of rolling updates to minimize downtime. Of course, you need more than one instance of a service running in order to perform a rolling update.

We first need to change the --update-delay flag on the following services (or any that you want to apply a rolling update):

• Traefik
• Pi-hole
• Cloudflared

We will use 120s (2 minutes) as the delay from one service instance to the next. Pi-Hole and Cloudflare services can take up to 30 seconds to reach ready-state and process new requests. 120 seconds should give us ample time for a new Docker image to download and update the first instance while the other instances remain unchanged and continue to process new requests.

In the Service details view, click on Update configuration on the right-hand menu to scroll to the section shown in the image above. Update Delay should be entered as 2m or 120s.

To update a service, scroll to the top and click on Update the service and toggle Pull latest image version on the confirmation screen. Now Docker will update each running instance one-by-one 120 seconds apart.

If you've noticed from the Service list view screenshot, I am only running 1 instance for all my services since it's a home network and I don't need the redundancy or load balancing at the moment (although I might scale Pi-hole and Cloudflared to 2 instances each as these containers have halted in the past). These services will be scaled to 2 or 3 before performing a service update in order to utilize the rolling update. Once all instances are updated, I will downscale them back to the initial scale.

Happy adblocking!

FAQ

Q: Where can get more blocklists?

Filterlists is a good place to start (https://filterlists.com/)

Q: What the hell are you on about?

For anyone starting from scratch, the initial steps not mentioned in this guide are:

1. Installing Ubuntu server (https://ubuntu.com/download/server), for Raspberry Pi (https://ubuntu.com/download/raspberry-pi)
2. Enable SSH after booting into Ubuntu: sudo apt install openssh-server. This will allow remote terminal login to your server via ssh. e.g. ssh your-username@your-server-ip-address from another computer with an SSH client. Windows 10 has SSH built-in or you can use PuTTY.
3. Installing Docker-CE: https://docs.docker.com/engine/install/ubuntu/

You do not need to install Ubuntu for the Rasberry Pi, there's also Raspian OS. Just make sure to do steps 2 and 3.

Q: How do I deal with the port 53 conflict error with the Traefik container?

There are two approaches to this. I will guide based on Ubuntu. You can use this as a guide for your system.
First, we need to find out what's using port 53 and determine if we can stop, or disable it. If you can't then read further for the alternative.

sudo netstat -pna | grep 53

If you're on Ubuntu 18.04 (I believe) and newer, it will most likely be caused by systemd-resolved.

If it is then:

1. sudo systemctl stop systemd-resolved to stop the service
2. sudo nano /etc/systemd/resolved.conf
3. Uncomment #DNS= and replace it with our DNS of choice. It shouldn't matter as this is for your host. I chose cloudflare. It should then look like this DNS=1.1.1.1
4. Uncomment #DNSStubListener=no or add the line if it's not there.
5. Save and exit the text editor.
6. sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf creates a symbolic link and replaces the current link in the existing path.
7. Check that DNS still works by doing a ping github.com
8. If you see an error about sudo: unable to resolve host ubuntu: Name or service not known then edit your hosts file and add 127.0.0.1 your-machine-name below 127.0.0.1 localhost.

If you cannot stop the service that's currently using port 53 then you can update the mgmt stack docker-compose.yaml to use docker's loopback.

traefik:
  image: traefik:latest
  ports:
    - target: 53
      published: 53
      protocol: tcp
      mode: host
    - target: 53
      published: 53
      protocol: udp
      mode: host
    - target: 80
      published: 80
      protocol: tcp
      mode: host

Unfortunately, IP address format is not supported in long-form port declaration. It will need to change to:

traefik:
  image: traefik:latest
  ports:
    - 0.0.0.0:53:53/udp
    - 0.0.0.0:53:53/tcp
    - target: 80
      published: 80
      protocol: tcp
      mode: host

A caveat of using this method is that Traefik won't be able to forward the real IP of the requestor in the request header for X-Forwarded-For and X-Real-Ip. That IP will be a docker NAT IP (test this out using http://whoami.test.local). It's not important for our purpose, but if you were to deploy another web service that utilizes this then it could be an issue.

Q: Pi-Hole client id in the query log only shows Traefik's hostname

When a request comes in through a reverse proxy via HTTP, the request's originating IP is passed along in the request headers as X-Forwarded-For and X-Real-Ip. However, with DNS, there's no such flag. There's been a proposal for DNS X-Proxied-For but that's not a standard. https://tools.ietf.org/id/draft-bellis-dnsop-xpf-03.html
Pi-hole only makes notes of which immediate client sent the request, and in this case, it's Traefik. If you really want to make this work then you can implement a macvlan network and attach your Pi-Hole instance(s). This would allow a Docker container to receive an IP on your router's subnet and allow direct communication.
If pi-hole was not in swarm mode then this would not be an issue (this was my previous setup). In swarm mode and behind a load balancer, you would need to allocate a block of IPs to dedicate to the swarm (IP assignment is not supported in swarm mode) while the load balancer would only apply to port 80 traffic to pi-hole's web UI.

Q: How do I test my adblocking capability?

https://d3ward.github.io/toolz/src/adblock.html is a good place to start. It should show you whether you are blocking major players on the DNS level.

You could also turn off your browser ad block extension and go to a known site with embedded ads. https://tweaktown.com is a good place to test.

Without a browser Adblock extension, it doesn't look pretty but it saves on your bandwidth usage since ads do not load at all. Use this in combination with a browser-based adblocker to remove those HTML elements. Without Pi-Hole ads will be hidden with browser adblockers but they would still load.

Table of contents

• Terraform
  • Linode
  • Cloudflare
  • Variables & Definitions
• Docker
  • Ghost Stack
  • Rclone
• Deployment

At the beginning of the pandemic, I decided to build a custom keyboard (post on this later). The project led to me wanting to add a Blackberry trackball and this would require a special way to mount it to the keyboard PCB. After weeks of trying to jury-rigged a mount from common parts to attach the trackball to the keyboard, I realized that it would be much easier to design and 3D print the necessary part.

I also needed a better way to document my projects and starting a blog is better for visibility and accessibility than a markdown README file on Github. Ghost is a good candidate, offering a lightweight straightforward writing platform (with markdown support). There's also a robust list of third-party integrations which would be good down the road if I needed to scale.

I also wanted to be able to easily backup my data in case I needed to move to a different host. At first, I was eyeing an AWS S3 bucket or perhaps Linode's offering called "Object Storage." At the moment, I don't need that much backup space. A free option would be best. This is where Rclone is handy as it would allow me to tarball essential ghost files and have a daily backup. I could also deploy Rclone on another server, such as a local NAS, as an added backup destination.

I chose Terraform in order to automate the setup tasks so that I can easily switch hosts in the future. The one plus side with Terraform is that it uses declarative language to define tasks and this is, to me, a lot easier to read and understand in the future. If you haven't looked at your code in over six months, it may as well been written by someone else.

Project Overview

• Terraform deployment
• Cloudflare proxied DNS
• Docker
  • Nginx reverse proxy
  • Let's Encrypt for automatic SSL
  • Ghost 4
  • Rclone backup blog data to Google Drive

This project can be found on my github. Clone and follow along:

git clone https://github.com/foureight84/ghost-linode-terraform.git && cd ghost-linode-terraform

Terraform

A declarative configuration-based infrastructure as code API wrapper. There are a few pros and cons to consider. Keep in mind that this is my first time using Terraform these are surface-level points observed.

Pros:

• Quick and straightforward (standardized) configuration syntax declaring what tasks to perform.
• Easy to read and double-check my work. Terraform's 'plan' feature to check for syntax errors and preview the final result.
• Configurations can be changed and applied quickly without having to go through service consoles.
• All pieces in my stack can be managed through Terraform (see cons).

Cons:

• Providers often offer web APIs to perform the same tasks, and at best, there will be feature parity. But there is a chance that feature-parity on Terraform is a second priority to a provider and support lags behind.
• Not all services may have Terraform support, which could lead to the additional complexity of having to manage multiple tools. This is often the case in a real-world scenario.

• Applying my Terraform configuration update doesn't always behave as expected. The functionality limitation depends greatly on the provider. For example, after a Linode is created, applying changes to certain properties such as root password will result in Terraform destroying the instance to create a new node using updated configurations.

Before diving into specific files in the project, here is a brief overview and explanation of the project structure:

[..root]
│  cloudflare.tf                   //resource module for CloudFlare rules
│  data.tf                         //templates ex. docker-compose.yaml, bash scripts, etc...
│  linode.tf                       //resource module for Linode instance setup
│  outputs.tf                      //handle specific data to display after terraforming
│  providers.tf                    //tokens, auth keys, etc required by service providers
│  terraform.tfvars.example        //example answers for input prompts 
│  variables.tf                    //defined inputs required for terraforming
│  versions.tf                     //declaration of providers and versions to use
│
└─[scripts]
  ├─[linode]
  │       docker-compose.yaml      //main stack. nginx proxy, letsencrypt, ghost
  │       stackscript.sh           //boot time script specific to Linode to setup env
  │
  └─[rclone]
    │     backup.sh                //cron script for backing up ghost blog directory
    │     docker-compose.yaml      //rclone docker application
    │
    └──[config]
            rclone.conf.example    //rclone configuration for cloud storage

Linode

linodes.tf, stackscript.sh, data.tf

This is a straightforward configuration to create a Linode with the use of Linode's Stackscript. Which is essentially a run-once bash script that gets executed on the first boot.

The stackscript.sh lives in the ghost-linode-terraform/scripts/linode/ directory and is parsed by Terraform as a data template. Terraform's templating syntax needs to be taken into consideration when parsing text files. Most notable are variables and their escape characters. Any $string or ${string} notation will be regarded as a template variable, while $$string and $${string} are escaped bash variables. More about strings and templates.

script = "${data.template_file.stackscript.rendered}"

Alongside the Stackscript, the "linode_instance" resource block also includes stackscript_data property. This is a way of providing data to the one-time boot script. The key-value assignments within this block correspond with the 'User-defined fields' at the top of stackscript.sh.

#!/bin/sh
# <UDF name="DOCKER_COMPOSE" label="Docker compose file" default="" />
# <UDF name="ENABLE_RCLONE" label="(Bool) Flag to turn on RClone Support" default="false" />
# <UDF name="RCLONE_DOCKER_COMPOSE" label="RClone docker compose file" default="" />
# <UDF name="RCLONE_CONFIG" label="RClone configuration file" default="" />
# <UDF name="BACKUP_SCRIPT" label="Backup script" default="" />

stackscript_data = {
    "DOCKER_COMPOSE" = "${data.template_file.docker_compose.rendered}"
    "ENABLE_RCLONE" = var.enable_rclone
    "RCLONE_DOCKER_COMPOSE" = "${data.template_file.rclone_docker_compose.rendered}"
    "RCLONE_CONFIG" = "${data.template_file.rclone_config.rendered}"
    "BACKUP_SCRIPT" = "${data.template_file.backup_script.rendered}"
  }

data "template_file" "docker_compose" {
  template = "${file("${path.module}/scripts/linode/docker-compose.yaml")}"

  vars = {
    "ghost_blog_url" = "${var.ghost_blog_url}"
    "letsencrypt_email" = "${var.letsencrypt_email}"
  }
}

In the snippet above, docker-compose.yaml for the Ghost stack is parsed as a data template and ghost_blog_url and letsencrypt_email variables get evaluated and then passed as a string to the Stackscript at runtime.

At the time of deployment, the Stackscript will be created before the Linode instance. Terraform allows direct referencing named values. This can be seen in the "linode_instance" block where linode_stackscript.ghost_deploy.id is assigned as the stackscript ID to include when creating the Linode instance. Finally, other parsed data templates such the docker-compose.yaml are assigned to stackscript user-defined fields and sent as stackscript_data.

Cloudflare

cloudflare.tf

The Cloudflare configuration is relatively straightforward. Since the foureight84.com domain is already managed by Cloudflare, a lookup is performed and the named value is referenced with each resource call as zone_id property.

data "cloudflare_zones" "ghost_domain_zones" {
  filter {
    name   = var.cloudflare_domain
    status = "active"
  }
}

An 'A' record is created for the blog (blog.foureight84.com) and end-to-end HTTPS encryption is enforced (ssl="strict") along with requiring all HTTPS origin pull requests only come from Cloudflare. Nginx-proxy-companion will generate a validation file under the path blog.foureight84.com/.well-known/<some random string> for SSL certification requests. Let's Encrypt validation of this generated file accepts both HTTP and HTTPS (ports 80 and 443), a page rule is created to ensure that the request does not get blocked.

Variables and Definitions

variables.tf

As seen in linode.tf, cloudflare.tf, and especially data.tf, ${var.<some string>} are used throughout. These are references to declared variables in variables.tf such as API tokens for our services, domain names, and etc. These variables show up as input prompts when performing Terraform plan, apply, or destroy actions. Variables contain properties such as data type, descriptions, default values, and custom validation rules. More about input variables.

terraform.tfvars.example

This Terraform deployment requires 17 data inputs in order to perform its tasks and that's 17 possible chances to introduce error. Luckily, Terraform supports dictionary referencing in the form of .tfvars files. Tfvars are key-value files where the key is the variable name. A .tfvars file is referenced during run-time in order to provide required inputs. For example:

terraform plan -var-file="defined.tfvars"

Docker

This project has two separate docker-compose environments. The first is our blog stack, and the second is Rclone to perform data backup. I decided to separate the Rclone docker service from the primary Ghost stack for two reasons:

1. I want to be able to mount the cloud storage prior to running the Ghost stack so that data restore can be performed should a backup exists (see stackscript.sh).
2. Cloud drive mount should always stay active. Changes to my Ghost stack should not impact Rclone's availability.

Ghost Stack

docker-compose.yaml

This runs 3 containers using the following images:

• jwilder/nginx-proxy
• jrcs/letsencrypt-nginx-proxy-companion
• ghost:4-alpine

The default directory is /root/ghost. This can be changed via Terraform during deployment from the tfvars file. Since this is a set default value, Terraform does not prompt for the input value.

jwilder/nginx-proxy

This is the "front" facing container on the origin server (Linode) sitting behind Cloudflare. Visitors' incoming requests for https://blog.foureight84.com will go through Cloudflare, which is forwarded to our reverse-proxy, where the request is relayed to the running Docker service matching the VIRTUAL_HOST value. Finally, nginx-proxy will collect the served content from the Ghost service and send it back to the visitor.

The reverse proxy will be listening to exposed ports 80 and 443.

jrcs/letsencrypt-nginx-proxy-companion

Nginx-proxy-companion handles automatic SSL registration for the running Docker services. The important environment variables to keep in mind are:

environment:
  - LETSENCRYPT_HOST=${ghost_blog_url}
  - LETSENCRYPT_EMAIL=${letsencrypt_email}

The above environment variables are added to Docker services that require SSL certificates (Ghost container in this example). In the future, if I need to add additional subdomains, such as www.foureight84.com, then that service will require LETSENCRYPT_HOST=www.foureight84.com. I can avoid having to repeatedly define LETSENCRYPT_EMAIL by setting the DEFAULT_EMAIL environment variable in the nginx-proxy-companion instead.

By default, a production certificate will be requested. Let's Encrypt has a limit of 10 cert requests every 7 days for normal users. All requested certs are stored in the path /etc/acme.sh which is mounted to the acme Docker volume.

The nginx-proxy and nginx-proxy-companion share certs, vhost.d, and nginx.html volumes. If I need to move to a different host, then the files store in certs and acme docker volumes will need to be backed up. The former is where the generated private key used for SSL certificate requests is stored, whereas the latter is contains the generated certificates and checked against upon service startup. Without the original private key then the SSL certificate cannot be renewed and without SSL certificate would force a new request.

Avoid performing a docker volume prune or docker system prune without proper backup. They are technically not essential unless the weekly certificate request limit has been reached. To get the path to these volumes, run the terminal command docker volume inspect <volume name>.

ghost:4-alpine

This is an all-in-one image. I believe prior versions used MySQL. With Ghost v4, SQLite is now the recommended DB. This works out better for my requirement as it is easier to backup.

ports:
  - 127.0.0.1:8080:2368

I am just remapping the default port 2358 to 8080. This will be internal and not required as the Ghost service will be not directly exposed to public traffic. As mentioned earlier this will be handled by the reverse proxy where incoming requests will be directed to the matching VIRTUAL_HOST.

Rclone

docker-compose.yaml

The default directory is /root/rclone but can be changed using Terraform tfvars at runtime.

Rclone is a command-line application that enables cloud storage to be mounted on the host filesystem. I believe it supports over 30 well-known services such as Google Drive, Dropbox, AWS S3, Amazon Drive, etc. I decided to stick with Google Drive since I have 100GB of underutilized storage. I plan on incorporating Rclone in a TrueNAS SCALE self-built NAS as a redundant backup in the future.

Here are the steps to setting up Google Drive with Rclone:

• Start with this: https://rclone.org/drive/#making-your-own-client-id
  • Don't forget to submit app for verification. It will be an automatic approval. The generated auth token will not renew if the app is in development mode.
• Then follow this guide to attach the Google Drive account to Rclone: https://rclone.org/drive/

volumes:
  - ${rclone_dir}/config:/config/rclone
  - ${rclone_dir}/mount:/data:shared
  - /etc/passwd:/etc/passwd:ro
  - /etc/group:/etc/group:ro

/etc/passwd and /etc/group mounts are required for FUSE to work properly inside the container. Additionally, a premade configuration from another Rclone instance is needed. This can be done by completing a Rclone setup wizard for Google Drive. Make sure to copy the configuration to the project's folder: ghost-linode-terraform/scripts/rclone/config/

In my setup, the Google Drive configuration is called 'gdrive.' This needs to reflect in the docker-compose's command block:

command: "mount gdrive: /data"

The default mount path is /root/rclone/mount.

backup.sh

This script is responsible for creating tarballs from the ghost blog directory on the host machine. By default, the script maintains a rolling 7-day backup with latest.tgz being the last run archive.

A crontab is added through Linode's Stackscript and is set to run daily at 11PM (system time). Keep in mind that UTC is the default system time zone.

Deployment

If you have not done so, install Terraform CLI on your localhost. Follow this installation guide.

Clone project

git clone https://github.com/foureight84/ghost-linode-terraform.git && cd ghost-linode-terraform

Initialize Terraform workspace

terraform init

Create your tfvars definition file

cp terraform.tfvars.example defined.tfvars

Open defined.tfvars and fill in all required values

//project_dir = ""                                // default /root/ghost
//rclone_dir = ""                                 // default /root/rclone

linode_api_token = ""
linode_label = ""
linode_image = "linode/ubuntu20.04"               // see terraform linode provider documentation for these values
linode_region = "us-west"                         // see terraform linode provider documentation for these values
linode_type = "g6-nanode-1"                       // see terraform linode provider documentation for these values
linode_authorized_users = [""]                    // user profile created on linode with associated ssh pub key. https://cloud.linode.com/profile/keys
linode_group = "blog"
linode_tags = [ "ghost", "docker" ]

linode_root_password = ""

cloudflare_domain = ""                           // requires that your domain is already managed by cloudflare. value ex: foureight84.com
cloudflare_email = ""
cloudflare_api_key = ""                          // not to be mistaken with cf api token

letsencrypt_email = ""

ghost_blog_url = ""                              // ex. blog.foureight84.com

enable_rclone =                                  // boolean (default false). change to true if using rclone. see README.md in rclone directory on how to setup config beforehand

Double-check that everything is correct and get a deployment preview

terraform plan -var-file="defined.tfvars"

Apply Terraform changes to production

terraform apply -var-file="defined.tfvars"