The Error That Nearly Broke Our Deployment

Three hours into our Kamal deployment, we were stuck in a loop:

ERROR Failed to boot web on {ip_address}
  INFO First web container is unhealthy on {ip_address}, not booting any other roles

The container would start, but Kamal’s health check kept failing. After 30 seconds, Kamal would kill the container
and retry, creating an endless loop.

We spent hours debugging deployment scripts, PostgreSQL configurations, and Rails settings. The fix turned out to
be much simpler: DNS configuration.

The Root Cause: Broken DNS Resolution

What Was Happening

When Kamal tried to verify container health, it performed this sequence:

1. Container starts → my_app-web-abc123 boots
2. Traefik (Kamal proxy) tries to check /up endpoint
3. DNS lookup → Resolve my_app-web-abc123 to an IP address
4. Health check fails → DNS resolution times out or fails
5. Container killed → Kamal marks it as unhealthy

The DNS Failure

The Traefik container’s /etc/resolv.conf showed:

nameserver 127.0.0.53
  search members.linode.com
  options edns0 trust-ad ndots:0

Problem: 127.0.0.53 is the host’s systemd-resolved DNS server. It’s not accessible from inside
Docker containers!

When Traefik tried to resolve my_app-web-abc123:

• It queried 127.0.0.53 (systemd-resolved)
• The query failed with “connection refused”
• Health check failed
• Container was killed

The Solution: Proper Docker DNS Configuration

What We Fixed

We configured Docker’s DNS settings in /etc/docker/daemon.json:

{
    "dns": ["127.0.0.11", "8.8.8.8", "1.1.1.1"]
  }

Why This Works

1. 127.0.0.11 (Docker’s Internal DNS) – First Priority

• Resolves container hostnames automatically
• Handles inter-container communication
• Always available inside Docker networks

2. 8.8.8.8 (Google DNS) – Second Priority

• Resolves external domains (APIs, gems, etc.)
• Fast and reliable
• Global infrastructure

3. 1.1.1.1 (Cloudflare DNS) – Third Priority

• Privacy-focused external DNS
• Backup if 8.8.8.8 fails
• No query logging

How Docker Uses This

Docker’s DNS resolution order:

1. Try 127.0.0.11 (internal) → container names
2. If that fails → 8.8.8.8 (external) → domains
3. If that fails → 1.1.1.1 (external) → domains

The IPv4/IPv6 Issue

While debugging, we discovered another subtle problem:

The IPv6 Trap

The server setup script used:

SERVER_IP=$(curl -s ifconfig.me || echo "ip_address_goes_here")

Problem: ifconfig.me returned an IPv6 address:

2600:3c03::...

This IPv6 address was used in PostgreSQL’s pg_hba.conf:

host my_app_production my_app_user 2600:3c03.../32 md5

PostgreSQL had issues with this IPv6 address, causing authentication failures.

The Fix

Force IPv4 detection:

SERVER_IP=$(curl -s -4 ifconfig.me || echo "ip_address_goes_here")

The -4 flag ensures we always get an IPv4 address, which PostgreSQL handles reliably.

The PostgreSQL Network Isolation Issue

The Problem

Kamal uses a separate Docker network (172.18.0.0/16) for containers, while PostgreSQL is on the host’s Docker
bridge network (172.17.0.0/16).

The firewall only allowed 172.17.0.0/16:

5432/tcp  ALLOW  172.17.0.0/16

The Fix

Add the Kamal network to both firewall and PostgreSQL config:

Firewall (ufw):

sudo ufw allow from 172.18.0.0/16 to any port 5432

PostgreSQL (pg_hba.conf):

host my_app_production my_app_user 172.18.0.0/16 md5

Complete Fix in our setup script

IPv4 Fix

SERVER_IP=$(curl -s -4 ifconfig.me || echo "ip_address_goes_here")

Kamal Network Firewall Rule

sudo ufw allow from 172.18.0.0/16 to any port 5432

PostgreSQL Kamal Network Rule

host $DB_NAME $DB_USER 172.18.0.0/16 md5

Docker DNS Configuration

{
    "dns": ["127.0.0.11", "8.8.8.8", "1.1.1.1"]
  }

Key Takeaways

1. DNS is Critical for Container Orchestration
   • Always configure Docker’s DNS properly
   • Include both internal and external DNS servers
   • Test DNS resolution from containers
2. Network Isolation Matters
   • Docker networks are isolated by default
   • PostgreSQL must allow connections from all Docker networks
   • Firewall rules must match
3. IPv4 vs IPv6 Can Break Things
   • PostgreSQL works better with IPv4
   • Force IPv4 when detecting server IPs
   • Test both IPv4 and IPv6 connectivity
4. Health Checks are Essential
   • The /up endpoint is critical for Kamal
   • DNS must work for health checks to succeed
   • Timeout settings matter (30s default)

Troubleshooting DNS Issues

If you encounter “First web container is unhealthy”:

1. Check Container Logs
   docker logs my_app-web-abc123
2. Check Traefik/Kamal Proxy Logs
   docker logs kamal-proxy | grep -i healthcheck
3. Test DNS Resolution

   # From inside Traefik container
     docker exec kamal-proxy getent hosts my_app-web-abc123
     docker exec kamal-proxy getent hosts google.com

4. Verify DNS Configuration

   # Check daemon.json
     cat /etc/docker/daemon.json
   
     # Check container's resolv.conf
     docker exec kamal-proxy cat /etc/resolv.conf

5. Check PostgreSQL Connectivity

   # From kamal network
     docker run --rm --network kamal postgres:16 psql \
       -h 172.17.0.1 -U my_app_user -d my_app_production -c "SELECT 1"

Results

After implementing all fixes:

• DNS resolution works (internal and external)
• Health checks pass (Traefik can reach containers)
• PostgreSQL connections work (from both Docker networks)
• Deployments succeed (consistent, reliable)
• IPv4 detection works (no IPv6 issues)

Final Thoughts

The “First web container is unhealthy” error can be a DNS configuration issue, not a deployment or application
problem.

By understanding how Docker networks work, how DNS resolution functions, and how PostgreSQL authentication works, we can prevent this issue from ever occurring again.

Key files to review:

• /etc/docker/daemon.json – Docker DNS configuration
• /etc/postgresql/16/main/pg_hba.conf – PostgreSQL authentication
• /etc/ufw/rules.conf – Firewall rules

The fix is now automated in our setup script, ensuring new servers have proper DNS and network configuration from
day one.

Somewhat unsurprisingly, given Oracle reputation, they disabled my supposedly always free resources two months later. Turns out, they disable the resources created while you were on trial. However, you can still make a clone of your boot volume, then terminate your previous instance, then recreate your instance from the cloned volume. It works so far.

In this tutorial, we’ll set up a free Minecraft server using a free tier Oracle Cloud.

Setting up an Oracle Free Tier

Signup for Oracle free account. Since this is Oracle, you probably want to check on the billing from time to time to make sure they didn’t start your changing for what they promised to be free.

Create a new compute instance. For max compatibility, we’ll use Ubuntu 18.04. You can try your luck with a newer version.

For the shape, select Ampere with 4 OCPU and 6GB Ram. This should still qualify for the free tier and should still be plenty for a Minecraft server.

Add your public SSH keys in the “Add SSH keys” section.

Create the instance.

After it’s created, go to the instance details, find “Primary VNIC” section, and open the subnet link (or create a new one).

Open Default Security List (or create a new one if one doesn’t exist yet)

Add Ingress Rules to open TCP/UDP ports 19132 for Bedrock and 25565 for Java edition (or both). Use CIDR for Source Type, 0.0.0.0/0 for Source CIDR, 19132 for Destination port. Repeat for TCP. Repeat for 25565 if planning to use Java edition.

SSH to your server

Upgrade all packages:

sudo apt-get update
sudo apt-get upgrade

Let’s reset the firewall rules and open the ssh and Minecraft ports:

sudo iptables -P INPUT ACCEPT
sudo iptables -P FORWARD ACCEPT
sudo iptables -P OUTPUT ACCEPT
sudo iptables -F
sudo iptables-save
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp
sudo ufw allow 19132/udp
sudo ufw allow 19132/tcp
sudo ufw allow 25565/udp
sudo ufw allow 25565/tcp
sudo ufw enable
sudo ufw status

Optionally install zsh and vim:

sudo apt install zsh
sudo apt install vim

Bedrock

SSH to your server: ssh ubuntu@ip_address

We’ll use this script to setup the server: https://github.com/TheRemote/MinecraftBedrockServer

curl https://raw.githubusercontent.com/TheRemote/MinecraftBedrockServer/master/SetupMinecraft.sh | bash

Use the following commands to start/restart the service (you specified the service name when you ran the installation script):

sudo systemctl stop minecraft_service
sudo systemctl start minecraft_service
sudo systemctl restart minecraft_service

screen -R
# To disconnect (do not do ctrl-c or it may kill the minecraft service):
ctrl-A-D

Java Edition

This section is only for Java edition. Don’t use it if you need the Bedrock version.

We’ll use this script to install the Java edition Paper server: https://github.com/TheRemote/RaspberryPiMinecraft.

SSH to your server and run curl https://raw.githubusercontent.com/TheRemote/RaspberryPiMinecraft/master/SetupMinecraft.sh | bash. All default settings should be fine.

If you want Bedrock users to use your server, we can install Geyser plugin:

We’ll download the latest Geyser-Spigot server from https://ci.opencollab.dev/job/GeyserMC/job/Floodgate/job/master/.

On the server run:

cd minecraft/plugins/
curl -O https://ci.opencollab.dev/job/GeyserMC/job/Geyser/job/master/lastSuccessfulBuild/artifact/bootstrap/spigot/target/Geyser-Spigot.jar

If you also want Bedrock users to be able to login with their Microsoft account without requiring a separate Java account, we can also install a floodgate plugin:

curl -O https://ci.opencollab.dev/job/GeyserMC/job/Floodgate/job/master/lastSuccessfulBuild/artifact/spigot/build/libs/floodgate-spigot.jar

Restart the Minecraft server with sudo systemctl restart minecraft.service

DNS Records with Cloudflare

Add an A record for your server IP. For example, if you own example.com and want to connect to your server using mct.example.com, then add an A record for mct pointing to your server IP.

Add 2 srv records for each port (19132 and 25565):

Use mct for Name, _minecraft for service, 0 for both Priority and Weight, UDP for protocol, mct.example.com for the target. mct is just the subdomain that you can change to whatever you want. Repeat for TCP and then for each port (19132 and 25565)

sudo ncdu /

Clean up unused stuff:

sudo apt-get clean
sudo apt-get autoclean
sudo apt-get autoremove

clean: clean clears out the local repository of retrieved package files. It removes everything but the lock file from /var/cache/apt/archives/ and /var/cache/apt/archives/partial/. When APT is used as a dselect(1) method, clean is run automatically. Those who do not use dselect will likely want to run apt-get clean from time to time to free up disk space.

autoclean: Like clean, autoclean clears out the local repository of retrieved package files. The difference is that it only removes package files that can no longer be downloaded, and are largely useless. This allows a cache to be maintained over a long period without it growing out of control. The configuration option APT::Clean-Installed will prevent installed packages from being erased if it is set to off.

autoremove: is used to remove packages that were automatically installed to satisfy dependencies for some package and that are no more needed.

See a related question on askubuntu: https://askubuntu.com/questions/3167/what-is-difference-between-the-options-autoclean-autoremove-and-clean

File: /etc/nginx/sites-enabled/default

server {
        listen 80;
        listen [::]:80;

        server_name domain.com;

        location / {
                root /var/www/html/your_site;
                try_files $uri /index.html;
        }
}

server {
  listen 80;
  listen [::]:80;

  server_name subdomain.domain.com;

  location / {
     proxy_redirect off;
     proxy_pass http://127.0.0.1:4000/;
  }
}

Start with regular updates

apt-get update && apt-get upgrade

Set up fail2ban and Firewall

I’m installing fail2ban 0.10 since it supports ipv6. At the time of this post, it is not available as a regular package.

wget https://github.com/fail2ban/fail2ban/archive/0.10.0.tar.gz
tar -xvzf 0.10.0.tar.gz
python3 setup.py install

#To enable fail2ban as an automatic service, copy the script for your distro from the files directory to /etc/init.d.

cp files/debian-initd /etc/init.d/fail2ban
update-rc.d fail2ban defaults
service fail2ban start

#Add local jail
awk '{ printf "# "; print; }' /etc/fail2ban/jail.conf | sudo tee /etc/fail2ban/jail.local
vim /etc/fail2ban/jail.local

uncomment sshd section and add
enabled = true

sudo apt-get install sendmail iptables-persistent
sudo service fail2ban start

Firewall ( allow established connections, traffic generated by the server itself, traffic destined for our SSH and web server ports. We will drop all other traffic):

sudo service fail2ban stop
sudo iptables -A INPUT -i lo -j ACCEPT
sudo iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 22 -j ACCEPT
sudo iptables -A INPUT -p tcp -m multiport --dports 80,443 -j ACCEPT
sudo iptables -A INPUT -j DROP

# easy way to rate-limit ssh with ufw:
# technically, we could do all of the iptables stuff with ufw
ufw enable
ufw limit ssh

If using IPv6:

ip6tables -A INPUT -p tcp --dport 80 -j ACCEPT
ip6tables -A INPUT -p tcp --dport 443 -j ACCEPT
ip6tables -A INPUT -p tcp --dport 22 -j ACCEPT # (replace with your undisclosed port)
ip6tables -A INPUT -p icmpv6 -j ACCEPT
ip6tables -A INPUT -j REJECT
ip6tables -A FORWARD -j REJECT

View iptables rules:

sudo iptables -S

Save iptables rules:

sudo dpkg-reconfigure iptables-persistent
sudo service fail2ban start

SSH

vim /etc/ssh/sshd_config

#Add or uncomment (if using Ubuntu < 17.04)
protocol 2

#Add allowed ciphers
Ciphers aes128-ctr,aes192-ctr,aes256-ctr
KexAlgorithms ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256
MACs hmac-sha2-256,hmac-sha2-512

Restart and test ssh config:

service sshd restart
#returns nothing if everything configured properly
sshd -t

NGINX

sudo apt-get install software-properties-common
sudo add-apt-repository ppa:nginx/stable
sudo apt-get install nginx
service nginx status

Update /etc/nginx/sites-enabled/default

root /var/www/html/your_site;

location / {
# Some comments...
try_files $uri /index.html;   # ADD THIS
}

sudo service nginx restart

Installing React Dependencies

# install yarn
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
sudo apt-get update && sudo apt-get install yarn
 
#install node (apt-get repo has an older version of Node)
curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs
sudo apt-get install -y build-essential

GIT Push Deploy

Let’s set up git on the server

apt-get install git-core
mkdir repos && cd repos
mkdir your_site.git
cd your_site.git
git init --bare

Set up a post-push hook
cd /repos/your_app.git/hooks
touch post-receive

#!/bin/bash -l

GIT_REPO=$HOME/repos/your_app.git
TMP_GIT_CLONE=$HOME/tmp/git/your_app
PUBLIC_WWW=/var/www/html

git clone $GIT_REPO $TMP_GIT_CLONE
cd $TMP_GIT_CLONE
yarn install
yarn build
rm -rf $PUBLIC_WWW/your_app_bup
mv $PUBLIC_WWW/your_app $PUBLIC_WWW/your_app_bup
cp -a build/. $PUBLIC_WWW/your_app
rm -Rf $TMP_GIT_CLONE
exit

Run on post-receiv:

chmod +x post-receive

On your local machine:

git remote add linode root@remote_server_address:repos/your_app.git
git push linode master