However longer inputs sometimes would generate errors. Longer inputs meant longer system prompts and longer responses.

The logs showed:

Analysis failed: expected ',' or '}' after object value

Claude was generating valid-looking JSON that wasn’t actually valid. A dropped comma deep in a large response object. The longer the response, the more likely this happened.

The Old Approach: Prompt Engineering + Defensive Parsing

My system prompt included a 28-line block demanding JSON output:

SYSTEM_PROMPT = <<~PROMPT
  ...
  CRITICAL INSTRUCTIONS:
  - You MUST ALWAYS respond with valid JSON. No exceptions. No explanations outside JSON.
  - NEVER respond with plain text - always use the JSON format.

  You MUST ALWAYS respond in JSON format with the following structure (no exceptions):
  {
    "score": 0-100,
    "level": "low|medium|high",
    "summary": "Brief overall assessment",
    "items": [
      {
        "title": "Issue name",
        "description": "What's wrong",
        "severity": "low|medium|high"
      }
    ],
    "recommendations": [
      "Specific actionable suggestion 1",
      "Specific actionable suggestion 2"
    ]
  }
PROMPT

Despite all the shouting in the prompt, Claude would sometimes:

• Wrap the JSON in markdown code fences (```json ... ```)
• Add explanatory text after the closing brace
• Drop commas in deeply nested objects on long responses
• Return plain text when it decided the input wasn’t suitable for analysis

So I built a pipeline of defensive code to handle all of this.

Step 1: Extract JSON from whatever Claude returned. A brace-matching parser that stripped markdown fences, found the first {, tracked nesting depth while respecting string escaping, and separated trailing notes:

def extract_json_and_note(text)
  text = text.strip
  if text.start_with?("```")
    text = text.sub(/A```(?:json|JSON)?s*/, "").sub(/s*```z/, "").strip
  end

  start_idx = text.index("{")
  return [text, nil, false] if start_idx.nil?

  brace_count = 0
  in_string = false
  escape_next = false

  text[start_idx..].each_char.with_index do |char, idx|
    if escape_next
      escape_next = false
      next
    end
    case char
    when "\" then escape_next = true if in_string
    when '"'  then in_string = !in_string unless escape_next
    when "{"  then brace_count += 1 unless in_string
    when "}"
      brace_count -= 1 unless in_string
      if brace_count == 0
        end_idx = start_idx + idx
        json_text = text[start_idx..end_idx]
        note = text[(end_idx + 1)..].strip.presence
        return [json_text, note, true]
      end
    end
  end

  [text, nil, false]
end

Step 2: Normalize missing fields because Claude might omit arrays for edge cases:

def normalize_response!(result)
  result["score"] ||= 0
  result["level"] ||= "unknown"
  result["summary"] ||= "Analysis complete"
  result["items"] ||= []
  result["recommendations"] ||= []
  result["score"] = result["score"].to_i if result["score"].is_a?(String)
end

Step 3: Validate the structure because even after parsing, I couldn’t trust it:

def validate_response_structure!(result)
  required_keys = %w[score level summary items recommendations]
  missing_keys = required_keys - result.keys
  raise "Invalid response structure: missing keys #{missing_keys.join(', ')}" if missing_keys.any?

  score = result["score"]
  unless score.is_a?(Integer) && score >= 0 && score <= 100
    raise "Invalid score: must be integer 0-100, got #{score.inspect}"
  end

  %w[items recommendations].each do |key|
    unless result[key].is_a?(Array)
      raise "Invalid #{key}: expected array, got #{result[key].class}"
    end
  end
end

All of this existed because I was asking an LLM to format its own output as JSON via natural language instructions. I was writing a fragile parser for a format the model was never constrained to produce.

The Fix: Tool Use

Anthropic’s tool use API (also called function calling) lets you define a JSON schema that Claude must conform to. Instead of asking Claude to output JSON, you tell the API: “call this function with these typed parameters.” Claude’s response is guaranteed to match the schema.

Here’s the schema definition:

ANALYSIS_TOOL = {
  name: "analyze",
  description: "Return the structured analysis results",
  input_schema: {
    type: "object",
    required: ["score", "level", "summary", "items", "recommendations"],
    properties: {
      score: { type: "integer", description: "Overall score from 0 (safe) to 100 (dangerous)" },
      level: { type: "string", enum: ["low", "medium", "high"] },
      summary: { type: "string", description: "Brief overall assessment" },
      items: {
        type: "array",
        items: {
          type: "object",
          required: ["title", "description", "severity"],
          properties: {
            title:       { type: "string" },
            description: { type: "string" },
            severity:    { type: "string", enum: ["low", "medium", "high"] }
          }
        }
      },
      recommendations: { type: "array", items: { type: "string" } }
    }
  }
}.freeze

The API call adds two parameters:

response = client.messages.create(
  model: model,
  max_tokens: max_tokens,
  system: [{ type: "text", text: system_prompt }],
  messages: [{ role: "user", content: user_message }],
  tools: [ANALYSIS_TOOL],
  tool_choice: { type: "tool", name: "analyze" }
)

tools: defines the schema. tool_choice: with type: "tool" forces Claude to use it — no chance of returning prose instead.

Response extraction is three lines:

tool_block = response.content.find { |b| b.type.to_s == "tool_use" }
raise "No tool_use block in response" unless tool_block
result = tool_block.input.transform_keys(&:to_s)

That’s it. tool_block.input is already a parsed hash. No JSON.parse, no brace matching, no markdown stripping, no comma repair.

The Result

Deleted: ~160 lines from the service, ~250 lines from tests..

Added: ~30 lines for the schema definition, 2 parameters on the API call, 3 lines of response extraction.

The system prompt shrank too. The 28 lines of “YOU MUST RESPOND IN JSON” instructions disappeared entirely. The prompt now focuses on what to analyze, not how to format the output.

The user message went from "Analyze this and respond with JSON only:" to just "Analyze this:".

When Should You Use This?

Any time you want structured output from an LLM. If you’re writing regex to fix JSON commas, building brace-matching parsers, or adding “RESPOND IN JSON ONLY” to your prompts: switch to tool use. The schema is self-documenting, the output is guaranteed valid, and you delete code instead of writing it.

The one caveat: tool use constrains the structure but not the content. Claude can still put whatever it wants in a string field. You still need to validate that a score is in a sensible range or that enum values match your expectations. But “validate the values” is a much smaller problem than “parse arbitrary text that might be JSON.”

Stripe can send the same webhook event more than once. Network timeouts, retries, and infrastructure hiccups all mean your endpoint might process the same event twice, charging a customer double, creating duplicate subscriptions, or corrupting your data.

The fix is idempotency: making your webhook handler safe to call multiple times with the same event.

The Problem

Every Stripe event has a unique ID like evt_1abc123. Stripe guarantees this ID is unique, but your endpoint has no such guarantee about delivery. From the Stripe docs:

Webhook endpoints might occasionally receive the same event more than once. You can guard against duplicated event receipt by making your event processing idempotent.

Without protection, a retried invoice.paid event could credit a user’s account twice or trigger duplicate emails.

The Solution

Track every processed event ID in your database. Before handling an event, check if you’ve already seen it. If yes, skip it.

Step 1: Create the Table

# db/migrate/xxx_create_stripe_webhook_events.rb
class CreateStripeWebhookEvents < ActiveRecord::Migration[8.1]
  def change
    create_table :stripe_webhook_events do |t|
      t.string :stripe_event_id, null: false
      t.string :event_type
      t.datetime :processed_at
      t.timestamps
    end

    add_index :stripe_webhook_events, :stripe_event_id, unique: true
  end
end

The unique index on stripe_event_id is the key. Even if two requests arrive simultaneously, the database constraint guarantees only one INSERT succeeds.

Step 2: Create the Model

# app/models/stripe_webhook_event.rb
class StripeWebhookEvent < ApplicationRecord
  validates :stripe_event_id, presence: true, uniqueness: true

  def self.process(stripe_event_id, event_type:)
    create!(
      stripe_event_id: stripe_event_id,
      event_type: event_type,
      processed_at: Time.current
    )
    true
  rescue ActiveRecord::RecordNotUnique, ActiveRecord::RecordInvalid
    false
  end
end

process returns true if this is a new event, false if it’s a duplicate. We rescue both exceptions because:

• RecordNotUnique — the DB unique constraint catches concurrent duplicate inserts
• RecordInvalid — the model-level uniqueness validation catches sequential duplicates

Step 3: Use It in Your Webhook Handler

# app/services/stripe_webhook_handler.rb
def process!
  event = verify_signature!

  unless StripeWebhookEvent.process(event.id, event_type: event.type)
    Rails.logger.info("Skipping duplicate webhook: #{event.id}")
    return { status: "success", duplicate: true }
  end

  handle_event(event)
end

The check goes right after signature verification and before any business logic. If it’s a duplicate, we return a success response (so Stripe doesn’t keep retrying) and skip processing.

Step 4: Return 200 for Duplicates

This is important — always return a 2xx status for duplicates:

# app/controllers/webhooks_controller.rb
def stripe
  handler = StripeWebhookHandler.new(
    payload: request.body.read,
    signature: request.env["HTTP_STRIPE_SIGNATURE"]
  )

  result = handler.process!
  render json: result, status: :ok
rescue StripeWebhookHandler::WebhookError => e
  render json: { error: e.message }, status: :bad_request
end

If you return an error for a duplicate, Stripe will keep retrying — which is the opposite of what you want.

Cleanup

Over time, the stripe_webhook_events table will grow. Add a periodic cleanup job to prune old records:

# Keep 90 days of webhook history
StripeWebhookEvent.where("created_at < ?", 90.days.ago).delete_all

Stripe retries happen within hours, not months, so 90 days is more than enough.

Summary

1. Create a table with a unique index on stripe_event_id
2. Attempt an INSERT before processing — if it fails, it’s a duplicate
3. Always return 200 for duplicates so Stripe stops retrying
4. The database constraint handles race conditions that application-level checks can’t

Why I Built This

I’ve signed a enough contracts over the years, and I was never quite sure if I was missing something important buried in the legal language. I’d skim through them, but let’s be honest – I didn’t understand half of it. Getting a lawyer to review every contract is also just not something I’m going to do, unless it’s something really big.

I figured: AI is pretty good at reading and understanding text now. And based on how many of these AI contract analysis tools are out there, it’s the new TODO app in the age of AI.

The Stack

I went with a Rails 8 monolith because I wanted something I could ship quickly and maintain solo:

• Rails 8 with Hotwire (Turbo + Stimulus)
• Anthropic’s Claude API
• Solid Queue for background jobs with priority queues (paid users get faster processing)
• Solid Cache for caching and rate limits
• Stripe for subscriptions and billing
• Tesseract OCR for extracting text from scanned images (JPG, PNG, WebP, HEIC)
• Kamal for deployment

Everything runs in Docker containers. No separate frontend framework, no microservices. Just a straightforward Rails app that does one thing well.

Technical Security Challenges

1. File Processing Security
   • Magic byte validation – Don’t trust file extensions. I check the actual file signature to verify it’s really a PDF or DOCX.
   • Size limits – DOCX files are zip archives, so I enforce size limits before decompression to prevent zip bombs.
   • Immediate deletion – Original files are deleted right after text extraction. No long-term storage of sensitive documents.
   • Command injection prevention – Only use safe extraction tools, never shell out with user-provided filenames.
2. Server Access
   • Firewall at the provider level
   • Firewall at the node level (ufw)
   • ssh through certs only, limit access to specific IP’s
   • Cloudflare
3. Application Security
   • Devise authentication – Industry-standard auth framework
   • CSRF protection – Rails CSRF tokens on all POST/PUT/PATCH/DELETE requests
   • UUID-based URLs – Guest contracts use UUIDs (prevents enumeration attacks)
   • Rate Limits (Rack::Attack)
   • CSP Policy
     • No unsafe-eval – Prevents eval() attacks
     • Whitelisted script sources – Only self, HTTPS, Stripe, Cloudflare allowed
     • No object embeds – object_src :none blocks Flash/plugin attacks
     • Nonce-based scripts – Importmap scripts use session-based nonces
     • HTTPS enforced – All resources loaded over HTTPS
   • Input Validation
4. Fraud Prevention
   • Email history tracking – SHA256 email hashing – Email hashes stored, not plain emails
5. Payment Security
   • Stripe webhook verification – Signature validation on all webhook events
   • No card storage – Stripe handles all payment details
6. Secret Management
   • Rails credentials – All secrets in encrypted credentials.yml.enc
7. XSS Prevention
   • Automatic HTML escaping
   • CSP headers – Content Security Policy blocks inline scripts
8. Transport Security
   • HTTPS everywhere – All resources loaded over HTTPS
   • Secure cookies – Session cookies marked secure in production
   • HSTS headers – Forces HTTPS connections
9. DoS Prevention
   • Job queues – Background processing prevents request timeouts
   • Priority queues – Paid users get separate high-priority queue
   • Rate limiting – Comprehensive rate limits across all endpoints
   • Query optimization – Indexed queries prevent slow lookups

Try it

Demo (no signup): https://clausyapp.com/contracts/new?demo=hn

Full app: https://clausyapp.com

It’s not legal advice – I’m very explicit about that – but it can help you spot things you might want to ask a lawyer about.

The Error That Nearly Broke Our Deployment

Three hours into our Kamal deployment, we were stuck in a loop:

ERROR Failed to boot web on {ip_address}
  INFO First web container is unhealthy on {ip_address}, not booting any other roles

The container would start, but Kamal’s health check kept failing. After 30 seconds, Kamal would kill the container
and retry, creating an endless loop.

We spent hours debugging deployment scripts, PostgreSQL configurations, and Rails settings. The fix turned out to
be much simpler: DNS configuration.

The Root Cause: Broken DNS Resolution

What Was Happening

When Kamal tried to verify container health, it performed this sequence:

1. Container starts → my_app-web-abc123 boots
2. Traefik (Kamal proxy) tries to check /up endpoint
3. DNS lookup → Resolve my_app-web-abc123 to an IP address
4. Health check fails → DNS resolution times out or fails
5. Container killed → Kamal marks it as unhealthy

The DNS Failure

The Traefik container’s /etc/resolv.conf showed:

nameserver 127.0.0.53
  search members.linode.com
  options edns0 trust-ad ndots:0

Problem: 127.0.0.53 is the host’s systemd-resolved DNS server. It’s not accessible from inside
Docker containers!

When Traefik tried to resolve my_app-web-abc123:

• It queried 127.0.0.53 (systemd-resolved)
• The query failed with “connection refused”
• Health check failed
• Container was killed

The Solution: Proper Docker DNS Configuration

What We Fixed

We configured Docker’s DNS settings in /etc/docker/daemon.json:

{
    "dns": ["127.0.0.11", "8.8.8.8", "1.1.1.1"]
  }

Why This Works

1. 127.0.0.11 (Docker’s Internal DNS) – First Priority

• Resolves container hostnames automatically
• Handles inter-container communication
• Always available inside Docker networks

2. 8.8.8.8 (Google DNS) – Second Priority

• Resolves external domains (APIs, gems, etc.)
• Fast and reliable
• Global infrastructure

3. 1.1.1.1 (Cloudflare DNS) – Third Priority

• Privacy-focused external DNS
• Backup if 8.8.8.8 fails
• No query logging

How Docker Uses This

Docker’s DNS resolution order:

1. Try 127.0.0.11 (internal) → container names
2. If that fails → 8.8.8.8 (external) → domains
3. If that fails → 1.1.1.1 (external) → domains

The IPv4/IPv6 Issue

While debugging, we discovered another subtle problem:

The IPv6 Trap

The server setup script used:

SERVER_IP=$(curl -s ifconfig.me || echo "ip_address_goes_here")

Problem: ifconfig.me returned an IPv6 address:

2600:3c03::...

This IPv6 address was used in PostgreSQL’s pg_hba.conf:

host my_app_production my_app_user 2600:3c03.../32 md5

PostgreSQL had issues with this IPv6 address, causing authentication failures.

The Fix

Force IPv4 detection:

SERVER_IP=$(curl -s -4 ifconfig.me || echo "ip_address_goes_here")

The -4 flag ensures we always get an IPv4 address, which PostgreSQL handles reliably.

The PostgreSQL Network Isolation Issue

The Problem

Kamal uses a separate Docker network (172.18.0.0/16) for containers, while PostgreSQL is on the host’s Docker
bridge network (172.17.0.0/16).

The firewall only allowed 172.17.0.0/16:

5432/tcp  ALLOW  172.17.0.0/16

The Fix

Add the Kamal network to both firewall and PostgreSQL config:

Firewall (ufw):

sudo ufw allow from 172.18.0.0/16 to any port 5432

PostgreSQL (pg_hba.conf):

host my_app_production my_app_user 172.18.0.0/16 md5

Complete Fix in our setup script

IPv4 Fix

SERVER_IP=$(curl -s -4 ifconfig.me || echo "ip_address_goes_here")

Kamal Network Firewall Rule

sudo ufw allow from 172.18.0.0/16 to any port 5432

PostgreSQL Kamal Network Rule

host $DB_NAME $DB_USER 172.18.0.0/16 md5

Docker DNS Configuration

{
    "dns": ["127.0.0.11", "8.8.8.8", "1.1.1.1"]
  }

Key Takeaways

1. DNS is Critical for Container Orchestration
   • Always configure Docker’s DNS properly
   • Include both internal and external DNS servers
   • Test DNS resolution from containers
2. Network Isolation Matters
   • Docker networks are isolated by default
   • PostgreSQL must allow connections from all Docker networks
   • Firewall rules must match
3. IPv4 vs IPv6 Can Break Things
   • PostgreSQL works better with IPv4
   • Force IPv4 when detecting server IPs
   • Test both IPv4 and IPv6 connectivity
4. Health Checks are Essential
   • The /up endpoint is critical for Kamal
   • DNS must work for health checks to succeed
   • Timeout settings matter (30s default)

Troubleshooting DNS Issues

If you encounter “First web container is unhealthy”:

1. Check Container Logs
   docker logs my_app-web-abc123
2. Check Traefik/Kamal Proxy Logs
   docker logs kamal-proxy | grep -i healthcheck
3. Test DNS Resolution

   # From inside Traefik container
     docker exec kamal-proxy getent hosts my_app-web-abc123
     docker exec kamal-proxy getent hosts google.com

4. Verify DNS Configuration

   # Check daemon.json
     cat /etc/docker/daemon.json
   
     # Check container's resolv.conf
     docker exec kamal-proxy cat /etc/resolv.conf

5. Check PostgreSQL Connectivity

   # From kamal network
     docker run --rm --network kamal postgres:16 psql \
       -h 172.17.0.1 -U my_app_user -d my_app_production -c "SELECT 1"

Results

After implementing all fixes:

• DNS resolution works (internal and external)
• Health checks pass (Traefik can reach containers)
• PostgreSQL connections work (from both Docker networks)
• Deployments succeed (consistent, reliable)
• IPv4 detection works (no IPv6 issues)

Final Thoughts

The “First web container is unhealthy” error can be a DNS configuration issue, not a deployment or application
problem.

By understanding how Docker networks work, how DNS resolution functions, and how PostgreSQL authentication works, we can prevent this issue from ever occurring again.

Key files to review:

• /etc/docker/daemon.json – Docker DNS configuration
• /etc/postgresql/16/main/pg_hba.conf – PostgreSQL authentication
• /etc/ufw/rules.conf – Firewall rules

The fix is now automated in our setup script, ensuring new servers have proper DNS and network configuration from
day one.

PG::DatatypeMismatch: ERROR:  column "blah" cannot be cast automatically to type boolean
HINT:  You might need to specify "USING blah::boolean".

This just tells you that you need a rule to convert your string to boolean. You can fix with using synthax. For example, if you want all columns to change to false:

change_table :table_name do |t|
  t.change :column_name, :boolean, using: 'false', default: false, null: false
end

Or if you want to convert your existing values from your column, you could do something like this:

change_table :table_name do |t|
  t.change :column_name, :boolean, using: 'cast(column_name as boolean)', default: false, null: false
end

Let’s assume your database default timezone is UTC (can check it with show timezone;). Thus, we are making the assumption that your dates are stored in UTC.

Since timestamp does not have any timezone data, we first need to read it in UTC, so it adds the timezone data. Then we can convert it to the desired timezone:

select timestamp_field AT TIME ZONE 'UTC' AT TIME ZONE 'America/New_York'

Now, we have the timezone and proper daylight-saving offset.

Just for fun, let’s check if the current hour matches the hour from the saved timestamp field in the ‘America/New_York’ timezone:

EXTRACT ( HOUR FROM timestamp_field at time zone 'UTC' at time zone 'America/New_York') = EXTRACT ( HOUR FROM now() at time zone 'America/New_York')

Notice, that we do not read now() in UTC timezone first, because our DB default timezone is UTC and now() already has all of the timezone data.

Somewhat unsurprisingly, given Oracle reputation, they disabled my supposedly always free resources two months later. Turns out, they disable the resources created while you were on trial. However, you can still make a clone of your boot volume, then terminate your previous instance, then recreate your instance from the cloned volume. It works so far.

In this tutorial, we’ll set up a free Minecraft server using a free tier Oracle Cloud.

Setting up an Oracle Free Tier

Signup for Oracle free account. Since this is Oracle, you probably want to check on the billing from time to time to make sure they didn’t start your changing for what they promised to be free.

Create a new compute instance. For max compatibility, we’ll use Ubuntu 18.04. You can try your luck with a newer version.

For the shape, select Ampere with 4 OCPU and 6GB Ram. This should still qualify for the free tier and should still be plenty for a Minecraft server.

Add your public SSH keys in the “Add SSH keys” section.

Create the instance.

After it’s created, go to the instance details, find “Primary VNIC” section, and open the subnet link (or create a new one).

Open Default Security List (or create a new one if one doesn’t exist yet)

Add Ingress Rules to open TCP/UDP ports 19132 for Bedrock and 25565 for Java edition (or both). Use CIDR for Source Type, 0.0.0.0/0 for Source CIDR, 19132 for Destination port. Repeat for TCP. Repeat for 25565 if planning to use Java edition.

SSH to your server

Upgrade all packages:

sudo apt-get update
sudo apt-get upgrade

Let’s reset the firewall rules and open the ssh and Minecraft ports:

sudo iptables -P INPUT ACCEPT
sudo iptables -P FORWARD ACCEPT
sudo iptables -P OUTPUT ACCEPT
sudo iptables -F
sudo iptables-save
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp
sudo ufw allow 19132/udp
sudo ufw allow 19132/tcp
sudo ufw allow 25565/udp
sudo ufw allow 25565/tcp
sudo ufw enable
sudo ufw status

Optionally install zsh and vim:

sudo apt install zsh
sudo apt install vim

Bedrock

SSH to your server: ssh ubuntu@ip_address

We’ll use this script to setup the server: https://github.com/TheRemote/MinecraftBedrockServer

curl https://raw.githubusercontent.com/TheRemote/MinecraftBedrockServer/master/SetupMinecraft.sh | bash

Use the following commands to start/restart the service (you specified the service name when you ran the installation script):

sudo systemctl stop minecraft_service
sudo systemctl start minecraft_service
sudo systemctl restart minecraft_service

screen -R
# To disconnect (do not do ctrl-c or it may kill the minecraft service):
ctrl-A-D

Java Edition

This section is only for Java edition. Don’t use it if you need the Bedrock version.

We’ll use this script to install the Java edition Paper server: https://github.com/TheRemote/RaspberryPiMinecraft.

SSH to your server and run curl https://raw.githubusercontent.com/TheRemote/RaspberryPiMinecraft/master/SetupMinecraft.sh | bash. All default settings should be fine.

If you want Bedrock users to use your server, we can install Geyser plugin:

We’ll download the latest Geyser-Spigot server from https://ci.opencollab.dev/job/GeyserMC/job/Floodgate/job/master/.

On the server run:

cd minecraft/plugins/
curl -O https://ci.opencollab.dev/job/GeyserMC/job/Geyser/job/master/lastSuccessfulBuild/artifact/bootstrap/spigot/target/Geyser-Spigot.jar

If you also want Bedrock users to be able to login with their Microsoft account without requiring a separate Java account, we can also install a floodgate plugin:

curl -O https://ci.opencollab.dev/job/GeyserMC/job/Floodgate/job/master/lastSuccessfulBuild/artifact/spigot/build/libs/floodgate-spigot.jar

Restart the Minecraft server with sudo systemctl restart minecraft.service

DNS Records with Cloudflare

Add an A record for your server IP. For example, if you own example.com and want to connect to your server using mct.example.com, then add an A record for mct pointing to your server IP.

Add 2 srv records for each port (19132 and 25565):

Use mct for Name, _minecraft for service, 0 for both Priority and Weight, UDP for protocol, mct.example.com for the target. mct is just the subdomain that you can change to whatever you want. Repeat for TCP and then for each port (19132 and 25565)

Acknowledgments: shout out to Steven Jones at Syncro for helping me better understand how Foreign Keys work in PostgreSQL.

Foreign keys are great for maintaining data integrity. However, they are not free.

As mentioned in this blog post by Shaun Thomas:

In PostgreSQL, every foreign key is maintained with an invisible system-level trigger added to the source table in the reference. At least one trigger must go here, as operations that modify the source data must be checked that they do not violate the constraint.

https://bonesmoses.org/2014/05/14/foreign-keys-are-not-free/

What this means is that when you modify your parent table, even if you don’t touch any of the referred keys, these triggers are still fired.

In the original post from 2014, the overhead made the updates up to 95% slower with 20 foreign keys. Let’s see if things changed since then.

These are slightly updated scripts we’ll use for testing

CREATE OR REPLACE FUNCTION fnc_create_check_fk_overhead(key_count INT)
RETURNS VOID AS
$$
DECLARE
  i INT;
BEGIN
  CREATE TABLE test_fk
  (
    id   BIGINT PRIMARY KEY,
    junk VARCHAR
  );

  INSERT INTO test_fk
  SELECT generate_series(1, 100000), repeat(' ', 20);

  CLUSTER test_fk_pkey ON test_fk;

  FOR i IN 1..key_count LOOP
    EXECUTE 'CREATE TABLE test_fk_ref_' || i || 
            ' (test_fk_id BIGINT REFERENCES test_fk (id))';
						
  END LOOP;

END;
$$ LANGUAGE plpgsql VOLATILE;


CREATE OR REPLACE FUNCTION fnc_check_fk_overhead(key_count INT)
RETURNS VOID AS
$$
DECLARE
  i INT;
BEGIN
  FOR i IN 1..100000 LOOP
    UPDATE test_fk SET junk = '    blah                '
     WHERE id = i;
  END LOOP;

END;
$$ LANGUAGE plpgsql VOLATILE;


CREATE OR REPLACE FUNCTION clean_up_overhead(key_count INT)
RETURNS VOID AS
$$
DECLARE
  i INT;
BEGIN
  DROP TABLE test_fk CASCADE;

  FOR i IN 1..key_count LOOP
    EXECUTE 'DROP TABLE test_fk_ref_' || i;
  END LOOP;
END;
$$ LANGUAGE plpgsql VOLATILE;

To validate that the overhead is caused strictly by the presence of the foreign keys, and not from the cost of looking up the child records, after the first benchmark, we’ll modify the first function and add indexes on each foreign key:

CREATE OR REPLACE FUNCTION fnc_create_check_fk_overhead(key_count INT)
RETURNS VOID AS
$$
DECLARE
  i INT;
BEGIN
  CREATE TABLE test_fk
  (
    id   BIGINT PRIMARY KEY,
    junk VARCHAR
  );

  INSERT INTO test_fk
  SELECT generate_series(1, 100000), repeat(' ', 20);

  CLUSTER test_fk_pkey ON test_fk;

  FOR i IN 1..key_count LOOP
    EXECUTE 'CREATE TABLE test_fk_ref_' || i || 
            ' (test_fk_id BIGINT REFERENCES test_fk (id))';
						
		EXECUTE 'CREATE index test_fk_ref_index_' || i ||
            ' on test_fk_ref_' || i || '(test_fk_id)';
  END LOOP;

END;
$$ LANGUAGE plpgsql VOLATILE;

We’ll run on on Mac i9, 2.3 GHz 8-Core, 64 GB Ram

PostgreSQL version 12

-- without an index

select fnc_create_check_fk_overhead(0);
SELECT fnc_check_fk_overhead(0); -- 2.6-2.829
select clean_up_overhead(0)


select fnc_create_check_fk_overhead(20);
SELECT fnc_check_fk_overhead(20); -- 3.186-3.5. ~20% drop
select clean_up_overhead(20)


-- after updating our initial function to add an index for each foreign key:
select fnc_create_check_fk_overhead(0);
SELECT fnc_check_fk_overhead(0); -- 2.6-2.8
select clean_up_overhead(0)

select fnc_create_check_fk_overhead(20);
SELECT fnc_check_fk_overhead(20); -- 3.1 same ~20% drop
select clean_up_overhead(20)

As we see from the benchmark, the drop in update performance on a parent table is about 20% after adding 20 tables with a foreign key to the parent. It’s not quite as bad as 95% in the original post, but the overhead is still clearly there.

Start with standard docker-compose file. If using custom SQL mode, specify the necessary options in the command options:

version: "3.7"
services:
    mysql:
        build:
            context: .
            dockerfile: dev.dockerfile
        restart: always
        command: --sql_mode="STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE"
        environment:
            MYSQL_ROOT_PASSWORD: root_password
            MYSQL_DATABASE: dev
            MYSQL_USER: dev_user
            MYSQL_PASSWORD: dev_password
        ports:
            - 3306:3306

Add dev.dockerfile:

FROM mysql:8.0.17

ADD init.sql /docker-entrypoint-initdb.d/ddl.sql

Finally, add your init.sql file. Let’s give all privileges to our dev_user and switch the default caching_sha2_password to mysql_native_password (don’t do it unless you rely on older packages that require the less secure au

Finally, add your init.sql file. Let’s give all privileges to our dev_user and switch the default caching_sha2_password to mysql_native_password (don’t do it unless you rely on older packages that require the less secure mysql_native_password authentication method):

GRANT ALL PRIVILEGES ON *.* TO 'dev_user'@'%';
ALTER USER 'dev_user'@'%' IDENTIFIED WITH mysql_native_password BY 'dev_password';

If you want to access the database container from other containers, while running them separately, you can specify host.docker.internal as the host address of your database.