Skip to content

skoenig/mysql-replica-http-healthcheck

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

36 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

MySQL Replica HTTP Healthcheck

This is a simple HTTP-based MySQL replica health check, consisting of a bash script, a systemd socket and a companion service. The check not only shows whether the database service is running or not, but also how much replication is lagging behind.

Motivation

HTTP as the protocol for this health check is used to provide rich status information beyond simple connectivity checks, allowing validation of replication lag.

The health check is implemented using systemd socket activation, to provide a simple way to serve HTTP requests. This approach conserves system resources by starting the service only when needed, while systemd handles all low-level socket management. For some background on socket activation, see:

The healthcheck script is also checking for the existence of a flag file to signal maintenance mode, e.g. when a replica needs to be replaced or updated: touch /var/tmp/replica-maintenance.

Build VM Image

As prerequisites, you need Hashicorp Packer and QEMU/KVM. This will allow you to build the image from the latest Debian locally:

packer build -var headless=false mysql.pkr.hcl

To base the build on a specific Debian version and name the resulting image like in the GitHub Actions pipeline, adapt the env vars in .env and run:

source .env
packer build -var headless=false $(scripts/create-tag.sh) .

Demo Scenario

If you want to run the Percona MySQL distribution on GCP, you need to build your own image and provision VMs with it because this flavor is not available as a managed service like Cloud SQL.

The Terraform manifests in infra/ create the infrastructure for this scenario within a Google Cloud Platform (GCP) project. They set up an internal application load balancer that polls port 9876 of the MySQL replica instance group to determine their health based on the returned HTTP status code.

To deploy this demo:

  1. Install Terraform or OpenTofu CLI.
  2. Create a infra/.env file with your BILLING_ACCOUNT_ID, REGION, and ZONE.
  3. Spin up the demo project: cd infra/ && ./setup.sh.
  4. Acquire your GCP credentials: gcloud auth application-default login
  5. Initialize / Plan / Apply Terraform: cd infra/ && terraform init && terraform plan && terraform apply.
  6. After the deployment completes, Terraform will output the load balancer's IP address (lb_ip) and DNS name (lb_dns).
  7. You can then access the MySQL replicas through the load balancer's IP address on port 3306 from within your VPC, or resolve the DNS name from within the VPC.

Optional - cleanup: cd infra/ && ./teardown.sh <PROJECT_ID>

About

health checking MySQL replicas over HTTP

Resources

Stars

Watchers

Forks

Packages

No packages published

Contributors 2

  •  
  •