The Deployment That Takes Down Production
You’ve been there. A Friday afternoon database migration. A run of ALTER TABLE that takes minutes — locking the table, queueing up requests, and eventually timing out your entire application. By the time you roll back, you’ve already lost customers.
Database deployments are the single riskiest operation in most SMB infrastructures. Unlike stateless application code, databases carry state. A failed migration can corrupt data, lock tables for hours, or force a restore from backup — and if your backups haven’t been tested recently, that restore might fail too.
Yet most SMBs treat database migrations as an afterthought. Code changes go through CI/CD reviews and staging environments, but schema changes are still applied manually via mysql or psql on a production console.
In this post, we’ll show you how to build a zero-downtime database deployment pipeline — even if you’re running on a single server with limited resources.
Why Traditional Database Migrations Fail
The root cause of most database deployment failures is a mismatch between how applications are deployed and how databases work:
| App Deployment | Database Migration |
|---|---|
| Immutable — old version replaced instantly | Mutable — data must be transformed in-place |
| Rollback means redeploying previous version | Rollback means reversing schema and data changes |
| Stateless — new instances can be created | Stateful — schema is shared across all instances |
| Canary deployments reduce risk | Schema change applies to entire database at once |
For SMBs, the problem is compounded by:
- Limited staging environments — often a smaller copy of production that doesn’t catch data-volume-related issues
- No dedicated DBA — the same engineer writing application code writes migrations
- Single-server deployments — no read replicas to absorb traffic during schema changes
- Tight deployment windows — migrations are squeezed into the same change window as application updates
The Zero-Downtime Migration Framework
The key insight is that database migrations should be decoupled from application deployments. You should be able to deploy schema changes and application code independently, in any order, without downtime.
Here’s the three-phase framework we recommend for SMBs:
Phase 1: Expand (Backward-Compatible Changes)
The first phase of any migration introduces changes that are fully compatible with the current application version. This means:
- Add columns with NULL defaults — never add NOT NULL columns without a default
- Add new tables — doesn’t affect existing queries
- Add indexes —
CREATE INDEX CONCURRENTLYin PostgreSQL, online index creation in MySQL 8.0+ - Expand column types — VARCHAR(100) → VARCHAR(255) is safe; shrinking is not
-- SAFE: Add column with NULL default
ALTER TABLE users ADD COLUMN phone VARCHAR(20) NULL;
-- SAFE: Add index concurrently (PostgreSQL)
CREATE INDEX CONCURRENTLY idx_users_email ON users(email);
-- SAFE: New table
CREATE TABLE audit_logs (
id BIGSERIAL PRIMARY KEY,
user_id INT REFERENCES users(id),
action VARCHAR(50) NOT NULL,
created_at TIMESTAMPTZ DEFAULT NOW()
);
Run this phase at least one deployment cycle before the application code that uses the new schema is deployed.
Phase 2: Migrate (Dual-Write Pattern)
Once the new schema is in place, update the application to write to both old and new fields simultaneously, while still reading from the old structure:
# OLD: Write to phone_no
# NEW: Write to phone_no AND phone
# READ: Still read from phone_no
def save_user(user_data):
# Dual write
db.execute(
"UPDATE users SET phone_no = %s, phone = %s WHERE id = %s",
(user_data['phone'], user_data['phone'], user_data['id'])
)
def get_user(user_id):
# Read from old field (backward compatible)
row = db.query_one("SELECT phone_no FROM users WHERE id = %s", (user_id,))
return row['phone_no']
During this phase, run a backfill job to populate the new field for all existing records:
# Backfill in batches to avoid locking
UPDATE users SET phone = phone_no WHERE phone IS NULL LIMIT 1000;
Phase 3: Switch (Cut-Over)
Once the backfill is complete and verified:
- Deploy code that reads from the new field instead of the old one
- Run a consistency check — compare old vs. new values for discrepancies
- After a monitoring period (typically 1–7 days), drop the old column
-- FINAL: Drop old column (can be done later, during maintenance window)
ALTER TABLE users DROP COLUMN phone_no;
Automating Migrations with CI/CD
The Expand-Migrate-Switch pattern works best when automated. Here’s how to integrate it into your pipeline:
# .github/workflows/migrations.yml
name: Database Migrations
on:
push:
paths:
- 'migrations/**'
jobs:
migrate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run Expansion Migrations
run: |
# These should always be safe to run
for f in migrations/expand/*.sql; do
psql "$DATABASE_URL" -f "$f"
done
- name: Run Data Backfill
run: python migrations/backfill.py
- name: Verify Consistency
run: python migrations/verify.py
Tools like Flyway, Liquibase, or dbmate can manage your migration versions. We recommend dbmate for SMBs — it’s a single binary with no dependencies, and it supports MySQL, PostgreSQL, SQLite, and ClickHouse.
Essential Tools for SMBs
- gh-ost (GitHub): Online schema migrations for MySQL without table locks. Used on production tables with millions of rows.
- pgroll: Zero-downtime migrations for PostgreSQL using the Expand-Migrate-Switch pattern natively.
- SchemaHero: Kubernetes-native database schema management — declare your schema as a Kubernetes custom resource.
- bytebase: Open-source database CI/CD tool with SQL review, schema versioning, and rollback support.
Database Deployment Checklist for SMBs
- Always test on a copy of production data — not just the schema, but with realistic data volumes
- Never deploy schema changes on Fridays — give yourself at least a full business day for rollback if needed
- Use transactions for reversible changes — wrap related DDL in a transaction when the database supports it
- Monitor database connections during migrations — lock waits can cascade into full outages
- Have a tested rollback script ready — before you run the migration, write the
DOWNscript and test it - Start with the least risky change — run non-blocking operations first, then progressive changes
What About NoSQL Databases?
The same principles apply to MongoDB, DynamoDB, and Firestore:
- Additive changes first — add new fields, don’t rename or remove existing ones
- Dual-write during transition — write to both old and new field names
- Backfill and verify — populate new fields for existing documents
- Remove old schema after verification — clean up in a separate change window
Need help building a database deployment strategy for your team?
We help SMBs implement zero-downtime migration pipelines and database reliability practices without a dedicated DBA.
Book a free consultation and let’s review your current deployment workflow.