RDS Multi-AZ Failover: What Happens and How to Test It

An Amazon RDS Multi-AZ failover promotes the standby in another Availability Zone to primary and repoints the database endpoint at it, with no change to your connection string. A Multi-AZ DB instance deployment typically completes in one to two minutes; a Multi-AZ DB cluster typically completes in under 35 seconds. Your application still has to reconnect, so the failover is only as fast as your connection handling allows. Test it with a forced failover before you rely on it.

This matters most right after a database migration. You moved an on-prem database onto RDS, ticked the Multi-AZ box, and now you are responsible for an availability promise you have never actually exercised. The box does not test itself.

What RDS Multi-AZ failover actually does

When the primary becomes unavailable, RDS does not rebuild anything. It promotes a replica that is already running in a second Availability Zone and moves the endpoint to it. <cite index="1-29">When failing over, Amazon RDS flips the canonical name record (CNAME) for your DB instance to point at the standby, which is in turn promoted to become the new primary.</cite> Because the DNS name your application connects to stays the same, there is nothing to reconfigure on the client. What changes underneath is the IP address that name resolves to.

RDS triggers an automatic failover for infrastructure problems, not application problems. <cite index="1-55,1-56">It fails over on loss of availability in the primary Availability Zone, loss of network connectivity to the primary, a compute unit failure on the primary, or a storage failure on the primary.</cite> It deliberately does not fail over for things that look like database problems from the outside. <cite index="1-60">Amazon RDS Multi-AZ deployments do not fail over automatically in response to database operations such as long running queries, deadlocks, or database corruption errors.</cite> If your incident is a runaway query or a lock pileup, Multi-AZ will not save you, and expecting it to is a common and expensive misread.

Failover is also the mechanism behind maintenance, which is a point in its favour. <cite index="1-57">When operations such as DB instance scaling or system upgrades like OS patching are initiated for Multi-AZ deployments, they are applied first on the standby prior to automatic failover, so your availability impact is limited to the time required for failover to complete.</cite>

Multi-AZ instance versus Multi-AZ cluster

These are two different products with the same marketing word, and the difference changes your failover time, your cost, and whether the standby is useful for anything.

A Multi-AZ DB instance deployment has one standby in a second AZ, kept in sync synchronously. <cite index="2-46">The standby is only a passive failover target for high availability and serves no read traffic.</cite> <cite index="2-48">A new primary is available to serve your workload in as quickly as 60 seconds, and failover time is independent of write throughput.</cite> In practice, <cite index="1-30,1-31">failovers typically complete within one to two minutes, and failover time can also be affected by whether large uncommitted transactions must be recovered.</cite>

A Multi-AZ DB cluster deployment is a different shape. <cite index="3-1,3-2">It is a semisynchronous, high availability deployment with a writer and two readable replica DB instances across three separate Availability Zones in the same Region.</cite> <cite index="3-4">Semisynchronous replication requires acknowledgment from at least one reader before a change is committed, not acknowledgment from all replicas.</cite> Those two standbys are not idle. <cite index="3-5">Reader DB instances act as automatic failover targets and also serve read traffic to increase application read throughput.</cite> Failover is faster: <cite index="1-3">failover times are typically under 35 seconds, and failover completes when the reader DB instances have applied outstanding transactions from the failed writer.</cite>

The cluster has one subtlety you should plan around. <cite index="3-11,3-13">Because any failover must first resolve replica lag before promoting a new writer, monitoring and managing replica lag is a real consideration, and for PostgreSQL clusters the reader with the lowest replica lag must apply its unapplied transactions before it is promoted.</cite> A backed-up replica is a slower failover.

If you just need availability, the instance deployment is simpler and runs on one fewer node. If you also want to push read traffic off the writer and you are on MySQL or PostgreSQL, the cluster earns its extra instance.

What to check before you rely on it

The failover is RDS's job. Reconnecting is yours, and this is where otherwise-correct setups lose minutes they did not budget for.

Connection retry and DNS caching

Your application must reconnect after the endpoint flips, because existing sockets are pointed at an instance that is now gone or read-only. AWS is explicit that this is an application responsibility. <cite index="2-29">The guidance is to implement database connection retry at the application layer.</cite> If your pool holds dead connections and hands them out without validation, your outage lasts until something forces those connections to recycle, not until RDS finishes the failover.

DNS caching is the other trap, and it is worse on the JVM than most people expect. <cite index="1-42,1-43">The failover changes the DNS record to point to the reader, so you need to re-establish connections, and in a JVM environment the Java DNS caching mechanism may require you to reconfigure settings.</cite> <cite index="1-46">On some Java configurations the default TTL never refreshes DNS entries until the JVM is restarted, so if the IP changes while your application runs it cannot use the resource until you restart the JVM.</cite> The fix is a small one. <cite index="1-45">AWS recommends configuring your JVM with a TTL value of no more than 60 seconds so that when a resource's IP address changes, your application can requery DNS and use the new address.</cite>

# $JAVA_HOME/jre/lib/security/java.security
networkaddress.cache.ttl=60

Check this before a failover, not during one. A 10 second RDS failover behind a JVM that caches DNS forever is a total outage until a human restarts the process.

Know when it actually happened

You cannot measure what you do not observe. <cite index="2-18">Amazon RDS emits a DB instance event when automatic failover occurs, which you can see in the Events section of the console or retrieve with the DescribeEvents API.</cite> Wire RDS event subscriptions to your alerting so a failover pages you with a cause, rather than surfacing as a wall of connection errors with no explanation. Building that kind of signal into on-call is exactly the work our reliability and SRE practice does alongside teams that have just inherited a managed database.

How to test a failover safely

You test Multi-AZ by causing a controlled failover and watching how long your application takes to recover. RDS gives you a first-class way to do this.

For a Multi-AZ DB instance, use reboot with failover. <cite index="5-10">If the DB instance is in a Multi-AZ deployment you can reboot with a failover, which is useful to simulate a failure of a DB instance or to restore operations to the original Availability Zone after a failover.</cite> On the CLI this is the --force-failover flag. <cite index="6-6">The ForceFailover parameter specifies whether the reboot is conducted through a Multi-AZ failover, and you cannot enable it if the instance is not configured for Multi-AZ.</cite>

aws rds reboot-db-instance \
  --db-instance-identifier my-prod-db \
  --force-failover

Be clear-eyed about what a reboot-with-failover does to clients. <cite index="5-15,5-16">An outage occurs for both Single-AZ and Multi-AZ instance deployments even when you reboot with a failover, and the operation interrupts the database abruptly so client sessions may not have time to finish cleanly.</cite> That is the point: it is a realistic test. Run it in a maintenance window the first time, time the recovery from the application's perspective, then decide whether the number is acceptable.

For a Multi-AZ DB cluster, there is a separate command. <cite index="1-14,1-17">If you manually fail over a cluster, RDS terminates the primary, the internal monitoring system detects it is unhealthy and promotes a readable replica, and failover times are typically under 35 seconds.</cite>

aws rds failover-db-cluster \
  --db-cluster-identifier my-prod-cluster

Measure the metric your users feel, which is end-to-end request error duration, not the time the console takes to show the new AZ. <cite index="1-4">After failover completes, it can take additional time for the RDS console to reflect the new Availability Zone,</cite> so the console lags reality. Trust your application logs and your synthetic checks.

The trade-off, and what it costs later

Multi-AZ means paying for standby capacity that serves no writes. On the instance deployment, that standby does nothing but wait. That is the honest trade: you buy automatic recovery from AZ and host failures, and you pay for idle redundancy to get it.

Three decisions compound later if you skip them now.

First, the single-AZ to Multi-AZ conversion is online but not free of effect. <cite index="2-26,2-27">When you convert, RDS takes a snapshot, creates a standby in another AZ from it, and configures synchronous replication, so there is no downtime, but you might see increased latency while the standby catches up.</cite> Do the conversion deliberately, not during peak.

Second, if a one to two minute reconnect window is too long, the next lever is RDS Proxy rather than a fancier database. <cite index="7-2">RDS Proxy reduces database failover times by up to 66 percent and preserves application connections during failovers.</cite> The measured improvement depends on the engine. <cite index="8-2">In AWS testing, RDS Proxy reduced client recovery time after failover by up to 79 percent for Aurora MySQL and by up to 32 percent for RDS for MySQL.</cite> It is a real reduction, not a cure, and it adds a component to run and pay for.

Third, Multi-AZ is a single-Region construct. All three AZs are in the same Region, so a Region-level event is outside what Multi-AZ covers. If your recovery objective spans Regions, that is a different design involving cross-Region replicas, and it belongs in the migration plan from the start rather than bolted on after cutover. Getting these availability decisions right during the move is the core of our cloud migration work, where the goal is that the cutover, and every failover after it, is boring.

Frequently asked questions

How long does an RDS Multi-AZ failover take?

It depends on the deployment type. A Multi-AZ DB instance deployment can complete <cite index="2-48">in as quickly as 60 seconds</cite>, though large uncommitted transactions push it toward one to two minutes. A Multi-AZ DB cluster is faster, <cite index="1-3">typically under 35 seconds</cite>. In both cases your real recovery time includes how long your application takes to notice and reconnect.

Does the standby in a Multi-AZ deployment serve read traffic?

Only in a Multi-AZ DB cluster. In a Multi-AZ DB instance deployment the <cite index="2-46">standby is only a passive failover target and serves no read traffic</cite>. In a Multi-AZ DB cluster, the <cite index="3-5">two reader instances act as failover targets and also serve read traffic</cite>. If you need read scaling as well as availability, that distinction decides which product you pick.

How do I test an RDS failover without waiting for a real outage?

Force one. For a Multi-AZ instance, run reboot-db-instance with --force-failover, which <cite index="5-10">is designed to simulate a failure of a DB instance</cite>. For a Multi-AZ cluster, use failover-db-cluster. Do it in a maintenance window the first time and measure recovery from your application's point of view.

Will Multi-AZ fail over if my database is slow or deadlocked?

No. RDS fails over for infrastructure failures, but <cite index="1-60">Multi-AZ deployments do not fail over automatically in response to database operations such as long running queries, deadlocks, or database corruption</cite>. Those are application and schema problems, and Multi-AZ is not the mechanism that resolves them.

Why does my application stay down after RDS says the failover finished?

Almost always connection handling. RDS repoints the endpoint, but <cite index="2-29">you have to implement connection retry at the application layer</cite>, and a long DNS TTL makes it worse, which is why <cite index="1-45">AWS recommends a JVM DNS TTL of no more than 60 seconds</cite>. Check your pool's dead-connection handling and your DNS cache before blaming the database.