When was the last time you checked your calendar? It’s 2017, modern data centers run Layer3 (L3) networks while getting rid of Layer2 (L2) as much as possible. I was always interested how to run MySQL service with high availability for read/write workloads.

Yeah yeah, you can say, it’s easy, let’s run few read replicas, load balance read traffic through L4 load balancer (e.g. HAProxy). Or, you can use L7 proxy (e.g. ProxySQL) to split read/write traffic and direct it to a certain node. This works unless you need HA for writes. With L7 proxy you can shard a table or a database to many write nodes, but what if I have only one and need HA for it?

Implementation

I picked few well-known tools for this task:

• Zookeeper
• ExaBGP

Implementation is simple. ExaBGP forks a new process. Process tries to acquire a mutually exclusive lock to Zookeeper. If it succeeds - ExaBGP will announce anycast BGP route and write information to Zookeeper about the master node (e.g. hostname or something more important). If it fails - ExaBGP will withdraw route and spin around trying to acquire mutex once again.

Zookeeper has a quorum at a minimum from three nodes. Even if Zookeeper nodes are partitioned, this solution is resistant, because it does health checks every arbitrary number of seconds to verify lock.

How about the speed to converge, because BGP is slow for that? The best answer to this - it depends. If your application tolerate database failover in 5-10 seconds, then it’s fine. In other words, keep rule #1 of good engineering - know your problem.

ExaZK

I have implemented this PoC using Ruby script named ExaZK which is spawned by ExaBGP as a child process.