This is the fourth post in a series of posts on setting up a high availability web application system on a shoestring budget. The goal is to have no single point of failure and to not spend a lot of money. Read the first post, the second one, and the third one.
A brief recap: I gave up on postgres and pgpool, and after evaluating some alternatives I am now going to try good old mysql and MMM.
MMM is a system to manage a multi-master mysql installation. You put an agent script on each database server and a monitoring script runs on a separate server. The interesting thing about it is that it uses a bunch of virtual floating ip addresses to manage failover. In a typical scenario, you need one floating ip that gets assigned to whichever master is going to handle the writes. You also need a floating ip for each reader. In my two server setup (both masters), I need three virtual ip addresses, one for the writer and two for the readers.
When you start it up, db0 will have the writer ip address and one reader ip address. db1 will have a reader address. The client application has these three addresses and they always maintain the same role. If db1 goes down, the MMM monitor tells db0 to take over the reader ip that was assigned to db1. If db0 goes down, the MMM monitor tells db1 to use the writer and reader ip addresses that were assigned to db0. The client application never knows that anything is going on (but it will need to reestablish any existing connections to the ip addresses that moved, which is unavoidable).
The simplicity of the system appealed to me. All MMM is doing is shuttling ip addresses around. I also like that I can add more database servers to the system easily. If I add a slave server, I just add another floating reader ip address.
And, as an added bonus, the installation guide is very well written and it all works. I’ve tested taking servers out of the pool using the mmm_control script, shutting down mysql, and shutting down the server. MMM worked flawlessly each time.
Using mysql and MMM has satisfied all my needs:
- There is no single point of failure. Either database server can go down and the application will still work.
- Both database servers are using their cycles productively.
- Failover and recovery is automatic.
- More servers can easily be added if necessary.
- All the technologies in use have been around for a while.
- It’s cheap: it only requires two database servers and another server to run the monitor (more on where I put it coming up).
Now that the database is (finally) highly available, the next topic will be load balancers in front of the web servers.