Where am I?

Have you ever thought about setting up a cluster of software nodes using Amazon's EC2 infrastructure? You may have wondered what happens if Amazon's underlying hardware should fail. That hardware runs the hypervisor for your EC2 instance, and if it fails so does your instance. If you have a cluster of N nodes, what happens to the cluster?

Most cluster software includes some mechanism to handle host failures. But the cluster was usually designed to run on real hosts, not virtual machines. In most cases a hardware or software fault should take out exactly one host. But in a virtualized environment the same fault might affect more than one virtual machine. This is especially true in a cloud environment like EC2, where you have no visibility into the hypervisor.

At first this may not seem to be a problem. If you create four EC2 instances at the same time, they usually seem to start on different hypervisors. So a hypervisor failure will only affect one instance, and the cluster failover mechanisms will operate as usual.

But when an instance does fail, you will want to replace it. How can you ensure that the replacement instance won't be co-resident with any of your existing instances? If the replacement instance is co-resident with an existing cluster instance, and another hypervisor fault affects both instances, the cluster will go down. Depending on the circumstances, you may lose data too.

So how can we prevent this? One idea is to include a pool of spare instances when we create the cluster instances. But this makes the cluster more expensive. We cannot stop the spare instances either, because when we start them again they may end up co-resident with another instance. Over time we may also exhaust this pool of spares, leaving us with the original problem again. How can we avoid co-resident cluster instances?

If we could detect co-residence, we could simply create instances until we get one that is not co-resident. This approach is crude, but should be effective.

1. Create instance (or start existing stopped instance).
2. Check co-residence for each existing instance.
3. If co-resident, stop the instance and return to step #1.

Any instance that makes it past step #3 is not co-resident with any existing member of the cluster instance, and can join the cluster. This leads to a new problem: how can we test for co-residence?

AWS does not appear to provide an easy way to do this. Allowing instances visibility into the hypervisor could lead to security holes. This problem has been discussed by Ristenpart, Tromer, et al. My own experiments suggest that AWS has changed its network topology since 2009, but I did find a technique that seems to work.

From what I can tell, each AWS hypervisor acts as a gateway for its guest instances. Each guest instance runs on its own subnet. So knowing the IP address alone tells us nothing, but knowing the network hops from one instance to another could be revealing.

If each hypervisor acts as a gateway, then packets sent between two co-resident instances will always need exactly two hops. Packets sent between instances that are not co-resident will need more hops.

We can conclude that ip-10-87-117-148 and ip-10-87-69-9 are co-resident.

We can conclude that ip-10-91-9-230 and ip-10-87-117-148 are not co-resident.

Note that this technique may not be foolproof. If Amazon believes that this information compromises security, they could probably reconfigure their hypervisors so that new instances would hop through a random number of extra gateways. This would add some latency to the network, but more importantly would keep the guest instance from using this technique to determine whether or not it is co-resident with some other instance. That would be a shame for those of us using AWS for highly-available cluster applications... unless AWS also added a supported technique for avoiding co-residence.