Amazon, please hire better network engineers

I’ve been evaluating network performance of AWS services for a while now, especially their globally spread solutions such as CloudFront and Route53 – these services go way beyond official AWS regions with points of presence in most important (Internet-wise) countries and major cities.

Queries for your domain are automatically routed to the nearest DNS server, and thus answered with the best possible performance.
Route53 description on Amazon.com

So, on paper, or to be more accurate, on screen they promise to route you to the closest edge server. To achieve the goal, Amazon currently employs anycast IP addresses for Route53 and GeoDNS technique for Route53 – both approaches are pretty standard and have been implemented by other companies for years, if not a decade.

Here we go – Amazon’s implementation of aforementioned sucks big time. Who would expect that one of the biggest (in terms of the capitalisation) companies on planet couldn’t implement right a pretty standard pattern? I will show further in this post how irrational Route53 and CloudFront routing policies are and how AWS seriously underperforms as compared to other, sometimes free, services.

Note that it’s not a temporary state of things. I’ve been eyeing (read tracerouting) them for at least 2-3 years now. This is a permanent network policy issue.

One more rant before we dive into analytics: Amazon, wake up!

Route53

When you add a new zone on Route53, you are assigned 4 AWS nameservers, just like these:


ns-1105.awsdns-10.org.
ns-1689.awsdns-19.co.uk.
ns-588.awsdns-09.net.
ns-75.awsdns-09.com.

Note that AWS uses 4 different gTLDs which, honestly, doesn’t make much sense here. As a matter of fact, it will slow down name resolution a little bit. First of all, neither of these 4 gTLDs use 100% anycast nameservers so your queries can take up to 200+ milliseconds. Secondly, it’s unlikely that’s some bad boy is going to successfully DDoS more than one gTLD so using 4 different tLDs as a failover method is perhaps an overkill. But okay, 4 different gTLDs it be.

These nameservers have the same IP addresses when looked up from any place in the world, but those IP addresses are ‘magic’ anycast IP addresses – meaning they will be routed to different edge locations depending on the source. Ideally, as in case with CloudFlare and others, it should point you somewhere really close – like the capital of your country or something, but let’s see what do we get with AWS Route53.

Anycast routing done the AWS Route53 way

Let’s take 4 nameservers that we mentioned before and ping them from different points in the world and then calculate average response time. DNS clients choose a nameserver from the list randomly so using an average is pretty fair in this context. At the same time, we will compare results with two other DNS options – CloudFlare and SoftLayer, both of them are free.

Latency average AWS CloudFlare SoftLayer
Jakarta 73ms 13ms 14ms
Kuala Lumpur 144ms 9ms 20ms
Singapore 194ms 1ms 1ms
San Francisco 36ms 2ms 3ms
Northern Virginia (AWS) 6ms <1ms 34ms
New Jersey 8ms 2ms 7ms
Sydney #1 102ms 1ms 1ms
Sydney #2 106ms 1ms 1ms
Sydney #3 (AWS) 74ms 11ms <1ms
Melbourne 215ms <1ms <1ms
Bangalore 97ms 9ms 230ms
Frankfurt 48ms <1ms <1ms
London 6ms 1ms 2ms
Moscow 37ms 24ms 38ms
Novosibirsk 88ms 83ms 95ms
Hiroshima 117ms 16ms 16ms
Rio de Janeiro 128ms 9ms 163ms
São Paulo 283ms 2ms 180ms

Note: Testing against DigitalOcean nameservers (also free) will give you the same exact figures as CloudFlare because DigitalOcean runs their DNS on CF infrastructure.

Note 2: It seems that Route53 usually routes you quite decently to two out of 4 nameservers. And quite often one is somewhere really far. It’s not clear whether they are doing this on purpose. If it’s a failover measure, it’s a very questionable one

Note 3: Some of the requests in Brazil are routed to India. How drunk a network engineer should be to route a packet that far, seriously?

Note 4: Route53 performs worse than competitors even when used from within the AWS network. Even in Northern Virginia, Amazon’s default and primary region, CloudFlare beats Route53, how cool is that? Not cool at all!

Other aspects

Having said and analysed all that, there are some extra details to note:

Pros:

  • Route53 offers advanced features like GeoDNS. Neither of the competitors mentioned does

Cons:

  • Route53 doesn’t support IPv6. Both competitors do
  • Every your domain will have a different set of nameservers. Not critical but not very convenient

Conclusions

If you live in São Paulo or Melbourne – dear, please don’t use Route53! You could as well just place your nameservers in Bulgaria or Peru, same effect.

Route53 underperforms in ALL points of test, and it takes more time to even start a query with Route53 because they use four different gTLDs. On top of that, you have to pay for such a misconfigured service.

Unless you need features like GeoDNS (which are handy sometimes, no doubt) using free alternatives like DigitalOcean, CloudFlare or SoftLayer will make your customers happier – your websites will open faster, your APIs will respond quicker and your mailbox will receive spam (no, this is not a real one, but your emails will deliver faster though).

In the next article we will do a similar analysis-slash-rant on CloudFront.

Post by Denis Mysenko

Born in the snows of Siberia