Amazon, please hire better network engineers
I’ve been evaluating network performance of AWS services for a while now, especially their globally spread solutions such as CloudFront and Route53 – these services go way beyond official AWS regions with points of presence in most important (Internet-wise) countries and major cities.
So, on paper, or to be more accurate, on screen they promise to route you to the closest edge server. To achieve the goal, Amazon currently employs anycast IP addresses for Route53 and GeoDNS technique for Route53 – both approaches are pretty standard and have been implemented by other companies for years, if not a decade.
Here we go – Amazon’s implementation of aforementioned sucks big time. Who would expect that one of the biggest (in terms of the capitalisation) companies on planet couldn’t implement right a pretty standard pattern? I will show further in this post how irrational Route53 and CloudFront routing policies are and how AWS seriously underperforms as compared to other, sometimes free, services.
Note that it’s not a temporary state of things. I’ve been eyeing (read tracerouting) them for at least 2-3 years now. This is a permanent network policy issue.
One more rant before we dive into analytics: Amazon, wake up!
When you add a new zone on Route53, you are assigned 4 AWS nameservers, just like these:
ns-1105.awsdns-10.org. ns-1689.awsdns-19.co.uk. ns-588.awsdns-09.net. ns-75.awsdns-09.com.
Note that AWS uses 4 different gTLDs which, honestly, doesn’t make much sense here. As a matter of fact, it will slow down name resolution a little bit. First of all, neither of these 4 gTLDs use 100% anycast nameservers so your queries can take up to 200+ milliseconds. Secondly, it’s unlikely that’s some bad boy is going to successfully DDoS more than one gTLD so using 4 different tLDs as a failover method is perhaps an overkill. But okay, 4 different gTLDs it be.
These nameservers have the same IP addresses when looked up from any place in the world, but those IP addresses are ‘magic’ anycast IP addresses – meaning they will be routed to different edge locations depending on the source. Ideally, as in case with CloudFlare and others, it should point you somewhere really close – like the capital of your country or something, but let’s see what do we get with AWS Route53.
Anycast routing done the AWS Route53 way
Let’s take 4 nameservers that we mentioned before and ping them from different points in the world and then calculate average response time. DNS clients choose a nameserver from the list randomly so using an average is pretty fair in this context. At the same time, we will compare results with two other DNS options – CloudFlare and SoftLayer, both of them are free.
|Latency average AWS||CloudFlare||SoftLayer|
|Northern Virginia (AWS)||6ms||<1ms||34ms|
|Sydney #3 (AWS)||74ms||11ms||<1ms|
|Rio de Janeiro||128ms||9ms||163ms|
Note: Testing against DigitalOcean nameservers (also free) will give you the same exact figures as CloudFlare because DigitalOcean runs their DNS on CF infrastructure.
Note 2: It seems that Route53 usually routes you quite decently to two out of 4 nameservers. And quite often one is somewhere really far. It’s not clear whether they are doing this on purpose. If it’s a failover measure, it’s a very questionable one
Note 3: Some of the requests in Brazil are routed to India. How drunk a network engineer should be to route a packet that far, seriously?
Note 4: Route53 performs worse than competitors even when used from within the AWS network. Even in Northern Virginia, Amazon’s default and primary region, CloudFlare beats Route53, how cool is that? Not cool at all!
Having said and analysed all that, there are some extra details to note:
- Route53 offers advanced features like GeoDNS. Neither of the competitors mentioned does
- Route53 doesn’t support IPv6. Both competitors do
- Every your domain will have a different set of nameservers. Not critical but not very convenient
If you live in São Paulo or Melbourne – dear, please don’t use Route53! You could as well just place your nameservers in Bulgaria or Peru, same effect.
Route53 underperforms in ALL points of test, and it takes more time to even start a query with Route53 because they use four different gTLDs. On top of that, you have to pay for such a misconfigured service.
Unless you need features like GeoDNS (which are handy sometimes, no doubt) using free alternatives like DigitalOcean, CloudFlare or SoftLayer will make your customers happier – your websites will open faster, your APIs will respond quicker and your mailbox will receive spam (no, this is not a real one, but your emails will deliver faster though).
In the next article we will do a similar analysis-slash-rant on CloudFront.