top | item 6057736

Network Solutions' DNS was down

72 points| mikegirouard | 12 years ago |networksolutions.com

61 comments

order
[+] pilif|12 years ago|reply
This is one of the reasons why I self-host DNS. Even with tons of users, the resources it takes to serve DNS requests pale compared to what you have to put behind your application servers.

Of course if you are using a CDN to provide your users with better locality, you might want to look into a service that provides localized DNS distribution, but the inherent caching feature of the DNS protocol might make that an unneeded additional burden (I remember a very old Stack Exchange podcast episode where they were talking about this, coming to the conclusion that self-hosted DNS and dns-internal caching is good enough for them).

When you self-host your DNS, you ensure that no provider can suddenly redirect to an ad-filled parking pages if users mistype hostnames and you can make sure that you can fix DNS when it's down. It's also much easier to transfer domains between registrars because there's no need to export and import DNS config - instead, your server just keeps serving the exact same data.

Finally, this allows you to keep the DNS config together with all other configuration files in puppet/chef/git/whatever you use. which further helps future deployments and/or configuration changes.

[+] druiid|12 years ago|reply
You'd think that it would be a good idea... until someone decides YOU are going to be the target of a DDoS attack. I speak from experience. I have many (at least 15) years of experience dealing with DNS and hosting of it, yet when a bot-net decided to attempt to use our DNS servers for a reflection attack. Obviously the servers were setup to not allow this, but the sheer volume of the traffic making attempts was still enough to saturate the connection with our provider at the time.

This is only one of many possible scenarios with self-hosted DNS. Essentially there are services out there (generally Anycast are best) that do only DNS and have had very high up-time. I'd suggest if you are anything over a 'small' target to look into these services. Sadly it's one of the few things I basically suggest not self-hosting at this point simply due to the unrealistic requirements to scale it yourself under attack scenarios.

[+] esw|12 years ago|reply
This is workable for small sites, but are you really equipped to deal with a denial of service attack? There are plenty of small DNS providers (Zerigo comes to mind) that can't even stay up during attacks.
[+] dholowiski|12 years ago|reply
I thought that even if you self host - isn't it your registrar that points the dns records to your dns servers? Would someone with self-hosted DNS still be working in this situation? (doesn't the DNS query first go to network solutions, who refers it to your name servers?).

I'm not trying to argue, I genuinely need to know the answer as I'll have to explain it to many other people, later today...

[+] dedward|12 years ago|reply
What's important is that you treat the DNS management aspect of your business as a proper thing to be managed by itself. Whether you host it yourself, or contract it out to a service that specifically does this for you is a matter of risk tolerance and budget.

A common setup is a private nameserver that you host yourself, wherever, that isn't publicly used. You then contract UltraDNS/Neustar or their brethren and set them up as your public servers, doing zone transfers from your private ones. You get the advantages of direct management of your zones, and the global scale infrastructure needed to largely eliminate nameserver issues.

[+] thejosh|12 years ago|reply
What happens when someone doesn't like you and decides to DDoS your DNS servers?

Though if it's only your site on the DNS Servers, they might as well just DDoS your website.

[+] simonsarris|12 years ago|reply
Honest question from a networking beginner:

So suppose right now I've got two name servers configured, NS93.worldnic.com and NS94.worldnic.com. These are down as they're the part of the Network Solution's name servers that are having issues.

If I had added more, for instance if I used Amazon's Route53 and added two name servers of theirs in addition to the *.worldnic.com ones, would my site be reachable right now?

[+] area51org|12 years ago|reply
Yes, but there may be delay, because the nameservers are tried in random order, and so each server that is down will have to time out before users move on to the next.
[+] throwit1979|12 years ago|reply
In general this ought to "work", but most dns check services and uppity sysadmins will complain loudly and bitterly that there's more than one SOA if you don't do a proper primary-secondary with IXFR.
[+] mikegirouard|12 years ago|reply
This[0] FB post is the best I can come up with as an explanation.

> Yesterday, some Network Solutions customer sites were compromised.

The funny thing is, they have a link in their post going back to their site, which of course doesn't work.

[0]: https://www.facebook.com/networksolutions/posts/101514668014...

[+] muraiki|12 years ago|reply
That explains my friend's site yesterday. Requests to his site sometimes resulted in the display of a banner page by "Islamic Ghosts Team," but not all the time. I noticed that Network Solutions was apparently running Apache 2.2.22, which has a few security flaws (I'm pretty sure he doesn't use a VPS).
[+] nodata|12 years ago|reply
Site down? Don't link to it. Bad netiquette. Jesus christ.
[+] billsinc|12 years ago|reply
Seeing this as well, appears to be DNS only at this point.
[+] asr2bd|12 years ago|reply
Our site has been sporadically up and down for the past few hours. Glad I know the culprit finally. Hope this gets resolved soon
[+] dholowiski|12 years ago|reply
Oh crap. Today is going to be a bad day for me.
[+] jryce|12 years ago|reply
If the mail server goes down again it'll be a bad day for us too.
[+] esmale|12 years ago|reply
I've been in panic mode trying to figure out why our applications have gone down. At least now I know why.
[+] mikegirouard|12 years ago|reply
Ditto. I just spent an hour hacking at my firewall rules before I tried the obvious thing: going to the server's IP.

It doesn't matter how long I do this... DNS problems always get me.

[+] astrodust|12 years ago|reply
And to think you paid extra for this service.
[+] dangayle|12 years ago|reply
Just what I wanted to wake up to. Client emails asking me why their sites were down. Sigh.
[+] SubZero|12 years ago|reply
Why did I click on the link to a down website? What did I expect? Shame on me.
[+] dholowiski|12 years ago|reply
This appears to be a problem again this afternoon (1:44PM MST)
[+] jryce|12 years ago|reply
Their mail server appears to be online.