First, some background… My company and many of our clients host websites with a company called Mediatemple (MT) in Los Angeles. My ISP is Cox Communications.
Tuesday Night: Bad things started happening. I coudn’t connect to our company mail server. I couldn’t load our company website. So, I tried to login the admin side of our server. Can’t do that either. Figured I’d give it some time.
Wednesday Morning: Still no dice. Client who hosts with MT and lives in the Florida panhandle says he can’t check email or load his website (his ISP is Cox also). Okay… so I tried going to Mediatemple’s site to put in a support request. Guess what? Can’t load their page either. I know what you’re thinking, my internet connection is down. Nope! I can reach other websites, check email from other servers, etc.
Wednesday Night: Having worked in tech support I went through the normal checks and fixes. Restart computer, clear cache, check other browsers, try without firewall, restart router, restart cable modem… All to no avail.
Luckily I had Mediatemple’s number and called tech suppport. I explained the situation in detail. After holding for a minute the support guys asks, “Is your ISP Cox?” Well of course it is… “Ya, they seem to be blocking all our IPs.” Supper! So I call Cox Support. I spend 50 minutes on the phone with this absolute moron lady who has me do all the checks I did before. I explain what Mediatemple told me and she says Cox would never block anything! Yeah, right! Four times she tells me she can not help any more and I should call my PC maker or Microsoft or Norton or Santa Clause or the Easter Bunny ’cause maybe they can help more than her. She doesn’t understand we had the same problem on a Mac and at different locations. The problem is not my computer and not the server, damnit!
Finally, I get to second level support and he has me ping the domains and IPs that I can not reach. Obviously, I can’t ping them. He eventually gives me a ticket number and I go to a higher level support. Once I reach them and explain the situation in detail, the tech says, “Did they not tell you we are having DNS problems?” No, they didn’t. But that would really make sense. He kindly adds the domains to the list of problems.
Thursday Morning: All is well. So, what have we learned. First and second level support are not only untrained monkeys, they are not informed of these outages. And I never got an answer as to how all IPs hosted by Mediatemple got blocked/lost/messed up. Sounds fishy. I later learned other sites including MySpace had the same problem. What a bunch of COX!