Qwest truly sucks
By CosmicConservativeSo my internet service provider is Qwest. Qwest is what is left of US West, one of the original seven Regional Bell Operating Companies, which were spawned in the early 80s when the federal government split up the monotlithic Bell Telephone Company.
Disclaimer: I used to work for Qwest. Although I worked in the yellow pages printing division, not the "phone company" part of Qwest.
Yesterday, sometime between noon and about 4:00 pm our internet service died. By the time I got home it had been out a couple of hours.
Now, as a short aside, we used to have internet service through a local ISP called "wispertel". Wispertel was a small entrepeneurial company that provided microwave internet connectivity in remote locations that had not yet been "connected" through the much more common internet connectivity such as DSL lines. Wispertel probably had a couple dozen employees tops, if even that many. During our time with Wispertel we had some outages, but the longest outage we had with Wispertel was probably a few hours. And that was from a small startup company that was more or less run on a shoestring.
As of this moment I have not had internet connectivity at my home for 18 hours at least, and Qwest "technical support" (which is a whole different sort of joke) hasn't got a clue about when they will have service restored. They say it should be "within 48 hours" but that their "trained and professional engineering team" is working diligently on the problem.
Now here's the deal. I work for a company that provides internet and direct network connectivity for millions of businesses and customers worldwide. We process financial transactions at the rate of millions per day. In fact our infrastructure is quite comparable to that of a Qwest. I know quite a bit about the internal data center details of Qwest having spent some time working with the Qwest data center people. In fact I have been inside the main Qwest data centers in the Denver area on more than one occasion. I haven't been inside the data centers of my current company, but my job is to help manage the consolidation of data centers so I have seen detailed diagrams of the data centers and have a good idea of how they compare to Qwest.
The reason that is relevant is because if we had any one of our key customer-facing network services down for more than an hour or two, it would be seriously bad news. We require redundancy and disaster recovery for all critical applications. Something like "residential internet connectivity" would certainly qualify as a "critical application" for us, and presumably, for Qwest.
The "technical support" I talked to (which was obviously an offshore call center manned by people whose technical expertise is limited to knowing which buttons to push on their computer screen) told me that their "system" had an entry from the "engineer" working on the outage which said simply that the outage was due to the need to replace a server, and that the server replacement would take no more than 48 hours.
Now what this reveals is rather stunning considering that Qwest likes to style itself as a "world-class communications company." What it means is that they have internet connectivity for probably tens of thousands of their customers (entire towns are affected by this) apparently supported by a SINGLE server which has NO REDUNDANCY and NO BACKUP and whose replacement appears to require PURCHASING AND SHIPPING A NEW SERVER TO THEIR DATA CENTER.
As I said, I work in this business and I know how long it takes to stand up equipment of this sort, and that is usually measured in minutes or hours, not DAYS. When you get into DAYS for this sort of thing, that is virtually guaranteed to mean that the company is having to PURCHASE equipment to fix the problem.
This is frankly a STUNNING display of HIGH LEVEL corporate incompetence. This isn't a problem with some engineer, this is a problem with a complete lack of planning for hardware problems at the MOST BASIC data center level. When the "tech support" flunkie said "our trained and professional engineers are working on this" I laughed in his face. I told him that if Qwest HAD any "trained and professional engineers" that this sort of thing would never happen. This is baby data center stuff. This is the sort of thing that small startup companies deal with. This is simple basic redundancy. For Qwest to have ONE SERVER fail and have this sort of impact is gross incompetence.
But the bottom line is that I can't do anything about it because there is NO OTHER high speed internet connectivity in my area. When Qwest finally got DSL in our area, that pretty much killed all the smaller internet companies such as the microwave internet company. So I can't go to Comcast, for example. There IS a satellite internet that I could go to, but that is expensive and requires installing satellite dishes on my roof. So I will probably stick with Qwest in spite of this stunning display of root level incompetence in their data center management.
Sheesh.... Hopefully I will have connectivity back before Thursday morning when I have to be working from home....
Comments (9)