This past weekend brought a second unusually heavy snowstorm to the mid-Atlantic area where I live. Unlike the just-before-Christmas storm, however, this one had heavy wet snow that left many of us without power from Friday afternoon until after midnight Sunday.
Heat, refrigeration and water (from our well pump) were provided by our emergency generator until power was restored. But communications lasted only 24 hours, which is the reason for this brief tale.
In the days before the Internet and cell phones and when the only phone provider was "Ma Bell", landline phone service seemed like it never went out. I could be misremembering, but when a hurricane knocked out power back in the 60’s, we were always able to pick up the phone, get a dial tone and check in with friends and family.
In this storm, our landline phone lasted about 24 hours before the dial tone vanished. And just about the same time, our cell phone started alerting that is was constantly exiting and entering its service area—probably because our closest cell tower had exhausted its backup power. And, of course, our DSL line died along with our dial tone, as did 3G broadband service along with the cell signal.
So in the course of 24 hours we went from being in touch with the world via four communications systems to having one (our cell phone) that worked only if we bundled up and trudged through the snow to a particular spot in our yard. And as I write this, almost twelve hours after power was restored, the iffy cell phone remains our only communications link. My DSL modem is showing that it’s synched and ready, but it’s waiting for some other link in the chain to come back online so that it can issue me an IP and a working connection.
So as I sit watching the Skype icon in my System Tray spin away as it tries to reach out and connect, I have to wonder how we got here. In our rush to push broadband and cell service out the suburban and semi-rural masses (forget the really rural folks, who still languish, frustrated, in the U.S. communications backwater), do we really have essential communication lifelines that can’t stay up more than 24 hours without centralized power?
From where I sit, it sure looks like it. We may have COLTs and micro-cells and roadside phone and DSL exchange cabinets. But, apparently, they have only 24 hours of backup power before they turn into useless piles of chips and wires.
I know that all designs have limits and that you can’t practically design for every adverse scenario. But how about making 7 days the design goal for backup power, or hell, maybe even half that?
Robust system design practice would dictate taking into account the MTBF and MTTR of key interdependent subsystems and designing accordingly. That’s why servers that need to be up 24/7 have redundant hot-swappable power supplies, hard drives and fan panels. While I’ll grant that distributed communications systems are more complex and difficult to failure-harden than a server rack, the same principles apply.
Sadly, the real reason for these fragile, yet essential systems is most likely the misguided profit-through-cost-reduction focus that seems to dominate today. Make it fast, flashy, cheap and quick and rake in the dough. And if it falls down in a heap under the slightest pressure, you can just keep charging customers for services they can’t even use because, well, because you can.
There’s a business opportunity in here somewhere, since that’s the only way these issues get addressed any more. We have the technology to fix this. We just need the will to do it.