PSA: Centurylink outage

[mdlbear]

Last night around 11pm I was awakened by an alert on my phone telling me that 911 service was down, and giving me an alternat number to call. By morning, it was clear that it wasn't a local problem. A quick search showed that the problem was caused by CenturyLink, which tweeted, blaming it on a a network element that was impacting customer services and saying that they estimated it would be fixed in about four hours.

It was more like twelve here on Whidbey Island, and some parts of the country are still (as of 2pm) offline, according to Outage.Report. The FCC is investigating.

If you live in Washington, king5.com has a handy list of numbers to call, by county. (The news article also has auto-playing video - you may want to mute your speakers.)

Notes & links:

  : C-link back up here.  Here are the emergency numbers to use when 911 is down:
    if 911 down: call 360-679-9567 360-675-3418 360-320-5795 360-320-5735 (Whidbey Is.)
    @ FCC investigating major CenturyLink outage and 911 disruptions - The Verge
      FCC to investigate CenturyLink as [...] outage continues, 911 calls disrupted
      CenturyLink: Service outage that caused 911 failures not related to hacking |
      king5.com (auto-play with audio) has list of emergency #s by county
      CenturyLink Outage Map - Outage.Report

Another fine post from The Computer Curmudgeon (also at computer-curmudgeon.com).

Comments

[siliconshaman]

I wonder if it's related to the Con-ed explosion in NYC? That knocked out power over half the city I hear.

[mdlbear]

That's what I thought, when I saw the two articles together. But I doubt it.

If the problem had been just the power outage, Centurylink would almost certainly have said so; they're under a huge amount of pressure for explanations, and it would have been a good one. But the article in the New York Times pegs the explosion at 9:12pm, which translates to 6:12pm Pacific.

This timeline (which I should have searched for) puts CL's first tweet about the problem at 10:06am Pacific time, so the Centurylink outage would have been in progress for over 8 hours by the time of the explosion.

Outages are also widely scattered, and IIRC some of the earliest maps I can remember showed most of the outage reports in the northwest.

My best guess is that it's some kind of configuration problem, possibly having to do with routing, but whatever it was it's going to make interesting reading.

[siliconshaman]

Hmm... I wonder if it's an accident, or if someone was testing something again.

Outage.report

[jesse_the_k]

is a v helpful service! Geographic outage maps per service! Global FB down-ness, for example

I knew that one drawback to cellular service was the lack of autoreporting geolocation, but I didn't realize that the 911 switching network required an ISP to be up.