I see multiple posts here speculating on cyberattack—as opposed to "we pushed a bad configuration update which messed everything up irreparably"—you know, like it has been every other time before this.
E.g., Cloudflare, Meta (who in doing so also locked themselves out of the building), and didn't some bumbling major Canadian telco knock themselves offline for like a week not too long ago?
Yeah, the Canadian telco was Rogers. Total recovery took multiple days. From the Wikipedia writeup:
> In a letter to the CRTC, Rogers stated that the deletion of a routing filter on its distribution routers caused all possible routes to the internet to pass through the routers, exceeding the capacity of the routers on its core network.
I took my kids swimming that day and the pool couldn't take our money since the payment terminal was on the Rogers cellular network, so it was a free family swim.
Cyberattacks are a good scapegoat for any large incompetent non-tech company that is unable to admit a mistake. (tech companies are more open to admitting actual mistakes - and reluctant to disclose cyberattacks even if there actually was one - where as non-tech ones would rather allude to an attack than admit a mistake)
Cyberattack scenarios pretty much never make sense in case of complete outages; if you have the access required to cause such an outage it’s always more profitable to keep this access and use it for covert spying/targeted attacks or save it for later than to burn it by causing a massive, visible problem.
Verizon had issues routing calls to a provider I'm aware of yesterday, and had to make some sort of change today to fix it. I'm definitely thinking bad configuration update.
For those who weren't aware: Verizon got a new CEO late last year and laid off 15% of the workforce (15,000 people). This included people working in network, IT and cyber security.
15% seems like the magic number. It seems like many corporations layoff approximately that fraction of their workforce; it's hard to believe it's coincidence.
Like all the doom and gloom after the Twitter layoffs predicting the site would implode and go permanently offline "within a month" which...never happened.
It's also ironic in the sense it implies the indignant people were so bad at their jobs they designed and built a system so fragile it would collapse without constant intervention from thousands of individuals.
You do realize it's possible for an organization to be overstaffed?
If you’ve got 100k people to run something that should run essentially on autopilot, you’ve got much deeper problems where merely laying off selected chunks of people will no longer help. The whole company is rotten and the only way is to start from scratch and not make the mistakes that led you to accumulate 100k people.
I can't even begin to imagine what those 100k people actually do. For starters, none of the telcos actually develop their own equipment - they buy pre-made from vendors like Ericsson. Often that includes ongoing maintenance too. The only "engineering" is building the back-office and customer-facing UIs, and even that is often outsourced (as a rule of thumb, if something can be outsourced, telcos will do it: https://berthub.eu/articles/posts/5g-elephant-in-the-room/).
Customer service might be part of that number (assuming that too isn't outsourced), but even then 100k feels extreme.
10k is ok although leaning on the more bloated side. But 100k?
Recently downloaded bitchat so I can contact my family members if outages like this happen when we’re out and about. I got a few T1000 cards for meshtastic but there’s just too much friction to teach my spouse and others how and when to use it. I wish haloW was built into phones which would make long range local communication much better.
Does bitchat work during a phone outage? Range on BLE is pretty low, right, so I'd expect it to not do much unless you happen to live in an area where there are bitchat users every 100 ft or so.
I'm in a similar boat to you in wanting a LoRa mesh. I tried out MeshCore on the LilyGo T-Deck+ hoping it would be a device I could hand out to family members, but I found the hardware and software disappointing.[0] But I'm weirdly tempted to try the LilyGo Pager.
I got sniped away from Visible (Verizon MVNO) by US Mobile (multi-carrier MVNO) during a Black Friday sale. USM has an interesting thing where you can actually get a separate eSIM for each of the major carriers, and switch between them. I was curious so I signed up for all 3. It's been interesting to see how the signals vary from location to location, and at least a couple times I've been able to get significantly better signal by switching.
The main downside is that you have different numbers for each eSIM, but that doesn't really affect me because I use Google Voice for SMS.
Unfortunately GV has been having a ~week outage of outgoing MMS group messages. Well, let's say a brownout – many messages make it through to some recipients.
I switched to US Mobile a long time ago just due to pricing. I was WFH most days of the week (before Covid) so I could do minimal data and pay around $100/yr for unlimited talk/text. Now, I do unlimited everything as I commute 5 days each week, but it's only $200/yr. Still significant savings.
I like US Mobile a lot. That they're able to get postpaid priority on Dark Star is amazing.
I use them for my work phone, but there are a few things keeping me from switching away from T-Mobile for my personal lines (and I'd VERY MUCH like to switch):
- I have a family plan wherein I pay something like $254/mo for seven voice/data lines, two smartwatch lines and two tablet data lines. The phone lines all have unlimited data at full speed.
- T-Satellite just launched (wherein your phone uses Starlink when terrestrial towers aren't available). I'm not in this situation often but it can be useful.
- My plan provides free Wi-Fi on United. I fly a lot and use this benefit all of the time. Losing it would negate the savings I'd rack up from switching to US Mobile.
I really love US Mobile. I was a flagship postpaid $90/mo VZW customer for a decade and was so hesitant to switch. It's been 4 years now, and all I can say is that I can't believe I waited so long.
Lol it's network core. Postpaid users on Verizon branded accounts affected. Most MVNO's just fine (checked US Mobile and Visible, different cores... and they are working fine on the Verizon towers)
https://downdetector.com/ shows verizon, tmobile, and att. BUT if you look at the magnitude of the outages for tmobile and att vs verizon it's fractions of a percent. Likely those people with tmobile and att reporting when they have trouble communicating with verizon customers.
(Note, you have to click on the providers to see absolute magnitude -- the graphs are scaled to show relative outage over time within a given provider; order of 150k verizon vs 1.5k others)
"Verizon engineering teams are continuing to address today's service interruptions. Our teams remain fully deployed and are focused on the issue. We understand the impact this has on your day and remain committed to resolving this as quickly as possible."
As someone responsible my whole career for uptime and network response, I really feel for the engineers, at the same time hoping my service comes back up soon. SOS
I find this kind of odd. Yesterday, 2026-01-13, I - who lives in the greater Washington D.C. area - experienced A LOT of Verizon disruption. However, today, my service has been excellent. Maybe I'm from the future and don't know it. Did anybody else in my general area experience outages yesterday?
I've got a Verizon Network Extender and it appears to be online - the tunnel is up back to VZW's security gateway, but all of my phones are refusing to register to it.
I did manage to roam onto an international network on the boarder near me in New York/Canada, so some bits of the core seem functional for authentication.
When I roam internationally I appear to be on Telus's 3G network (no LTE) for data and voice is falling even further back it looks like.
My service was out for approximately 7 hours. But I pose a question that I've been wondering since:
I have a Moto. Updates are explicitly disabled. A lot of stuff has been deactivated via ADB. While Moto does try to force sneak updates against my settings and will, on occasion, I just so happened to grab my phone today and at approximately the very moment of the outage, noticed an unsolicited Moto update, which I cancelled. But at that moment, cellular was or went down (~1400) and remained so until ~2200.
Was this a coincidence, or could something much more interesting have happened, eg a serious security breach? An interesting coincidence, for me. I spent the day convinced that Moto sent a kill sig wanting me to buy a new phone.
Since this outage is still going on just a town away from their HQ... porting still works to get you going again on the same towers. You can get the device unlock pin from Verizon postpaid sent through a push message to the device so it doesn't rely on SMS.
Ironically I was planning to port my parents from Verizon to US Mobile to save them some money since they aren't financing any devices (there's a sale from them that ends today) and I've just done that on the first line and I now have service on that line with Verizon, where the remaining Verizon postpaid is still dead.
It's also weird that some people here in the office on Verizon work fine, and others are on SOS. No correlation between phone versions or hardware that I could deduce. I also see the same on X: https://x.com/CPTholen/status/2011520566159982758
It came up for around 5-10 minutes at 15:00 EST, but is currently still down.
If they're on different phones, they could be on different bands, different towers, and different paths, one or more of which could be impacted by whatever the underlying problem is. iPhone vs Android would be the most blatant tell that something like this is at fault, but it could also be different configurations from different stores causing them to interact with the cellular network in different ways.
Something like a routing configuration, BGP failure, or underlying network misconfiguration would cause seemingly bizarre results with some phones working and some not with no obvious correlation. Compare Access Point Names under Mobile Settings on android, and whatever the equivalent is on iPhones, and check things like whether 5G allowed and data roaming is enabled.
If it's a cyber attack of some sort, then there's all sorts of different attack vectors that would cause these outcomes.
My wife and I are on the same Verizon family plan. One of us can be down while the other is fine, then 30 minutes later it's the opposite. It's been like that all day.
i know alot are joking / sarcastic about its a cyber attack- that said, Wouldn't it make more sense that whenever there is a "cyber attack" its more likely it would only affect one provider? ie, each has to have different systems / security postures ect, such that a non-public vuln useful to attack Verizon would likely not be exploitable/exposed at AT&T (or vise versa)?
I get the sentiment, but at least for me at home, iOS iMessage still works fine with Wifi. So it's not impacted, and in fact I had to relogin to a client machine with a very persnickety 2FA and it had no issues.
I have two Verizon phones on very different networks and both have not been working well since Tuesday - anyone also having this? I kept restarting my phones, airplane mode on/off etc
Yeah, this is notable. If true, every mobile carrier getting hit at the same time says "coordinated incident", from whatever source and for whatever purpose.
Edit to say: my Verizon FioS and cell service are both working fine, no noticeable interruption at any point today.
Second edit to say: never mind, downdetector's home page normalizes report spikes so 1k and 100k both look identical.
I had a Verizon outage Monday. I called my neighbor who also has Verizon; hers was out.
But hers came back on when she paid her bill, oops! So...
I went to their website, which said "You have a local outage! Click here to know more...". I clicked there, and it said "There are no outages in your area." OK, so we've established that Verizon's website doesn't know shit.
I started troubleshooting with a tech support person ("I am in ... New... Jersey" - sure you are...), and discovered my optical interface whatsit was hanging by its cables in my basement, mysteriously knocked off its wall bolts. This happened sometime in the last 10 days, so it's a smoking gun.
I called for a repairman, because obviously the fall damaged the fiber alignment...
... except it didn't. The tree limb across the street that broke the cable coming into my house did cause a problem.
So... Verizon outages are complicated, and despite all their technical know-how, they don't really know shit.
The US would never purposefully cut of it's own communications, right? What would be the tells if that would have happened? And what could the average citizen do to be able to communicate regardless? Might be useful to always have that knowledge around, even if it isn't a personal threat to yourself today.
Get into Meshtastic or Meshcore! (So far I prefer meshtastic)
It's a citizen-run mesh network that allows text communication. It's not perfect and not 100% reliable (signal and hops limit delivery guarantees) but it's better than 0 ways to communicate in this kind of event!
schmuckonwheels|1 month ago
E.g., Cloudflare, Meta (who in doing so also locked themselves out of the building), and didn't some bumbling major Canadian telco knock themselves offline for like a week not too long ago?
jandrese|1 month ago
embedding-shape|1 month ago
shimman|1 month ago
These network topologies are incredibly complex and edges you think wouldn't exist have ways of suddenly appearing when things go awry.
cheeseprocedure|1 month ago
> In a letter to the CRTC, Rogers stated that the deletion of a routing filter on its distribution routers caused all possible routes to the internet to pass through the routers, exceeding the capacity of the routers on its core network.
https://en.wikipedia.org/wiki/2022_Rogers_Communications_out...
mikepurvis|1 month ago
I took my kids swimming that day and the pool couldn't take our money since the payment terminal was on the Rogers cellular network, so it was a free family swim.
Nextgrid|1 month ago
Cyberattack scenarios pretty much never make sense in case of complete outages; if you have the access required to cause such an outage it’s always more profitable to keep this access and use it for covert spying/targeted attacks or save it for later than to burn it by causing a massive, visible problem.
ocdtrekkie|1 month ago
unknown|1 month ago
[deleted]
venturecruelty|1 month ago
CGMthrowaway|1 month ago
mikeweiss|1 month ago
JacoboJacobi|1 month ago
mmooss|1 month ago
schmuckonwheels|1 month ago
Like all the doom and gloom after the Twitter layoffs predicting the site would implode and go permanently offline "within a month" which...never happened.
It's also ironic in the sense it implies the indignant people were so bad at their jobs they designed and built a system so fragile it would collapse without constant intervention from thousands of individuals.
You do realize it's possible for an organization to be overstaffed?
Nextgrid|1 month ago
I can't even begin to imagine what those 100k people actually do. For starters, none of the telcos actually develop their own equipment - they buy pre-made from vendors like Ericsson. Often that includes ongoing maintenance too. The only "engineering" is building the back-office and customer-facing UIs, and even that is often outsourced (as a rule of thumb, if something can be outsourced, telcos will do it: https://berthub.eu/articles/posts/5g-elephant-in-the-room/).
Customer service might be part of that number (assuming that too isn't outsourced), but even then 100k feels extreme.
10k is ok although leaning on the more bloated side. But 100k?
syntaxing|1 month ago
mtlynch|1 month ago
I'm in a similar boat to you in wanting a LoRa mesh. I tried out MeshCore on the LilyGo T-Deck+ hoping it would be a device I could hand out to family members, but I found the hardware and software disappointing.[0] But I'm weirdly tempted to try the LilyGo Pager.
[0] https://mtlynch.io/first-impressions-of-meshcore/#testing-th...
crims0n|1 month ago
apitman|1 month ago
The main downside is that you have different numbers for each eSIM, but that doesn't really affect me because I use Google Voice for SMS.
peaseagee|1 month ago
https://www.google.com/appsstatus/dashboard/incidents/THtvkD...
BirAdam|1 month ago
nunez|1 month ago
I use them for my work phone, but there are a few things keeping me from switching away from T-Mobile for my personal lines (and I'd VERY MUCH like to switch):
- I have a family plan wherein I pay something like $254/mo for seven voice/data lines, two smartwatch lines and two tablet data lines. The phone lines all have unlimited data at full speed.
- T-Satellite just launched (wherein your phone uses Starlink when terrestrial towers aren't available). I'm not in this situation often but it can be useful.
- My plan provides free Wi-Fi on United. I fly a lot and use this benefit all of the time. Losing it would negate the savings I'd rack up from switching to US Mobile.
tony_cannistra|1 month ago
Meekro|1 month ago
edoceo|1 month ago
joecool1029|1 month ago
cormorant|1 month ago
daveguy|1 month ago
https://downdetector.com/ shows verizon, tmobile, and att. BUT if you look at the magnitude of the outages for tmobile and att vs verizon it's fractions of a percent. Likely those people with tmobile and att reporting when they have trouble communicating with verizon customers.
(Note, you have to click on the providers to see absolute magnitude -- the graphs are scaled to show relative outage over time within a given provider; order of 150k verizon vs 1.5k others)
noncoml|1 month ago
Looking at the replies some of the people who are more worried about the outage seem to be the OF models
Not judging or trying to make a point. Just find it interesting the ways things are interconnected
relium|1 month ago
I noticed that a little while ago too. Very bad UI. Tempted to post in r/dataisugly.
_nickwhite|1 month ago
"Verizon engineering teams are continuing to address today's service interruptions. Our teams remain fully deployed and are focused on the issue. We understand the impact this has on your day and remain committed to resolving this as quickly as possible."
As someone responsible my whole career for uptime and network response, I really feel for the engineers, at the same time hoping my service comes back up soon. SOS
downrightmike|1 month ago
mmooss|1 month ago
NoSalt|1 month ago
gjstein|1 month ago
asne11|1 month ago
kotaKat|1 month ago
I did manage to roam onto an international network on the boarder near me in New York/Canada, so some bits of the core seem functional for authentication.
When I roam internationally I appear to be on Telus's 3G network (no LTE) for data and voice is falling even further back it looks like.
wildzzz|1 month ago
threecheese|1 month ago
jedberg|1 month ago
eth0up|1 month ago
I have a Moto. Updates are explicitly disabled. A lot of stuff has been deactivated via ADB. While Moto does try to force sneak updates against my settings and will, on occasion, I just so happened to grab my phone today and at approximately the very moment of the outage, noticed an unsolicited Moto update, which I cancelled. But at that moment, cellular was or went down (~1400) and remained so until ~2200.
Was this a coincidence, or could something much more interesting have happened, eg a serious security breach? An interesting coincidence, for me. I spent the day convinced that Moto sent a kill sig wanting me to buy a new phone.
_nickwhite|1 month ago
joecool1029|1 month ago
Ironically I was planning to port my parents from Verizon to US Mobile to save them some money since they aren't financing any devices (there's a sale from them that ends today) and I've just done that on the first line and I now have service on that line with Verizon, where the remaining Verizon postpaid is still dead.
unknown|1 month ago
[deleted]
johnisgood|1 month ago
eob|1 month ago
I assume state on state cyber attacks are commonplace but get minimized to avoid public fear.. perhaps this will be the first notable one.
in0v8r|1 month ago
mosura|1 month ago
Made worse by the fact Estonia is a more networked society than, for example, the US.
swsieber|1 month ago
https://downdetector.com/status/t-mobile/ ~ 1,600
https://downdetector.com/status/att/ ~ 1,500
https://downdetector.com/status/verizon/ ~ peaked at ~169k, dropped to 67k
mrits|1 month ago
_nickwhite|1 month ago
It came up for around 5-10 minutes at 15:00 EST, but is currently still down.
observationist|1 month ago
Something like a routing configuration, BGP failure, or underlying network misconfiguration would cause seemingly bizarre results with some phones working and some not with no obvious correlation. Compare Access Point Names under Mobile Settings on android, and whatever the equivalent is on iPhones, and check things like whether 5G allowed and data roaming is enabled.
If it's a cyber attack of some sort, then there's all sorts of different attack vectors that would cause these outcomes.
Meekro|1 month ago
bobbob1921|1 month ago
dylan604|1 month ago
kensai|1 month ago
_nickwhite|1 month ago
fron|1 month ago
Scubabear68|1 month ago
nunez|1 month ago
hprotagonist|1 month ago
tantalor|1 month ago
malfist|1 month ago
Vaslo|1 month ago
downrightmike|1 month ago
refulgentis|1 month ago
rcpt|1 month ago
asne11|1 month ago
Scubabear68|1 month ago
syntaxing|1 month ago
sumtechguy|1 month ago
unknown|1 month ago
[deleted]
lbcadden3|1 month ago
ardit33|1 month ago
memcg|1 month ago
alexzenla|1 month ago
RandallBrown|1 month ago
neb_b|1 month ago
reb|1 month ago
Edit to say: my Verizon FioS and cell service are both working fine, no noticeable interruption at any point today.
Second edit to say: never mind, downdetector's home page normalizes report spikes so 1k and 100k both look identical.
swsieber|1 month ago
IAmBroom|1 month ago
But hers came back on when she paid her bill, oops! So...
I went to their website, which said "You have a local outage! Click here to know more...". I clicked there, and it said "There are no outages in your area." OK, so we've established that Verizon's website doesn't know shit.
I started troubleshooting with a tech support person ("I am in ... New... Jersey" - sure you are...), and discovered my optical interface whatsit was hanging by its cables in my basement, mysteriously knocked off its wall bolts. This happened sometime in the last 10 days, so it's a smoking gun.
I called for a repairman, because obviously the fall damaged the fiber alignment...
... except it didn't. The tree limb across the street that broke the cable coming into my house did cause a problem.
So... Verizon outages are complicated, and despite all their technical know-how, they don't really know shit.
matltc|1 month ago
Some server in NJ is down
morpheos137|1 month ago
adsfsadf|1 month ago
unknown|1 month ago
[deleted]
unknown|1 month ago
[deleted]
andrewinardeer|1 month ago
embedding-shape|1 month ago
fractal618|1 month ago
jerlam|1 month ago
BART Defends Cutting Off Cell Service In Subway
https://www.npr.org/2011/08/16/139664637/bart-defends-cuttin...
And will happen again.
chankstein38|1 month ago
It's a citizen-run mesh network that allows text communication. It's not perfect and not 100% reliable (signal and hops limit delivery guarantees) but it's better than 0 ways to communicate in this kind of event!
Meekro|1 month ago
unknown|1 month ago
[deleted]