top | item 45139270

A computer upgrade shut down BART

234 points| ksajadi | 5 months ago |bart.gov

363 comments

order

Some comments were deferred for faster rendering.

VonGuard|5 months ago

Sooooo much snark, and so little interest into what BART actually runs on!

Originally, BART was a master stroke of digital integration in the 70's, and it's digital voices announcing the next trains were a thing of the future: An early accessibility feature before we even knew what those were, really.

Reading:

https://www.bart.gov/about/history

https://www.bart.gov/about/projects/traincontrol#:~:text=To%...

dlcarrier|5 months ago

I know what it runs on! It's a 5' 6" in gauge, usually used in India, and used no where else in the US.

bbaron63|5 months ago

I believe in one of the Planet of the Apes sequels, they used a BART construction site, because of how futuristic it looked.

esalman|5 months ago

I lived mostly car free in Atlanta because the Marta station is one flight of stairs down from the airport terminal, and I could get to my lab in GSU in downtown Atlanta in less than 30 minutes, midtown Georgia tech campus in similar time, my first apartment in Lindberg in 40 minutes, and my second apartment in Sandy Springs on the other side of the city in less than an hour from the airport. Commute to and from my school/lab/apartment was always under 30 minutes and always faster by train compared to car.

These days I fly to the bay area to my office in East Bay. It's 2+ hours commute from either SFO or even OAK because you need to change buses 2 or 3 times. Add 1 more if you count taking the airport shuttle to the BART station. And SJC does not even have a BART connection.

There's fundamental design flaw in public transportation in the US, they almost never connect the population centers. Part of the reason why people are discouraged from using them and they don't get the funding to stay up to date.

linguae|5 months ago

I travel to Japan twice a year for business and for vacation, and coming back to the Bay Area and dealing with its transportation infrastructure is always jarring.

I find the Bay Area very difficult to get around. The roads are jammed with commuters who live far from their workplaces due to the housing situation. There is not enough housing near job centers, which bids up the prices of available housing to very high levels that requires FAANG-level salaries to clear unless one wants to have an army of roommates. Thus, many people have to commute, some from far-flung exurbs and even from Central Valley cities like Stockton and Modesto.

Public transportation in the Bay Area is better than most American cities, but it’s still underpowered for the size of the metro area. Not all residences are served by trains, and bus service is often infrequent and subject to delays. Missing a connection can lead to major inconveniences (such as a long 30-60 minute wait) or even being unable to reach your destination without an über-expensive Uber or Lyft ride. There’s also matters of safety and cleanliness on public transportation; every now and then I smell unpleasant odors like marijuana and urine, and occasionally I see sketchy people.

It’s a major step down from Tokyo, where public transportation is ultra-convenient, reliable in non-emergency situations, impeccably clean, and generally safe.

The sad thing is the reason the Bay Area lacks Tokyo-style transit is not technology, but social and political issues. If it were merely technology, we’d have solutions by now.

kulahan|5 months ago

I don't think this is a very big reason. I'm absolutely convinced people in the US are just used to cars, and like with any new piece of software, it has to be 10x better in some way for people to start using it en masse.

Maybe it's a matter of breaking down the costs for everyone to see, or maybe it's a matter of the city providing bus wifi so you can get some guaranteed access to the internet while riding, or maybe it's a matter of putting a police officer on every train.

But busses, aside from rush hour in probably the 10 largest cities in the nation, are always going to be way less convenient than a car. It has to stop a million times, there's no good way to guarantee you'll arrive on time (it's impossible to create a bus route where they stay evenly spaced like a train might handle better), and they never actually get you where you're going - just kinda nearby. Maybe you can transfer onto a bus now, but that's two modes of transportation. And God forbid there's a number of people combining their bus usage with a bicycle. Gotta wait for them to walk around front, unhook it, and hopefully put the bike rack back up so the driver doesn't have to get out and do it himself... etc, etc, etc.

Plus, I'm too busy to find it at the moment, but there's a study showing most people just want public transit so some other people use it and get off the highway. As in, they just want public transit so their car commute improves.

This will almost certainly never get major support; it's just too miserable of a system to overtake our already-crazy-convenient cars.

dylan604|5 months ago

Part of the reason why people are discouraged s/from/by/ using them and they don't get the funding to stay up to date.

People are constantly being encouraged to take public transpo, but once they finally do, they realize why they hadn't before.

kelnos|5 months ago

That's a problem in San Francisco proper too. If I think about my trips to Japan, in the population centers, at best, a car trip will take about the same time as a public transit trip, but the majority of the time transit will be faster, sometime significantly so.

But inside SF, even during rush hour, it'll still nearly always be faster for me to drive (or get an Uber). The reason is because there's precious little transit infra that doesn't share the road to some extent, and even when there are dedicated bus lanes or off-street train tracks, there's still traffic lights, and the buses and trains are slow and make enough stops that any gains are lost. Then on top of that, transfers take time, and if you're even slightly off on your timing, you might have to wait for up to 15 minutes for the next bus at your transfer point.

I agree with your assessment of inter-city trips as well; SFO airport to my house in SF is also so frustrating, because I live a few blocks from a Caltrain station, but having to go from SFO->BART->Caltrain->home... that transfer in the middle is a killer. My home is only a few blocks farther to the freeway than to the train station, so even in rush-hour traffic it's still only a 20 minute drive, while BART+Caltrain will take 30-45 minutes, and that's during a time of day the Caltrain trains run at their most frequent.

I've lived in SF for 15 years, and I think I've only taken BART/Caltrain to or from SFO a handful of times. I can't even remember the last time; it's been at least 10 years (probably before Uber/Lyft was a thing). Nowadays I always take a Lyft, and while I cringe at the price ($30-$50, depending on time of day), it's so worth it when a) I'm worried about not making my flight if transit is slower than I expect, or b) I'm getting back home and just want to be home.

And yeah, I get that I'm privileged enough to be able to afford to take a car. Many people aren't; they have to pay with their time, which just really sucks. We never get that time back.

halfmatthalfcat|5 months ago

Chicago (Blue Line from O'Hare) and NYC (M60 from Laguardia or Skytrain to MTA/LIRR from JFK) are also good in that regard.

Buuntu|5 months ago

Everyone here blaming BART and bureaucracy for being inefficient when in reality it's starved for funding due to our own voting (and zoning preventing housing/badly needed ridership near transit stops). Yes it's expensive to build transit just like it's expensive to build anything in America, which we should fix but that is not unique to BART.

It's quite possible the system will collapse next year if we don't pass increased taxes to fund it in 2026 https://www.bart.gov/about/financials/crisis.

Just last year we failed to pass a common sense bill to make it so we only need a 51% majority for transit bills in the future, indicative of how opposed we still are to transit in the Bay Area https://www.cbsnews.com/sanfrancisco/news/california-proposi....

Not to mention the fact that Silicon Valley opted out of BART and chose car dependent sprawl instead.

So let's be clear, most of the issues with BART are due to anti-transit and suburban voters starving it of support.

dilyevsky|5 months ago

Hilarious that from 2020 and to this day ridership has collapsed but BART operating expenses went up despite that and all the efficiencies they talk about in your link. Kind of tells you everything you need to know about where the money is actually going...

Just to compare with another expensive city - BART serves 1/20th of London's Tube rides while operating on 1/5th of the Tube's budget.

kqgnkqgn|5 months ago

I wouldn't consider myself anti-transit - before Covid I took BART every work day and currently walk to my office. And have never regularly commuted by car in the Bay Area. But in SF, we seem to keep throwing money at transit orgs through ballot measures, and getting little tangible results in return. I voted for funding increases for Muni for years, with supposed reliability / service enhancements that never seemed to materialize. It's disappointing that rather than hearing that voters are more hesitant to fund this now vs previously, the reaction would be to try to lower thresholds to get things passed.

Even with the new Central Subway that opened in SF (which I assume cost billions given how long it took to develop), wasn't a clear net-win. Muni closed other Metro routes when those opened. Depending on where you're going, you might be worse off now.

While RTO may be increasing ridership numbers, Covid did change population and commuting dynamics. Transit orgs need to adapt, and maybe accept downsizing / focusing more on a smaller scope. In Bart's case, maybe it would be wiser to focus on the core Bart system, and not the more recent expansions (the East Bay trains that are totally separate from the rest of Bart, and the Oakland airport train). Maybe a stronger look should be taken at merging the disparate transit organizations themselves, to reduce administrative overhead?

Caltrain seems to be doing better than others - they have financing worries themselves, but are on a better track from my understanding. Pun semi-intended :)

Transit is important, and I feel like the current organizations keep letting us down.

jjice|5 months ago

> when in reality it's starved for funding due to our own voting

Everyone wants more services and lower taxes, but they vote for the lower taxes and get made when there are no services. Those things often don't go together. It's okay to either accept fewer services with less tax burden, or higher taxes with more services (the side I generally lean towards, within reason).

chuckadams|5 months ago

It's pretty hard to keep from drowning in despair when one realizes that almost everywhere else in the USA except maybe NYC, the situation is worse.

nradov|5 months ago

The failure of Proposition 5 doesn't indicate that California voters are opposed to transit. That was a very broad proposition which lowered the voting threshold for local governments to issue bonds for a wide variety of projects, not just transit. Local governments are already facing debt problems and making it easier to take on more debt would set them up for serious future fiscal problems.

vondur|5 months ago

It looks like BART usage is way down from pre-pandemic levels, around half of what it used to be, and to top it off the BART system has added over 300 additional employees since 2019. It may be a tough sell to convince taxpayers to fork over more money to them.

dylan604|5 months ago

> Not to mention the fact that Silicon Valley opted out of BART and chose car dependent sprawl instead.

Didn't bigTech start buses going directly to their campus as a perk?

crooked-v|5 months ago

Good ol' Prop 12, guaranteeing that everything will be underfunded one way or another.

Animats|5 months ago

Are there any technical details yet? What was upgraded?

linguae|5 months ago

I’m at a conference at Stanford University right now. I was going to take BART and the Dumbarton Express to avoid having to drive in traffic, but when I drove to the Dublin BART station, I found out BART wasn’t running. I ended up having to drive to Stanford, since the only public transportation over the hills separating Dublin/Pleasanton from the inner East Bay is the Altamont Commuter Express, which is much less convenient due to its few runs. Thankfully traffic wasn’t that bad today, but going home is going to be a traffic nightmare since it’s a Friday.

I wish there were more bus options that connected the outer East Bay (Dublin, Pleasanton, San Ramon, Walnut Creek, etc.) to the inner East Bay.

dylan604|5 months ago

> but going home is going to be a traffic nightmare since it’s a Friday.

I've never understood the Friday traffic issue. Are there people that normally stay in the city during the week and only go home on Fridays causing more traffic? How is there more traffic on Friday and the rest of the week? Friday being one of the forced RTO days, but the Friday traffic thing was known well before WFH/RTO fights. Then again, the root cause of most traffic always seems much more anticlimatic

coldest_summer|5 months ago

sorry using throwaway for this.

When GitHub was constantly failing, I finally got fed up and now I use my own private Gitea. It’s near-zero maintenance and has never had any unexpected downtime. Never looked back.

Stories like these make me feel the same way about California, which I called home for almost 20 years. So much to love, so many reasons never to live there again. Great place to visit when there’s not an active disaster unfolding.

outlore|5 months ago

mate we can't self host BART

jasonjmcghee|5 months ago

I'm curious what percent of HN is based in the bay area for this to hit the front page so quickly. I suppose it could in part be that it was posted when people are commuting in?

MBCook|5 months ago

A tech failure taking down a big government thing is certainly HN worthy. And BART is relatively famous, as such things go. It’s a name people know, as opposed to of it was the Minnesota DMV system. That would be a fine story too but no one knows the name for that.

nottorp|5 months ago

I'm not even on the same continent but I'm still reading this, including the comments...

0xffff2|5 months ago

HN has always had a huge bay area focus.

darth_avocado|5 months ago

Is it causation or correlation? Maybe bart being down caused all the people to browse HN while waiting for the issue to be resolved, thereby making this show up on the main page.

RcouF1uZ4gsC|5 months ago

Public transportation is inherently centralized.

Cars are anti-fragile and decentralized.

Cars fail open in the short term.

loire280|5 months ago

Buses are the resilient backup for trains, especially if road infrastructure has been designed to prioritize transit (e.g. Chicago highways with shoulders designed to let Pace buses bypass traffic jams).

rafram|5 months ago

Tell that to someone in a two-hour traffic jam on the highway.

formerly_proven|5 months ago

Traditional train systems themselves are extremely decentralized, though scheduling is not. Traditional interlockings form a mirroring mesh network parallel to the physical network of steel rails itself.

namuol|5 months ago

No. Cars rely on centralized road systems.

lxe|5 months ago

Was this related to the CBTC rollout by any chance?

EDIT: It was not.

slowhadoken|5 months ago

Yeah the BART needs some love.

mdaniel|5 months ago

I'm on my phone and not able to readily dig up the link, but there was a linguistic study about the differences between the audience who use definite articles for highways/freeways/interstates and the transit system, versus those who just use the distinctive noun ("101", "80", "Bart")

I thought it was interesting and I'm sorry a hint at it is all I can offer right now

benced|5 months ago

I’m not frustrated that this happened, I’m frustrated that it seems likely this won’t get better (witness their protracted incredibly high constructions costs that have not improved). I hope they prove me wrong.

diebeforei485|5 months ago

Incompetence. Someone should be fired for this.

giardini|5 months ago

Windows, upgrading again?

fmbb|5 months ago

Nah, we upgraded the network configuration. Should have no impact. No there is no source control.

phkahler|5 months ago

It'd be pretty cool if busses and trains were local-first.

gjsman-1000|5 months ago

If you can't send updated schedules or emergency alerts through the system, I also don't want service started. It doesn't have to be an individualized problem to render local-first useless.

Also, what do you mean by trains being local-first? Trains by definition need to share the same tracks with catastrophic consequences for getting it wrong. You can't figure out if a train is going to possibly be on the same route locally, or if your route has been obstructed. Somebody gets a schoolbus stuck on a crossing, it takes over a mile to stop a train.

ok123456|5 months ago

Did the upgrade also break scrolling on their site?

rafram|5 months ago

Your ad blocker is probably blocking a modal popup, badly.

hed|5 months ago

You'd think trains would use a rolling release

wavemode|5 months ago

They clearly need to rebuild this as a Rails app

CartwheelLinux|5 months ago

Also surprised they don't have the ability to rollback

cortesoft|5 months ago

Broken release train breaks train brakes

nova22033|5 months ago

Works on my local environment <points to train set> choo choo

dylan604|5 months ago

so you're saying BART should run in a container?

wills_forward|5 months ago

The cheap easy take: it's tragically ironic that the software running the infrastructure in Silicon Valley is such a problem

dilap|5 months ago

It's a shame that SF politics are so dysfunctional it can't have a metro at the same level of quality as, say, North Korea.

gdulli|5 months ago

Maybe expected though that high salaries there depress incentive to work in these jobs even more than other cities?

mleonhard|5 months ago

BART is a government organization and all California government employee pay is public. You can see that BART has about 40 software engineers and they earn about 70% of the market rate:

https://transparentcalifornia.com/salaries/search/?q=compute...

https://transparentcalifornia.com/salaries/search/?q=compute...

It seems to me that they are over-worked & under-paid and are doing a good job given the circumstances.

NIMBYs have blocked BART in Silicon Valley. BART doesn't reach Menlo Park, Palo Alto, Stanford, Mountain View, Sunnyvale, Los Altos, Santa Clara, or Cupertino. A few years ago, it finally reached San Jose.

A separate train (CalTrain) goes from SF through Silicon Valley. Last year they switched to electric trains which are faster and run more frequently. The SF CalTrain station is inconvenient (20-mins walk from downtown, under a highway), but they are working to extend CalTrain to the central SF station: https://en.wikipedia.org/wiki/Salesforce_Transit_Center#Futu... .

So Silicon Valley transit is getting better, slowly.

jerlam|5 months ago

BART barely goes into Silicon Valley. Fremont was the closest stop up until 2017. Now it gets to North San Jose. Even if was funded, any further extension wouldn't be complete for over a decade.

some-guy|5 months ago

I'll bite: Silicon Valley isn't known for good infrastructure, we are just able to roll back changes very easily. The cost of getting software wrong for BART is far higher than if my div is padded incorrectly.

ninetyninenine|5 months ago

I mean despite it's history the snark is well deserved. With so many companies and people in the bay paying taxes, where the hell does all the money go?

Interesting, tidbit you added here. But snark is needed for this situation.

jeffbee|5 months ago

It certainly doesn't go to Bay Area software companies. When BART originally began letting out the contract to redesign the then-already-obsolete control system in 1992, they awarded it the Hughes Aircraft. That project failed. The current attempt to deploy CBTC was awarded to Hitachi. The supplier of their fare gate system integration was originally IBM and is now CUBIC, a San Diego defense contractor.

If anything the Bay Area has utterly failed to provide systems software of lasting value to address public needs like these.

IshKebab|5 months ago

Yeah I was pretty blown away when I visited San Francisco just how archaic it was. In the same place you have driverless cars you have a metro payment system from like 70s USSR or something.

octernion|5 months ago

your tax money broadly speaking doesn't go to BART; it's massively underfunded. not sure why they are the target of the snark.

buckle8017|5 months ago

A significant amount of BARTs budget goes to inflated salaries for operators and ticketing staff.

They have very little money left for paying engineering and construction staff.

SuperHeavy256|5 months ago

snark is not productive.

ants_everywhere|5 months ago

It's hard to think of a single situation that's ever been improved by snark. Or passive aggressiveness in general.

bravetraveler|5 months ago

break things and don't move at all

edit: lmao, so many upvotes yet my comment has been moved so low. No more snark than a loving brother would provide. TY for your attention to this matter

kelnos|5 months ago

I generally downvote jokes, even the clever ones, because I don't come here for that (reddit is generally more fun for that; even though many are low effort, there are many that are just so clever), and I'm mainly interested in thoughtful prose and discussion. (Which sometimes is hard to come by here, too, but jokes are obviously not it.)

Probably what's happened is you ended up with a lot of upvotes, but also a lot of downvotes. I would expect HN's software to downrank "controversial" posts, since those are likely to lead to flamewars. So even if you see +30 on your comment, the overall tallies might be something along the lines of +100 -70.

dayyan|5 months ago

[deleted]

rvnx|5 months ago

[deleted]

QuercusMax|5 months ago

Alternatively, get good at doing rolling releases so you don't take down the entire system and have some sort of canarying process.

blamarvt|5 months ago

This is not how software works. Although I guess this isn't quite as catchy:

Assume all software is broken at all times. Constantly try to ensure it works and is secure. Sometimes updates break things. Test before production. Ensure test environments are similar to production. You're going to break things.

xyst|5 months ago

[deleted]

nimbius|5 months ago

The bart is basically the crowning achievement of US public transit. As for the solutions coming from everyone's favourite bay Aryan Elon Musk, they are...somewhat lacking.

You're probably not going to believe this but the Hyperloop in Las Vegas:

- is now just "the loop."

- only has 8 stops

- doesnt go to the airport

- most stations are unprotected park benches in the desert sun

- vehicles arent driverless

- speeds are 26 miles per hour instead of 155

- it can take up to 20 minutes for a ride to show up

- it does not go to or from the airport.

- it only runs for 11 hours a day at some stations.

- cost taxpayers fifty-three million dollars.

thephyber|5 months ago

[flagged]

jama211|5 months ago

Public infrastructure is requested and funded by the government, not voluntarily done by companies that happen to base themselves nearby. Sounds more of an issue of government.

nradov|5 months ago

BART is unique and doesn't share much in the way of infrastructure with any other public transit system. You can't build a scalable startup targeting BART because you'd have a maximum of one customer.

The Boring Company has attempted to develop tunnel boring technology which theoretically could someday allow for cheaper expansion of all subway and light rail systems. Although in practice they haven't accomplished much and their existing projects aren't even used for rail transit.

https://www.boringcompany.com/

There are also several eVTOL startups aiming to improve quality of life through rapid point-to-point transportation. But I doubt they'll succeed on any widespread basis due to battery and noise limitations.

johnebgd|5 months ago

This is government procurement being broken not the companies themselves.

Gibbon1|5 months ago

The problem with the US is it's been taken over by finance capitalists, and lets be honest, VC's are finance capitalists. And finance capitalists are essentially slumlords.

Their first reflex when it comes to paying for infrastructure and maintenance is to think what that'll do to their short term CAP rates. And then they get angry.

nilsbunger|5 months ago

Seems like BART should do these upgrades only at low traffic times, like overnight Saturday night.

ForOldHack|5 months ago

They did. They started before yesterday's shutdown, and worked all night, they tried to bring up the system for startup, and it came up, then crashed.

It was state of the art on 1962 when it was designed, and remained state of the art until the 1980s, when the signal system started breaking down, and the the late 80s upgrade which had a train presence glitch, which caused almost all the system to get resignaled.

So by the 2000s again it's showing its age, and they got a 32 processor zSeries mainframe.

Brake problem last week, and the this on Friday? Now it's getting like New York, even more. Whatsmatteryou?