top | item 32884814

How mechanics keep 50-year-old BART trains running: Windows 98, eBay, and scraps

210 points| Stratoscope | 3 years ago |mercurynews.com | reply

192 comments

order
[+] jalk|3 years ago|reply
There is something wrong with the description of the laptop use (besides calling DOS a language) - on one hand they are running windows 10 with Win98 virtualized. Then they mention Thinkpads circa 2000. I'm pretty sure Win10 won't run on those machines, so I guess the old laptops are for the interface to the train computers - RS-232 is my guess (could be the parallel port). If all they are using the DOS software for is to dump a log from the train, then it sound to me, that sniffing the protocol would be fairly simple. Calling people from bay-area hacker spaces to give them a hand in creating something more modern.
[+] parker_mountain|3 years ago|reply
> that sniffing the protocol would be fairly simple

ah yes, "fairly simple". The protocol is not the issue, the data within is. There are hundreds of datapoints, various switching logic, and tons more of 50 years of accumulated cruft. At least. All in a well tested, stable, and understood package. The documentation is probably pretty good but likely has small (but crucial) errors.

It's easy to assume that this would be a simple rewrite, but real engineering (not the kind we do on hacker news! lol) requires tons of planning, testing, and phasing in. This is dealing directly with people's safety after all! It's interfacing with a train control system, not a webapp that turns markdown into memes.

also

> RS-232 is my guess (could be the parallel port)

RS-232 is inherently a serial protocol. If you're using RS-232 since basically ever, it's either a built in serial port, some sort of serial headers, or some sort of usb-to-serial adapter. The parallel port has never been directly used for RS-232 - they are fundamentally different technologies.

[+] RajT88|3 years ago|reply
Would agree with all of this.

The problem is that BART doesn't employ talented reverse engineers. There is now way they could afford them in that area.

Also: Talented reverse engineers are hard to find on a good day in any given area. I used to think that my mentors from the start of my tech career were just the baseline level of sophistication you'd expect of any decent software engineer. 20 years later I have seen how few people have the mindset to get into that sort of thing. (And it is a mindset much more than a level of intellect, IMO)

[+] origin_path|3 years ago|reply
If you read to the end of the article you see:

"Finding replacement parts won’t be a problem and DOS is no longer necessary for Allen and his crew, but it will still be bittersweet to say goodbye to trains that they have lovingly kept alive."

So they don't actually use DOS anymore, maybe, or possibly this just means they don't care because the new trains will soon replace the old ones and then DOS will be gone too. It's a poorly written article even allowing for the usual tech mistakes but either interpretation would be reasonable.

[+] Tanoc|3 years ago|reply
The big problem it seems like is Windows Vista dropped legacy support drivers and 16-bit support fifteen years ago, and Windows 10 dropped even more legacy support drivers. So they keep the ThinkPads around because they can't find the drivers for some things anymore. I've run into that a few times, where old drivers were made specifically for DOS 7/8 or even DOS 6 to interface with custom fabricated CNC machines or injection mould machines. In the end the company that originally created those few hundred units no longer exists, and nobody has the floppies that held the original drivers or software. So you keep cloning the drives for when they fail and hoping there's enough near-spec hardware spares to hold out until a total system replacement.
[+] honkdaddy|3 years ago|reply
>Calling people from bay-area hacker spaces to give them a hand in creating something more modern.

Tragically, from the small amount of time I've spent in SF, my understanding is this is unlikely to be their first instinct, even if it's a great idea. Doesn't the SF local government have a testy, if not outright hostile relationship with the tech worker/hacker/startup scene?

I've met people from either side of the discussion who pretty much entirely blame the opposite group (hackers or city workers) for why SF is so different now than 'back in the day.'

[+] carlivar|3 years ago|reply
I'm surprised the article didn't mention that BART is the only train in North America on the uncommon 5'6" gauge. This is another factor making parts and equipment more difficult to source.
[+] MAGZine|3 years ago|reply
I don't think most parts and equipment are swappable anyhow. Maybe tracks themselves. But as far as the trains are concerned, you're pretty much buying all of the trains bespoke from the manufacturer.

If SF used standard gauge rail, it's not like they could just buy some surplus rail cars, put some seats in them, and call it a day. Subways are generally lower to the ground because it improves stability and reduces roll. They corner better. I'm not sure that any subway system uses regular old heavy rail train cars.

And besides, if you're ordering thousands of cars for replacement from bombardier (or whomever), you can order spare parts while you're at it.

[+] jabl|3 years ago|reply
From what I've read (I think there was discussion on HN about it a few months ago?), it's not only the gauge. The entire system was a 'clean sheet' design, made by aerospace engineers. Everything is totally different from what you'd expect in any normal (heavy, light, subway) train.
[+] djbusby|3 years ago|reply
It also makes the trains stable; wide stance.
[+] bluGill|3 years ago|reply
Also known as India broad gauge, which while not common isn't uncommon either.
[+] sidewndr46|3 years ago|reply
My understanding is BART also uses 1000 VDC for power rather than the more common 600 VDC.
[+] mgraczyk|3 years ago|reply
BART is pretty unreliable in part due to maintenance problems, but also due to system design.

For example two days ago there was a horrific final-destination-esque crash in which a motorcyclist happened to be in front of the train.

https://twitter.com/sfbart/status/1570997016347680768

This crash caused the entire system to be disrupted for hours, even very far away from the crash. According to BART, "We don’t have extra tracks to maneuver around problems. That’s why our system comes to a halt from one incident."

[+] lostdog|3 years ago|reply
"We shared the info [of an early report of someone being on the tracks] and some took to twitter declaring that the person was a junkie on our tracks and it is BART's fault.

We hope you have all since deleted your tweets."

I am looking forward to the future of aggressive-aggressive tweeting from public transit agencies.

I've also always wondered if there was any way to help BART, sort of in a USDS style, or if the problems really are paltry budgets and mismanagement, and so joining in an engineering capacity wouldn't help at all.

[+] _skel|3 years ago|reply
Caltrain also has limited ability to route around accidents, and in some cases has to run trains the "wrong way" on the other side of the tracks to get past certain areas.

They also have no communication at all. Back when I had a Caltrain commute I would use Twitter to find out if there were accidents or delays, because the station signs were alway wrong and almost all of the stations are unstaffed. I would see posts on Twitter from people who were on a train that crashed into a car and the station signs would still say everything was on schedule.

[+] dehrmann|3 years ago|reply
> For example two days ago there was a horrific final-destination-esque crash in which a motorcyclist happened to be in front of the train.

Isn't Bart entirely grade-separated?

But yes, no ability to single-track is an issue.

The non-standard rail gauge is also an ongoing issue. Everything has to be custom. Compare this to Caltrain. Several years ago, they wanted to add capacity. They bought off-the-shelf Bombardier cars from an agency in Socal. I'm not sure if they ever even painted them, but they could use them almost immediately.

[+] Rebelgecko|3 years ago|reply
IIRC that's also why the hours of BART are so limited. They have to do all their maintenance when the train isn't running since they can't route around it
[+] chrisdhoover|3 years ago|reply
BART has grade crossings? Isn’t entirely underground or elevated? Caltrain has grade crossings along its entire length.
[+] oneplane|3 years ago|reply
It's almost like the people in charge aren't really affected by it, and therefore don't really want to put any focus and effort (money, time, attention) in to making it a bit more reliable and attractive.
[+] planetsprite|3 years ago|reply
As is the case for 90% of public services in the United States.
[+] clintonb|3 years ago|reply
What are you talking about? The "Fleet of the Future" (or Deloreans, as I call them) now comprises a third of the cars in operation. The older cars (space shuttles) are being phased out.
[+] LatteLazy|3 years ago|reply
Just to be "that guy" for a minute...

Windows 98? The DLR (Docklands Light Railway) still uses computers from the 80s for their self driving and signalling systems. They have a room full of them and when the last one dies, that's it.

London Underground were at one point the worlds largest manufacturer of vacuum tubes for their signal systems (can't buy them, I guess we have to build them)

Train systems are one of the ultimate examples of people not thinking long term, complex systems and infrastructure that cannot be closed even for a few days. So you often get these weird special cases.

[+] Symbiote|3 years ago|reply
If the DLR has a room full of spare computers, doesn't that show they are thinking long-term?

Stockpiling £50,000 of computers might well be cheaper and lower-risk than upgrading a signalling system.

[+] DIARRHEA_xd|3 years ago|reply
Not sure if this is still true after SelTrac upgrades
[+] Aloha|3 years ago|reply
In fairness, they're in the midst of a major fleet replacement which will theoretically be done by next year.
[+] dang|3 years ago|reply
I hope so. The difference between the old cars and the new ones is striking enough that I've noticed myself holding my breath waiting to see which I'm about to get in.

I fear it won't take long for the new ones to degenerate, though; I've even noticed some signs of that already.

[+] V-eHGsd_|3 years ago|reply
i expect a bart bond measure in basically every cycle.
[+] fibonacc|3 years ago|reply
Its puzzling how the world's richest country spends more than a country like South Korea on public transit and still stuck using Windows 98 and other ancient antics.

For instance the Seoul Metro yearly budget is 10x less than New York's yet it runs far longer and more efficient. It's also safer.

[+] midoridensha|3 years ago|reply
Construction costs (for buildings or tunnels) in NYC are also many times higher than anyplace else in the world, largely due to corruption.
[+] MAGZine|3 years ago|reply
when something needs to run day-in and day-out every day, it is difficult to justify swapping things just because they're old.

as it turns out, old things have stood the test of time and generally work quite well.

vancouver bc upgraded its own train control systems a handful of years ago. unsurprising to anyone who has tried to do a migration, there were service incidents. trains stuck in the middle of nowhere sort of thing.

It's all done, upgraded, and modernized yet. But there is a growing pain and growing cost. The tricky thing to know is when the juice is worth the squeeze.

[+] quake|3 years ago|reply
Not to mention the amount of similarities between SkyTrain and BART with the automated system and custom parts. It didn't even strike me as odd that the trains didn't have drivers until a dump of snow hit the rails and someone had to open the front compartment of the train to access the manual controls to drive it into Columbia Station (why is it always Columbia?)
[+] nspattak|3 years ago|reply
If only there have been a more appropriate idea to avoid proprietary software and protocols.

Oh wait, there is free software/open source!

it is really astonishing how people are still willing to be locked-in and go to extreme lengths to continue to be locked-in.

[+] wildrhythms|3 years ago|reply
Such is the case when government projects are built on the idea of 'the lowest bidder' making a quick buck.
[+] rayiner|3 years ago|reply
The decline of BART and Metro is a good metaphor for the decline of American government competence. The 1970s saw a wave of state of the art automated train systems built in SF, DC, and ATL. When we moved to DC in 1989 the Metro offered a smooth, comfortable, computer-controlled ride.

Today those systems are in shambles. The automatic train control infrastructure in the DC metro has deteriorated so much that it was turned off in 2009. Today a DC Metro ride is uncivilized, like the ancient system in NYC, with some poorly trained driver jerking you to a stop at each platform. Not only that, but the 6 minute headways built into the original system were relaxed to 8 minutes, and even that is not achieved in practice.

[+] LeafItAlone|3 years ago|reply
> Today a DC Metro ride is uncivilized

This is quite the exaggeration. I use the DC metro multiple times a week (but no longer daily). I also prefer to use public transit while traveling. The ride is similar to elsewhere in the US and about 50% of abroad systems.

[+] wycy|3 years ago|reply
> Today a DC Metro ride is uncivilized, like the ancient system in NYC, with some poorly trained driver jerking you to a stop at each platform.

I ride DC metro regularly and this seems like quite the exaggeration.

[+] sys_64738|3 years ago|reply
I've not been on BART for 25 years and it felt antiquated back then.
[+] zhala|3 years ago|reply
As someone who did IT at a public transit agency for a medium sized city briefly, this is all pretty accurate and similar at most public transit agencies. Even brand new busses are still using old hardware under the hood that requires special deutsch to serial cables that require windows 7 machines to interface.

Likewise all the software that runs transit agencies is extremely dated and heavy.

[+] tibbydudeza|3 years ago|reply
Reminds me of the McLaren F1 sports car - a bespoke once off run V12 BMW engine so it uses a custom McLaren developed PCMCIA adapter as the main diagnostic interface.

Interface and software only works with a Compaq LTE 5280.

[+] jschveibinz|3 years ago|reply
There is gold in specializing in 40+ year old technology and programming environments for some lucky consultants.
[+] marcodiego|3 years ago|reply
The screenshot makes it look like they are running something in DOS. So, maybe they can upgrade to FreeDOS?
[+] oneplane|3 years ago|reply
If they depend on direct hardware access, there are other operating systems that can do that. And if they can do multi-tasking, they can switch to pretty much any other operating system.

But in this case I bet it's a combination of them not actually having any option other than to comply with the requirements of some old piece of software that is outside of their control. If they were to demand that all software used in those systems were delivered with sources and compilers, they could at least maintain it and port to newer operating systems and hardware. But they don't.

[+] dehrmann|3 years ago|reply
Since I guess it's a DOS program, I wonder if FreeDOS would work out-of-the-box on modern hardware?
[+] tyingq|3 years ago|reply
The article mentions they don't need DOS anymore, so they seem to be past that particular problem. It could have been as simple as something like FreeDOS on modern hardware didn't work because it was written to use a real serial port and not something like a USB to serial adapter. And perhaps until recently, they didn't have anyone to help figure out how to bridge something like that through an emulator or vm passthrough device, etc.