I emailed them. I live I'm SF. This is the sake, solved, problem we have in the machine tool world (milling machines, lathes, etc). Similar to old keyboards, you can just buy and use a floppy emulator.
They emailed me back, they said that the floppy thing makes a good headline but is really just the tip of the iceberg. It's really the whole system that's like this at every layer, it needs replacing they say.
This was my thought. Floppy emulators are a dime a dozen thanks to the retrocomputing community. Emulators for the custom boards full of discrete logic, opaque ROM chips, PLAs, and so on from long dead companies are going to be a much bigger challenge.
If they had good schematics for all of the parts it might be possible to keep the the system running for a long time with a couple of smart EEs who are comfortable with the scope and soldering iron, but eventually they're going to run out of some obscure part and be up a creek.
Or maybe they could replace entire boards with home designed versions condense all of the old logic down to one chip and a handful of support components and start in-place upgrading without a total system revamp. Still an expensive process, and one that requires some hard to find engineers on staff, but theoretically spreads out the upgrade process over many years. It also loses out on functional improvement opportunities while your system is made up of a hodgepodge of old and new hardware.
A once a generation refresh seems okay in my book. The Breda trains that came in 1996 were being phased out about twenty years into their tenure, most generally seem to think that is about reasonable a time to do something like that. I don’t see software related regimes as inherently that different.
Since I experienced 5.25 inch floppy disk era...and even the occasional bernoulli disk...we could simply say: the system had its run, replacement is reasonable. A lot of stuff had changed since then, and not just in storage media.
The real scope of the project is a mind-boggling $600 million to track the light rail trains inside and outside the tunnels and to control the trains inside the tunnels. Divided by the 250 light rail vehicles (apparently does not include historic streetcars), it works out to $2.6 million per car. Plus $36M for consulting (2023-11-07 board presentation from the documents that I linked earlier https://www.sfmta.com/sites/default/files/reports-and-docume...).
The PC and the floppy disk should be running as a virtual machine. ( The Vm software and host OS and the hardware would all be current supported versions of course )
That doesn't result in bigger budgets. If the goal of a manager is to increase their budget and hence scope of responsibility, any solution which doesn't result in a bigger budget isn't really a solution. Same reason it's going to cost a hundred million, yet a single indie game dev can write a train scheduling simulator by themselves.
the state-of-good-repair grift will continue until the morale improves! SelTrac is fine. and if it ever actually breaks (dubious) they can fall back to block signaling which comfortably handles more trains than muni is capable of running. the MBTA does just fine with it on the green line, and when the MBTA is making you look incompetent you've got to question where it all went wrong.
this is the agency that brought you the central fucking subway, a $300M sewer project masquerading as some red paint on van ness, that picked the already-mostly-grade-separated M Ocean over the N Judah to subway-ify and that is incapable of flicking the traffic preemption switch on the rest of the T to the on position without a decades long pedestrian detection LIDAR project for the unique in the world needs of san francisco.
It sounds like they let this problem fester until it has reached this existential end. They clamor for new technology, which they get, then they use the most risk adverse management strategy and never upgrade or change it, until it reaches this problem state.
They either need significant third party help deploying and managing this system, or they should go back a few generations of technology and use the simplest possible system that meets their needs. Pen and paper should be considered if it can be made more efficient.
I guess the people who work on public transit aren't interested in having the best public transit system available. They're only interested in keeping it running for as long as possible with zero changes or responsibility.
This is why I'm strongly doubtful on public transportation in the US. Our bureaucracy can't handle it.
My company provides new compute modules for another major global city's infrastructure that still relies on Intel CPU's from the early 80's. That is, we are building them brand new boards populated with chips that are over 40 years old.
They haven't shown any interest in updating the system. It works, they can get service, and get "new" replacements for things that go bad.
What they might not know though is that there is basically just one engineer we have (and probably the only one on Earth) who knows how to work on these things. He's getting old, and obviously none of the younger engineers really have an interest in learning ancient forgotten systems.
LOL:) this story seems to resurface every couple of years. My favorite part of the article:
"Jeffrey Tumlin: "It's a question of risk. The system is currently working just fine but we know that with each increasing year risk of data degradation on the floppy disks increases and that at some point there will be a catastrophic failure."
This seems to imply they have been using the exact same disk for the past 20 years (absurd), they have absolutely no idea what is written on the disk and how it can be safely backed up or restored. This would be a problem regardless of the medium used.
Although I hold the line at using paper tape there is nothing wrong with using floppies other than it seems antiquated. It certainly is reliable and cheap. Maybe the only thing that needs replacing is the people running the Muni.
One of my older jobs relied on ancient hardware, and the life of the company was measured by parts in the warehouse and vendors who still cared. From their perspective they'd have a modern system as long as your company delivers.
> hey haven't shown any interest in updating the system.
So expensive to update that there's a calculated end-of-life to the system. They'd love to know about your engineer situation. That'd trigger plans put in place a while ago.
Your company could do a last big batch for the city and send the old guy out with a nice bonus.
Nah getting you fee engineer interested isn't that hard, there's always a fun challenge in mastering something. You just have to give them time to figure it out.
At my workplace, we still ship devices with AMD 8086 processors to the military. And AMD still makes us 8086 processors on special order.
We got a few guys under 40 that work on that project once in a blue moon when a change is required.
> This system was designed to last 20 to 25 years. SFMTA's director Jeffrey Tumlin said upgrading the system will take another decade and cost hundreds of millions of dollars.
This is the problem, not that they're using a floppy. This isn't web dev where you get to rewrite everything every 6 mos. Systems have to have decades long life cycles BUT THEY EVENTUALLY NEED TO BE REPLACED and that's not happening quickly enough here.
Edit: It was last updated in 1998, so it's due now not a decade from now.
I'd like to see a breakdown of that "hundreds of millions of dollars" and which companies provided the quotes, plus who owns the companies and who are their husbands/wives/uncles/nephews/nieces/etc.
Still, when you have a 20-25 year replacement cycle being 10 years behind is pretty significant. And that's assuming their estimate of needing another decade is correct. Which seems doubtful considering they don't have a contractor, they don't have the money, and the previous system probably took a decade to develop considering their choice of 5.25" floppies (3.5" floppies overtook 5.25" floppies in sales in 1988).
What's really happening is going to be:
Emerging new use-cases and rising costs from specialized vendors for replacement parts that match the existing system are driving a desire for the agency to replace the control system. The floppy system bit can 100% be replaced with a solution that isn't a floppy, but that wouldn't help them with the rest of what they want.
>Turns out that in 1998, SFMTA had the latest cutting edge technology when they installed their automatic train control system.
> "We were the first agency in the U.S. to adopt this particular technology but it was from an era that computers didn't have a hard drive so you have to load the software from floppy disks on to the computer,"
In 1998, most personal computers already had hard drives [0]. From Wikipedia "The IBM PC/XT in 1983 included an internal 10 MB HDD, and soon thereafter, internal HDDs proliferated on personal computers."
The 3.5" floppy is from the mid 80's, again from Wiki [1] "In the early 1980s, many manufacturers introduced smaller floppy drives and media in various formats. A consortium of 21 companies eventually settled on a 3½-inch design..."
Why would a hard drive be better? If the floppy fails just grab another off the shelf and try it (surely they have more than one copy). Downtime measured in seconds.
The other good thing about the floppy is it can't hold very much code. So the system has a tight upper bound on how bloated and complex it can get. Simpler systems are more maintainable.
These things seem like great assets for maintaining critical infrastructure.
EDIT: Another great thing is such a system will be stateless. No disks, no filesystems, no databases. Sign me up.
Why do I have to do this research instead of the "journalist"?
Ratchet the snark back. The journalist was referring to the train control system not home computers in someone's basement. And, yes, twenty five years ago SelTrac was cutting edge. Moving block systems were basically unheard of back then.
Given that the agency seems not to be able to cope with change at a reasonable pace, I wonder if 1998 was just when the control system project finished, having been planned and started several years prior.
I had the displeasure to operate a sort of industrial/automotive rack computer running very simple Simulink data logging that booted only from a floppy, and I think it was bought around 2008. For a greenfield project.
Probably cost like hell and was decade or two behind in technology, but I didn't buy it and definitely wouldn't have. Salespeople get people to do stupid stuff.
I'd guess these are used a lot in industrial settings where the code and task is actually very simple. Would run no problem with a $1 microcontroller and a hundred lines of C and with a lot less hassle. Likely vendors have gotten a reputation and keep on selling them with FUD.
Well... I feel slightly better about Boston constantly pushing back being able to use our phones as tickets after seeing the timeline for the transition and how just that they are still doing this.
What is it about public transit in the US that it is so... bad? Inadequate funding seems to be the easy one, but the MBTA (Boston) doesn't even handle the funds it has well. Yeah it needs more funding but there is also just a core issue to how it's run.
It is sad to see the state of public transit in this country, particularly in dense urban areas where we should be discouraging Car use as much as possible.
I am very curious what other countries are doing that we are not.
I am an elected official (12 years and elected six times in Illinois) unfortunately seeing this process work over 12 years is discouraging. For example President Obama allocated 8 billion dollars for high sped passenger rail service improvements. By 2017 almost 2 billion of this money had been spent on railroad improvements / railroad crossings and line inspections.
Fast forward to 2024 no high speed rails have actually been built. For example Chicago Illinois was to get a line from Iowa City to Quad Cities to Chicago (straight line). This project was fully funded at one point but Iowa and Illinois had (still have) a disagreement on bridge repairs connecting the states.
Now Governor Pritzer has authorized a new high speed rail commission to try and get this project going again.
In a nutshell .. Rail project between Iowa City and Chicago was fully funded as of 2016… all that actually happened is some rail road crossing were improved.. lines were inspected and no actual new lines constructed. Instead 1 out of the 2 billion was spend on “engineering” costs and compliance paperwork … which now has expired and need to be redone if the project is to be completed.
The amount of money spent on compliance paperwork and “engineering” is staggering. Many six figure salaries depend on slightly altering existing engineered projects to meet compliance requirements for projects that are never built. From waste treatment upgrades, water treatment, road improvement, traffic studies, on and on. The amount of money spent on services that are not finished or lead to an actual project is absolutely staggering … entire industries depend on this inefficiency and lobby effectively to keep things “obtuse”.
I think about this all the time every time I see an lcd screen in my local train stations. If people who designed them actually used transit, maybe they'd have the signs you can only see as you walk out of the station say something about the bus lines you'd transfer to, instead of when the train behind you that just departed from is set to arrive. At some stations the screens they installed don't display anything useful at all, not when the next train will come, just the date and time as if everyone doesn't have that in their pocket, and this needs to be displayed every 25 feet on the platform.
A lot of transit could be fixed by just taking a regular routine user, empowering them to become a dictator for a week and point out all the friction points they hit actually using the system. But then that would make the entire bureaucratic system that is the transit agency look like idiots who don't understand their own jobs, so it will unfortunately never be done.
Japan's transit is private and semi-competitive. (there are 10+ train companies in Tokyo), (4+ in Kyoto), I haven't counted Osaka's
It works because the companies own land and facilities around every station. Grocery stores, office buildings, shopping centers (stores at many stations), apartments, etc. This creates a virtuous cycle where the more riders the more people use their other services and visa-versa.
As for semi-competitive, at least in Tokyo, Osaka, and Kyoto there are enough lines that you often have a choice. For example Tokyo (area) to Yokohama there's JR, Keikyu, Tokyu, (3 different companies). They have slightly different routes so if you're closer to one you might take one or the other but they do advertise trying to get you ride their's over the other's. To Hanada there's Keikyu and the Tokyo Monorail (it's own company). To Narita there's the Narita Express (JR) and Keisei (a different company) as well as local lines from both. Same in Kyoto. You can go to Kyoto to Nara via JR or via Keihan. You can go Kyoto to Osaka via JR or Hanshin.
So, if one company offers easy pay methods and another doesn't it quickly gets the reputation and "a crappy old line" (who wants to live there, open an office there, etc...)
- political point-scoring - blame-oriented cultures discourage experimentation and incremental improvement
- political polarization - one-party areas can more easily slide into cronyism, and fighting between parties makes it hard to compromise even on things like fixing infrastructure
- classism - in a lot of places, transit is for the poors
- racism - many don't want transit bringing Those People around
- manager culture, not engineer culture - as we see with Boeing, standard MBA thinking doesn't work well for long-term safety and reliability; the focus on short term metrics, mostly financial ones, leads to underinvestment and decay of infrastructure
> What is it about public transit in the US that it is so... bad?
Massive money and entrenched behind preserving a car-based life.
Car companies, auto workers unions, railroads can siphon off funds from promised improvements to cover deferred maintenance and give themselves bonuses for being so clever… Hell, AAA which one might expect to spend its income on serving its members instead diverts money to proactively lobbying against improvements in public transit in an ongoing display of cynical self-preservation
When BART was built in the 60s/70s it was innovative. The construction techniques for the Transbay Tube were unique at the time. The system was billed as a transit system built from the ground up for the space age. It was equipped with, at the time, a state of the art Automation Train Control system.
Upgrading those systems is expensive and difficult, so it's not surprising old prices of equipment hung around until the end of their useful life. Especially since BART is going through a massive budget crisis at the moment (and this isn't the only one in its history).
I've shared this experience/opinion with folks nearly verbatim :) Especially after having seen what I took to be a very well-integrated transit system in Chicago, the fragmented and confusing Bay Area experience was quite different from what I expected.
SF Voters rejected Proposition A in 2022 [1], which would have included funding to upgrade Muni's control systems (among many other projects). We'll eventually have to find the money somewhere else when the system fails.
The entire system has reached the end of it's design life and the floppy is just the shocking bite to get people to click through. Really this is the regular replacement of an old system with a newer control system that will make the trains run better and more reliably.
Can someone just ask Foone or lcamtuf to help them get it running off an internet-connected toothbrush or something? It seems like there are so many people doing complex reverse-engineering of ancient stuff _just for kicks_ that this floppy-disk issue just shouldn't require a massive project.
yes, I know safety-critical systems are different. I also expect that the floppy-disk issue is just the easiest problem to explain of a long chain of terrible legacy lock-ins. However, if they're literally holding their breath every morning when it's time to IPL the system off a floppy...that part sounds solvable.
The hypercompetence of tech enthusiasts and the mind-numbing idiocy & dishonesty of politial governance don't mix, so cities have to spend tens of millions with parasitic contractors for what a few clever hackers could do as a side-project.
> Turns out that in 1998, SFMTA had the latest cutting edge technology when they installed their automatic train control system.
> "We were the first agency in the U.S. to adopt this particular technology but it was from an era that computers didn't have a hard drive so you have to load the software from floppy disks on to the computer," said Mariana Maguire, SFMTA Train Control Project.
> SFMTA's train control system relies every morning on 5 inch floppy disks.
This doesn’t make any sense. 5.25-inch floppy disks and no hard disks was not “cutting edge technology” in 1998. It arguably wasn’t even “cutting edge technology” in 1988
Not cutting edge personal computer technology, obviously, but was it not cutting edge train control technology? TFA isn't making any claims about PC technology. If I had to come up with a computer system that needs to last O(1 century) I'd avoid disks, I think.
Frustratingly, though the article goes into how replacing/upgrading the whole control system will be an big, expensive project, they don't attempt to say why they can't just update the "read from floppy" part of the system to read the same info from a modern component.
Floppies aren't the problem, the main problem is lack of support. The MTA moved off of OS/2 back in 2008. Yeah, halfway into its intended lifespan Alcatel/Thales basically stopped supporting the whole mess. If I had to guess they're using an equally unsupported version of Windows now.
Yeah, the article reads as if the real problem is a lack of a tested system of backups. Or I don't know how else to parse this line:
- "It's a question of risk. The system is currently working just fine but we know that with each increasing year risk of data degradation on the floppy disks increases and that at some point there will be a catastrophic failure."
"Data degradation on the floppies" should not, by itself, cause any sort of "catastrophic failure" in a sound system. If one software disk fails, you should have five identical copies in a drawer, five more in a different room, ten more off-site, plus a disk image on cloud storage that you can write fresh floppy disks from.
I mean, it's probably a good idea to change the storage medium too—but that's not the root problem.
I believe that the article is about this RFP for “Contract No. SFMTA-2022-40 FTA” to upgrade the Communications-Based Train Control System (CBTC) or Advanced Train Control System (ATCS). The resolution to create the RFP for the supplier was approved 2023-01-17 (https://www.sfmta.com/reports/1-17-23-mtab-item-14-communica...), and a $36M resolution to create an RFP for a consultant was approved 2023-11-07 (https://www.sfmta.com/reports/11-7-23-mtab-item-11-train-con...). I’m not sure where the actual RFPs are pubished though.
SF is a joke. 20 years of being one of the richest cities in the world (probably in the history of human beings) and it comes out worse than it was before.
Upgrade? More like launch the process of planning for drafting a bid solicitation announcement. If I were 3.5" floppies, I wouldn't worry about retirement any time soon.
"The San Francisco Municipal Transportation Agency (SFMTA), which runs the city's Muni Metro light rail, claims to be the first US agency to adopt floppy disks. But today, the SFMTA is eager to abandon its reliance on 5¼-inch floppy disks—just give it about six more years and a few hundred more million dollars."
> "Wow. I mean I thought we were moving on to AI. So why are we doing floppy disk," said Katie Guillen, SFMTA passenger.
This naivety is not Katie's fault. We who work in tech are to blame for constantly pushing our half-baked experimental garbage as if it was "engineering" on par with civil or aeronautical systems. We can't blame people for occasionally believing the lies.
The tech-washed version of this quote might go something like "wow I thought everything was moving to the multi-cloud serverless kooberneetus now, why is it still running on a computer?"
> It is easy to run a secure computer system. You merely have to disconnect all dial-up connections and permit only direct-wired terminals, put the machine and its terminals in a shielded room, and post a guard at the door[1].
This is the kind of thing I want running the trains. Give it ECC RAM too, please.
> "It's like if you lose your memory overnight, and every morning, somebody has to tell you hey 'this is who you are and what your purpose is what you have to do today,'" said Maguire.
Yeah, that's called a cold boot. Moving to not-floppies doesn't mean you can avoid this. Clearly it's off of floppies instead of ROM so you can more easily update the software, but I am wondering how often that ended up happening. Maybe EEPROMs would have been better.
> Luz Pena: "How dire is it to change the system to upgrade it from a floppy disk to a wireless system?"
I agree that floppies aren't the peak of reliability, but "a wireless system" also sounds like a disaster. I don't want critical urban infrastructure running on extremely hackable OTA updates. For the love of god, SF, you can avoid pretty much all potential cybersecurity problems by just not putting your trains online.
I feel like neither the interviewer nor the interviewee really had the technical expertise to speak to this. This entire piece is just, "oooooo, floppies are old. Old bad! Why not new yet? New good."
I don't think that is actually a concern. The article is written in a way to imply that it is, but I doubt the people who are running the system actually think that is the risk.
I am sure they have copies of the floppy disk, and likely have images on other machines that can be used to create a new floppy even if all the copies are lost.
I feel like the SFMTA is mainly using the existence of the floppy disks as a marketing point for their desire to get funding to update the system. It is something that the average person will be able to look at and know it is out of date in a visceral way.
The reporter seems to read this as if we are one bad floppy away from failure, but that is not the actual case.
You are correct that this would be much simpler if the only goal was to replace the floppy disk part of the system.
The floppy disk angle is there to make a good headline. The article makes it clear this is a much bigger project than just replacing the floppy disks.
"The detail[ed] project schedule will be finalized once we have a contractor onboard. This is effectively a multi-phase decade long project that starts with pieces of market street subway and pieces in the surface. Ultimately our goal is to have a single train control system for the entire rail system," said Tumlin.
Playing video games on an old Apple II via a floppy emulator has a different set of requirements than a safety-critical application.
Lawyers will probably spend a year wrangling liability concerns.
I could probably replace a floppy drive on any system within a couple of weeks. I would not accept legal or financial liability for any such solution without an extremely thorough and slow review of all aspects of the project. edit: and an astronomically high paycheck.
> However, budget challenges put the project's timeline into question. The SFMTA's train upgrade project isn't just a migration off of floppy disks but also a "complete overhaul of the current train control system and all its components, including the onboard computers, central and local servers, and communications infrastructure," Roccaforte said.
> Much more critical than the dated use of floppy disks is the system's loop cable, which transmits data between the central servers and the trains and, according to Roccaforte, "has less bandwidth than an old AOL dial-up modem."
Also, they're already late as hell, what's 5 more years? 5 1/4 isn't going to become more obsolete than it is at this point.
There are other concerns that need to be addressed where a hack wouldn't help:
>The transportation body says the train control system was built to last for just 20 to 25 years, meaning it surpassed its expected lifetime in 2023. In 2020, the Muni Reliability Working Group, said to be composed of local and national transit experts, recommended replacing the transit control system within five to seven years. [...] "We have to maintain programmers who are experts in the programming languages of the '90s in order to keep running our current system, so we have a technical debt that stretches back many decades," Tumlin told San Francisco's KQED in February 2023.
I think the government is involved which means the task must primarily be an effort to funnel large amounts of money around to just the right people while facilitating all the best changes to whatever they can find within N degrees of separation from the project itself, (where N approaches infinity.)
Speaking naively (I have no insight into the situation), I would guess that this is probably the result of the officials in charge not having technical understanding and contracting it all out. Contractors aren’t going to be choosing the quick or cost-effective option, because that’d undercut their profits.
>""The system is currently working just fine, but we know that with each increasing year, risk of data degradation on the floppy disks increases and that at some point there will be a catastrophic failure," Tumlin told ABC7."
They haven't even settled on a contractor yet? Maybe the problem here is that they're trying to write one check to fix all their problems at once instead of taking an incremental approach.
This takes deep expertise and a system oriented, long term view with a very precise eye towards the details of what's technically possible at every individual step.
This kind of knowledge and experience doesn't come cheap, and we all know how much US city governments pay. I was at one point, very briefly, motivated to apply to a city job before I learned the pay is approximately 1/3 to 1/4 of what the private sector pays. The USDS routinely posts on "who wants to get hired" and the comments on that and other thread also mention a 66% to 75% salary reduction from baseline.
This is even before we get to legal liability and political risk shifting that can happen when there is a fully responsible contracted company involved.
The longer a thing has not failed, the less likely it is to fail at any given moment. On the other hand, the older a thing is, the more likely it is to fail at any given moment.
They do use 3.5 inch disks; this is a misstatement: “SFMTA's train control system relies every morning on 5 inch floppy disks.” The SFMTA Board presentation justifying the >$600M upgrade shows a photo of the real 3.5 inch floppy disks in use that were written in 2021. https://www.sfmta.com/sites/default/files/reports-and-docume...
Trains are expected to last for decades, so this really isn't surprising. Developing a new system with the latest hardware to replace the floppy disk system would probably cost millions and might introduce bugs and security issues.
Just a few short hours ago I was hearing some Muni workers at West Portal station talk about this story. One said to the other, "young people don't know what a floppy disk is anymore".
Raspberry pi controller $30, couple of people to make it work - let's say $500k.
Sure... there are loads of things that probably need attention, but maybe you don't need to fix them all at once?
Remember when, less than 3 years ago, SF projected a $108M surplus for the next two years?
Woulda been a nice time to clean up some of this technical debt!
Or how about the SF Emergency Sirens, taken offline in late 2019 for a "2 year" upgrade plan that officeholders implied was already in place?
In August 2023, with no progress whatsover, with the Maui fire disaster fresh on their minds, Mayor Breed & Supervisors President Peskin touted they'd finally funded a plan to return them to service soon: https://www.sf.gov/news/mayor-breed-and-board-president-pesk...
Of course, this was just more blatant self-exonerating bullshit from our local political machines immune from any real accountability for incompetence in basic public functions.
A mere 6 months later in February 2024, nothing's been started, Peskin admitted "we don't even have a plan", the department is still waiting until "funding is identified", and the cost estimate has ballooned to $20.5m: https://abc7news.com/san-francisco-sirens-emergency-911-aler...
That works out to $170K+ for each of 119 units – units that each could probably just be a weatherized consumer-grade handheld device with multiple mobile/packet/sat radios, & a simple authenticated-playback app, mounted on existing poles that presumably already have power and even loudspeakers.
Sirens don't even need to be that complicated. A motor, a noise generator wheel (basically just a big spinning squirrelcage fan), and a horn. You could toggle them on/off with a simple relay/contactor.
> It's a question of risk. The system is currently working just fine but we know that with each increasing year risk of data degradation on the floppy disks increases and that at some point there will be a catastrophic failure.
The key words in the sentence are: "it's working just fine".
Data degradation of floppy discs is easy: just copy them to fresh ones, and verify that you have a good copy. The images should be safely backed up so they can be regenerated. (Plus there are emulators; a topic covered elsewhere under this submission.)
I mean, are they really using the same 30 year old floppy discs over and over again until they degrade?
iancmceachern|1 year ago
They emailed me back, they said that the floppy thing makes a good headline but is really just the tip of the iceberg. It's really the whole system that's like this at every layer, it needs replacing they say.
jandrese|1 year ago
If they had good schematics for all of the parts it might be possible to keep the the system running for a long time with a couple of smart EEs who are comfortable with the scope and soldering iron, but eventually they're going to run out of some obscure part and be up a creek.
Or maybe they could replace entire boards with home designed versions condense all of the old logic down to one chip and a handful of support components and start in-place upgrading without a total system revamp. Still an expensive process, and one that requires some hard to find engineers on staff, but theoretically spreads out the upgrade process over many years. It also loses out on functional improvement opportunities while your system is made up of a hodgepodge of old and new hardware.
fdr|1 year ago
Since I experienced 5.25 inch floppy disk era...and even the occasional bernoulli disk...we could simply say: the system had its run, replacement is reasonable. A lot of stuff had changed since then, and not just in storage media.
yonran|1 year ago
kazinator|1 year ago
It may feel emotionally like this at every layer, but the layers that are not floppy discs are completely different from floppy discs.
Other than electrolytics caps (easy replacement items), old electronics is reliable.
Moving parts need service; that's life.
> needs replacement
Not believable without detailed justification.
In the present article, someone is literally quoted as everything working just fine.
JohnTHaller|1 year ago
asdefghyk|1 year ago
nostrademons|1 year ago
andbberger|1 year ago
this is the agency that brought you the central fucking subway, a $300M sewer project masquerading as some red paint on van ness, that picked the already-mostly-grade-separated M Ocean over the N Judah to subway-ify and that is incapable of flicking the traffic preemption switch on the rest of the T to the on position without a decades long pedestrian detection LIDAR project for the unique in the world needs of san francisco.
boringg|1 year ago
SllX|1 year ago
(they probably are…)
snvzz|1 year ago
Hrnrurj|1 year ago
https://www.businessinsider.com/nyc-mta-subway-delay-2017-6
akira2501|1 year ago
It sounds like they let this problem fester until it has reached this existential end. They clamor for new technology, which they get, then they use the most risk adverse management strategy and never upgrade or change it, until it reaches this problem state.
They either need significant third party help deploying and managing this system, or they should go back a few generations of technology and use the simplest possible system that meets their needs. Pen and paper should be considered if it can be made more efficient.
I guess the people who work on public transit aren't interested in having the best public transit system available. They're only interested in keeping it running for as long as possible with zero changes or responsibility.
This is why I'm strongly doubtful on public transportation in the US. Our bureaucracy can't handle it.
Workaccount2|1 year ago
They haven't shown any interest in updating the system. It works, they can get service, and get "new" replacements for things that go bad.
What they might not know though is that there is basically just one engineer we have (and probably the only one on Earth) who knows how to work on these things. He's getting old, and obviously none of the younger engineers really have an interest in learning ancient forgotten systems.
strangattractor|1 year ago
"Jeffrey Tumlin: "It's a question of risk. The system is currently working just fine but we know that with each increasing year risk of data degradation on the floppy disks increases and that at some point there will be a catastrophic failure."
This seems to imply they have been using the exact same disk for the past 20 years (absurd), they have absolutely no idea what is written on the disk and how it can be safely backed up or restored. This would be a problem regardless of the medium used.
Although I hold the line at using paper tape there is nothing wrong with using floppies other than it seems antiquated. It certainly is reliable and cheap. Maybe the only thing that needs replacing is the people running the Muni.
gopher_space|1 year ago
> hey haven't shown any interest in updating the system.
So expensive to update that there's a calculated end-of-life to the system. They'd love to know about your engineer situation. That'd trigger plans put in place a while ago.
Your company could do a last big batch for the city and send the old guy out with a nice bonus.
delfinom|1 year ago
At my workplace, we still ship devices with AMD 8086 processors to the military. And AMD still makes us 8086 processors on special order.
We got a few guys under 40 that work on that project once in a blue moon when a change is required.
culopatin|1 year ago
throwaway984393|1 year ago
[deleted]
HumblyTossed|1 year ago
This is the problem, not that they're using a floppy. This isn't web dev where you get to rewrite everything every 6 mos. Systems have to have decades long life cycles BUT THEY EVENTUALLY NEED TO BE REPLACED and that's not happening quickly enough here.
Edit: It was last updated in 1998, so it's due now not a decade from now.
markus_zhang|1 year ago
wongarsu|1 year ago
dblohm7|1 year ago
bdamm|1 year ago
What's really happening is going to be: Emerging new use-cases and rising costs from specialized vendors for replacement parts that match the existing system are driving a desire for the agency to replace the control system. The floppy system bit can 100% be replaced with a solution that isn't a floppy, but that wouldn't help them with the rest of what they want.
"because life cycles" is a lazy description.
WaitWaitWha|1 year ago
> "We were the first agency in the U.S. to adopt this particular technology but it was from an era that computers didn't have a hard drive so you have to load the software from floppy disks on to the computer,"
In 1998, most personal computers already had hard drives [0]. From Wikipedia "The IBM PC/XT in 1983 included an internal 10 MB HDD, and soon thereafter, internal HDDs proliferated on personal computers."
The 3.5" floppy is from the mid 80's, again from Wiki [1] "In the early 1980s, many manufacturers introduced smaller floppy drives and media in various formats. A consortium of 21 companies eventually settled on a 3½-inch design..."
[0] https://en.wikipedia.org/wiki/Hard_disk_drive
[1] https://en.wikipedia.org/wiki/Floppy_disk
Why do I have to do this research instead of the "journalist"?
jcgrillo|1 year ago
The other good thing about the floppy is it can't hold very much code. So the system has a tight upper bound on how bloated and complex it can get. Simpler systems are more maintainable.
These things seem like great assets for maintaining critical infrastructure.
EDIT: Another great thing is such a system will be stateless. No disks, no filesystems, no databases. Sign me up.
compuguy|1 year ago
Edit: Even the news article hints at 5 inch floppy disks...which makes 1998 make absolutely no sense to me at least.
https://abc7news.com/san-francisco-train-system-has-been-run....
inferiorhuman|1 year ago
abeppu|1 year ago
jampekka|1 year ago
Probably cost like hell and was decade or two behind in technology, but I didn't buy it and definitely wouldn't have. Salespeople get people to do stupid stuff.
I'd guess these are used a lot in industrial settings where the code and task is actually very simple. Would run no problem with a $1 microcontroller and a hundred lines of C and with a lot less hassle. Likely vendors have gotten a reputation and keep on selling them with FUD.
nerdjon|1 year ago
What is it about public transit in the US that it is so... bad? Inadequate funding seems to be the easy one, but the MBTA (Boston) doesn't even handle the funds it has well. Yeah it needs more funding but there is also just a core issue to how it's run.
It is sad to see the state of public transit in this country, particularly in dense urban areas where we should be discouraging Car use as much as possible.
I am very curious what other countries are doing that we are not.
roody15|1 year ago
Now Governor Pritzer has authorized a new high speed rail commission to try and get this project going again.
In a nutshell .. Rail project between Iowa City and Chicago was fully funded as of 2016… all that actually happened is some rail road crossing were improved.. lines were inspected and no actual new lines constructed. Instead 1 out of the 2 billion was spend on “engineering” costs and compliance paperwork … which now has expired and need to be redone if the project is to be completed.
The amount of money spent on compliance paperwork and “engineering” is staggering. Many six figure salaries depend on slightly altering existing engineered projects to meet compliance requirements for projects that are never built. From waste treatment upgrades, water treatment, road improvement, traffic studies, on and on. The amount of money spent on services that are not finished or lead to an actual project is absolutely staggering … entire industries depend on this inefficiency and lobby effectively to keep things “obtuse”.
asdff|1 year ago
A lot of transit could be fixed by just taking a regular routine user, empowering them to become a dictator for a week and point out all the friction points they hit actually using the system. But then that would make the entire bureaucratic system that is the transit agency look like idiots who don't understand their own jobs, so it will unfortunately never be done.
nox101|1 year ago
It works because the companies own land and facilities around every station. Grocery stores, office buildings, shopping centers (stores at many stations), apartments, etc. This creates a virtuous cycle where the more riders the more people use their other services and visa-versa.
As for semi-competitive, at least in Tokyo, Osaka, and Kyoto there are enough lines that you often have a choice. For example Tokyo (area) to Yokohama there's JR, Keikyu, Tokyu, (3 different companies). They have slightly different routes so if you're closer to one you might take one or the other but they do advertise trying to get you ride their's over the other's. To Hanada there's Keikyu and the Tokyo Monorail (it's own company). To Narita there's the Narita Express (JR) and Keisei (a different company) as well as local lines from both. Same in Kyoto. You can go to Kyoto to Nara via JR or via Keihan. You can go Kyoto to Osaka via JR or Hanshin.
So, if one company offers easy pay methods and another doesn't it quickly gets the reputation and "a crappy old line" (who wants to live there, open an office there, etc...)
wpietri|1 year ago
- chronic underfunding - we all know how problems build up when maintenance is deferred
- waterfall planning - Mary Poppendieck has a nice talk on how much trouble this causes: https://www.infoq.com/presentations/tyranny-of-plan/
- political point-scoring - blame-oriented cultures discourage experimentation and incremental improvement
- political polarization - one-party areas can more easily slide into cronyism, and fighting between parties makes it hard to compromise even on things like fixing infrastructure
- classism - in a lot of places, transit is for the poors
- racism - many don't want transit bringing Those People around
- manager culture, not engineer culture - as we see with Boeing, standard MBA thinking doesn't work well for long-term safety and reliability; the focus on short term metrics, mostly financial ones, leads to underinvestment and decay of infrastructure
rangestransform|1 year ago
other cities (in the east, with the best public transit) don't let unions grab the entire city by the balls
government employees in other cities aren't as hellbent on extracting their pound of flesh from the taxpayer
electric_mayhem|1 year ago
Massive money and entrenched behind preserving a car-based life.
Car companies, auto workers unions, railroads can siphon off funds from promised improvements to cover deferred maintenance and give themselves bonuses for being so clever… Hell, AAA which one might expect to spend its income on serving its members instead diverts money to proactively lobbying against improvements in public transit in an ongoing display of cynical self-preservation
It’s basically hopeless.
Solvency|1 year ago
[deleted]
bigyikes|1 year ago
I was expecting it to be some kind of utopia, with futuristic technology on every corner.
In reality, it is roughly the same as anywhere else in America. A bit of a let down. The innovation does not take place in the infrastructure.
Beautiful place, though!
patmorgan23|1 year ago
Upgrading those systems is expensive and difficult, so it's not surprising old prices of equipment hung around until the end of their useful life. Especially since BART is going through a massive budget crisis at the moment (and this isn't the only one in its history).
BHSPitMonkey|1 year ago
kccqzy|1 year ago
*: Many people don't. Self-driving cars are better than human-driven cars, but affordable and reliable trains are even better.
aluminum96|1 year ago
[1] https://www.sfchronicle.com/sf/article/S-F-voters-narrowly-r...
shrubble|1 year ago
Sounds like they are using the floppy as an excuse to push for an upgrade that has nothing to do with the floppy drives.
rtkwe|1 year ago
sevenseventen|1 year ago
yes, I know safety-critical systems are different. I also expect that the floppy-disk issue is just the easiest problem to explain of a long chain of terrible legacy lock-ins. However, if they're literally holding their breath every morning when it's time to IPL the system off a floppy...that part sounds solvable.
gojomo|1 year ago
throwaway74432|1 year ago
That's not the 3.5" floppy disk in the video. This is the old floppy disks[1]
1. https://www.digitaltreasures.ca/img/level2_floppy_525.jpg
prewett|1 year ago
The OG floppy disks were 8 inches, though, 5 inch floppies were the small ones (hence why 3.5" disks were "microfloppies"):
https://images.techhive.com/images/article/2015/12/floppy_di...
skissane|1 year ago
> "We were the first agency in the U.S. to adopt this particular technology but it was from an era that computers didn't have a hard drive so you have to load the software from floppy disks on to the computer," said Mariana Maguire, SFMTA Train Control Project.
> SFMTA's train control system relies every morning on 5 inch floppy disks.
This doesn’t make any sense. 5.25-inch floppy disks and no hard disks was not “cutting edge technology” in 1998. It arguably wasn’t even “cutting edge technology” in 1988
sitharus|1 year ago
Perhaps ATC was layered on top of the original signalling system, which is the part that uses the floppy disks?
jcgrillo|1 year ago
abeppu|1 year ago
inferiorhuman|1 year ago
Aloha|1 year ago
The technology works, there is a replacement outlined, there is no shortage of floppy disks - even 5 1/4 ones.
perihelions|1 year ago
- "It's a question of risk. The system is currently working just fine but we know that with each increasing year risk of data degradation on the floppy disks increases and that at some point there will be a catastrophic failure."
"Data degradation on the floppies" should not, by itself, cause any sort of "catastrophic failure" in a sound system. If one software disk fails, you should have five identical copies in a drawer, five more in a different room, ten more off-site, plus a disk image on cloud storage that you can write fresh floppy disks from.
I mean, it's probably a good idea to change the storage medium too—but that's not the root problem.
yonran|1 year ago
ericmcer|1 year ago
Smoosh|1 year ago
ilya_m|1 year ago
ForOldHack|1 year ago
"The San Francisco Municipal Transportation Agency (SFMTA), which runs the city's Muni Metro light rail, claims to be the first US agency to adopt floppy disks. But today, the SFMTA is eager to abandon its reliance on 5¼-inch floppy disks—just give it about six more years and a few hundred more million dollars."
yea for Unicode Character “¼”
ForOldHack|1 year ago
jcgrillo|1 year ago
This naivety is not Katie's fault. We who work in tech are to blame for constantly pushing our half-baked experimental garbage as if it was "engineering" on par with civil or aeronautical systems. We can't blame people for occasionally believing the lies.
The tech-washed version of this quote might go something like "wow I thought everything was moving to the multi-cloud serverless kooberneetus now, why is it still running on a computer?"
> It is easy to run a secure computer system. You merely have to disconnect all dial-up connections and permit only direct-wired terminals, put the machine and its terminals in a shielded room, and post a guard at the door[1].
This is the kind of thing I want running the trains. Give it ECC RAM too, please.
[1] https://en.m.wikipedia.org/wiki/Robert_Morris_(cryptographer...
sitkack|1 year ago
VyseofArcadia|1 year ago
Yeah, that's called a cold boot. Moving to not-floppies doesn't mean you can avoid this. Clearly it's off of floppies instead of ROM so you can more easily update the software, but I am wondering how often that ended up happening. Maybe EEPROMs would have been better.
> Luz Pena: "How dire is it to change the system to upgrade it from a floppy disk to a wireless system?"
I agree that floppies aren't the peak of reliability, but "a wireless system" also sounds like a disaster. I don't want critical urban infrastructure running on extremely hackable OTA updates. For the love of god, SF, you can avoid pretty much all potential cybersecurity problems by just not putting your trains online.
I feel like neither the interviewer nor the interviewee really had the technical expertise to speak to this. This entire piece is just, "oooooo, floppies are old. Old bad! Why not new yet? New good."
JTbane|1 year ago
zitterbewegung|1 year ago
cortesoft|1 year ago
wongarsu|1 year ago
nvahalik|1 year ago
The retro community has proven reliably that a simple Raspberry PI can easily bit-bang floppy controllers. We have myriad floppy-to-SD card adapters.
Surely a plug-and-play solution that removes the area of most concern (reliance on the media itself) should be easily achievable in a few months?
cortesoft|1 year ago
I am sure they have copies of the floppy disk, and likely have images on other machines that can be used to create a new floppy even if all the copies are lost.
I feel like the SFMTA is mainly using the existence of the floppy disks as a marketing point for their desire to get funding to update the system. It is something that the average person will be able to look at and know it is out of date in a visceral way.
The reporter seems to read this as if we are one bad floppy away from failure, but that is not the actual case.
jdblair|1 year ago
The floppy disk angle is there to make a good headline. The article makes it clear this is a much bigger project than just replacing the floppy disks.
snakeyjake|1 year ago
Lawyers will probably spend a year wrangling liability concerns.
I could probably replace a floppy drive on any system within a couple of weeks. I would not accept legal or financial liability for any such solution without an extremely thorough and slow review of all aspects of the project. edit: and an astronomically high paycheck.
pessimizer|1 year ago
> Much more critical than the dated use of floppy disks is the system's loop cable, which transmits data between the central servers and the trains and, according to Roccaforte, "has less bandwidth than an old AOL dial-up modem."
Also, they're already late as hell, what's 5 more years? 5 1/4 isn't going to become more obsolete than it is at this point.
lepus|1 year ago
>The transportation body says the train control system was built to last for just 20 to 25 years, meaning it surpassed its expected lifetime in 2023. In 2020, the Muni Reliability Working Group, said to be composed of local and national transit experts, recommended replacing the transit control system within five to seven years. [...] "We have to maintain programmers who are experts in the programming languages of the '90s in order to keep running our current system, so we have a technical debt that stretches back many decades," Tumlin told San Francisco's KQED in February 2023.
ammo1662|1 year ago
Clearly no one wants to do that.
luxuryballs|1 year ago
jwells89|1 year ago
FpUser|1 year ago
Do they know that floppy can be backed up?
DerekL|1 year ago
anigbrowl|1 year ago
maxvt|1 year ago
This kind of knowledge and experience doesn't come cheap, and we all know how much US city governments pay. I was at one point, very briefly, motivated to apply to a city job before I learned the pay is approximately 1/3 to 1/4 of what the private sector pays. The USDS routinely posts on "who wants to get hired" and the comments on that and other thread also mention a 66% to 75% salary reduction from baseline.
This is even before we get to legal liability and political risk shifting that can happen when there is a fully responsible contracted company involved.
wongarsu|1 year ago
mathgradthrow|1 year ago
_trampeltier|1 year ago
https://www.theverge.com/2019/10/25/20931800/usa-nuclear-8-i...
rootusrootus|1 year ago
therealmarv|1 year ago
yonran|1 year ago
wakawaka28|1 year ago
t1c|1 year ago
asveikau|1 year ago
ornornor|1 year ago
And in the video, she says “on 3x 5 inch floppy disks like this one <shows a 3.5 inch floppy>”
ChrisArchitect|1 year ago
rainbowzootsuit|1 year ago
asdefghyk|1 year ago
sourcecodeplz|1 year ago
fifteen1506|1 year ago
m3kw9|1 year ago
time4tea|1 year ago
gojomo|1 year ago
Woulda been a nice time to clean up some of this technical debt!
Or how about the SF Emergency Sirens, taken offline in late 2019 for a "2 year" upgrade plan that officeholders implied was already in place?
In August 2023, with no progress whatsover, with the Maui fire disaster fresh on their minds, Mayor Breed & Supervisors President Peskin touted they'd finally funded a plan to return them to service soon: https://www.sf.gov/news/mayor-breed-and-board-president-pesk...
In that same August 2023 timeframe, Peskin said the plan would bring this "need to have" system "up and running" & to "state of the art" by end of 2024, for $5.5M: https://www.cbsnews.com/sanfrancisco/news/san-francisco-city...
Of course, this was just more blatant self-exonerating bullshit from our local political machines immune from any real accountability for incompetence in basic public functions.
A mere 6 months later in February 2024, nothing's been started, Peskin admitted "we don't even have a plan", the department is still waiting until "funding is identified", and the cost estimate has ballooned to $20.5m: https://abc7news.com/san-francisco-sirens-emergency-911-aler...
That works out to $170K+ for each of 119 units – units that each could probably just be a weatherized consumer-grade handheld device with multiple mobile/packet/sat radios, & a simple authenticated-playback app, mounted on existing poles that presumably already have power and even loudspeakers.
paradox460|1 year ago
kazinator|1 year ago
The key words in the sentence are: "it's working just fine".
Data degradation of floppy discs is easy: just copy them to fresh ones, and verify that you have a good copy. The images should be safely backed up so they can be regenerated. (Plus there are emulators; a topic covered elsewhere under this submission.)
I mean, are they really using the same 30 year old floppy discs over and over again until they degrade?
JoyousAbandon|1 year ago
"it was from an era that computers didn't have a hard drive"
Absolute BS. Pretty much every computer had a hard drive in 1998, and most had CD-ROM.
Then they referred to the 3.5" disk as a "5-inch floppy."
<sigh>
jcgrillo|1 year ago
> <sigh>
</sigh>
throwaway984393|1 year ago
[deleted]
unknown|1 year ago
[deleted]
YawnNews|1 year ago
[deleted]