top | item 23497842

Brilliant Hardware in the Valley of the Software Slump

183 points| mpweiher | 5 years ago |craigmod.com

97 comments

order
[+] Tom4hawk|5 years ago|reply
The problem is that both hardware and software are garbage.

Spectre/Meldown & friends are just the tip of an iceberg. We have layers & layers of indirection/abstraction everywhere. We have hardware that lies and tells you that it has certain properties when in reality it doesn't (example: sector sizes in hard drives/NVMs, processors still pretending that they behave like PDP-11), we have hardware that is flat out broken. We try to fix those issues in software.

But in the software, we have another dump of workarounds, dependencies, abstractions with a sprinkle of backward compatibility. We are now creating "minimalist" applications with a fraction of functionality of the software from 30 years ago but using so many layers that total amount of code used to make it work is many orders of magnitude larger than what we had back then.

I know that most of the programmers did not work with systems where it's very, very easy to debug the whole stack and you can learn it in a short period but it's amazing when you have knowledge about EVERY part of the system in your head.

There are some good things going on (like strive to replace C with something which has similar performance characteristics but without its flaws) but it's not enough.

Here are two things worth watching:

https://www.youtube.com/watch?v=pW-SOdj4Kkk - Jonathan Blow - Preventing the Collapse of Civilization

https://www.youtube.com/watch?v=t9MjGziRw-c - Computers Barely Work - Interview with Linux Legend Greg Kroah-Hartman

[+] trimbo|5 years ago|reply
> The problem is that both hardware and software are garbage.

I think it's incredible that, in my lifetime, computers went from giant mainframes with dedicated hard-line terminals to always-connected supercomputers in everyone's pocket, worldwide. Furthermore, anyone can use the internet to learn how to program.

Maybe that's garbage compared to some mythical ideal but in terms of impact on the world it's incredible.

> I know that most of the programmers did not work with systems where it's very, very easy to debug the whole stack and you can learn it in a short period but it's amazing when you have knowledge about EVERY part of the system in your head.

Well, you can tell from the above that I was around then. I started programming with a single manual and the Beagle Brothers "Peeks, Pokes and Pointers" cheatsheet[1].

People forget that the software itself had to do much less than it does today. Here's just one angle to judge: security. We did not have a worldwide interconnected network with so many people trying to steal data from it. We all used rsh and packets were flying around in cleartext, no problem. But today, all software will have to incorporate TLS.

And far fewer people built that simpler software. EA's first titles were written by one or two people. Now a typical EA title has hundreds of people working on it.

Things will get better than where they are today. In the future, the industry will have invest more money in "10X" productivity and reliability improvements. Eventually, I think that will happen as productivity continues to slow on large codebases.

[1] - https://downloads.reactivemicro.com/Apple%20II%20Items/Docum...

[+] matheusmoreira|5 years ago|reply
> we have hardware that is flat out broken

Reading Linux driver code is very informative. Sometimes hardware just doesn't do what is expected and the driver must try and fix it up so that user space can have a reasonable interface.

A simple example:

  /* The ITE8595 always reports 0 as value for the rfkill button. Luckily
   * it is the only button in its report, and it sends a report on
   * release only, so receiving a report means the button was pressed.
   */
  if (usage->hid == HID_GD_RFKILL_BTN) {
      input_event(input, EV_KEY, KEY_RFKILL, 1);
      input_sync(input);
      input_event(input, EV_KEY, KEY_RFKILL, 0);
      input_sync(input);
      return 1;
  }
[+] scroot|5 years ago|reply
Simplicity and good design take lots of time and money. Our culture is not truly ready to make these kinds of investments in the manner required. Why would they? There is a whole universe of FOSS out there upon which anyone can cheaply create "working" software. If your goals are short term (quarterly earnings, looking only a year or two down the road) this is "good enough." Worse, that FOSS foundation is typically filled to the brim with complexity. We have created a computing culture that is premised on pushing the extremes of the teletype model of computing, and tacking what customers think they want on top of it.

We have good alternate examples from the past (Oberon, Lisp machines, Hypercard, Smalltalk systems, etc). How often does the new generation of computing people get exposed to these ideas?

[+] bob1029|5 years ago|reply
It would seem there may be a lack of appreciation for how powerful a modern x86 CPU actually is. Even when you apply every side-channel mitigation in the book, these processors are incredibly powerful when used appropriately. Somehow x86 is being branded as this inferior compute facility, always at odds with the GPU/ARM/et.al., and I feel it mostly boils down to shitty software engineering more than anything else.

I believe a lot of this can be attributed to rampant, unmitigated abuse of the NUMA architecture exposed by the x86 stack, as well as a neglect for the parallelism offered by CPUs of the last 3-4 generations. Most developers are not following principles which align with what the hardware can actually do, rather, they are following principles which align with what academia/cult-wisdom says you should do. These two worlds are vastly at odds when it comes to keeping your L1/2/3 caches as hot as possible and keeping all of those cores busy with useful work.

For example, most software engineers would tell you that busy waiting is a horrible practice, but when you step back and pay attention to the hardware reality, you have 32+ threads to burn, why not put your ultra-high-performance timer as a high-priority process on one of those threads and let it check trigger conditions in a tight loop? You can get timing errors measured in 10s of nanoseconds with this approach, and it's dead simple. Now you only have 31 threads remaining. 30 if you really want to make sure that timer runs smooth. Running at 3+ ghz, that one thread could service an incredible number of events per second. There are new ways to think about how we build things given newer hardware capabilities.

I feel a lot of the high-performance software revolution is going to come out of some ideas that have been floating around in fintech. Frameworks like the LMAX Disruptor (and the ideology behind it) can serve as the foundation for a UI framework capable of transacting tens of millions of aggregate user events per second and with peak latencies measured in microseconds. I have personally started to dabble in this area, and the results after just a few weekends have been very encouraging. You would be surprised what even high level languages (Java/C#) are capable of when their use is well-aligned with the hardware.

[+] pwdisswordfish2|5 years ago|reply
Jonathan Blow: "Software has been freeriding on hardware."

True or false? If I make the code that does a particular task smaller, faster, less resource-intensive, then I am not freeriding.

He says people do not reference the "five nines" anymore. True? I do not work in the industry anymore. I had no idea this has disappeared. That is really sad.

"Developer time" versus "user time". What is more important? Are they equally important?

Recently someone posted a video of a recent talk from Margo Seltzer. She said users, e.g., in the Physics Department, do not care about computer languages and these things that computer scientists and programmers think are so important. They care about how fast the program runs. That's all. "Make it go faster".

The incentives seem backwards. We pay programmers more today to do less than they did in the past. There is a lot of "busy work" going on.

[+] ponker|5 years ago|reply
In spite of all this "garbage" I carry around a $500 machine which fits in my pants pocket and give me directions to anywhere I can think of, a live video call (!) with my friends or family, a vast trove of knowledge about millions of different topics, and a camera that basically matches my $3000 DSLR from six years ago. And these are available whenever and wherever I want, so I can do this video call at 2am on a mountaintop if I want. So... I love this garbage.
[+] ecf|5 years ago|reply
I too sometimes have the same feeling that the software stack we have today is just a tower of cards waiting to crumble.

How long do you think it would take to get back to where we are if everything was scrapped and we restarted with the initial binary -> assembly jump?

[+] taneq|5 years ago|reply
This is kind of like someone living during the Crusades yelling metallurgy is shiiiiit.
[+] sharkjacobs|5 years ago|reply
This doesn't seem to have anything to do with the article.

Am I wrong to wish that the top voted comment was by someone who read the article before they posted?

[+] darepublic|5 years ago|reply
Well is the solution more tech (like improving on C language) or is it making sure we have robust non technical fallbacks as a civilization
[+] bcrosby95|5 years ago|reply
I find it interesting that your two links seem to contradict each other.
[+] rcoder|5 years ago|reply
Semi-related to this and the recent Haiku R1/beta2 announcement: just for giggles, I installed that inage on a USB3 external SSD and fired it up under KVM, and was suddenly greeted with a faster, more polished desktop environment than Windows 10 or Ubuntu offer out of the box. Consistent UI patterns, applications that open in less than the 250ms perception threshold, and some truly useful utilities installed (a fast POSIX terminal, programmer's editor, media viewers, basic but serviceable WebKit-based browser, etc.). All of it happy to run in 1-2 GB of RAM and two cores of my six-core laptop, even under virtualization.

Running on an Atom-based SBC I had on the workbench, it's even more responsive.

Yes, building native apps requires at least a basic knowledge of C++. No, it won't seamlessly run the latest React SPAs as well as Chrome. The driver situation isn't as good as (say) Linux or FreeBSD.

And yet, the focus on providing a productive environment for normal computing tasks instead of endless up-selling to an app store, countless background updaters, and vendor-bundled crapware is like a breath of fresh air.

This is what we've lost in the move to impose ever more layers of services, unique per-app GUIs, and ubiquitous (even gratuitous) packaging of "webapps" in lieu of targeted, native apps.

To be clear: this isn't a unique property of Haiku. A nice clean install of FreeBSD or Debian has many of the same properties, and I have reason to believe that a Windows 10 "distro" based on something like Server Core could be similarly light and responsive.

Unfortunately, the major platform providers seem deeply uninterested in building systems that don't push you aggressively towards newer hardware, loads of subscription services, and "strategic" software bundles no one asked for.

[+] yellowapple|5 years ago|reply
Yeah, I feel like as soon as Haiku has support for encrypted partitions it'll be my daily driver (even if it's a performance hit, I'll take that over my data being in the clear should someone snag my laptop). Implementing support for it is something I'd love to try my hand at as soon as I can figure out how I'd go about doing so (namely: how to make an encrypted partition look like a normal partition to the overall system while still providing some mechanism to prompt for a passphrase or key).
[+] PaulHoule|5 years ago|reply
I think it's the real problem of the industry. It used to be that Intel had a strong brand because you'd buy a new computer in three years that would knock your socks off. It isn't that way anymore, and from the top of the bottom of the stack we should be addressing perceived performance.

My #1 frustration as a Windows user is that every so often an applications or the whole system freezes up for a few seconds. I use MacOS sometimes and also Linux and I don't think either one is much better.

For instance when I launch a command prompt often it is up in <1s, but sometimes it takes 10s. That's a very simple program, but often there are security checks involved; even ssh logins to a linux server on a LAN have a long-tail distribution in time-to-prompt because maybe some daemon was swapped out.

I would be satisfied I had some visual indication that progress was happening (ALWAYS AND <100ms LATENCY) and some insight into the process (e.g. real progressbars... if hadoop can do it, why can't pandas?)

We don't want to trouble people with details, but what's the harm in telling the user we are waiting on a 30 second timeout?

[+] RcouF1uZ4gsC|5 years ago|reply
> For instance when I launch a command prompt often it is up in <1s, but sometimes it takes 10s.

On Windows, a reason for this might be virus scanners. Get rid of all third party virus scanners and tune the settings on the built in ones.

I personally am of the opinion that virus scanners are a waste of computing resources, in that if you reach the point where a binary you run has been infected you are probably compromised in ways a virus scanner cannot fix anyways.

[+] buboard|5 years ago|reply
People need to try some old systems, dos or windows 3 or even XP. Responsiveness does matter, a lot
[+] yummypaint|5 years ago|reply
Ive recently been noticing that g suite apps in firefox tabs are routinely grabbing more than 500MB of ram, it's often more than 1GB just to have a gmail window open. Just a few years ago this wasn't the case, but nothing has been improved as far as functionality. Anyone have insight into how we got to a place where a google sheet with 4 entries can use more memory than the entire OS? I used to routinely perform identical tasks on much lesser hardware with greater speed, so how do these seemingly broad regressive changes happen at the ground level?
[+] hedora|5 years ago|reply
I blame the Software as a Service model. Clearly, you should just downgrade to the old version of google docs (or whatever) that used 10% as many resources, and had a better UI.

Since you don’t get to control which version you use, teams don’t have to compete with last year’s version. Therefore their management doesn’t need to worry about extreme regressions in functionality, so they don’t take action to avoid them.

Since the service provider is the development team, you’re locked in, and won’t switch.

Usage metrics look good, management gets credit for shipping ${feature}, and gets their promotion. This happens a thousand times, and everyone wishes the irreparably broken ${megacorp} would just go out of business already.

This pattern doesn’t play out to the same extent with hardware, where people have to pay big lump sums to upgrade, and can hold on to older models / switch vendors instead.

[+] nateburke|5 years ago|reply
Brilliant post.

I think this trend will continue, sadly, until a crafty company is able to steal 1B of revenue from a flaky incumbent by building a reliable and fast version of a core utility.

I am reminded of the early google.com homepage. I can remember setting it as my default homepage in IE because it got to the point that Yahoo would take MINUTES to load, if not just crash the browser.

Or maybe we need hardware to just suck again, to force the issue.

[+] tonymet|5 years ago|reply
make a law requiring software developers to use 15 year old hardware when doing development
[+] phlhar|5 years ago|reply
Electron apps are easy to develop, but the performance as it is a browser is pretty bad compared to native. The performance af all electrons apps I have used (Slack, Discord, Spotify, Twitch, ...) is always worse than native apps, and not just a little bit. You will notice it! I understand it is easy to develop a electron app when you come from a web background, but it still seems wrong to me.
[+] plehoux|5 years ago|reply
Co-founder at Missive, an email client built entirely in HTML/JS; Electron on desktop; Cordova on mobile.

You can try the app, it's way faster than Mail from Apple. Apple even featured the app multiple times on the mobile App store; again all HTML/JS.

Making fast software in HTML/JS is definitely possible.

An email client is mostly HTML/Style rendering so our use case might be more aligned.

You can learn more about how we made it fast by listening to :

https://syntax.fm/show/184/desktop-and-mobile-apps-with-a-si...

[+] entropicdrifter|5 years ago|reply
My grandfather used to say, "The people who sit up front catch all the bugs in their teeth"

If you're trying to move your full desktop workflow to a platform that only added the option to work that way recently and as an afterthought compared to its original use-case, you're going to be catching bugs for quite a long time. If you're unlucky, your particular use-case for that hardware will never catch on enough for the value of working out those kinks to outweigh the cost.

I know techies like most of us on here love to be early adopters, but you have to draw a line between using tech as a toy and using it as a tool. If you're using something that you don't expect to be stable 99.9% of the time, odds are that it's a toy and you should hold your expectations a hell of a lot lower for any sort of productivity.

[+] Nextgrid|5 years ago|reply
The problem is that in some cases you are forced to upgrade your tool and turn it into a toy. Windows 10 is an example. macOS Catalina (required for latest Xcode which itself is required to target latest iOS) is another one and I've been bitten by it.
[+] _bxg1|5 years ago|reply
Economics, economics, economics. Companies only put the amount of effort into their software that's necessary to optimize profits.

Startups are incentivized to move fast and break things, and then to keep adding to their broken prototype instead of rebuilding the product, because it's more affordable.

OS vendors benefit from lock-in and hardware is fast enough that the vast majority of consumers don't notice. If something breaks, you just take it to the Apple store and they reset it. It's cheaper for everybody involved.

Online ad vendors have no incentive to create a less-than-terrible web experience because it's not their site that's being trashed.

On top of it all, there's no regulatory or institutional quality standard. It's left to be a race to the bottom.

I don't know what the fix is, or whether there is one at all, but we should at least stop being surprised. We shouldn't really be blaming it on "kids these days" either, which is an all-too-common refrain. There simply isn't a business incentive to invest in quality.

[+] bcrosby95|5 years ago|reply
Of course, that hardware receiving a lot of praise has a lot of software (firmware) behind it too. The line between great hardware and software isn't really that distinct in my mind. Lots of things that may have once been mechanical in nature have software behind it instead because of that amorphous trait of it brings about great flexibility.
[+] tonymet|5 years ago|reply
I feel that we as software engineers need a hippocratic oath of software development. One of the commitments should be to write efficient and responsive (responding to user input) software.

I feel both angered and embarrassed at software inefficiency. Trivial Electron "apps" require 1-2GB + of ram. Seconds of Input latency like typing on the keyboard or when pressing a button. Not to mention the gigawatts of electricity and gigatons of CO2 being wasted server-side on poor code , logging, encode-decode & other nonsense.

My desktop experience peaked on Windows 2k and has been declining rapidly for the last 10 years.

[+] bombela|5 years ago|reply
I don't know why you are being down-voted. I think you make a good point here.

If we were real engineers; not just by pretension; maybe we would have to abide to a minimal set of rules that respect our users and theirs wallet.

[+] robomartin|5 years ago|reply
> Something strange is happening in the world of software: It’s slowly getting worse. Not all software, but a lot of it. It’s becoming more sluggish, less responsive, and subtly less reliable than it was a few years ago.

> What baffles about these software moans is that Apple’s hardware is ever-more refined. While far from flawless, the entire lineup is now (finally) largely free from these “foundational” issues you see in software.

The answer to this is very simple (at some level): It is impossible to produce mechanical designs that are the equivalent of the software engineering abominations the world is stuck with today.

Going back to iPhone 3 days, I remember coding a genetic solver in Objective-C that was an absolute dog. I optimized as much as I could and could only squeeze so much performance out of it.

I finally gave up and re-coded the entire thing in clean C++. The code was somewhere in the order of 290 times faster. Objective-C was an absolute dog because, at a minimum, every data type you had to use was an object oriented mess. This was absolutely unnecessary, the proof of it being that my C++ equivalent performed exactly the same function and lacked nothing. In fact, it was so fast that it allowed the use of this genetic solver in real time in the context of this app. Objective-C, er, objectively speaking, had no reason to exist. Yet, it did, and lots of people believed it had to exist and likely many do today. They are wrong.

Another way to look at this is that the clean solution used somewhere in the order of 200~300 times less energy. This is something hardware engineers are keenly aware of. Software consumes energy and badly written software is even worse. Think of it this way: A bit transition costs energy. Inefficient code requires more bit transitions per unit time, therefore, more energy and more power dissipation.

Imagine the mechanical engineering equivalent. Some monstrosity where, instead of using a simple screw and a nut one ends up using a complex fastener that has layers upon layers of mechanisms and unnecessary complexity. A screw with the ability to grow and shrink to any length and diameter, including all the wrenches, tools, nuts, washers and devices needed to install, tighten and test them. Very quickly a simple product that could mechanically fit in the palm of your hand would quickly become the size of a car.

And so, in this way, mechanical and industrial design is always "low level", like always coding in assembler (not proposing we do that). Sure, materials and manufacturing techniques improve, yet, at the most fundamental level, excellent mechanical and industrial design is clean, uncomplicated, easy to understand and easy to manufacture. It's machine language, or C. Not Objective-C.

Software engineers who are not exposed to this reality, through no fault of their own, are not aware of these issues. I don't blame them. If most of what someone sees in school amounts to Python, their view of reality will be skewed.

My son is studying Computer Science at one of our top universities and has less than a year to graduate. He is home now for both the summer and due to the virus. I've been working on several embedded projects and showed him a few tricks to improve performance. He was amazed to see how you could optimize code for execution speed (with sometimes dramatic results) by making simple high level choices. For example, a down counter in a "for" loop ("i=10; i!=0; i--") is generally faster than the typical up-counter: "i=0; i<10; i++". This is due to the fact that processors have instructions like "DJZ" or "DJNZ" (Decrement and Jump if Zero / Decrement and Jump if Not Zero) that don't require loading a comparison value and engaging the ALU and sometimes even fetching that value over and over again from memory.

Software engineering doesn't have the same physical constraints found in mechanical engineering unless someone is coding for an application and a domain where excess really matters. One example of this is writing code for space hardware, where, aside from reliability, you have to be aware of the fact that every bit you are flipping will cost a quantum of energy and you might not have enough to be careless about it. Energy quickly translates to mass in the form of large cooling radiating surfaces that must be launched into space along with the hardware.

It's an interesting problem this issue of bad or bloated software. Not sure if there's a solution. There won't be until there's a forcing function that requires a change in perspective and approach.

EDIT: To be clear, my point isn't to pick on a specific language or languages but rather than use easy examples to highlight one of the problems that has been building up since the introduction of object-oriented programming. I remember the early days of OO. It was a mess. Much like the early days of desktop publishing where people loaded-up documents with every font they could find.

[+] zozbot234|5 years ago|reply
Objective C will always be an absolute dog for high-performance work because of how everything has to go through dynamic dispatch and indirection steps. It's why even Apple is trying to replace it with Swift - that name is not coincidental! And the Rust language by Mozilla is even better performance than Swift while supporting a great set of principled, higher-level language features if you want them.
[+] machiaweliczny|5 years ago|reply
I thoughr this trivial stuff like loops doesn't really matter anymore due to compilator optimizations. Isn't that the case?
[+] shakermakr|5 years ago|reply
You know what’s impressed me software wise has been Microsoft on Mac.

For home office I pretty much use their software constantly: Office 365, Teams, SharePoint, VSCode, and it’s not just rock solid, but pretty enjoyable as it is rock solidly integrated together.

Would never have thought I’d say MSFT having rock solid anything, nevermind on a Mac, but credit where it’s due.

[+] Agathos|5 years ago|reply
An Apple keyboard is a funny example to open with. Weren't we just recently celebrating the end of Apple's ill-advised MacBook keyboard experiment?
[+] codeisawesome|5 years ago|reply
The end of that failed experiment was marked by the release of a much better keyboard - which is what is shipped in the device featured.

If anything, it further illustrates the author's point that hardware has to respond to customer complaints, but software can seemingly get as awful as it wants and customers can't really vote with their feet, ironically.

[+] centimeter|5 years ago|reply
The complexity of software has far exceeded the ability of traditional strategies to manage it. We know how to reliably scale our software: the principled application of formal methods. More advanced type systems, proofs of correctness, principled math-based strategies for software construction, etc. Yes, all of these are locally more expensive than the slipshod strategies most companies are using now, but I am confident that if the cost of buggy software was correctly internalized, it would become clear that improving the processes we use to create software would leave us a lot better off in the end.
[+] asdfman123|5 years ago|reply
It could have to do with talent shortage. If you're a company that wants to hire a team of people like that, how much are you going to have to pay for it?

And if you're making, say, a chat app, are you sure you're not going to be beaten by a bunch of 20 year olds who slap together JS and Electron, ultimately winning because they get that people want funny reaction gifs built in?

Maybe it freezes for a half second every so often and takes 10 seconds to load, but no one cares because 1) it gets to market three times faster and 2) it has funny reaction gifs.

[+] entropicdrifter|5 years ago|reply
The biggest issue with this mindset is that attempting to deploy self-discipline at scale is a losing strategy, both on the individual project level (if they're large enough) and at the corporate level. That's why the most successful large projects always have a 'dictator' acting as the standard-bearer for quality.

Often it can be quite advantageous for individual instances, no doubt, but expecting humans to be disciplined en-masse without serious external oppression is a bit like expecting cats to neatly line up for treats.