Ask HN: What is the single top-priority software engineering problem?

[abrax3141]

If you have unlimited time and/or resources, what single software engineering problem would you address? I'm not talking about "Peace on Earth"-type problems, but rather real world practical problems facing software engineering that could be actually solved if you could pay an rationally large team of serious hackers a rationally large amount of money for a rationally long period of time. Another way of asking this is: What's the most important piece of technical debt across software engineering, that could practically be solved if we put enough energy into it?

[simonw]

Development environments. The amount of time and hassle I've seen lost to getting a working development environment up and running is incredible.

Every time I talk to someone who's learning to program I tell them: "Setting up your development environment will be a nightmare. I have been doing this for twenty years and it's STILL a nightmare for me every time I do it. You are not alone."

Docker is reasonably good here... except you have to install Docker, and learn to use Docker Compose, and learn enough of the abstractions that you can fix it when something breaks.

https://glitch.com/ is by far the best step forward I've seen on this problem, but it's relatively limited in terms of what you can accomplish with it.

[amasad]

This is my calling. I'm the cofounder of Repl.it and I've dedicated my career to solving this problem.

Simon -- since you're the cocreator of Django, you might get a kick out of this: From loading up a Django environment to adding and rendering the first view takes less than a minute: https://gifs.amasad.repl.co/django.gif

Before starting this company I was a founding engineer on the React Native team where I focused speed of setting up and taking the drudgery out of cross-platform mobile dev. And before that I was a founding engineer at Codecademy where we were attacking the same problem from an education angle.

With Repl.it, however, we're determined to simultaneously solve the "getting started" problem while scaling the environment to allow real work. It's incredibly challenging but we've made great progress. If anyone is interested in working with us, collaborating, or if you're simply passionate about this problem and want to chat then I'd love to hear from you: [email protected]

[darzu]

My dream is a "Open in VSCode" button on each GitHub repo that would create a container dev environment (either locally or hosted), and that environment would be ready to run all test, CI, localhost etc.

You could trivially create a new environment to test out someones PR or work on a feature branch, etc. If you use hosted environments, then you could connect from any client. If you only have a web browser, then you could work from VSCode in the browser (it's all JS+HTML anyway).

Others could join your hosted environment for peer-programming Google Docs style.

On hosted environments, normally long builds could be done very quickly and smart caching on the backend could share incremental compilation state between your teammates and you.

We're pretty close to this already with VSCode Remote Containers[0], Visual Studio Online[1] and VSCode Live Share[2].

(Disclosure: I work at MSFT but not on these technologies.)

[0] https://code.visualstudio.com/docs/remote/containers

[1] https://code.visualstudio.com/docs/remote/vsonline

[2] https://visualstudio.microsoft.com/services/live-share/

[silviogutierrez]

Nix is the closest I've come to this dream.

I like docker, but it's too resource heavy on a Mac. I need it running directly and not in a VM.

So a simple nix file, and nix-shell shebangs[1] has been life changing. Throw in --pure and now you really know the environment works everywhere.

My repos all have a shell.nix with every dependency. Just call nix-shell and it works in Linux and OSX. Anywhere.

You can even bundle it into a docker image and use nix in there too. For production, etc.

Documentation and tutorials are lacking, but if you put in some time, I've found nothing comparable.

Version it in, pin the Nix channel[2], and you can come back a year later in a brand new machine and start where you left off.

[1] https://gist.github.com/travisbhartwell/f972aab227306edfcfea

[2] Highly counterintuitive: if you don't pin a Nix channel, it's a moving target and dependencies may go missing, be updated, etc.

[lame88]

A constantly cycling proliferation of different languages, frameworks, libraries, etc. that all do the same things in a different way and are most often mutually incompatible with each other and have entirely different ecosystems with their own comparative advantages but also major pitfalls. This causes tech workers’ investment in skills to get more out of shallow knowledge and trivia than on deeper concepts, creates silos of employment opportunity based on the trivial knowledge workers have, and hinders the ability for the software engineering field as a whole to have a large pool of shared knowledge and develop and evolve stable, relatively timeless systems and tools of high quality, both as end products and in intermediate tooling toward those ends.

[closeparen]

Learning new skills is easy enough, what kills me is the enormous manpower invested in churning working code.

Easily 30% of my company's total engineering effort is just treading water, migrating from a deprecated platform to another one that will be deprecated by the time the migration is complete. Typically because the original team's standard 18-month tenure has elapsed and the new guy was under-leveled at hiring so he needs impact for promotion.

It's a great disservice to our field that people so deep in the stack are so comfortable changing their minds all the time. The Python 2.7 thing feels like the Library of Alexandria. Burning down mountains of perfectly good working code just because we can.

Backwards compatibility is tragically underrated.

[Viliam1234]

I was thinking about the reasons why this happens, and I see two major factors:

1) Tension between abstraction and optimization. To put it shortly, abstraction is about ignoring the details, optimization is about fine-tuning the details; you can't do both at the same time. Which is why different programming languages make different kinds of compromise. You could make a beautiful language or framework with elegant abstractions, then look at the performance and cry. Or optimize for performance, and then cry while reading and debugging the code.

2) Tension between mathematical elegance and the cost of hiring people who are great at math. Some developers care about having the code elegant from mathematical perspective, but the companies optimize for having a product to sell, as cheaply as possible, which involves a lot of cutting corners. A product full of bugs can still make you millions of dollars. And what's the point of having a mathematical proof of correctness of your current code, when the business requirements are going to change tomorrow anyway.

[hanoz]

And conversely, the problematic consequence and cause of all that: Resume Driven Development.

[zelly]

You don't actually have to invest in these new shiny ecosystems. It's just a choice you made. There are still people making a living on Fortran, C, C++03, Java <5, Angular. All of that still works on modern hardware and will for the next 100 years. You just probably won't be in a coastal place with pool tables and free food.

[mamcx]

I think this is a manifestation of a more bigger issue: C, C++, JS.

This happend because our foundations are shaking, and is requiere to put on top some sanity. But nothing can work because: This industry not let go of the old and bad (C, C++, JS, Bash, etc) and refuse to do a significant change.

Too much change is bad.

Too little change is bad.

Have both things at odds in a fierce battle is worse.

[fmjrey]

Totally agree, this is a problem I also see plaguing our industry, and solving it is indeed a priority. We developers or engineers like wasting our time in abstractions, i.e. virtual worlds of isolated castles and babel towers of different languages and expertise. In such virtual worlds we can create and fantasize on all the abstractions we want, be the expert, play god, decide on which constraints matter, and build a world based on that. Such is the power of the non material software world indeed. Who does not want to be god? To caricature I'd say a significant part of the IT industry is about writing virtual entertainment: video games to be sold to the wider public, and toys developers and engineers can play with: languages, IDEs, ecosystems, frameworks, etc.

Earlier in this thread I wrote about data being the top priority [1], the raw material, that we rarely give data the primary focus it deserves and instead focus too much on the processing side. More concretely: dashboards showing instrumented processing clusters give a biased view that does not focus on what matters at the end of the day. What we also need are dashboards showing data flowing between sinks and sources, data quantity, data quality, etc. Sure resource utilization and efficiency matters, but only after we can validate we still have the right output, and that input is of proper quality. If output contains garbage, is it bad processing or is it garbage from the input? And if something is wrong with processing, do we know the impact downstream? In other words instrumentation should include data sensors, not just processing sensors: data counters, validation points, invariants, etc. Because at the end of the day, when the power goes off, do you know what's left to recover? Do you prefer your customers telling you about an unfulfilled order or do you rather want it to be detected earlier? If you get audited for GDPR, do you have a map of your sensitive data? In terms of security, is it about protecting clusters and containers, or is it about protecting the data? Once you get the data side right many things become simpler, but if you get it wrong, as we often do, we create a world of problems. Giving data its proper place in our engineering practices will certainly change our industry for the better and bring it closer to "reality" with less danger of veering into the virtual for the sake of it.

In a world where software is increasingly involved in human activities this would have a great impact. However I don't think we should stop there, as I believe this is part of a larger trend I'm concerned about: the idea that, not just in software development but in most human endeavors, we're increasingly favoring spending time in virtual/man-made spaces and activities, at the expense of the real world, the place and time we're at, nature and the environment. As if we want to escape the physical conditions we're in: whether it's our body, our environment, society, the work we do, etc. When one can't see a way to influence the real world a tendency would be to start operating in a virtual one where we get the illusion to have an effect, make some money, be an expert, etc. Oh and let's not forget this desire to put as much tech between us and the real world, as if we don't want to experience it directly, it's too icky, and instead need devices to offer an indirect perception: wearable tech, navigation by GPS, remote controlling tractors in vast industrialized agricultural fields, etc.

A friend working in a supermarket chain told me this story: he often advised a younger manager to consider better maintenance procedures of their A/C system, but to no effect. Now that my friend is retiring soon this younger manager is proposing to make an excel chart to track energy consumption in order to optimize setpoints, and asks my friend for his approval. Really? Approve perception to be limited to an excel chart? Isn't that the same as leaving the windows open and wanting to change the setpoint?

[1] https://news.ycombinator.com/item?id=22277875

[gdsdfe]

Do you work in software? It's making new things with new things that software makers better at what they do.

[dfabulich]

We need a faster web framework that generates HTML on mobile phones with no JS on the main thread.

The web is the "single top-priority" software platform, but it's in big, big trouble.

On mobile, users spend less than 7% of their time on the web. https://vimeo.com/364402896 All of the rest of their time is in native apps, where big corporations decide what you are and aren't allowed to do.

As a result, the money is going to native apps. The ad money is going there, the dev time is going there, and the mobile-web developer ecosystem is in peril.

The biggest reason people use native apps instead of mobile web apps is performance. Developers design web apps for fast desktop CPUs on fast WiFi data connections, and test their sites on top-of-the-line smartphones costing 5x-10x as much as the cheap smartphones people actually carry around.

Web developers have to solve this performance problem the way we've always solved our problems: with a new framework. ;-)

But specifically we need a framework designed to generate HTML with no JS at all, and designed to run in a Service Worker, which is a little web server that runs directly on the user's phone.

This style of app is often called a "Progressive Web App," and there are plenty of frameworks that support PWAs, but they generate PWAs on top of a single-page app framework that downloads megabytes of JavaScript running on the main thread. PWA is an afterthought for most frameworks, but we need it to be the centerpiece of the design.

[fmjrey]

What matters most is what you want to still be there when the power goes off: data.

No amount of processing power matters if you don't have the data.

Everyone in the industry focuses too much on the processing side: objects, functions, containers, VMs, k8s, etc. but nobody really gives proper attention to data, its provenance, where it stays and where it goes, etc. I'm not saying engineers don't think about these things, they obviously have to think about it at some point. It's just that data is always accessory to the story. It's like processing is the cool kid and data is the stinky one nobody wants to approach unless you have to. Look at the 12 factor principles for example, where is data in there? How easy is it to take data from one place/cloud/database to another? Data is the raw material, it needs to be the primary concern in programming languages and architectures, not objects or functions, containers or whatever, those come after, not first.

[Areading314]

SQL and its continued success is a testament to how valuable this approach can be. Could be something here though, as I have found maintaining databases and datasets in a sane way to be difficult, even today.

[pesfandiar]

The problems around data are in general harder to address than the processing part. Most people dabbling in software engineering don't have the skills or attention span to work on those, and most of the issues are already solved by extremely complicated systems (e.g. DBs, Apache projects, ...).

As for 12 factor principles, it was first touted by a PaaS provider. The whole idea is that they take care of the nitty gritty, while you can focus on the exciting (and technically easier) parts of software systems.

[Rafuino]

As someone in the hardware world where CPUs and fancy accelerators are all the rage, it's the same story when it comes to thinking about non-volatile memory (AKA NAND, Optane, PMem, etc.)

[pmlnr]

I would add the problem of long term archiving - both as the physical medium and file formats into this. Will my jpgs be accessible 50 years dpwn the line? If yes, who will remember and have the equipment to read 100gb blu-ray mdisc?

[dane-pgp]

A piece of technology that seems to be missing is a global decentralised anonymous identity and trust framework. It probably requires a leap of engineering comparable to the invention of the blockchain (although I don't think that a blockchain is necessarily the model to follow here, since the data should be encrypted).

Every website and app and network service seems to be reinventing the wheel here, and end up creating little silos of trust data, whereas in principle it should be possible to receive a (locally) consistent answer to the question of "Is this person in good standing with the other humans they interact with?" whether that person is sending you an email, or writing a software library you are downloading, or creating an account on your website, or selling you goods on Ebay, or offering you a lift through a ride sharing app.

[say_it_as_it_is]

I would fund a rigorous study of frontend development with a team of academics who would gather details from a very large sample of organizations about frontend development projects. My theory that I would seek to validate is that the vast majority of frontend work was unnecessary, overly complex, costly, and shouldn't have ever been funded. Chasing new approaches creates new problems. Following FAANG solutions to problems no one has is costing everyone time, effort, and money. Myths need to be debunked. Anecdotal evidence consisting exclusively of success stories is concealing the truth. It feels as if the entire frontend world has gone crazy and received financial support to fund an expensive addiction to unnecessary complexity.

[ketzo]

Can you give some specific examples of this problem? Frontend developer curious what kind of trends/solutions you mean.

[arexxbifs]

The decline of usability, recognizability and coherence in desktop user interfaces. I honestly think we reached peak UX some time in the mid-90s. With the advent of touch devices, paradigms are mixing in a way that's directly hostile to productivity.

[blihp]

I agree with your take that mid-90's was probably peak UX. But I think that has more to do with that being about the time when companies stopped trying to strike a balance between accommodating both power users and casual users. Since then, much more effort has been put into providing a polished/slick UI at the expense of things like automation. Of course, that also plays into the advertising model which I believe is diametrically opposed to things like automation. (i.e. if you're not clicking/touching, they can't tell if you're looking at the ads)

Touch capabilities can be a nice addition to many types of desktop productivity software without detracting from what's already there. Companies reworking desktop applications to look and feel like mobile apps (with all of the related pros and cons) is what causes productivity to suffer rather than anything inherent in the paradigm.

[crispinb]

It's pretty shocking that we are where we are in 2020. That year sounds like the future to me, but in computer interface terms it's definitely dystopia. The market failure to cap all market failures!

[jspaetzel]

Understanding the problem.

Most of the time the users haven't taken enough time to understand their own problems and have trouble articulating them in a way that's meaningful for a product manager or developer. And on the opposite side of that product managers and developers often do not develop sufficient domain experience to understand the problems users are trying to express.

This is why the absolute best software is built by people who are developing for themselves. You know when it's not solving your problem and you fix it because you know nobody else will.

[cs02rm0]

I seem to spend far more time on the deployment of code than I ever used to, more time writing CloudFormation and ansible than the Java backend, python lambdas and static UI that it deploys.

That seems nuts to me. I'm not sure how you fix it without being Amazon/Google. People moan about Terraform too before that gets mentioned.

[thom]

No idea where this fits on the priority list, but I think a lot of problems around stream processing still aren't solved, and it's holding us back from a really productive programming paradigm. Handling updates/retractions elegantly is hard or impossible on many platforms, handling late (sometimes _extremely_ late) data can be very inefficient. Working with complex dependencies between events (beyond just time-based windows), in realtime, can be really tough. As the saying goes, cache invalidation is one of the hardest problems in software engineering. Having a simple platform to represent processing as a DAG, but fully supporting both short and long term changes transparently would make event sourcing architectures trivial and extremely productive. The closest we've come seems to be:

https://github.com/frankmcsherry/differential-dataflow

Lots of very active CS research in this area though.

[btown]

When humans enter the mix, it becomes really complicated. Your event sourcing code can no longer look at events like "Human A changed the record with primary key B to have property C equal to D at time T" because that event is actually a function of the snapshot of how Human A saw the entire state of the interface at time T. And what if someone edited the record with primary key B to represent an entirely different entity (from the perspective of a rational human observer)? Is it still the case that a prior event on this record should be retroactively interpreted to affect the new identity? All code becomes recursively dependent on all prior code, and all events become recursively dependent on prior events. Figuring out how to code in a paradigm where you can't retire old code is very difficult and I'd welcome people pushing the envelope.

Martin Kleppman's articles and research is a great place to start for anyone interested: https://martin.kleppmann.com/2015/02/11/database-inside-out-...

[3pt14159]

Unlimited time and resources? I'd try to tackle the global security issues related to advanced cyberattack. It's such a complex problem I don't even know if it's possible, but it would require hardening software update servers, networks, and utilities (especially electrical power distribution) to the point where a single bad Windows update doesn't take out the economy.

[tzs]

Documentation.

Far too often when I want to learn something new I either find there is not much documentation or find that there is a large mass of documentation that probably has everything I need but is so disorganized that I can't figure out how to approach it.

Another documentation problem, especially with open source projects, is that development and documentation are often loosely coupled, if at all. The people doing documentation usually don't have the resources to keep up with development, and so even if there is good well organized comprehensive documentation it is usually obsolete.

[thepenguinco]

Hey there, I'm actually currently working on something in the documentation space that will benefit both companies and open source projects.

Would love to chat with you further! Shoot me an email over if you're interested at [email protected]

[rodolphoarruda]

Having my phone as my ultimate CPU, immediately connectable to commodity peripherals (monitor, keyboard, printers) at home, office and on the street. Content would be stored on the phone following the "local fist" principle to favor speed and security.

Something like Ubuntu Edge for those who remember it, but with more local storage -- TBs maybe -- more connectivity out of the box.

[arexxbifs]

I envisioned something similar around the time of the 2nd gen touch phones: just plonk your phone into a dock with keyboard, mouse and monitor.

I realize now it’ll probably never happen, since the companies making the phones are the same companies that make computers.

[fsflover]

Here you go: https://puri.sm/products/librem-5/

[dyeje]

I talk about this all time. Our phones are strong enough, just give me a stupid terminal to plug in to for a keyboard and screen.

[AndrewKemendo]

Yeah I'd love to just dock my phone into a desktop setup.

Not sure why this doesn't exist for Apple/Android. It seems so obvious that they're must be Some critical flaw I'm overlooking

[ronyfadel]

Decent nocode, or some sort of nocode holy grail.

I want to be able to plug APIs together, process user input, have persistence and identity, without writing so much boilerplate.

[keenmaster]

If we don't take software engineers as a fixed population, but rather everyone on Earth with sufficient intelligence to code well with the right no-code framework (which currently doesn't exist), then this is indeed the biggest priority. Of course, I am under no illusion that no code will work for everything, such as developing ML applications from scratch. Even then, it can be a way to interface with pre-built ML apps, and use them in various industries. It would also serve as a pipeline for non-STEM workers into full-blown programming and STEM.

Think about what Excel did to productivity, and multiply that by 3x or more. That's what no code could do as the 60% solution for a 200x larger labor pool (more like 80% solution for anyone who can't access engineering talent). It would also make its creators obscenely rich. With the amount of processing power that we have, the time for no code is now.

By the way, there's a large chance that Big Tech incumbents won't be the ones to create the no-code holy grail. They have institutional handcuffs that will make that difficult.

[tomc1985]

Ah, the impossible holy grail of businessmen and marketeers everywhere

[AndrewKemendo]

This seems like an antigoal to me.

Abstracting away infrastructure just means it will become increasingly centralized as a commodity and less likely/harder to create independent services.

[js8]

Formally specify all the commonly used languages, runtimes and APIs, so that we could translate programs between them. Then make sure that all programs such translated can be compiled, so that we don't waste resources by running inefficient VMs.

[belltaco]

>What's the most important piece of technical debt across software engineering, that could practically be solved if we put enough energy into it?

Being able to update libraries, tools etc. automatically and without friction. Right now upgrading is so tedious, error prone and painful that most places just keep using ancient versions that are not only lacking bug fixes and newer features but are a huge attack surface.

[naasking]

A true build once run anywhere platform, that doesn't sacrifice security or verifiability, and that would also scale from embedded systems (that are often married to their own limited C compiler), up to distributed systems and mobile code (like browsers acting as a remote UI typical of web apps).

WASM is a great advance over what came before in many ways, but still has some room for improvement. These are all problems with known good solutions, but there is no holistic platform that integrates these smoothly. It's largely a hodge podge of various programming languages, configuration systems, build systems, scripts, etc. Look elsewhere in this thread for people complaining about how difficult it still is to setup a development environment. Embedded programming is typically even worse, though it's improved dramatically since the rise of Arduino.

It needn't be this way though. A well designed programming language can be used for configuration, scripting and more. Racket is a good example on the dynamically typed side, and F# is a good example for a statically typed language along these lines.

[GrumpyYoungMan]

Build the tools and infrastructure necessary to make FPGA accelerator programming accessible to the average programmer. Moore's Law is mostly dead; we are getting more cores but we are not going to get significantly faster cores in the near future. What will bring us next big jump in computing performance is unclear but FPGA acceleration seems like one of the few promising directions.

[petters]

A computer and OS that boots in 100ms. Every user action gives a response in 10ms.

[snisarenko]

Documentation!

I am willing to go out on a limb and say that as much as 25% of software engineering time worldwide is wasted due to poor documentation.

It's an asymmetric problem too. If someone benevolently funds a team of engineers for a couple of months to write great docs (with detailed examples) for top 500 libraries, frameworks, APIs. They could increase global productivity of software engineers by %25 percent.

[pbedat]

A very conservative estimation! And it's not only documentation of software but also the requirements and other parts of the whole lifecycle. It's incredible how many companies hold meetings after meetings to keep up the oral tradition only to run in circles. Instead of POs who churn out dozens of irrelevant tickets, we need people who can tell and write stories - real stories not "user stories". They offer insight, motivation, engagement and the right understanding to break down the work into reasonable and deliverable bits.

[thepenguinco]

I'm actually currently working on something in the documentation space that will benefit both companies and open source projects. Would love to chat with you further! Shoot me an email if you're interested. (li.eric00 at gmail)