It seems too powerful, though. I could see it quickly becoming intractable to find where the code lives that is causing a certain outcome.
The root problem it's trying to solve is the fact that things often don't fit neatly in a single bucket, but our traditional idea of modules as mutually-exclusive and hierarchical categories forces you to choose just one place to "put" something. An idea I've had is to get rid of hierarchical modules altogether. Really they're a relic of directories and files. Instead, one could "tag" a function/class/constant as belonging to multiple modules, and one's editor could aggregate all members of a cross-cutting module as needed. Organization within actual files on disk becomes an implementation detail.
> Instead, one could "tag" a function/class/constant as belonging to multiple modules, and one's editor could aggregate all members of a cross-cutting module as needed. Organization within actual files on disk becomes an implementation detail.
> I could see it quickly becoming intractable to find where the code lives that is causing a certain outcome.
(Author here.) The tangler I describe inserts #line directives into the output. You can open up the generated C++ and see where a line comes from. Conventional C++ tools understand line directives as well, so the debugger experience is sane.
I'm sure there are gaps in this tooling, but they're not intractable. Tooling around conventional modules has matured over decades. My approach hopefully shows some benefits of a road not taken.
Well this is interesting, I recently spent a weekend creating a 'splicing compiler', which could take a typescript project split up into features, where a feature was written in just one single typescript file. Each section of code could then specify new 'code hooks', for future code to insert itself at during compilation.
My original inspiration was that English language is spoken sequentially, but ends up in a graph database (i.e. the brain's neural network). So you'd write code like a thesis, and the hooks you specify would allow the splicer to form the code graph.
In theory it seemed like it would be useful, so I built Pacman with it and, well, it didn't really work in practice! Things ended up being more abstract in the end, and I'd be mentally juggling where hooks were created and what the final code would look like. Even showing the code output to the right wasn't that helpful, as the way the code was spliced together put things in not the most readable order.
Additionally how to split up the code wasn't the most obvious, i.e. when should a hook be added? It often felt I'd only know after writing the code in full. At which point it just felt like extra work. I'd often find myself to be inserting code all over the place to get something working, and so the refactoring needed at the end became overwhelming. Ultimately these reasons lead me to abandon the idea.
I've written 35k LoC with this tool, and it does require some taste but not intractably so. A couple of rules of thumb:
- Don't use layers in too fine-grained a manner. Often if I need to change two lines in a function but they're separated by 6 lines I err on the side of replacing the entire paragraph or even the entire function. Each layer should be readable in a self-contained manner as much as possible.
- I never invalidate tests in an earlier layer. You never want to be in a situation where a layer lies to you about what desired behavior is.
It seems like a lot of programming ideas are just ways to manage spaghetti rather than reduce the spaghetti. I worry that a lot of programming is a sort of local minima, where people dig themselves into a hole that they can't iteratively dig themselves out of. Something like this, which allows you to write code full of cross-cutting concerns and lets the programmer fool themselves into thinking they wrote clean code because they swept everything under the rug - that kind of tool is just going to dig deeper holes.
> It seems like a lot of programming ideas are just ways to manage spaghetti rather than reduce the spaghetti.
Agreed! This is why I built this approach in a ~1000 lines of code. The ability for tools to create new problems was very much on my mind.
My current project using this approach is a bootstrapped computing stack from scratch: https://github.com/akkartik/mu#readme. It's at ~40k LoC (of machine code) and self-hosted, and it can create bootable images (when packaged with an OS kernel). I expect to get to approximately C's level of expressivity in another ~20k LoC or so. So I care very much about what I call the "thrust to weight ratio" of a codebase. The less code you have, the less spaghetti you can create.
My implementation of layers will keel over and fail terribly for large projects with many team members. I consider that constraint to be a good thing.
Sometimes I think this is a good idea but generally it seems like refactoring in a way that the code base always makes sense, even to a newcomer, is the correct goal. History can be kept in source control, if it's necessary to follow the history of code development it's probably a sign that the abstractions are not chosen correctly.
The analogy to `git log` is just that, an analogy. Here's a concrete thing that you get with layers that you don't get with `git log` or conventional abstractions and refactoring: alternatives. Often we see comments in codebases that some complex, hairy function is really a much simpler function with certain optimizations applied. There's been lots of work done to try to separate out functional spec from optimization so that the two can be managed separately. But this is all open research. Layers let you get the value of this separation today, with very little code (just a lot of reorientation in mindset). If you adopt layers you can use the simple version in an earlier layer, and then replace it in a later layer. Your CI system will exercise both, and flag if either stops passing tests. What used to be a code comment now stays up to date.
Well, what if you could have approach described in the article but there would be a defined patch interface so that if you looked at it, each patch makes sense also?
So you‘re trying to remove complexity by making it invisible? Sounds great while things work, but doesn‘t that turn debugging into a nightmarish exercise? Because I never get to see the code that‘s actually executing, the pieces of a function might be scattered over a dozen different layers. Strikes me as the ultimate recipe for self-modifying spaghetti code... Or am I missing something here?
I'm not sure what you mean by 'invisible'. Notice that I make zero claims of reducing complexity. Layers are only a way to manage and stage complexity.
I think the core problem is that one of the root causes of the patchwork hard to understand code is lack of spending the effort to keep the code understandable. Refactor into new models, move around code so it makes more sense given changes in requirements, etc.
If that level of maintenance isn't being performed (which is almost a given, assuming we are looking at a problem with such a patchwork problem), then this new tool won't be fully followed and the end result will be a patchwork code base that reaches nightmarish levels where some features are added in this layered approach, but other features are added directly to the code, and even other features are added directly to prior layers.
If the tooling force people to use this in the recommended way then it may have interesting payoffs that are worthwhile. But lacking such an enforced requirement, I don't see applying being beneficial except in the code bases that least benefit from it.
I find this approach to be easier than conventional approaches, and my motivation and effort levels seem constant on both sides. You're right that there's a social component here. This tool tries to make things easier for authors under the assumption that people try to use it tastefully.
There seems to be some confusion between layers and AOP. Layers are a static mechanism - all the layers are composed to a final program before compilation. The layers have no effect at runtime (unlike AOP). If the all the layers are composed (tangled) without the '#line' directives, the code should look (and be) the same as a normally-written program. Modifying the program most likely will require adding a new layer. (If "layer" == "git commit" then this is only option under existing programming practice.) Programming with layers means the programmer has the additional option of modifying previous layers. Whether this can scale to large programs is an open (and interesting) question.
Reminds me of Layom, which is something my phd supervisor worked on in the nineties. Later aspect oriented programming (which is similar) became a thing with languages like AspectJ around 1999 or so.
These days you see a weaker form of this in the form of annotations in e.g. Python or Java or macros in Rust. Spring actually uses an AOP library that has its roots in AspectJ.
I actually prefer a more recent trend in e.g. Kotlin to use internal DSLs instead of injected magic. The advantage is that it is simpler to debug and part of normal type checks, autocomplete, etc. since technically all you are doing is working with the language instead of augmenting it via generated code.
Or you have to keep the codebase small. I'm not convinced an OS or programming language or database needs millions of lines of code.
I'm not alone in this scepticism. Check out Alan Kay's STEPS project from a few years ago. The goal was to create a complete computing stack in 20k LoC.
You mentioned tech debt. I think I have a more aggressive definition of tech debt than most: any dependency you rely on that you don't understand the internals of is tech debt. It's fine to take on in the short term, but you should be planning to pay it off in the long term. The approach modern software takes, of continually adding dependencies (each of which is itself growing monotonically complex) faster than anyone can internalize them, this approach feels utterly insane. Under this definition, my tiny tool is the least of someone's problems if say they depend on 60 Ruby Gems.
Yes, I was aware of both as well as AOP when I built this. Some comments on OP discuss them as well.
The big difference between my approach and these is the weight-to-thrust ratio. These academic approaches tend to assume people don't want to change the pieces, just combine them. And they create huge tools to make that possible. Why can't people just change the pieces? I've never understood that. I don't need an algebra of program transformations intermediating between me and my code. I'd much rather hold the code directly in my mind. And that keeps the tooling needed really, really small. Just a hundred or so lines of C.
Tooling is a double-edged thing. Adding tooling often encourages increases in codebase scale. Keeping tooling rudimentary can be a good thing by keeping codebases small.
Fascinating idea. I've definitely written and encountered programs where functions were more complicated than they could've been, in order to support features separate from the "main idea" of the program. Like FPS calculation, Ctrl+C handling, passing intermediate audio data to extra buffers for visualization...
> There's a new constraint that has no counterpart in current programming practice — remove a feature, and everything before it should build and pass its tests:
> build_and_test_until
Looks like added tooling is needed, but seems worthwhile.
Interesting, although the features might leak into other codebases and services too, at which point compiler directives won't be enough.
I believe this is a lot more tractable with Unision's[0] content addressable hashing. It's kinda made for that although more with an eye towards updates to distributed systems.
About 15 years ago, I built a system like this for a web application. I defined a series of core extension points such as the request router and authentication and authorization providers. Modules provided extensions that connected to the core extension points, as well as define their own extension points that other modules could connect to. Dependency injection was used throughout so that implementations could be easily swapped (mostly used for the tests). It took quite a bit of time to develop this framework, but, as a result, everything was modularized, contained, and composed.
It was a complete nightmare to reason about and work with, even for me, and especially for newcomers. A natural way of trying to understand a system is to look at its entry point. With a system like this, it looks like a skeleton:
let di := DependencyContainer.make()
let moduleSystem := ModuleSystem.with_container(di)
di.register_singleton[IModuleSystem](moduleSystem)
moduleSystem.find_and_register_modules()
Where do we go from here? The next logical step is into `find_and_register_modules` or into the documentation for the module system (just kidding: I was a "senior engineer," I didn't write docs).
We are left with a lot of questions. How does anything happen? It's somewhere inside the modules, but which one? Does the order that modules are loaded matter? The answer is maybe. Clearly, there are places where order of operations matter. For example, the main navigation should be ordered. How do we accomplish that? (The answer for my system was to have the interface for navigation extensions specify there was a `readonly weight: Float64` attribute. But then to actually understand why things are in the order they're in, and to get things ordered correctly, you need to look up the values for other navigation items. It's secret coupling.)
More recently, I saw a team at my previous company build a system like this. There were dozens of interfaces and implementation classes and code for composing all of this, and the end result was you couldn't just go somewhere and see what was happening. What we really want to see is:
Do you have to make a change here whenever something gets added? Yes. You do. But the idea that your program needs infinite flexibility in all areas is simply wrong. And, in fact, everything ends up secretly coupled and inscrutable.
For this reason, my advice has long been to limit extensibility to specific, narrow use cases (for example, filters in an image editing program). Do not build entire systems around extensibility. You are not building an abstract system, so don't waste your time with unnecessary and obscuring abstractions.
Author here. I read your comment and just think I'm not a very good writer. Layers are absolutely not about adding extension points, and layers have nothing to do with modules with fixed interfaces. Often when I want to extend behavior in my layered programs, I just modify the line in place.
You can find a list of layers here: https://akkartik.github.io/mu1 (URL is slightly different; it's a previous prototype. But should suffice for this thread.)
[+] [-] _bxg1|6 years ago|reply
It seems too powerful, though. I could see it quickly becoming intractable to find where the code lives that is causing a certain outcome.
The root problem it's trying to solve is the fact that things often don't fit neatly in a single bucket, but our traditional idea of modules as mutually-exclusive and hierarchical categories forces you to choose just one place to "put" something. An idea I've had is to get rid of hierarchical modules altogether. Really they're a relic of directories and files. Instead, one could "tag" a function/class/constant as belonging to multiple modules, and one's editor could aggregate all members of a cross-cutting module as needed. Organization within actual files on disk becomes an implementation detail.
[+] [-] akkartik|6 years ago|reply
Yes, this is interesting. Are you thinking of something like Ruby's modules and `include` directives? (https://ruby-doc.com/docs/ProgrammingRuby/html/tut_modules.h...) Is there a reason the editor has to do this rather than the language?
[+] [-] akkartik|6 years ago|reply
(Author here.) The tangler I describe inserts #line directives into the output. You can open up the generated C++ and see where a line comes from. Conventional C++ tools understand line directives as well, so the debugger experience is sane.
I'm sure there are gaps in this tooling, but they're not intractable. Tooling around conventional modules has matured over decades. My approach hopefully shows some benefits of a road not taken.
[+] [-] hacker_9|6 years ago|reply
My original inspiration was that English language is spoken sequentially, but ends up in a graph database (i.e. the brain's neural network). So you'd write code like a thesis, and the hooks you specify would allow the splicer to form the code graph.
In theory it seemed like it would be useful, so I built Pacman with it and, well, it didn't really work in practice! Things ended up being more abstract in the end, and I'd be mentally juggling where hooks were created and what the final code would look like. Even showing the code output to the right wasn't that helpful, as the way the code was spliced together put things in not the most readable order.
Additionally how to split up the code wasn't the most obvious, i.e. when should a hook be added? It often felt I'd only know after writing the code in full. At which point it just felt like extra work. I'd often find myself to be inserting code all over the place to get something working, and so the refactoring needed at the end became overwhelming. Ultimately these reasons lead me to abandon the idea.
[+] [-] akkartik|6 years ago|reply
- Don't use layers in too fine-grained a manner. Often if I need to change two lines in a function but they're separated by 6 lines I err on the side of replacing the entire paragraph or even the entire function. Each layer should be readable in a self-contained manner as much as possible.
- I never invalidate tests in an earlier layer. You never want to be in a situation where a layer lies to you about what desired behavior is.
[+] [-] c3534l|6 years ago|reply
[+] [-] akkartik|6 years ago|reply
Agreed! This is why I built this approach in a ~1000 lines of code. The ability for tools to create new problems was very much on my mind.
My current project using this approach is a bootstrapped computing stack from scratch: https://github.com/akkartik/mu#readme. It's at ~40k LoC (of machine code) and self-hosted, and it can create bootable images (when packaged with an OS kernel). I expect to get to approximately C's level of expressivity in another ~20k LoC or so. So I care very much about what I call the "thrust to weight ratio" of a codebase. The less code you have, the less spaghetti you can create.
My implementation of layers will keel over and fail terribly for large projects with many team members. I consider that constraint to be a good thing.
[+] [-] zwieback|6 years ago|reply
[+] [-] akkartik|6 years ago|reply
The analogy to `git log` is just that, an analogy. Here's a concrete thing that you get with layers that you don't get with `git log` or conventional abstractions and refactoring: alternatives. Often we see comments in codebases that some complex, hairy function is really a much simpler function with certain optimizations applied. There's been lots of work done to try to separate out functional spec from optimization so that the two can be managed separately. But this is all open research. Layers let you get the value of this separation today, with very little code (just a lot of reorientation in mindset). If you adopt layers you can use the simple version in an earlier layer, and then replace it in a later layer. Your CI system will exercise both, and flag if either stops passing tests. What used to be a code comment now stays up to date.
[+] [-] joe_the_user|6 years ago|reply
[+] [-] veddox|6 years ago|reply
[+] [-] akkartik|6 years ago|reply
I responded to tooling questions at https://news.ycombinator.com/item?id=21766557#21767393
[+] [-] SkyBelow|6 years ago|reply
If that level of maintenance isn't being performed (which is almost a given, assuming we are looking at a problem with such a patchwork problem), then this new tool won't be fully followed and the end result will be a patchwork code base that reaches nightmarish levels where some features are added in this layered approach, but other features are added directly to the code, and even other features are added directly to prior layers.
If the tooling force people to use this in the recommended way then it may have interesting payoffs that are worthwhile. But lacking such an enforced requirement, I don't see applying being beneficial except in the code bases that least benefit from it.
[+] [-] akkartik|6 years ago|reply
[+] [-] mdewing|6 years ago|reply
[+] [-] akkartik|6 years ago|reply
[+] [-] jillesvangurp|6 years ago|reply
These days you see a weaker form of this in the form of annotations in e.g. Python or Java or macros in Rust. Spring actually uses an AOP library that has its roots in AspectJ.
I actually prefer a more recent trend in e.g. Kotlin to use internal DSLs instead of injected magic. The advantage is that it is simpler to debug and part of normal type checks, autocomplete, etc. since technically all you are doing is working with the language instead of augmenting it via generated code.
[+] [-] cryptonector|6 years ago|reply
Instead you have to properly clean tech debt from time to time.
[+] [-] akkartik|6 years ago|reply
I'm not alone in this scepticism. Check out Alan Kay's STEPS project from a few years ago. The goal was to create a complete computing stack in 20k LoC.
My current project has a similar goal: https://github.com/akkartik/mu#readme
You mentioned tech debt. I think I have a more aggressive definition of tech debt than most: any dependency you rely on that you don't understand the internals of is tech debt. It's fine to take on in the short term, but you should be planning to pay it off in the long term. The approach modern software takes, of continually adding dependencies (each of which is itself growing monotonically complex) faster than anyone can internalize them, this approach feels utterly insane. Under this definition, my tiny tool is the least of someone's problems if say they depend on 60 Ruby Gems.
Look at my definition of x86 instructions: http://akkartik.github.io/mu/html/013direct_addressing.cc.ht... If this is tech debt, I'll stay in debtors' prison, because it's safer inside than outside.
[+] [-] canadaduane|6 years ago|reply
Context-Oriented Programming: https://arxiv.org/pdf/1105.0069.pdf
Feature-Oriented Software Development: https://en.wikipedia.org/wiki/Feature-oriented_programming
[+] [-] akkartik|6 years ago|reply
The big difference between my approach and these is the weight-to-thrust ratio. These academic approaches tend to assume people don't want to change the pieces, just combine them. And they create huge tools to make that possible. Why can't people just change the pieces? I've never understood that. I don't need an algebra of program transformations intermediating between me and my code. I'd much rather hold the code directly in my mind. And that keeps the tooling needed really, really small. Just a hundred or so lines of C.
Tooling is a double-edged thing. Adding tooling often encourages increases in codebase scale. Keeping tooling rudimentary can be a good thing by keeping codebases small.
[+] [-] _fbpt|6 years ago|reply
Could "splicing in" code it be implemented using aspect-oriented programming frameworks? https://en.wikipedia.org/wiki/Aspect-oriented_programming I've never used them though.
> There's a new constraint that has no counterpart in current programming practice — remove a feature, and everything before it should build and pass its tests: > build_and_test_until
Looks like added tooling is needed, but seems worthwhile.
[+] [-] daralthus|6 years ago|reply
I believe this is a lot more tractable with Unision's[0] content addressable hashing. It's kinda made for that although more with an eye towards updates to distributed systems.
[0] https://github.com/unisonweb/unison
[+] [-] spoondan|6 years ago|reply
It was a complete nightmare to reason about and work with, even for me, and especially for newcomers. A natural way of trying to understand a system is to look at its entry point. With a system like this, it looks like a skeleton:
Where do we go from here? The next logical step is into `find_and_register_modules` or into the documentation for the module system (just kidding: I was a "senior engineer," I didn't write docs).We are left with a lot of questions. How does anything happen? It's somewhere inside the modules, but which one? Does the order that modules are loaded matter? The answer is maybe. Clearly, there are places where order of operations matter. For example, the main navigation should be ordered. How do we accomplish that? (The answer for my system was to have the interface for navigation extensions specify there was a `readonly weight: Float64` attribute. But then to actually understand why things are in the order they're in, and to get things ordered correctly, you need to look up the values for other navigation items. It's secret coupling.)
More recently, I saw a team at my previous company build a system like this. There were dozens of interfaces and implementation classes and code for composing all of this, and the end result was you couldn't just go somewhere and see what was happening. What we really want to see is:
Do you have to make a change here whenever something gets added? Yes. You do. But the idea that your program needs infinite flexibility in all areas is simply wrong. And, in fact, everything ends up secretly coupled and inscrutable.For this reason, my advice has long been to limit extensibility to specific, narrow use cases (for example, filters in an image editing program). Do not build entire systems around extensibility. You are not building an abstract system, so don't waste your time with unnecessary and obscuring abstractions.
[+] [-] akkartik|6 years ago|reply
I mentioned a couple of rules of thumb for when I create new layers here: https://news.ycombinator.com/item?id=21766557#21767499
Here's the entry point for my current project which uses layers: http://akkartik.github.io/mu/html/000organization.cc.html
You can find a list of layers here: https://akkartik.github.io/mu1 (URL is slightly different; it's a previous prototype. But should suffice for this thread.)
[+] [-] crazypython|6 years ago|reply
[+] [-] akkartik|6 years ago|reply
It's like AOP with user-definable join points.
[+] [-] canadaduane|6 years ago|reply
[+] [-] bruth|6 years ago|reply