As a scientist that ends up working closely with actual professional software engineers... lots of the stuff they do looks like this do me, and I can't for the life of me make sense of why you'd do it.
I have seen a single line of code passed through 4 "interface functions" before it is called that call each other sequentially, and are of course in separate files in separate folders.
It makes reading the code to figure out what it does exhausting, and a few levels in you start to wonder if you're even looking at the right area, and if it will ever get to the part where it actually computes something.
This is actually really bad practice and a very “over eager junior engineer” way of writing software. You’re not off base at all that it seems excessive and confusing. It’s the kind of thing that seems technically complex and maybe even “elegant” (in isolation, when you first write the “interesting” code) at first but becomes a technical nightmare when used in real software that has to grow around and with it. You’re actually more on point in worrying about the understandability and debuggability this introduces.
I spent the better part of two years unfucking some Go software that (among other things) misused channels. The problem with channels is that you rarely actually need them, but can use them for a lot of different things without too much initial difficulty.
I think a good litmus test for proper use of channels is if you answer no to “could this be done with a direct function call instead?” and “can I use a wait group or mutex instead”, and yes to (zooming out a bit to think about what kind of decisions you previously made that led you to think about using channels) “am I really benefitting from concurrency/parallelism enough to justify the technical complexity of debugging concurrent code”.
Sometimes there's a scary lack of understanding and competency where you'd expect to find it.
As an undergrad, I once spent about half an hour peer programming with a computer science PhD - it was enlightening.
He didn't have the slightest understanding of software - calling me out for things like not checking that the size of a (standard library) data structure wasn't negative.
But other times these things are done for a reason; sometimes it's actually sane and sometimes it's just a way to deal with the lunacy of a codebase forged by the madmen who came before you.
The coder spectrum, scientists on one end, software engineers on the other. Only balance can save us.
I have read code used in research papers. The theoretical math usually goes beyond my comprehension, so I always dive into the code to better understand the logic, only to find... it's way worse... unintelligible.
At the end of the day we are used to what we do, and anything different will be foreign to us.
> I can't for the life of me make sense of why you'd do it.
Over-engineering is a common cause: simple solutions can be deceitfully difficult to find. That being said, additional indirection layers are usually justified by the overall architecture, and — assuming they're reasonable — can't always be appreciated locally.
« I'm always delighted by the light touch and stillness of early programming languages. Not much text; a lot gets done. Old programs read like quiet conversations between a well-spoken research worker and a well-studied mechanical colleague, not as a debate with a compiler. Who'd have guessed sophistication bought such noise? » (Dick Gabriel)
On the contrary, and I do agree that software engineers take the abstraction too far when they don’t know better, I don’t hold the code produced by people who aren’t software engineers by profession in particularly high esteem either.
You’re looking at two extremes: the codebase that is spread out too much with too much abstraction, and the codebase with zero abstraction that is basically a means to an end. In both cases they are difficult to work with.
I’ve certainly dealt with enough python, JS and PHP scripts that are basically written with the mindset of ‘fuck this, just give me what I want’, whereas people working in the code day to day need the abstractions to facilitate collaboration and resilience.
> I have seen a single line of code passed through 4 "interface functions"
I once had to deal with a HTTP handler that called `Validate` on interface A which called `Validate` on interface B which called `Validate` on interface C which called `Validate` on interface D which finally did the actual work. There was a lot of profanity that month.
I think something people forget is that computer programming is a craft which must be honed.
A lot of people are introduced to it because computers are such an important part of every discipline, but unfortunately the wealth of mistakes maintaining many a code base occur from those who, quite honestly, simply lack experience.
In the authors case, they don’t explain the simple understanding that every pointer is simply adding a dimension to the data.
int *data; // one dimensional vector
int **data; // two dimensional matrix
int ***data; // three dimensional matrix
Which is one way to interpret things. The problem is that when folks learn computer programming using Python (or another high level language), it’s like using power tools bought from the hardware store compared to some ancient, Japanese wood working technique. The latter takes time to understand and perfect.
Ignoring inexperience/incompetence as a reason (which, admittedly, is a likely root cause) domain fuzziness is often a good explanation here. If you aren't extremely familiar with a domain and know the shape of solution you need a-priori all those levels of indirection allow you to keep lots of work "online" while (replacing, refactoring, experimenting) with a particular layer. The intent should be to "find" the right shape with all the indirection in place and then rewrite with a single correct shape without all the indirection. Of course, the rewrite never actually happens =)
Contrary to the "over-engineering" claims, I'll put this explanation up for consideration: it's a result of fighting the system without understanding the details. Over-engineering absolutely exists and can look just like this, but I think it's mostly a lack of thought instead of too much bad thinking.
You see the same thing with e.g. Java programmers adding `try { } catch (Exception e) { log(e) }` until it shuts up about checked exceptions (and not realizing how many other things they also caught, like thread interrupts).
It's a common result of "I don't get it but it tells me it's wrong, so I'll change random things until it works". Getting engs in this state to realize that they're wasting far more time and energy not-knowing something than it would take to learn it in depth has, so far, been my most successful route in dragging people into the light.
(Not surprisingly, this is one of my biggest worries about LLM-heavy programmers. LLMs can be useful when you know what you're doing, but I keep seeing them stand in the way of learning if someone isn't already motivated to do so, because you can keep not-understanding for longer. That's a blessing for non-programmers and a curse for any programmer who has to work with them.)
This is a popular pattern in apps or other "frameworky" code, especially in C++.
I can think of at least two open source C++ apps I used where every time I checked in the IRC channel, the main dev wasn't talking about new app features, but about how to adopt even newer cooler C++ abstractions in the existing code. With the general result that if you want to know how some algorithm backing a feature works, you can't find it.
There's an element of what you might call "taste" in choosing abstractions in software.
Like all matters of taste, there are at least two things to keep in mind:
(1) You can't develop good taste until you have some experience. It's hard to learn software abstractions, and we want engineers to learn about and practice them. Mistakes are crucial to learning, so we should expect some number of abstraction mistakes from even the smartest junior engineers.
(2) Just because something is ugly to a non-expert doesn't mean it's necessarily bad. Bebop, for example, has less mass appeal than bubblegum pop. One of the things that makes bebop impressive to musicians, is the amount of technical and musical skill it takes to play it. But if you're not a musician those virtues may be lost on you whereas the frenetic noise is very apparent.
One of the things Google does better than other huge tech companies (IMO) is demonstrate good taste for abstractions. Those abstractions are often not obvious.
[Obviously the bebop comparison
breaks down, and showing off technical skills isn't a virtue in software. But there are other virtues that are more apparent to experts, such as maintainability, library reviews, integration with existing tooling or practices etc.]
As a software engineer, this is something I get onto with my team. There is a such thing as too much abstraction and indirection. Abstraction should serve a purpose; don’t create an interface until you have more than one concrete implementation (or plan to within a PR or two). Premature abstraction is a type of premature optimization, just in code structure instead of execution.
> I have seen a single line of code passed through 4 "interface functions" before it is called that call each other sequentially, and are of course in separate files in separate folders.
Procrustination. n. the act of writing an infinite regression of trivial nested 'helper' functions because you're not sure how to actually attack the problem you're trying to solve.
I worked with a guy that did this.
int CalcNumber(int input) { int val = GetNumberFromInput(input); return val; }
int GetNumberFromInput(int num) { int calcedNumber = 0; for (int i=0; i<1; i++) calcedNumber += NumberPart(i, num); return calcedNumber; }
int NumberPart(int part, int seed) { return actuallyGenerateNumberPart(part, seed); }
int actuallyGenerateNumberPart(int part, int seed) {
// todo
return 0;
}
this is the classic over abstraction problem so that you can change things behind an interface at some point down the line if you ever need to while being totally opaque to any consuming code.
A lot of languages force you to start with this from day one, unless you want to go refactor everything to use an interface later on, so people just do it even when there will literally never be a reason to (and for testability, sometimes).
The cool thing about Go is the interface system is inverted kind of like duck-typing, so if you write purely idiomatic Go, then the thing receiving an arg in a function call is the one specifying the interface it must meet, rather than the implementing code having to declare every interface that some implementation meets.
People screw this up a lot though, especially if they came from Java/C# backgrounds.
It happens so that we don't have "a single 15,000 line file that had been worked on for a decade". We don't have the luxury of asking the GitHub team and John Carmack to fix our code when we are forced to show it to the stakeholders.
I mean we are in “midwit meme” territory here. 4 levels of indirection look fine to an idiot. A “midwit” hates them and cleans them up. But a very seasoned engineer, thinking about the entire system, will happily have 4 layers of indirection.
Like anyone using a HashSet in Rust is already doing four layers of indirection: rusts HashSet is actually wrapping a hashbrown::HashSet. That set is wrapping a HashTable and the HashTable is wrapping an inner Table.
If you’re having trouble comprehending such code then a good IDE (or vim) that can navigate the code on a keypress should help.
There is a Go developer that defends using channels four times like in this blog, he also made some great medium posts in which he details real use cases.
His name is Christopher, if you want to find more about him, just Google “chris chan 4chan”.
The irony of the comments in this thread decrying this as a classic example of too much abstraction is that the reason you rarely see three-star variables in C is that, while it's common to have long chains of pointers (and in languages where ~everything is a pointer~ most things are pointers by default, like Python, Java, or JavaScript, you definitely have pointer chains that are much longer than 4!), what is uncommon (but not nonexistent) is the need to manipulate more than two levels at once — you'll have one, maybe two, levels of pointers, but anything deeper is probably going to be hidden in a struct whose internals you don't have to care about (i.e. abstracted away). The same argument applies to channels, which are roughly to concurrent code what pointers are to sequential code: the fatal flaw here, if there is one, is not too much abstraction, but perhaps too little abstraction.
But then, I don't know the codebase — maybe a `chan chan` was exactly the right abstraction for what they were trying to write :) Certainly in highly-concurrent languages like Erlang, it is very standard practice to send a PID to another PID, for example to identify the process that should receive the reply to the message.
The "chan chan Value" or "chan struct{resp chan Value}" is actually a pattern I've used in a very specific situation. It's a situation where a message bus probably could have been used instead, but then you'd have a message bus to handle...
channel of channel is a normal pattern, although it more commonly looks like a channel of struct vals, and the struct type has a field that's a channel. it's lets you send requests that are completed, whose values can be awaited on out of order, to avoid head of line blocking. something like `type request struct { params, reply chan response }`, where you send those requests on a channel, they get dispatched to some worker, and then the worker does the work and puts the result on the reply chan. But two is as high as I've ever found useful, I don't think I've ever seen a use case for a chan chan chan.
Here's a counterpoint blog using a chan that sends a chan in order to implement a dynamic dispatch mechanism. This is in Limbo not Go. But same concepts. Maybe the complexity proves the point. https://ipn.caerwyn.com/2007/07/lab-78-dynamic-dispatch.html...
The inner channel represents a future, while the outer channel is fed by threadpool workers to a finalization reader. This way the ordering doesn't get corrupted by parallelism
That’s some pretty Go code, I like it. Just out of interest, why have a channel for subscribe/unsubscribe? Did you want to avoid a mutex on the subscriber map?
The closest thing to this I've seen in actual code is BlockingQueue<BlockingQueue<List<T>>>, which was used to wait for an element from multiple blocking queues simultaneously, where each element is a list of messages.
* Currently sendChanChan() takes 2 params: a 4chan and a 3chan. I think it can be simplified by just taking a 3chan. The send on the 4chan can be done before calling sendChanChan().
* Why does the code have 2 levels of loops in receiveChanChanChan()? I think a single loop would be fine. Especially since there are 0 loops in the place where receiveChanChanChan() is called. So currently, a single goroutine performs N receives all the values from the 4chan, but we have N goroutines to receive values for each 3chan (so each goroutine will perform on average 1 receive). That's inconsistent.
In LabVIEW code we sometimes do this as a means of receiving asynchronous response data; instead of dumping responses in a queue, the message you pass includes a callback event channel, which you can use to notify exactly the same caller that the operation is done, and what the results are. A little wasteful on memory, but since theyre single use and are closed upon responding, the ergonomics are worth it
That image with the pointer types is also derived from a frequently-posted-on-4chan picture from the youtuber "Laurie Wired" aka "the queen of /g/": https://0x0.st/XvfB.jpg
This is either parody or the making of someone who’s never done real work.
Source: if you do this stuff or even dare to name another microservice after a Harry Potter character when 11.5 million people rely on it, I’m going to dedicate the rest of my life to having you fired from everywhere.
[+] [-] UniverseHacker|1 year ago|reply
I have seen a single line of code passed through 4 "interface functions" before it is called that call each other sequentially, and are of course in separate files in separate folders.
It makes reading the code to figure out what it does exhausting, and a few levels in you start to wonder if you're even looking at the right area, and if it will ever get to the part where it actually computes something.
[+] [-] weitendorf|1 year ago|reply
I spent the better part of two years unfucking some Go software that (among other things) misused channels. The problem with channels is that you rarely actually need them, but can use them for a lot of different things without too much initial difficulty.
I think a good litmus test for proper use of channels is if you answer no to “could this be done with a direct function call instead?” and “can I use a wait group or mutex instead”, and yes to (zooming out a bit to think about what kind of decisions you previously made that led you to think about using channels) “am I really benefitting from concurrency/parallelism enough to justify the technical complexity of debugging concurrent code”.
[+] [-] yarg|1 year ago|reply
Sometimes there's a scary lack of understanding and competency where you'd expect to find it.
As an undergrad, I once spent about half an hour peer programming with a computer science PhD - it was enlightening.
He didn't have the slightest understanding of software - calling me out for things like not checking that the size of a (standard library) data structure wasn't negative.
But other times these things are done for a reason; sometimes it's actually sane and sometimes it's just a way to deal with the lunacy of a codebase forged by the madmen who came before you.
[+] [-] eddd-ddde|1 year ago|reply
I have read code used in research papers. The theoretical math usually goes beyond my comprehension, so I always dive into the code to better understand the logic, only to find... it's way worse... unintelligible.
At the end of the day we are used to what we do, and anything different will be foreign to us.
[+] [-] mbivert|1 year ago|reply
Over-engineering is a common cause: simple solutions can be deceitfully difficult to find. That being said, additional indirection layers are usually justified by the overall architecture, and — assuming they're reasonable — can't always be appreciated locally.
[+] [-] ljm|1 year ago|reply
You’re looking at two extremes: the codebase that is spread out too much with too much abstraction, and the codebase with zero abstraction that is basically a means to an end. In both cases they are difficult to work with.
I’ve certainly dealt with enough python, JS and PHP scripts that are basically written with the mindset of ‘fuck this, just give me what I want’, whereas people working in the code day to day need the abstractions to facilitate collaboration and resilience.
[+] [-] zimpenfish|1 year ago|reply
I once had to deal with a HTTP handler that called `Validate` on interface A which called `Validate` on interface B which called `Validate` on interface C which called `Validate` on interface D which finally did the actual work. There was a lot of profanity that month.
[+] [-] jayd16|1 year ago|reply
[+] [-] stroupwaffle|1 year ago|reply
A lot of people are introduced to it because computers are such an important part of every discipline, but unfortunately the wealth of mistakes maintaining many a code base occur from those who, quite honestly, simply lack experience.
In the authors case, they don’t explain the simple understanding that every pointer is simply adding a dimension to the data.
int *data; // one dimensional vector
int **data; // two dimensional matrix
int ***data; // three dimensional matrix
Which is one way to interpret things. The problem is that when folks learn computer programming using Python (or another high level language), it’s like using power tools bought from the hardware store compared to some ancient, Japanese wood working technique. The latter takes time to understand and perfect.
Ten thousand hours, so I’ve heard ;)
[+] [-] Misdicorl|1 year ago|reply
[+] [-] Groxx|1 year ago|reply
You see the same thing with e.g. Java programmers adding `try { } catch (Exception e) { log(e) }` until it shuts up about checked exceptions (and not realizing how many other things they also caught, like thread interrupts).
It's a common result of "I don't get it but it tells me it's wrong, so I'll change random things until it works". Getting engs in this state to realize that they're wasting far more time and energy not-knowing something than it would take to learn it in depth has, so far, been my most successful route in dragging people into the light.
(Not surprisingly, this is one of my biggest worries about LLM-heavy programmers. LLMs can be useful when you know what you're doing, but I keep seeing them stand in the way of learning if someone isn't already motivated to do so, because you can keep not-understanding for longer. That's a blessing for non-programmers and a curse for any programmer who has to work with them.)
[+] [-] astrange|1 year ago|reply
I can think of at least two open source C++ apps I used where every time I checked in the IRC channel, the main dev wasn't talking about new app features, but about how to adopt even newer cooler C++ abstractions in the existing code. With the general result that if you want to know how some algorithm backing a feature works, you can't find it.
[+] [-] ants_everywhere|1 year ago|reply
Like all matters of taste, there are at least two things to keep in mind:
(1) You can't develop good taste until you have some experience. It's hard to learn software abstractions, and we want engineers to learn about and practice them. Mistakes are crucial to learning, so we should expect some number of abstraction mistakes from even the smartest junior engineers.
(2) Just because something is ugly to a non-expert doesn't mean it's necessarily bad. Bebop, for example, has less mass appeal than bubblegum pop. One of the things that makes bebop impressive to musicians, is the amount of technical and musical skill it takes to play it. But if you're not a musician those virtues may be lost on you whereas the frenetic noise is very apparent.
One of the things Google does better than other huge tech companies (IMO) is demonstrate good taste for abstractions. Those abstractions are often not obvious.
[Obviously the bebop comparison breaks down, and showing off technical skills isn't a virtue in software. But there are other virtues that are more apparent to experts, such as maintainability, library reviews, integration with existing tooling or practices etc.]
[+] [-] withinboredom|1 year ago|reply
[+] [-] taneq|1 year ago|reply
Procrustination. n. the act of writing an infinite regression of trivial nested 'helper' functions because you're not sure how to actually attack the problem you're trying to solve.
I worked with a guy that did this.
[+] [-] jdc0589|1 year ago|reply
A lot of languages force you to start with this from day one, unless you want to go refactor everything to use an interface later on, so people just do it even when there will literally never be a reason to (and for testability, sometimes).
The cool thing about Go is the interface system is inverted kind of like duck-typing, so if you write purely idiomatic Go, then the thing receiving an arg in a function call is the one specifying the interface it must meet, rather than the implementing code having to declare every interface that some implementation meets.
People screw this up a lot though, especially if they came from Java/C# backgrounds.
[+] [-] LudwigNagasena|1 year ago|reply
[+] [-] lowbloodsugar|1 year ago|reply
Like anyone using a HashSet in Rust is already doing four layers of indirection: rusts HashSet is actually wrapping a hashbrown::HashSet. That set is wrapping a HashTable and the HashTable is wrapping an inner Table.
If you’re having trouble comprehending such code then a good IDE (or vim) that can navigate the code on a keypress should help.
[+] [-] stonemetal12|1 year ago|reply
https://en.wikipedia.org/wiki/Law_of_Demeter
[+] [-] theideaofcoffee|1 year ago|reply
That meme in the first few paragraphs made me actual lol as a recovering C programmer.
That is all.
I love seeing odd contortions of languages like this, and C has opportunity aplenty, interesting to see it in Go.
[+] [-] Lucasoato|1 year ago|reply
[+] [-] zachmu|1 year ago|reply
[+] [-] noman-land|1 year ago|reply
[+] [-] nomorewords|1 year ago|reply
[+] [-] ge96|1 year ago|reply
[+] [-] HideousKojima|1 year ago|reply
[+] [-] tomcam|1 year ago|reply
If you're a Go programmer, this is about a guy who fucked his mom and has nothing to do with Go programming.
[+] [-] spookie|1 year ago|reply
We are still living in those days.
[+] [-] Twey|1 year ago|reply
But then, I don't know the codebase — maybe a `chan chan` was exactly the right abstraction for what they were trying to write :) Certainly in highly-concurrent languages like Erlang, it is very standard practice to send a PID to another PID, for example to identify the process that should receive the reply to the message.
[+] [-] lemursage|1 year ago|reply
[+] [-] kardianos|1 year ago|reply
[+] [-] zemo|1 year ago|reply
[+] [-] caerwy|1 year ago|reply
[+] [-] jhoechtl|1 year ago|reply
[+] [-] malkosta|1 year ago|reply
[+] [-] __s|1 year ago|reply
The inner channel represents a future, while the outer channel is fed by threadpool workers to a finalization reader. This way the ordering doesn't get corrupted by parallelism
[+] [-] 9cb14c1ec0|1 year ago|reply
[+] [-] twp|1 year ago|reply
https://github.com/twpayne/go-pubsub/blob/master/pubsub.go
[+] [-] cedws|1 year ago|reply
[+] [-] PhilipRoman|1 year ago|reply
[+] [-] Thorrez|1 year ago|reply
* Currently sendChanChan() takes 2 params: a 4chan and a 3chan. I think it can be simplified by just taking a 3chan. The send on the 4chan can be done before calling sendChanChan().
* Why does the code have 2 levels of loops in receiveChanChanChan()? I think a single loop would be fine. Especially since there are 0 loops in the place where receiveChanChanChan() is called. So currently, a single goroutine performs N receives all the values from the 4chan, but we have N goroutines to receive values for each 3chan (so each goroutine will perform on average 1 receive). That's inconsistent.
[+] [-] david2ndaccount|1 year ago|reply
[+] [-] __s|1 year ago|reply
[+] [-] ijustlovemath|1 year ago|reply
[+] [-] ranger_danger|1 year ago|reply
[+] [-] ianpenney|1 year ago|reply
Source: if you do this stuff or even dare to name another microservice after a Harry Potter character when 11.5 million people rely on it, I’m going to dedicate the rest of my life to having you fired from everywhere.
[+] [-] enneff|1 year ago|reply
[+] [-] fsndz|1 year ago|reply