> Yes, I had failed to see the proper solution: a class generator — so that I didn't have to manually copy code again
No, please don't. This is jumping from the frying pan into the fire. If you think abstract base classes can be clever and hard to understand, code generators can be even more so. In addition, because code generators are a one way conversion, and the generated code evolves independently and the code generator itself is evolving independently, you end up with a bunch of things that are subtly different yet somewhat related in a way that you will need an advanced degree in evolutionary biology to understand.
As a blog post with the title "Don't be clever" ending up with the author's answer being an even clever-er solution, than talking about the hidden fight against complexity in software, was an odd ending note.
My understanding is that in this case, the code generator is merely a boilerplate generator that isn’t meant to keep the code in sync, but just do the initial copy/paste work.
I think code generators are a perfectly acceptable solution in cases like this, when the starting point all looks the same, and it needs to diverge from there. Especially if the generation logic is fairly straightforward.
There are a lot of IDEs that have boilerplate generators out of the box.
So much about programming in a larger sense is just getting abstractions correct. Too loose and they don't standardize/remove enough boilerplate. Too strict and they break or multiply when changes are needed.
I am curious, since my own backend experience is limited (obviously there will be many opinions on this) but it seems to me like his mistake was using the inheritance of classes.
If he had simply had a CRUD layer - that only standardized the actions of creating, removing, updating, or deleting a record (or records) from a database. This would accomplish standardizing the payload, as well as possibly doing an RBAC/Authorization check.
He could have just written individual controllers that invoke that layer with custom business logic. He would even have more space to add comments indicating why the business logic is implemented this way. When someone needed to make a change, they could just go straight to the specific controller.
Putting the impracticality of having to override specific methods aside, isn't it also a waste to instantiate a class full of methods that essentially do the same thing when you can just point to existing ones? Well I guess it is pointing to existing ones via inheritance, but it seems harder to reason about. But this is coming from someone who doesn't write traditional object oriented code very often.
Genuinely curious to hear from backend people about this, it's not my area of expertise but I've learned a lot this year.
Generating code is fine, if the generated code strictly never evolves independently of what it is generated from. For instance generating libraries from .proto files (or other declarative schema definition solutions) works really well. If the schema changes, you throw away the old generated code and generate brand new code, no problem.
But if you want to make even a single tiny modification to one of the generated files, you're busted, you need a different solution.
The funny thing is the last thing I did code generation for just generated classes which inherited abstract base classes. Then we had to inherit the generated code to extend it. It massively increased the complexity.
Then someone came along and invented partial classes in C# which made this problem go away. Well it would if anyone wanted to do all the maintenance legwork which they didn't so half of it's a 5 class inheritance tree, some of it's partial classes and someone got really fed up with this shit and just arbitrarily stuffed Dapper in there one afternoon.
I think one lesson newer programmers fail is that sometimes copy/paste is not only OK, but preferred.
One time this is good is when helping others get up to speed. Other people can copy/paste your examples and be functioning right now.
But if you have a function that you put in to eliminate some duplication, they have to understand this new abstraction layer before they can start working.
One good thing with a code generation approach is that we can actually inspect into the generated code and see what's going to happen in a relatively easier way. And it's usually working nicely with static analysis. For abstract base class or whatever, it's kind of hard to do.
Of course, code generator (or even compiler) is harder to maintain, especially when you want the output to be human readable. So it's always about trade-off. Think about how many lines of code it's going to generate. If it's an order of 100k then it's usually worth the cost. 10k might be good to go. 1k, probably not.
Code generators can work if you .gitignore the target folders, so everything generated must be used as-is or augmented by a separate handwritten file like a subclass. And to integrate them tightly into your build process so the generator tool runs transparently.
But even then they're a nightmare because your errors don't match your source files.
"generate and edit" is suicide. "generate and treat the outputs as intermediate compilation objects" is merely an incredibly painful tool of last resort.
Yeah surely a better approach is... rather than inheriting from an all powerful class, just compose controllers out of modules. Some used in almost every controller. Some pretty specific to one or two edge cases.
Like the problem was having an all-or-nothing inheritance model for shared functionality, which also ended up very long and intimidating.
I also don't really understand why emerging edge cases caused him to complicate his 90%-perfect parent class instead of just... overriding specific functions in the children. (The specifics of these overrides will probably make you want to refactor your parent class so that the overrides have clean joints but you shouldn't just dump the new child-specific logic into your parent)
IMHO, I don’t think there are enough code generators, however, a good code generator’s output should be a library, not editable source. I’ve had a lot of good experience using IDLs that could generate libraries for multiple languages that other code could be used with (e.g. define your REST url scheme and supported operations and a base class that gets generated that allows filling in logic).
At least in Java world, annotation seemed to have taken over, but the mixture of code and interface definition kicks you into the language and allows too much flexibility that really make it no different than writing registration code. Having a nice declarative language without business logic sneaking in.
This whole article is about whether to do or not, and I dig the lesson here. The author built on monstrosity they hate, and they propose perhaps creating another. How bold & brave!
Of course lots and lots of people love telling us why not to do things. This is the cosmic-brain point of Steve Yegge's "Notes from the Mystery Machine Bus", of what we let guide us. Is it trying and hope and doing? When and where do we allow reservation in? https://gist.github.com/cornchz/3313150
I use code generators all the time and it has saved me a huge amount of work. I routinely generate 80% of the code needed to implement typical business applications.
Having said that, code generators are not all the same. Some are awesome, others are downright awful. There is an art to writing good code generators. Not surprising really. The same can be said for most other software.
Is that what he's talking about? I thought he was talking about some pattern using first class classes rather than ever programmatically touching source?
"Write code that writes code" -- Tip 29, The Pragmatic Programmer[1]
I think we need to be a little more nuanced, here.
Don't be clever in your code generation, to be sure. But let's not act like you need advanced degrees for all meta-programming.
What do you think phased compilers do? They generally transform the AST produced by your code into expanded versions or rewrites of that AST that are more amenable to machine interpretation or transformation into assembly.
In a very real sense, nearly all production code goes through some kind of codegen process. It isn't clever, it isn't new, it generally isn't complicated (if this then that else that else ... when you get down to it). It is repeatable and reproducible and reliable enough method that the whole economic system of software is built on top of it. Most of the problems come with recursive applications of codegen and strange interactions between different subsystems of codegen.
Here's a tip: Don't do recursive codegen in your user land metaprogramming.
User code is _easier_, simply because it isn't as reusable as the language primitives themselves. But if you understand a problem well enough to essentially reduce it to copy, paste, and replace one or two items, codegen is superior because a bug fix during maintenance need only happen in a single place.
My primary language recently added generic derivation to the language itself (as opposed to in macro libraries) and it is totally worth it as I almost never have to write a ser/de by hand. I never have to test the ser/de round trip. I don't have to pepper my code with second-class annotation primitives or keywords. I never misspell `asJson`, etc.
Not everything should be done with codegen in user land code (code outside the compiler and language tooling). But if I see thousands of lines of json/avro/protobuf/test example copypasta, that's a code smell that needs to be addressed or it will lead to production bugs and take hours instead of minutes to fix.
And when you write these things, you come to find out that they are just like any other data transformation task. They turn `A`s into `B`s, systematically.
My mentor, when I became a professional coder, taught me that if you do something more than three times, its probably worth taking the time to generalize it and do it once before you push the changes into the version repository. It's been 18 years now since I began my career, and that simple principle has rarely steered me wrong across probably a dozen languages that I've written production code in.
I have swung both ways and I think I now settle somewhere near "boring is good" and "repetition is harmless (compared to the astronomic costs of wrong abstraction)".
Especially repetition seems to be hated with the might of a thousand suns and while I get it, because I myself hated it, I now can see the beauty of it.
What is currently a superficial repetition - a bunch of endpoint handlers, some forms - will often turn out to be similar-looking instances of completely different problems. Repetition is quite often deeper than just "code looks the same".
I've seen my share of cases where the repetition was only because the original author didn't know how to abstract the problem say, using a simple struct. Or even easier, a simple function.
I've sometimes taken on factoring the repeated code using common abstractions, and when that was done it often turned out that there was much more to factor out. And that the code quality could massively be improved because there are often some tricky situations that can only be properly handled when everything goes through a single place with all the necessary context.
As an example, we had a config file with an ini-like structure that was parsed by some central parser code. So far, so good, but after the parse, the resulting key-value pairs were processed like this:
if (string_is_equal(config_key, "foo_setting"))
config->foo = parse_number(config_value);
else if (string_is_equal(config_key, "bar_setting"))
config->bar = parse_string(config_value);
Repeating this pattern dozens of times is decidedly not cool. It lacks expression of the common structure, it imposes and prevents code optimizations and fixes (such as error handling).
Sometimes I was told that abstraction isn't worth the hassle and boring and repetitive is good. By factoring the code allegedly we risk breaking it and make it harder to fix. Those people didn't see how broken and incomplete the code is precisely because there is a lack of abstraction. We can't even know that there isn't some "quux_setting" with broken handling code because of course there aren't proper tests for each config setting or combination of settings.
I hate repetition because it's nearly always laziness - it takes less thought/time to copy and paste a few lines of code than it does to factor them out into a reusable function and decide where to put it (and with what name). I'm taking about scenarios where the business logic needs to be exactly the same in both cases, there just happens to multiple ways to reach that point.
On the other hand I also hate having to deal with shared functions that have been repeatedly adapted and extended to be able to deal with all the various edge cases to the point they have 20 cryptically named parameters and no reasonable way of guessing what the output should be for given set of inputs.
But I'd still say more of my time is used up dealing with problems caused by lazy copying & pasting than by shared functions becoming overcomplicated or buried under excessive layers of abstraction.
There are also valid use cases for preferring boring repetition.
Most of the time when I do any kind of coding work, it's as a single person project for companies/orgs with absolutely zero tech people. For instance, the last project I took on was for a bra shop of 3 people. No IT. Nobody on staff who has any idea how anything tech works - the closest is the person who taught herself the basics of using Shopify + advertising platforms.
Which means that in the future if they need somebody to look at or update anything I've done, I can't assume they're going to have access to a well-trained, talented person - it's equally likely they'd go "hey so and so's kid does 'computer stuff' let's see if they can fix it for us." Knowing that the next person who looks at my work might have very basic level skills leads me to prefer being repetitive and 'simple' over the more efficient solutions I can think of.
It's the difference between a regular Wikipedia article and the Simple English Wikipedia - if I'm writing something that is likely to be maintained by beginners or apprentices, making it easy to understand and work with matters a lot more.
I have what I call the "10 second rule". The rule is that an experienced programmer (ie. someone who has written the type of code your codebase is written in, whether Python, JS, etc) should be able to look at a code snippet, any code snippet in your code, and figure out what it does in about 10 seconds. There are obviously exceptions to this where complexity can't be avoided but overall I found the tradeoff is worth it especially on a team of multiple devs.
I currently work for a client that has a very very large react native app that is pretty hard to wrap your head around. I can't tell you how many millions of dollars they have wasted in dev hours because it takes a dev way to long to even figure out what the code is doing before they can start making changes.
I thought about this a while and concluded that the right question is whether the repetitions are intended to do the same thing. If they are, deduplicate them. If not, leave them alone to evolve independently.
I feel the same way. Repetition is fine when you're working within a solid, well established framework. If you're writing 100s of GET endpoints, and most of them requires you to prefix your endpoints withs some @GET and @RequriesToken decorators, so be it.
If there are other forms of repetition within the user-code, there are ways of dealing with it. But writing a super class to solve superficial code repetition is most often the wrong way to go.
I've come to the same conclusion as you. Code is typically read more times than it is written/modified, so it is better to optimise for the common case, i.e. code that is tedious to write but simple to read and follow.
At a consulting gig I did, we were bootstrapping a brand new python engineering team for a new line of products. We chose the frameworks, set standards via decision records, wrote a template service that you would copy paste, and build on top. Cross cutting concerns were pulled out into a library that all of these services installed. Most things were standardized, all APIs felt like they were written by a single person, and yet little was abstracted. It was simple and beautiful. An engineer would come onboard, look at one repo and be able to navigate all service repos going forward. You could see and feel the repetition, aside for the business logic, it was like reading the same book over and over again. The team was delivering at a crazy velocity.
Then a year or so later the client brings us back in to help diagnose and fix why their delivery dates continue slipping and number of bugs in production is increasing. Finds out a new, very vocal hire, came in, saw the repetition and decided to build a "service service". A service, into which you would pass a single python file of pure business logic encoded as a b64 string, with some decorators, and it would set up the endpoints, serialization, DB, etc automagically.
If you feel confident refactoring some code then you have been able to read and understand it easily. If you can read and understand it easily then it’s good code, leave it alone.
There are cases where doing the "same" work twice is sufficient, preferred even...
Somehow some colleagues found it absurd to parse some code twice, but don't see the insanity of parsing code, getting an AST, serializing, sending it, and then deserializing correctly.
Never mind the fact that the serialized AST is 250x larger, and deserializing arbitrary constructs without codegen is a whole different can of worms, and especially harder when the code is running in 2 different programming languages.
From why you've said it sounds like maybe the issue was really how they factored out the common code (Python loading base64 encoded Python is clearly insane) rather than the fact that they did.
It's difficult to draw conclusions from these sorts of things though because the answer is "sometimes you should copy and paste a bit, sometimes you shouldn't" and you just have to have skill and taste to know the difference.
I really dislike when someone considers my code "clever" because it always means they don't approve of it or think it's too confusing. No, I wasn't trying to be clever, but to create the most appropriate solution I could imagine.
The author's problem isn't being overly clever, but that they had applied an inappropriate yet imtellectually-satisfying programming pattern that is notorious for being difficult to make exceptions for. The quote about the correct solution being a "class generator" shows that the author misdiagnosed their problem. If you continue down the path of making everything a class, you're gonna have a worse one than if you made most of your functionality into compostable functions, reserving classes for things that truly deserve it.
So many issues in programming would be diminished if programmers toned down the thinginess of their code, focusing more on procedures and data shapes.
Making things worse, I suspect there's at least two (and probably more) definitions for 'clever' that are thrown around and the intent half the time is that one definition is masquerading as the other.
One plausible definition for clever code is that it's code that only works because of a non-obvious dependency on some other fact being true in the code base or outside of the code base. Once the fact is no longer true the code will be broken or subtly broken (or it will break or subtly break other code). "Ah ha, very clever, but we shouldn't do this."
Another plausible definition is approximately, "I don't understand the feature or pattern you're using, and I don't want to understand it or form a coherent argument against it."
> So many issues in programming would be diminished if programmers toned down the thinginess of their code, focusing more on procedures and data shapes.
Exactly my reaction too. The code is complicated and requires boilerplate because it tries to adhere to a pattern that doesn't need to exist.
Whenever something becomes entangled and complex, decompose it into pieces first (functions and data) and then put it together again, piece by piece. The right abstractions (if needed) emerge naturally from simple code.
Larry Wall said something like "if you have a complex problem, the complexity will come up somewhere -- either you have a complex language and a simple program, or a simple language and a complex problem. The complexity can't be wished away." Being Larry, he said it far better but I can't find the exact quote.
All programming is making models: representing a chaotic and changing analog world in bits, data structures, and algorithms. For every model there's something that breaks that model. Attempting to incorporate every exception into your model leads you to the path of tuples + a workflow engine, and at that point all you've done is build a different database and programming language and move the complexity further up the stack.
As requirements change, the model you built will no longer prove appropriate because now you're modeling different situations. There's an art to building a model that doesn't box you in to the point where the first change request doesn't break your model. But at some point, every model is going to be inadequate.
A senior programmer brings to this: knowing the common ways things change, the costs of changing different types of model, and the understanding that there's no such thing as Perfect so don't proclaim your framework to be such.
So many times I've had people regurgitate this blog post[1], or KISS, or some other general piece of advice to "keep things simple" and it just means they want to influence the shell game of managing complexity so the complexity is someone else's problem--and they can stop thinking about it. I've come to really hate when someone pulls this out as some general principle, because it so frequently means they want to push complexity from an implementation (which maybe has a hope of hiding some of the complexity) out into callers or configurations or some other kind of leaf, where it becomes a breeding ground for manually synchronized configurations, copy-and-paste code promotion (not reuse, which is bad enough, but the pattern where some thing that you're forcing people to manually compile or explode has to be copy-and-modified through an artifact promotion.
Frankly, KISS and its ilk have become more often "things uninsightful people say to make them sound clever and vaguely iconoclastic" than something that can be actually useful as guidance: because people think they remove nuance rather than adding it (KISS is great if you honor its nuances like "and no simpler..." and "considered as a whole"; but instead it's a thought-terminating cliché). I've heard people use these sound bites to justify, e.g., the entire notion of automated testing (too complicated), or "infrastructure as code"--though I also dislike this term and favor "software-defined infrastructure", though maybe if it became popular, objections that software-defined infrastructure isn't really software would become as common as objections that infrastructure-as-code isn't really code.
-----
[1] I don't mean this literal blog post, I mean the same kinds of "don't be too clever" platitudes.
Very well put. Effective businesses are good at communicating and sharing these models so everyone is able to reason about them, and if they’re lucky anticipate future articulation points.
I think these stories and aversion to "clever" code end up hurting programmers overall. I wish we spent more time teaching how to choose the correct time and place to be clever instead. Searching for ways to make things better instead of mindlessly copy pasting how it was done by the person before me is how I learned my most valuable skills.
It doesn't help that we don't have a good definition for what 'clever' means when referring to code.
This article talks about a time someone made an abstract class and then took DRY a little too far. If this is an example of 'being clever' then anyone using map instead of writing a for-loop would have to be at gaussian levels of cleverness.
[Also a class generator is implied to be less clever than an abstract class ... I'm not sure I can accept the usage of 'clever' in the article as meaningful.]
A new pattern may seem "clever" or "difficult" if you haven't encountered it before; but if you spend time with it and work with it, it can look like the most natural thing in the world.
I am not sure if teaching this skill would help or is possible.
But I do agree with the direction of your thought. The reading material at least needs to be inherently balanced rather than writing itself getting balanced out by 50% of articles talking about "Using proxies to prevent null pointer errors"(guilty of writing this) and the other half being "Maybe proxies are a bad idea".
Code was devised to address a specific problem (DRY), which it did quite elegantly. Then, the situation evolved, rendering the solution inappropriate. When a controller starts to deviate from the standard approach, it's best to refrain from inheriting from the base controller and instead create custom ones. Once all controllers operate independently, the base class can be safely removed.
The key lesson here is not to avoid cleverness but to be aware of when the initial problem becomes obsolete.
What about making all of the "clever" things composable and optional, instead of tightly coupled? That way when you get a special case you can just make that part a special handler. You neither need to throw out the whole "clever" thing nor make the "clever" thing "more clever".
My first lesson around this was in manufacturing. It has stuck with me much more strongly than any sort of software industry jargon. It is an incredibly simple phrase too: "Value-add".
I just stay focused on that item. "Is my proposal/refactor/shiny thing going to make a customer want to spend more money, sign additional/longer contracts, or somehow smooth out support & operations?"
If I get an "ehhh im not sure" or a more honest "no", then I'd be inclined to shitcan the proposal unless a very clear argument can be put forth for why this refactor will somehow eventually result in downstream value-add.
Arguments along the axis of "because its the right thing to do", "this is the technical best practice" or "it's cool to look at" are rejected by default unless additional supporting evidence can be brought forth. Proposals that bring additional vendors & operational liabilities without also bringing obvious value are to be considered malicious suggestions.
This probably sounds draconian as hell to some of HN, but if you are working with truly complex software, you cannot be playing games in traffic, especially if you are already in production for many customers and under half-decade-long contracts.
The time to be clever is on your side projects. Do not conflate the thing that results in your paycheck with entertainment. It might seem like there could be some happy co-existence, but I've personally never seen it work out.
I love best practices, and I'm even a fan of cargo cult development to a certain degree, staying with what's tested, reducing novelty, making things repeatable and declarative, choosing a million line framework over 1000 I'm house lines.... But it's not worth trying to maintain your own Ansible in your own custom language with a GUI in your own GUI toolkit, just to avoid some ugly hacks and workarounds you'd need if you just used the popular tool.
Apparently the author hasn't learned the most important lesson: Think thrice before you use OO. And if you do, keep your hierarchies flat and simple. Prefer composition over inheritance. Make invalid states unrepresentable.
I once worked in a role where my boss made me rework commits that were too DRY. I had to completely decouple features that were early in development, then reopen the PR. I thought it was excessive and pointless at the time, but I've seen enough stories like these to get it now.
1. Make sure there's always a trap-door where you can implement special cases
2. A good way to always have (or to create) a trap door is to compose out of smaller pieces; then if you need something to be different in one case you just "reassemble it" with pieces added, removed, or rearranged
In the author's case, maybe CRUDController's key pillars of functionality could be broken out, still bundled together as CRUDController for normal cases, but then for special cases they could just assemble custom solutions out of those pieces instead of bending CRUDController into shape
Not disagreeing with the core assertion in this post because I think the author is correct that you can absolutely over-engineer things. But one thing I don't quite understand.
> Some controllers had to do some things a little differently.
....
> And young me? I just kept going. Adding the proverbial knobs and pulls to my abstract class
Aren't the ways in which the concrete implementations vary supposed to be in the subclasses? One of the points of the abstract base class is to encapsulate common behavior. For things that differ, those belong in the concrete implementations. So _one_ of the mistakes here was to try to write a god class that could anticipate all of the ways in which the subclasses differ and to encode those into the abstract class in order to leave the subclasses as simple as having to specify the entity type.
I think the point of the author was that as a young - implied: inexperienced - developer he was blinded by the power of being able to describe the world in code; and inadvertently got stuck in the rabbit hole.
Learning to take a step back and discern the best fitting abstractions that allow modeling the business domain is a learned skill. There's a difference between learning about object inheritance in school, and actually applying it in big projects where the effects only become apparent over the course of months.
To expand on the article from my personal experience, sometimes, there's a very thin line between "clever code" as in "overengineered", and "clever code" as in "I don't understand it, so I don't like it".
Imagine situation where a team of Java developers need to create a component in Python. No problem!
We come to the following code:
```
list_of_resources = method_a()
if not list_of_resources:
handle_empty_list()
```
Now, is the code above "clever", or not? As Java developers, you could get into arguments like "we should use `len(list_of_resources) == 0` instead" vs "it's not Pythonic and there's no reason to use len instead truth/falsey-ness".
Of course, the whole problem gets exacerbated by unit testing, where `list_of_resources` can be a MagicMock, and therefore `if not list_of_resources` will never trigger.
Of course, probably nobody will argue that the above snipped of code is overengineered. But, somehow, the same logic of "clever code bites you back" applies to more situations than just insane class hierarchies. And it's those very tiny fragments that seems so simple yet there are hidden issues that I find the most interesting to focus on.
> you could get into arguments like "we should use `len(list_of_resources) == 0` instead" vs "it's not Pythonic and there's no reason to use len instead truth/falsey-ness".
these kinds or arguments trigger me so badly i should probably talk to a therapist. I've been involved in a couple and it usually ends, rightly or wrongly, with me pulling rank and saying this is what it's going to be, this is what it's always going to be, move on to bigger things or gtfo of my team.
> the whole problem gets exacerbated by unit testing, where `list_of_resources` can be a MagicMock, and therefore `if not list_of_resources` will never trigger
Not a python or javascript programmer, but that alone already points at using `len` as a better (more testable) solution.
In my limited understanding of python, there's no type-checker that will shout if someone in another team changes method_a() to sometimes return a `None` or `False` in addition to a List. So even if in python empty lists are falsy, the code will be a bit more robust if it doesn't rely on that feature to begin with. The code will error on the if instead of continuing and trying to use a `None` in a place where a list is expected later on.
Disclaimer: I work with a language with only two falsy values (nil and false), so I might be biased
I have a softer rule for myself, which is "If you are going to be clever, make a (depends on the size of the abstraction): README, detailed API docs, possibly even a landing page. Specify behavior, write diagrams, explain what and why, document edge cases. Also, periodically have someone who's never seen it give honest feedback based on the docs only."
If that sounds like too much work, then its not worth it.
Yeah, like me (DevOps) writing our dev environment manager tool in Python when all the devs write nodejs. Then not being able to just ask them how to solve a problem in node, instead having to spend far longer figuring stuff out myself.
Good for learning, bad for velocity. Although I do know a buttload more Python now.
Yeah, clever! As my old man said, so sharp you cut yourself.
My most productive time working on DB applications was when I worked with a code generator.
You laid out the entry screens, specified the tables and columns, fired off the generator and magic happens.
You could slam out a basic maintenance CRUD screen, with search, foreign key popups in an hour!
But here’s the meat of if. If you ever needed to go back and update that program, you could do it in minutes.
Add some validation logic? Mechanically trivial. Say a column was added to the table and you just wanted to populate but not even show it on the screen. 5 minutes. If you wanted to add it to the screen the hard part was deciding where to put it, making room for it, putting it in the right focus order. But coding it? Once it was part of the screen the rest came for free.
See the trick was that unlike pretty much every other code generator I’ve seen (which I think are better termed wizards), this code generator was not a one shot deal.
Your code was in a separate file and the generator merged your changes with the generated code to create the final result. So for simple changes, your code file was quite small as the generator built up all the boiler plate around it.
A very (very) crude example is imagine using a typical one shot generator, making changes to the result, capturing those as a diff patch file, and in the future you did all your work in the patch file, applying it any time you needed to rerun the code generator.
That won’t work in practice, but when the patch files uses more semantic indicators, it does work. It worked really well.
This makes the code generator part and parcel to the workflow, and you get to leverage it again and again.
In many of these systems, and especially modern frameworks, they have a happy path that can do 80+% of the common task, but more often than not you hit that 81% mark and you have to toss the baby out with the bath water. Not only is the tooling not able to take you where you want to go, it can’t even take you partway and you’re on your own and your stuck with 100% of the work instead of the extra 20%.
I never really encountered that using this toolset. I had few examples where you should not rerun the generator any more, but it was really rare.
I agree that you should strive to not make "clever" code since it requires more cognition to parse later (and none of us have perfect memories). I try to strike a balance between reusable code that doesn't have 1 million flags/params to handle all the cases and writing straightforward code.
It's easy for developers to look at 2 components and see the overlap and a lot of, normally junior, developers go overly-DRY and the readability and usability suffers for it.
I'll take verbose over clever every day but at the same time I think this blog author goes a bit too far in assuming there was no use for a base class at all. I agree that the base class does not need to support every eventuality but a good base class would provide the base implementation with an easy way for subclasses to override the logic as needed. And no, tons of hooks probably aren't the way to do that either.
We have some code where I work that I wrote as a base implementation of some CRUD logic but it requires your implementation to create stubs for get/put/delete/iterate/etc that call the internal logic. I got some pushback on this because "I have to write this boilerplate when I just want a straight passthrough" but I'd seen this play out enough times to hold my ground and, while I'm biased of course, I think it's proven successful in the long term. At the start you might just need to do a straight pass-through but business logic always changes or evolves and while on day 1 you might not appreciate having to write "boilerplate" you will be thanking yourself 1 month, 6 months, 1 year down the line when you need to modfiy default flow. If a class never goes pass the "boilerplate" then great! But as soon as it does you'll be thankful for how easy it is to modify.
As always, when making "one-size-fits-all" statements, my answer is "It depends."
In many cases, I use inheritance and polymorphism, as well as extension, to make my code incredibly adaptable. It's basically habit.
It is not always so easy to scrutinize (even by Yours Truly), but I document it well, and the structure can sometimes make tasks that might take days, take minutes (This has happened many times in the project I'm working on, now).
A couple of weeks ago, one of the team needed a fairly large change to the operation of one of my screens. I was pretty reluctant, because it was one of those "Just one little change" things that actually translates to a massive codebase hit.
However, when I reviewed the code (which I hadn't touched in a while), I found that I had left some fairly good hooks in the base class, so it actually took about five minutes to give them what they wanted.
Right now, I'm working on two databases that almost do the same thing. One is legacy, the other is the new and improved. I can't delete the old one because there are several teams that generate their reports from it, I can't delete the new one because new functionality depends on it.
When we add a new feature, we run update queries on both databases (but not always). We need several months of work and create training for everyone to move to the new DB. The business does not recognize the work as valuable so we continue to do this dance.
How did we get here? Someone with great intentions thought "I hate this crappy mssql db, I'll create a new modern one". That person has left the company many years ago. This is now my real life example for "... now you have two problems"
I think this is because most literature focusses on exactly those things. Design patterns to be one step ahead of some obscure future usage of your API, FP patterns to abstract your API into a 'something' that you could not just use to aggregate customer account balances in the database but, say, merge arbitrary lists as well etc.
It's rare to find literature that tells you -not- to use new frameworks or design patterns, and if it exists, if probably doesn't sell very well because it's just not sexy.
I don't think this particular person would have learned the lesson.
> Yes, I had failed to see the proper solution: a class generator — so that I didn't have to manually copy code again. It actually existed back then, I simply didn't know about it, no one told me about it, and I wasn't smart enough to question myself once I started going down a certain path.
I shudder for the next gig where they get to apply ^
> Yes, I had failed to see the proper solution… It actually existed back then, I simply didn't know about it, no one told me about it, and I wasn't smart enough to question myself once I started going down a certain path.
And there’s your problem right there: Assuming that nobody has ever tried to solve this problem before so not bothering to look around at existing solutions.
This is a failure of lack of context, in two ways:
1. The author lacked knowledge about the various possible solutions and the strengths and weaknesses of each one.
2. The author lacked knowledge of the context into which the solution would be applied, e.g. a sophisticated solution might not make sense for an unsophisticated team to maintain.
Found solutions also tend to be super clever. Because they're packaged in a generalized library or white-paper, where the author doesn't have to worry about implementation details.
Clever works great outside of your actual business code.
I would argue you can be clever in one place if it lowers the overall complexity of the code.
In the Lisp community, we try to be "clever by composition" and not "clever by subclassing" and... it makes code where you have to have more than one thing in mind at a time, but if you can do that it's often MUCH simpler than subclasses.
A practical example might be a collections "class" (actually just a few functions) in old-school lucid lisp to do the same thing Java/Smalltalk collections did. Instead of subclassing for Set, Dictionary, OrderedCollection, etc. We had an "insert" function that could call functions to see if we could insert a particular element and where to insert it if we could. If you wanted Set semantics, you returned false from the first function if the element was in the collection. If you wanted OrderedCollection semantics, you did a binary search to find where to insert the element.
It worked well if you were the type of programmer who couldn't remember the Collections class hierarchy semantics but could remember how to write two functions. I'm sure we had a few helper functions for common cases.
It's not a panacea, but every now and again you can lower the overall complexity by raising the complexity of a small portion of code. It is, as the author has discovered, hard to figure out how to do it right the first time through.
At a previous job we used ActiveAdmin on top of Rails, which is a sort of CRUDController as described in the article. Wonderful productivity when you are just starting out, but it got so bloated that it took over a minute to hot reload code. You basically had to kill the server and restart it when changing anything.
Nobody dared doing anything about it and nobody had any idea how to fix the issues. But there were scripts to make it easier to kill and restart the development server...
So at some point I wrote a script that read the ActiveAdmin parts and used a library to convert it into its abstract syntax tree. Now I could manipulate the tree and shape it into regular Rails controllers and plain old HTML views.
That took care of about 90% of the work. The remaining 10% was of course the weird exceptions. Finishing it took a lot of testing, a ballsy deploy to production, and a further hour of frantic bug fixing, but then it was done.
No more productivity destroying hassling with the smart admin framework. I think I would have quit my job way earlier if I wasn't able to fix that, and had to live with it instead.
> It's such a classic example, many of you probably recognise it. I wouldn't say my coding skills were bad, by the way.
Oh, coding skills. This isn’t about coding skills, this is about maintenance skills, which are wholly almost completely different than coding skills.
Maintenance skills are about creating systems where you can:
- extend, add or remove with low risk
- quickly read and understand
- quickly find a relevant bit
- keep modules simple enough that you can consider all possible cases and build them robustly to handle those cases
I see a lot in test code where engineers get overly clever and start writing looping test scripts and whatnot and pretty soon, you don’t have a clear picture of what’s even being tested anymore, or it becomes difficult to remove or change a part of the test.
Testing is _not_ an area you want to get clever with. There is already plenty of built in complexity in introspecting and mocking runtime code to be adding a layer of cleverness to it.
An engineer of a complex system doesn’t strive to make things that are clever or novel. They strive to make things that are maintainable (as defined above). If in DRYing up code, or creating a solution, it’s not making the code more maintainable, you’re going the wrong way.
This is a junior issue fixable with proper social alignment with the team or business process. If the team has no adequate know-how transfer process then there is no right way to properly abstract and understand any business model.
Saying "Don't be clever" is like saying, let me give you a piece of advice: "Don't be mistaken". I've personally find that level of vagueness very annoying.
I don't understand why he added all those exceptions to the abstract class. That's not what abstract classes are for. The initial code example shows an implementation class inheriting from the abstract class; that seems like the perfect place for exceptions. Why not simply use that most basic feature of OOP? Override the parts of your abstract class that you need to override.
I guess he wanted to make sure that the subclass code would be as simple as possible. So when he wanted to add a new entity to his model, as he mentioned, he only had to extend the abstract class, without writing any new business logic.
Clearly what you're saying is correct, however, it would mess up his original intent, which was frictionless extensibility.
The problem isn’t copying code, the problem is if the code has to evolve and then has to be changed in a dozen places consistently. Each place by now having its own particular knobs and special cases doesn’t make it easier.
In the context of code maintenance and evolution, there is a tension between duplication and abstraction. Both have their drawbacks. There is no silver bullet. One can overdo both.
Which is why I always use the term „clever by half”. Simply clever is actually good and a term of praise. Clever until an unforeseen (but foreseeable!) consequence bites back, is, well, not clever enough.
Everyone always wants simple beautiful code they can understand, I want boring code, that implements a beautiful UI users can understand.
And if that means calling some black box of horrors mega library, that's fine as long as I don't have to look in the box.
My first approach to embedded was an installer script that configures a stock distro image. Then I moved to a premade CustomPiOs image, but that's a nightmare and adds another large artifact to build and manage and maintain and isn't portable. Now I'm back to the installer script.
Technical merits have nothing to do with the choice really. Nobody else was interested in learning anything about the customized image but they already know basic scripting.
As much as I love high tech, and pretty much always use the megaproject latest hyper neophile stuff if I can.... I like not having to maintain custom stuff even more.
This describes our legacy codebase, which is quite cleverly optimized for the original business case our company had ~15 years ago, and which is now a massive millstone dragging everything down into a black abyss. Just try to add a feature, I dare you.
I would say: design for flexibility rather than cleverness in most cases.
Fifteen years is a long time for code to remain unchanged. Either you constantly work on it and change as the needs change, or you have a boom and bust cycle of productivity.
Another way to look at it would be: holy shit, this company made it 15 years without investing in rearchitecting their codebase even as they took on new problems. That's amazing! Now we have 15 years worth of use cases to accommodate as we remodel/rearchitect/refactor/rewrite ... do you think that new codebase will last 15 years before it needs major work?
If the amount of time it takes to type your code is the biggest bottleneck in your project and you need to optimise for amount of characters written you're either the most genius 100x coder ever or ... not one of the best.
You should always optimise your (work) code for readability, even if it takes longer to type. I've spent way too many hundreds of hours of my 25 year career untangling overly clever code just to figure out what's wrong.
If you can't be sure the person who has to open up your clever code at 3AM to fix a live production issue can instantly see what's going on with the code, write more readable code. Preferably ask someone to review the pull request and see if they can understand what's going on.
You can be as clever and terse you want on your own time, but on company time write readable code even if it's verbose.
The only time being clever has helped me has been putting out a fire in production, to stop the bleeding, and to buy us time for implementing a better fix in the next release.
Being clever during feature development is a red flag to me. I’m a huge fan of the YAGNI principle.
The CRUDController example is interesting because the problem is not exactly cleverness, but more a design without good escape hatches. It's bad because the pathway from a CRUDController to a controller that does more is a total rewrite.
I have a slightly different view and I think expressing it in front of the right people may have gotten me a job. Programmers ought to be clever, but clever is something you hold in reserve. Choose the boring, the staid, the obvious. Clever is for when you're backed into a corner by odd requirements, edge cases that the obvious will not cover, and so on. Then, you comment your clever, journalism-style. Why were you clever? Where is the clever? Just how did you do it? When do you apply this? And so on.
While I agree with the sentiment to keep things simple, I feel that saying "don't be clever" is just a way for developers to sound actually clever. :-D
So much of the horribly unmaintainable code I've come across was the result of overengineering that whenever I see an abstraction I have a compulsive urge to scream into the void. Some (the simplest ones) are ok, but other times an abstraction that seems like a good idea on day 0 evolves into an unmanageable hodgepodge of hacks that makes an innocent developer that bumps into it years later question their choice of career.
An abstraction creates a new language, a new semantic layer, that has a precise definition. You can express laws and properties from its definition. This lets you think in the new semantic layer without thinking about the lower levels of complexity.
But yes… the pattern class problem is real. a good number of us have all done it once or twice.
The takeaway seems less about nifty coding and more about working in isolation. The author found a path he thought would solve the problem and was 'clever', but never really got feedback from peers. I've done that myself; been so enamored with my own idea that I failed to get feedback and ultimately delivered a non-optimal solution.
things like this always have some underlying notion that there are objectively correct combinations of lines of code. There isn't. The cleverness problem can live at any level in a system.... this line is really clever because in <language> <data type> is really just a <other data type>, so if we use this <unrelated feature> we actually can do this all in one line!
You can also have system level cleverness, which is fine as long as you can explain the cleverness in a few sentences. Sometimes the data access layer is clever and the other stuff isn't....
at the end of the day, as long as you leave things such that they are easy to change, or at least you're brave enough to change them, none of it really matters. Someone is going to replace the "clever" code with other code that does more or less the same thing, and then next engineer will replace that code with something else.
I’ve been working with Prisma and React Query recently and I wonder if those libraries haven’t fallen victim to the same trap. They seem to want to ”do everything you could possibly want to do, in a clean wrapper”.
But I wonder if they are trying to wrap domains that are too big to wrap, and should just be left to a programming language.
It seems, in some subcultures the FactoryFactory or ServiceService approach to reducing repetition and increasing complexity is common. In Lisps/ Scheme the best approach is usually the copy&paste until you established it is a problem and then introduce some wrapper function or a macro or whatever.
I read this like "if I was was really clever back then I would do what now I think is clever" and my humble opinion about the ultimate solution for it really horrifies me (the class generator).
Maybe I'm not clever enough. Maybe I don't hate copy and pasting enough.
The real hidden message in the article is: no matter how great you think your code is, when you revisit it a few years later, it will (nearly) always be crappy.
The lesson is to keep it simple so everyone can recognize that it's crappy and help fix it/make it worse.
As a corollary, quite often in code there is a 'happy path', or small set of operations that people are going to be using 90% of the time. That code is going to get read even more, so make it as high quality and as simple as possible, don't mess with it unless you're very sure...
Oh no, the conclusion is so funny. That's the problem with "magic frameworks" like ASP.NET, where it's not obvious how to extend things if need be (if even possible).
Code generations is just metaprogramming. The worst case, where you have to compile twice. The first one to generate the code and the second to compile the generated code.
Ha, I ran into one of these once. I didn't write it.
Even our database records of "OAuth clients" went through it. I had wanted to make an optimization to those: I wanted to inline the OAuth client IDs for the mobile & web applications into the source code¹. The beast that was the CRUD controller stopped me in my tracks. I eventually gave up — too much complexity all muddled into one thing.
It was also "cleverly" general in that our admin interface built off it, and the UI thus built the table showing you what records existed … generically! Similarly, you could generically CRUD records. And one day … someone deleted the mobile client's record from the OAuth client's table, which of course breaks the mobile app and caused an outage, and lead me to wonder "why do we even have this lever, Kronk?" (While maybe it made sense to delete some random script's OAuth client … not so much the mobile app¹.)
Treating everything as a generic object that can be CRUD'd is one of those extremely leaky abstractions, and as the article gets at, it's usually the business logic that will reveal where the holes in the hull are.
We had another class of "objects" which were user uploads, and they went through about 4 different storage models (they started as rows in the database, and that didn't scale well at all — they ended up as S3 objects which was okay-ish). But my predecessors, when they added a newer, better way of storing these … didn't migrate the old ones. So the read logic had to be ready for a v1 upload or a v4 upload. A minor nightmare. So add "the DB model changes and is never migrated" to the list of things that screw these things up.
(I still see devs doing this — adding newer, better means of storing something but not migrating the old records — and it's massive tech debt, every time I've seen it. Inevitably, I've had someone want to do a "give me all the data" style query, and that necessitates dealing with all the caked on layers of upgrades.
My current company has this in the form of audit records. They started in the database … which … didn't … scale … and now they're in … a bucket in a cloud's blob storage, which is okay-ish.
So if you ever find yourself in one of these … please, please migrate the old data.)
I don't think codegen is the answer. I don't think codegen is ever the answer … "what happens when the generator changes?" is something I never see addressed. The old code rots, typically.
(¹because a.) unless the company folded, they weren't going to change, so I figured that a commit to remove the literal mobile app was fine in that case and b.) that meant that, for those clients, we didn't need to make DB requests to that table. IIRC we did this on every query.)
OK, I grasp what the author is getting at -- there are certainly times when I've gone needlessly complex expecting a set of problems that not only never materialized, but the ones that actually materialized were poorly handled by my design. It sucks. Lesson learned.
I only have first-person anec-data based on a diverse career in software development (big/small/solo/small/big team/process-heavy/haphazard) but it's the opposite circumstance that usually burns me far worse.
I describe this as "they asked for this really small thing and now it runs half the company" problem. It's an exaggeration, but the story from my past was the temporary system that was set up for an (unnamed) major long-distance carrier to "pay the bill via an EDI setup"[1]. Ten years later, shortly after we moved a huge data-center to another time zone the panic alarm bells rang over a few million dollars missing from said carrier.
The temporary-turned-permanent setup was an IBM NetVista machine hooked up to a 1200 baud modem with a dedicated phone line running a DOS program at boot off of a hard drive that had to have its disk parameters manually provided via BIOS every reboot because the little battery inside had sent its last electron down the line and -- rather than replace the battery with an abundance of spares -- the technician decided printing a label with the values would suffice.
A more apt example would be a dashboard that I created a simple service for which produced a list of "the top 20 endpoint threats by vulnerability patching level" ... oh did the execs love this[2], which led to creating a mess of other dashboards and a set of small services that produced just the hostnames. They were all variations like "Just locations with desktop support staff on-site by location", "Servers", "Domain Controllers" ... I want to say I had something like 30 variations at the end ... and thousands of lines of code. I wasn't clever. This "stupid service" took me away from things I needed to finish and several others that I enjoyed working on much more, so this was a model of spaghetti-code-copy-pasta(-marinara). And then they wanted something added to the output of everything. I finished it a few days before leaving the department ... along the way I figured out that more than half of the reports produced results that were anywhere from "kind of close" to what they said they did to several which produced results for the wrong building for each endpoint type (and -- as a result -- was completely ignored). I bit the bullet and took this 5,000 multi-file ASP (vbscript!) monster down to 6-ish classes, less than 500 lines, easy to extend/expand and re-use what's there[3].
Unfortunately, this leads to the totally useless advice of "Be Clever When It Will Help." When will it help? Beats the hell out of me. I didn't imagine a world where a VP calling my desk asking for "a quick web page that shows the worst machines on the network for a slide I'm working on" would become an integral reporting component of an in-house massive networking auditing tool, so I can't help much, there...
[0] Admittedly, I only discovered I encountered the opposite circumstance after that circumstance presented itself, so the whole thing benefited from hindsight bias.
[1] I was not directly involved in the software involved, only in the resurrection of the dead service.
[2] I had the dubious distinction of accidentally creating "a new operations process" whereby every week it was expected that the top 20 machines on Monday (...which became "from each list"...) would be fully resolved by Friday. Desktop had to have dolls in my shape with knives sticking out of them.
[3] Demonstrated on the final revision by adding a single class that altered the retrieval process to WeakReference cache all views (there were reasons an output cache wasn't useful due to it now being part of a bigger system that often required previously materialized values on other threads but never touched an HTTP endpoint).
The problem OP describes (multiple classes like that other class, but with a twist each) is a textbook use case for inheritance, but after decades of prominent programmers indoctrinating „inheritance bad”, I can see why he would not consider it at all.
On a similar vein, Kernighan’s Law: Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.
jbandela1|2 years ago
No, please don't. This is jumping from the frying pan into the fire. If you think abstract base classes can be clever and hard to understand, code generators can be even more so. In addition, because code generators are a one way conversion, and the generated code evolves independently and the code generator itself is evolving independently, you end up with a bunch of things that are subtly different yet somewhat related in a way that you will need an advanced degree in evolutionary biology to understand.
wallmountedtv|2 years ago
jader201|2 years ago
I think code generators are a perfectly acceptable solution in cases like this, when the starting point all looks the same, and it needs to diverge from there. Especially if the generation logic is fairly straightforward.
There are a lot of IDEs that have boilerplate generators out of the box.
dimmke|2 years ago
I am curious, since my own backend experience is limited (obviously there will be many opinions on this) but it seems to me like his mistake was using the inheritance of classes.
If he had simply had a CRUD layer - that only standardized the actions of creating, removing, updating, or deleting a record (or records) from a database. This would accomplish standardizing the payload, as well as possibly doing an RBAC/Authorization check.
He could have just written individual controllers that invoke that layer with custom business logic. He would even have more space to add comments indicating why the business logic is implemented this way. When someone needed to make a change, they could just go straight to the specific controller.
Putting the impracticality of having to override specific methods aside, isn't it also a waste to instantiate a class full of methods that essentially do the same thing when you can just point to existing ones? Well I guess it is pointing to existing ones via inheritance, but it seems harder to reason about. But this is coming from someone who doesn't write traditional object oriented code very often.
Genuinely curious to hear from backend people about this, it's not my area of expertise but I've learned a lot this year.
sanderjd|2 years ago
But if you want to make even a single tiny modification to one of the generated files, you're busted, you need a different solution.
baz00|2 years ago
Then someone came along and invented partial classes in C# which made this problem go away. Well it would if anyone wanted to do all the maintenance legwork which they didn't so half of it's a 5 class inheritance tree, some of it's partial classes and someone got really fed up with this shit and just arbitrarily stuffed Dapper in there one afternoon.
m463|2 years ago
One time this is good is when helping others get up to speed. Other people can copy/paste your examples and be functioning right now.
But if you have a function that you put in to eliminate some duplication, they have to understand this new abstraction layer before they can start working.
moffkalast|2 years ago
summerlight|2 years ago
Of course, code generator (or even compiler) is harder to maintain, especially when you want the output to be human readable. So it's always about trade-off. Think about how many lines of code it's going to generate. If it's an order of 100k then it's usually worth the cost. 10k might be good to go. 1k, probably not.
Pxtl|2 years ago
But even then they're a nightmare because your errors don't match your source files.
"generate and edit" is suicide. "generate and treat the outputs as intermediate compilation objects" is merely an incredibly painful tool of last resort.
robomc|2 years ago
Like the problem was having an all-or-nothing inheritance model for shared functionality, which also ended up very long and intimidating.
I also don't really understand why emerging edge cases caused him to complicate his 90%-perfect parent class instead of just... overriding specific functions in the children. (The specifics of these overrides will probably make you want to refactor your parent class so that the overrides have clean joints but you shouldn't just dump the new child-specific logic into your parent)
jonhohle|2 years ago
At least in Java world, annotation seemed to have taken over, but the mixture of code and interface definition kicks you into the language and allows too much flexibility that really make it no different than writing registration code. Having a nice declarative language without business logic sneaking in.
jauntywundrkind|2 years ago
Of course lots and lots of people love telling us why not to do things. This is the cosmic-brain point of Steve Yegge's "Notes from the Mystery Machine Bus", of what we let guide us. Is it trying and hope and doing? When and where do we allow reservation in? https://gist.github.com/cornchz/3313150
deterministic|2 years ago
Having said that, code generators are not all the same. Some are awesome, others are downright awful. There is an art to writing good code generators. Not surprising really. The same can be said for most other software.
eternityforest|2 years ago
OhMeadhbh|2 years ago
jackcviers3|2 years ago
I think we need to be a little more nuanced, here.
Don't be clever in your code generation, to be sure. But let's not act like you need advanced degrees for all meta-programming.
What do you think phased compilers do? They generally transform the AST produced by your code into expanded versions or rewrites of that AST that are more amenable to machine interpretation or transformation into assembly.
In a very real sense, nearly all production code goes through some kind of codegen process. It isn't clever, it isn't new, it generally isn't complicated (if this then that else that else ... when you get down to it). It is repeatable and reproducible and reliable enough method that the whole economic system of software is built on top of it. Most of the problems come with recursive applications of codegen and strange interactions between different subsystems of codegen.
Here's a tip: Don't do recursive codegen in your user land metaprogramming.
User code is _easier_, simply because it isn't as reusable as the language primitives themselves. But if you understand a problem well enough to essentially reduce it to copy, paste, and replace one or two items, codegen is superior because a bug fix during maintenance need only happen in a single place.
My primary language recently added generic derivation to the language itself (as opposed to in macro libraries) and it is totally worth it as I almost never have to write a ser/de by hand. I never have to test the ser/de round trip. I don't have to pepper my code with second-class annotation primitives or keywords. I never misspell `asJson`, etc.
Not everything should be done with codegen in user land code (code outside the compiler and language tooling). But if I see thousands of lines of json/avro/protobuf/test example copypasta, that's a code smell that needs to be addressed or it will lead to production bugs and take hours instead of minutes to fix.
And when you write these things, you come to find out that they are just like any other data transformation task. They turn `A`s into `B`s, systematically.
My mentor, when I became a professional coder, taught me that if you do something more than three times, its probably worth taking the time to generalize it and do it once before you push the changes into the version repository. It's been 18 years now since I began my career, and that simple principle has rarely steered me wrong across probably a dozen languages that I've written production code in.
1. https://www.goodreads.com/book/show/4099.The_Pragmatic_Progr...
slim|2 years ago
aldanor|2 years ago
SanderNL|2 years ago
Especially repetition seems to be hated with the might of a thousand suns and while I get it, because I myself hated it, I now can see the beauty of it.
What is currently a superficial repetition - a bunch of endpoint handlers, some forms - will often turn out to be similar-looking instances of completely different problems. Repetition is quite often deeper than just "code looks the same".
It takes a while to get to know these nuances.
jstimpfle|2 years ago
I've sometimes taken on factoring the repeated code using common abstractions, and when that was done it often turned out that there was much more to factor out. And that the code quality could massively be improved because there are often some tricky situations that can only be properly handled when everything goes through a single place with all the necessary context.
As an example, we had a config file with an ini-like structure that was parsed by some central parser code. So far, so good, but after the parse, the resulting key-value pairs were processed like this:
Repeating this pattern dozens of times is decidedly not cool. It lacks expression of the common structure, it imposes and prevents code optimizations and fixes (such as error handling).Sometimes I was told that abstraction isn't worth the hassle and boring and repetitive is good. By factoring the code allegedly we risk breaking it and make it harder to fix. Those people didn't see how broken and incomplete the code is precisely because there is a lack of abstraction. We can't even know that there isn't some "quux_setting" with broken handling code because of course there aren't proper tests for each config setting or combination of settings.
wizofaus|2 years ago
On the other hand I also hate having to deal with shared functions that have been repeatedly adapted and extended to be able to deal with all the various edge cases to the point they have 20 cryptically named parameters and no reasonable way of guessing what the output should be for given set of inputs.
But I'd still say more of my time is used up dealing with problems caused by lazy copying & pasting than by shared functions becoming overcomplicated or buried under excessive layers of abstraction.
Mezzie|2 years ago
Most of the time when I do any kind of coding work, it's as a single person project for companies/orgs with absolutely zero tech people. For instance, the last project I took on was for a bra shop of 3 people. No IT. Nobody on staff who has any idea how anything tech works - the closest is the person who taught herself the basics of using Shopify + advertising platforms.
Which means that in the future if they need somebody to look at or update anything I've done, I can't assume they're going to have access to a well-trained, talented person - it's equally likely they'd go "hey so and so's kid does 'computer stuff' let's see if they can fix it for us." Knowing that the next person who looks at my work might have very basic level skills leads me to prefer being repetitive and 'simple' over the more efficient solutions I can think of.
It's the difference between a regular Wikipedia article and the Simple English Wikipedia - if I'm writing something that is likely to be maintained by beginners or apprentices, making it easy to understand and work with matters a lot more.
_fat_santa|2 years ago
I currently work for a client that has a very very large react native app that is pretty hard to wrap your head around. I can't tell you how many millions of dollars they have wasted in dev hours because it takes a dev way to long to even figure out what the code is doing before they can start making changes.
sanderjd|2 years ago
jprete|2 years ago
lopatin|2 years ago
If there are other forms of repetition within the user-code, there are ways of dealing with it. But writing a super class to solve superficial code repetition is most often the wrong way to go.
hliyan|2 years ago
toshk|2 years ago
Most of time you want business logic in the same place. In a big & old codebase it can be very hard to fix things if there are multiple places.
But certain things can be easily repeated.
jdjdjdhhd|2 years ago
Pxtl|2 years ago
lijok|2 years ago
Then a year or so later the client brings us back in to help diagnose and fix why their delivery dates continue slipping and number of bugs in production is increasing. Finds out a new, very vocal hire, came in, saw the repetition and decided to build a "service service". A service, into which you would pass a single python file of pure business logic encoded as a b64 string, with some decorators, and it would set up the endpoints, serialization, DB, etc automagically.
sjducb|2 years ago
PartiallyTyped|2 years ago
Somehow some colleagues found it absurd to parse some code twice, but don't see the insanity of parsing code, getting an AST, serializing, sending it, and then deserializing correctly.
Never mind the fact that the serialized AST is 250x larger, and deserializing arbitrary constructs without codegen is a whole different can of worms, and especially harder when the code is running in 2 different programming languages.
catchnear4321|2 years ago
you built a system for cogs.
someone hired a thinker.
IshKebab|2 years ago
It's difficult to draw conclusions from these sorts of things though because the answer is "sometimes you should copy and paste a bit, sometimes you shouldn't" and you just have to have skill and taste to know the difference.
ravenstine|2 years ago
The author's problem isn't being overly clever, but that they had applied an inappropriate yet imtellectually-satisfying programming pattern that is notorious for being difficult to make exceptions for. The quote about the correct solution being a "class generator" shows that the author misdiagnosed their problem. If you continue down the path of making everything a class, you're gonna have a worse one than if you made most of your functionality into compostable functions, reserving classes for things that truly deserve it.
So many issues in programming would be diminished if programmers toned down the thinginess of their code, focusing more on procedures and data shapes.
Verdex|2 years ago
One plausible definition for clever code is that it's code that only works because of a non-obvious dependency on some other fact being true in the code base or outside of the code base. Once the fact is no longer true the code will be broken or subtly broken (or it will break or subtly break other code). "Ah ha, very clever, but we shouldn't do this."
Another plausible definition is approximately, "I don't understand the feature or pattern you're using, and I don't want to understand it or form a coherent argument against it."
dgb23|2 years ago
Exactly my reaction too. The code is complicated and requires boilerplate because it tries to adhere to a pattern that doesn't need to exist.
Whenever something becomes entangled and complex, decompose it into pieces first (functions and data) and then put it together again, piece by piece. The right abstractions (if needed) emerge naturally from simple code.
assbuttbuttass|2 years ago
gnat|2 years ago
All programming is making models: representing a chaotic and changing analog world in bits, data structures, and algorithms. For every model there's something that breaks that model. Attempting to incorporate every exception into your model leads you to the path of tuples + a workflow engine, and at that point all you've done is build a different database and programming language and move the complexity further up the stack.
As requirements change, the model you built will no longer prove appropriate because now you're modeling different situations. There's an art to building a model that doesn't box you in to the point where the first change request doesn't break your model. But at some point, every model is going to be inadequate.
A senior programmer brings to this: knowing the common ways things change, the costs of changing different types of model, and the understanding that there's no such thing as Perfect so don't proclaim your framework to be such.
partdavid|2 years ago
Frankly, KISS and its ilk have become more often "things uninsightful people say to make them sound clever and vaguely iconoclastic" than something that can be actually useful as guidance: because people think they remove nuance rather than adding it (KISS is great if you honor its nuances like "and no simpler..." and "considered as a whole"; but instead it's a thought-terminating cliché). I've heard people use these sound bites to justify, e.g., the entire notion of automated testing (too complicated), or "infrastructure as code"--though I also dislike this term and favor "software-defined infrastructure", though maybe if it became popular, objections that software-defined infrastructure isn't really software would become as common as objections that infrastructure-as-code isn't really code.
-----
[1] I don't mean this literal blog post, I mean the same kinds of "don't be too clever" platitudes.
amanvir_|2 years ago
thom|2 years ago
report-to-trees|2 years ago
Verdex|2 years ago
This article talks about a time someone made an abstract class and then took DRY a little too far. If this is an example of 'being clever' then anyone using map instead of writing a for-loop would have to be at gaussian levels of cleverness.
[Also a class generator is implied to be less clever than an abstract class ... I'm not sure I can accept the usage of 'clever' in the article as meaningful.]
PartiallyTyped|2 years ago
A new pattern may seem "clever" or "difficult" if you haven't encountered it before; but if you spend time with it and work with it, it can look like the most natural thing in the world.
_iostreamer_|2 years ago
But I do agree with the direction of your thought. The reading material at least needs to be inherently balanced rather than writing itself getting balanced out by 50% of articles talking about "Using proxies to prevent null pointer errors"(guilty of writing this) and the other half being "Maybe proxies are a bad idea".
AnimalMuppet|2 years ago
ygouzerh|2 years ago
[deleted]
BoxFour|2 years ago
Code was devised to address a specific problem (DRY), which it did quite elegantly. Then, the situation evolved, rendering the solution inappropriate. When a controller starts to deviate from the standard approach, it's best to refrain from inheriting from the base controller and instead create custom ones. Once all controllers operate independently, the base class can be safely removed.
The key lesson here is not to avoid cleverness but to be aware of when the initial problem becomes obsolete.
sjducb|2 years ago
orblivion|2 years ago
bob1029|2 years ago
I just stay focused on that item. "Is my proposal/refactor/shiny thing going to make a customer want to spend more money, sign additional/longer contracts, or somehow smooth out support & operations?"
If I get an "ehhh im not sure" or a more honest "no", then I'd be inclined to shitcan the proposal unless a very clear argument can be put forth for why this refactor will somehow eventually result in downstream value-add.
Arguments along the axis of "because its the right thing to do", "this is the technical best practice" or "it's cool to look at" are rejected by default unless additional supporting evidence can be brought forth. Proposals that bring additional vendors & operational liabilities without also bringing obvious value are to be considered malicious suggestions.
This probably sounds draconian as hell to some of HN, but if you are working with truly complex software, you cannot be playing games in traffic, especially if you are already in production for many customers and under half-decade-long contracts.
The time to be clever is on your side projects. Do not conflate the thing that results in your paycheck with entertainment. It might seem like there could be some happy co-existence, but I've personally never seen it work out.
eternityforest|2 years ago
tempodox|2 years ago
nperez|2 years ago
brundolf|2 years ago
1. Make sure there's always a trap-door where you can implement special cases
2. A good way to always have (or to create) a trap door is to compose out of smaller pieces; then if you need something to be different in one case you just "reassemble it" with pieces added, removed, or rearranged
In the author's case, maybe CRUDController's key pillars of functionality could be broken out, still bundled together as CRUDController for normal cases, but then for special cases they could just assemble custom solutions out of those pieces instead of bending CRUDController into shape
xwowsersx|2 years ago
> Some controllers had to do some things a little differently.
....
> And young me? I just kept going. Adding the proverbial knobs and pulls to my abstract class
Aren't the ways in which the concrete implementations vary supposed to be in the subclasses? One of the points of the abstract base class is to encapsulate common behavior. For things that differ, those belong in the concrete implementations. So _one_ of the mistakes here was to try to write a god class that could anticipate all of the ways in which the subclasses differ and to encode those into the abstract class in order to leave the subclasses as simple as having to specify the entity type.
I guess that was the author's point ha?
CaptArmchair|2 years ago
Learning to take a step back and discern the best fitting abstractions that allow modeling the business domain is a learned skill. There's a difference between learning about object inheritance in school, and actually applying it in big projects where the effects only become apparent over the course of months.
BossingAround|2 years ago
Imagine situation where a team of Java developers need to create a component in Python. No problem!
We come to the following code:
```
list_of_resources = method_a()
if not list_of_resources:
```Now, is the code above "clever", or not? As Java developers, you could get into arguments like "we should use `len(list_of_resources) == 0` instead" vs "it's not Pythonic and there's no reason to use len instead truth/falsey-ness".
Of course, the whole problem gets exacerbated by unit testing, where `list_of_resources` can be a MagicMock, and therefore `if not list_of_resources` will never trigger.
Of course, probably nobody will argue that the above snipped of code is overengineered. But, somehow, the same logic of "clever code bites you back" applies to more situations than just insane class hierarchies. And it's those very tiny fragments that seems so simple yet there are hidden issues that I find the most interesting to focus on.
chasd00|2 years ago
these kinds or arguments trigger me so badly i should probably talk to a therapist. I've been involved in a couple and it usually ends, rightly or wrongly, with me pulling rank and saying this is what it's going to be, this is what it's always going to be, move on to bigger things or gtfo of my team.
otikik|2 years ago
Not a python or javascript programmer, but that alone already points at using `len` as a better (more testable) solution.
In my limited understanding of python, there's no type-checker that will shout if someone in another team changes method_a() to sometimes return a `None` or `False` in addition to a List. So even if in python empty lists are falsy, the code will be a bit more robust if it doesn't rely on that feature to begin with. The code will error on the if instead of continuing and trying to use a `None` in a place where a list is expected later on.
Disclaimer: I work with a language with only two falsy values (nil and false), so I might be biased
spion|2 years ago
If that sounds like too much work, then its not worth it.
bloopernova|2 years ago
Good for learning, bad for velocity. Although I do know a buttload more Python now.
Yeah, clever! As my old man said, so sharp you cut yourself.
rbreaves|2 years ago
charles_f|2 years ago
That your code is complex to change is a good sign that it's not design in an appropriate way.
Switching from "this code is repetitive" to "therefore I will automate the production of this code" sounds unfortunate though.
1: https://en.m.wikipedia.org/wiki/Composition_over_inheritance
dmichulke|2 years ago
> Yes, I had failed to see the proper solution: a class generator
Looks to me like the problem is actually object orientation
whartung|2 years ago
You laid out the entry screens, specified the tables and columns, fired off the generator and magic happens.
You could slam out a basic maintenance CRUD screen, with search, foreign key popups in an hour!
But here’s the meat of if. If you ever needed to go back and update that program, you could do it in minutes.
Add some validation logic? Mechanically trivial. Say a column was added to the table and you just wanted to populate but not even show it on the screen. 5 minutes. If you wanted to add it to the screen the hard part was deciding where to put it, making room for it, putting it in the right focus order. But coding it? Once it was part of the screen the rest came for free.
See the trick was that unlike pretty much every other code generator I’ve seen (which I think are better termed wizards), this code generator was not a one shot deal.
Your code was in a separate file and the generator merged your changes with the generated code to create the final result. So for simple changes, your code file was quite small as the generator built up all the boiler plate around it.
A very (very) crude example is imagine using a typical one shot generator, making changes to the result, capturing those as a diff patch file, and in the future you did all your work in the patch file, applying it any time you needed to rerun the code generator.
That won’t work in practice, but when the patch files uses more semantic indicators, it does work. It worked really well.
This makes the code generator part and parcel to the workflow, and you get to leverage it again and again.
In many of these systems, and especially modern frameworks, they have a happy path that can do 80+% of the common task, but more often than not you hit that 81% mark and you have to toss the baby out with the bath water. Not only is the tooling not able to take you where you want to go, it can’t even take you partway and you’re on your own and your stuck with 100% of the work instead of the extra 20%.
I never really encountered that using this toolset. I had few examples where you should not rerun the generator any more, but it was really rare.
joshstrange|2 years ago
It's easy for developers to look at 2 components and see the overlap and a lot of, normally junior, developers go overly-DRY and the readability and usability suffers for it.
I'll take verbose over clever every day but at the same time I think this blog author goes a bit too far in assuming there was no use for a base class at all. I agree that the base class does not need to support every eventuality but a good base class would provide the base implementation with an easy way for subclasses to override the logic as needed. And no, tons of hooks probably aren't the way to do that either.
We have some code where I work that I wrote as a base implementation of some CRUD logic but it requires your implementation to create stubs for get/put/delete/iterate/etc that call the internal logic. I got some pushback on this because "I have to write this boilerplate when I just want a straight passthrough" but I'd seen this play out enough times to hold my ground and, while I'm biased of course, I think it's proven successful in the long term. At the start you might just need to do a straight pass-through but business logic always changes or evolves and while on day 1 you might not appreciate having to write "boilerplate" you will be thanking yourself 1 month, 6 months, 1 year down the line when you need to modfiy default flow. If a class never goes pass the "boilerplate" then great! But as soon as it does you'll be thankful for how easy it is to modify.
ChrisMarshallNY|2 years ago
In many cases, I use inheritance and polymorphism, as well as extension, to make my code incredibly adaptable. It's basically habit.
It is not always so easy to scrutinize (even by Yours Truly), but I document it well, and the structure can sometimes make tasks that might take days, take minutes (This has happened many times in the project I'm working on, now).
A couple of weeks ago, one of the team needed a fairly large change to the operation of one of my screens. I was pretty reluctant, because it was one of those "Just one little change" things that actually translates to a massive codebase hit.
However, when I reviewed the code (which I hadn't touched in a while), I found that I had left some fairly good hooks in the base class, so it actually took about five minutes to give them what they wanted.
firefoxd|2 years ago
When we add a new feature, we run update queries on both databases (but not always). We need several months of work and create training for everyone to move to the new DB. The business does not recognize the work as valuable so we continue to do this dance.
How did we get here? Someone with great intentions thought "I hate this crappy mssql db, I'll create a new modern one". That person has left the company many years ago. This is now my real life example for "... now you have two problems"
max_hammer|2 years ago
Also, I would put other DB in keep the lights on (KTLO) and not provide any support or patches. You need to make your problem thier problem.
turnsout|2 years ago
misja111|2 years ago
It's rare to find literature that tells you -not- to use new frameworks or design patterns, and if it exists, if probably doesn't sell very well because it's just not sexy.
foobarian|2 years ago
> Yes, I had failed to see the proper solution: a class generator — so that I didn't have to manually copy code again. It actually existed back then, I simply didn't know about it, no one told me about it, and I wasn't smart enough to question myself once I started going down a certain path.
I shudder for the next gig where they get to apply ^
jt2190|2 years ago
And there’s your problem right there: Assuming that nobody has ever tried to solve this problem before so not bothering to look around at existing solutions.
This is a failure of lack of context, in two ways:
1. The author lacked knowledge about the various possible solutions and the strengths and weaknesses of each one.
2. The author lacked knowledge of the context into which the solution would be applied, e.g. a sophisticated solution might not make sense for an unsophisticated team to maintain.
koromak|2 years ago
Clever works great outside of your actual business code.
retrocryptid|2 years ago
In the Lisp community, we try to be "clever by composition" and not "clever by subclassing" and... it makes code where you have to have more than one thing in mind at a time, but if you can do that it's often MUCH simpler than subclasses.
A practical example might be a collections "class" (actually just a few functions) in old-school lucid lisp to do the same thing Java/Smalltalk collections did. Instead of subclassing for Set, Dictionary, OrderedCollection, etc. We had an "insert" function that could call functions to see if we could insert a particular element and where to insert it if we could. If you wanted Set semantics, you returned false from the first function if the element was in the collection. If you wanted OrderedCollection semantics, you did a binary search to find where to insert the element.
It worked well if you were the type of programmer who couldn't remember the Collections class hierarchy semantics but could remember how to write two functions. I'm sure we had a few helper functions for common cases.
It's not a panacea, but every now and again you can lower the overall complexity by raising the complexity of a small portion of code. It is, as the author has discovered, hard to figure out how to do it right the first time through.
dajonker|2 years ago
So at some point I wrote a script that read the ActiveAdmin parts and used a library to convert it into its abstract syntax tree. Now I could manipulate the tree and shape it into regular Rails controllers and plain old HTML views. That took care of about 90% of the work. The remaining 10% was of course the weird exceptions. Finishing it took a lot of testing, a ballsy deploy to production, and a further hour of frantic bug fixing, but then it was done. No more productivity destroying hassling with the smart admin framework. I think I would have quit my job way earlier if I wasn't able to fix that, and had to live with it instead.
dclowd9901|2 years ago
Oh, coding skills. This isn’t about coding skills, this is about maintenance skills, which are wholly almost completely different than coding skills.
Maintenance skills are about creating systems where you can:
- extend, add or remove with low risk
- quickly read and understand
- quickly find a relevant bit
- keep modules simple enough that you can consider all possible cases and build them robustly to handle those cases
I see a lot in test code where engineers get overly clever and start writing looping test scripts and whatnot and pretty soon, you don’t have a clear picture of what’s even being tested anymore, or it becomes difficult to remove or change a part of the test.
Testing is _not_ an area you want to get clever with. There is already plenty of built in complexity in introspecting and mocking runtime code to be adding a layer of cleverness to it.
An engineer of a complex system doesn’t strive to make things that are clever or novel. They strive to make things that are maintainable (as defined above). If in DRYing up code, or creating a solution, it’s not making the code more maintainable, you’re going the wrong way.
sebastianconcpt|2 years ago
Saying "Don't be clever" is like saying, let me give you a piece of advice: "Don't be mistaken". I've personally find that level of vagueness very annoying.
mcv|2 years ago
duderific|2 years ago
Clearly what you're saying is correct, however, it would mess up his original intent, which was frictionless extensibility.
layer8|2 years ago
In the context of code maintenance and evolution, there is a tension between duplication and abstraction. Both have their drawbacks. There is no silver bullet. One can overdo both.
valenterry|2 years ago
Nothing to see here, please move on...
WesolyKubeczek|2 years ago
otikik|2 years ago
eternityforest|2 years ago
And if that means calling some black box of horrors mega library, that's fine as long as I don't have to look in the box.
My first approach to embedded was an installer script that configures a stock distro image. Then I moved to a premade CustomPiOs image, but that's a nightmare and adds another large artifact to build and manage and maintain and isn't portable. Now I'm back to the installer script.
Technical merits have nothing to do with the choice really. Nobody else was interested in learning anything about the customized image but they already know basic scripting.
As much as I love high tech, and pretty much always use the megaproject latest hyper neophile stuff if I can.... I like not having to maintain custom stuff even more.
slifin|2 years ago
So everything is a class
The majorities language's community has this super weird Java obsession The other half has a Rails obsession
The best career decision I ever made was to watch the Rich Hickey talks
2x my wage in a few years and don't have to care about any of this obvious mis-match ever again
dpcx|2 years ago
hu3|2 years ago
karaterobot|2 years ago
I would say: design for flexibility rather than cleverness in most cases.
gnat|2 years ago
Another way to look at it would be: holy shit, this company made it 15 years without investing in rearchitecting their codebase even as they took on new problems. That's amazing! Now we have 15 years worth of use cases to accommodate as we remodel/rearchitect/refactor/rewrite ... do you think that new codebase will last 15 years before it needs major work?
theshrike79|2 years ago
You should always optimise your (work) code for readability, even if it takes longer to type. I've spent way too many hundreds of hours of my 25 year career untangling overly clever code just to figure out what's wrong.
If you can't be sure the person who has to open up your clever code at 3AM to fix a live production issue can instantly see what's going on with the code, write more readable code. Preferably ask someone to review the pull request and see if they can understand what's going on.
You can be as clever and terse you want on your own time, but on company time write readable code even if it's verbose.
lolsal|2 years ago
Being clever during feature development is a red flag to me. I’m a huge fan of the YAGNI principle.
strken|2 years ago
contravariant|2 years ago
The author spent so much time thinking about what the CRUDController should be able to do they forgot to think about what it even is.
at_a_remove|2 years ago
neontomo|2 years ago
vbo|2 years ago
0x4164|2 years ago
Be DRY enough. DRY what needed only, you will keep debugging easy.
agentultra|2 years ago
An abstraction creates a new language, a new semantic layer, that has a precise definition. You can express laws and properties from its definition. This lets you think in the new semantic layer without thinking about the lower levels of complexity.
But yes… the pattern class problem is real. a good number of us have all done it once or twice.
TheIronMark|2 years ago
code_runner|2 years ago
You can also have system level cleverness, which is fine as long as you can explain the cleverness in a few sentences. Sometimes the data access layer is clever and the other stuff isn't....
at the end of the day, as long as you leave things such that they are easy to change, or at least you're brave enough to change them, none of it really matters. Someone is going to replace the "clever" code with other code that does more or less the same thing, and then next engineer will replace that code with something else.
erikpukinskis|2 years ago
But I wonder if they are trying to wrap domains that are too big to wrap, and should just be left to a programming language.
kaliszad|2 years ago
unknown|2 years ago
[deleted]
pachico|2 years ago
Maybe I'm not clever enough. Maybe I don't hate copy and pasting enough.
DaanDL|2 years ago
Cutman|2 years ago
The lesson is to keep it simple so everyone can recognize that it's crappy and help fix it/make it worse.
bqm|2 years ago
Usually, when you do that, you end up with simple logic and potential repetitions over complicated abstractions.
sweezyjeezy|2 years ago
burglins|2 years ago
robertlagrant|2 years ago
motoboi|2 years ago
janmarsal|2 years ago
Sujeto|2 years ago
sam0x17|2 years ago
GTP|2 years ago
log101|2 years ago
FreshStart|2 years ago
Aeolun|2 years ago
Khelavaster|2 years ago
imdsm|2 years ago
bellajbadr|2 years ago
_samjarman|2 years ago
mock-possum|2 years ago
deathanatos|2 years ago
Even our database records of "OAuth clients" went through it. I had wanted to make an optimization to those: I wanted to inline the OAuth client IDs for the mobile & web applications into the source code¹. The beast that was the CRUD controller stopped me in my tracks. I eventually gave up — too much complexity all muddled into one thing.
It was also "cleverly" general in that our admin interface built off it, and the UI thus built the table showing you what records existed … generically! Similarly, you could generically CRUD records. And one day … someone deleted the mobile client's record from the OAuth client's table, which of course breaks the mobile app and caused an outage, and lead me to wonder "why do we even have this lever, Kronk?" (While maybe it made sense to delete some random script's OAuth client … not so much the mobile app¹.)
Treating everything as a generic object that can be CRUD'd is one of those extremely leaky abstractions, and as the article gets at, it's usually the business logic that will reveal where the holes in the hull are.
We had another class of "objects" which were user uploads, and they went through about 4 different storage models (they started as rows in the database, and that didn't scale well at all — they ended up as S3 objects which was okay-ish). But my predecessors, when they added a newer, better way of storing these … didn't migrate the old ones. So the read logic had to be ready for a v1 upload or a v4 upload. A minor nightmare. So add "the DB model changes and is never migrated" to the list of things that screw these things up.
(I still see devs doing this — adding newer, better means of storing something but not migrating the old records — and it's massive tech debt, every time I've seen it. Inevitably, I've had someone want to do a "give me all the data" style query, and that necessitates dealing with all the caked on layers of upgrades.
My current company has this in the form of audit records. They started in the database … which … didn't … scale … and now they're in … a bucket in a cloud's blob storage, which is okay-ish.
So if you ever find yourself in one of these … please, please migrate the old data.)
I don't think codegen is the answer. I don't think codegen is ever the answer … "what happens when the generator changes?" is something I never see addressed. The old code rots, typically.
(¹because a.) unless the company folded, they weren't going to change, so I figured that a commit to remove the literal mobile app was fine in that case and b.) that meant that, for those clients, we didn't need to make DB requests to that table. IIRC we did this on every query.)
bronxpockfabz|2 years ago
uwagar|2 years ago
10g1k|2 years ago
mdip|2 years ago
OK, I grasp what the author is getting at -- there are certainly times when I've gone needlessly complex expecting a set of problems that not only never materialized, but the ones that actually materialized were poorly handled by my design. It sucks. Lesson learned.
I only have first-person anec-data based on a diverse career in software development (big/small/solo/small/big team/process-heavy/haphazard) but it's the opposite circumstance that usually burns me far worse.
I describe this as "they asked for this really small thing and now it runs half the company" problem. It's an exaggeration, but the story from my past was the temporary system that was set up for an (unnamed) major long-distance carrier to "pay the bill via an EDI setup"[1]. Ten years later, shortly after we moved a huge data-center to another time zone the panic alarm bells rang over a few million dollars missing from said carrier.
The temporary-turned-permanent setup was an IBM NetVista machine hooked up to a 1200 baud modem with a dedicated phone line running a DOS program at boot off of a hard drive that had to have its disk parameters manually provided via BIOS every reboot because the little battery inside had sent its last electron down the line and -- rather than replace the battery with an abundance of spares -- the technician decided printing a label with the values would suffice.
A more apt example would be a dashboard that I created a simple service for which produced a list of "the top 20 endpoint threats by vulnerability patching level" ... oh did the execs love this[2], which led to creating a mess of other dashboards and a set of small services that produced just the hostnames. They were all variations like "Just locations with desktop support staff on-site by location", "Servers", "Domain Controllers" ... I want to say I had something like 30 variations at the end ... and thousands of lines of code. I wasn't clever. This "stupid service" took me away from things I needed to finish and several others that I enjoyed working on much more, so this was a model of spaghetti-code-copy-pasta(-marinara). And then they wanted something added to the output of everything. I finished it a few days before leaving the department ... along the way I figured out that more than half of the reports produced results that were anywhere from "kind of close" to what they said they did to several which produced results for the wrong building for each endpoint type (and -- as a result -- was completely ignored). I bit the bullet and took this 5,000 multi-file ASP (vbscript!) monster down to 6-ish classes, less than 500 lines, easy to extend/expand and re-use what's there[3].
Unfortunately, this leads to the totally useless advice of "Be Clever When It Will Help." When will it help? Beats the hell out of me. I didn't imagine a world where a VP calling my desk asking for "a quick web page that shows the worst machines on the network for a slide I'm working on" would become an integral reporting component of an in-house massive networking auditing tool, so I can't help much, there...
[0] Admittedly, I only discovered I encountered the opposite circumstance after that circumstance presented itself, so the whole thing benefited from hindsight bias.
[1] I was not directly involved in the software involved, only in the resurrection of the dead service.
[2] I had the dubious distinction of accidentally creating "a new operations process" whereby every week it was expected that the top 20 machines on Monday (...which became "from each list"...) would be fully resolved by Friday. Desktop had to have dolls in my shape with knives sticking out of them.
[3] Demonstrated on the final revision by adding a single class that altered the retrieval process to WeakReference cache all views (there were reasons an output cache wasn't useful due to it now being part of a bigger system that often required previously materialized values on other threads but never touched an HTTP endpoint).
jasmer|2 years ago
[deleted]
WesolyKubeczek|2 years ago
WesolyKubeczek|2 years ago
Jun8|2 years ago
Discussed on HN a few years ago: https://news.ycombinator.com/item?id=22348386