top | item 20357203

Fighting Complexity in Software Development

257 points| atsapura | 6 years ago |github.com | reply

115 comments

order
[+] Chyzwar|6 years ago|reply
In mature OOP you have ways to write nice models and have good validations. https://guides.rubyonrails.org/active_record_validations.htm...

I will argue that the complexity of software development is not because of OOP vs Functional. Tooling, documentation, quality of libraries and people are what matter most. Ruby was a massive success is largely attributed to above. We are humans, we can understand and deal with a fixed amount of complexity if I can offload some of it to a framework, library or tool I will have more time to work on my problem.

Every time I try to play with anything Functional I got hit by a bus of undocumented frameworks(Erlang), multiple standard libraries (ocaml), competing half finished implementations (lisp), arcane tooling (scala), no tooling (Haskell) and broken tooling (F# on Linux).

[+] jes5199|6 years ago|reply
but have you ever had to maintain a ruby project after the first year? The cost just goes up and up, and I think it’s because the language is so hostile to static analysis.
[+] cottsak|6 years ago|reply
Agree. You can write terse, maintainable, well encapsulated, testable and readable code in most any language. The problem is about the humans not the language they select.

But yes, complexity is also a major problem.

[+] jacknj|6 years ago|reply
When you mention "no tooling (Haskell)" what do you mean? I am using Haskell (mostly for hobby projects) and am wondering what I am missing out on, since I feel the tools available is sufficient.
[+] apta|6 years ago|reply
This is already achievable using annotations in Java or attributes in C#, in a less verbose way. You just tag your method parameter with `@Valid` or `[Valid]` or what have you, and the framework you're using automatically ensures that the validations you specified on the data model are valid at that point in time.
[+] pjmlp|6 years ago|reply
The irony is that Smalltalk already offered many FP patterns.

In the end multi-paradim languages will win.

What many FP advocates fail to acknowledge is that all FP languages that got some kind of mainstream adoption, might be FP first, but they are actually multi-paradigm.

[+] merlincorey|6 years ago|reply
> competing half finished implementations (lisp)

As well as one of the most complete specifications in ANSI Common Lisp.

[+] walshemj|6 years ago|reply
I think the argument is that for a lot of cases OOP isn't the right paradigm
[+] flukus|6 years ago|reply
I'm sure this is partly because I don't read F#, but it looks like they've moved all the complexity into meta-programming madness, this is just being way to clever to play code golf at a high level, this is exactly the sort of complexity we should be fighting against.

Even the initial c# version was over complicated. The complex fluent interface with lambdas and callbacks could be done with a few if statements that would be simpler, faster and require no knowledge of the FluentValidation library. Unnecessary getters and setters to satisfy the encapsulation gods.

If you want to fight complexity got back to basics, you can have a static method returning a validation result with code like this:

  if (!string.IsNullOrEmpty(card.CardNumber) && CardNumberRegex.IsMatch(card.CardNumber))
    validations.add("Oh my");
Converting if statements to more elaborate constructs is creating complexity not fighting it.
[+] kazinator|6 years ago|reply
The main source of complexity is requirements. Gatekeeping against the influx of requirements will keep complexity down.

Then there is unnecessary complexity from doing incomplete refactorings and rewrites. If some code cannot handle the addition of a new requirement, it should be replaced. Otherwise you add complexity that roughly takes the logical (if not actual) form if (these cases) { new code } else { old code }. And there is overlap! new code has taken over requirements for which old code still exists, but because of some lingering requirements that only the old code handled, all of it is still there (due to laziness, dependencies or whatever). It's not obvious that some of that code is never used; someone diving into it faces the complexity of figuring out what is the real payload in production now and what is the historic decoy.

[+] marcc|6 years ago|reply
It's easy for us to blame "requirements" as the main source of complexity. This isn't accurate. Software exists to serve the needs of the business. Depending on the maturity and stage of the business or the software itself, it's possible that there's a changing set of requirements. As developers, it's out job to figure out how to deliver, not to say "no" to new enhancements and requirements.

The main source of complexity is how we write software, not that the software has requirements.

[+] Aeolun|6 years ago|reply
I feel like I’ve responded this before, but I feel like people often attribute their increased knowledge of how to develop systems without bugs to the new fancy language they switched to.

Fact is they could build better software in the old language as well, assuming they started from scratch.

[+] DecoPerson|6 years ago|reply
I strongly disagree. We use a strongly typed Lua-like language at my company and it has everything you need to build decent applications, but we hit so many bugs. It took me 12 hours over 5 days to make a simple modification to the business logic (half of that was figuring out and fixing bugs). It took me 4 hours to write something far more complex in Rust with virtually zero bugs; I attribute this almost entirely to sum types (Rust enums), a better type system, an unforgiving compiler, async/await, a better module system, lifetime checking, and an ecosystem of easy-to-grok libraries.

These things just make bugs disappear.

When it comes to IDE experience, the parts that I use often are mostly the same between Rust and our language.

Edit: I'd say it's both, in a multiplicative manner. You need experience and a good set of tools (the language itself being the most important tool) to write good code fast.

[+] codr7|6 years ago|reply
Which is a longish way of giving the answer no one wants to hear: Experience is everything.
[+] collyw|6 years ago|reply
In my opinion it's usually worse using a new framework or language as you don't know the ins and outs of it. Your first project is going to be a learning experience in that technology. I have seen plenty of Python code that was clearly written by Java developers wanting to try out something new.
[+] hinkley|6 years ago|reply
Figuring out the Mikado method was one of the bigger shocks of my career. I thought I already knew all this stuff, and of course once I saw it I could explain it all. But knowing something is true and seeing it first hand can be a very different experience.

The simpler solution is often hard to see. We get attached to the wrong details or suffer sunk cost fallacies.

When you switch languages the cost of porting is higher, so it shouldn’t be a surprise that you end up with something much simpler. And if the target language attracted you because it makes some part of the problem simpler, that’s important but maybe not the dominant contributing factor to the experience.

[+] kh7uky|6 years ago|reply
I'd agree, but some tools just make it easier to design and build a complex system whatever the level of experience of the programmer(s), designers and so on. This programming language and IDE https://en.wikipedia.org/wiki/Clarion_(programming_language) is behind some of the biggest databases in the world. Because of the openess of the IDE, in one instance it was possible to migrate one country's main cancer charity app from ISAM files to MS SQL, and rewrite it from procedural to OOP code in just two hours! Admittedly it took a week to build a program to do the coding changes, but that program became a tool in its own right to migrate other programs, but the original devs thought it would take a human 3months to do the work, which is already several months less than if the program was written in another language!

These are just some of the big corps who use Clarion. https://en.wikipedia.org/wiki/LexisNexis https://en.wikipedia.org/wiki/DBT_Online_Inc. https://en.wikipedia.org/wiki/Experian

Various banks and other stock market listed companies. Even various military use it for their own top secret work.

The key to its success is the template language, which enables the programmers to work at a higher level of abstraction which for some reason just doesnt seem popular amongst many programmers. You can use the templates to write code in other languages, including Java, PHP, ASP.net, javescript and more.

Its safe to say, that everyone in the Western world will have some of their details stored in a Clarion built database, and its not just limited to building databases, its even been used to build highly scalable webservers. Theres also C/C++, Assembler and Modula-2 built into the compiler, so you can get right down to low level coding if required, and there's a Clarion.net version which is mainly like C# but has some of the data handling benefits of F#.

[+] mixedCase|6 years ago|reply
I can attest to this but it's definitely misleading. Some languages have features and patterns that are straightforward and easy to learn in that language, that once you've learnt you can emulate/replicate in almost every other language but only because you already know how it works, why it works and where the limits of your abstraction are.

And these abstractions will be often be overlooked or misused by developers who have not used them in languages where they're native; making them a net negative instead of an obvious benefit.

[+] davesmith1983|6 years ago|reply
"Started from Scratch". I have never seen any project where something has been started from scratch actually turn out well.
[+] agumonkey|6 years ago|reply
It does play but no. Languages and paradigms do make a difference. I had to come up with a tiny DP answer for a java interview and it was a massive burden. Even though I could write the same (quality and perf) version in other languages in 2 minutes.
[+] kazinator|6 years ago|reply
This can be validated or refuted by going back to that language and building something. If the same old roadblocks reappear, it was the darned language, after all.
[+] twodave|6 years ago|reply
On the C# API I develop for we overcome these issues in a few ways.

1. Our way of implementing DDD helps us organize code into infrastructure and domain. Domain objects typically aren’t allowed to have external dependencies. Infrastructure code is primarily for data access and mapping. Our API code (controllers and event handlers) ties the two together.

2. Given the above we are able to write a) very clear and concise unit tests around domain objects and API endpoints and b) integration tests that don’t have to bother with anything but data and other external dependencies.

The result is that when we go to ask, “How does the system respond given X?” we can either point to a test we have already written or else add a new test or test case to cover that scenario.

We can even snapshot live data in some critical areas that we can then drive through our high level domain processes (we process payroll so it’s a lot of data to consider). If someone wants to know how a particular pay run would play out, they can just craft the data snapshot and write some assertions about the results.

We also use FluentValidation (on API objects only) and test those as well (but only if the rules are non-trivial).

[+] realshowbiz|6 years ago|reply
I’m quite happy to be seeing conversations about the benefits of simplicity, boring tech, etc lately.

It’s a breath if fresh air from the sadly too common (IMO) flavor of the month new tech promotion.

[+] tw1010|6 years ago|reply
Sometimes it's hard to disentangle if a conversation is having an upward trending trajectory, or if you just happen to pay attention more to the links that mention some subject you happened to have caught an interest in.
[+] UK-AL|6 years ago|reply
Domain modelling made functional is fantastic book(What this article is inspired by). Taught me a lot about encoding business logic using functional programming techniques.

Functional programming is basically my goto when there is complicated business logic involved now.

[+] redact207|6 years ago|reply
I really love the concepts provided by Domain Driven Design (DDD), regardless if you choose OOP or FP to implement it with.

It's fine if you want to choose C#, and there're better ways of addressing the approaches to validation in OOP than were provided in the examples. Value objects are a nice way to ensure strong immutable types like credit cards can be created and passed around without requiring separate validation classes or wild abstract base classes.

I like exceptions in C# - when I used to code that I'd make a lot of domain/business exceptions that the code would throw anytime there was a violation. Here I think Java is a lot stronger in that you are forced to declare what types of errors can be thrown from a function so you have a chance of handling them. In C#, Typescript, I'm finding myself having to lean on codedoc "@throws" to do the same thing (though not as reliably).

That said, I generally am fine for most exceptions to not be handled and instead bubble up "globally". If it happened because of an API request? Let middleware map it back to a 400 Bad Request with the error body. If it happened because of a message handled? Log it, retry the message until it gets dumped to the DLQ. If it's not a violation, then it may not be an exception in the first place, in which case it can be returned with a compensating action performed.

I really like F#, but I struggled to find the actual benefit of it in this article from a DDD perspective.

[+] tomxor|6 years ago|reply
I think the problem with OOP is that it is merely a pattern and yet has been integrated into many languages in a very prominent way. This can mislead people into thinking it's something more fundamental and generally applicable - but it's not, it's just another pattern, which for some things works great, and for others is terrible. Just like when you read that aweful code that someone wrote just after reading a book on pattern x and forced it upon a project, the difference is OOP is forced on almost everything.

Once you realise this it's fine, you just don't use those features when it doesn't make sense.

[+] jshowa3|6 years ago|reply
An excellent click bait article for functional programming. For example, the brief talk of validators, why couldn't you just put the validator in the credit card object? The credit card validator should only be used in the credit card object so the discussion about not being able to see it can be accessed by simply navigating to the classes that compose the credit card. Putting it in the constructor or other credit card methods fixes the not being forced aspect (when I look at the FP example, it looks like it's doing just this). And you can use the validator anywhere in the CC object.
[+] yogthos|6 years ago|reply
In my experience, one of the best ways to fight complexity is by reducing the amount of things you have to keep in your head. In practical terms this translates into being able to do local reasoning about your code.

I find that imperative OO style naturally leads to complexity. Passing references to mutable data all over the place creates tight coupling across your entire application. This makes it impossible to guarantee that any change you make is local without considering every other place that references the data. Meanwhile, objects are opaque state machines and programs are structured by creating many interdependent objects.

These aspects make it pretty much impossible to tell what any particular piece of code is doing just by reading it in large applications. The only option is to fire up the debugger, get the app in a particular state and look at the data. However, there are typically many ways to get into any particular state, and it's really hard to know that you've accounted for them all. So, a debugger is a heuristic at best.

FP and immutability tackle both of these problems head on. Immutable data directly leads to the ability to do local reasoning about your code, and allows you to write pure functions that can be reasoned about independently. Meanwhile, data is not being abstracted inside opaque state machines that provide ad hoc DSLs as their API. Instead, it's explicitly passed through function pipelines to transform it.

[+] tabtab|6 years ago|reply
Much of the time the "big picture" isn't local. That's just the nature of the big picture by definition. I find it better to put "big picture" stuff in the RDBMS, including UI issues (see "Table-Oriented" nearby), and keep only local details in code.

For example, the menus and navigation can almost all be tracked and managed in the RDBMS. It's easier to query and study the structure that way because I can sort, search, group, and filter it by any way --I-- please for any given need; I don't want to be stuck with YOUR single grouping; I want to be the Grouping God when studying the app. File-centric code can't do that (at least not without an IDE that reinvents a database). Therefore, don't do it. Use code where code is best, and RDBMS where RDBMS is best.

Code sucks at the big-picture and FP won't change that.

[+] petra|6 years ago|reply
Reasoning locally, means maintenance on FP software should be easy.

So beginners could solve simple bugs and add simple features to FP applications.

Yet it's not a common thing.

Why ?

[+] twhitmore|6 years ago|reply
Is it just me? I feel this is confused and adding complexity in some areas, not genuinely optimal for simplicity.

I tend to feel that unexpected, unrecoverable exceptions are best treated as just that -- exceptions. Applying a C-style function return value check seems backwards.

And the "Interpreter"?? Proper purpose of Interpreter or DSL is for dynamic (configurable or user-input) code, not to implement basic sequential flow and the 'if' statement which the underlying language already provides.

[+] couchand|6 years ago|reply
Seems like a pretty classic case of second system syndrome to me.
[+] tabtab|6 years ago|reply
I agree that OOP is limiting, but Functional is not the fix. Table-Oriented-Programming is the future. Your code snippets for validation etc. could be associated how your domain prefers, and you can query them to be together by field or by any other grouping as needed. You just have to make sure you have the proper meta-data in place, such as field name/ID, entity, screen, event type, etc.

File-centric code forces a hierarchical big picture structure, but many relationships are either not hierarchical, or need additional non-hierarchical ways to view/group them. Relational is more powerful and more flexible than file systems. (It has some rough areas, but they can be worked around or fixed.)

Start backward next time and think how you would LIKE your code organized, forgetting about frameworks you know. If you do this often enough, you'll realize RDBMS-based code management is where we should be heading. About 90% of validation and field management could also be attribute-driven: data-dictionaries would do most of the grunt work.

With OOP and FP you are forced into choices such as "should this be its own class, or an object instance, or a group of classes for composition?" etc. etc. When tablizing your event & validation snippets, you are not forced to choose. They are grouped "by" anything you want, and by multiple groupings at the same time: multiverse. I agree that FP is probably more flexible than OOP, but it's also less disciplined: large FP systems look like the spaghetti-pointer databases that preceded RDBMS. Hierarchical, logical, and pointer-based DB's thrived for a while, but relational won, for a reason.

[+] timClicks|6 years ago|reply
One really excellent resource that I'm making my way through currently is "A Philosophy of Software Design" by Ousterhout. It's very practical and provides several strategies for reducing complexity and identifying practices that contribute to it.
[+] ChicagoDave|6 years ago|reply
Reducing complexity starts with abstracting bound contexts, understanding relationships between them, identifying ubiquitous languages, understanding the autonomous bubble pattern, and then worrying about code.

I’d add DAL should be shelved for a repository pattern and business layer shelved for root aggregates and value objects.

It’s all in Eric Evans’ Domain Driven Design book that still carries enormous weight.

[+] austincheney|6 years ago|reply
Perhaps the best way to take this seriously is to ensure developers RTFC (Read The Fucking Code). While that sounds like a given it’s taken for granted that developers actually do that before forming all manners of biased or incomplete assumptions.
[+] eloff|6 years ago|reply
That's necessary, but the aim of good software practice should be to make it as easy as possible for the reader of the code to follow and understand it. Including skipping over well named functions which should not nest unintuitive behavior.
[+] collyw|6 years ago|reply
Having recently jumped into a project with MongoDb as the database, I have realized how much easier a schema definition makes understanding the project. It's so much easier to understand a number of create table statements than it is to get that information from code.
[+] hardwaresofton|6 years ago|reply
I really support this kind of writing, I don't think this kind of stuff is written about enough, and it's exactly what everyone learns the hard way working on software projects. Skill wise, knowing the kinds of reasoning/techniques that articles like these discuss is the difference between "junior" and "senior" developers (though I very much dislike those terms).

One thing I want to point out -- if at all possible do not use decimals for money:

> We could use decimal (and we will, but no directly), but decimal is less descriptive. Besides, it can be used for representation of other things than money, and we don't want it to be mixed up. So we use custom type type [<Struct>] Money = Money of decimal .

The custom type is a great idea (try to write code in languages that make this concept easy, I suggest Haskell & type/newtype). The problem here is that decimal is the wrong type for storing money[0]. Your first IEEE754 floating point bug teaches you this, but in general trying to write code around manipulating decimals can get very messy really quickly when precision is involved in any case. Another example is JSON, JSON numerics are actually all floats under the covers, so this means if you store more precision or a bigger number than it can handle, things can get wacky if you're not careful -- this is one of the places where being "stringly typed" (and defining your own unpacking to go with your domain types) can be very helpful.

Libraries like dinero.js[1] exist because of how surprisingly hard this problem is, kind of like how moment[2] exists due to how hard dealing with time can be.

[0]: https://stackoverflow.com/questions/3730019/why-not-use-doub...

[1]: https://github.com/sarahdayan/dinero.js

[2]: https://momentjs.com

[+] jorams|6 years ago|reply
> The problem here is that decimal is the wrong type for storing money

Note that the code here is .NET, where decimal is a type explicitly for use in financial calculations[0]. It is still floating point, which means you still need to really watch what you are doing, but it's very high precision.

In general though, if your application allows for it, you should store money using integers representing cents (or the relevant smallest unit for the currency).

[0]: https://docs.microsoft.com/en-us/dotnet/csharp/language-refe... (this is a page for C#, but it is more useful than the general page for Decimal)

[+] marcc|6 years ago|reply
Agreed. I dislike the terms "sr" and "jr" developer, but I think we too often call someone a senior because they know a lot of technology or they've been around for a while.

I've been thinking a lot about these terms and have been defining these roles internally as:

junior: can build relatively simple solutions, still gaining experience.

mid-level: can build complex systems, but the solution is going to be complicated.

senior: builds simple solutions to solve complicated problems.

I mentor internal devs to help shift their focus from learning more complicated technologies toward learning how to produce much cleaner and simpler solutions. This is the career path that I think helps them the most and also helps our company scale.

[+] lacampbell|6 years ago|reply
I love F#, but using F# instead of (type|java)script would add a lot of complexity to my 'full stack'. Let's see I replaced my node.js backend with an F# one, the following issues would arise:

1. I now have two different languages for client and server, and can't share e.g. validation code.

2. Dealing with 2nd class linux support (no REPL)

3. No library that combines the maturity and simplicity of express.js. Yes I am aware of ASP.NET Web API and no I do not think that compares.

So as much as I love pipes and partial application and concise syntax, F# would create an explosion of complexity, not fight it.

EDIT: This also follows the common trend I see of starry eyed functional programmers who - to put it bluntly - don't seem to know what they are criticizing.

Of course some things from here we can do in C#. We can create CardNumber class which will throw ValidationException in there too.

A Result datatype with all the bells and whistles is what, a few hundred lines in C#? I agree that it sucks that there isn't one built in, but if you like them and you're stuck in C#, code one up and forget about the 'issue' ever again.

But that trick with CardAccountInfo can't be done in C# in easy way

That example looks trivially translatable to an abstract class with two concrete sub-classes to me.

EDIT 2:

The F# docs are awful. Struggling to find the API for the Result module. There's a guide on how to use it, but when you look at the core namespace it's missing.

[+] avinium|6 years ago|reply
I really don't think it adds complexity. Sure, it's going to be different, so there's a mental hurdle to clear in learning how a new stack works. That doesn't mean it's going to be more complex.

> 1. I now have two different languages for client and server, and can't share e.g. validation code.

You could use Fable to transpile F# to Javascript, keeping a single language

> 2. Dealing with 2nd class linux support (no REPL)

FSI is available under Linux, but I admit it's got some hairs on it. (FWIW, FSI on Windows needs a lot more TLC too).

> 3. No library that combines the maturity and simplicity of express.js. Yes I am aware of ASP.NET Web API and no I do not think that compares.

This is purely a matter of taste. While express.js is very accessible, I don't think it's any more so than Giraffe (a wrapper around ASP.NET Core).

I'm all-in on F# now - once you're over the (mild) initial learning curve, you see huge dividends from the smaller codebase, static typing, fewer null-checks and drastically less testing required.

[+] arwhatever|6 years ago|reply
What would the concrete implementation of the at class that represents the deactivated credit card look like?
[+] mapcars|6 years ago|reply
``` type CardNumber = private CardNumber of string with member this.Value = match this with CardNumber s -> s static member create str = ```

Haha, no thank you

>OOP languages for modern applications gives you a lot of troubles, because they were designed for a different purposes

For which purposes if not software development exactly?

[+] atsapura|6 years ago|reply
> For which purposes if not software development exactly?

Well, if you recognize only 1 category "software development", I can't help you. C was designed for software development as well, but you wouldn't chose it to do web development, I hope? It's a good tool when you need to develop something small and resource efficient. OOP fits good when you don't have much of concurrency, but you manage complex states in memory (which you do with help of objects and inheritance). And functional programming fits well when you need to manage data flow applications.

And truth is in a big complex system you need a decent support of both FP and OOP. Point is that languages like C# decently support only OOP.

[+] GorgeRonde|6 years ago|reply
Fighting stupidity, laziness and a misplaced love for typography in software developers