Most of the enterprise developers' job is connecting components together. I hate it. I hate modifying existing systems and hate connecting them together. It is usually hard, but the kind of hard which is not really fulfilling for me. It is about fighting with accidental complexity created by other people. It is not why I started programming when I was 12 years old. But this is what needs to be done: it is rational from business perspective. Of course at least in my side project I create something new from scratch (a component-based UI system painted on HTML5 Canvas).
It's not that difficult when you work with the right tools.
Most of these problems boil down to a couple operations.
map
reduce
filter
Most of the data comes from a couple places:
Some kind of SQL system
Text files
Webservices (of the JSON/XML/HTML variety)
Most of the data goes back to the same places that it came from:
Some kind of SQL system
Text files
Webservices (of the JSON/XML/HTML variety)
All you need to make are four types:
inputs: () -> Some<T>
outputs: Seq<T> -> ()
pipes: Seq<T> -> Seq<Q>
tees: Seq<T> -> () -> Seq<T> -> Seq<Q> (once partially applied the tee becomes a pipe)
Using the usual sequence operators pretty much anything is possible with those inputs, outputs, and transforms (pipes) and you avoid a lot of overhead making data pipelines.
To continue the plumber metaphor: some systems are like black boxes with a lot of pipes sticking out - and they will route their raw sewage into your freshwater connector depending on barely documented and sometimes non-deterministic configuration parameters.
The API of 10 year old closed source enterprise systems that have grown over time by buying this solution provider and that and where documented methods may work or not... shudders, no, that's not plumbing at all, that is more like playing Minecraft in a toxic waste dump.
What about those all enterprise integration tools (visual mappers for xml/relational/web-services) like Altova Mapforce, MicroSoft Biztalk Mapper and Stylus Studio? Oracle and IBM have similar tools.
I haven't used these - but they look like they'd nail this problem. Is there a problem with them?
Definitely. An awful lot of programming is "Take data from place A (in format 1) and place B (in format 2) do magic thing J to it and put it in location Z (in format 3)".
Process J is usually the simple bit - the hard bit is taking it from A and B, converting (1) and (2) so that they can be used together, dealing with all the different eventualities that can cause things to go wrong when fetching the data, parsing the data into (3) and then writing to Z (while dealing with all the things that can go wrong when writing to it).
"Yeah, meanwhile Z is temporarily unavailable and by the time it's available again, the data from A and B is no longer consistent, so you have to re-query again, ouch, your pointers to last read data on A are no longer valid anymore and yeah, we forgot to tell you this little funny thing about B... <the dramatic music in the background pauses here>" - this is what you should be thinking when looking at a small arrow drawn on a whiteboard.
I don't think this makes sense. If it were true, that would imply that it would be easier to rewrite J to work on format 1, which is the flip side of claiming the conversion is the more difficult part. And if the arrows are more expensive than the components they are pointing to, this remains true for all such arrows, so adding more arrows means it's even cheaper to rewrite J n times than to write n converters; you can't make up the loss in volume.
This sounds superficially appealing but I'm not sure there's any actual wisdom here.
Funny anecdote, last time I heard about "plumber programmers", the meaning was very different and actually quite pejorative. The person I was talking to was referring to the type of programmers that can only write applications by assembling third party components such as ruby gems while lacking the algorithmic skills to solve problems that haven't been solved before.
I've talked about this too. Sadly, this is what programming has started to become: slapping third-party components together and calling yourself a developer.
It's one of the reasons I tend to stay away from things like Django and RoR. It makes it easy to develop applications quickly, but I many times have to create hacks to make any changes that don't fit into the one-size fits all libraries.
A well-architected application (they do exist) where a lot of work was put into designing the arrows and boxes will suffer from this effect less so than one where the arrows and boxes were added on haphazardly. In fact, I would argue that spending too much time worrying about the arrows is a sign that something's likely wrong with your architecture.
Of course, that's not to say that architecture can cure this altogether. In particular, enterprise software tends to focus on integration a lot, and mostly because enterprises have a lot of boxes that need to be put together.
Startups will tend to do a lot less of this, but it's still important.
For the past 8 years I've been designing .NET enterprise integration solutions and I love it. I love the problem solving that is required and working on getting square pages to fit seamlessly into round holes. Recently I switched companies and was working with jQuery and PHP doing front end things like UI validation and was losing my mind. It's not the pencil and paper type of challenge as integration design.
I'm with you on this one, we're a rare breed. I think people underestimate how much good software engineering can go into these "boring" enterprise projects. Of course, there are a lot of shops where programming skill is lacking, so you're actively discouraged from doing anything the least bit interesting. But if you can find a gig that welcomes skill and creativity, you can end up having a lot of fun doing these types of projects.
For me, an interesting project is not the topic but the technical challenges behind it. Your typical web front end stuff just doesn't have any appeal. It's just programming gruntwork.
I've used the "plumber" term to refer to my own deeds, if only because they don't resemble the shiny and showy stuff which is out there. I don't make games, and I don't generate graphics. My stuff tends to be the glue which makes other things happen in the first place.
I'm far more likely to write the server underneath that huge multiplayer game and deal with all of the systems stuff, in other words. The actual gameplay, graphics, sound, and user interface? That's someone else's realm.
"Object-oriented systems are all about interfaces and interactions. An object’s internal state is an implementation detail and not part of its observable behavior. As such, it is more subject to change than the object’s interface. We can therefore keep specs more flexible and less brittle by avoiding reference to the internal state of an object."
This 'Plumber Programmer" article is meant to be more general than object-oriented code, but I think the analogy applies. Building systems is about interaction and communication - system state is just an illusion that gives us a first approximation of how things talk to one another.
[+] [-] nadam|14 years ago|reply
[+] [-] fleitz|14 years ago|reply
Most of these problems boil down to a couple operations.
Most of the data comes from a couple places: Most of the data goes back to the same places that it came from: All you need to make are four types: Using the usual sequence operators pretty much anything is possible with those inputs, outputs, and transforms (pipes) and you avoid a lot of overhead making data pipelines.[+] [-] chokma|14 years ago|reply
The API of 10 year old closed source enterprise systems that have grown over time by buying this solution provider and that and where documented methods may work or not... shudders, no, that's not plumbing at all, that is more like playing Minecraft in a toxic waste dump.
[+] [-] 6ren|14 years ago|reply
I haven't used these - but they look like they'd nail this problem. Is there a problem with them?
[+] [-] js2|14 years ago|reply
[+] [-] AndrewDucker|14 years ago|reply
Process J is usually the simple bit - the hard bit is taking it from A and B, converting (1) and (2) so that they can be used together, dealing with all the different eventualities that can cause things to go wrong when fetching the data, parsing the data into (3) and then writing to Z (while dealing with all the things that can go wrong when writing to it).
[+] [-] praptak|14 years ago|reply
[+] [-] matwood|14 years ago|reply
[+] [-] jerf|14 years ago|reply
This sounds superficially appealing but I'm not sure there's any actual wisdom here.
[+] [-] uniclaude|14 years ago|reply
Funny anecdote, last time I heard about "plumber programmers", the meaning was very different and actually quite pejorative. The person I was talking to was referring to the type of programmers that can only write applications by assembling third party components such as ruby gems while lacking the algorithmic skills to solve problems that haven't been solved before.
[+] [-] rick888|14 years ago|reply
It's one of the reasons I tend to stay away from things like Django and RoR. It makes it easy to develop applications quickly, but I many times have to create hacks to make any changes that don't fit into the one-size fits all libraries.
[+] [-] j_baker|14 years ago|reply
A well-architected application (they do exist) where a lot of work was put into designing the arrows and boxes will suffer from this effect less so than one where the arrows and boxes were added on haphazardly. In fact, I would argue that spending too much time worrying about the arrows is a sign that something's likely wrong with your architecture.
Of course, that's not to say that architecture can cure this altogether. In particular, enterprise software tends to focus on integration a lot, and mostly because enterprises have a lot of boxes that need to be put together.
Startups will tend to do a lot less of this, but it's still important.
[+] [-] mullethunter|14 years ago|reply
[+] [-] zheng|14 years ago|reply
[+] [-] hackinthebochs|14 years ago|reply
For me, an interesting project is not the topic but the technical challenges behind it. Your typical web front end stuff just doesn't have any appeal. It's just programming gruntwork.
[+] [-] timclark|14 years ago|reply
[+] [-] hezekiah|14 years ago|reply
[deleted]
[+] [-] rachelbythebay|14 years ago|reply
I'm far more likely to write the server underneath that huge multiplayer game and deal with all of the systems stuff, in other words. The actual gameplay, graphics, sound, and user interface? That's someone else's realm.
[+] [-] jonmc12|14 years ago|reply
"Object-oriented systems are all about interfaces and interactions. An object’s internal state is an implementation detail and not part of its observable behavior. As such, it is more subject to change than the object’s interface. We can therefore keep specs more flexible and less brittle by avoiding reference to the internal state of an object."
This 'Plumber Programmer" article is meant to be more general than object-oriented code, but I think the analogy applies. Building systems is about interaction and communication - system state is just an illusion that gives us a first approximation of how things talk to one another.
[+] [-] run4yourlives|14 years ago|reply
[+] [-] gurtwo|14 years ago|reply
[+] [-] dexen|14 years ago|reply
Again, there is more value in the connections than in the individual programs themselves.
[+] [-] anamax|14 years ago|reply
[+] [-] hezekiah|14 years ago|reply
[deleted]