The trick to making a rewrite work is to only rewrite what you know how to rewrite. What I mean by that is a lot of people will be working on a project and will at some point say to themself "This project is really hard, I can't get anything done, I have no idea what is going on. My only solution is to do a complete code rewrite". If this is your mindset going into a large refactor, you are bound to either fail, or end up with a system that is just as badly composed as it was before.
On the other hand, if you're saying to yourself "This code really sucks, they wrote their own complex CSV parser, I can rewrite this to use the CSV library in the standard lib" and things like that, no matter how big the task may seem, it'll work out in the end.
One thing to pay attention to is the never-quite-done migration. For example, i worked on a deep refactoring (near rewrite) of a room reservation system where the data model needed restructuring for faster querying (but with identical data). The migration was done by making the old data model automatically write to the new from triggers, and then migrating the read code to point to the new. Ofcourse, once the performance bottleneck from the read code was solved, suddenly migrating the write code seemed way less important. Higher priorities intervened, and the code is still in a halfway state two years down the road.
Another danger i ran across were moving goal posts. Long rewrites will typically be interrupted to extend the old system with new features, which increase the effort to rewrite.
I don't know the specifics (obviously) but this reads like creating a better index on an existing table.
The gross part is (I'm guessing, obviously) that now each write to this data store incurs an additional write and some cpu load on the data store servers (you mentioned it was done via a trigger). But now the value of finishing the migration that you started is just recovering this extra write and cpu.
There is one common mistake that teams tend to make when deciding on a rewrite: committing to the rewrite too early.
Here's some ways to do a rewrite well:
Set up the rewrite as an alternate, competing product and let the two compete in the marketplace. This requires having the resources to support developing two such products at the same time using separate teams for each, which is not something many companies can afford. The advantage to doing it this way is that it gives the rewrite time to mature before it's fully replaced the old system. What most people tend to not realize is that mature software with fundamental flaws can often be superior to immature software on a better foundation. Real-world use is orders of magnitude more effective as a QA process than any sort of in-house testing, and always well be, products that have been tried by fire and made the cut should always be respected no matter how horrible their underlying code is.
If you look at a lot of the major switchovers in software you'll see this sort of trend fairly often. For example, consumer Windows took several generations and almost an entire decade to fully switch over to the NT core.
Another good way is to rewrite piece-wise, in place (refactoring). There are lots of different ways to do this regardless of architecture, ultimately you're essentially creating a bubble of new code that grows and grows until it completely replaces all the old code. The advantage of this is that it takes much less effort than a full rewrite and you can do it incrementally and slowly. Often it's easier to mature each new component in isolation than to try to mature an entire new product.
And, of course, there's the hybrid approach. Work component-wise in an "add + deprecate" fashion. Create the new component, have it work in parallel along with the old component, and then deprecate the old component and work toward moving client code to the new component.
> This requires having the resources to support developing two such products at the same time using separate teams for each, which is not something many companies can afford.
This is definitely the ideal situation, but as you said most companies do not have the resources to greenfield a separate project. Additionally convincing the higher-ups that its worth it is a difficult task.
> Another good way is to rewrite piece-wise, in place (refactoring). There are lots of different ways to do this regardless of architecture, ultimately you're essentially creating a bubble of new code that grows and grows until it completely replaces all the old code.
> Work component-wise in an "add + deprecate" fashion.
Everywhere that I have worked this has been the route we've taken. We isolated the old behavior until we were able to fully ween clients off the product (or perform a data conversion and completely use the new code).
Having been the person responsible for migrating data from an old system to a shiny new one, I wish I had read this a few years ago. However, since I will be embarking on rewrite from scratch project in a couple of weeks at a new position, I am glad I read this now. Migrating data is hard and should be given the same level of importance as the back-end or front-end development. I remember vividly the cycle of "sleeping" with my laptop next to me churning away over night, waking up every hour to check the process and having it fail at odd times. As the author said, make your migration process repeatable, testable and logged. Good luck.
Over the years, I've noticed that many programmers who want to re-write, just aren't comfortable modifying other people's code, or it wasn't written in their favorite language. So, they are critical of the code and urge for a re-write. These people generally make bad programmers. If you have people on your staff that want to re-write this program, or switch to this framework (because it's what they are used to), then watch out. That's a bad sign.
Competent programmers can maintain other people's code, those who cannot want to re-write everything. Train wreck in progress. Run from these people, or fire them if you're in a position to do so.
I've come to a situation in the past, where there would be endless lines upon lines of PHP code intermixed with random static functions and HTML markup.
I'm talking 40, 2000 LoC files that all called the database whenever they felt like it, all echoed out data and modifed each other in wierd ways.
I spent about two weeks trying to untangle the mess and threw the towerl in at day 15. We just rewrote it using a PHP framework that not only let us query the database using a sane ORM, but also gave the project STRUCTURE and added mantainability value.
What an excellent why of shutting down critique of your choices and protect your self from having to learn new ideas.
Probably why most big companies have such a horrible time getting good people to work for them: who really wants to work in a place like that if you have options somewhere else?
I totally agree that much of the time, it's just people wanting to work on something they find familiar, being incapable of maintaining code other than their own.
But then again, sometimes, the problem is just that the legacy code is shit. In a project last year, for instance, I had to rewrite every class I touched; the whole thing was so fragile that it was impossible to change anything without the rest breaking. So, I did this “incrementally” rather than all at once, and factored this into the cost of the features I was implementing for the client.
Absolutely agree, and I personally use this as a potential red flag for new people we hire.
Legacy systems have gone through so many fixes and shifts that it's almost impossible to recreate them without deep understanding of the business demands it meets.
The business part, I would say, is critical to being able to maintain (or re-write) a project.
Competent programmers known when its time to maintain code or to re-write it and theres no easy answer. But i definitely agree with your point that insisting on a switch of framework just because its more familiar with out any real compelling reasons is a very bad sign.
We have a pattern we call shrink ray. It's a graph of how much the old system is still in place.
Immediately brought to mind the first large shop I worked in out of school (about 1992) where they had a big chart on one wall titled "Punch Card Elimination" with a downward trending line. I was actually shocked that a large, multinational manufacturer was using punch cards for anything, particularly when my project involved building cutting-edge (at the time) client/server systems that included wireless handheld devices on the client side. My introduction to the counterintuitive world of enterprise IT: you will often see opposite ends of the technology spectrum in use, simply because the value proposition of eliminating the old was never high enough before.
It ceased to be a prototype when it went into production. No dissembling can change that fact at that point, all the caveats of how and what you can rewrite take effect whether you like it or not.
Data migration becomes a lot easier the closer to "raw" data you store. The farther away you are, the tricker it will be to write those migration scrips. And, if what you're storing is rolled-up enough, you may not even have the information you want/need any more. Note that the source data doesn't have to be what you're serving queries from; it could be an archive with batch scripts to re-process that's only dusted off in case of disaster/migration.
This was great. I don't really code but I have been trying to migrate handcoded websites to Wordpress since dinosaurs ruled the earth. I still haven't figured out what I need to do to pull that off satisfactorily. Copying pages requires me to then strip out html. I feel like I have never wrapped my brain around getting the new navigation to really work even though navigation problems (with an overly long and growing list of pages) was one reason for the migration.
I am wondering what sorts of info I need to go looking for to get this finished. I have thought about it before, contemplated asking around, and realized I didn't even really know what questions I needed to ask.
I've always just dumped the Wordpress database using the export tool. After I have the data I rewrote the template.
Most of the time the Wordpress template is so wonky that its better to just rewrite it (especially if you're migrating away from Wordpress in the first place).
In the past, we used subsets of data to test our migration tool. Always. ALWAYS, there was something bizarre in the the excluded subset that caused the migration to break.
Rewrite are bad, true. But sometimes unavoidable. Really.
When you have a PHP project where there's a mix of procedural function (the code is at the top and the html at the bottom, so it's easy to split right ? Hum and those includes, what do they do exactly ?), ... classes (We used classes to put our functions in so we do OOP, right ? Err... no) and parts of a heavily customized Wordpress.
Plus two-three half started in-code reorg/rewrite that went nowhere because the devs jumped the ship.
You're told you have to I18N the whole spaghetti meatball soup. You try a few weeks and then see you're going nowhere and start to realize that you're doomed to a rewrite. Oh, and you're a team of two. Sometimes you really have to. And migrating data .... such a pain in the ass ! Test encoding I'm looking at you ! At of hand int code (would not call that enum) I'm looking at you too.
But. It's not always the case. And right now I'm starting the reorganize and refactoring a project that was already in a descent shape. And it's much more interesting, less straining and business owner have more visibility on the work done and are happier.
So yes, whenever you can, incremental rewrite is the way to go. But never say never. There are contexts where doing a rewrite is just plain necessary because the code is in such a bad shape you just can't add any feature anymore.
Voted you up because you basically just described the situation I'm in now.
While successfully in production, the web servers were crashing multiple times a day (still Apache-modphp). So, I started to work on the PHP upgrade from 4-5 to get onto php-fpm + Nginx to stabilize the site. Lo and behold, the legacy db abstraction code (poor man's ORM) was not playing well with later PHP releases so I started down the path to patch what I could.
I successfully upgraded PHP, fixed a few of their outstanding issues, but I was weeks into this project and had not completed any of the new feature sets on my plate yet. Looking into the horizon I imagined months of wrestling with legacy code and the deeper I went (there was a SOAP layer I hadn't even looked at yet, they wanted REST/JSON) the more worried I became.
We evaluated a bunch of options and opted for a re-write. We're still deep into it but there is light at the end of the tunnel and I'm pretty sure it will be worth it in the end.
Incrementalism is sometimes called Burkean or Oakeshottian conservatism in political circles.
The analogy used is of a ship at sea. Replacing the ship in a single step, while it is underway, is impossible. Replacing too many planks at once is foolhardy. (I thought the analogy was Oakeshott's but I can't find the original source).
So instead you replace the planks one at a time. Yes, this is tremendously slow, wasteful and frustrating. But when the ship is a system on which you utterly rely, there is no other safe path.
Hayek expanded on the problems of pure rationalism more generally; it's a trap we as a profession tend to fall into. We substitute our preferences (for a tidy, orderly system) for an appraisal of actual reality. Real systems have accumulated complexity that cannot be wished away. Worse: the complexity is often essential and not merely accidental.
Thank God I've never HAD to do a complete re-write of anything.
What the author describes sounds like a complete nightmare.
On the projects I've worked on, I would like to think that a rewrite will not be needed.
If it does, it will be because we've now understood the problem space so well that we can start from scratch and avoid all the 'pivots' that happened in the original code base.
We fought off a complete re-write for a critical component in one of our products for many months due to knowing the amount of effort/complexity required, however in the end it was not the best decision to delay. The decision was to work on product features rather than fix the known scalability issues as we believed we still had enough headroom.
We grew faster than expected and ended up in a position where we had exhausted most of our software optimisation options, database tweaks and even hardware options with upgrading to SSD's, RAM etc.
In the end the component was re-written around the clock by a few engineers to leverage redis and msmq in a couple of months and it solved all our problems thanfully - a very stressful time.
Lesson learnt. It was a difficult call when managing a limited set of resources deciding where to focus the attention.
I've done a few rewrites, in all cases they were projects that were written entirely by junior developers (company took a while to learn the costs of cheap developers) and with absolutely no tests whatsoever.
If you're rewriting a project then you're probably going to have to bin the unit tests. If you're rewriting it's because you want to make some drastic structural changes and unit tests are just too tightly coupled. Integation tests on the other hand can be an absolute godsend, you can swap out the entire stack and they'll still provide just as much value as before.
If you are starting with some new language or technology that you are not very fluent at, and months later you land in that same project with a wider experience in the language, would you rewrite that code?
If you're reading about surviving a ground-up rewrite when you're supposed to be _doing_ a ground up rewrite, you're doing it wrong. Read about design patterns, focus on your core business, and stop trying to focus on the "five things to make your rewrite successful".
[+] [-] freework|13 years ago|reply
On the other hand, if you're saying to yourself "This code really sucks, they wrote their own complex CSV parser, I can rewrite this to use the CSV library in the standard lib" and things like that, no matter how big the task may seem, it'll work out in the end.
[+] [-] krichman|13 years ago|reply
[+] [-] Joeri|13 years ago|reply
Another danger i ran across were moving goal posts. Long rewrites will typically be interrupted to extend the old system with new features, which increase the effort to rewrite.
[+] [-] philsnow|13 years ago|reply
The gross part is (I'm guessing, obviously) that now each write to this data store incurs an additional write and some cpu load on the data store servers (you mentioned it was done via a trigger). But now the value of finishing the migration that you started is just recovering this extra write and cpu.
[+] [-] InclinedPlane|13 years ago|reply
Here's some ways to do a rewrite well:
Set up the rewrite as an alternate, competing product and let the two compete in the marketplace. This requires having the resources to support developing two such products at the same time using separate teams for each, which is not something many companies can afford. The advantage to doing it this way is that it gives the rewrite time to mature before it's fully replaced the old system. What most people tend to not realize is that mature software with fundamental flaws can often be superior to immature software on a better foundation. Real-world use is orders of magnitude more effective as a QA process than any sort of in-house testing, and always well be, products that have been tried by fire and made the cut should always be respected no matter how horrible their underlying code is.
If you look at a lot of the major switchovers in software you'll see this sort of trend fairly often. For example, consumer Windows took several generations and almost an entire decade to fully switch over to the NT core.
Another good way is to rewrite piece-wise, in place (refactoring). There are lots of different ways to do this regardless of architecture, ultimately you're essentially creating a bubble of new code that grows and grows until it completely replaces all the old code. The advantage of this is that it takes much less effort than a full rewrite and you can do it incrementally and slowly. Often it's easier to mature each new component in isolation than to try to mature an entire new product.
And, of course, there's the hybrid approach. Work component-wise in an "add + deprecate" fashion. Create the new component, have it work in parallel along with the old component, and then deprecate the old component and work toward moving client code to the new component.
[+] [-] johnbellone|13 years ago|reply
This is definitely the ideal situation, but as you said most companies do not have the resources to greenfield a separate project. Additionally convincing the higher-ups that its worth it is a difficult task.
> Another good way is to rewrite piece-wise, in place (refactoring). There are lots of different ways to do this regardless of architecture, ultimately you're essentially creating a bubble of new code that grows and grows until it completely replaces all the old code.
> Work component-wise in an "add + deprecate" fashion.
Everywhere that I have worked this has been the route we've taken. We isolated the old behavior until we were able to fully ween clients off the product (or perform a data conversion and completely use the new code).
[+] [-] sps_jp|13 years ago|reply
[+] [-] 16s|13 years ago|reply
Competent programmers can maintain other people's code, those who cannot want to re-write everything. Train wreck in progress. Run from these people, or fire them if you're in a position to do so.
[+] [-] sergiotapia|13 years ago|reply
I'm talking 40, 2000 LoC files that all called the database whenever they felt like it, all echoed out data and modifed each other in wierd ways.
I spent about two weeks trying to untangle the mess and threw the towerl in at day 15. We just rewrote it using a PHP framework that not only let us query the database using a sane ORM, but also gave the project STRUCTURE and added mantainability value.
Sometimes a rewrite is the best cure. ;)
[+] [-] tomjen3|13 years ago|reply
Probably why most big companies have such a horrible time getting good people to work for them: who really wants to work in a place like that if you have options somewhere else?
[+] [-] eli|13 years ago|reply
[+] [-] jonsterling|13 years ago|reply
But then again, sometimes, the problem is just that the legacy code is shit. In a project last year, for instance, I had to rewrite every class I touched; the whole thing was so fragile that it was impossible to change anything without the rest breaking. So, I did this “incrementally” rather than all at once, and factored this into the cost of the features I was implementing for the client.
[+] [-] PixelPusher|13 years ago|reply
Legacy systems have gone through so many fixes and shifts that it's almost impossible to recreate them without deep understanding of the business demands it meets.
The business part, I would say, is critical to being able to maintain (or re-write) a project.
[+] [-] dllthomas|13 years ago|reply
[+] [-] katbyte|13 years ago|reply
[+] [-] ams6110|13 years ago|reply
Immediately brought to mind the first large shop I worked in out of school (about 1992) where they had a big chart on one wall titled "Punch Card Elimination" with a downward trending line. I was actually shocked that a large, multinational manufacturer was using punch cards for anything, particularly when my project involved building cutting-edge (at the time) client/server systems that included wireless handheld devices on the client side. My introduction to the counterintuitive world of enterprise IT: you will often see opposite ends of the technology spectrum in use, simply because the value proposition of eliminating the old was never high enough before.
[+] [-] derefr|13 years ago|reply
What about when what you're rewriting is basically an intensely-overgrown prototype that was never meant to run in production in the first place?
You're supposed to completely rewrite those, aren't you?
[+] [-] stormbrew|13 years ago|reply
[+] [-] unknown|13 years ago|reply
[deleted]
[+] [-] dkuebric|13 years ago|reply
[+] [-] Mz|13 years ago|reply
I am wondering what sorts of info I need to go looking for to get this finished. I have thought about it before, contemplated asking around, and realized I didn't even really know what questions I needed to ask.
I also found this, which struck me as helpful but not quite what I wanted to read about: https://www.linkedin.com/today/post/article/20121226171356-6...
[+] [-] johnbellone|13 years ago|reply
Most of the time the Wordpress template is so wonky that its better to just rewrite it (especially if you're migrating away from Wordpress in the first place).
[+] [-] chiph|13 years ago|reply
In the past, we used subsets of data to test our migration tool. Always. ALWAYS, there was something bizarre in the the excluded subset that caused the migration to break.
[+] [-] philsnow|13 years ago|reply
[+] [-] skastel|13 years ago|reply
[+] [-] ldng|13 years ago|reply
When you have a PHP project where there's a mix of procedural function (the code is at the top and the html at the bottom, so it's easy to split right ? Hum and those includes, what do they do exactly ?), ... classes (We used classes to put our functions in so we do OOP, right ? Err... no) and parts of a heavily customized Wordpress. Plus two-three half started in-code reorg/rewrite that went nowhere because the devs jumped the ship.
You're told you have to I18N the whole spaghetti meatball soup. You try a few weeks and then see you're going nowhere and start to realize that you're doomed to a rewrite. Oh, and you're a team of two. Sometimes you really have to. And migrating data .... such a pain in the ass ! Test encoding I'm looking at you ! At of hand int code (would not call that enum) I'm looking at you too.
But. It's not always the case. And right now I'm starting the reorganize and refactoring a project that was already in a descent shape. And it's much more interesting, less straining and business owner have more visibility on the work done and are happier.
So yes, whenever you can, incremental rewrite is the way to go. But never say never. There are contexts where doing a rewrite is just plain necessary because the code is in such a bad shape you just can't add any feature anymore.
[+] [-] purephase|13 years ago|reply
While successfully in production, the web servers were crashing multiple times a day (still Apache-modphp). So, I started to work on the PHP upgrade from 4-5 to get onto php-fpm + Nginx to stabilize the site. Lo and behold, the legacy db abstraction code (poor man's ORM) was not playing well with later PHP releases so I started down the path to patch what I could.
I successfully upgraded PHP, fixed a few of their outstanding issues, but I was weeks into this project and had not completed any of the new feature sets on my plate yet. Looking into the horizon I imagined months of wrestling with legacy code and the deeper I went (there was a SOAP layer I hadn't even looked at yet, they wanted REST/JSON) the more worried I became.
We evaluated a bunch of options and opted for a re-write. We're still deep into it but there is light at the end of the tunnel and I'm pretty sure it will be worth it in the end.
[+] [-] jacques_chester|13 years ago|reply
The analogy used is of a ship at sea. Replacing the ship in a single step, while it is underway, is impossible. Replacing too many planks at once is foolhardy. (I thought the analogy was Oakeshott's but I can't find the original source).
So instead you replace the planks one at a time. Yes, this is tremendously slow, wasteful and frustrating. But when the ship is a system on which you utterly rely, there is no other safe path.
Hayek expanded on the problems of pure rationalism more generally; it's a trap we as a profession tend to fall into. We substitute our preferences (for a tidy, orderly system) for an appraisal of actual reality. Real systems have accumulated complexity that cannot be wished away. Worse: the complexity is often essential and not merely accidental.
[+] [-] PixelPusher|13 years ago|reply
What the author describes sounds like a complete nightmare.
On the projects I've worked on, I would like to think that a rewrite will not be needed.
If it does, it will be because we've now understood the problem space so well that we can start from scratch and avoid all the 'pivots' that happened in the original code base.
[+] [-] nugz|13 years ago|reply
[+] [-] japhyr|13 years ago|reply
If so, do you rewrite those tests? Or do you keep them, no matter how ugly they are, because new code still has to pass all the old tests?
[+] [-] ollysb|13 years ago|reply
If you're rewriting a project then you're probably going to have to bin the unit tests. If you're rewriting it's because you want to make some drastic structural changes and unit tests are just too tightly coupled. Integation tests on the other hand can be an absolute godsend, you can swap out the entire stack and they'll still provide just as much value as before.
[+] [-] rlbisbe|13 years ago|reply
[+] [-] ukandy|13 years ago|reply
Get your data source in order, then refactor in small achievable projects from there. Better for motivation and better for end users.
[+] [-] bengotow|13 years ago|reply
[+] [-] k3n|13 years ago|reply
edit: unflagged after modifications
[+] [-] dshah|13 years ago|reply
I think the content is really good, so title changed -- don't think the author will mind (he had two alternate titles for the original post).
[+] [-] unknown|13 years ago|reply
[deleted]
[+] [-] unknown|13 years ago|reply
[deleted]