> It’s hard to describe the experience of doing a major refactoring, having a really helpful and friendly compiler guide you along step by step and when finally everything compiles it just works.
A truly large refactoring is one that you want to try long before getting everything to compile.
"Hmm, I have 75 operations involving data structure A. I want it to be B. Let's try it with five operations; I know that such and such features of the program only use those. Let's build the inconsistent program and try it out with with the B structure to the extent that it is possible."
The dynamic world lets us have inconsistent code that partially works, rather than forcing us to follow up on all the places reached by the ripple effect of a change. Maybe the change turns out to be a bad idea in the end; then it was a waste of effort.
Psychologically, having to deal with a big ripple effect is discouraging; it creates an "activation potential barrier" which opposes change.
This is why Haskell recently got a flag to turn type errors into warnings: now you came make a change, get a bunch of type errors (as warnings!) and fix them one at a time, loading and testing each part as you change it. You don't have to update everything to play around with your module.
This isn't dynamic typing though: if you tried to use an expression that did not typecheck, you will get a runtime value immediately. It's the moral equivalent of automatically replacing the broken expression with an `undefined` exception.
> A truly large refactoring is one that you want to try long before getting everything to compile.
So I thought this for a good long while - and then I started coding in Rust on a major project. Turns out that once I'd written my serialization/deserialization functions on the boundaries of my program, it's remarkably difficult to get the rest of my code to compile and not do what I was intending it to do.
It's not magic - the compiler doesn't guess at what I want - but it's close enough that I'm comfortable making larger changes than with Python.
> A truly large refactoring is one that you want to try long before getting everything to compile.
That's a luxury that may not be available :-)
Here's a case study of industrial use of a refactoring tool provided with a dynamically typed language:
…
"A very large Smalltalk application was developed at Cargill to support the operation of grain elevators and the associated commodity trading activities. The Smalltalk client application has 385 windows and over 5,000 classes. About 2,000 classes in this application interacted with an early (circa 1993) data access framework. The framework dynamically performed a mapping of object attributes to data table columns.
…
Analysis showed that although dynamic look up consumed 40% of the client execution time, it was unnecessary.
…
A new data layer interface was developed that required the business class to provide the object attribute to column mapping in an explicitly coded method. Testing showed that this interface was orders of magnitude faster. The issue was how to change the 2,100 business class users of the data layer.
…
A large application under development cannot freeze code while a transformation of an interface is constructed and tested. We had to construct and test the transformations in a parallel branch of the code repository from the main development stream. When the transformation was fully tested, then it was applied to the main code stream in a single operation.
…
Less than 35 bugs were found in the 17,100 changes. All of the bugs were quickly resolved in a three-week period.
…
If the changes were done manually we estimate that it would have taken 8,500 hours, compared with 235 hours to develop the transformation rules.
…
The task was completed in 3% of the expected time by using Rewrite Rules. This is an improvement by a factor of 36."
…
from “Transformation of an application data layer” Will Loew-Blosser OOPSLA 2002
> A truly large refactoring is one that you want to try long before getting everything to compile.
For me, from experience, this is a sign that I or my coworker is creating an unreviewable tarball of a diff that will break something, can't be incrementally rolled back, and may not be easily fixed.
A truly large refactoring is one that I want to break up into individual smaller refactorings, each with a more reviewable diff, aided by automatic refactoring tools that help offload my workload. While it's possible to perform enough static analysis of dynamically typed source code for such tools to exist and aid the programmer, they're less likely to be as rock solid as statically typed source code refactoring tools, for something as simple as a method rename.
You can still do experiments with some adjustment to your work habits: create the new data structure with a different name, try it it out in some places via renames, and if necessary comment things out. Delete the old one and rename back again when you're done.
The one that still annoys me, though, is having to comment out unused imports in Go. I agree they should always be removed before submit, but in the meantime it's annoying. (Tools that automatic remove unused imports are also annoying in their own way.)
Leaving Haskell's deferred type errors aside for a moment; after you are done with your partially working code, what then? Relying on (the surely preexisting) 100% test coverage?
> The dynamic world lets us have inconsistent code that partially works, rather than forcing us to follow up on all the places reached by the ripple effect of a change. Maybe the change turns out to be a bad idea in the end; then it was a waste of effort.
Except you can mostly do this reasoning just using the types. So you're rarely updating everything, you're just updating a few type definitions to see if your idea meshes well, and then you have to do the large refactoring.
If lots of things change though... Couldn't that be a smell of bad boundaries? Like the change would not have been so drastic with the right abstractions?
That part made me chuckle, because it made me realize that a typing zealot is exactly what I've become! I used to be a huge fan of Ruby and I was using it for everything, and now 10 years later I'm a huge fan of Haskell instead, I give talks about how JavaScript is horribly error-prone and how Elm is so much better because not only is it statically-typed, it's also purely-functional, and yadi-yadi-ya.
The transition occurred so smoothly that I didn't notice it. Beware! You might not be a zealot now, but if you continue on this path, you might unintentionally become one too :)
Whatever Scala devotees may believe, I think FP and OOP are irreconcilable opposites which means if you really get into FP you're going to hate doing what the industry demands of you in your day job, ie. OOP. Maybe FP will one day become the dominant paradigm but until then treat it like alcohol - too much and you cease to function in the "real" world. I exposed myself to Rich Hickey's Sermons From The Hammock a few years ago and Clojure just blew my mind. Since then I get a churning feeling in the stomach when I have to dealt with OOP, ie. Ruby, PHP, Python. The effects are irreversible so beware - immutable data mixed with pure functions is a potent drug.
As you said, this is slowly happening to me. I love being able to prototype systems in languages like python and matlab, but at the same time it feels like building on sand, especially when other developers get involved.
Recently while writing some matlab found a great example of the sort of trouble a decent type checking system will save you. The predict method takes a machine learning model, and applies it to some data. The output however, is up to the model, and not all models return the same data type (e.g. column vector of doubles, or cell array of strings), making it a pain in the butt to do any abstraction over models.
I am excited for languages like lbstanza that let you have your cake and eat it too.
> It’s hard to describe the experience of doing a major refactoring, having a really helpful and friendly compiler guide you along step by step and when finally everything compiles it just works.
This is my experience in a nutshell, my "home language" has been Python since high school after trying to learn C/C++ and persisted through learning Java in college. It was easy to use and get started with, and didn't require a ton of extraneous typing in Vim or looking up class types and returns every time you tried to use something.
For simple projects as a beginner dynamic/implicit typing is wonderful because you need fewer references. Having gone the way of the IDE, however, having good autocomplete, safe refactoring, error flagging before compilation/runtime, and an easy reference for your libraries has made me incredibly efficient in explicitly typed languages. I'm not sure how much longer I'll be using Python as my home language...
To be fair, at least good autocomplete + easy reference is quite possible in an IDE with dynamic languages.
I have it for Common Lisp in emacs for example(it requires running a live repl and having the editor connect to it).
Pre compilation errors are harder, and I personally don't feel safe refactoring in any language unless there are good automated tests. Static typing makes it better, but nowhere near full proof.
Python is apparently the first language you used in anger and now you want to use other things, but that simply doesn't mean that it is unsuitable for experienced people or complex projects.
> I’m convinced that functional programming vs imperative programming is a much more important concern...
Except that you can be functional and imperative, like Clojure, Erlang and even OCaml. Those are functional-imperative languages (you can even be pure and imperative yet not functional, like Esterel and maybe Céu).
[+] [-] kazinator|9 years ago|reply
A truly large refactoring is one that you want to try long before getting everything to compile.
"Hmm, I have 75 operations involving data structure A. I want it to be B. Let's try it with five operations; I know that such and such features of the program only use those. Let's build the inconsistent program and try it out with with the B structure to the extent that it is possible."
The dynamic world lets us have inconsistent code that partially works, rather than forcing us to follow up on all the places reached by the ripple effect of a change. Maybe the change turns out to be a bad idea in the end; then it was a waste of effort.
Psychologically, having to deal with a big ripple effect is discouraging; it creates an "activation potential barrier" which opposes change.
[+] [-] tikhonj|9 years ago|reply
This isn't dynamic typing though: if you tried to use an expression that did not typecheck, you will get a runtime value immediately. It's the moral equivalent of automatically replacing the broken expression with an `undefined` exception.
[+] [-] hacker_9|9 years ago|reply
This is precisely why I have nightmares about refactoring dynamic codebases.
[+] [-] vertex-four|9 years ago|reply
So I thought this for a good long while - and then I started coding in Rust on a major project. Turns out that once I'd written my serialization/deserialization functions on the boundaries of my program, it's remarkably difficult to get the rest of my code to compile and not do what I was intending it to do.
It's not magic - the compiler doesn't guess at what I want - but it's close enough that I'm comfortable making larger changes than with Python.
[+] [-] igouy|9 years ago|reply
That's a luxury that may not be available :-)
Here's a case study of industrial use of a refactoring tool provided with a dynamically typed language:
…
"A very large Smalltalk application was developed at Cargill to support the operation of grain elevators and the associated commodity trading activities. The Smalltalk client application has 385 windows and over 5,000 classes. About 2,000 classes in this application interacted with an early (circa 1993) data access framework. The framework dynamically performed a mapping of object attributes to data table columns.
…
Analysis showed that although dynamic look up consumed 40% of the client execution time, it was unnecessary.
…
A new data layer interface was developed that required the business class to provide the object attribute to column mapping in an explicitly coded method. Testing showed that this interface was orders of magnitude faster. The issue was how to change the 2,100 business class users of the data layer.
…
A large application under development cannot freeze code while a transformation of an interface is constructed and tested. We had to construct and test the transformations in a parallel branch of the code repository from the main development stream. When the transformation was fully tested, then it was applied to the main code stream in a single operation.
…
Less than 35 bugs were found in the 17,100 changes. All of the bugs were quickly resolved in a three-week period.
…
If the changes were done manually we estimate that it would have taken 8,500 hours, compared with 235 hours to develop the transformation rules.
…
The task was completed in 3% of the expected time by using Rewrite Rules. This is an improvement by a factor of 36."
…
from “Transformation of an application data layer” Will Loew-Blosser OOPSLA 2002
[+] [-] MaulingMonkey|9 years ago|reply
For me, from experience, this is a sign that I or my coworker is creating an unreviewable tarball of a diff that will break something, can't be incrementally rolled back, and may not be easily fixed.
A truly large refactoring is one that I want to break up into individual smaller refactorings, each with a more reviewable diff, aided by automatic refactoring tools that help offload my workload. While it's possible to perform enough static analysis of dynamically typed source code for such tools to exist and aid the programmer, they're less likely to be as rock solid as statically typed source code refactoring tools, for something as simple as a method rename.
[+] [-] skybrian|9 years ago|reply
The one that still annoys me, though, is having to comment out unused imports in Go. I agree they should always be removed before submit, but in the meantime it's annoying. (Tools that automatic remove unused imports are also annoying in their own way.)
[+] [-] pka|9 years ago|reply
[+] [-] naasking|9 years ago|reply
Except you can mostly do this reasoning just using the types. So you're rarely updating everything, you're just updating a few type definitions to see if your idea meshes well, and then you have to do the large refactoring.
[+] [-] ionforce|9 years ago|reply
[+] [-] gelisam|9 years ago|reply
That part made me chuckle, because it made me realize that a typing zealot is exactly what I've become! I used to be a huge fan of Ruby and I was using it for everything, and now 10 years later I'm a huge fan of Haskell instead, I give talks about how JavaScript is horribly error-prone and how Elm is so much better because not only is it statically-typed, it's also purely-functional, and yadi-yadi-ya.
The transition occurred so smoothly that I didn't notice it. Beware! You might not be a zealot now, but if you continue on this path, you might unintentionally become one too :)
[+] [-] cutler|9 years ago|reply
[+] [-] jlgray|9 years ago|reply
Recently while writing some matlab found a great example of the sort of trouble a decent type checking system will save you. The predict method takes a machine learning model, and applies it to some data. The output however, is up to the model, and not all models return the same data type (e.g. column vector of doubles, or cell array of strings), making it a pain in the butt to do any abstraction over models.
I am excited for languages like lbstanza that let you have your cake and eat it too.
[+] [-] Programmatic|9 years ago|reply
This is my experience in a nutshell, my "home language" has been Python since high school after trying to learn C/C++ and persisted through learning Java in college. It was easy to use and get started with, and didn't require a ton of extraneous typing in Vim or looking up class types and returns every time you tried to use something.
For simple projects as a beginner dynamic/implicit typing is wonderful because you need fewer references. Having gone the way of the IDE, however, having good autocomplete, safe refactoring, error flagging before compilation/runtime, and an easy reference for your libraries has made me incredibly efficient in explicitly typed languages. I'm not sure how much longer I'll be using Python as my home language...
[+] [-] mordocai|9 years ago|reply
I have it for Common Lisp in emacs for example(it requires running a live repl and having the editor connect to it).
Pre compilation errors are harder, and I personally don't feel safe refactoring in any language unless there are good automated tests. Static typing makes it better, but nowhere near full proof.
[+] [-] RodericDay|9 years ago|reply
[+] [-] pekk|9 years ago|reply
[+] [-] pron|9 years ago|reply
Except that you can be functional and imperative, like Clojure, Erlang and even OCaml. Those are functional-imperative languages (you can even be pure and imperative yet not functional, like Esterel and maybe Céu).