It's easy to call Excel the cause, but it's not, any more than the existence of memory management is the cause of memory leaks.
Putting a very powerful (if archaic) tool in the hands of people without the experience or understanding to use it effectively and then passing around the results for editing by other people with similarly (or differently) deficient ability with the program is the problem.
The tool is simply too flexible to be used for many of the things it's doing by many of the people who are using it. This is one of the good reasons that we (application developers) do our best to gather requirements: Excel is usually good at what it does, but it does so many things that understanding what it does that you don't need it to do, or how and why it does things you've accidentally done or need to watch out for, has become as -- if not more -- important as/than understanding what it was you wanted to do in the first place.
So yes... Excel plays a role in some very bad things. But no, it's not Excel's fault.
Are people taking this article seriously? I'm fairly certain that it's a tongue-in-cheek jab at various other articles recently blaming Excel for the Reinhart-Rogoff thing.
I don't know how anyone could see the last bit about Native Americans and think that the article was actually serious.
Excel is part of the cause. It simply mangles and hides programming. So it creates just the atmosphere were people think they can "avoid" programming and use Excel. Not realising that they are programming but in a very bad way.
So don't try to hide and dumb down the programming part. This of course doesn't mean that mistakes wouldn't happen with better tools.
This reminds me of Edward Tufte's criticisms of PowerPoint [1,2]. Both PowerPoint and Excel are widely abused, in part due to their flexibility and initial ease of use.
However, in addition to problems caused by its users, there have also been problems resulting from design deficiencies in Excel: automatically and irreversibly converting strings representing gene names into dates [3], displaying some numbers incorrectly [4], and incorrectly evaluating some statistical procedures [5].
"Too flexible" sounds like someone implemented all the requirements. A programming language is too flexible too. That's not really the problem, which is more about the opaqueness and lack of DRY.
In the end it's always "pilot error", right? But some blame rests on Excel for creating a tool that engendered a level of confidence that was unwarranted.
Excel is designed to be used for mission critical applications under certain circumstances. And it does a piss-poor job of delineating what those circumstances are, it deserves a lot of criticism for encouraging people to misuse it.
The problem with Reinhardt-Rogoff has nothing to do with Excel. It was a programing error that could have happened anywhere. The problem is that it took them years to reluctantly share their spreadsheet. Open data research and computation is the solution to that problem.
It was one of many problems. The biggest problem, in my opinion, was excluding some years from their analysis without any explanation or reasoning behind why they did so. Again, peer review would have crushed this paper before it got out the door.
The really amazing thing is that the authors have not retracted the study, and, last I heard, they claimed that their analysis was correct in spite of the errors and exclusion of data which did not support their thesis.
Once again: this business with Excel mistakes seems like a red herring. The bigger problem is that the Reinhart-Rogoff result never made much sense to begin with. Correlation isn't causation; in fact, there's a strong intuitive case that it's the reverse with debt and growth.
Yup, but the Excel error is the soundbite. Anyone can understand it and it makes the authors look ridiculous, which is so much more potent than just being wrong, especially if what you're wrong about is something abstruse that in the end will only get reported as "economists disagree".
I've often wondered about this aspect of political discourse. It's not rational, it's symbolic. Yet most of the time when a symbolic soundbite takes off, even though it's wrong or misleading per se, there's still some poetic justice to it. That is, it's rare that such a soundbite is completely unfair; usually it turns out to be a metonym for an arguably legitimate criticism. This is a case of that.
Both situations are intuitive. High debt causes high servicing of debt causes inefficient use of private market resources paid in taxes.
Additionally, slow growth causes lower govt revenue causes less incurring of debt to make up the shortfall in tax base.
Debt and growth are both inputs and outputs that are codependent. The degree to which they impact the overall financial picture depends HEAVILY upon the opportunity cost of the servicing of the incurred debt.
For bonus, consider what will happen to heavily indebted countries when interest rates begin to rise. Servicing of debt will become more difficult leading to a decrease in our ability to grow the economy, leading to more difficulty in servicing the debt, and on and on.
Obviously, Excel isn't exactly at fault -- it's the people using it, as echoed by many comments here.
HOWEVER, spreadsheets are rather unique in the fact that they hide the very formulas used to generate everything, unless you have a specific cell selected.
So, opposite of computer programs, they're almost impossible to "read through" and verify that everything makes sense. If 99 cells have identical formulas, and 1 in the middle has a slight typo, how on earth are you ever going to see that?
I wonder if there are other similar spreadsheet-like tools that one might use, that make a point of showing everything going on in the spreadsheet, rather than trying to hide it?
If we want to enter 99 identical (or relatively identical) formulas into 99 cells, then we should be able to select all 99 cells, and enter the formula once without any dragging. If we want to change that formula, we should be able to change it once and not need to remember to redrag. We should be able to enter a formula into the column heading and know it'll apply whenever relevant to every cell in a column. That we can't means Excel (and Lotus and OpenOffice) is defective.
What QA is to blame for isn't letting unfinished spreadsheets out the door, but rather for letting users in a company use defective tools like Excel and Lotus and OpenOffice in the first place.
Many of us developers learnt the hard way that visual programming tools from the 1990's don't work, which is why scripting languages became popular. Users in corporations (and the IT staff who authorize what tools that can use) need learn the same lesson.
After reading your comment, I did some googling to find something that could convert an existing spreadsheet into code. I found a few commercial solutions that turn Excel into C++ or Java. I think the idea is that Excel becomes the prototyping tool for non-coders. That seems like a better idea than treating Excel as the final product. I feel like there's a new paradigm for both spreadsheets and programming in there somewhere but I need a bigger brain to see it.
Related to this is that there are no descriptive variable names either. Just a massive three dimensional grid of, non typesafe, memory indexed by letter and number: sheet/row/column.
Imagine programming like this:
A5 = (C6*C6 * Sheet5:D1 + Sheet6:D1 / B3)
I'm not saying you have to use excel like this, there are options to do for example Sum(myTable[myRow]) and pivottable reduces alot of manual equations also. However both of these alternatives require that you format everything nicely into one table per sheet like a proper database and use the "Format as table" option, you almost need a DBA to use excel properly! The problem is that normal people don't use excel like this, they paste random data around everywhere, they want to have the sum, the average and lots of other calculations both below and on the right side of the data so you can see both simultaneously but once you do this you've screwed up your table and we're back on square one.
If you've got 99 cells with identical formulas, and one without, it's not far fetched for Excel to speculate that there may be a mistake (particularly if they're within X% of being identical). At which point it can provide a simple, non-intrusive hint.
And going further up the wishful thinking tree, into more difficult territory, Excel should be able to grasp what you're attempting to do by understanding, say, the top 10,000 (arbitrary number) tasks commonly performed by people using spreadsheets. That 'understanding' could then tip Excel toward deciding whether there's a problem, and whether it should leave you a nice little illuminated icon at the top right of the program with an exclamation mark.
Microsoft should have insane amounts of data on everything people do with spreadsheets, and they should be able to make it radically more intelligent (without being obnoxious or intrusive - it should be optional and highly passive).
HOWEVER, spreadsheets are rather unique in the fact that they hide the very formulas used to generate everything, unless you have a specific cell selected.
Unless you click the button to "show formulas" in Excel, then you can view them all at once.
I got screwed again by my damn notepad. I needed a quart of buttermilk for a recipe, but wrote a pint instead on the memo pad.
If only I had used a recipe management package that would have automatically told me that I wouldn't have enough buttermilk to make waffles for 5 kids.
Excel is an incredibly powerful environment to code. I learned it many years ago; I started thinking it was some arcane tool that accountants used and ended being able to deliver amazingly powerful tools to users orders of magnitude faster than systems developers. Yes they had bugs, but... so did the industrial systems!
Excel's real limitations came around scalability of developers (beyond one developer you are in a bad place) and performance that can drop of a cliff beyond a certain size. But it is an astonishingly powerful tool.
Let us all stop pretending that people were pursuing austerity because of some paper by Reinhart and Rogoff. Cutting government services to citizens for the sake of repaying foreign debt has been going on as long as there has been foreign debt. And that is longer than three years.
Reinhart and Rogoff were important because their idea supported an existing financial theory. If their paper had reached an opposite conclusion, there still would have been austerity.
Now that their paper has been shown to be in error, there will still be austerity. And that will be because the people who benefit from it have the power to impose it. Where they don't have the power, it will not be imposed.
In bioinformatics work, Excel auto-formatting can screw up genomic data http://nsaunders.wordpress.com/2012/10/22/gene-name-errors-a... . The problem with it is that formatting will modify entered text by default, and the user has to be actively aware of this behavior to undo it. Regardless, spreadsheets are invaluable for visualizing and processing tabular data -- but default behavior and user-end awareness are the lessons here.
When I started my thesis there was a graph passed around essentially outlining the difference between Latex and traditional word processing, and it essentially looked like http://bit.ly/ZAtwNi, with word processing packages starting off easier but exponentially getting more difficult to use and Latex being harder to start with, but easier in the long run.
I feel like the same thing applies to Excel. Essentially what gets built with Excel are small programs where all the variables are unnamed, everything passed around is a numerical value and there's almost no capacity to see the entire 'program'.
These always get built because Excel is easier to start with (and you can always pass an Excel spreadsheet around and let other people 'run' it), but it means in the long run with the bigger programs you hit exponential complexity pretty quickly and by then you're invested in Excel, even though R or Pandas would have been more appropriate for the size of the program. Just my two cents (coming from businesses that LOVE excel for completely insane purposes).
From a recent Nassim Taleb facebook posting: "Rejecting a macroeconomic idea (Rogoff and Reinhard) over an excel error is exactly like falsifying astrology over a computer glitch."
Yeah, whatever. The only thing worse than Excel is the typical corporate-developed big Java application produced by a bloated team of mediocre programmers grown to further some middle manager's empire-building desire rather than to produce quality software. I'm not surprised that people doing organizationally key work give up on interfacing with little wannabe-CIOs counting subordinates and just get the work done with a spreadsheet.
Not to mention Excel often ends up having better performance than most home-grown solutions, especially now its core functions are properly multithreaded.
The problem is people being satisfied with a free application for doing mission critical calculations. Yes, you buy an MS Office licence and Excel comes free. That is fine for run of the mill everyday calculations, but it is not OK for mission critical work and it is NOT ok for the CFO or anyone who advises the CFO on his decisions. Those important financial people should use a spreadsheet like this http://www.resolversystems.com/products/resolver-one/ backed up by a team of software developers who write unit tests for the spreadsheets and validate the calculations.
TLDR is that the ResolverONE spreadsheet writes Python code as you manipulate the spreadsheet. A software developer can then organize that Python code, write unit tests for the code and validate that the calculations do what the CFO intended. Once this extra work is done, then the business can have confidence in the results of the calculations. But the user interface, i.e. the spreadsheet that end users see, is the same as it was originally.
This allows mission critical software development discipline to be applied to the development of spreadsheets. I suspect that the existence of a tool like ResolverONE is one of the factors that led the SEC in the USA to require Python source code for new derivatives that are created, so that there is a standard way to express the calculations of the derivative.
I'd say Excel is superior to those systems for any one-person project. Especially monsters like Bugzilla, which shine for huge projects but impose a lot of overhead on tiny projects.
I look at excel like a car. What's happening is that most people who use don't have a license. What's worse they've mainly learned to drive on the farm.
Now rather than hire a professional or take performance driving courses they try and do the same in high end usage and things go wrong.
It's not the sports car that's at fault by someone driving it at the edge of its performance without the proper skill. If the just drove it around town they'd be fine, but taking a high performance car to the track with no experience, well bad things happen. We've all seen youtube clips of high performance crashes from inexperience ;)
I wondered if there might be a market for an Excel correctness checker. If billions are on the line, it might be worth a few bucks. But apparently Excel can advise you of common problems like these already.
Probably that means such an add-on would not be sellable? However it wouldn't be the first time someone paid lots of money for something that software they owned already did just fine.
There is probably not much you could check. You could probably do some type checking but beyond that I see - ad hoc - not much one could do. Excel does already catch syntax errors and a few other errors mentioned in the link. Maybe you could make these errors more prominent. But should we really encourage more development using office software? Shouldn't we solve the actual problem?
The hard part would be designing a machine-readable and illiterate-writable language for formal specification of Excel spreadsheet requirements to validate against. And getting said illiterates to use it.
There is no other way to detect errors involving somebody using SUM instead of AVG or aggregating data from wrong fields (examples from the article).
Use a hammer. Hit your thumb instead of the nail. Blame the hammer.
Use a table saw. Cut off three fingers instead of wood. Blame the table saw.
Go scuba diving. Return to the surface too fast. Get the bends. Blame the equipment.
Drive a car. Have an accident while texting. Blame the phone.
Go fishing. Be careless. Put a hook through your hand. Blame the fishing rod, the reel and the hook.
Use Excel. Don't make an effort to learn what you are doing. Mindlessly copy and paste. Do not seek out peer review. Cause massive financial losses. Blame Excel.
I'm reminded of a similar situation involving treasury secretary Timothy Geithner and TurboTax. He initially blamed the software for underpayment of tax (or at least he said he used turbotax) and later admitted that he fudged the numbers.
Huh, a second ago there was a comment here asking how this article was on the front page... then it was disappeared. Call Alex Jones.
I wanted to voice my thought anyway... this article prominently features the information that the mainly cited study showing that national debt is bad is no longer valid. Those wanting to fight for higher debt levels are absolutely ecstatic and will up-vote any article that even tangentially mentions the Reinhart/Rogoff failure.
[+] [-] obviouslygreen|13 years ago|reply
Putting a very powerful (if archaic) tool in the hands of people without the experience or understanding to use it effectively and then passing around the results for editing by other people with similarly (or differently) deficient ability with the program is the problem.
The tool is simply too flexible to be used for many of the things it's doing by many of the people who are using it. This is one of the good reasons that we (application developers) do our best to gather requirements: Excel is usually good at what it does, but it does so many things that understanding what it does that you don't need it to do, or how and why it does things you've accidentally done or need to watch out for, has become as -- if not more -- important as/than understanding what it was you wanted to do in the first place.
So yes... Excel plays a role in some very bad things. But no, it's not Excel's fault.
[+] [-] Negitivefrags|13 years ago|reply
I don't know how anyone could see the last bit about Native Americans and think that the article was actually serious.
[+] [-] fafner|13 years ago|reply
So don't try to hide and dumb down the programming part. This of course doesn't mean that mistakes wouldn't happen with better tools.
http://www.pages.drexel.edu/~bdm25/excel2007.pdf
[+] [-] jamessb|13 years ago|reply
However, in addition to problems caused by its users, there have also been problems resulting from design deficiencies in Excel: automatically and irreversibly converting strings representing gene names into dates [3], displaying some numbers incorrectly [4], and incorrectly evaluating some statistical procedures [5].
[1] http://www.edwardtufte.com/bboard/q-and-a-fetch-msg?msg_id=0... [2] http://www.wired.com/wired/archive/11.09/ppt2.html [3] http://www.biomedcentral.com/1471-2105/5/80 [4] http://www.joelonsoftware.com/items/2007/09/26b.html [5] http://dl.acm.org/citation.cfm?id=635312
[+] [-] justincormack|13 years ago|reply
[+] [-] InclinedPlane|13 years ago|reply
Excel is designed to be used for mission critical applications under certain circumstances. And it does a piss-poor job of delineating what those circumstances are, it deserves a lot of criticism for encouraging people to misuse it.
[+] [-] pesenti|13 years ago|reply
[+] [-] wyday|13 years ago|reply
[+] [-] inthewoods|13 years ago|reply
The really amazing thing is that the authors have not retracted the study, and, last I heard, they claimed that their analysis was correct in spite of the errors and exclusion of data which did not support their thesis.
[+] [-] tptacek|13 years ago|reply
[+] [-] gruseom|13 years ago|reply
http://krugman.blogs.nytimes.com/2013/04/20/the-good-glitch/
I've often wondered about this aspect of political discourse. It's not rational, it's symbolic. Yet most of the time when a symbolic soundbite takes off, even though it's wrong or misleading per se, there's still some poetic justice to it. That is, it's rare that such a soundbite is completely unfair; usually it turns out to be a metonym for an arguably legitimate criticism. This is a case of that.
[+] [-] sopooneo|13 years ago|reply
[+] [-] crusso|13 years ago|reply
Additionally, slow growth causes lower govt revenue causes less incurring of debt to make up the shortfall in tax base.
Debt and growth are both inputs and outputs that are codependent. The degree to which they impact the overall financial picture depends HEAVILY upon the opportunity cost of the servicing of the incurred debt.
For bonus, consider what will happen to heavily indebted countries when interest rates begin to rise. Servicing of debt will become more difficult leading to a decrease in our ability to grow the economy, leading to more difficulty in servicing the debt, and on and on.
[+] [-] james1071|13 years ago|reply
[+] [-] crazygringo|13 years ago|reply
HOWEVER, spreadsheets are rather unique in the fact that they hide the very formulas used to generate everything, unless you have a specific cell selected.
So, opposite of computer programs, they're almost impossible to "read through" and verify that everything makes sense. If 99 cells have identical formulas, and 1 in the middle has a slight typo, how on earth are you ever going to see that?
I wonder if there are other similar spreadsheet-like tools that one might use, that make a point of showing everything going on in the spreadsheet, rather than trying to hide it?
[+] [-] vorg|13 years ago|reply
If we want to enter 99 identical (or relatively identical) formulas into 99 cells, then we should be able to select all 99 cells, and enter the formula once without any dragging. If we want to change that formula, we should be able to change it once and not need to remember to redrag. We should be able to enter a formula into the column heading and know it'll apply whenever relevant to every cell in a column. That we can't means Excel (and Lotus and OpenOffice) is defective.
What QA is to blame for isn't letting unfinished spreadsheets out the door, but rather for letting users in a company use defective tools like Excel and Lotus and OpenOffice in the first place.
Many of us developers learnt the hard way that visual programming tools from the 1990's don't work, which is why scripting languages became popular. Users in corporations (and the IT staff who authorize what tools that can use) need learn the same lesson.
[+] [-] celticjames|13 years ago|reply
[+] [-] Too|13 years ago|reply
[+] [-] adventured|13 years ago|reply
If you've got 99 cells with identical formulas, and one without, it's not far fetched for Excel to speculate that there may be a mistake (particularly if they're within X% of being identical). At which point it can provide a simple, non-intrusive hint.
And going further up the wishful thinking tree, into more difficult territory, Excel should be able to grasp what you're attempting to do by understanding, say, the top 10,000 (arbitrary number) tasks commonly performed by people using spreadsheets. That 'understanding' could then tip Excel toward deciding whether there's a problem, and whether it should leave you a nice little illuminated icon at the top right of the program with an exclamation mark.
Microsoft should have insane amounts of data on everything people do with spreadsheets, and they should be able to make it radically more intelligent (without being obnoxious or intrusive - it should be optional and highly passive).
[+] [-] jodrellblank|13 years ago|reply
Unless you click the button to "show formulas" in Excel, then you can view them all at once.
http://i237.photobucket.com/albums/ff263/dharma-putra/Excel2...
http://www.lytebyte.com/wp-content/uploads/2008/04/view-form...
[+] [-] Spooky23|13 years ago|reply
If only I had used a recipe management package that would have automatically told me that I wouldn't have enough buttermilk to make waffles for 5 kids.
[+] [-] yummyfajitas|13 years ago|reply
In Scala one can certainly build this. You could certainly do it in Haskell, though it wouldn't be as nice syntactically.
[+] [-] roblev|13 years ago|reply
Excel's real limitations came around scalability of developers (beyond one developer you are in a bad place) and performance that can drop of a cliff beyond a certain size. But it is an astonishingly powerful tool.
[+] [-] brudgers|13 years ago|reply
Reinhart and Rogoff were important because their idea supported an existing financial theory. If their paper had reached an opposite conclusion, there still would have been austerity.
Now that their paper has been shown to be in error, there will still be austerity. And that will be because the people who benefit from it have the power to impose it. Where they don't have the power, it will not be imposed.
[+] [-] CatMtKing|13 years ago|reply
[+] [-] mrmagooey|13 years ago|reply
I feel like the same thing applies to Excel. Essentially what gets built with Excel are small programs where all the variables are unnamed, everything passed around is a numerical value and there's almost no capacity to see the entire 'program'.
These always get built because Excel is easier to start with (and you can always pass an Excel spreadsheet around and let other people 'run' it), but it means in the long run with the bigger programs you hit exponential complexity pretty quickly and by then you're invested in Excel, even though R or Pandas would have been more appropriate for the size of the program. Just my two cents (coming from businesses that LOVE excel for completely insane purposes).
[+] [-] justin66|13 years ago|reply
[+] [-] ddfu|13 years ago|reply
Not to mention Excel often ends up having better performance than most home-grown solutions, especially now its core functions are properly multithreaded.
[+] [-] mbreese|13 years ago|reply
[+] [-] protomyth|13 years ago|reply
[+] [-] memracom|13 years ago|reply
TLDR is that the ResolverONE spreadsheet writes Python code as you manipulate the spreadsheet. A software developer can then organize that Python code, write unit tests for the code and validate that the calculations do what the CFO intended. Once this extra work is done, then the business can have confidence in the results of the calculations. But the user interface, i.e. the spreadsheet that end users see, is the same as it was originally.
This allows mission critical software development discipline to be applied to the development of spreadsheets. I suspect that the existence of a tool like ResolverONE is one of the factors that led the SEC in the USA to require Python source code for new derivatives that are created, so that there is a standard way to express the calculations of the derivative.
[+] [-] crusso|13 years ago|reply
Bug tracking with Jira, Bugzilla, Mantis, or Trac? Heck no, Excel is the tool for the job!
[+] [-] klodolph|13 years ago|reply
[+] [-] gcr|13 years ago|reply
Excel is quite sufficient for many tasks.
[+] [-] anigbrowl|13 years ago|reply
[+] [-] FollowSteph3|13 years ago|reply
Now rather than hire a professional or take performance driving courses they try and do the same in high end usage and things go wrong.
It's not the sports car that's at fault by someone driving it at the edge of its performance without the proper skill. If the just drove it around town they'd be fine, but taking a high performance car to the track with no experience, well bad things happen. We've all seen youtube clips of high performance crashes from inexperience ;)
[+] [-] neilk|13 years ago|reply
http://office.microsoft.com/en-us/excel-help/correct-common-...
Probably that means such an add-on would not be sellable? However it wouldn't be the first time someone paid lots of money for something that software they owned already did just fine.
[+] [-] danbruc|13 years ago|reply
[+] [-] qb45|13 years ago|reply
There is no other way to detect errors involving somebody using SUM instead of AVG or aggregating data from wrong fields (examples from the article).
[+] [-] robomartin|13 years ago|reply
Use a table saw. Cut off three fingers instead of wood. Blame the table saw.
Go scuba diving. Return to the surface too fast. Get the bends. Blame the equipment.
Drive a car. Have an accident while texting. Blame the phone.
Go fishing. Be careless. Put a hook through your hand. Blame the fishing rod, the reel and the hook.
Use Excel. Don't make an effort to learn what you are doing. Mindlessly copy and paste. Do not seek out peer review. Cause massive financial losses. Blame Excel.
Boy, do I like the sound of the word "moron".
[+] [-] niggler|13 years ago|reply
Excel is sometimes the culprit (Excel 97 bug: http://blogs.office.com/b/microsoft-excel/archive/2007/09/25... ), but usually manual error is to blame.
[+] [-] crusso|13 years ago|reply
I wanted to voice my thought anyway... this article prominently features the information that the mainly cited study showing that national debt is bad is no longer valid. Those wanting to fight for higher debt levels are absolutely ecstatic and will up-vote any article that even tangentially mentions the Reinhart/Rogoff failure.
[+] [-] anigbrowl|13 years ago|reply
[+] [-] unknown|13 years ago|reply
[deleted]