As an electrical-engineer turned machine-learning-grad-student, linear transformations have been involved in most everything I do since my first year of undergrad. But all this time, I've done matrix multiplication the way I was taught in high school: "The (i,j) element of AB is what you get by walking right across the i'th row of A while you walk down the j'th column of B, taking the sum of products as you go."
It works, but there's no connection between that process and the intuition of a linear transformation; it's just a rote computation. And checking a long string of matrix multiplications to see if they intuitively make sense (shouldn't everything intuitively make sense?) is especially aggravating when you constantly have to interrupt your intuition to switch to a rote calculation.
I never thought to think of the columns of B as vectors that physically travel through A; to think of a dataflow or pipeline from right to left on the page. Sure, it's not a cure-all, but it'll be a useful mental tool to have.
Oh, and it's also an excellent introduction to the subject, although the Linear Operations section gets a bit muddled... first something's not a linear operation, and then it is, wat? Still, an excellent post.
You might get some mileage out of thinking of a row in the matrix product as a dot product, with the intuition that goes along with that. The dot product is perhaps easier to tie into the geometric intuition you have for linear transformations.
Can't say enough good things about Prof. Gilbert Strang http://www-math.mit.edu/~gs/ He is one of my heros, and yeah, I have copied his Linear Algebra lectures to CD so I can speed them up, stop them, and in case they ever disappear from the net. Not to take away from the OP, who I think has some good ideas for making this stuff intuitive.
I'm watching them now, and they are simply superb, probably the best math class I've ever "taken". I particularly liked his "4 ways of matrix multiplication" (lecture 3) and explanation of fourier series (lecture 24), but his exposition of every concept from determinants to eigenvalues has been intuitive and memorable.
> Linear algebra gives you mini-spreadsheets for your math equations.
Okay, that lets you visualize it (in the finite case) but it's a terrible way to sell it. Spreadsheets are booooring. Did you know that functions are vectors? Okay, better. Did you know that quantum mechanics is all about linear algebra? Okay, sold!
1. Almost any time you work in more than one dimension you will want linear algebra in your toolbox. There are a zillion methods for solving (non-linear) equations out there, and in more than one dimension, they use linear algebra. Newton's method? Incredibly useful in practice due (quadratic convergence rocks!), and with some linear algebra sauce BOOM you have Newton's method in as many dimensions as you can sneeze at.
2. Oh, by the way... did you know that the Fourier transform is linear?
3. Back to quantum mechanics... there's a thing you can do with a linear operator (a matrix is a kind of linear operator) where you get the "spectrum" of the linear operator. It's useful for making sense of big matrices. But in QM, the wavefunctions for electrons are described as eigenfunctions of a linear operator, and taking the "spectrum" of the linear operator gives you the actual spectrum of light that the chemical under study emits. Hence the name, "spectral theorem". It may be linear algebra on paper, but it's laser beams and semiconductors in the real world.
4. Oh hey, want to learn about infinite-dimensional vector spaces? Maybe some other time..
5. It's hella useful for modeling. Any model is wrong, but Markov processes are useful. Say you run an agency that rents out moving vans, and you have facilities in 30 cities. Vans rented in city A have a 10% chance of being dropped off in city B, 7% in city C, 9.2% in city D, etc. At this rate, how long till you run out of vans in city F? It's a differential equations problem with like 30 different equations! Or you could rewrite it as a single equation with matrices. You'll end up with weird things like 'e^(A*t)' where A is a matrix, and you thought "no way I can exponentiate to the power of a matrix" but spectral decomposition is like "yes way!" and you can solve the equation by diagonalization. Radical! (Basically, linear algebra rescues differential equations from the pits of intractability. I'm using rental vans as an example, but it could be a chemical reaction or a nuclear reaction or a million subway riders or whatever you want.)
So the question is:
Do you find economics, quantum physics, chemistry, engineering, classical mechanics, machine learning, statistics, etc. useful?
Hate to do this (but will have to anyway because you're misrepresenting that authors article),
- you've mixed in with nonlinear phenomena (most physical processes) with linear, and there's less and less reason to pretend linearity with increasing computer speeds (although in fairness you alluded to models being wrong)
- poo pooed spreadsheets without explaining why (you alluded to QM etc. but they don't care about useless proofs in linear algebra unless they're pencil pushing time wasters)
- implied infinite-dimensions is practically useful when it isn't (unless you're a mathematician/theoretical-somethingist seeking to extract tax-payer money)
There's little need to get linear algebra in you unless you want to waste time.
I hope that this matches how some people think enough to help them.
For me it is too computational. I prefer understanding the topic from first principles as described at http://news.ycombinator.com/item?id=4086325. (Then again I don't particularly like spreadsheets either.)
After going through that course I finally understood things like eigenvectors, null spaces, and projections. Now I see them everywhere (unless you think that's a curse)
Funny, I'm actually working on a game that intends to teach some of this stuff to a non-mathematical audience.
Linear algebra is so far-reaching, I find it surprising that other branches of mathematics often seem to get preferential treatment (usually normal algebra and geometry), in spite of the fact that linear algebra is both:
a) fairly advanced (i.e. not often taught in school, at least not the deeper stuff)
b) not very difficult to learn (unlike lots of other 'introductory' topics in mathematics).
Perhaps there is something about matrices (being mere tables of numbers for most folks) that people find unattractive, almost statistics-like.
(On the other hand, it could be a simple extension of the symbol barrier [1], given those long vertical brackets.)
"The eigenvectors are the axes of the transformation" = mind blown. After several engineering courses, studying eigenvectors in an advanced math class and still no one could put it this simple. This guy is amazing.
His description of the determinant, too. I hadn't heard that explanation until a second semester of real analysis, when learning the proof of the Inverse Function Theorem (an amazing thing to study, by the way, connects many the dots between linear algebra and calculus).
Even then, it was a question I had to ask my brilliant, constantly-pissed looking young professor. "Hey, uh, the Jacobian... what does the determinant mean, uh, geometrically?". He looked at me like a slug, before explaining it was the measure of the newly mapped unit square. Fireworks went off in my head. Two linear algebra classes before only ever explained it by its algorithm or its usefulness (e.g., ∃ A^(-1) for A \in R^{n,n} iff det(A) != 0)
Side note: that professor had the most effective teaching style for pure math I've ever seen. Besides lectures that expanded on the contents of Rudin and interesting problem sets, he gave us a list of a hundred theorems, propositions, and exercises. Told us the final exam would be six problems, four of which would come from that list, another of which would be a clever new one, the last something truly hard.
Never learned analysis better than when sitting down and working through (not memorizing) each of those proofs and theorems for possible later recapitulation.
What the author uses as his strawman is a linear algebra course for engineers.
As mathematicians, we didn't do any of this matrix and vector stuff with numbers when introducing linear algebra in university. There were a bunch of axioms, and you proved things. That's how you know.
What the author sees as abstract "(2d vectors! 3d vectors!)" was way more applied than the stuff we dealt in.
But, granted, the purpose wasn't learning about how to get mini-spreadsheets for equations. It was about how to rigorously navigate a useful axiomatic setting.
(Later on, we proved that you can find a base, and write down your linear transformation as a bunch of numbers and call that a matrix; same for points and vectors. But we always saw that as somewhat ugly, and anyway limited to the finite dimensional case.)
I had studied it under a similar approach. Of course, after taking Abstract Algebra course, and seeing Polynomials over Abstract Fields, which form a vector space, while computing the degree of finite field extensions[1], I realized how much powerful Linear Algebra was.
Also during a Computational Complexity course, when we were studying de-randomization techniques, computing the Spectrum of an expander graph[2] and looking into the eigenvalues and eigenvectors of the graphs' adjacency matrix to understand topological properties from graphs, I realized how powerful and general Linear Algebra can be.
2d and 3d vectors over the Field of Real Numbers are easy to see, and a good way to get started. But many beautiful and powerful things come from this theory.
Linear Algebra was at the core of almost every math and cs course I took. I ended up taking several semesters of it and by the end we were approaching highly relevant problems in subjects like quantum mechanics and machine learning. I cannot recommend studying it highly enough!
If anyone is interested in building an understanding of Linear Algebra from the ground up, I would suggest some of Israel Gelfand's work: http://www.scribd.com/doc/35985818/Gelfand-Lectures-on-Linea...
It's not for the light hearted, but when you get through it, you will be deeply changed and have a new perspective on even basic math, such as integration.
> What the author uses as his strawman is a linear algebra course for engineers.
If that's true, I'm sorry for the engineers who took this course.
I did not see a single number in my algebra course, neither 2d or 3d vectors. As a matter of fact, we followed the same program that math students (axioms, theorems, linear spaces, rings, ...). Even some teachers were teaching in both faculties, and our exams usually were more difficult (this was just a policy, there were too many engineering students).
Now, I'm in a different university where the maths future engineers study are much more "practical". It's sad seeing that students cannot understand, for example, that a principal direction in a stress tensor is just an eigen-value or that a rotation is just a change of base and, therefore, there are many theorems and properties you can apply to these concepts just making use of some basic algebra.
It's always good to know well the tools you have to use and, for an engineer, algebra is one of the most important tools.
Also, the common 1st year texts (Anton, Lang, Hoffman/Kunze and Friedberg/insel/Spence) can be found easily for cheap, used. The old edition of Strang I used to have was good too, but some people react really strongly when you bring it up. There's lots of ways ot sequence LA and the needs of EE's, econometrics/game theory, prob/stats and applied mathies are different from physics/math majors. Look at ToC's and read the Holyoke prof's writeup:
Other things that might be nice to include are dot products (projecting one vector onto another as a measure of co-linearity) and rotation matrices (you could keep it to 3x3).
For the mathematicians here. I am taking Linear Algebra classes as a CS undergrad. Can someone recommend a very good book?
I am looking for the kind of book that will make you fell in love with Linear Algebra. For Calculus, I used Piskunovs Differential and Integral Calculus which was miles away from what my classmates were using. That along with Maple help to double check that my stuff was correct proved a good combo. My current Linear Algebra book is an honest book but it is a boring book, it fails to entertain or to amaze or to give you those moments of insight that puts a grin in your face.
I think I just wish I had better books as an undergrad :-(
OK can you explain kernels and their role in transformations (f.e. the dimension formula)? No you can't this way. And it truly has an INTUITIVE explanation (so much more than the intuitive ess of a spreadsheet!).
Every now and again I think my highfalutin college courses were an overpriced waste, and then a conversation like this comes along and I see that the fundamentals of my courses are apparently radical and unheard of most other places.
Do schools really not teach the underlying concepts of math, or do people just fail to understand them the first time through and then blame their teachers?
/took linear algebra in the math department, then TAed it.
>Do schools really not teach the underlying concepts of math, or do people just fail to understand them the first time through and then blame their teachers?
It is absolutely the case that a great many students will not remember seeing material that they definitely were exposed to. As someone who's TA'd physics classes for a few years, I'd often ask a recitation whether they'd covered some particular subject yet in lecture, and they'd often say (as a class) no.
I'd ask the professor later, and of course they had -- but students rarely go into lecture having done the reading or prep work, and so have a piss-poor retention rate.
I entirely agree. People tend to run away from abstract math, with the impression that it's difficult. But the whole point is that a more abstract view of these things makes them easier to understand, not harder.
[+] [-] btown|13 years ago|reply
It works, but there's no connection between that process and the intuition of a linear transformation; it's just a rote computation. And checking a long string of matrix multiplications to see if they intuitively make sense (shouldn't everything intuitively make sense?) is especially aggravating when you constantly have to interrupt your intuition to switch to a rote calculation.
I never thought to think of the columns of B as vectors that physically travel through A; to think of a dataflow or pipeline from right to left on the page. Sure, it's not a cure-all, but it'll be a useful mental tool to have.
Oh, and it's also an excellent introduction to the subject, although the Linear Operations section gets a bit muddled... first something's not a linear operation, and then it is, wat? Still, an excellent post.
[+] [-] wging|13 years ago|reply
[+] [-] valgaze|13 years ago|reply
[+] [-] jackfoxy|13 years ago|reply
[+] [-] fferen|13 years ago|reply
[+] [-] klodolph|13 years ago|reply
Okay, that lets you visualize it (in the finite case) but it's a terrible way to sell it. Spreadsheets are booooring. Did you know that functions are vectors? Okay, better. Did you know that quantum mechanics is all about linear algebra? Okay, sold!
1. Almost any time you work in more than one dimension you will want linear algebra in your toolbox. There are a zillion methods for solving (non-linear) equations out there, and in more than one dimension, they use linear algebra. Newton's method? Incredibly useful in practice due (quadratic convergence rocks!), and with some linear algebra sauce BOOM you have Newton's method in as many dimensions as you can sneeze at.
2. Oh, by the way... did you know that the Fourier transform is linear?
3. Back to quantum mechanics... there's a thing you can do with a linear operator (a matrix is a kind of linear operator) where you get the "spectrum" of the linear operator. It's useful for making sense of big matrices. But in QM, the wavefunctions for electrons are described as eigenfunctions of a linear operator, and taking the "spectrum" of the linear operator gives you the actual spectrum of light that the chemical under study emits. Hence the name, "spectral theorem". It may be linear algebra on paper, but it's laser beams and semiconductors in the real world.
4. Oh hey, want to learn about infinite-dimensional vector spaces? Maybe some other time..
5. It's hella useful for modeling. Any model is wrong, but Markov processes are useful. Say you run an agency that rents out moving vans, and you have facilities in 30 cities. Vans rented in city A have a 10% chance of being dropped off in city B, 7% in city C, 9.2% in city D, etc. At this rate, how long till you run out of vans in city F? It's a differential equations problem with like 30 different equations! Or you could rewrite it as a single equation with matrices. You'll end up with weird things like 'e^(A*t)' where A is a matrix, and you thought "no way I can exponentiate to the power of a matrix" but spectral decomposition is like "yes way!" and you can solve the equation by diagonalization. Radical! (Basically, linear algebra rescues differential equations from the pits of intractability. I'm using rental vans as an example, but it could be a chemical reaction or a nuclear reaction or a million subway riders or whatever you want.)
So the question is:
Do you find economics, quantum physics, chemistry, engineering, classical mechanics, machine learning, statistics, etc. useful?
Then get some linear algebra in you!
[+] [-] marshallp|13 years ago|reply
- you've mixed in with nonlinear phenomena (most physical processes) with linear, and there's less and less reason to pretend linearity with increasing computer speeds (although in fairness you alluded to models being wrong)
- poo pooed spreadsheets without explaining why (you alluded to QM etc. but they don't care about useless proofs in linear algebra unless they're pencil pushing time wasters)
- implied infinite-dimensions is practically useful when it isn't (unless you're a mathematician/theoretical-somethingist seeking to extract tax-payer money)
There's little need to get linear algebra in you unless you want to waste time.
[+] [-] btilly|13 years ago|reply
For me it is too computational. I prefer understanding the topic from first principles as described at http://news.ycombinator.com/item?id=4086325. (Then again I don't particularly like spreadsheets either.)
[+] [-] lomendil|13 years ago|reply
http://linear.axler.net/
After going through that course I finally understood things like eigenvectors, null spaces, and projections. Now I see them everywhere (unless you think that's a curse)
[+] [-] hdivider|13 years ago|reply
Linear algebra is so far-reaching, I find it surprising that other branches of mathematics often seem to get preferential treatment (usually normal algebra and geometry), in spite of the fact that linear algebra is both:
a) fairly advanced (i.e. not often taught in school, at least not the deeper stuff)
b) not very difficult to learn (unlike lots of other 'introductory' topics in mathematics).
Perhaps there is something about matrices (being mere tables of numbers for most folks) that people find unattractive, almost statistics-like.
(On the other hand, it could be a simple extension of the symbol barrier [1], given those long vertical brackets.)
[1] Prof Keith Devlin introduces this concept here: http://profkeithdevlin.org/2012/02/22/how-to-design-video-ga...
[+] [-] elteto|13 years ago|reply
[+] [-] textminer|13 years ago|reply
Even then, it was a question I had to ask my brilliant, constantly-pissed looking young professor. "Hey, uh, the Jacobian... what does the determinant mean, uh, geometrically?". He looked at me like a slug, before explaining it was the measure of the newly mapped unit square. Fireworks went off in my head. Two linear algebra classes before only ever explained it by its algorithm or its usefulness (e.g., ∃ A^(-1) for A \in R^{n,n} iff det(A) != 0)
Side note: that professor had the most effective teaching style for pure math I've ever seen. Besides lectures that expanded on the contents of Rudin and interesting problem sets, he gave us a list of a hundred theorems, propositions, and exercises. Told us the final exam would be six problems, four of which would come from that list, another of which would be a clever new one, the last something truly hard.
Never learned analysis better than when sitting down and working through (not memorizing) each of those proofs and theorems for possible later recapitulation.
[+] [-] tomrod|13 years ago|reply
[+] [-] sadga|13 years ago|reply
[+] [-] eru|13 years ago|reply
As mathematicians, we didn't do any of this matrix and vector stuff with numbers when introducing linear algebra in university. There were a bunch of axioms, and you proved things. That's how you know.
What the author sees as abstract "(2d vectors! 3d vectors!)" was way more applied than the stuff we dealt in.
But, granted, the purpose wasn't learning about how to get mini-spreadsheets for equations. It was about how to rigorously navigate a useful axiomatic setting.
(Later on, we proved that you can find a base, and write down your linear transformation as a bunch of numbers and call that a matrix; same for points and vectors. But we always saw that as somewhat ugly, and anyway limited to the finite dimensional case.)
[+] [-] DanielRibeiro|13 years ago|reply
Also during a Computational Complexity course, when we were studying de-randomization techniques, computing the Spectrum of an expander graph[2] and looking into the eigenvalues and eigenvectors of the graphs' adjacency matrix to understand topological properties from graphs, I realized how powerful and general Linear Algebra can be.
2d and 3d vectors over the Field of Real Numbers are easy to see, and a good way to get started. But many beautiful and powerful things come from this theory.
[1] http://en.wikipedia.org/wiki/Degree_of_a_field_extension
[2] http://en.wikipedia.org/wiki/Expander_graph
[+] [-] sidupadhyay|13 years ago|reply
If anyone is interested in building an understanding of Linear Algebra from the ground up, I would suggest some of Israel Gelfand's work: http://www.scribd.com/doc/35985818/Gelfand-Lectures-on-Linea... It's not for the light hearted, but when you get through it, you will be deeply changed and have a new perspective on even basic math, such as integration.
[+] [-] yiyus|13 years ago|reply
If that's true, I'm sorry for the engineers who took this course.
I did not see a single number in my algebra course, neither 2d or 3d vectors. As a matter of fact, we followed the same program that math students (axioms, theorems, linear spaces, rings, ...). Even some teachers were teaching in both faculties, and our exams usually were more difficult (this was just a policy, there were too many engineering students).
Now, I'm in a different university where the maths future engineers study are much more "practical". It's sad seeing that students cannot understand, for example, that a principal direction in a stress tensor is just an eigen-value or that a rotation is just a change of base and, therefore, there are many theorems and properties you can apply to these concepts just making use of some basic algebra.
It's always good to know well the tools you have to use and, for an engineer, algebra is one of the most important tools.
[+] [-] marshallp|13 years ago|reply
edit: downvoters downvote instead of providing refutation. A lot in common with fundamental religionists.
[+] [-] ced|13 years ago|reply
[+] [-] achompas|13 years ago|reply
Thanks for the link!
[+] [-] gtani|13 years ago|reply
http://linear.ups.edu/xml/latest/fcla-xml-latest.xml#fcla-xm...
http://joshua.smcvt.edu/linearalgebra/book.pdf
Also, the common 1st year texts (Anton, Lang, Hoffman/Kunze and Friedberg/insel/Spence) can be found easily for cheap, used. The old edition of Strang I used to have was good too, but some people react really strongly when you bring it up. There's lots of ways ot sequence LA and the needs of EE's, econometrics/game theory, prob/stats and applied mathies are different from physics/math majors. Look at ToC's and read the Holyoke prof's writeup:
http://www.amazon.com/Introduction-Linear-Algebra-Fourth-Gil...
(i think they're 1st year texts, my Dad's a physics prof and he started talking about determinants around 7th grade)
[+] [-] mturmon|13 years ago|reply
Other things that might be nice to include are dot products (projecting one vector onto another as a measure of co-linearity) and rotation matrices (you could keep it to 3x3).
[+] [-] textminer|13 years ago|reply
[+] [-] unknown|13 years ago|reply
[deleted]
[+] [-] sadga|13 years ago|reply
[+] [-] dsrguru|13 years ago|reply
http://www.cns.nyu.edu/~eero/NOTES/geomLinAlg.pdf
[+] [-] soapdog|13 years ago|reply
I am looking for the kind of book that will make you fell in love with Linear Algebra. For Calculus, I used Piskunovs Differential and Integral Calculus which was miles away from what my classmates were using. That along with Maple help to double check that my stuff was correct proved a good combo. My current Linear Algebra book is an honest book but it is a boring book, it fails to entertain or to amaze or to give you those moments of insight that puts a grin in your face.
I think I just wish I had better books as an undergrad :-(
[+] [-] pfortuny|13 years ago|reply
[+] [-] marshallp|13 years ago|reply
[+] [-] ktf|13 years ago|reply
http://nostarch.com/linearalgebra
:D
[+] [-] reyan|13 years ago|reply
[+] [-] Evbn|13 years ago|reply
Do schools really not teach the underlying concepts of math, or do people just fail to understand them the first time through and then blame their teachers?
/took linear algebra in the math department, then TAed it.
[+] [-] shardling|13 years ago|reply
It is absolutely the case that a great many students will not remember seeing material that they definitely were exposed to. As someone who's TA'd physics classes for a few years, I'd often ask a recitation whether they'd covered some particular subject yet in lecture, and they'd often say (as a class) no.
I'd ask the professor later, and of course they had -- but students rarely go into lecture having done the reading or prep work, and so have a piss-poor retention rate.
[+] [-] marshallp|13 years ago|reply
[+] [-] verroq|13 years ago|reply
[+] [-] backprojection|13 years ago|reply