I realized that derivatives are linear

[dan-robertson]

So the point of this article is that differentiation is linear. That is, the operator D which takes f to d f/d x is linear. The author points out that one can write this down in as a matrix with respect to a basis of polynomials, which is nice for suitably well behaved functions and I think nice for understanding. Other operators one might look at are integration, Fourier or Laplace transforms, or more exotic integral transforms which are linear. One can view a Fourier transform like a change of basis.

In another sense, derivatives themselves are linear: for a function f: U -> V of vector spaces, the derivative (at some point) is a linear map from U -> V, (i.e. the derivative of the functions is a function Df: U -> L(U,V)) and this extends the concept of derivative to multiple dimensions as f(x+h) = f(x) + (Df)(x)h + o(h).

This seems ok at first derivatives but can become unwieldy as they became tensors higher rank.

Another question one might ask on learning that differntiation is a linear operator is what it’s eigenvalues are. For differentiation these are functions of the form f(x) = exp(ax). But one can construct other linear operators and from this you get Sturm–Liouville theory which is fantastic.

One final note is that much of this multidimensional derivatives and tensor stuff becomes a lot easier if one learns suffix notation (aka Einstein notation, aka index notation, aka summation convention), as well as perhaps a few identities with the kronecker delta or Levi-Civita symbol. Notation can break down a bit with arbitrary rank tensors: $a_{i_1,...,i_k}$ becomes unwieldy but writing $a_{pq...r}$ is ok.

[sampo]

The derivative is a linear operator, but it's not a bounded operator. That is, for example, the vector norm of f(x) = k·sin(x/k) → 0 when k→0, but the norm of d/dx f(x) does not. This also means that it's not continuous.

Of the mappings between vector spaces, the most well behaving are the bounded linear operators, and the derivative doesn't belong to these. But yes, it's linear.

Edit: Originally wrote f(x) = k·sin(k·x), but meant f(x) = k·sin(x/k).

[hgibbs]

Depends on what space you define the derivative on. It is of course a bounded (and therefore continuous) operator from C^k to C^{k-1} for any positive integer K.

Additionally, it only really makes sense to talk about bounded operators between topological vectors spaces (as you need to make sense of what it means to be bounded), of which the most commonly dealt with are Banach spaces.

[giomasce]

You cannot speak of boundedness of operators if you don't specify the domain and codomain spaces. The derivation operator is for example bounded between C^1(R) and C^0(R), or between C^\infty(R) and C^\infty(R). Your example correctly proves that it is not bounded between C^0(R) and C^0(R).

[soVeryTired]

Anyone have a concrete example of an open subset of C^k whose preimage under differentiation is not open?

(Also, what's the most natural norm on C^k? The sup norm? I haven't done any functional analysis in years)

[msie]

So what level of mathematics is needed to understand the following thread? Undergrad real analysis? Grad math courses?

[azernik]

This was an example used in my linear algebra class as soon as they started introducing the vector spaces in an abstract sense.

I think this post may still be too wedded to the idea of linear spaces and vectors being arrays of objects - specifically in insisting on decomposing functions like sin and cos to Taylor Series. In fact, you can have a vector space where, in addition to polynomial terms, there are also dimensions for sin(x), tan(x), sin(x - pi), e^x, etc. The fact that you can't enumerate these dimensions, or even describe the set of them until given a set of vectors you're trying to describe, doesn't keep this from being a vector space.

[tzahola]

Hm. Interesting.

I always viewed real functions as infinite-dimensional vectors in the "canonical" basis, that is, shifted Dirac impulses. I guess it can be transformed into your representation with a change of basis with some handwaving.

[chombier]

Well this is the whole point of derivatives (i.e. tangent maps): to be linear approximations of functions.

So yes, a linear approximation of a linear function is the function itself.

[giomasce]

You are right, but what you are saying has nothing to do with the author's point: what he is saying is that the differentiation operator itself is linear, which is a meaningful and true fact even in spaces where you have no idea of what a linear function is.

[geoalchimista]

That is my feeling after reading this article. For people who have learned a little bit of Analysis, it is like saying "the earth is round". There is nothing new. The Wikipedia page of Derivative (https://en.wikipedia.org/wiki/Derivative) has a detailed description on the linearity. (Well, I do appreciate the author's way of presenting the idea, but I don't think it deserves an in-depth discussion on Hacker News.)

[thaumasiotes]

If a function f passes through some point (a,b), then the tangent to f through that point is given by

    (y-b) = f'(a)·(x-a)

and that function is affine but usually not linear. (For the tangent curve to be a linear function, you would need a·f'(a) = b, so that the tangent goes through the point (0,0).)

It's not at all obvious to me that this means that the function d(f) = df/dx is linear. It is linear, but I don't see how the tangent curve demonstrates it.

[wodenokoto]

The first headline for this was something along the lines of "I realized that derivatives are linear" making it clear that this is not a new discovery, but rather a person sharing a lightbulb moment.

I feel a lot of comments are saying "well of course they are!", not realizing that this is not about a new discovery.

[anujsharmax]

Be careful when using knowledge from this post. These are special cases, not the general rules of differentiaton (or calculus).

For example, for multi variable calculus, the results would be very different.

Let's take the example of W.X

d/dx (W.X) = X.d/dx(W) + W.d/dx(X)

since W is not dependent on x, the first term is zero and we get the answer the author got.

Before drawing conclusions from the post, please remember the assumptions the author has taken.

[ohazi]

Integrals and Fourier Transforms are also linear...

[ndh2]

There's a ² missing. Should be dC/dx = sum d/dx |...|².

[jasonszhao]

Yep, fixed!

[vole]

>Most of the other non-polynomial functions have an equivalent Taylor polynomial

_analytic_ functions have a Taylor _series_, but it would be incorrect to say that "most" functions have a taylor series, and a taylor series is not a polynomial.

[speedplane]

There seems to be a misconception that linear transformations have to look like lines.

[blattimwind]

One easy way to see that "linear" does not mean "line-like" is looking at matrix transforms: All matrix transforms are linear. And we can do a lot of stuff easily with those, like rotating, distorting, even perspective (with w-normalization).

[fwdpropaganda]

HN continues to confuse me to no end.

Mention some mathematically advanced idea: out come the pitchforks about how you don't need that, all you need is code/market size/scalability/product fit/investment/execution.

Mention a banality that anyone who studied algebra knows: frontpage.

[dang]

Could you please not post like this? It's a shallow dismissal and it's tediously meta.

There's nothing wrong with the OP at all—someone sharing the excitement of discovering something for themselves.

[viraptor]

You may be overestimating the banality. I studied CS with many math courses and some of the article goes over my head. (It would be clearer 10y ago) I don't expect many people here actually studied math as their main goal.

[rs86]

My sentiments exactly. I see how the author felt thrilled by his discovery. But hn is getting less interesting each month. Otoh lobsters and Lbda the ultimate keep going strong.

[anonytrary]

This is why you take linear algebra and calculus before doing machine learning.

[mduerksen]

On the contrary - ML is a great motivator to finally grapple the "prerequisites".

During school I never understood what the math was for, so my unconscious brain never saw the necessity to actually learn it. Now I want to learn - with hugely better results.

This mechanism should be utilized much more often instead of shoving seemingly unrelated knowledge into peoples ears without letting them feel the need for it first.

[dang]

"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something."

https://news.ycombinator.com/newsguidelines.html

[Xcelerate]

I think it's great that machine learning is exciting enough that people want to learn the math behind it.

[bbeonx]

The dude/dudette figured it out on their own which is what math is actually about. Who cares if it's well known? Knowing a fact because you learned it in a class is cool and all but it's not the only way to acquire knowledge, and I would argue not even the best way---just the most efficient fact/hour ratio.

[sgt101]

I used to teach fencing, in old days people would do months of footwork before being allowed to have a go at the actual sport. This changed because once the possibility of fatal encounters disappeared almost no one was motivated enough to go through the prerequisite.

But being clear, the footwork is the fundamental and can't be skipped.

[OscarCunningham]

This is HN, they call it "backpropagation".

[enriquto]

hey, it is better to be nice in that case! Because you don't know anything when your are born, everyday thousands of people learn about well-known cool things that are new for them: https://www.xkcd.com/1053/

This guy even explained his rediscovery in a beautiful form.

[zeofig]

Heyyy man, what if the universe is LINEAR?

[Ace17]

"Classification of mathematical problems as linear and nonlinear is like classification of the Universe as bananas and non-bananas."

[rs86]

Quantum computing is expressible as linear transformations of qubit superpositions