[extasia]

This is not quite right; you are actually losing information about each of the dimensions and your mental model of reducing the dimensionality by one is misleading.

Consider [1,0] and [x,x] Normalised we get [1,0] and [sqrt(.5),sqrt(.5)] — clearly something has changed because the first vector is now larger in dimension zero than the second, despite starting off as an arbitrary value, x, which could have been smaller than 1. As such we have lost information about x’s magnitude which we cannot recover from just the normalized vector.

[Scene_Cast2]

Well, depends. For some models (especially two tower style models that use a dot product), you're definitely right and it makes a huge difference. In my very limited experience with LLM embeddings, it doesn't seem to make a difference.

[extasia]

Interesting, I hadn’t heard of two tower modes before!

Yes, I guess it’s curious that the information lost doesn’t seem very significant (this also matches my experience!)

[atorodius]

That‘s exactly the point no? We lost 1 dim (magnitude). Not so nice in 2d but no biggie in 512d

[extasia]

Magnitude is not a dimension, it’s information about each value that is lost when you normalize it. To prove this normalize any vector and then try to de-normalize it again.

[renewiltord]

There's something wrong with the picture here but I can't put my finger on it because my mathematical background here is too old. The space of k dimension vectors all normalized isn't a vector space itself. It's well-behaved in many ways but you lose the 0 vector (may not be relevant). Addition isn't defined anymore, and if you try to keep it inside by normalization post addition, distribution becomes weird. I have no idea what this transformation means for word2vec and friends.

But the intuitive notion is that if you take all 3D and flatten it / expand it to be just the surface of the 3D sphere, then paste yourself onto it Flatland style, it's not the same as if you were to Flatland yourself into the 2D plane. The obvious thing is that triangles won't sum to 180, but also parallel lines will intersect, and all sorts of differing strange things will happen.

I mean, it might still work in practice, but it's obviously different from some method of dimensionality reduction because you're changing the curvature of the space.