It's not hijacked, the formulation is the same. For any layer there is a state-space formulation
h = Ah + Bx
y = Ch + Dx
where x is the input, y is the output, and h is the state. They use "h" instead of "s" for the state variables because they're called "hidden states" in the literature. edit: it is obnoxious they've flipped the convention for A/B/C/D which is the one thing controls people agree on (we can't even agree on the signs and naming of transfer function coefficients!).
Where this diverges from dynamical systems/controls is that they're proving that when x/h/y are represented with finite precision numbers, the model is limited in the problems it can represent (no surprise from controls perspective), and they prove this by using an equivalence to the state-space formulae that's consistent with evaluating it on massively parallel hardware.
The classical controls theory is not super applicable here, because what controls people care about (is the system stable, is its rise/fall time in bounds, what about overshoot, etc) is not what ML researchers care about (what classes of AI problems can be modeled and evaluated using this computational architecture).
Interesting. This is how I sometimes feel about physics
The field completely appropriated and redefined so many terms of common language, that now it’s hard to talk in plain language with someone formally trained in physics about physical phenomena
For example, everyone has some sort of intuitive idea about what Energy is. But if you use that word with a physicist, watch out, for them it means something super specific within the context of assumptions and mathematical models and they will assume you don’t know what you are talking about because you are not using their definitions from their models
I mean, if you like state space models, then you should read the paper on Mamba if you haven’t already! Because it quite literally uses state spaces… and you will probably think it’s a really cool application of state spaces!
Apologies if you know the following already, but maybe others reading your comment feeling similarly will not be familiar and might be interested.
At least intuitively, I like to motivate it this way- pick your favorite simple state space problem. Say a coupled spring system of two masses, maybe with some driving forces. Set it up. Perturb it. Make a bunch of observations at various points in time. Now use your observations to figure out the state space matrices.
There’s fundamentally not really anything different (in my opinion) with using Mamba (or another state space model) as a function approximation of whatever phenomenon you are interested in. Okay Mamba has more moving parts, but the core idea is the same: you are saying that on some level, a state space is an appropriate prior for approximation of the dynamics of the quantities of interest. It turns out being pretty remarkable the number of things this can work out quite well for. For instance, I use it to model the 15-min interval data for heating, cooling, and electricity usage of a whole building given 15 min weather data, occupancy schedules, and descriptions of the building characteristics (eg building envelope construction, equipment types, number of occupants, etc).
duped|1 year ago
Where this diverges from dynamical systems/controls is that they're proving that when x/h/y are represented with finite precision numbers, the model is limited in the problems it can represent (no surprise from controls perspective), and they prove this by using an equivalence to the state-space formulae that's consistent with evaluating it on massively parallel hardware.
The classical controls theory is not super applicable here, because what controls people care about (is the system stable, is its rise/fall time in bounds, what about overshoot, etc) is not what ML researchers care about (what classes of AI problems can be modeled and evaluated using this computational architecture).
throwaway42668|1 year ago
nico|1 year ago
The field completely appropriated and redefined so many terms of common language, that now it’s hard to talk in plain language with someone formally trained in physics about physical phenomena
For example, everyone has some sort of intuitive idea about what Energy is. But if you use that word with a physicist, watch out, for them it means something super specific within the context of assumptions and mathematical models and they will assume you don’t know what you are talking about because you are not using their definitions from their models
Same thing happens with infinite in math
szvsw|1 year ago
Apologies if you know the following already, but maybe others reading your comment feeling similarly will not be familiar and might be interested.
At least intuitively, I like to motivate it this way- pick your favorite simple state space problem. Say a coupled spring system of two masses, maybe with some driving forces. Set it up. Perturb it. Make a bunch of observations at various points in time. Now use your observations to figure out the state space matrices.
There’s fundamentally not really anything different (in my opinion) with using Mamba (or another state space model) as a function approximation of whatever phenomenon you are interested in. Okay Mamba has more moving parts, but the core idea is the same: you are saying that on some level, a state space is an appropriate prior for approximation of the dynamics of the quantities of interest. It turns out being pretty remarkable the number of things this can work out quite well for. For instance, I use it to model the 15-min interval data for heating, cooling, and electricity usage of a whole building given 15 min weather data, occupancy schedules, and descriptions of the building characteristics (eg building envelope construction, equipment types, number of occupants, etc).
3abiton|1 year ago
aerospace_guy|1 year ago