It's scary to think that while we have the Transformer and it seems extremely complex, things like this throw it all away and give us a glimpse into what the neural (or even non-neural) networks of the future will be like. Perhaps even just simulating full biological systems as well as a brain-like system.
While ANNs are rather trained for unidirectional propagation, action potential propagation in biological neurons is symmetric e.g. ”it is not uncommon for axonal propagation of action potentials to happen in both directions” ( https://journals.aps.org/pre/abstract/10.1103/PhysRevE.92.03... ).
Also, while current ANNs use guessed parametrizations, objectively available is joint distribution - biological neuron should be evolutionarily optimized to exploit, and it is relatively simple in approach from this arXiv.
Such joint distribution neurons bring additional training approaches - maybe some of them are used by biological neural networks?
First, this work needs work. The English needs to be improved before I would recommend wading into the contents deeply.
Second this assertion or citation is wrong:
>for biological neurons e.g. "it is not uncommon for axonal propagation of action potentials to happen in both directions" - suggesting they are optimized to continuously operate in multidirectional way.
What is true is that dendritic spikes can propagate bidirectionally in some neurons (but can also fade or be blocked).
What we often forget is that spikes are a kludge to enable faster INTRAcellular communication (not needed in retinal processing).
The classic action potential connects the axon hillock (the spike initiation zone) to a variable subset of responsive presynaptic sites that may or may not release neurotransmitters that may or may not modulate behaviors of neighboring processes and cells.
Sure biological NN are much more complicated, but basically action propagation can travel in both directions, and evolution should optimize for that.
In contrast, current ANNs are focused on unidirectional propagation, and are much worse at training from single samples - to reach abilities of biological, maybe it is worth to start thinking about multidirectional?
Neurons containing joint distribution model can do propagate conditional distributions in various direction, and it is not that difficult to represent - maybe something like that could be hidden in biological (?)
However, it degenerates to ~KAN if restring to pairwise dependencies (can consciously add triplewise and higher), and gives many new possibilities, like multidirectional propagation, of values or probability distributions, with novel additional training approaches like through tensor decomposition.
bingbingbing777|1 year ago
jarekd|1 year ago
Also, while current ANNs use guessed parametrizations, objectively available is joint distribution - biological neuron should be evolutionarily optimized to exploit, and it is relatively simple in approach from this arXiv.
Such joint distribution neurons bring additional training approaches - maybe some of them are used by biological neural networks?
robwwilliams|1 year ago
>for biological neurons e.g. "it is not uncommon for axonal propagation of action potentials to happen in both directions" - suggesting they are optimized to continuously operate in multidirectional way.
What is true is that dendritic spikes can propagate bidirectionally in some neurons (but can also fade or be blocked).
What we often forget is that spikes are a kludge to enable faster INTRAcellular communication (not needed in retinal processing).
The classic action potential connects the axon hillock (the spike initiation zone) to a variable subset of responsive presynaptic sites that may or may not release neurotransmitters that may or may not modulate behaviors of neighboring processes and cells.
jarekd|1 year ago
In contrast, current ANNs are focused on unidirectional propagation, and are much worse at training from single samples - to reach abilities of biological, maybe it is worth to start thinking about multidirectional?
Neurons containing joint distribution model can do propagate conditional distributions in various direction, and it is not that difficult to represent - maybe something like that could be hidden in biological (?)
rajnathani|1 year ago
cs702|1 year ago
Doesn't look like it.
jarekd|1 year ago
However, it degenerates to ~KAN if restring to pairwise dependencies (can consciously add triplewise and higher), and gives many new possibilities, like multidirectional propagation, of values or probability distributions, with novel additional training approaches like through tensor decomposition.
unknown|1 year ago
[deleted]