[MathYouF]

The tone of this betrays a possibly more argumentative than collaborative conversation style than that which I may want to engage with further (as seems common I've noticed amongst anti-connectionists), but I did find one point intersting for discussion.

> Parameters are just the database of the system.

Would any equations parameters be considered just the database then? C in E=MC^2, 2 in a^2+b^2=c^2?

I suppose those numbers are basically a database, but the relationships (connections) they have to the other variables (inputs) represent a demonstrable truth about the universe.

To some degree every parameter in a nn is also representing some truth about the universe. How general and compact that representation is currently is not known (likely less than we'd like of both traits).

[mjburgess]

There's a very literal sense in which NN parameters are just a db. As in, it's fairly trivial to get copyrighted verbatim output from a trained NN (eg., quake source code from git co-pilot, etc.).

"Connectionists" always want to reduce everything to formulae with no natural semantics and then equivocate this with science. Science isnt mathematics. Mathematics is just a short hand for a description of the world made true by the semantics of that description.

E=mc^2 isnt true because it's a polynomial, and it doesnt mean a polynomial, and it doesnt have "polynomial properties" because it isnt about mathematics. It's about the world.

E stands for energy, m for mass, and c for a geometric constant of spacetime. If they were to stand for other properties of the world, in general, the formulae would be false.

I find this "connectionist supernaturalism" about mathematics deeply irritating, it has all the hubris and numerology of religions but wandering around in a stolen lab coat. Hence the tone.

What can one say or feel in the face of the overtaking of science by pseudoscience? It seems plausible to say now, today, more pseudoscientific papers are written than scientific ones. A generation of researchers are doing little more than analysing ink-blot patterns and calling them "models".

The insistence, without explanation, that this is a reasonable activity pushes one past tolerance on these matters. It's exasperating... from psychometrics to AI, the whole world of intellectual life has been taken over by a pseudoscientific analysis of non-experimental post-hoc datasets.

[politician]

This discussion (the GP and your response) perhaps suggests that a way to evaluate the intelligence of an AI may need to be more than the generation of some content, but also citations and supporting work for that content. I guess I'm suggesting that the field could benefit from a shift towards explainability-first models.

[jsharf]

I'm not anti-connectionist, but if I were to put myself in their shoes, I'd respond by pointing out that in E=MC^2, C is a value which directly correlates with empirical results. If all of humanity were to suddenly disappear, a future advanced civilization would re-discover the same constant, though maybe with different units. Their neural networks, on the other hand, probably would be meaningfully different.

Also, the C in E=MC^2 has units which define what it means in physical terms. How can you define a "unit" for a neural network's output?

Now, my thoughts on this are contrary to what I've said so far. Even though neural network outputs aren't easily defined currently, there's some experimental results showing neurons in neural networks demonstrating symbolic-like higher-level behavior:

https://openai.com/blog/multimodal-neurons/

Part of the confusion likely comes from how neural networks represent information -- often by superimposing multiple different representations. A very nice paper from Anthropic and Harvard delved into this recently:

https://transformer-circuits.pub/2022/toy_model/index.html

[ctoth]

Related: Polysemanticity and Capacity in Neural Networks https://arxiv.org/abs/2210.01892