I like this approach. It's not that intelligence is in the brain or the model, and it should not be a noun. It is a process, a search process based on exploration and learning.
Think of a river - the water carves the banks, the banks channel the water. Which is the real river?
A model is like the river banks, it is not intelligent in itself. Neither is activity by itself. Intelligence comes from their co-constitutive relation.
You can see the structure+flow coupling in many places. The model weights (banks) vs the training set (water). The contextual tokens (banks) vs the mass of probability for the next token (water). Agent environment (banks) vs agent activity (water).
The trick here is that neither is capable of explaining the process, or more fundamental than the other. It is a recursive process rewriting its own rules.
And we know from Godel, Turing and Chaitin that recursive processes exhibit incompleteness, undecidability and incompressibility. There is no way to predict them from outside, they can only be known only through unrolling, you have to be inside, or you have to be it to know it.
86
6