Vaishnavi B Mohan, Vijayakrishna Naganoor, Yashas Annadani +1cs.CV
Vision transformers (ViTs) have become the de facto standard for image encoding across many perception tasks. Despite their empirical success, it remains mechanistically unclear how they encode low-level features, given their lack of inductive biases: ViTs process information globally rather than relying on local structure. Biological visual systems, in contrast, build low-level features, such as orientation selectivity in the primary visual cortex, by combining information from small, localized regions of the visual field. These features are general-purpose representations, shared and required across multiple specialized neural pathways, unlike higher-level, task-specific semantic features. This raises the question if such biologically-grounded features arise in ViTs. In this work, we systematically study how orientation selectivity emerges in ViTs by introducing a suite of neuroscience-inspired metrics: representational similarity score (RSS), orientation recruitment score (ORS), and orientation tuning bandwidth to quantify how orientation is encoded in representational geometry and as a function of model depth. Through extensive analysis, we find that: (1) the training paradigm is the strongest determinant of orientation selectivity, with models sharing an objective, peaking at comparable relative depths regardless of scale (2) many units are orientation-selective early in training, with early-to-middle layers recruiting more such units over time, while deeper layers lose selectivity and broaden their tuning toward semantic encoding and (3) our metrics offer a mechanistic heuristic for how many layers to unfreeze for best downstream generalization. Our framework presents a way to track biologically-grounded features during ViT training, probes how desired properties are encoded in transformer representations, and builds a systematic understanding of how ViTs generalize across tasks.
The emergence of orientation selectivity in the primary visual cortex (V1) remains a central question in computational neuroscience. Shirazi's Bayes-Markov model proposed a probabilistic explanation for how orientation-selective inhibition can arise from non-oriented lateral geniculate nucleus (LGN) inputs through local inference. In that formulation, the activity pattern of striate cortical inhibitory (SCI) cells is estimated from the LGN activity pattern by a maximum a posteriori (MAP) criterion over a two-layer hierarchical Markov random field, and the resulting inference is implemented through a local parallel relaxation algorithm. We provide a computationally explicit re-implementation and quantitative simulation study of this framework. We reconstruct the mathematical model, describe its fully LGN-driven update rule, and implement a vectorized simulation framework that preserves the original local clique operations while making systematic parameter sweeps feasible. We evaluate the model using orientation tuning curves, an orientation selectivity index (OSI), controlled LGN noise perturbations, contrast tests, and model-variant comparisons. We further add a spiking SCI-layer realization using leaky integrate-and-fire and Hodgkin-Huxley neurons to examine whether the rate-coded SCI field can be expressed through temporally explicit neural activity. The simulations support the central qualitative behavior of the Bayes-Markov framework: sharp orientation selectivity, robustness to moderate LGN noise, and a biologically interpretable proof-of-concept spiking realization of the inferred inhibitory field.