Mechanistic Interpretability of Structure-Aware Numerical Reasoning in LLaMA 3.1 8B

Recent work has shown that large language models (LLMs) exhibit strong numerical sequence modeling capabilities and show promise in time-series prediction. While LLMs display in-context learning capabilities, the mechanisms with which they accomplish time-series prediction remain unclear. Specifically, whether they truly understand the underlying structure, which at a minimum requires reasoning over first differences in the sequence of numbers. To study this, we investigate Llama 3.1-8B from a mechanistic interpretability point of view. Mechanistic interpretability is an emerging field concerned with the reverse engineering of the algorithms learned by neural networks such as LLMs. To assess Llamas' numerical sequence modeling capabilities and to facilitate our mechanistic interpretability analysis, we create a sequence modeling task that cannot be solved without picking up structural cues. Specifically, we sample n random numbers and repeat them with an offset. We find that Llama displays strong performance on our tasks suggesting that it can pick up on the underlying structure. To understand the mechanisms that allow it to do so, we perform probing experiments and activation patching based counterfactual analysis. Probing reveals that the model computes and stores first differences in its internal representations without explicit supervision, indicating that it tracks structural information about the sequence. Activation patching reveals that Llama retrieves the relevant first-difference with a mechanism similar to an induction circuit and subsequently adds it to the current value. Notably, our work represents one of the first studies to identify this form of concept induction in LLMs.

Approximate reservoir computing with a semiconductor laser for reducing energy consumption

Photonic reservoir computing is a promising physical machine-learning technique for predicting time-series data. The quantization of the response signal from the reservoir is required for the implementation of photonic reservoir computing, and the number of quantization bits and sampling frequency need to be optimized to achieve high performance and low energy consumption. However, few studies have been reported to investigate the effect of bit quantization and sampling frequency. In this study, we introduce a concept of approximate reservoir computing with a semiconductor laser by quantizing the amplitude of node states in the reservoir and output weights. We evaluate the performance of a chaotic time-series prediction task and energy consumption per sample. We achieve significant reduction of energy consumption by optimizing the number of quantization bits, the sampling frequency, and the injection current of the semiconductor laser, while maintaining the prediction performance.

Photonic reservoir computing with complex networks

Photonic reservoir computing has attracted increasing attention as a fast and low-cost approach for time-series prediction. Photonic reservoir computing utilizes the high speed, broad bandwidth, and spatial parallelism of light. However, the effect of the internal connection structure (network topology) on the computing performance has not been investigated for large-scale photonic reservoirs. In this study, we experimentally and numerically demonstrate photonic reservoir computing using a spatial light modulator to systematically evaluate the relationship between the network topology and the performance of reservoir computing. We introduce complex network structures such as small-world and scale-free network topologies of the internal nodes in the reservoir. We perform the memory capacity measurement and the one-step-ahead prediction task of the chaotic time series to compare the performance. We found that the small-world network exhibits the maximum memory capacity and the best prediction performance. Our numerical calculations reveal that the performance of the time-series prediction can be optimized by changing the rewiring probability of the network and the leak rate of the reservoir. We also implement photonic human brain network as a reservoir, which is designed by the connectomes of human brain activities. We found that the network topology strongly affects the performance of reservoir computing, and the small-world network structure outperforms the other configurations.

Nonlinear Bias-Compensated Adaptive Filter and Its Application for Time-Series Prediction

Most existing nonlinear adaptive filtering algorithms only account for output noise, neglecting the fact that input noise is also prevalent in practice. Although the recently proposed bias-compensated kernel least mean square (BCKLMS) algorithm addresses input noise in the nonlinear errors-in-variables (EIV) model, it still suffers from two major limitations. First, the use of a fixed-size dictionary restricts network growth but also prevents it from fully capturing the characteristics of the input signal. Second, as an least mean square (LMS) based algorithm, it exhibits poor robustness in the presence of non-Gaussian noise in the output signal. To overcome these issues, this paper proposes the random Fourier bias-compensated filter under general adaptive function (RFFBCGA) algorithm. Within the random Fourier feature based bias-compensated (RFFBC) framework, the proposed algorithm not only maintains a fixed network structure and effectively mitigates input noise interference through the BC term, but also achieves improved characterization of the input signal. Moreover, by leveraging the flexible form of the general adaptive (GA) function, the algorithm's robustness across various noise scenarios is further enhanced. Extensive simulations, including real-world time series prediction tasks, demonstrate the superiority of the proposed method.