Neehal Tumma

Hi, I'm Neehal! I am a second-year PhD student at MIT CSAIL, advised by Prof. Daniela Rus. I also work at Liquid AI on scaling efficient foundation models.

My research interests broadly lie in architecture design for sequence modeling, particularly LLMs. Within this space, I primarily focus on recurrent architectures as alternatives to attention-based ones. I am interested in exploring these recurrent primitives in various settings, including test-time training, continual learning, and efficient long-context modeling.

I graduated from Harvard in 2024 with an SM in Computer Science and an AB in Mathematics and Computer Science. During my undergrad, I was fortunate enough to work with Prof. Nir Shavit and Prof. Finale Doshi-Velez.

Email  /  Google Scholar  /  Twitter  /  GitHub  /  LinkedIn

We introduce ParaTTT, an algorithm for parallelizing test-time training layers with deep nonlinear memories through iterative refinement. The exact online recurrence is a fixed point of the iteration. With two refinement steps, ParaTTT trains 4.9× faster than TTT-MLP at 340M scale while improving recall over linear recurrent models.

We show that decayed recurrent models develop weaker residual outliers than softmax Transformers, making their weights and activations easier to quantize. For the recurrent state, we introduce LoReS, combining scale grouping, decay-based mixed precision, and stochastic rounding to support robust 8-bit state quantization and faster decoding in vLLM.

We identify seven functional invariances that preserve the full-precision computation of linear recurrences while making their operands easier to quantize. LinQuant uses these transformations to improve low-precision prefill across recurrent architectures, improving perplexity at matched latency on Mamba-2 and Nemotron-H.

We show that with least-squares preconditioning, DeltaNet and linear attention reduce to the same recurrence. Using this insight, we introduce preconditioned variants of DeltaNet (PDN), Gated DeltaNet (PGDN), and Kimi Delta Attention (PKDA), demonstrating gains across synthetic recall benchmarks and language modeling at the 340M and 1B scale.

We introduce LFM2, a family of compact foundation models (350M–8.3B parameters) optimized for edge deployment. Using hardware-in-the-loop architecture search, LFM2 achieves up to 2x faster prefill and decode on CPUs compared to similarly sized models, with extensions to vision-language, speech, and retrieval.

We introduce effective state-size (ESS), a principled metric grounded in signal processing and control theory for quantifying how sequence models utilize memory. ESS provides interpretable insights into memory dynamics across attention, convolutions, and recurrent architectures, enabling improved initialization, regularization, and distillation.

We propose a statistical sliding window framework for analyzing neural connectivity in large-scale connectomics datasets using both the connectivity graph and functional activations. Applied to the MICrONS dataset, we find that the V1-RL border region exhibits greater synaptic connectivity and more synchronous activity, acting as a bridge between visual areas.

We explore how the recurrent connectivity of RNNs, parameterized by rank and sparsity, influences robustness in closed-loop control. We show that low-rank, sparse connectivity induces desirable network dynamics, enabling closed-form continuous-time networks (CfCs) with fewer parameters to outperform their full-rank counterparts under distribution shift.

We introduce the Label-Indexed Neural Topic Model (LI-NTM), the first effective upstream semi-supervised neural topic model. LI-NTM outperforms existing neural topic models in document reconstruction, with the strongest results in low labeled data regimes.