From Statistical Models to LLMs: A Comprehensive Survey of Language Model Evolution

Abstract

The evolution of language models marks one of the most transformative trajectories in the history of Natural Language Processing (NLP). This survey aims to provide a structured overview of key developments, tracing the progression from early statistical models to deep learning approaches, and culminating in the rise of Transformer-based architectures and Large Language Models (LLMs). We categorize and synthesize key contributions based on algorithmic paradigms, performance metrics, and systemic challenges. Specifically, we examine contributions from foundational models such as n-gram and Hidden Markov Models (HMMs), advances enabled by Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks, and the paradigm shift introduced by self-attention mechanisms in Transformer architectures. Additionally, the survey discusses how LLMs have expanded the capabilities of NLP systems in tasks including text generation, translation, and dialogue modeling. Alongside these achievements, we critically highlight ongoing challenges, including model bias, interpretability, computational costs, and environmental impacts, drawing on recent literature and evaluation frameworks. Emerging trends toward improving model efficiency, fairness, and societal alignment are also explored. By mapping historical progress and identifying open questions, this article offers a comprehensive reference for researchers and practitioners interested in the evolving landscape of language models.