Explainer
Small language models explained: what they are and why they matter
A small language model is one compact enough to run on an ordinary consumer device; NVIDIA researchers said in 2025 that most models under 10 billion parameters could count as small.[1] Examples include Microsoft's 3.8-billion-parameter phi-3-mini, which its makers say scores close to GPT-3.5 on benchmarks while fitting on a phone, and Apple's roughly 3-billion-parameter on-device model.[2][3] Running a model locally keeps data on the device, and serving a 7-billion-parameter model costs an estimated 10 to 30 times less than a much larger one.[4][5]
Small language models (SLMs) are language models compact enough to run on a common consumer device such as a phone or laptop.[1] Labs have released several since 2024, including Microsoft’s phi-3-mini, Meta’s smallest Llama 3.2 models and Apple’s on-device model.[2][6][3]
What counts as small
NVIDIA researchers use a working definition: a small language model can fit on a common consumer electronic device and respond quickly enough to be practical for one user’s requests.[1] They said in 2025 that they would consider most models below 10 billion parameters to be small.[1] What parameters are is explained in how large language models work.
Examples
Microsoft’s phi-3-mini, described in April 2024, has 3.8 billion parameters and was trained on 3.3 trillion tokens of heavily filtered web data and synthetic data.[2] Its makers report benchmark results that rival GPT-3.5 and Mixtral 8x7B, while the model is small enough to be deployed on a phone.[2]
Meta’s Llama 3.2, released in September 2024, included lightweight 1-billion and 3-billion-parameter text models designed for edge and mobile devices.[6] Apple’s 2025 foundation models include an on-device model of about 3 billion parameters alongside a server-based model.[3]
How models are made smaller
Knowledge distillation trains a small “student” model using a larger “teacher” model, on the idea that the small model can do better with a teacher than if trained from scratch.[7] A 2015 paper by Geoffrey Hinton and colleagues developed this further, compressing the knowledge of a group of models into a single model that is much easier to deploy.[7] Meta built its 1B and 3B Llama 3.2 models from larger Llama models using pruning followed by distillation.[6]
Quantization stores a model in fewer bits: Apple compressed its on-device model to 2 bits per weight using quantization-aware training.[3] The chips that run these models on devices are covered in GPUs, TPUs and NPUs and AI PCs.
Why small models matter
Privacy is one reason: Meta says running models locally keeps data such as messages and calendar information on the device instead of sending it to the cloud.[4] Cost is another. NVIDIA researchers estimate that serving a 7-billion-parameter model is 10 to 30 times cheaper in latency, energy consumption and computation than serving a 70-to-175-billion-parameter model.[5]
In a 2025 position paper, the same researchers argue that small models are powerful enough, better suited and more economical for many of the repetitive, specialised tasks inside AI agent systems, and that agents can call larger models when broad conversation is needed.[8] How agents combine models and tools is explained in how AI agents work.
Read next
How fine-tuning works: adapting AI models with LoRA and more
How fine-tuning adapts a pre-trained AI model to a task: full fine-tuning, LoRA and QLoRA, instruction tuning, distillation, and the risks of forgetting.
Developing in AI & Computing
Questions readers ask
What is a small language model?
NVIDIA researchers define it as a language model that can fit on a common consumer device and respond fast enough to serve one user's requests. They said in 2025 that most models below 10 billion parameters could be considered small.[1]
Can a language model run on a phone?
Yes. Microsoft's phi-3-mini has 3.8 billion parameters and is small enough to be deployed on a phone, Meta's Llama 3.2 includes 1-billion and 3-billion-parameter models built for edge and mobile devices, and Apple runs a roughly 3-billion-parameter model on its devices.[2][6][3]
How are small models made?
Common methods include knowledge distillation, where a larger "teacher" model trains a smaller one, and pruning, which Meta combined with distillation for its smallest Llama 3.2 models. Apple compressed its on-device model to 2 bits per weight using quantization-aware training.[7][6][3]
Are small language models more private?
Running a model locally means data such as messages and calendar information does not have to be sent to the cloud, as Meta notes for its on-device models.[4]
Why use a small model instead of a large one?
Cost and speed. NVIDIA researchers estimate that serving a 7-billion-parameter model is 10 to 30 times cheaper in latency, energy and computation than serving a 70-to-175-billion-parameter model, and argue that small models suit many repetitive tasks in AI agents.[5][8]
Sources
· 8 claims from 5 publishers
Each numbered claim is a statement we checked against the sources listed with it. Tap a number in the text to see its sources in place.
- [1]
NVIDIA researchers define a small language model as one that can fit on a common consumer device and respond quickly enough to serve one user's requests, and said in 2025 that most models below 10 billion parameters could be considered small. confirmedas of 2025-06-02
- Small Language Models are the Future of Agentic AI · arXiv (Belcak et al., NVIDIA Research) · Primary source · 2025-06-02 · Section 2, Definitions (WD1) (retrieved 2026-11-14)
- Small Language Models are the Future of Agentic AI · arXiv (Belcak et al., NVIDIA Research) · Primary source · 2025-06-02 · Section 2, Definitions (retrieved 2026-11-14)
- [2]
Microsoft's phi-3-mini, described in April 2024, has 3.8 billion parameters, was trained on 3.3 trillion tokens of heavily filtered web data and synthetic data, and scored close to GPT-3.5 on academic benchmarks while being small enough to run on a phone. confirmedas of 2024-04-22
- Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone · arXiv (Microsoft) · Primary source · 2024-04-22 (retrieved 2026-11-14)
- Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone · arXiv (Microsoft) · Primary source · 2024-04-22 (retrieved 2026-11-14)
- Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone · arXiv (Microsoft) · Primary source · 2024-04-22 (retrieved 2026-11-14)
- Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone · arXiv (Microsoft) · Primary source · 2024-04-22 (retrieved 2026-11-14)
- [3]
Apple's 2025 foundation models include an on-device model of about 3 billion parameters, which Apple compressed to 2 bits per weight using quantization-aware training. confirmedas of 2025-06-09
- Updates to Apple's On-Device and Server Foundation Language Models · Apple Machine Learning Research · Primary source · 2025-06-09 (retrieved 2026-11-14)
- Updates to Apple's On-Device and Server Foundation Language Models · Apple Machine Learning Research · Primary source · 2025-06-09 (retrieved 2026-11-14)
- [4]
Meta says running models locally keeps data such as messages and calendar information on the device instead of sending it to the cloud. confirmedas of 2024-09-25
- Llama 3.2: Revolutionizing edge AI and vision with open, customizable models · Meta AI · Primary source · 2024-09-25 (retrieved 2026-11-14)
- [5]
The NVIDIA researchers estimate that serving a 7-billion-parameter model is 10 to 30 times cheaper in latency, energy and computation than serving a 70-to-175-billion-parameter model. confirmedas of 2025-06-02
- Small Language Models are the Future of Agentic AI · arXiv (Belcak et al., NVIDIA Research) · Primary source · 2025-06-02 · Section 3 (economy argument) (retrieved 2026-11-14)
- Small Language Models are the Future of Agentic AI · arXiv (Belcak et al., NVIDIA Research) · Primary source · 2025-06-02 · Section 3 (economy argument) (retrieved 2026-11-14)
- [6]
Meta's Llama 3.2 release in September 2024 included lightweight 1-billion and 3-billion-parameter text models designed to fit on edge and mobile devices, which Meta built from larger Llama models using pruning and knowledge distillation. confirmedas of 2024-09-25
- Llama 3.2: Revolutionizing edge AI and vision with open, customizable models · Meta AI · Primary source · 2024-09-25 (retrieved 2026-11-14)
- Llama 3.2: Revolutionizing edge AI and vision with open, customizable models · Meta AI · Primary source · 2024-09-25 (retrieved 2026-11-14)
- Llama 3.2: Revolutionizing edge AI and vision with open, customizable models · Meta AI · Primary source · 2024-09-25 (retrieved 2026-11-14)
- [7]
Knowledge distillation uses a larger "teacher" network to train a smaller one, on the idea that the small model can perform better with a teacher than if trained from scratch; a 2015 paper by Geoffrey Hinton, Oriol Vinyals and Jeff Dean further developed the approach of compressing the knowledge of an ensemble of models into a single model that is much easier to deploy. confirmedas of 2024-09-25
- Llama 3.2: Revolutionizing edge AI and vision with open, customizable models · Meta AI · Primary source · 2024-09-25 (retrieved 2026-11-14)
- Distilling the Knowledge in a Neural Network · arXiv (Hinton, Vinyals and Dean) · Primary source · 2015-03-09 (retrieved 2026-11-14)
- [8]
In a 2025 position paper, NVIDIA researchers argue that small language models are powerful enough, better suited and more economical for many of the repetitive, specialised tasks in AI agent systems, and that agents can call larger models when general conversation is needed. confirmedas of 2025-06-02· interpretation
- Small Language Models are the Future of Agentic AI · arXiv (Belcak et al., NVIDIA Research) · Primary source · 2025-06-02 · Abstract (retrieved 2026-11-14)
- Small Language Models are the Future of Agentic AI · arXiv (Belcak et al., NVIDIA Research) · Primary source · 2025-06-02 · Introduction (retrieved 2026-11-14)
- Small Language Models are the Future of Agentic AI · arXiv (Belcak et al., NVIDIA Research) · Primary source · 2025-06-02 · Abstract (retrieved 2026-11-14)
Revision history (1)
- Page created.
Created Nov 14, 2026. Last reviewed by an editor on Nov 14, 2026. Next scheduled review: Jun 8, 2027.
Cite this page
"Small language models explained: what they are and why they matter." ContentLora, updated Nov 14, 2026. https://contentlora.com/explain/small-language-models-explained
Spotted an error? Suggest a correction or emailcorrections@contentlora.com.