Skip to content
ContentLora

    Tip: press / anywhere to search.

    Explainer

    Small language models explained: what they are and why they matter

    A small language model is one compact enough to run on an ordinary consumer device; NVIDIA researchers said in 2025 that most models under 10 billion parameters could count as small.[1] Examples include Microsoft's 3.8-billion-parameter phi-3-mini, which its makers say scores close to GPT-3.5 on benchmarks while fitting on a phone, and Apple's roughly 3-billion-parameter on-device model.[2][3] Running a model locally keeps data on the device, and serving a 7-billion-parameter model costs an estimated 10 to 30 times less than a much larger one.[4][5]

    Editor reviewedUpdated Explainer8 claims · 100% primaryArtificial intelligence, Frontier AI, Computing

    Small language models (SLMs) are language models compact enough to run on a common consumer device such as a phone or laptop.[1] Labs have released several since 2024, including Microsoft’s phi-3-mini, Meta’s smallest Llama 3.2 models and Apple’s on-device model.[2][6][3]

    What counts as small

    NVIDIA researchers use a working definition: a small language model can fit on a common consumer electronic device and respond quickly enough to be practical for one user’s requests.[1] They said in 2025 that they would consider most models below 10 billion parameters to be small.[1] What parameters are is explained in how large language models work.

    Examples

    Microsoft’s phi-3-mini, described in April 2024, has 3.8 billion parameters and was trained on 3.3 trillion tokens of heavily filtered web data and synthetic data.[2] Its makers report benchmark results that rival GPT-3.5 and Mixtral 8x7B, while the model is small enough to be deployed on a phone.[2]

    Meta’s Llama 3.2, released in September 2024, included lightweight 1-billion and 3-billion-parameter text models designed for edge and mobile devices.[6] Apple’s 2025 foundation models include an on-device model of about 3 billion parameters alongside a server-based model.[3]

    How models are made smaller

    Knowledge distillation trains a small “student” model using a larger “teacher” model, on the idea that the small model can do better with a teacher than if trained from scratch.[7] A 2015 paper by Geoffrey Hinton and colleagues developed this further, compressing the knowledge of a group of models into a single model that is much easier to deploy.[7] Meta built its 1B and 3B Llama 3.2 models from larger Llama models using pruning followed by distillation.[6]

    Quantization stores a model in fewer bits: Apple compressed its on-device model to 2 bits per weight using quantization-aware training.[3] The chips that run these models on devices are covered in GPUs, TPUs and NPUs and AI PCs.

    Why small models matter

    Privacy is one reason: Meta says running models locally keeps data such as messages and calendar information on the device instead of sending it to the cloud.[4] Cost is another. NVIDIA researchers estimate that serving a 7-billion-parameter model is 10 to 30 times cheaper in latency, energy consumption and computation than serving a 70-to-175-billion-parameter model.[5]

    In a 2025 position paper, the same researchers argue that small models are powerful enough, better suited and more economical for many of the repetitive, specialised tasks inside AI agent systems, and that agents can call larger models when broad conversation is needed.[8] How agents combine models and tools is explained in how AI agents work.

    Developing in AI & Computing

    Go deeper

    Questions readers ask

    What is a small language model?

    NVIDIA researchers define it as a language model that can fit on a common consumer device and respond fast enough to serve one user's requests. They said in 2025 that most models below 10 billion parameters could be considered small.[1]

    Can a language model run on a phone?

    Yes. Microsoft's phi-3-mini has 3.8 billion parameters and is small enough to be deployed on a phone, Meta's Llama 3.2 includes 1-billion and 3-billion-parameter models built for edge and mobile devices, and Apple runs a roughly 3-billion-parameter model on its devices.[2][6][3]

    How are small models made?

    Common methods include knowledge distillation, where a larger "teacher" model trains a smaller one, and pruning, which Meta combined with distillation for its smallest Llama 3.2 models. Apple compressed its on-device model to 2 bits per weight using quantization-aware training.[7][6][3]

    Are small language models more private?

    Running a model locally means data such as messages and calendar information does not have to be sent to the cloud, as Meta notes for its on-device models.[4]

    Why use a small model instead of a large one?

    Cost and speed. NVIDIA researchers estimate that serving a 7-billion-parameter model is 10 to 30 times cheaper in latency, energy and computation than serving a 70-to-175-billion-parameter model, and argue that small models suit many repetitive tasks in AI agents.[5][8]

    Sources

    · 8 claims from 5 publishers

    Each numbered claim is a statement we checked against the sources listed with it. Tap a number in the text to see its sources in place.

    1. [1]

      NVIDIA researchers define a small language model as one that can fit on a common consumer device and respond quickly enough to serve one user's requests, and said in 2025 that most models below 10 billion parameters could be considered small. confirmedas of 2025-06-02

    2. [2]

      Microsoft's phi-3-mini, described in April 2024, has 3.8 billion parameters, was trained on 3.3 trillion tokens of heavily filtered web data and synthetic data, and scored close to GPT-3.5 on academic benchmarks while being small enough to run on a phone. confirmedas of 2024-04-22

    3. [3]

      Apple's 2025 foundation models include an on-device model of about 3 billion parameters, which Apple compressed to 2 bits per weight using quantization-aware training. confirmedas of 2025-06-09

    4. [4]

      Meta says running models locally keeps data such as messages and calendar information on the device instead of sending it to the cloud. confirmedas of 2024-09-25

    5. [5]

      The NVIDIA researchers estimate that serving a 7-billion-parameter model is 10 to 30 times cheaper in latency, energy and computation than serving a 70-to-175-billion-parameter model. confirmedas of 2025-06-02

    6. [6]

      Meta's Llama 3.2 release in September 2024 included lightweight 1-billion and 3-billion-parameter text models designed to fit on edge and mobile devices, which Meta built from larger Llama models using pruning and knowledge distillation. confirmedas of 2024-09-25

    7. [7]

      Knowledge distillation uses a larger "teacher" network to train a smaller one, on the idea that the small model can perform better with a teacher than if trained from scratch; a 2015 paper by Geoffrey Hinton, Oriol Vinyals and Jeff Dean further developed the approach of compressing the knowledge of an ensemble of models into a single model that is much easier to deploy. confirmedas of 2024-09-25

    8. [8]

      In a 2025 position paper, NVIDIA researchers argue that small language models are powerful enough, better suited and more economical for many of the repetitive, specialised tasks in AI agent systems, and that agents can call larger models when general conversation is needed. confirmedas of 2025-06-02· interpretation

    Revision history (1)
    1. Page created.

    Created Nov 14, 2026. Last reviewed by an editor on Nov 14, 2026. Next scheduled review: Jun 8, 2027.

    Cite this page

    "Small language models explained: what they are and why they matter." ContentLora, updated Nov 14, 2026. https://contentlora.com/explain/small-language-models-explained

    Spotted an error? Suggest a correction or emailcorrections@contentlora.com.