Skip to content
ContentLora

    Tip: press / anywhere to search.

    Explainer

    GPUs, TPUs and NPUs explained: the chips that run AI

    AI runs mostly on three kinds of chip. Graphics processing units (GPUs) are general parallel processors that NVIDIA opened to general computing with CUDA in 2006.[1] Custom ASICs such as Google's TPUs are built for the matrix maths in machine learning.[2] Neural processing units (NPUs) are efficient AI engines built into PCs.[3] Training a model and running it (inference) put different loads on these chips, and memory bandwidth is often the limit.[4][5]

    Editor reviewedUpdated Explainer29 claims · 97% primaryNext-gen computing hardware, Semiconductors, Artificial intelligence
    Reading level: 5 passages differ; the rest is the same for everyone.

    This explainer covers the main chip types used for AI, what training and inference ask of them, and why memory matters. Choose a reading level in each section. For one chip family in depth, read Google TPU. For the business question of whether custom chips threaten NVIDIA, read the custom chips debate.

    Training and inference

    An AI model is built in two stages. In training, the model makes guesses on example data, measures how wrong it was, and adjusts itself to do better. This repeats until the errors are small enough.[6] Training takes enormous amounts of maths. NVIDIA says training Llama 3.1, a 405-billion-parameter model, took about 3.8 x 10^25 operations.[7]

    In inference, the finished model is put to work answering questions or making predictions on new data.[4] A chatbot answering a user in real time is one kind of inference.[4]

    Training runs a forward pass over a batch, computes a loss, and uses an optimizer to update parameters, iterating until loss is acceptable.[6] Inference uses the trained weights on new inputs and needs only the forward pass.[4] The compute bill for frontier training is very large: NVIDIA puts Llama 3.1 405B at roughly 38 yottaflops of total operations.[7] Google split its eighth-generation TPU into a training chip and an inference chip, citing rising inference demand, and gave the inference chip more memory bandwidth for latency-sensitive serving.[8]

    GPUs: the general-purpose workhorse

    NVIDIA says it invented the GPU in 1999.[9] In 2006 NVIDIA released CUDA, software that opened the GPU’s parallel processing to general computing.[1] Today NVIDIA earns most of its revenue from data center chips.[10] AMD also sells data center GPUs, such as its Instinct MI400 series.[11]

    Modern processors increasingly pair general-purpose cores with accelerators such as GPUs to improve performance per watt.[12] CUDA, introduced in 2006, exposed GPU parallelism to general compute-intensive workloads.[1] GPUs keep their flexibility: Google’s own guidance recommends GPUs for models with many custom PyTorch or JAX operations, or with operations a TPU does not support.[13] NVIDIA’s Data Center segment brought in $89.0 billion in the quarter ended July 26, 2026.[10]

    TPUs and other custom ASICs

    An ASIC, or application-specific integrated circuit, is a chip built for one kind of job. Google’s Tensor Processing Units are ASICs built for the matrix maths at the heart of machine learning.[2] Google’s first TPU went into its data centers in 2015 to run trained models.[14] Other large companies design their own AI chips too: Amazon sells access to Trainium3[15], Meta runs hundreds of thousands of its MTIA chips[16], and OpenAI is designing accelerators with Broadcom.[17]

    The first TPU was built around a 65,536-unit 8-bit multiply-accumulate array with 28 MiB of software-managed on-chip memory.[18] Google’s 2017 paper reported it was on average 15 to 30 times faster than a contemporary K80 GPU or Haswell CPU on production inference, with 30 to 80 times better performance per watt.[19] The authors framed this as part of a wider shift toward domain-specific hardware.[20] The trade-off is narrower scope: Google says TPUs are not suited to high-precision arithmetic or code with frequent branching.[13] Some are designed with partners: Broadcom has a long-term agreement to develop and supply Google’s future TPUs.[21] Meta’s newest MTIA chips will mainly serve generative AI inference.[22]

    NPUs: AI on your own device

    A neural processing unit is an AI engine built into a PC. Microsoft describes it as a chip designed for the maths in AI models.[3] It handles features such as live translation and image generation, and uses less power for AI tasks than the main processor or graphics chip, which saves battery.[23]

    Microsoft says Copilot+ PCs have NPUs that can perform more than 40 TOPS.[3] The NPU works alongside the CPU and GPU, and Windows assigns work to whichever is most suitable.[23] Windows NPU support spans Arm-based Qualcomm devices as well as Intel and AMD devices.[24] Microsoft calls running AI tasks on the NPU inference, and recommends its Windows ML framework for it.[25]

    Why memory matters: HBM

    A fast chip is useless if it waits for data. AI chips therefore sit next to stacks of high-bandwidth memory, or HBM, joined on one package.[26] Compute power has grown faster than memory speed, so memory is now often the limit, especially when serving models to users.[27][5] Read more in the memory wall.

    A 2024 analysis found peak server FLOPS grew about 3.0x every two years against 1.6x for DRAM bandwidth, and argued memory bandwidth is the main bottleneck for serving.[27][5] Packaging such as TSMC’s CoWoS places processors and several HBM stacks on a shared interposer.[26] HBM4 doubles the I/O count to 2,048 connections.[28] Chip designers trade on this: Google’s TPU 8i pairs 288 GB of HBM with 384 MB of on-chip SRAM to keep a model’s working set close to compute.[29] How these chips are built is covered in how advanced chips are made, and the supply chain in why AI chip supply is concentrated.

    Next in the crash course

    More in Computing hardware

    Developing in AI & Computing

    Go deeper

    Questions readers ask

    What is the difference between a GPU and a TPU?

    A GPU is a parallel processor that NVIDIA opened to general computing in 2006 with CUDA. A TPU is Google's custom ASIC for machine learning, built for large matrix operations. Google's guidance says TPUs suit models dominated by matrix maths, while GPUs suit models with many custom PyTorch or JAX operations.[1][2][13]

    What is an NPU?

    Microsoft describes a neural processing unit as a specialized chip for the deep learning maths in AI models. Copilot+ PCs have NPUs rated at more than 40 trillion operations per second, and they run AI tasks more efficiently than the CPU or GPU.[3][23]

    What is the difference between AI training and inference?

    Training repeatedly adjusts a model's parameters to reduce its errors on training data. Inference uses the finished model to produce outputs from new data and needs only a forward pass through the model.[6][4]

    What is HBM and why does it matter for AI chips?

    High-bandwidth memory is stacked memory placed next to the processor. A 2024 analysis found compute grew about 3x every two years while memory bandwidth grew about 1.6x, and argued memory is now the main bottleneck, especially for serving models.[26][27][5]

    Do companies other than Google make custom AI chips?

    Yes. Amazon makes Trainium, Meta makes MTIA chips, and OpenAI is designing its own accelerators with Broadcom.[15][22][17]

    Sources

    · 29 claims from 18 publishers

    Each numbered claim is a statement we checked against the sources listed with it. Tap a number in the text to see its sources in place.

    1. [1]

      NVIDIA introduced CUDA in 2006, opening the parallel processing capabilities of its GPUs to general compute-intensive applications. confirmedas of 2026-02-25

    2. [2]

      Google describes TPUs as its custom-developed application-specific integrated circuits (ASICs) for accelerating machine learning, built for the large matrix operations common in machine learning and equipped with on-chip high-bandwidth memory (HBM). confirmedas of 2026-10-26

    3. [3]

      Microsoft describes a neural processing unit (NPU) as a specialized chip designed to execute the deep learning math in AI models, and says Copilot+ PCs have NPUs that can perform more than 40 trillion operations per second (TOPS). confirmedas of 2026-11-07

    4. [4]

      AI inference is the use of a trained model to make predictions or generate outputs from new data, including live interactions such as chatbots; unlike training, it needs only a forward pass through the model. confirmedas of 2026-11-07

    5. [5]

      The same analysis argues that memory bandwidth, rather than compute, has become the primary bottleneck for AI workloads, especially when serving models. confirmedas of 2024-03-21

      • AI and Memory Wall · arXiv (published in IEEE Micro) · Primary source · 2024-03-21 · Abstract (retrieved 2026-10-10)
    6. [6]

      In model training, a model makes predictions on training data, a loss function measures the error, and an optimization algorithm updates the model's parameters to reduce it, repeated until the loss is acceptably low. reportedas of 2026-11-07

    7. [7]

      NVIDIA says training Llama 3.1 with 405 billion parameters took roughly 38 yottaflops, or 3.8 x 10^25 math operations, over the whole training run. confirmedas of 2026-11-07

    8. [8]

      Google says it split the generation because it anticipated rising inference demand and AI agents; TPU 8t has more compute throughput and scale-up bandwidth for training, while TPU 8i has more memory bandwidth for latency-sensitive inference, though both can run either kind of workload. confirmedas of 2026-04-22

    9. [9]

      NVIDIA describes itself as having invented the GPU (graphics processing unit) in 1999. confirmedas of 2026-10-10

    10. [10]

      In the second quarter of fiscal 2027, NVIDIA's Data Center revenue was $89.0 billion, up 117% year over year, and Edge Computing revenue was $7.2 billion, up 27% year over year. confirmedas of 2026-07-26

    11. [11]

      AMD has unveiled its Helios rack-scale system and Instinct MI400 Series GPU family, which includes the MI455X. confirmedas of 2026-08-04

    12. [12]

      More recently, heterogeneous designs pair general-purpose cores with specialized accelerators such as GPUs to improve performance per watt. confirmedas of 2026-10-10

    13. [13]

      Google's guidance says TPUs suit models dominated by matrix computations, large models that train for weeks or months and models with very large embeddings, while GPUs suit models with many custom PyTorch or JAX operations, and TPUs are not suited to high-precision arithmetic or code with frequent branching. confirmedas of 2026-10-26

    14. [14]

      Google's first TPU was a custom ASIC deployed in its data centers from 2015 to accelerate the inference phase of neural networks. confirmedas of 2017-04-16

    15. [15]

      In December 2025 AWS made EC2 Trn3 UltraServers generally available, powered by Trainium3, its first 3nm AI chip, scaling to 144 chips and up to 362 FP8 petaflops, with Anthropic among the customers using Trainium. confirmedas of 2025-12-02

    16. [16]

      Meta says it deploys hundreds of thousands of MTIA chips for inference across content and ads, and that MTIA is more cost-efficient than general-purpose chips for those workloads. confirmedas of 2026-03-11

    17. [17]

      In October 2025 OpenAI and Broadcom announced a collaboration for 10 gigawatts of OpenAI-designed AI accelerators, with Broadcom deploying racks of accelerators and Ethernet networking from the second half of 2026 through the end of 2029. confirmedas of 2025-10-13

    18. [18]

      The first TPU was built around a 65,536-unit 8-bit multiply-accumulate matrix unit with a peak of 92 trillion operations per second and 28 MiB of software-managed on-chip memory. confirmedas of 2017-04-16

    19. [19]

      Google's 2017 paper reported that on its production inference workloads the first TPU was on average about 15 to 30 times faster than a contemporary Nvidia K80 GPU or Intel Haswell CPU, with 30 to 80 times higher performance per watt. confirmedas of 2017-04-16

    20. [20]

      The 2017 TPU paper argued that many architects believe major improvements in cost, energy and performance must now come from domain-specific hardware. confirmedas of 2017-04-16

    21. [21]

      In April 2026 Broadcom disclosed a long-term agreement to develop and supply custom TPUs for Google's future TPU generations, plus a supply agreement for networking and other components for Google's next-generation AI racks, through up to 2031. confirmedas of 2026-04-06

    22. [22]

      In March 2026 Meta said it is developing four new MTIA chip generations, the MTIA 300, 400, 450 and 500, within two years; MTIA 300 is in production for ranking and recommendation training, and the later chips will mainly serve generative AI inference into 2027. confirmedas of 2026-03-11

    23. [23]

      Microsoft says a PC's NPU runs AI tasks more energy-efficiently than the CPU or GPU, works alongside them while Windows assigns tasks to the most appropriate processor, and is used for on-device AI features such as real-time translation and image generation. confirmedas of 2026-11-07

    24. [24]

      Microsoft says Windows 11 NPU support covers Arm-based Qualcomm devices as well as Intel and AMD devices. confirmedas of 2026-11-07

    25. [25]

      Microsoft describes running AI tasks on a device NPU as inference and recommends Windows ML for it. confirmedas of 2026-11-07

    26. [26]

      TSMC's CoWoS packaging places processor chips and multiple high-bandwidth memory (HBM) stacks on a shared interposer, which can exceed twice the reticle size (about 1,700 mm²). confirmedas of 2026-10-10

      • CoWoS · TSMC · Primary source · Overview; the page renders the unit as mm with a superscript 2 (retrieved 2026-10-10)
      • CoWoS · TSMC · Primary source (retrieved 2026-10-10)
    27. [27]

      A 2024 analysis found peak server hardware FLOPS grew about 3.0x every two years, while DRAM bandwidth grew about 1.6x and interconnect bandwidth about 1.4x over the same period. confirmedas of 2024-03-21

      • AI and Memory Wall · arXiv (published in IEEE Micro) · Primary source · 2024-03-21 · Abstract (retrieved 2026-10-10)
    28. [28]

      On September 12, 2025 SK hynix said it had completed HBM4 development and prepared for mass production, doubling bandwidth with 2,048 I/O connections, improving power efficiency by more than 40% and exceeding the JEDEC 8Gbps standard speed with more than 10Gbps. confirmedas of 2025-09-12

    29. [29]

      Google says TPU 8i pairs 288 GB of HBM with 384 MB of on-chip SRAM, three times the previous generation, to keep a model's active working set on the chip. confirmedas of 2026-04-22

    Revision history (1)
    1. Page created.

    Created Nov 7, 2026. Last reviewed by an editor on Nov 8, 2026. Next scheduled review: Mar 7, 2027.

    Cite this page

    "GPUs, TPUs and NPUs explained: the chips that run AI." ContentLora, updated Nov 7, 2026. https://contentlora.com/explain/gpus-tpus-and-npus-explained

    Spotted an error? Suggest a correction or emailcorrections@contentlora.com.