Explainer
GPUs, TPUs and NPUs explained: the chips that run AI
AI runs mostly on three kinds of chip. Graphics processing units (GPUs) are general parallel processors that NVIDIA opened to general computing with CUDA in 2006.[1] Custom ASICs such as Google's TPUs are built for the matrix maths in machine learning.[2] Neural processing units (NPUs) are efficient AI engines built into PCs.[3] Training a model and running it (inference) put different loads on these chips, and memory bandwidth is often the limit.[4][5]
This explainer covers the main chip types used for AI, what training and inference ask of them, and why memory matters. Choose a reading level in each section. For one chip family in depth, read Google TPU. For the business question of whether custom chips threaten NVIDIA, read the custom chips debate.
Training and inference
An AI model is built in two stages. In training, the model makes guesses on example data, measures how wrong it was, and adjusts itself to do better. This repeats until the errors are small enough.[6] Training takes enormous amounts of maths. NVIDIA says training Llama 3.1, a 405-billion-parameter model, took about 3.8 x 10^25 operations.[7]
In inference, the finished model is put to work answering questions or making predictions on new data.[4] A chatbot answering a user in real time is one kind of inference.[4]
Training runs a forward pass over a batch, computes a loss, and uses an optimizer to update parameters, iterating until loss is acceptable.[6] Inference uses the trained weights on new inputs and needs only the forward pass.[4] The compute bill for frontier training is very large: NVIDIA puts Llama 3.1 405B at roughly 38 yottaflops of total operations.[7] Google split its eighth-generation TPU into a training chip and an inference chip, citing rising inference demand, and gave the inference chip more memory bandwidth for latency-sensitive serving.[8]
GPUs: the general-purpose workhorse
NVIDIA says it invented the GPU in 1999.[9] In 2006 NVIDIA released CUDA, software that opened the GPU’s parallel processing to general computing.[1] Today NVIDIA earns most of its revenue from data center chips.[10] AMD also sells data center GPUs, such as its Instinct MI400 series.[11]
Modern processors increasingly pair general-purpose cores with accelerators such as GPUs to improve performance per watt.[12] CUDA, introduced in 2006, exposed GPU parallelism to general compute-intensive workloads.[1] GPUs keep their flexibility: Google’s own guidance recommends GPUs for models with many custom PyTorch or JAX operations, or with operations a TPU does not support.[13] NVIDIA’s Data Center segment brought in $89.0 billion in the quarter ended July 26, 2026.[10]
TPUs and other custom ASICs
An ASIC, or application-specific integrated circuit, is a chip built for one kind of job. Google’s Tensor Processing Units are ASICs built for the matrix maths at the heart of machine learning.[2] Google’s first TPU went into its data centers in 2015 to run trained models.[14] Other large companies design their own AI chips too: Amazon sells access to Trainium3[15], Meta runs hundreds of thousands of its MTIA chips[16], and OpenAI is designing accelerators with Broadcom.[17]
The first TPU was built around a 65,536-unit 8-bit multiply-accumulate array with 28 MiB of software-managed on-chip memory.[18] Google’s 2017 paper reported it was on average 15 to 30 times faster than a contemporary K80 GPU or Haswell CPU on production inference, with 30 to 80 times better performance per watt.[19] The authors framed this as part of a wider shift toward domain-specific hardware.[20] The trade-off is narrower scope: Google says TPUs are not suited to high-precision arithmetic or code with frequent branching.[13] Some are designed with partners: Broadcom has a long-term agreement to develop and supply Google’s future TPUs.[21] Meta’s newest MTIA chips will mainly serve generative AI inference.[22]
NPUs: AI on your own device
A neural processing unit is an AI engine built into a PC. Microsoft describes it as a chip designed for the maths in AI models.[3] It handles features such as live translation and image generation, and uses less power for AI tasks than the main processor or graphics chip, which saves battery.[23]
Microsoft says Copilot+ PCs have NPUs that can perform more than 40 TOPS.[3] The NPU works alongside the CPU and GPU, and Windows assigns work to whichever is most suitable.[23] Windows NPU support spans Arm-based Qualcomm devices as well as Intel and AMD devices.[24] Microsoft calls running AI tasks on the NPU inference, and recommends its Windows ML framework for it.[25]
Why memory matters: HBM
A fast chip is useless if it waits for data. AI chips therefore sit next to stacks of high-bandwidth memory, or HBM, joined on one package.[26] Compute power has grown faster than memory speed, so memory is now often the limit, especially when serving models to users.[27][5] Read more in the memory wall.
A 2024 analysis found peak server FLOPS grew about 3.0x every two years against 1.6x for DRAM bandwidth, and argued memory bandwidth is the main bottleneck for serving.[27][5] Packaging such as TSMC’s CoWoS places processors and several HBM stacks on a shared interposer.[26] HBM4 doubles the I/O count to 2,048 connections.[28] Chip designers trade on this: Google’s TPU 8i pairs 288 GB of HBM with 384 MB of on-chip SRAM to keep a model’s working set close to compute.[29] How these chips are built is covered in how advanced chips are made, and the supply chain in why AI chip supply is concentrated.
Read next
Next in the crash course
TSMC (Taiwan Semiconductor Manufacturing Company)
TSMC is the Taiwanese contract chipmaker that builds chips designed by other companies. Its history, scale, process technology and 2026 results.
Questions readers ask
What is the difference between a GPU and a TPU?
A GPU is a parallel processor that NVIDIA opened to general computing in 2006 with CUDA. A TPU is Google's custom ASIC for machine learning, built for large matrix operations. Google's guidance says TPUs suit models dominated by matrix maths, while GPUs suit models with many custom PyTorch or JAX operations.[1][2][13]
What is an NPU?
Microsoft describes a neural processing unit as a specialized chip for the deep learning maths in AI models. Copilot+ PCs have NPUs rated at more than 40 trillion operations per second, and they run AI tasks more efficiently than the CPU or GPU.[3][23]
What is the difference between AI training and inference?
Training repeatedly adjusts a model's parameters to reduce its errors on training data. Inference uses the finished model to produce outputs from new data and needs only a forward pass through the model.[6][4]
What is HBM and why does it matter for AI chips?
High-bandwidth memory is stacked memory placed next to the processor. A 2024 analysis found compute grew about 3x every two years while memory bandwidth grew about 1.6x, and argued memory is now the main bottleneck, especially for serving models.[26][27][5]
Sources
· 29 claims from 18 publishers
Each numbered claim is a statement we checked against the sources listed with it. Tap a number in the text to see its sources in place.
- [1]
NVIDIA introduced CUDA in 2006, opening the parallel processing capabilities of its GPUs to general compute-intensive applications. confirmedas of 2026-02-25
- NVIDIA Corporation Form 10-K for fiscal year ended January 25, 2026 · NVIDIA (SEC filing) · Primary source · 2026-02-25 · Item 1. Business (retrieved 2026-10-10)
- [2]
Google describes TPUs as its custom-developed application-specific integrated circuits (ASICs) for accelerating machine learning, built for the large matrix operations common in machine learning and equipped with on-chip high-bandwidth memory (HBM). confirmedas of 2026-10-26
- Introduction to Cloud TPU · Google Cloud documentation · Primary source (retrieved 2026-11-07)
- Introduction to Cloud TPU · Google Cloud documentation · Primary source (retrieved 2026-11-07)
- Introduction to Cloud TPU · Google Cloud documentation · Primary source (retrieved 2026-11-07)
- [3]
Microsoft describes a neural processing unit (NPU) as a specialized chip designed to execute the deep learning math in AI models, and says Copilot+ PCs have NPUs that can perform more than 40 trillion operations per second (TOPS). confirmedas of 2026-11-07
- Copilot+ PCs developer guide · Microsoft Learn · Primary source (retrieved 2026-11-07)
- Copilot+ PCs developer guide · Microsoft Learn · Primary source (retrieved 2026-11-07)
- [4]
AI inference is the use of a trained model to make predictions or generate outputs from new data, including live interactions such as chatbots; unlike training, it needs only a forward pass through the model. confirmedas of 2026-11-07
- What is AI inference? · IBM Think · Secondary source · Opening (retrieved 2026-11-07)
- What is AI inference? · IBM Think · Secondary source · Training vs inference section (retrieved 2026-11-07)
- What is AI inference? · IBM Think · Secondary source · Online inference (retrieved 2026-11-07)
- What's the Difference Between Deep Learning Training and Inference? · NVIDIA Blog · Primary source · 2016-07-29 (retrieved 2026-11-07)
- [5]
The same analysis argues that memory bandwidth, rather than compute, has become the primary bottleneck for AI workloads, especially when serving models. confirmedas of 2024-03-21
- AI and Memory Wall · arXiv (published in IEEE Micro) · Primary source · 2024-03-21 · Abstract (retrieved 2026-10-10)
- [6]
In model training, a model makes predictions on training data, a loss function measures the error, and an optimization algorithm updates the model's parameters to reduce it, repeated until the loss is acceptably low. reportedas of 2026-11-07
- What is AI inference? · IBM Think · Secondary source · Training vs inference section (retrieved 2026-11-07)
- What is AI inference? · IBM Think · Secondary source (retrieved 2026-11-07)
- [7]
NVIDIA says training Llama 3.1 with 405 billion parameters took roughly 38 yottaflops, or 3.8 x 10^25 math operations, over the whole training run. confirmedas of 2026-11-07
- What's the Difference Between Deep Learning Training and Inference? · NVIDIA Blog · Primary source · 2016-07-29 (retrieved 2026-11-07)
- [8]
Google says it split the generation because it anticipated rising inference demand and AI agents; TPU 8t has more compute throughput and scale-up bandwidth for training, while TPU 8i has more memory bandwidth for latency-sensitive inference, though both can run either kind of workload. confirmedas of 2026-04-22
- Our eighth generation TPUs: two chips for the agentic era · Google (The Keyword) · Primary source · 2026-04-22 (retrieved 2026-11-07)
- Our eighth generation TPUs: two chips for the agentic era · Google (The Keyword) · Primary source · 2026-04-22 (retrieved 2026-11-07)
- Our eighth generation TPUs: two chips for the agentic era · Google (The Keyword) · Primary source · 2026-04-22 (retrieved 2026-11-07)
- Our eighth generation TPUs: two chips for the agentic era · Google (The Keyword) · Primary source · 2026-04-22 (retrieved 2026-11-07)
- [9]
NVIDIA describes itself as having invented the GPU (graphics processing unit) in 1999. confirmedas of 2026-10-10
- NVIDIA corporate timeline · NVIDIA · Primary source · 1999 (retrieved 2026-10-10)
- [10]
In the second quarter of fiscal 2027, NVIDIA's Data Center revenue was $89.0 billion, up 117% year over year, and Edge Computing revenue was $7.2 billion, up 27% year over year. confirmedas of 2026-07-26
- NVIDIA Announces Financial Results for Second Quarter Fiscal 2027 · NVIDIA Newsroom · Primary source · 2026-08-26 (retrieved 2026-10-10)
- NVIDIA Announces Financial Results for Second Quarter Fiscal 2027 · NVIDIA Newsroom · Primary source · 2026-08-26 (retrieved 2026-10-10)
- [11]
AMD has unveiled its Helios rack-scale system and Instinct MI400 Series GPU family, which includes the MI455X. confirmedas of 2026-08-04
- AMD Reports Second Quarter 2026 Financial Results · AMD Newsroom · Primary source · 2026-08-04 · Recent highlights (retrieved 2026-10-10)
- [12]
More recently, heterogeneous designs pair general-purpose cores with specialized accelerators such as GPUs to improve performance per watt. confirmedas of 2026-10-10
- A 20-Year Retrospective on Power and Thermal Modeling and Management · arXiv · Primary source · 2025-08-07 · Introduction (retrieved 2026-10-10)
- [13]
Google's guidance says TPUs suit models dominated by matrix computations, large models that train for weeks or months and models with very large embeddings, while GPUs suit models with many custom PyTorch or JAX operations, and TPUs are not suited to high-precision arithmetic or code with frequent branching. confirmedas of 2026-10-26
- Introduction to Cloud TPU · Google Cloud documentation · Primary source · When to use TPUs (retrieved 2026-11-07)
- Introduction to Cloud TPU · Google Cloud documentation · Primary source · When to use TPUs (retrieved 2026-11-07)
- Introduction to Cloud TPU · Google Cloud documentation · Primary source · When to use TPUs (retrieved 2026-11-07)
- Introduction to Cloud TPU · Google Cloud documentation · Primary source · When to use TPUs (retrieved 2026-11-07)
- [14]
Google's first TPU was a custom ASIC deployed in its data centers from 2015 to accelerate the inference phase of neural networks. confirmedas of 2017-04-16
- In-Datacenter Performance Analysis of a Tensor Processing Unit · arXiv (Jouppi et al., Google) · Primary source · 2017-04-16 · Abstract (retrieved 2026-11-07)
- [15]
In December 2025 AWS made EC2 Trn3 UltraServers generally available, powered by Trainium3, its first 3nm AI chip, scaling to 144 chips and up to 362 FP8 petaflops, with Anthropic among the customers using Trainium. confirmedas of 2025-12-02
- Trainium3 UltraServers now available: Enabling customers to train and deploy AI models faster at lower cost · Amazon (About Amazon) · Primary source · 2025-12-02 (retrieved 2026-11-07)
- Trainium3 UltraServers now available: Enabling customers to train and deploy AI models faster at lower cost · Amazon (About Amazon) · Primary source · 2025-12-02 (retrieved 2026-11-07)
- Trainium3 UltraServers now available: Enabling customers to train and deploy AI models faster at lower cost · Amazon (About Amazon) · Primary source · 2025-12-02 (retrieved 2026-11-07)
- Trainium3 UltraServers now available: Enabling customers to train and deploy AI models faster at lower cost · Amazon (About Amazon) · Primary source · 2025-12-02 (retrieved 2026-11-07)
- [16]
Meta says it deploys hundreds of thousands of MTIA chips for inference across content and ads, and that MTIA is more cost-efficient than general-purpose chips for those workloads. confirmedas of 2026-03-11
- Expanding Meta's custom silicon to power our AI workloads · Meta Newsroom · Primary source · 2026-03-11 (retrieved 2026-11-07)
- Expanding Meta's custom silicon to power our AI workloads · Meta Newsroom · Primary source · 2026-03-11 (retrieved 2026-11-07)
- [17]
In October 2025 OpenAI and Broadcom announced a collaboration for 10 gigawatts of OpenAI-designed AI accelerators, with Broadcom deploying racks of accelerators and Ethernet networking from the second half of 2026 through the end of 2029. confirmedas of 2025-10-13
- OpenAI and Broadcom announce strategic collaboration to deploy 10 gigawatts of OpenAI-designed AI accelerators · Broadcom and OpenAI (GlobeNewswire press release, via MFN) · Primary source · 2025-10-13 (retrieved 2026-11-07)
- OpenAI and Broadcom announce strategic collaboration to deploy 10 gigawatts of OpenAI-designed AI accelerators · Broadcom and OpenAI (GlobeNewswire press release, via MFN) · Primary source · 2025-10-13 (retrieved 2026-11-07)
- [18]
The first TPU was built around a 65,536-unit 8-bit multiply-accumulate matrix unit with a peak of 92 trillion operations per second and 28 MiB of software-managed on-chip memory. confirmedas of 2017-04-16
- In-Datacenter Performance Analysis of a Tensor Processing Unit · arXiv (Jouppi et al., Google) · Primary source · 2017-04-16 · Abstract (retrieved 2026-11-07)
- [19]
Google's 2017 paper reported that on its production inference workloads the first TPU was on average about 15 to 30 times faster than a contemporary Nvidia K80 GPU or Intel Haswell CPU, with 30 to 80 times higher performance per watt. confirmedas of 2017-04-16
- In-Datacenter Performance Analysis of a Tensor Processing Unit · arXiv (Jouppi et al., Google) · Primary source · 2017-04-16 · Abstract (retrieved 2026-11-07)
- In-Datacenter Performance Analysis of a Tensor Processing Unit · arXiv (Jouppi et al., Google) · Primary source · 2017-04-16 · Abstract (retrieved 2026-11-07)
- [20]
The 2017 TPU paper argued that many architects believe major improvements in cost, energy and performance must now come from domain-specific hardware. confirmedas of 2017-04-16
- In-Datacenter Performance Analysis of a Tensor Processing Unit · arXiv (Jouppi et al., Google) · Primary source · 2017-04-16 · Abstract (retrieved 2026-11-07)
- [21]
In April 2026 Broadcom disclosed a long-term agreement to develop and supply custom TPUs for Google's future TPU generations, plus a supply agreement for networking and other components for Google's next-generation AI racks, through up to 2031. confirmedas of 2026-04-06
- Broadcom Inc. Form 8-K (Google TPU long-term agreement and Anthropic capacity), April 6, 2026 · Broadcom (SEC filing) · Primary source · 2026-04-06 (retrieved 2026-11-07)
- Broadcom Inc. Form 8-K (Google TPU long-term agreement and Anthropic capacity), April 6, 2026 · Broadcom (SEC filing) · Primary source · 2026-04-06 (retrieved 2026-11-07)
- [22]
In March 2026 Meta said it is developing four new MTIA chip generations, the MTIA 300, 400, 450 and 500, within two years; MTIA 300 is in production for ranking and recommendation training, and the later chips will mainly serve generative AI inference into 2027. confirmedas of 2026-03-11
- Expanding Meta's custom silicon to power our AI workloads · Meta Newsroom · Primary source · 2026-03-11 (retrieved 2026-11-07)
- [23]
Microsoft says a PC's NPU runs AI tasks more energy-efficiently than the CPU or GPU, works alongside them while Windows assigns tasks to the most appropriate processor, and is used for on-device AI features such as real-time translation and image generation. confirmedas of 2026-11-07
- Copilot+ PCs developer guide · Microsoft Learn · Primary source (retrieved 2026-11-07)
- Copilot+ PCs developer guide · Microsoft Learn · Primary source (retrieved 2026-11-07)
- Copilot+ PCs developer guide · Microsoft Learn · Primary source (retrieved 2026-11-07)
- [24]
Microsoft says Windows 11 NPU support covers Arm-based Qualcomm devices as well as Intel and AMD devices. confirmedas of 2026-11-07
- Copilot+ PCs developer guide · Microsoft Learn · Primary source (retrieved 2026-11-07)
- [25]
Microsoft describes running AI tasks on a device NPU as inference and recommends Windows ML for it. confirmedas of 2026-11-07
- Copilot+ PCs developer guide · Microsoft Learn · Primary source (retrieved 2026-11-07)
- [26]
TSMC's CoWoS packaging places processor chips and multiple high-bandwidth memory (HBM) stacks on a shared interposer, which can exceed twice the reticle size (about 1,700 mm²). confirmedas of 2026-10-10
- [27]
A 2024 analysis found peak server hardware FLOPS grew about 3.0x every two years, while DRAM bandwidth grew about 1.6x and interconnect bandwidth about 1.4x over the same period. confirmedas of 2024-03-21
- AI and Memory Wall · arXiv (published in IEEE Micro) · Primary source · 2024-03-21 · Abstract (retrieved 2026-10-10)
- [28]
On September 12, 2025 SK hynix said it had completed HBM4 development and prepared for mass production, doubling bandwidth with 2,048 I/O connections, improving power efficiency by more than 40% and exceeding the JEDEC 8Gbps standard speed with more than 10Gbps. confirmedas of 2025-09-12
- SK hynix Completes World's First HBM4 Development and Readies Mass Production · SK hynix Newsroom · Primary source · 2025-09-12 (retrieved 2026-10-10)
- SK hynix Completes World's First HBM4 Development and Readies Mass Production · SK hynix Newsroom · Primary source · 2025-09-12 (retrieved 2026-10-10)
- [29]
Google says TPU 8i pairs 288 GB of HBM with 384 MB of on-chip SRAM, three times the previous generation, to keep a model's active working set on the chip. confirmedas of 2026-04-22
- Our eighth generation TPUs: two chips for the agentic era · Google (The Keyword) · Primary source · 2026-04-22 (retrieved 2026-11-07)
- Our eighth generation TPUs: two chips for the agentic era · Google (The Keyword) · Primary source · 2026-04-22 (retrieved 2026-11-07)
Revision history (1)
- Page created.
Created Nov 7, 2026. Last reviewed by an editor on Nov 8, 2026. Next scheduled review: Mar 7, 2027.
Cite this page
"GPUs, TPUs and NPUs explained: the chips that run AI." ContentLora, updated Nov 7, 2026. https://contentlora.com/explain/gpus-tpus-and-npus-explained
Spotted an error? Suggest a correction or emailcorrections@contentlora.com.