Technology

What GPUs are and why AI needs so many of them

A GPU is a chip built to do thousands of simple calculations at once, which happens to be exactly what AI models need. Here is how it differs from a CPU, why memory and networking matter as much as raw speed, why Nvidia leads, and what India's subsidised GPU pool offers.

Chander Prakash

By Founder & Editor

8 min read

Illustrative cover: What GPUs are and why AI needs so many of them
Illustration: Pointales

A GPU, or graphics processing unit, is a chip designed to do a very large number of simple calculations at the same time. It was built to colour millions of screen pixels at once for video games, but the same trick turns out to be what artificial intelligence needs. Training and running an AI model comes down to multiplying huge grids of numbers, and a GPU can split that work across thousands of small processing units in parallel. That is why GPUs, not ordinary processors, have become the scarce, expensive engine of the AI boom.

CPU vs GPU, in plain terms

Every computer has a CPU, the central processing unit. It is a generalist: a small number of powerful cores, each very good at working through a sequence of different instructions quickly, with lots of logic for making decisions and juggling tasks. A GPU takes the opposite approach: many more, much simpler cores that all do similar arithmetic on different pieces of data at once.

Nvidia’s own CUDA programming guide puts the difference neatly. A CPU is designed to run a sequence of operations “as fast as possible and can execute a few tens of these threads in parallel”, while a GPU is designed to run “thousands of threads in parallel, trading off lower single-thread performance to achieve much greater total throughput”. GPUs, it adds, “devote more transistors to data processing units, while CPUs dedicate more transistors to data caching and flow control.”

A rough analogy: a CPU is a handful of senior chefs who can cook anything; a GPU is a kitchen of thousands of assistants who can each only chop, but all chop at the same time. If the job is a million onions, the assistants win.

CPUGPU
Design goalFinish one task fast (low latency)Finish many identical tasks together (high throughput)
CoresFew, powerful: AMD’s top 5th-generation EPYC server chip has 192 coresMany, simple: Nvidia lists 16,896 “CUDA cores” plus 528 Tensor Cores on its H100
Good atOperating systems, databases, web servers, branching logicGraphics, simulations, and the matrix maths behind AI
MemoryLarge, cheaper system RAMSmaller, much faster memory on the card (HBM or VRAM)
Role in AICoordinates the job, prepares dataDoes the heavy arithmetic

The core counts are not a like-for-like comparison, because a GPU “core” is a far simpler unit than a CPU core. The point is the shape of the design, not the number.

Why AI is mostly matrix multiplication

A neural network, the kind of model behind chatbots and image generators, stores what it has learned as billions of numbers called parameters, arranged in grids (matrices). When text goes in, it is converted to numbers and multiplied through layer after layer of those grids. Our guide to how large language models work explains the steps.

Nvidia’s deep learning performance guide calls general matrix multiplications “a fundamental building block for many operations in neural networks”. Multiplying two big matrices means doing millions of small multiply-and-add sums that do not depend on each other, so they can all run at once. That is a perfect fit for a GPU. Modern GPUs add Tensor Cores, units built specifically for matrix maths, and support for lower-precision numbers (16-bit or 8-bit instead of 32-bit), which let them do more sums per second.

Training vs inference: both need GPUs

Training is when a model learns. It reads trillions of words, makes predictions, measures its errors and adjusts every parameter slightly, over and over. Meta’s Llama 3 paper is one of the few to disclose the scale: its largest model was trained on “up to 16K H100 GPUs”, using 3.8 × 10²⁵ floating-point operations. Meta’s model card puts the training at 30.84 million GPU hours for that one model. On a single GPU, that would take about 3,500 years.

Inference is when a trained model is used: every chatbot reply, every AI search answer. Each response is smaller work, but it happens millions of times a day, must come back in seconds, and grows when tools look things up first or run as AI agents that take many steps per task. That is why companies keep buying GPUs even after their models are trained, and why Google describes its newest TPU chip as built for “high-volume, low-latency AI inference”. Not all inference needs a data centre, though: small models can now run on the chip inside a phone.

The real bottleneck: memory and connections

Raw arithmetic speed is only half the story. A GPU can only work on numbers it can reach quickly, and feeding it data is often the slower part.

Memory on the chip. AI GPUs use high-bandwidth memory (HBM): memory chips stacked on top of each other and placed right beside the processor, so data travels a very short distance. Gaming cards use a related type called VRAM. Nvidia’s H100 has 80GB of memory with 3.35 terabytes per second of bandwidth. AMD’s Instinct MI355X lists 288GB at 8 TB/s. Nvidia’s guide notes that many AI operations are “memory limited”: the GPU sits waiting for data rather than doing sums.

Why one GPU is not enough. Simple arithmetic shows the problem. A model with 405 billion parameters, stored at 2 bytes each, needs about 810GB just to hold its weights: more than ten 80GB GPUs before it has processed a single word. Training needs several times more, for the running calculations and corrections.

Connections between GPUs. So the model is split across many GPUs, and they must constantly swap results. Nvidia’s NVLink connects GPUs inside a server at 900 GB/s on the H100, against 128 GB/s for a standard PCIe connection, according to its spec sheet. Its GB200 NVL72 rack links 72 Blackwell GPUs that Nvidia says act “as a single, massive GPU”, with 13.4TB of combined GPU memory. Beyond the rack, thousands of servers are joined by high-speed networks; Llama 3 used 400 Gbps links between GPUs.

At this scale, hardware failure becomes routine. In a 54-day stretch of Llama 3 training, Meta recorded 466 job interruptions, and said about 78% of the unexpected ones were due to confirmed hardware issues.

Why so many: size, data and time

The number of GPUs a project needs comes from three things multiplied together:

  1. Model size. More parameters means more memory and more sums per word.
  2. Training data. Llama 3 was trained on about 15 trillion tokens; each token must be run forwards through the whole model and then backwards to correct it.
  3. Time. A run that would take millennia on one chip must finish in weeks or months, so the work is spread across thousands.

Then inference adds a second, ongoing fleet sized to the number of users. Even well-run training does not use the chips fully: Meta reported using 38% to 43% of the GPUs’ theoretical peak.

Why Nvidia dominates, and the alternatives

Nvidia’s lead is as much about software as silicon (for what chip labels such as “3nm” actually mean, see our explainer on 3nm and 2nm). In 2006 it introduced CUDA, a platform that let programmers use GPUs for general calculations rather than just graphics. Two decades of libraries, tools, tutorials and trained engineers have been built around it. Switching chips can mean rewriting and re-testing code, which is a real cost for any team.

There are serious alternatives, each with trade-offs:

  • AMD sells Instinct GPUs with large memory, and its open-source ROCm software aims to run the same AI frameworks.
  • Google designs its own Tensor Processing Units (TPUs), available through Google Cloud. Its seventh-generation Ironwood scales to 9,216 chips in one “superpod”, according to Google.
  • Custom chips from cloud companies, such as Amazon’s Trainium, which AWS describes as “purpose-built” for AI training and inference.

Performance comparisons between these chips come mostly from the vendors themselves, often using test conditions that favour their own product, so treat headline “X times faster” claims with care. Nvidia’s H100 and AMD’s MI355X are both made by TSMC in Taiwan, according to their makers, a dependence our explainer on India’s semiconductor push discusses.

Energy and cost

GPUs are power-hungry. Each H100 is rated at up to 700 watts, and a training cluster runs thousands of them, plus cooling. Meta wrote that when its GPUs all paused or restarted together, power use across the data centre swung by “tens of megawatts”, “stretching the limits of the power grid”.

The International Energy Agency’s Energy and AI report (April 2025) estimated that data centres used about 415 terawatt-hours of electricity in 2024, around 1.5% of the world’s total, and projected this to roughly double to about 945 TWh by 2030 in its base case, with AI the main driver. That cost in chips, power and cooling is why AI companies charge per token and why compute has become a matter of national policy.

India’s angle: shared, subsidised GPUs

Few Indian start-ups or universities can buy thousands of GPUs. The IndiaAI Mission, approved with an outlay of about ₹10,372 crore, instead pays private data-centre providers to offer GPU time to eligible users at a subsidised rate.

According to a PIB note of 10 February 2026, more than 38,000 GPUs had been onboarded and were available at ₹65 per GPU-hour, “nearly one third of the global average cost”, along with 1,050 TPUs. A PIB factsheet dated 13 August 2026 said shared capacity had grown to more than 45,000 GPUs as of June 2026, and that 237 projects had used 93.18 lakh (about 9.3 million) GPU hours by August 2026.

For scale, that national total is roughly a third of the 30.84 million GPU hours Meta reports for training one large model, though the chips involved differ. It is a meaningful start for Indian researchers and start-ups, but it shows how much compute frontier AI consumes. Rates and capacity change with each procurement round, so check the IndiaAI compute portal for current terms.

The point: A GPU is a chip of thousands of simple cores built for parallel arithmetic, and AI is mostly parallel arithmetic, which is why the two fit. What makes AI need so many is model size, data volume and deadlines, with memory and networking as the real limits. Nvidia leads largely through its CUDA software, and India’s answer so far is a shared, subsidised GPU pool rather than ownership.

Sources

Share