Running a language model on your own machine has stopped being exotic. The question that decides whether it is pleasant or miserable is not the processor, and it is not the number of TOPS on the box. It is memory, in two separate senses, and both of them are arithmetic you can do in your head before downloading a nine gigabyte file.
Capacity decides whether it runs at all. Bandwidth decides how fast it answers. Almost every disappointing local-AI experience is one of those two numbers being wrong, and neither of them appears in laptop marketing.
Multiplication one: does it fit
A model is a pile of numbers called parameters. How much space it takes is the parameter count times the size of each number, and the format decides the size:
| Format | Bytes per parameter | An 8B model | Quality |
|---|---|---|---|
| FP32 | 4 | 32 GB | The training format. Nobody runs this locally. |
| FP16 / BF16 | 2 | 16 GB | The reference. What "full precision" usually means in practice. |
| 8-bit | 1 | 8 GB | Very close to the reference for most uses. |
| 4-bit (common k-quants) | ~0.5 to 0.6 | ~4.5 to 5 GB | The usual choice. A small, measurable loss most people do not notice. |
That is the whole rule: parameters times bytes per parameter. A 3B model at 4-bit is under two gigabytes. A 70B model at 4-bit is around forty. The 4-bit column is slightly above the theoretical 0.5 bytes because these formats keep some values at higher precision where it matters, which is why a "4-bit" 8B file on disk is nearer five gigabytes than four.
What else takes memory
The weights are the big number, not the only one.
The context cache. As the model reads your prompt and writes its answer it keeps a cache of intermediate state for every token in the conversation. It grows with the length of the conversation, and on a small model with a long context it can reach a gigabyte or more. This is why a model that started fine gets slower and heavier deep into a long chat, and why raising the context window is not free.
The runtime and the operating system. Windows and a browser will happily use six to eight gigabytes between them before you start. Memory that is not there gets paged to disk, and a model that spills to disk does not run slowly, it stops being usable.
Multiplication two: how fast can it possibly be
This is the part almost nobody explains, and it predicts your experience better than any benchmark list.
To produce one token, the model reads all of its weights. Every token. So the ceiling on generation speed is simply:
tokens per second = memory bandwidth / model size in memoryA 4.5 GB model on a machine with 135 GB/s of memory bandwidth: 135 / 4.5 = 30 tokens per second, and that is the number you cannot exceed even with a perfect implementation. Real output typically lands somewhere between half and three quarters of it.
Two consequences follow immediately, and both are counterintuitive until you have seen the formula.
A smaller model is faster in direct proportion. Halve the model and you double the ceiling. That is why 4-bit quantization does not merely make a model fit, it makes it roughly twice as fast as the same model at 8-bit on the same hardware.
A faster processor does not help. If the bottleneck is reading weights out of memory, more arithmetic capacity has nothing to do. This is the single reason a laptop with an impressive TOPS figure can be slower at chat than a desktop graphics card of the same age.
The same sum on real machines
Memory bandwidth is bus width times transfer rate, so you can compute it rather than look it up. A 128-bit bus is 16 bytes wide, so LPDDR5X at 8533 MT/s gives 8533 × 16 = about 136 GB/s. A 384-bit GDDR6X bus is 48 bytes wide, so 21 Gbps per pin gives 48 × 21 = about 1008 GB/s.
| Machine | Bandwidth (approx) | Ceiling on a 4.5 GB model | Ceiling on a 20 GB model |
|---|---|---|---|
| Typical thin laptop, DDR5-5600 dual channel | ~90 GB/s | ~20 tok/s | does not fit in a 16 GB machine |
| Copilot+ class, LPDDR5X-8533 on 128 bits | ~136 GB/s | ~30 tok/s | ~7 tok/s if it fits |
| Desktop card with 384-bit GDDR6X at 21 Gbps | ~1008 GB/s | ~220 tok/s | ~50 tok/s |
| Desktop card with 512-bit GDDR7 at 28 Gbps | ~1792 GB/s | ~400 tok/s | ~90 tok/s |
The gap between the top and bottom rows is roughly twentyfold, and none of it is about how clever the chip is. It is about how many bytes per second can be dragged out of memory. That is also the honest answer to "should I buy an AI laptop or a graphics card": if the goal is a large model answering quickly, buy bandwidth.
Why the NPU does not rescue this
An NPU is very good at what it was built for, and this is not it.
On a laptop the NPU usually has no memory of its own. It borrows system RAM, which is why Windows reports NPU Shared Memory rather than dedicated memory for most machines. Sharing the RAM means sharing its bandwidth, so the ceiling above applies to the NPU exactly as it applies to the CPU.
What the NPU changes is the power bill. The same small model, running continuously, costs a fraction of the watts it would on the CPU or the integrated GPU, and the rest of the machine stays free and quiet. For a model that runs for hours in the background, that is the whole point. For a large model you are waiting on, it is not the lever you need. Our guide on what TOPS actually measures covers why the headline number never mentions any of this.
There is a second, more practical obstacle. A model only runs on the NPU if it has been quantized and compiled for that hardware, and Windows ML falls back to the GPU or the CPU when it has not been. So the honest question is not "is my NPU fast enough for this model" but "does this model exist in a build for my NPU at all". Today, for most models, the answer is no.
What to run on what you have
| You have | Comfortable | Possible but tight | Do not bother |
|---|---|---|---|
| 8 GB total RAM | 3B at 4-bit | 7-8B at 4-bit with nothing else open | anything above 8B |
| 16 GB total RAM | 7-8B at 4-bit | 14B at 4-bit | 30B and above |
| 32 GB total RAM | 14B at 4-bit | 30B at 4-bit, slowly | 70B |
| 8 GB graphics card | 7-8B at 4-bit, fast | 14B at 4-bit partially offloaded | anything that spills to system RAM |
| 24 GB graphics card | 30B at 4-bit | 70B at 4-bit, partially | — |
"Tight" in that table means it runs and you will feel it: less room for a long conversation, and a noticeable slowdown as the context grows. Splitting a model between a graphics card and system RAM works, and it runs at the speed of the slower half, which is the system RAM.
Where PC Workman fits
The part of this you cannot estimate from a table is what your machine is already using before the model starts, and that is measurable today. PC Workman reads memory pressure, what is holding it, and whether the system has started paging to disk, which is the moment a local model goes from slow to unusable.
It does not read NPU counters and it does not benchmark models. What it can tell you is whether the eight gigabytes you were planning to spend on a model are actually free on your machine, or whether a browser and a launcher have already taken them.
Related: why is my RAM usage so high · how much VRAM you need for games · all guides →
The capacity and bandwidth figures here are arithmetic you can repeat: bytes per parameter times parameter count, and bus width in bytes times transfer rate. Platform context from Microsoft, Copilot+ PCs developer guide (quantization to integer formats, NPU shared memory) and Windows ML execution providers (provider selection and fallback to GPU or CPU).