What Is Quantization in Local AI in Plain English

By Austin Little

Quantization is the reason a model trained in a data center can run on your kitchen-table laptop at all. It's a kind of careful rounding. This guide explains what's being rounded, why that rounding hurts some jobs more than others, and how to run your own side-by-side test so you're not taking anyone's word for it, including ours.

AI disclosure. This page was drafted with AI assistance and edited for Apiary. Facts that need a live check are marked. We don't invent quotes, stats, people, or events.

Where this fits with our other guides

Apiary already has two close companions, and you might want one of them instead:

If you're staring at a download list with labels like Q4_K_M and Q8_0 and want to know which button to click, read What Is a Quantized Model (Q4, Q8) in Plain English. That one decodes the labels and walks through picking a file for your memory.
If you want to know what the big .gguf file itself is, read What Is a GGUF File and Why Local AI Uses It.

This piece is about the idea of quantization and about testing it. What does it do to the model's "thinking"? Why does it barely touch a grocery list but sometimes trip up arithmetic or careful instructions? How do you check, on your own machine and with your own tasks, whether the smaller version is good enough? Think of it as the "why and how to judge it" article, with the label decoder living next door.

The short answer

An AI language model is, underneath, a very large pile of numbers. People call them weights or parameters. When the model writes a sentence, your computer does a huge amount of arithmetic with those numbers, word piece after word piece.

When a model is first trained, those numbers are usually stored with a lot of precision, many digits after the decimal point. That precision takes space. Quantization stores the same numbers with fewer digits. The model gets smaller on disk and in memory, and it usually runs faster on ordinary hardware. The cost is a little accuracy, because rounding throws away detail.

The llama.cpp project, which a lot of local AI tools are built on, puts it plainly in its quantization tool's documentation: quantization "reduces the precision of model weights (e.g., from 32-bit floats to 4-bit integers), which shrinks the model's size and can speed up inference," and "may introduce some accuracy loss" (llama.cpp quantize README).

That's it. Everything else in this article is about what "a little accuracy" means in practice and how to measure it for yourself.

A kitchen analogy that holds up

Picture a recipe binder from a very fussy test kitchen. Every measurement is written to absurd precision: 4.2371 grams of salt, 118.304 milliliters of milk. The full binder fills a shelf.

Now you want a version that fits in your apron pocket. You round. 4.2371 grams becomes "a scant teaspoon." 118.304 milliliters becomes "half a cup." Most dishes still come out the same. A soup doesn't care about the fourth decimal place of salt.

But a few recipes are touchy. Baking chemistry, candy temperatures, a sauce that breaks if the ratio drifts. Round those too hard and the result changes. Not every time, and not dramatically, but you notice.

That's quantization. The model's "recipes" for predicting the next word are mostly forgiving. A few kinds of tasks are touchy. And just like in the kitchen, rounding gently (keeping more digits) almost never hurts, while rounding hard (keeping very few) starts to show on the touchy jobs.

One more thing the analogy gets right: a good cook doesn't round every line the same way. They keep the baking soda precise and round the parsley. Modern quantization methods do something similar, keeping some parts of the model at higher precision than others. That's what some of those label letters are hinting at, and the label guide covers it.

How much smaller, really? One documented example

We don't publish our own size tables, because sizes depend on the model, the method, and the file. But the llama.cpp documentation gives one concrete example you can rely on. For the Llama 3.1 family, it lists:

Model	Original size	Quantized size (Q4_K_M)
8B	32.1 GB	4.9 GB
70B	280.9 GB	43.1 GB
405B	1,625.1 GB	249.1 GB

Source: llama.cpp quantize README, "Memory/Disk Requirements" section, fetched 2026-10-04. Note the "original" column there is the full-precision file the tool starts from.

Look at the first row. A file that would never fit on a typical laptop becomes something many home machines can hold. That gap is the whole reason local AI is practical for regular people. Without quantization, "run it at home" would mostly mean "own a server."

What actually gets rounded (and what doesn't)

You don't need the math, but a rough picture helps you predict what will break.

The weights: the model's long-term knowledge

The weights are what the model learned during training. They're the bulk of the file. When people say a model is "quantized to 4-bit," they almost always mean the weights. This is the part that decides how big your download is.

The memory of your conversation: a separate thing

While the model is reading your prompt and writing an answer, it keeps a working scratchpad of what it has already seen. Engineers call it the K/V cache. It grows as your conversation gets longer. Some tools let you quantize this too, separately from the weights.

Ollama's FAQ describes this as an optional setting, OLLAMA_KV_CACHE_TYPE, with f16 as the default. It says q8_0 "uses approximately 1/2 the memory of f16 with a very small loss in precision," and q4_0 uses "approximately 1/4 the memory" with "a small-medium loss in precision that may be more noticeable at higher context sizes" (Ollama FAQ). It also notes this is a global setting that applies to all models, and that it works with Flash Attention enabled.

Why mention it? Because people sometimes change two things at once, a smaller weights file and a squeezed scratchpad, and then can't tell which one made answers worse. Change one at a time. More on that in the testing section.

What quantization does not change

It doesn't change what the model knows about. A quantized model wasn't retrained. It has the same training data and the same knowledge cutoff as the original. If the original didn't know something, the quantized one won't either.
It doesn't change the model's license. A smaller copy is still that model under the same terms. See What Is a Local AI Model's License and What You Can Do With It.
It doesn't make the model safer or less safe in any designed way. It's compression, not new rules.
It isn't the same as a "smaller model." A 3-billion-parameter model and a heavily quantized 8-billion-parameter model are different things. One has fewer "recipes." The other has the same recipes written more roughly. We'll come back to this, because it's the most common decision people face.

Why some jobs notice and others don't

Here's the part most explainers skip. "A little accuracy loss" is not spread evenly. In everyday use, here's the pattern many people report. Treat it as a hypothesis to test, not a law.

Jobs that usually tolerate rounding well

Rewording and tone changes. "Make this email warmer." "Shorten this paragraph." There are many acceptable answers, so small drift doesn't matter.
Summaries of text you paste in. The facts are sitting right there in your prompt. The model is mostly rearranging.
Brainstorming lists. Names for a bake sale, ideas for a garden sign. Variety is the point.
Casual Q&A on common topics. Where the original model was already confident.

Jobs that are more sensitive

Arithmetic and counting. Adding a column of numbers, converting units, scaling a recipe. (Honestly, models of every size slip here. Use a calculator. See Free AI for Recipe Scaling and Kitchen Math.)
Following long, picky instructions. "Use exactly these five headings, no bullet points, under 200 words, British spelling." Heavier rounding can make a model drop one of the rules.
Exact formats. Producing valid JSON, a spreadsheet formula, or code that runs.
Rare facts and less common languages. The model's grip on these was weaker to begin with, so rounding can loosen it further.
Long conversations. Small errors can compound as the chat goes on, especially if the scratchpad is also squeezed.

The practical takeaway: you can't judge a quantization level by a quick "hello, how are you" chat. Every version will pass that. You judge it on your touchy jobs.

The big decision: a bigger model rounded hard, or a smaller model rounded gently?

Most people hit this fork the first time they browse a model library. Your computer can hold a file of roughly a certain size. You could fill that space with:

Option A: a larger model, heavily quantized, or
Option B: a smaller model, lightly quantized.

There's no universal winner, and we won't pretend there is. Here's how to think about it:

A larger model generally has more knowledge and better reasoning to start with. Rounding takes some of that away, but it may still come out ahead.
A smaller model at gentle rounding keeps what it has more faithfully, but it had less to begin with.
Very aggressive rounding (the lowest-bit options) is where people more often report the larger model falling apart in odd ways: repeating itself, losing the thread, garbling formats.

The honest answer is "test both on your jobs," and the next section shows you how. Our companion Small Language Models Explained makes the case for smaller models at home, and How to Run Two Local AI Models and Compare Their Answers gives you a fuller comparison routine.

The taste test: judge quantization on your own work

This is the core of the article. It takes about an hour the first time, and after that you'll never have to guess.

Step 1: Pick one model and two versions of it

Choose a model you already like, then get two quantization levels of the same model. For example, a lighter-rounded version and a heavier-rounded version. Same family, same size, same maker. Only the rounding should differ.

In Ollama, model library pages usually list several tags for one model, and the tag name often includes the quantization. In LM Studio, the download panel shows multiple files per model. In both cases, the label guide helps you read the names.

Keep an eye on disk space. Two copies of a model take two model-sized chunks of your drive.

Step 2: Write down five real tasks, including two touchy ones

Not demos. Things you actually do. Here's a sample set:

A rewrite. Paste a paragraph you wrote and ask for a shorter, friendlier version.
A summary. Paste a page-long article and ask for five bullet points.
A rule-following task (touchy). "Write a 120-word notice for our church bulletin about the bake sale. Use exactly three short paragraphs. Do not use exclamation points. Include the date Saturday, October 17, and the time 9 a.m. to noon."
A format task (touchy). "Turn this list into a table with columns Item, Quantity, Aisle." Then paste a messy shopping list.
A knowledge question you can check. Something with a known answer you can verify from a reliable source.

Save these prompts in a text file so you can paste the exact same words each time. Changing the wording between versions ruins the comparison.

Step 3: Hold everything else still

Quantization isn't the only thing that changes answers. Models also have a randomness setting called temperature, and a seed that controls the random choices. If you leave those at defaults, you'll get different answers every run even from the same file, and you'll mistake randomness for quality differences.

In Ollama's interactive chat, you can set these for the session:

/set parameter temperature 0
/set parameter seed 42

Ollama's Modelfile reference describes seed as making the model "generate the same text for the same prompt" when set to a specific number, and lists temperature as the creativity setting (Ollama Modelfile reference). Setting temperature low makes answers more repeatable. You can turn it back up afterward for creative work.

Also keep the context length the same for both versions. If you're not sure what that is, see What Is an AI Context Length Setting and When to Change It. And leave the K/V cache setting alone during this test.

Step 4: Run each task on both versions and score blind

Run task 1 on version A, then on version B. Paste both answers into a document, labeled "Answer 1" and "Answer 2," and shuffle which is which. Better yet, have someone else do the labeling. Then score each answer:

Correct? (yes / partly / no)
Followed every instruction? Count the rules it broke.
Would I use this with light edits? (yes / no)

Repeat for all five tasks. A simple tally on paper is fine.

Step 5: Note the speed and the fit

Quality isn't the only thing that matters. Notice how long each version takes to start answering and how fast it writes. In Ollama, running a chat with ollama run <model> --verbose, or typing /set verbose inside a chat, shows timing statistics after each answer. Also run ollama ps in another terminal while the model is loaded. Ollama's FAQ explains that the PROCESSOR column shows whether a model is "100% GPU," "100% CPU," or split between them (Ollama FAQ).

If the lighter-rounded version spills over onto CPU and the heavier-rounded one fits on the GPU, the heavier-rounded one may feel dramatically faster. That's a legitimate reason to choose it.

Step 6: Decide with a simple rule

If the heavier-rounded version scored the same on your touchy tasks, keep it. You save space and probably speed.
If it broke rules or formats the lighter one didn't, and those are tasks you do often, keep the lighter one, or keep both and use each for what it's good at.
If both struggled on the touchy tasks, the problem probably isn't rounding. It's the model, or the task needs a calculator or a human.

Write the result down with the date. Model files get updated, and you may want to re-run the test later. Our model update guide talks about keeping a known-good version.

A worked example: the bake sale notice

Here's what this looks like in practice, described rather than quoted, since your results will differ.

Say you run the bulletin task from Step 2 on two versions of the same mid-size model. Both write pleasant notices. The lighter-rounded version gives you three paragraphs, no exclamation points, the right date and time. The heavier-rounded version also reads nicely, but it sneaks in one exclamation point and uses four paragraphs.

Is that a dealbreaker? It depends on you. If you always edit the result anyway, maybe not. Deleting one exclamation point takes two seconds. But if you were hoping to paste directly into a template, the lighter version earns its extra disk space.

Now run the shopping list table task. Suppose both get it right. And the summary task: both fine. The rewrite: you slightly prefer the heavier-rounded one's phrasing, which happens, because randomness and small differences can cut either way.

Final tally: the heavier-rounded version is "fine except for picky rules." That's useful knowledge. You might keep it as your everyday model and switch to the lighter one when you need strict formatting.

The point isn't that this exact result will happen. It's that you now have evidence about your jobs, instead of a forum argument.

Sample prompts that expose rounding problems

If you want sharper tests, these prompts tend to reveal differences between versions. Run them with temperature at zero and compare.

Instruction stacking:

Write a packing list for a two-night camping trip. Rules: exactly 12 items, numbered, alphabetical order, no item longer than three words, and the last item must be "first aid kit."

Count the rules it broke. Alphabetical order plus a forced last item is a deliberate conflict. Watch how each version handles it. A careful answer might point out the conflict.

Format fidelity:

Convert this to valid JSON with keys "name", "phone", "notes". Output only the JSON. Martha Jones, 555-0142, brings rolls Dev Patel, 555-0199, needs a ride

Paste the result into any free online JSON checker, or just look for missing quotes and commas.

Careful reading:

Here is a paragraph. List every date mentioned in it, in the order they appear, and nothing else.

Paste a paragraph with four or five dates. Missing or invented dates are a red flag.

Consistency over a long chat:

Start a conversation, tell the model three facts about an imaginary event ("The picnic is at Elm Park, starts at 1 p.m., and Ruth is bringing lemonade"), chat about other things for ten or fifteen exchanges, then ask it to repeat the three facts. Heavier rounding, especially combined with a squeezed K/V cache and a long chat, can make the recall shakier. Compare.

Troubleshooting: symptoms and what to try

"The model is too big. It won't load or my computer freezes."

Try a heavier-rounded version of the same model, or a smaller model. Close other big programs. Check ollama ps to see if it's split between CPU and GPU. Our 8GB RAM guide covers tight machines. We deliberately don't give a "you need X GB" rule here, because it depends on the model, the rounding, your context length, and your hardware.

"It loads but answers are gibberish or repeat forever."

This is often not a rounding problem. More common causes are a broken or incomplete download, the wrong chat template, or a file that doesn't match your app's version. Very aggressive rounding can contribute, so try a lighter-rounded version as one test. Our guide What to Do When Your Local AI Gives Gibberish walks through the other causes first.

"The smaller version is worse at math."

All models are unreliable at arithmetic. Rounding can make it worse, but the fix isn't a bigger file. It's a calculator, or asking the model to show steps you check yourself.

"It was fine for short chats but drifts in long ones."

Check whether you changed the K/V cache type. If so, set it back to the default and test again. Also check your context length setting. If the conversation is longer than the model's working memory, it will start forgetting the beginning regardless of rounding.

"I made my own quantized file and it's terrible."

If you used llama.cpp's tool, its documentation warns that re-quantizing a file that's already quantized, using --allow-requantize, "can severely reduce quality compared to quantizing from 16bit or 32bit" (llama.cpp quantize README). Start from the highest-precision source you can get.

"Two people online swear by opposite choices."

They're probably doing different jobs on different machines. That's why the taste test exists.

Should you ever quantize a model yourself?

Most home users never need to. Model publishers and community uploaders already provide several versions of popular models, and your app downloads one with a click.

If you're curious, here's the landscape, briefly:

llama.cpp includes a llama-quantize tool. Its documentation describes a two-phase process: convert the original model to GGUF, then quantize that file. It also mentions an "importance matrix" (imatrix) option that uses sample data to decide which parts of the model to protect, which "can minimize" accuracy loss (llama.cpp quantize README).
The same README points to a Hugging Face space called GGUF-my-repo that builds quantized files without local setup.
Ollama's import guide says Ollama "does not quantize GGUF models during import," so you'd prepare a GGUF file with a tool like llama.cpp first (Ollama import docs). Ollama's command-line create command also has a quantize option for some imports.

Doing it yourself mainly makes sense if you want an unusual size that nobody uploaded, or you're learning. It needs plenty of free disk space for the intermediate files. Respect the model's license either way. Making a quantized copy for yourself is usually different from sharing it, and the license decides what's allowed.

Common misunderstandings, gently corrected

"Quantized means pirated or knock-off." No. It's a standard, widely used technique. Many official model releases include quantized versions.

"4-bit means it's four times dumber." No. The bits describe storage precision, not intelligence. Quality loss is usually much smaller than the size savings would suggest, though it grows as rounding gets more aggressive.

"More bits is always better, so I should always get the biggest file." Only if it fits comfortably. A lighter-rounded file that overflows your memory and crawls is worse in daily use than a heavier-rounded one that runs smoothly.

"Quantization makes the model forget recent events." No. Knowledge cutoff comes from training. See How to Check if an AI Answer Is Out of Date.

"If it passes my 'hello' test, it's fine." Every version passes that. Test the touchy jobs.

Safety and good sense

Quantization doesn't change the basic rules for using any AI model:

Don't paste passwords, account numbers, or medical records into prompts unless you've thought hard about why. Local is more private than cloud, but files and chat logs still live on your disk.
Treat answers about health, money, law, electrical work, or anything that can hurt you as unverified until you check with a qualified person or a primary source. A smaller file doesn't make those answers more or less trustworthy in any predictable way. The model was never a professional.
Download model files from sources you trust. The GGUF guide has tips on checking where a file came from.

A one-page checklist

[ ] I know quantization means storing the model's numbers with fewer digits.
[ ] I know it shrinks the file and can speed things up, at some cost in accuracy.
[ ] I know the cost is uneven: rewording and summaries tolerate it, math, strict formats, and picky rules are more sensitive.
[ ] I picked two versions of the same model to compare.
[ ] I wrote five real tasks, including two touchy ones, and saved the exact prompts.
[ ] I set temperature and seed so randomness doesn't fool me.
[ ] I scored blind and noted speed and fit with ollama ps or my app's equivalent.
[ ] I changed one thing at a time (weights file or K/V cache, not both).
[ ] I wrote down which version won for which job, with the date.

FAQ

Is quantization the same as compression, like a zip file? Not quite. A zip file gives you back exactly the original when you unzip it. Quantization throws away some detail for good. It's closer to saving a photo at lower quality: smaller, usually fine, occasionally visibly worse.

Can I "un-quantize" a model to get the quality back? No. The rounded-away detail is gone. If you want the original quality, download a higher-precision version.

Does quantization affect privacy? No. Privacy depends on where the model runs and where your chats are stored, not on rounding. See How to Keep AI Chats Private (Local First).

Why do some download pages have a dozen versions of one model? Different people have different machines and different jobs. The many versions are different points on the size-versus-accuracy trade-off.

Is a quantized model safe to use for important decisions? No AI model, quantized or not, should be the one making important decisions. AI explains; it doesn't decide. Use it to draft, summarize, and think out loud, then check anything that matters with a primary source or a qualified professional.

Sources

llama.cpp, quantize tool README: https://github.com/ggml-org/llama.cpp/tree/master/tools/quantize (fetched 2026-10-04)
Ollama FAQ: https://docs.ollama.com/faq (fetched 2026-10-04)
Ollama Modelfile reference: https://docs.ollama.com/modelfile (fetched 2026-10-04)
Ollama, Importing a Model: https://docs.ollama.com/import (fetched 2026-10-04)