There's a persistent belief that running a language model on your own hardware requires an expensive graphics card, and it stops a lot of curious people before they start. It isn't true. The laptop you already own — the ordinary one with 8GB of RAM and whatever integrated graphics came with the processor — can run a capable language model locally, offline, for free. It won't match a frontier hosted model, and it won't be instant. But it will work, the setup takes about fifteen minutes, and the experience of watching a model generate text on a machine with no dedicated GPU changes how most people think about what this technology actually requires. This guide walks through the whole process with Ollama, explains the arithmetic that determines how fast your particular laptop will go, and is honest about where the limits sit.
Why This Works Without a Graphics Card
The reason "you need a GPU" became conventional wisdom is that it's true for training models, and training is what most of the coverage has focused on. Training involves running enormous batches of data through a network repeatedly and adjusting billions of weights, which is a massively parallel problem that graphics cards are purpose-built for. Running a model that someone else already trained — inference — is a different workload with different bottlenecks.
During inference, generating each token requires reading every weight in the model once and doing a relatively small amount of arithmetic with it. The processor isn't the constraint. The constraint is how fast your machine can move the model's weights from RAM into the CPU. This distinction matters enormously, because it means a fast processor doesn't help much, and it means the honest answer to "how fast will this run" comes from a memory bandwidth calculation rather than a benchmark of your CPU.
Two developments made this practical on modest hardware. The first is quantization, which stores each weight with fewer bits than the original training precision — typically 4 bits instead of 16. A 7-billion-parameter model at full precision needs around 14GB of memory. The same model quantized to 4 bits needs roughly 4.4GB. That single change is what moves these models from "server hardware" to "the laptop on your desk." The quality cost is real but small; most people find the difference against the full-precision original hard to notice in ordinary use.
The second is llama.cpp, the inference engine that sits underneath Ollama. It's a highly optimised C++ implementation with a genuinely good CPU code path, using vectorised instructions available on essentially every processor made in the last decade. Ollama wraps this in a clean interface that handles downloading, storage, and model switching, so you never have to think about the machinery.
What Ollama Actually Is
Ollama is a free, open-source tool that makes running local models feel like using a package manager. You type a command with a model name, it downloads the model, and it starts a chat session. Behind that simplicity it manages quantized model files, memory allocation, prompt templates for each model family, and a local API server that other applications can talk to.
The thing to appreciate about Ollama is what it removes. Before it existed, running a local model meant sourcing a model file from a repository, choosing among a dozen quantization variants with cryptic names, compiling an inference engine with the right flags for your hardware, and constructing the correct prompt template by hand — because a model given the wrong template produces confused, rambling output that looks like the model is broken when the model is fine. Ollama handles every one of those steps. You can download it from ollama.com, and it runs on Windows, macOS, and Linux.
Checking Whether Your Laptop Qualifies
The requirements are more forgiving than people expect, and the meaningful number isn't your total RAM — it's your free RAM.
On an 8GB machine, Windows and its background services typically occupy 3 to 4GB before you open anything. A browser with a handful of tabs can easily take another 2GB. That leaves perhaps 2 to 3GB genuinely available, which is why an 8GB laptop that "should" fit a 7GB model often doesn't. Before you start, open Task Manager on Windows or Activity Monitor on macOS and look at how much memory is actually free with your normal applications running. That number decides which model you should pick.
The rule of thumb is straightforward: you need free RAM equal to the model file size plus roughly 1 to 2GB for the context window and runtime overhead. A 2GB model wants around 3.5GB free. A 4.4GB model wants around 6GB free, which on an 8GB laptop realistically means closing your browser.
You'll also want around 10GB of free disk space to give yourself room for a couple of models, and any processor from roughly 2015 onward will work — it will have the vector instructions llama.cpp needs. A genuinely old machine may still run, just slowly.
Choosing a Model: The Most Important Decision You'll Make
This is where most first attempts go wrong. Someone reads that Llama or Mistral is good, pulls a 7B model onto an 8GB laptop, waits ninety seconds for a two-sentence reply, and concludes local models are useless. The model wasn't the problem. The choice was.
On 8GB with no GPU, the sensible starting point is a 3B-class instruction-tuned model. Llama 3.2 3B is the common recommendation and a reasonable default. Small Phi and Gemma variants occupy similar territory, and the Phi family in particular was trained with a deliberate focus on getting strong reasoning out of a small parameter count, which makes it punch above its size on structured tasks.
Models in the 1B range exist and are genuinely fast, but the quality drop is noticeable enough that most people find the 3B tier a better trade. Models at 7B — Mistral 7B and similar — are meaningfully more capable and will technically run on 8GB, but only with other applications closed, and the speed cost is roughly double. Treat 7B as something to graduate to after you've confirmed a 3B model isn't good enough for your specific task, not as your starting point.
The practical advice: start smaller than you think you need. A 3B model that answers in eight seconds gets used every day. A 7B model that answers in forty seconds gets abandoned within a week, regardless of how much better its answers are.
Installing Ollama Step by Step
The installation itself is unremarkable, which is the point.
Step 1 — Download the installer. Go to ollama.com and download the build for your operating system. The Windows installer is a standard executable and the macOS version is a normal application bundle. On Linux, the site provides a single shell command.
Step 2 — Run the installer. There are no meaningful options to configure. On Windows, Ollama installs itself and starts a background service that runs on localhost:11434. You'll see a small icon in the system tray indicating it's running.
Step 3 — Confirm it's alive. Open a terminal — PowerShell on Windows, Terminal on macOS — and run:
ollama --versionA version number means the installation succeeded and the command is on your PATH. If the command isn't found, close and reopen your terminal so it picks up the updated PATH.
Step 4 — Pull your first model. Now the real step:
ollama run llama3.2Ollama checks whether you have the model, doesn't find it, and starts downloading — a couple of gigabytes, with a progress bar showing speed and remaining time. On a reasonable connection this takes a few minutes. When the download finishes, the model loads into memory and you get a chat prompt. Type a question and watch the response generate.
Step 5 — Get out. Type /bye to exit the chat and release the model from memory. The model stays on disk, so the next ollama run llama3.2 skips the download and starts in seconds.
That's the entire setup. Five steps, most of the elapsed time spent waiting on a download.
The Commands Worth Knowing
Ollama's command surface is small enough to learn in one sitting.
ollama list— Shows every model stored locally with its size and modification date. Useful for finding out where your disk space went.ollama ps— Shows which models are currently loaded in memory. Genuinely important on 8GB, because a model you forgot to unload is quietly holding RAM you need.ollama pull <model>— Downloads a model without starting a chat. Handy for queuing a download while you work.ollama rm <model>— Deletes a model from disk. Models accumulate faster than you'd expect while experimenting.ollama run <model> --verbose— The important one. Starts a chat and prints timing statistics after every response. This is how you measure your own machine rather than trusting anyone else's numbers./bye— Exits a chat session and unloads the model.
Understanding Speed: The Arithmetic Behind Your Tokens Per Second
Here's where this guide is going to be more useful than a benchmark table, and more honest.
Published token-per-second figures for local models are close to meaningless across machines, because the number depends almost entirely on your laptop's memory bandwidth — a spec most people have never looked up and which varies by a factor of three or more between otherwise similar-looking machines. Two laptops with the same processor and the same 8GB of RAM can differ by double in generation speed, purely because one is running dual-channel memory and the other has a single stick.
So rather than quoting numbers from someone else's hardware, here's the calculation that produces them. Generating one token requires reading the entire model from memory once. Therefore:
Theoretical tokens per second ≈ memory bandwidth ÷ model size
Take a common configuration: DDR4-3200 in dual channel, which is roughly 51 GB/s of theoretical bandwidth. Running a 3B model quantized to 4 bits — about 2GB — the arithmetic gives 51 ÷ 2, or about 25 tokens per second in theory. Real-world efficiency on CPU inference typically lands somewhere around half to two-thirds of theoretical bandwidth, because memory access isn't perfectly sequential and the processor has other work to do. So the honest expectation is somewhere in the region of 10 to 16 tokens per second — comfortably faster than reading speed, which is what makes it feel conversational.
Run the same arithmetic for a 7B model at 4 bits, around 4.4GB: 51 ÷ 4.4 gives about 11 tokens per second theoretical, so perhaps 5 to 7 in practice. Usable, but you'll feel every word.
Now the case that catches people out. Single-channel DDR4 halves your bandwidth to roughly 25 GB/s, which halves every number above. This is why some 8GB laptops feel dramatically slower than others with identical processors, and it's the single most common reason someone's experience doesn't match a guide they read.
The pattern worth internalising is that halving the model size roughly doubles your speed. That relationship is why model selection matters more than anything else you can control, and why the advice to start small is about usability rather than caution.
Measuring Your Own Machine
Since your numbers are the only ones that matter, measure them. Start any model with the verbose flag:
ollama run llama3.2 --verboseAsk something that produces a decent amount of output — "explain how a bicycle stays upright in about two hundred words" works well, because a longer response gives a more stable measurement than a one-line reply. When the response finishes, Ollama prints a block of statistics. Two lines matter:
Prompt eval rate is how fast the model read your input. This is usually much faster than generation, because input tokens are processed together rather than one at a time. It's why a long prompt costs less time than you'd expect.
Eval rate is the one that counts. It's how many tokens per second the model generates, and it directly determines how fast text appears on screen. For reference, comfortable human reading is roughly 5 to 8 tokens per second, so anything above 10 feels responsive and anything below 3 feels like waiting.
Run the same prompt three times and take the middle result. The first run is often slower because the model is still loading, and background activity introduces noise. A median of three is far more trustworthy than a single measurement.
Then do the useful experiment: measure a 3B model, pull a 7B model, measure that, and compare. You'll see the size-versus-speed relationship in your own numbers, on your own hardware, which is worth more than any table someone else publishes.
When It's Slower Than It Should Be
There's a specific failure that produces genuinely unusable performance, and it's worth recognising because the fix is easy once you see it.
If your model doesn't fully fit in free RAM, the operating system starts swapping — moving parts of the model between RAM and disk. Since every token requires reading the whole model, and part of it now lives on disk, every single token triggers disk reads. Performance doesn't degrade gracefully; it collapses. You go from six tokens per second to less than one, your disk activity light stays solid, and the whole laptop becomes sluggish.
The tells are unmistakable: constant disk activity during generation, the system feeling unresponsive, and speeds far below what the bandwidth arithmetic predicts. The fixes, in order:
- Close your browser. Genuinely. A browser with a dozen tabs can hold 2GB or more, and reclaiming that is often the entire difference between swapping and not.
- Run
ollama ps. Check whether another model is still loaded from an earlier session and holding memory you need. - Drop to a smaller model. A 3B model that fits will annihilate a 7B model that doesn't, by a factor of five or more. This isn't a compromise — it's the correct choice.
- Reduce the context window. The context takes memory proportional to its length. If you're only having short exchanges, a smaller context frees up meaningful headroom.
If the model fits and you're still seeing modest speeds, that's not a bug — you're seeing your memory bandwidth ceiling, and it's the expected result.
Where Your Models Live, and Moving Them
Models are stored in your user directory by default — under %USERPROFILE%\.ollama\models on Windows and ~/.ollama/models on macOS and Linux. On laptops with a small system drive, which frequently accompanies an 8GB configuration, several models will fill it faster than you'd like.
Ollama reads an OLLAMA_MODELS environment variable that lets you point storage at a different drive. Set it to a path on your larger drive, restart Ollama so it picks up the change, and future downloads land there. Existing models can be moved manually to the new location. This is worth doing early, before you've accumulated models in a place you'll have to clean out later.
What These Models Are Genuinely Good At
Setting expectations correctly is the difference between a tool you keep and one you uninstall in a week.
A 3B model running on your laptop is a capable assistant for contained tasks. It summarises documents you paste in. It rewrites text in a different tone. It reformats messy data into clean structure. It answers general questions about how things work. It drafts routine emails. It explains code you don't recognise. For this class of work — bounded, self-contained, not requiring deep reasoning or current facts — it's genuinely useful, and it does all of it with no account, no API key, no rate limits, and no data leaving the machine.
It's meaningfully weaker at hard reasoning: multi-step logic, mathematics beyond the straightforward, and problems requiring careful chains of inference. It struggles with long context, both because small models handle it poorly and because context costs memory you don't have. Its factual knowledge is limited and frozen at its training cutoff, with no ability to look anything up. And code generation is workable for small, common tasks but well short of what a large hosted model produces.
The honest framing is that a local 3B model isn't a replacement for a frontier hosted model. It's a different tool with a different set of advantages — privacy, offline operation, zero marginal cost, no rate limits — that you reach for when those advantages matter or when the task is simple enough that the capability gap doesn't.
"The value of a local model isn't that it matches the hosted ones. It's that it's yours, it's free, it works on a plane, and nothing you type ever leaves your laptop. For a surprising number of everyday tasks, that trade is clearly worth it."
Who Should Actually Bother
Local inference isn't for everyone, and it's worth being direct about who benefits.
People handling sensitive material get the clearest win. If you're working with confidential documents, client information, unpublished writing, or anything covered by a policy that prohibits sending data to third-party services, a local model sidesteps the question entirely. The data never leaves the machine, and you can verify that by pulling the network cable and watching it keep working.
People who want to understand the technology get something no amount of reading provides. Watching a model load, seeing memory consumption rise, measuring tokens per second, and feeling the difference a smaller model makes builds an intuition for what these systems actually are. They stop being magic and become software with understandable constraints.
Developers building AI features get a free, unmetered, rate-limit-free endpoint for development. Ollama's local API at localhost:11434 means you can build and debug against a real model without burning API credits on every test run, then swap to a hosted model when you need production quality.
People with unreliable connectivity get an assistant that works on a plane, on a train, in a basement, or anywhere the network doesn't reach.
Conversely, if you have reliable internet, no privacy constraints, and want the best possible answers, a hosted model is the better tool and there's no shame in that. Local inference is a trade, not an upgrade.
Common Mistakes Worth Avoiding
The failure patterns are consistent enough to name directly.
Starting with the biggest model your RAM might tolerate. It's the natural instinct and it's the wrong move. Start small, confirm the workflow, and only move up if you've established that the smaller model genuinely isn't sufficient. Most people discover that 3B handles what they actually wanted.
Testing quality on the wrong tasks. Asking a 3B model an obscure factual question and concluding it's bad is like testing a bicycle on a motorway. Test it on what it's for: summarising, rewriting, reformatting, explaining. Judge it there.
Leaving models loaded. A model you forgot to unload holds gigabytes indefinitely. On 8GB that's the difference between your next task running well and swapping. Use ollama ps, and use /bye.
Comparing against benchmarks from different hardware. Someone's 40 tokens per second came from a machine with an entirely different memory configuration. Measure your own. Your number is the only one that predicts your experience.
Judging speed on the first run. The first generation after loading includes startup costs. Run it a few times before forming an opinion.
Where Local Models Are Heading
The trajectory here is unusually favourable for anyone on modest hardware, and it's worth understanding why.
Small models keep getting better at a rate that outpaces the broader field. The 3B models available now would have been considered remarkable at 13B a couple of years ago. Training techniques have improved, data curation has improved, and there's serious commercial interest in models that run on phones and laptops — which means real investment in making small models good rather than treating them as consolation prizes.
Quantization also continues to improve. Getting more quality out of fewer bits is an active research area, and each advance directly translates into either better output at the same memory footprint or the same output in less memory. Both outcomes help the 8GB laptop.
Hardware is moving too. Neural processing units are now standard in new consumer processors, and inference engines are progressively learning to use them. Memory bandwidth on new machines is climbing, and unified memory architectures blur the CPU/GPU distinction that this article has treated as fixed. The 8GB no-GPU laptop of a few years from now will run models that would surprise anyone reading this today.
The direction is clear: the hardware bar for useful local inference has been falling steadily and shows no sign of stopping. Which makes now a reasonable time to learn how it works, because the skills transfer directly and the models only get better.
Frequently Asked Questions
Can you really run an LLM without a GPU?
Yes. Ollama uses llama.cpp underneath, which has a fully functional CPU inference path. A GPU makes generation faster, but it isn't required. On an 8GB laptop with no dedicated graphics card, small quantized models in the 1B to 4B range run at conversational speeds, and 7B models run slowly but usably. The limiting factor on CPU is memory bandwidth rather than raw processing power, which is why a fast processor doesn't help as much as you'd expect and why memory configuration matters more than most people realise.
How much RAM do you need to run a local LLM?
The practical rule is that you need free RAM equal to the model file size plus roughly 1 to 2GB for the context window and runtime overhead. A 3B model at 4-bit quantization is about 2GB on disk, so it wants around 3.5GB free. A 7B model at 4-bit is about 4.4GB, needing roughly 6GB free. On an 8GB machine the operating system and browser typically consume 3 to 4GB, which is why 3B models are the comfortable choice and 7B models require closing other applications first.
Which model should I use on an 8GB laptop with no GPU?
Start with a 3B-class instruction-tuned model such as Llama 3.2 3B, or a small Phi or Gemma variant. These fit comfortably, leave room for a reasonable context window, and generate fast enough to feel like a conversation rather than a wait. Move to a 7B model only if you've tested a 3B model first and found its output genuinely insufficient for your task, because the speed cost is roughly double and on 8GB it means working with everything else closed.
Why is my local LLM so slow?
The most common cause on 8GB is that the model doesn't fully fit in available RAM, so the system swaps model weights to disk on every token. Performance collapses rather than degrading gracefully — often from several tokens per second to below one. Close memory-heavy applications, run ollama ps to check nothing else is loaded, and drop to a smaller model or a smaller context window. If the model does fit and it's still modest, you're seeing the memory bandwidth ceiling of CPU inference, which is normal and expected.
What does quantization mean and does it hurt quality?
Quantization stores each weight using fewer bits than the original training format, typically 4 instead of 16. This cuts memory footprint by roughly four times and speeds up inference proportionally, at the cost of a small amount of accuracy. The widely used 4-bit variants are generally considered a strong balance point, and most people find the difference against the full-precision model hard to notice in ordinary use. Quantizing more aggressively — to 3 or 2 bits — produces noticeably worse output, and isn't a trade most people should make.
Is Ollama free to use?
Ollama is open source and free to download and run. The models it pulls are also free to download, though each carries its own license from whoever released it, and some place restrictions on commercial use — worth checking if you're building something you intend to sell. There's no account, subscription, or per-token charge for local inference. The only real costs are disk space and electricity.
Does Ollama send my data to the internet?
Once a model is downloaded, inference happens entirely on your machine and your prompts don't leave it. Ollama contacts the network when you pull a model and may check for updates. If privacy is your main reason for running locally, you can verify the behaviour directly: disconnect from the network entirely and confirm the model still responds. It will.
How do I measure how fast my local model actually is?
Run the model with the verbose flag: ollama run llama3.2 --verbose. After each response, Ollama prints timing statistics including the prompt evaluation rate and the generation rate, both in tokens per second. The eval rate is the number that determines how fast text appears on screen. Run the same prompt three times and take the median — the first run includes loading overhead and single measurements are noisy.
Can I use a local LLM instead of ChatGPT?
For some tasks, yes. Small local models handle summarisation, rewriting, formatting, simple extraction, and casual question answering reasonably well. They're meaningfully weaker at complex reasoning, long-context work, current factual knowledge, and difficult code generation. The realistic framing is that a local model is a capable offline assistant for contained tasks, not a drop-in replacement for a frontier hosted model. Plenty of people run both and choose based on the task.
How much disk space do local models take?
A 3B model at 4-bit is roughly 2GB, and a 7B model at 4-bit is roughly 4.4GB. Models are stored once and reused, but they accumulate quickly while you're experimenting. Use ollama list to see what's stored and ollama rm to delete what you don't want. If your system drive is small, set the OLLAMA_MODELS environment variable to a path on a larger drive before you start downloading.
Will running an LLM damage or overheat my laptop?
Inference is a sustained heavy CPU workload, comparable to video encoding or compiling a large project. A laptop in good working order with clear vents handles this safely, and modern processors throttle before reaching damaging temperatures. The practical effects are a hot chassis, audible fans, and fast battery drain — so mains power on a hard surface is sensible. Sustained thermal throttling will lower your tokens per second, which is a performance issue rather than a safety one.
Final Thoughts
The gap between "I've heard you can run these locally" and "I have one running" is about fifteen minutes, most of it spent waiting for a download. That's the main thing worth taking away, because the perceived difficulty stops far more people than the actual difficulty ever would.
What you get at the end is genuinely modest and genuinely useful at the same time, and holding both of those ideas simultaneously is the key to being happy with it. A 3B model on an 8GB laptop won't write your best code or reason through your hardest problem. It will summarise a document, rewrite a paragraph, explain an unfamiliar concept, and reformat a mess of data — offline, instantly available, costing nothing, with no possibility of your input being logged or transmitted anywhere. For a lot of everyday work, that's the whole job.
The deeper value, for a lot of people, turns out to be understanding rather than utility. Once you've watched a model load into memory, measured its output rate, worked out why a smaller model runs faster, and felt the cliff when something doesn't fit in RAM, language models stop being an abstraction. They become software with constraints you can reason about — files of numbers being read from memory at a rate your hardware determines. That intuition is worth the fifteen minutes on its own, and it's not something you can get from reading about it.
If you have a laptop and a spare afternoon, the barrier to trying this is essentially zero. Head to ollama.com, install it, run ollama run llama3.2 --verbose, and see what your own machine does. Then run the arithmetic in this article against your result and see how close it lands.
Start smaller than you think you need, measure your own hardware rather than trusting anyone else's numbers, and judge the model on the tasks it's actually built for. Do those three things and local inference on modest hardware goes from disappointing to genuinely useful.