You've got a laptop or gaming desktop, a folder of documents you'd rather not upload, and a model download that looks small enough to fit. Then the fans spin up, the chat window stalls, or the runtime complains that you're out of memory.
Learning how to run an LLM locally isn't mainly about installing an app. It's about making three decisions correctly: which model fits your available memory, which runtime suits your work, and who can access the resulting service. Get those right, and installation becomes the straightforward part.
Match Model Size to Your Memory
Matching a model's download size to your computer's total memory is the first mistake to avoid. The model's weights aren't the only things your runtime needs to hold.
Using the weight-memory estimates described in Hugging Face's bitsandbytes documentation, bf16 requires approximately 2 bytes per parameter; the table applies that arithmetic and approximate quantization reductions, not complete system requirements.
Approximate weight memory by model size and precision
| Model size | bf16 weights | 8-bit weights | 4-bit weights |
| 1B | About 2 GB | About 1 GB | About 0.5 GB before quantization overhead |
| 8B | About 16 GB | About 8 GB | About 4–5 GB |
| 70B | About 140 GB | About 70 GB | About 35–40 GB |
Find the model size you're considering, then budget separately for the key-value cache, runtime buffers, and operating system. The cache holds information used during generation, so a longer conversation can require more memory even though the model file hasn't changed.
Check available memory, not just installed capacity. On Windows, inspect Task Manager's Memory and GPU panels, including dedicated GPU memory. On macOS, use Activity Monitor and watch memory pressure while the model runs.
Pick a precision before a bigger model
For quality-sensitive work, start with a higher-precision quantized version when it fits. Hugging Face's quantization comparison describes 8-bit methods as about twice as memory-efficient as bf16 and generally very close in accuracy, while 4-bit methods are about four times smaller and usually retain high accuracy.
For a modest computer, I would start with Gemma 3 1B or Llama 3.2 1B rather than download the largest model available. An 8B model at four-bit precision is a practical candidate for an 8 GB GPU, provided you leave room for inference rather than assuming the weights tell the whole story.
On Apple silicon, unified memory also serves the operating system and other apps. Treat a four-bit 7B–8B model on a 16 GB laptop as a candidate to test, not a guaranteed comfortable fit.
Practical rule: Choose a checkpoint comfortably below your free memory. A model that barely fits at startup isn't a dependable working setup.
Before buying memory, check your laptop's RAM compatibility with Steel City IT's guide. Desktop owners can also review how to enable XMP, but memory profiles don't create additional capacity.
Follow this order: estimate weights, choose a quantized checkpoint, reserve headroom, load, then test realistic prompts. Where the runtime supports it, use automatic device placement when distributing weights across accelerators.
Choose Between Ollama, LM Studio and llama.cpp
Local LLM tools compared
| Tool | Interface | Best for | Model source |
| Ollama | Command line and local API | Developers and scripts | Its own model library |
| LM Studio | Desktop app with chat window | Beginners who want a GUI | Built-in model browser |
| llama.cpp | Command line and built-in server | Tuning and custom setups | GGUF files you download yourself |
Choose LM Studio if you want to chat without managing a terminal. Its model browser and loading controls make it easier to see which file you're selecting. You still need to inspect the model family, quantization, and memory requirements rather than trust the download button alone.
Choose Ollama if you're building something around a local model. Its command-line workflow is convenient for pulling models, testing prompts, and connecting scripts to a local service. The trade-off is that convenient model tags can hide details you should verify before comparing results.
Use llama.cpp when control matters
llama.cpp exposes settings such as GPU offload, context size, and server behavior more directly. That makes it useful for tuning one machine, but you're responsible for choosing compatible files and understanding the runtime flags.
For GGUF selection, NVIDIA's local-AI guidance recommends Q4_K_M checkpoints with llama.cpp as a practical balance between performance and output quality. Its capacity table associates GeForce RTX systems with up to 32 GB of VRAM with models up to roughly 50 billion parameters, RTX PRO systems with up to 96 GB with roughly 150 billion per device, and systems with up to 128 GB of unified memory with roughly 120 billion parameters.
Those are vendor capacity estimates, not speed promises. A runtime may load a model successfully yet produce responses too slowly for interactive work.
All three tools support Windows, macOS, and Linux. Choose by workflow first: LM Studio for visual interaction, Ollama for application integration, llama.cpp for explicit control. Changing tools won't compensate for a model that's too large for the machine.
How to Install a Runtime and Load Your First Model
Use Ollama for a reproducible terminal-based start, or take the LM Studio route if you prefer a graphical interface. Don't install several runtimes at once, because you'll make it harder to identify which process owns the memory or serves your application.
The installation sequence is straightforward when you separate downloading the runtime from downloading the model.
Install and verify Ollama
- Check storage and close heavy apps. Leave space for the checkpoint, downloads, and any additional model versions you plan to compare. Close games, virtual machines, and unused browser windows before loading.
- Install the runtime. Get the installer for your operating system from Ollama's official site and follow its platform-specific instructions. Avoid repackaged installers, especially those bundled with extensions you didn't choose.
- Open a fresh terminal and verify installation. Run
ollama -vto confirm that the command is available. If your shell can't find it, reopen the terminal before changing environment settings or reinstalling. - Select an explicit model tag. For a four-bit 8B starting point, check the library entry for
llama3.1:8b-instruct-q4_K_M, then runollama pull llama3.1:8b-instruct-q4_K_M. Confirm the tag and download size in the library before proceeding, since a family name alone doesn't identify the precision. - Start the model. Run
ollama run llama3.1:8b-instruct-q4_K_Mand ask a plain question, such as explaining the difference between RAM and storage. A coherent answer confirms the basic path works, but it doesn't establish useful performance. - Inspect what's running. Use
ollama listto inspect downloaded models andollama psto inspect loaded models. Record the exact model tag so you can reproduce the setup instead of relying on memory.
Take the graphical route if you prefer
In LM Studio, search for the same model family, select a Q4_K_M GGUF file, download it, and load it in the chat interface. Enable GPU offload where your hardware and backend support it, then inspect the loading information rather than assuming acceleration is active.
Keep the initial context setting conservative and use the default chat template for the model. Don't simultaneously change sampling settings, context, offload, and system instructions.
A healthy first run loads successfully and answers coherently. There's no universal stopwatch target that applies to every SSD, model, and backend.
If loading finishes but generation never begins, inspect the runtime's error output before downloading another checkpoint. A failed first prompt is easier to diagnose while the setup is still simple.
How to Run a First Prompt and Measure Speed
A quick greeting proves that text generation works. It tells you little about document processing, coding, or whether the model will remain responsive during a real session.
Use a task with an answer you can check. A summarization workflow makes a useful starting example.
Paste a representative email or article and ask for a short summary containing the main decision, outstanding questions, and next action. Check whether the model preserved names, dates, and qualifications rather than merely producing fluent prose.
Record a usable baseline
- First-token latency: Measure the wait before output begins, distinguishing a cold load from an already-loaded model.
- Generation throughput: Record the tokens per second reported by the runtime.
- Peak memory: Watch dedicated GPU memory or unified-memory pressure throughout generation.
Repeat with a longer input while keeping the model and settings unchanged. In Ollama, use ollama run llama3.1:8b-instruct-q4_K_M --verbose to display timing details after the response.
For scale, a 2026 consumer-GPU benchmark published on arXiv found that Gemma 3 1B and Llama 3.2 1B exceeded 170 tokens per second on an NVIDIA RTX 4060 Ti 16GB. That result describes those small models on that configuration, not an expected rate for larger checkpoints.
Power deserves its own measurement: a 2026 local-inference study measured maximum draw on a MacBook Pro M2 of 43.0 W for LLaMA-3.1 and 33.6 W for the compressed Gilda model. For laptop use, test sustained sessions on your own hardware rather than judging battery impact from a brief reply.
For a visual walkthrough alongside your measurements, use this demonstration.
Fixing Slow, Crashing and Out-of-Memory Runs
Out of memory during loading is usually a capacity problem, not a reason to reinstall the runtime. Check which checkpoint you downloaded and whether another model remains loaded.
Make the smallest useful change first. Starting over with a different tool creates more variables without necessarily fixing the underlying constraint.
| Symptom | Likely cause | First action |
| Model won't load | Insufficient available memory | Unload other models and close heavy apps |
| Output is consistently slow | CPU execution or partial offloading | Inspect placement and choose a smaller checkpoint |
| Long conversations crash | Growing inference allocations | Start a fresh session and reduce context |
| First response is unusually delayed | Cold loading or substantial prompt processing | Compare warm requests and trim unnecessary input |
| Answers worsen after changing files | Different quantization or chat template | Verify the template and compare a higher-quality build |
Separate quick fixes from replacements
Quick fixes include closing apps, unloading an idle model, starting a clean conversation, and removing an oversized system prompt. Run the same test prompt after each change so you can identify what helped.
Replacement work includes downloading a smaller model, switching quantization, updating a backend, or rebuilding llama.cpp with the appropriate accelerator support. Don't call that work a full model rebuild unless you're rebuilding software or producing a new checkpoint.
For quality regressions, Hugging Face's quantization documentation warns that methods below four bits can produce more noticeable accuracy loss. Compare a less aggressively compressed checkpoint before trying to repair poor answers with elaborate prompting.
Change one variable, rerun the same workload, and keep the result. Otherwise, you're collecting impressions rather than diagnosing a fault.
A whole-computer shutdown is different from a runtime error. Check temperatures, drivers, and hardware stability; on a desktop that loses power under sustained GPU load, testing the power supply is more relevant than repeatedly shortening the prompt.
Lock Down the Local Setup Before Sharing It
Local does not automatically mean private. Inference can stay on your computer while a web interface, plugin, log file, or exposed server still leaks sensitive information.
An empirical study of real-world LLM deployments found that more than 40% of observed endpoints used plain HTTP; some Ollama deployments answered more than 35% of unauthenticated API requests, with roughly 15% of those requests exposing model or system information. Downloading a model doesn't establish an access-control boundary.
Match protection to the deployment
- Single-user laptop: Keep the service bound to localhost, review extensions, and protect the operating-system account.
- Household LAN server: Require authentication before allowing other devices to connect. Don't assume every device on your home network is trustworthy.
- Remote workstation: Restrict inbound access with a firewall and use encrypted connections or an authenticated tunnel.
- Team deployment: Define access permissions, prompt-retention rules, and audit procedures before colleagues submit real documents.
Ollama binds to localhost by default. Changing its host configuration to listen beyond the machine expands who may reach it, so establish authentication and network restrictions before enabling access.
Use this checklist before sharing the service.
Verify model files against published hashes and check signed releases where available. A matching hash establishes that your download matches the publisher's artifact, not that the artifact is safe.
Run with least-privilege permissions, remove unnecessary plugins, and encrypt the disk containing chat history. Decide what gets logged, how long it remains, and how to delete it, including copies in backups.
Finally, test access from another device. A configuration that looks private in a settings window may still be reachable through a separate interface or proxy.
Choose the Setup Your Real Work Needs
Choose by workload, not by the largest model you can load. Interactive chat needs responsive first-token output; coding assistance needs correct results on your files; document Q&A needs reliable retrieval as well as useful generation.
Batch summarization favors throughput, while multi-user serving requires room for concurrent sessions. Evaluate sustained performance, not just an isolated successful request.
Keep the cheapest setup that passes
Reuse your latency, throughput, and peak-memory measurements, then add task quality, energy per completed request, and monthly hardware-plus-electricity cost. Measure energy over the request rather than treating instantaneous watts as energy consumption.
Stanford's local-model profiling retrospective, covering more than one million queries, found local accelerators 1.4–7.4 times slower than cloud accelerators on identical workloads; its hybrid router reduced energy, compute, and cost by 60–80% against a batched cloud baseline. That supports testing selective routing, provided the requests sent remotely are appropriate to share.
Build a test set of 20 real prompts and compare four-bit and eight-bit versions. Keep the least expensive configuration that passes your quality bar, and check the specific model's license before commercial use or redistribution.
NeoTeo offers practical hardware, software, AI, and security guides for readers who want to build and troubleshoot their own setups. Visit NeoTeo for tutorials that can help you check your hardware, tune your runtime, and secure your next local AI project.