Run LLM Locally on Windows, macOS, or Linux

Memory sets the limit. A practical local deployment needs enough RAM or VRAM for the model weights, the context cache, the runtime, and whatever else the operating system refuses to stop running.

Start with a 7 billion or 8 billion parameter model in a 4-bit quantization. Its file is commonly around 4GB to 6GB, although context length and model architecture push actual memory use higher. A 14 billion parameter model usually wants roughly twice that weight memory. Larger models become awkward quickly.

• 8GB system RAM: try a 1B to 3B model and close other applications.
• 16GB system RAM: a 7B or 8B model at 4-bit is the sensible starting point.
• 32GB system RAM: 14B models become practical, with room for a longer context.
• 8GB dedicated GPU memory: many 7B or 8B quantized models can fit mostly or entirely on the GPU.
• Apple silicon with 16GB unified memory: begin at 7B or 8B, not 14B.

Disk space is different. A model can fit on an SSD and still fail during generation because its weights and key-value cache cannot fit in available RAM or VRAM.

CPU inference works. It is merely slower, and memory bandwidth often matters more than the processor's advertised boost clock. GPU acceleration significantly improves generation performance when the runtime supports the device and can offload enough layers.

Install Ollama first. It is the least troublesome route for running free and open source LLM models locally because it handles downloads, model configuration, a local server, and hardware acceleration behind one command-line interface.

Ollama supports Windows, macOS, and Linux, while its local API gives development tools and chat applications a consistent endpoint. Use the installer or shell command in the Ollama documentation↗, not a third-party download mirror.

1. Install Ollama for your operating system.
2. Open Terminal, PowerShell, or a Linux shell.
3. Confirm the runtime is available with ollama --version.
4. Start a modest model with ollama run llama3.2:3b.
5. Wait for the model layers to download.
6. Confirm that a >>> prompt appears.
7. Type a short request.
8. Enter /bye when finished.

That first command downloads several gigabytes. If it fails with a connection error, check free disk space, proxy settings, DNS resolution, and whether security software is blocking Ollama's network access.

The runtime exposes an AI-compatible API for local clients, but that does not make every chat private by magic. A desktop chat interface, extension, retrieval plug-in, or web-search feature may still contact cloud-based services, so inspect its documentation before entering proprietary data.

Choose the weights deliberately. Match the license, parameter count, quantization, context length, model architecture, and prompt format to the runtime and available memory rather than downloading the largest model on the page.

Start with an instruction-tuned 7B or 8B model in a 4-bit build on a 16GB machine. Quantization stores weights with fewer bits, reducing memory requirements and often improving local performance, although aggressive 2-bit and 3-bit conversions can damage output quality.

1. Read the model provider's license and usage restrictions.
2. Pick an instruction model for chat, not a base model intended for fine-tuning.
3. Prefer a 4-bit tag for the first attempt.
4. Check that Ollama has a model definition for it.
5. Pull the exact tag with ollama pull model:tag.
6. Start that tag with ollama run model:tag.

Names are slippery. “Open weights” does not necessarily mean the model uses an Open Source Initiative-approved license, permits every production deployment, or publishes its training data and engineering recipe.

Manual control has its place. llama.cpp↗ runs GGUF language models across CPU and several GPU back ends, and it can expose an OpenAI-compatible local server. It is the better runtime when you need explicit layer offloading, repeatable flags, or a particular GGUF quantization. I would not start there unless Ollama lacks the configuration you need.

Watch the hardware counters. A successful prompt should raise GPU activity or allocated memory, while the runtime log should report an accelerated back end instead of placing every model layer on the CPU.

On an NVIDIA system, leave nvidia-smi running in another terminal while generating a long response. Ollama should appear as a process and consume VRAM. Low utilization between generated tokens is normal; zero allocated VRAM throughout the request is the warning sign.

1. Start the model and request a response long enough to observe.
2. Run nvidia-smi on Windows or Linux with a supported NVIDIA driver.
3. Open Task Manager > Performance > GPU on Windows if the command is unavailable.
4. Open Activity Monitor > Window > GPU History on macOS.
5. Check Ollama's server output for its selected compute library and offloaded layers.

Shared memory muddies this. Integrated graphics and Apple silicon do not present memory exactly like a dedicated GPU, while operating-system graphs can display the wrong engine by default. In Windows Task Manager, select a Compute or CUDA graph instead of trusting the headline 3D percentage.

Acceleration can fail after a driver update, inside an unsupported virtual machine, or when the model exceeds available VRAM and spills into system memory. The prompt may still finish, just much slower. A working chat window proves generation, not GPU use.

Call it directly. A request to Ollama's loopback API proves that the local server, selected model, JSON payload, and text generation path work without involving a separate chat interface.

First confirm the model is installed with ollama list. Then send a non-streaming request, which returns one JSON object and is easier to inspect than the default stream of partial responses.

Look for the requested text in the response field and done: true. If curl reports “Failed to connect,” start Ollama and confirm that another application has not changed or blocked port 11434. If the server returns “model not found,” copy the exact name and tag from ollama list.

Keep the service on loopback. Exposing an unauthenticated local LLM server to a local network creates resource-abuse and data-handling risks, particularly for organizations subject to healthcare, finance, or retention regulations.

A local API removes per-request cloud model fees, but production deployments still need authentication, request limits, logging policy, monitoring, updates, and business continuity planning. Local does not mean maintenance-free.

Switch after measuring failure. Change the model when its answers are inadequate, but change the runtime when acceleration, model format, API behavior, or operating-system support is the actual constraint.

Bad output does not automatically require a larger model. Check the prompt template, instruction-tuned variant, context truncation, sampling configuration, and quantization first, because each can make a capable large language model behave like a broken one.

• Use a smaller model when generation spills from VRAM or makes the desktop unresponsive.
• Use a larger model when the smaller one follows prompts but lacks the required reasoning or domain knowledge.
• Use llama.cpp when you need direct GGUF control or explicit GPU layer settings.
• Use MLX↗ or an MLX-based runtime for development aimed specifically at Apple silicon.
• Keep Ollama when straightforward installation, model management, and API compatibility matter more than tuning every inference flag.

Do not use fine-tuning merely to insert current documents or private records. Retrieval-augmented generation is usually the cleaner first move, while fine-tuning changes model behavior and requires a different evaluation job. The distinction is covered in RAG versus fine-tuning.

Cloud-based generation remains rational for occasional access to larger models, bursty workloads, or machines without enough memory. Local LLMs make more sense when privacy, offline availability, predictable repeated use, or control over deployments outweigh electricity and administration.