Running AI Models Locally: Hardware and Software Basics

You do not have to send every request to a cloud service. Open models can run on your own computer, giving you privacy, offline access and no per-request fees. The trade-off is that you need adequate hardware and you will usually accept somewhat lower quality than the very largest cloud models. Here is what you need to know to get started.

Why run locally?

• Privacy: prompts and documents never leave your machine.
• Offline use: works without internet, useful in places with unreliable connections.
• No usage fees: after setup, running costs only electricity.
• Control: choose the model, settings and version; no sudden changes.
• Learning: you see how models behave and can experiment freely.
• Customisation: combine with your own data and tools.

What hardware do you need?

The main constraint is memory (RAM or GPU VRAM), because the model's parameters must fit in memory.

• A rough rule: memory needed ≈ number of parameters × bytes per parameter, plus overhead for the context.
• A 7-billion-parameter model at 4-bit precision needs around 4 to 5 GB. At 16-bit it needs about 14 GB.
• A 13B model at 4-bit needs around 8 GB; a 70B model at 4-bit needs roughly 40 GB or more.

Types of machines

• Laptop or desktop with 16 GB RAM, no GPU: can run small models (1B to 8B parameters, quantised) at modest speed using the CPU.
• Computer with a gaming GPU (8 to 24 GB VRAM): runs 7B to 30B class models much faster.
• Apple Silicon Macs: their unified memory lets the GPU use the large shared RAM, which makes them popular for local models. A Mac with 16 to 32 GB or more of memory handles many models comfortably.
• Workstations and servers: several high-memory GPUs for large models.
• Phones and edge devices: very small models (about 1B to 3B) with heavy optimisation.

Speed is measured in tokens per second. Anything above about 5 to 10 tokens per second feels usable for reading; faster is smoother.

Quantisation

Quantisation stores weights with fewer bits (8-bit, 5-bit, 4-bit) to reduce memory and speed up inference, with a small loss in quality. For local use it is almost always worth it. Common model file formats include GGUF (used by llama.cpp and Ollama) and formats for GPU libraries. Names like "Q4_K_M" indicate quantisation levels.

Popular tools

Ollama

A simple way to download and run models with one command and an HTTP API. Good for beginners.

# after installing Ollama from its website
ollama pull llama3.2          # download a model (choose any model from the library)
ollama run llama3.2           # chat in the terminal

llama.cpp

A fast C++ engine that runs quantised models on CPUs and GPUs, the basis of many other tools. Offers fine control and a server mode.

LM Studio and similar desktop apps

Graphical applications to browse, download and chat with models, and to start a local server compatible with common API formats.

Hugging Face Transformers

The Python library for loading many models, with more flexibility for developers and researchers, often needing more memory than quantised GGUF setups.

vLLM and text-generation servers

Production-oriented servers for serving open models efficiently to many users on GPUs.

Calling a local model from Python

Ollama exposes a local REST API on your machine. This example sends a prompt using the requests library:

import requests

response = requests.post(
    "http://localhost:11434/api/generate",
    json={
        "model": "llama3.2",
        "prompt": "Explain what an API is in two sentences.",
        "stream": False,
    },
    timeout=120,
)
response.raise_for_status()
print(response.json()["response"])

Many local servers also provide a compatible chat endpoint so that existing client code can point to localhost with minimal changes.

Choosing a model

• Size: start with 3B to 8B for ordinary laptops.
• Task: some models are tuned for chat, others for code, others for embedding or vision.
• Language: test Nepali carefully; many small models are weaker there.
• Licence: check it allows your intended use.
• Context length: longer context needs more memory.
• Benchmarks: use them as hints, but test with your own questions.

Expect trade-offs

• Smaller local models may make more mistakes and follow complex instructions less reliably.
• Long documents are slow on CPUs.
• You manage updates, security and backups yourself.
• Laptops may heat up and drain the battery.

Practical tips

1. Close memory-hungry applications before running a large model.
2. Start with a small quantised model to test your workflow.
3. Use a GPU or Apple Silicon if possible.
4. Keep models and tools updated.
5. Restrict network access to the model server if others share your network.
6. Combine local models with retrieval from your own documents for private question answering.
7. Measure speed and quality on your real tasks before committing to hardware purchases.

Using it in a small business

For tasks like summarising internal notes, drafting replies or searching private documents, a modest local model on an office computer can be enough, and customer data never leaves the building. For heavy workloads or the most demanding tasks, a hybrid approach with a cloud model may be better.

Frequently Asked Questions

Can I run these models without a GPU?

Yes, on a CPU with enough RAM, using small quantised models, but speed will be lower.

Is a local model as good as the big cloud models?

Usually not on the hardest tasks, but for many everyday jobs the difference is small.

Are local models safe and private?

Your data stays on your device, but you must still secure your computer and download models from trusted sources.

How much disk space do I need?

Quantised models range from about 1 GB to tens of GB each, so plan for storage if you try several.

Conclusion

Running AI locally is practical with modern tools: choose a model that fits your memory, use quantisation and a tool like Ollama or llama.cpp, and test on real tasks. You gain privacy and independence in exchange for some convenience and peak quality.