You don't need a cloud account, a subscription, or even a signal bar to use AI anymore. The same class of language models that used to require a data center can now run entirely on the phone in your pocket. Download the model once, switch on airplane mode, and start chatting.
This guide walks through the whole process: what your phone needs, how to choose a model, the actual steps to get one running, and how to get good answers out of it. No hype, no invented benchmarks, just what works.
What "running an LLM on your phone" actually means
When you use a typical AI app, your prompt travels to a server, the model runs there, and the answer travels back. Running an LLM offline cuts the server out completely:
- You download the model weights once, usually a few gigabytes, over Wi-Fi.
- Inference happens on your phone's chip — the CPU, GPU, or the dedicated neural processor (NPU).
- Everything after that works offline. Planes, tunnels, rural areas, roaming abroad: if the app is genuinely on-device, none of that matters.
The key phrase is "genuinely on-device." Some apps download a model for show but quietly send your questions to the cloud. The airplane-mode test settles it in seconds: turn on airplane mode, and if the app still answers, it is really offline.
Step 1: Check what your phone can handle
You don't need the latest flagship, but hardware does set the ceiling. Here's the practical breakdown:
iPhone. Anything with an A14 chip or newer (iPhone 12 and up) runs on-device models well, thanks to Apple's Neural Engine. 3-8B parameter models run comfortably on recent iPhones; older ones should stick to 1-3B models.
Android. Recent flagships with Snapdragon 8-series, Google Tensor, or MediaTek Dimensity chips handle 3-8B models. Mid-range phones with less RAM should start with 1-3B models, which are genuinely useful for everyday writing and Q&A.
RAM and storage matter more than the brand. A quantized 1-3B model needs about 1-3 GB of free storage; a 7-8B model needs 4-6 GB. You also want a few hundred megabytes of free RAM headroom while the model runs. If your phone is constantly full, delete unused models rather than squeezing — a starved model will crash or crawl.
Battery. Running models on-device does use battery, roughly comparable to gaming or video streaming for the minutes you're actively generating text. It is not a background drain: when you're not chatting, the model isn't doing anything.
Step 2: Pick a model (and understand quantization)
This is where most beginners get lost, so here's the short version of what matters.
Parameters (the "B" number). Model names usually include their size: 1B, 3B, 7B, 8B. Bigger generally means more capable, but also slower and hungrier for storage. On a phone, bigger is not automatically better — a well-made 3B model often beats a sloppy 8B one in real use.
Quantization. This is the trick that makes phone LLMs possible. Quantization compresses the model's weights (for example from 16-bit to 4-bit precision) so the model shrinks to a fraction of its original size with only a small quality trade-off. You'll see labels like Q4 or Q8 on downloads. For phones, 4-bit quantized models are the sweet spot: small enough to fit, fast enough to feel responsive.
Chat vs. base models. Always download the instruct or chat version of a model (names ending in "-Instruct" or similar). These are tuned to follow instructions and hold conversations. Base models without that tuning will ramble.
Model families worth knowing. The open-weight ecosystem moves fast, but compact models from families like Gemma, Llama, and Phi are consistently good choices in the 1-8B range. Look for recent releases — a current 3B model usually outperforms a two-year-old 7B.
A practical starting point: download one 1-3B chat model first. Get a feel for speed and quality on your hardware, then try a larger one and compare. Most good apps let you keep several models installed and switch between them.
Step 3: Install an app and download your first model
The exact taps vary by app, but the flow is always the same:
- Install an offline-capable AI app from your app store. Look for one that clearly states models run on-device, needs no account, and lets you choose which models to download.
- Connect to Wi-Fi and pick a model. Start with a 1-3B chat model — a few gigabytes, downloads in minutes.
- Wait for the download to finish completely. Don't try to chat mid-download; partial weights produce garbage.
- Turn on airplane mode. This is both the test and the point. If the app works now, you're running a real offline LLM.
- Say hello. Ask it something simple: "Explain what a black hole is in one paragraph." Watch the speed. If tokens stream out smoothly, your hardware is happy with this model size.
That's it. You're running an LLM on your phone with no internet.
Step 4: Get good answers from small models
Phone-sized models are capable, but they're not frontier cloud models, and prompting them well makes a big difference:
- Be specific and give context. "Summarize this email in three bullet points for my boss" beats "summarize this."
- Break hard tasks into steps. Ask for an outline first, then expand each section, instead of demanding a perfect long document in one shot.
- Use them for what they're good at. Drafting, rewriting, summarizing, brainstorming, translation, explaining concepts, coding help, and roleplay are all strong suits.
- Don't ask for current events or facts you're unsure about. Small offline models have a knowledge cutoff and no web access. They will confidently invent details if pushed — verify anything important.
- Keep conversations focused. Very long chats eat into the model's context window and slow it down. Start a fresh chat for a new topic.
What works great offline — and what doesn't
Excellent on-device: everyday chat, writing and editing, summaries, translation, brainstorming, study help, journaling prompts, coding assistance, and voice transcription with models like Whisper.
Possible but limited: image generation (quantized diffusion models work on recent phones), voice chat, and short music or video generation on high-end hardware.
Still needs the cloud: live web search, current news, frontier-level reasoning and math, very long documents, and anything requiring up-to-the-minute facts.
The honest framing: offline AI replaces the large majority of daily AI use. Keep a cloud option around for the rest, and use each where it shines.
Privacy and security: keeping it actually private
On-device inference is private by physics — your prompts can't reach a server that is never contacted. But a few checks keep it that way:
- Pass the airplane-mode test for every feature you care about, not just chat.
- No account, no problem. The most private apps need no sign-up. An account means your usage can be tied to an identity.
- Check permissions. An offline AI app has no business asking for contacts, location, or an advertising ID.
- Open source is the strongest trust signal. Anyone can verify that an open-source app sends nothing anywhere. Closed apps can still be fine, but you're taking their word for it.
- Remember the model download itself. Fetching model weights needs internet once. Use Wi-Fi you trust, from an app you trust — after that, you're on your own hardware.
Troubleshooting: when it's slow or crashes
- Try a smaller model. Nine times out of ten, "slow" or "crashy" means the model is too big for the phone. Drop from 8B to 3B and compare.
- Free up RAM. Close heavy apps running in the background before a long session.
- Free up storage. A nearly-full phone throttles everything. Delete models you don't use.
- Keep the phone cool. Sustained generation warms the chip, and thermal throttling slows tokens down. Shorter answers and breaks help on long sessions.
- Restart the app, not the download. If output turns to gibberish mid-chat, the context window is probably full — start a new conversation.
A note from us: LLM Hub
Since this is our blog, here's the honest plug. We built LLM Hub because we wanted exactly what this guide describes: a private, on-device AI app for Android and iOS where you download the models you want and everything runs locally.
It ships with 15+ downloadable models — chat models in the 1B to 8B range, plus on-device image generation, Whisper-based transcription, a translator, voice chat, music and video generation, an on-device coding sandbox (Vibes Coder), a scam detector, and custom AI personas. No account, no tracking, no cloud fallback, open source, and it passes the airplane-mode test because there's nothing to phone home to. Free on Google Play and the App Store at www.llm-hub.app.
Whether you use our app or another one, the steps in this guide are the same. The important part is the idea: your AI doesn't have to live on someone else's computer.
The bottom line
Running an LLM on your phone takes about ten minutes: check your hardware, pick a small chat model, download it over Wi-Fi, and verify with airplane mode. After that, you have a capable AI assistant that works in a tunnel, on a plane, or abroad — with your words never leaving your device.
Start small, try a couple of models, and keep the cloud for the few things phones can't do yet. The gap shrinks every year; the privacy advantage is already total.
