Imagine asking your phone "What was the total on last month's invoice?" and getting the answer straight from the PDF sitting in your files — with no upload, no internet connection, and no part of that document ever leaving your device. That is what local RAG makes possible, and it is simpler than the jargon makes it sound.
What RAG actually is
RAG stands for retrieval-augmented generation. Strip away the acronym and it is a two-step trick:
- Retrieval. When you ask a question, the system searches your own documents for the passages most likely to contain the answer.
- Generation. Those passages are handed to an AI model along with your question, so the model answers based on your files — not just on what it learned during training.
A language model trained in 2025 cannot know what your rental agreement says. But if the relevant paragraphs of your rental agreement are pasted into its context at answer time, it can read them like anyone else and answer accurately. RAG is simply that process, automated.
Why not just paste the whole document into the chat yourself? Because phone-sized models have limited context windows — a 3-8B model might comfortably handle tens of pages, not a 400-page manual. RAG solves this by finding the right paragraphs first, so the model only reads what matters.
How search works: embeddings in plain English
The clever part of RAG is the search. Keyword search is too dumb for this job: if your question says "refund policy" but the document says "returns and reimbursement," a keyword search misses it entirely.
Instead, local RAG uses an embedding model — a small AI whose only job is to convert text into a list of numbers (a vector) that captures the meaning of the text. Two passages with similar meanings get similar vectors, even when they share no words. "Refund policy" and "returns and reimbursement" end up close together in this number-space, so the search finds both.
Here is the whole pipeline, in the order it actually runs:
- Split. Your documents are chopped into small chunks — usually a paragraph or two each. Small chunks are easier to match precisely to a question.
- Embed. Each chunk is run through the embedding model once and converted into a vector.
- Store. All the vectors go into a local database on your phone (called a vector store). This is the one-time indexing step; it happens while your phone charges.
- Search. When you ask a question, the question itself is embedded with the same model, and the database returns the chunks whose vectors are closest.
- Answer. Those chunks plus your question go to the language model, which writes a grounded answer.
The embedding models used for this are genuinely tiny — far smaller than a chat model. Google's EmbeddingGemma and similar models run comfortably on phones, and the same on-device embedding approach already powers features like searchable photo and video galleries in apps that run everything locally.
Why keep RAG local
Cloud document-chat services work fine, but think about what documents you actually want to chat with: employment contracts, tax paperwork, medical reports, legal filings, a company's internal docs. The whole point of asking AI about these files is that they are private.
Local RAG keeps the pipeline on your hardware:
- Nothing is uploaded. Your documents are indexed on the device and the vectors never leave it.
- Works anywhere. On a plane, in a basement archive, in a country with expensive roaming — if the index is built, the answers work.
- No subscription meter. You are not paying per query or per page of uploaded documents. The compute cost is your phone's battery and your patience during indexing.
The tradeoff is honesty about capacity: a phone indexes a personal library comfortably, not an entire corporate archive. A few hundred PDFs or a few thousand pages of notes is the sweet spot on current hardware.
What your phone needs to run it
Local RAG has two resource demands, and they are different from just chatting with a model:
- Storage. The language model needs 2-4 GB, the embedding model a few hundred megabytes, and the vector index roughly scales with your library — a few hundred megabytes for a personal collection. This is where most phones feel it first.
- RAM. Both models need to fit in memory at answer time. Modern flagship phones with 8-12 GB of RAM manage fine. On phones with 4 GB or less, expect swapping and slower answers.
- Indexing time. Building the index for a few hundred documents can take tens of minutes on a phone processor. Do it once, on the charger, and afterwards queries are fast — the slow part is the initial pass, not the answering.
NPUs (the AI accelerators in modern phone chips) increasingly help here: embedding models and small language models map well onto them, which is why recent flagship phones run this noticeably better than phones from a few years ago.
Where local RAG stumbles (honest limitations)
Local RAG is genuinely useful, but it is not magic, and the failure modes are worth knowing:
- Garbage in, garbage out. Scanned PDFs with bad OCR, image-heavy documents with no extractable text, and badly formatted tables all index poorly. The system can only retrieve what it could read.
- The retrieval step can miss. If the answer is phrased in your documents in a way that is semantically distant from your question, the right chunk may not be retrieved. Rephrasing the question usually fixes this — you are searching a database, not reasoning with the model, at this step.
- The model can still overreach. A small model given retrieved chunks can still blend in its training knowledge or phrase things confidently. Asking it to "answer only from the provided passages" helps, and treating answers as drafts to verify against the source document is good practice.
- Very long documents are harder. Matching the right chunk out of thousands works well; reasoning across dozens of scattered chunks to synthesize one answer pushes the limits of small models. For a single manual or contract, though, this rarely comes up.
How to get good answers: practical tips
If you use a local document-chat feature, a few habits make a big difference:
- Feed it clean text. Prefer proper PDFs or text files over photos of documents. The cleaner the text, the better the retrieval.
- Ask specific questions. "What is the notice period in section 4?" beats "Tell me about my contract." Specific questions embed closer to the right chunks.
- Ask for citations. "Quote the relevant passage" forces the model to stay grounded and gives you something to verify.
- Rephrase when it misses. If the answer looks wrong or generic, try different wording. Retrieval is a search problem; sometimes the second query finds what the first missed.
- Index in the background. Let the app finish indexing before judging it. A half-built index gives half answers.
The bigger picture
RAG is one of those ideas that sounds like research-lab material until you realize your phone already has all the pieces: a small language model for answering, a tiny embedding model for searching, and enough storage for a personal library's worth of vectors. The shift from cloud RAG to local RAG is the same shift happening across all of mobile AI — the models got small enough and the chips got fast enough that the round trip to a server stopped being necessary.
For anyone handling documents they would rather not upload anywhere, that is not a marginal improvement. It is the difference between using the feature and not using it at all. Local RAG means your files stay yours — and your AI can finally read them.
LLM Hub runs its AI entirely on your device — chat, image, video and music generation, translation, transcription, and more — with no accounts and no tracking. The same on-device embedding technology behind local RAG also powers its media search, letting you find photos and videos in your gallery by describing them, all without the cloud.
