Most prompting advice on the internet was written for giant cloud models with enormous context windows and near-infinite patience. If you run AI on your phone instead — small models in the 3-8B range running entirely offline — that advice often underperforms. Not because small models are bad, but because they respond to a different prompting style. Here is what actually works.
Small models are literal, so be literal
The biggest mindset shift: a small on-device model is less forgiving of ambiguity than a frontier cloud model. Where a huge model can infer what you "really meant" from a vague request, a small one will answer the question you literally asked — including any accidental vagueness.
Weak: "Tell me about France."
Stronger: "Give me a 5-sentence summary of France's geography and economy, written for a 12-year-old."
The second version wins not because it is longer, but because it removes decisions the model would otherwise guess at: how long, about what, for whom. With small models, every decision you leave open is a decision made randomly. Spell out format, length, audience, and the specific angle you want.
Put the task first, context second
Phone-sized models pay more attention to the start and the end of a prompt than to the middle (the same recency-and-primacy effect affects all language models, but it bites harder when capacity is smaller). So lead with the actual instruction, then supply supporting context.
Less effective: three paragraphs of background, then "so anyway, summarize this."
More effective: "Summarize the following notes into 3 bullet points:" followed by the notes.
This ordering is the cheapest possible upgrade to any prompt. It costs you nothing and reliably improves output quality on small models.
One task per message
Big cloud models can juggle compound requests — "draft the email, then translate it, then suggest subject lines." Small models do much better when you chain one task per message:
- "Draft a polite follow-up email about the invoice."
- "Now translate it to Spanish."
- "Now suggest three subject lines."
Each message gets the model's full attention instead of a fraction of it. On a phone this also matches how people actually use chat: quick exchanges, not batch jobs. If you are used to the cloud workflow of stuffing three asks into one prompt, splitting them is the single biggest habit change that improves on-device results.
A short system prompt beats a long one
A system prompt (the persistent instruction that shapes every answer) is a genuine advantage of apps that let you set one — but on small models, brevity wins. One to three sentences defining role and format is the sweet spot:
"You're a concise writing assistant. Answer in short paragraphs, avoid jargon, and ask a clarifying question instead of guessing when something is ambiguous."
Resist the urge to paste a page of rules. A long system prompt eats into the already-limited context window and, worse, a small model will follow it inconsistently — obeying rule 2 in one answer and rule 7 in the next. Fewer rules, followed consistently, beats more rules half-remembered.
Few-shot examples: the highest-impact trick
If you take one technique from this guide, make it this. Small models are excellent pattern-matchers, and one or two short input-output examples show them exactly what you want — far better than describing it in prose.
Example:
Convert these to calendar entries:
"coffee with Sam tmr 10" -> {title: "Coffee with Sam", date: "tomorrow", time: "10:00"}
"dinner mom fri 7pm" -> {title: "Dinner with Mom", date: "Friday", time: "19:00"}
Now: "dentist thu 2"
Two examples, and a small model will follow the pattern reliably. Keep examples brief — each one consumes context window, and two sharp examples beat five rambling ones. This technique works across summarizing styles, tone matching, classification, and structured extraction.
Break reasoning into steps, but ask for them
Small models are weaker at silent multi-step reasoning. The classic "think step by step" instruction genuinely helps smaller models, with a catch: you often want to see only the final answer. A practical workaround:
"First work through the math step by step, then give me only the final number as the answer."
This gives the model the reasoning space it needs while keeping the output clean. For planning and logic tasks on-device, this is night and day: unprompted, a small model may jump to a wrong conclusion; with explicit step-by-step instructions, it frequently gets there.
Constrain the output format explicitly
Small models wander less when you fence them in. Specify exactly what you want:
- "Answer in exactly 3 bullet points."
- "Reply with yes/no and one sentence of reasoning."
- "Format as: Title, then 2-3 sentences, then one tip."
Open-ended questions get open-ended answers, and with limited capacity, open-ended often means rambling. Structured output requests — lists, tables, JSON-ish formats — are small-model-friendly because they turn generation into a fill-in-the-blank exercise.
Rephrase instead of re-sending
When a small model gives a weak answer, hitting send on the same prompt again rarely helps — you will get another weak answer with different words. Instead, diagnose and rephrase:
- Too generic? Add a constraint ("focus only on costs, in plain English").
- Too long? Add a hard limit ("in under 50 words").
- Off-topic? Restate the task more narrowly, not the background.
This is also where conversation history helps: refer back to the previous answer ("make that version more formal") rather than starting over. The model keeps your earlier context, and iterative refinement is cheap when everything runs locally on your phone with no per-message cost.
Use follow-ups instead of rewriting
Because on-device chat is free and instant — no network round-trip, no per-message cost — you can treat prompting as a conversation rather than a one-shot command. Start deliberately rough: a quick, imperfect first draft from the model gives you something concrete to react to, and short follow-ups ("shorter", "add an example", "now in a friendly tone") are easier for a small model to execute than one giant master prompt. Think of the first message as a sketch and the next three as the polish. Users who adopt this loop get more out of small models than users who spend ten minutes engineering one perfect prompt.
Know the honest limits
Good prompting closes much of the gap between small and large models, but not all of it. Small on-device models still struggle with: very long documents (limited context), niche expert knowledge (smaller training footprint), and complex multi-hop reasoning (ask them to show steps, and break the problem up yourself). When a task keeps failing after three rephrased attempts, it is probably beyond what that model size can do reliably — no prompt trick will fix it. That is fine: the right tool for the job might be a bigger model on the same phone, or simply a simpler version of the task.
The 60-second checklist
Before sending a prompt to a small on-device model, run through this:
- Task stated first, in plain words?
- Format, length, and audience specified?
- One task per message?
- One or two short examples included (for patterned tasks)?
- Step-by-step instruction added (for reasoning tasks)?
- Short system prompt set, not a rulebook?
Do that consistently and you will get noticeably better answers from the same model on the same phone — no upgrade required.
Want to practice? All of these techniques work offline in LLM Hub, which runs models like Gemma, Llama, Phi, and Mistral entirely on your device. Good prompting is the skill that turns a pocket-sized model into a genuinely useful assistant.
