Adding AI Features to an Existing App: LLMs, Cost and UX

Adding "AI" to an app is a one-line item on a roadmap and a three-front engineering problem in reality: the experience has to feel instant, the answers have to be grounded in something real, and the per-user cost has to survive your pricing. We have shipped AI inside our own products — including a speech-to-text captioning pipeline — and the same patterns keep working.

UX: design for the wait

LLMs are not instant, so honest UX beats fake speed: stream tokens as they arrive, show meaningful progress for long jobs, and always give the user something to do or read while waiting. For mobile, background the heavy work and notify on completion rather than pinning users to a spinner.

Equally important: an escape hatch. Every AI output should be editable, retryable and dismissible — users forgive imperfect drafts they can fix; they uninstall over confident nonsense they cannot.

Cost: unit economics before launch

• Route by difficulty: small/cheap models for classification and short tasks, frontier models only where quality is the feature.
• Cache aggressively — identical and near-identical requests are common in real usage.
• Cap usage per tier, and meter it visibly so limits feel fair rather than sneaky.
• Log tokens per feature from day one; "AI costs" as one blended number hides the leak.

Quality: grounding and evaluation

Features grounded in the user's own data (their documents, catalog, history) outperform general chat by miles — retrieval is usually more important than model choice. And before launch we build a small evaluation set of real prompts with expected behaviour, so upgrades are tested against evidence instead of vibes.

Latency budgets users will tolerate

• Inline suggestions (smart reply, autocomplete): under 1 second, or users stop waiting for them.
• Chat responses: first token within ~1.5s, streaming — the perceived speed of streaming beats a faster-but-silent response.
• Document/summary generation: up to ~10s with visible progress and a preview of partial output.
• Batch jobs (captioning a video, processing a file): background it, notify on completion; never hold a screen hostage.

Model routing cheat-sheet

• Classification, tagging, extraction, short rewrites → smallest model that passes your eval set (often 10–20× cheaper).
• User-facing chat and generation where tone matters → mid-tier model with your style prompt.
• Complex reasoning, code, high-stakes output → frontier model, cached aggressively.
• Identical repeat requests → cache; near-identical → normalize inputs first, then cache. Real apps see 20–40% cache-hit rates.

Where to start in an existing app

Pick the screen where users already do the tedious thing, and make AI do the first 80% there. One well-placed feature — smart reply, auto-summary, auto-caption, search that understands meaning — moves retention more than an "AI tab" bolted to the side ever will.