Shipping a self-hosted bilingual AI assistant: iOS app, website and a local LLM server
A bilingual Arabic and English AI assistant for a private client in the UAE now signs people in, streams answers on iOS and the web, searches the web privately and runs on one desktop GPU. Follow-up answers that used to fail after 22 seconds now arrive, correct, in about 7.
- Sign-in and safe deploys
- Streaming bilingual chat
- Local LLM with web search
Challenge
I was the sole contracted engineer, working with a non-technical founder. When I took over, the app could not sign anyone in. The production server's working tree had hundreds of uncommitted changes, so any deploy risked overwriting work nobody had recorded. Once chat worked, real use showed a second problem: the answers were shallow. Search found the right pages, but the reply was one sentence, and a follow-up question got the same sentence back.
What I did
- Fixed sign-in end to end: email, Google and guest, including an expired guest session that made the first tap fail
- Built a deploy process for a server with hundreds of uncommitted changes: git bundles, a byte-for-byte comparison before replacing any file, dated backups, a restart, a health check and a written log of every deploy
- Migrated the schema of the live auth database, with a backup first and row counts checked afterwards
- Added live QR scanning in the camera view on every platform with a pure-Dart decoder, tested against a synthetic camera feed
- Made chat feel live: streaming replies, stop and retry, text direction set per message for mixed Arabic and English, and a translate action on each reply
- Gave the model web research it can use: privacy-guarded queries, page reading, answers without links in the text, and evidence carried into follow-up turns
- Fixed CI when the hosted Flutter version jumped overnight and deprecated a core import: pinned the toolchain and shipped the same day
- Found a false outage: an old hostname pointed at a retired tunnel while the real one was healthy. Counting requests at the tunnel proved it, so nobody restarted a healthy server
- Took the Flutter tests from 213 passing and 45 failing to 318 passing, with the 1,600+ backend tests kept green
How I worked
I measure before I change anything. The shallow answers were the clearest case. Instead of rewriting the prompt, I captured the exact messages the server sent to the model. The system prompt and the conversation history came to about 2,100 tokens, and the model's context window was 2,048. The instructions were being cut off without any error, so the model never saw most of what it was told. I raised the window to 8,192 tokens, which still fits in the 8 GB GPU's memory. Follow-up turns went from failing after 22 seconds to correct answers in about 7 seconds.
I test prompt changes against the real model before they reach production, and no production change happens without a backup, a health check and my approval. I used Claude Code as a pair engineer under those rules: it read code, drafted changes and ran checks, and I reviewed and approved every step.
I push back on scope with numbers. I advised against adding a graph database and a document server until a real question needs them. And I keep a dated status log, so any engineer can pick up the project where I left it.
Results so far
People sign in with email, Google or as a guest, and new builds reach TestFlight through Codemagic. Replies stream on iOS and the web, in Arabic, English or both. Follow-up questions get full answers in about 7 seconds instead of failing after 22, on one consumer desktop. Every production change has a backup and a logged health check behind it. The client's code and data are confidential, so this page describes the work rather than the product.
Stack
- Flutter for iOS and the web
- FastAPI on Python 3.12
- Ollama running an 8B open-weight model on one desktop with an 8 GB GPU
- SQLite
- Firebase Auth
- Self-hosted SearXNG for web search
- Cloudflare Tunnel, Tailscale and SSH
- Codemagic CI to TestFlight
What I would do next
- Build a knowledge engine: a vector store with local embeddings, so the assistant looks up what it knows before it answers
- Store facts with their source and date, and rate sources by simple, written rules
- Benchmark two embedding models on real Arabic and English questions before choosing one
- Add a graph database or a document server only when a real question needs one