Stack
- React Native (Expo)
- Next.js
- NestJS
- AWS
- OpenAI Realtime API
- RevenueCat
- Swift / Kotlin
01/
“The AI knew everything except what just happened”
- The hard problem
- The companion runs on OpenAI’s Realtime API for natural, low-latency voice. But the model’s knowledge stops at its training cutoff — so the moment a user asked about anything recent, it would answer confidently and wrong. Bolting on a normal web-search step would have meant pausing the conversation, running a query, and resuming — shattering the one thing that made the product feel alive: a continuous, one-on-one call.
- What the team built
- A live web-search tool woven directly into the realtime stream. It detects when a user’s intent needs current information, fires an internet search mid-conversation, and folds the fresh results back into the model’s response — without breaking the audio flow. The user just hears the companion answer, accurately and in real time, as if it simply knew. Latency stayed conversational; the seam is invisible.
- Why it matters
- Real-time voice and live external knowledge normally fight each other — one demands speed, the other demands a round-trip. Resolving that tension without a perceptible pause is the hard part, and it’s what separates a demo from a product that feels like a companion.
02/
“An expert that stays in its lane — and can’t be hijacked”
- The hard problem
- Users can create their own specialized agents — upload a document and a prompt and get, for example, a gym instructor that behaves only as a gym instructor. That raises two hard problems at once. Scope: a custom agent has to stay in character and in-domain, not drift into a generic chatbot the moment the conversation wanders. And safety: the instant you let users supply their own prompts and documents, you’ve opened the door to prompt injection — someone trying to jailbreak the agent, override its instructions, or turn it into something it shouldn’t be.
- What the team built
- An agent-creation system where a user uploads a document plus a prompt, and the companion becomes a specialized, in-character agent bounded to that domain — while still using live web search, so it’s not just on-topic but current within its specialty. In front of all of it sits a three-layer defense against prompt injection and malicious input: (1) input sanitization — untrusted user input is cleaned before it ever reaches the model; (2) instruction hierarchy — system guardrails are structured so user-supplied prompts can’t override or escalate above them; (3) filtering and flagging — user-informed content is explicitly marked to the model as untrusted, and anything suspicious is flagged, logged, and surfaced in an admin portal for review.
- Why it matters
- User-generated AI agents are easy to demo and hard to ship safely. Keeping an agent both in scope and un-hijackable — while it’s also pulling live information inside a real-time voice loop — is the kind of problem most teams discover the hard way in production. Building the safeguards in from the start, with visibility into attempts, is the difference between a feature and a liability.
03/
“A companion has to remember you — privately”
- The hard problem
- A companion that forgets you every session isn’t a companion. But conversation history is deeply personal, and storing it on a server creates a real privacy liability.
- What the team built
- Per-user context persisted on-device with end-to-end encryption — the companion remembers each user across conversations without their private history ever living in readable form on a server.
- Why it matters
- Memory is what makes the relationship feel real; on-device E2E encryption means that intimacy doesn’t cost the user their privacy.
04/
“Make it feel like someone, not something”
- The hard problem
- Voice alone isn’t presence. To feel like a real companion, the AI needed to emote, to fit naturally into however a user actually listens and speaks, and to be able to actually do things — not just talk.
- What the team built
- Three things came together: an emotive 3D avatar animated across the full range of human emotions, synced to the AI’s voice so the companion visibly reacts as it speaks; native audio-routing modules (custom Swift / Kotlin) letting users pick any input or output — phone speaker, Bluetooth, or any connected device — so the conversation moves with them; and an action layer that, on voice command mid-conversation, records and schedules messages (send now or later) and sends email reminders. Also shipped: subscription monetization via RevenueCat (entitlements and paywalls), with a React Native (Expo) frontend and Next.js + NestJS backend on AWS — designed, built, and shipped solo to both app stores.
- Why it matters
- The combination — emotive avatar, real voice, private memory, live knowledge, and the ability to actually do things — is what crosses the line from “AI chatbot” to something that genuinely feels like a companion.
“I asked for a companion that feels like a person, not a chatbot. They shipped real-time voice, private memory, and an emotive avatar to both app stores — solo.”
Want something like this built?
Book a call and tell us what you're shipping — we'll talk scope, timeline, and the fastest path to production.
Book a Call