Building a Voice-First Property Search with OpenAI Realtime

"I'm looking for a two-bedroom apartment in Brooklyn, somewhere around $3,000 a month. My partner and I are moving from San Francisco, and we really need a place with good light and outdoor space."

That is how people describe what they want. They do not start with 15 form fields. They talk about a move, the person coming with them, the light in the living room, and the things they refuse to compromise on.

Homi's voice agent turns that conversation into actions inside a property-search collection. The browser uses AI SDK Realtime with OpenAI's gpt-realtime-2 model. Voice and text receive tools from the same typed Homi registry.

What the server owns

The browser talks directly to OpenAI over a Realtime WebSocket. Homi's server handles the work that requires trust:

• Authenticate the user.
• Check access to the selected collection.
• Compile the tools allowed for that user and collection.
• Mint a short-lived OpenAI client secret.

The flow is deliberately small:

Browser -> POST /api/voice/session
        -> Homi checks the user and collection
        -> Homi compiles collection-aware tools
        -> OpenAI returns a short-lived client secret
        -> AI SDK Realtime connects over WebSocket
        -> Voice and text use the same Homi tools

The permanent OpenAI API key never reaches the browser.

Mint a short-lived client secret

Homi's session route checks the signed-in user and requires write access to the collection before it contacts OpenAI.

export async function POST(request: Request) {
  const session = await auth.api.getSession({
    headers: request.headers,
  });

  if (!session?.user.id) {
    return NextResponse.json({ needsAuth: true }, { status: 401 });
  }

  const requestUrl = new URL(request.url);
  const collectionId = requestUrl.searchParams.get("collectionId");
  const body = await request.json();
  const { collection, permissions } = await loadCollectionWithAccess(
    collectionId,
    session.user.id,
  );

  if (!permissions.canWrite) {
    return NextResponse.json({ error: "No write access" }, { status: 403 });
  }

  const { tools, toolsContext } = buildToolRuntime({
    collectionId,
    userId: session.user.id,
    collection,
    surface: "voice",
  });
  const toolDefinitions = await experimental_getRealtimeToolDefinitions({
    tools,
    toolsContext,
  });

  const token = await openai.experimental_realtime.getToken({
    model: "gpt-realtime-2",
    sessionConfig: {
      ...body.sessionConfig,
      tools: toolDefinitions,
    },
  });

  return NextResponse.json({
    ...token,
    tools: toolDefinitions,
  });
}

OpenAI's current client secrets normally last about a minute. That is enough time to establish the connection. It is too short to treat as a stored browser credential.

The AI SDK maps OpenAI's client-secret response into its provider-neutral { token, url, expiresAt } contract. The browser never needs to know OpenAI's raw response shape.

Let AI SDK own the realtime session

Our first version created the peer connection and Realtime event handlers through OpenAI's Agents SDK. That left Homi with one AI SDK tool registry and a second adapter for voice.

Homi now uses experimental_useRealtime from @ai-sdk/react:

const stream = await navigator.mediaDevices.getUserMedia({
  audio: {
    echoCancellation: true,
    noiseSuppression: true,
    sampleRate: 24000,
  },
});

const realtime = experimental_useRealtime({
  model: openai.experimental_realtime("gpt-realtime-2"),
  api: {
    token: `/api/voice/session?collectionId=${collectionId}`,
  },
  sessionConfig: {
    instructions: buildInstructions(),
    voice: "verse",
    inputAudioTranscription: {
      model: "gpt-4o-transcribe",
      language: "en",
    },
    turnDetection: {
      type: "server-vad",
      threshold: 0.7,
      prefixPaddingMs: 300,
      silenceDurationMs: 700,
    },
  },
  onToolCall: ({ toolCall }) =>
    dispatchRealtimeTool(collectionId, toolCall),
});

await realtime.connect();
realtime.startAudioCapture(stream);

Homi still captures the microphone stream so the interface can display a live input level. AI SDK Realtime handles audio encoding, playback, provider events, interruption, and tool outputs.

Use one tool registry for voice and text

The first voice implementation ran a separate structured extraction loop over the transcript. That could turn speech into a budget or location, but it duplicated business rules from text chat.

The current system compiles Realtime definitions from the same AI SDK 7 tool registry used by text:

const { tools, toolsContext } = buildToolRuntime({
  collectionId,
  userId,
  collection,
  hasListings,
  isAnonymousUser,
  surface: "voice",
});

const definitions = await experimental_getRealtimeToolDefinitions({
  tools,
  toolsContext,
});

A voice request can update a budget, save a listing, or change collection criteria through the same server-side implementation and permission checks as a typed request.

The session endpoint returns those definitions unchanged. onToolCall sends the model's arguments to Homi's authenticated voice endpoint. The server validates them with the same strict application schema used by text chat before executing anything.

Refresh context after writes

Collection state changes during a call. A tool might update the budget or add a property, which makes the instructions created at connection time stale.

After each state-mutating tool finishes, the hook waits for the shared UI state to receive the result. It then rebuilds the collection context and refreshes the Realtime session instructions:

realtime.sendEvent({
  type: "session-update",
  config: { instructions: buildInstructions() },
});

Without that refresh, the model can ask for information the user just supplied or reason from an old shortlist.

Keep voice and text in one conversation

Voice is another input mode inside the same Homi conversation. When a session starts, Homi includes the recent completed transcript in its assembled context. This keeps the provider-neutral Realtime client free of OpenAI-specific history events.

const history = recentConversationContext(unifiedMessages);
const instructions = assembleHomiContext({
  surface: "voice",
  sections: [persona, collectionState, history],
}).text;

AI SDK Realtime produces ordinary UIMessage values as speech and tools finish. Homi writes completed parts into the shared message stream. A small display projection changes dynamic-tool into the existing tool-{name} UI part, so the same cards render for voice and text.

The SDK exposes product-level events for the interface:

const realtime = experimental_useRealtime({
  onEvent(event) {
    if (event.type === "speech-started") setState("listening");
    if (event.type === "response-created") setState("processing");
    if (event.type === "audio-delta") setState("speaking");
  },
  onError() {
    setState("error");
  },
});

The voice hook can focus on Homi's state instead of translating raw Realtime wire events.

Make an invisible interface legible

People need to know whether the system is listening, thinking, or speaking. Homi combines a small state machine with live microphone levels:

type VoiceAgentState =
  | "idle"
  | "connecting"
  | "connected"
  | "listening"
  | "speaking"
  | "processing"
  | "error";

The interface changes its label and motion for each state. Microphone levels drive the listening animation. Realtime events and the SDK's isPlaying state track processing and playback. A silent voice interface with weak status feedback feels broken even when the connection is healthy.

What we learned

Provider protocol belongs below the product hook

AI SDK Realtime owns the WebSocket, audio buffers, normalized events, and tool-result round trips. Homi's hook deals with collection context and interface state.

One tool contract is easier to trust

The old voice adapter rewrote parts of the tool schema before OpenAI saw it. Direct AI SDK definitions preserve the names, descriptions, and inputs generated by Homi's action compiler. The strict server validator still protects the execution boundary.

State must be refreshed after every meaningful write

Realtime conversation feels continuous, but application state does not update itself inside the model's context. Refreshing instructions after mutating tools prevents a surprising number of repetitive questions.

Real conversations find the awkward edges

People pause, correct themselves, change cities halfway through a sentence, and talk over the assistant. Synthetic transcripts rarely reproduce that rhythm. We learned more from actual calls than from tidy fixtures.

Voice should remain optional

Some people prefer typing. Others are in public or cannot use a microphone. Homi keeps both modes in one conversation so switching does not discard context.

The migration detail worth remembering

If an older example sends a request to /v1/realtime/sessions, it documents the beta API. New integrations should create client secrets at /v1/realtime/client_secrets. AI SDK's experimental_realtime.getToken() handles that provider call and returns the normalized token contract used by experimental_useRealtime.

The old endpoint still appears in search results and copied snippets. That is why we have kept the migration note explicit instead of silently deleting the history.

---

Want to hear it work? Start a Homi search, create a collection, and open voice chat.