Building a Voice-First Property Search with OpenAI Realtime

"I'm looking for a two-bedroom apartment in Brooklyn, somewhere around $3,000 a month. My partner and I are moving from San Francisco, and we really need a place with good light and outdoor space."

That is how people describe what they want. They do not start with 15 form fields. They talk about a move, the person coming with them, the light in the living room, and the things they refuse to compromise on.

Homi's voice agent turns that conversation into actions inside a property-search collection. This article describes the production system we run in July 2026: OpenAI's GA Realtime API, the Realtime Agents SDK, WebRTC, and the same typed tools that power text chat.

What the server owns

The browser talks directly to OpenAI over WebRTC. Homi's server handles the work that requires trust:

• Authenticate the user.
• Check access to the selected collection.
• Compile the tools allowed for that user and collection.
• Mint a short-lived OpenAI client secret.

The flow is deliberately small:

Browser -> POST /api/voice/session
        -> Homi checks the user and collection
        -> Homi compiles collection-aware tools
        -> OpenAI returns a short-lived client secret
        -> RealtimeSession connects over WebRTC
        -> Voice and text use the same Homi tools

The permanent OpenAI API key never reaches the browser.

Mint a short-lived client secret

Homi's session route checks the signed-in user and requires write access to the collection before it contacts OpenAI.

export async function POST(request: Request) {
  const session = await auth.api.getSession({
    headers: request.headers,
  });

  if (!session?.user.id) {
    return NextResponse.json({ needsAuth: true }, { status: 401 });
  }

  const { collectionId } = await request.json();
  const { collection, permissions } = await loadCollectionWithAccess(
    collectionId,
    session.user.id,
  );

  if (!permissions.canWrite) {
    return NextResponse.json({ error: "No write access" }, { status: 403 });
  }

  const { tools, toolsContext } = buildToolRuntime({
    collectionId,
    userId: session.user.id,
    collection,
    surface: "voice",
  });
  const toolDefinitions = await experimental_getRealtimeToolDefinitions({
    tools,
    toolsContext,
  });

  const response = await fetch(
    "https://api.openai.com/v1/realtime/client_secrets",
    {
      method: "POST",
      headers: {
        Authorization: `Bearer ${env.OPENAI_API_KEY}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        session: { type: "realtime", model: "gpt-realtime-2" },
      }),
    },
  );

  const data = await response.json();
  return NextResponse.json({
    model: "gpt-realtime-2",
    sessionToken: data.value,
    expiresAt: data.expires_at,
    tools: toolDefinitions,
  });
}

OpenAI's current client secrets normally last about a minute. That is enough time to establish the connection. It is too short to treat as a stored browser credential.

The old beta response nested the ephemeral key under client_secret.value. The GA endpoint returns value and expires_at at the top level. That small response-shape change is easy to miss during a migration.

Let the SDK own WebRTC

Our first version created the peer connection, data channel, SDP offer, and event handlers by hand. It worked. It also left a lot of protocol code in a product hook.

Homi now uses @openai/agents-realtime:

const sessionData = await fetch("/api/voice/session", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ collectionId }),
}).then((response) => response.json());

const stream = await navigator.mediaDevices.getUserMedia({
  audio: {
    echoCancellation: true,
    noiseSuppression: true,
    sampleRate: 24000,
  },
});

const tools = buildRealtimeTools({
  collectionId,
  definitions: sessionData.tools,
});

const agent = new RealtimeAgent({
  name: "Homi",
  instructions: buildInstructions(),
  tools,
  voice: HOMI_VOICE_CONFIG.voice,
});

const transport = new OpenAIRealtimeWebRTC({ mediaStream: stream });
const realtime = new RealtimeSession(agent, {
  apiKey: sessionData.sessionToken,
  transport,
  model: sessionData.model,
  config: {
    audio: {
      input: {
        transcription: HOMI_VOICE_CONFIG.transcription,
        turnDetection: HOMI_VOICE_CONFIG.turnDetection,
      },
      output: { voice: HOMI_VOICE_CONFIG.voice },
    },
    reasoning: { effort: "low" },
  },
});

await realtime.connect({ apiKey: sessionData.sessionToken });

We still capture the microphone stream ourselves because the interface displays a live input level. The SDK owns the connection, SDP exchange, data channel, and Realtime protocol.

WebRTC is a good fit for a browser voice interface. It handles live audio without making Homi proxy the stream through its own server.

Use one tool registry for voice and text

The first voice implementation ran a separate structured extraction loop over the transcript. That could turn speech into a budget or location, but it duplicated business rules from text chat.

The current system compiles Realtime definitions from the same AI SDK 7 tool registry used by text:

const { tools, toolsContext } = buildToolRuntime({
  collectionId,
  userId,
  collection,
  hasListings,
  isAnonymousUser,
  surface: "voice",
});

const definitions = await experimental_getRealtimeToolDefinitions({
  tools,
  toolsContext,
});

A voice request can update a budget, save a listing, or change collection criteria through the same server-side implementation and permission checks as a typed request.

The browser receives data-only definitions. buildRealtimeTools adds a generic dispatcher that sends each call back to Homi. We do not keep a second client-side schema in sync.

Refresh context after writes

Collection state changes during a call. A tool might update the budget or add a property, which makes the instructions created at connection time stale.

After each state-mutating tool finishes, the hook rebuilds the collection context and refreshes the session instructions:

realtime.on("agent_tool_end", (_context, _agent, tool) => {
  if (STATE_MUTATING_TOOLS.current.has(tool.name)) {
    refreshInstructions();
  }
});

Without that refresh, the model can ask for information the user just supplied or reason from an old shortlist.

Keep voice and text in one conversation

Voice is another input mode, not a separate assistant. When a session starts, Homi seeds completed text messages into the Realtime history:

if (seedItems.length > 0) {
  realtime.updateHistory(seedItems);
}

As speech is finalized, the hook writes user and assistant transcripts back into the shared UI message stream. Tool results use the same message parts as text chat, so existing tool cards render without a voice-only copy.

The SDK exposes product-level events for the interface:

realtime.on("history_updated", handleHistoryUpdated);
realtime.on("audio_start", () => setState("speaking"));
realtime.on("audio_stopped", () => setState("connected"));
realtime.on("audio_interrupted", () => setState("listening"));
realtime.on("agent_start", () => setState("processing"));
realtime.on("agent_end", () => setState("connected"));
realtime.on("error", () => setState("error"));

The voice hook can focus on Homi's state instead of translating raw Realtime wire events.

Make an invisible interface legible

People need to know whether the system is listening, thinking, or speaking. Homi combines a small state machine with live microphone levels:

type VoiceAgentState =
  | "idle"
  | "connecting"
  | "connected"
  | "listening"
  | "speaking"
  | "processing"
  | "error";

The interface changes its label and motion for each state. Microphone levels drive the listening animation, while audio_start and audio_stopped track playback. This matters more than decorative polish. A silent voice interface with weak status feedback feels broken even when the connection is healthy.

What we learned

Protocol code belongs below the product hook

Managing WebRTC manually taught us how the system worked. Keeping that code in the application hook made every Realtime API change expensive. The Agents SDK now owns the transport details.

Tool consistency matters more than transcript extraction

A second extraction schema looks simple at first. It becomes another source of truth for permissions, field names, and mutations. Sharing the tool registry lets voice and text improve together.

State must be refreshed after every meaningful write

Realtime conversation feels continuous, but application state does not update itself inside the model's context. Refreshing instructions after mutating tools prevents a surprising number of repetitive questions.

Real conversations find the awkward edges

People pause, correct themselves, change cities halfway through a sentence, and talk over the assistant. Synthetic transcripts rarely reproduce that rhythm. We learned more from actual calls than from tidy fixtures.

Voice should remain optional

Some people prefer typing. Others are in public or cannot use a microphone. Homi keeps both modes in one conversation so switching does not discard context.

The migration detail worth remembering

If an older example sends a request to /v1/realtime/sessions, it documents the beta API. New integrations should create client secrets at /v1/realtime/client_secrets. The current OpenAI Realtime API reference also exposes /v1/realtime/calls for direct call creation, while the Realtime Agents SDK handles the browser WebRTC flow used here.

The old endpoint still appears in search results and copied snippets. That is why we have kept the migration note explicit instead of silently deleting the history.

---

Want to hear it work? Start a Homi search, create a collection, and open voice chat.