Voice agent function calling: real-time data lookup during a call

Function calling is how a voice agent answers "where's my order?" instead of transferring the caller to someone who can. One order-status lookup on the inbound telephony agent pattern takes two round trips, three function calls, and a spoken line for every way it can fail.

Time is the binding constraint, since a cascaded streaming pipeline reached first audio at 755ms in Salesforce AI Research's tutorial before any database read. In this guide you'll work through the latency budget, the three failure lines, a complete order-status flow, and the smoke test to run before launch.

Five decisions separate a lookup that sounds like a conversation from one that sounds like a dropped call.

• CRM lookup latency lands on top of whatever the pipeline already spends reaching first audio.
• In this pattern, agent_filler handles filler as a callable function.
• An order-status question costs two lookup round trips (find_customer, then get_orders) and three function calls when agent_filler is included.
• Every failure path (timeout, empty result, wrong record) ships with a spoken line written before launch.
• In the tutorial's server-side pattern, an endpoint in a function definition routes execution through Deepgram, while server credentials stay on your server.

Two things change on a call. The caller hears the wait, and the parameters arrive as audio instead of keystrokes. Function calling otherwise works the same whether your agent types or talks.

You get no typing indicator on a phone line, so whatever your handler takes, the caller experiences as dead air.

A spoken ORD0089 reaches your handler only after the model reconstructs it from audio, so the dangerous failure is a confident lookup against the wrong ID. Text fields hand the same ID to a chat agent intact.

The tutorial's find_customer description is mirrored in functions.py. It tells the model to turn a spoken "42" into CUST0042, and to strip spaces, dashes, and parentheses from a phone number before the handler sees it. Recognition accuracy at the ASR layer still sets the ceiling.

Whether a function runs on your side or through Deepgram depends on one field in its definition. Leave endpoint out and your application receives a FunctionCallRequest, then returns a FunctionCallResponse. Add endpoint (url, method, headers) and the Voice Agent API routes the call to your URL instead.

The function call docs point server-side execution at anything sensitive, including database reads and third-party service calls. Credentials stay on your server. Each item in the request schema includes a client_side boolean, which your code can use to identify which functions still need a response.

Your database read gets whatever time is left once the pipeline reaches first audio, and the pipeline usually takes most of it. Plan the fill before you plan the query.

Dead air is what the caller hears while your handler works, and it costs you fast. Ratings of perceived willingness drop off noticeably at 600ms in the classic 2013 gap-tolerance study from Roberts and Francis, and the difference turns statistically significant between 700 and 800ms. InjectAgentMessage is what covers that silence in the reference pattern.

Deepgram's pipeline comparison cites the same Salesforce AI Research tutorial for its first-audio figure. Add your backend read time, and a routine turn lands past the threshold.

The filler is a function the model has to call, not a phrase it can improvise, so every "let me check that" routes through agent_filler. The prompt template in Deepgram's reference tutorial forbids the model from saying "One moment please" without a function call.

It requires the model to call agent_filler with message_type="lookup" first, then the lookup function, then speak only once it has real information. The filler handler returns two parts: a function_response with status: "queued", and an InjectAgentMessage carrying the spoken phrase. A 2024 filler study measured a 1.1-point gain on a Likert scale in participants' impression of response delay, a statistically significant result.

Chained lookups can't run in parallel, so the caller waits for both hops plus the model's decision between them. The get_orders definition requires a customer_id that must come from find_customer first, so identity resolves before the order does.

Write the spoken line for each of the three failure modes before launch, or the model improvises one on the call. A timeout the caller hears as silence costs you the call, even when your logs show the retry succeeded. Deepgram's reference prompt covers all three. Never expose technical detail, and say something like "I'm having trouble accessing that information right now."

Set the clock yourself; the documented workshop pattern provides no numeric limit. Deepgram's workshop warns that the agent waits indefinitely if your code never sends a FunctionCallResponse. It tells you to wrap slow lookups in a timeout.

When the timer fires, return content the agent can speak instead of throwing. Example line: "That's taking longer than usual on my end. Would you like to hold for another moment, or can I take your number and call you back?"

Give the model speakable data when no account matches. Deepgram's reference prompt tells the agent to ask for a different phone number or email rather than report a failure. Return a boolean like found: false so the model can distinguish an empty result from an error.

Ask the caller to repeat or spell rather than guess. Example line: "I couldn't find an account under that number. Could you read it to me once more, or give me the email on the account instead?"

When the lookup returns a real order belonging to someone else, the agent reads back a stranger's purchase and your logs look clean. Resolve identity before data. You must bind the customer_id returned by find_customer to the verified caller identity in your backend.

Requiring that parameter in the get_orders schema doesn't enforce identity or authorization. Put authorization in the backend. OWASP's LLM06 guidance says to implement it in downstream systems rather than letting an LLM decide what's allowed. Example line: "Before I read anything out, can you confirm the name on the account and the zip code it ships to?"

In this flow, getting from the caller's question to a spoken answer takes three calls: filler, customer resolution, and order retrieval. Your Settings message defines them, your handlers run them, and the response payload feeds the read-back.

The linked configuration example places both lookups under agent.think.functions in the Settings message. It shows each with a name, description, parameters object, and optional endpoint, per the configuration docs. agent_filler is a third function definition in the complete flow. The description does real work, since it's where the model learns identifier formats and ordering rules.

{
  "name": "get_orders",
  "description": "Retrieve order history for a customer. Always verify you have the customer's account first using find_customer before checking orders.",
  "parameters": {
    "type": "object",
    "properties": {
      "customer_id": {
        "type": "string",
        "description": "Customer's ID in CUSTXXXX format. Must be obtained from find_customer first."
      }
    },
    "required": ["customer_id"]
  }
}

Beside it, find_customer accepts customer_id, phone, or email.

Your handler reads the call's arguments and returns a payload the model can speak. It pulls functions[].arguments (a JSON-encoded string) and functions[].id from the FunctionCallRequest, then echoes that ID back. Deepgram's handler is short:

async def get_orders(params):
    customer_id = params.get("customer_id")
    if not customer_id:
        return {"error": "customer_id is required"}
    result = await get_customer_orders(customer_id)
    return result

Wrap get_customer_orders in the timeout from the previous section. The reply is a FunctionCallResponse payload with four fields. Its content is a string, so serialize before you send:

{
  "type": "FunctionCallResponse",
  "id": "<id from the request>",
  "name": "get_orders",
  "content": "{\"found\": true, \"orders\": [...]}"
}

Say the identifier slowly and the status conversationally, because a number read back wrong makes the caller doubt the status too. Deepgram's Aura-2 prompting guidance recommends natural pauses between groups of 2–4 letters when reading identifiers.

Its example is "To confirm, is your referral code Queue Why. Eigh Beee?" The reference prompt shapes the order summary as "I can see you have two recent orders. Your most recent order from [date] for $[amount] is currently [status]."

Every round trip ends in a system you need to measure directly. The numbers below are alarm thresholds and engineering benchmarks.

Your CRM flags its own slowness long after the caller has noticed. The Salesforce Trust status guide counts a 500ms average over five consecutive minutes as core-service degradation, with some API services flagged at 300ms. Integration architecture sits in the voice agents Salesforce guide.

Your order system is the hop with no published number, so measure it yourself. Take the P95 from the agent's region, including any gateway in front of it. Pipeline design for CRM write-back sits in the CRM integration pipeline guide.

Token validation adds a network hop unless the signing keys are already cached. Auth0's guidance is to cache JWKS for 5–10 minutes so validation skips the network. Warm that cache when the call connects, not when the lookup fires.

Run the happy-path call and both controlled failures before you point a phone number at the agent. If all three sound like a conversation, the lookup path is ready.

Prove the happy path before forcing the controlled failures:

1. Use a record that exists to prove the happy path and the read-back.
2. Test a missing record to prove your empty-result line fires instead of an exception.
3. Stall a handler past your limit to prove the fallback line plays instead of silence.

Log id, name, arguments, and client_side on every FunctionCallRequest, plus whether the argument the model extracted matched what the caller said.

Four fields per frame give you enough to replay the whole call: request_id from the Welcome message, a monotonic seq, a ts timestamp, and direction. Tap the WebSocket and persist every non-audio frame this way, as Deepgram's observability guide recommends.

When the toolset grows, selection accuracy falls. The HumanMCP study measured a drop from 98% at 10 tools to 88% at 100. Add functions one at a time and re-check selection accuracy after each. One lookup, three failure lines, and a filler function make a complete first release.

Try the lookup on a real call. Create a free Deepgram account and put your $200 free credits against the order that always comes back wrong.

Your telephony platform decides that for you, and its no-input timer is the hard ceiling. It reprompts or disconnects without consulting your code, so look up the configured value in your own stack. Set filler timing from your measured first-audio and backend latency so the filler plays well before that boundary.

Inspect the client_side boolean on each item in the incoming request. Use it to route response ownership and avoid sending a client response for calls assigned to the server-side pattern.

The lookup keeps running, so your client halts playback and decides whether that in-flight call finishes or cancels. UserStartedSpeaking is the signal to stop audio and discard what's buffered. In the Flux STT pattern, that logic triggers from StartOfTurn, which carries a non-empty transcript. With Nova-3 streaming, it waits for an interim result with at least 2 words.

There's no threshold number, since accuracy degrades gradually and overlapping descriptions hurt more than raw count. Two functions with similar descriptions degrade selection faster than ten with distinct ones. Test the phrasings that could plausibly map to two of your functions, and split or merge definitions until each has one obvious trigger.

Yes, at two hours, and an in-flight lookup won't hold the session open. You get a Warning with MAXIMUM_SESSION_LENGTH_APPROACHING at 1 hour 55 minutes, then an Error with MAXIMUM_SESSION_LENGTH_REACHED at the two-hour limit. KeepAlive doesn't extend it, and the docs don't cover a pending FunctionCallRequest at close. Finish or abandon it, then reconnect with agent.context.