Skip to main content

Measuring first-token latency: how we define and test it

DataArc TeamPublished · 5 min read

First-token latency in streaming recognition is often summed up in one line: how long after you start speaking the first character appears. That leaves three things open: where the clock starts, where it stops, and under what conditions the test ran. Change the conditions and the same system can produce very different numbers. This article explains how we measure first-token latency in our benchmark table, and how it differs from the live reading on the real-time speech page.

1. How we define first-token latency

A streaming recognition server returns partial results while it is still receiving audio. A partial result is the text recognized so far for the sentence being spoken, and it may be revised later.

First-token latency looks only at the first partial result whose text is not empty, that is, the first recognized character. An empty result does not count, because the user still sees nothing.

first-token latency
= time the first character returns
− time the first frame is sent

2. Why we measure inside the data center

The clock does not start when someone taps the microphone. The user may not have started speaking yet, and counting from the tap would measure human reaction time. So no person takes part in the test: a test program sends the audio of the test set to the service frame by frame.

The test program and the recognition service run in the same data center, so the network round trip is short. What the number mainly measures is the server's time from receiving audio to returning the first character: decoding, recognition and producing the first result. Capture buffering on the user's device and the network between the user and the data center are not in this number.

3. Which conditions are fixed

A different GPU, or a few more concurrent streams, changes first-token latency. We fix the following conditions during evaluation and publish them with the number:

  • Server GPU: [GPU model]
  • Concurrent streams: [N] streams
  • Model version and test date: [version], [date]
  • Test set: [test set]

If any one of these is missing, the number cannot be compared directly with numbers from elsewhere.

4. What P50 and P90 each tell you

We measure one first-token latency for every test clip and sort all results from smallest to largest.

P50: 50% of runs are this fast or faster
P90: 90% of runs are this fast or faster

P50 is the median and reflects typical performance. P90 captures tail latency: one run in ten is slower than this mark. Users notice the slow runs, so looking only at the average or P50 makes a system look smoother than it feels in practice. Averages get skewed upward by occasional outliers while masking tail latency behind fast runs, so we do not use them. The latency table on the public benchmarks page reports both P50 and P90.

5. The reading on the real-time speech page

When you speak on the real-time speech page, the "First token" chip shows the value measured in your session. Before you start speaking, it shows the P50 from the benchmark table.

That value is timed in the browser with performance.now(), and it is measured differently from the table:

  • The clock starts at the first audio frame that carries speech energy. When the normalized root mean square (RMS) of a frame reaches 0.015, someone is speaking, and the beginning of that frame is the start. If no frame in the whole session reaches the threshold, the start falls back to the beginning of the first frame sent.
  • Capture buffering is included. The browser collects 2048 samples before it hands over a frame: 128 ms at 16 kHz, about 43 ms at 48 kHz. The start is the beginning of the frame, so a whole frame has already passed by the time it is delivered.
  • The network between you and the server is included, and so is server processing.

So the reading reflects your device and network at that moment. It is useful for getting a feel for the whole path, not for comparing with the table. If background noise is loud enough to cross the threshold, the start moves earlier, to where the noise begins, and the reading comes out higher.

6. Questions to ask about anyone's numbers

  • When does the clock start: at the tap, at the first audio frame, or at the first frame that contains speech?
  • When does it stop: at the first character, or at the final result for the sentence?
  • Where was it measured: inside the data center, or in the user's browser? On what hardware?
  • Is it P50, P90 or a mean?

When all four questions have answers, two numbers can be compared. To see the reading on your own device, speak a few sentences into the microphone on the real-time speech page.

DataArc Team

Related articles