How To Measure And Manage Caller Experience

agents, and eliminates repeat calls by increasing first call
resolution.
How To Measure And Manage
Caller Experience
For the call center, caller experience metrics have several
advantages over customer satisfaction scores. Caller
experience is directly controlled and delivered by the call
center, whereas customer satisfaction is influenced by many
other factors. For most companies, maximizing customer
satisfaction beyond levels that impact loyalty and revenue
only serves to drive excess cost. Optimizing caller
experience, on the other hand, reduces the total call duration
required to achieve support, service and sales goals - thereby
reducing costs and improving customer satisfaction.
Caller Experience is a hot topic because it's
a predictive indicator of both customer
loyalty and total cost per call. But what
exactly is Caller Experience? How is it
different from existing metrics? How do you
measure it and make it actionable?
Limitations of Traditional Metrics
In the 35 years since New York Telephone started using a
modified 5XB switch as the first large ACD, call centers have
developed a plethora of metrics to manage agents, queues,
and the supporting technology infrastructure. But, call
centers today are being asked to do more. It's no longer
sufficient to manage just service levels and handle times. Call
centers must actively contribute to creating customer value
and driving operational efficiency.
Traditional call center metrics provide each team and
functional area with measures to manage and report on their
individual performance. Each center or skill group tracks
handle time, service levels, and schedule adherence. The
control desk manages arriving call volumes and queue times.
Other groups manage training and hiring. The technology
team tracks IVR containment, port/trunk status, and desktop
response time. And, there are hundreds more.
Instead of focusing on the agent's activity before, during and
after a call, companies now seek to optimize the caller's
activity before, during and after speaking with an agent.
Reducing call volume and optimizing the cradle-to-grave
success of inbound calls delivers both lower total costs and
higher customer satisfaction. In contrast, call centers today
typically focus on the individual cost and performance of each
agent and the availability of agents.
Despite this abundance of data, call centers remain challenged
to boost efficiency, increase sales and improve customer
satisfaction. The reason for this gap is simple. Call centers
have become complex systems of people, processes and
technology - yet none of the traditional metrics are system
level measures. Operations executives manage portfolios of
component reports and metrics, with the mis-placed trust
that if the parts are each working, then the system must be
working too. With good scorecards from each participant in
the call, companies struggle to understand escalating
customer service costs, declining satisfaction scores, and
conversion rates that do not meet expectations.
Defining Caller Experience
Caller Experience is the caller's entire journey starting from
the moment they dial the phone until the moment they hangup. In many companies, this journey traverses sites, partners
and continents. It includes time spent interacting with IVRs,
waiting in queues and speaking with agents.
In response, several new measurement strategies have been
developed. Chief among them are the increased use of
surveys, and various ways to quantify first call resolution.
Unfortunately, each has significant shortcomings.
Caller Experience drives the total cost envelope of the call.
Improving caller experience reduces call volume and total call
duration - and therefore reduces all associated costs. In
today's business world, companies must optimize every
expense, not just the large cost of agent labor. Optimizing
caller experience reduces the need for toll-free minutes,
switch ports, IVR/speech ports, agent labor and all per-seat
costs.
Surveys are appealing because they provide direct customer
feedback. Indeed, verbatim responses can be particularly
insightful and compelling. However, aggregate survey scores
are susceptible to bias. In many centers, agents have learned
how to control whether a caller receives a survey and thus
avoid surveying calls that may reflect poorly on their
performance. Response rates are also higher from disgruntled customers which bias survey results.
Caller Experience drives customer satisfaction. An optimized
caller experience takes less time from the caller's day,
removes negative experiences with both automation and
Copyright 2008
First call resolution is an important strategy to both reduce
costs and improve satisfaction. However, it is very difficult to
Page 1 of 4
www.bbn.com/avoke
measure. Agent scoring typically provides a more favorable
first call resolution rate than reported by customer surveys.
Repeat calls generally do not count calls that fail before
nd
reaching an agent. A 2 call within a certain time interval
does not necessarily indicate that the agent on the 1st call had
an opportunity to do more for the caller. Finally, callers who
attrit or churn after a dis-satisfactory experience may be
counted as "resolved" because they didn't call back. Like
surveys, first call resolution is a lagging indicator. Unlike
surveys which typically under-report true satisfaction levels,
first call resolution typically overstates the true resolution
rates.
Total Caller Experience Metrics
Over the past decade, researchers at BBN Technologies have
developed a new set of metrics to quantify caller experience.
The metrics and accompanying methodology were proven
and reduced to practice by BBN's call center professional
services team. They currently provide a design center for the
AVOKE Call Browser, which is BBN's new platform for caller
experience analytics.
To characterize the complete caller experience in a
comprehensive fashion, we utilize the following four basic
metrics:
1. Call reason: the need that prompted the call.
2. Call completion: how the call was handled – in selfserve, with one agent, multiple agents? Or, did the
caller abandon or reach a dead end?
3. Call resolution: was the caller’s reason resolved? If not,
why?
4. Caller invested time: how much total time elapsed
from dialing to hangup?
These measures can be monitored and integrated in to
weekly management reports. And, they can be mined in a
systematic fashion to identify the highest impact improvement
opportunities in a call center.
Reason For Call
The call reason is critical because the caller's experience
depends entirely on the caller's specific need. For example, a
call center's strategy may be for balance inquiries to be
satisfied entirely in self-service, while technical support calls
are to be escalated to specially trained skill groups. The call
reason can be determined in a variety of ways. Call centers
sometimes have agents document the call reason. The
caller’s selections in the IVR can be used as evidence of the
call reason. For self-served calls, the reason can be inferred
from the category of self-service information that the caller
obtained. For agent-handled calls, speech analytics can
Copyright 2008
provide evidence of the call reason, though the most accurate
classification is obtained by using auditors to annotate a
random sample of calls.
Call Completion
Call completion describes how the caller was served and
which resources were utilized in the process. When broken
down by call reason, this metric enables actual caller behavior
to be compared to the intended call handling process. For
example, a large number of balance inquiries that complete
after interacting with an agent suggests a problem with the
intended process to resolve all such calls in self-service. The
call completion metric also reveals calls which fail to reach
either an agent queue or a self-service application. In most
call centers, such un-served calls are either not captured in
existing reporting, or they are wrongly assumed to be selfserved since they terminated within the IVR. Call completion
is typically reported in the following categories:
Self-Served - Caller hung-up after obtaining useful
information in the IVR.
One Agent - Call terminated after being handled by
one agent
Multiple Agents - Call terminated after being handled by
two or more agents
Queue Abandon - Caller hung-up while waiting for an
agent
IVR Abandon - Caller hung-up in the IVR without
obtaining any useful information
Dead-End - Call terminated after the caller
encounters an obstacle and is unable to
proceed.
Figure 2 shows actual results from a call center. This center
has an unusually large number of callers that encounter deadends (4% of all inbound calls) and that give up in the IVR
before getting anything useful done (15% of all inbound calls).
Abandoned
in Queue
3%
Abandoned
in IVR
15%
Dead-end
4%
Self-served
5%
Many Agents
2%
Two Agents
10%
One Agent
61%
Figure 2: Call Completion Chart
One advantage of the call completion metric is that it can be
How To Measure And Manage Caller Experience
Page 2 of 4
www.bbn.com/avoke
automatically determined by software that detects and
interprets sequences of events in the caller's entire dialing to
hang-up experience.
Call Resolution
As stated previously, call resolution is critically important yet
difficult to measure. The most actionable measures of call
resolution assess resolution for calls without waiting to see if
there is a repeat call. Agents can indicate whether the call
was resolved during wrap-up. However, centers generally
strive to minimize wrap-up tasks. And, the agent's view of
resolution may be biased by performance goals - or may
simply be interpreted differently than the caller. For example,
the first agent in a call may consider their activity with the
caller resolved when they transfer the call to the proper
specialist. Yet, the specialist may not in fact have been able
to resolve the call.
Call resolution can be automatically classified for a portion of
the total inbound call volume. Referring to Figure 2, calls that
abandon or encounter dead-ends are clearly unresolved.
Calls that achieve successful self-serve and then end normally
can be deemed to have been resolved. For agent-handled
calls, the most accurate determination is achieved using
auditors to annotate a statistically significant random sample
of the inbound call volume.
Call resolution is particularly powerful when combined with
call completion. In figure 2, for example, one may assume
that a large fraction of agent-handled and self-served calls are
resolved. Yet, figure 3 reveals that only 48% of calls are
resolved despite the fact that 78% are handled by agents or
self-service.
Abandoned in
Queue 3%
Dead-end
4% Self-served
5%
Abandoned in
IVR 15%
Caller Invested Time
Caller invested time is the total amount of time the call took
from the caller's day. It includes all time spent in IVR(s),
queue(s), and with agent(s). It can be correlated with both
total cost of the interaction and with the caller's satisfaction
with the call. It can also be used to manage differentiated
levels of service for different types of calls. For example,
Figure 4 shows caller invested time broken down by how the
caller routed themselves in the IVR.
Figure 4: Caller Invested Time By IVR Routing
Putting It All Together - Case Study
To illustrate how caller experience metrics can reveal issues
that can't be discerned using traditional metrics, we'll
examine two appointment centers for a large health care
insurance provider. The company wanted to identify the best
strategy for achieving consistent call handling. Average handle
time metrics by skill group could not be compared because of
differences in the skill definitions. But comparing caller
invested time by call reason revealed that one center had a
significantly higher total call time for scheduling appointments
and leaving messages for a doctor.
Leave Message (Center A)
Self-served, then
Resolved with
Agent 2%
Resolved, one
agent 34%
Leave Message (Center B)
Schedule Appointment (A)
Schedule Appointment (B)
Change Appointment (A)
Change Appointment (B)
Unresolved
24%
Transferred,
Unresolved 3%
Resolved, two
agents 5%
Reached
Voicemail 3%
Resolved, >2
agents 2%
Cancel Appointment (A)
IVR
Queue(s)
Agent(s)
Cancel Appointment (B)
0
2
4
6
8
10
Caller Invested Time [min.]
Figure 3: Call Resolution By Completion Type
Figure 5: Caller Invested Time By Call Reason For 2 Centers
Copyright 2008
How To Measure And Manage Caller Experience
Page 3 of 4
www.bbn.com/avoke
The call completion metrics provided additional insights to
understand these differences. Center A had more transfers mostly for scheduling appointments with specialists. Center
B had trained their floor agents to handle specialist
appointments. And Center A spent more time leaving
messages for doctors because several facilities had difficulty
locating doctors which extended the time spent waiting
before leaving a message.
Conclusion
Caller experience is a new strategy and approach to achieve
operational excellence. It builds upon well established
methods for measuring and managing the individual teams and
technologies that make up the call center. Caller experience
adds a process management layer to provide visibility of the
caller's experience and behavior from dialing to hang-up.
With this visibility, new opportunities are revealed to
optimize the interactions between automation and agents,
between companies and business partners, and between the
call center and the line of business.
Caller experience is also quantifiable and measureable. In
particular, there are 4 metrics that can be used to
characterize caller experience: (a) Call Reason, (b) Call
Completion, (c) Call Resolution, and (d) Caller Invested
Time. These metrics provide business level monitoring of
caller behavior and call center performance - as well as a rich
data mining opportunity to discover and quantify high impact
improvement opportunities.
About AVOKE Caller Experience Analytics
AVOKE Caller Experience Analytics addresses 2 critical
needs: managing caller experience & driving strategic process
improvement. Both require an end-to-end view of caller
experience, which is provided by the AVOKE Call Browser™
system. AVOKE Analytics customers include 8 electric/gas
utilities, 3 large telecom/cable companies, 4 of the largest
healthcare/insurance providers, the 3rd largest credit agency,
one of the largest airlines, and 3 diversified financial services
companies.
For More Information
AVOKE Caller Experience Analytics is one of many
innovative solutions developed by BBN Technologies. For
more information, call (617) 873-1600, send email to:
avoke@bbn.com, or visit: www.bbn.com/avoke
Copyright 2008
How To Measure And Manage Caller Experience
Page 4 of 4
www.bbn.com/avoke