View PDF of Webinar Slides

Five Hard-Won Lessons…
…In Performance, Load, and Reliability Testing
Ferrari 360 race from www.Fast-Autos.net
www.LeeHayward.com
Cosmo
Wh
ti M
tt ?
Why D
Does PLR T
Testing
Matter?
Why do performance,
performance load
load, and reliability testing?
Inadequate performance can lead to user dissatisfaction,
user errors, and lost business
S stem failure under load often means loss of ssystem
System
stem
functionality when the need is highest
Reliability problems damage your reputation and your
business
Perhaps the more important question PLR testing is,
“Are y
you sure that there are not critical risks to y
your
business related to performance, load, or reliability?”
p
“Experience
is a dear teacher,, but some will learn from no
other.” – Ben Franklin, 1700s-era wit and US Revolutionary
Five Lessons in PLR Testing
www.rbcs-us.com
Copyright (c) 2004-2007 RBCS
Page 2
G
t iin PLR Ri
k
Greatt M
Moments
Risks
If 10% of an ee-commerce
commerce site
site’ss customers abandon
their shopping cart due to sluggish performance,
how long will that company stay in business?
One bank estimated its average cost of field failure at
over €100,000—often related to PLR
The Victoria’s
Victoria s Secret 1999 Internet holiday lingerie
show—good concept; terrible execution—mistake:
insufficient performance, load, and reliability testing
On introduction, a new 999 (emergency) phone
service in the UK suffered from reliability (and other)
problems that resulted in late ambulance arrivals and
consequent fatalities
Five Lessons in PLR Testing
www.rbcs-us.com
Copyright (c) 2004-2007 RBCS
Page 3
“I SSee th
k S Wh
?”
the Ri
Risk—So
Whatt D
Do I D
Do?”
I can
can’tt tell you everything you need to know—no
know no one
can—but I can give you some ideas
The following pages will illustrate five lessons I’ve
learned in PLR testing
Case studies show both success and failure
II’ll
ll indicate tips for success with a
I’ll indicate warnings for common mistakes with Ì
All figures and case studies are real (some details
omitted)
itt d) th
thanks
k tto th
the ki
kind
d permission
i i off my clients
li t
and my associates on the projects
Five Lessons in PLR Testing
www.rbcs-us.com
Copyright (c) 2004-2007 RBCS
Page 4
Fi
ti
Five L
Lessons iin PLR T
Testing
Configure performance
performance, load
load, and reliability test
environments to resemble production
Generate loads and transactions that model various
real-world scenarios
Test the tests with models and simulations—and vice
versa
Invest in the right tools but don’t waste money
Start modeling,
g, simulation,, and testing
g during
g
design—and continue throughout the lifecycle!
g ppersonal
These are hard-won lessons,, learned through
experience with successful projects—and expensive failures
Five Lessons in PLR Testing
www.rbcs-us.com
Copyright (c) 2004-2007 RBCS
Page 5
L
li ti E
i
t
Lesson 11: R
Realistic
Environments
Software engineering is not yet like other
engineering fields
Civil engineering
g
g example:
p Bridges
g
Aeronautical engineering: Scale models
Software engineering counterexample: Binary
trees, normalized relational databases
It can be difficult to extrapolate results from
smaller
ll environments
i
t under
d scaled-down
l dd
loads into larger environments
Five Lessons in PLR Testing
www.rbcs-us.com
Copyright (c) 2004-2007 RBCS
Page 6
U
li ti E
i
t
Unrealistic
Environment
One client
client’ss systems:
Support large, diverse networks
Gather, integrate, and
control complex data
Tested in a minimal
environment
For a large customer with a
complex network and data, one
query that was to run overnight,
overnight
each night, took 25 hours!
ÌFor systems
y
that deal with large
g data-sets,, don’t forget
g that the
data is part of the test environment
Five Lessons in PLR Testing
www.rbcs-us.com
Copyright (c) 2004-2007 RBCS
Page 7
R
li ti E
i
t
Realistic
Environment
For a banking application,
application we built a test
environment that mimicked the production
environment
Differences were limited to the bandwidth between
the call center (where the clients were) and the data
center ((where the servers were))
We used testing and modeling to ensure that these
differences would not affect the PLR results in the
production
d ti environment
i
t
aUnderstand all differences between actual and test
environments and how they might affect your results
Five Lessons in PLR Testing
www.rbcs-us.com
Copyright (c) 2004-2007 RBCS
Page 8
T t andd P
Test
Production
d ti E
Environments
i
t
Tests were run in steps
where each step added 20
users to the load, looking
for “knees”
knees in the
performance
The only differences
between the test and
production environments
were in the bandwidth and
throughput limits on this
wide-area network that
tied the call center to the
data center.
Five Lessons in PLR Testing
Realistic loads
put into the
system from
the call center
side of the
interface
www.rbcs-us.com
Copyright (c) 2004-2007 RBCS
Page 9
L
l W ld L
d
Lesson 22: R
Real-World
Loads
A realistic test environment makes possible the next
level of realistic PLR testing: realistic transactions
and loads
Load scenarios should include…
Regular events (like backups)
Seasonal events (e.g., holiday shopping, year
year-end
end closing,
etc.)
Different classes of users (power, novice, etc.)
Growth for the future
External factors (such as WAN or Internet load)
aTry “tip over tests”: Increase load until the system fails to
check for graceful degradation and unexpected bottlenecks
Five Lessons in PLR Testing
www.rbcs-us.com
Copyright (c) 2004-2007 RBCS
Page 10
Li
ht U
p
t ti L
d
Light,
Unrepresentative
Loads
For an interactive voice response server
server, the
developers tested subsystem performance by
generating load using half of the telephony cards
The system load imposed by the load generators was
below that of the telephony subsystems under test
Warnings from test professionals
about the meaninglessness of
such tests were ignored
The tests “passed”, until…
ÌPLR testing is a specialized, complex task that requires
expertise and study: Careless OJT can kill your project
Five Lessons in PLR Testing
www.rbcs-us.com
Copyright (c) 2004-2007 RBCS
Page 11
R
p
t ti L
d
Representative
Loads
We built a load generator
that ran on an identical
but separate host
We loaded all
telephony ports
The tests failed,
failed revealing
project-threatening design
problems
And when we cranked
up the
h lloadd here…
h
Crash!!!
aPrefer non-intrusive load generators for most PLR testing
ÌPLR testing of subsystems in unrealistic system settings
yields misleading results
Five Lessons in PLR Testing
www.rbcs-us.com
Copyright (c) 2004-2007 RBCS
Page 12
Wh
lf G
tdL
When SSelf-Generated
Loadd iis Ok
Okay
“MaxConfig”
MaxConfig
A cluster of 31 systems
A mix of mainframes and
PCs running Unix
Load generators
Simple
p Unix/C p
programs
g
Used files, interprocess
communication, process
migration,
g
and other
cross-network activities
Created high, random
loads for up to 48 hours
aIntrusive (self-generated)www.rbcs-us.com
load works for load, reliability tests
Five Lessons in PLR Testing
Copyright (c) 2004-2007 RBCS
Page 13
L
t
d th
dl
Lesson 33: T
Testt th
the T
Tests—and
the M
Models
If realism of environment and load is
important, how can we be sure we got it
right—before
g
we g
go live or ship?
p
Models can help us “test our tests”
The tests also test the models
A model which is validated through testing is
useful for p
predicting
g future p
performance
aSpreadsheets are good for initial PLR modeling and design
aDynamic simulations allow for more “what-if” scenarios and
are useful during design, implementation, and testing
Five Lessons in PLR Testing
www.rbcs-us.com
Copyright (c) 2004-2007 RBCS
Page 14
T
if
Test,t bbutt V
Verify
For an Internet project,
project we
tested at 25K, 40K, and 50K
We used a production
environment and realistic
load generator
Functional testing of the
devices during PLR tests
gave us a sense of user
experience
i
under
d load
l d
We compared our test
results against the
simulation results
Five Lessons in PLR Testing
www.rbcs-us.com
Copyright (c) 2004-2007 RBCS
Page 15
D
bl Ch ki
Double-Checking
An extract from our PLR results presentation
Simulation data for IMAP servers
45% server utilization at 25K users
75% server utilization (saturation) at 40K users
Worst-case snapshots (25K users)
Server
CPU Idle
MTA1
MTA2
Maildb1
Maildb2
IMAP1
IMAP2
68%
79%
67%
89%
59%
55%
∴ Test data supports simulation
results @ 25K users
www.rbcs-us.com
Five Lessons in PLR Testing
Copyright (c) 2004-2007 RBCS
Page 16
A
bl Ch ki
Andd M
More D
Double-Checking
We did a detailed analysis of performance,
performance too
Transaction
Connect
Banner
Authorize
Select
Fetch
Server
IMAP
IMAP
IMAP
IMAP
IMAP
Frequency
1 per connect
1 per connect
1 per connect
1 per connect
1 per connect
Simulation
Time (ms)
0.74
2.57
19.68
139.87
125.44
Observed
Time (ms)
0.77
13.19
19.08
163.91
5,885.04
aBe careful to design your tests and models so that
you can compare the results.
results
Five Lessons in PLR Testing
www.rbcs-us.com
Copyright (c) 2004-2007 RBCS
Page 17
L
l D ’t Spl
Lesson 44: IInvestt iin T
Tools—Don’t
Splurge
Tools are necessary for PLR testing
There are many types of suitable tools
Commercial
Open-source
Custom-developed
p
For complex test situations, you may well
need a combination of the above
aCheck www.tejasconsulting.com and sourceforge.org for
open-source tools
ÌDon’t overload on free tools; even free tools have costs
Five Lessons in PLR Testing
www.rbcs-us.com
Copyright (c) 2004-2007 RBCS
Page 18
M
t d on T
l
Money W
Wasted
Tools
One client’s
client s all
all-too-often-repeated
too often repeated test tool tale
They selected a commercial tool based solely on a salesperson’s
representations that it would work for them
They
y hired a consultant to automate all their tests for them,
without any transition plan to train staff, maintain tests
They tried to automate at an unstable interface (in this case, the
GUI)
The consultant left, and the day after the tool and the tests were
found to be unusable by the staff
Total cost? About $500,000!
aFirst, design your PLR test system (cases, data, automation
strategy, etc.), then identify the specific tool requirements for
your system,
t
andd only
l then
th go shopping
h
i for
f tools
t l
Five Lessons in PLR Testing
www.rbcs-us.com
Copyright (c) 2004-2007 RBCS
Page 19
M
ll Sp t
Money andd Ti
Time W
Well-Spent
For a complex system with a wide
wide-area
area network of
interactive voice response servers connected to a
large call center, we used the following tools
QA Partner (now
(
Silk)
lk) to d
drive the
h call
ll center applications
l
through the graphical user interface
Custom-built load generators to drive the interactive voice
response server applications
l
Custom-built middleware to tie the two sides together and
coordinate the testing end-to-end
Free tools such as TCL as building blocks for our test
components
Total cost? Under $$500,000
,
Let’s see how…
Five Lessons in PLR Testing
www.rbcs-us.com
Copyright (c) 2004-2007 RBCS
Page 20
A hit t off the
Architecture
th System
S t Under
U d Test
T t
ACD
PBX
N x T1
Telephone
CSA Workstation
ACD
IVR
City
IVR
CT-Connect
Server
Telephone
CM Workstation
CSA-Server
WAN
CM Server
Pub/Su
Telephone
Voice Repository
CSA Workstation
Five Lessons in PLR Testing
www.rbcs-us.com
Copyright (c) 2004-2007 RBCS
City
IVR
Page 21
T t SSystem
Test
t A
Architecture
hit t
ACD
PBX
N x T1
CSA Workstation
QA Partner
drove the
call center
GUI
application
ACD
IVR
City
IVR
CT-Connect
Server
CM Workstation
CSA-Server
WAN
CM Server
Pub/Su
Voice Repository
Custombuilt load
generator
drove the
IVR
application
CS Workstation
CSA
City
IVR
Custom-built testing middleware coordinated the tests end-to-end
Five Lessons in PLR Testing
www.rbcs-us.com
Copyright (c) 2004-2007 RBCS
Page 22
L
l
d St
Lesson 55: St
Startt E
Early—and
Stay on C
Course
Performance, load
Performance
load, and reliability problems often
arise during initial system design
A bad unit or subsystem can affect the entire system
Interface problems between subsystems can create
PLR problems
Slow brittle,
Slow,
brittle unreliable system architectures can
can’tt be
patched into perfection
aU models
aUse
d l tto identify
id tif andd fix
fi PLR problems
bl
during
d i design
d i
aInclude PLR in unit, subsystem, integration, and system test
ÌThrowing more hardware at the problems might not work
Five Lessons in PLR Testing
www.rbcs-us.com
Copyright (c) 2004-2007 RBCS
Page 23
F
ili tto Sti
k with
ith th
Failing
Stick
the P
Program
For the interactive voice response server
testing mentioned on slides 11 and 12, one
senior technical staff member had built a
spreadsheet model that predicted many of the
problems we found in later testing
However, there was no follow-up during
design, implementation, or subsystem testing
Late discovery of problems during
integration and system test lead to “patching”
off th
the IVR server…
Five Lessons in PLR Testing
www.rbcs-us.com
Copyright (c) 2004-2007 RBCS
Page 24
P
bl with
ith P
t hi PLR B
Problems
Patching
Bugs
Patching of a problem
would improve
performance
However, the next
bottleneck would be
discovered during
g
testing
“Peeling the onion” was
a slow
l
process which
hi h
didn’t resolve the
performance p
p
problems
Five Lessons in PLR Testing
Lines on the axes indicate desired response
ti andd supported
time
t d load
l d level
l l
www.rbcs-us.com
Copyright (c) 2004-2007 RBCS
Page 25
PLR
l T
PLR: T
Testt E
Early,
Testt Oft
Often
For the Internet project discussed on slides 15
15-17
17, we
started with a spreadsheet during initial server
system design
That initial server design was turned into a
simulation, which was fine-tuned as detailed design
work continued
Performance testing of units and subsystems was
compared against the simulation
Few surprises were found during system testing
aIteratively improve the models and the tests to resolve
discrepancies between the predicted and observed results
Five Lessons in PLR Testing
www.rbcs-us.com
Copyright (c) 2004-2007 RBCS
Page 26
Wh
ti St
d?
Where D
Does Y
Your PLR T
Testing
Stand?
Assess your PLR testing:
How well—or poorly—does your company do in
applying
pp y g each of these lessons?
Which common mistakes are you making?
Which tips for success could you apply?
Avoid expensive, unforeseen field failures by
assessing and improving your PLR risk
mitigation
iti ti strategy
t t
th
throughout
h t th
the lif
lifecycle
l
ÌDon’t let PLR field failures be your first sign you’re at risk
aDirect questions to info@rbcs-us.com
Five Lessons in PLR Testing
www.rbcs-us.com
Copyright (c) 2004-2007 RBCS
Page 27
…Contact
C t t RBCS
For over a dozen years, RBCS has delivered services in consulting, outsourcing and training
f software
for
ft
andd hardware
h d
testing.
t ti
Employing
E l i the
th industry’s
i d t ’ mostt experienced
i
d andd
recognized consultants, RBCS conducts product testing, builds and improves testing groups
and hires testing staff for hundreds of clients worldwide. Ranging from Fortune 20
companies to start-ups, RBCS clients save time and money through improved product
development, decreased tech support calls, improved corporate reputation and more. To
learn more about RBCS, visit www.rbcs-us.com.
Address:
RBCS, Inc.
31520 Beck
B kR
Road
d
Bulverde, TX 78163-3911
USA
Phone:
+1 ((830)) 438-4830
Fax:
+1 (830) 438-4831
E-mail:
info@rbcs-us.com
Web:
www.rbcs-us.com
Five Testing Best Practices
www.rbcs-us.com
Copyright (c) RBCS
Page 28