Why On-Demand Provisioning Enables Tighter Alignment of Test and Production Environments •

• Cognizant Reports
Why On-Demand Provisioning Enables
Tighter Alignment of Test and Production
Environments
Executive Summary
Steve Smith looks like a worried man, moving restlessly in his chair and making frantic phone calls
late on a Friday evening, inquiring about what
went wrong with the software his team had developed. All seemed well during testing, but errors
surfaced as soon as the application was deployed,
leaving Steve and his team in a quandary.
Tales such as Steve’s abound in the software testing arena. From banking applications to mobile
phones and airport systems, software defects
affect organizations’ business-critical and customer-facing applications, impacting their daily
business operations and resulting in huge losses.
According to the National Institute of Standards
and Technology (NIST), software flaws cost the
U.S. economy a whopping $60 billion annually.
One of the primary reasons for such unpleasant situations — in which errors are discovered in
production or only after end-users are affected
— is low-quality testing, often impacted by poorly
built test environments. Test environments rarely
match production environments due to the huge
cost and effort involved in mimicking them. Further, issues such as the time taken to build test
cognizant reports | november 2013
environments, unplanned environment downtime
and inaccurate configuration of test environments
impact testing effectiveness. These result in application roll-out delays and test managers signing
off on projects with caveats, leading to applications issues once put into production.
Organizations are currently exploring various
ways to mitigate these risks. Attempts to use virtualization to overcome test environment challenges have met with limited success. Also on the
rise is “on-demand provisioning” on the cloud
and traditional environments, as well as build
and deployment automation. Our experience
shows that cloud-based test environments offer
better results by providing testing teams with
on-demand capacity. Cloud-based offerings also
enable organizations to generate standard builds
that are more closely and cost-effectively aligned
with real production environments.
On-demand provisioning has much to offer
organizations seeking to instill greater discipline
in managing builds and elevating code quality.
Application quality depends on test environments that support functional testing requirements; moreover, it is also important to have
a production-like environment to complete the
testing process without an exception. This synchronization of test and production environments is achieved by building a service framework around the test environment, supported by
cloud-based computing grids. This approach, in
our view, has already delivered great benefits to
organizations that are piloting and enacting fullfledged on-demand test environment strategies.
Test vs. Production Environments
According to a 2008 Aberdeen study, only 42%
of over 200 organizations surveyed were happy
with the performance of their business-critical
applications.1 Primary reasons for this include
the inability to identify issues before end-users
are impacted and the challenges of testing application performance in pre-production stages
(see Figure 1). Sadly, our experience with companies across industries indicates that not much
has changed. At a time when companies are
laser-focused on operational cost-cutting and
time-to-market acceleration, software defects
that emerge in production environments can
severely undercut financial performance and
cause reputational damage.
With applications becoming increasingly complex,
the scope of testing has expanded tremendously,
creating unforeseen challenges for testing teams.
For instance, to assess their likely performance
in real-time, Web and mobile applications must
be tested for multiple operating systems and
updates, multiple browser platforms and versions,
different types of hardware and a large number
of concurrent users. Further, in the case of new
applications, it may not be possible to predict all
the likely scenarios in which the application will
be deployed.
The conventional approach of manually creating
in-house testing environments that fully mirror
these complexities and multiplicities consumes
huge capital and resources. This has caused
application developers to take the risky route
of conducting testing in their production environments. However, the Sarbanes–Oxley Act, as
well as other regulations, restricts access to
production systems, which has forced organizations across industries to conduct testing
in separate environments. Further, it is not
uncommon for organizations that cannot afford
large-scale test environments to use sizing/
extrapolation techniques during performance
testing to understand how an application behaves
in the real world. This approach, however, does
not guarantee success.
Poorly built test environments often limit the
ability of testing teams to test applications
for various scenarios. Test environments differ
from production environments in terms of OS,
patch levels, software version, configuration,
etc. The wider the gap between test and production environments, the greater the chance of an
application misbehaving once deployed or for a
defect to leak into live systems. The following are
among the most common issues that afflict test
environments:
Creating test environments can be
time-consuming and delay-prone. It can
sometimes take weeks or month to set up
a simple test environment due to delays in
Top Challenges for Application Performance Management
Inability to identify issues before
end-users are impacted
57%
Increase in complexity of applications
51%
Inability to measure SLAs around
application performance
37%
Inability to test application
performance in pre-production stages
34%
Increase in complexity of
network traffic
32%
Source: Aberdeen Group
Figure 1
cognizant reports
2
finding servers, procuring licenses for programs and testing tools and installing them.
Roughly 30% of defects arise due to
inaccurate test environment configurations,
studies show. Further, configuration changes
made in response to errors often go untracked,
causing errors when applications are moved
into production.
Scheduling of the test environment is
crucial, especially when environments
are shared by two projects. If not properly
managed, this can lead to unnecessary delays.
Testing teams often clone production databases and write extract scripts to create
test data. This approach is resource consumptive and error prone due to a lack of audit trails.
Simulating the production issue in the
test environment to identify the root cause
is challenging due to a lack of test environment alignment with production versions at all
system component layers.
Test environment downtime and changes
are often not controlled and communicated
properly. This poses challenges in managing
testing projects and achieving KPIs.
Lack of a single view makes the environment less flexible to support changes due
to code modifications during the test cycle.
Further, it is difficult to tune a complex test
environment to suit the advancements in software development methods such as Agile.
The test environment refresh is not
considered while planning for the test
project execution phase. This narrows the
opportunity to maintain an up-to-date test
environment.
Loosely managed asset control necessitates
undue investments in the infrastructure.
Inadequate access control reduces confidence
in the state of the environment and its configuration control. Together, this adds to the cost
of the project and to the risk-based sign-off of
the test plan.
Automation: Production systems have
adopted automation to a great extent. The
non prods lacked automations relatively due
to the higher cost of automation tools. Build/
deployment and orchestration solutions have
greatly improved the test environment build
timelines. This has also brought in agility
features to test environments.
The Impact
Test environment issues can impact delivery
schedules and result in the risk of subsequent
software failures, as well as additional business
expenditures. It is estimated that 40% of the
effort involved in a software development lifecycle goes toward managing defects that arise due
to environment-related issues.2 The ever-present
pressure to meet service level agreements (SLAs)
can force IT managers to resort to risk-based
sign-off on a product/application release. This
means that the product/application is tested for
only those risks that have a high probability of
occurrence.
This risked-based sign-off can have the
unintended consequence of putting the
organization’s reputation and financial condition at risk (i.e., errors that cause customers to
question product quality — or, worse, cause
monetary harm — can be grounds for lawsuits).
Further, with the cost of fixing defects in production being significantly greater than in the
early stages (see sidebar below), testing teams
Cost of Fixing Defects
According to a study by dynaTrace on application performance management, about 50% of
software problems are discovered in production only after they have affected end users and
that resolving these issues can involve huge costs.3
Business losses from system downtime can exceed $1 million per hour. The majority of defects
creep into code during the early stages of the development lifecycle. The longer a defect goes
undetected, the greater the costs of fixing it. According to studies, the ratio of the cost to fix a
defect at different stages of a project is 1:10:100. This means the cost of fixing a coding defect in
production is 10 times the cost of fixing it during testing and 100 times the corresponding cost of
fixing it during the design or requirements phase (see Figure 2, next page). According to Gartner,
“The cost of fixing defects ranges from a low of approximately $70 (cost to fix a defect at the
requirements phase) to a high of $14,000 (cost to fix a defect in production).”4
cognizant reports
3
Index of the Relative Cost of Fixing Defects
110
90
70
50
30
10
0
Requirements
Design
Code
Test
Production
Source: “The Economics of Testing,” www.riceconsulting.com
Note: Analysis based on base cost of fixing a defect in the requirements and design stages, measured as one unit (hour, dollar, etc.).
Figure 2
must ensure that defects do not pass through to
later stages.
On-Demand Provisioning
The criticality of business applications for today’s
organizations goes without saying. As such,
delivering quality applications requires a proper
test strategy complemented by an efficient test
environment. Testing helps minimize errors but
does not guarantee their removal. Organizations
strive to deliver quality applications by reducing
the majority of defects through rigorous testing
while preparing themselves to fix minor bugs
during production. In the case of a new version
of an existing application, organizations must
ensure that the new version works without disturbing the current systems environment. This
creates the need for a production-like test environment, which might not offer the same processing capacity but delivers the same functionality
and enables all required components to interact
as they would in a production environment.
Organizations have tried numerous options to
improve the test process. Recently, system virtualization has provided some respite, but it is not
a panacea. It has met base-level operational and
financial objectives by eliminating the need for
large capital investments in hardware, software
and related infrastructure; however, given the
requisite set-up costs, many pioneering companies have yet to achieve the operational flexibility
and scalability required to deliver the forecasted
productivity and ROI on assets.
cognizant reports
We believe that on-demand provisioning of test
environments via the cloud offers an effective
solution for achieving better ROI on assets. Cloud
computing minimizes the volume of issues that
plague current approaches to test environments.
Cloud-based testing leverages on-demand
computing grids that can be more easily and
flexibly deployed, eliminating the need for the
upfront capital expenditures (Cap-Ex) required
to procure hardware and software. The pay-peruse model provides several benefits, including
providing testing organizations with freedom
from owning assets; enhanced collaboration
among testers, developers and managers;
greater levels of efficiency; and, importantly,
reduced time-to-market for key business
applications. These benefits are delivered by
leveraging access to cloud-based resources over
the Internet without compromising security
requirements.
The pursuit to reduce time-to-market and the
expense of application releases is often hampered by huge costs and long delays in setting up
an appropriate test environment. Production-like
environments at reduced costs can be achieved
through on-demand provisioning, combined with
efficient processes and governance that creates
the discipline to develop quality build images and
version control.
To a large extent, the testing model can also
eliminate configuration errors. This will help
organizations achieve their test plan objectives
4
and minimize defect leakages when applications
enter production due to incorrect test environment setup/configurations. Organizations would
do well to combine on-demand provisioning
with industry best practices like shift-left (see
sidebar below) to improve the overall quality
of software and reduce the cost of application
development, maintenance and testing.
Shift-Left Practice
Finding coding flaws early in the development lifecycle is one way to reduce the number of defects
that surface in production. Shift-left is a practice where efforts are made to identify defects early
and fix them before submitting the code to a testing team. It also includes sharing the project’s
progress with clients at regular intervals, which ensures that the project is heading in the right
direction. This relieves testing teams from incurring additional costs and effort required to fix
defects at later stages, when teams are hard-pressed for time.
The process can be time-consuming, and care must be taken to ensure it does not cause
unnecessary delays in product development. Automating tests and running automated tests with
every build can help in identifying bugs quickly.
Footnotes
“Application Performance Management: The Lifecycle Approach Brings IT and Business Together,”
Aberdeen Group, June 30, 2008.
1
Julian Brook, “Testing Environment Management Challenges With Virtualisation,” ComputerWorldUK,
Sept. 6, 2008.
2
Alois Reitbauer, “Is There a Business Case for Application Performance?” Dynatrace, May 26, 2010.
3
4
“Hype Cycle for Application Development, 2011,” Gartner, July 27, 2011.
Bibliography
Martin Perlin, "Notes from the Trenches: Obstacles and Challenges to IT Environment Stability,"
www.evolven.com, October 2011.
Wayne Ariola, "The Next Generation of Test Environment Management," Virtualization Journal,
July 12, 2011.
"Maximizing the Value of Good Testing Practice in an Agile Environment: Delivering On Time, In Scope,
On Budget and At the Tight Level of Quality," Sogeti, September 2010.
“Is Shift-Left Agile? And Death by Build,” AgileFall, March 22, 2010.
Vinaya Kumar Mylavarapu, “Taking Testing to the Cloud,” Cognizant, March 2011.
"Official Fix for the Zune 30 Fail," www.gizmodo.com, December 31, 2008.
Bojan Simic, "Application Performance Management: The Lifecycle Approach Brings IT and Business
Together," Aberdeen Group, June 2008.
"Improving Software Quality to Drive Business Agility," IDC, June 2008.
"The Economics of Testing," www.riceconsulting.com, 2001.
cognizant reports
5
Author and Analyst
Vinaya Kumar Mylavarapu, Cognizant Research Center
Subject Matter Expert
Saravanan Kumar, Associate Director, Integrated Services Leader
Navin Rao, AVP, QE&A
About Cognizant
Cognizant (NASDAQ: CTSH) is a leading provider of information technology, consulting, and business process
outsourcing services, dedicated to helping the world’s leading companies build stronger businesses. Headquartered
in Teaneck, New Jersey (U.S.), Cognizant combines a passion for client satisfaction, technology innovation, deep
industry and business process expertise, and a global, collaborative workforce that embodies the future of work.
With over 50 delivery centers worldwide and approximately 166,400 employees as of September 30, 2013, Cognizant
is a member of the NASDAQ-100, the S&P 500, the Forbes Global 2000, and the Fortune 500 and is ranked among
the top performing and fastest growing companies in the world.
Visit us online at www.cognizant.com or follow us on Twitter: Cognizant.
World Headquarters
European Headquarters
India Operations Headquarters
500 Frank W. Burr Blvd.
Teaneck, NJ 07666 USA
Phone: +1 201 801 0233
Fax: +1 201 801 0243
Toll Free: +1 888 937 3277
Email: inquiry@cognizant.com
1 Kingdom Street
Paddington Central
London W2 6BD
Phone: +44 (0) 207 297 7600
Fax: +44 (0) 207 121 0102
Email: infouk@cognizant.com
#5/535, Old Mahabalipuram Road
Okkiyam Pettai, Thoraipakkam
Chennai, 600 096 India
Phone: +91 (0) 44 4209 6000
Fax: +91 (0) 44 4209 6060
Email: inquiryindia@cognizant.com
­­© Copyright 2013, Cognizant. All rights reserved. No part of this document may be reproduced, stored in a retrieval system, transmitted in any form or by any
means, electronic, mechanical, photocopying, recording, or otherwise, without the express written permission from Cognizant. The information contained herein is
subject to change without notice. All other trademarks mentioned herein are the property of their respective owners.