Introducing Large Scale Agile testing John Coogan, Oct 2013 Slide title

Introducing Large
Scale Agile testing
John Coogan, Oct 2013
Ericsson oss
› Operator Support System
› 1 Billion Subscribers
› 320 Operators
› Over 1000 Deployments
› PDU OSS Based in Athlone
Complex Networks, Simple Management
Ericsson Internal | 2012-01-31 | Page 2
Background
› Incremental waterfall
projects
› 2 major releases per
year
› Back-ended test
› Fault profile
Ericsson Internal | 2012-01-31 | Page 3
Cumulative
› Heavy handover process
RFA RFS GA
Why Agile?
› New product and architecture
› New opportunity
Change
› Culture
› Environment
› Increased output
› Shorter TTM
› Maintain high level of quality
Speed
› Decrease dependencies
› Customer feedback
Flexibility
An Opportunity to Excel
Ericsson Internal | 2012-01-31 | Page 4
Ericsson Internal | 2012-01-31 | Page 5
Test Strategy: Key points
Quality in Teams
› Development Team has test
responsibility and ownership
› Every sprint potentially releasable
› Test and automated part of “done”
Continuous Integration
› Continuous Integration and test at
all levels
› Fast Failure, fast feedback
Customer Collaboration
› Reduced TTM
› Customer focus - always on
› Release Verification Clean Room
and system level performance
Agile Principles: Better Quality
Ericsson Internal | 2012-01-31 | Page 6
Agile Test Quadrants
Business Facing
Automated
Manual and
Automated
TAF
Usability Tests
Exploratory Tests
Critique the Product
Supporting the Team
Acceptance Tests
Security Tests
Q2
Q3
Q1
Q4
Static Tests
Performance Tests
Unit Tests
Load Tests
Basic Integration Tests
ility Tests
Automated
Tools
Dev Env
TAF
Technology Facing
Ericsson Internal | 2012-01-31 | Page 7
Team Structure
APP1 SM
Continuous Integration
Release
Verification
APP2 SM
e2e
…
System
Performance
Deployment
APPn SM
Ericsson Internal | 2012-01-31 | Page 8
Deployment
Validation
Challenges – Test Profile
# of Resources
› Teams from various
backgrounds
› Dedicated testers versus
generalists
› Different levels of adoption
› Reliance on others
› Mindset!
Ericsson Internal | 2012-01-31 | Page 9
Pure Test Focus
Development/Test
Competence
Test/Design Competence
Pure Design Focus
Challenges – Integration
› Parallel development of CI
Framework
› Manual deployment of test
servers
› Early and frequent check-in
› User story size
› Big bang!
Ericsson Internal | 2012-01-31 | Page 10
Challenges – Test
Environment
› Parallel development of platform
and deployment procedures
› Complex production
environments
› Evolving toolset
› Competence level
Ericsson Internal | 2012-01-31 | Page 12
Challenges - Scale
› Large number of teams and
rising
› Strong focus on individual
teams
› Geographical diversity
› Vision gets lost
Ericsson Internal | 2012-01-31 | Page 13
BenEfits
Ownership
Automation and Efficiency
Fast Feedback
Customer Engagement
Quality Improvement
Knowledge Sharing and Collaboration
Ericsson Internal | 2012-01-31 | Page 14
What have we Learned?
› Full versus Incremental
Devolution of Test Responsibility
› Whole Team Test Approach is,
sometimes, not enough
› The Value of Release Verification
› Strong central strategy,
decentralized execution
› Test Coaching, Support to
Teams
› Dedicated Testers in
Development teams
Ericsson Internal | 2012-01-31 | Page 15
What have we Learned?
› Central provision of complex
customer like test environment
› Building a CI Framework and
populating it with quantity and
quality test cases takes time
› Task force needed to drive key CI
implementation
› Performance test in development
teams, small wins
› Running Scrum ceremonies is the
easy part…
Ericsson Internal | 2012-01-31 | Page 16
Principles behind the Agile Manifesto.
Welcome changing requirements, even late in
development. Agile processes harness change for
the customer's competitive advantage.
Deliver working software frequently, from a
couple of weeks to a couple of months, with a
preference to the shorter timescale.
Business people and developers must work
together daily throughout the project.
Our highestBuildpriority
to individuals.
satisfy the customer
projects around is
motivated
Give them the environment and support they need,
trust them
to get the job done.
through earlyandand
continuous
delivery
The most efficient and effective method of
conveying
information tosoftware
and within a development
of valuable,
quality
team is face-to-face conversation.
Working software is the primary measure of progress.
Agile processes promote sustainable development.
The sponsors, developers, and users should be able
to maintain a constant pace indefinitely.
Continuous attention to technical excellence
and good design enhances agility.
Simplicity--the art of maximizing the amount
of work not done--is essential.
Ericsson Internal | 2012-01-31 | Page 17
Questions
Ericsson Internal | 2012-01-31 | Page 18