end to end automation for Continuous Delivery

White Paper
The Importance of Automating
the End to End Pipeline
for Continuous Delivery
The Importance of Automating the End to End Pipeline for Continuous Delivery
Executive Summary
Continuous Delivery (CD) can transform businesses
by enabling innovation, improving product quality,
and dramatically decreasing time to market, risks and
costs. To implement an effective CD methodology,
enterprises must integrate, orchestrate, automate, and
manage potentially hundreds of disparate point tools
and environments used across their development,
operations, and IT organizations.
Some believe that they have implemented an
automated CD methodology just because their
individual groups (silos) are using automated tools.
But, if one examines the flow between these groups,
what is more likely found is a scenario riddled with
manual handoffs. Besides being error prone, this
manual interaction is slow, does not provide a ‘single
pane of glass’ visibility and control of the entire end-toend pipeline, and doesn’t efficiently implement CD.
ElectricFlow is an enterprise-ready CD automation
platform that automates and integrates the product
pipeline across development, operations, and IT.
ElectricFlow provides unique value to an organization
by enabling orchestration, visibility, and control of all
phases of the end-to-end software delivery process as
well as providing a platform to integrate a disparate set
of tools and processes. ElectricFlow enables CD agility,
predictability and security across build-test-deploy
pipelines.
Avoiding Common Mistakes
2
is this not scalable, but these scripts are difficult to
maintain but they consume resources that would be
better spent on adding business value.
• Creating an infrastructure that isn’t flexible to support
changing business needs. Managing the end-to-end
pipeline is as much about creating business value is it
is about releasing software product. As such, flexibility
is needed to support changing business needs and
objectives.
• Creating an infrastructure or IT requirements that
impose limitations on the various teams that are part
of the end-to-end pipeline. This stifles innovation and
productivity.
• Trying to utilize tools for tasks that they were not
designed to do. Examples of this include using
continuous integration tools for deployment,
configuration management tools for end-to-end
orchestration, or using point tools designed for single
project usage to manage an entire organization.
Implementing Continuous Delivery
For the operations and deployment side of the
pipeline, IT often manages hundreds or thousands
of computing resources in various combinations of
physical hardware, private cloud, and public cloud
machines. When combined with the many variations of
environments, versions of application and middleware,
this becomes an almost impossible task to manage
and track.
When a business decides to automate their enterprise
CD initiative, these problems will likely surface:
At first glance, automating CD in an enterprise
environment that uses hundreds (or thousands)
of different tools, processes, environments, and
workflows may seem a daunting task. The first step
in doing this is to avoid the common mistakes that are
often made:
• Tool duplication: tools from different vendors are
used to perform similar tasks, or the same tool is used
in different ways by different groups.
• Investing resources on custom scripts and ‘glue’ to
integrate and automate the product pipeline. Not only
• Tool scalability: managing multiple instances or
versions of the same tool chains is difficult because
• Tool misuse: trying to use tools to do more than
their intended purpose such as attempting to use a
configuration management tool for automation and
orchestration.
those tools were not designed for enterprise usage.
• Process flexibility: business needs often dictate
that different processes are used in different parts of
the organization; therefore flexibility in automation is
required.
• Control and visibility of the entire pipeline: CD
covers the entire pipeline from development through
delivery. Being able to orchestrate and monitor this
entire pipeline is crucial. Integration with a wide
variety of tools is essential.
• Governance: Ensuring that internal and external
standards are adhered to and that appropriate data is
available for audit purposes is mandatory (even if that
data comes from many different tools).
• Large Scale Deployment: Home-grown scripts and
point tools not designed for enterprises do not scale
well. Achieving CD, controlling costs, and managing
hundreds or thousands of computing resources
becomes problematic with these alternative solutions.
This is especially an issue when the computing
resources are deployed globally and consists of a
combination of physical machines, private cloud, and
public cloud virtual machines.
The elusive last mile of implementing an effective
enterprise continuous delivery initiative requires
integrating, automating, managing, and orchestrating
the many different tools and processes that are
used across the enterprise. Unfortunately, this is
not a simple problem to solve and one that should
not be undertaken by in-house IT resources writing
homegrown glue. Instead, businesses must consider a
purpose-built, end-to-end, highly-available, enterpriseready automation platform that:
• Provides flexibility – utilizes open standards and
integrates with many different tools; easily implements
and automates different types of business logic
supporting any development/release/deployment
cycles including manual gates and dynamic
processes.
• Provides scalability & high availability – the platform
must be purpose built for enterprises and offer
redundancy, predictability, and high availability. It must
support a global scale on the magnitude of 1000s
of people, servers, and teams located globally. It
must offer the ability to control and share resources
equitably globally, helping to maximize the existing
infrastructure.
• Provides extensibility – the platform must support
open standards and have a robust API, support
3
two-way communication between tools effectively
enabling a disparate set of tools to be chained
together
• Provides visibility – at any given moment, the
platform must be able to provide the high-level
status of the entire application pipeline, available
environments, and log information for hundreds or
thousands of builds. In addition, the platform must
immediately notify when an error occurs and provide
adequate data to fix the problem. The platform must
be able to report on the exact configuration of all
components that are used in any particular build.
• Provides governance – at all times, compliance with
internal and/or external regulations is required. The
platform must also ensure security and provide data
needed for any audits.
ElectricFlow
Electric Cloud’s ElectricFlow is an enterprise-ready
DevOps process automation platform. ElectricFlow
provides distributed teams shared control and
visibility into infrastructure, tool chains and processes.
It accelerates and automates the software delivery
process to enable agility, predictability and security
across build-test-deploy pipelines.
ElectricFlow with Continuous Integration Tools
One of the key features of ElectricFlow is its ability
to orchestrate and manage a wide variety of existing
infrastructure and tools. As an example of this,
consider the continuous integration tool that many
development organizations use called Jenkins.
Jenkins was purpose-built for continuous integration
(CI) on a small project basis. CI is only one part of
continuous delivery. A common problem occurs
when Jenkins is used on multiple projects and that
is some organizations discover that the low cost
and simplicity of setting up a Jenkins server leads to
a proliferation of one-off Jenkins instances. These
“islands of innovation” are the result of a “get it done/
shadow IT” mindset among quick-moving development
teams. This decentralization of Jenkins CI activities
is sometimes referred to as “Jenkins sprawl,” and
presents organizations pursuing a Continuous
Delivery model with challenges. What is needed
is an orchestration and management tool, such as
The Importance of Automating the End to End Pipeline for Continuous Delivery
ElectricFlow, to provide the necessary orchestration
and automation layer of many Jenkins instances
to enable truly automated, end-to-end continuous
delivery flow at an enterprise level.
a click of a button, ElectricFlow can dynamically
provision a set of machines with OpenStack, configure
them, deploy an application to them, run integration
tests, and ask a group for manual approval.
By adopting ElectricFlow, organizations can:
• Extend the continuous integration infrastructure that
Jenkins provides with an end-to-end automated
continuous delivery infrastructure
• Gain a centralized view of Jenkins instances in the
enterprise
• Prioritize and migrate projects to ElectricFlow to
increase collaboration and sharing of artifacts across
the enterprise
• Accelerate organization-wide CD with a buildtest-deploy Center of Excellence to establish best
practices and leverage pooled enterprise resources
ElectricFlow with Configuration Management
Systems
Configuration management systems such as Chef,
Puppet, and Ansible enable infrastructure to be
represented as code which then is versionable,
buildable, and repeatable. While these tools are very
good at configuration management, this represents
only one part of end-to-end CD. ElectricFlow
integrates these tools into an end-to-end continuous
delivery process. With ElectricFlow, one can
orchestrate the configuration of virtual machine
instances as part of the environment set-up portion of
the CD workflow. All of this can easily be packaged
into a reusable procedure called from the main
automation pipeline for either build/test or application
deploy needs.
Managing Virtualization Platforms
The promise of a non-proprietary and extensible
private cloud (e.g. OpenStack), as well as the use
of public and hybrid clouds, has revolutionized IT
operations. This ties in well with another trend in
the industry, continuous application delivery, which
automates steps between code-check-in and
production deployment to provide a more predictable
path to application deployment.
ElectricFlow allows infrastructure and applications
to be integrated in a scalable and flexible way that
maximizes compute resources. For example with
4
360 Degree View of your Product Pipeline
The above section mentions just a few of the tools
that are used to move software from development to
delivery. In reality, many tools are used throughout the
various parts of an organization. Each of these tools
have their own UI, their own logins, and may require
a different skillset in the user. ElectricFlow allows all
of these tools to be orchestrated into a reusable highlevel pipeline process, alleviating manual handoff from
process to process.
Besides the problems associated with orchestrating
a disparate set of tools, there is a problem of data
visibility. When trying to collect data for internal
audits or for regulatory purposes, the data resides in
many different tools and is extracted out in a variety
of ways. However, using ElectricFlow alleviates this
problem by providing visibility in one single place - in
ElectricFlow. By doing so, from a single place, one
has access to all of the details about what changes
went into a build, which unit tests passed, what the
code quality was, the version of the artifacts used in
the build, the environments and applications deployed,
how many tests passed/failed, etc. ElectricFlow also
captures manual approvals that occur along the way
in the pipeline. Because all data is available in a single
system, one has a 360 control of the process and
visibility into the process.
Governance
ElectricFlow addresses governance issues by
providing:
• Reusable processes that can be shared between
teams to eliminate process duplication, enabling agility
across the organization
• Access control can be used to restrict who has access
to what
• Full access to information in a single pane of glass
which allows users to add automated actions based
on custom logic
• Process auditing and the ability to instantly roll back to
earlier versions
The Importance of Automating the End to End Pipeline for Continuous Delivery
ElectricFlow Automates Continuous Delivery for the Enterprise
• Flexible execution: ElectricFlow provides a flexible workflow automation engine easily allowing third party
products to be integrated, orchestrated, and managed.
• Visibility: Since all CD activities occur within ElectricFlow, activity can be viewed at various levels of granularity,
from executive overviews to process debugging.
• Infrastructure as code: Because all applications, environments, workflows are modeled and are represented by
code, the scripts and structure that make up the CD process can live in source code control.
• Integrations: ElectricFlow integrates with any tool that exposes an API via REST, SOAP, HTTP, CLI, etc. For
example: SCM tools like Git, Subversion, or Perforce; CI tools such as Jenkins; configuration management
tools such as Chef and Puppet; virtual machine infrastructure such as OpenStack, VMware, Amazon, Azure.
ElectricFlow provides a rich API that can be used to pull all the information needed to provide visibility to the end
users.
• Deployment automation: Applications, components, environments, and automated deployment processes are
modeled in ElectricFlow. By linking application tiers to environment tiers with tier maps, processes are reused
across environments of different sizes and configurations (development, integration, staging, production, etc.)
ElectricFlow provides easy access to determine which application versions are running on each environment.
• Resource management: Run processes on an explicit resource, pick a resource from a pool, or broadcast
it across an environment. Dynamically spin up and tear down as resources are needed. Resources/pools/
environments can be managed as an elastic cloud to maximize hardware utilization.
• Artifact management: ElectricFlow provides a built-in mechanism to publish and retrieve artifacts. Artifact
publishers and retrievers are tracked by the system to ensure visibility and auditability. Artifacts dependencies
are a very powerful mechanism used to combine and test several individually developed components into
applications or products. ElectricFlow also integrates with major artifact repositories like Artifactory and Nexus.
• Preflight: ElectricFlow provides a preflight mechanism whereby an end-user can run a production build/test/
deploy process to test their code changes before committing them. ElectricFlow simulates the commit by
using a clean source code copy and overlaying the user’s changes (which may include product source code,
configuration files, infrastructure as code, etc.) The preflight mechanism eliminates the chance of an invalid
change because code is not committed if the preflight fails.
• Access & security: ElectricFlow provides both coarse and fine-grained access control to manage which users can
read, modify, and execute the various objects. LDAP/ActiveDirectory groups and access control can be used to
set up role-based access control (RBAC). ElectricFlow provides a mechanism for securely storing credentials and
controlling access to processes to which they are attached.
• Enterprise scalability: Runs processes at an enterprise scale – supports any number of applications, resources,
environments, and teams on ElectricFlow’s multi-tiered platform. The horizontally scalable ElectricFlow server
allows you to add multiple servers behind a load balancer to increase throughput.
Conclusions
ElectricFlow is purpose-built to support the complete product lifecycle from development, to testing, to
deployment at an enterprise scale. With the core goal of providing new capabilities to customers quickly,
every aspect of the full end to end software delivery needs to be optimized rather than living in isolated silos.
ElectricFlow allows organizations to optimize the entire process from the time developer has committed code until
the delivery of new features.
ElectricFlow provides the ability to automate and orchestrate a disparate set of processes and tools that support
continuous integration, continuous delivery, automated release automation, and deployment. By leveraging the
following capabilities, ElectricFlow enables faster time to market, high quality, increased productivity and resource
utilization, and reduced costs.
5
The Importance of Automating the End to End Pipeline for Continuous Delivery
About Electric Cloud
Electric Cloud powers Continuous Delivery. We help mobile, embedded systems
and enterprise web/IT providers deliver better software faster by automating and
accelerating build, test, and deployment processes at scale. Industry leaders like
Qualcomm, SpaceX, Cisco, GE, Gap, and E*TRADE use Electric Cloud solutions
and services to boost DevOps productivity and Agile throughput.
For more information, visit electric-cloud.com.
Corporate Headquarters
Electric Cloud, Inc.
35 S. Market St, Ste 100, San Jose, CA 95113
T: 408.419.4300 F: 408.419.4399
info@electric-cloud.com
www.electric-cloud.com
Electric Cloud Europe
1650 Arlington Business Park
Theale, Reading
Berkshire RG7 4SA United Kingdom
T: +44 (0) 0207.872.5500
europe.info@electric-cloud.com
Electric Cloud Japan KK
22F Shibuya Mark City West
1-12-1 Dogenzaka, Shibuya-ku
Tokyo 150-0043 Japan
T: +81.3.4360.5375
japan-info@electric-cloud.com
©2015 Electric Cloud, Inc. All rights reserved. Electric Cloud, Electric Make, ElectricAccelerator, ElectricCommander and ElectricInsight are trademarks of Electric Cloud, Inc. Microsoft and Visual Studio are
either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. All other names are used for identification purposes only and are trademarks of their respective
companies. Microsoft, Visual Studio, and the Visual Studio Logo are trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries.