Agile Operations with Puppet

Agile Operations with
Puppet
How to ensure feedback loops at every level
Peter Simon <pesimon@team.mobile.de>
Uwe Stuehler <uwe@bsdx.de>
Our Site Operations Team
eBay company, most visited online
market place for vehicles in Germany
Germany's most visited
classifieds market place
~900 systems, multiple data centre
multiple OS
5-7 live deployments / day
Kanban
„DevOps“
Infrastructure Management Flow
Provisioning
Configuration
Deployment
Internal
Framework
Internal
Tool
inventory
network,
dns, dhcp config
starts install
Deployment
Pipeline &
Continuous
Delivery
Fully Automatic
Installation (FAI)
Solaris
Jumpstart
Apache
ZooKeeper
Before Puppet
●
FAI on Debian (even OpenBSD)
●
Re-installation to reflect changes, or the old for-loop
$ vi fai/config/classes/DBMASTER
$ for host in dbmaster46-1 dbmaster47-1; do
>
ssh -l root $host fai -v softupdate
> done
●
It's hard to write idempotent shell scripts :)
...and then came another platform
Configuration management for Solaris/sparc was needed (Apache tier)
●
JumpStart not even as useful as FAI
●
Puppet tried, as proof-of-concept
●
Got PuppetLabs private training...
How we're using Puppet now: Overview
Feedback Cycle #1: Edit, Compile, Run
Feedback Cycle #2: Publish, Verify, Revise
Feedback Cycle #3: Release, Monitor, React
Edit, Compile, Run
Per-User Puppet Environments
●
Personal environments + /home on NFS, on all Puppet masters:
●
Every admin has one, automatically
[ustuehler]
manifestdir=/home/ustuehler/puppet/manifests
modulepath=/home/ustuehler/puppet/modules
●
Puppet runs against personal environments, on any server:
●
Smoke tests (in production) with –noop
●
Test after editing, before commit
dbmaster46-1$ sudo puppet agent -t --env=ustuehler
Standard Puppet Environments
●
production, testing, development – standard Puppet environments
mapped to Git branches
●
Cloned from Gerrit, always clean, local filesystem
●
Updated via cron
cd /etc/puppet/production
git fetch gerrit
git reset gerrit/master
git clean -ffdx
git reset --hard >/dev/null
git submodule update --init
Working with Git – Workspace Setup
●
We use submodules for almost all modules
$ git clone --recursive gerrit:puppet
●
Workspaces are managed by Puppet
define site::admin_user($home)
{
git::clone { “${home}/puppet”:
source => 'gerrit:puppet'
}
}
Working with Git – Custom Shell Prompt
●
Top-level directory, branch, HEAD, stash-level:
puppet38-1:~/puppet/modules [master 5f532b7 stash@{0}]$
●
Detached head warning:
puppet38-1:~$ cd puppet/modules
WARNING: Repository ~/puppet is in detached head state.
puppet38-1:~/puppet/modules [475f8a8]$
Editor Support – Syntax Highlighting
file { '/hello.txt': content => "Hello World" }
vs.
file { '/hello.txt': content => "Hello World" }
vs.
file { '/hello.txt: content => "Hello World" }
Editor Support – Style Checks
●
Stick to the Puppet Labs Style Guide
●
vim hooks for puppet-lint
puppet-lint
https://github.com/rodjek/puppet-lint
vim-puppet & co.
https://github.com/rodjek/vim-puppet
The whole setup: http://jedi.be/blog/2011/12/05/puppet-editing-like-a-pro/
Publish, Verify, Revise
Gerrit Dashboard
http://code.google.com/p/gerrit
Gerrit Change
Jenkins CI
http://jenkins-ci.org/
https://wiki.jenkins-ci.org/display/JENKINS/Gerrit+Trigger
Check Syntax & Style (puppet-lint, again)
Catalog Compilation
Module Smoke Tests
cucumber-puppet
Di
sc
on
tin
https://github.com/nistude/cucumber-puppet
ue
d
Back to Gerrit, for a Change...
The Human Element
●
Review gets us valuable peer feedback
●
Review keeps everyone „in the loop“
●
Encourages us to „think twice“
●
Helps us to avoid mistakes
●
Helps us to write better code
●
Review is (almost) mandatory
Oh, and history becomes really useful...
commit 1e5469515245721996c3a23071882f7db8dfe24f
Author: Ingo Dyck <idyck@team.mobile.de>
Date:
Mon Sep 24 17:03:10 2012 +0200
zone_hostname and correct nameserver for integras
Change-Id: I6cef82ef352a0948e48ef49e6c6e49fe540996a9
Reviewed-on: https://gerrit/gerrit/2161
Tested-by: Jenkins CI <jenkins@team.mobile.de>
Reviewed-by: Peter Simon <pesimon@team.mobile.de>
Reviewed-by: Sascha Curth <scurth@team.mobile.de>
Reviewed-by: Ingo Dyck <idyck@team.mobile.de>
Tested-by: Ingo Dyck <idyck@team.mobile.de>
Release, Monitor, React
Nagios Checks
●
●
check_puppet_run (passive service check)
●
triggered via cron
●
freshness threshold: 24 hours
●
applies the catalog
●
reports success or failure
check_puppet_config
●
●
$environment == 'production' ?
check_puppet_state
●
Is puppet disabled locally?
Recap
We're hiring!
All links can be found here:
bit.ly/agile_ops