NYC_UserGroup_2

New York City Optimizely User Group
Charlotte Weiskittel
DonorsChoose.org
Photo credit: Scott Hudson!
Title text
AGENDA:
• 
Introductions
• 
Idea generation, prioritization & building a testing culture
- - Catherine Pao, Blue Apron
- - Marcus Im, Bark Box
- - Sergei Kogut, FSAstore.com
• 
Small group discussion
Introductions!
Name, position & company, favorite productivity tip or app
How many years have you been doing A/B testing?
<1
|
1
|
2-3
|
4-7
|
8+
What tool do you use for prioritization?
1
|
2-3
|
4-7
|
8+
How many tests in your backlog?
14
12
10
1
8
6
4
|
2-3
|
4-7
|
2
0
< 15
16 - 100
Too many /
∞
8+
Opticon!
June 17 & 18, 2015
•  www.optimizely.com/opticon/!
!
Training Sessions!
•  Training sessions will be held on June 16, 2015 at Optimizely
HQ in SF!
•  Topics: Strategy, Technical, Mobile!
!
7 Integrations Launched Year-To-Date!
Audience!
Analytics!
Content/CMS!
Heat Mapping!
Create personalized company & industry specific content for your
website & optimize those messages over time.!
Iron Mountain leveraged this integration to yield an increase of 123%
engagement with their home page into product specific pages. This also
allowed them to decrease their form size which led to 25% more form
conversions.
•  Test how different layouts and changes to your page impact the
viewability of your ads
•  Immediately impacts revenue, as you can charge higher rates for
spaces that result in higher viewability Title text
Idea generation, prioritization &
building a testing culture
- - Catherine Pao, Blue Apron
- - Marcus Im, Bark Box
- - Sergei Kogut, FSAstore.com
Prioritization
Experiment Backlog/ Pipeline
Idea Generation
1.  Marketing
2.  PMs
3.  Learnings from past experiments
4.  Heavily debated assumptions from internal
stakeholders about customer behaviors
Weekly Priorities Meeting
Sprint Planning - Asana
Detailed
Experiment
Data
Learnings Doc
Coupon Messaging as Header Landing Page Copy
1. Hypothesis: Moving the coupon
body message as the main header copy
to replace the more “explanatory” text
will increase conversion by highlighting
the offer text.
VS.
Coupon Messaging as Header Landing Page Copy
Great Resource
- 
Lot of inspiration from Brian Balfour’s methodology - “How To Design &
Track Viral Growth Experiments”:
http://www.slideshare.net/500startups/02-brian-balfour-hub-spot-final
Starting to Test!
How to get started?"
Sure, we have a product backlog,!
but we kept asking ourselves:!
“WHY?” and “HOW?”"
• 
WHY? are we working on the projects we are?"
• 
HOW? are we going to accomplish these?"
WHY?"
Build a KPI Tree"
• 
Which Metrics? What are the biggest opportunities? No time for everything."
• 
Goal Decomposition? Increased focus produces better ideas and makes
tracking progress more accurate."
WHY?"
Define Your Metrics"
• 
What Do We Mean? Agree with set of metric definitions. What is LTV? NPS?"
• 
Where Do We Find It? Decide on uniform data sources."
• 
How Do We Share It? Build universal reports for alignment and motivation."
HOW?"
Assign Teams & Go!"
• 
Who Does What? Where are your resources best spent?"
• 
How Will You Get There? Discovery, prioritization, testing."
• 
Is This Right For Us? Don’t overcomplicate, and be ready to adjust."
Marcus Im!
Product!
marcus@barkbox.com"
@marcusim"
FSAstore.com
An easy, convenient way for employees
to use and manage their FSA
Exclusively stocked with
FSA Eligible Products
FSA Tools to Empower Account Holders
Learning Center
FSA Tools to Empower Account Holders
FSA Calculator
FSA Tools to Empower Account Holders
Prescription Processing Feature
FSA Tools to Empower Account Holders
Eligibility List
Ideation – Team Meetings
Ideation – List of Ideas
Ideation – Open for Suggestions
Prioritization – Step in the Funnel
“The Conversion Funnel”
(Good)
BEST!
(Better)
Prioritization – Depth of the Page
Prioritization – Landing Pages
Prioritization – Degree of Difficulty
1) Easiest: using Optimizely UX
2) Slightly harder: writing
custom JS code
3) Can take longest: help from
Developers
Testing Culture
Example #1 – Counterintuitive Results
My Account page
+ FSA Deadline
+ Grace Period
+ TPA Name
Receipt Page
+ Carryover option
+ FSA Contribution Amount
All Landing Pages
+ Email Address
Example #2 – Unexpected Winner
Example #3 – Less is More
Thank you!
Your turn!
Where do you get ideas and how do you prioritize?
What are some challenges you’re facing?
Next event:
What do you want to learn about?
+
Would you be interested in presenting
or speaking on a panel?
(Please write down name & email if so)