BI Survey 12 Results updated

THE
lk
BI SURVEY
The Customer Verdict
The world’s largest survey of
business intelligence software users
12
11
11
A preview of The BI Survey 12: The Results
For more information, visit: www.BI-Survey.com
Free Preview
This document is not to be shared, distributed or reproduced in any way without
first purchasing licensing rights and the express prior permission of BARC.
2 The BI Survey 12 – The Results
Table of Contents
Introduction ............................................................................................................................4
Business Benefits and the BBI ...............................................................................................7
BBI by peer group and vendor ...........................................................................................9
Goal Achievement ................................................................................................................17
Deployment ..........................................................................................................................22
Usage in different company size classes .........................................................................22
Number of BI users ..........................................................................................................24
BI usage by departments .................................................................................................25
Product usage for application scenarios ..........................................................................29
The selection process ..........................................................................................................33
Evaluation methods .........................................................................................................33
Reasons to buy ................................................................................................................37
How products perform in the selection process................................................................42
Buying BI software ...............................................................................................................45
Licenses ..........................................................................................................................45
Number of users ..............................................................................................................47
Cost .................................................................................................................................48
Implementing BI and Satisfaction .........................................................................................54
Implementation time ........................................................................................................54
Project scope ...................................................................................................................58
Support ............................................................................................................................61
Recommendation.............................................................................................................62
Chosen as standard.........................................................................................................65
Problems in BI projects ....................................................................................................67
Query Performance and Data Volume ..................................................................................69
Query Performance results and user satisfaction .............................................................69
Data volumes handled per product ..................................................................................73
3 The BI Survey 12 – The Results
Front ends for multidimensional databases ..........................................................................75
Infor ION BI OLAP Server ................................................................................................76
Microsoft SSAS ...............................................................................................................76
Oracle Essbase ...............................................................................................................78
IBM Cognos TM1 .............................................................................................................79
SAP BW ..........................................................................................................................80
Trending topics in BI.............................................................................................................82
Three-step approach to analyzing trends .........................................................................82
The frequent inaccuracy of plans .....................................................................................83
User perception of IT and software trends .......................................................................84
Status and plans for trending topics in BI .........................................................................87
Visual Analysis & Data Discovery ....................................................................................88
Collaboration ...................................................................................................................90
Mobile BI .........................................................................................................................91
Cloud BI/BIaaS ................................................................................................................94
4 The BI Survey 12 – The Results
Introduction
This document provides an overview of the key product-related results from The BI
Survey 12. The Survey provides a detailed quantitative analysis of why customers
buy BI tools, what they are used for, how successful they are and why they
eventually abandon them. This edition has increased the range of products
reviewed, including not only products from well-known BI giants, but also specialist
tools from much smaller vendors and open source vendors.
The BI Survey 12 follows on from ten successful editions of The BI Survey (formerly
The OLAP Survey). It is important to note that this year the numbering of The BI
Survey has changed so that The Survey number now aligns to the year in which the
data was collected.
Based on analysis of real-world experiences of nearly 3,000 respondents, the value
of The Survey is dependent on a sufficiently large, well-distributed and unbiased
sample. The Survey is the largest and most thorough fact-based analysis of the BI
market currently available, using eleven years of experience to analyze market
trends and debunk many myths surrounding the BI industry.
After data cleansing and the removal of responses from participants unable to
answer specific questions on BI tools, we were left with a sample of 1872 end users,
283 consultants and 512 vendor employees. Participants from all over the world took
part in The BI Survey 12. Around 20 percent of respondents had an IT job function,
15 percent were Heads of BI, and 65 percent had various lines of business job titles.
The BI Survey is not based on anecdotal accounts or personal opinions, unlike much
analyst research. The Survey is not intended to be a measure of market shares and
unlike our research publication, ‘The BI Verdict’, it does not include product reviews
or qualitative product comparisons. The BI Survey does not attempt to forecast
future trends, instead providing evidence undermining the reliability of many such
forecasts.
In the past the vast quantity of data collected by The Survey has been presented in
one large document. However, The BI Survey 12 heralds the introduction of several
bitesize documents, each focusing on a specific area of The Survey.
5 The BI Survey 12 – The Results
Detailed information on the sample and Survey methodology can be found in the
‘Sample, Products and Methodology’ document. For advice on selecting, managing
and implementing products into your organization see the document ‘The BI Survey
12: Best Practices’.
Document
Description
The BI Survey 12 - The Results
Provides an overview and analysis of the most important results
of The BI Survey 12
The BI Survey 12 – Trending
Topics series
Provides in-depth analysis of the following trending topics:
SaaS/Cloud BI, Mobile BI, Big Data, Social BI, Collaboration,
Self Service, Analytical Databases
The BI Survey 12 – Vendor
Performance Summaries
A series of short executive reports highlighting all the productrelated results for each vendor.
The BI Survey 12 – Best
Practices
This report features recommendations on how best to choose a
BI product and the most important and relevant tool selection
criteria.
The BI Survey 12 - Sample,
Products, Methodology (this
document)
Provides details of the sample and an overview of our
methodology including the entire questionnaire and details of
our calculation methods.
The BI Survey 12 - KPIs and
Dashboards
This document outlines the definitions and calculation methods
of the KPIs used in The BI Survey 12.
Figure 1: Overview of The BI Surve y 12
The BI Survey Analyzer is an online tool allowing users to carry out their own
analyses of The Survey data. Using results from this year’s Survey, the tool also
offers users the ability to drill down using product and demographic filters.
2012 also sees the exciting introduction of a mobile version of The BI Survey
Analyzer for the iPad. The mobile BI Survey Analyzer focuses on product-related
results based on criteria (KPIs) such as business benefits achieved, implementation
costs, functional usage, product competitiveness, innovation, performance, customer
satisfaction and agility.
6 The BI Survey 12 – The Results
Figure 2: Screenshot of The BI Surve y Anal yz er
Figure 3: Screenshot of The BI Surve y Anal yz er iPad App
7 The BI Survey 12 – The Results
Business Benefits and the BBI
The BI Survey asks key questions regarding the benefits of BI projects. Respondents
are asked to indicate the level of achievement gained from a list of eleven potential
benefits. A scoring system is then used to derive a composite weighted score for
each benefit, based on the level of benefit achieved. This system is called the BBI
(Business Benefits Index). For further information on the calculation methods used,
see the ‘Sample, Products and Methodology’ document.
Figure 4 shows the overall breakdown of responses to the Business Benefits
questions. The six levels of achievement, along with their weightings, are shown on
the horizontal axis. Results are RAG (red, amber, green) rated with green indicating
Weighting
Do not know
(5024)
Got worse/more
expensive (226)
Not achieved
(2495)
Informally
suspected (2741)
Formally
claimed, but not
verified (2186)
Proven, but not
measured (5643)
Proven and
quantified (4961)
higher scores and red denoting lower scores.
10
8
5
3
-2
-6
0
Better business decisions
27.9%
39.5%
12.6%
10.7%
1.8%
0.2%
7.4%
Faster reporting, analysis or planning
45.2%
35.5%
7.4%
5.0%
2.0%
0.3%
4.6%
Improved customer satisfaction
17.7%
29.0%
13.8%
13.8%
5.3%
0.3%
20.1%
Improved data quality
29.6%
32.2%
10.6%
10.5%
6.2%
0.5%
10.3%
Improved employee satisfaction
22.8%
33.9%
13.5%
13.7%
5.3%
0.7%
10.1%
Increased revenues
10.3%
13.3%
11.0%
19.2%
9.0%
0.2%
36.9%
More accurate reporting, analysis or planning
33.6%
38.9%
8.9%
8.7%
1.9%
0.4%
7.6%
Reduced external IT costs
16.6%
14.2%
7.5%
11.8%
14.7%
3.4%
31.7%
Saved business headcount
11.4%
9.5%
5.9%
13.2%
25.7%
0.9%
33.4%
Saved IT headcount
11.1%
9.2%
4.9%
9.0%
29.3%
2.9%
33.6%
8.3%
11.4%
7.2%
13.9%
16.7%
0.8%
41.7%
Saved other non-IT costs
Figure 4: Frequenc y of Business Benefits (n=2116)
8 The BI Survey 12 – The Results
Figure 5 shows business benefits listed in order of the most commonly achieved.
Faster Business Intelligence
7.82
More accurate Business Intelligence
7.12
Better business decisions
6.85
Improved data quality
6.23
Improved employee satisfaction
5.93
Improved customer satisfaction
5.06
Increased revenues
3.03
Reduced external IT costs
3.03
Saved other non-IT costs
2.14
Saved business headcount
2.02
Saved IT headcount
1.60
0
2
4
6
8
10
Figure 5: Benefits overview (n= 2116)
Figure 5 Key Findings:
Overall, ‘Faster Business Intelligence’ rated higher than accuracy and more
than 80 percent of respondents stated that faster reporting, analysis and
planning had been achieved. 72 percent reported that ‘More accurate
reporting, analysis and planning’ had been proven and 67 percent of
respondents, up from 59 percent last year, said that ‘Better business
decisions’ was another benefit likely to be achieved. The ability to make
better business decisions is a highly desirable benefit. However, it is a benefit
that cannot be accurately scoped when developing the business case for a
project. While all BI projects would hope to gain this benefit, few projects
would be cost-justified against the possibility that it ‘might’ one day be
achieved.
‘Improved data quality’ is a benefit that can be measured directly. Although
often one of the most difficult factors to achieve in a business intelligence
project, it ranked fourth in the list with ‘Employee satisfaction’ following in fifth
place, a benefit that typically arises when users have fast access to data and
reports.
Benefits with the lowest level of achievement were ‘Saving headcount in IT
departments’, ‘Saving headcount in business departments’, ‘Saving non-IT
costs’ and ‘Reducing external IT costs’. This is in line with our experience that
companies are still looking to extend their BI capabilities, and are adding
resources to implement and run them. If BI were a mature and saturated
market we would see more investment in replacing existing systems but until
9 The BI Survey 12 – The Results
this pattern changes we would not expect savings in headcount and costs to
move up the list of benefits.
In contrast to previous results, this year the overall BBI decreased slightly from 4.89
to 4.62. However, we do not think this the beginning of a long-term trend of declining
benefits from BI projects. Instead we suspect that this small variation can be
accounted for by the fact that The Survey sample varies from year to year.
BBI by peer group and vendor
The difference between business benefit achievement according to the type of
product in use is reflected in the following chart. In order to compare product types
we group products into specific peer groups as explained in the ‘Sample, Products
and Methodologies’ document.
Visual Analysis & Data Discovery
Small & Medium Project Vendor
Dashboard
Performance Management
OLAP Analysis
IT Giants
BI Giants
Large Project Vendor
Enterprise Reporting
5.06
4.80
4.77
4.61
4.50
4.41
4.24
4.13
4.10
0
2
4
6
Figure 6: Business benefits anal yzed b y peer group (n=changing bases)
Visual Analysis & Data Discovery tools achieved the best BBI results reflecting their
user friendliness and flexibility in allowing end-users to quickly load data, perform
user analysis and take advantage of advanced visualization features. The userfriendly nature of these tools relates to business benefits achieved. Small and
Medium Project vendors rank second, Dashboard vendors rank third and
Performance Management products rank fourth. This is a result we have consistently
observed in previous editions of The BI Survey. Specialists, or smaller vendors, tend
to show better BBI results than the larger BI Giants, Large Project vendors and
Enterprise Reporting vendors. There is a straight correlation between the size of the
BI project (measured in data volume used or number of users served) and the size of
the tool vendor. With few exceptions, smaller vendors serve smaller projects and
products from larger vendors are used in larger projects.
10 The BI Survey 12 – The Results
In our opinion, there are several reasons for the good results achieved by smaller
and specialist vendors:
Some customers may achieve more business benefits when working with
smaller vendors with whom they have a close working relationship, compared
to the more arm’s-length, impersonal relationship that users tend to have with
large vendors. Survey results show that users usually experience better
vendor support, satisfaction and recommendation with smaller vendors than
with many of the large vendors.
The prospect of smaller vendors being removed from a company more
quickly if projects are not successful leads to better than average results for
the ones that remain in use. Conversely, large vendors are able to keep
customers with whom they have long-term, strategic relationships, even if
individual projects or products deliver disappointing results.
Products from small vendors are more likely to be selected in competitive
evaluations. Users expending more effort selecting solutions show a higher
level of benefit achievement. This could be because the extra diligence taken
to ensure that the product fits their needs increases the probability of the
project delivering business benefits. There is also the simple fact that some
projects are managed better than others.
But, for whatever reason, it is clear that the customers of small vendors such as
Yellowfin, Dimensional Insight and Phocas are reporting more business benefits than
those of larger vendors like SAP, Oracle, Microsoft and IBM. That said, several
products from large vendors have above average BBI scores, most notably the
multidimensional databases Oracle Essbase, IBM Cognos TM1 and Microsoft SSAS.
11 The BI Survey 12 – The Results
Figure 7 shows the business benefits scores for all products in the BI Giants peer
group.
Oracle Essbase
IBM Cognos TM1
QlikTech
Microsoft SSAS
MicroStrategy
Information Builders
Microsoft SSRS
Microsoft Excel
Infor
IBM Cognos BI
SAP BW
Oracle OBIEE
SAP BO WebI
4.92
4.90
4.65
4.64
4.56
4.50
4.28
4.23
4.11
3.76
3.55
3.45
3.33
0
2
4
6
Figure 7: Business benefits for the BI Giants peer group (n=949)
The BI Giants peer group includes companies with annual revenues of more than
$200 million and a truly international reach. These tools are usually used in
enterprise scenarios.
Figure 7 Key Findings:
The multidimensional database Oracle Essbase ranked first in the BI Giants
peer group. All five top ranking products offer in-memory database
technology, while the traditional ROLAP tools from SAP, Oracle and IBM
deliver the least business benefits.
Three of the four multidimensional databases in this peer group (all except
Infor) appear in the top five, including QlikTech (with its own in-memory
database technology) and MicroStrategy, having recently expanded its
caching technology.
The correlation between user self service and fast query performance with
business benefits has been evident in previous BI Surveys.
12 The BI Survey 12 – The Results
Arcplan
5.24
MicroStrategy
4.56
Information Builders
4.50
Microsoft SSRS
4.28
IBM Cognos BI
3.76
SAP BW
3.55
Oracle OBIEE
3.45
SAP BO WebI
3.33
0
2
4
6
Figure 8: Business benefits for the Enterprise Reporting peer group (n=555)
The Enterprise Reporting peer group includes products that can provide standard
formatted reporting in a large scale enterprise situation. Arcplan, a very flexible tool
for designing reporting applications, ranks highest in this peer group, followed by the
specialist vendors MicroStrategy and Information Builders. Users of the large
vendors SAP, Oracle and IBM report lower business benefits.
Yellowfin
6.31
Dimensional Insight
5.52
Arcplan
5.24
BOARD
5.19
QlikTech
4.65
MicroStrategy
4.56
Information Builders
4.50
Decisyon
3.58
Oracle OBIEE
3.45
0
2
4
6
8
Figure 9: Business benefits for the Dashboard vendor peer group (n=634)
The Dashboard peer group includes products that are focused on creating advanced
dashboards.
Users of tools from small vendors report the highest business benefit scores in the
Dashboard peer group. This group is led by Yellowfin, a tool with a modern
13 The BI Survey 12 – The Results
architecture and look and feel, with most of its customers being small companies.
Dimensional Insight, another specialist, comes in second, followed by two more
small vendors: Arcplan and Board.
BOARD
5.19
TARGIT
4.94
Oracle Essbase
4.92
IBM Cognos TM1
4.90
Bissantz
4.73
Microsoft SSAS
4.64
MicroStrategy
4.56
Jedox
4.52
Cubeware
4.49
Evidanza
4.35
Microsoft Excel
4.23
Pentaho
4.11
Infor
4.11
SAP BW
3.55
0
2
4
6
Figure 10: Business benefits for the OLAP Anal ysis peer group (n=939)
The OLAP Analysis peer group includes products that support analysis in
dimensional and hierarchical data models. This peer group is crowded and highly
competitive with many vendors scoring good business benefits results. The peer
group is led by Board, followed by three products with very similar results: Targit,
Oracle Essbase and IBM Cognos TM1. As in all previous editions of The BI Survey,
SAP BW users report considerably lower business benefit achievement than any
other product’s users.
14 The BI Survey 12 – The Results
Dimensional Insight
5.52
Phocas
5.30
Tableau
5.24
Cyberscience
4.97
QlikTech
4.65
0
2
4
6
Figure 11: Business benefits for the Visual Analysis & Data Discover y peer group
(n=532)
The Visual Analysis & Data Discovery peer group includes products that provide
advanced visualization features. Dimensional Insight rates top in this small peer
group of specialist vendors, followed by Phocas and Tableau. QlikTech, one of the
most successful BI vendors over the last five years and now considered to be a BI
giant, is starting to show similar results to the large vendors it used to comfortably
beat in this area.
BOARD
5.19
Oracle Essbase
4.92
IBM Cognos TM1
4.90
Jedox
4.52
Cubeware
4.49
Evidanza
4.35
Infor
4.11
0
2
4
6
Figure 12: Business benefits for the Performance Management peer group (n=420)
Figure 12 Key Findings:
The Performance Management peer group includes companies whose
products are predominantly used in planning projects. This peer group shows
above average business benefits results and is led by BOARD followed by
Oracle Essbase and IBM Cognos TM1.
Jedox and Cubeware hold fourth and fifth places, also scoring excellent
results.
15 The BI Survey 12 – The Results
Yellowfin
Dimensional Insight
6.31
5.52
Phocas
5.30
Tableau
BOARD
5.24
5.19
Cyberscience
4.97
TARGIT
4.94
IBM Cognos TM1
4.90
Bissantz
4.73
QlikTech
4.65
Microsoft SSAS
4.64
Jedox
4.52
Cubeware
4.49
Evidanza
4.35
Microsoft SSRS
4.28
Pentaho
Infor
4.11
4.11
Decisyon
3.58
0
2
4
6
8
Figure 13: Business benefits for the Small and M edium Project vendor peer group
(n=1308)
Figure 13 Key Findings:
The SME Project vendor peer group includes products that are primarily used
for small and medium sized projects (<500 users). The sample of Yellowfin’s
customers in The BI Survey 12 reveals its clear focus on small companies
with less than 100 employees. Yellowfin appears to serve its customers well,
scoring exceptionally high business benefits, head and shoulders above the
competition.
Dimensional Insight and Phocas follow in second and third places
respectively.
16 The BI Survey 12 – The Results
Arcplan
5.24
Oracle Essbase
4.92
MicroStrategy
4.56
Information Builders
4.50
IBM Cognos BI
3.76
SAP BW
3.55
Oracle OBIEE
3.45
SAP BO WebI
3.33
0
2
4
6
Figure 14: Business benefits for the Large Enterprise vendor peer group (n=535)
The Large Enterprise Project peer group includes products that are primarily used for
large enterprise projects (>500 users). In the Large Enterprise projects peer group, a
leading group of four products scored excellent business benefits: Arcplan, Oracle
Essbase, MicroStrategy and Information Builders.
Next Steps
The BI Survey Analyzer Demo
The BI Survey Analyzer is an interactive online tool which enables you to perform
your own custom analysis of the survey data.
Click here to register for an online demo and find out how effective The BI Survey
Analyzer can be in helping you to understand the strengths and weaknesses of BI
products and trends in the BI market.
Purchase
To purchase The BI Survey 12, visit our website or contact us at:
Email: sales@bi-survey.com
Tel: +44 (0)20 8133 8987