Usability and User Experience Feedback Report

Usability and User Experience Feedback Report

PEACOX – Persuasive Advisor for CO2-reducing cross-modal trip planning
Project Reference: 288466
FP7-ICT 2011: 6.6 Low carbon multi-modal mobility and freight transport
Project Duration: 1 Oct 2011 – 30 Sept 2014
D7.2 Usability and User Experience
Feedback Report
[CURE – Center for Usability Research & Engineering]
Author(s)
Sebastian Prost (CURE)
Mariella Hager (CURE)
Matei Capatu (CURE)
Kathrin Röderer (CURE)
Date: 27/06/2013
Dissemination level: (PU, PP, RE, CO): PU
Project Co-Funded by the European Commission within the 7th Framework Program
27/06/2013
Abstract
This deliverable reports and describes the first usability and user experience evaluations for
the PEACOX journey planner app. It gives an overview of the applied user interface design
recommendations and a heuristics evaluation of the first mock-ups of the app on
consistency, controls and other usability features. It also shows the changes to the
application after the expert review. The deliverable describes the trials and usability test,
mentioning the task the participants had to go through, in order to evaluate the app in its
entirety. First reviews from the user evaluations show a general contentment with the
concept of the project and the application; still, there were some difficulties with the
handling of the app.
Page 2 / 33
27/06/2013
Table of Contents
Abstract ........................................................................................................................................................... 2
1.
Introduction ............................................................................................................................................ 4
1.1
Background ............................................................................................................................................... 4
1.1.1
1.2
2.
2.1
Analysis and Creation of Patterns ............................................................................................................. 6
2.2
Creation of Personas, Scenarios and Use Cases ........................................................................................ 6
2.2.1
PEACOX Personas ............................................................................................................................ 6
2.2.2
PEACOX Scenarios ........................................................................................................................... 8
2.2.3
PEACOX Use Cases......................................................................................................................... 11
5.
3.1
Application description ........................................................................................................................... 12
3.2
Expert-based evaluation methods .......................................................................................................... 14
3.2.1
Heuristic evaluations ..................................................................................................................... 14
3.2.2
Cognitive walkthrough .................................................................................................................. 16
3.2.3
User group walkthrough ............................................................................................................... 16
Results ..................................................................................................................................................... 17
Usability Test ........................................................................................................................................ 19
4.1
Changes in the Application after Expert Review ..................................................................................... 19
4.2
Description of the trial and tasks ............................................................................................................ 21
4.3
Results ..................................................................................................................................................... 22
Conclusion ............................................................................................................................................ 25
5.1
6.
User Interface Mock-ups Design Process ................................................................................................ 12
Expert review ........................................................................................................................................ 12
3.3
4.
The UID Expert Working Group ................................................................................................................. 4
User Interface Design – Process and Outcome ........................................................................................ 6
2.3
3.
Scope of this Deliverable ................................................................................................................. 4
UX Feedback Modification Positive User Comments: ............................................................................. 25
References ............................................................................................................................................ 27
Appendix ........................................................................................................................................................ 28
Page 3 / 33
27/06/2013
1.
Introduction
1.1 Background
Task 7.2 is described in the Description of Work as follows:
“To support the agile development process throughout the whole project on-going usability
feedback on the developed concepts and components will be provided. Different methods
such as expert evaluations, rapid user tests or card sorting, will be applied. Once the
developed concepts and prototypes become more mature they are tested in more controlled
lab and field test settings. Feedback is targeted to be given as soon as possible to inform
development in early stages where changes can be made easily and minor resources.
Feedback will be applied whenever modules, elements and procedures become available.
Feedback will be given rather informally, e.g. verbally in phone conferences or through use of
simple to do-lists, etc. However, to not lose the rationality of design choices, the results of
these feedback processes are summarized in a feedback report, that is updated at least
quarterly during the main development phases.”
In this 1st PEACOX Evaluation, the first PEACOX-mock-ups, as well as an application for
comparison (the AnachB-app) will be evaluated for the first time through potential endusers and usability experts.
1.1.1 Scope of this Deliverable
This deliverable provides an implementation concept for the 1st User Evaluation in the
PEACOX-project.
1.2 The UID Expert Working Group
Starting point of the development of the PEACOX user interface was the formation of the
UID expert working group. The group was constructed to integrate different skills and expert
perspectives and to jointly develop the first version for the PEACOX user interface design. It
consists of 7 experts (see Figure 1) coming from different scientific fields (computer sciences,
design, psychology, and sociology).
Page 4 / 33
27/06/2013
Through regular expert meetings and workshops the current PEACOX user interface design
was created. Personalised eco-feedback and persuasive eco-feedback strategies were
presented, discussed, developed and integrated during the creation and design of the user
interfaces.
Between the aforementioned official meetings multiple informal design critique and
feedback sessions have taken place both at CURE and at Fluidtime (FLU). While the main
design iterations took place after the kick-off (first iteration), workshop I (second iteration),
and workshop II (third iteration), several smaller iterations have been generated based on
these informal sessions.
Figure 1: Core Participants and responsibilities of the User Interface Design working group.
Page 5 / 33
27/06/2013
2.
User Interface Design – Process and Outcome
The initial situation of the joint work was a first rough draft of an interaction/interface
design, which was also presented at the consortium meeting in Dublin in May 2012. The
initial draft in form of an interaction design was a starting point for further work of the
taskforce. The main purpose of the rough draft pointed out the main features of the PEACOX
app. It served as a basis for discussion and further elaboration of functionalities.
The Deliverables D2.1 Description of User Groups and Travelling Context, D2.2 Requirements
Document and D2.3 Stakeholder and Technical Requirements built an important basis of the
work on functionalities and requirements in relation to the PEACOX app.
2.1 Analysis and Creation of Patterns
As a basis for the creation of the mock ups, already existing basic patterns from related apps
(provided by FLU) were partly used. The patterns were analysed, amended and completed.
New patterns had to be developed, e. g. for the implementation of persuasive elements, for
personalisation, for the creation of avatars, and for sharing in social networks. A detailed
description of the patterns is beyond the scope and therefore not part of this deliverable.
2.2 Creation of Personas, Scenarios and Use Cases
There were various personas, use cases and scenarios developed and described.
2.2.1 PEACOX Personas
A persona is a description of a user archetype that can serve as a guide in the design process.
The descriptions are summarised below including education, goals, skills, attitudes, job
description and personal details such as name, education, marital status, or favourite sport
to bring the character to life.
Different personas (male and female, different age groups, diverse marital status, with or
without children, different jobs, diverse environmental attitudes) in different situations
should contribute to a variety of use cases and scenarios. Their characteristics allow us to
reach a wide target group for PEACOX functionalities.
The following PEACOX personas were created and described:
 Rihanna, 33 years of age, 1 child, unemployed, married, ecological aware
 Jack, 40 years of age, independent employee, single
Page 6 / 33
27/06/2013
 Michael, 53 years of age, manager, married, no kids, status conscious
 Emily, 23 years of age, student, in a relationship, ecological aware
Rihanna, 33 years of age, married, ecological aware
Rihanna is married and has a daughter Mina. She is momentarily unemployed since she
wants to spend more time with her daughter. She is concerned about climate change and is
therefore interested in ecological way of living. Rihanna wants to use ecological friendly
transport modes in the future.
Jack, 40 years of age, single
Jack is 40 years of age and lives on the outskirts of a city. He works as an independent
employee in a downtown lawyer office. He likes to travel with public transportation to be
able to read and work while travelling.
Michael, 53 years of age, married, no kids, status conscious
Michael is 53 years old and C.E.O. of a middle-sized industrial company producing car engine
parts. He is a home owner in the outskirts of town with only minimal public transportation
available. He drives a big company car and is well aware of the status it represents. He is not
a climate change sceptic, but he needs the car for business trips. He likes its comfort,
privacy, and speed. In his opinion, only congestion in rush hour traffic represents a
disadvantage of the car usage.
Emily, 23 years of age, in a relationship, ecological aware
Emily is a 23 year old studying communication and politics. She is living in a shared flat, from
which she usually takes her bike to university or other places when she visits friends, goes
for a swim or go to see her sister. In case of bad weather she usually takes public
transportation. As she doesn’t know the network that well, she frequently needs to look up
connections and time tables to find her way.
Page 7 / 33
27/06/2013
2.2.2 PEACOX Scenarios
A scenario is a narrative explaining how the user uses PEACOX in his or her daily life. It uses
plain language free of technological specificity, but detailing the user’s intentions and
motivation. The scenarios below show how the user is solving a problem, getting decision
support, and changing behaviour by using PEACOX. The aim is to tell a common story of
what PEACOX will do without entangling in implementation specificities that will be tackled
in a next step. In particular, specific user interfaces are not part of the scenarios.
Different scenarios were created in order to cover an exemplary subset of PEACOX
functionalities and to make it conceivable. The scenarios cover various aspects of user’s life
(e. g. leisure, business, and official routes) and demonstrate how PEACOX functionalities can
be used in practice.
The following scenarios were created and described:
 In a hurry to the dentist
 Reviewing personal travel behaviour
 Creating a profile
 Navigate to the supermarket
 On the way to the office
 Switching over to public transportation
 Getting aware of your impact
 Rediscovering the city by bike
 Challenging your environmental impact
In a hurry to the dentist
Rihanna has an appointment at the dentist and is late. She wants to use her new
smartphone app PEACOX to find the shortest way to the dentist by use of public transport.
The dentist is at the other end of the town. She switches on her smartphone and starts the
PEACOX application.
Rihanna uses PEACOX to learn about different routes suggested by PEACOX between her
home and the dentist. Rihanna is in a hurry, so she wants to take the fastest route. She
Page 8 / 33
27/06/2013
compares all possible routes with each other in order to have a basis for decision making,
which means of public transport and which route she wants to take.
PEACOX offers information and feedback about the eco-friendliness of the suggested routes.
As she is running late she decides to take the fastest route (underground).
Reviewing personal travel behaviour
Rihanna arrives at the dentist and has to wait. She starts the PEACOX app to view the routing
details, the produced emissions and the travel time of her latest routes. She is happy as the
routes she chose are eco-friendly and decides to post her travel behaviour on Facebook. She
is very proud about the fact that she made a small contribution to saving our planet.
Creating a profile
Rihanna has to pick up her daughter from the kindergarten but she recently sold her car.
Therefore, she is not interested in car routing information anymore. She personalises her
profile to get optimised routing information. She starts the PEACOX app and requests the
route to the kindergarten by public transportation. She compares the routing possibilities.
Because she has to carry heavy bags with her she decides to take the route with the shortest
pathway from the station to the kindergarten.
On the way to the office
Jack has to leave home early to be on time to a meeting with a very important client.
Because he had to work very late yesterday he needs to prepare while travelling and decides
to go by public transportation. To make sure he will be at the office on time he starts the
PEACOX application. He already is in a hurry and does not want to enter a Route. He starts
the Live-Tracking the time he walks through his door. The app tells him the real time
information of the public transportation in his vicinity. He discovered that there is a
disturbance on the subway he usually takes and decides to take the bus and the tram. During
travelling the application tells him which bus and tram he uses and how many stations are
still left. He prepares for the meeting while he travels comfortably on a bus and a tram to
his office. When he leaves the tram in front of his office building he stops the Live-Tracking.
Because he has still some minutes left until the meeting starts and he already is well
prepared, he decides to drink a coffee and relax a minute before his client arrives.
Page 9 / 33
27/06/2013
Going to work / use park + ride (P+R) to reach the office
Michael is on his way to work, an office tower in the city. It’s morning and once again he is
stuck in heavy traffic. Recently he started using PEACOX because it offers turn-by-turn
navigation and live traffic data. He hoped it would guide him around the traffic jams to get
him to work quicker. Some days however, there is just no faster way. Suddenly PEACOX
points out a P+R facility a few blocks down the road, and estimates that switching to subway
there would save him at least 15 minutes in time. He never tried P+R before, but is intrigued
to do so, considering the time savings. Once at the P+R PEACOX tells him witch trains he has
to take to arrive at his company’s headquarter.
At the end of the day, he takes the same connection back to the P+R and from there his car
to get home, avoiding 5 o’clock rush hour. He decides that from now on he will more
frequently use P+R in combination with PEACOX.
Getting aware of your impact
After using the app for a while, Michael finally decides to connect the app to Facebook to
share his P+R discovery with his friends and family. The app allows him to see his trip history
and what emissions he produced. He is impressed to see how big a difference it made
switching from car-only to P+R. Also, he realises that during his daily traffic jam he was
exposed to bad emissions from the other vehicles. Air quality certainly improved once he got
away from the street and started using the subway.
Rediscovering the city by bike
It’s a sunny day and Emily is riding her bike to University. After the lecture she decided to
visit her friend Sue, who recently moved to a new flat. Although she knows the city well, she
doesn’t always know the best – or most comfortable – route for bicycles, especially if it’s a
place she hasn’t been to before. Recently she downloaded the PEACOX app onto her
smartphone, because it nicely integrates trip planning for bikes and public transport, which
are her usual ways of getting around in the city. She tries PEACOX this time to see if there is
an alternative, safer route she could take to her friend. PEACOX suggests a nice route she
hasn’t taken before. She also has the option to get turn-by-turn instructions while riding and
having her phone mounted on the handle bar.
Page 10 / 33
27/06/2013
Challenging your environmental impact
At her friend’s place Emily talks about the PEACOX app and demonstrates the trip planner.
Being able to see how little carbon emissions riding a bike or taking a train produces
compared to using the car is a good reinforcement and motivates her to discuss this with
some of her friends who are frequent car users. PEACOX not only tells her how much an
individual trips saves, but with the tracking function she can see how much she has emitted
last week or from the beginning of use. While playing with the app, she discovers the
“challenges” offered by PEACOX. She and her friend both decide to commit to the challenge
“Ride your bike at least 3 days a week”. Once completed, they will earn a reward, which they
can share on Facebook. They compete against each other and can see who is being more
environmentally friendly.
2.2.3 PEACOX Use Cases
A use case is a methodology used in system analysis to identify, clarify, and organize system
requirements. The use case is made up of a set of possible sequences of interactions
between systems and users in a particular environment and related to a particular goal. The
use case should contain all system activities that have significance to the users (Margaret
Rouse 2007). Due to different requirements of the PEACOX project partners involved, input
from the partners was important.
In the framework of the UID working group a table in form of an excel sheet was developed,
in which the PEACOX partners wrote all important use cases and scenarios. This enabled and
guaranteed the acquisition of all necessary functionalities for the PEACOX app. The following
high level use cases were defined:
 Personalisation
 Route information
 Navigation
 Live-Tracking
 Feedback
 Challenges/ rewards
Detailed information about the PEACOX use cases will be found in Deliverable 6.1 Report
about Use Case Scenarios which will be available at the end of month 14 (November 2012).
Page 11 / 33
27/06/2013
2.3 User Interface Mock-ups Design Process
Based on the results of WP2, the initial ideation concepts, together with the identified
persuasive strategies, ideas have been incorporated into paper drafts that explored design
possibilities for an Android-based PEACOX smartphone app. Travel planner, live-feedback
component, user profile, trip history, and social sharing and challenging components were
included in these drafts.
The first iteration drafts were discussed at Workshop I of the UID expert working group
concerning the design itself and its technical feasibility. After this initial evaluation, softwarebased mock-ups were designed as a second iteration. After several internal revisions as well
as an interim session of the persons responsible for design from CURE and FLU, the
improved design was again presented and discussed in Workshop II of the UID expert
working group. The third large iteration that followed this workshop was again internally
refined and resulted in a pseudo-interactive mock-up to be presented to all partners at the
consortium meeting in Athens 24 and 25 September 2012.
The work of the UID expert working group will continue and after further feedback and input
from all partners at the consortium meeting a forth iteration will be generated. The final
design of the PEACOX app will be included in the second version of this deliverable available
in month 25, (October 2013).
3. Expert review
3.1 Application description
The following Figure 2-Figure 6 show screenshots of the first prototype of the PEACOX trip
planner application.
Page 12 / 33
27/06/2013
Figure 2: Start screen showing the nearest
Figure 3: Route screen, quick view over settings,
underground station, City Bike availability and
which can be changed when clicked on. Settings
today's CO2 statistics. Route planner in the top
include: person options, modality, means of
right corner, timeline in the left bar, personal
transportation, time, and the selection of start
data, settings, challenges and other at the bottom
and finish
Figure 4: Start selection screen. View of
Figure 5: View of calculated routes. Route show
predetermined addresses and starting point in the
means of transportation, as well as departure and
green rectangle. Route summary on top shows
arrival time and the duration right next to them.
selected settings.
Page 13 / 33
27/06/2013
Figure 6: Screen showing a timeline of events
and various statistics, accessed by clicking on
the logo on the top left corner.
3.2 Expert-based evaluation methods
In this sub-section we provide an overview of usability methods, which are used by HumanComputer-Interaction (HCI) experts for evaluating the usability of the system.
3.2.1 Heuristic evaluations
The goal of a heuristic evaluation1 is to uncover most usability problems of software without
the involvement of end-users. A heuristic evaluation is normally conducted by a handful of
experts who are evaluating a system and assigning the found errors to a list of heuristics.
These heuristics are rules of the thumb based on long-term HCI experience. HCI experts
evaluate if certain software fulfils most of these heuristics. The higher the fulfilment of the
heuristics the better the usability of the software is.
The heuristics used are mainly based on the work on usability heuristics, e.g., by Molich and
Nielsen (1990) and Mandel (1997). These heuristics were extended by several other usability
heuristics based on CURE’s long-term experience.
1
A quick and easy user evaluation method based on certain „rules of thumb” of good user interface design.
Page 14 / 33
27/06/2013
The following usability heuristics for conducting the heuristic evaluations were used in the
PEACOX project:

Consistency: Consistency describes a common design of elements and processes
from the users‘ point of view; all user interface concepts should thus be consistently
designed

Feedback: Feedback means that users expect a sufficient system reaction to all of
their actions and interactions

Efficiency: The user interface must enable the users to carry out their tasks efficiently

Flexibility: The system must allow different users to work differently, or a single user
to work differently if she wishes or needs to, in order to accomplish goals of the
users.

Clearly marked exits: The user must always know how to leave a specific context,
window or display when working with a user interface, and how the user can return
to the starting position

Wording in the users‘ language: The wording of the user interface must be known
and easily understandable to the user

Task orientation: The user interface should always be designed to suit as perfectly as
possible the users‘ tasks; never should a user need to adapt to a system

Control: The user must always be in control of the system; the user must never have
the feeling of being controlled by the system

Recovery and forgiveness: The system must prevent the user from (unknowingly)
taking severe actions; the user should be able to undo changes or actions easily

Minimise memory load: The user should be able to totally focus on the task, not
being troubled with the user interface as such; therefore the user interface must
require as little cognitive effort as possible

Transparency: The user must always know what will happen when the user takes an
action - the user interface must be transparent

Aesthetics and emotional effect: Everything has an emotional effect; if a user
interface has an inappropriate emotional effect, it will interfere with the user’s tasks
Page 15 / 33
27/06/2013
3.2.2 Cognitive walkthrough
The cognitive walkthrough method proposed by Wharton et al. (1994) is intended to give
insights into problems a novice user is expected to have. A usability expert “walks” through
the smartphone-application according to a certain task. Based on the task analysis the expert
takes different routes through the application with the mind-set of a novice user and
analyses every interaction step of the tasks. The focus of the method is “ease of learning”, in
particular “learning by exploration”. Through this method a usability expert is able to
uncover, for example, problems in the workflow of an application.
3.2.3 User group walkthrough
For the user group walkthrough, the PEACOX user groups [D2.1] were used for an early
expert-based evaluation using the cognitive walkthrough method. In the user group
walkthrough, an expert – or a group of experts – steps through a system according to predefined context scenarios.
The expert evaluators imagine themselves to be a user group and the scenarios are created
from the user group perspective. The single tasks of the scenarios represent the user group’s
typical interaction with the interface and are selected according to the user groups
attributes. Therefore the evaluator is able to see the system through the eyes of the user. In
this case, the user is not only one possible and loosely defined person, but resembles a welldefined target group of the system. User group walkthroughs can be conducted in three
different steps. Either they are used for a rapid evaluation of a system, which can take about
one to two hours, or they can be used for a more formal review with more detailed tasks.
Another possibility is to use the user group walkthrough as part of larger design efforts. In
PEACOX user group walkthrough will be used for feedback sessions between HCI researchers
and developers. The conduction of this kind of walkthrough will ensure a strong focus on the
needs and wishes of the users of our target groups.
Employing user group walkthrough, user-typical design issues can be detected early in the
design process. Furthermore, also the entire user experience and learnability of a system can
be investigated. The outputs of the user groups based cognitive walkthrough are usability
issues, user experience flaws and concerns, but also detailed suggestions for improvement
of the system.
Page 16 / 33
27/06/2013
The goal of the user group walkthrough is to not only get insights into the efficiency of a
workflow of the application, but rather to focus on the workflow of a defined user.
3.3 Results
Based on the expert evaluations methods, the following results could be obtained.
Navigation: No consistent navigation concept
•
Problem: No main navigation, users have to search for the main functions in different
places: Route planner in the top right corner, Timeline in left bar, Personal data,
settings, challenges etc. from the bottom.
•
Comment: Users need a consistent navigation concept for easy locating.
•
Solution: Offer a main navigation menu in the left bar (currently contains the
timeline).
•
The main menu should contain Route planner, Timeline, Challenges etc. and
My Account
•
Their contents should open in the content area
•
The start screen should be the Route planner (as the main functionality) or a
dashboard that offers a clear value for users (direct links to primary
functionality like the route planner, relevant status, events, notifications)
Navigation: Icons
•
Problem: Meaning of icons is not clear
•
Comment: Icons, especially if they are used without text and for main functions,
should be easily recognizable
•
Solution: Use recognizable icons.
– Main functionality such as the route planner should (additionally) be available
in a main menu with text label. Currently users might expect additional
settings or something similar behind the access to the route planner.
Low contrast of texts
•
Problem: Text has low contrast in some areas, e.g. in the My Account area, Route
planner
•
Comment: Colour contrast of text should be high to ensure good legibility.
•
Solution: Take care of sufficient colour contrast.
Page 17 / 33
27/06/2013
Usefulness of wizard concept
•
Problem: The concept of the route planner promotes a wizard-like navigation which
is an inefficient way to reach the user’s goal.
•
Comment:
– Users want to get their results as quickly and easily as possible. Typically they
need only a few settings to form their request.
– Once a user enters the editing process, the wizard navigation leads through
all the remaining steps, no matter if needed or not.
– Especially if previous requests are reused, the wizard is much more inefficient
than editing single search parameters.
– It is possible to edit a single attribute and go back to the request page using
the HW back button, but this way to navigate is not obvious and not
sufficiently supported by the user interface.
•
Solution:
– Offer a short, efficient way for simple travel requests, and make additional
parameters optional
– Keep the idea to present the request on one page, maybe (but not
necessarily) as a sentence, but support users in editing individual attributes as
they like (contrary to guiding them through all of them).
– Omit choices that do not have a value for users (number of adults/children. If
a ticketing function will be added, let users chose the number of travellers
after they have chosen a route, and only if they chose public transport)
– Offer an accessible travel option on the request page, not hidden in the
easy/comfortable/fast category, since it is essential for those who need it, no
matter of balancing and trading off like the other attributes.
Results List: Easy Scanning of Results
•
Problem: Users cannot easily distinguish time and length of the trip
•
Comment: The result list should be designed in a way that users can easily scan it and
find the trip that is best for them.
•
Solution: Make time and length of the trip easily distinguishable and comparable
over the different results by using 2 lines and different weight.
Page 18 / 33
27/06/2013
4. Usability Test
4.1 Changes in the Application after Expert Review
Several changes were made after the expert evaluation of the PEACOX mock-ups. They can
be obtained from Figures 7–12.
Figure 7: Menu screen accessed by clicking on the
Figure 8: Settings when establishing a route
logo on the top left corner. Screen shows
describing the options; choice between a more
following functions: home, route, account, and
comfortable, barrier free or faster route.
about.
Page 19 / 33
27/06/2013
Figure 9: Start selection screen showing
Figure 10: When clicking the footsteps on the left
predetermined addresses, including "here".
side of the map, the application shows a bottom
Symbols can indicate address or station
up information on the route, displaying the
different stops step-by-step.
Figure 10: Route selection screen, on the left showing the different travel possibilities including means of
transportation, times, and CO2 feedback. The right screen shows a detailed description of the route. The
button “Go” on the top right shows the route on the map.
Page 20 / 33
27/06/2013
Figure 11: Account and vehicle screen display personal information and the possibility to enter various values
regarding the user's car.
4.2 Description of the trial and tasks
The application was tested on a HTC Sensation Smartphone running on Android OS 4.0. Five
participants took part in the usability and user experience trial, three of them being women.
The average age was 34.8 years. The requirement for the participation to this trial was the
usage of a smartphone for at least three months. This criterion was fulfilled by all the
participants.
Participant
Sex
Age
Use of mobile route
planer
Use of web-based
route planer
1
2
3
4
5
M
F
M
M
F
22
29
45
37
41
Daily
Several times a week
Never
Several times a month
Several times a month
Several times a week
Several times a week
Several times a month
Rarely
Rarely
Table 1: Sociodemographic data of the trial participants
The duration of the trial was 45 minutes and the participants were asked to perform various
tasks to test different aspects of the application. They were asked to comment their actions,
thus acting in accordance with the thinking aloud concept, and they were filmed performing
their tasks using a screen capture device. Before the start of the trial, the subjects were
given a questionnaire collecting demographic information. Then, participants were asked to
Page 21 / 33
27/06/2013
try the application freely and describe their first impression. They were asked about what
functionalities they could recognise, the colours, readability, and structure of the
application.
Afterwards, the participants tried out the route planning. They were asked what means of
transportation they used to arrive at trial site. Their task was to recreate the route from their
departure location to the trial site. Then, the participants were asked whether or not the
resulting route recommendations made sense and if they coincidentally had been using one
of the displayed routes. They were also asked if they recognised the CO2 information and to
describe their impression of it. To conclude this task, they were asked to describe their
impressions of structure and traceability of this function.
Next, the participants had to answer various questions regarding their choice of transport
modes and the subjective relevance of CO2 consumption. Afterwards, they planned a route
of their choice and were asked, if they would choose a route or transport mode based on the
CO2 consumption displayed. Again, they were asked about colours, readability, and also
display of the CO2 information.
The next task was to have a look at the options of the “Account” function and comment on
the functionalities. The participants were asked concluding questions regarding the
application.
Finally, the subjects had to fill in a short questionnaire containing the usability scales HED/UT
(Voss, Spangenberg, & Grohmann, 2003) and SUS (Sauro, 2011). The specific description of
the trial tasks, as well as the questionnaires can be found in the appendix.
4.3 Results
Usability
A first difficulty appeared when using the Back-button on the home screen: the application
was closed and had to be restarted. The participants, however, felt the need for a Backbutton in order to navigate more easily through the app.
The setup of the planning was easy, although there were many difficulties in confirming the
chosen address or station. The participants first tried to press the Enter-button or the
magnifier-symbol to select the “from” and “to” points of the route, before pressing the
green address button.
Page 22 / 33
27/06/2013
The map feature was not so well received, as it didn’t show the route automatically. Instead,
it showed the GPS location on the map, which differed from the route in some cases.
The daily CO2 feedback was not found, as it could only be accessed when clicking a rather
small icon on the map view of the application or after restarting the app.
The participants often asked about a navigation function and speech output, sometimes
forgetting the fact that the application provided just finding a route. They said that it would
be difficult to remember the route. This may be because they could not always find the
direction window after choosing a route.
Colours and design
The colours were perceived as gentle and pleasant most of the time, the darker route finding
screen was being questioned for its lighting conditions. While the text was generally easy to
read, it was commented that the space was rather compressed due to the big taskbar at the
top of the screen. It was easy to understand and liked that the colour scheme was chosen for
the eco-friendliness of the application. The colours and symbols for the different means of
transportation were also liked, especially in case of the metro, as it showed the colours of
each line.
Route
The main factors for choosing transport modes were:
• Travel time (including time of day to anticipate congestions)
• Weather
• Costs
Secondary factors included: comfort, barrier-free circumstances, and information on traffic
jams and disruptions but also the task, e.g. the car would be preferred when transporting a
heavy load.
Choosing barrier-free routes in the route options was evaluated positively, but options on
walking time were not liked, e.g. walking comfortably resulted in 18 minutes walking time.
Also, it would have been more helpful to show suggestions when typing an address.
Page 23 / 33
27/06/2013
Another commentary on the route and its options was to have the possibility to sort the
route results by various factors. As an alternative, the participants named the possibility to
view the different routes for just one transport mode.
The majority of the subject arrived by foot or public transport and agreed on the routes
suggested by the planner when evaluating them in retrospect. Some additional suggestions
were:
• Finding a location without route
• Having information on travel costs (ticket, gas prices, and others)
The information on the city bikes was interesting, but more information on using them as
well as their position and where to park them was desired.
CO2-Feedback
When first viewing the CO2 output, the participants’ first impression was that this
information was interesting. The information was understood well and the position on the
screen was positively evaluated. Additionally, the colouring and emphasis of the most
economic friendly route was praised. Still, when asked if it had any relevance on the route
choice, the answers were ambiguous. When comparing the different routes for an indicated
route option, participants chose the most time efficient route.
When asked if the display of CO2 information alongside the calculated route would influence
their choice, the answers were again ambiguous. Although initially interested, the duration
and means of transport weighed more, e.g. the bicycle would be chosen in perfect weather
and traffic conditions.
Settings and features
The information in the user account was perceived as unnecessary.
The input of vehicle data was not interesting, also other vehicles or vehicle information
could not be added. It was mentioned that it was not second nature to know all the data in
this option window. The subjects also said that adding information here is rather useless, as
it would be the least eco-friendly option anyway.
Following functions or options were listed as more helpful for the settings window:
• Option for favourites
Page 24 / 33
27/06/2013
• Option for statistics, e.g.: consumption, costs, and logbook
• General settings: Font size, colours, photo and others
• Time with calendar to take in account schedules for calculation of the time of arrival
In conclusion, the expectations were fulfilled and participants would like to continue using
the route planer because of the ease of use and the multifaceted functions it has. They
noted that such an application is indeed useful for smartphones but also said that their use
would depend on the costs.
The things that were favoured the most were the possibility of showing routes for multiple
means of transport, including the walking distances, and the option to choose barrier-free
route planning and other route finding settings. They were also pleased with viewing the
most eco-friendly route.
The things that were not positively evaluated were the use of the keyboard and buttons.
Especially using or knowing when to use the Back-button and choosing the addresses were
perceived as difficult. The route summary was rated as rather difficult to understand and felt
that they would like an “A to B” route as a quicker option when confronted with long loading
times. The participants generally liked the application, but asked for additional information
to be added, such as information about city bike stations, parking zones, problems related to
public transport or traffic jams.
5. Conclusion
5.1
UX Feedback Modification
Positive User Comments:
–
Highlighting of most eco-friendly route
–
Routes for multiple means of transport
–
Routes were mostly correct
–
Barrier-free options
–
Colours were mostly pleasant
Negative User Comments:
–
Search “statement”: unclear when start, destination, preferences are entered; no
Page 25 / 33
27/06/2013
Confirm-button
–
Map difficult to use: centred at current position, not route
–
Cycling routes sometimes longer than necessary
Observed Difficulties:
–
Back-button
–
daily CO2 feedback
–
Confirming the address
Daily CO2 Feedback
–
Difficult to find
Route CO2 Feedback
–
Not top priority when planning
–
Interesting and understood
Requested settings:
–
Favourites
–
Statistics (consumption, cost, log)
–
More information on city bikes
Page 26 / 33
27/06/2013
6. References
Brooke, J., 1996. SUS: A quick and dirty usability scale. In: Jordan, P. W., Weerdmeester, B.,
Thomas, A., Mclelland, I. L. (Eds.), Usability evaluation in industry. Taylor and Francis,
London.
Hassenzahl, M., Burmester, M. und Koller, F. (2003). AttrakDiff: Ein Fragebogen zur Messung
wahrgenommener hedonischer und pragmatischer Qualität. In: Ziegler, J. und Szwillus,
G. (Hrsg.), Mensch & Computer 2003, Interaktion in Bewegung, S. 187-196. Stuttgart:
Teubner.
K. E. Voss, E. R. Spangenberg, and B. Grohmann (2003). Measuring the hedonic and
utilitarian dimensions of consumer attitude. Journal of Marketing Research, 40(3), 310 –
320.
Sauro, J. (2011). Measuring Usability with the System Usability Scale (SUS)
Page 27 / 33
27/06/2013
Appendix
Manual for Usability-Test and questionnaires
1. Allgemeine Informationen
●
Termin: 06.-09.05.2013
●
Ort: Live-Observation Room
●
Anzahl der TeilnehmerInnen: 5 Personen > 18 Jahre, Smartphone-NutzerInnen
●
Aufwandsentschädigung: 40 Euro
2. Einleitung (0' – 5')
●
Begrüßung und Vorstellung der TestleiterInnen (Sebastian Prost, Matei Capatu)
●
Allgemeine Einführung und Vorstellen des Ablaufs
●
○
Dauer: 45 Min
○
Prototyp testen
Vorstellung des Themas:
Bei der heutigen Testung wird es um einen mobilen Routenplaner gehen. Dieser soll Ihnen das
Finden der optimalen Route erleichtern. Wir bitten Sie, diesen Prototyp der Applikation zu
evaluieren.
Es gibt weder richtige noch falsche Antworten, allein Ihr persönlicher Eindruck zählt. Nicht Sie
werden getestet, sondern die Applikation. Wenn Ihnen etwas unklar ist, Sie Fragen haben,
Ihnen etwas nicht gefällt (oder gut gefällt), dann bitte ich Sie das einfach laut zu sagen.
Generell bitte ich Sie, jeden Gedanken den Sie haben, einfach laut auszusprechen, also laut
mitzudenken.
●
Videoaufnahme: Fragen, ob die teilnehmenden Personen etwas dagegen haben, dass der Ton
der Testung aufgezeichnet wird. Außerdem nimmt eine kleine Kamera den Bildschirm des
Gerätes auf. Dies dient nur zu Zwecken der Auswertung und wird nicht außerhalb des Projekts
weitergegeben.
●
Aufzeichnungserlaubnis ausgeben und unterschrieben wieder einsammeln.
●
Vorfragebogen:
o Geschlecht
o Alter (in Jahren)
o Wie lange besitzen Sie schon ein Smartphone?
_____________ Monate/Jahre
Page 28 / 33
27/06/2013
o Nutzen Sie derzeit einen mobilen Routenplaner?
☐ Ja
☐ Nein
o Wenn ja, welche/n?
_____________ ____________________
o Nutzen Sie Routenplaner im Internet?
☐ Ja
☐ Nein
o Wenn ja, welche/n?
_____________ ____________________
o Wie häufig nutzen Sie
den mobilen Routenplaner?
den webbasierten Routenplaner?
o ☐ seltener als 1x im Monat
☐ seltener als 1x im Monat
o ☐ mehrmals im Monat
☐ mehrmals im Monat
o ☐ mehrmals in der Woche
☐ mehrmals in der Woche
o ☐ täglich
☐ täglich
o Bitte geben Sie an, welchen Anteil die jeweiligen Verkehrsmittel an täglichen
Wegen haben:
____% Auto ____ % Öffis ____ % Fahrrad ____ % zu Fuß
3. Präsentation und Bearbeitung der Tasks (5' –35')
Bitte halten Sie sich vor Augen, dass hier nicht Sie getestet werden, sondern der Prototyp und falls Sie
auf irgendwelche Probleme stoßen, dies nicht an Ihnen liegt, sondern dass das bedeutet, dass wir
etwas an dem System ändern müssen.
Bevor wir die App fertigstellen, wollen wir im Vorhinein gerne Ihre Meinung dazu haben, wie gut die
Darstellung der Benutzeroberfläche für Sie funktioniert. Wir werden Ihre Verbesserungsvorschläge
festhalten und versuchen diese umzusetzen. Ihre Rückmeldungen helfen uns die App
benutzerfreundlicher zu machen. Gibt es von Ihrer Seite noch irgendwelche Fragen?
Wie schon erwähnt, ist dies hier der Prototyp eines mobilen Routenplaners. Nehmen wir also einmal
an, Sie haben das Handy schon eingeschaltet und die Applikation gestartet.
Sie bekommen nun mehrere Aufgaben gestellt, die Sie so gut es geht lösen sollen.
Page 29 / 33
27/06/2013
Aufgabe 1: Freies Ausprobieren & Allgemeiner Eindruck (5min)

Bitte nehmen Sie sich ein paar Minuten Zeit, um die App auszuprobieren. Wenn Sie fertig
sind, sagen Sie bitte Bescheid.

Wie ist ihr erster Eindruck?

Welche Funktionen können Sie erkennen? Falls TP nicht alles nennt: Haben Sie a) das
Benutzerkonto b) die Sidebar c) die Karte d) das CO2-Feedback e) die Routensuchfunktion
bemerkt?

Hat etwas gefehlt, das Sie aber erwartet haben?

Was sagen Sie zu den Farben und Kontrast? Lesbarkeit des Textes?

Was sagen Sie zum Aufbau der Applikation?
Aufgabe 2: Eine Route planen (10min)

Mit welchem Verkehrsmittel sind Sie heute zu uns gekommen?

Planen Sie bitte im Nachhinein Ihre Route von Ihrem Ausgangspunkt zu CURE,
Modecenterstraße 17. Wählen Sie die Verkehrsmittel aus, die Ihnen auch tatsächlich zur
Verfügung stehen. Stellen Sie ein, dass Sie noch rechtzeitig vor Testbeginn ankommen
möchten.

Machen die vorgeschlagenen Routen Sinn? Könnten Sie sich vorstellen, eine der
vorgeschlagenen Routen auszuprobieren?

Deckt sich vielleicht eine der Routen mit Ihrem Weg hierher?

Gibt es weitere für Sie sinnvolle Routen, die nicht angezeigt werden?

Haben Sie die CO2 Informationen gesehen? Was halten Sie davon?

Was denken Sie über die Auffindbarkeit der Suchfunktion? War etwas zu versteckt oder war
es leicht zu finden?

Hat etwas gefehlt, das Sie aber erwartet haben?

Was sagen Sie zum Aufbau der Suchfunktion? (Anmerkung: Dass der Wizard den User Schritt
für Schritt durchführt)

Wenn Sie eine Sache an der Suchfunktion ändern könnten, was wäre das?
Aufgabe 3: CO2-Verbrauch ansehen (10min)

Welche Faktoren sind Ihnen persönlich bei der Verkehrsmittelwahl wichtig? (Falls keine
Antwort: Beispiele: Dauer, Komfort, Kosten, Barrierefreiheit, Umweltfreundlichkeit, etc.)

Die Applikation zeigt Ihnen an, wie viel CO2 Sie auf Ihren Wegen tatsächlich ausstoßen.
Wie sieht ihre heutige CO2 Bilanz aus?

Sind die CO2-Informationen verständlich? Umfassend genug? Würden Sie sich eine andere
Darstellung wünschen?

Welche Relevanz hat der CO2-Ausstoß bei Ihrer Verkehrsmittelwahl?
Page 30 / 33
27/06/2013

Planen Sie eine Route von hier zum Karlsplatz mit allen möglichen Verkehrsmitteln (Auto,
Öffis, Fahrrad, zu Fuß).

Welche Alternative würden Sie wählen?

Wie hoch ist der Ausstoß der jeweiligen Alternativen?

Können Sie sich vorstellen, auf Grund der CO2-Information auf ein anderes Verkehrsmittel
umzusteigen? Warum, oder warum nicht?

Was denken Sie über die Auffindbarkeit der CO2-Informationen? War etwas zu versteckt oder
war es leicht zu finden?

Was sagen Sie zu den Farben und Kontrast? Lesbarkeit des Textes?

Wenn Sie eine Sache an der CO2-Information ändern könnten, was wäre das?
Aufgabe 4: Einstellungen (5min)

Legen Sie ihre Heimatadresse als Ort im Benutzerkonto an.

Was denken Sie über die Auffindbarkeit des Benutzerkontos? War etwas zu versteckt oder
war es leicht zu finden?

Hat etwas gefehlt, das Sie aber erwartet haben?

Was sagen Sie zu den Farben und Kontrast? Lesbarkeit des Textes?

Was sagen Sie zum Aufbau des Benutzerkontos? (Falls User nicht alle Tabs findet/nicht
erwähnt: Was halten Sie von der Unterteilung in die einzelnen Abschnitte (Konto, Orte, Trips,
Herausforderungen, Statistik)?

Wenn Sie eine Sache am Benutzerkonto ändern könnten, was wäre das?
4. Abschlussfragen (35’-40’)

Wurden Ihre Erwartungen an den Routenplaner erfüllt?

Würden Sie den Routenplaner gerne weiterhin benutzen? Warum oder warum nicht?

Nennen Sie drei Dinge, die Ihnen am meisten gefallen haben.

Nennen Sie drei Dinge, die Ihnen am wenigsten gefallen haben.

Gibt es generelle Punkte, die Sie an der Applikation ändern würden?

Haben Sie sonst noch Anmerkungen?
5. Fragebogen (40‘-45‘)
Page 31 / 33
27/06/2013
Nach der letzten Aufgabe wird folgender Fragebogen vorgelegt:
Die Benutzung des mobilen Routenplaners war für mich…
Überhaupt
nicht
etwas
mittel
ziemlich
sehr
Brauchbar
O
O
O
O
O
Praktisch
O
O
O
O
O
Zweckmäßig
O
O
O
O
O
Hilfreich
O
O
O
O
O
Effizient
O
O
O
O
O
Aufregend
O
O
O
O
O
Spaßig
O
O
O
O
O
Unterhaltsam
O
O
O
O
O
Spannend
O
O
O
O
O
Fröhlich
O
O
O
O
O
Bitte beziehen Sie das Wort “System” auf Ihren Umgang mit dem mobilen Routenplaner. Bitte geben
Sie bei den folgenden Aussagen an, wie sehr Sie diesen jeweils zustimmen. Kreisen Sie die
entsprechenden Ziffern ein!
Ich stimme dem zu …
Ich denke, dass ich dieses System gerne häufig
benutzen würde.
Ich fand das System unnötig komplex.
Ich hatte den Eindruck, dass das System leicht
zu benutzen ist.
gar nicht
wenig
mittel
ziemlich
sehr
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
Ich denke, dass ich die Unterstützung einer
technisch versierten Person benötigen würde,
um das System verwenden zu können.
Ich hatte den Eindruck, dass die verschiedenen
Page 32 / 33
27/06/2013
Funktionen des Systems gut zusammenspielen.
Ich fand das System zu wenig stimmig.
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
Ich kann mir vorstellen, dass die meisten Leute
sehr schnell lernen würden, mit dem System
umzugehen.
Ich fand die Bedienung des Systems sehr
umständlich.
Ich fühlte mich in der Bedienung des Systems
sicher.
Ich müsste sehr viel lernen, bevor ich mit
diesem System umgehen könnte.
6. Debriefing
●
Bedanken für die Teilnahme
●
Ausgabe der Aufwandsentschädigung
Page 33 / 33