MovieFone: User Test Report

 MovieFone: User Test Report P r a t t I n s t i t u t e L I S 6 4 4 – U s a b i l i t y o f D i g i t a l I n f o r m a t i o n S p r i n g 2 0 1 3 Laura Liang Chris Mullin Samantha Raddatz Gili Warsett Pratt Institute MovieFone: User Test Report M a y 2 , 2 0 1 3 Table of Contents 1. Executive summary............................................................................................................. 2
a. Introduction............................................................................................................ 2
b. Evaluation Objectives............................................................................................ 2
c. Findings.................................................................................................................. 2
2. Introduction......................................................................................................................... 3
a. Overview................................................................................................................ 3
b. Target Population....................................................................................................4
c. Methodology.......................................................................................................... 4
3. Findings and Recommendations......................................................................................... 5
a. Summary of Findings..............................................................................................5
b. Walkthrough Screenshots........................................................................................5
c. Additional Findings.................................................................................................9
4. Conclusions....................................................................................................................... 10
5. References......................................................................................................................... 11
6. Appendix............................................................................................................................12
Pratt Institute I MovieFone User Test Report 1 Pratt Institute MovieFone: User Test Report 1 Executive Summary__________________________________________________ A. Introduction Moviefone provides users with movie news, movie theater information, showtimes for
upcoming movies, information about movies and DVDs that will be released in the near
future. Since its launch as a call-in service, and now through its many permutations
online, Moviefone has become an invaluable resource for people who are interested in
acquiring details about films.
Although Moviefone has most frequently been used to find current movie listings by
location and time, the website and mobile app continue to develop as a movie guide
database complete with user and critic reviews, information about the cast and crew,
interviews with celebrities, and video clips.
B. Evaluation Objectives The purpose of this report is to evaluate the usability of http://www.moviefone.com/.
Moviefone organizes and categorizes contemporary cinema for both the frequent and
infrequent movie-goer. Using a search box, a “Browse” drop-down menu, and a “Get
Showtimes” button, Moviefone’s website offers users with varying levels of computer
literacy the ability to find information through different tool. It is the goal of the authors
to highlight some of the ways in which Moviefone could become more user-friendly to
frequent browsers and to create a more welcoming site for users who are less familiar
with the website’s myriad of features. This report provides recommendations to increase
the Moviefone’s navigability and overall usability.
C. Findings This report presents implementations for creating a more accessible and user-friendly
database. It is the goal of the research team to strengthen Moviefone through actionable
suggestions and to therefore increase traffic to the website.
Eight research participants agreed to perform the same three tasks on Moviefone in order
for the research team to study the usability of the website. This report provides the
findings and recommendations of the team of evaluators.
M a y 2 , 2 0 1 3 The research team found three main problems:
1. Cast and crew information is difficult to locate.
2. The desired screening date changes when a search result is chosen.
3. Users had difficulty locating a movie’s genre.
Pratt Institute I MovieFone User Test Report 2 Pratt Institute MovieFone: User Test Report 2 Introduction___________________________________________________________ A. Overview Moviefone provides a list of current and upcoming showtimes for movies and
information about newly released DVDs. In addition, Moviefone offers users information
about celebrities including film credits and movie news.
This user test report offers findings and recommendations based on an evaluation of
Moviefone’s website. Four evaluators conducted this study through Pratt Institute’s
School of Information and Library Science in New York. The evaluation process focused
on three tasks, which took into account the scope of Moviefone. The ways in which each
of the eight users completed or failed to complete the three tasks, as well as their verbal
feedback as they used the site, provided insight into the users’ experiences with
Moviefone. The participants in the study had varied degrees of knowledge and
experience using Moviefone prior to participating in this research project. Some of their
initial responses to the website were:
“I can track movies that I want to see, and movies that are out, and I can
read about them and possibly find out fun details about the actors and
actresses and keep track of them all."
“The website is too congested and moving too fast.”
“I think it looks much better than it looked before. I like the current
design. I like that it’s got a dark grey background. I think it’s legible. It
really lets the movie posters shine.”
“It doesn't feel to me like I'd be certain how to navigate it without just kind
of fooling around first."
B. Target Population The target population is any web user who would like to see a movie. This pertains to
movie-goers who want to see movies at a theater as well as those who would like to know
what is available on DVD.
M a y 2 , 2 0 1 3 C. Methodology This user test was conducted by four students of LIS-644: Usability of Digital
Information at Pratt Institute in the Spring of 2013. The participants were half male and
half female. Four out of ten had used Moviefone.com previously and the majority of
users watch movies at least once a month. 80% were IMDB users.
When conducting the test the moderators followed a script, which is attached here as
Appendix A. Each participant filled out pre- and post-test questionnaires (see Appendix
C and D) as well as a consent form (Appendix B). The participants used the “thinkPratt Institute I MovieFone User Test Report 3 Pratt Institute MovieFone: User Test Report aloud” method and described their actions to the moderator while Silverback recorded
their screen, clicks, audio, and facial expressions. Data was taken from ten user tests to
compile this report.
M a y 2 , 2 0 1 3 Please note that think-aloud testing is not faultless. A study by Norgaard and Hornbaek
(2006) found that “evaluators end up focusing too much on already known problems, and
that the questions they ask during a test seem to concern problems that the user expects,
rather than problems actually experienced during the test” (p. 217). That being said, the
site problems outlined in this report were actually ones not foreseen by the moderators.
Also, lab condition testing with the think-aloud method results in identification of twice
as many problems as remote testing methods (Bruun, Gull, Hofmeister, & Stage, 2009).
Pratt Institute I MovieFone User Test Report 4 Pratt Institute MovieFone: User Test Report 3 Findings and Recommendations___________________________________ A. Summary of Findings
All of the test participants found the Moviefone website dynamic to use and a few
commented specifically on the eye-catching layout and the interactive nature of the
pages. A couple of participants were even interested in using the site in the future to
search for movies in theaters. However, improvements could be made to the cast and
crew information, date selection in search results, and finding movies by genre that
would increase user satisfaction with the interface.
B. Evaluation Screenshots
M a y 2 , 2 0 1 3 The following are a series of screenshots of the notable issues that pertain to the use of
the site as noted by the evaluators. A brief problem description below each screenshot is
given. Also included below each screenshot are the recommendations to improve or
correct the problem.
Pratt Institute I MovieFone User Test Report 5 Pratt Institute MovieFone: User Test Report B.1 – Cast and Crew Information is Difficult to Locate
M a y 2 , 2 0 1 3 Almost all of the participants experienced varying degrees of frustration when asked to
locate the names of producers involved in the making of 42. The sentiments expressed
ranged from self-doubt to confusion. The website failed to meet the participants’
expectations for where to find this information, especially in relation to their experiences
with sites similar to Moviefone. One user didn’t “understand why the producers would
not be listed in the ‘Read More’ section along with film director.” Supporting that user’s
argument, another found it “surprising, seeing that the producers are essentially
responsible for the film’s existence.” The ‘Read More’ section, where most users clicked
first in an attempt to find producers, contained some helpful cast and crew information,
but no clear, direct way to retrieve further information.
Recommendation:
Include ‘Producer’ heading within a film’s ‘Plot and Details’ listings. Heading should be
situated in between ‘Director’ and ‘Starring’. In addition, include a link to further Cast
and Crew information. These additions would allow the users to easily navigate any
possible personnel related information they would like to access from the ‘Read More’
link.
Pratt Institute I MovieFone User Test Report 6 Pratt Institute MovieFone: User Test Report M a y 2 , 2 0 1 3 B.2 – Date Changes When a Search Result is Chosen
All of the users were able to locate and utilize the ‘Change Date’ option in the red menu
bar within the search results page. However, some users became frustrated when they
chose specific theatres or movies from the results page and the date defaulted back to the
current day. One participant said, “Showing today? I had a different date. Do you
remember I had a different date?” Another user did not even notice that the date changed
when they chose a film they wanted to attend, which in real life could have resulted in
them arriving to the theater at an incorrect time.
Recommendation:
Direct the site to “remember” a user’s date change until they select another. Never default
back to ‘today’s date’ on any page unless the user specifically makes that request. This
will remove the need for the user to re-select their desired attendance date on every page
and minimize confusion when planning in advance.
Pratt Institute I MovieFone User Test Report 7 Pratt Institute MovieFone: User Test Report B. 3 – User’s Cannot Locate Movie’s Genre
M a y 2 , 2 0 1 3 Navigating to movie showtimes via the search bar or “Get Showtimes” tab was easy for
all of the participants. But, once they found a theater they wanted to go to, users were
unable to easily find the genre of the movies playing. They often tried to find the genre
by reading through the synopsis or watching the trailer. A few users located the genre
hyperlink after trying multiple other methods to discern the genre. After locating it, one
participant said, “It was there the whole the time, I just never saw it!”
Recommendations:
- Move the genre hyperlink closer to the film’s synopsis. This is where most participants
immediately went to in an attempt to find the genre, and is most likely where they would
easily notice it.
- Another solution is to move the Facebook ‘Like’ and ‘Send’ buttons to another location
on the page so that the movie title/year/rating and genre lines are closer to the rest of the
film information. Users are more likely to notice the genre if it is closer to the film’s
description.
Pratt Institute I MovieFone User Test Report 8 Pratt Institute MovieFone: User Test Report C.
Additional Findings
A seven-question exit questionnaire submitted by each participant provided the study’s
team with additional insight into user perception and overall impressions and
experiences. With the exception of Question 7, all determinant values were reached by
calculating both real and estimated averages of individual questions, which were used to
determine overall user satisfaction.
The additional conclusions drawn from the exit questionnaire are as follows:
M a y 2 , 2 0 1 3 1. Overall, Moviefone.com offers a moderately satisfactory level of usability.
2. Five of eight study participants did not provide written responses to Question 7,
reinforcing moderate satisfaction with respect to site usability.
a. Whereas high and low levels of overall satisfaction tend to motivate users to
respond at length, moderate overall satisfaction by definition tends not to inspire
“good or bad” reaction or sentiment.
b. The three written responses, in varying levels, offer complimentary or otherwise
constructive commentary based on individual experience.
3. Users are more likely to use Moviefone.com again themselves than they are to
recommend the site to their friends.
Pratt Institute I MovieFone User Test Report 9 Pratt Institute MovieFone: User Test Report 4 Conclusions____________________________________________________________ This usability study found three main problems with Moviefone. First, the cast and crew
information is difficult to locate, particularly information about people involved with a
movie who are not the director(s) and the main actors. Second, the screening date defaults
to ‘today’ when a search result is chosen and, third, users tend to have difficulty locating
a movie’s genre.
M a y 2 , 2 0 1 3 Moviefone.com has come a long way since it was predominately used as a call-in movie
listing service. The company has branched out to include information about movie
theaters, celebrities, DVD releases, and movie news. The research team’s findings and
recommendations are presented in this report to support Moviefone’s brand and to
increase accessibility and usability of the website.
Pratt Institute I MovieFone User Test Report 10 Pratt Institute MovieFone: User Test Report 5 References______________________________________________________________ Bruun, A., Gull, P., Hofmeister, L., & Stage, J. (January 01, 2009). Let Your Users Do
the Testing: A Comparison of Three Remote Asynchronous Usability Testing
Methods. Chi Conference-, 3, 1619-1628.
M a y 2 , 2 0 1 3 Norgaard, M. & Hornaek, K. (2010). ”Working Together to Improve Usability: Exploring
Challenges and Successful Practices”, International Journal of Technology and
Human Interaction, 6(1), 33-53.
Pratt Institute I MovieFone User Test Report 11 Pratt Institute MovieFone: User Test Report Appendix A_-­‐ Moderator Script________________________________________________
User Test Moderator Script Spring 2013 Prepared by:
Laura Liang
Chris Mullin
Samantha Raddatz
Gili Warsett
This document is the moderator script for the Moviefone user test. It was prepared by Pratt SILS students Laura Liang, Chris Mullin, Samantha Raddatz and Gili Warsett under the direction of Pratt SILS Professor Craig MacDonald. Before the Session M a y 2 , 2 0 1 3 1.
2.
a.
b.
c.
d.
e.
3.
Arrive early and set up the testing environment:
Open Silverback on computer and create new project
Organize and arrange all necessary documents and materials:
Two pens or pencils and paper for the participant
Two copies of printed Consent Form (one for you, one for the participant)
Printed Moderator script
Paper and pen for your notes
Laptop with charger (if needed)
Open Google Chrome:
.
Clear the browser history and cache
a.
Open all necessary links in different tabs:
i. Moviefone website: http://www.moviefone.com/
ii. Pre-­‐Test Questionnaire: http://bit.ly/moviefonepre
1. Enter participant’s assigned ID number and leave tab open
iii. Post -­‐Test Questionnaire: http://bit.ly/moviefonepost
1. Enter participant’s assigned ID number and leave tab open
4. Greet your participant! Pratt Institute I MovieFone User Test Report 12 Pratt Institute MovieFone: User Test Report Appendix A_-­‐ Moderator Script________________________________________________ Introduction & Informed Consent Thank you for agreeing to take part in our study. My name is {Name} and this is {Second moderator’s name}. I am going to walk you through the session today. Before we begin, I have some information for you. I am going to read it to make sure that I cover everything. I’d like to remind you that you’re here today because we’re testing the MovieFone website. Here’s how the test will work: I’m going to give you a few tasks to complete while “thinking aloud,” meaning I would like you to think out loud while working, talking about your thoughts and opinions as you complete each task. “Thinking aloud” will help us to evaluate the results by knowing what your thought process was. As you go through the tasks, I’m going to ask about your current emotional state. You’ll also be asked to fill out two brief questionnaires: one prior to beginning the tasks and one upon completion of the tasks. Overall, the session should take about 30 minutes.
We are testing the website, not you. In no way will your information seeking methods, preferences or tendencies reflect negatively on you or the value of your contribution. There is no penalty for mistakes, frustrations expressed or even for decision(s) to break momentarily or to request to cease conducting your session. All that is asked of you as a participant is to respond with honest, descriptive feedback related to your experience.
[Give participant consent form] At this point, please consult this “User Consent Form,” which has been drafted specifically for this user test. In addition to including a more detailed review of the general points we’ve discussed, the user consent form outlines the tasks you will be asked to conduct. Please take as much time as you need to review the form; feel free to ask any questions. Once you feel satisfied, please sign your consent form with your full name along with today’s date. [Let them read the form] M a y 2 , 2 0 1 3 Before you sign the form, do you have any questions?
[Have participant sign the form. Give them a copy to keep.]
Thank you! Here’s a copy for you. Pre-­‐Test We’d like to learn a little bit more about you so we’d like you to take a few minutes to fill out a brief questionnaire. While you are free to answer as many or as few questions as Pratt Institute I MovieFone User Test Report 13 Pratt Institute MovieFone: User Test Report Appendix A_-­‐ Moderator Script________________________________________________ you desire under no penalty, we ask you to provide as many responses as possible, since your confidential data will assist test conductors with obtaining the optimal test data for thorough analysis. Your participant ID number has already been filled in. Please fill in the rest of the questions. [Click to Pre-­‐Test Questionnaire Tab] [Make sure the users clicks ‘Send Form’] Thank you!
As part of this session we’d like to record the screen and video of your reactions to the website. The recordings will only be used for research purposes and will not be made public. Do we have your permission to record both screen and video? [If Yes: Start Silverback recording. If No: Place post-­‐it note over the camera.] While you’re completing the following tasks we’d like to remind you to “think out loud” or verbally express your thoughts and opinions, both positive and negative, as you navigate the site. [Click to the Moviefone website tab] Please take a moment or two to look around this main page. Tell us about your initial impressions of the site. Feel free to scroll around the page, but please do not click any link or fill in any search fields. What are your first impressions of this website?
What do you think this website is about and used for?
M a y 2 , 2 0 1 3 [Give participant about a few minutes to give their impressions]
Thank you. Now I’m going to read you a series of tasks. I will read you the first task and then you will have time to go through the task before we move on to the second one. You don’t need to memorize the tasks. Feel free to ask me to repeat them at any time. As you go through the steps to complete these tasks please remember to talk me through what you’re doing and why. Task #1 You and your friends are planning to see a comedy that is screening this Friday around 8 pm at a theater close to zip code 10003. You’re in charge of logistics. Please find a movie, showtime, and theater. Pratt Institute I MovieFone User Test Report 14 Pratt Institute MovieFone: User Test Report Appendix A_-­‐ Moderator Script________________________________________________
Task #2 Please go back to the main page.
You haven’t decided if you want to stay in or go out tonight. Find a movie that is still in theatres AND is also out on DVD. You may select any movie that is currently listed on MovieFone’s website.
Task #3 From the main page, please find the producers for the new movie 42. Prior to the release of 42, what other movies were the producers involved with making?
Post-­‐Test OK, now we’d like to learn more about your experience using the Moviefone website so please take some time fill out this questionnaire. [Open post-­‐test questionnaire tab]
Your participant ID number has already been filled in. Once again all of the questions are optional. [Give participant a few minutes to fill it out] [Make sure they click the ‘Send Form’ button] Great! Do you have any other comments or questions about the website?
[Use this time for any open ended follow up questions regarding areas of the site they had particular troubles with or any areas you want to know more about]
M a y 2 , 2 0 1 3 Debrief We really appreciate your time and feedback. Please remember that all information you provided will remain anonymous and confidential and will not be shared with anyone outside the research team.
If you think of any other questions or comments, please feel free to contact me or Samantha Raddatz. Her information can be found on your copy of the consent form. Thank you again for your time! [Stop Silverback recording] Pratt Institute I MovieFone User Test Report 15 Pratt Institute MovieFone: User Test Report Appendix A_-­‐ Moderator Script________________________________________________
After The Session M a y 2 , 2 0 1 3 [Talk with your partner and make a list of all the problems you observed during the session. Problems will include things the participant said and things they did that were unexpected or incorrect.]
Pratt Institute I MovieFone User Test Report 16 Pratt Institute MovieFone: User Test Report Appendix B_-­‐ Consent Form____________________________________________________ MovieFone.com Usability Study – Consent Form Purpose: The purpose of this study is to evaluate the usability of MovieFone.com. The study is part of the Usability of Digital Information graduate level class, under the supervision of Professor Craig MacDonald. Procedure: If you agree to be in this study, you will be asked to do the following: 1. Find a movie, showtime, and theater. 2. Find a movie that is still in theatres AND is also out on DVD. 3. Find the producers for the new movie 42 and see what other movies they’ve been involved in. Benefits/Risks to Participant: The participant will learn about the potential uses of www.MovieFone.com and will help contribute to the body of knowledge in usability research. There is a possible risk of viewing unsettling images. Voluntary Nature of the Study/Confidentiality: You are voluntarily taking part in this study. At any point you are welcome to tell the moderator that you would like to discontinue your participation in the study. You may also ask the moderator any questions at any time during the study. Your name and personal information is completely confidential and will only be accessible to the members of the project team. Contacts and Questions: If you have any questions after the study is complete, you may contact Samantha Raddatz: SRaddatz@Pratt.edu 646-­‐734-­‐6882 Statement of Consent: M a y 2 , 2 0 1 3 I have read the above information. I have asked any questions I had regarding the experimental procedure and they have been answered to my satisfaction. I consent to participate in this study. Signature of Participant: Your name: Date: Age: (Note: You must be 18 years of age or older to participate in this study.) Pratt Institute I MovieFone User Test Report 17 Pratt Institute MovieFone: User Test Report Appendix C_-­‐ Pre-­‐Test Questionaire__________________________________________ MovieFone: Pre-­‐Test Questionaire
M a y 2 , 2 0 1 3 Please fill out this form to the best of your ability. There are no right or wrong answers to these questions. Feel free to leave any answers blank. Participant ID Number: ______ Gender: _____ Have you used Movie Fone.com before? __ Yes __ No If yes, When was the last time you used it? __ In the last week __ In the last month __ In the last six months __ In the last year __ In the last 5 years __ More than 5 years ago How often do you go to a theater to watch a movie? __ Once a week __ Once a month __ Once every few months __ Once a year __ Less than once a year How often do you watch movies on DVD or via subscription service? i.e. Netflix, Hulu Plus __ Once a week __ Once a month __ Once ever few months __ Once a year __ Less than once a year Not including MovieFone, check any of the following websites or apps you use to retrieve information about movies and showtimes: __ IMBD __ Fandango __ Rotten Tomatoes __ Option 4 __ Other: _____________________________ Pratt Institute I MovieFone User Test Report 18 Pratt Institute MovieFone: User Test Report Appendix D_-­‐ Post-­‐Test Questionaire_________________________________________ MovieFone: Post-­‐Test Questionaire
M a y 2 , 2 0 1 3 Please fill out this form to the best of your ability. There are no right or wrong answers to these questions. Feel free to leave any answers blank. Participant ID Number: ______ Overall, using MovieFone.com was: Difficult to use 1 2 3 4 5 Easy to use How interesting did you find the content to be? Very uninteresting 1 2 3 4 5 Very interesting What was your impression of the appearance of MovieFone.com? It’s Ugly 1 2 3 4 5 It’s beautiful Locating the buttons and links you wanted was: Difficult 1 2 3 4 5 Easy How likely are you to use this website again? Very unlikely 1 2 3 4 5 Very likely If you are unlikely to use this website again, please tell us what MovieFone could offer that would make you more likely to use it. _____________________________________________________________________________________________________
_____________________________________________________________________________________________________
_____________________________________________________________________________________________________
_____________________________________________________________________________________________________ How likely is it that you would recommend MovieFone.com to a friend? Very unlikely 1 2 3 4 5 Very likely Pratt Institute I MovieFone User Test Report 19 Pratt Institute MovieFone: User Test Report Appendix E_-­‐ Table of Problems Found During User Test_________________ M a y 2 , 2 0 1 3 Problem Description The location results weren't obvious to the participants as being results possibly because
the results were in hot blue.
Participants went to "Read More" for cast and crew but couldn't find producers, only
found actors and directors
Participant had difficulty finding change the date tab. If participant clicks off of theater
information it changes back to today's date
None of them could find genre either by browse function or they didn't see the genre
above movie thumbnail once they clicked on the movie
No breadcrumbs
Can't type in a number even if it's title of the movie without selecting from the
dropdown or it doesn't understand the search
Browse arrow for pages is too small
Location problems: The site does not take into account that if you’re looking for a
movie in Manhattan, there is little chance that you care if the closest theater where it’s
playing is in Ontario or Minnesota.
Preferred IMDB for cast and crew
The participants did not utilize the “browse” dropdown at all.
Participants highly disliked infinite scroll.
Pratt Institute I MovieFone User Test Report 20 Pratt Institute MovieFone: User Test Report M a y 2 , 2 0 1 3 Appendix F_-­‐ Responses to Pre-­‐Test Questionnaire___________________________ Pratt Institute I MovieFone User Test Report 21 Pratt Institute MovieFone: User Test Report M a y 2 , 2 0 1 3 Appendix G_-­‐ Responses to Post-­‐Test Questionnaire__________________________ Pratt Institute I MovieFone User Test Report 22