iFlix final report - College of Letters and Science, UC Berkeley

iFlix
is246 Multimedia Metadata Final Project
Milestone 5: Complete Description of our iFlix Design
Cecilia Kim, Nick Reid, Rebecca Shapley
Table of Contents
IFLIX
1
Table of Contents
2
Introduction to iFlix
What is iFlix?
Relationship to the Apple iLife application suite
The Keyframe Summary: content metadata
Sorting and Browsing: object metadata
A static surrogate to start a revolution
3
3
3
3
4
4
The iFlix audience
5
Our Motivations for iFlix
5
Relationship of iFlix to Previous & Current Work on Multimedia Information
6
About This Project
7
User Scenario
Organize: The Catalog View
Organize: Import metadata for a new flick
Organize: Rendevous with Local Macs
Play: Viewing a Flick
Organize: Watching for New Flicks
Summary: Editing a Flick Summary
Organize: Share Flix Summaries and FlixLists
8
8
8
8
8
9
9
9
Screen Descriptions
Organize – Import, Organize and Share Flix listings
Summary – View and Edit a Flick’s Keyframe Summary
Play – Watch a Flick
10
10
14
18
System Diagrams
Information Flows including iFlix
19
19
Interviews with Users
Methods
Alpha Screenshots
Conclusions
21
21
22
23
References
User Preference for Static Summaries
Keyframe Selection for Video Summaries
Navigating video using keyframes
Navigating among keyframes
Music-recognition services
On the audiences
24
24
24
25
25
25
25
Introduction to iFlix
What is iFlix?
iFlix is designed to be the next member of the Apple iLife suite of applications. iFlix is a
media player and cataloging application, but perhaps most interestingly, is the first
application to provide a way for consumers to get involved in media content annotation
with almost no learning curve. iFlix’s Keyframe Summary is a simple, flexible concept
for annotating media content that is scalable to the user’s interest in annotation while
being easy to create, organize, and share.
iFlix is a tool for organizing flicks, from Hollywood movies to homemade flash movies
shared over the Internet. Flicks are time based multimedia clip products of any common
file type, including flash files (.swf), quicktime files (.mov), real player streaming pointer
files (.ram), or "bookmark" pointer files (.url), located either on the user’s local hard
drive or as a streaming movie on a remote server.
Relationship to the Apple iLife application suite
The iLife applications are designed to bring everyday computer users into the world of
digital media consumption and production. The users of these applications are motivated
by their desire to organize their personal collections of media and to share the results
within their social circles. Apple has intentionally set the learning curve quite low in
these applications, at the expense of features that professional media producers might
consider unacceptable. Additionally, there is little room in these consumer –focused
applications for creating and editing highly-structured, computer-parsable MPEG-7
metadata about media content. However, we feel that iFlix will both serve a consumer’s
growing need to organize their video media collections and introduce the concept of
annotating media content into the general computer using community while
simultaneously providing the tool for doing so easily.
Audio
Images
Video
Organize
iTunes
iPhoto
iFlix
Produce
GarageBand
(camera)
iMovie
Throughout the interface, our design choices leverage users’ experience with many of the
interactivity concepts developed in the iLife suite applications : iTunes, iPhoto, iMovie,
and even Safari. Novel to iFlix is the simple but powerful concept of the Keyframe
Summary, which is designed to be exactly what the user wants it to be.
The Keyframe Summary: content metadata
There’s a reason why the iFlix box - an application to organize video - is the last to be
filled in. To organize a collection will involve displaying lists of the colllection’s content.
While commercial music is described with a standard metadata of title, artist and album,
and images can be represented with a thumbnail, there isn’t an obvious answer to what
best represents a video. The variety of length and variation in flicks is astounding, and
many intelligent people have done a lot of brilliant work to figure out how to summarize
a video in a catalog listing.
A Keyframe Summary consists of a selection of still images and associated text in a
specific order. iFlix automatically generates three-image shot-optimized Keyframe
Summaries from the video (a.k.a. “flick”), accompanying the images with text created
by OCR of text in the flick and by speech recognition. By adding images and editing the
text, users can edit these Keyframe Summaries to their heart’s delight, collecting multiple
versions of summaries for a given flick. Keyframe Summaries can be exchanged by email, distributed via RSS feed, shared on a webpage, and downloaded from authoritative
sources.
Sorting and Browsing: object metadata
iFlix aims to identify the key simple object metadata users want to sort and search flick
objects by. This object metadata is either automatically generated by content analysis,
contained in id3 tags with the file, entered by users locally or retrieved from a shared
Gracenote-style online database. Examples of object metadata include filesize, date, title,
author, distributor, names of actors, characters, and other participants in the flick’s
production, “my rating,” and play count.
While designing, we quickly discovered that there is no definite or default way of
categorizing flicks. Unlike music there is generally not one definite named creation agent,
or the creator’s name is unknown. Outside the world of Hollywood content, even the title
of a flick maybe ambiguous. Unlike personal pictures, the use of time to catalog flicks
isn’t powerful either because the knowledge doesn’t exist or the creation date is only
mildly relevant to the media’s meaning.
In our design, we’ve privileged Names and Roles as a way of browsing an iFlix
collection. Names include both those of real people and those of characters that appear in
the flick. Because this is such a huge list with a variety of implications, it is refined with a
Role keyword. Selecting the Name Spike Lee will pull up the Roles Actor and Director,
and all the flicks he has been involved in. Selecting a particular Role will narrow the list
of flicks to browse.
We feel this aspect of the design is still open to more refinement. The iLife applications
are good at narrowing object metadata down to the specific essentials that users want and
will create or maintain.
A static surrogate to start a revolution
We made a specific design choice that iFlix would create static summaries of the timebased cataloged visual media, feeling from our own experience and user interviews that
static surrogates can be reviewed at the user’s own pace. Research suggests that although
people can interpret various time-based summaries, if they are not generated masterfully
(like movie trailers) people prefer static summaries. The Keyframe Summary is scalable
to the amount of effort the user wishes to invest (from nil to highly-customized) and can
be put to various uses, including cataloging, navigation, summarizing a video,
bookmarking interesting moments, and more. We even specifically designed an
interface that doesn’t require the keyframes to stay in the same order as the original flick,
allowing maximal storytelling flexibility such as that often leveraged by Hollywood
movie trailers! As users find value and satisfaction in interacting with content
annotations, we trust that future applications will be able to leverage a combination of
human and automatic information to eventually bring the computer-parsable media
revolution to the average computer user.
The iFlix audience
iFlix is designed to be usable by people who enjoy flicks. There is a very broad range of
media consumers, but our ideal users for building a promising future for computable
media are those who find sharing flick information to be satisfying expressions of social
connectivity.
The application supports a range of interaction with the user's collection of flicks, from
simply cataloging, grouping, browsing, and searching among a collection, to adding
value to the metadata about a particular flick, to sharing lists of flicks, metadata about
flicks, and the location of flicks with friends. We expect iFlix to be used by:
"The Organizer" – people who are active browsers of media content on the Internet or
are in media production and have a need to organize the various video clips and
streaming file pointers on their computer
“The Fan” – the small but highly-dedicated, highly-organized group of fans of various
media series, such as Star Wars or Star Trek fans, who currently keep an amazingly
extensive mental catalog of specific interesting moments in various episodes, and would
be able to externalize and organize this catalog.
"The Socializer" – perhaps college undergraduates (ages 18 to 25) - people for whom
collecting, commenting on and sharing video experiences is a social activity.
Additional audiences for iFlix include sports fans sharing or selling annotations of key
moments in sports events footage; video ethnography production teams collectively
annotating reams of footage; and interest groups seeking feedback from their member
networks on proposed commercials or documentaries.
Our Motivations for iFlix
Motivation behind iFlix is simply addressing the real problems that we have been facing
with iLife applications. We are all Mac users that have been impressed with powerful yet
simple designs of the various iLife applications, and we believe they are excellent entrylevel applications. There is an “Organizer” and a “Socializer” on our design team, so
we’re designing for our own challenges with finding and keeping track of the various
versions of media that we have produced and consumed. Management and sharing of
metadata hasn’t even been a possibility, until iFlix.
As iTunes has been the solution for music files, we’d like iFlix to be the solution for
video files. iFlix' features are inspired by 1) the visual and sharing features of iPhoto, 2)
the cataloging, metadata, and sharing features of iTunes, and 3) the function of media
player and the concept of identifying pointers and keyframes within a flick for which
iMovie pioneered a simple interface.
How should users organize their media? Their way. For any time-based media cataloging
solution, some sort of summary representation of the media is required for browsing and
evaluating the contents of lists, or the results of searches. Much successful attention has
focused on automated techniques for generating these representations for use in retrieving
video from large collections, but many challenges remain. One of those challenges is
bringing media interactivity to regular people, where the organization is informed by
their own interests instead of an algorithm. We believe users are interested in non-timebased summaries that can be browsed and don’t need to be played. We are interested in
providing an interface that facilitates human users’ contributions to optimize the
summary representation of a flick towards their particular communication goal. Not only
does iFlix’ combination of automatic and human-moderated summarization serve a broad
user audience, it also explores new territory in terms of encoding semantics into video
summaries.
Relationship of iFlix to Previous & Current Work on Multimedia
Information
We take our inspiration from the envelope-pushing work being done within the
Informedia project, the Open-Video Digital Library project, and FXPal’s Manga and
Mbase projects. These projects have shown us an impressive range of what is possible
for interactivity with static summaries of multimedia content. iFlix wouldn’t exist without
our ability to say with confidence, for example, that the task of generating an keyframe
summary with text annotations automatically from video content in a way that is useful
for a browsing human is a solved problem. However, these projects all focus on
searching within video libraries fed by automatically captured content, best described as
the large, impersonal, automated information retrieval system. In this information
retrieval context, automatic summarizing systems must seek the one summary to
represent a video.
With iFlix, we’re interested in the personal and social digital video library. Personal
digital libraries means people already know they care about the content. Shared personal
digital library index means someone you know cares about the content. This application
is NOT optimized for helping filter through long meeting videos for important moments,
for example….although if someone has, they might share their index via iFlix. iFlix is
about being able to make a summary, and another, and another…each one appropriate for
a given expression about the flick’s content.
The Open-Video Digital Library project began to bring human mediation into their
summary creation process, to improve the quality of the summaries. But this still remains
behind the scenes; a task that the video librarians do while cataloging a video. iFlix aims
to make video content annotation with keyframes a front-of-house operation: easy,
ubiquitous, and sharable.
Our few user interviews supported what Manga also discovered: people think about static
keyframe summaries as a way to navigate within content, much like the chapters in a
DVD. Manga also points to another approach to a static summary – the collage or
stained glass summary – where keyframes are arranged in two dimensions and in
different sizes on a single page. We considered this approach in iFlix during our design
brainstorming, envisioning an additional mode screen with a full-page customizable
Manga-type collage or poster for a video. It could be a creative outlet for generating a
collage from the still keyframe images, re-sizing them, even merging the boundaries
between them to accomplish some of the effects of a movie poster.
We decided to focus on the linear keyframe summary for a few reasons. First, we
recognized that other applications already provide the artistic environment well, and we
didn’t want to re-develop Adobe Photoshop, for example. The keyframe summary is
something new and very flexible for a variety of users, and we have also provided paths
to and from iPhoto so keyframe images can be used in art, and art can be brought in as a
keyframe. Second, we were concerned about the readability of complex poster frames in
a listing, and yet having the keyframe visuals in the catalog listing is an important part of
the purpose of iFlix. Finally, our aim for simplicity and low-learning curve in the iFlix
design requires that the method for annotating multimedia content be revolutionary in
power but “Doh!”-obvious in concept, and similar to things users already know or do.
Hence, we didn’t develop the single-page collage idea in this project.
Mbase obviously has developed a powerful navigation tool, and we’ve designed a
similar, simpler one appropriate for the iLife target audience.
The goal is to provide video users with an easy interface for marking which parts of the
video they specifically care about, rather than guess what metadata everyone will care
about. iFlix is designed to support media content annotation becoming an easy and
ubiquitous activity. Within iFlix, the value in annotation comes through being able to
navigate to moments in the video that you find important, and sharing those with others
in your social networks.
About This Project
The design of iFlix’s user interface and features is being conducted as a final project for a
graduate-level course in multimedia informatics.
Our process to date has included:
•
defining iFlix,
•
brainstorming designs,
•
composing the alpha versions of the Organize and Summary screens,
•
seeking feedback from our peers and through think-aloud interviews with
potential users,
•
revising to the beta/current versions of the Organize, Summary and Play screens
•
collecting more feedback from professional advisors to the multimedia course,
and
•
ensuring the current designs of screens and features are thoroughly communicated
as a foundation for future work.
There is obviously much more work that should be done to further develop the iFlix
external application design. There are many potential audiences for iFlix’ functionality,
but its development will benefit from focusing on serving a particular audience well.
Additional needs assessment work should be done to characterize user audiences and
select one. The proposed interaction for adding keyframes and comments to summaries
should be tested with users from the selected audience. Central features of iFlix such as
the Keyframe Summary editing tools and the sharing mechanisms should undergo
multiple additional iterations.
Thinking more broadly, our professional advisors envisioned that in the age of TiVo, a
keyframe summary annotation interface might become an interesting and integral part of
watching television: see a funny line in the Simpsons, grab the keyframe, send it to a
friend to start watching from that point; close-to-realtime annotations of sports
broadcasts; even a different way to collect Neilsen ratings or viewer impressions of show
episodes. A media-invested site like Yahoo! or MSN might also provide an online
version of iFlix –type functionality as a personalized interface for users to organize,
consume, and recommend media available on the site.
User Scenario
Organize: The Catalog View
Georgia just moved into a new group house at U. C. Berkeley. She’s met a few nice folks
here already, and Ryan has offered to share a flick with her. She plugs in his USB
Minidrive to her iBook, copies over the flick’s file, and opens iFlix. Dragging the file’s
icon over the Organize view, she adds Ryan’s flick to the rest of her listings of funny
short flicks, mostly attachments that she and her friends have sent around by e-mail.
Organize: Import metadata for a new flick
Georgia clicks Yes when prompted about importing the Summary from the FMDB - a
database of shared, user-entered metadata about flick files including authors, website
source, or whatever else is available. iFlix automatically generates a three keyframe
summary of Ryan’s flick, so Georgia can recognize it visually. She and Ryan watch the
flick…another funny one!
Organize: Rendevous with Local Macs
The group house also has a LAN, and she now notices a bunch of additional flicks are
listed in her iFlix catalog. These additional flicks are iFlix metadata entries from her new
housemates’ computers. Symbols next to the listings distinguish the origins and
accessibility of the flicks: pointers to streaming files and websites on the internet,
copyable files on a friend’s hard drive, and non-copyable files on a friend’s hard drive.
Georgia can review the summary of a flick, and if she likes it, she can click to copy the
pointer or copyable file into her iFlix library. She has a thing for penguins. She imports
the files and their metadata listings for a few copyable funny penguin flicks to her hard
drive. She sees another penguin flick is posted at Ebaum’s World (a flick content
aggregation site), so she imports the listing. She also sees that a housemate has a copy of
Amelie, (which features a gnome, not a penguin, but she likes it anyway) and she sends
them an e-mail, proposing a flick-watching night.
Play: Viewing a Flick
She selects the listing for the penguin flick from Ebaum’s World in the catalog, and
clicks the “Play” button. iFlix switches to the Play screen. iFlix natively plays any
QuickTime format, or provides screenspace for other media players on Georgia’s
computer to display a file. The penguin flick is streamed using RealPlayer, and iFlix
displays the RealPlayer player within the larger context of iFlix. Whatever controls are
available through the media player are also available within iFlix’ Play mode.
Organize: Watching for New Flicks
Georgia had heard of Ebaums’ World, but never gone to visit the website. Having seen
the penguin flick and lots of other listings from there among her housemates, she goes to
the website to check it out. They have a special “Penguin-Lovers” theme, and she can’t
resist…she wants to know about every penguin flick they have! She switches back to
iFlix, clicks on Import, and adds Ebaums’s world’s Penguin-Lovers webpage to her list
of flick sources to track. Now these flicks will appear as listings in her catalog just as her
housemates’ did, and she can import any new flicks to her personal library whenever they
appear.
Summary: Editing a Flick Summary
Fond of the new penguin flick she discovered , she wants to share it with her friends. To
help them evaluate whether it’s worth downloading the flick to their mobile phone or
over a dial-up connection, Georgia wants to send them the Summary with the weblink.
iFlix has automatically generated a three keyframe summary of the flick. Looking at the
Summary, she sees that one especially funny idea in the flick is missing. She thinks she
can add it in without giving away the punch line…
She selects the penguin flick’s listing in the catalog, clicks Summary, and iFlix changes
to the Keyframe Summary editing screen. She clicks the timeline in about the right place,
watches the flick and stops it near the right frame. She types in an intriguing caption for
the keyframe, and clicks Save. The new keyframe appears on the Summary timeline, and
she reviews it…yeah, it works. She shows it to Ryan, who suggests an additional
keyframe. They add that one, too.
Organize: Share Flix Summaries and FlixLists
With the flick selected, she clicks E-mail and sends the flick’s URL with the Summary
file out to her other penguin-loving friends. Ryan is impressed with her exhaustive list of
penguin flicks, and points out that the group house has a web server. She selects her
penguin FlixList and clicks Homepage, setting up a webpage for herself to share her list
of penguin flicks with the whole world…and her new housemates.
Screen Descriptions
Organize – Import, Organize and Share Flix listings
Screenshot
Overview
The Organize view, similar to the iTunes interface or iPhoto’s Organize view, provides
the user an overview of the contents of their flix collection. By clicking within the
hierarchical structure of the folders in the Sources column (left panel), the user can select
if the catalog reflects all flix available for playing including networked resources, or to
list only items on the user’s local hard drive. Flix can be searched for, browsed for,
organized into playlists, and imported from various sources.
iFlix provides extensive support for social sharing of video media within a networked
community, while respecting ownership models for digital property. Support features
include rapid search across the community network, and status icons that help the user
understand if a flick is local or remote, importable or streamed-only, or available for
purchase.
Feature and Functionality Descriptions
Import: Bringing data into iFlix
Importing Flicks
Flicks from the user’s hard drive can be added with drag and drop onto the catalog
window, or using the Import feature of the File menu. Flicks available at a webpage or
via a streaming media pointer file can be imported in the same fashion, using the
browser’s drag-n-drop bookmark icon, the resulting .url or .ram file, or by pasting the
URL into the menu-triggered Import dialogue box. Importable Flicks from network
resources may also be imported with a click on the status icon (see Status Icons).
Generating Listing Entry
After import, iFlix will automatically check the FMDB (FlixMetaDataBase) for existing
object metadata and Keyframe Summary files, or a combination of the program and the
user’s input can generate title, Keyframe Summary, and other object and content
metadata to populate the newly imported flick’s listing. Network resources may also
provide their listing data for a flick. We encourage the future developers of iFlix to use
appropriate parts of the MPEG-7 standard, such as the navigation/summary description
standards.
Source Management Panel
The upper-left panel , entitled “Source”, has specifically broken with the two-tiered
folder convention from iPhoto and iTunes, creating a third “Library” tier. Now with one
click, the catalog display can now encompass all local and LAN iFlix listing resources .
The ability to focus the catalog display on just the locally maintained iFlix library or on
any particular networked resource is maintained.
Extending the idea of the iTunes music store, we envision a variety of sources for
discovering and downloading or purchasing flicks can be added to this Source
Management Panel. By including them under an additional tier “Stores,” search efforts
may be able to act across vendors and locate the desired media.
Network Search and Catalog Sync
A Catalog Sync feature allows subscribing to networked resources’ iFlix listings to
facilitate rapid search for flicks across all available listings in the networked
environment. When a Catalog Sync is established with a frequent Rendevous partner, the
partner’s iFlix catalog (listed under “Shared” in the Source Management Panel) is
cached locally, and updated in the background. The Catalog Sync feature is accessed and
managed through menu commands.
Understanding and Customizing the Catalog
Mode Menu
With a flick selected, click Play to enter media player mode and see the video, or
Summary to view and edit the Keyframe Summary on the Summary editing screen.
Flix Status Icons
Status icons help the user navigate a variety of possible situations. The listing of a flick
available over the network could be a pointer file in a friend’s catalog, an actual file on
their hard drive. The flick can be a file you can have a copy of, or a file you are only
allowed to view while they are online.
A flick’s non-local status is indicated with the “streaming” icon from iTunes. Clicking on
it will play it as a stream from the LAN or Internet networksource. If the flick is
available to be copied to the user’s local hard drive, a > arrow icon will appear, and the
file can be imported by clicking and holding on the icon. When a networked flick is
added to a local playlist or its Keyframe Summary is edited, the metadata listing is
imported. If the user requests to play it when the network resource is unavailable, a ! icon
(also from iTunes) appears to indicate the absence of the original file.
Making FlixLists
Just like iTunes and iPhoto, custom collections of flicks can be created, named, added to
by drag-n-drop, and deleted. Flicks with particular themes, playlists for a flick-watching
party, or “funny one-liners” can be brought together in memorable FlixLists, which can
be exported and shared with friends. For flicks with multiple summaries, a particular
summary can be specified to represent the flick in a given list (not shown). Meanwhile all
flicks listings remain safely ever-present in the Personal Library.
Search and Browse to find Flicks
Browsing in iFlix is inspired by iTunes. Click the Browse eye to show/hide the browsing
panel. The browsing panel has two sections, Names/Roles, and Type.
Flicks may be full-length feature films, trailers, TV episodes, commercials, animation
shorts, flash movies, and any number of additional types. Whichever of these types exist
in the metadata of the library or libraries selected on the Source Management Panel will
appear on the Type browsing list. Select one of them to limit the list of flicks to only that
type.
Similarly, Names/Roles pulls from the metadata in the selected source library. This
metadata covers named people associated with flicks, and their role(s). For example,
Robert Redford might be listed under Names. Associated with his name are the Roles
“Actor” and “Director.” The names of the characters he played as an actor might also
appear in the Names list. Click any of these Names or Name/Role combinations to filter
the flicks shown in the catalog listing.
Search in iFlix is also inspired by iTunes and Mail.app, but with a small twist. Typing
any text into the search box will start iFlix looking in all text fields for matches. The pull
down-menu allows a user to limit to the search to particular text fields. And the twist is
that the searched field is displayed in grey text in front of the search text, providing the
user feedback that the limit has been set.
Customizing the Catalog Listing Display
By default, the catalog listing display shows certain columns, including displaying the
first three keyframes in the Summary column. Use the icon size slider (from iPhoto! And
using the Mac OS X Aqua rendering engine!) to show larger keyframes. Adjust column
widths, for example to show more keyframes from summaries. Sort the listings by one
click on the column header. Customize which columns appear in the catalog listings
using the View>Columns menu settings (not shown, but modeled after those inMail.app).
Finally, expose or hide the Browse panel by clicking the Browse eye.
Future designs should include an indicator of when the number of keyframes present in a
summary is greater than the number shown in the Flix column. The Organize view should
also provide the ability to select which summary to display in a given list.
Sharing FlixLists and Keyframe Summaries
Sharing Tools
With one or more flicks selected in the catalog listing, the user can share the selection in
various ways. Where relevant, these features implement the iTunes model of intellectual
property protection for flicks.
Play is an additional way of entering Play mode with a selected flick.
E-mail will compose an e-mail in your preferred e-mail application with the selected
metadata file(s) or FlixList as an attachment. Summaries might also be shared via IM
(not shown).
Homepage facilitates publication of FlixLists or download pages for metadata files to
webpages, including blogs. Like browsing someone’s bookshelf or movie collection in
the physical world, people share their movie titles on Friendster profiles, webpages and
blogs as a way of identifying their perspective on the world. Sharing your collection of
keyframe summaries can be a social activity just as sharing photos is.
Import is an additional way of activating the import of flick(s) to your hard drive, or their
metadata listing to your iFlix library, according to their status.
Burn supports the creation of video CDs or DVDs from a FlixList or selected flick.
Summary – View and Edit a Flick’s Keyframe Summary
Screenshot
Overview
Each flick has an automatically generated summary composed of three keyframes and
associated text and music. The Summary mode provides for viewing, navigating, and
enhancing Keyframe Summaries of a flick. Each flick can have multiple summaries, so
you can make your own and share them with friends!
To create the automatic summary, three keyframes are selected from evenly spaced
intervals in the flick, then optimized using the practices refined by the Informedia project.
The associated text is generated with speech recognition on any audio track,
supplemented with OCR of any video text. iFlix will also attempt to identify music
within the soundtrack, and provide a link to iTunes to purchase it.
Flicks have a variety of different aspect ratios. iFlix will not perform pan-n-scan,
letterboxing, or other adjustment to the flick’s aspect ratio. The Current Frame Screen
and the Context Screens will automatically scale to fit into the are provided by the
window size while adjusting for aspect ratios. Keyframes will adjust to fit the aspect ratio
into the same height of the Summary Display.
Add a Keyframe to a Summary
Custom summaries can be downloaded from friends and selected for viewing from the
Version bar. Make your own summary by adding to a copy of an existing summary.
Simply playback the flick to an appropriate frame, type in any relevant text, and click
“+”. The summary display shows a preview of where your keyframe will be added.
Rearrange the keyframes on the timeline to create the most compelling summary.
Feature and Functionality Descriptions
Mode Menu
Click Organize to see the catalog listings, or click Play to start the media player with the
current flick. Clicking on a source or FlixList in the Source Management Panel to the left
will exit Play and show the catalog in Organize view.
Frame Navigation Panel
Current Frame Screen
The current frame or the active playback of the current flick is the main feature of the
upper half of the Summary screen. While stopped to show a specific frame, the comment
text box is available for editing at the bottom of the image. Just below it is a line
displaying the name of the music track, or that a music track has been detected near this
frame.
Context Screens
To the right and left of the current frame screen, smaller displays show the frame
immediately preceding and following the current frame. This helps when deciding which
frame makes the perfect keyframe.
Use the arrow button below each context preview to move the flick one frame forward or
backward at a time. Click and hold on a button to bring up a menu for selecting a number
of frames to jump greater than one.
Scrub bar
The scrub bar presents a timeline view of the current flick. Light lines indicate shot
breaks detected automatically by iFlix, or imported as part of the flick’s metadata.
A point-down triangle indicates the location of the currently-displayed frame within the
flick’s timeline. Click and drag to relocate the playhead.
Point-up triangles indicate where existing keyframes have been drawn from in the flick.
Click one of these triangles to move the playhead to that point in the flick.
Playback Buttons
Control the playback of the current flick with the Fast-forward, Play, and Fast-reverse
buttons. Use the Fast-rate Controller in the Summary Tools panel below to adjust the rate
at which the Fast-forward and Fast-reverse buttons skim through the flick.
During playback, the comment text box and Context screens are inactivated.
Add a new Keyframe to a Summary
With the flick’s playback paused, click the + button to add it to the summary currently
displayed in the Summary Display. A gap between two keyframes in the summary
indicates where the new keyframe will appear.
Summary Display
Versions bar
Inspired by Safari’s Bookmarks Bar, the Summary Display’s versions bar shows the
names of the existing Keyframe Summaries for the current flick. By default, iFlix creates
the Auto three-keyframe summary, and places it first on the bar.
If the user changes the Auto summary or clicks the + button, a new summary is created.
E-mailed summary files can be added by drag-and-drop onto the catalog window or this
window. Menu actions (not shown) provide for the creation of automated summaries with
different defaults (more keyframes, or one per shot) and importing summaries from a
URL or file location.
Summary names highlight on rollover, and clicking the name will display that summary.
The name of the currently-displayed summary is shown as an editable text box (except
Auto). Summary names (except Auto) can be dragged into any order, or dragged off the
bar to delete them.
Whichever summary is selected when the user switches away from the Summary view
will be the one used to represent the flick in the catalog listing.
If the flick is streamed and the summary file has been imported, the Summary screen may
need to take a minute to contact the flick source and collect the keyframe images before
displaying the summary.
Future designs might tackle ways to display aggregate comments, addressing issues like
how to identify differences in comments or keyframe selection between highly-similar
summaries; ways to collate or view multiple summaries; and allowing users’ comments
on a keyframe to reflect a social commenting chain, much like a chat room dialog or
blog entry and comments.
Keyframes on Display
Each summary is displayed as a series of Keyframe images and the first 15 characters of
the comment text. A scroll bar will appear if all of the keyframes can’t fit on the screen.
If music has been identified in the soundtrack around the keyframe, a music note icon
will appear (see Check Music).
Keyframes can be re-ordered by drag-and-drop, just like on the iMovie clip ordering
timeline. Clicking the little X in the upper right corner will delete the keyframe, text, and
music information from the summary.
Navigating with a Summary
Clicking on any keyframe in the currently displayed summary will move the Frame
Location Panel’s Main Screen display to that keyframe’s location in the flick. Keyframe
Summaries can be used as a way to provide DVD-like chapters for media.
Keyframe location preview
Two keyframes will be spaced further apart to indicate where the currently displayed
frame on the Frame Navigation Panel will go in the Summary sequence if it is added as a
keyframe. By default, this will occur between the keyframe that most recently proceeds
the location of the playback head on the scrub bar, and the keyframe that follows it in the
sequence.
Summary Tools
Fast-Rate Control
Use this slider to adjust how fast “Fast” is for the Fast-Forward and Fast-Reverse controls
on the Frame Navigation Panel. The tortoise on the right is the slowest setting, while the
hare on the left is the fastest.
Check Music
Check Music identifies any commercial music in the flick’s soundtrack, adding this
information to the current keyframe. Grabbing a few seconds of the music from before
and after the current keyframe, Check Music uses an online music identification service
to gather the name and other metadata about the music. This metadata is added in a
special library in iTunes, and the keyframe will now display the track name and a music
note icon to indicate a link to iTunes. iFlix uses this same tool on keyframes
automatically selected as part of the Auto Keyframe Summary.
If iFlix does not detect any music within the soundtrack of the flick, this button will be
inactivated.
Get Text
Get Text uses optical character recognition (OCR) to identify text within the image.
Titles or credits present in the image can be entered into the Comments with one click.
iFlix uses this same tool on keyframes automatically selected as part of the Auto
Keyframe Summary.
Get Speech
Get Speech uses speech-to-text technology on any speech it recognizes within the
soundtrack, starting a few seconds before and ending a few seconds after the selected
keyframe. This length is modified by the density of surrounding keyframes and the
natural phrasing of the speech to optimize the coherence of the text results. iFlix uses this
same tool on keyframes automatically selected as part of the Auto Keyframe Summary.
If iFlix does not detect any speech within the soundtrack of the flick, this button will be
inactivated.
Slideshow
Because Keyframe Summaries may have more images than can comfortably be viewed at
once in the Summary viewer, the Slideshow functionality presents the images full-screen,
with the comment text at the bottom of the picture. Moving the mouse during the
Slideshow brings up an overlay menu with slideshow controls (like in iPhoto). Pausing
the slideshow allows the text for the current keyframe to be edited.
iPhoto import/export
This tool provides various functionality for importing and exporting photos between iFlix
and iPhoto. The images in an entire Keyframe Summary may be imported as a new
album in iPhoto. Conversely, a photo from iPhoto can be imported to replace the visual
image of an existing keyframe, allowing the creation of custom poster frames or section
titles.
Play – Watch a Flick
Screenshot
Overview
Any flick can be watched in Play mode. iFlix will either play the media for the formats it
directly supports, or present the media player for the given file format seamlessly for the
user, including whichever controls the media player supports.
Feature and Functionality Descriptions
Mode Menu
Click Organize to see the catalog listings, or click Summary to view and edit the
Keyframe Summary for the current flick. Clicking on a source or FlixList in the Source
Management Panel to the left will exit Play and show the catalog in Organize view.
Playing a flick in iFlix media player
The iFlix media player is a QuickTime player, presented through the iFlix application and
very similar to iMovie. A point-down triangle on the timeline bar indicates where the
current frames are in the flick’s playback (not shown). Click and drag the triangle to
change the current location in the flick.
Playback controls are located in the lower panel. Click the large right-pointing arrow to
play the flick. During playback , it will change to a square, and clicking it will stop
playback. Smaller buttons with double arrows will fast-forward or fast-rewind the flick.
Click the + button to add the current frame to the flick’s Keyframe Summary.
Activate full-screen playback by clicking the button with a soft rectangle around the play
arrow (not shown). Moving the mouse in full-screen mode will bring up an overlay menu
with the playback controls.
Future design work could optimize and streamline the ability to select and or share a
keyframe with others directly from Play mode.
Playing a flick in another media player
Flicks in file formats other than those supported by QuickTime can often still be played
in iFlix. When entering Play mode, iFlix will check the player associated with the file
type and either open the player window and associated playback controls within iFlix’
Play screen, or automatically start up the other player and switch to that application.
Otherwise, iFlix may prompt the user to identify an application on the local hard drive
that can play the selected file. Full-screen playback and keyframe addition will be
supported when possible.
System Diagrams
Information Flows including iFlix
iFlix is an application designed to help collect, organize, and pass on information about
time-based video assets within a community of users. Information flows through iFlix in
the following manner:
In more detail:
Interviews with Users
Methods
We showed alpha screenshots of iFlix to potential users, and asked them to describe what
they thought any given screen widget was for, and how they might use such an
application. The potential users are all undergraduate UC Berkeley students between the
ages of 19 and 24, residents of a 150-person community house with an active mediasharing community over the local LAN. They are representatives of “The Socializer”
audience for iFlix.
In a separate document, we present a rough transcript of their comments. Our own
questions or clarifications are in [ ]. Names have been changed to protect the innocent.
Alpha Screenshots
Organize
Summary
Conclusions
Much of the search, browse, network/share, mediaplayer, and keyframe functionality is
intuitive to potential users. They are able to translate effectively from their experience
with iTunes and movie editing software.
The static keyframe summary seemed to be a natural entity, and something many could
enjoy “consuming”, and a few might enjoy creating. Sharing them would be an essential
part of the point of making them.
The keyframe summary would be a nifty way of navigating through a flick.
Collections of highly-specific custom keyframe summaries could be a way for fans to
record their preferred combinations of TV series moments.
The music-recognition and purchasing feature is exciting to users, but not intuitively
presented yet in the alpha version of the Summary mode screen.
The Stain Glass is not named correctly, and although it seems cool, doesn’t seem to have
as much motivation to be created as the keyframe summary does. This could be an
artifact of the absence of having a screenshot for it for these interviews, however.
The iPhoto and Homepage features have potential; people have ideas about what they
might be, although those ideas aren’t as clearly formed as the keyframe summary.
These interviews were not a good test of the proposed interaction for actually editing
keyframe summaries. Future interviews should give users a sense of how OCR and
speech-to-text would help generate comments for keyframes.
References
User Preference for Static Summaries
Gary Marchionini’s work
http://ils.unc.edu/~march/
also http://www.open-video.org/
slide show from AVI '98
http://ils.unc.edu/~march/avi98/sld016.htm
Summary slide appears to indicate that users prefer static summary displays of video.
How Fast is Too Fast?
How Fast is Too Fast? Evaluating Fast Forward Surrogates for Digital Video
ACM/IEEE 4Joint Conference on Digital Libraries, May 29, 2003 (Talk for the
Vannevar Bush Best Paper Award)
http://ils.unc.edu/~march/jcdl2003/How_Fast_files/v3_document.htm
Slide 7 has a list of different types of surrogates for video. Slide 17 lists a summary of
results that suggests while people can understand video in extreme fast-forward, they
don’t like it. Also that users should have control of video summaries, but with appropriate
defaults.
Keyframe Selection for Video Summaries
Automated
fxpal work on Manga and MBase
http://www.fxpal.com/?p=manga
http://www.fxpal.com/?p=mbase
Time-Constrained Keyframe Selection Technique. Andreas Girgensohn, and John
Boreczky. In Multimedia Tools and Applications, 11(3), pp. 347-358, 2000.,August 1,
2000 http://www.fxpal.com/?p=abstract&abstractID=42
Keyframe-Based User Interfaces for Digital Video Andreas Girgensohn, John Boreczky,
and Lynn Wilcox. IEEE Computer, 34(9), pp. 61-67, September 1, 2001
http://www.fxpal.com/?p=abstract&abstractID=6
Video Manga: Generating Semantically Meaningful Video Summaries. Shingo
Uchihashi, Jonathan Foote, Andreas Girgensohn, and John Boreczky. In Proceedings
ACM Multimedia, (Orlando, FL) ACM Press, pp. 383-392, 1999., October 30, 1999
http://www.fxpal.com/?p=abstract&abstractID=136
Informedia
http://www.informedia.cs.cmu.edu/dli2/index.html
This project has also developed good principles for selecting appropriate phases among a
speech soundtrack to associate with a keyframe.
Virage
Automated plus human input
The Open Video Digital Library
Gary Marchionini and Gary Geisler. The Open Video Digital Library. D-Lib
Magazine. December 2002 V8:12
http://www.dlib.org/dlib/december02/marchionini/12marchionini.html
Figures 1 and 2 show interfaces similar to those we are proposing. The article also
describes creating keyframe summaries through a combination of automated and
manual actions, as well as efforts to capture what others think about videos and who
has used a video for what.
Navigating video using keyframes
fxpal work on Manga
http://www.fxpal.com/?p=manga
An Interactive Comic Book Presentation for Exploring Video. John Boreczky, Andreas
Girgensohn, Gene Golovchinsky, and Shingo Uchihashi In CHI 2000 Conference
Proceedings, ACM Press, pp. 185-192, 2000., April 1, 2000
http://www.fxpal.com/?p=abstract&abstractID=86
Navigating among keyframes
fxpal work on MBase
http://www.fxpal.com/?p=mbase
Music-recognition services
MusicBrainz, an open-source music recognition initiative - http://www.musicbrainz.org/
“Philips and Gracenote Launch Gracenote Mobile(SM) -- First Global Music Recognition
and Content Delivery Service for Mobile Phones” – Marketwire,
http://www.marketwire.com/mw/release_html_b1?release_id=61431
Shazam is a UK-based phone service provider that offers music recognition, facilitating
ringtone and CD purchases. http://www.shazam.com/uk/do/home
On the audiences
The “Fan” audience type is described in: Jenkins, H. Textual Poachers: Television
Fans & Participatory Culture. Routledge, New York, 1992;
The “Socializer” audience type is a media-centered variation on the mobile-phone giftgiving social culture described in: Taylor, A. S. & Harper, R. (2003). The gift of
the gab? A design oriented sociology of young people's use of mobiles.
Computer Supported Cooperative Work, 12, 267-296.