of Human0Computer"Interaction

A"Glimpse"of
Human0Computer"Interaction
hci.ucsd.edu/hollan
A"Story"About"Activity0Enriched"Computing
Jim Hollan
Email: hollan@ucsd.edu
Web: hci.ucsd.edu/hollan
IEEE Computer, October, 2012
hci.ucsd.edu
Computers Are Special
Computers are special in that they provide a new kind of stuff
out of which to fashion dynamic interactive systems to assist
thought, communication, collaboration, and social interaction
“The computer is the first meta-medium, and as such it has
degrees of freedom for representation and expression never
before encountered and as yet barely investigated.”
-- Alan Kay
Computers Are
Special
Computation provides the most plastic medium for representation,
interaction, and communication we have ever known
Mimic existing media (e.g., books, newspapers, magazines, photographs, audio
recordings, and films)
Create new media and modify the form of existing media,
Create models that represent, with ever increasing fidelity, the physical world
Provide virtual worlds that range from the simple metaphorical desktop of the
graphical user interface to the amazing digital effects and virtual characters of
current games and films
Combine the real and the virtual, as with computer-augmented surgery in which
images of internal structure are projected onto a patient's body to guide surgery
and robotic-assisted controls remove the tremors from the surgeon's hands
Moving beyond the desktop computer
Increasingly we have multiple and we don’t think of many
of them as computers
Connected to computers, sensors, and people all over the
world
Web is changing our professional, personal, and social lives
Beyond the desktop and
onto the desk
ObjecTop: Occlusion Awareness of
Physical Objects on Interactive Tabletops
Mohammadreza Khalilbeigi (Darmstadt),
Jürgen Steimle (MIT), and Jim Hollan (UCSD)
Boundary between physical and
digital worlds is becoming
permeable
Also computer is morphing form
Changing form of
computers
Monolithic computer is
coming apart and being
reassembled in myriad
new forms
New device ecologies and
ways of interacting
For good and for ill, our
activities are increasingly
mediated by computers
As Computers Morph into New Forms
There is also a Data Revolution
•
•
Inexpensive digital recording devices,
sensors, and storage facilities are
revolutionizing data collection in the
behavioral sciences
Many disciplines are taking advantage
of inexpensive digital video to assemble
extensive data collections of human activity
captured in real-world settings
Fun example: capturing
behavior of people playing
Spore (the outtakes)
Data Collection Revolution
•
Inexpensive digital recording devices,
sensors, and storage facilities are
revolutionizing data collection in the
behavioral sciences
•
Extending data collection into
situations that have not typically been
accessible
•
Enabling examination of the fine detail
of action captured in meaningful
settings
This makes real world activity an object of
scientific scrutiny in ways never before possible
and at a scale that until recently was unimaginable
boltpeters.com,,
My personal story of capturing activity
history starts in an auto repair shop in
Austin Texas
Activity Histories
Wear on menus,
buffers, email, ...
Instrumenting activities with sensors
Edit Wear and
Read Wear
Hill, Hollan, Wroblewski, and
McCandless
History-Enriched
Digital Objects
Hill and Hollan
Attribute-Mapped
Scrollbars
Intelligent Driver Support System
10 Cameras
Video
Streams
A Critical Challenge as well as an
Unprecedented Opportunity
ChronoViz:
A tool for supporting navigation of time-coded data
But more data cannot be the whole answer, since most
researchers feel that they are drowning in data
A critical challenge is how to fully capitalize on the
opportunity provided by the revolution in activity data
capture
Multiple sources
of video
Sensor/Sim Data
Geographic/Spatial
Data
Annotations
Transcript Data
Digital Notes
Activity Trails: Episodic Access to
Digital Activity
Activity-Enriched
Computing
Little of the history of computer-mediated activity is
currently captured or made available
Computing devices only record the consequences of our
actions
Mobile devices record even less, but increasingly have
access to even more of our day-to-day activity
Activity Trails: Episodic Access to
Digital Activity
Year-long field study in a law office by Gaston Cangiano:
observation and analysis of activity, auto-confrontation
with videos of screen activity, role of episodic memory in
work practices
<frame'time="23:37:13">'
<keyBuffer'time="5222055"'words="get'current">etSHTcurrentRETESC</keyBuffer>'
<desktop'numWins="6">'
<window'exeJile="dexplore.exe"'minimized="yes"'module="wndclass_desked_gsk"'
rect="O32000,O31840,O32000,O31975"'title="set'Members'O'MSDN'Library'O'Visual'Studio'2005'O'
Microsoft'Document'Explorer"></window>'
A Conjecture:
Aiding Context Reinstatement
May require less than one imagines
Needs to be interpretable only by you
Exploit visual memory
Brady, Konkle, Alvarez, and Oliva,
Visual long-term memory has a massive
storage capacity for object details. Proceedings
of the National Academy of Sciences, 105,
14325–14329.
Example#Test#Pairs#(##correct)#
Ss#presented#with#2,500##
pictures#for#3#seconds##each.#
Over#5.5#hours.#
#
Repeat;Detection#Task#during#
presentation#
#
2;alternative#forced;choice#
#
Three#conditions#
Example#Test#Pairs#(##correct)#
Ss#presented#with#2,500##
pictures#for#3#seconds##each.#
Over#5.5#hours.#
#
Repeat;Detection#Task#during#
presentation#
#
2;alternative#forced;choice#
#
Three#conditions#
#
! 
! 
! 
Novel#
Exemplar#
State#
Performance:##
Novel#(92.5%,#1.6%),#
Exemplar#(87.6%,#1.8%),##
State#(87.2%,1.8%)#
#
! 
! 
! 
Novel#
Exemplar#
State#
Performance:##
Novel#(92.5%,#1.6%),#
Exemplar#(87.6%,#1.8%),##
State#(87.2%,1.8%)#
Summary
Computers are special
Moving beyond the desktop: onto the desk and mobile
Increasingly permeable boundary between physical and digital
Data revolution: capture real-world activity for scientific scrutiny
Activity Histories: read/wear edit/wear; real-world driving
ChronoViz: aid annotation, visualization, and analysis
Activity-Enriched Computing: can we help restoring context of interrupted
activities by using visual memory