Proactive, Context-Aware Assistants

Proactive, Context-Aware Assistants
Sunil Vemuri
Co-Founder & CPO, reQall Inc.
@sunilvemuri vemuri@reQall.com
Traffic is increasing. You should leave for your 8am mee8ng with Don Norman in San Francisco.
Directions
Don asked, “Would you like to get breakfast?”
Yes
No
OK, I replied to Don and men8oned a few restaurants you might like.
Search-less search
… most people don't want Google to answer their
questions. They want Google to tell them what they
should be doing next.
Eric Schmidt, Google
Vision
Increasing human capabilities
through an automated personal assistant
that tells you what you need to know and do,
and helps you get it done
Sep. 8. 2011
Confidential
4
http://www.youtube.com/watch?v=pzWofXvIcW0
Near Term
6 actions detected from 80 emails at 2:30 pm Wednesday
Under the Hood
Example: Business Meeting
meet <who> <date> <location>
lunch <who> <date> <location>
•
Before
Negotiate time/place with other
attendees
Meal/Drink? Recommend someplace
suitable
Reserve the location
Add to calendar
Set agenda
Review/complete pending action
items before meeting (e.g., NDA)
Prepare presentation?
Biographies of attendees
Latest activity of attendees
Research
Refresh memory about previous
meetings/interactions with attendees
Strategize among those in your party
Check traffic & Commute to location
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
During
Silence phone
Note taking
Minimize interruptions
Keep to agenda
Keep to time
Distribute Action items
After
Un-silence phone
Debrief among those in your party
Distribute meeting minutes
Send follow-up emails
Complete/Delegate Action Items
Enter business cards into Contacts
Accomplish any non-meeting-related
ToDos while in the neighborhood
Commute to next location
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Example: Attending a Conference
•
Before
Travel
Air
Hotel
Ground Transport
Weather
Packing
Clothes
Medicines
Phone(s)
Computer(s)
Camera
Wallet
Business Cards
Pre-correspondence
Setup meetings
Visit friends/family
Speaker?
Prepare/rehearse
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
During
Registration
Scheduling
Tweet about it
Networking
Note taking
Remembering Names/
relationships
Eating
Exercise
Tipping customs
After
Summarize
Post-correspondence / follow-up
Return Travel
Gifts
Trip report
Expense report
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Detect Goals: What consumers want to do
•
•
•
•
•
•
•
•
•
•
Buy more snack bars for office at
Target
Rent Everybody's fine from Netflix
Buy The Increment by David
Ignatius from Amazon.com
Call Sandy about changing the car
insurance
Haircut tomorrow at 1:15 pm
Add Mad Men to Tivo by August
14
July 28th Susan’s Birthday
Extra strength Tylenol
Backup home computer
July 23rd 11 am, Ann arrives, meet
her at Dulles
•
•
Buy Cheerios
•
•
Change AT&T wireless plan
•
•
•
•
•
•
•
Fix bath tub diverter
Book flight to NYC, make sure no
blackout dates
Get feedback from Tom at 4 pm
on Thu
Put bible in car
Pay Bank of America bill
Tires for BMW
Transfer funds to 529 account
Buy tent
Propose trip to Hawaii Oct 11
through Oct 14
Detect Goals: What consumers want to do
•
•
•
•
•
•
•
•
•
•
Buy more snack bars for office at
Target
Rent Everybody's fine from Netflix
Buy The Increment by David
Ignatius from Amazon.com
Call Sandy about changing the
car insurance
Haircut tomorrow at 1:15 pm
Add Mad Men to Tivo by August
14
July 28th Susan’s Birthday
Extra strength Tylenol
Backup home computer
July 23rd 11 am, Ann arrives, meet
her at Dulles
•
•
Buy Cheerios
•
•
Change AT&T wireless plan
•
•
•
•
•
•
•
Fix bath tub diverter
Book flight to NYC, make sure no
blackout dates
Get feedback from Tom at 4 pm
on Thu
Put bible in car
Pay Bank of America bill
Tires for BMW
Transfer funds to 529 account
Buy tent
Propose trip to Hawaii Oct 11
through Oct 14
Context
Spoken Query
Location
Calendar
Soc. Net
Contacts
Data Mining
User Plan
Email
ToDos
Activities
User Profile
Speech Reco.
NLP
Assistants
Recommender
Searchless
Search Engine
Here and Now (Push)
Intent Engine
Personalized News
Local Deals
Meeting Prep
Person Bios
Social Trends
Email Assist
Commute
Restaurant
Nearby ToDos
etc. (3rd party experts)
Voice Answers (Pull)
Frequency
Detect Goals: The long tail
Email Appt.
Buy Call Bday
TiVo Bank Xfer
Fix Book Flight
Topic
Ski conditions
Backup PC
Change phone plan
Future
Future: More Assistants
Future: new context sensors
FUTURIST GAZETTE
News before you know you need it
ASSISTANT SPENDS $10,000 WHILE
OWNER ASLEEP
FUTURIST GAZETTE
News before you know you need it
HI-TECH THIEVES USE YOUR
ASSISTANT TO ROB YOU
FUTURIST GAZETTE
News before you know you need it
ASSISTANT DISCOVERS HUSBAND
CHEATING. TELLS WIFE
FUTURIST GAZETTE
News before you know you need it
HIGH COURT RULES NO
OWNER-ASSISTANT PRIVILEGE
Spoken Interfaces
I still think a voice driven interface has problems.
The fact you can say something instead of write it is
an advantage particularly for a mobile device say if
you’re doing a query. But it brings us back to the
command line interface. You have to compose your
commands instead of just seeing something that you
recognize like an icon without being prompted.
I’m not very optimistic that natural language
recognition will be truly good enough. We have the
image of Star Trek where you talk to the computer
and it seems like a nice interface, but notice that the
main reason for the Star Trek interface is for the
audience watching the show.
Jakob Nielsen, Dec 2011
Future: Personality
Asimov’s 3 Laws of Robotics
1. A robot may not injure a human being or,
through inaction, allow a human being to
come to harm.
2. A robot must obey the orders given to it by
human beings, except where such orders
would conflict with the First Law.
3. A robot must protect its own existence as
long as such protection does not conflict with
the First or Second Laws.
reQall Rover Code: vanguard
Search the Android Marketplace for “reQall Rover” to
download app
32
Traffic is increasing. You should leave for your 8am mee8ng with Don Norman in San Francisco.
Directions
Don asked, “Would you like to get breakfast?”
Yes
No
OK, I replied to Don and men8oned a few restaurants you might like.