ASK NJIT LIBRARY - New Jersey Library Association

ASK NJIT LIBRARY: A Natural Language
Knowledge Base Self-Service Solution
Haymwantee P. Singh
Technical Reference Librarian
New Jersey Institute of Technology
singhh@njit.edu (973) 596-8498
http://library.njit.edu/
NJLA CUS 2009 Technology Innovation Award
NJLA 2009 Conference, Technology Innovation Forum
Tuesday April 28 2009, Long Branch, NJ
Innovation/Use of Technology
•
•
•
•
Benefits to library users
Benefits to the Information Community
Impact on the Library
Impact on Perception of the Library
How do I place a book on hold?
How do I place a book on hold?
Self-learning and
information Literacy
Rate the Q
Email us
Suggested Q’s
ASK NJIT LIBRARY: A Natural Language
Knowledge Base Self-Service Solution
Answering service
that answers repetitive questions – ready
reference questions.
ASK NJIT LIBRARY: A Natural Language
Knowledge Base Self-Service Solution
provides a
24/7 website self-service experience that
makes it easy for website visitors to type a
question in natural language and get one right
answer, powered by IntelliResponse
www.intelliresponse.com
IntelliResponse is an Instant Answer Agent,
implemented by more than 100 higher
education institutions.
Intelliresponse Marketing, October 17th 2008
www.intelliresponse.com
IntelliResponse captures every question, and
analyzes and reports on the data in a variety of
ways. This can help administrators, registrars,
marketers and department heads understand
exactly what students’ top needs are.
Intelliresponse Marketing, October 17th 2008
www.intelliresponse.com
IntelliResponse has been successfully
deployed campus wide as well as in support of
individual department(s)
Intelliresponse Marketing, October 17th 2008
Background of the Program
• commenced in August 2007
• took approximately 200 hours over seven
months
• 10 hours of training of 1 professional librarian
• creation of a knowledge base of about seventy
questions
• focus group testing on February 12, 2008
• product was launched March 18, 2008
Knowledge Base
Tasks
1. Knowledge base creation was the most challenging undertaking. Librarians
across all departments produced a list of over 100 questions.
2. List included questions stated in natural language and questions with
similarities.
3. Questions were also obtained from our library FAQ list and from other
library websites.
4. Many answers or responses to questions had to be integrated with existing
websites and many new answers had to be created.
5. Focus group consisted of 15 end users who were undergraduates, graduate
students and faculty.
6. Focus group feedback captured by IntelliResponse was analyzed over a
two weeks period (average of two hours per day).
7. Feedback helped to improve existing questions and uncover new content
opportunities
8. Work in progress
System Performance
System Performance Overview
- As of Mar 17, 2009
System Performance Overview
- Oct 28, 2008
How do I cite my sources?
Where can I find information on workshops?
Response Hit Frequency Report
48
How do I renew my
books online?
System Usage Distribution Chart
Total Questions Asked by Hour of Day
Total Questions Asked by Hour of Day
500
450
400
350
300
250
200
150
100
50
0
12:00
AM
1:00
AM
2:00
AM
3:00
AM
4:00
AM
5:00
AM
6:00
AM
7:00
AM
8:00
AM
9:00 10:00 11:00 12:00
AM AM
AM
PM
1:00
PM
2:00
PM
3:00
PM
4:00
PM
5:00
PM
6:00
PM
7:00
PM
8:00
PM
9:00 10:00 11:00
PM PM
PM
System Usage Distribution Chart
Total Questions Asked by Day of week
What are the Library Hours?
Natural language terms used
• calendar
• sched*
• hour*
• Hrs
• Hors – note spelling
• hr
• (what time)
• Clos*
• Open*
• Tim*
Quarterly Statistics - March 18,
2008 to March 17, 2009
Quarter
Total
Questions
System Health
Jan-Mar '08
202
62%
Apr-Jun '08
1483
77%
Jul-Sep '08
2046
74%
Oct-Dec '08
2031
83%
Jan-Mar '09
1899
83%
Total
7661
09=1143
Cost
Total questions per year
Number of minutes per question
Total hours for 6000 questions
Annual cost at student rate of $8.00 per hour
Annual rate at Librarian rate of $30.00 per hour
6000
5
500
$4,000.00
$15,000.00
Benefits to Library Users
•
•
•
Accessibility anywhere, anytime from the
web
Increased accuracy and consistency of ready
reference answers
Reduction of user time:
1.
2.
3.
4.
5.
to email
to make phone calls
to chat
to go the library
to navigate library website, Q&A or FAQ solutions
Benefits to the Information Community
Introducing a model:
• for library self-service
• that permits natural language query
• that can make it possible for libraries to collaborate and build
and share questions and answers in areas such as information
literacy as well as ready reference rather than try to build Q&A
from scratch. For example, it might be possible to collaborate
and build a folder of all the possible questions and answers
with respect to the question “How do I cite a website?”. It
would be possible to link both a web page as well as a short
video explaining how to do this.
Impact on the Library
•
•
•
•
Provision of better quality and more consistent answers to
questions
Unanswered questions are captured and can be used to create
new knowledge
Reveals implied customer needs without surveys or focus
groups
Reduction of staff/librarian’s time to:
1. answer emails
2. take phone calls
3. chat
4. navigate library website
Impact on Perceptions of the Library
• Library staff themselves find the
knowledge base valuable that even some
of them have begun to use it to obtain
answers to frequently asked questions.
• Enhanced library service – anecdotal
evidence favors the system
What Have We Learned?
• Some of our websites:
– may be inaccurate
– may need a makeover
– could be more user friendly
• Our library policies may need revision
• Our users wish to have their indepth
homework questions answered
• Our Intelliresponse knowledgebase will be a
continuous WIP.
Is there another library using IR?
University of
Western
Ontario
Library
Some Customers of IR
•
•
•
•
•
•
•
•
•
•
NJIT
Penn State
Ohio State
Harvard University
Hofstra University
Budget Trucks
Bell Canada
ING Direct
Johns Hopkins
University of Toronto
Similar Programs to Intelliresponse
•
•
•
•
RealDialog from Astute Solutions
Rightanswers
RIGHTNOW
Talisma
ASK NJIT LIBRARY: A Natural Language
Knowledge Base Self-Service Solution
NJLA CUS 2009 Technology Innovation Award
NJLA 2009 Conference, Technology Innovation Forum
Tuesday April 28 2009, Long Branch, NJ