IBVRE Digital Paper Evaluation Report

Integrative Biology VRE
Work Package 7: Digital Paper Evaluation
Report
Matthew Mascord
Marina Jirotka
Annamaria Carusi
Intended Audience
The purpose of this report is to give an account of the evaluation of Anoto digital pen and paper
technology conducted as part of the Integrative Biology VRE project at the University of Oxford. The
audience for this document includes the digital pen and paper developers and manufacturers, as well as
those researching paper augmentation or replacement technologies. It is also likely to be of interest to
the wider VRE development community, especially those projects funded as part of the JISC VRE2
programme.
Issued: 17 April 2007
1
Executive Summary
The JISC funded Integrative Biology VRE project aimed to investigate and develop the Virtual Research
Environment concept in a computational biology setting by extending and complementing the work of the
EPSRC Integrative Biology (IB) project in the areas of user interfaces and collaborative tools. IB is
building e-Infrastructure to address the needs of an interdisciplinary and globally distributed community of
heart and cancer modellers researching the causes of colorectal cancer and cardiovascular disease.
Following an analysis of research processes across the IB user community we chose to evaluate Anoto
Functionality, a leading digital pen and paper technology, to address the community's pressing need to
better manage their paper-born material, generated through a range of key research practices such as
mathematical modelling, data taking, and the keeping of laboratory notebooks.
Twelve Logitech IO2 digital pens were distributed to heart and cancer modellers in the IB community to
use for three months. Their experiences in using the technology were reported back through a miniworkshop on digital paper, one-to-one interviews, and through an online survey. The following gives a
summary of the key benefits, issues and requirements that emerged.
Conclusions and Recommendations
1.
Anoto Functionality can bring significant benefits to the researcher who uses paper as a fundamental
component of their research practice, for example, by providing a backup, facilitating sharing, making
material easier to locate, and offering opportunities for capturing information digitally in situations
where a tablet or PDA is unavailable or cumbersome.
2.
Although the sample community in this study expressed little desire for a structured online repository
for digital notes, experiences reported by other projects, such as the evaluation at CERN, have shown
this concept to be highly relevant in situations where there are already well established and justified
practices of regularly sharing paper-based material amongst collaborators.
3.
The significant interest in the possibility of converting handwritten equations to LaTeX (a
mathematical typesetting language) appears to merit further research in this area, building on work
such as PenMath at the Ontario Research Centre for Computer Algebra.
4.
If handwriting recognition is to be offered to the scientific community as a key value proposition in
future products, further work is required to make it more reliable for scientific text. Support for Greek
and common short forms needs to be supported by default, as well as simple table and column
recognition. There also needs to be a straightforward way to tag sections to be excluded from this
processing.
5.
Special tags to associated pages together, a document annotation solution, a Mac/Linux version, and
integrated search were the requirements deemed most important by evaluators and as such should be
seriously considered by those developing tailored solutions for this community in future.
6.
In the scientific community there are well established practices of building and exploiting home-grown
software to meet highly particular needs not catered for by commercial suppliers. We would strongly
encourage Anoto and its partners to publish their file formats so that those in the scientific community
can build on the technology, making it more useful and facilitate a wider take-up.
7.
To encourage a wider adoption, Anoto technology suppliers should consider allowing end-users to
print out their own paper, and to implement greater interoperability between paper formats produced
by different suppliers. Allowing end users to print out their own paper on scrap could alleviate some
of the environmental/cost concerns associated with the use of the special paper.
2
Introduction
The Integrative Biology VRE project was a JISC funded Virtual Research Environment project based at the
University of Oxford that ran for two years between April 2005 and March 2007. Led by Professor David
Gavaghan, it aimed to investigate and develop the VRE concept in a computational biology setting by
extending and complementing the work of the EPSRC Integrative Biology project.
The Integrative Biology (IB) project is a second-round EPSRC e-Science Pilot project which commenced
in February 2004 and is funded until January 2008. It received initial funding of £2.4M to support the
research of nine post-doctoral researchers, and to provide training for 10 PhD students, across seven UK
Institutions (the Universities of Oxford, Nottingham, Leeds, UCL, Birmingham and Sheffield, and
CCLRC), IBM, and the University of Auckland. The IB project's scope is a combination of science and
technology: on the science side, it aims to further understanding of the causes of cancer and cardiovascular
disease through a combination of mathematical and computational modelling; on the technology side, it is
building the underlying Grid infrastructure required to support this research. This infrastructure follows a
service-oriented Grid architecture, and is utilising many of the middleware developments within the UK eScience Programme and in the wider Grid community, and within the JISC Middleware programmes.
Whereas the IB project has focused on constructing a set of generic middleware services that can be
composed to support a diverse range of activities, the Integrative Biology VRE project was set up
specifically to investigate and address the user community's requirements in the areas of graphical user
interfaces and collaborative tools. In an initial requirements gathering exercise carried out at the start of the
project [3], one of the highest priority requirements to emerge was that of providing better management of
paper-based material produced through a range of research practices within the IB community.
This report describes the findings of an evaluation of a bespoke technology designed to address this - Anoto
Functionality - a digital pen and paper solution based on the use of real pen and paper.
3
The Integrative Biology Community
The two main user groups within the IB community are the heart and cancer modellers. The heart
modellers have a long history dating back some forty years to the pioneering work of Denis Noble (1962).
In contrast, cancer modelling, particularly the modelling of colorectal cancer currently being tackled by the
IB cancer modelling Tumour Task Force (TTF) is relatively new.
Heart modelling groups within the community tend to be globally distributed but tied together through the
common use of a large-scale simulation package, used on a day-to-day basis to perform in silico
experiments. There are several examples of these in use by the IB community. However, the most
commonly used are: Memfem [4], used by groups at the Johns Hopkins University, University of Oxford,
and Tulane University; and CMISS [5], written by Peter Hunter and researchers at Auckland University,
New Zealand.
In contrast, the cancer modelling groups are largely UK-based, at Nottingham, Birmingham, and Oxford
Universities. The aim of the IB project is to support this community in its initial attempts to build a
comprehensive model of cancer development across multiple spatial and temporal scales. However, at
present, the majority of their work is mathematical, and fundamentally paper-based, with some use of
symbolic manipulation packages such as Mathematica or Maple to provide quick answers. Individual
researchers do perform small-scale simulation work in packages such as MatLab, but this work is mainly to
validate and provide proof of concept for the fledgling mathematical models.
4
The Use of Paper within the Community
Within our initial analysis of research processes across the IB community, we found that paper was used by
researchers in two quite distinct contexts: either as the principal medium through which research is
conducted, or to provide a record or diary of work that has been, or is in the process of being carried out.
Paper was also often used to help an individual manage research tasks on a moment-to-moment basis, for
example, in the form of 'To do' lists, or Post-it® notes stuck to a computer monitor.
When paper is the primary tool used for carrying out the research, as is the case with the mathematical
modelling of cancer tumors, there did not appear to be a need to keep a separate record. Typically, these
modellers produce mathematical formulae on scraps of paper or whatever is at hand. These scraps may
then be collected some time later and ‘filed’ in boxes or drawers. Some order may be imposed by stapling
together chunks of related of material before filing, for example, a combination of notes, plots, figure cutouts, and calculations.
Further uses of paper, relevant to both the cancer and heart modellers included the drafting of academic
papers, recording thoughts, note-taking at seminars and making notes from papers. Whether or not material
is kept in the long run is an archival decision taken by the individual researcher; for some materials the
decision is easy, for example, previous drafts of papers are not usually kept, as anything worth keeping
would be typed up. However, while this preliminary or incomplete material may not be useful for a current
project, it can often turn out to be very useful in the future, when it can be further expanded upon.
Paper tends to be used as a record-keeping medium where the work has been conducted outside the
medium of paper, for example, performing experiments in a lab, or the computer-centric design of in silico
experiments. The usual form for this record-keeping activity is the traditional lab book. These books have
been identified as forming a critical part of the modellers’ research practice. One heart modeller whose
research examines the electrical activity of the heart through computer simulations, said ,'it would be a
disaster to lose my book'. In these settings, entries represent the evolution of scientific discovery and as
such the experimenter notes the procedures followed, methods used, results and any ideas or observations
that occur during the work within the lab or outside of it. It is used to record thoughts and input parameters
to experiments; build up and record reading lists, stick in figure cut-outs from papers and results. Selected
pages from the book were sometimes photocopied and shared with others if people needed them.
It is important to note that the lab books are written for specific groups of audiences, namely other
scientific researchers, so that others may follow the steps and approaches taken. Consequently, specific
practices have evolved, for example, that entries are chronological not logical. This means that if a solution
to a past problem emerges it should not be added to the pages dealing with that problem. Rather it should
be listed chronologically, on the day it occurs, with annotations to show to what it refers. In addition, they
may also be used as a source of evidence to establish intellectual property [6] [8] [15].
Other researchers use online blogs or wikis, as an alternative, or in addition to, their paper-based lab book.
The reasons for using paper rather than an online equivalent are sometimes simply because the researcher
prefers it, but also fundamentally due to its convenience - often it may be impractical to use anything else.
For example, heart modellers when performing experiments on heart patients, use lab books to record data
during the experiments. Though they may use PDAs on a regular basis, paper is preferred in these
situations as it is seen as more flexible and reliable in the narrow time windows available for these
experiments - which are conducted in the middle of heart surgery. In this case, the lab book is used as an
interim storage solution; the data is always subsequently scanned in for analysis by colleagues.
These findings resonate with research in CSCW and workplace studies on the affordances of paper [7].
This research has suggested that often technological devices such as PDAs, as in this example, become the
‘interactive domain of interest’ instead of the activity the practitioner is engaged in such as the heart
surgery in this example. Entering data onto a PDA or other technological device requires the practitioner to
examine the screen to look at what is being typed in. Writing with pen on paper is more immediate and
does not require the extra step of checking.
5
Anoto Functionality
There have been several technological approaches to addressing the problem of capturing handwritten text
digitally. Such solutions rely either on the use of special paper, special pens or some combination of the
two. Prior work, for example SmartTea [16], has addressed this through the development of bespoke
digitising tablets. However, we sought a solution that would be as close as possible to the look and feel of
writing on real paper, and so chose to evaluate Anoto functionality [9], the market leader in digital pen and
paper solutions based on the use of real paper.
Anoto technology is based on a special pen, and paper imprinted with a barely visible dot pattern. The pen
uses ink, and writes just like a normal ballpoint pen; equally, the paper is real. However, the pen is also
equipped internally with a digital camera, image processing unit, and communication device. When the
user writes on the specially imprinted paper, the digital camera, by recognising the dot pattern, captures the
movements of the pen across the paper, as well as the pressure and angle of the pen on the paper.
Data stored in the pen can be transferred to a computer for view, storage, or manipulation through a
docking mechanism, or for those pens that support it, bluetooth. Bluetooth functionality also allows data to
be sent to others while on the go. For example, the Nokia Digital Pen SU-1B [10] allows handwritten notes
to be sent to others with compatible devices via a mobile phone's multimedia messaging service (MMS).
A wide range of commonly used paper products have been manufactured to include the Anoto dot pattern.
For example, Black 'n' Red notebooks, A5 spiral bound notebooks, A4 pads, and even Post-it® notes. Such
products often include printed checkboxes that control the operation of the pen itself. For example, all
notebooks have a 'New Notebook' checkbox, which must be checked when starting a new book to inform
the software that all new pen data should be stored in a new file. Another example is the use of gatefold
pages containing checkboxes to control the colour and thickness of the writing as it appears in the digital
pen software.
The types of digital pen products adopting Anoto functionality range from those sold with consultancy by
business-to-business system integrators constructing bespoke solutions intended to be tightly integrated into
a particular organisation's business processes, to products sold by consumer facing firms providing more
plug-and-play, multi-purpose solutions. As we did not want to make any assumptions about the types of
activities the technology would be used for, the main criteria for technology selection were that the
technology should be multi-purpose, support a wide variety of paper types, yet be very easy to set up. This
rationale led to the decision to use the Logitech IO2 Digital Writing system [11].
6
Evaluation Methodology
The overall intention of the evaluation was two-fold. It was first to address a range of needs at a very
practical level, but it was also to identify requirements on how this technology might be developed further,
particularly with respect to its integration within VREs.
The methodology chosen aimed to examine the use of the technology in as naturalistic setting as possible.
Rather than prescribing set tasks, we wanted to allow for the use of the technology for more freeflowing
activities. By embedding the pens in the researcher's context, they acted as a probe [12] or intervention in
the researcher's environment, allowing us to get at this context. Hence, we adopted a diary study approach.
In diary studies [13], participants using a technology are asked to record observations of their use of a
technology in a diary throughout the day. In this case, however, the very technology the researchers were
trialling was used to record observations. Participants were asked to keep an informal log of their
experiences of using the pen and paper at the back of the supplied Anoto notebooks.
In April 2006, we conducted a two week preliminary study, involving two members of the Mathematical
Institute at Oxford. The purpose of this was both to inform our main evaluation but also to contribute to a
poster and demo at the UK e-Science All Hands Meeting 2006. In the main evaluation, carried out between
July and September 2006, twelve members of the IB community were involved, mostly heart and cancer
modellers at various academic levels, from graduate student to professor. Experiences of participants
involved were elicited at a mini-workshop on digital paper, follow-up interviews, and through an online
survey.
7
Preliminary Study
At the start of the preliminary study in April 2006 we had not yet ordered the Logitech IO2 Pens but
already had to hand a pen supplied by Maxell, and so we decided to use it for this initial investigation. The
DP-201 Bluetooth/USB pen, packaged inside the Maxell Digital Pen Evaluation Kit, also shipped with a
cradle for docking, a spiral-bound A5 notebook, and the Digital Note Viewer software.
Docking the DP-201 pen caused the pen data to be uploaded to the Digital Note Viewer software, and
deleted from the pen. In common with other products adopting Anoto functionality, the data was saved as
a set of vectors within a Pen Generated Document (PGD) file, recording the movements of the pen across
the paper. This data storage strategy meant that file sizes are typically much smaller than an equivalent
image of a scanned page, even when the image is compressed.
The Digital Note Viewer software presented the pen data as a virtual notebook, a series of pages mapping
directly to the physical book, and allowed the data to be viewed, manipulated, and converted to a variety of
image formats for transfer to standard office applications. Pen strokes could be viewed as-is or 'replayed'
within the software, exactly as they were created (see figure 1).
Figure 1 - Digital Note Viewer
In this preliminary study, the pen was used for a wide range of activities, for example, drafting sections of a
paper to be submitted, checking calculations, note-taking in a library, and small amounts of cancer
modelling. It was also used in supervisions with students and post doctoral researchers.
Design of the Pen
The researchers found the pen reasonably easy to use while slightly fatter than pens they would normally
use. One researcher commented that her handwriting was slightly worse when using the pen but that this
was not a major issue. There were no problems with the life of the battery but, apart from the example of
its use in a library, pens tended to be used in the office where they would be docked (and hence charging)
when not in use.
Design of the Software
The software provided as part of the evaluation kit, the Digital Note Viewer, was seen as basic but easy to
use. The PGD file could either be kept untouched or manipulated as required: this was seen as both an
advantage and a disadvantage. One modeller liked the fact that pen data could be manipulated as it allowed
her to simulate, in the digital note viewer, the protocols she had developed with physical paper of gathering
together, and stapling related material. However, it was seen as crucially important to always keep a readonly master copy, recording everything that the pen writes sacrosanct, so that it is always possible to go
back to the original in the event of hardware, software, or human error. It was suggested that the software
might do this automatically, offering only copies for manipulation. Another safeguard suggested was that
the software should automatically save the pen data to the PGD file upon docking, rather than the
responsibility for this lying with the user. The software as it stood left open a small risk of data loss for
users not properly educated on the importance of saving the PGD file immediately upon docking: this was
encountered after losing a small amount of material this way.
Small file size was seen as a major benefit, although this was only realised when all manipulation of pen
data was performed within the Digital Note Viewer; when exported to other packages, the size of the pen
data increased considerably. Further features requested were for some support for highlighting material in
the viewer, and being able to paste in figures/cut-outs from papers, or plots from mathematics packages,
ideally from within the Digital Note Viewer software to retain the advantages of the small file size.
Paper Formats
There were two aspects of the paper format that were considered: the size, and the binding. For taking
notes from seminars and papers, the A5 size was seen as ideal, and the spiral binding useful, to ensure
pages do not go missing. However, for mathematical modelling, the A5 format was seen as too small, with
the researchers finding they were running out of space. Additionally, the binding was seen as an
impediment, as some researchers like to arrange pages in front of them whilst working to be able to see
everything at a glance. A4 pads appeared to be the ideal because they are large enough for the mathematics
and also allow pages to be ripped out. Whether or not the paper is lined, plain or squared was regarded as
unimportant.
According to the help documentation provided with the digital pen, it was not possible to use multiple
books of the same type at the same time. Using multiple books of the same type with the same pen would
result in an irreversible 'double exposure' effect, with pen data from multiple books overlaid within the
same page in the Digital Note Viewer software. One way round this, was to always tick the 'New
Notebook' tickbox, before switching to a different notebook - forcing the software to prompt for a new
PGD file upon docking, removing the possibility of the double exposure effect, but at significant user
overhead. This was considered to be a real limitation of the technology as it is common practice to keep
different notebooks for different projects, and separate books for seminar notes, notes from papers, as well
as for work with students. Having only one A4 pad on the go at once would be less of a problem, as pages
within these are not normally kept together as one unit, but are removed and stapled with related material.
Organising Material for Efficient Retrieval
It was thought that the pen would assist in the management of large amounts of hand-written material, for
example, by keeping track of the volume, being able to categorise it, and keeping it safe. 'Safe' was defined
here in terms of being able to find it again, rather than protecting it from physical destruction. Researchers
may encounter difficulties in organising the paper material, and can spend a large amount of time looking
for material but not finding it. Some find electronic documents easier to organise than paper, mainly
because material can be named and organised into directories. For this reason, some type up a lot of
material simply in order to be able to find it again. It was commented however, that a longer study would
be required in order properly to assess these benefits, to allow a larger volume of material to be
accumulated.
It was thought that Optical Character Recognition (OCR), if good, would be useful, but problematic as
mathematical calculations do not tend to consist of long strings of words but are more likely to consist of
isolated words interspersed with equations and diagrams. Its main use might be to enable the search for
seminar and paper notes, rather than to convert to text. There was however, interest in any tool that could
convert hand-written mathematics directly into LaTeX [14], a typesetting language for mathematics, as this
would save everyone a lot of time.
Using Digital Paper for Email
There was also strong interest in the idea of using the technology in conjunction with email, particularly
when collaborating with non-local collaborators. Difficulties are encountered using email for mathematics
because it is a predominantly text-based medium, whereas in order to communicate mathematical ideas
effectively, an equation or diagram is nearly always required. It was reported that some groups of
mathematicians try to get round this limitation by writing commands from LaTeX directly in the email, for
example '\int' for the integral sign. A far better solution, however, would be support for sending handwritten equations or pictures directly in email, a feat trivially accomplished with digital pen technology.
When probed as to why mathematicians did not use scanners or faxes to achieve this same objective,
researchers maintained that this would entail a lot more work due to the extra step of finding and using the
scanner or fax machine which are rarely at hand. However, there were limitations as the technology worked
only on Windows, and some researchers used a text-only email client, Pine, on a Linux desktop machine.
The problem here would be that the pen-generated image would need to be transferred from the laptop to
the Linux desktop over FTP before it could be sent in an email. This extra step might well be a significant
disincentive. However, having the capability to send pen data over email might be an incentive to switch to
a graphical email interface.
Another related possibility that wasn't explored as part of this evaluation, was the use of digital paper in
conjunction with instant messaging (IM) applications.
Using Digital Pens for Training
Some researchers tended to do less work individually, and saw the main use of the technology as aid to
supervisions of students or post doctoral researchers. During these meetings, students bring work they have
completed and student and researcher work together on a problem. Usually the student will take the
resulting material away at the end of the meeting and the researcher does not see it again until the next time
they meet, assuming that students are organised enough to bring it.
If the digital pen were used in these meetings, there would be a digital record of everything that was
written, even if the paper record was taken away by the student, and so it would be easier for the researcher
to keep track of exactly what was discussed. One limitation of the digital pen technology, however, is that
it is not possible to use multiple pens with the same pad, and so the pen would have to be shared by both
the supervisor and supervisee. This was not seen to be a problem as the researcher would normally be
doing most of the writing. The only place where the technology would not work would be when writing
over manuscripts.
8
Findings from the Main Evaluation
The mini-workshop brought together those who had evaluated the technology as part of the Integrative
Biology VRE project, as well as a number of those who had evaluated it, or were planning to evaluate it, in
related outside projects. The aim here was to allow these people to share their experiences and help define
priorities for future development, ascertaining any need for the tailoring of this technology specifically for
the IB community, or integration of the technology within a VRE.
Those invited from outside the project included Paul Luff, co-ordinator of the EU PaperWorks project at
Kings College London; François Grey, head of the IT Communications Team at CERN; and Jeremy
Leighton-John, Curator of Digital Manuscripts and Scientific Curator at the British Library. Both Paul and
François had evaluated the technology in previous projects and were interested in the experiences of those
working within an integrative biology context; Jeremy was interested in how digital paper might change the
practices of note taking in the field. Ruth Kirkham, Project Manager of the Building a VRE for the
Humanities (BVREH) project, also attended in the hope that the workshop might help inform a planned
evaluation of Anoto Functionality with humanities researchers.
Workshop Agenda
The agenda was designed to allow everyone who had evaluated the technology at least ten minutes to
describe their experiences, before participating in a session dedicated to prioritising any emerging issues or
requirements. Paul Luff helped frame the discussion initially by introducing a number of projects and
possibilities relating to paper augmentation. There was also a final session, aimed more at those
researching the technology, to address the wider question of paper augmentation and replacement.
Round-the-table Summaries
In this session, eight individual experiences were presented. A good range of usage situations were
described, from basic note taking in meetings to the more advanced examples of using the digital pen and
paper to record data during experiments. The most common use was using the pens to take notes in
meetings and at conferences. One of the more innovative examples was using the pen to prepare material
for presentations.
Figure 2 - A Mathematician using Digital Paper for Cancer Modelling
Evaluators' attitudes to the technology following this evaluation were wide ranging, from the fairly negative
to the highly positive. On the negative side, there were examples of users experiencing serious problems
with the software conflicting with other applications running on their computers. In one instance this led to
an evaluator giving up after only one week. On the more positive side, many participants were very
enthusiastic and fully intended to continue to use the technology even after the end of the formal evaluation
period.
Key Benefits
The aspect of the technology people liked the most was having a digital copy backup of the hardcopy and
vice versa. One mathematical modeller used the pen for keeping notes at conferences and mathematical
modelling, and found it useful that even if the hardcopy was thrown away or lost, the digital copy would
remain. On the other hand, another participant described how she had had problems with her PDA that had
led to data being lost – an advantage with the digital paper is that even if the digital copy is erased, the
hardcopy remains.
The other major benefit reported was how much easier sharing material became. Examples were cited of
mathematicians sometimes scanning or digitally photographing material and sending it. The main benefit
of the digital pen is that it does not rely on having a scanner or digital camera to hand - if the pen is being
used on a day-to-day basis, material can be sent instantly over email, as-is. The fact that digital notes are
stored in a vector graphics-like file format, also makes such emails more space efficient than an email with
an equivalent JPEG or TIFF attachment.
Other positive things evaluators said about the technology included:
•
•
•
•
•
•
•
•
It was useful when the tablet/PDA was unavailable or cumbersome.
It was good for writing letters and emails, and simple diagrams.
It was good for writing equations even though it doesn't convert them.
The saved output is very close to the written page.
A presentation prepared using the pen looked nice on screen.
It had great battery life.
The memory was ample.
The pen size easy to get used to.
Quality of Handwriting Recognition
The most universal complaint was the quality of the handwriting recognition system, a feature absent from
the digital pen software used in the preliminary study. The handwriting recognition appeared to be reliable
only for plain handwritten text, without any scientific short forms, equations, diagrams, dates, or Greek
characters. Even for plain text, participants had mixed results, with reliability appearing to depend on the
person, and their handwriting style. One of the evaluators thought he had bad handwriting and was
surprised that the pen software managed to recognise it. On the other hand, another found that the system
would not recognise her handwriting even after running the training application four times. An evaluator
based at the Oxford Physiology department reported that the plain text conversion tended to perform worse
with 'decorated' pages, for example, where the text contains doodles or diagrams, or where certain words
are circled.
When questioned further about their need for handwriting recognition, it often appeared that this derived
from the requirement to be able to search the digital notes - a feature absent from the Logitech IO2
software. Not everyone in the group was fully aware of the Logitech Google Desktop plugin that has the
capability to transparently index and OCR digital notes residing on a computer in the background.
Related to handwriting recognition was the requirement for the recognition of simple tables. An
experimentalist based in the Physiology department described how she frequently writes down data in
tabular columns during experiments in the lab, and found that the Logitech IO2 software was unable to
recognise them. She had also tried an evaluation version of another package, MyScript, which provided
basic table support but still lacked recognition of scientific short forms.
Other Issues
The most serious issue with the Logitech pen encountered was related to an intermittent fault with the pen
ink sensor. Four of the participants experienced this fault to greater or lesser degrees. The sensor appeared
to stick, so that even when the pen was lifted off the page to start a new mark, it would continue to register
the underlying pattern, resulting in a scrawling effect
Figure 3 - Example of Scrawling Problem - the intended text here is 'Scrawl test'.
A solution to the issue was eventually identified, though too late to be of use to those involved in the
evaluation. Applying a small amount of lubricant, in this case WD40, to the pen ink refill seemed to free
the mechanism allowing the pen to register correctly when the pen was in contact with the page. Note that
there were no instances of this issue experienced with the Maxell DP-201 pen.
A rare but serious issue affected the performance of the software itself. In two cases there was a conflict
with other software running on the evaluators' machines resulting in 100% CPU utilisation. A third
participant didn't encounter this problem but would have preferred it if the software were not memory
resident and for it to be possible to manually upload notes from the pen rather than this happening
automatically every time the pen is docked
In terms of more fundamental issues with the technology, one concern expressed was related to the cost and
environmental consequences of the use of the special paper. One of the mathematical modellers described
how many mathematicians write on any paper they have to hand, mostly scrap, and even napkins. Unless
useful, much of this material is thrown away. It was felt that using such high quality paper at all times
would be both costly - as the paper is more expensive than premium standard paper - and wasteful. The
other related issue here is the fact that whilst you might have the pen to hand, you may not have a
convenient supply of Anoto paper.
Suggestions
The mathematicians liked to spend time organising notes after upload, for example, by distilling conference
notes to make them more useful. One mathematician when preparing a presentation using the technology
liked the way text could be moved around, and was particularly pleased with how the end-result looked on
screen.
On the other hand, many did not like the way the software stored each notebook page as a separate file they would have preferred it if file organisation could be controlled from the paper itself through the use of
special tags, to enable, for example, each talk to be stored as a separate file, rather than every page.
Other suggestions included the following:
•
Ability to write over the top of presentations while delivering them, for example, as an alternative
to OHPs.
•
Ability to record audio notes on the pen where writing things down is too time consuming
•
Support for Linux and Mac - all evaluators had access to Windows but many used a Linux
machine as their main workstation.
•
A wider choice of ink colour, beyond blue and black, to allow for example the use of different
colours in diagrams: one for the equations, and another for the boundary conditions.
Experiences at CERN
François gave a general overview of science carried out at CERN, noting some of the larger particle
physics experiments and their need for a distributed computing infrastructure due to the vast amounts of
experimental data generated. He also introduced work by summer school students on the OpenLogBook
project [17], to develop an electronic lab book solution for a number of scientific groups at CERN. The
bulk of his presentation concerned work to extend OpenLogBook with support for digital pen and paper,
and the evaluation of this technology with three research groups at CERN.
The biggest breakthrough for the technology was with the Athena antimatter experiment [18], a
collaboration of around 100 people and 20 institutes. Athena use pen and paper log books as a central part
of their data taking; in the course of a year, 30 paper log books are filled in by control room operators,
recording key qualitative data relating to the experiment. At the end of the year, a secretary takes the stack
of thirty books, copies them, and distributes to the 20 partners. Scientists in each institute are then able to
see all data captured throughout the year.
In 2004, Anoto digital pen and paper technology was introduced into the Athena collaboration as a
replacement for their paper log books [19]. In addition, a simple password protected web site was
developed to allow digital notes to be deposited and shared amongst partners. The introduction of this
technology meant that the log book information could be shared immediately with remote partners,
increasing the interactivity scientists had with the people in the control room. The fact that the digital notes
were time-stamped also introduces the possibility of making connections with other related digital data.
In 2006, a more general evaluation of the technology was conducted at CERN. 25 Logitech IO2 pens,
provided free by Logitech were given out to test users. The outcomes from this were as follows:
Benefits:
•
Sharing paper on web very useful (as mentioned)
•
Password protected.
•
Portability.
•
The ability to quickly compose and send hand-drawn sketches.
Weaknesses:
•
No ability to associate printouts, and graphics into the digital notes.
•
No annotation solution.
•
No Linux/Mac solution.
•
No solution for annotating the online version of the notes, only the original paper copy.
François also mentioned the fact that they would like to get the stroke timestamp from the Anoto XML file,
which is currently encrypted. This led to François later indicating the desirability, from his perspective, of
Anoto publishing its file formats as open standards, to allow such information to be interpreted and used
more widely.
Outcomes from Open Discussion
This open discussion, facilitated by Marina Jirotka, was designed specifically to prioritise the requirements
and issues already established. As only three of the original evaluators could stay for this session the
outcomes were not, unfortunately, representative. The session turned out to be more concerned with
elaborating previously mentioned issues/requirements, as well as uncovering further requirements that had
not already been raised through the round the table summaries.
According to the evaluators remaining, the most important issue to address was the scrawling. One of
those experiencing this felt that if she continued to get it all the time, she would stop using the pen.
Next on the list was addressing the post-processing issues. It was described how the Logitech IO2
software's method of saving each individual hardcopy page as an individual file led to the need for a lot of
post-processing work to associate related material together as one file. There was the suggestion of
introducing unobtrusive formatting marks to control the treatment of notes after they are uploaded into the
software. By drawing lines in the margin, the user could indicate which sections of each page to go into
one file, and the name of the file. For example, the latter half of page 2, page 3 and the first half of page 4
should go into a file called X. The group also felt it might be useful to have tags to indicate sections to
completely exclude from upload.
Another related issue was the ability to associate print-outs or diagrams with the digital notes. In the hardcopy, printouts and diagrams would be stapled to related mathematical material. In the Logitech software
there was no ability to associated non-digital note material in the editor.
Although none of the participants expected the technology to recognise mathematical equations, the
possibility of translating these was briefly discussed. Though most doubted the feasibility of
accomplishing this, those present thought that if it was possible, LaTeX would be the most useful format to
convert to. It was commented that other experimentalists may want conversion to other mark-up languages
such as ChemML. Prior work to convert handwriting to MathML [20] was mentioned, and there was the
suggestion that this work could be extended to work with the digital pens, by integrating existing open
source tools to convert MathML to LaTeX.
Some small but important usability issues with the pen were raised. Two participants found that the pen
would buzz and beep in certain situations. One thought it was related to the scrawling problem; another
thought it occurred when writing too close to the centre divide of the Anoto notebook. It was reported that
this buzzing and beeping would seriously hinder their concentration, preventing them from using the
technology. Another issue reported was the way in which the Logitech IO2 pen would automatically turn
itself off after a period of time, if the cap was left off. Two reported this problem and found that it led to
note loss because it was not always obvious that the pen was off.
In terms of integrating digital notes into a VRE, there was some discussion about the need for providing
some form of online repository for digital note data, such as the one developed by François Grey's group at
CERN. One participant who was not involved in the IBVRE evaluation, described how this would be a
very good idea from his perspective because his group frequently share data between collaborators. Notes
are taken, scanned in, and PDFs sent across - a shared web page that everyone can log into would be a
really good idea. The question of tagging notes to be viewable by others was raised, but none of the
participants expressed an immediate need for this.
Although only mentioned near the end of the discussion, the ability to print out documents on the digital
paper and annotate was raised as a key feature missing from the Anoto digital pen and paper technology.
One of the mathematical modellers described how he would often receive a draft via email but could not
send handwritten comments back, which results in him posting or writing the changes into an email. Drafts
get bounced about a lot. If the pen can only be used for the first iteration, it is not so useful. The group felt
sure there would be people working on this and mentioned work with publishers to carry out their editing
through the use of digital paper. The standard set of proof editing marks could be recognised by the digital
paper software.
Near to the end of the discussion, the question of the possible personal identification of digital notebooks
was raised. It was noted that the Logitech books have bar codes, and that other people had been
experimenting with RFID tags. One of the participants felt it might be possible to integrate an identifier
into the dot pattern itself. It was pointed out that the Logitech pen can hold the user's name and email
address, and the potential for privacy concerns. No one among the remaining evaluators expressed a
particular need for this.
Online Survey
An online survey was developed following the workshop to help establish a prioritised list of issues and
requirements. This was a short structured survey built using the free SurveyMonkey [21] web-based tool.
It was based on the list of usage situations, issues, and requirements gathered through the workshop, and
follow-up investigations. Ten out of the twelve evaluators responded.
Figure 4 - Digital Pen Survey
Respondents were asked to rate each issue and requirement in order of importance, from 'Very Important'
to 'Not Important'. In order to translate this importance matrix into an ordered list, a simplistic scoring
algorithm based on weighted sums was introduced. This assigned 'Very Important' a weight of three,
'Somewhat Important' a weight of two, and 'Important' a weight of one. An issue considered by all
respondents to be 'Very Important' would therefore score thirty whereas an issue considered 'Not Important'
by all respondents would score zero.
Score =
3 x <Number of Respondents Rating as Very Important> +
2 x <No. Respondents Rating as Important>
1 x <No. Respondents Rating as Somewhat Important>
The following list of usage situations validates the findings from the workshop that the most common use
was for taking notes in meetings and conferences. It is also useful to bear in mind while considering the
issues and requirements users felt most important.
Rank
1
2
3
4
5
6
7
8
9
10
11
12
Uses
Taking notes in meetings
Taking notes at conferences.
Mathematical modelling.
Preparing material for presentations.
As a replacement for a traditional laboratory
notebook.
Taking notes from papers.
Writing down data relating to simulations.
Writing letters and emails
Drafting proposals
To work through a problem in supervisions.
Writing down data during experiments in the lab.
Checking calculations in a to-be-submitted paper.
Occurrences
7
6
4
4
3
2
2
2
1
1
1
0
13
Drafting papers
0
As the following ordered list of issues shows, the problems with the handwriting recognition were ranked
as most important by the majority due to the fact that it was a problem encountered by virtually everyone.
The scrawling and software conflict problems, though critical for those experiencing them, were only a
problem for a minority of participants.
Rank
1
2
3
4
5
6
7
8
8
9
10
10
10
11
Issue
Handwriting recognition limited/faulty
Time spent organising after upload
Spidering/scrawling problem
Handwriting recognition: scientific shortforms/greek not recognised
Upload time
Handwriting recognition: 'is' converted to 13
Handwriting recognition: dates not recognised
Bulky
Capture frequency too low
Pen going to sleep if cap left off leading to loss of notes.
Beeping/buzzing
Capture resolution too low
CPU Utilisation Problem
Cost/environmental issues
Score
22
21
20
19
18
16
15
14
14
13
11
11
11
10
The following table gives a list of requirements for future development, in order of perceived importance.
Ability to search came out top, with the ability to do more sophisticated post-processing next on the list.
Ability to annotate documents was also a requirement very near the top of the list.
Interestingly, one of the features we expected users to want most, the provision of a form of shared online
repository for the digital notes, came out relatively low in the priority list. This finding points to a low
level of interest in the integration of this technology in a collaborative virtual research environment, at least
within this cross-section of the heart and cancer modelling community.
Rank
1
2
3
4
4
5
6
7
7
8
8
9
9
10
Requirement
Ability to search digital notes.
Tags to indicate which pages are saved as a single file.
Ability to print out documents on digital paper and annotate.
Customizable tags
Equation recognition and conversion to LaTeX.
Support for Mac/Linux
Tags to exclude material from handwriting recognition
Diary/schedule
More sophisticated options for cutting and pasting digital notes (in
addition to rectangle, freehand and timeline)
Tags to exclude material from upload.
Timestamped notes to tie to a particular dataset.
Reference management facilities e.g. reference auto-completion,
auto-generation of BibTeX/Endnote databases
Tying printouts to digital notes (simulate stapling action)
Pressure sensitivity - change colour according to how hard you
press.
Score
24
22
20
19
19
18
17
16
16
15
15
14
14
13
10
11
12
13
14
14
Tags to indicate privacy of digital notes.
Ability to write over a powerpoint when presenting it.
Online repository to share digital notes
Other coloured inks (in addition to blue and black)
Ability to record audio notes and associate with a page.
Pencil version
13
12
11
10
7
7
Paradoxically, when users were asked in the survey, 'Who, if anyone, do you intend to share the material
captured by the pen with?', willingness to share with others was indicated in six out of ten cases.
Presumably, these users were happy with the use of email or IM to achieve this same objective rather than
through more tailored online repositories.
Rank
1
2
3
4
9
Requirement
Close collaborators/friends
No-one - the material is completely private to me.
Other - Students I teach, supervisor, collaborators
General public
Occurrences
5
4
1
0
Conclusions and Recommendations
1.
Anoto Functionality can bring significant benefits to the researcher who uses paper as a fundamental
component of their research practice, for example, by providing a backup, facilitating sharing, making
material easier to locate, and offering opportunities for capturing information digitally in situations
where a tablet or PDA is unavailable or cumbersome.
2.
Although the sample community in this study expressed little desire for a structured online repository
for digital notes, experiences reported by other projects, such as the evaluation at CERN, have shown
this concept to be highly relevant in situations where there are already well established and justified
practices of regularly sharing paper-based material amongst collaborators.
3.
The significant interest in the possibility of converting handwritten equations to LaTeX (a
mathematical typesetting language) appears to merit further research in this area, building on work
such as PenMath at the Ontario Research Centre for Computer Algebra.
4.
If handwriting recognition is to be offered to the scientific community as a key value proposition in
future products, further work is required to make it more reliable for scientific text. Support for Greek
and common short forms needs to be supported by default, as well as simple table and column
recognition. There also needs to be a straightforward way to tag sections to be excluded from this
processing.
5.
Special tags to associated pages together, a document annotation solution, a Mac/Linux version, and
integrated search were the requirements deemed most important by evaluators and as such should be
seriously considered by those developing tailored solutions for this community in future.
6.
In the scientific community there are well established practices of building and exploiting home-grown
software to meet highly particular needs not catered for by commercial suppliers. We would strongly
encourage Anoto and its partners to publish their file formats so that those in the scientific community
can build on the technology, making it more useful and facilitate a wider take-up.
7.
To encourage a wider adoption, Anoto technology suppliers should consider allowing end-users to
print out their own paper, and to implement greater interoperability between paper formats produced
by different suppliers. Allowing end users to print out their own paper on scrap could alleviate some
of the environmental/cost concerns associated with the use of the special paper.
10 References
[1] Integrative Biology VRE Project, http://www.vre.ox.ac.uk/ibvre/
[2] Integrative Biology project, http://www.integrativebiology.ac.uk/
[3] IBVRE Initial Analysis Report,
http://www.vre.ox.ac.uk/ibvre/IBVRE%20Initial%20Analysis%20Report.pdf
[4] Memfem, http://virtualheart.bmen.tulane.edu/pmwiki/pmwiki.php/Main/LabSoftware
[5] CMISS, http://www.bioeng.auckland.ac.nz/cmiss/cmiss.php
[6] McCormack,J.B., Morrow, R.K., Bare, H.F., Burns R.J., Rasmussen J.L. 1991. The complementary
roles of laboratory notebooks and laboratory reports, IEEE Trans Education, vol. 34. 133-137.
[7] Paul Luff, Christian Heath, Moira C. Norrie, Beat Signer, Peter Herdman: Only touching the surface:
creating affinities between digital content and paper. CSCW 2004: 523-53
[8] schraefel, m.c., ., Hughes, G., Mills, H, Smith, G., and Frey, J. 2003. Breaking the Book: Translating
the Chemistry lab Book into a Pervasive Computing lab Environment. Proceedings of CHI 2004 Vienna
Austria
[9] Anoto Functionality, http://www.anotofunctionality.com/
[10] Nokia Digital Writing Solution, http://www.nokia.co.uk/
[11] Logitech IO2 Digital Writing System, http://www.logitech.com/
[12] Gaver, B., Dunne, T., Pacenti, E. 1999 Design: Cultural Probes. Interactions Vol 6, Issue 1. 21-29.
[13] Brown, B., Sellen, A.J., and Ohara, K. 1999. A Diary Study of Information Capture in Working Life.
HP Labs Technical Report.
[14] LaTeX, http://www.latex-project.org/
[15] Zanón. E. 2001 El Diario de Laboratorio en Diseño Electrónico y su uso en el Ámbito Universitario
En: Mundo electrónico, , Nº 321, 60-6
[16] SmartTea, http://www.smarttea.org/
[17] OpenLogBook, http://openlogbook.sourceforge.net/overview/index.html
[18] The ATHENA experiment, CERN, http://athena.web.cern.ch/athena/
[19] ATHENA tests the digital logbook, CERN Courier, Volume 45, Number 1,
http://cerncourier.com/main/article/45/1/13
[20] PenMath, Pen-Based Interfaces for Mathematics, Ontario Research Centre for Computer Algebra,
http://www.orcca.on.ca/PenMath/
[21] SurveyMonkey, http://www.surveymonkey.com/
11 Acknowledgements
The Integrative Biology VRE project, led by the University of Oxford, was funded by the JISC Virtual
Research Environment programme from April 2005 to March 2007. It involved three departments at the
University of Oxford: the Computing Services (through the Research Technologies Service), the
Computing Laboratory, and the Oxford e-Science Centre (now the Oxford e-Research Centre). It also
involved members of the e-Science Centre at the Council for the Central Laboratory of the Research
Councils (CCLRC).
We would like to thank all those who participated in the evaluation: Carina Edwards, Jon Chapman, Gil
Bub, Alex Rowley, Alex Walter, Gary Mirams, Inge van Leeuwen, Helen Byrne, Blanca Rodriguez,
Sharon Lloyd, Rebecca Rowland, and Alan Garny. We would also like to thank Paul Luff, Jeremy
Leighton John, François Grey, and Susheel Varma for their valuable input to the workshop on digital paper,
and for commenting on previous drafts of this evaluation report.