Deliverable (public)

Transcription

Project Number
IST-1999-10375
Project Title
INTELLECT
Document Type
Deliverable
Document Title
Documentation on pilots and usability test
Document Number
D15
Due Date
31/05/2001
Delivery Date
01/03/2002
Document Status
FINAL
Workpackage(s)
WP4
Security
Public
File Name
INT-D15FINAL.doc
Short
Description
Keywords
The deliverable D15 documents experiences gathered during the three INTELLECT
pilots (tasks 4.1, 4.2 and 4.3). Specific points of interest are:
•
Functionality testing
•
Performance benchmarking
•
Usability testing
• Recruitment of pilot users
Pilots, demonstration, testing
Partners owning
BIBA
Partners contributed
All Partners
Made available to
Jorge Gasós (CEC)
INTELLECT
IST-1999-10375
Table of contents
1
Executive Summary.......................................................................................................................... 5
2
Introduction ....................................................................................................................................... 6
3
Definition of Usability Testing ........................................................................................................... 7
3.1
Usability testing in standardisation and literature...................................................................... 7
3.1.1
Further Usability factors ..................................................................................................... 7
3.1.2
Usability Testing as a tool for software development ........................................................ 8
3.1.3
Usability Testing as an evaluation mechanism for existing systems ................................. 8
3.2
3.1.3.1
Speed and Error.......................................................................................................... 9
3.1.3.2
Controllability .............................................................................................................. 9
3.1.3.3
Suitability for the task.................................................................................................. 9
3.1.3.4
Error tolerance ............................................................................................................ 9
3.1.3.5
Subjective assessment ............................................................................................. 10
3.1.3.6
Knowledge type and structure .................................................................................. 10
Usability Testing methods ....................................................................................................... 10
3.2.1
Heuristic evaluation of User Interfaces ............................................................................ 10
3.2.1.1
Approach................................................................................................................... 10
3.2.1.2
Advantages and Disadvantages ............................................................................... 10
3.2.1.3
Applicability to INTELLECT ...................................................................................... 11
3.2.2
Interviews ......................................................................................................................... 11
3.2.2.1
3.2.3
Qualitative Analysis in the Usability Laboratory ............................................................... 12
3.2.3.1
3.2.4
Prompted recall / Critical incident .................................................................................... 13
3.2.4.1
Approach................................................................................................................... 13
3.2.4.2
Advantages and Disadvantages ............................................................................... 13
3.2.4.3
3.2.5
Formal Usability ............................................................................................................... 13
3.2.5.1
3.2.6
Guideline-oriented Evaluation methods ........................................................................... 14
3.2.6.1
4
Pilot User Scenarios ....................................................................................................................... 16
4.1
Current situation ...................................................................................................................... 16
4.1.1
Common elements ........................................................................................................... 16
4.1.2
Blauwerk........................................................................................................................... 16
4.1.3
HiTEC............................................................................................................................... 17
4.1.4
InterSet............................................................................................................................. 18
4.2
Pilot approach.......................................................................................................................... 18
4.2.1
INT-D15FINAL
Common preconditions .................................................................................................... 18
© The INTELLECT consortium – 2002
Page 2 of 51
INTELLECT
IST-1999-10375
4.2.2
Blauwerk Pilot Scenario ................................................................................................... 19
4.2.2.1
eShop........................................................................................................................ 20
4.2.2.2
Configurator .............................................................................................................. 20
4.2.2.3
VR Applet .................................................................................................................. 20
4.2.2.4
Order Processing ...................................................................................................... 20
4.2.2.5
Help Desk Module..................................................................................................... 20
4.2.3
4.2.3.1
eShop........................................................................................................................ 21
4.2.3.2
Configurator .............................................................................................................. 21
4.2.3.3
VR Applet .................................................................................................................. 22
4.2.3.4
Order Processing Module ......................................................................................... 23
4.2.3.5
Help Desk Module..................................................................................................... 23
4.2.4
5
HiTEC Pilot Scenario ....................................................................................................... 20
InterSet Pilot Scenario ..................................................................................................... 23
4.2.4.1
eShop........................................................................................................................ 23
4.2.4.2
Configurator .............................................................................................................. 24
4.2.4.3
VR Applet .................................................................................................................. 24
4.2.4.4
Order Processing Module ......................................................................................... 24
Usability Testing Results ................................................................................................................ 25
5.1
Effectiveness ........................................................................................................................... 25
5.1.1
Prototype platform ............................................................................................................ 25
5.1.2
Prototype architectural components ................................................................................ 28
5.1.3
Prototype functionality in the pilot scenarios.................................................................... 30
5.2
5.1.3.1
INTELLECT eShop ................................................................................................... 30
5.1.3.2
INTELLECT Configurator.......................................................................................... 31
5.1.3.3
INTELLECT VR Applet ............................................................................................. 32
5.1.3.4
INTELLECT Order Processing ................................................................................. 33
5.1.3.5
INTELLECT Help Desk ............................................................................................. 34
Efficiency ................................................................................................................................. 35
5.2.1
eShop performance.......................................................................................................... 35
5.2.1.1
Serving pre-generated HTML pages ........................................................................ 36
5.2.1.2
Serving HTML pages dynamically generated by Cocoon......................................... 36
5.2.1.3
Compiling, executing and serving XSP pages.......................................................... 37
5.2.1.4
Executing and serving pre-compiled XSP pages ..................................................... 37
5.2.1.5
Comparison between URLs from the three INTELLECT pilots ................................ 38
5.2.1.6
Comparison of different servlet engines ................................................................... 40
5.2.2
Configurator performance ................................................................................................ 41
5.2.2.1
Pure Java Implementation ........................................................................................ 42
5.2.2.2
SQL Implementation ................................................................................................. 44
5.2.2.3
SQL vs Java.............................................................................................................. 47
5.2.2.4
Conclusion ................................................................................................................ 48
INT-D15FINAL
Page 3 of 51
INTELLECT
IST-1999-10375
5.2.3
Order Processing performance ........................................................................................ 48
Database transactions .............................................................................................. 48
5.2.3.2
XML parsing.............................................................................................................. 48
5.2.3.3
XSL transformations ................................................................................................. 48
5.2.3.4
XPATH addressing ................................................................................................... 48
5.2.3.5
XML transformations................................................................................................. 48
5.2.3.6
Backoffice functionality ............................................................................................. 49
5.2.3.7
Results ...................................................................................................................... 49
5.2.4
Help Desk performance ................................................................................................... 49
5.2.5
VR Applet performance.................................................................................................... 49
5.3
6
5.2.3.1
Satisfaction .............................................................................................................................. 50
References ..................................................................................................................................... 51
INT-D15FINAL
Page 4 of 51
INTELLECT
IST-1999-10375
1 Executive Summary
This document contains deliverable D15 of the IST project INTELLECT - Intelligent Online Configuration of Products by Customers of Electronic Shop Systems. The objective of the project is to enable
the suitable representation of products including all practicable variants in electronic commerce systems to achieve the most realistic possible visualisation.
The deliverable D15 documents experiences gathered during the three INTELLECT pilots (tasks 4.1,
4.2 and 4.3) in Work Package 4 “Integration of Pilot” in which the INTELLECT system was validated
within realistic business scenarios. The deliverable D15 summarises the pilots as executed based on
the INTELLECT implementation described in D13. D15 has a two-fold focus namely both a technical
as well as a usability-oriented viewpoint. The technical part focuses on issues like performance
benchmarking that tested the efficiency and applicability of the INTELLECT platform. Furthermore the
document includes the strategy for recruiting pilot users, the members of the pilot groups and the
methods chosen for usability testing.
INT-D15FINAL
Page 5 of 51
INTELLECT
IST-1999-10375
2 Introduction
The subject of WP4 was the preparation and execution of the prototype pilot phase and the evaluation
of the systems by means of usability testing. The deliverable D15 – “Documentation on pilots and usability tests” – summarises the activities and results of the WP related to evaluation.
D15 is structured as follows: Chapter 3 “Definition of Usability Testing” offers a quick overview of the
concept of usability testing, the criteria related to such evaluation and the specific methods involved.
Chapter 4 contains a presentation of the three user scenarios that were the basis of each of the three
pilots. This chapter also tries to give a realistic view of scenario related aspects i.e. specific difficulties
and measures taken to overcome them. Chapter 5 describes the INTELLECT prototype used for the
pilots, specific points of interest are the components and functionality of the system. Chapter 5 then
focuses on the results of the usability testing conducted in conjunction with the pilots. According to the
definition of usability testing presented in chapter 3 results are presented in the areas of Effectiveness,
Efficiency and Satisfaction. Effectiveness is primarily concerned with the functionality and general
quality of the software, Efficiency covers the actual applicability of the solution to concrete problems
and Satisfaction covers the vague and subjective area of general user satisfaction with the solution.
INT-D15FINAL
Page 6 of 51
INTELLECT
IST-1999-10375
3 Definition of Usability Testing
Usability testing concerns itself with methods for the measurement of usability. Such methods are
roughly categorised in two groups by literature in the area of ergonomics [1 p.19]. These groups are
methods for usability testing and methods for usability evaluation. Usability testing is seen as a test
that focuses solely on software-ergonomic criteria, while completely disregarding issues related to the
functionality or efficiency of the software at hand. An evaluation of the usability of a system on the
other hand has a broader scope. Here the goal is to evaluate the usefulness of a piece of software for
a certain group of users and a specific function. Both the group of users, and the work which is assigned to them play a major role in this process. This holistic approach allows for more pragmatic and
realistic results that offer substantial insight into the inner workings of the system. We therefore
choose to address a broad range of criteria in the sense of usability evaluation. Usability testing is
therefore to be understood as a procedure for the evaluation of the quality of a software; further quality
criteria would be functionality, efficiency, reliability, portability, testability, etc. Usability testing is used
today both for the optimisation of software during the development process, and for the classification
of existing systems.
3.1 Usability testing in standardisation and literature
Definition according to ISO 9241, Part 11: Usability (Principles), 3.1:
-
“usability: The quality of interaction between a user and other parts of the work system which
is measured by the effectiveness, efficiency and satisfaction with which specified users
achieve specified goals in particular environments.“ [2] According to ISO 9241 usability is a
measure of the quality of the interaction between a user and his/her working environment.
This working environment can be regarded as a socio-technical system. A specific point of interest is the usability of software. The word usability suggests a tool perspective: It is not the
machine, but rather the human users of the software, that are the focal point. [3, p. 177]
Usability as a measurable quantity for the quality of software according to ISO 9241 is therefore split
into three groups of criteria:
-
“effectiveness: The accuracy and completeness with which specified users can achieve
specified goals in particular environments.“ – This criterion represents the functionality offered
by a specific system or the environment in general for the solution of specific problems by a
specified group of users. The amount and type of functions offered by the software is clearly
the main criterion to be considered here.
-
“efficiency: The resources expended in relation to the accuracy and completeness of goals
achieved.” – This criterion tries to define the necessary amount of resources for the fulfilment
of the specified goals. The following equation is suggested as a rough way of demonstrating
the relationship between efficiency, effectiveness and resources.
-
“satisfaction: The comfort and acceptability of the work system to its users and other people
affected by its use.“ – This last criterion directly addresses the one of the hardest areas of
Software Ergonomics, namely the subjective satisfaction derived by a user in his interaction
with the system. This very vague but also extremely important criterion for the success of a
software system can in most cases be addressed only very vaguely and on a purely subjective
manner.
3.1.1 Further Usability factors
“In other words a system is considered more useable if the user can master the system more quickly,
complete standard tasks more swiftly and with more enjoyment, and develop a greater understanding of the workings of the system.” [4]
“The quality of a system depends on it meeting various criteria, which are traditionally vague and nonconstructive in design: usability is ‘the degree’ to which specified users can achieve specified goals in
specified environments, subject to various comfort, effectiveness, efficiency, and acceptability criteria.“ [5]
INT-D15FINAL
Page 7 of 51
INTELLECT
IST-1999-10375
This expanded definition introduces two new criteria missing in the original ISO definition of the term
Usability. These are the cost of personnel training and general transition to the new system as well as
the general transparency of the system. The transition phase from one system to another – be it software, hardware or even non-technical – should be clearly as simple as possible in order to reduce or
avoid additional short term costs for the organization and discomfort for the individual user.
The “increased user understanding” criterion introduced by Briggs [5] on the other hand is a criterion
with long term consequences. Systems that give the user the feeling that they act independently of his
wishes and his commands tend to create a negative working climate that has a considerable impact
on satisfaction and – even more important – on productivity at last.
3.1.2 Usability Testing as a tool for software development
Up-to-date software engineering does not follow anymore the top-down approach which got out of
style during the so-called “software crisis”[7], but uses an iterative proceeding to create software. It is
no longer assumed that at the time of the beginning of a project all information needed to build the
software product is available. During the development process, the present achievements have to be
evaluated and enhanced step-by-step. As early as 1985 the following principles for the creation of
computer systems were established in [7]:
-
“Early Focus on Users and Tasks” - As early as possible the target group and those works,
which users of the system to be developed have to (or want to) execute, should be led into the
focal point of consideration. It is particularly important that the users themselves are integrated
into the development process instead of operating with virtual users, which exist exclusively in
the conception of software developers - and have amazingly similar needs and views to those
of the software developers them selves. INTELLECT followed this approach by executing user
requirements capturing and analysis studies early in the project. WP1 produced extensive material that provided invaluable input all through the duration of the project. Constant communication with the end-users and the whole developer groups allowed the developers to stay upto-date concerning the needs and preferences of the end-users.
-
“Empirical Measurement” - Exactly these users should try out the software during an early
phase of the project. The software can be present e. g. in the form of simulations or prototypes. It has to be taken into account that the trying out the software is recorded and analysed
empirically. Purely subjective decisions are to be avoided. INTELLECT demonstrations were
organised as soon as the first prototype was available. This provided input for constant improvement of the software.
-
“Iterative Design” - If flaws are detected during trying out the software, these are to be removed. In the course of the project the software is gradually improved, as tests are executed
time and again by users. The principle of Empirical Measurement can be enforced with different procedures of Usability Testing. A cost-effective method for evaluating software, wherein
the developer does not come directly into contact with users, but nevertheless can obtain
feedback, was particularly forced by the rapid development in the internet. The unfinished
software is distributed to many volunteers, who discover a multitude of errors. Similarly, the
acknowledgements of users of software already delivered can be analysed for the development of future program versions (“Collect feedback from field use”). The distributed developing
effort behind INTELLECT consisting of four different partners in three European countries and
roughly 15 developers provided a good basis for this sort of testing. Technical flaws were
normally immediately discovered as soon as the flawed version of the software was distributed
to the developing partners. Deployment on the local prototypes of the four partners gave immediate input that allowed the consortium to further improve the software in an iterative development process.
3.1.3 Usability Testing as an evaluation mechanism for existing systems
Numerous enterprises and authorities, which want to introduce new data processing, rely also on procedures of Usability Testing, in order to meet a selection. Obviously development patterns like the
Iterative Development are not suitable for such tasks. In this case, systems are gone through with a
fine-tooth comb, in order to determine the exact status of a set of characteristics. We will try here to
present the most important criteria for the determination of Usability of such systems.
INT-D15FINAL
Page 8 of 51
INTELLECT
IST-1999-10375
3.1.3.1 Speed and Error
Efficiency was recognized as fundamental criterion of all definitions of Usability. The increase of the
productivity of users should be thus a primary target of the system. But how does one determine as
Usability Engineer, which effect the system has on productivity?
A simple measurement of the time needed in order to complete a working cycle, would be the first
step. Obviously the used time alone is no sufficient criterion [4], because it is influenced by various
factors, which usually have little to do with the efficiency of the system. To ensure the validity of the
results, also the number of the errors and unsuccessful attempts must be considered. A formal handling would be quite advantageous in this case (see 3.2.5 Formal Usability).
Chapter 5 offers a detailed overview of the activities executed and results collected in order to fulfil this
criterion for the various performance critical parts of the INTELLECT prototype. I is to be noted that
these activities offered new insight into the inner workings of the various modules that led to further
optimisation during and after the pilots themselves.
3.1.3.2 Controllability
Computer systems are “tools of wit”, and just like conventional tools, computer systems can be used
meaningfully and efficiently only if their mode of operation is mastered by the users. This means that
such a system must have a certain degree of transparency. It must be thus able to make clear for the
user without larger effort how it solves its tasks. Users who achieve a deeper understanding for the
systems they operate every day, have the legitimate feeling that they determine the working process.
Thus, by transparency both the knowledge status of a human and his/her ability to master critical
situations can be improved as well as working morale.
The INTELLECT user interface fulfils two basic criteria of software ergonomics in order to achieve
“controllability” by the user. All INTELLECT functions are presented in a format that is as similar as
possible to that of popular eCommerce system. With this the developers hope to achieve recognition
of said functions by the user thereby enabling the user to re-use existing experience. Furthermore the
INTELLECT concept allows for a simplification of the user interface to the point were the customer is
only confronted with data and functions he really needs and can cope with. In Configuration, being one
of the most complex and difficult to understand (for the user) areas of functionality, the INTELLECT
prototype allows the user to configure the product by simply determining witch pert is to be exchanged
and then offering a list of possible components in return. All such components fit both soft and hard
criteria defined by the user and are thus compatible with the product. This simple interactive interface
eliminates the need for complex handling of the product and its components.
3.1.3.3 Suitability for the task
This point may appear self-evident or even trivial, but it represents a large and real existing problem.
Today's software packages have with few exceptions such extensive functionality that for customers a
realistic estimation is practically impossible. The complexity of these systems together with the complexity of the tasks to be mastered, can easily lead to situations where the customer decides for a
product, or requests one, which does not correspond at all to his/her needs. It is thus the job of a Usability Engineer to determine first of all whether the software is at all suitable for the actual task. INTELLECT addresses this point with a reduction of the functionality presented to the user. Following
the concept “less is more” INTELLECT hides complexity and offers intelligent concepts for aggregating
complex functionality into a few simple controls (see Controllability).
3.1.3.4 Error tolerance
No human is perfect, even the most clever expert will make mistakes in interacting with the computer
– even repeatedly. These more or less important errors are therefore somewhat ordinary and must as
such factor be intercepted by the developers. This means that in the software balancing mechanisms
must be integrated, which can prevent losses by a failure of the user.
All data related to activities in the INTELLECT shop is stored in the INTELLECT database. This allowed the developers to implement a persistent data handling that makes data loss due to user or
system failure highly improbable. Persistent product data stays effectively in the database until deleted
by the administrator.
INT-D15FINAL
Page 9 of 51
INTELLECT
IST-1999-10375
3.1.3.5 Subjective assessment
An investigation, which deals with the evaluation of a man-machine interface, should contain of course
also the subjective opinion of the users. Here has to be differentiated between the individual interpretation of the system and the feeling of personal satisfaction [4]. Of our interest at the moment is the
latter one, i. e., how content the user feels while using the system.
Obviously a high working morale and a pleasant work atmosphere presuppose a smooth interaction
between human and machine. This means again that more energy and creativity can be invested into
work - energy, which was devoured so far by a senseless fight against the own computer.
The objective of INTELLECT is to enable the suitable representation of products including all practicable variants in electronic commerce systems to achieve the most realistic possible visualisation. The
projects goals therefore are inherently oriented towards a user friendly presentation. The simple interactive user interface (see Controllability and Suitability for the task) helps improve the user experience.
3.1.3.6 Knowledge type and structure
Pursuant to current knowledge within the field of the psychology, the human ability to deal with complex devices is based on mental models. Those models contain our whole knowledge over a field and
map as exactly as possible the functionality of the actual matter. With the help of these models we are
able to make decisions and even predict their consequences.
The key to a product of quality excellent is thus the selection of the models, which are to be evoked in
the user. The knowledge structures already available in the software must be concise, understandable, detailed, accurate and above all appealing, so that the developing models can successfully fulfil
their function. (see Controllability and Suitability for the task)
3.2 Usability Testing methods
Procedures of Usability Testing are divided into two categories: techniques concentrating on the machine, and work patterns in which humans come to the fore [4].
Machine oriented methods regard the system and look for the already known weak points as well as
for characteristics, which studies regard as positive. This information results in an overall view on the
software status regarding Usability.
User-centred modes of operation observe the reactions of the user and try to classify his/her performance into areas such as efficiency, satisfaction, knowledge status. These results can then be used to
draw conclusions on the status of the system.
3.2.1 Heuristic evaluation of User Interfaces
3.2.1.1 Approach
The following type of evaluation is called heuristic:
A not necessarily qualified person comes in contact with the product in the course of the investigation
and formulates afterwards a statement about the interface. Ideally, scientific studies and industrial
standards should serve as background for the reports of the correspondents [8].
3.2.1.2 Advantages and Disadvantages
The advantages of the heuristic method are the following: It is cheap, intuitive, easily applicable and
can become part of a development process.
However, it has a large disadvantage: inherent inefficiency. Investigations have shown that a person
testing heuristically an interface will discover 20-50% of the weaknesses or problems [8]. Experience
within the area of Usability Engineering can be useful, but also in this case the results do not look
much better.
INT-D15FINAL
Page 10 of 51
INTELLECT
IST-1999-10375
A further advantage of this method is however that researchers, who operated independently, encounter problems which differ to a large extent (see fig. 1). This means that a group of scientists operating
parallel to each other could achieve a quite high percentage (see fig. 2).
Figure 1 – Distribution of usability problems
Each column of the above table contains problems (represented by squares) which were discovered
by a person. It is obvious that many overlaps exist, however it can be seen clearly that almost all problems emerge at least once
Figure 2 – Performance of groups containing 1 to 30 participants
This table depicts that a relatively small number of persons who operate heuristically can be very successful. An investigation executed by five persons would detect about 70% of all problems. Ten persons would achieve even more than 80% [8].
The number of ten proves as a general highest boundary. It turned out that further coworkers can improve the result only marginally.
3.2.1.3 Applicability to INTELLECT
Heuristic evaluation because of its obvious advantages formed the backbone of the evaluation of the
INTELLECT pilots. “Evaluators” were in this case the end-users working with the software and the
developers consulting them.
3.2.2 Interviews
An obvious possibility of getting information from users about their experiences with a software is to
simply interview them. We differentiate here between three forms of questioning:
INT-D15FINAL
Page 11 of 51
INTELLECT
IST-1999-10375
In a free discussion users are asked what they noticed using the software. This procedure is particularly suitable for the discovery of problems occurring during use..
During a structured questioning the person placing the questions proceeds according to a pattern,
questions can however be answered at will.
If users fill out questionnaires and the possibilities of the responses are limited, a quantitative analysis is simplified.
In the everyday life of programming a questioning is quite frequently used - often even without consciousness for the fact that a Usability Evaluation of the software is done by that. Regarding the participation of users at the development process, or analysis process of a software, questioning may be
the most direct kind of communication between Usability Engineers and “laymen”.
Interviews between developers in the form of “free discussion” as well as “semi structure interviews“
also took place with the limits of the heuristic evaluation presented in the previous section.
3.2.3 Qualitative Analysis in the Usability Laboratory
The performance concept already addressed (at which expenditure can certain work be executed by
certain users) is based on the view that it is meaningful to analyse Usability tests qualitatively.
A qualitative analysis presupposes that tests are executed and statistically analysed. There are different possibilities in which way these tests can be executed; here, we would like to describe possibilities, which exist in Usability Laboratories. In a Usability Laboratory, situations which occur during the
use of the system to be tested, are usually re-enacted, i. e. simulated. At the same time, processes
are logged, which takes place in different manners:
Figure 3 – Usability laboratory (taken from [10])
One procedure for logging man-machine interactions, which can be technically implemented with relative easy, is performed by recording modifications of interfaces, e. g. keyboard, mouse and display
during the use. The larger the liberty is, which the user is given by the input medium, the larger will be
the data quantity resulting from this procedure, and the more difficult will it be to analyse this data - in
some cases it will be impossible. In principle, this procedure is not limited to the application in the Usability Laboratory, it could run as background process any computer. However, this method is at least
INT-D15FINAL
Page 12 of 51
INTELLECT
IST-1999-10375
doubtful regarded from the angle of data security, since easily a profile about the user could be created – after all, it is the software Usability and not the user efficiency to be determined.
A further possibility of analysing Usability qualitatively is to record the process of using a software with
a video camera. Here rather the human and fewer the machine step into the focal point of observation.
Behaviours can be held with the help of the video recording and be shown later either the users or
developers. So problems can be described, resp. insights be supported - e. g. by showing a developer
a 10-minute sequence, in which some poor user desperately tries to quit a word processor.
To take a log is a third possibility of recording user activities. In the Usability Laboratory, this is usually
not performed by software users, but by observers trained in software ergonomics and able to detect
critical situations. Here results can be obtained the fastest, however are they less easy to be given
reasons - humans believe rather what they can see (“You have to show pictures” [9, S.323]).
Tests in Usability laboratory apparently have proved themselves, where they were already used [9,
S.326 ]. A problem is that the human behaviour in the test situation possibly differs from that in working life. Often users feel in such a manner as if it is them and not the machines which they use are
tested, as the users are observed. Thus the users serve in principle solely as measuring instrument for
the quality of a technical system.
If it succeeds to withdraw this feeling from users, then with the help of a Usability Laboratory important
results could be obtained.
The risks and overhead related to this approach led the INTELECT consortium to decide against this
approach. All pilots were conducted on the premises of the end-users in order to profit from the close
proximity to the end-users production environment.
3.2.4 Prompted recall / Critical incident
3.2.4.1 Approach
The two methods treated here concentrate on the research of mental models of the user. Assumed
that critical situations, problem cases and the like are the results of gaps in the user’s model, then we
can interpret these just as well as guideposts, because a knowledge gap of the user can be finally
attributed to a Usability weakness of the system.
In order to retrieve this information, the users are questioned (logs already available can very much
facilitate the work of the Usability Engineer). Target of this interview is to learn as much as possible
experienced about problem cases in which users had difficulties. Those cases are then examined
more exactly to determine the actual flaw in the system.
The other method has a pre-emptive character. The examiners try to simulate a hypothetically critical
situation by imagining the user with various auxiliary means into such a situation. Most different auxiliary means can be used - from simple cardboard representations to realistic simulations of the program. Resulting reactions are analysed afterwards.
3.2.4.2 Advantages and Disadvantages
The weakness of this method, and its strength at the same time, is that it detects only certain - although very serious - problems. It is thus no general procedure, with which Usability value of a system
can be determined.
Critical Incidents during the pilots were reported to the developer partners and proved to be an important source of input for further refinement of the INTELLECT prototype.
3.2.5 Formal Usability
Usability is generally empirical and extremely context specific in nature, but developers could all the
same profit greatly from estimations of formal origin. User interfaces become more and more extenINT-D15FINAL
Page 13 of 51
INTELLECT
IST-1999-10375
sive and complicated. The high measure of complexity and the meanwhile high requirements of the
industry regarding Usability require that developers make more and more critical decisions. In the
ideal case these decisions should be supported by empirical investigations, however is this mostly not
possible out of time and cost reasons. Formal Usability solutions able to automatically make predicates about aspects of the software, could be very useful here.
A system can be abstracted in various ways, yet we preferred to concentrate on one type of representation. Graph-theoretical knowledge is particularly common among computer scientist; this characteristic and the intuitive nature of graphs and trees let appear graphs and trees as particularly suitable for
this purpose.
Each system can be represented as a graph, with the help of a quite simple mapping function. All
states the software can assume are represented as nodes, and all actions which make it possible to
achieve such a state are represented as edges. A weighting of the edges is additionally possible [11].
Algorithms on graphs, e. g. the well-known algorithm of shortest path by Dyjkstra could be applied to
such a graph in order to find the simplest way between two nodes (states) and to determine parallel
also its length. It would be even conceivable to compare several concepts not implemented yet together in this way. A set of questions, which are illustrated on appropriate algorithms, could be used
even as “benchmarks”. Such questions would be e. g. [11]:
•
the user knows nothing, how long will he/she need?
•
the user is confused, how long does he/she need to orientate again?
The data for similar benchmarks could be collected even empirically. Examples:
•
the user calls the on-line help, how long does he/she occupy with it?
•
How extensive is the documentation?
Even due to simple observations, developers could gain a deeper insight into the structure of the system (see picture 4).
Figure 4 – A system displayed as a weighted graph
The simple nature (user friendly, interactive) of the interface presented to the customer made the development of a formalised description of the software (i.e. in the form of a weighted graph) unnecessary.
3.2.6 Guideline-oriented Evaluation methods
With a guideline-oriented evaluation method software can be evaluated by entering certain characteristics of the software into lists, which is usually carried out by experts trained in software ergonomics.
A list of quite superficial test criteria can be taken from the standards DIN 66234-8, or ISO 9241-10;
these leave much room for interpretation. There are several check lists which widely transcend the
requirements of the standards and specify test criteria more exactly (an enumeration of examples is
can be found in [12 p.14]).
INT-D15FINAL
Page 14 of 51
INTELLECT
IST-1999-10375
Figure 5 - Check list, taken from [12]
Often a recommendation regarding the procedure of filling out is attached to check lists, in some
cases even a program that trains the checking person - e. g. EVADIS II [12].
The application of a guideline-oriented evaluation method is narrowed to that extent that already when
creating the lists characteristics of the software to be checked should be known - this procedure does
not measure up to innovative software. Another problematic is the relatively rigid determination of the
quality of a software, which is of course expressed by the sum of the individual characteristics. However, a checking person often gets to know whether software has flaws in it during the determination
of its characteristics.
The material gathered through the various other methods for Usability Testing eliminated the need for
further input through a formalised questionnaire. Co-operation within the INTELLECT pilots in the form
of heuristic evaluation, interviews and prompted recall provided sufficient input for usability testing.
INT-D15FINAL
Page 15 of 51
INTELLECT
IST-1999-10375
4 Pilot User Scenarios
The INTELLECT pilot scenarios were designed to closely resemble the day to day usage situation in
the core business of the individual end-user. These scenarios were therefore based on the current
business models of the end-users. The following sections give a rough overview of said business
cases and then continue to present the resulting pilot scenarios, followed by the prototype functionality
playing a critical role.
4.1 Current situation
4.1.1 Common elements
Blauwerk, HiTEC, and InterSet rely on an extensive network of partners in order to bring their product
to the end customers. These partners offer a diverse amount of services to their customers including
pre-sales consulting, after-sales support, guarantee services, delivery, etc. None of the three INTELLECT end-users is currently in a position to supply these consumer services on a broad basis through
its own infrastructure. This is one of the main reasons why none of the three companies actively compete with local business partners. On the other hand the Internet presents a separate sales venue
aimed at consumers that are not being attended to by any local business partner. These could be
customers living in regions or countries where no such stores currently exist (i.e. new markets or isolated areas) or customers in need of very specific products, configurations or services that can not be
delivered by the local dealers.
Most of the products offered by the INTELLECT end users are highly configurable, some of them even
require for the consumer to possess special skills in order to integrate them in existing configurations.
This characteristic has a number of consequences depending on the development cycle for new products, the nature of the components or products, and the marketing strategies of the specific company.
Nevertheless all of the end users could greatly profit in different ways from a configuration process
simplified through intelligent software and visual representation of the data. Functionality of this nature
would even enable less experienced consumers to produce their own configurations or individualised
variations based on existing models.
4.1.2 Blauwerk
The Sidewalker is a product that does not solve a specific problem or address a need. It is rather a
lifestyle product that aims to entertain the end-user. The company offers a number of products, each
based on a distinctive image (city roller, outdoor model, down-hill model, etc.). Material like the official
catalogue emphasises the special features of each different model and tries to convey a certain feeling in order to capture the customers’ interest and get them personally involved.
Figure 6 - The Blauwerk Sidewalker
Blauwerk’s business model currently heavily depends on a rigid hierarchy composed of distributors
that have the exclusive right to sell Sidewalkers in the region or country they are located and are only
loosely co-ordinated by Blauwerk. These business partners order Sidewalkers directly from the subcontracting plant in Taiwan. The Taiwanese producer then informs Blauwerk accordingly and delivers
the product directly to the distributor. Distributors pay the production fee directly to the producer and a
INT-D15FINAL
Page 16 of 51
INTELLECT
IST-1999-10375
separate royalty fee to Blauwerk. This process is a lengthy one as it requires roughly 2 months for the
production of the Sidewalkers and roughly 3 months for delivery, forcing Blauwerk to concentrate on a
relatively small number of configurations/models. Even local retailers are supported and managed by
the authorised distributor. They don't have the flexibility to offer arbitrary configurations or models to a
private customer. Instead they have to orientate themselves according to the Sidewalkers their distributor has stocked. This distributor is responsible for storage, processing orders, delivery, technical
support, local events, setting local prices, etc.
This structure forces Blauwerk to treat its distributors in many respects as equal partners. Blauwerk
itself has no Sidewalkers stocked and does not handle any orders, so the company has no need for an
inventory control system or a full featured back office solution. Such data, if needed, can be retrieved
from the back offices of the regional distributors.
Blauwerk intends to change this business model to achieve greater flexibility and address a larger part
of the market through more models and more configurations. First and most important goal is to give
retailers, distributors and even private customers equal flexibility in configuring and ordering Sidewalkers. The first step in this direction will be to move production of the product to Europe so as to reduce
delivery costs and time and reduce dependence on the American dollar and the Taiwanese producers.
4.1.3 HiTEC
HiTEC bikes are definite high end products (high performance, high cost) with prices ranging well over
€ 4000. This fact makes them rather exclusive and at the same time more difficult to sell. HiTEC therefore aims at clearly emphasising the advantages stemming from the technical superiority of the product as well as its high degree of configurability and adaptability to individual requirements.
Figure 7 - Typical HiTEC full suspension bike
All products currently offered through the internet site or the printed catalogues are essentially preconfigured, and thus fail to clarify the great number of possible technical variations or the feeling of
individuality that the customer should associate with the product. This as well as the lack of active
support during configuration via the Internet make the process unattractive for less technical or less
informed customers.
HiTEC products are currently sold through distributors, retailers, the internet or on site. Distributors are
generally just larger retailers that are in a position to order larger quantities and supply other smaller
retailers as well. Retailers represent the main distribution channel mainly because they play the role of
a local “service centre” offering a number of important services to the consumer (i. e. maintenance and
repair services). The internet is currently only an alternative market with a rather small turnover consisting of customers that do not have a local dealer or have too advanced requirements. These consumers can directly order from HiTEC after consulting the current WWW site of the company. The
bikes are then delivered via courier service; retailers and distributors ordering larger quantities also get
the product via rail or forwarding agency. The prices offered by HiTEC are always comparable and
most of the time even somewhat higher than the “official” price recommendations handed to the retailers. This strategy is supposed to reduce the danger of HiTEC competing against its own business
partners.
INT-D15FINAL
Page 17 of 51
INTELLECT
IST-1999-10375
All HiTEC bikes are made to measure for each specific customer in the HiTEC workshop in Austria.
Most of the construction is done by HiTEC but some of the tasks are subcontracted. The spare parts
themselves are ordered from many different suppliers world-wide and have varying delivery times. The
back office system employed is rather simple and does not offer sufficient stock control functionality.
Parts of the process are thus handled manually.
4.1.4 InterSet
InterSet is a computer hardware and peripherals wholesaler, its products are largely commodities.
InterSet’s customers are either retailers, franchise stores, chain stores or distributors. Distributors are
generally other wholesalers and are regarded as partners by InterSet.
Retailers and franchise- or chain-stores as a group are basically very similar and represent the main
category of InterSet’s customers. Such stores basically have no stock, they simply order components
or complete systems on demand from InterSet and hand them over to the customer. The exception to
this rule are items roughly up to € 150. These items are normally available locally. Delivery times for
the more expensive products are very short as both InterSet and most of the stores are located in
Athens. Components or peripherals are delivered within one day and completely individually configured systems after a minimum of 2-3 days. Deliveries to the province take longer and are transported
mostly by public transportation buses.
InterSet is very interested in providing assistance to retail partners in any possible way, provided this
helps boost sales of their products. One of the main problems retail stores and ultimately InterSet are
facing is the lack of product awareness. Product awareness and brand recognition is very low in
Greece, the average non-informed customer as a rule doesn’t have any special preference when it
comes to specific labels or models, so the salesperson needs the technical knowledge and motivation
necessary to actively try to convince the consumer that a specific product is better or more desirable
than the rest. This is often a problem as stores generally have barely trained personnel due to the high
staff throughput customary in this branch in Greece.
One of the most prominent characteristics of the computer hardware industry is the extremely short
product life-cycle. Consumer products can generally be considered state of the art only for a few
months up to maybe a year before they get replaced by other even more advanced versions with new
and more attractive features. This forces dealers to invest considerable resources and personnel in a
continuous configuration process that incorporates new components in the existing products. InterSet
plans to reorganise and formalise this process in order to minimise the cost of configuration.
Added value is a very important factor in a branch where price is nearly the only important criterion
distinguishing one supplier from another. InterSet is currently successful with various financing
schemes as a service to its customers for products that cost more than € 150. The company plans to
offer more innovative services making its products more interesting to private customers and enhancing customer loyalty. InterSet as an exclusive importer for some labels has the means to effectively
control the prices and the profit margin on of some of its products. This control has been used to force
a minimal profit margin of 15% for these components and peripherals. This is a strong motive for retailers to support InterSet products in a branch where profit margins as a rule range from 5% to 12%.
InterSet’s back office systems are currently minimal and constitute of manually operated spreadsheets
and a certified accounting software which use is regulated by the Greek government. A specialised
inventory control system does not exist, this functionality is handled by a simple spreadsheet. This
system is not going to be replaced in the foreseeable future.
4.2 Pilot approach
4.2.1 Common preconditions
All of the INTELLECT end users rely heavily on a network of business-to-business partners such as
retailers or distributors. This structure is not likely to change in the foreseeable future because of the
considerable investments the companies would have to undertake in order to offer the same level of
services over large areas that is currently being offered by their business partners. Competing against
these partners is clearly against the best interest of the suppliers. E-Commerce is meant to be an alternative only in cases where the local dealers can not satisfy demand or are simply inexistent. The
INTELLECT business model respects this and therefore retains such major attributes. On the other
INT-D15FINAL
Page 18 of 51
INTELLECT
IST-1999-10375
hand co-operation and support towards retailers and distributors would enable them to more effectively sell the product supplied by the INTELLECT end users. INTELLECT as a world wide accessible
information centre could offer assistance in this respect by offering a state of the art information repository based on current multimedia technology.
The eShop should offer a great deal of information as added value in order to offer better pre- and
after-sales support as well as promote the image and lifestyle the company wants to associate with
their product. Such information would be technical data (data sheets, case studies, reviews, etc.),
usage tips and tricks (new applications, performance tuning, etc.), supporting software (drivers,
demos, screensavers, etc.), multimedia features (animations, music, art, etc.) and much more, depending on the actual product. The consumer should be able with a minimum of effort to get complete
and comprehensible information on all the products that interest him as well as determine the next
available retail outlet. An eShop has the possibility to offer adapted versions of the information to each
customer according to the country or region he is coming from. This feature could be used to adjust
the page’s contents to the specific locale (language, prices, units), to direct interested customers to an
appropriate retail outlet, etc.
An innovative sort of added value would be the 3D configuration support integrated in INTELLECT.
This feature is meant to both assist salespersons in a retail store in producing and ordering correct
individual configurations according to the customers wishes, but also to allow for private customers to
experiment in a user-friendly fashion with new variants on their own. This feature reduces the requirements on the salespeople on one hand, and at the same time involves the consumer personally,
giving him the feeling that he is ordering something individual and special.
Configuration should always be based on an existing model in order to make it easier for the consumer. The basis for the 3D representation should be a 3D wire frame model that represents an idealised form of the product. The software should be able to freely rotate and move this model in 3D
space. The user should then have the possibility to add or change various components to this wire
frame thereby “filling” it piece by piece. The categorisation of components in groups (e. g. Compulsory,
Useful but optional, Luxury) according to their importance would also support less experienced buyers.
Component models should be low detail 3D models of the actual components or of other components
with a similar function. It is not required that every make of every component have its own detailed 3D
model. This process should ultimately lead to the creation of an individual configuration.
The system should offer a great variety of configuration criteria ranging from “hard” factors, e. g. price,
size, weight, performance criteria, to “soft” factors like a certain reputation, image, popularity, etc. Configuration based on non-technical criteria would be a service to consumers interested in an individually
configured product, but lacking the technical knowledge to assemble the configuration on their own.
The system should i.e. be capable of configuring computer systems that are explicitly using the most
“trendy” components or are primarily suitable i.e. for playing games or running scientific simulations,
etc. Other criteria like the system with the fastest CPU under a certain budget or the best possible
system built around a specific graphics card would also be interesting.
4.2.2 Blauwerk Pilot Scenario
From Blauwerk’s perspective, one major aim of INTELLECT is to give retailers, distributors and even
private customers equal flexibility in configuring and ordering Sidewalker scooters. Therefore the
eShop provides functionality to duplicate and extend the functions of the printed Blauwerk catalogue
(which is normally used by salespersons as a starting point when consulting a customer) to serve as
an online multimedia product catalogue for potential customers to explore.
As the company’s goal is to use the internet to showcase their Sidewalker scooter, offer information
and transport the innovative and trendy image to the consumers, INTELLECT offers the possibility to
include information material on the product, as blueprints, posters, pictures, brochures, animations,
etc. Hereby, product components are showcased according to the available amount of such material.
Ordering over the Internet is possible as well. But since most people will not buy such a product over
the Internet without first having actually experienced it, users can, after having been stimulated by the
eShop, be directed to a local shop to try a Sidewalker first hand.
The software is able to interface with the back office systems of all distributors in order to have access
to the necessary data and functions. Business partners however have no means to change the contents or format of the INTELLECT site.
INT-D15FINAL
Page 19 of 51
INTELLECT
IST-1999-10375
4.2.2.1 eShop
The INTELLECT eShop contains a main part controlled by Blauwerk and several other smaller pages,
one for every distributor. The main page is completely localisation-independent (though it can be displayed in various languages) and presents the full palette of Blauwerk’s products as well as the 3D
configurator and 2D/3D multimedia material and information. The prices displayed on this page are the
standard Blauwerk prices. This part of the shop is also able to handle orders from consumers outside
of regions served by a distributor. The eShop of Blauwerk includes the user profiling, configuration of
3D objects, basket, and multi-language support.
For the establishment of the pilot the full pages of Blauwerk have to redesigned from Optinet. That
means, Blauwerk itself was not able to create new HTML pages, because they have no knowledge
about Internet technologies. Therefore, during the pilot phase we had the delay of designing web
pages in HTML and XM/XSL too. Also the navigation structure was not clear from the beginning and
had to developed.
4.2.2.2 Configurator
Blauwerk has a relatively small product palette composed of few parts (up to 10) that can mostly be
combine to create very simple variations. Entering the product data for the scooters into the product
database of the Configurator posed no great difficulty.
4.2.2.3 VR Applet
The VR applet was used in a very simple way, because only one scooter was able to configure with
different components in different colours.
4.2.2.4 Order Processing
Blauwerk had no backend system, that could have been integrated into the order processing module,
because they use proprietary software for accounting and warehousing, that has no open interfaces
that can be accessed via Java.
So we decided to represent the actual handling of an order as a workflow definition in the order processing module. Emails are the medium of communication between the different people involved in the
order handling. Because of the possible loss of an email we implemented a timeout functionality, so
that the order reminds the responsible person after a configurable amount of inactivity.
Blauwerk has to split orders, that contain both configured products and pre-configured products, because the configured products will be assembled in their headquarter in Vienna, whereas preconfigured products can be delivered from a retailer; so Blauwerk can forward the split orders independently to different parties via mail.
Blauwerk had also the requirement to sort the order confirmation emails for the merchant by the country, because Blauwerk wants to subsume orders from the same country into one package and forward
them all to the associated retailer.
4.2.2.5 Help Desk Module
After Blauwerk saw the features of the HiTEC Help Desk implementation, they also requested an
online list of all customers with the information of the online time (see 4.2.3.5).
4.2.3 HiTEC Pilot Scenario
HiTEC’s unique selling point and competitive edge, the high configurability of their high performance
products compared to those of the competitors, is promoted by the eShop. The customers can involve
themselves in the design of a product and thus get the feeling of obtaining additional (even exclusive)
information. This is made possible by extensive configuration functionality allowing the end-user to
produce new variants on the product and to individualise them according to his taste, physical size etc.
These characteristics establish an image of competence. The eShop offers an extensive database of
technical data and specifications as well as multimedia presentations demonstrating the principles and
INT-D15FINAL
Page 20 of 51
INTELLECT
IST-1999-10375
advantages at work behind some of the most interesting components of a bike. These services are
provided free of charge even if the customer is not directly interested in buying.
Since HiTEC’s network of partners consists mainly of a network of retailers that operate largely independently from one another INTELLECT operates as an information centre for potential customers
and a meeting place for consumers that already own HiTEC products.
Resellers can use the eShop to adjust their palette and supplement their collection with new and different models while at the same time ensuring that no dysfunctional product combination is assembled.
4.2.3.1 eShop
Consumers choosing to purchase their bike directly from HiTEC through the eShop receive accurate
information about product availability and delivery times. In order to achieve this, INTELLECT offers
appropriate interfaces to the back-office software HiTEC employs. Components currently not available
are marked as such during configuration and the system presents information concerning delivery
times if the user decides to use them in spite of their unavailability. Orders and payment information
are as well automatically routed through to the back-office system in order to ensure prompt processing. The end user has the possibility to track his order and is automatically informed every time the
order has reached an important stage.
In anticipation of the configuration capabilities, the web-pages of HiTEC was completely redesigned.
The company’s strategy to offer different variants of the same bike was already apparent in the recent
versions of the site, but different components were only shown in text form.
The new web-pages was designed for highlighting the different variations and to show a picture
(maybe later a 3d model) of every key component. There was the goal to provide a clear structure for
the configuration of different bikes based on standard models, which could be upgraded/stripped down
by the prospective buyers.
This aspect was also considered by the organisation of the bike’s components, which was elaborated
for as a basis for configuration module of the INTELLECT e-shop. It had to enhanced from a simple
part list to a multi-level component structure, where each level consists of – interchangeable – parts,
which are combined to modules being the parts for the next level.
HiTEC sells a rather complex product that is composed of a very large number of parts that have
many complex individual properties. Furthermore the actual configuration of the product is accomplished with the help of so called “component groups”. HiTEC has sorted the components comprising
a bike in seven groups. This technique is meant to reduce the amount of possible combinations and
thus simplify the configuration process for both the customer/salesperson and the HiTEC technicians
that are going to build the specific product.
This is a list of the HiTEC component groups:
•
Brakes
•
Saddle
•
Wheels
•
Gears
•
Gear-system
•
Steering-wheel
•
forks/suspension forks
This peculiarity was modelled using the “component list” construct in the INTELLECT Product Data
Model. In essence a HiTEC product does not contain only one list as any other product normally
would. Instead it contains 7, one for each component group. This allows for configuration of the various individual components while still having the possibility of exchanging whole component groups.
INT-D15FINAL
Page 21 of 51
INTELLECT
IST-1999-10375
4.2.3.3 VR Applet
The mountain bikes produced by HiTEC Trading consist of parts purchased from renowned suppliers
like Shimano, Marzocchi, Manitou. When the representatives of these producers were contacted for
the first time the first half of the year 2000 with information about the INTELLECT Project, there were
positive and encouraging reactions. The industry appreciated the potential of promoting high-end
equipment through appealing VR/3D representation. Industry representatives especially stressed the
necessity of a detailed presentation of the product features to justify the higher prices of professional
components.
When the first prototype had to be filled with real data, the first problems were encountered. According
to the producers there were component data available, both in 2d and 3d format. Some of the companies even had rendered product files for simulation purposes. However, the company representatives
were reluctant to give this information to outside parties.
HiTEC contacted suitable producers on numerous occasions:
•
Marzocchi Spa (Bologna/IT) was contacted on the Taipei Cycle Show in April 2002 on management level. The result was, that the company would reconsider their point of view – after a few
weeks an numerous meetings with the Austrian representative, HiTECs request was turned down,
•
A formal meeting with Shimano Europe Gmbh. (the European branch of the world market leader in
gear shifting and brake equipment) took place at the “EuroBike” Fair in Friedrichshafen. Also the
European General Distributor participated in this meeting. Again no decision was taken at this
event, but was postponed for further internal consultations. After a few weeks and two meetings
with the head of the marketing department Mr. Zeilberger only a CD with component images was
sent with an apology that this is all what Japan was ready to distribute.
•
For the negotiation with Answer Products USA (Manitou etc.) HiTEC was in a better position, being the general representative of the company in Austria. The first request for data, with which 3d
models could be generated, was communicated to Answer at the Taipei fair of 2001. Through the
regular contact with the European Management the headquarters in the States was pushed even
more. Nevertheless the request was rejected. Only after an intervention in a personal meeting with
Glen Miller, Answer’s president an assent could achieved. It took until 11/2001 until one file with
the construction data of a 2 two year old fork reached HiTEC.
The situation is maybe bets explained by a mail sent to HiTEC by Manitou Inc. during the negotiation
phase, in which the development department explains that “.... any competitor would be able to use
the data for copying our product to the very detail. It even would be able to feed the information to
production machines and get a bootleg product within a few hours”.
The lacking co-operation from the suppliers was a major drawback for the overall goals of HiTEC and
endangered the whole strategy of bike configuration via the Internet.
As a consequence HiTEC Trading tried to get 3D data from other sources. First there were attempts to
capture construction data by copying them by a CAD system. HiTEC technicians, though experienced
with system used hours of drawing time without promising results. 2D construction plans could be
generated quite easily, because it was similar to the development work being done for the components designed by HiTEC personnel. When it came to adding the third dimension to these drawings,
the problems could not be overcome. Firstly an abundance of additional data would have been necessary to give in. Secondly, additional program components would have been necessary together with
lengthy qualification measures for the employees involved.
All in all, no economic way of generating 3D in-house could be found. So, contacts with some 3rd
party suppliers of rendering services were established. But very quickly it became apparent, that outsourcing the production of 3D models of existing parts would also become a very expensive task.
There were bids for creating a detailed 3D model of key parts like gear shifts components at about
3.000 Euro, which would rise the price of a fully equipped VR-mountain bike summed to over 20.000
euro.
As a potential alternative, there were contacts to a company providing 3D scanning services. But also
in this case, the cost was way too high for a economic production of the necessary data (app. €5000
per hour including personnel, with the potential outcome of 1-2 parts per hour). Purchasing a 3D
Scanner in order to perform the operations within the consortium also turned out to be impractical. A
scanner of sufficient quality costs roughly €2500 to €5000. Furthermore Atlantide the partner responINT-D15FINAL
Page 22 of 51
INTELLECT
IST-1999-10375
sible for the VR Applet made clear that they do not posses the required expertise for using such a
machine to create the required material.
Also within the consortium there were attempts to generate the necessary data for the visualisation
module using. Existing blueprints were transferred to CAD systems and tested by developer partners
for their usability as source material for rendering. Such material also proved inadequate as it only
contains data of two out of three dimension on the components. Three dimensional blueprints as previously described could not be obtained from suppliers.
4.2.3.4 Order Processing Module
Also HiTEC had no back-office system that could have been integrated into the order processing
module. Similar to the Blauwerk prototype we based the workflow of HiTEC on emails that give the
responsible person the possibility to affect the further routing of the order. We defined different roles in
the workflow, that perform the different tasks of verifying, preparing, assembling and delivering the
ordered products. The timeout feature for orders, that reminds the responsible person via email after a
specified time of inactivity was also highly appreciated by HiTEC.
The main adaptation on the HiTEC order workflow was the insertion of “parts lists”. HiTEC provided us
with a database in dBase format, that contained parts information that correspond to the component in
HiTEC’s warehouse. We implemented a business logic component that enhances the existing order
with this parts list data. Every entry for a component gets a subordinate data structure, that describes
the parts of the component. After adapting the stylesheets for email and document generation the
parts list information was visible in emails for the assemblers and assembler documents.
An additional kind of business logic component was implemented to have a look at dynamically generated order documents in a browser. The workflow can send an email, that contains a link to the
server generated document.
4.2.3.5 Help Desk Module
HiTEC wished to have a list of all customers in the shop. They liked too see the name and the time,
that the customer has already spent in the eShop. With this information a help desk agent can efficiently select a customer that has been spending rather much time in the shop without buying something. This could be easily implemented via the Help Desk integration with the eShop module.
4.2.4 InterSet Pilot Scenario
The interest of InterSet is to make the functionality of INTELLECT available to all its retail business
partners in the form of a “mini-shop”, giving them also extensive administrative privileges to individually configure their own separate INTELLECT pages. Every InterSet partner that wishes to use INTELLECT in his store is given the freedom to configure elements like his logo, the design of the page,
the prices, etc. Stores that wish to offer the “mini-shop” as a fully-fledged internet shop have to provide
their own payment mechanisms.
Through INTELLECT, InterSet offers many forms of information-oriented added value. Many of these
services are meant to simply give customers easy access to data that is either not available to nonprofessionals or simply difficult to get and complicated to understand, as e. g. technical data and
specifications on components and peripherals, including “dealer only” tips and tricks, relevant information on how to combine components effectively, performance tips on how to achieve best results, case
studies for many components, etc. The system also offers a well sorted library of official material including drivers, BIOS upgrades, brochures, manuals and other paraphernalia normally offered by the
original manufacturer.
InterSet wishes to fully base its internal operations on INTELLECT by exclusively using the system as
means for producing new configurations but also as a repository for current models. The company
expects to free up valuable resources that are currently fully occupied with such tasks.
4.2.4.1 eShop
InterSet adapted the eShop perfectly on their new web pages. They created a new design and a
XML/XSL format. In agreement with OptiNet they set new buttons and functionalities into the existing
INT-D15FINAL
Page 23 of 51
INTELLECT
IST-1999-10375
web pages. Therefore they are using multi-language support, user profiling, basket, and the shopping
cart.
The main difficulty faced by the Configurator during the InterSet pilot was the fact, that InterSet was
not happy with the requirement formulated early in the project concerning the modus operandi of configuration. Instead of simply offering the user the possibility to exchange components InterSet required
the possibility of adding and removing entirely new components. This requirements violated the basic
assumption that a product stays viable as long as every part exchanged is replaced by one with the
same features and interfaces. Adding this functionality required major changes in the configuration
strategy followed by the Configurator. The module was extended in a way that it allows to “know” to
which components each component is actually connected, thereby giving it the possibility of directly
removing or adding individual components.
4.2.4.3 VR Applet
The VR Applet has been used in this environment at a relative short time. On the one hand abstract
3D objects are available to present different computer types, and on the other hand, InterSet created
on her self 3D objects for the visualisation.
4.2.4.4 Order Processing Module
Because Interset planned to change their proprietary back-office system to a standard solution from
SAP, the implementation of the order processing module for Interset was postponed; up to now, Interset did not carry through the switch because of company internal problems.
Because Anecon has the know-how of Java/SAP integration from a former customer project, this
would have been the chance to test the integration of a real world back-office system with the order
processing module.
The set of business logic components that already was implemented for the Blauwerk and the HiTEC
pilot sufficed the needs of Interset.
INT-D15FINAL
Page 24 of 51
INTELLECT
IST-1999-10375
5 Usability Testing Results
Chapter 5 presents the findings of the evaluation activities conducted in preparation, during and after
the INTELLECT pilots according to the usability testing criteria presented in chapter 3. The first section
“effectives” directly addresses “The accuracy and completeness with which specified users can
achieve specified goals in particular environments.“ [2] For this we present the functionality offered by
INTELLECT in relation to the tasks set for each pilot followed by concrete technical data on benchmarks demonstrating the efficiency of the INTELLECT solution. The second section “efficiency” addresses “The resources expended in relation to the accuracy and completeness of goals achieved.” [2]
For this we match goals identified during the user requirements phase of the project against the functionality listed in the previous section as it was tested in the pilots in order to demonstrate “the accuracy and completeness of goals achieved”. The last section contains findings in the area of satisfaction and describes experiences of the end-users during the INTELLECT pilot phase.
5.1 Effectiveness
This criterion represents the functionality offered by the INTELLECT pilot within the three pilot scenarios formulated after the requirements of the three end-users. The first criterion to be considered is the
amount and type of functions offered by the software followed by detailed performance data on all
relevant parts of the INTELLECT prototype. Benchmark results presented here are not related to a
specific pilot instead they apply to all scenarios.
The subject of this section is to summarise the prototype-related aspects of the pilot scenario. The
INTELLECT prototype aims to provide an enhanced solution including all practicable electronic commerce features, and offering a qualitative representation of the product based on available VRML material. The following paragraphs present a rough overview of the components comprising the INTELLECT prototype. We then proceed to present the architecture of the prototype scenario of the INTELLECT pilot network.
5.1.1 Prototype platform
The INTELLECT prototype is based around the INTELLECT Application Server the software responsible for hosting the shops for the three INTELLECT pilots (see Figure 8). This server consists of a
group of Open Source Java based products that manage all INTELLECT Enterprise Java Beans and
Servlets as well as handle the XML files of the various Shops, generate HTML pages and serve them
to the client. The exact components of the Application Server and their specific function are described
in the following paragraphs. The INTELLECT Application Server used during the Pilots was installed
on a machine with permanent Internet access at BIBA. This gave the developers and the end-users
the possibility of updating and working with the system on a 24/7 basis. All three INTELLECT pilots
were installed on the same prototype server and were realised using the same database and application Server.
Producer
Internet
Helpdesk
Helpdesk
Client
INTELLECT
Application
Server
VR
VR
Client
Client
DB
Figure 8 – INTELLECT Pilot Topology
INT-D15FINAL
Page 25 of 51
INTELLECT
IST-1999-10375
The actual components of the prototype were the following:
•
Debian GNU/Linux Version 2.2 (potato)
The operating system employed as the main development and testing platform was Debian
GNU/Linux. The System was chosen for its performance and flexibility as well as popularity in
ASP/ISP circles targeted by the INTELLECT exploitation strategy. Regular tests on Microsoft
Windows NT4 and Windows 2000 have revealed no interoperability issues.
•
Oracle Enterprise Edition Version 8.1.7.
Oracle Enterprise Edition for Linux (http://www.oracle.com) is a best of class RDBMS providing efficient, reliable, secure data management for high-end applications such as high volume
on-line transaction processing (OLTP) environments, query-intensive data warehouses, and
demanding Internet applications. It should be emphasized, that INTELLECT uses none of the
proprietary Oracle extensions and can be ported to a different relational database (i.e. PostgreSQL) with a minimum of effort.
DB2
DB
Oracle
DB
Postgres
DB
Database
EJB Application
Server
jBoss
IPlanet
Apache/Tomcat
Jetty
Internet
Web/Servlet/JSP
Server
WWW
Figure 9 – INTELLECT Application Server
•
Apache Software Foundation Jakarta Tomcat Version 3.2.1.
Tomcat (The Apache Jakarta Project http://jakarta.apache.org/) is a servlet container and
JavaServer Pages(tm) implementation. It may be used stand alone, or in conjunction with
several popular web servers (Apache HTTPd, Microsoft Internet Information Server, Microsoft
Personal Web Server, Netscape Enterprise Server, etc.)
•
Apache Software Foundation Cocoon Version 1.8.2
Cocoon (The Apache XML Project http://XML.apache.org/) is a 100% pure Java publishing
framework that relies on new W3C technologies (such as XML and XSL) to provide web content. Apache Cocoon is an XML publishing framework that raises the usage of XML and XSLT
technologies for server applications to a new level. Cocoon interacts with most data sources,
including: filesystems, RDBMS, LDAP, native XML databases, and network-based data
sources. It adapts content delivery to the capabilities of different devices like HTML, WML,
PDF, SVG, RTF just to name a few. Cocoon is integrated currently as a Servlet. Specific features of interest are:
INT-D15FINAL
o
Database connection pooling
o
Servlet 2.0 spec compatibility
o
Good performance for heavy load conditions
o
High speed for XSP pages which include complex nodes
o
IBM Jikes Java compiler support for faster XSP page compilation
Page 26 of 51
INTELLECT
•
IST-1999-10375
jBoss Version 2.0
JBoss (The jBoss Project http://www.jboss.org) is an Open Source, standards-compliant,
J2EE application server implemented in 100% Pure Java. jBoss 2.0 is licensed under the
LGPL license. In our case jBoss 2.0 is working as an EJB-server.
According to the INTELLECT requirements analysis the requirements for the client should as low as
possible. Any current browser supporting Java either natively or with the Sun Java pluggin can display
the INTELLECT Shop-pages and make use of all the 3D presentation and configuration functions. All
testing during development was conducted with Microsoft Internet Explorer 5.5 and Netscape 6.0.
Testing during the pilot phase was based mainly on Internet Explorer.
The facilities for running the prototype INTELLECT marketplace during the three INTELLECT pilots
were provided by BIBA. The task of BIBA under the guidance of Anecon (being responsible for the
demonstration platform) as marketplace operator was to document and to solve problems that may
arise if the server is 24 hours per day in operation and has to answer the concurrent requests of buyers and sellers accessing the server. The experiences gained by running the marketplace will be used
as input for further refinement and debugging of the overall system.
The users of the INTELLECT marketplace mainly comprise the end users of the INTELLECT consortium as well as users that accessed the INTELLECT prototype via the link provided by the INTELLECT
Internet Web-Site. All users interested in configuring and buying products through INTELLECT needed
to access the marketplace by using a common browser (e.g. MS Internet Explorer or Netscape Communicator) including the Java 3D Java extension Library. Customers were provided with appropriate
installation instructions through the INTELLECT Web-Site.
Companies aiming to products through INTELLECT have to make use of the administrative tools developed by the INTELECT developer partners in order to enter new product data into the INTELLECT
database. This process was heavily supported by the INTELLECT developers as it was seen as too
complex for end-users with scarce technical experience to handle.
The main task performed by the INTELLECT Marketplace is to transform the entered product information into a browsable product catalogue containing configurable products. Such products can be ordered thereby generating orders in a format compatible to the back-office and workflow system of the
end-user (producer/distributor) systems. Support is provided through am online help-desk system that
offers video-conferencing and CMCW functionality.
INT-D15FINAL
Page 27 of 51
INTELLECT
IST-1999-10375
5.1.2 Prototype architectural components
The INTELLECT prototype is composed of five modules:
•
The e-shop as the e-shop as the main interface visible to the customer supplies the general
infrastructure of an e-commerce system (basket, catalogue, secure communications, localisation, multimedia presentations, etc.).
•
The Virtual Reality module provides a 3D visualisation of the product, and enables the customer to configure its own product by choosing each of its components.
•
The Configurator supports the generation of product variants as well as implements a front
end to the database thereby offering infrastructure for the administration of products and components.
•
The help-desk provides on-line help to the customer using features like page mirroring, voice
over IP, and video conferencing.
•
The order-processing module as a front end to the back-office system manages the orders,
and offers an interface between the INTELLECT application, and the back office systems.
Client
Client
Distributor
Internet
Support
Helpdesk
Helpdesk
Backoffice
INTELLECT
Configurator
VR
VR
Producer
Client
DB
E-Shop
DB
Ordering
Client
Supplier
DB
Client
Figure 10 - INTELLECT Module architecture
The core modules use a common product database for all product and customer data. The database
contains all product and order related information partitioned into separate areas one for each module.
It is populated and updated by using information already available in the supplier systems. This information is updated on-line using standards for the exchange of product data (XML, SOAP). In addition,
a database of multimedia data (volumes models, sounds, videos, textures, etc.) exists on the HTTP
Server serving the shop pages and is used for product demonstrations within the product catalogue of
the eShop.
Furthermore the Help Desk and VR modules offer extended functionality that is implemented with
software running on the customers computer. This software consists mainly of the browser and a Java
Virtual Machine (JVM). In order to make use of the Videoconferencing and CM/CW features of Intellect a H.323 compatible client (i.e. Microsoft Netmeeting) is required. The VR modules being the main
front-end to the Configurator offers:
•
Full VR navigation functionality based on VRML scene description data and Java3D 3D graphics technology
•
Individual configuration of the viewed objects
•
Textual and audio visual explanations
•
The possibility of defining preferences that influence configuration
INT-D15FINAL
Page 28 of 51
INTELLECT
IST-1999-10375
If the customer requires further assistance or simply wishes to speak to a salesperson a video or audio conference session with the company's hot line or call centre can be initiated via the Help-desk
module. The Help-Desk provides a direct link between the browser of the customer and the browser of
the salesperson thereby making it possible for the salesperson to view the same configuration the
customer is working on. The salesperson also has the possibility to take part in the configuration process and actively change the product variant in order to help the customer.
The electronic shop module finally takes the order of the customer over a secure connection via the
Internet. At this time, a human employee could carry out a spot check of the order to confirm the correctness of the order. An acknowledgement with complete details of the order is then returned automatically to the customer.
Customer
BROWSER
Web page
Cocoon
Servlets/
JSP‘s
EJB‘s
3D Applet
HD Applet
DB
Netmeeting
H.323
Helpdesk
BROWSER
Web page
3D Applet
HD Applet
Netmeeting
H.323
Figure 11 - INTELLECT EJB/Servlet Architecture
All INTELLECT Server modules are implemented as Java Enterprise Java Beans (EJB) that are installed on one or more EJB compliant application servers. It is important to note, that the various
beans can function separately from one another. This allows for a distributed setup with some beans
being installed on the server of an ISP and some being only partially online on the producers or distributors site. A further possibility is the use of more than one bean of a certain type (i.e. ordering) in
order to facilitate the integration of distributed back-office infrastructures.
All communication between the beans and the customers browser is filtered through appropriate Java
Servlets that serialize Java objects and tunnel them through a standard communications port (TCP
Port 80) used for HTTP transactions. This allows the use of INTELLECT through any standard type of
firewall or packet-filter employed by the customer.
A more detailed view of the INTELLECT Modules is presented in “D13: Pilot Integration Documentation and Handbook”.
INT-D15FINAL
Page 29 of 51
INTELLECT
IST-1999-10375
5.1.3 Prototype functionality in the pilot scenarios
The main functionality of the marketplace prototype to be demonstrated and tested during the pilot
phase is summarised in the following sections.
5.1.3.1 INTELLECT eShop
Extensible Server Pages (XSP) are XML pages with Java code in them. In other words it is possible to
put whole programs written in Java into an XSP page in order to execute a Java programme that generates dynamic content. XSP combines therefore the functionality offered by Java with the modularity
and transparency of XML. The result of this process is at the end an XML page. This allows the handling of XSP generated pages like any other XML data stream, e.g. by applying stylesheets to it.
Unlike other technologies like JSP (JavaServer Pages), PHP (PHP: Hypertext Preprocessor) or ASP
(Advanced Server Pages) which also produce dynamic pages are not as flexible as XSP directly generates XML documents that can be easily transformed using standard XML techniques into an arbitrary format or medium. The separation of the documents structure, content and layout, i.e. the way
the document finally appears allows for a multitude of possibilities. The same page could e.g. be presented as web page (HTML) or a PDF document with only a different XML processing component.
This degree of separation of semantics from structure makes it easy for different kinds of specialists to
work together in order to maximize efficiency. Somebody who's good at designing data structures can
do this, while somebody else produces content using the structures created by the first one. Finally the
third person does the layout for the data.
In the INTELLECT project these advantages are used to build the shop GUI in a modular fashion. One
can think of these modules like objects which can be combined in a specified way according to a template to meet the needs of the specific shop. Let's say for instance you want to build a shop, that has a
catalogue, a shopping cart and a Help Desk. Then you simply plug together these modules and you
have what you want. Because of the separation of structure, content and style using these three modules you can build many variations of the same shop which will all provide the same functionality but
will all have different content and look different. This feature was used to produce the shop pages for
the various pilots on the same prototype application server and can be furthermore used to easily and
efficiently produce “mini-shops” for various retailers as variation of the main shop of i.e. a wholesaler
(like InterSet).
Multimedia-object
Navigation-object
MasterDetailsView-object
Navigation:
NEWSFLASH:
MainMenu
Headline
MainMenu
MainMenu
New Bike Race in Vienna
Special Brakes available
Bla Bla Bla
Tralalalalala
MainMenu
SubMenu
SubMenu
MainMenu
MainMenu
Date
LOGO 2
1.7.00
2.6.00
1.6.00
9.5.00
Click on the News article to see details below!
LOGO 1
SubMenu
SubMenu
Location
Vienna
Oslo
Rennes
Bremen
Selected News article:
New Bike Race in Vienna
Picture 2
Vienna, 1.7.00:
Bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla
bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla bla
bla bla
Order Tracking
You already ordered
products from us
and want to know, if
it has already been
delivered?
Log in here:
User:
Password:
Special offer of this week:
Get the new Shimano Break for only $ 245,-!!!!
Click here to get details....
Multimedia-object Offer-object
Picture 1
Forgot your
password? Click
here!
OrderTracking-object
Figure 12 – Object oriented shop-GUI design
INT-D15FINAL
Page 30 of 51
INTELLECT
IST-1999-10375
5.1.3.2 INTELLECT Configurator
Configuration functionality in these scenarios reduces the workload on the salespeople on one hand,
and involves the consumer personally giving him the feeling that he is ordering something individual
and special on the other. The use of such functionality in an e-commerce context and the resulting
requirements impose restrictions, but also offer new possibilities concerning the implementation of
existing configuration methods. A short overview of these requirements is presented here:
•
Interactivity: The customer must be able to interact with the e-commerce System.
The aim is to allow him to construct the product piece by piece according to his own
preferences and wishes thereby giving the customer the feeling that he is a part of the
construction process of an individual product.
•
Multimedia: The customer must be able to receive visual feedback of his configuration efforts.
•
Product independence: The Configurator must be in a position to handle arbitrary
products. Main issues here are the handling of the attributes (the quantity and type of
the components attributes should be handled dynamically) and the components of the
product (a product’s composition should not be restricted by any non configuration related issues).
•
Performance: The system must be able to handle large numbers of concurrent configuration requests. This implies a scalable and simplified architecture that can operate with a minimum of product data in order to ensure acceptable performance and
make efficient use of bandwidth.
•
Soft criteria: The use of a Configurator in an e-commerce environment imposes additional requirements on usability and user friendliness. Configuration should not restrict
itself on mechanical or “hard” criteria, but should also consider soft criteria expressed
by the customer. Relevant configuration criteria are “hard” factors like whether a component fits in an existing product as well as “soft” factors like personal preferences of
the user as well as other non operability related information (i.e. price, size, weight,
performance criteria, image, popularity, etc.).
Hard
Categories
Soft
Categories
Component
Types
Components
HardDisks
Mainboards
AVEquipment
High-End
Low-End
IBM
DCAS3440
Maxtor
4567
IBM
DORS3240
U2W, 8 GB,
5400rpm
U, 4 GB,
5400rpm
UW, 2 GB,
5400rpm
Figure 13 – INTELLECT Product Data Model
INT-D15FINAL
Page 31 of 51
INTELLECT
IST-1999-10375
5.1.3.3 INTELLECT VR Applet
The figures in this section show the client VR applet in action. The customer sees in the main window
(see Figure 14) the pre-defined product he has chosen in the eShop, and is able to interact with it and
manipulate it in various ways.
Three modes are available for the user: Translate, Rotate and Interact. The translate mode allows the
user to use the mouse to translate the object in the current visualisation plan, the rotation mode allows
the user to rotate the object with the mouse, and the interaction mode disables navigation with the
mouse (but not the other navigation means), so that the user can use the mouse to interact with the
scene (select an anchor or a component, or move an anchor).
To make it easier for the user who may not be used to use the mouse for moving objects, the application calculates a list of predefined ones (Face, Back, Right, Left, Up, Down), using rotations from the
initial viewpoint. Another viewpoint allows the user to see all the objects of the scene.
The user also has the possibility to zoom in or out with the Zoom functionality. He can give directly the
zoom value, increment or decrement the existing one, or reset the zoom. All these navigation possibilities allow the user to view the object in several points of view even if he isn’t very used to navigation,
and to move objects with the mouse, which may seem difficult for a novice, but which bring the sensation of manipulating the object.
Configuration of the product is accomplished as follows. The user can select the component he wants
to modify in the components list or in the viewing area, if he is in the interaction mode, and if the 3D
model is available. The customer indicates then, that he wants to replace this component, thereby
opening the component exchange dialog (see Figure 15).
Figure 14 – Client VR configuration applet
The component exchange dialog displays
the list of possible components that can
replace the selected one. This component
list is generated by the Configurator from
the product database and represents the
subset of available components that will fit
into the product physically and mechanically and at the same time meet all the
“soft” criteria specified for this product, or
manually specified by the user. Selecting a
component in this list shows a 3D view of
the component (if available) and its technical description (see Figure 15).
Figure 15 – Component exchange
dialog
INT-D15FINAL
Page 32 of 51
INTELLECT
IST-1999-10375
5.1.3.4 INTELLECT Order Processing
The Ordering Module is an XML-based workflow environment. Anecon tried to implement as much as
possible of the back-office processing as XML transformations, as this is the most open and flexible
approach to handle data up to now. The following points were the main arguments to base the order
processing module onto XML documents:
•
Most vendors of backoffice systems provide an XML interface, so there is little integration development necessary.
•
Logic that is implemented in XML/XSLT is easier to port to other programming environments than
logic that is implemented as code. So this module is a better investment for future reusability, if
Anecon will implement a similar product based on Microsoft’s .NET framework.
•
For remote access “soap” offers simple possibilities to send and receive XML content.
•
There are XML front-ends to all significant databases
The eShop provides an order as an XML document, that conforms to a DTD that was defined in cooperation with OptiNet. To stay completely independent from the structure of the document, which is
subject to change, the relevant information within the document is addressed via XPath expressions,
that can be configured in a configuration file. The business processes, that are initiated by the order
processing module get their information also via configurable XPath expressions, so this module is
very flexible in respect to the structure of the XML document and is therefore not restricted to the use
within shop environments, but can handle every kind of objects, that have a XML representation. This
greatly improves reusability.
The adaptation of the order processing module to a specific shop environment consists mainly in the
definition of a DTD and the related XPath expressions; additionally the end-user specific business
logic has to be implemented. This can be accomplished either by reusing and configuring existing
business logic components or by implementing new business logic components. A programmer implementing a new business logic component can reuse the a library of XML utilities and business logic
component base classes, that reduce the development process to implementing only the bare business functionality. If external systems have to be integrated, of course a respective library must be
supplied.
The following business logic components are already implemented by the module:
Email with content generated from the order data
Email with the possibility for the user to affect the further routing of the order
Splitting of an order
Parts list generation
Document generation out of the content of the order data
Conditional routing of the order depending on order data
INT-D15FINAL
Page 33 of 51
INTELLECT
IST-1999-10375
5.1.3.5 INTELLECT Help Desk
The Help Desk module had the problem, that it could not be tested with the integrated Tomcat 4 until
beginning of 2002. The jBoss group needed nearly a year since the appearance of the first version of
tomcat 4 until they integrated it into their application server. Tomcat 4 implements servlet spec 2.3,
which introduces the concept of servlet filters. Servlet filters are the basis for the Help Desk module,
they provide a transparent way to intercept the browser communication with a web application which is
vital for the implementation of the page mirroring functionality.
So we developed the Help desk module independent of jBoss with the available beta versions of Tomcat 4 and tested it with other web applications developed by Anecon. Tests of compatibility with the
eShop module and the virtual reality module had to be postponed.
Our recent tests with the available jBoss/Tomcat4 bundle showed that the integration is still not finished completely, the virtual reality applet does not function properly, because no static contexts can
be defined in Tomcat until now, and the VR module depends on this feature. The current status for the
help desk module is, that it does not function properly with the virtual reality module.
The parts of the help desk module, that could already be integrated into the recent jBoss/Tomcat4
bundle are:
Voice and Chat integration
Video integration
Online user list
Page mirroring
Mirroring direction change (customer’s browser is controlled by help desk agent)
Because there was not much time left for tests and further development, the help desk module has not
the production quality properties like the other modules.
INT-D15FINAL
Page 34 of 51
INTELLECT
IST-1999-10375
5.2 Efficiency
The resources expended in relation to the accuracy and completeness of goals achieved.” - This criterion tries to define the necessary amount of resources for the fulfilment of the specified goals.
5.2.1 eShop performance
The INTELLECT eShop benchmarks focused in the measurement of the performance related to the
generation of dynamic web-pages based on an XML infrastructure this being the technically most challenging task as well as the most interesting feature in terms of innovation for the end-users. The two
Servers involved in this process are Cocoon and Tomcat.
The role of Tomcat is to initialize the Cocoon Servlet, and to receive the HTTP Request and pass it on
to the Cocoon Servlet. On reception of a HTTP Request, Tomcat calls the Cocoon Servlet.service(HttpRequest, HttpResponse) method. When the Cocoon Servlet gets a HTTP Request
from the servlet engine, it sets up an Environment (a HttpEnvironment in this case) and passes that to
Cocoon. The Environment consists of a Request, Response, and some servlet info (such as requested URI and the servlet's path). This Cocoon object lets the Environment decide which sitemap to
use, and passes the sitemap filename along with the Environment to a Manager. This one puts a
Handler to work: it checks whether there already exists a Handler with a compiled version of the sitemap. If not, it creates one.
In order to test this functionality meaningfully four scenarios were identified:
•
Serving pre-generated HTML pages
o
•
Serving HTML pages dynamically generated by Cocoon
o
•
XML pages without logic i.e. no XSP pages, this is a pure conversion of XML and XSL
to HTML
Compiling, executing and serving XSP pages
o
•
Plain HTML pages z.B. www.ist-intellect.com
Tomcat executes the Cocoon Servlet, Cocoon compiles the eShop servlet
Executing and serving pre-compiled XSP pages
o
Tomcat as a Servlet engine executing the Cocoon servlet
In a realistic scenario the application server will generally generate all required pages in a relatively
short time depending on the traffic the site is subjected to. Effectively all pages both HTML pages and
Servlets will be pre-generated resp. pre-compiled. Therefore it is safe to assume that compilation or
generation time will normally not influence the performance of the site. In the following paragraphs will
we nevertheless present data on both serving pre-generated material and on serving material that is
being generated on-line.
In order to evaluate the INTELLECT eShop prototype a traffic simulation tool called “Web Stress” was
used (http://www.paessler.com/WebStress/webstress.htm). This tool is an HTTP client that offers the
possibility of starting multiple requests at the same time in order to simulate a situation of heavy traffic.
The tool also allows for extensive configuration of the behaviour of the simulated client and compile
and generate statistics describing the response of the server. Each test is configured to run for one
minute generating requests through 5 parallel clients.
All benchmarks were carried out on the INTELLECT on-line prototype server. This machine is a dual
Pentium II, 400MHz with 256 MB RAM and SCSI disk subsystem. The operating system was Debian
GNU/Linux 2.2 (potato) with Linux Kernel 2.2.18. All clients were connected via LAN to the server.
INT-D15FINAL
Page 35 of 51
INTELLECT
IST-1999-10375
5.2.1.1 Serving pre-generated HTML pages
In this benchmark the client is accessing plain HTML files and Tomcat is working as a plain Webserver not doing any processing of the files served. Each test is configured to run for one minute generating requests through 5 parallel clients. The request times ranging from 100ms to 250ms with an
average request time of roughly 150ms are somewhat high for a plain HTTP-Server, but nevertheless
acceptable. Tomcat answered successfully all requests with no errors. The occasional spikes in the
performance of the server can only be explained by internal Java Virtual Machine (JVM) fluctuations
that are not visible to the Java programme or the operating system.
Figure 16 –Request times for a single plain HTML URL
Figure 17 –User wait time and users per second
Figure 17 is a different view on the same test and demonstrates the effectiveness of Tomcat in terms
of the total number of users that can be served at the same time before the server starts losing requests (ranging from 500 to 800 users per second).
5.2.1.2 Serving HTML pages dynamically generated by Cocoon
In this scenario XML pages without logic i.e. no XSP pages are served to the client. This is a pure
conversion of XML and XSL to HTML in order to provide the browser with a medium that it can understand. The same way a different format i.e. PDF could be produced for a different application, context
or output device or client.
INT-D15FINAL
Page 36 of 51
INTELLECT
IST-1999-10375
This functionality is present but has not been used in a realistic scenario as all INTELLECT pilot pages
contain dynamic data and have to therefore be generated dynamically (see 5.2.1.4 Executing and
serving pre-compiled XSP pages).
5.2.1.3 Compiling, executing and serving XSP pages
When accessing the XML–pages via Cocoon the servlet executed by Tomcat has to be compiled first.
This servlet is generated from the XML- and XSL- pages in the first 9 seconds of the test and is then
used without modification thereby accelerating the process to roughly 1 to 2 seconds per request. The
compilation of the Servlet only happens once after the startup of the application server or modification
of the page.
Figure 18 – Request time including compilation for a single client
Figure 19 – Request time including compilation for multiple clients
5.2.1.4 Executing and serving pre-compiled XSP pages
In this scenario the page is generated by the shop servlet with is pre-generated by Cocoon and simply
executed by Tomcat. This benchmark represents a “normal” usage scenario, were access to the INTELLECT pages is frequent and thus does not require permanent compilation of the Servlet.
INT-D15FINAL
Page 37 of 51
INTELLECT
IST-1999-10375
Figure 20 – Request times without compilation
Figure 21 - Average user wait time without compilation
5.2.1.5 Comparison between URLs from the three INTELLECT pilots
In the following scenario “Web Stress” targeted three different URLs for each of the 5 simulated users
one for each of the three different Intellect shop-pilots. The red line shows the request time for a page
without any graphics. The green line represents a page from the Blauwerk pilot containing a lot of
graphics. This page has to be compiled first and creates therefore the typical initial overhead.
INT-D15FINAL
Page 38 of 51
INTELLECT
IST-1999-10375
Figure 22 –Request time for multiple URLs
Figure 23 –Request duration without compilation
Figure 24 clearly demonstrates that the stress test caused the server to operate at maximum capacity
for a large part of the benchmark duration. This indicates that performance issues related to the
eShop servlets are at least to some degree related to performance issues with the server hardware.
INT-D15FINAL
Page 39 of 51
INTELLECT
IST-1999-10375
Figure 24 – Response time and CPU usage table
5.2.1.6 Comparison of different servlet engines
Previous benchmarks have shown that servlets generated and compiled an average request time of
about 1000 – 1200ms require. Such high response times are caused by the Servlet engine in Tomcat.
Tomcat while being an internationally acclaimed Servlet/JSP Server and the J2EE platform Reference
Implementation for Servlet/JSP technology is also known for its relatively bad performance. Jakarta
Tomcat Version 3.2 is a software mainly aimed at developers seeking a 100% standards compliant
implementation that offer many development related features and is not meant to be used as a deployment platform for commercial rollout. Jakarta Tomcat 4 is meant to address this problem and is
expected to offer much higher performance. Nevertheless the extremely short-term availability of the
new Tomcat version did not allow for further benchmarks with the new application server.
The following benchmark presents the performance of Tomcat 3.2 compared to that of two other popular products, Orion (http://www.orionserver.com/), Resin (http://www.caucho.com/).
Figure 25 – Performance comparison between Tomcat 3.2, Resin and Orion
INT-D15FINAL
Page 40 of 51
INTELLECT
IST-1999-10375
5.2.2 Configurator performance
The performance of the Configurator is heavily influenced by the underlying Product Data Model. This
insight had a major impact on the design of the Configurator as well as the underlying Product data
model for all the INTELLECT modules. The INTELLECT Product Data Model is designed to reduce
overhead caused by complex operations during product data processing [6]. Furthermore the INTELLECT product data model is constructed in a way that allows the mapping of configuration functions to
SQL sequences that can be executed by an RDBMS much faster and more efficiently than executing
the processing in the Configurator itself. This strategy further eliminates the need for extensive data
transfer between the database and the Configurator during configuration, thereby further reducing the
amount of resources (i.e. bandwidth) and reaction times required.
Following sections will focus on the main configuration function of the Configurator Bean, namely getPossibleComponents(). This function delivers all compatible components for a specific product while
at the same time taking into account all soft and hard criteria specified by the user. This function directly or indirectly uses all other Configurator functionality and is the one most frequently called by the
other modules. It is also the one with the highest resource consumption.
Moreover we will be presenting benchmarks for two implementations of the Configurator, one consisting of a pure Java algorithm that simply uses the database as a data-repository and one that maps
configuration requests to SQL as described in the previous passages. The direct comparison between
the results from both versions makes the advantage of “Lean Configuration” the Configuration concept
developed by the INTELLECT project evident.
All benchmarks were carried out on the INTELLECT on-line prototype server. This machine is a dual
Pentium II, 400MHz with 256 MB RAM and SCSI disk subsystem. The operating system was Debian
GNU/Linux 2.2 (potato) with Linux Kernel 2.2.18. All clients were connected via LAN to the server. The
product database contained the following data:
•
5 Products
•
5 Component lists
•
74 Components
•
44 Component types
•
67 Component categories
INT-D15FINAL
Page 41 of 51
INTELLECT
IST-1999-10375
5.2.2.1 Pure Java Implementation
The version of the Configurator tested here consists of a pure Java algorithm that simply uses the
database as a data-repository.
1st Scenario
In this benchmark five groups of clients with one up to five clients per group from five different client
machines are performing requests for getPossibleComponets in parallel for a product with a continuously increasing amount of components. The first call targets a product with one component, the second a product with two components, etc. The following graph demonstrates the average time required
for each client group to receive the configuration information for products with a specific amount of
components.
12000
10000
8000
time (ms)
1 client
2 clients
6000
3 cleints
4 clients
5 clients
4000
2000
0
1 component 2 components 3 components 4 components 5 components 6 components 7 components 8 components 9 components
10
components
Figure 26 - Average request time for a specific amount of components (Java)
Figure 26 clearly demonstrates the following:
•
Response time is roughly 4 Seconds per request.
•
Response times are not dependent on the complexity of the product. Or in other words complex
products are handled just as well as simple products.
•
The overall performance of the system depends on the number of concurrent requests. The performance difference on the testbed hardware was roughly 1 sec per further client. This is to be attributed to the performance of the server as every concurrent connection requires a further Java
thread to be started. CPU usage on the server in all tests reached or exceeded 80-90% with three
or more concurrent connections.
INT-D15FINAL
Page 42 of 51
INTELLECT
IST-1999-10375
2nd Scenario
machines are performing requests for getPossibleComponets in parallel for a product with a single
component. The following graph demonstrates the average time required for each client group to receive the configuration information.
14000
12000
time (ms)
10000
8000
1 client
2 clients
3 clients
6000
4 clients
5 clients
4000
2000
0
1
2
3
4
5
6
7
8
9
10
Figure 27 - Average request time for a single component (Java)
•
Response times are not related to the state of the Java Virtual Machine (JVM) or the database.
The first and the last request have in general roughly the same response times. This precludes the
involvement of a cache/internal compilation or other event that impacts performance.
INT-D15FINAL
Page 43 of 51
INTELLECT
IST-1999-10375
5.2.2.2 SQL Implementation
This implementation of the Configurator maps configuration requests to SQL as described in the previous passages[6]. The direct comparison between the results from both implementations follows in
the next section and should make the advantage of “Lean Configuration” the Configuration concept
developed by the INTELLECT project evident.
1st Scenario
machines are performing requests for getPossibleComponets in parallel for a product with a continuously increasing amount of components. The first call targets a product with one component, the second a product with two components, etc. The following graph demonstrates the average time required
for each client group to receive the configuration information for products with a specific amount of
components.
1400
1200
time (ms)
1000
1 client
800
2 clients
3 clients
4 clients
600
clients
400
200
0
1
2
3
4
5
6
7
8
9
10
Figure 28 - Average request time for a specific amount of components (SQL)
•
Response time is roughly 400ms per request, or in other words one 10 of the response times for
the pure Java implementation.
•
•
The overall performance of the system does not depend on the number of concurrent requests.
The performance difference on the testbed hardware was roughly 20ms per further client.
•
The decline in response time after the first queries indicates that the Database employs a cache
that improves performance as long as queries request the same information.
th
INT-D15FINAL
Page 44 of 51
INTELLECT
IST-1999-10375
2nd Scenario
machines are performing requests for getPossibleComponets in parallel for a product with a single
component. The following graph demonstrates the average time required for each client group to receive the configuration information.
1600
1400
1200
1000
time(ms)
1 client
2 clients
800
3 clients
4 clients
5 clients
600
400
200
0
1
2
3
4
5
6
7
8
9
10
Figure 29 - Average request time for a single component (SQL)
•
The first and the last iteration have roughly the same response time. This precludes the involvement of a cache/internal compilation or other internal optimisation that impacts performance.
•
Performance is heavily dependent on the performance of Oracle. Oracle reaches a CPU Workload
of over 50% with a single client connecting. The server being a dual process or machine can accommodate two Oracle processes under these circumstances. The second client can thus also be
accommodated with only a slight performance loss, but further clients then proceed to overburden
the server thereby considerably degrading performance.
INT-D15FINAL
Page 45 of 51
INTELLECT
IST-1999-10375
CPU Usage
machines are performing requests for getPossibleComponets in parallel for a product with a continuously increasing amount of components. The first call targets a product with one component, the second a product with two components, etc. The following graph demonstrates the average CPU load
caused by each client group for products with a specific amount of components.
100,00%
90,00%
80,00%
70,00%
CPU Usage%
60,00%
1 client
2 clients
50,00%
3 clients
4 clients
5 clients
40,00%
30,00%
20,00%
10,00%
57
55
53
51
49
47
45
43
41
39
37
35
33
31
29
27
25
23
21
19
17
15
13
9
11
7
5
3
1
0,00%
time (ms)
Figure 30 – Total CPU usage with up to five concurrent clients
100,00
90,00
80,00
CPU Usage %
70,00
60,00
Oracle
50,00
Java
40,00
30,00
20,00
10,00
0,00
1
2
3
4
5
6
7
8
9
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
time(ms)
Figure 31- Oracle/JVM CPU usage with up to five concurrent clients
INT-D15FINAL
Page 46 of 51
INTELLECT
IST-1999-10375
Figure 30 and Figure 31 clearly demonstrate the following:
•
Performance is heavily dependent on the performance of the Server. The Server reaches a CPU
Workload of over 50% with a single client connecting. Further clients then proceed to overburden
the server thereby considerably degrading performance. The 100% mark is never achieved due to
the process management properties of the operating system.
•
Oracle requires the major part of available CPU resources thereby acting as the main bottleneck.
A setup where the Application server runs on a separate physical server than the RDBMS and
possibly addresses multiple RDBMS servers would definitely increase performance by dramatically.
5.2.2.3 SQL vs Java
This section contains a combined benchmark for both implementations of the Configurator. The direct
comparison between the results from both version makes the advantage of “Lean Configuration” the
Configuration concept developed by the INTELLECT project evident.
Overview
In this benchmark a single client performs requests for getPossibleComponets for a product with a
product with a continuously increasing amount of components. The first call targets a product with one
component, the second a product with two components, etc.. The following graph demonstrates the
average time required for each client to receive the configuration information for products with a specific amount of components.
6000
5000
time(ms)
4000
SQL
3000
Java
2000
1000
0
1
2
3
4
5
6
7
8
9
10
Figure 32 – Difference in response times between the SQL and pure Java implementations of the INTELLECT Configurator
•
Response time of the pure Java implementation is roughly 4 Seconds per request.
•
Response time of the SQL implementation is roughly 400ms per request, or in other words one
th
10 of the response times for the pure Java implementation.
INT-D15FINAL
Page 47 of 51
INTELLECT
IST-1999-10375
5.2.2.4 Conclusion
The direct comparison between the results from both Configurator versions makes the advantage of
“Lean Configuration” [6] the configuration concept developed by the INTELLECT project evident:
•
Response time is roughly 400ms per configuration request.
•
•
The overall performance of the system does not depend on the number of concurrent requests, as
long as the hardware can accommodate the number of clients. The performance difference on the
testbed hardware was roughly 20ms per further client.
•
The first and the last iteration have roughly the same response time. This precludes the involvement of a cache/internal compilation or other internal optimisation that impacts performance.
•
Performance is heavily dependent from the performance of Oracle. Oracle reaches a CPU Workload of over 80% with a single client connecting on the testbed server.
5.2.3 Order Processing performance
The work of the order processing starts, after the shop has accepted an order from a customer. Therefore the performance of the order processing module has no impact on the user expierience of the
customer, all the processes in this module occur offline.
We identified the following performance-critical areas within the Ordering-Module:
5.2.3.1 Database transactions
When the state of an order changes as a result of an incoming event, the order's state is reflected in 3
columns of the corresponding table: two of them are simple types (numeric and character data), the
XML-data of the order is saved in a CLOB ("character large object"). The duration of the update
statement is proportional to the length of the XML data. Benchmarks showed, that the part of these
database transactions are negligible in comparison to the transformations of XML documents. This is
further elaborated in the following paragraphs.
5.2.3.2 XML parsing
Parsing of the XML document is done with the Xerces reference implementation of the Apache Software Foundation. This implementation is free, but is known to be rather slow. There are commercial
implementations that are about five times faster than Xerces. Performance with the test XML was
about 5 to 7 milliseconds per parse.
5.2.3.3 XSL transformations
Generation of outgoing documents is accomplished via XSLT operations. For this, the XML document
has to be parsed into a tree representation by an XML DOM parser and after that transformed by an
XSLT engine. Both processes are rather time-consuming.
XSL-Transformation with a simple XSL stylesheet needed about 13 to 17ms.
5.2.3.4 XPATH addressing
XPath expressions are used within the workflow to get the information for the routing of orders. Xpath
evaluations are expensive, when the XML document is not already parsed; as the module needs the
parsed XML-document anyway the XPath addressing is also negligible.
5.2.3.5 XML transformations
Some backoffice functionality consists of XML-transformations; e.g. for the HiTEC pilot insertion of
parts list data was implemented, which resides in the database, into the order data. The time needed
INT-D15FINAL
Page 48 of 51
INTELLECT
IST-1999-10375
for these modifications of the DOM representation also had a negligible impact on the overall performance. We tested this with the HiTEC functionality of adding parts list information into the XML document from the database, this function required 600 to 830ms to perform on the test XML file.
5.2.3.6 Backoffice functionality
The integration of existing back-office systems is also part of the scope of the INTELLECT order processing module. As none of the end-users had no back-office system with necessary interfaces for
integration (they were all using proprietary products, that had no open interfaces) we tested this with
Anecon's self-developed "Admin Tool", which processes incoming and outgoing invoices.
These tests showed that the amount of time for communicating and processing the request in the
back-office system took much longer than any of the aforementioned points and was in the range of 5
to 9 seconds, depending on the desired back-office functionality.
For the pilots of our three end users sending of an email was the preferred back-office functionality,
because they implemented their workflow via sending "intelligent emails", which can trigger other
tasks. And even sending a short email via Anecon's SMTP server took 3 to 5 seconds.
5.2.3.7 Results
Our benchmarks showed, that the performance critical parts of the Order Processing module are not
located within the module, but by the external processes i.e. the external back-office systems that are
integrated by the module.
So the numbers for transaction throughput will depend mainly on the performance of the integrated
backoffice system. Business rules that are executed in the module (i.e. XML transformations like HiTEC’s parts list integration) can also influence the throughput if the logic is very complex. Compared to
the performance results of the eShop module and the configurator module, which expierience a much
higher load, because the number of shop and configurator transactions are between one and two
magnitudes over the number of completed purchases.
All performance tests were performed on the following system configuration:
•
Win2k Server, 256 MB RAM, 20 GB Harddisk, Intel Processor Pentium 3 800MHz
•
Java SDK 1.3.1
•
JBoss/Tomcat application server bundle, jBoss version 2.2, Tomcat 3.2
•
Xerces XML parser, version 1.4.3
•
Xalan XSLT processor version 2.1.0
•
XML test document : size is 25kB (this corresponds to a basket with 20 different items)
5.2.4 Help Desk performance
Because of the unexpected long time the jBoss group needed to integrate the next version of the servlet container we had no time for performance benchmarks until now (see 5.1.3.5).
5.2.5 VR Applet performance
The Virtual Reality module as a front-end for the configuration module shows a view of the product,
which must be as realistic as possible. The user can interact with the product: turn, zoom, select and
configure a component; the possible configurations are given by the configuration module. Excepted
few cases where some simple configurations (colour, size) are directly available from the e-shop, the
customer must go in the Virtual Reality module to configure its product. The 3D view of the products is
obtained by assembling 3D models of its components stored in VRML files provided by the shop operator. If no 3D model is available for some of the components, configuration via the VR Applet is not
possible.
The use of Java3D, which is an API for Java, permits to render a 3D world in a Java application or a
Java applet. Java3D provides a collection of high-level constructs for creating, manipulating and rendering 3D geometric objects in a virtual universe. The Java3D world corresponds to a scene graph,
INT-D15FINAL
Page 49 of 51
INTELLECT
IST-1999-10375
which is an arrangement of 3D objects in a tree structure, that completely specifies the contents of the
virtual universe, and the way it is rendered.
VRML files can be loaded and added to a Java3D universe thanks to the VRML97 library, defined by
the VRML-Java3D working group.
The performance of the 3D Applet is therefore completely dependent on the performance of the
Java3D implementation, the 3D graphics language used (i.e. OpenGL, DirectX), the 3D driver for the
graphics hardware of the client and last but not least the graphics hardware itself.
In cases were the hardware offers appropriate 3D functionality (all current chipsets) which is supported by the graphics driver and the graphics language libraries installed on the operating system
(default in all modern platforms) the 3D Applet offers fluid navigation through the VR-World.
5.3 Satisfaction
The comfort and acceptability of the work system to its users and other people affected by its use.“ –
This last criterion directly addresses the one of the hardest areas of Software-Ergonomics namely the
subjective satisfaction derived by a user in his interaction with the system. This very vague but also
extremely important criterion for the success of a software system can in most cases be addressed
only very vaguely and on a purely subjective manner.
INT-D15FINAL
Page 50 of 51
INTELLECT
IST-1999-10375
6 References
[1]
Wolfgang Dzida, Marion Wiethoff, Albert G. Arnold, ERGOguide - The Quality Assurance
Guide to Ergonomic Software, Delft University of Technology, German National Centre of
Computer Science, April 1993
[2]
ISO 9241 - Ergonomic requirements for office work with visual display terminals (VDTs), Part
10: Dialogue principles & Part 11: Usability (Principles), International Organization for
Standardization, 1993
[3]
Frieder Nake, Von der Interaktion - Über den instrumentalen und den medialen Charakter des
Computers, aus Die erträgliche Leichtigkeit der Zeichen – Ästhetik Semiotik Informatik, agis
Verlag Baden-Baden, 1993
[4]
Pamela Briggs, Usability Assessment for the office: Methodological Choices and their
Implications, Human Computer Interaction in the Work Place, Elsevier Science Publishers B.
V. (North-Holland), 1987
[5]
Shackel, B., Ergonomics in design for usability, HCI’86, pp.44-64
[6]
Ioannis Fikouras, Ewgeni Gisbrecht, Lean Configuration: Interactive Configuration for the
Internet, eBusiness and eWork 2001, Venice, 17-19 October
[7]
John D. Gould, Clayton Lewis, Design for Usability: Key Principles and What Designers Think,
Communications of the ACM, March 1985
[8]
Jakob Nielsen, Rolf Molich, Heuristic Evaluation of User Interfaces, CHI ’90 Conference
Proceedings, Special Issue of the SIGCHI Bulletin
[9]
Angelika Herzig, Usability-Testing: Put it into practice, Das Usability Labor der
Schweizerischen Kreditanstalt, aus Berichte des German Chapter of ACM 39,
Software-Ergonomie '93
[10]
Klaus Neugebaurer, Nicola Spielmann, Das Ergonomie-Labor der SAP - empirische
Ergonomie in der Praxis, aus Berichte des German Chapter of ACM 39, Software-Ergonomie
'93
[11]
Harold Thimbleby, Formulating usability, SIGCHI Bulletin, April 1994, pp.69-73
[12]
R.Oppermann, B.Murchner, H.Reiterer, M.Koch, Software-ergonomische Evaluation - Der
Leitfaden EVADIS II, Walter de Gruyter, Berlin, New York 1992
[13]
Lynette Hirschman and Donna Cuomo, Evaluation of Human Computer Interfaces, A
Report from an ARPA Workshop, SIGCHI Bulletin, April 1995, pp.28-29
[14]
Randolph G. Bias, Deborah J. Mayhew, Cost-Justifying Usability, Academic Press, Inc., 1994
[15]
Deborah A. Mitta, A Methodology for Quantifying Expert System Usability, Human Factors,
April 1991, 33(2), pp.233-245
[16]
comp.human-factors, comp.groupware, comp.cog-eng, USENET newsgroups
[17]
Marja-Riitta Koivunen, Marko Nieminen, Sirpa Riihiaho, Launching the Usability Approach
Experience at Helsinki University, of Technology, SIGCHI Bulletin, April 1995, pp.54-60
[18]
Fabio Paterno, A Formal Approach to the Evaluation of Interactive Systems, SIGCHI Bulletin,
April 1994, pp.59-64
INT-D15FINAL
Page 51 of 51

Deliverable (public)

Transcription

Similar documents

Intellect-powered security at Abu Dhabi National Oil Company

Investors Presentation - Intellect Design Arena Ltd

Deliverable (public)

Froggy Drawing handout2 - Mid Mon Valley Transit Authority

Deliverable (public)

The Difference Between the Brains of Men and Women

Smoke Signals, Oct 2010

KOROBRI LA HOMSTED Danny Marmina Jentian

Judge Me By My Past.