A programming Library for multi-party Collaborative Applications

MultiCross –
A programming Library for multi-party
Collaborative Applications
Bachelor Thesis
Silvan Troxler
University of Fribourg 2011
Professor:
Prof. Rolf Ingold
Head of the DIVA Research Group
Advisors:
Dr. Denis Lalanne
Dr. Bruno Dumas
Matthias Schwaller
Table of Contents
1
2
Introduction .............................................................................................. 4
1.1
Context / Situation...................................................................................................... 4
1.2
Motivation ................................................................................................................... 5
1.3
Goal ............................................................................................................................. 5
1.4
Approach and Report Organisation ......................................................................... 6
Environment ............................................................................................. 7
2.1
2.1.1
Wiimote .................................................................................................................. 7
2.1.2
DiamondTouch ...................................................................................................... 7
2.2
3
Software ...................................................................................................................... 7
2.2.1
WiiRemoteJ ........................................................................................................... 7
2.2.2
BlueCove ............................................................................................................... 7
2.2.3
WIDCOMM Bluetooth Stack .................................................................................. 8
2.2.4
DiamondTouch Developer’s Kit ............................................................................. 8
2.2.5
Choice of programming language – Java .............................................................. 8
Development Process.............................................................................. 9
3.1
Client/server architecture .......................................................................................... 9
3.1.1
Design .................................................................................................................... 9
3.1.2
Implementation .................................................................................................... 11
3.1.3
Messages ............................................................................................................ 12
3.2
Developing the Library ............................................................................................ 13
3.2.1
Application Data and Logic .................................................................................. 14
3.2.2
Application specific messages ............................................................................. 14
3.2.3
Adding multiple devices support .......................................................................... 14
3.3
4
Hardware ..................................................................................................................... 7
Library Improvements ............................................................................................. 15
MultiCross Library ................................................................................. 17
4.1
Package Client .......................................................................................................... 17
4.1.1
Class App ............................................................................................................ 17
4.1.2
Class AppPanel ................................................................................................... 17
4.1.3
Client-Class ......................................................................................................... 18
4.1.4
Thread-Classes ................................................................................................... 18
4.2
Package Server ........................................................................................................ 19
4.2.1
Class Server ........................................................................................................ 19
4.2.2
Class ClientThread .............................................................................................. 19
4.2.3
Class User ........................................................................................................... 20
4.2.4
Class AppInfo ...................................................................................................... 20
4.3
Extending the MultiCross Library ........................................................................... 21
1
5
4.3.1
Adding a new device type .................................................................................... 21
4.3.2
Adding a new event ............................................................................................. 21
Validation ................................................................................................ 22
5.1
5.1.1
Game Description ................................................................................................ 22
5.1.2
Creating a Server ................................................................................................ 22
5.1.3
Creating a Client .................................................................................................. 23
5.1.4
Adding behavior ................................................................................................... 25
5.1.5
Adding Images ..................................................................................................... 25
5.2
6
Use case 1: „Ligretto“-Game .................................................................................. 22
Use case 2: “DiffusionBarrier”-Game .................................................................... 26
5.2.1
Game description ................................................................................................. 27
5.2.2
Display technique ................................................................................................ 27
5.2.3
Simultaneous actions ........................................................................................... 27
5.3
Performance Test ..................................................................................................... 27
5.4
Validated Features ................................................................................................... 28
Discussion .............................................................................................. 30
6.1
Review Development Process ................................................................................ 30
6.2
Review MultiCross Library ...................................................................................... 30
6.2.1
Achievements ...................................................................................................... 30
6.2.2
Improvement capability ........................................................................................ 31
7
Conclusion.............................................................................................. 32
8
Bibliography ........................................................................................... 33
9
Annex ...................................................................................................... 34
9.1
Acknowledgment...................................................................................................... 34
9.2
How to get the MultiCross library work.................................................................. 34
9.3
Client-Server messages of Ligretto-application ................................................... 35
9.4
Client-Server messages of DiffusionBarrier-application ..................................... 36
9.5
Source Code ............................................................................................................. 37
2
Abstract
This report illustrates the development process of a the java library MultiCross, which allows
developers to easily create their own multi-party applications using a stable communication
over the network and using the DiamondTouch Table, Wiimotes as well as the standard
mouse as input devices. Such a library simplifies the developers’ work when creating a collaborative application. Beside the native support for the mentioned devices, the MultiCross
Library can be extended in order to work with different devices and to add or modify behaviour to the application.
The whole server/client architecture as well as the library itself were firstly planned and then
implemented. Parallel to the library, two applications were designed and implemented as a
help for building and improving the library, but also as use cases to validate the MultiCross
Library at the end.
The development and the result were discussed critically and improvement ideas were given.
The goals of having more than one user per machine and more than one client in the network, but also having support for the DiamondTouch and Wiimotes were fully fulfilled. The
newly formed library can be extended in order to include new devices, new events and therefore match the different applications’ needs.
This report documents the whole development process and therefore gives you an idea of
how the goal was achieved and what the major challenges were. Furthermore, with the help
of the two use cases, an example is given of how the library can be used in order to build a
new application and to adjust it for one’s own need.
3
1 Introduction
1.1 Context / Situation
Applications are improved all the time and are getting better and better. Developers are always looking for new kinds of applications to improve them even more. A fairly new approach
of the last several years are collaborative applications, where several people can use the
same application at the same time, working together at one thing and share the application’s
resources.
At the University of Fribourg, different collaborative applications have been developed. For
example there is a mindmap application running on the DiamondTouch Table. This application allows up to four people to work on a mindmap, all of them using the DiamondTouch. Another application developed in the DIVA group of the University of Fribourg is the so-called
“Communication Board”, which allows two people on different machines to work on the same
application, always seeing the opponent’s actions as well as the person on the other machine
by using videoconference. Additionally, this application can be controlled by a Wiimote.
Both applications are pretty limited in their use. The mindmap application is a single-machine
application and can only be run on the DiamondTouch Table. The “Communication Board”application can indeed be used in the network but is limited to two machines and one user per
machine.
This is not only the trend at the University of Fribourg, we can also see this tendency at all the
collaborative applications.
Generally, there are four different kinds of collaborative applications as Figure 1.1 shows.
Figure 1.1: The main types of collaborative applications
1
1
Inspired by Robert Johansen (1988): Groupware. Computer Support for Business Teams,
The Free Press, New York and London, 1988.
4
Concentrating on the left part of this graphic, we can see that nowadays, there are principally
two kinds of synchronous collaborative applications: Face to Face and Remote Interaction
Applications.
Face to Face or Single Display Groupware are applications in which multiple users collaborate on a single machine only. For the development of such applications different toolkits
have been programmed in order to support multiple mouse pointers (B.B.Bederson, 1999;
S.Greenberg, 2004).
Different approaches are followed in this field. Hutterer et al. added multiple input device support to the X.Org server to mention only one example. Therefore, not only multiple mouse
pointers but also multiple keyboard interaction is possible (P.Hutterer, 2007).
Remote Interaction Applications are applications, which allow two or more users to interact
simultaneously over a distance. This is done by using a network or the Internet to pass data
from one computer to another with the aim of supporting efficient collaboration over a distance. There are already lots of remote interaction software, mainly in the electronic meeting
and videoconference area, network games are also examples of applications of this kind.
What all of these applications has in common is the fact, that they only provide support for
one single user per machine.
1.2 Motivation
The two applications described above, developed at the University of Fribourg, as well as the
most today’s collaborative applications are good in their kind but not flexible enough. Especially the code of the mentioned applications is barely reusable and doesn’t support the developer by doing his work. For developers, it would be great to have a more general solution
in order to develop multi-party collaborative applications and therefore also close the gap between collocated and remote applications.
1.3 Goal
In this project, the two different approaches - having more than one user per machine on the
one hand, and to have more than one machine in the network on the other hand - are combined. The result will be a collaborative multidevice network application with the following
characteristics:
-
At least 10 people can interact at the same time
-
Indefinite number of machines in the same network
-
DiamondTouch and Communication Board/Wiimote Support
The idea of the project is to program two different multidevice network applications and to
develop a small library to make it easier for other developers to create such collaborative multidevice network applications.
5
Furthermore the network as well as device performance can be tested with the programmed
applications.
1.4 Approach and Report Organisation
The project was be divided into four major parts:
1. The Design & The Implementation of a client/server architecture
2. The Card-Game Application “Ligretto”
3. The Second Application “DiffusionBarrier”
4. Developing the Library in parallel.
After having dealt with the hard- and software used for this project, the client/server architecture is explained. In chapter 4, the library is described. The two different applications explained in chapter 5 provide good examples of how to use the library.
At the end of this report, you will find a short evaluation as well as a conclusion part.
6
2 Environment
2.1 Hardware
2.1.1
Wiimote
1
The Wiimote (short for Wii Remote) is the controller of the game console Wii of the jJpanese
company Nintendo. It is equipped with an infrared camera and an acceleration sensor. Different libraries in many languages are available in order to use the Wiimote for computer applications.
2.1.2
DiamondTouch
2
The DiamondTouch Table of Circle Twelve Inc. allows multiple people to interact simultaneously without disturbing each other. This feature sets the DiamondTouch Table apart from
other products. In order to distinguish the people the DiamondTouch uses capacitive coupling.
2.2 Software
2.2.1
3
WiiRemoteJ
In order to use the Wiimote as a control device for an application, there are different Java libraries available. After having looked at some of them, I decided to use WiiRemoteJ since it
has multiple Wiimote support, which is mandatory to achieve the project’s goal. To get
WiiRemoteJ, and therefore the Wiimote as a device, working on windows, a few points should
be taken into account:
-
BlueCove is needed
-
The WIDCOMM bluethooth stack has to be installed
-
A bluetooth dongle based on a broadcom chip must be used
2.2.2
BlueCove
4
BlueCove is a Java library for Bluetooth. It is needed to work with the WiiRemoteJ library to
connect the Wiimotes via Bluetooth to the computer. It should be considered that there are
different distributions for different platforms.
1
http://en.wikipedia.org/wiki/Wii_Remote
2
http://www.merl.com/projects/DiamondTouch/
3
http://www.world-of-cha0s.hostrocket.com/WiiRemoteJ/
4
http://bluecove.org/
7
2.2.3
WIDCOMM Bluetooth Stack
1
To get WiiRemoteJ working properly, it is essential that the WIDCOMM bluetooth stack is installed on the machine. This software is free and can be downloaded on the website of
Broadcom. Unfortunately, the WIDCOMM bluetooth stack doesn’t work with all bluetooth devices but only with those which have a broadcom chip. As I have seen, it’s not always easy to
find out which chipset was used, since the product descriptions of the bluetooth dongles rarely give information about it. At present, Belkin is using Broadcom chipsets in their dongles.
2.2.4
DiamondTouch Developer’s Kit
2
Circle Twelve Inc. provides a Developer’s Kit for their DiamondTouch Table. This Kit simplifies the development of DiamondTouch-applications, since it comes with great features like
getting the device, getting the device’s state or providing the point where a touch occurs.
2.2.5
Choice of programming language – Java
Since Java is well known and widely used, it is useful to develop this library in this language.
Most of the modern input devices, such as the DiamondTouch, provide Java support. Also,
with the Sockets (see section 3.1.2), Java is providing a good tool for message passing from
one computer to another.
1
http://www.broadcom.com/support/bluetooth/update.php
2
http://www.merl.com/projects/dtsdk/
8
3 Development Process
In order to develop the library, two applications were programmed. Even if this required a lot
of work and most notably a lot of revision during the whole process, it was a good way to see,
which functions the library should provide and how it can be improved in order to be useful for
different applications.
The development of the library and the two applications can be divided in several parts:
1. Designing and implementing of a client/server architecture
2. Designing and implementing application 1 and the library
3. Designing and implementing application 2 and improving the library
In this chapter, the whole development approach with its major problems is going to be shown
and explained. The results of this process, thus the library and the applications themselves
are then presented in the following two chapters.
3.1 Client/server architecture
3.1.1
Design
For a collaborative application that can be executed on several computers in a network at the
same time, it is important to have a good client/server architecture.
There are several possibilities of how the server and the clients can communicate with each
other. All machines could be interconnected and therefore pass messages to any other client
or server.
Figure 3.1: Network with interconnected clients
1
1
inspired by http://decode-abap.blogspot.com/
9
Another approach would be to have one central server and have all messages pass through
this server in order to get to another client. In this case, for every change a client wants to
share with another client, it has to send a message to the server and is not able to inform the
client directly. The server handles all incoming messages, executes the desired action and
then informs the other clients.
Figure 3.2: Network with central server
1
For this project, the second approach was chosen with the following advantages:
•
This implementation provides a better control of what the application is actually doing,
as all the messages have to pass through the server.
•
Imagine the clients could communicate directly to each other. It could happen, that
two clients are sending messages at the same time, but the other clients in the network are not going to receive the messages in the same order and therefore, they
would have different program states at the same time.
Additionally, for a multi-party application, we want to have more than one device per client.
Beside the “server” and the “client”, we therefore need to have a third entity “user”. One client
could have multiply users and handles all user interactions and communicates them to the
server if necessary.
To sum up, let’s have a look at the most important entities for the collaborative application:
1
http://decode-abap.blogspot.com/
10
Entity
Description
Server
The central unit which handles the whole application
Client
One single machine
User
One single Input Device, such as a hand on the DiamondTouch or a
Wiimote at the communication board. One Client can have more than
one User.
Table 3.1: Most important entities for client/server structure supporting multi-party interaction
3.1.2
Implementation
1
Inspired from an existing open source code found on the Internet a solution, which includes
all the needs mentioned in the section before, was implemented.
This solution utilises the well-known observer pattern. The server is the observer, observing
all registered clients. In the java.util package, the two abstract classes Observer and Observable are available and can be subclassed in order to use this design pattern.
The following figure shows the observer pattern and the adjusted implementation for the client/server architecture used for this library.
Figure 3.3: Observer Pattern (top) and adjusted implementation (bottom)
Every time a Client connects to the Server, a new instance of the class ClientThread is created. This is mandatory, since in Java a ServerSocket object can only receive a connection request but not handle any messages sent from the client. For each new connection, a separate socket is created. Over this socket, messages can be passed from one end to the other.
In this case, the socket is forwarded to the clientThread which can then receive and send
1
http://www.developer.com/print.php/1356891
11
messages from and to one client. Since the clientThread is observed by the server, it can also
notify the server in order to send a message over the different clientThreads to all clients.
3.1.3
Messages
Thus, a clientThread is responsible to receive messages from the client. Also, it parses the
messages and decides how to respond or what to do.
All messages are composed the same way. The first (and sometimes the second) word is the
keyword with the help of which the clientThread decides what to do. All other words or numbers are additional information which might be needed. All the words and numbers are separated by a semicolon.
The following table lists all possible messages between client and server and explains what
they stand for. A practical example is given afterwards.
Client to Server - Messages
IP;<IP-address>
This is always the first message sent from the client to the server. It is a registration on the
server, telling the server that a new client wants to join the application. The server will then
respond to this request by giving the client a specific client number.
USER;NEW; device_nr>
For each input device at a client’s machine, a user should be created. For this purpose, the
client sends a message to the server, informing the server about the device. The device
number can be chosen by the client itself, but has to be unique. Therefore, it is recommended
to use the client number multiplied by 10 and add 1 for each new device so that we have for
example the device number 21 and 22 on the client number 2.
POSITION;<device_nr>;<x>;<y>
This is the most basic message. It consists of the keyword “POSITION”, the device number
as well as the x and y position of the cursor. This message is used to indicate a change of the
coordinates of the cursor of a device.
Server to Client - Messages
RESP;<client_nr>
Whenever a new client wants and is allowed to join the application, the server respond with
this message. <client_nr> would then be replaced by the actual client number which is used
in messages afterwards. This message is only sent to the client, who wanted to register on
the server.
USER;NEW;<client_nr><device_nr>
Whenever a new user is created, the server responds to all its clients by sending this message, in order to ensure that all clients are aware of the fact that a new user has joined the
application.
12
POSITION;<dev_nr>;<x>;<y>
After receiving a cursor position change from a client, the Server automatically sends this
message to all its clients in order to inform them about the change.
ERROR;<message>
If an error occurs while handling the received message, a error message can be passed to
one or all clients by using this message. After the keyword “Error” the error message is
passed as a string.
Here is an example of how a usual communication between the server and its clients could
look like. Suppose we have the server and two clients communicating with each other.
Figure 3.4: client/server communication
In this case, the first client registers itself on the server. Then the second client is doing the
same thing. Client 1 then tells the server, that there is a new user with device id 11. The server responds to all known clients and informs them that a new user at client 1 with device nr 11
was created. A moment later, the position of the user with device id 11 is changed and client
1 who is aware of that change, sends an adequate message to the server which again informs all its clients about the position change by sending exactly the same message to all
clients.
3.2 Developing the Library
The Library and the first application were developed in parallel. Therefore, the first step of
developing the library was actually to design the first application. After having dealt with all
the needed classes and functions, the application specific parts were separated from the
more general parts which could also be used for other applications. Also the client/serverimplementation was included in the library.
13
Since the Library as well as the Applications are explained in detail later, only the structure
and the important points in the development process are treated at this point.
3.2.1
Application Data and Logic
One of the major decisions was to determine whether the application is running on the server
or independently on each client and therefore, where to store the application data. Since for
the first application a shuffle-function is needed, it makes more sense to have run it on the
server side. Suppose each client would shuffle an array itself. Every client would then have a
different order in this array and synchronisation would not be possible. However, if the server
shuffles the array and then passes the order to all its clients, they will all have the same state
at the end. For this reason, the decision was made to run the application on the server, or at
least to let the server keep track of the most important things in order to ensure synchronisation.
In the shuffle example above, the server would shuffle the array and pass the order to all its
clients. However, whenever the clients need the next element from the array, they don’t have
to ask the server again, since all of them have exactly the same elements in the same order.
Thus, some data is stored both on the server and at each client, but the data on the server is
effective in case of an asynchronous state.
While the server is therefore in charge of the application data, the visualisation of the data is
the clients’ business.
With this approach, another great advantage arises. Since the server knows all about the application, it can restore the application state at any time. If, for example, a client is joining the
application later, the server can send all the needed information in order to update this client
about the current application state.
3.2.2
Application specific messages
Of course, in addition to the messages used for the client/server architecture, the developer
should be able to define application specific messages. Since any message can be sent from
and to a client, one only has to be sure the client or the server can decode the message. For
this purpose, the parsing-function of the ClientThread as well of the Client class has to be
adjusted, adding the keywords to recognize the message type and assigning an action to perform.
3.2.3
Adding multiple devices support
The library should be able to handle different devices, such as a normal mouse, the DiamondTouch Table as well as Wiimotes. For this purpose the Client class was sub classed for
each type of input device in order to add device specific functions. The client then creates a
separate thread that listens to all inputs on the device. The performed actions on a given input (for example a click or a dragging), have to be specified in the run-method of the thread.
14
Figure 3.5: Simplified class diagram of the client package
3.3 Library Improvements
While programming the second application, the strength and the weakness of the existing
library could easily be seen. Since the aim was that the library code has not to be changed to
write an application, but can easily be extended, all the changes, which must have been
made to the library code while implementing the second application, were the library’s weak
points and required a revision.
The following list shows the main problems and their solutions or rather adjustments made to
the library.
The application specific messages had to be added to the ClientThread class.
An additional Parser class was created. Any subclass of this Parser class can now be set as
the parser of the application. Every time the server or the client receives a message, the
evaluate-method of this object is called.
To set the application specific parser, an instance of the subclass has to be passed as a constructor argument when instantiating a new client or accordingly assigned to the parameter
“parser” of the server object.
Duplicated code in the different client classes
A parent class was created for the different Client classes. This can avoid duplicated code on
the one hand and provide an easy way to integrate new devices on the other hand.
15
In order to have a convenient way to create one of the client’s subclasses, a ClientFactory
class was created.
Inflexible thread classes in order to extend the event based behaviours
The thread classes were revised. For each device, the input is analysed by this thread class
and click, press, drag and release events are filtered. Whenever one of this event occurs, a
corresponding method is called. This method can also be changed when subclassing the
thread class. This approach gives more freedom to the developer and is hiding the devicespecific input treatment.
Thus, the programmer is free to add the same behaviour for all devices or to react differently
to the events depending on the device type.
Not enough freedom in using the library
At some points the programmer had to choose between using the library and to disclaim of
his freedom or to program it himself. Since this is not desirable some hooks were included
were the developer can add own behaviour to some functions, but still has all the library’s
advantages.
Extra-Features
To simplify the programmer’s life, a few extras were included in the library, such as an extended JFrame which provides a good starting point for the applications’ graphical output as
well as some useful functions and a standard interface for choosing the device type.
Also, a logging was included in order to simplify the debugging. All messages received over
the network communication are added in the log-file. For the server as well as for each client,
it can be determined whether the logging is turned on or off.
16
4 MultiCross Library
The Library is divided into two packages, the client and the server package. While the classes
of the client package mainly serve the application’s client, the classes of the server class are
important for the server of the application. While only the main classes and functions are presented in this chapter, examples how to use the different classes are given in chapter 5 of this
report. Furthermore, the complete source code and the javadoc of either the library and the
two use cases can be found on the enclosed CD.
Figure 4.1: Simplified UML diagram of MultiCross Library
4.1 Package Client
4.1.1
Class App
Every application which makes use of the MultiCross library, has to be started by creating an
App object or an object of an App’s subclass. This Class provides some useful functions to
the developer such as opening the interface to choose a device type or to connect to the
server.
The JFrame class is extendend by this class. Therefore, a new instance automatically creates
a JFrame. The content pane of this window has to be of the type AppPanel (or a subclass
thereof), and can either be passed as an argument to the constructor or set afterwards by
calling the method setAppPanel(appPanel).
4.1.2
Class AppPanel
The AppPanel class extends a JLayeredPane of the java.swing package. The only mandatory
thing is to specify the size of the JLayeredPane by setting the variable dimension of type Dimension as well as to define a JPanel named cursorPanel as one Panel of the JLayeredPane, on which by default, the cursor is displayed afterwards. Apart from that, the developer
is absolutely free in creating his own layout and his own design.
17
4.1.3
Client-Class
The Client class describes a usual client of an application. Such a client is capable of connecting to, as well as communicating with, a server.
When instantiating a new Client Object, an application panel of type AppPanel, a parser of
type Parser and a Boolean, to indicate whether logging is on or off, has to be passed as arguments to the constructor.
Here is a list of the most important and most often needed methods of the Client class.
connect(String hostName, int port)
is called to connect to the server. This method needs to be executed before messages can be
sent to the server. After calling this method, the client is listening to all messages comming
from the server.
sendmessage(String msg)
sends a message to the server if it is connected.
afterUserCreation(User u)
hook method to specify an action after a new user has been created.
Table 4.1: The most often needed methods of the class Client
The MultiCross library already includes three subclasses for the three different device types
“Standard Mouse”, “DiamondTouch Table” and “Wiimotes”. Each of this subclass includes
some methods especially needed for the corresponding device. Also, each Client class creates a new Thread object (see section 4.1.4) and starts it in order to listen to the input on the
specific device.
4.1.4
Thread-Classes
For each device, a separate thread class has to be written in order to listen to the device input
and filter the different events which can occur. The library already includes the thread classes
for the supported device mentioned in the section before. For all of these devices, the following events inspired from the standard mouse events are handled:
•
Click
•
Press
•
Release
•
Drag
•
Moved
If one of this event occurs, the standard actions are performed and the corresponding handlefunction is called. For example if a click event occurs, the position of the device is also updated automatically by sending an appropriate message to the server. Then, the method
handleMouseClick() is called which, by default, does not perform any action. However, this
18
method can be filled with actions when subclassing the class and thus can be adapted to realise the programmer’s intentions.
4.2 Package Server
4.2.1
Class Server
As explained in section 3.1, the server class is responsible for the clients’ registrations and for
creating clientThreads to listen to this clients’ messages. There are only a few methods which
could be useful to use in your server’s Parser class.
hasUser(int device_id)
returns true if a User with the passed device_id exists. Returns false otherwise.
getUser(int device_id)
Returns an object of type User with the device_id passed as an argument. Before using this
method, the method hasUser(device_id) should have been called in order to get sure that the
user exists.
hasClient(InetAdress s)
returns true if a client with the InetAdress s exists, false otherwise.
getClient(InetAdress s)
Returns an object of type Client with the InetAdress s. Before using this method, the method
hasClient(InetAdress s) should be used in order to get sure that the client exists.
Table 4.2: The Server class' most useful methods
After instantiating the server, the start()-method has to be called. Before calling the start()method the server’s appInfo and parser should be set using the two function setAppInfo(AppInfo info) and setParser(Parser parser).
4.2.2
Class ClientThread
For each client a clientThread instance is created on the server. One clientThread is always
listening to one particular client, receives the messages of this client and can send a message to this client as well as to all clients registered on the server.
Whenever a message is received, the evaluateString()-method of the specified parser is
called automatically. In this method the field clientThread is a reference of the clientThread
which received the message. Over this variable, messages can be sent back to the client using the method sendToClient(String msg) or to all clients using the method sentToAll(Srting
msg).
19
4.2.3
Class User
The User class holds information about the users using the application. Usually no change
has to be made to this class. Nevertheless, you can find a brief overview of the most important methods below.
setVisible(Boolean visible)
This method sets the variable visible to the value passed as an argument. This variable can
then be used in your AppPanel class to specify whether the user’s cursor should be visible or
not.
getClientNr()
Returns the number of the client on which the user is working.
getDeviceNr()
Returns the number of the user’s device.
setColor(Color c)
The first eight users automatically get a color from the MultiCross library. The following user
will all get the color black. Using this method, the color can be set manually.
Table 4.3: the most important methods of the class User
4.2.4
Class AppInfo
This class is an empty abstract class. It’s only advantage is to have a new type which can be
subclassed. For a new application, this class has to subclassed and can be filled with both
fields containing all the data needed by the application’s server and functions used to manipulate the data.
Figure 4.2: UML of the Server Package
20
4.3 Extending the MultiCross Library
4.3.1
Adding a new device type
To include a new device type, two classes have to be written. First of all a new Thread class
has to be created in order to listen to the new device’s input. This class has to implement the
interface Runnable and therefore provide a run-method. At least the events click, press, release, drag and depending on the device the event move should be catched and the corresponding handle-functions have to be created.
Additionally a new subclass of the Client class has to be created, which is in charge of starting a new thread with the before created Runnable object.
As well as the ClientFactory class, the chooseDevicePanel class must also be updated.
4.3.2
Adding a new event
Maybe a developer wants to add a new event such as entered or exited. In this case, all
thread classes have to be updated, specifying for each device type when the new event appears and what should happen by its occurrence. It is also recommended to create a new
handle function for each new event type.
21
5 Validation
Even if the two applications and the library were developed in parallel meaning both the library and the application grew as an entity, the two applications are a good indication for the
library’s gain. The first use case, the Ligretto game, is in need of an efficient system since it
depends on fast actions. The second use case on the other hand validates the collaborative
capabilities of the library.
5.1 Use case 1: „Ligretto“-Game
Use case 1 is the famous card game Ligretto, which was implemented using the MultiCross
library.
5.1.1
Game Description
The rules of the game are pretty easy. Every player has 40 cards, numbered from 1 to 10 in 4
different colors. The aim of the game is to get rid of all the cards as fast as possible. All players are playing simultaneously and therefore attention to the cards being played by others as
well as one's own cards is needed.
When the game starts, players simultaneously discard cards in the middle of the table, building colored piles in ascending numerical order according to color, and only starting new piles
with a '1' card.
Every player has three “open cards”, a “stack” of ten cards and the rest of the 40 cards remain in the player’s hand. While the cards on the stack are covered, the “open cards” are visible and can be played. If an “open card” is played, a card from the “stack” replaces it. As
soon as the stack of one player is empty, the round is finished. Whenever a player can’t play
any of his “open cards”, he can look at every third card from the card in his hand and always
play the card on the top of this pile. Every played card counts one point while each remaining
card on the “stack” at the end of the round counts minus two points for this player.
5.1.2
Creating a Server
At first, let us have a look in how one can use the MultiCross library to create a new server for
its own application. To do this, a new class has to be created which includes the main function and therefore is the starting point of the application.
For the Ligretto Application, this class is called LigrettoGameServer and was created as a
singleton class such that the main-function looks like that:
LigrettoGameServer g = LigrettoGameServer.getInstance();
The constructor of this class then does actually do the main task.
server = new Server();
gameInfo = new LigrettoGameInfo();
22
server.setAppInfo(gameInfo);
server.setParser(new LigrettoServerParser(gameInfo));
server.startServer();
On line 1 the new Server is instanciated. Line 2 is responsible for creating a new instance of
the class LigrettoGameInfo which subclasses the AppInfo class of the library. This object is
then set as the server’s appInfo.
Also a parser has to be set in order to parse the application specific message which the clients will send to the server. For this purpose an instance of the parser class (or a class subclassed from this class) has to be set as the server’s parser (line 4).
Once the appInfo and the parser are set, the server can be started (line 5). From now on,
new clients can register at the server.
5.1.3
Creating a Client
The client side of the application is implemented similarly. First of all, a new subclass of the
class App has to be created. This class is the entry point of the application and contains the
main method. As well as the LigrettoGameServer class, the LigrettoGameClient class is designed as a singleton class. Therefore the main method starts with the following statement
LigrettoGameClient game = LigrettoGameClient.getInstance();
Since the super class of this class, the App class of the MultiCross library, expects a argument of type AppPanel, a corresponding subclass LigrettoGamePanel was created and the
unique instance of this class is passed as an argument to the GameLigrettoClient class’ constructor. So this is how the method getInstance() of the GameLigrettoClient class looks like:
public static LigrettoGameClient getInstance() {
if(instance == null) {
instance = new LigrettoGameCl
ent(LigrettoGamePanel.getInstance());
}
return instance;
}
Please note that creating a new object of the type LigrettoGameClient is not creating an actual client, but is a subclass of a JFrame. In order to create a client which can communicate to
23
the server, two more steps are mandatory. Firstly a device has to chosen. The superclass of
the LigrettoGameClient provides a graphical interface to let the user choose his device.
Therefore we call
game.chooseClient();
after having created an object of the LigrettoGameClient class and assigned it to the variable
game. Before calling this method, a ClientFactory class for your application must have been
created, and set as the clientFactory of your own App class.
Then, after having chosen the client type, the application can connect to the server by calling
game.chooseServer(String defaultServerIp);
this method lets the user edit the server ip address on the one hand and connects to the
server on the other.
Once the client is successfully connected to the server, the content pane of the application’s
JFrame is set to the specified appPanel. In the case of the Ligretto game, the content of the
LigrettoGameClient is set to the instance of LigrettoGamePanel. The Application is now fully
started.
For a better understanding of the explained process, the activity diagram of starting the Ligretto application is shown in Figure 5.1.
Figure 5.1: Activity Diagram of Application Start
24
5.1.4
Adding behavior
After having a server and a client, the behaviour should be implemented. Let’s have a look in
how that was made in the Ligretto application using the drag event in order to move a card.
The thread classes already provide some hook method we can use in order to program behaviour (see chapter 4.1.4: Thread-Classes). Therefore, all we have to do is to specify for
each device which action to perform whenever the drag event occurs, So we have to override
the method handleMouseDragged(int x, int y, int device_nr) In the subclass of each thread
class.
Since for the Ligretto game the behaviour should be the same for all the different devices, the
handleMouseDragged()-method looks the same for all thread subclasses. Because we also
don’t want to have duplicate code, the instructions, which have to be performed by a dragging
event, were outsourced in the class LigrettoGameClient. Thus, the handleMouseDragged()method of all thread subclasses is looking like this:
@Override
public void handleMouseDragged(int x, int y, int device_nr) {
LigrettoGameClient.getInstance().handleMouseDragged(x,
y, device_nr, bigboss.clientNr);
}
In the handeMouseDragged()-method of the LigrettoGameClient, the actual behaviour was
finally implemented. In this particular case, it is checked if there is a card on the starting point
of the drag event and whether the user, which triggered the event, can move this card. If so, a
new message is generated and sent to the server in order to move the card to the next position.
In order to ensure synchronisation, no action is performed without the server’s permission. So
when the client wants to move a card, a message with the keyword “MOVE” is sent to the
server. The server then redirects this message to all its clients, so that the change is made at
all clients at the same time.
5.1.5
Adding Images
The last step of the development of the first application was to include graphics so that the
game actually looks similar to the real Ligretto game. Since the MultiCross library provides a
lot of useful functions, but doesn’t constrain the developer in the graphical part, it was easy
and fast to include the images.
25
Figure 5.2: Screenshot of the Ligretto application
5.2 Use case 2: “DiffusionBarrier”-Game
The second application, the DiffusionBarrier-Game is implemented more or less the same
way than the Ligretto application. Therefore, it won’t be explained in detail again at this point.
However, there are some points that extend the previous section and differ from the Ligretto
application. Those points should be pointed out here.
Figure 5.3: Screenshot of the DiffusionBarrier application
26
5.2.1
Game description
DiffusionBarrier simulates a very simple cell in which different ions are diffusing. The players
are the membrane of the cell, allowing some ions either to pass or not.
It is a four-player game where always two players form one team. The team on the left side
should try to get all green ions on the left side, while the team on the left should bring all blue
ions on the right side. Both teams should avoid having red ions in their zone on the left or rather the right margin.
In order to control whether the ions may pass or not, each player can move the hole in his
diffusion barrier by moving the mouse, the Wiimote or the finger up or down. All ions can only
pass the barrier at its hole. Therefore, in order to pass the diffusion barrier, both players of
one team have to place the hole on the same spot. The aim is, to keep the good ions on the
players’ zone while getting rid of the bad red ions. Depending on the color, the ions have
positive or negative scores, which are counted separately for each team. The team with the
higher score after one minute wins the round.
5.2.2
Display technique
While for the Ligretto application a layoutmanager and java.swing objects were used to design the game, absolute positioning and geometric forms such as rectangles and circles were
used for the diffusionBarrier application. This shows, that the library is flexible and lets the
programmer freely choose between the different design approaches.
5.2.3
Simultaneous actions
For the diffusionBarrier game, it must be checked whether two clicks of different users are
made within a small space of time. For this purpose, the ClientThread class of the MultiCross
library provides the method checkSimultaneousTime(). The following parameters have to be
passed as arguments:
•
Timestamp t1: Timestamp of first action
•
Timestamp t2: Timestamp of second action
•
Int msec: the maximal number of milliseconds between the two actions.
The method then returns true, when the difference between the two timestamps t1 and t2 is
smaller than msec and returns false otherwise.
This method was used to implement the functionality of the freeze-button of the diffusionBarrier game. Only if the two players of the same team are pressing the freeze button at the
same time (space of time = 1000ms), the balls in the middle area are actually frozen. Otherwise, nothing happens.
5.3 Performance Test
One of the most important questions is whether the application, mainly the network speed, is
fast enough to run the applications. Of course, the speed depends on the network type. But it
27
still is important that the application itself is fast enough. Therefore, some tests where made
in order to evaluate the university’s network capacity on the one hand and the application
speed on the other.
For this purpose, the logging of the application was used. Since the exact time (Timestamp) is
recorded on every message a client sends to the server and also on every message the client
receives, we can see how long it takes to actually perform an action.
Let us observe the following case on the Ligretto game:
One player wants to move a card. In order to do that, the client is sending a message to the
server. The Server than sends this message to all its clients. Nothing happens on the client
until it receives this message. How long does it takes from starting the move-action until it
actually performs?
The following table shows the recorded amount of time passed between the user’s dragging
action and the actual move of the card, once when the server is running on the same machine as the client, and once when they run on two different machines within the network of
the university.
#
Local
Network
1
8ms
16ms
2
16ms
135ms
3
1ms
16ms
4
3ms
31ms
5
6ms
31ms
6
7ms
15ms
7
7ms
184ms
8
2ms
8ms
9
16ms
22ms
10
12ms
88ms
Average
7.8ms
54.6ms
As it may have been expected, the communication between the client and server is much
faster when both parties are running on the same machine. But even if they have to communicate over the network, it is fast enough to have a fluently running application.
5.4 Validated Features
To sum up the validation of the library we have seen that two different applications could be
implemented without making any change on the library itself. The library is therefore independent from the application and can be used for any kind of collaborative applications. The
library has the following services:
28
•
Multidevice capability: The library can handle more than one client and different input
device types.
•
Multipoint capability: The library can handle multiple users per client.
•
A stable client/server communication with an unlimited number of clients and users
•
Extendable behaviours for all input events
•
New input events can be added
•
New Device Types can be added
•
Logging functionality
Figure 5.4: People playing Ligretto on the DiamondTouch table and the CommunicationBoard.
29
6 Discussion
The two use cases indicate the gain of the MultiCross library. Nevertheless, a critical view
should highlight the good as well as the bad points in the development process and judge the
capability of the library, also bring up the library’s weak points and it could be improved.
6.1 Review Development Process
Even if there were made some UML class diagrams before the library and the applications
were implemented, this part could have been done better. For such a project, it is important to
have a clear conception of what should be achieved and how it should be implemented, before you actually start programming. A lot of time can be saved and could also have been
saved in this project if the planning part had been carried out even more thoroughly.
After having planned the things to perfection, it also is necessary to always keep in mind the
overall structure and the overall goals of a project. When the first application was programmed, it was programmed like this would be the single part of this project. A lot of things
could already have been implemented if the larger idea of having a library at the end had
been taken into account even more.
One of the best parts on the other hand was to program the second application. In order to
have a useful library at the end, it was extremely useful to have not only one, but two use
cases, which were implemented while the library was developed. This gave the possibility to
see what is needed in order to create a library that can be used for different applications without the need of changing the library’s code. A lot of the library improvements could be done
while implementing the second application. Of course, this needed some rework on the first
application. But with this effort only, it was possible to have two applications at the end, based
on the exactly same library without having changed one single line in the library’s source
code.
6.2 Review MultiCross Library
As the two use cases illustrate, the library is basically working fine for different applications.
Nevertheless there is always room for improvement. The good points should pointed out first
and the ideas for improvement were mentioned afterwards.
6.2.1
Achievements
A stable implementation of a client/server architecture could be programmed. The chosen
implementation allows having an infinite number of clients and users. This increases the value of network applications a lot. Also, the server is running independently, not having the
function of both the server and a client on the same time. Therefore, the client and the server
can be developed totally independently.
30
Another outstanding point is the ability to adapt the library for its own use. On several positions in the library, so-called hook methods are added in order to give the developer the possibility to add the behaviour of the library without changing the library’s code directly. This allows the developer great latitude and makes the library more useful.
6.2.2
Improvement capability
Even if the library already provides some flexibility and freedom for the developer, the usability could still be improved. The developer is indeed free in adding his own behaviour to the
application, but as the case may be, he has to sublass and edit several classes in order to do
so. A better handling of the behaviour would increase the usability of the library a lot. Ideally,
the developer could add the desired behaviour at one single place in order to get it work on
every device. However, it should not prohibit the possibility of having different behaviour for
different device types.
Further, the debugging support of the library could be improved. Errors relying on the library,
such as connection errors, should be captured and included in the log-file. A great thing for
developers would be a level based debugging where one can specify the captured error
types.
At that time, the library automatically takes care of the client and user creation. But by default
nothing is done when a client leaves the application. It would be an improvement to have also
an automatic system taking care of client and user destruction.
31
7 Conclusion
The product of this bachelor thesis is the MultiCross library, which allows developers to easily
create their own multi-party applications using a stable communication over the network and
using the DiamondTouch Table, Wiimotes as well as the standard mouse as input devices.
The whole server/client architecture as well as the library itself were firstly planned and then
implemented. Parallel to the library, two applications were designed and implemented as a
help for building and improving the library, but also as use cases to validate the MultiCross
Library at the end. The development and the result were discussed critically and improvement
ideas were given. The goals mentioned at the beginning of the project, namely to have more
than one user per machine and more than one client in the network, but also to have support
for the DiamondTouch and Wiimotes were fully fulfilled. The newly formed library can be extended in order to include new devices, new events and therefore match the different applications’ needs.
1
The whole library is available on sourceforge.net . Therefore everyone can download and use
the library, but also edit it in order to improve the library and therefore make it even more useful for other developers.
The momentary version of the library provides a good base for further development and more
services. For example, video and audio conferencing could be included. The challenge of this
feature would be how the different users on one machine are handled.
Further, a great improvement would be to integrate keyboard input such that the applications
are not confined to click events.
1
http://sourceforge.net/projects/multicross
32
8 Bibliography
•
B.B.Bederson, J. J. (1999). Architecture and Implementation of a Java Package for
Multiple Input Devices (MID) (Vols. 99-08). Maryland, USA: University of Maryland.
•
P.Hutterer. (2007). Groupware support in the windowing system. In Procedding of the
eight converence on Australasian user interface. Balarat, Australia: Australian
Computer Society.
•
S.Greenberg, E. a. (2004). Rapidly prototyping single display groupware through the
SDGToolkit in Proceeding of the fifth converence on Australasian user interface.
Dunedin, New Zealand: Australian Computer Society.
33
9 Annex
9.1 Acknowledgment
Firstly I would like to thank Denise Lalanne who supported me during the last two semesters
and provided me assistance for my bachelor project. His kindly manner and effort in helping
me find ideas and solutions for any kind of problem made it pleasurable for me to work with
him on my bachelor thesis.
Further, I also own thanks to Matthias Schwaller for supporting me in the more technical area
and for handing out the lab keys about a 100 times.
Thanks also go to Bruno Dumas who helped me particularly in the first period of the project,
to the Departement of Informatics of the University Fribourg, as well as to all other people that
were in any form part of my project.
9.2 How to get the MultiCross library work
In order to use the MultiCross library, follow the instructions below.
1. Copy all needed libraries from the enclosed CD.
Please consider that there are two versions of Bluecove:
BlueCove1.1.0 for Windows
BlueCove1.1.1-SNAPSHOT for Mac OS X
2. Also copy the MultiCross, if you don’t need to change the library. If you want to improve or adapt the library, you can find the source code on the CD as well.
3. Include all the needed libraries in the Java Build Path.
4. For using the Wiimotes on Windows, the WIDCOMM Bluetooth-Stack should be
downloaded and installed. Alternatively you can also find a version on the CD. For
Mac OS X, the AvantanaBluetooth and RTXTxomm-libraries are needed. These libraries has to be copied to the /Library/Java/Extensions-folder.
The library and the use case applications were tested on the following platforms:
•
MouseClient on Windows XP/Vista/7, Mac OSX 10.6
•
WiimoteClient only on Windows Vista, using the WIDOMM Bluetooth Stack and Belkin Bluetooth dongle
•
DTouchClient on Windows XP
34
9.3 Client-Server messages of Ligretto-application
Server to Client messages
PLAYER;NEW;<device_nr>;<player_nr>;<pos_x>;<pos_y>
PLAYER;UPDATE;<device_nr>;<player_nr>;<pos_x>;<pos_y>
START
MOVE;<player_nr>;<card_nr>;<pos_x>;<pos_y>
DROP;<player_nr>;<card_nr>;<stack_nr>
LAY;<player_nr>;<card_nr>;<card_attributes>
RESET;<player_nr>;<card_nr>
SCORE;<player_nr>;<score>
STOP;<player_nr>
Client to Server messages
PLAYER; NEW;< device_nr>;<player_nr>;
PLAYER;UPDATE
START
GET;<player_nr><card_nr>
MOVE;<player_nr>;<card_nr>;<x>;<y>
DROP;<player_nr>;<card_nr>;<stack_nr>
RESET;<player_nr>;<card_nr>
LAY;<player-nr>;<card_nr>;<card_parameters>
35
SCORE<player_nr>;<score>
9.4 Client-Server messages of DiffusionBarrier-application
Server to Client messages
ION;NEW;<id>;<color>;<x>;<y>;<dir_x>;<dir_y>;<frozen>;<visible>
ION;UPDATE;<id>;<color>;<x>;<y>;<dir_x>;<dir_y>;<frozen>;<visible>
PLAYER;NEW;<device_nr>;<player_nr>;<pos_x>;<pos_y>
START
CLOCK;<sec>
FREEZE;<sec>
STOP
Client to Server messages
PLAYER; NEW;< device_nr>;<player_nr>;
ION;NEW
START
FREEZE;<device_nr>
36
9.5 Source Code
The complete source code of the library as well as the two use case applications can be
found on this CD as well as at http://sourceforge.net/projects/multicross.
37