Pool Table Analyzer SDP09 Team Mettu

1
Pool Table Analyzer
SDP09 Team Mettu
Dave Fraska, Doug Frazer, Mitchell Kendall, and Timothy Langlois
Abstract—The Pool Table Analyzer is designed to enhance the
experience of a game of billiards. It will analyze the positions
of the balls and suggest the best shot to each player. This will
help amateur players to think about advanced concepts, such
as cue ball placement, that they do not normally consider. The
system will have an intuitive interface which anyone will be able
to understand. Four cameras will be mounted above a pool table
and connected to a computer. The computer will analyze the
images from the cameras to suggest the best shot. The computer
will also use the images from the cameras to track the location
of the cue stick as a player makes his/her shot.
I. I NTRODUCTION
T
HE Pool Table Analyzer is a system designed to watch a
pool table as a game is played. The analyzer will suggest
the best shot possible to each player on their turn. This will
enhance the game experience for players.
A. Problem Statement
A system is to be designed that will track the placement of
pool balls in a game of pool. The system will allow users to
see, on a screen, what balls are remaining, where they are on
the table, whose turn it is, and the score in real time. There
will be a user interface which will show the best shot for the
active player and the consequences of such a shot.
A system as such could attract more players to a hall, not
having to worry about remembering the rules, whose turn it
is, and other tedious tasks during game play. Similar systems
are in place in bowling alleys, to keep track of scores. Such a
system will lower the learning curve for new players, allowing
them to concentrate more on making shots and less on the
game play mechanics.
B. Context of Project
Our project is designed to be used wherever pool tables are
used, including pool halls, bars, and homes. The installation
will be fairly simple, mostly consisting of mounting the
webcams on the lighting fixture. The installation will be simple
enough that home users can install it themselves. Anyone who
plays pool will be able to use the system.
C. Requirements Specification
•
•
•
The analyzer shall suggest the best shot to each player
The pool cue module shall communicate with computer
up to 20 ft away
The pool cue module battery shall last for at least 10
hours
•
•
The analyzer shall be able to place the cue to make
straight shots with 5cm accuracy
The system shall make a shot suggestion within 5 seconds
of when all the balls stop moving
II. D ESIGN
A. System Overview
Our system will consist of four webcams mounted on the
lighting fixture above a pool table. The webcams will be
connected to a computer using usb cables. The computer will
analyze the images from the webcams to locate the balls and
the cue stick. The system will determine the positions of the
balls and then decide the best shot the player can make, based
on difficulty and usefulness of the shot.
B. Block Diagram
Our block diagram is shown in Figure 1 on page 2.
C. System Specification
1) Cameras: Our project will use four Logitech Quickcam
Pro webcams. Each webcam will cover one quadrant of
the billiard table. With a resolution of 1600x1200 each, the
cameras will give an accuracy of about 1 pixel per mm. The
cameras will constantly send images to the image recognition
software, which will process the images to determine the
locations of the balls. The cameras will be mounted in the
lighting fixture above the table.
2) Image Recognition Software: The image recognition
software is responsible for analyzing the images from the
webcams. There are three objectives this software must accomplish. The first is motion detection. It must detect when balls
are moving and when they stop: this signals when players are
changing turns, which is when the system must calculate the
best shot. The second objective is determining the positions of
the balls on the table. This will be accomplished by using the
Hough transform. We are using an open source implementation
of the transform which we tuned to more accurately detect the
balls [1]. The third objective is to determine the location of the
cue stick when the player is taking their shot. This will also
be done by using the Hough transform provided by OpenCV
[1]. This will be used to send signals to the cue stick module
to tell the player if their shooting angle needs to be adjusted.
We have designed a flow chart of how our software will
operate. Please refer to Figure 2 on page 3.
2
Fig. 1.
Block Diagram
3) Best Shot Suggestion Software: The purpose of the best
shot suggestion software is to suggest the easiest shot that
leaves the table in a useful state. This piece of the software is
based on the algorithm developed by Michael Smith in his
Master’s Thesis [2]. The best shot detection software will
have two main stages. The first is move generation, where
possible shots are found. Each class of shots will be searched
separately, in order of difficulty, so all straight shots will be
found first. To generate straight shots, a shot will be considered
for each ball for each pocket. If the ball is between the cue
ball and the pocket, and there are no other balls in the way, the
shot is possible. Since the difference in difficulty is so large
between classes, if any straight shots are found, other classes
will usually not be searched, unless the available straight shots
are fairly difficult.
The second stage is best shot suggestion. All the possible
shots will be evaluated based on difficulty and usefulness, i.e.,
how much does this shot help the player? Usefulness is mainly
based on the cue ball position after the shot. Difficulty of a
shot depends mainly on four variables: the cut angle, the object
ball-pocket angle, the cue-object ball distance, and the object
ball-pocket distance. Difficulty values will be precomputed by
setting up shots with certain values for these four variables
and then simulating the shots after adding noise.
To suggest the best shot, a Monte-Carlo search algorithm
will be used. For each possible shot, the software will determine the parameters needed to make the shot. Then zero-mean
Gaussian noise will be added to the parameters to simulate
the player. The noise will create i possible outcomes for each
particular shot. The move generation algorithm will be run
on each possible outcome state, the possible shots will be
sorted based on difficulty, and the easiest 3 will be chosen.
The usefulness score for outcome state i will be
ui = d1 ∗ p1 + d2 ∗ p2 + d3 ∗ p3
(1)
where d1 , d2 , d3 are weighting factors. Most of the time the
easiest shot will be the best shot to take, but having other
good shots on the table is good also. We are starting with
d1 = 1, d2 = .33, d3 = .15, but we may tweak these. The
average of the ui , uavg is the usefulness score of the shot.
The score of the shot is
s = uavg ∗ p
(2)
where p is the difficulty (probability) of correctly making the
shot. The shot with the highest score is the best shot.
This software also uses a physics library designed specifically for billiards [3]. The poolfiz library is a free, closed-
3
Fig. 2.
Flow Diagram
source library for simulating billiards. This library also uses
openGL to draw the table and balls as well as animate shots.
4) Pool Cue Positioning Software: After the best shot is
found, the pool cue positioning software will calculate the
angle the cue should be in to execute the shot. As the player
positions the cue stick, the system will monitor the position
and send the data to the pool cue positioning software through
the cameras and image recognition software. It does this by
using the hough transform provided by OpenCV. The line that
is closest to the cue ball is the line that is chosen as the cue
stick. The pool cue positioning software will calculate whether
the pool cue needs to rotate left or right and send this data
through the bluetooth to the cue.
5) Graphical User Interface: In order to combine all of our
different libraries together we have designed a graphical user
interface. This interface uses the poolfiz library to draw the
pool table and animate the pool balls being hit. This library is
written in OpenGL using the GLUT toolkit. We draw the pool
table into one frame and then draw more information specific
to our application in the surrounding frames, for example
information about the current angle of the pool cue or the
velocity that the player should hit the ball at. There is also
a series of buttons that allow the user to interact with the
user interface through their mouse. We provide to the poolfiz
library the positions of all the pool balls that we have found
and place their positions accordingly, and then allow the user
to either design their own shot and animate it to see what will
happen or have our AI suggest a shot to them. After animation
the pool table in the user interface will revert to the current
table state that is representative of the table. During this mode
of user interaction the library is also constantly polling the
position of the cue ball to see if it has moved, which would
imply a shot has been made. If this event occurs, then the state
of the user interface will change to an idle state while it waits
for the pool balls on the table to stop their movement. Both
the detection of the cue ball motion and the motion of all other
balls on the table is done by doing simple RGB comparisons
of low resolution images of the table. Once the user interface
has decided that the table is in a stable state, it will return to
the original user mode state with an updated view of the table
representing the current state of the table.
The user interface will also compare the current state to the
previous state to check if any balls were scored and if they
were it will notify the user of the current turn based on which
balls were sunk.
6) Bluetooth Transmitter: The transmitter is a generic Bluetooth USB dongle using serial port protocol. It will receive
signals from the pool cue positioning software and relay the
signals to the bluetooth receiver on the cue stick.
7) Bluetooth Receiver: The bluetooth receiver is the RN41 from Roving Networks. It will receive the control signals
from the bluetooth transmitter and send these signals to the
microcontroller.
8) Microcontroller: The microcontroller will also be located on the cue stick. It is an ATmega168. It will decode
the signals from the bluetooth receiver and determine which
LED should be lit up.
4
9) LEDs: The three LEDs will be mounted on the front of
the cue stick. They will show the player which way he or she
should rotate the cue stick to have the best chance of correctly
executing the shot.
D. Design Alternatives
We originally wanted to use only one camera. This would
have simplified the fixture, and also the image detection
because we would only have one image to process. However,
after completing the error analysis, we realized that one web
camera would not give us enough accuracy, so we decided
to use four. Also, we planned on using the GTK+ library
to build the GUI. However, we discovered that the GTK+
library was very poorly documented and difficult to use. We
found another library called Qt that did the same thing but was
better documented and easier to use, so we used that instead.
Originally we had wanted to find the balls using a resolution
of 1600x1200, but this was too slow, so we ended up using
960x720.
III. I MPLEMENTATION
A. Hough Transform
The Hough transform is a mathematical technique used in
image processing to isolate or locate certain shapes in an
image. This transform is classically used to find geometric
shapes such as squares, lines, circles, ellipses etc. however
there is a generalized Hough transform which can be specified
to search for arbitrary shapes. In either situation, the transform
works by finding a series of points of interest, which changes
based on the implementation, and searching the remaining
linear spaces for possible solutions.
1) Hough Transform for Lines: When the Hough transform
is used to find lines it will need to find a series of points
of interest, (xi , yi ) and search all possible curves that this
point can create given some limiting parameters. In order to
find these (xi , yi ) points, generally a canny filter is applied
to the image to highlight possible edges. By using the canny
representation of an image we need to try and find a series of
enough points which are collinear and may represent a line.
If we search in the polar coordinate system, where a line is
defined as: x ∗ cos(Θ) + y ∗ sin(Θ) = r. The transform will
create a permutation of all possible lines that can be created
in this method, given certain discretizing parameters such as
minimum length, minimum step for Θ and r. However, varying
values of r and Θ are represented by sinusoids. By plotting
these points in r, Θ space, which is a series of sinusoids,
it becomes possible to find points which are collinear in
Cartesian space by finding intersections of the sinusoids in
(r, Θ) space. To find the possibles X,Y points that may be
considered as lines the transform first runs the image through
a canny filter which makes it more obvious what the lines may
be.
2) Hough Transform for Circles: Much like the Hough
transform for lines, the Hough transform for circles attempts
to vary the possible parameters that would represent a circle.
These parameters are the radius of the circle and its center
in two dimensions. This provides three dimensions to search
over instead of two which was required for the lines, so it
takes an order of time longer to run the algorithm. The same
as the line finder we apply some canny filtering to the image
before we search for images. However, by drawing possible
circles along points of interest we can identify possible circles
by finding intersections of sufficient number of circles in our
search space. This point of intersection will represent the
center of a circle of the radius that was being searched for at
that point in time. Using the hough transform for circles can
search for arbitrary sized circles, however for our application
we have a specific goal to accomplish and have limited the
parameters to only search for balls of the radius of a standard
pool ball with a small offset δ1 .
B. Color Histograms
In order to determine the number of each pool ball we
attempt to identify the color of the ball. Assuming we have
successfully found the locations of the pool balls to begin with,
we can search the circle where the center is identified through
the image detection and the ball radius is a known calculation
based on the resolution. We will evaluate each pixel within this
circle and create a histogram of the colors. This histogram is
created by grouping pixels into discretized groups, which is
inherit based on the fact that the 8-bit RGB values correspond
to 256 different groups. To increase accuracy we have played
with the size of the bucket and have settled on grouping each
R, G, or B value into 16 discrete groups, which effectively
evaluates to dropping the 4 least significant bits of each
RGB value. After the histogram of the ball is created, it is
compared to a histogram that was previously created under
controlled circumstances where the ball numbers are known,
and whichever histogram best matches the histogram that we
just measured is returned as the ball number. We repeat this
process for each ball. This algorithm will run on the order of
the number of pixels that represent a ball.
C. Error Calculation
When calculating the error, we are only concerned with the
error perpendicular to the shot direction. The error comes from
the accuracy of the cameras. With one pixel per mm, we can
only expect to be accurate to one mm. The image detection
will also add additional error. In Figure 3, this error is e. From
our observations so far, the maximum e is 4 pixels. Since the
angle of the suggested shot will not change, the player will
shoot the cue ball (white) at the same angle that the system
suggests. This means the cue ball will strike the object ball
(red) at an offset from where the software expected, causing a
deviation from the expected path. In Figure 4, we show how
this offset will affect the direction of the object ball. If the
cue ball is striking the object ball at an offset of e, we can
draw an identical triangle on the other side of the object ball
and extend it. For example, if the object ball is struck at an
offset of e and is traveling about half the length of the table
(≈ 950mm), we extend the triangle out. Since 950 ≈ 33 ∗ r,
the object ball will be 33∗e away from where it was supposed
to be at the end of the shot. With e = 1mm, the object ball
5
a
b
Fig. 5.
l
c
e
w
Fig. 3. Blue arrow is the intended shot. Red arrow is the actual shot. a ≈
57mm, b ≈ 142mm, c ≈ 117mm, l ≈ 1900mm, w ≈ 952mm
PDF of a single ball
will have an error of 33mm. With e = 4mm, the shot will
have an error of 4 ∗ 33 = 132mm.
The bulk of our error will be in detecting the center of
the balls. We will assume for now that the error is uniformly
distributed in a circle of radius emax centered at the true center
of the ball. Then the probability density function will be
1
x2 + y 2 < e2max
πe2max
(3)
fx,y (x, y) =
0
otherwise
However, we only care about the error perpendicular to the
direction of the shot. This is the error which will cause offset
when the balls collide, causing error in the path of the object
ball. To find the pdf of the error in one direction, we use the
equation
Z
∞
fx (x) =
fx,y (x, y)
(4)
−∞
e
r
Using equation 4 with our probability density function we
obtain
Z √e2max −x2
1
fx (x) =
dy
√2
2
− emax −x2 πemax
1 p 2
2
=
2
e
−
x
(5)
max
πe2max
x
r
e
Of course fx (x) is only valid where |x| < emax because emax
is the maximum amount of error.
Since the cue ball and the object ball will have an error in
the detected location, the total error will be the difference of
the two errors.
etotal = e1 − e2
(6)
The pdf of the total error will be the convolution of the two
separate errors. Since both separate errors have the pdf in Eq
4, the pdf of the total error is
Fig. 4.
A ball when it is hit
fe (e)
= fx (x) ∗ fx (x)
Z ∞p
q
4
2
2
2
emax − τ
e2max − (x − τ ) (7)
dτ
=
π 2 e4max −∞
6
Fig. 6.
PDF of the total error
A numerical approximation of this integral is shown in Figure 6. Through approximation in Matlab, we have calculated
the following statistics:
Probability
|etotal | < 2
.497
|etotal | < 3
.687
|etotal | < 4
.830
Now let d = distance between object ball and pocket, r =
radius of ball, and s = d/r. Then according to Figure 4, the
error at the end of the shot will be ef inal = s ∗ etotal . Using
the pdf for etotal ,
Probability
2
|ef inal | < 2 ∗ s = ∗ d
r
3
|ef inal | < 3 ∗ s = ∗ d
r
4
|ef inal | < 4 ∗ s = ∗ d
r
So we will be able to correctly make shots from a distance
of 712mm with a probability of .497, from a distance of
475mm with a probability of .687, and from a distance of
356mm with a probability of .83. Also note that we believe
this to be a minimum estimation. Through fine tuning the
image recognition software, we should be able to decrease
emax for a single ball from 4 pixels to hopefully 1 pixel.
Repeating the previous calculations for emax = 1, 2, 3, we
obtain the following statistics for pshot , the probability of
correctly executing the shot:
d(mm)
356
475
712
1425
emax = 4 emax = 3 emax = 2 emax = 1
pshot
pshot
pshot
pshot
.830
.946
.999
.999
.687
.829
.978
.999
.497
.629
.830
.999
.263
.345
.497
.832
(8)
.497
.687
D. Requirements Specification Status
.830
Following is how well we have met our requirements
specification.
7
•
•
•
•
•
MET - The system suggests shots to the player.
MET - The pool cue can communicate with the computer
up to 20ft away.
MET - The pool cue battery lasts for at least 10 hours.
NOT MET - Have a 5 cm accuracy
ALMOST MET - Make a suggestion within 5 seconds
of when the balls stop moving - it takes between 6 and
7 seconds
IV. P ROJECT M ANAGEMENT
A. Roles of Team Members
We divided the necessary work as evenly as possible. Dave
is responsible for the bluetooth module on the cue stick. Dave
will also help with the lighting fixture, and when that is
finished he will work on the GUI. Doug is mainly responsible
for the GUI, although he is also helping with the lighting
fixture and the bluetooth module. Tim is responsible for the
physics engine and the best shot algorithm. He is also assisting
with the image recognition software. Mitch is responsible for
the image recognition software and the webpage. He is also
helping with the physics, best shot algorithm, and lighting
fixture.
B. Gantt Chart
Please refer to Figure 7 on page 8. This chart shows all the
tasks we have identified for this project, when we expect to
start each task, and how long we expect each task to take to
complete.
V. S UMMARY AND C ONCLUSION
We are pleased with the outcome of our project. Although it
is not perfect, we feel that we have learned and accomplished
a lot.
R EFERENCES
[1] “Opencv:
Open
computer
vision
library,”
http://opencv.willowgarage.com/wiki.
[2] M. Smith, “Pickpocket: An artificial intelligence for computer billiards,”
Master’s thesis, University of Alberta, 2006.
[3] W. Leckie and M. Greenspan, “An event-based pool physics
simulator,” Eleventh Advances in Computer Games: Lecture
Notes on Computer Science, no. 4250, pp. 247–262, Sep. 2006,
http://rcvlab.ece.queensu.ca/ greensm/downloads.html.
8
Fig. 7.
Gantt Chart