Thesis Documentation

El Rompecabezas
An Interactive Puzzle Installation
by
Rafael Mejia
A thesis submitted to the faculty of Parsons The New School for Design,
in partial fulfillment of the requirements for the degree of
Master of Fine Arts in Design and Technology
Faculty: Marko Tandefelt,Thesis Studio, Spring 2007,
Marko Tandefelt, Thesis Studio, Fall 2006
Advisor: Joshua Goldberg
Thesis production website:
http://a.parsons.edu/~rmejia/thesis
©Rafael Mejia
2007
2
Very special thanks to the following
people for all their help, support,
and understanding.
Josh Goldberg
Alice M. Rodriguez
Eddy Rodriguez
Marko Tandefelt
Carmen Torregrosa
Jaime Valle
Loretta Wolozin
Jenny Wu
And to my loving family,
Rafe, Milly, Alberto, and Natalie.
I dedicate this thesis project to you.
3
El Rompecabezas consists of an interactive video puzzle. The user faces a screen with
a live feed video of himself or herself, which is split into pieces and scattered randomly
across the screen. He or she must assemble the pieces back together, but with every
piece that is put into place, the original image gets more and more distorted. By the time
the user has assembled the last piece, what is displayed is a new and completely altered
image from the one at the beginning.
The project is set up as a tabletop touch screen. The technologies used for this project
consist of Max/MSP/Jitter, and Javascript. Max/MSP/Jitter is used for processes such as
splitting a video image, video playback and manipulation, and navigation throughout the
different scenarios. The Javascript code goes inside of the Max/MSP/Jitter patch for
controlling certain elements and objects directly from the script, which would otherwise
be extremely difficult to handle. The end result is an interactive piece in which the user
needs to assemble a puzzle by placing small pieces where they belong on the grid.
The intent of the project is to thwart expectations by giving the user a familiar image that
they need to reconstruct, but the more they try, the more elusive it becomes. This puzzle
will act as an abstract way of seeing and understanding how we deal with the process of
solving many challenges, issues, and frustrations we constantly face in life.
4
Table of Contents
Chapter 1: Introduction……………………………………………………………………… 7
1.1 Concept ………………………………………………………………………………….... 7
1.2 Impetus…………………………………………………………………………………….. 8
1.3 Concept Formulation……………………………………………………………………....8
1.4 Design Questions…………………………………………………………………………10
Chapter 2: Domains and Precedents……………………………………………………..11
2.1 Main Domains…………………………………………………………………………… 11
2.1.1 Jigsaw Puzzles ……………………………………………………………… 11
2.1.2 Order and Disorder………………………………………………………….. 12
2.1.3 Elusiveness……………………………………………………………………13
2.2 Precedents……………………………………………………………………………….. 14
Chapter 3: Methodology……………………………………………………………………. 19
3.1 Thesis Components………………………………………………………………………19
3.2 Form………………………………………………………………………………………..19
3.3
Building and Process Iterations………………………………………………………...19
3.3.1 Gathering Ideas……………………………………………………………….20
3.3.2 Iteration 1: Reconstruction of Oneself………………………………………21
3.3.3 Iteration 2: Puzzle Game……………………………………………………..22
3.3.4 Iteration 3: Modular Puzzle Tiles…………………………………………….23
3.3.5 Iteration 4: Flow of Time……………………………………………………...24
3.3.5.1
Time’s Disorder……………………………………………………..24
3.3.6 Back to Basics: Finalizing my Thesis…………………………………….....26
3.3.6.1
General Idea……………………………………………………......26
3.3.7
First Prototype………………………………………………………………..26
3.3.8
Second Prototype………………………………………………………..….. 27
5
3.3.9
Third Prototype………………………………………………………………..28
3.3.10 Final Version…………………………………………………………………..29
3.3.11 User Interaction……………………………………………………………….32
3.4
Behind the Magic: Technologies…………………………………………………………34
3.5
Usability Tests……………………………………………………………………………...36
3.5.1
Procedure……………………………………………………………………...37
Chapter 4: Evaluation……………………………………………………………………….…39
4.1 Criteria for Evaluating Success………………………………………………………..…...39
4.1.1 Purpose…………………………………………………………………….……..39
4.1.2 Aesthetic………………………………………………………………….……….39
4.1.3 Usability and Interaction……………………………………………….………...40
4.1.3.1 Interest…………………………………………………….…………...40
4.1.3.2 Playfulness…………………………………………….………………41
4.1.3.3 Frustration……………………………………………..……………….41
4.1.4 Problem Solving………………………………………………..………………….41
4.1.5 Future of the Project…………………………………………..…………………..42
4.2 Feedback and Concluding Thoughts…………………………………..……………………42
4.3 Reflecting on the Thesis Process……………………………………...…………………….43
Works Cited………………………………………………………………………………..……...45
6
Chapter 1: Introduction
1.1 Concept.
My thesis project consists of an interactive puzzle installation displayed in a tabletop
touch screen. When the user approaches the screen, he or she will see a moving image
split into 25 pieces, which are scattered randomly across the screen. The image being
displayed is a video live feed of the user who is
interacting with the piece. The user must
assemble the pieces back together, but with
every piece that is put into place, the original
image gets more and more distorted. By the time
the user has assembled the last piece, what is
displayed is a new and completely altered image
from the one at the beginning.
After all the pieces are together as one, the user is rewarded with a backwards
progression of the distortion of the image, starting at the last screen and clearing itself all
the way to the first screen. The last thing the user will see is his or her live image
completely clear and assembled into place, before the image is re-split into 25 pieces,
and the cycle starts again.
What users experience is an elusive, unpredictable result, regardless of what they do
and counter to their expectation, especially since they’re working with the familiar image
of themselves via the live feed. My intent is to thwart expectations. Users succeed when
they realize that my puzzle is a bit oppositional. Does this mean life’s puzzles cannot be
solved? No. Only this one is elusive. What can the user do? Move on. I hope.
The reason I want to develop this project is to create an interactive tool in the form of a
puzzle that thwarts the user’s expectations. The user builds the puzzle based on his or
her own image, which provides a familiar framework and sets up the idea that all he or
she needs to do is put the pieces where they belong, yet the puzzle becomes more
difficult and frustrating with every step of the process. This serves as a metaphor to life’s
7
sudden twists, turns, and unexpected surprises. This puzzle will act as an abstract way
of seeing and understanding how we deal with the process of solving many challenges,
issues, and frustrations we constantly face in life.
1.2 Impetus.
As I started to think about what was it that I wanted to do as my thesis project, I
narrowed some domains and concepts as I tried to find a topic of interest. I knew I
wanted to explore things like puzzles and brain games, portable devices, and interactive
usability. I continued to investigate different domains and fields of study that have always
interested me, and I studied some of the most popular forms of puzzles, ranging from
simple and fun aimed to entertain users, to the purely logical, mathematical, and brain
enhancing.
Having been a puzzle aficionado my entire life, I’ve had the opportunity to play with
different kinds of puzzles, but mostly jigsaw and logic ones. One element that I always
wish I could control whenever I assemble jigsaw puzzles is their displayed image.
Generally, when a user assembles image puzzles such as jigsaw, he or she already
knows what the end result will be, whether it’s because they use the image from the box
where the puzzle came from as a reference tool, because they have a general sense of
what it is by looking at the individual pieces, or because someone else told them, among
many other reasons.
1.3 Concept Formulation.
Once the puzzle is assembled, the user is typically rewarded with the resulting image.
But what if there was a jigsaw puzzle in which the user gets a result unlike what he or
she expected? Or even better, what if the puzzle image somehow changed or evolved
into something different? Asking myself these kinds of questions, I realized that I was
intrigued by the idea of a creating an image puzzle that would somehow change as the
user interacts with it. A question that was raised was whether or not the user should
have direct control of the changes. Would they be caused by direct reactions based on
the user’s interaction (the user has control over the changes)? Or are they just random
8
changes triggered by certain events, in which the user only has control of the event but
not of the changes themselves? I decided that the latter would work better with my
original concept and ideas.
As I started to think about the imagery of my puzzle, I contemplated the idea of making it
very personal for whoever was assembling it. Having this in mind, I came up with the
idea of the image being a live feed of the user who is interacting with the puzzle.
Whether it’s a still image or a live feed, I was fascinated by the idea of the users
assembling what to them are bits and pieces of an unclear shape or silhouette, and
eventually discovering it is indeed themselves what they are trying to solve and fix. It
would be interesting to see their reaction as they discover it is indeed themselves who
they are trying to assemble.
Another strong element of my thesis is the elusive feeling involved in the process of
reconstructing the puzzle. As the user puts the pieces together and gets closer to fully
assembling the puzzle, the image gets more and more distorted until what is left is a
warped unrecognizable image of himself or herself. In a way, the puzzle is never
completely solved. The user will start with chaos and disorder, and the more he or she
works to solve it, the more distorted the image will get. In other words, it goes from
disorder, to another form of disorder. The reasoning behind this is that sometimes we try
to organize and bring order into issues or problems we face in our daily lives, but the
more we do the worse things get. There is a feeling of frustration as the user tries to
assemble himself or herself together, because the more he or she tries, the worse the
process gets. I wish to convey this message and feeling across into the user. The puzzle
becomes a metaphor for real-life problem-solving frustrations – a kind of “anti-puzzle”.
After the user has joined all the pieces where they belong, the distorted image will slowly
revert back to its clearer form as whole (not split into pieces). A few seconds later, the
image will split back once again into many pieces, triggering the process all over again.
Those few seconds in which the user sees himself or herself assembled correctly will act
as a “moment of clarity”. In other words, it will be clear what the problem was, the
process he or she went through, and how it got solved, right before the process starts
again. This moment of clarity will serve as a reward for the user.
9
Personally, I see this process as a metaphor for my own personal experiences
developing this project, and with many other frustrating or elusive aspects of problems in
life. This topic is something that everyone can relate to, and hopefully after interacting
with it the message will get across.
1.4 Design Questions.
1. Does the experience result in the user’s understanding of the messages I’m
trying to convey?
2. Will the user feel frustrated? If so, can this frustration be the means to the end of
his or her realization that his or her frustration and how he or she copes with it is
the point of the problem?
3. Will the puzzle act as an abstract way of seeing and understanding how we deal
with the process of solving many challenges, issues, and frustrations we
constantly face in life?
4. Will the users be curious enough to keep trying to assemble the puzzle as it
becomes more and more elusive?
5. Does the process of the image getting distorted resemble a new process of
disorder and chaos?
6. What visual effects can I incorporate into the piece that match the look and feel of
the piece?
10
Chapter 2: Domains and Precedents
When I started to analyze and develop my different ideas in order to find out and
understand what I wanted to do as my thesis project, I had a general sense of the topics
I wanted to embark on, such as puzzle games, elusive elements in games, playing with
perceptions, human interaction, portability, and physical computing. From these, solving
puzzles and brainteasers was the idea that appealed to me the most, since it’s a topic
that has always fascinated me, and I liked the idea of somehow incorporating and
applying them into my thesis. Shortly after, I came to the conclusion that I wanted to
build a puzzle device or installation.
2.1 Main Domains.
After carefully analyzing the different topics and fields that interested me the most, I
decided that these are the domains that best fit my thesis: jigsaw puzzles, order and
disorder, and elusiveness.
2.1.1 Jigsaw Puzzles.
There are many kinds of puzzles, ranging from simple and fun games aimed to entertain
users, to the purely logical, mathematical, and brain enhancing. The types of puzzles
that have always interested me the most are jigsaw and logical (better known as
brainteasers). Between these two, I decided to work with jigsaws.
By definition, a jigsaw puzzle is “a tiling puzzle that
requires the assembly of numerous small, often
oddly shaped, interlocking and tessellating pieces.
Each piece has a small part of a picture on it; when
complete, a jigsaw puzzle produces a complete
picture. Jigsaw puzzles were originally created by
painting a picture on a flat, rectangular piece of
wood, and then cutting that picture into small pieces with a jigsaw, hence the name.
11
John Spilsbury, a London mapmaker and engraver, is credited with commercializing
jigsaw puzzles around 1760” (Spilsbury).
There are some key elements about jigsaw puzzles that motivate me to work with them.
First of all, they are constrained to a static image. The point is to put back together a big
image that is divided into small pieces. The reward is to discover what the image looks
like as a whole once it has been assembled in its entirety. The levels of challenges and
difficulty are mostly based on 2 things: the number of pieces, and how clear the image is.
In the case of my project, what interests me the
most is the idea of finding alternate ways to
create jigsaw puzzles that don’t necessarily rely
on a static image, but of an image that keeps
constantly changing and evolving into something
else. For example, by using a live feed from a
video camera as source image for this jigsaw
puzzle, you get a moving image that will always
change and vary according to what is at the other side of the lens. The fact that it’s a live
image provides a sense of movement that is unique and inexistent in current puzzles of
the same category.
2.1.2 Order and Disorder.
A key element when assembling a jigsaw puzzle is that of bringing pieces in disorder
into order to see the bigger image. When I started the thesis process, I kept thinking of
this topic as I tried to apply it to my different thesis iterations.
Everything about my thesis project involves this process of order and disorder, not just
the assembly of the individual pieces. The quality of the split image starts being clear,
but with every piece that is placed into the grid, the image gets more and more distorted.
In other words, the image goes from order to disorder, while the arrangement of the
pieces goes from disorder to order. There is always balance in the system.
12
2.1.3 Elusiveness.
An interesting element when building a puzzle is how the users feel as they progress.
The hardest part of assembling a puzzle is at the beginning when users don’t have a
clear idea of where the pieces go, or what part is it that they’re starting to build. As they
go along, the more they build, the less complicated the process becomes. When it’s
almost completed, it’s easier for the users since they have a better sense of the image in
general and where the remaining individual pieces need to go. Naturally, the more a
person assembles a puzzle as a whole, the better he or she feels about the process.
The interest I have in relation to this domain is to invoke an opposite effect or feeling in
the user who builds the puzzle. In other words, the more the user tries to assemble a
puzzle, the harder and more frustrating it becomes. A puzzle that becomes more and
more elusive.
This is a domain I discovered as I worked on my thesis during the semester and tried to
understand what were the feelings behind my concept. The more the users struggle to
solve the puzzle the more frustrating it becomes, as their inability to clearly see and build
the image increases with every effort to organize it.
One clear example of elusive systems that
have inspired me is what is known as Room
Escape games.
Most
of
these games
operate from a first-person perspective, and
they begin when the character mysteriously
wakes up in an unknown room with no idea
how they got there. The room doesn’t have
any way out of it, except for a locked door.
The objective is for the user to escape the room by finding clues, hidden objects, and
solving puzzles. Most of these games become extremely frustrating to interact with since
the clues and objects are discoverable by clicking on them, but the programmers make
sure to hide them in such a way that is not apparent to the user unless he or she has
spent a lot of time clicking throughout the screen. In addition, most of the puzzles within
13
the games can be extremely complicated and hard to understand and figure out.
Typically, the more the users try to escape the room, the more challenges and new
puzzles they will find. Usually, after playing for 10 minutes, these games start to become
extremely frustrating. Users who claim to have spent hours stuck on the game swamp
message boards through the Internet for help and solutions.
Even though I don’t want to make my thesis based on these games, I want to
incorporate this feeling of elusiveness. The more we try to solve the problems or beat the
objectives, the harder and more elusive they become.
2.2 Precedents.
As I worked with these ideas discussed above and investigated all my domains, I started
to identify precedents for them. These precedents work as a guide to see what has been
done that relates to my work-- the technologies involved, and sometimes, as inspirational
references.
The first example is an electronic
physical interactive puzzle or game
called the Tiletoy (Tammenpää and
Blackburn). The Tiletoy consists of
“a set of modular blocks with LED
displays, which come together as
multiple games and puzzles for users to interact with”. The elements that I like the most
about this project are the content and the technologies involved in it, since it is a physical
puzzle game that users can play and interact with. Each block works independently from
the other, but reacts in a specific manner according to the block adjacent to it, all in a
wireless fashion. In a way, the blocks act as a jigsaw puzzle since you have to place
them in the order in which they belong in order to uncover the big solution, or in this
case, the big LED animation. The liberty and the element of playfulness that users have
with these blocks is certainly an element I constantly thought about when thinking about
ideas for my own puzzle project.
14
Another example is Drawn (Lieberman), which consists of virtually interacting and
manipulating a drawing made out of a hand-made sketch by using your hands and
fingers. The idea of being able to move and manipulate a virtual projection of something
that doesn’t exist there fascinates me. I started to think of how the user can interact with
the puzzle pieces in the same fashion of using hands and fingers to move them around.
The pieces do not physically exist, but the users manipulate their position by touching
them on the screen.
Four-Hand Composition (Cho) shares many elements that
interest me. Hsin-Ming (Jimmy) Cho was interested in doing a
thesis involving hand gestures. The result was a game in which
two users can interact with a virtually projected board using
only their hands. The hand gestures and movements would be
captured and used to alter the game’s projection. This is yet
another example in which users affect a virtual object with hand
movement, as with the example seen above.
One example of a microworld that has inspired my thesis is the game Fl0w (Chen). The
“game” doesn’t really have an introductions, no instructions, and the game starts of with
a little creature swimming around. As you eat other smaller creatures, you start to notice
how your own creature changes and evolves. There are other creatures that have
different specific purposes (go deeper or dive back). There is no real purpose as to
15
"beating the game" besides that of evolving your creature. The narrative and the game
play are really plain and simple. The user needs to follow a certain order: navigate the
stages and eat the creatures so the creature can evolve a bit, then find the creature that
transports you back and forth within the stages. The background colors blend in with the
setting, getting darker as the user navigates deeper. The creatures vary in size, and you
can try to eat bigger ones but sometimes they will kick you back some stages before. In
other words, there's an order of progression regarding eating these creatures.
This progression (level jump) is an element I wanted to add to my thesis. As the user
assembles the puzzle pieces in their corresponding places on the gird, the image within
the piece gets affected. Each time the user places a piece where it belongs, the level of
complexity increases and it becomes harder to progress. This is similar to the stage
navigation in Fl0w.
Another Fl0w element I wanted to incorporate into my thesis is the mood of the game.
Every movement, progression, and change within the game is done in a subtle beautiful
way that goes in "flow" with the rest (it doesn't feel forced, but more like it’s a part of that
beautiful realm). I also love the music, the sound, and the peaceful and quiet ambience.
The short film called Powers of Ten (Eames) serves as another inspirational precedent
that deals and explores different scales of the Universe (outside and within us) by
navigating through layers in factors of ten. Each layer is rich in visuals, mood, and feel,
independent from the rest. What I like about this piece so much is how each layer is
visually stunning and can stand independently on its own from the rest, same as what I
was looking for with each layer of distortion in my thesis.
16
Having talked about Room Escape games before,
there’s a particularly set that are extremely popular
among fans of this recently emerged genre. These
games are Blue Chamber, Crimsom Room, Viridian
Room, and White Chamber (Fasco-cs). These games
are known for being extremely good and challenging,
and each is characterized by the particular color of the
room. In all of them, the user’s objective is to escape the room by finding clues and
hidden objects. These games have an elusive nature,
since the puzzles and clues can be extremely
challenging and well hidden. In order to uncover them,
you have to click on the exact same spot at different
intervals of the game, which prompts the user the
need to inspect the same areas over and over again.
Most times there is an order in which events on the
room unfold (one must solve certain clues in order to
uncover new ones), and what was an empty space at
one time can be occupied by a minuscule “dot” later
on which users need to find. This exploration process
can be quite frustrating.
These games have an elusive nature, especially for inexperienced users or those with
little or no patience who are trying it for the first time. Nonetheless, even though the
17
games can be tedious and frustrating, users still play because they want to know what is
behind the door. They want to discover what happens once they escape. Will there be
salvation for the character? Will all the time and frustration invested pay off? Similarly, I
want to incorporate this feeling of elusiveness into my thesis. I want users to feel
frustrated as they interact and “play” with it, and this frustration should grow more and
more. I want them to keep interacting and solving the puzzle no matter how elusive it
might turn out to be, in order to discover what is the outcome or reward of all their effort.
18
Chapter 3: Methodology
3.1 Thesis Components.
The thesis installation consists of a table top touch screen monitor with an activated
video camera above it. When a person approaches the screen, the live feed taken by the
camera splits into pieces that are displayed in random order around a grid in the middle
of the screen. The user will attempt to assemble the original image by placing the pieces
in their respective places inside of the grid. With every piece that the user puts back on
its right place, the general image gets more distorted. In a way, the puzzle acts as a
broken mirror that the user attempts to reassemble, but the final image will be completely
unrecognizable.
3.2 Form.
The main forms and elements being used to shape the project are: physical computing,
screen based interactive installation, and elusive play.
Since the thesis project consists of a puzzle installation, the mentioned forms are vital for
its execution. The physical computing form involves having touch screen interface and a
mounted camera to capture the users who interact with the piece. There will also be a
wall projection displaying what is shown on the screen, which allow other people (who
may not necessarily be interacting with the piece) to observe what is going on. The
screen based interactive installation form means that the users will participate and
interact only with what is being displayed on the screen. These forms depend on specific
software such as Max/MSP/Jitter and the Javascript programming language to make the
interaction work according to its design intent.
Lastly, the play form comes together in the puzzle itself, through puzzle- building and
usage. The interaction experience becomes more elusive as the user progresses.
3.3 Building and Process Iterations.
From the moment I started this project, I created many iterations and possibilities
regarding what the project could do -- how it could work. Below is a general description
19
of these basic ideas and iterations, which lead up to the final delivered project.
3.3.1 Gathering Ideas.
As the semester began, I knew I wanted to work
with a puzzle device. Being inspired by classic
puzzle games such as electronic Simon (Morrison),
Rubik’s Cube (Rubik), and even jigsaw puzzles, I
wanted to create a portable device a person could
interact with while he or she had a fun time. I wanted
to do something a person can take or leave as he or she pleases. I created a very basic
and simple diagram of a possible interaction with said device.
The next step I underwent was to think of how could
I engage the user even more. I wanted the
experience to be somewhat addictive, or at least
leave a small desire in the user to go back and
experiment with it some more. I thought about
usability, particularly how multiple users could
benefit and enhance the experience. This led me to
think about tabletop gaming with multiple users.
After exploring some of the tabletop learning benefits (and many interactive tabletop
precedents), and tabletop games, the possibility of doing a tabletop puzzle installation
20
was more reasonable than doing a portable device. With this in mind, I started to design
different interactions and usages. Most importantly, I started to think about the content of
the puzzle itself.
3.3.2 Iteration 1: Reconstruction of Oneself.
After consulting many resources related to
my project, I came up with my first project
concept (or iteration): a user stands in front
of a table with white pieces on top of its
surface. A camera takes a live feed from
the user, which is processed and split into
pieces.
These split
images
are then
randomly projected on top of the white
pieces lying on the table. As the user
moves the white pieces, the images being projected on top of them follow accordingly.
The user tries to assemble the pieces, from chaos to order, and he or she finally
discovers that the projection is their own reflection.
Top View
Bottom View
As seen on the diagrams, the projection is made on top of the white tile pieces, while the
color tracking is done using the colors on their bottom side. Using Max/MSP/Jitter, I
21
found solutions to some of the technical challenges such as image splitting, video
capturing, and motion and color tracking. I also experimented with touch screens, and
discovered important technical details that are vital for my project, such as acquiring
coordinate numbers of the places in which the user touches a screen.
3.3.3 Iteration 2: Puzzle Game.
As I kept trying to explore and understand my ideas regarding the project, I ventured into
another possibility as a form altogether. It still involved a puzzle, but this time focusing on
it being a game.
The user needs to use the movement and
placement of the tiles in order to be able to play
correctly and make progress. Nine grids can be
rotated in order to allow a ball to travel from start
to finish. The method in which the ball travels is
by “jumping on trampolines” from grid to grid,
until it reaches the end. The user has a limited
number of rotations he or she can make;
calculating that the ball will travel correctly once
it leaves the starting point. If the user rotated the
grids successfully, the ball will reach its destination. Otherwise, the ball will go “outside”
the grids, and the player will loose.
After analyzing this style of game play, I found that other than being a game in which
users would need to move and place tiles in certain order, it did not posses most of the
some of the key ideas and elements that I wanted to explore in the first place. Most of
these ideas were essential as part of my thesis exploration process, so I immediately
decided to go back and continue working with the first concept.
22
3.3.4 Iteration 3: Modular Puzzle Tiles.
I started to think about the design and physical
shape of how the tiles, and how exactly would they
interact when joined together. On the right, you can
see a very simple example: when these tiles are
separate, they show parts of a train acting on their
own. As soon as you join the pieces, the train
responds. If you place the tiles together in the right
order, the train will move away. If you place them
incorrectly, the train will collide and blow up.
Train parts away
Train collides and explodes
The content of the train is merely an example to illustrate the kind of modularity involved
in this iteration. Eventually I would develop new contents that would fit better in this
process.
As I started to understand how the dynamic of
the
system
should
work,
I
developed
a
Max/MSP/Jitter patch that would allow me to get
a more real sense of how the pieces would act
and interact. When I first did Iteration 1, I
discovered how to track an object by its color by
identifying the coordinates of where the colors
23
were located on the screen. Using this method, I incorporated new content based on the
process of plant growth. As seen on the picture, when the tiles are separated from each
other, video will play for each one of them.
When two or more colors came close to each other, different videos played according to
the number of pieces and their position. The result was a very complex mapping process
consisting of video arrays that were constantly triggered with their position and proximity.
Red (water) and blue (soil) create mud.
Blue (soil) and green (seed) eventually dry up
because of lack of water.
As I developed this iteration, I faced various technical problems involving color tracking
of the pieces, slow and jumpy video playback, and light conditions. On top of that, I was
unhappy with the way the project was developing up to that moment, so I decided to
move on.
3.3.5 Iteration 4: Flow of Time.
After studying and analyzing all the ideas I had worked with so far, I started to think
about concepts such as the flow of time, cause-and-effect reactions when assembling
different pieces, and ideas and topics of order and disorder. I decided to incorporate
these into my thesis, and came up with a whole new concept.
3.3.5.1
Time’s Disorder.
For this iteration, the project was going to be set as a tabletop touch screen. When the
user approached the screen, he or she was going to be presented with a set of images
24
(either moving or still), split and disordered across the screen. The user would have to
assemble these pieces from disorder into order. Once this was done, the image evolved
and formed a smaller part of the next scenario that the user must interact with. The
messages portrayed within each scenario consisted of different perceptions or ways in
which time flows. When all the scenarios were completed, the process would repeat
itself by going back to the first scenario, connecting seamlessly with the last stage in the
same manner, by zooming out and presenting the puzzle image that started it all. This
process would continue infinitely. The following is the storyboard of this process.
The main software tools used in this iteration were After Effects, MAX/MSP/Jitter, and
Javascript. After Effects was used for the preparation of the content of the videos, while
Max/MSP/Jitter and Javascript was used for processes such as splitting a video image,
video playback and manipulation, and navigation throughout the different scenarios.
As I started to deliver the first scenarios of this iteration, I still felt it wasn’t quite what I
had envisioned as my final thesis project. Eventually I decided to go back to my first
iteration, the one consisting of assembling your own image, but to also take some of the
most important and significant elements of the latest iterations, such as loops, causeand-reaction, deconstruction and reconstruction.
25
This process was vital for my thesis project, since if it weren’t for all of the iterations and
steps I had to go through, I would have never ended with the final thesis project as it is.
3.3.6 Back to Basics: Finalizing my Thesis.
After going through this process of developing a thesis project for almost a year, I
realized that I wanted to go back and work with the very first iteration of reconstructing
oneself, while incorporating some of the elements I explored and worked with in prior
iterations.
3.3.6.1 General Idea.
The
current
thesis
consists
of
an
interactive puzzle installation displayed in a
tabletop touch screen. When the user
approaches the screen, he or she will see
25 pieces of a split moving image, which
are scattered randomly across the screen.
The source of the split image being
displayed is a video live feed of the user
who is interacting with the piece. Users
must assemble these split pieces into a grid, and as they do, the image within the pieces
will become more and more distorted until it becomes virtually unrecognizable. There will
be a wall projection of what is being displayed in the monitor.
Below are the different prototypes that were developed prior to the finalized version.
3.3.7 First Prototype.
As part of the first prototype, I created a stage and incorporated the pieces with drop
shadows into it. The pieces would float randomly across the screen. When two pieces
that go next to the other are brought close to each other, they snapped into place
creating one new piece.
26
The main problem I faced with this prototype is that once the pieces snapped into
position, they couldn’t be moved around anymore. If the user assembled different sets of
pieces across the screen, there wasn’t any way that the puzzle could be solved, since
the sets would be snapped into different positions. Also, if the user started the puzzle far
away from the center, it was likely that the placement of some pieces would have ended
up being off-screen.
3.3.8 Second Prototype.
For the second prototype, a grid was incorporated so that the users could place the
pieces in their corresponding positions. The benefits of incorporating the grid were that
ideally the user would have a better sense of the placement of the pieces, eliminating the
position issues from the previous prototype.
27
After testing this prototype, users had a better sense of how to assemble the puzzle
together, even though there still was one big challenge: how does the user know where
to start? All the times this prototype was user tested, the users didn’t know where to
place the first piece, so they would start randomly in any place of the grid only to realize
later on that they had to relocate all the pieces.
3.3.9 Third Prototype.
Trying to find a way to solve the first-piece location issue described above, I came up
with a temporary solution. It consisted on notifying the user when a piece goes in the
right place of the grid by slightly changing its color to a light green. In other words, the
user would grab a piece and start hovering it over the different places of the grid, and
when the piece turned green that meant the user could let go of it and the piece would
snap into place.
Once the users let go of the pieces, or moved them out of their corresponding areas, the
color of the pieces turned back to normal. If the users let go of the piece in or near their
corresponding places on the grid, the pieces snapped into place and remained locked
from that point on.
This “color-lock” dynamic was incorporated into the puzzle after observing how users
28
spent long amounts of time trying to understand where to start placing the pieces. Even
worse, sometimes they would realize that they have “assembled” around 90% of the
puzzle in the wrong place just because they didn’t start in the correct spot.
This solution was soon discarded since it had one major flaw: the users would not
assemble the puzzle by its content anymore, but instead by looking for where the piece
turned green. This happened more and more as the users progressed through the
building process.
3.3.10 Final Version.
After doing some final changes, I developed a version of the puzzle that solved all the
previous issues encountered before. First of all, when the puzzle starts all the pieces are
randomly arranged around the grid, except for one which is placed in a random position
inside of the grid. The piece inside of the grid helps the user have a better idea of how
and where to start. They would need to find the neighboring piece to the one that is
already in place. It serves as a starting point.
29
In addition, when users place pieces in the wrong part of the grid, the piece turns red for
a second and jumps into a random place off the grid. When the piece is placed in the
right place, it turns green for a second after it snaps into place.
Once the first piece snaps into place, the image blurs. The image will continue to change
and become more unrecognizable as the puzzle is assembled.
30
By the time the user has assembled the last piece, what is displayed is a new and
completely altered image from the one at the beginning. When the user starts to build
the puzzle, the pieces are scattered across the screen in complete disorder, yet the
image is clear. As the user assembles the pieces more and more, he or she is creating
order yet the image gets more and more unclear. There is an element of balance in this
process.
31
After all the pieces are together as one, the user is rewarded with a backwards
progression of the distortion of the image, starting at the last screen and clearing itself all
the way to the first screen. The last thing the user will see is his or her live image
completely clear and assembled into place, before the image is re-split into 25 pieces,
and the loop cycle starts again.
3.3.11 User Interaction.
The following illustrations show an ideal representation of how does the user interaction
works.
The user approaches the installation. The image is split into pieces.
32
The user sees himself or herself split into the pieces scattered through the screen. He or
she starts to assemble them into the grid.
As the user puts the pieces in their rightful place, the image gets more and more
distorted. At the end, the image is completely unrecognizable.
33
3.4 Behind the Magic: Technologies.
As previously stated, the technologies used for this project consist of Max/MSP/Jitter,
and Javascript. The Javascript code goes inside of the Max/MSP/Jitter patch for
controlling certain elements and objects directly from the script, which would otherwise
be extremely difficult to handle.
Below are screen capture shots of what the Max/MSP/Jitter patches look like while
running the application.
This is the main Max patch (above). As soon as it opens, it launches the function objects
that control rendering and drawing elements on the display window, draws the main grid
and the individual puzzle pieces, and opens the live feed from the camera. In addition, it
invokes the Javascript code (the object that says “js mything14.js near the middle of the
screen).
34
The Javascript code is vital to the Max patch since it handles the properties of each one
of the pieces of the puzzle, such as color, shape, size, texture, and position. In addition,
the code controls that whenever the patch is started, the pieces load randomly around
the grid (not inside of it). It also checks constantly for the position of the pieces, in order
to do operations such as changing colors and locking the pieces in their rightful place.
The second patch acts as a sub-patch of the first (above). It controls all the effects that
are added to the video whenever the pieces are snapped into position. In the image,
there are 11 effects on (crosses on the boxes). The resulting image can be seen on the
right side.
35
The third patch (sub-patch from the second) controls the rewind process of the effects
(above). In other words, once all the pieces are snapped into position, this patch will
activate and turn off all the effects (the boxes in the second patch). Once all the effects
are off, this patch will make a 5 second countdown (the time the user sees himself or
herself clearly) before restarting the process all over again.
3.5 Usability Tests.
During the building process of each one of the prototypes of the puzzle up to the
finalized version, I allowed outside users to interact with it and give me feedback related
to the experience and the usability of it. The testers included design students from the
Design and Technology department (both Undergraduates and Graduates at the time),
and other friends with different degrees and backgrounds in design and arts.
36
3.5.1. Procedure.
The first thing I did when users tested the puzzle was to observe how they initially
interacted with the pieces when they are not familiar with the system’s rules. One trait all
users shared was that the first thing they did was click on the piece that caught their
attention the most. Most times they would go straight for the piece showing their eyes, or
a lot of movement.
Once they grabbed the first piece, I started to time how much time they took until they
placed it in the right place. One of the biggest concerns I before my finalized version was
the problem I described before in regards to how the users start to assemble the puzzle.
Most users took the first piece that caught their attention, and placed it randomly across
the grid. They didn’t know where the piece belonged for sure, and even worse, they
didn’t know there was a way to know according to how the piece belonged (it changed
colors when it hovered where it was supposed to go). I usually had to point these details
out to the users. After they placed the first piece in its rightful place, they had a general
sense of how the puzzle worked and would proceed to build the rest of it. The downfall of
this was that after they placed the first piece, they continued to assemble the puzzle by
looking where the pieces changed colors, rather than by trying to find out where it went
based on the content image of the piece. This detail made the piece lose some of its
meaning, which is of the constant struggle to assemble ourselves and that which
surrounds us.
In my finalized version, I made sure to approach these issues and I corrected them. I
modified the color indicators so that they would only turn on when users placed the
pieces in the right or the wrong spots. This way, the users were forced to assemble the
puzzle based on its content, not by the color changing of the piece. In addition, the
puzzle starts with a piece already in place, which gave them an idea of the general
placement of all the pieces.
These changes made all the difference in the process of building the puzzle, and it
solved my main challenges. Users tested the puzzle, and their behavior was as I
expected it to be. First, they looked for the piece that in their opinion goes right next to
37
the one already on the grid. That first piece served as their starting point. Second, they
had a clear idea on whether pieces were placed correctly or not based on the color
response and reaction (locking into grid vs. jumping outside). They were struggling to
find what pieces belonged in the right spots, based on what their content looked like. All
of them informed me that as they progressed, the puzzle definitely got more and more
difficult and frustrating, but they kept going since they wanted to see what the next effect
would be like. Also, there would be fewer pieces, which made things easier for them as
they got closer to discovering what the end result would be.
All of the users liked the reward (or outcome) and thought it was a fair payoff for the
process they just went through. My project has succeeded.
38
Chapter 4: Evaluation
After building my thesis project and testing it with friends and people who were unfamiliar
with it, I got various reactions and feedback that allowed me to think and analyze how
my project has succeeded. When I showed the project in my final thesis presentation to
the teacher, students, guests and outside critics, their reactions were extremely positive.
The following are a set of criteria points that I set at the beginning and through the year
process in order to help me keep in track of it.
4.1 Criteria for Evaluating Success.
4.1.1
Purpose.
These are some key ideas and questions related to the reason why I’m doing this
project. They can be considered general design questions.
a. What is the purpose of my piece?
b. How do I incorporate my desire to build an interactive puzzle with the
ideas and feelings I want to convey?
c. Will the message and the feeling I’m trying to convey through this piece
be clear?
d. Will the user understand the purpose and the importance of building the
puzzle, and will he or she learn or acquire a new perspective?
4.1.2
Aesthetic.
This is an important element of the project since the message is all about how we
see ourselves. The aesthetic of the piece in general, and of its independent
components of it (such as the pieces and the grid), should go in par with the
concept.
The pieces start up looking nice, clear, sharp and clean. As the puzzle evolves,
so does the look of these pieces. The image within the pieces starts to loose
39
focus, shape, and form. The colors re-adjust, the movements become less
apparent. The grid and the surrounding areas are still neutral, but now the
content within the piece is different from the beginning. This goes in par with the
personal struggle within the user. He or she starts thinking he or she has a clear
idea of what is to be done, only to be surprised by a new unexpected change
each time.
The color identifiers that let know the user if the piece is on the right place or not
are green and red, colors we all associate with “stop and go”, green for good and
red for bad.
In addition, as with some of the precedents I mentioned in Chapter 2 of this
document, I wanted to incorporate feelings of cleanliness, harmony, pureness,
and a futuristic feel. This has proven to be a challenge, due to the fact that either
the pieces are scattered throughout the screen in disorder, or the pieces are
assembled into one big image that looks fuzzy and unrecognizable. Both of these
possibilities work opposite from my desire to convey a “clean” piece, so I started
to explore alternate ways of expressing these aesthetic desires, such as music or
ambient sounds, and background colors. In the end I was happy with my finalized
version of the aesthetics of the project, so I decided to leave it as it was.
4.1.3
Usability and Interaction.
Different elements related to the user’s interaction.
4.1.3.1 Interest.
As I researched on my domains, I always kept it mind questions such as:
•
How do I make users aware and interested in my project?
•
Once I have the user’s attention, how do I keep them
interested?
•
What elements in games and puzzles enhance the
experience enough to keep a person interested? Which of
40
these elements do I need to adapt into my piece?
From previous research on this topic, I knew that one important element
in order to keep the user interested was surprise. This fits well into my
puzzle, since it keeps changing the content of the image as the user
interacts more and more. Every piece that is put together brings a new
surprise twist into the puzzle, and makes it more challenging.
4.1.3.2 Playfulness.
Inasmuch as the project is in the form of a puzzle, I wanted it to invoke
the same feeling of playfulness as assembly of a physical jigsaw or
picture puzzle.
4.1.3.3
Frustration.
As users finished interacting with the puzzle, most of them express how
frustrating it was. Particularly, the more they progressed, the more
frustrating it got.
Having the user feel frustration as he or she builds the puzzle is a key
element. The fact that he or she feels frustration as they build it goes
hand-in-hand with my concept of struggle and trying to find oneself.
4.1.4
Problem Solving.
Part of the ongoing process. Some pointers are:
•
As I build and test the project, I need to keep testing and solving technical
and interaction issues, given the limited amount of time prior to the end of
thesis year.
•
After designing and implementing this interactive puzzle, have I answered
my main design question and most of my domain questions? What have I
learned as part of the process?
41
4.1.5
Future of the Project.
Below are some questions regarding my thinking about the future of the project:
•
How will this project evolve next, particularly after the Thesis Show is
over?
•
What are possible elements I can incorporate into it to enhance the
message or the experience?
•
Do I want it to be part of future exhibitions? Or should it be just a part of a
bigger project?
4.2 Feedback and Concluding Thoughts.
After presenting my thesis to the guests and critics, the general response seemed to be
extremely positive. Comments by the critics included the suggestion that I should look
new methods of distortion such as altering colors, noise, image blurs, and zooming
capabilities. Critics also suggested that the image restoration should be less forced once
the puzzle is solved. A possibility might be something that looks more like a subtle and
smooth progression of image clarity, rather than the “forced turning-off” effects. They
also said that it might be better if the project didn’t resolve with such clarity. In other
words, that there’s inconsistency there. Although these ideas sound logical based on the
progression, aesthetics, mood, and feel of the project, I don’t think they go in par with the
message I’m trying to get across, and the project’s reward. The idea is that once you
have struggled and fought with the frustration, the user gets rewarded with a clear
image. It symbolizes clarity, understanding, and harmony. They can finally get a glimpse
and see the clear picture, not just parts of it.
Another suggestion by a critic on a private meeting was to incorporate an effect that
would consist of the pieces getting out of place according to the movement within the
pieces. For example, if the user starts to jump and shake this would cause the pieces to
shake and fall off towards the sides. Although this idea sounds very interesting, I don’t
see how it would fit or relate with my project’s ideas and concept. Perhaps it’s related as
a part of the struggle process, in which users need to keep control of themselves? It is
my opinion this would be adding levels of complexity to an already complex project.
42
If user interaction is directly with a mouse-driven screen based puzzle and not a physical
puzzle, an important element is taken away from it, which is that of tangibility. With
analog puzzles, just by touching the pieces and physically comparing them to others
makes the experience very different from playing with an interactive puzzle that depends
on a mouse. This is the reason why I decided to incorporate a touch screen into the
project, so that the user feels like he or she is really touching and dragging the piece by
using his or her fingers. After gathering some thoughts and feedback on this perspective,
users in general agree that a touch screen, or even a digital pen tablet, would help to fill
this gap. In the case of the pen tablet, even though it would act in a similar way as a
mouse, it is still less commonly used, which would make it a little less generic. It would
also have a tip side used to “write” (or in this case move the pieces around) and an
“eraser” side, which could be used to restart the puzzle if desired.
4.2.1 Reflecting on the Thesis Process.
After a whole year of struggle and hard work, I believe that I have succeeded in creating
a fully working thesis project. Not only that, but I learned many things as part of the
process of studying the topics at hand.
As I started to think and study the puzzles, I kept telling myself “my thesis itself is a
puzzle”. Why? First of all I started in zero, not knowing what I wanted to do or even less
how was I going to do it. Soon enough, I started to see the pieces and ideas scattered
throughout my mind, and I knew I had to order them, yet I didn’t know how. Using the
thesis meetings and papers as a framework (a grid) for these ideas, I kept struggling
constantly trying to see the clear final image, trying to understand what exactly was it
that I wanted to do. The more I tried the more my thoughts became more and more
unclear. At one point of the last semester, when I had no idea anymore how to proceed
or what to do next, I decided to follow my instincts and look back at the process I had
endured up to that moment. I reviewed all the steps from the very beginning, all the “ups
and downs”, and that is when I had a moment of clarity and saw the bigger picture. I
knew what I had to do.
43
Curiously enough, I never imagined that my final project would be precisely about the
process of said project. It ended up being a reflection of what I went through as I tried to
find what it was that I wanted to do for my thesis, and as I tried to find myself within this
process. It was something I had to go through, and would have never occurred to me at
the beginning of the semester. There was no way I could have solved the puzzle without
first having a direct experience and understanding what I was up against.
Same as with my final project, I tried to discover myself and what is it that I wanted to do
as I tried to put the pieces into place. The more I tried the worse it got, and the more
elusive the process became. But in the end I was rewarded with clarity. I gained a clear
understanding of the process I went through, realized where I was at the beginning, and
how that is different from where I am now.
Hopefully it will be a long while before the pieces break up again, and the never ending
process starts once again.
44
Works Cited
Chen, Jenova. Flow in games. 14 April. 2006. 8 February. 2007 <http://
intihuatani.usc.edu/cloud/flowing/>
Cho, Hsin-Ming. Four-Hand Composition. May. 2006. 26 Sept. 2006 <http://
a.parsons.edu/~hmcho/thesis/index.htm>
Eames, Charles, and Ray Eames. Powers of Ten. Powers of Ten. 2006. 18 May. 2007
<http://www.powersof10.com/>
Electroplankton. Electroplankton. Wikipedia.org. 26 January. 2007. 27 January. 2007
<http://en.wikipedia.org/wiki/Electroplankton>
Fasco-cs. FASCO-CS. 30 March. 2006. 18 May. 2007 <http://www.fasco-csc.com/index_e.php>
---. Blue Chamber. Takagi, Toshimitsu. 2004. 18 May, 2007 <http://www.fascocsc.com/works/bluechamber/index_e.php>
---. Crimson Room. Takagi, Toshimitsu. 2004. 18 May. 2007 <http://www.fascocsc.com/works/crimson/crimson.php>
---. Viridian Room. Takagi, Toshimitsu. 2004. 18 May, 2007 <http://www.fascocsc.com/works/viridian/index_e.php>
---. White Chamber. Takagi, Toshimitsu. 2006. 18 May. 2007 <http://www.fascocsc.com/works/white/white_e.php>
Lieberman, Zachary. Drawn. 2006. 26 Sept. 2006 <http://www.thesystemis.com/
drawnInstallation/index.html>
Microworlds. Inquiry Based Learning through “Microworld” Exploration: An Exploration of Emergent Systems
in our Environment. University of Maine Department of Computer Sciences. 8 February. 2007
<http://www.umcs.maine.edu/~larry/microworlds/microworld.html>
Morrison, Howard J., and Ralph Baer. Simon. 13 Sept. 2006. 26 Sept. 2006 <http://
en.wikipedia.org/wiki/Simon_(game)>
Rieber, Lloyd P. Seriously Considering Play: Designing interactive learning environments based on the
blending of microworlds, simulations, and games. Educationan Technology Research & Development,
44(2), 43-58.
45
Rubik, Erno. Rubik’s Official Online Site: Brief History of the Cube. 1998-2007. 15 April. 2007 <http://
www.rubiks.com/lvl3/index_lvl3.cfm?lan=eng&lvl1=inform&lvl2=medrel&lvl3=histry>
Spilsbury, John. Jigsaw Puzzle. 14 May. 2007. 18 May. 2007 <http://en.wikipedia.org/wiki/Jigsaw_puzzle>
Tammenpää, Tuomo, and Daniel Blackburn. Tiletoy. 26 June 2006. 26 Sept. 2006
<http://www.tiletoy.org/>
46