from idc2014.org

NoteCubes: Learning Notes and Rhythms with Spatial
Sense
Wanfang Diao
Ali Momeni
Aisling Kelliher
Carnegie Mellon University
Computational Design Lab
Carnegie Mellon University
School of Art
Carnegie Mellon University
School of Design
wanfangd@andrew.cmu.edu
momeni@cmu.edu
aislingk@andrew.cmu.edu
ABSTRACT
Learning key concepts in music theory, such as rhythm and
harmony, pitch, and counterpoint, is a complex process for young
learners. Problems encountered in building the perceptual
knowledge of these concepts lie in a lack of flexibility in
practicing these concepts without mastering a musical instrument.
This paper describes NoteCubes, a set of tangible interactive
construction toy blocks for children to explore and build
understanding of musical concepts by creating simple melodies of
their own based on the spatial reasoning with a playful interface.
We also report initial user feedback from a small-scale exhibition
and discuss future directions for further development and
refinement of this tangible musical toy.
their own based on this spatial understanding. The cubes can be
arranged freely to form different 3D geometries. When a cube is
triggered by light, it can play a note or an empty beat and trigger
its "neighbor cubes" (left or right & up & down) to play notes.
NoteCubes encourage children to create melody and rhythms by
exploring different spatial arrangements of cubes based on the
understanding of musical concepts.
Instead of using complex and invisible wireless technologies—
e.g. Bluetooth or RFID--we implement the information
communication among blocks with Visible Light Communication
(VLC).
Categories and Subject Descriptors
K.3.1 [Computers and Education]: Computer Uses in Education
General Terms
Design, Experimentation
Keywords
1. INTRODUCTION
1.1 Problem Addressed
(a)
(b)
Learning concepts in music theory, such as pitch, rhythm and
chords, is a complex process. Problems encountered in building
the perceptual knowledge of these concepts lie in the difficulty
with practicing such concepts prior to instrument mastery. In
comparison, mastering a musical instrument requires an embodied
understanding of mapping between gestures and musical concepts.
Therefore, the iterative learning process that builds such mastery
on traditional musical instruments tends to be very physically
demanding, gesturally unintuitive and potentially frustrating to the
novice; these parameters post an obstacle for musical education in
very young children.
Our idea is to tackle the frustration barrier and provide a play
experience for practicing musical concepts with an intuitive
mapping metaphor between physical operation and musical
elements. This musical intervention is aimed at children aged 5-12
with the goal of helping them build interest and curiosity in music.
(c) Structure created in (a)
1.2 Concept
To address the problem, we apply the metaphor of construction
with building blocks, to the composition of musical melodies.
This process allows young children to develop musical curiosity
by connecting augmented objects within physical space with the
sense of sound and time.
We created a set of tangible interactive construction toy blocks for
children to explore and build understanding of concepts in music
(such as notes, chords and rhythms) and create simple melodies of
Copyright 2010 ACM 1-58113-000-0/00/0010 …$15.00.
(d) Structure created in (b)
Figure 1. Playing with NoteCubes to Create Melody
2. RELATED WORK
2.1 Tangible Musical Toys
The field of music education has produced several other examples
of smart musical toys as physical shape and space can help
children better understand abstract concepts in music such as
rhythm, chord and pitch. BeatTable [3] is a physical table with a
digital environment controlled by tangible objects, which give
auditory and visual feedback to learners building concepts of
rhythms and ratios. It has features that use physical activity to
build representational mappings by offering the user cognitively
ergonomic ways to dive into music, which is similar to
NoteCubes. What differentiates BeatTable from NoteCubes is that
BeatTable only provides a 2D graphic mapping and 2D visual
feedback, although the primary elements are tangible. NoteCubes
can be arranged freely to form different 3D geometries and map
3D volumes to intervals of sounds. In addition NoteCubes not
only helps learners to understand tempo and rhythm, the system
also introduces the concept of pitch. Another related example is
Zoundz [4], which includes a sound board and a small set of
pawns. Users can create riffs by placing pawns on "hot spots" on
the sound board. However, instead of providing freedom of
expression and creativity, Zoundz puts fixed complex riffs in each
pawn and only allows users to superimpose the riffs. It lacks the
educational function of building understanding of musical
elements by creating melodies based on these understandings,
which is one of the central affordances of NoteCubes.
NoteCubes encourage children to create melody and rhythms by
exploring different spatial structures of cubes based on the
understanding of musical concepts. For example, in Figure 1.a, a
child put five cubes as structure shown in Figure 1.c to create a
piece of melody. Figure 1.c also shows how the structure mapping
to melody’s time track. In Figure1.b, another child built a 3D
structure. The triggering path and time track are shown in Figure
1.d.
In the first round of iterative prototyping we developed four
categories of cubes:
• Start cube: The first cube does not play a sound but can
emit light by pressing its button to trigger other cubes.
(Figure 2.a)
• Pitch cube: When trigger by receiving face (embedded
with photo sensors), play a note on certain pitch in one
beat and light up LED on sending face (embedded with
LEDs) to trigger another. (Figure 2.b)
• Empty cube: Similar to the Pitch cube, but plays silence in
one beat. It’s for creating rhythm patterns.
• Broken Chord cube: when triggered, it can play a broken
chord repeatedly. The tempo of the broken chord can be
adjusted via knob. (Figure 2.d)
2.2 Construction Kits
Augmented construction kits have been used in educational
setting for many years [7]. However, most of the research in this
area is focused on mechanical, electronic or programing
education, as opposed to musical education. Cubelets [1] is a
robot construction kit that “combines sensor, logic and actuator
blocks.” The blocks can be snapped together to make different
kinds of robots. Flow blocks [2] are wooden blocks with
embedded electronics and electromechanical contacts that allow
children to play with and reconfigure a complex causal system.
Unlike Flow blocks, NoteCubes takes full advantage of the 3dimensional feature of the block form and can be stacked to build
a 3D spatial model. Both the visual and audio feedback come
from a 3D structure, which truly mobilize children’s spatial sense
to learn and create.
a
b
c
d
Figure 2. NoteCubes
3. DESIGN
3.1 Inspiration: Learning by Doing
Toy building blocks are the primary inspiration for this project.
By playing with toy blocks, children acquire not only spatial
skills, but math skills are also promoted [5]. What if we also
injected music learning into the process of playing with toy
blocks? Our goal is therefore to design a set of graspable and
interactive musical blocks connecting the spatial sense with
the sense of sound/time to aid young children in building curiosity
and interest in music.
3.2 Description
NoteCubes is a set of interactive and augmented building blocks
for children to explore musical notes, chords, rhythms and
melodies. The series of cubes can be arranged freely to form
different three-dimensional geometries. When a cube is triggered
by an external light source (i.e. a flashlight), it plays a note at a
designated pitch and it triggers its "neighbor cubes" (left or right
& up & down) to play notes. An example is shown in Figure 1.
Figure 2. An example of NoteCubes structure
3.3 Features
3.3.1 Integrate Visual Feedback with Information
Transfer
Instead of using wireless technology like Bluetooth or RFID, we
integrate the information transformation input with visual
feedback as a LED light. NoteCubes use light as the information
transformation media and the user can also see the feedback on
other sending face when cubes are triggered. This approach was
used for cost-saving purposes.
We used different colors of LED to represent different notes.
Therefore, the visual feedback is a chain of blinking LEDs
embedded on the cubes surfaces. In addition, there is also a
metaphor between pitches in musical note and the light colors: the
note pitch is a frequency-related property of a sound wave [11];
the light color is also a frequency-related property of a light wave.
Finally, as a medium for information transfer, light can carry
much more information by varying patterns and frequencies. This
provides great potential for inter-block communication in our
future development.
(in our next round prototyping, we consider to make the chord
metaphor as a stack of cubes. More details are in 5.2).
4. SYSTEM DESIGN
The system diagram is shown in Figure 4.
3.3.2 Multi Sound Sources
There is a speaker in each cube. We believe that the playing
experience is different between a single audio source and sound
directly from the cube itself. When children move the cube’s
physical location, the location of audio feedback will also be
moved. So the feedback is strongly fixed with users operation thus
emphasizing the metaphor of each cube having a musical voice of
its own.
3.3.3 Logic Thinking Promotion
The triggering path can form various logic structures such as
branch and loop. Figure 1.c and Figure 1.d are simple examples.
More complex example are shown in Figure 3. NoteCubes also
help the development the logic thinking of children.
3.3.4 Tangible User Interface
Our NoteCubes project is influenced by the vision of tangible user
interface that “exploring the interactive techniques … go beyond
the current GUI paradigm” [8]. Compared with existing computer
and mobile music learning applications, NoteCubes’ TUI offer
other learning opportunities:
•
•
More physical collaboration opportunities are offered.
Especially when using NoteCubes in classroom or
home, physical collaboration and interaction between
child and child, child and teacher or child and parent
can help improve learning.
Tangible toys can enrich the learning experience for the
children [6]. More sensory experiences are provided to
children such as touching, moving, stacking, hearing
and watching.
3.4 Metaphor
3.4.1 Cube and Note
We choose cube to represent a note, because a cube is a basic unit
in space. It has the same width in three dimensions. Each cube
includes three receiving surfaces (sensor embedded) and sending
surfaces (led embedded). By connecting sending and receiving
surface, cubes can be both arranged as a line or stacked together.
It can be freely stacked to form a 3 dimensional structure, just like
a note is a unit in melody and can be arrange freely in a timeline
to form a melody.
In addition, blocks as a form of traditional toy bring an existing
cognitive and emotional association when children play with
them. The traditional form can “take advantage of children’s deep
familiarity with (and deep passion for) these objects”. [9]
3.4.2 Rhythm, Chord and More
The spatial construction metaphor for musical composition offers
intuitive analogies for important musical concepts like melody,
harmony, and rhythm. Based on the metaphor between cubes and
notes, more concepts in music have a counterpart in NoteCubes.
For examples, a piece of melody is addressed to be a threedimensional geometry; a piece of rhythm is a queue of cubes with
an ordered pattern of pitch cubes and empty cubes; a chord is a
group of cubes which are triggered simultaneously by a same cube
Figure 4. System Diagram
The system in each cube includes a micro-controller (Trinket),
three photo sensors, an audio amplifier, a speaker and three LEDs.
Photo sensors are embedded on the center of three receiving
surfaces and LEDs are embedded on the center of sending
surfaces.
Trinket is a lowest-cost Arduino-IDE programmable board. It is
programed to trigger speaker to play a note when LED lights from
other cubes trigger one of the photo sensors. Here we used Trinket
for rapid prototyping. ATtiny family of Atmel [13], which are
cheaper and tinier, may have a better fit for this project. The
cube's shell is made by hardboard by laser cutting. We used laser
cutter to print music sign like “C” on the surface of the cube.
However, in the second round prototyping, instead of design one
cube with one fixed pitch, we added a slide bar to adjust the pitch
in each cube (Figure 2.c) because we find that limit number of
cubes reduces the flexibility of creation and learning.
5. DISCUSSION AND CONCLUSION
5.1 User Feedback
NoteCubes was exhibited at Assemble Gallery in Pittsburgh on
Dec. 6th 2013. Seven children aged between 5-12 attended the
show. Although we did not conduct controlled experiments, we
were able to observe how children played with the NoteCubes, the
process they used to learn the features of the cubes, and some
initial feedback. Most of the children appeared curious and
interested in NoteCubes and gravitated towards the exhibit.
Before we explained how it worked, children were willing to go
through experiments trying to figure it out. Without
communicating the features of NoteCubes, one boy figured out
how to make a melody using the blocks (Figure 1.a). Most of the
children were willing to build more structures themselves after
understanding the basic feature of NoteCubes. One limiting factor
we observed was confusion about the different functionality of
sending surfaces and receiving surfaces. We also observed some
children used the “start cube” as a tool to check if the other cubes
work well before triggering the whole cube chain.
5.2 Future Work
As a work-in progress, there are several aspects of the NoteCube
system we are currently working on. We are considering how to
bring NoteCubes into a formal music classroom, as NoteCubes
doesn’t need any support equipment or platform. It is suitable for
music teaching in primary school music class, but more features
need to be designed to fit music teaching process. For example,
we are prototyping the use of small white boards on cube surfaces,
which can be changed with markers by teacher and students.
Deeper question arise about whether a 3 dimensional timeline is a
good way to teach western music concepts. Moving forward, we
are considering how the stacking feature of the blocks can be used
for chords making, which has more direct mapping with chords in
stave. In addition to the concepts in western music theory,
NoteCubes may also have potential in exploring Non-western
music concepts or other music learning modes, such as recording
sound, replay/restructure them or involve the concepts like
"Scales and/or Modes" and "Consonance vs. Dissonance." There
is still much space for improving the appearance of the cubes and
timbre of notes. As discussed in 3.4.1 there are many advantages
in making the shape as a cube. But based on the basic shape in the
first round prototyping, we can design more details to differentiate
the sending and receiving surfaces.
5.3 Conclusion
Based on our observations above, we can make a preliminary
conclusion that the concept of musical construction block toy has
attraction to children. Most children have curiosity and interest to
play with these interactive blocks and explore their embedded
features. As we discussed in 5.2, there are still much works and
also potential in future development. We believe that tangible
interactive block is a novel and potential solution to children’s
music learning problem.
6. REFERENCES
[1] Cubelets. https://www.modrobotics.com/cubelets
[2] Zuckerman O, Grotzer T, Leahy K. Flow blocks as a
conceptual bridge between understanding the structure and
behavior of a complex causal system[C]//Proceedings of the
7th international conference on Learning sciences.
International Society of the Learning Sciences, 2006: 880886.
[3] Bumbacher E, Deutsch A, Otero N, et al. BeatTable: a
tangible approach to rhythms and ratios[C]//Proceedings of
the 12th International Conference on Interaction Design and
Children. ACM, 2013: 589-592
[4] Zoundz. http://www.youtube.com/watch?v=6kqSIvUBeB0
[5] Kersh J, Casey B M, Young J M. Research on spatial skills
and block building in girls and boys[J]. Contemporary
perspectives on mathematics in early childhood education,
2008: 233-251.
[6] Hengeveld B, Voort R, van Balkom H, et al. Designing for
diversity: developing complex adaptive tangible
products[C]//Proceedings of the 1st international conference
on Tangible and embedded interaction. ACM, 2007: 155158.
[7] Blikstein P. Gears of our childhood: constructionist toolkits,
robotics, and physical computing, past and
future[C]//Proceedings of the 12th International Conference
on Interaction Design and Children. ACM, 2013: 173-182.
[8] Ishii H. Bottles: A Transparent Interface as a Tribute to Mark
Weiser[J]. IEICE Transactions on information and systems,
2004, 87(6): 1299-1311.
[9] Resnick M, Martin F, Berg R, et al. Digital manipulatives:
new toys to think with[C]//Proceedings of the SIGCHI
conference on Human factors in computing systems. ACM
Press/Addison-Wesley Publishing Co., 1998: 281-287.
[10] Fitzmaurice, G.W., Ishii, H. & Buxton, W. Bricks:
Laying the Foundations for Graspable User Interfaces,
in Proceedings of CHI'95, 442-449.
[11] Anssi Klapuri and Manuel Davy (2006). Signal processing
methods for music transcription. Springer. p. 8. ISBN 978-0387-30667-4.
[12] Trinket http://learn.adafruit.com/introducingtrinket/introduction
[13] ATtiny
http://www.atmel.com/products/microcontrollers/avr/tinyavr.
aspx