Music Aided Design - Felix Faire Dissertation

Music Aided Design:
The Foundations of Spatial Music
Felix Faire BA Arch Dissertation - Click to open pdf
2
Acknowledgements:
I would like to thank Prof. François Penz for his expertise, guidance and humour;
Amir Soltani for his enthusiasm and support;
And all the friends who volunteered to be experimented on:
Adam, Sohanna, Matt, Max, Lucy, Song, Jazz, Lauren, Katie, Mica, Iain, Immy, Fiona,
Daniel, Elly, Miranda, Charlie, Sam, Freddy, Emma, Phoebe, Sophia, Alice, Kitty, Livvy
and Francois.
9353 words
3
Abstract.
Key words: Embodied music cognition, sound, space, interaction, association
This dissertation hypothesises that our perceptions of music and space are
cognitively connected through embodied interaction with the world. These
perceptual connections will be initially revealed by looking at our structures of
language and speech, and then examining physical and audible precedents in the
work of Iannis Xenakis, Oskar Fischinger and Mark Applebaum. The hypothesis will
be further developed by exploring the shared neurological processes in listening
to music and the navigation of spatial environments. The theoretical framework
developed from the research is tested with a series of spatio-musical interactions
tested by 24 volunteers. The results are discussed in relation to their application to
musical composition, architecture and urbanism, speculating the potential for a new
paradigm in Music Aided Design.
4
Contents.
Introduction:
5
Precedents:
Music Signification:
7
The Musical Score:
9
Animated Music:
13
Sound Gestures:
15
The Psychogeography of Music:
18
Experimental Research:
21
Pitch and Height:
22
Sonic Compass:
23
Sounds in Space:
24
Spaces of Sound:
25
Music and Motion:
26
Haptic Music:
27
Experiment Data:
29
Discussion:
30
List of Illustrations:
33
Bibliography:
34
Appendix:
36
5
Introduction.
“To most of us music suggests definite mental images of form and colour. The
picture you are about to see is a novel scientific experiment - it’s object is to convey
these mental images in visual form.” - Oskar Fischinger 1938
“It is now becoming clear through scanning technologies that the various senses
also share higher order cerebral networks, or perceptual supramodalities that
engage a crossover of sensory inputs from one sense to another”
- Harry Francis Mallgrave 2010
Over the past hundred years, increasingly refined theories of perception have
emerged. The top down philosophical theories are beginning to coalesce with the
bottom up scientific theories giving us greater insight into the study of experience1.
The relatively recent theory of Embodied Cognition has shed light on how the mind,
the body and the world form a conscious system of dynamic symbiosis. This recent
view of perception implies that experiences originally considered to reside solely
in the human mind such as music are in fact intrinsically connected to our bodies
and environment. Marc Leman describes this concept of embodied cognition:
“In contrast to dualism, the concept of mind is seen as an emerging effect of the
brain perceiving its own actions in relation to a physical environment.2 From that
perspective the subjective world of mental representations is not an autonomous
category but a result of an embodied interaction with the physical environment.”3 He
also suggests how our perception of music is tied to other embodied experiences:
“The multimodal aspect of musical interaction draws on the idea that the sensory
systems – auditory, visual, haptic as well as movement perception – form a fully
integrated part of the way the human subject is involved with music”. 3 This
dissertation will not focus on specific emotional processes involved in making and
listening to music. The primary aim of this discussion is to reveal and test how our
embodied interaction with the world develops perceptions of music and sound that
are fundamentally connected to our perceptions of space.
The history of phenomenology through to embodied cognition is described in further detail in
Dourish (2004) p.102-126
2
Ernelling and Johnson (2005)
3
Leman (2008) p.13
1
6
To develop this hypothesis, I will examine the works of Iannis Xenakis, Oskar
Fischinger and Mark Applebaum. I will observe how these artists have been able to
transpose between sound, image and gesture by utilising and adapting the intrinsic
spatial inferences that music creates. These works will be examined based on the
principles of embodied cognition to find the theoretical basis for the vocabulary and
intersubjectivity of each medium. The discussion will also look briefly towards shared
neurological processes in music and space perception, particularly the narrative
and memory functions of the hippocampus necessary for both spatial navigation
and musical experience.
The research is consolidated with a series of spatio-musical interactions in the form
of volunteer tested experiments. The experiments use the theoretical framework set
out in the first chapters to actively engage the volunteer’s spatio-musical ability and
develop new kinds of intersensory interactions. I will then discuss the potential value
and opportunities given by this frame of mind towards architectural discourse.
For the purposes of this text I will use a broad definition of “music” as: any sequence
of sounds that are arranged within a larger temporal structure, and the term “sound”
to describe any audible form.
7
Music Signification.
Listening to music is a vastly complex mental phenomenon that activates many
different parts of the brain. However, we can start to reveal the spatial aspects of
music perception by examining the ways in which we signify musical concepts.
The genealogy of music perception has consistently been related to the evolution
of language.4 Verbal communication, as well as refining the sophisticated pitch,
rhythm and pattern recognition necessary for music appreciation, uses higher level
representations to signify different experiences.
Linguistic descriptions of music (such as Hatten’s texts about Beethoven’s works
1994) rely heavily on metaphors to create intersubjective representations of melodies,
dynamics and timbres. These metaphors (such as Escalating, Swirling, Collapsing
and Sweeping) are not simply means to describe music but actively display how
we perceive music through the language of other experiences. George Lakoff and
Mark Johnson illustrate how our entire language is constructed from cross modal
inferences­and that these associations form the essential rudiments in which we can
conceptualise and think about the world.5,6 They describe the metaphor of “affection
is warmth” as emergent from a child’s concurrent experiences of affection and warmth
in the embrace of a parent.6,7 This association between conceptual and sensory
inferences forms what Lakoff and Johnson call an “Embodied concept”.8 Our ability
to think is facilitated by both the brain and the nervous system7,8,9. The prevalence of
physical and spatial metaphors use to describe musical attributes and features, such
as dancing melodies, sweeping and swirling phrases, rising and falling scales etc.
allow us to see that our conceptions of music rely heavily on spatial understanding
and embodied concepts.
Brown (1999)
Mallgrave (2010) p.175-180.
6
Lakoff and Johnson (1980) p.255-256
7
The neurological basis for metaphors in language is described by Lakoff and Johnson (1999) using
Donald O. Hebb’s theory of concurrent sensory information and synaptic growth (Hebb 1949), This is
explained fully in Mallgrave (2010) p.175-180.
8
Lakoff and Johnson (1999) p.20
9­
Tversky (2008)
4
5
8
The words “high” and “low” used to denote audible pitch as well as vertical
position is a prime example of an audio-spatial metaphor that we can suggest a
possible etymology from an embodied perspective. Perhaps the most direct way
we interact with sound from the moment we are born is through the use of our
vocal chords. In producing or replicating different pitched sounds we must perform
certain biomechanical operations that are coordinated with our auditory sense.
Our vocal production of sound affects our spatial perception of pitch through the
perceived location of vocal noises relative to the ears. The position of the larynx
in the throat moves vertically to allow a higher or lower vocal range, this physical
change in position affects the perceived location of the produced sound. Singers
use the terms “Chest Voice”, “Middle Voice” and “Head Voice” to represent these
different positions as this is where the sound is felt to be coming from, this clearly
reinforces the perception that lower pitches are physically positioned below higher
pitches in space. This embodied explanation displays that the signification of
the words “high” and “low” to audible pitch is not an arbitrary selection of words
but is heavily reinforced by the spatial perception of our voices, catalysing this
association between pitch and height over time. This is one example of how an
embodied action produces cross modal inferences between seemingly disparate
sensory perceptions. This multimodal perception is consolidated and symbolised
as polysemous homonyms in language. This vertical mapping of pitch in space was
initially observed by C.C Pratt in 1930 after observing that the specific succession
of tones in a musical phrase can generate a sensation of vertical movement.10
These ideas have since been experimentally demonstrated by Lidji et al. 2007 and
Rusconi et al. 2006 and show the potential that audible properties can be mapped
successfully into spatial representations.11,12 I have highlighted just one example of
how spatial terms are used to describe audible characteristics in speech; however
the discipline of music has developed other means of signification to represent and
translate musical attributes in visual and spatial terms.
Pratt, C.C. (1930)
Lidji, P., Kolinsky, R., Lochy, A., & Morais, J. (2007)
12
Rusconi E, Kwan B, Giordano B.L, Umilta C, Butterworth B (2006)
10
11
9
The Musical Score.
The musical score is a syntax and grammar used to translate music from a dynamic
temporal form to a static representation in 2 dimensions and vice versa. The 2
dimensions of space used in musical scores traditionally represent time read from
left to right in the x axis, and pitch arranged vertically on the y axis. The traditional
score clearly utilises the aforementioned spatial inference by signifying relative
pitches with vertical positions. The length of the note or rest is symbolised using
conventions of notation that must be learned in order to read the musical text.
Whilst the general form of instrumental music can be effectively transcribed with
these traditional graphical languages, the strict use of both spatial dimensions and
limited syntax of expression are abstracted from the experiential qualities of the
music13. This further asks the question whether there are alternatives to the standard
graphical musical signification that are able to capture the more experiential
qualities of music.
The composer Iannis Xenakis started to experiment with spatial representations of
sounds to reveal wider experiential truths within his music. His work Metastasies
originated with a graphical score that maintained the conventional use of
representational axis yet aimed to become more than just an instructional text for
performance (Fig. 1). Xenakis was particularly interested in the aspect of time in
Fig. 1 - Graphical glissandi from Xenakis’ Metastasis.
music, the contemporary Einsteinian view had shown that time was relative to mass
and energy, in that any change in the contents of time would change time itself.
Xenakis aimed to emulate/explore this abstract idea in his music through the act of
capturing it in a static space.14 This negation of real time allowed the comprehension
of the whole in the instance of seeing the score. Xenakis used the example of
gunshots in a battlefield to describe the nonlinearity of these particular musical
Linguistic descriptions and conventional notations are critiqued by Bengtsson, I & Eggebrecht, H.
(1973), and Leman, M. (2008)
14
Hofmann, B. (2005)
13
10
experiences; the exact order of each bullet fired is irrelevant as the resulting sound
as a whole will undoubtedly be that of “gunfire”. This concept of the musical form
being something other than a sum of its parts is particularly evident in Xenakis’
Mycenes Alpha (1978).The graphical score of Mycenes Alpha is comprised of many
(often straight) lines which collectively define larger, curved undulating masses (Fig.
2). The image that results is a collection of undulating conglomerate forms. The
graphical score was translated into audio by Xenakis’ own UPIC system15, whilst
Fig. 2
the system translates each individual line into a note; the resulting cacophony of
Excerpt from Mycenes
Alpha.
whole. Xenakis was fascinated by the form of the Hyperbolic Paraboloid (Fig. 3)
all the combined sounds also generates the perception of an undulating dynamic
which too features this characteristic explored in his music: whilst every element
in a hyperbolic paraboloid is a straight line, the perceived whole is a twisting and
smoothly curved surface. These ideas were manifested spatially in the design of the
Fig. 3
Example Hyperbolic
Paraboloid.
Philips Pavilion (Fig. 4). This is an example of how a perceived audial experience
has become effectively formalised to generate similar visual (and then spatial)
experiences. However, Xenakis was not the only proponent of the graphical score
and whilst his methods of representation reveal some ways in which the experience
of music can be perceived in static representations, others began to extend the
boundaries of how 2 dimensional spatial representations in fact become sonic
experiences in themselves.
Mark Applebaum’s Metaphysics of Notation (2010) rejects the standard
use of dimensional axis to represent specific attributes of sound and time.
His work instead relies solely on subjective associations and metaphorical
Fig. 4
synaesthetic interpretations. This indeterminate notation does not specify any
Philips Pavilion
Xenakis & Le Corbusier.
instrumentation or temporal structure and thus, the individual performers of The
Metaphysics of Notation produce varying audial results. The musicians’ process
of transforming/analogizing the shapes and forms on the page into temporally
arranged soundscapes displays the brains ability to interchange sensory forms
to produce meaning. This ‘ability’ of multimodal chiasm is not simply a skill of the
conscious brain but is in fact a fundamental property of how the brain works.16
The scores themselves break so far from traditional scores that many of the
interpretations are completely unique, however some aspects of an intersubjective
The computerised UPIC system translated the vertical position of a mark into pitch and its
horizontal length into duration.
16
Mallgrave, H. F. (2010)
17
Arnold, R. (2010)
15
11
associative language are revealed by intrinsic similarities between the performers’
interpretations of certain passages of shapes.17 Some of these are simply carried
across from the performers’ traditional training of scores; sweeping and curving
lines become long notes changing in pitch respectively, and sections of repetitive
Fig. 5
spots seem to unanimously represent individual notes or rhythmic musical events
in time (Fig. 5). Some similar responses that aren’t traditional of standard notation
are that the size of the elements are commonly attributed to volume, this is used in
the standard crescendo symbol but is also a basic perception that larger objects
generally make louder sounds when interacted with. Associations derived from
common perceptual experiences of actions and sounds in the world form a key
aspect of the sensory chiasm that allows the translation of these images into sounds.
Shape, texture and formal characteristics are also passed between our experiences
of sound and sight; they are intrinsic to the legibility of the graphical scores and
have been observed experimentally.
Fig. 5 - Excerpt from The Metaphysics of Notation
The Kiki/Bouba effect first observed by Wolfgang Kohler 1929 gives great insight
into how shape and formal characteristics of objects produce strong inferences to
types of sound.18 These synaesthetic connections are essential subconscious tools
that the musicians employ when translating the many shapes of “The Metaphysics
of Notation” into music. Kohler tested this idea by presenting subjects with two
shapes, a smoothly curved anamorphous blob and a jagged, angular star-like shape
(Fig. 6). When asked which shape was called “Kiki” and which was called “Bouba”,
Kiki.
Bouba.
Fig. 6
95% of subjects assigned “Kiki” to the spiky form and “Bouba” to the rounded blob.
Speculatively this could be connected to the visual graphemes used to represent
the verbal sounds. For example the letters of the word “Kiki” feature sharper, more
angular lines than those in the word “Bouba”, which ubiquitously features rounded
letters. This implies that there may be some aspect of reading the shapes in the
graphical score using the musician’s functions of verbal language, developed in
parallel to their musical training. However, further work on sound-shape correlations
by Daphne Maurer in 2006 observed that the Kiki/Bouba effect was also prevalent
in toddlers as young as 2.5 years old.19 Whilst the effect was slightly less prominent,
the fact that the toddlers had yet to develop reading skills shows that the shapesound inference is primarily based on the audible sound rather than inferences back
to a visual representation of the words. Ramachandran and Hubbard hypothesised
Köhler, W. (1929)
Maurer D, Pathman T & Mondloch C.J. (2006)
20
Ramachandran, V. S., & Hubbard, E. M. (2001)
18
19
12
that this association is built through the use and shape of the mouth in creating
the sounds of the nonsense words: the angular figure “mimics the sharp phonemic
inflections of the sound kiki, as well as the sharp inflection of the tongue on the
palate.”20 From this we can see how our ability to translate shapes into sounds
is also derived from an embodied application of using our mouths and lips to
produce different sounds. However, many musicians have refined interactions with
sound (other than using their vocal chords) that they use to interpret Applebaum’s
scores. For these examples we must look at the coupling of body to instrument,
and then to the affordances of the instrument in terms of the corporeal motion used
to produce sounds. The limitations and facilitations of the instrument will invariably
affect the interpretation of the visual form. The bodily motion that changes the
pitch of a trombone for example is very different to that of playing a cello or flute;
as such, associations of movement across the score could yield different audible
results. The empathic association that allows the symbols to be read as sound
generating movements is another factor in interpreting these scores, and can be
generated through the very act of drawing the score. The signification of the musical
expression has emerged from a motion, such as a sweeping brush stroke or flick
that has imprinted its temporal history into a static symbol. The intentionality of the
artist in interpreting the symbol can be read through their empathic comprehension
of how the mark was made, this can then be manifested as corporeal musical
expression.21 We have seen how forms on a 2 dimensional plane can infer audible
responses, through convention, “embodied concepts” and even inferences to
motion and gesture that inform the instrumental output. This inferred motion and
mental animation of the symbols on the score as they are interpreted displays the
limitations of 2D graphical representations. Music is dynamic and it appears that our
spatial representation and cognition of music can be further augmented through a
dynamic spatial medium.
Corporeal imitation, articulation and expression are further investigated in Leman, M. (2008)
p.103,123,141
21
13
Animated Music.
The medium of film and in particular the animations of Oskar Fischinger address
some of the dimensional abstractions of graphical scores as the visual space and
animation occupy the same temporal dimension as the music itself. Whilst the
visual metaphors between shape and sound such as size to volume and shape
to timbre etc. are still used to great effect in Fischinger’s films, the possibilities of
movement are released by the animated medium. Musical objects now exhibit
speed and characteristics of movement that bring with them their own physical
and aural associations. Objects that exist within the dimension of time (unlike the
images of Applebaum and Xenakis) can swoop, shake, dart, fold in on themselves
and perform a wide variety of dynamic behaviours; as a result the animations get
closer to synthesising these ‘intuitive’ connections between sound and visual form.
Susannah Knights writes of Oskar’s animations: “Both media unfold and engage the
audio and visual senses over time, similarly capable of evoking an automatic sense
of narrative expectation. Secondly, both create an illusion of a gestalt, through the
audience psychologically connecting component parts which run in a temporal
sequence.”22 The dynamic properties of Fischinger’s films reveal how the viewer’s
expectations of imminent musical and visual activity can become synchronised to
further enhance the perception of an audio-visual gestalt. Our ability to expect future
events based on current conditions is based on observing recurring phenomena
to build an understanding of how objects behave. If we continue to use embodied
experiences and interactions as a basis for generating audio-visual associations
22
Knights, S. (2012) p.13
14
we can start looking at our spatial expectations and see how these interact with our
musical expectations. For example, the resolution of objects in motion is for them to
fall downwards under gravity and come vertically to rest. We use this expectation
and understanding of the mechanics of our environment to be able to perform
actions, such as catching a falling ball. The same kind of resolution also appears
in chord sequences. Sustained chords and perfect cadences also create a tension
that is traditionally expected to be resolved. This effect of expected resolution is
evident in our melodies of speech, for example an upwards inflection at the end of a
Downbeat 00:37
sentence or melody creates a musical question that expects an answer. Susannah
Knights observes how Oskar exploits this sense of melodic narrative and physical
understanding: “although the first two strokes arch downwards on the audible
downbeats, the third, marking a quaver leading-note which sparks a short passage
Fig. 7
†
Downbeat 00:38
of syncopation, arcs upwards, briefly creating a similar expectation of a resolution.”23
(Fig. 7) This expectation is not only built with the motions that precede it, the visual
and audible upward inflection evokes the same expectation of resolution as a verbal
question or object being flung into the air. We can see that, as static shapes can
elicit audible timbres, we also perceive motions and expected trajectories of both
dynamic physical objects and musical phrases.
Expectation 00:39
Resolution 00:40
23
†
Knights, S. (2012) p.17 describing Fischinger’s Studie No. 7 00:37
Images are 20 frames of animation superimposed to show motion paths.
15
Sound Gestures.
Fischinger’s films visibly correlate to our embodied perception of music’s motions
and dynamic patterns far more effectively than static graphical symbols. However,
both mediums achieve their audio-spatial inferences through embodied concepts
and common sensory experiences. Mark Applebaum’s piece Aphasia (2010)
combines hand gestures with music to create an audiospatial gestalt that is created
around the body itself. The medium of hand gestures not only moves the experience
of the music into a dynamic 3 dimensional realm but immediately generates an
implication of action to the audience. We use our hands as a primary means to
interact with the world, therefore a movement of a hand implies an action is taking
place in which there will be an effect. Unlike the dancing shapes and sprites
of Fischinger’s films, that are simply characterised by motional behaviours that
complement the music, the movements of the hands are perceived to have a direct
causal relationship to the music that they accompany. The electronically altered
vocal samples and abstract audio snippets used to create the audio of the piece
do not remind us of any traditional instrumentation, however the hand gestures
appear so perfectly matched to the audio that it is difficult to imagine which is
derived from the other, or if they were composed in parallel. Applebaum’s advanced
and multidisciplinary instrumental experience has given him a sophisticated
understanding of how motions and hand actions produce sounds with objects. His
search for new musical interactions has led to the invention of bespoke instruments
such as the “Mousketeer” (Fig. 8). Applebaum avoids the motional associations that
Fig. 8 - Mark Applebaum playing his “Mouseketeer”.
accompany traditional instrument sounds (such as the smooth sweeping motions of
the string instruments visualised to great effect in Fischinger’s Studie Nr. 7 01:22).
Instead he creates an original vocabulary of sounds and motions that is strictly
adhered to in the performance of Aphasia. The lack of a perceptible time signature
augments the audience’s perception that the sounds are causally linked to the
actions as they appear spontaneous yet temporally synchronised. By repeating the
sound gestures, the audience signifies the visible motion to the audible response
16
and gradually learns the vocabulary of the piece as it progresses and new sounds
and motions are revealed. By producing the illusion that the motions actively create
the sounds, the actions that Applebaum uses appear to imbue the sounds with a
physical presence and spatial attributes. The piece opens with several disparate
beats to the chest, creating a short percussive knocking sound. After the sixth
instance of this action, instead of returning his hand to his lap his arm opens out
away from his body (Fig. 9 - Aphasia.mp4 - 00:34), the knock appears to echo
around this new space that the gesture has created. This perception localises the
Fig. 9
Echo Gesture.
sound of the knock to the space around the body that is perceived to be its source.
Similarly at (Fig. 10 - Aphasia.mp4 - 02:12) Applebaum strenuously pulls his hands
apart emulating a tension that is audibly reflected with a series of strained rubbery
squeaks. The perception that the sound is being tangibly stretched apart is a very
successful example of the sounds assuming a spatial and tangible form in the mind.
It must be noted at this point that a core aspect of embodied cognition is that for
any intentionality or experience to be embodied it must also be situated. “People’s
conceptions of space differ for different spaces and are a joint product of perception
and action appropriate for those spaces”24 This is the idea that behaviour is situated
in it’s architectural environment.25 A person might be more likely to dance if music is
played in a dance studio however would perhaps act differently in a church or office.
Fig. 10
Stretch Gesture.
The gestures of Applebaum’s Aphasia are not only afforded by the biomechanical
trajectories of the human body but also influenced (or indeed generated) by the
wider spatial context and the perception and action appropriate for the environment
itself. The piece is primarily viewed on a blank stage or in front of a black
background as shown in the filmed version. There are no visual clues to the spatial
context in terms of architecture, however the piece is by nature a performance so
there is an intrinsic factor of ‘Audience’ that informs the spatial representations of the
music that Applebaum creates. This audience is either a real audience in the stage
context or the camera in the filmed context. This directional connection between
audience and performer changes how the gestures are interpreted, it would
certainly not have the same effect if viewed from behind. We can see how the study
of spatio-musical chiasm in a dynamic 3 dimensional space demands embodied
and situated means of analysis; the spatial context and relative perspectives of
people within the system deeply affect the resulting musical interpretation.
Tversky, B. (2008) p.202
Behaviour and language are spatially situated but not environmentally determined Dourish, P.
(2004) p89
24
25
17
By initially looking at examples of prior art we have seen how static and dynamic 2
and 3 dimensional spaces have been used to naturally mediate our perceptions of
sound, music and space. I have also described how these gestalts are formed in
the mind by utilising the approach of embodied cognition26,27 and the action oriented
ontology of music described by Leman. The precedents have not only examined the
spatial inferences of individual sounds, but also touched on how the sounds (and
indeed shapes and gestures) in the music relate to each other in time and generate
expectations, moments and contrasts that are attuned to our embodied experience
of the physical world. This aspect of music distinguishes it from being simply
sequences of sound or noise and deserves greater examination as it displays more
fundamental ways in which space and music are shared in the brain.
26
27
Tversky, B. (2008)
Shapiro, L. (2010)
18
The Psychogeography of Music.
Marvin Minsky’s description of music as an experience of learning begins to illustrate
how the temporal structures of music are experienced and formed in the mind. He
uses the words “sonata as teaching machine”28, describing the traditional musical
sections of ‘exposition’, ‘development’ and ‘recapitulation’ as phases of observation,
assimilation and accommodation in learning how to listen to the rest of the music29.
In a simple sense the gradual exposition of initial rhythms, harmonies and scales
start to build a psychological temporal framework of time signatures and key
signatures to which the rest of the music can develop a relationship. Minsky makes
the distinction that “Learning to recognise is not the same as memorizing”30 and that
whilst we rarely memorise entire pieces of music, something remains in the mind
that allows phrases to trigger memories of the subsequent bars. He illustrates the
similarities between our use of sight and of music in building mental maps: “How do
both music and vision build things in our minds? Eye motions show us real objects;
phrases show us musical objects. We “learn” a room with bodily motions; large
musical sections show us musical “places.” Walks and climbs move us from room
to room; so do transitions between musical sections. Looking back in vision is like
recapitulation in music; both give us time, at certain points, to reconfirm or change
our conceptions of the whole.”31 This musical view of space not only applies to
“rooms and objects” but also has resonances with the views of navigation in larger
Fig. 11 - Front cover of Debord’s “Guide Psychogeographique de Paris” illustrates
mapping of ‘the city’ based on the relationships between spaces, not actual positions.
scale psychogeography. In navigating a city or country route, we do not memorise
the exact spatial arrangement of our environment, however moments, contrasts
and landmarks along familiar paths cue us in understanding how our current space
relates to other nearby spaces (Fig. 11). Minsky suggests that the parts of the brain
that allow us to map spaces in this way are the same ones used to appreciate
musical form. This leads to the hypothesis that our experience of music uses similar
mental processes of learning and cognition to those of spatial navigation.
28
29
30
31
Minsky M. (1981) In Clynes M, ed. Music, Mind and the Brain: The Neuropsychology of Music p.29
Stages of learning outlined in Piaget. (1970)
Minsky M. (1981) In Clynes M, ed. Music, Mind and the Brain: The Neuropsychology of Music p.30
Minsky M. (1981) In Clynes M, ed. Music, Mind and the Brain: The Neuropsychology of Music p.37-38
19
Cupchick et al tested the neurological aspects of this hypothesis in their paper
“Shared processes in spatial rotation and musical permutation”.32 The paper found
a direct correlation between a subject’s performance at a spatial rotation task and
their ability to detect whether a tune had been played backwards or inverted etc.
Cupchick’s paper illustrates the mental chiasm used to manipulate physical and
musical objects; this is neurological evidence of a mediating process that facilitates
our experiences of the precedents in the first chapter. However, these tests are
primarily focussed on short length musical phrases and technical cognition such as
pitch processing. They do not illustrate the wider temporal experiences of narrative,
expectation and contrast that are intrinsic to musical experience. The hippocampus
has been described as the essential area of the brain that allows the perception
of temporal and spatial narratives. “In addition to its outstanding role for memory
and spatial navigation,33,34 the hippocampus has been suggested to be involved in
novelty detection 35,36. Hippocampal novelty detection is based on a comparison of
actual sensory inputs with stored stimulus patterns.37,38,40”. 41This is clear evidence
for Minsky’s learning analogy of music. For example listening to a preliminary bar of
music is a “sensory input” that becomes a “stored stimulus pattern” in memory, as
the next bar is heard this new “sensory input” is compared to the growing structure
of the previously heard bars by the hippocampus. This is also a neurological basis
for the aforementioned expectation generated in Oskar Fischinger’s Studie No.
7. This emerging structure grows as we are exposed to more of the music, and
the subtle or substantial harmonic shifts and changes push and pull us in varying
degrees of expectation42. The hippocampus in this sense facilitates our perception
of rhythm, flow, continuity and surprise. This narrative experience separates what I
have called “sound” from “music” in the same ways that a single frame of a film is
different to the film itself, or a photograph of a street is different to walking through it.
It is now known that taxi drivers and musicians both develop larger hippocampi due
Cupchik, G. C., Phillips, K., & Hill, D. S. (2001)
Maguire, E.A. (2001)
34
Ekstrom A. D, Kahana M. J, Caplan J. B, Fields T. A, Isham E. A, Newman E. L, Fried I. (2003)
35
Knight, R. (1996)
36
Strange B.A, Fletcher P.C, Henson R.N, Friston K.J, Dolan R.J. (1999)
37
Gray J.A, Rawlins J.N.P. (1986)
38
Strange B.A, Dolan R.J. (2001)
39
Vinogradova, O.S. (2001)
40
Kumaran D, Maguire E.A. (2007)
41
Herdener, M., Esposito, F., di Salle, F., Boller, C., Hilti, C. C., Habermeyer, B., & Cattapan-Ludewig, K.
(2010) p.1
42
Huron, D. (2006)
32
33
20
to their practiced experience and reliance on both spatial and temporal narratives
and memory. 43,44,45,46 The increased hippocampal neuroplasticity clearly illustrates
the essentiality of the hippocampus in perceiving narrative and temporal contrast,
and thus in facilitating our higher level emotional responses to sensitively curated
temporal experiences.
This associative understanding of human navigation through space is currently in
common use by architects and urban planners to generate consistent elements
and extensions of urban form. The reason for this is not neurologically based
but has evolved as a refined human sensitivity to city making. Our experience of
the city is built on a framework of how spaces relate to each other. We learn the
relationships between roads, streets, alleys and open spaces as we move through
the city, these relationships are used as a subconscious mental framework that
we use to comprehend and operate within other cities and spaces. By examining
the processes and functions of the hippocampus, we have seen that music and
dynamic navigation of space share this cognitive process, yet the parallels between
sensitive urban design and musical compositions are rarely consciously used.
Many metaphors such as “harmony” and “rhythm” are used to discuss and design
architectural proposals and their relationships to their context, yet few actively
engage the processes of musical composition to augment the embodied experience
of the city that they are developing. Perhaps by further understanding the deeper
complexities of music and its interaction with mind, body and place we could use
Music Aided Design (MAD) to develop deeper and more emotionally engaging
experiences of architecture and the built environment. This specific area of spatial
musicality deserves a lot more research but we are beginning to see how musical
experiences and spatial experiences are not as distinct as previously thought.
Maguire, E.A. Gadian D.G, Johnsrude I.S, Good C.D, Ashburner J, Frackowiak R.S, Frith C.D. (2000)
Gaser, C., & Schlaug, G. (2003)
45
Rodrigues, A. C., Loureiro, M. A., & Caramelli, P. (2010)
46
Herdener, M., Esposito, F., di Salle, F., Boller, C., Hilti, C. C., Habermeyer, B. & Cattapan-Ludewig, K.
(2010)
43
44
21
Participant
Body Tracking Camera
Experimental Research.
We have examined how recent research and development in neuroscience, music
cognition and embodied cognition provide a framework for a spatial interaction
based ontology of music, this has been illustrated through basic observations
about how our human physiology affords particular experiences of the world and
also by examining similarities in neurological processes. A series of 6 interactive
experiments were developed in order to directly expand upon the multimodal
experiences described in the first chapters. The information and precedents
discussed suggest that we should also be able to form a symbiotic perception of
Fig. 12 - Experiment Setup.
Custom Software
space and music that uses our knowledge and experience of both to produce an
intuitively interactive gestalt.
Surround Audio
The 6 experiments featured 3 focussed on “sound” and 3 focussed on “music”
and were followed by a short questionnaire about the different interactions and the
user’s experiences (Fig. 12-13). The experiments were tested on students ranging
in musical ability and spatial awareness and were placed in a category of either
“Architect” (architecture students with less than 5 years musical experience),
“Musician” (non architecture student with over 5 years of musical experience),
“Musician Architect” (Architecture student with over 5 years musical experience)
or “Non Musician” (does not study architecture and has less than 5 years musical
experience). The main body of the questions required an X on a spectrum between
Agree and Disagree; the data of each questionnaire was calculated by measuring
the distance on the spectrum and dividing it by the length to create a coefficient of
Fig. 13 - Speaker placement and Camera field of view.
agreement (0 represents strong disagreement and 1 represents strong agreement).
(Film clips of the experiments are on the Data DVD at the back)
22
Pitch and Height.
Experiment 1
The first experiment the user was introduced to aimed to test the prevalence of the
embodied ‘Pitch is Height’ metaphor previously examined by C.C Pratt et al.10 The
user was effectively presented with an invisible plane of sound that was consistently
at arm’s length in front of them (Fig. 14) (PitchAndHeight.mp4); as the user extended
their arm out into this field an audible sine wave was produced. The frequency of
the sine wave was directly related to the user’s vertical hand position such that when
the user moved their hand up, a higher pitch was produced and vice versa a lower
position produced a deeper tone. The distance the user reached out correlated to
the sound’s volume, emulating the effect of force or pressure in creating the sound.
The pitch and volume would correlate to their hand’s movement and position in
real time. After exploring the relationships between sound and space for 1 minute
the relationship was reversed such that moving the hand towards the ground now
produced a higher pitched tone. The users were given another minute to play with
this interaction and were asked which interaction they found more intuitive and
natural. 92% of the subjects agreed that the deeper tones belonged below the
higher pitches in space and the average coefficient of agreement was 0.84. One
“Architect musician” subject felt very strongly that the higher pitches intuitively
belonged below the deeper pitches in space, it was soon made evident that this
particular user had played the cello for 12 years from the age of 6, the embodied
action involved in raising the pitch of a cello indeed requires a lower hand position
in space. This demonstrates the hypothesis that relationships between sound and
space are forged through corporeal interaction and experience.
Fig. 14 - “High” and “Low” Frequency Sound Plane Visualised.
23
Sonic Compass.
Experiment 2
The second experiment used a similar mechanic of reaching away from the body to
Fig. 15 - Reaching into Sound Compass.
produce a sine wave pitch yet the responsive area was arranged radially around the
center of the body as a ring at arm’s reach. The Highest pitches pointed towards the
front of the lecture room and the lowest pointed towards the rear of the lecture room.
The users were able to tap into this audial sound compass by extending their hands
out and feeling for direction with the sound (Fig. 15). The users were not told how
the sounds were oriented but were given 2 minutes to explore their affective audible
environment. The users were then blindfolded and disorientated by revolving on
the spot a number of times. The blindfolded, disoriented users were asked to
simply point towards the lectern at the front of the room. Despite only experiencing
this interaction for 2 minutes, 88% of the subjects were able to successfully and
instinctively use the sound as an audible sense of orientation and point in the correct
direction. This effectively shows how sound information can be imbued with spatial
orientation information and that even with only 2 minutes of exposure to the system
the users were able to use their augmented senses of proprioception and hearing to
make confident spatial decisions about their orientation in the space.
The speakers used for this experiment completely surrounded the user such that no sense of stereographic orientation could be used
47
24
Sounds in Space.
Experiment 3
This ability to map the experience of a sound into a 3 dimensional position through
bodily interaction illustrates the potential of sensory chiasm when perceiving spatial
objects and sounds. The aural connections between sound and shape discussed in
The third experiment featured three
the first chapter imply that the perceived shape of the sound would be affected by
differenty pitched string instrument sounds
the timbre and texture as well as just the interactive spatial experience. In the future,
that were hidden within the room. The
this experiment could be expanded by using different types of sound as well as just
sounds were modelled as spheres that
different positions, pitches and sizes to test if perceptions of shape and form were
emitted a continuous note when occupied
influenced by audible timbre.
by a hand. The volume of the sound
increased as the hand moved towards
the center of the sphere. The users had
to explore the space at different heights
until they made contact with one of the
sound volumes (Fig. 16). This interaction
relies solely on the connection between the
aural sense and the sense of the body’s
position through proprioception and sight.
The users were surprised to discover the
sounds at first but went on to discover the
others using various methods of scanning
the space (Fig. 17). 63% of the subjects
experienced the sounds they interacted with
as 3 dimensional forms in space, however
the perception of the actual shape of the
sounds in the air ranged from the accurate
description of spheres through to columns,
discs, walls and cubes. The experiment
was modelled such that the sounds were
not only different heights but varied in
radius between 30cm and 50cm, 67% of the
subjects perceived the change in physical
size of the sounds in the space. 83% of the
subjects were able to correctly relocate the
position of the invisible sounds in the room
and could even play them as chords using
both hands.
Fig. 16 - Searching and Finding Sounds (Film Stills)
Fig. 17 - Searching and Finding Sounds (Sounds visualised)
25
Photoshop image here!
Spaces of Sound.
Experiment 4
Experiment 4 tests how the presence of a modulating sound field can change a
user’s perception of a room as they move through it. The room was divided into 3
overlapping zones of music, such that as the inhabitants moved around the room,
the audible characteristics of the space changed with their position (Fig. 18). The
3 pieces of music had their tempo synced with each other such that there was a
Fig. 18 - Visualisation of 3 Continuous Sound Zones
smooth transition between each piece when the users moved between zones. The
users explored the room to discover the new audible significance of the different
spaces, most of them developed new preferences for particular positions in the
room. 96% of the users found that the music significantly changed the character of
the spaces, and 96% were able to perceive the spatial presence of a modulating
sound field. The experiment also caused some users to jump from one zone to
another at the end of a bar to further control the music as well as passively observe
the changes. The questionnaire required the users to draw a map of the sound
spaces on a floor plan of the room. 67% of the users accurately illustrated the
Fig. 19 - Correct Zone Diagram
spatial layout of the music zones (Fig. 19). The remaining participants produced
less accurate diagrams of how the room was divided, yet still picked up some
attributes of the zones such as the fact that the zones overlapped (Fig. 20). Only
54% of all architecture students we able to successfully illustrate the zone divisions,
whereas the other 46% had close representations but generally overcomplicated
the spatial arrangement; this suggests that the architect’s refined skill for thinking in
plan had not necessarily expanded their ability to map the sounds into space but
perhaps added extra complexity from other factors in the music. However, 70% of
the musicians were able to draw the zones in plan with near perfect accuracy. This
is evidence that instrumental experience builds and reinforces strong associations
Fig. 20 - Incorrect Zone Diagram
between corporeal action in space and audible reaction.
26
Music and Motion.
Experiment 5
The 5th experiment aimed to bring the temporal dimension of music into a physical 3
dimensional representation that the participants could interact with. The experiment
was developed by again looking to metaphor as a source of crossmodal chiasm,
particularly the words we use to talk about direction and time. We use the terms
such as “looking forward to something”, “backwards”, “behind” and “thinking
ahead” to talk about linear directions in time. Tversky describes the embodied
meaning of the word “forward” as the space in front of our bodies, or our primary
direction of movement.48 Using the principle that this linguistic metaphor has both
temporal and spatial associations, the experiment was developed such that any
forward motion of the user (such as walking) would cause the music to play; if
they stopped, the music stopped, and if the user walked backwards, the music
would also play backwards (Fig. 21). Through interacting with the space the users
were allowed to feel the alignment of their motion with the dimension of time and
therefore have the ability to move through or control it at will. 96% of the participants
agreed that the music moved in the same direction as their body with the strong
average agreement of 0.88 (MusicAndMotion.mp4). A common response to this
experiment was that the subjects tried to control the tempo of the music with how
fast they moved, this feature was omitted due to the technical challenges and time
constraints, however future versions of this experiment could be greatly enhanced
by allowing the subject to travel along the temporal dimension of music at their own
speed within the space as well as simply direction. These comments and statistics
of engagement clearly illustrate that we perceive both music and spatial motion as
dynamic temporal structures.
48
Fig. 21 - Walking through music, forwards and backwards.
Tversky, B. (2008) p.201
27
insulation will appear muffled and
Haptic Music.
Experiment 6
will lack the definition of the higher
frequencies. From this common
experience, muffled sounds carry
with them associations of spatial
Music, sound and noise are commonly experienced
as ubiquitous qualities of space. The sixth experiment
aimed to generate the perception that the music
playing in the space could be physically held and
manipulated with the hands and body. To generate
this perception of a physical and interactive sound
field, the actions that altered the sound were
designed to harness common sensory experiences
and associations to create an immediately intuitive
interface. The experiment focussed on the character
of muffling and dampening sounds and the spatial
factors that create this audible change. The sensory
ubiquity of sound and noise in the everyday
spaces we inhabit has built a comprehensive
perceptual understanding of how sounds act
within and permeate different environments. For
example the acoustic responses of sound in large
and small spaces have immediately recognisable
characteristics of reverb. This relationship also
allows artificial or reproduced sounds with greater
reverberation times to strongly elicit senses of
spatial scale without visual cues (such as the
aforementioned echo gesture in Aphasia). A similar
common experience of sound behaviour is the
natural muffling of sounds in enclosed spaces.
Obstacles between a listener and a sound source
will absorb and reflect different frequencies of the
audio depending on the form, depth and materiality
of the obstacle. For example, building walls tend to
transmit lower audio frequencies as their wavelength
exceeds the depth of the obstacle; as such, music or
noise from a neighbouring room with poor acoustic
Fig. 22 - Screenshots of “Haptic Music” program.
conditions such as being enclosed
in a box, underwater or hidden
behind a wall. The experiment
aimed to harness this spatial
association of muffled sounds
to generate a perception of a 3
dimensional source of pure music
that could be held in the hands,
gathered, enclosed, squeezed,
compressed and even released
back into the room. The interaction
was devised such that as the hands
of the user pressed together a low
pass frequency filter would act on
the music playing in the room, the
cutoff frequency of the filter was
directly proportional to the proximity
of the hands providing the illusion
that the hands were enclosing the
source of the sound. At the point at
which the hands came completely
together, the music had been
reduced to a deep, compressed
throb. If the hands were unclasped
rapidly the music appeared to have
“escaped” and all frequencies were
returned to normal. To capture the
music again the subject had to
extend their arms out to “gather”
the audio and then the music
was able to be compressed and
released at will.
Fig. 23 - Collecting and Releasing Music
(augmented video stills - HapticMusic1.mp4).
28
Haptic Music (Continued).
The music the subjects manipulated was chosen specifically to encourage
participation; the music featured a strong underlying rhythm with a catchy repetitive
chord progression and melody.49 The contemporary music used had very clear
dynamics, build ups and releases at the end of regular 16 bar phrases. The
In this experiment the subject was asked to stand in the middle of the space whilst
repetitive nature of the beat and chord progression was chosen in order to introduce
the music began playing through the speakers. The subject was then simply asked
the subjects to the temporal framework of the music quickly such that they were able
to “compress the music” but was given no visual or gestural cues of how this
to comprehend and act on their expectations of change by reacting with appropriate
might be done. The verbal cue to “compress” the sound elicited various different
body movements. By compressing the sound and reducing the music to the
responses from the subjects, some extended all their limbs before crouching
bassline and underlying beat, the subjects were able to contribute to the dynamic
and curling into a ball (Fig. 24), others raised their hands up then pressed them
changes of the music, generating a literal tension and anticipation of resolution that
together towards the floor and most of the subjects brought their hands together as
is felt both musically and physically. This is a similar effect to the stretching action
if squeezing a large ball (Fig. 25-26). These initial responses indicate the success
and audio used in Applebaum’s Aphasia. 83% of the musician subjects agreed that
of the application of a spatial metaphor to the manipulation of the music. As the
the structure of the music influenced their choice of motion, yet only 40% of the non-
subjects realised that their initial motions affected the music in an expected manner
musician non-architect group found this to be the case. The musicians had a clearly
they almost immediately began to interact with the music as an object or field in
refined awareness of corporeal action with regards to musical structures. Whereas
space. 96% of the subjects agreed that the interaction between body and music
the primary aim of the non-musicians’ actions was simply to examine the effect of
felt natural with a strong average agreement of 0.89. The experiment was not only
the resulting tone of the sound. 100% of the Architect musician group agreed that
devised to test whether a spatial sense of music could be achieved through intuitive
the structure of the music influenced their choice of movement with an average
interaction but also to test how the temporal structure of the music would influence
agreement of 0.93. This clearly illustrates a relationship between the subject’s
the subject’s choice of movement once the perceptual coupling between action and
musical and spatial experience and their ability to think in 4 dimensions.
effect had been made.
Fig. 24 - Crouching to hold the music (Video Stills).
Fig. 26 - Experiment photograph with overlaid illustration of sound.
The music was of a popular genre to the subject demographic yet not well known enough to distract
the subject from the spatial interaction
49
Fig. 25 - Compressing to hold the music (Video Stills).
Experiment 6:
Haptic Music
Table 1. - Experiments Overview.
• We can feel the spatial qualities of music
and sound with our ears.
• We can use previous understanding of
how sound works in space to manipulate
sound and music as physical objects.
Table 2. - Experiment Data.
Architects
Musicians
Architect Musicians
Non Either
100
100
60
100
92
0.94
0.86
0.65
0.84
Architects
Musicians
Architect Musicians
Non Either
% Agreement
0.84
88
83
100
80
88
0.79
0.77
0.87
0.64
0.77
Music and Motion
• We feel time as moving forwards.
• Music can become a navigable string of
movement in space.
Agreement Coefficient
Experiment 5:
88
33
80
40
63
0.79
0.64
0.85
0.39
0.68
75
50
80
60
67
0.80
0.61
0.75
0.61
0.71
75
83
100
80
83
0.66
0.75
0.87
0.75
0.74
60
26
45
32
47
40
59
60
58
60
34
43
37
21
30
40
60
59
40
47
40
54
50
30
100
83
100
100
96
0.91
0.69
0.96
0.97
0.88
60
46
15
57
43
40
42
59
60
60
56
60
47
55
50
60
60
59
57
60
60
44
52
60
100
83
100
100
96
0.87
0.82
0.87
0.93
0.87
60
57
60
57
46
30
42
60
60
60
33
60
36
55
50
60
45
59
59
57
60
44
53
49
7
88
100
100
100
96
0.89
0.79
0.93
0.94
0.88
58
47
43
60
46
50
59
60
60
60
47
60
40
30
50
60
60
59
60
55
58
40
60
49
5
8
5
5
17
0
0
42
2
0
0
0
2
0
15
0
6
0
7
12
0
4
5
12
5
10
9
0
17
0
0
4
0.02
0.24
0.09
0.08
0.10
• Audible attributes of a space significantly
change our experience.
• Sound fields can be perceived spatially
(and accurately described).
45
11
15
60
30
52
59
46
60
44
26
60
17
58
52
24
60
59
60
42
12
54
30
40
Spaces of Sound
59
44
58
51
44
15
33
60
60
60
57
60
45
56
50
60
52
42
60
57
55
45
60
48
10
100
83
100
100
96
0.89
0.69
0.92
0.93
0.85
6
75
83
100
40
75
0.80
0.74
0.93
0.48
0.75
58
56
15
60
30
54
47
30
11
60
57
60
40
56
60
30
60
59
46
11
13
53
58
52
11
100
83
100
100
96
0.95
0.86
0.96
0.91
0.92
57
54
30
59
60
50
45
59
60
60
57
60
50
56
60
60
60
60
60
54
52
56
53
53
88
100
100
80
92
0.85
0.84
0.98
0.82
0.87
60
55
50
51
42
52
46
16
60
60
58
60
54
56
60
60
60
60
59
50
30
45
53
53
General
12
13
• Spatially interacting with sounds can
perceptually give them a 3 dimensional
form.
60
30
30
60
30
30
59
0
50
30
34
60
45
58
50
60
60
54
60
4
47
55
13
3
4
6
Experiment 4:
Architecture
Physics
Engineering
Architecture
History
Chemistry
Architecture
Architecture
History of Art
Architecture
Architecture
Architecture
Architecture
Architecture
Architecture
Architecture
Architecture
Theology
Architecture
PPS
History
Engineering
PPS
Biological Sciences
63
83
40
60
Successfully orientated
1
1
1
0
1
1
0
0
0
1
0
1
1
0
0
0
1
1
0
0
0
1
0
0
Musician Subject
Sample size
8
6
5
5
F
M
M
M
M
F
M
F
F
M
M
F
F
M
F
F
M
F
F
F
F
M
F
F
M/F
• 88% of participants orientated themselves
using proprioception and hearing.
• Participants could functionally use spatial
arrangement of sounds to make decisions.
• Sounds are be imbued with spatial
meaning over time and experience.
59
54
60
37
0
57
59
51
10
59
47
60
37
58
8
60
45
60
60
45
45
45
52
40
5
Sounds in Space
4
Experiment 3:
3
Sonic Compass
3
Experiment 2:
3
5
1
6
5
3
6
2
5
6
3
5
5
4
6
6
6
6
3
4
6
6
4
4
2
6
3
3
1
2
4
3
3
4
6
6
1
5
4
4
4
1
6
6
4
3
6
6
6
2
2
2
2
4
2
4
2
2
5
3
4
3
3
1
5
2
2
1
1
2
5
3
5
3
5
4
4
6
3
5
6
3
2
4
6
6
2
3
3
3
4
3
3
4
2
2
1
1
4
1
3
1
1
1
1
1
1
1
3
1
1
2
1
4
1
2
2
1
1
1
6
4
6
6
4
6
3
4
5
3
1
5
3
2
4
1
2
5
5
2
1
1
2
5
Average rank of preference
6 4 5 3 1 2
4
4
6
5
6
5
5
6
4
5
4
2
2
2
5
5
2
5
5
5
5
5
3
5
1 2 3 4 5 6
• 92% of participants placed higher pitches
above lower pitches.
• Some musician’s instrumental experience
had reversed this placement.
• Spatial associations to pitch are built
through embodied actions.
60
53
60
50
46
57
59
60
34
0
50
60
15
58
60
54
60
37
60
57
55
55
50
55
2
2
Pitch and Height
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
1
1
Experiment 1:
Experiment >
Question Number >
Subject Number v
29
30
Discussion.
The results of the experiments clearly show that we have the ability to perceive
both music and sound as spatial forms. We naturally use metaphors to mediate
meaning between musical and spatial experiences yet rarely actively engage our
experiential knowledge of one with the other. This ability is not reserved to those with
neurological synaesthesia, it is built through our everyday experience. It is clear that
we have moved beyond the question “Are music and space cognitively connected?”
to “How can we apply this unified understanding of embodied perception to
design?” and particularly “How can the diciplines of music and architecture
augment and inform each other?”
I have demonstrated that by using emerging technologies and systems, both
spatial and audible presence can be combined to augment a user’s navigation and
experience of architecture. The results of the experiments speculate that these forms
of navigable music could become an architecture unto themselves. In experiments 3
and 4 the subjects used words such as “columns”, “corridors”, “walls” and “rooms”
to describe the sounds they interacted with. This shows the potential to create
an immaterial architecture made of music that is still spatially navigated. I have
illustrated that the instrumental experience of the participants heavily influenced
their audio-spatial ability; however the architecture itself now has the potential to
become the instrument that musicians and non-musicians alike will inadvertently
play (and thus practice) with their bodies as well as their eyes. This implies that even
non musicians could become musicians of space as they are exposed to this type of
architecture.
31
The experiments have not only shown that we perceive spatial inferences in sounds
but also that music can have an affective role in our spatial interaction when the two
are coupled. This affective quality of music could extend the idea of an architecture
made of sound to become active navigational sound architecture for the blind,
where sounds are naturally imbued with spatial characteristics. Ascending scales
of notes could have clear spatial connotations to obstacles such as stairs, or
volumes of sounds with different textures could be used to generate perceptions
of proximity to various objects in a room. The refined musical ability of temporal
pattern recognition could be attuned to the motions of the body as it moves through
space. Phrases of music can now actually become “musical places” which develop
and change as we move from one to another. This combines both uses of the
hippocampus discussed and could be used as a framework to distinguish different
spaces and their relationships to each other. In the same way as Minsky described
musical learning, this system of navigable music would perceptually build into
intuitive wayfinding devices that cue memories of what lies ahead as inhabitants
move through them. These are just a few examples of embodied music concepts
that could contribute to a much larger vocabulary and temporal grammar of intuitive
architectural sound elements.
The implementation and curation of these elements need not be functional and
could become new kinds of architectural ‘detail’. Experiment 4 illustrated how our
visual perceptions of space are overridden by the audible characteristics, in the
same way that film soundtracks significantly enhance or alter the visual narrative.
Pockets of music can now be placed in buildings that change and augment the
qualities of that particular space. The study of how and what music makes us feel is
a larger avenue of research but the architect has the opportunity to change the way
their buildings are listened to as well as seen. The music and the architecture have
the potential to be composed together: Michael Gondry’s film for “Star Guitar” by
the Chemical Brothers is a fantastic example of how music changes our perception
of images, and also how features of space could be heard and augmented in
this way (StarGuitarExperiment.mp4). Every element in the landscape of the train
journey is highlighted by a different sound; attention is drawn to every element in
the scene as it is revealed and audibly signified in the music. Architecture could
use similar technology to literally become audible music in this way, or alternatively
(and perhaps more interestingly) sequences of spaces could be designed with
these principles of Music Aided Design to actively stimulate the same temporal
continuities, contrasts and empathic motions that move us so deeply in music.
32
There is the opportunity to apply this musical intelligence during the appraisal
and brief stages of architectural design. During the contextual analysis of a
site, particular attention could be given to the details, rhythms and forms of the
surrounding spaces and buildings, analysing the developments and changes of
a dynamic perspective through the area. Musical composition could be used to
understand how these rhythmic and dynamic textural changes lead to a dynamic
architectural language of the area as well as just a static, stylistic or materialistic
language. The gap (or silence) of the undeveloped site in the routes through the
area could be conceptually developed based on the expectations, contrasts and
rhythms (to name a few) of the temporal moments either side. This could even be
achieved by composing the music of the context and experimenting with potential
surprises or continuities in the currently empty site. The approaches and departures
from the building could therefore be actively engaging the visual and spatial aspects
of the site with a larger musical narrative. This would continue inside the building
where languages of ‘exposition’, ‘development’ and ‘recapitulation’ (taking sonata
form for example) could be used to create gradual changes, moments of surprise,
steady continuities or moments to reconfirm or change our perceptions of the whole.
The results of this discussion also have implications in other disciplines. Similar
technologies and bespoke software could allow sounds to be actively sculpted,
stretched, smoothed, textured and manipulated as spatial objects. The sounds
could even be passed between multiple collaborators or moulded and developed
together. This could have huge potential in creating music through dance, and also
in electronic sound design and musical composition.
The precedent studies, neurological studies and experimental studies have all
revealed that they may be explored in far greater depth than I have done here.
The experiments deserve a greater sample size to critically examine the changes
between refined spatial awareness and musical ability. The study would also benefit
from exploring the spatial responses to many different types of music and sound
in each experiment. To extend this research, empathy and empathic involvement
with space, motion and music will also need more comprehensive analysis. This
could even extend into how mirror neurons activate the sensorimotor system just by
perceiving the sounds and motions of other objects and people. This dissertation
has been a proof of concept in what clearly has the potential to be a much larger
practical and theoretical pursuit in the disciplines of both architecture and music.
33
Illustrations & Tables.
Fig. 1: Excerpt from Xenakis’ Metastasis graphical score.
Fig. 2: Excerpt from Xenakis’ Mycenes Alpha graphical score.
Fig. 3: Hyperbolic Paraboloid, Straight lines creating curved forms, diagram.
Fig. 4: Xenakis and Le Corbusier’s Philips Pavilion, photograph.
Fig. 5: Excerpt from Mark Applebaums Metaphysics of Notation graphical score.
Fig. 6: Kiki and Bouba diagrams.
Fig. 7: Superimposed frames of Oskar Fischinger’s Studie Nr. 7 showing expectation and resolution. Fig. 8: Mark Applebaum playing his bespoke Mouseketeer instrument, photograph.
Fig. 9: Aphasia Echo gesture, film stills with illustrations.
Fig. 10: Aphasia Stretch gesture, film stills with illustrations.
Fig. 11: Guide Psychogeographique de Paris book cover, illustrates relative space mapping, photograph.
Fig. 12: Experiment Setup. Photograph with labels.
Fig. 13: Sketch plan of room and speaker layout, drawing.
Fig. 14: “High” and “Low” frequency sound plane, Illustrated film stills.
Fig. 15: Reaching into Sound Compass, Illustrated film frame - SonicCompass.mp4.
Fig. 16: Searching and finding sounds in space, film stills - SoundsInSpace.mp4.
Fig. 17: Searching and finding sounds in space, Illustrated film stills
Fig. 18: Visualisation of different audible music zones within the space, illustrated photograph.
Fig. 19: Participants correct zone diagram, drawing.
Fig. 20: Participants incorrect zone diagram, drawing.
Fig. 21: Walking backwards and forwards through the music, illustrated film stills.
Fig. 22: Screenshot of motion camera and “Haptic Music” program, screenshot.
Fig. 23: Collecting and releasing music, film stills - HapticMusic1.mp4.
Fig. 24: Crouching to collect the music, film stills.
Fig. 25: Compressing to hold the music, film stills.
Fig. 26: Tangible music, Illustrated photograph.
Table 1: Experiments overview.
Table 2: Experiment numerical data.
34
Ekstrom A.D, Kahana M.J, Caplan J.B, Fields T.A, Isham E.A, Newman E.L, Fried I. (2003).
Cellular networks underlying human spatial navigation. Nature 425:184–188.
Erneling, C. E., & Johnson, D. M. (Eds.). (2005). The mind as a scientific object: between
brain and culture. Oxford University Press: USA.
Bibliography.
Fishwick, P. A, (2006). Aesthetic Computing, Cambridge MA: MIT Press
Gaser, C., & Schlaug, G. (2003). Brain structures differ between musicians and non-
Arnold, R (2012). “Theres No Sound In My Head”, Online documentary about the
performances of Mark Applebaum’s “Metaphysics of Notation” https://vimeo.com/14469188
Bengtsson, I & Eggebrecht, H. (1973). Verstehen: Prolegomena zu einem semiotischhermeneutischen Ansatz. In P. Faltin & H.-P. Reinecke (Eds.), Music und Verstehen:
Aufsatze zur semiotischen Theorie, Asthetik und Soziologie der musikalischen Rezeption.
Cologne: Volk.
Borries, F (2007). Space Time Play: Computer games architecture and urbanism, Germany:
Birkhauser
Brown S (1999). The “Musilanguage” Model of Music Evolution. In Wallin NL, Merker B, and
Brown S, eds. The Origins of Music. Cambridge MA: MIT Press.
Clark, A (2008). Supersizing the mind, embodiment, action and cognitive extension, Oxford:
oxford university press
Cupchik, G. C., Phillips, K., & Hill, D. S. (2001). Shared processes in spatial rotation and
musical permutation. Brain and cognition, 46(3), 373-382.
Debord, G. (1957). Guide psychogéographique de Paris. Édité par le Bauhaus imaginiste.
Douglas, K.M., and Bilkey, D.K. (2007). Amusia is associated with deficits in spatial
processing. Nat. Neurosci. 10, 915–921
Dourish, P (2004). Where the Action Is, the foundations of embodied interaction, Cambridge
MA: MIT Press.
musicians. The Journal of Neuroscience, 23(27), 9240-9245.
Gray J.A, Rawlins J.N.P. (1986). The hippocampus, Comparator and buffer memory: an
attempt to integrate two models of hippocampal function. Plenum: New York, p159–201.
Hatten, R. S. (1994). Musical meaning in Beethoven: Markedness, Correlation, and
Interpretation. Bloomington: Indiana University Press
Hebb, D. O. (1949). The Organisation of behaviour: A Neuropsychological theory. New York:
John Wiley and sons.
Herdener, M., Esposito, F., di Salle, F., Boller, C., Hilti, C. C., Habermeyer, B., ... & CattapanLudewig, K. (2010). Musical training induces functional plasticity in human hippocampus.
The Journal of Neuroscience, 30(4), 1377-1384.
Hofmann, B (2005). Spatial Aspects in Xenakis’ Instrumental Works. In Makis Solomos,
Anastasia Georgaki, Giorgos Zervos (ed.), Definitive Proceedings of the “International
Symposium Iannis Xenakis” (Athens, May 2005), www.iannis-xenakis.org, October 2006
Howard, D (2009). Sound and Space in renaissance venice, China: Yale University Press
Huron, D. (2006). Sweet Anticipation: Music and the Psychology of Expectation. Cambridge,
MA: MIT Press.
Husserl, E, (1928). Lectures on the Phenomenology of Internal Time Consciousness,
Bloomington : Indiana University Press
Köhler, W (1929). Gestalt Psychology. New York: Liveright.
35
Knight, R (1996). Contribution of human hippocampal region to novelty detection. Nature
Pratt, C.C (1930). The spatial character of high and low tones Journal of Experimental
383:256–259.
Psychology. 13 (1930), pp. 278–285
Knights S, (2012). Visual Music: the films of Oskar Fischinger. Cambridge: History of Art
Merleau-Ponty, M. (1996). Phenomenology of perception. Motilal Banarsidass Publishe.
dept. Dissertation
Ramachandran, V. S., & Hubbard, E. M. (2001). Synaesthesia - a window into perception,
Kumaran D, Maguire EA (2007). Which computational mechanisms operate in the
thought and language. Journal of Consciousness Studies, 8(12), 3-34.
hippocampus during novelty detection? Hippocampus 17:735–748.
Rasmussen, S. E. (1964). Experiencing architecture (Vol. 2). Mit Press.
Lakoff, G., & Johnson, M. (1980). Metaphors we live by (Vol. 111). London: Chicago.
Rodrigues, A. C., Loureiro, M. A., & Caramelli, P. (2010). Musical training, neuroplasticity and
Lakoff, G. J., & Johnson, M. M.(1999). Philosophy in the Flesh. The Embodied Mind and Its
cognition. Dementia & Neuropsychologia, 4(4), 277-286.
Challenge to Western Thought. New York: Basic Books.
Rusconi E, Kwan B, Giordano B.L, Umilta C, Butterworth B (2006). Spatial representation of
Leman, M, (2008). Embodied Music Cognition and Mediation Technology. Cambridge MA:
pitch height: The SMARC effect. Cognition, 99 (2), pp. 113–129
MIT Press
Sacks, O, (2008). Musicophilia; Tales of music and the Brain, New York: Knopf
Lidji, P., Kolinsky, R., Lochy, A., & Morais, J. (2007). Spatial associations for musical
stimuli: A piano in the head?. Journal of Experimental Psychology: Human Perception and
Shapiro, L, (2010). Embodied Cognition. USA: Routledge
Performance, 33(5), 1189.
Stewart, L., & Walsh, V. (2007). Music perception: sounds lost in space. Current Biology,
Maguire, E.A. (2001). Neuroimaging, memory and the human hippocampus. Rev Neurol
17(20), R892-R893.
Paris 157:791–794.
Strange B.A, Fletcher P.C, Henson R.N, Friston K.J, Dolan R.J. (1999). Segregating the
Maguire, E.A. Gadian D.G, Johnsrude I.S, Good C.D, Ashburner J, Frackowiak R.S, Frith C.D.
functions of human hippocampus. Proc Natl Acad Sci U S A 96:4034–4039.
(2000). Navigation-related structural change in the hippocampi of taxi drivers. Proc Natl
Acad Sci USA 97: 4398-4403.
Strange B.A, Dolan R.J. (2001). Adaptive anterior hippocampal responses to oddball stimuli.
Hippocampus 11:690–698.
Mallgrave, H. F, (2010). The Architects Brain. UK: Wiley-Blackwell
Tversky, B. (2008). Spatial cognition: Embodied and situated. The Cambridge handbook of
Maurer D, Pathman T & Mondloch CJ (2006). The shape of boubas: Sound-shape
situated cognition, 201-217. USA: Cambridge University Press.
correspondences in toddlers and adults. Developmental Science 9 (3): 316–322
Van Gelder, T, (1998). The dynamical hypothesis in cognitive science, Behav. Brain Sci. 21,
Minsky, M (1981). Music, Mind and Meaning. In Clynes M, ed. Music, Mind and the Brain:
615–628. doi: 10.1017/S0140525X98521735.
The Neuropsychology of Music. Plenum: New York
Vinogradova, O.S. (2001). Hippocampus as comparator: role of the two input and two output
Piaget, J. (1970). Piaget’s theory.
systems of the hippocampus in selection and registration of information. Hippocampus
11:578–598.
36
Appendix:
(Volunteer data is recorded in hardcopy)
www.felixfaire.com