School of Arts, Culture & Environment
Msc in Design & Digital Media 2010-2011








Title: "What is synesthesia, understood clinically and
artistically? What are its implications for creating digital media
works?"








Essay by Georgios Aposkitis (s1052908)

Course: Media & Culture (ARCH11002)

Course organiser: Richard Coyne

---

Preface ..................................................................................................... 3
Synesthesia from a clinical viewpoint ....................................................... 4
Traces of synesthesia in artistic endeavors of the past ............................ 5
From the early visual masters to the digital art era: different means,
similar visions ........................................................................................... 7
Crucial views on perception in order to understand synesthesia .............. 8
Childhood, synesthesia and education (Were we all born synesthetes?
Can we return to that point?) .................................................................... 9
Selected bibliography and online resources ........................................... 11
Addendum .............................................................................................. 13

2

---

Preface

"In the green group, there are alder-leaf f, the unripe apple of p, and
pistachio t. Dull green, combined somehow with violet, is the best I can do for
w. The yellows comprise various e's and i's, creamy d, bright-golden y, and u,
whose alphabetical value I can express only by 'brassy with an olive sheen.'
In the brown group, there are the rich rubbery tone of soft g, paler j, and the
drab shoelace of h. Finally, among the reds, b has the tone called burnt
sienna by painters, m is a fold of pink flannel, and today I have at last
perfectly matched v with 'Rose Quartz' in Maerz and Paul's Dictionary of
Color. The word for rainbow, a primary, but decidedly muddy, rainbow, is in
my private language the hardly pronounceable: kzspygv."
(Vladimir Nabokov, Speak Memory)


Synesthesia is a term derived from the ancient Greek words
(which means together) and (which means sense), its meaning
however is associated with much more than the sum of its parts. There have
been many sorts of different approaches to this phenomenon which seems to
gradually attract more and more researchers and it was not only until very
recently -during the last two centuries and especially during the 2000s- that
scientists working on different fields, especially neurologists, linguists and
psychologists began to deal with this phenomenon as seriously being worth
looking into.
Synesthesia, though, as a term on its own or similar terms more or less
clearly implying synesthesia have been occurring for centuries in people's
everyday lives, in music, in art, in literature as well as in philosophy and
theosophy. Listening to colours, tasting shapes, seeing sounds - what could
such notions mean to the western world whose basis lies on the existence of
five separate senses? Is synesthesia actually a disorder or a special way of
perceiving? Why have so many different artists from different eras and diverse
groups approached or tried to approach the phenomenon of synesthesia?
Why do everyday people seem to find the idea of synesthesia so exciting?
Have we all been synesthetes at some point of our lives? Is it possible for
non-synesthetes to develop synesthetic skills? In which way is the creation of
digital art related to synesthesia and by which means could we be trained in
order to fully appreciate this sort of art?
Above are some of the topics that are going to be examined throughout
the present essay. Synesthesia regarded as a mental state, as well as what
could be regarded as synesthetic art. Hopefully, some light will be shed on a
topic that still remains controversial and is probably bound to remain this way,
unless the western world steps to new, unexpected directions - in terms of
how we insist on viewing the human mind- and a whole new way of thinking is
gradually formed to reveal another kind of reality.

3

---

Synesthesia from a clinical viewpoint

Most people have never heard of synesthesia. Many of those who
indeed experience synesthesia find the above so strange, in fact they think
that their way of perceiving is normal and cannot understand nor feel the way
in which the rest of the humanity does so. Early scientific approaches to the
phenomenon mainly fell to skepticism. The majority actually used to think of
synesthesia as a joke, as something impossible, as a certain form of drug-
induced (or not) illusion, as fantasy, as a product of the imagination of
deranged minds. Nowadays, after some extended experiments have been
made, as well as brain scans of synesthetes, synesthesia is generally taken
into consideration as a respectable phenomenon that is out there, even if it
cannot still be accurately explained.
Synesthesia is usually taken into account as a neurological defect, as a
brain defect, in which different regions of the brain that are not expected to
communicate with each other participate in what is described as "crosstalk".1
In that sense most of the definitions of synesthesia, at least in the past,
defined the phenomenon having to do with cross modal sensory experiences,
that is according to them synesthesia seems to occur at those cases where
the stimulation of one sensory or neurological pathway triggers the stimulation
of another, leading to automatic multi-sensory experiences. The way in which
we see the phenomenon is at the moment changing and is about to change
the more scientists talk to actual synesthetes and large-scale experiments are
held. Scientists are now starting to move towards the direction that the brain
integrates sensory information all the time2.
Listening to synesthetes speaking can be in many senses apocalyptic
for those who have not themselves actually witnessed the phenomenon.
Watching documentaries on-line and reading interviews of synesthetes, one
cannot but immediately notice that the following words are repeated and could
be considered to be common ground:

1. Synesthetes put great emphasis on the fact that synesthesia is
not like seeing music in colours, like tasting shapes, like
visualizing music but is exactly that, a unique conjunction of the
senses. In other words, synesthesia is not a metaphor, but a
completely different way of perceiving the world and should be
treated as being this way.
2. Most of the synesthetes find it hard to understand how come
one can be a non-synesthete. Coming back to point one,
synesthesia is a sui generis state, with its own rules, that is
probably why it was until recently treated as a disorder by
outsiders, for the same reason that the actual synesthetes
cannot explain the way in which non-synesthetes sense and
perceive.
3. Most of the people experiencing synesthesia believe they are
gifted, enjoy what their special ability has to offer and not only

1 Van Campen, 2008, 1
2 http://www.synesthesia.com.au/id1.html

4

---

would they not get rid of it at any cost but think of it as an
indispensable inborn quality of their actual being. For how long
could something like that continue to be treated as a disorder?

Synesthetes do share all of the aforementioned, but that does not in
any way mean that synesthesia occurs to each one of them in exactly the
same way. There is a great deal of diversity in the sense that every
synesthete hears his own colours, associates each sound with a different
number, which means that they may all experience the same phenomenon, it
is though customized to a certain degree and appears to different persons
with slight variations. For most of the synesthetes, synesthesia is a one-way
travel between the senses, which means they listen to sounds when viewing
colours but not the other way round, there are, however, cases where
synesthesia occurs as a two-way traffic as well. There is recent proof that
information is constantly traveling between senses, that meaning that the -
until some years ago- widely accepted idea of one-way transportation was
wrong.3
There have, finally, been quite a few recent estimates on the extent to
which synesthesia appears within a population, the most well-known being
that made by Julia Simner and colleagues at the University of Edinburgh in
2005 demonstrating that synesthesia is by far more common than it was
thought to be: 1 in 23 for any type of synesthesia.4 This cannot be considered
outside the frame of the increased interest in the phenomenon: the more
scientists and common people become aware of its existence, the easier it
becomes for actual synesthetes to identify themselves and not be afraid to
share their special way of perception. We must always, however, keep in
mind that distinguishing synesthetes from non non-synesthetes with the use
of scientific tests is in fact hard and whatever results come out cannot be
taken as granted as long as the term itself remains obscure.
Traces of synesthesia in artistic endeavors of the past

Synesthesia has been involved with the creation of different types of
music and art for many centuries and there are also quite a few examples of
great authors who have been synesthetes. The most prominent of those
cases linked to synesthesia will be briefly discussed in the following
paragraphs.
Vladimir Nabokov was a synesthete and synesthesia is stated or
implied in many of his works. There are innumerable expressions such as "a
sigh of delicious relief" in his most famous "Lolita" (1965), heroes and anti-
heroes "suffering" from synesthesia in many of his novels as well as direct
statements in his autobiography "Speak, memory" (1966) that not only himself
but also his mother and his son were synesthetes, which was groundbreaking
to declare at that moment, when synesthesia was generally considered to be
folly. It is still unclear if synesthesia is a trait passed on from a generation to
the next one, however such cases have been reported and Nabokov's is
probably the most well known.

3 Van Campen, 2008, 15-17
4 Cytowic, Eagleman, 2009, 8-9

5

---

Among the various types of synesthesia, the one having to do with
music, in particular experiencing colours when listening to, thinking of or
composing music is the most common and probably the most intense. Most of
the people who have experienced it are musicians, so they either are closer to
synesthesia or they seem to be more easily aware of it. 5
In rock music, especially in the context of 60s and 70s psychedelia, lots
of different bands experimented with hallucinatory drugs as well as the
projection of visuals and effects during concerts in order to try to provide their
listeners with multi-sensory experiences. The light shows of that era as well
as drug-induced hallucinations are nowadays considered not to have so much
in common with the real synesthetic phenomenon, due to the fact that they
are temporary and unstable. LSD and similar drugs sometimes -not always-
induce synesthetic or quasi-synesthetic effects and synesthetes seem to react
to the use of such substances to the same way as non-synesthetes.6 It is
interesting, though, that some synesthetes have found the visuals of Pink
Floyd exceptional and really close to the actual phenomenon, so one could
not be sure even for that. Generally, though, those cases as well as the more
recent veejaying that accompanies electronic music dj sets in clubs are
considered to be close to synesthesia but at the same time not quite
synesthetic because of the aforementioned reasons.7
Linda Perhacs and her album "Parallelograms" first released in 1970 is
a great example of non drug-induced, synesthesia-inspired music of that era,
blending acoustic instruments with electronic landscapes to create
idiosyncratic visual music.
Quoting her own words from an interview held through e-mails
especially for this essay: "First off, I have experienced synesthesia for as long
as I can remember. I could see light colors in music and had always thought
that others could experience this as well. I have the ability to see patterns of
music, dance, color and light in moving 3 dimensional patterns. Contrary to
what some people may think, none of my songs, including "Parallelograms"
were influenced in anyway by direct experience with hallucinogens. I had
seen what drugs do to people and wanted no part of it...if I want to paint with
sound, then the higher things are going to have a different wavelength, so I
literally drew it on a scroll with the understanding that I wanted three
dimensional shapes. But yes, it was a concept that came quickly, like a light
bulb going on. And I saw it all at once, as a full composition where you're
painting with sound, the words are coming out as sound creating those
shapes."
In classical music there have been quite a few different approaches to
what could be synesthesia, most of them though are now regarded with
skepticism and are thought to be close to the phenomenon but not the
phenomenon of synesthesia itself. Such is the case of classical program
music, such as for example Claude Debbusy's "La mer". What mainly
differentiates program music from true synesthetic experience is the fact that
contrary to synesthesia, with program music it is the linked thought and not
the stimulus itself that evokes the effect, as well the fact that with program

5 Sacks, 2007, 167-168
6 Cytowic, 2002, p.101
7 Van Campen, 2008, 17-18

6

---

music one can decide not to think of the bonds between colours and sounds,
whereas a synesthete cannot at all break those links even if he wants to.8
20th century classical music has been probably closer to synesthesia,
at least in terms of it being a fundamental source of inspiration for composing,
than any other music and composers such as Olivier Messiaen, Gyorgy Ligetti
and Michael Torke have clearly stated synesthesia directly inspiring the
creation of their works. The following quote from Claude Samuel's book "
Conversations with Olivier Messiaen" is characteristic:
" Claude Samuel: For you, a composer, is the presence of colour in
nature then as essential as that of sound?"
Olivier Messiaen: Both are linked. Without suffering from synopsia (as
did my friend the painter Blanc-Gatti, who had a disorder of the optic and
aural nerves which allowed him really to see colours and shapes when he
heard music), when I hear a score or read it, hearing it in my mind, I also see
in my mind's eye corresponding colours which turn, mix and blend with each
other just like the sounds which turn, mix and intermingle, and at the same
time as them..."
In art, there have been many examples of painters giving names that
imply synesthesia to their paintings but as was the case with program music,
thought intervenes between the stimulus and the (pseudo) synesthetic
experience. Moreover, whereas music is generally time-based and musical
compositions gradually build up through time, a painter cannot predict or
impose any certain sequence in which the observer is viewing his art.9
Just like 20th century classical music, certain artists mainly creating
abstract painting came closer to synesthesia than ever before. Wassily
Kandinsky and the whole Blaue Reiter team working in Germany during early
20th century tried not only to create synesthetic art but also to provide through
research -and Kandisnky's own insight into theosophy and the romantics-
certain aesthetic background that could support this sort of art. He moves
further, suggesting that those integrated sensual experiences are in harmony
with the awakening of the spirit. Quoting from his book "On the spiritual in art"
(1910): "...then admittedly, sight must be related not only to taste, but also to
all the other senses. Which is indeed the case. Many colours have an uneven,
prickly appearance, while others feel smooth, like velvet...Even the distinction
between cold and warm tones depends upon this sensation..." When reading
the Blaue Reiter Almanac, one can come across some more theory regarding
the sort of Gesamtkunswerk they intended to create as well as the sketches
for the stage play "The yellow sound. A stage composition", which is a really
interesting depiction of what Kandisky had in mind10.

From the early visual masters to the digital art era: different
means, similar visions

In early 20th century visual arts, especially inside the avant-garde
movements synesthesia has been explored in various ways. The collages and

8 Van Campen, 2008, 19-20
9 Brougher, Strick, Wiseman, Zilczer, Mattis, 2005, 18-19
10 Kandinsky, Mark eds, 1974, p.207

7

---

videos of Oscar Fischinger followed a path very similar to Kandinsky's vision
in the sense that images or moving images blended with sounds and the
actual process of musical composition. Disney's "Fantasia" was a more
mainstream approach to the creation of some sort of visual music (in fact O.
Fischinger had drawn some of the initial sketches), while Norman Mclaren's
avant-garde work in the 1940s, especially his work "Dots" definitely implies
synesthesia as well. Len Lye's work in the 1930s consisted of combinations of
abstract visual art -he painted directly on the film- with Latin and Brazilian
rhythms.
Hy Hirsh and Harry Smith in the 1960s kept this "direct film" and "rejection of
camera" approach. Being in parallel deeply influenced by the visuals and
aesthetics of Kandinsky, they tried to combine all of the above with popular
music. More recently Stan Brakhage created films that could be characterized
as visual music and James Whitney with his film "Lapis" (1963-1966)
produced one of the greatest visual - musical works till then. 11 All the above
artists worked with certain technical constraints but their artistic vision was so
strong that they were able to produce works that are still worth mentioning
and seeing.
In the digital era the means are much more powerful and the synthesis
of different types of arts seems easier and much more challenging.
Audiovisual fusion makes more sense than ever, since music and visuals are
united by not only the receiver but the artist as well in the sense that they are
made out of the same stuff which is no other than bits and pieces of
information. This might possibly in the future more concisely than ever
embody the synesthetic experience and lead to a fulfilled, coherent syntheses
of the arts.
This way, synesthesia is brought to the foreground and integrated art
that breaks boundaries between the conventional forms of art can more than
ever be achieved. Visual artists working together with sound designers
constantly create art installations that head towards this direction. Impressive
works stem from the same human vision to create pieces of art in which the
different forms fuse and merge into one and despite the fact that the use of
digital media in this context is only recent, there are signs that future art
passes through this union of the senses. Time will tell if the synesthetic
visions of the early pioneers will be finally fully embodied, helped by the
strengths of digital media.
Crucial views on perception in order to understand synesthesia

According to anthropological studies the so well known five-senses
system is culturally dependent. This classification is considered to be panacea
in the western world and people usually deal with the senses as five separate
channels. To take the division in five senses for granted is to a certain extent
misleading, as in fact there are whole cultures like certain tribes of Indians
and also contemporary neurologists and physicians contradicting this widely
established notion. Recent studies in the fields of biology, anthropology and
cultural history have shown that the boundaries between senses presumed by

11 Brougher, Strick, Wiseman, Zilczer, Mattis, 2005

8

---

those inhabiting the western world are not as clear as we tend to believe, but
dynamic and decided by means of established cultures.12
The French philosopher Maurice Merlau - Ponty in his works describes
the body as the source of perceiving and knowing our surroundings. For him
the senses are united in the following way, quoting from his book
"Phenomenology of perception" (1945, 1962 for the English edition) "The
intersensory object is to the visual object what the visual object is to the
monocular images of double vision, and the senses interact in perception as
the two eyes collaborate in vision. The sight of sounds or the hearing of
colours come about in the same way as the unity of the gaze through the two
eyes: in so far as my body is, not a collection of adjacent organs, but a
synergic system, all the functions of which are exercised and linked together
in the general action of being in the world, in so far as it is the congealed face
of existence."13 This same concept regarding perception brought to the
western thought via Merlau-Ponty can also be found in ancient Buddhist and
Hindu books, in which all the senses are described as stemming from the
same point, the same unity.
Childhood, synesthesia and education (Were we all born
synesthetes? Can we return to that point?)

"...The child is indifferent to the practical meanings since he looks at
everything with fresh eyes, and he still has the natural ability to absorb the
thing as such. Only later does the child by many, often sad, experiences,
slowly learn about the practical meanings. Without exception, in each child's
drawing the inner sound of the subject is revealed automatically. Adults,
especially teachers, try to force the practical meaning upon the child..."14
Researchers claim that all the newborns perceive their sensory
impressions in unite, as an unbreakable whole. No matter how hard is at
times for adults to think this way, babies do not just smell, taste or see, they
actually perceive their environment in this special way that is really close to
synesthesia and is usually called "sensory primordial soup". Recent scientific
evidence has proven that senses start to separate five or six months after the
birth. In this sense, one could state that we are all born synesthetes, even
though there are differences between this sort of childhood synesthesia and
adult synesthesia, the main one being that grown-up synesthetes still perceive
the world through separated senses even if those interact with each other.15
To be able to deeper understand the fascinating phenomenon of
synesthesia we should develop educational systems that are not that focused
on the enhancement of cognitive thought but also leave room for the senses
to develop. This reminds us of the emphasis put on by Rousseau during the
18th century on the importance of learning through experience, play and
experimentation. In this sense intuition could come to the foreground and
cognitive skills could still be taught but not to the extent that this is happening
in our times. Being open to uncommon experiences, building up a new system

12 Van Campen, 2008, 109-110
13 Merlau - Ponty, 1962, 271-272
14 Kandinsky, Marc eds, 1972, 174
15 Van Campen, 2008, 31-34

9

---

of thinking through education and last but not least believing in synesthesia
seem to be the keys in order to become able to feel it and understand it.




10

---

Document Outline

• Preface
• Synesthesia from a clinical viewpoint
• Traces of synesthesia in artistic endeavors of the past
• From the early visual masters to the digital art era: different means, similar visions
• Crucial views on perception in order to understand synesthesia
• Childhood, synesthesia and education (Were we all born synesthetes? Can we return to that point?)
• Selected bibliography and online resources
• Addendum

---