ISU Electrical and Computer Engineering Archives

Synesthetic music experience communicator

Hill II, Lewis Charles (2006) Synesthetic music experience communicator. PhD thesis, Iowa State University.

Full text available as:

PDF - Requires Adobe Acrobat Reader or other PDF viewer.


The Synesthetic Music Experience Communicator combines research in the areas of human computer interaction, music technology, and human perception to illustrate the experience of synesthesian mental imagery in response to musical sounds. Synesthesian musicians have reported positive benefits from their augmented awareness of sound in the areas of pitch identification, memorization, composition, and improvisation. This dissertation attempts to communicate both the experience and performance benefits associated with this cognitive phenomenon. Several virtual worlds have been developed to explore group education, rehearsal, and the artistic transformation of live performances into informative and entertaining visual presentations. Initial inspirations, background research, development stages, iterations, user evaluation, and future directions are discussed. Two virtual reality prototype systems are overviewed. The first demonstrates chromesthetic translations of real-time Musical Instrument Digital Interface (MIDI) events. The second demonstrates real-time transformations of multi-band Fast Fourier Transform processed audio into visual scenery. Observations and feedback about the initial prototype systems are summarized. Three systems are proposed which expand the initial prototypes and demonstrate concepts for computer augmented ensemble rehearsals. The Synesthetic Visualizer modules combine real-time audio and MIDI data to demonstrate synesthesia and photism-like imagery. The Computer Augmented Percussion Trainer and Small Ensemble Trainer are proposed to augment musical practices and rehearsals by providing real-time displays of ensemble members’ performances and displaying transformations between instrument fingerings. An introductory user study is conducted to determine which aspects of music are best communicated by the visual displays and to evaluate the potential benefits of this synesthetic approach. The user study asks participants to consider this research in relationship to existing music visualization and education methods. Exhibitions and publication efforts are reviewed. The user study, observations, and exhibitions serve to validate the core hypothesis of this research. The dissertation concludes by proposing an intelligent interactive synesthetic software agent to facilitate profile driven multimedia content creation.

EPrint Type:Thesis (PhD)
Additional Information:Dissertation Committee: James Oliver (Major Professor), Kris Bryden, Barry Larkin, Chiu-Shui Chan, Constance Hargrave, Gurpur Prabhu
Uncontrolled Keywords:synesthesia music virtual reality human computer interaction graphics visualization user study dissertation
Subjects:Computer Engineering > SOFTWARE SYSTEMS > Software Engineering
ID Code:290
Identification Number:Identification Number UNSPECIFIED
Deposited By:Lewis Charles Hill II
Deposited On:30 November 2006

Archive Staff Only: edit this record