top of page

Haptic Electronic Audio Research into Musical Experience (HEAR-ME)

HEAR-ME: A Multisensory Investigation into Enhanced Music Perception of Cochlear Implant Users


Dr Lauren Hayes (PI), assistant professor, Arts, Media & Engineering, ASU

Dr Xin Luo (PI), assistant professor, Department of Speech & Hearing Science, ASU

Gabriella Isaac, graduate student, MA Digital Culture, AME, ASU

Dominic Bonelli, student, BS Electrical Engineering, ASU

Slater Olsen, student, BA Digital Culture, AME, ASU

Kiana Gerard, ARES student, SHS, ASU

Firas Haddadin, student, SHS, ASU

Katie Pulling, research specialist, SHS, ASU

Marije Baalman, artist & researcher, ex-STEIM, Amsterdam


- Cochlear Implant (CI) is a medical device that partially restores hearing sensation to profoundly deaf people by electrically stimulating the surviving auditory neurons.

- CI users have trouble with speech recognition in noisy environments and various

aspects of music perception

- F0 is not faithfully encoded by CIs which presents problems in music perception

- Some CI users have residual low frequency acoustic hearing up to ~1000 Hz

- When a hearing aid used to provide low frequency acoustic cues for CI users, their speech recognition in noise and music perception greatly improve

- With no residual hearing, a tactile aid could help

- Evelyn Glennie:

Hearing is basically a specialized form of touch. Sound is simply vibrating air which the ear picks up and converts to electrical signals, which are then interpreted by the brain. The sense of hearing is not the only sense that can do this, touch can do this too.


- Vibration and vibrotactile feedback to accompany music and sonic cues can enhance the experience for performers and listeners alike

- performer-instrument relationship for augmented/digital instruments

Isaac, G, Hayes, L and Ingalls, T. 2017. Cross-Modal Terrains: Navigating Sonic Space through Haptic Feedback. In Proceedings of the International Conference on New Interfaces for Musical Expression. Copenhagen

Hayes, L. 2012. Performing Articulation and Expression through a Haptic Interface. In Proceedings of the 2012 International Computer Music Conference (pp. 400-403).

Hayes, L. 2011. Vibrotactile Feedback-Assisted Performance. In Proceedings of the International Conference on New Interfaces for Musical Expression (pp. 72-75). Oslo, Norway.

Hayes, L. 2013. Haptic Augmentation of the Hybrid Piano. Contemporary Music Review, 32(5). 499-509. London: Taylor & Francis.

- networked communication between musicians; cueing; conduction

Hayes, L and Michalakos, C. 2012. Imposing a Networked Vibrotactile Communication System for Improvisational Suggestion. Organised Sound, 17(1). 36-44. Cambridge University Press.

- audio-haptic compositions

Hayes, L. 2015. Skin Music (2012): an Audio-Haptic Composition for Ears and Body. In Proceedings of the 2015 ACM SIGCHI Conference on Creativity and Cognition (pp. 359-360). ACM.


- Vibrotactile thresholds tested at different sites

- Vibrotactile perception adaptation

- Frequency interval detection

- Melodic contour detection, improved with training

- worse than normal hearing, comparable to CI hearing

- approx. 7% improvement on melodic contour with haptic feedback and CI simulation

bottom of page