Pen-based Musical Interaction

The ease-of-learning is an important factor deciding upon the success of novel musical interfaces. However, reducing the complexity of the musical interaction runs the risk of a too restricted expressive power of the instrument. Traditional instruments draw their expressivity to a large extent from the fine-grained control possibilities, which they offer to the performer. Traditional instruments require that performes practice the instrument for long periods until they have reached this level of fine-grained control. In contrast, pen-based musical interfaces enable the performer to use the motor skills already learnt for writing. This empowers the user to reach a high level of control more quickly.

We are currently developing a novel pen-based musical interface for live performance which uses digital pens on a tabletop display. We use the Anoto digital pen technology. The Anoto pen is a ballpen with a built-in camera near the pen tip. By detecting and decoding a nearly invisible dot pattern, which is printed on the surface the pen is used on, the pen decodes its current position. In contrast to other approaches for capturing pen input, the Anoto technology provides for input on large surfaces with a very high resolution (about 850 dpi). In addition to the sound which is generated in real-time by a synthesizer, the tabletop display provides visual feedback to the performer.

Our main interaction concepts are as follows:

• The performer generates sounds by spanning virtual chords between two pens on the table surface.
• The instrument provides for collaborative use, letting multiple performers span chords between their personal pens.
• In addition to the tabletop display, the performer can use interactive paper surfaces for creating sounds. While using paper as an input medium does not provide for co-located feedback, paper is mobile and can be flexibly arranged in the physical space depending on the needs of the performer. As a consequence, interaction with the instrument is not restricted to the flat table surface, but extends to the 3D physical space.