Pen and Touchscreen Computing and Music in 2015

 (Maestoso: Handwriting-based music theory software. Screengrab from “Maestoso Prototype,” Laura Barreto, 1:15)

 

It seems like the last 12 months have seen great growth in the field of pen and touch computing, both for the computing industry in general and music technology in specific. The release of the first iPad in 2010 promised to launch widespread access to digital ink and handwriting for mainstream computer users, but the first few years of the iPad era demonstrated that finding a niche for handwriting apps was a slow process. Fast-forward five years, and a variety of software and hardware developments have seemingly re-energized the concept of digital handwriting, but I believe that the biggest development in the digital ink field is not a technical one, but rather a shift in the industry’s thinking that could lead to some very interesting developments for us in music. I met recently with a colleague in a large software company who has been watching the field of digital ink for many generations, and he said that the software and hardware industries hadn’t really fully committed to digital ink in the past because it wasn’t something in which they thought users were particularly interested. As a musician, educator and technologist, this point floored me– how many times have we looked at a file on the screen and wished we could just write directly on it rather than fumble for a way to comment, edit or annotate using the keyboard? I had taken for granted that the desire to write directly on an electronic document was universal, but my colleague’s observation was that the business world (still a major driver of the IT industry) preferred to edit documents by type. His suggestion was that type is “clean” (i.e., professional) and efficient for people to use and share. What his company was starting to realize, though, was that creative fields such as art, design, music, and education, wanted to write freely into a computer precisely because it was messy: the process of designing, creating, annotating and editing is much more about a fluid process than a final product. Where businesses are often more concerned with the professionalism of a final document, a musician is more concerned with personalizing a document to serve an internal purpose: creating a public purpose. This software company’s realization that education and creative/design fields had fundamentally different priorities in working with digital files was leading them to re-commit to designing software that let people use digital ink to collaborate, edit, communicate about and design digital work. Recently I’ve written about two examples of digital ink and handwriting already in the field: StaffPad and NotateMe. These two programs are each very exciting for what they demonstrate about the utility of touchscreen and digital ink for music composition and editing. If this shift in thinking is indicative of a larger industry trend (and I believe that it is), we will continue to see many more promising developments in the coming couple of years that will accelerate the development of software and hardware for digital inking. A couple of weeks ago, I attended the Workshop on the Impact of Pen and Touch Technology in Education, a symposium of educators, researchers, technology industry and computer scientists hosted by Microsoft. Viewing the research projects from around the world and across all computing platforms, I saw two major trends which I think have great promise for us as musicians.

 

The first headline was that document markup continues to be the largest use of digital ink. The idea of marking up a .PDF file on an iPad or iPhone is pretty unsurprising to us at this point, but the immediate future of document markup is in shifting from single-person to collaboration. Again, thinking about the arts, design and education fields, these users all share a high priority on collaboration, and having multiple people work on the same document simultaneously creates a tremendous capacity for rapid creation. In short, think of Google Docs with handwriting support (which, yes, can be done now with plugins, but is a bit limited). While highly speculative and far from release, a team from Cornell University demonstrated a system called RichReview which allows you to upload a .PDF document and share it with a group or team in order to mark it up. Their system also includes the ability to record audio annotations. Perhaps most surprisingly, the system inserts the marked-up comments into the document themselves by respacing all of the text to make room rather than just as a layer on top. It’s worth the two minutes to watch their demo video for a sense of how the system works. Using a system such as this, an ensemble could be sharing a common copy of a score, and everyone could edit or mark it up together– imagine the composition studio or rehearsal where the director can re-write the score on the fly and have it synced to all the musicians instantly.

 

The second headline to me was that handwriting recognition technology is extending beyond handwriting into symbols, drawings and non-standard languages such as music notation. The workshop featured multiple demonstrations of systems that decoded a user’s drawings or doodles and translated them into machine action. This is the process which currently powers handwriting-based notation systems such as StaffPad and NotateMe, but it seems obvious that we’ll see many more entrants into this space soon, and not just for notation software. I was particularly interested in a project from Texas A&M’s Sketch Recognition Lab called Maestoso. The research team of Paul Taele and Laura Barreto have built a music theory instruction program around handwritten notation recognition, noting that while notation programs are most useful for people who are already familiar with reading and writing music, beginning students will learn music theory and notation best by writing it. Akin to our beginning theory workbooks, this software automatically recognizes and evaluates the students’ written input and gives them the result of their work.

 

Neither of these headlines are “new” developments, but to my mind they show a renewed interest from industry and research in the use of digital ink for creative processes. We have seen great developments in this field recently, but they’ve been slow to push into the mainstream of computer users, even in a dedicated field such as music technology. For a handful of reasons, that seems to be changing quickly. Part of this is also simple market competition– where Apple dominated the touchscreen device world with iOS for the first three or four years, Microsoft’s entries with Windows 8 and the Surface and Google’s announcement that ChromeOS and touchscreen Chromebooks can run Android apps natively means that the software and hardware powering pen and touch will continue to build up steam. With software developers starting to look at the ways in which handwriting and digital ink serve the way that education, the arts, and creative fields use technology, that could mean big things for us in the next few months ahead.