I recently attended a User Experience conference which had many interesting presentations on all kinds of design and HCI tid bits. There was one particularly different talk that really caught my attention- experiments using biofeedback to gauge user reactions to Half Life 2, thus opening a whole array of thoughts about using a virtual reality world to address individual emotions (in this particular case, fear). For me, I was particularly interested in thinking about how this idea could be altered and used to measure human reaction to other mediums, music at the forefront of my thinking of course. Wouldn’t it be astounding to see the results of an individual’s emotional response to a particularly feeling piece of music?

Essentially the researchers, Erik Champion and Andrew Dekker, were strapping their test subjects to a pulse monitor (which measured heart rate and sweat on the finger tips) in an attempt to gauge the level of fear that the subject was currently experiencing whilst playing, analyzing these findings, then fiddling around with the game controls to personalize the game’s patterns.

It is interesting to note at this point that only 3 people in the conference room had played Half Life 2 when questioned- myself and another girl, and a guy. Not soon after this was discovered there appeared a rather amusing tweet on the conference feed which said excitedly: “Funny fact, three people in the audience had played HL2 – of which two were girls!”  Anyway, I digress

I myself have played a limited amount of Half Life 2, and have not touched the particular level that they used for the experiments- Ravenholm. But I have been assured that the experience is indeed a frightening one. Digressing again, but apparently humans are particularly fearful of zombies because of their close resemblance to the human form, and because they seem particularly attuned to ‘hunting you down’.

So, imagine you are playing HL2. You are running around the darkened nighttime level, a little bit apprehensive of what might come around the corner, when AGH! A wild zombie appears! *shoot shoot shoot* Well that was easy, you say to yourself. Nothing particularly frightening about your average, run-of-the-mill zombie. On you run. Suddenly, out of nowhere, some hellish headcrab infested zombie jumps out at you. You feel your heart rate increase as you struggle to get into a good position to dissemble this thing. Your palms are a little sweaty as you slip to get a firmer hold on the mouse. The BAMMMMM more of these creatures are appearing everywhere, your becoming overrun, they’re in the windows, coming down from the roof, everytime you shoot one a dozen more appear. Now you’re getting really freaked out.

And that’s how these guys worked their tests. If the subject didn’t fear a particular creature, then no more of those creatures would appear. If the subject freaked out over crabheadzombies (or whatever they are) then they’d generously send out a whole army of them. And when you got used to that, they alter the landscape- removing all the walls of the buildings so you could see everything and anything that was coming your way. And if that wasn’t frightening enough for you, they’d use the built in color/saturation/lighting altering module to change the entire atmosphere of the game. Shifting from calm color, to fearful red and black. Your own delightful level attuned to your most feared HL2 experiences, what could be more pleasant?

The results of their tests indicated (I guess not that surprisingly) that most preferred playing the level that was adapted to their personal reactions, rather than the standard level (they didn’t know which was which at the time). To be honest I’m not too sure I’d even want to play Ravenholm, let alone their demented version.

They did touch on the effect that the music has on the emotional response of the player, and their efforts of matching music to the game and user, but they definitely didn’t go into enough detail with it. I think it would have been fascinating to see if there was any difference in the levels of fear if they played with a blank soundtrack, as opposed to the normal accompaniment, and then perhaps with a soundtrack that matched the levels of intensity experienced throughout the level.

I’d really like to investigate more into this area, and I really do like the idea of games being more attuned to the individual. Interesting, Interesting, Interesting

‘The Shape of a Song’ is a piece of software, and its produced collection of diagrams/illustrations, that attempts to visually represent the music from the MIDI files that it is given as a sequence of translucent arches. Each of these arches connects two repeated passages of a composition, so if the first verse is compositionally repeated in the third verse, these two will be connected via an arch. By using these repeated passages as ‘signposts’, the diagram attempts to “illustrate the deep structure of the composition”. The more complex the composition, the more intricate the diagram.

The illustrations are quite nice to look at, pieces of art in themselves. But it’s particularly interesting to examine the diagrams of songs you are familiar with, and attempting to determine how the translucent arches are depicting a favourite melody.

The software itself, for my geeks, is written entirely in Java (woo), with an applet to handle the user interface and the visualization itself. A servlet handles the reading of the MIDI files from the web. The software uses MIDI files, instead of MP3, because it contains a description of the notes in the musical score that is simple and flexible enough to be useful for analysis, and because MIDI files can be divided into several tracks (each representing different instruments or voices) which allows the software to analyse the unique patterns of each instrument.

Here you can find the current 648 piece repertoire of ‘The Shape of a Song’, which means there are plenty of recognizable pieces already there for you to examine! From Chopin to Ben Folds- Go have a looksie!

'Battle' from Final Fantasy 7, Nobuo Uematsu

Day Three, Paper Session One.

First paper I saw in the morning session was the ‘LUSH: Organic Eco-Music System’ which attempted to thwart the general sequencer operation, which plays the assigned notes when they are hit by a bar traveling from left to right. It was a very pretty system, comprised of “flocks” and “boids” of circular notes, that flew and swam around the screen. Users could create their own barricading bar, which would act in the traditional manner- playing the notes as they hit the bar. The effect was, again, very pretty, creating a unique playing of a musical piece with the sense of the original, yet altered enough to be called your own. The system was described as a method of “hybrid music making through user interaction”, which I think was a very accurate way to describe it. I would have loved to have a go at it, and see if something as recognizable as The Imperial March (John Williams) would still have the same presence, yet sound quite different.

Next up was the ‘Twinkleball’. Yes, it was Japanese. It seemed a rather fun-looking ‘wireless musical interface for embodied sound media’ disguised as an everyday plastic ball that one would generally find in k-mart, with additional features such as LED lights! and sound generation! and light senses! This would make for a much more interesting sports game, however when questioned whether his Twinkleball could yet withstand the purpose that it’s name would suggest- the common ball-bounce,  the presenter replied somewhat shamefully that he had tried bouncing it multiple times. With the general result of broken Twinkleball.

Moving into Paper Session 2!

I’m going to mention this briefly. It is a rather large pet hate of mine, and I would generally tend to get carried away with it, but we’re short on time so I’m going to just mention it and move on. Rehersed presentations where you read from a piece of paper with lots of big words and even larger sentences are horrible. Do Not Do It.

</end>

The next presentation could not be further from the disappointment of the the one before! It was one of the best presented presentations (ha) that I have ever witnessed! Hats off to you Mr Collins-Ben-Folds-Wannabe! In my opinion the presentation was more interesting than the system that it was about. So what was so fantastic about it, you ask? Well, the title was “Contrary Motion: An Oppositional Interactive Music System”, and his first slide was simply text that read ‘Thank-you, any questions?’ from which he continued to read the entirety of his presentation backwards from end to start. The especially well executed part of this was how much he stayed in the feel of the presentation, constantly adding little statements like “as I was saying five minutes in the future…” It was just simply well done. The system itself basically allowed a computer to play in accompaniment with a musician- histogramming predictions and other nifty things to determine what a player may do next. Apparently for use in experimental music and composition.

Oooooooo! Now this was super cool! I can’t wait for this to be widely available! And, even more excitingly- this was being researched in our very own Wollongong! Ok, so, the title was “Images into Spatial Audio”, but really it was just extraordinarily awesome sound coming through epic speaker set-up system, that aimed to give the illusion of space and sound. Their research involved an image being “laid flat”, scaled, and then the listener is essentially placed in the middle of the image. Each pixel is then turned into a virtual speaker, giving an amazing encompassing sound. We were given a demo, but we had only the UTS speakers to hear it through in stereo. Even still I was blown away but the quality of the sound. I felt EXACTLY like I was in the middle of the sound, and that the sound existed in some form around me. It’s very hard to describe, but it was a rather fantastic sensation, and quite noticeably different to the normal audio sound. Think listening to music through normal headphones, and then switching to amazing Sennheisers. That level of difference in quality. The demo sound was kinda an Autobot train in the middle of the ocean.

This session was really excellent! The next presentation was also wonderfully presented! I got to chat with this particular American later too, and he was a really nice! His area of research was in creating musical instruments from a Mac trackpad. Yes, as I look down at my trackpad I get strange urges to play music with it too. You could do all the normal trackpad gestures- clicking, dragging etc, and it would analyze any points that were contacted, the relation of the single points to the whole gesture, the distance between points and the angles between the points, and then produce real-time sound as you interacted with the trackpad. What made it nice was that he had a really nicely designed interface to go with it- a black background with coloured points, links and angles. His future work is mapping dance to a similar system, so I’m looking forward to seeing how that progresses!

Break time!

*interval music plays*

And we’re back- Session 3!

Scorelight was quite a different interface compared to any that we’d seen yet. The presenter worked in physics, specializing in lasers, and Scorelight produced music as it traced objects around with a laser. I won’t go into too much detail with this, there are heaps of youtube videos that give a better demonstration, best to head over there.

Alright! Another one of my absolute favourites- Disky! A DIY Rotational Interface with Inherent Dynamics! aka- a mini turntable that everyone can build themselves out of discarded technology parts. The developer of Disky is extraordinarily passionate about reusing the countless discarded parts that find their way into junk yards when their technology becomes obsolete. Disky functions almost exactly the same asa turntable, and I’ve got to say, it’s EXTRAORDINARILY fun to play with! I made friends with Karl Yerkes, Disky’s developer, which meant I got plenty of opportunity to play with this cool little piece of tech toy. I probably stood there for a good 30mins, “scratching” with Disky, whilst talking to Karl. I want one. For anyone who is interested in building one for me, umm I mean themselves, you can find a complete list of parts and instructions here: http://karlyerkes.com/disky/

From here we moved into robots!  First up was a robot who could play the saxophone, and then a course from “CalArts” (Californian Institute of the Arts) that undertakes a collaboration between music and theatre design students to design and build their own musical robot for the ‘robot orchestra’ , meanwhile tackling the pedagogical paradigm for musical robots, and finally a robot musician who can interact with a human partner through initiative exhange.

The last sessions of the day involved mobile development, yes mostly iPhone. There was a nifty musical collaboration ballgame called Soundbounce, and some fairly interesting research into pressure-input which aimed to solve the disappointing lack of haptic dynamic feedback from the iPhone piano apps. They had this pretty nifty piece of development work that involved two iPhones back-to-back and pressure-based multi-touch, which essentially meant that they could control the object through both iPhones.

In the evening there were two sets of concerts where we finally got to see some interfaces in action! I must be frank here and say that I was at a rather loss as to what was happening during this concert. An introduction, a blurb in a program, a speech at the end of each demonstration! Any of these would have increased the quality and understanding of these performances dramatically, as I am sure I’m not the only one who went there without previously researching what we were going to see. Also, length. All three were FAR longer than was necessary. Anyway, one was a trumpet player and cello-type instrument player (who also sang), both of whom (I THINK) were being distorted remotely by the person who was in the on-screen live stream with lots of audio and mixing boards. Fairly nifty, except I liked the actual playing of the cello-type instrument, and the distortion ruined it. Next was a two people behind lecturns with Mac computers, one with a glove, who pointed at the audience, nodded at each other, and seemed to control a very loud distorted amount of audio clips (most of them radio news readings) somehow *shrugs*  Finally, there was a women dressed in white, decked with some sort of attachments, who repeatedly sang the words “What does a body know”, whilst moving her equipment-gloved hands, and creating some sort recorded feedback noise that sounded like whatever she just sang.

Then we all walked over to the ABC studios (awesome!) for the main concerts. I was usher and programme hander-outer. I was only able to stay for the first couple of performances, but each of these were wonderfully inspiring. The first performance involved a beautiful cellist performing baroque pieces, who was connected to a heart/brain activity monitoring system, which produced its own sounds based on this activity. Interesting! Next up was a flutist whose flute generated more wind-type sounds. And then Andrew Johnston’s fascinating system, demonstrated through a performance of the piece ‘A Touching Dialogue’ (found on Andrew’s website!) for trombone and clarinet. Definitely go and watch the videos of this system in action! Basically, the performer can interact with a linked halo of spheres which are mapped to pitch and intensity, to which the interface responds with its own sound and unique reaction. Go watch *makes shooing motions*  http://andrewjohnston.net/research.php

The next performance was by the once-again entertaining Nick Collins-Ben-Folds-Wannabe. He was using his system to generate ‘instructions’ for piano improvisation, instructions such as “tap  your shoulders”, “play Chopin” and “sign F off” It was really most entertaining, and his skill in playing the piano was evident. However, I was shocked when he (in a very Ben Folds like fashion) used his leg/foot to play/slam the ABC centres STEINWAY AND SONS GRAND PIANO. Yes! I know!

With that we finish Day 3, and move into the final Day 4…

This day was a little less exciting interest wise, which was rather disappointing. Some interesting presentations that did arise however are as follows…

The keynote, Nicholas Collins (not Nick Collins) presented “By Hand: A Personal Pre-History of NIME” and it was really amazing to see how this pioneer in the field of NIME has developed and soldered his own “hacked hardware” musical instruments. He was also a rather wonderfully humorous character, which made for a far greater interest-factor in his presentation. He acquired a chuckle out of me with his statement during his explanation of why humans enjoy fiddling with their hands, and how they find such tasks instinctive- “Rosary Beads, knitting, sewing, soldering… they’re all the same”  Maybe you had to be there. This one was also quite chuckle worthy, and came from him quoting a friend who was an audio engineer: “the professionalism of a band is directly proportional to the number of wireless mics they request for a performance”

We all just adore the reliableness of wireless though? Surely? No…? Oh.

There was a a Panel Discussion during lunch that discussed ‘The Future of New Electronic Musical Interfaces’, but lets leave that till last.

The final two papers that I’ll mention were both rather cool in the artistic sense, the cool sense. The first was a a type of musical performance that generated this distorted sound and image with a lighter. Veryyyyyyyyyyyyy cool. The presenter and his obvious compulsive tendency to flicking his lighter on and off was slightly disconcerting however. Apparently he’s in a band title ‘Crisis’ but I couldn’t find any videos of him performing the lighter show, dang it.

Finally! There was this really awesome Virtual OSC Controller from Italy that was a virtual reality system for DJing! You can assign shapes sound clips, and then touch them to activate the sound, then mix others in and out and all kinds of rather funky things! The video they showed sounded pretty decent as a set as well! I guess Italy produces some pretty good things in terms of techno music (think Technoboy and Davide Sonar)

And that was it. NIME ended… But what a wonderful week! I can honestly say that I thoroughly enjoyed it. This is one of the only conferences which I have found constantly engaging, and interesting! Cliched, but I can also honestly say that I learnt alot. I really want to go next year (to Oslo!), and join this NIME community <3

I’ll finish with a few ideas from the panel discussion, since this was really the most thought-provoking session for me.

The first speaker began by discussing how wonderful it would be to have access to the entire history of music- “We do, it’s called The Pirate Bay” interrupts Nicholas Collins. Discussion then progressed into how our NIMEs fit into this history of music (where their place is, if they have one), and how resistant musicians are in terms of the new- they like things that they know work musically, they don’t like our new interfaces because they are new and different.

“The future is getting silly” muses Nicholas Collins. The innovations and creations that a constantly being presented in some many varied fields today just seem so extraordinarily weird that its difficult to conceive them in any useful sense. But it is this that is the Avant-garde, the challenge to society, the demonstration of the new pushing the way we live.

I queried to the Panel what they answered to those that perceive what they, and indeed everyone at NIME, do as a waste of time. I explained that coming from a very practical-thinking IT base I constantly ran into barriers when I attempted to explain NIME. “But why?” they would constantly asked me “What purpose does it serve? What problem does it solve?” My dear IT people can understand improving an algorithm, or creating a program that directly improves a system, but with a NIME they couldn’t quite understand what functional base that it serves society.

“These people solve problems, improve on what already exists. We designers create”

I received the perfect answer. As cliched as it may be, sayings such as “music is the spice of life” are reality. Humans would live dry existences without such artistic modes of expression, we need them to survive, to breathe. We need to be able to express ourselves. An earlier presenter claimed that neanderthals could not create art- we survive, they didn’t. Whether or not this claim has any basis in fact, I think it still gives food for thought.

“We designers create“

This week is the exciting week that Sydney is holding the international annual conference of NIME (New Interfaces for Musical Expression – http://www.educ.dab.uts.edu.au/nime/). The most exciting part of this for me is that UTS is hosting it and I am a volunteer, which means I get to attend and help out at the entire conference. The conference itself is basically a week long demonstration of work in the field of new interfaces for musical expression (as you may have guessed by the name), with paper sessions during the day, and concerts at night. But what exactly is a NIME I hear you ask? Good question. And the answer is really whatever you want to be a NIME. Each year they seem to vary, and get more interesting, and more “out there”, than the year before. Some examples would be the well-known iPhone Ocarina, or the Reactable which Bjork has used during concerts, both of which were first presented at a NIME conference. Other, less well known, examples include a Tooka (2-person flute like instrument) or UTS’ own Andrew Johnston and his Partial Reflections work. In essence, I guess you could say that a NIME is a paring of music and technology, an interface that enables a new connection between these two fields and a representation of the link between sound and image.

The papers from each year are all online, so if you interested (as you should be) you can simply search NIME in Google, and find all the papers dating back from when the conference first began in 2001.

We’re now at the end of Day 2, and I tell you what, so far it has been fascinating for the NIME Newbie that is me. The keynote last night- Stelarc, was such a supremely unique individual that it is rather difficult to describe just how interesting that one hour of a presentation was. I am going to attempt to do my best in another post in the coming days. I am trying to get a hold of his presentation slides, which will make my eventual description that much more detailed and accurate than from my memory. For a little taster, let me just tell you that he will soon let a tiny Scorpian robot crawl into his mouth, that he has grown an ear on his arm, and that I am fairly confident that he will eventually be a cyborg. Did I mention that he was insane? Insane in the sense that he was quite awesome, but also in the original definition of the word. Throughout the presentation that was the only word I seemed to be able to stutter to my fellow NIME-attendee who was sitting next to me. And I honestly can not think of a better description than that for Cyborg Stelarc and his rather epic laugh.

Anyway. Much more to come about him.

Today was all Paper Sessions, and there were some pretty interesting things… Let’s see…

The Neurohedron! A non-linear sequencer interface, by a Mr T3DB0T. This thing was pretty (you can see some videos here http://log.liminastudio.com/itp/neurohedron), and what amazed me the most was the musical melodies that it created. What it sounds like is completely unpredictable and randomised each time- creating new “polyrhythms” dependent on randomized weights, although you can control the type of scale it uses through the interface. I got to speak to Mr T3DB0T, with his lurid red hair, during the break, and we started up a rather diverting conversation involving the pretty lights of the Neurohedron and the benefits of it using electroluminescent panels over LEDs to create them, as well as the wonderfully challenging world of programming in Pure Data (a rather excellently undocumented open source visual programming language that both he and I have used for our NIME projects). The Neurohedron is apparently currently in Manly, so check it out if you can.

Next up on my interest list was a software called Freepad, which you can actually go and download, which I was fairly excited about until he ruined it all by saying that it currently only operates on Windows. You should have heard the snorts and gasps from the crowd. Perhaps now is a good opportunity to mention that during my first observations of the lecture theatre I counted 22 laptops in my immediate surroundings. I think 3, if that, were not Mac. Not a smart move Mr Freepad. He tried to defend himself from the crowd’s harsh snorts by saying it was all their developers fault, because he could only use Visual Studio. Way to shift the blame. ANYWAY. The software was actually pretty impressive. Basically, all you needed was a piece of paper, a pen, a webcam, and this software, and away you go! You can draw any shapes you want on the piece of paper, use the webcam/software to recognise the paper as the design space, and then assign each shape a midi sound. You can then “hit” each of the shapes, and have them make the assigned sound. Quite amusingly, the presenter mentioned that he was unable to use his hands to hit the shapes, as the software couldn’t distinguish between the paper and his rather white hands.I was personally wondering if it allowed multi-hits, so if you drew a piano, you could essentially “play” this drawn piano, and have the appropriate music sound. They ran out of time for questions though, unfortunately.

I’ll talk about this one quickly, but the Cuebert was quite a cool new mixing board for musical theatre that mixed a digital touchscreen interface with a physical/analogue board. The presenter explained that digital interfaces can remove the physical constraints of analogue boards, but that it can also create a whole lot of new usability issues. Their newly designed interface allowed for a whole bunch of new cue functionality and I was pleased to note that in the demo they were obviously using Les Miserables as their theatre production, as the names “Javert” and “Valjean” seemed to keep popping up all over the place.

Finally, I was completely intrigued by the talk “Creating Meaningful Melodies from Text Messages”. This research was attempting to create an add-on for phones, chat, email, skype etc that would interpret the intention of the message (whether it be happy/sad/a question) and produce a melodic piece that would alert the user of the message intent. This way, the user will know what the message is “about” without needing to immediately stop what they are doing to check it. Analysis is based on keywords, and whether these words are generally associated with being happy/sad, and  a question mark obviously indicating a question. They include chords for punctuation, and use the major scale for a happy message, the minor for sad. We were given a demo test, and it was interesting that most of the audience were able to pick the first “musical piece” as a question, the second as a sad message. They then presented the results of a longer test they conducted over a period of two weeks, using around 20 users, and musically interpreting around 500 of their messages. The results were quite fascinating. There was apparently no significant difference in results between those who had musical knowledge and those that did not, and apparently people are far more likely to leave a message that is a question, while immediately wanting to answer a happy/sad message. Some issues were briefly discussed in the question time, such as its inability to distinguish between a commonly “sad” word used with a positive intent and that the system could simply not understand the intent of the message, rather only interpret it based on its previously acquired keyword memory. Another question mentioned the world project that attempts to glean a “Global Feeling” via analyzing twitter etc, and whether the presenter thought his system would be of use. His answer was something along the lines of “Yeah, that’s interesting. Perhaps.”

I could go on to explain every paper that was presented today, but I shan’t  I’m sure there will be plenty more tomorrow that I’ll want to  talk about.

I think I’m going to become a NIME addict.

Alright! Here I am! I have been thinking of starting a blog for a while now, and after three full days of deliberating over a name, I have finally begun! The name ‘Maiestas’ is Latin- I liked its uniqueness and, well, it looks pretty.

I know my following is rather small at the moment (so thanks if you’re reading this!) I am hoping to get more into the swing of it soon, and hopefully get a few more followers as I do! Basically, this blog will be a place where I can mention and discuss anything that I come across that I find rather interesting, mostly regarding design, music, film and user experience, but most probably also including a whole lot of miscellaneous bits of divertingness.

And so, ad initium…