Why the laptop torture sesh? Well, I’m back on the wildly careening Fast! Fourier! Transform! bandwagon (the exclamation marks are for extra excitement). FFTs are a way to break down a raw chunk of sound (for example, an MP3) into all of its individual frequencies. So for example, if you had a recording of a pure C chord, running an FFT on it would show that it had C, E and G tones in it.

It looks a lot more interesting when there is more stuff going on than that…

Click here to view the embedded video.

…but than it is also harder to see what is happening.

I’m excited about this again because I came across this page, in which they have an algorithm that can calculate the spectrograms waaaay faster than my old crummy one. I haven’t had time to pick apart why that is yet, but it’ll for SURE result in some sexy animations. Otherwise I want all my money back.

Click here to view the embedded video.

(widescreen here)

The motivation here was to get a program to work out when notes were being hit, just by looking at the volume. The idea is that when the volume suddenly increases, a note is being played. If I can work out exactly when all the notes are being hit in a recording, then I can map the analyzed score (that is, the raw notes from the sheet music) onto the timings, and make a neat animation.

The video shows a moving plot of the volume. Well sort of. It’s the absolute value (i.e., all negative values are made positive) of the amplitude of the waveform. You know how when you look at a speaker cone real close you can see it vibrating? The amplitude tells you how how far it is moving. If the amplitude is large, it is compressing a lot of air, which sounds loud. If the amplitude is very small, the speaker is barely moving, and we can hardly hear anything. This doesn’t exactly correspond to what we hear as volume, because there are lots of psychological effects which affect our perception (for example, hearing low and high pitches differently).

As you can see, it works pretty well when not much is happening, like during the first few minutes. It is interesting to see how the sound slowly drops off after each note is struck. When things get more hectic it gets way harder to separate the notes, since the sound level is continuously fairly high. Quieter notes get lost in the sustains from previous ones.

Imagine hitting a low C on the piano and then immediately hitting a high one, much more quietly. We could probably hear both because they have such different frequencies, despite the volumes. But if instead of hitting the high C you quietly hit the low one again, it would be really hard to hear. That’s pretty much what is happening here. We are not using any of the pitch information.

There is a way to do that, but it is much trickier to program.

I did have some time to play around a bit more with the score processing. Here’s my standard guinea-pig type piece, Beethoven Op. 111:

Click here to view the embedded video.

Do you love the glorious widescreen? Oh wait… the embedded player doesn’t work with widescreen yet. Well, if you (like me) are hot for 16:9 you can watch the full thing here. HOWEVER. It still won’t have any audio. Why not? It turns out that synchronizing the audio is actually the real crux of this score analysis malarkey.

You see, the notes in the video are a literal transcription of the score (extracted from MusicXML versions of MIDI files), but no-one ever plays a literal transcription of the score. The tempos vary all the time. So making the audio match the notes is a much more difficult problem than getting the notes themselves. I have about five different ideas for getting this to work, but all of them are several day long programming sessions.

Still, the video looks kind of pretty, right?

I only just came across this site, even though all the drama of it reappearing happened half a year ago. I didn’t notice it back then because my visits into sheetmusicland only occur rarely. In fact I only go in search of scores for one of about, errr, two reasons:

• I’ve decided to try and play the piano again.
• I am obsessing about a particular passage in a piece of music, and want to know the key, or time-signature, or which instrument is making that funny noise (like that weird buzzing in Shostakovich 4)

Well hang onto your three-cornered hats, because today it was for another reason. The reason of animation. I had this totally awesome idea for animating a piece of classical music in which you’d have an orchestra layout:

And then the different sections would light up when they were playing. I thought it’d be interesting to get a visual feel for how the melodies are getting passed around, like you do with a live performance. Of course, in order to do this you need a copy of the score (or an insanely good ear and lots of time).

Instead of doing it manually — that is, looking at the score and by hand turning that into frames of animation — I want to automate it, at least a little bit. In the most basic version you don’t even need to try and work out which exact notes are being played. All you need to do is draw a line (or rather, a rectangular box) vertically across the clef and see how many black pixels there are inside. When a note is being played it will be darker than average. If you do this for the whole score you should have a pretty good indication of when notes are being played by each instrument.

Of course, things like this are always way easier to describe then to actually accomplish, but I’m gonna give it a shot when I get a bit of free time that isn;t spent playing on teh internets.

This thesis reading came about due to me taking another poke at my previous attempts at music visualization (or more fancy-schmancily: MIR – Music Information Retrieval). Like so many research related items, although the general concept is fairly well-defined and straightforward the devil is definitely in the heapings of details.

There are all kinds of issues that rapidly start cropping up after the basics have been sort of sorted out.

For example, humans are pretty good at picking a melody. It’s hard to tell a computer to do that. As soon as you have more than one note being played at a time it’s a very non-trivial problem to work out which notes belong to the melody and which to the accompaniment. Particularly if you have, for example, a melody which switches from high to low notes, or passes between instruments, or has sharp changes in its dynamics.

In short, this means it is pretty easy to extract the melody from a monophonic performance of “Mary Had a Little Lamb”, but nearly impossible to extract it from anything else. At least automatically.

However, this brings up an interesting point. Can people even accurately identify which notes form the melody in a piece of music? Here’s a quote from the above thesis which alludes to what I’m getting at, which in this case is talking about automatically identifying musical instruments:

Martin [41] examines the classification of isolated samples from 37 instruments using
hand-picked features as inputs to a quadratic classifier … best-case accuracy is reported at 71.6%.

Martin [41] and Brown [51] also cite human performance for instrument identification tasks. Brown notes that human performance on isolated tones for her two-instrument identification task is comparable to her system’s performance. Martin finds that humans score much worse than his system (i.e., 50-67% versus 71.6%) on the instrument identification task and with comparable accuracy for instrument family identification.

Some MIR goals might not actually be possible. This is possibly true for 100% accurate two-instrument identification, but I bet it’s definitely true for melody extraction. In fact, the more I consider this problem the more I realize that it would be frickin’ impossible to say which of the notes in a piece from part of the melody.

It’s like when you try whistling a famous piece. Whistling should be a perfect example of melody recall, right? We whistle the melodies we have extracted from music, it’s our best attempt at melody identification. Well, anyway, when I try to do this the first part usually goes pretty well, you can sometimes make it all the way through the exposition without getting too confused. However, as soon as you hit the development section you’re basically screwed. You try to whistle three parts at once and it falls apart like a… a… buttercup.

At least, that’s what happens when I try. Now I’ll probably get smarmy eMails from music majors who can whistle every orchestral part of Beethoven’s Ninth all the way through. At once.

One nice feature of having a decent collection of blog posts up and active is that every now and then someone leaves you a nice comment on a piece you had pretty much forgotten about. Today this was a comment from chaika, who commented on this post back from the depths of last year in which I really, really wanted a piece of software which would automatically display the tonality of a piece of music as it progresses.

Miss/Mister commenter provided me an excellent lead on the subject: Elaine Chew at USC has published a bunch of papers which attempt to do exactly that. Not only are her papers relevant, but via her references I can now work out all the other important writings on the subject. Awesome.

So far it seems that her method for determining tonality is based on a spiral:

In which each point on the spiral is a major fifth higher than the last (and so each point vertically above ends up being a third higher, which is why those chords look like triangles, because it’s a point connected to a spiral-neighbor and a vertical-neighbor) That spiral looks kinda complicated, I know, and I’m feeling the pain a bit because I only know the most basic music theory. However, I’m fairly determined to get to grips with the ideas in this paper, and it’s actually a rather interesting (and effective) way to learn the theory for me: backward from the math.

installation proposal for artbots 2008 from ranjit on Vimeo.

It’s a robot which translates vocal patterns into musical sounds which it performs on a modified electric guitar and various pieces of percussion equipment. In the video linked above you can see how it interprets a brief scene from Citizen Kane. This is exactly the kind of thing that Janacek was into, apparently; trying to mimic the cadences of the human voice using instruments. Except without the electric guitar and microchips. Probably.

Additionally, reader Yvonne forwarded me a recent story about an attempt to geometrically visualize the structure of musical works. As regular readers are probably sick to death of hearing, I’m totally in love with the concept of theoretically-well-grounded visualization techniques. It’s a fairly holy grail. This press piece is tantalizingly light on exact details, but one of the most interesting tidbits for me was:

To some extent, we can represent the history of music as a long process of exploring different symmetries and different geometries.

Which I’m a tad suspicious of due to the “certain extent”, and might just mean that they got some pretty pictures which sort of look similar, but could be really neat. If I feel brave enough I’ll try and dig up the actual paper sometime this week and wade through acres of musical and mathematical theory which I do not understand. I can pretend to though. I’m getting really good at that after four years in grad school.

Click here to view the embedded video.

It’s at least passably interesting, isn’t it? Mostly I’m just relishing and wallowing gleefully in a minimal amount of proudness for working out how to animate a bunch of pictures with a line down the middle…

This shows the dominant sound frequencies present in the music, over a period of time. These frequencies correspond in a not completely linear way to the notes which are being played. This lets you see the shapes of the melodies as the piece progresses (from left to right in the spectrogram). I think with cleverer coloring, and animation, this will become much more apparent. Still, not bad for a first, errr, second try!

The recording I am using is from the wikimedia page on Beethoven. The software I am using to create it is all open-source: sox, python, numpy, pil.

At the moment each video frame looks like this:

See what I mean? In the next few days I’ll play around with graphics routines to try and generate a proper spectrogram. Then we’ll be in a bit more business.