Classical Convert

Score Processing, Part III

Here is the latest product of my epic battle to nicely animate a score to music:

(widescreen here)

The motivation here was to get a program to work out when notes were being hit, just by looking at the volume. The idea is that when the volume suddenly increases, a note is being played. If I can work out exactly when all the notes are being hit in a recording, then I can map the analyzed score (that is, the raw notes from the sheet music) onto the timings, and make a neat animation.

The video shows a moving plot of the volume. Well sort of. It’s the absolute value (i.e., all negative values are made positive) of the amplitude of the waveform. You know how when you look at a speaker cone real close you can see it vibrating? The amplitude tells you how how far it is moving. If the amplitude is large, it is compressing a lot of air, which sounds loud. If the amplitude is very small, the speaker is barely moving, and we can hardly hear anything. This doesn’t exactly correspond to what we hear as volume, because there are lots of psychological effects which affect our perception (for example, hearing low and high pitches differently).

As you can see, it works pretty well when not much is happening, like during the first few minutes. It is interesting to see how the sound slowly drops off after each note is struck. When things get more hectic it gets way harder to separate the notes, since the sound level is continuously fairly high. Quieter notes get lost in the sustains from previous ones.

Imagine hitting a low C on the piano and then immediately hitting a high one, much more quietly. We could probably hear both because they have such different frequencies, despite the volumes. But if instead of hitting the high C you quietly hit the low one again, it would be really hard to hear. That’s pretty much what is happening here. We are not using any of the pitch information.

There is a way to do that, but it is much trickier to program.