In a few words, this post explains my desire that the HTML5 audio tag provide an API for the reading and writing of sound spectrum data of that tag, natively, in the Firefox web browser ( and, hopefully, all others). It is my hope that this would involve a Javascript API similar to, or better than, the current API provided by Adobe’s Flash technology. If you wish to know why I think this may be important, please skip this link and read on. Otherwise, here is a miniscule sample of the drawing capabilities of Javascript when provided with spectrum data:

Note: this link is no longer functional. sorry. hopefully you’ll read on.
http://modern-carpentry.com/workshop/html5/jsmusic

long version

Over the past few years, my favorite interactions with programming have been in trying to visualize music. I’ve made a number of efforts at doing so ^{[1] [2] [3.1415926] [4] [5]} ( Nota bene: You may only have one of these experiments open in your browser at one time, otherwise they will not work ) and yet I feel it hasn’t been gotten right yet, so I’ll keep trying.

It is a goal of mine to build instruments– tools of understanding and play. I am most motivated in this goal by my wish to understand the phenomena of sound and music, and also see it visually, in a natural way. In these pursuits I have found that an investigation of randomness, and its uses in graphical programming are important. ^[6]

I’ve always had a peculiar feeling that the random is what is special, it is what we must pay attention to. The random separates a normal bus-ride from one in which a kind stranger gives you a compliment. The random is a divide between a normal hurricane and Katrina. The random is the placement of a bullet just left of your heart, in your shoulder, rather than otherwise. It makes life interesting in that, for most people, the most happy and the most sad moments of their lives may be random ones, rather than moments they had thought out and planned to have. An un-expected phone call, the great harbinger of random, initiates unplanned actions regardless of whether it involved good news or bad. But I’d not like to get into a discussion of ‘what is random’, because that will take much too long. I will only say that the beautiful, the terrible, and the sublime often involve randomness.

One goal of many programmers is to make things look ‘natural’. Though there is much structure involved in natural processes, and many repeatable, non-random procedures for emulating nature, the best simulations of nature, or the real, are a combination of non-random and random processes. While there is no such thing as programmatic randomness ( the pursuit of programming randomness is a rather absurd one ^[7] ) there are many methods whereby we might approach randomness. Often these methods are enough to trick the eye, and end up providing us with pleasant things.

In my pursuits of programming randomness I have heavily relied upon Perlin noise, and am often pleased with the results. The Perlin noise method, however, is computationally expensive ^[8], and, as such, is restricted in its value for us in situations where large amounts of processing power are not on hand ( on the web, on a mobile phone, etc ). Because of this, I have been investigating different ways to simulate randomness in order to provide natural seeming and beautiful experiences.

One approach of mine has been to use the prime number data set as a way to simulate randomness, because of the fact that there is no formula to predict prime numbers along the continuum of 1 to infinity ^[9][10]. I have not yet completed these investigations, but they have informed my approach.

Another approach I have taken, with perhaps a bit more success, is to use sound data as a source of randomness. ^[11] I have found that a frame by frame analysis of sound and music provides us with a pleasant data set for our uses in the approach of randomness and naturalness– the real. An analysis of the distribution of frequencies in a sound at a particular point in time can easily be done, and, while such an analysis does not provide us with true randomness, its results often look random, and perhaps even natural.

Because I have found sound data so interesting, and also useful in my attempts to create natural motion and shapes, I have filed a bug with the Mozilla Firefox browser in regards to their audio tag ^[12][13]. The current HTML5 audio tag does not give us the capability to analyze sound on the fly. I am confident that, if this feature is regarded as important enough, it will be implemented, and I look forward to the day when I can begin using Javascript rather than Actionscript in my efforts to build instruments.

works cited

1:http://modern-carpentry.com/music/proto

2:http://modern-carpentry.com/workshop/musicdraw3/

3.1415926: http://modern-carpentry.com/area51/musictool

4:http://modern-carpentry.com/workshop/musicdraw4/

5: http://modern-carpentry.com/workshop/html5/jsmusic

6: From the programming world, I have been mostly influenced by Ken Perlin, Jared Tarbell, Robert Hodgin, Mario Klingemann, and Andre Michelle. I have also enjoyed Nicholas Tasim Teleb’s book, The Black Swan, and its analysis of the random, and the probable. Most important are my studies of the late Robert K Merton and his investigations into the concept of serendipity.

7: the idea of programming randomness or nature, of manufacturing the real with code is downright impossible, a task for Sisyphus [14], a rock I gladly put my shoulder to.

8: http://modern-carpentry.com/workshop/perlintest

9: http://en.wikipedia.org/wiki/Prime_number

10:http://modern-carpentry.com/workshop/primes/

11: The flash application on this page uses sound data ( though the sound is unheard ) along with the x and y position and velocity of a reader’s mouse in order to draw shapes http://articulture.org/

12: bug: https://bugzilla.mozilla.org/show_bug.cgi?id=490705

13: about the bug: http://modern-carpentry.com/workshop/html5/jsmusic/about/

14: Camus, Albert. “The Myth of Sisyphus” Ed. Robert C. Solomon. New York: McGraw-Hill, 1974