A Flash audio output for real-time Javascript audio generation


In the not-too-distant future, it seems likely that all major browsers will provide built in support for real-time audio generation in Javascript. Standardization for the Web Audio API is well under way. It’s already supported in shipping versions of Chrome.

In the meantime though, if you want to do real-time audio synthesis in Javascript without requiring Chrome or special beta plugins for other browsers, one alternative is to use a Flash SWF that calls out to Javascript to do the audio generation. This post describes such a SWF.

Here’s the general idea: the ActionScript 3 code for the SWF uses the Sound class’ SampleDataEvent mechanism to periodically request sample data. The SampleDataEvent handler uses the ExternalInterface class to call a generateAudio() function implemented in Javascript; that function computes the audio sample values, and returns them to the SWF.

Code for a proof-of-concept can be found below. The full source code for the SWF is in one class, FlashAudioOutput. The Javascript code just generates a sine tone, but in principle you could of course generate whatever audio data you like.

The code is generally pretty obvious. The only tricky bit is how to transfer a block of audio sample data from Javascript to ActionScript. The most obvious way would be to use an Array, but this turns out to be horrendously inefficient. Incredible as it may seem, it’s actually more efficient to pass the data as a String!

That’s accomplished using Javascript code that converts an Array of normalized (-1,1) floating point numbers to a very long string of 16-bit integers represented in hexadecimal. That’s done in function convertToHexString() below. In the ActionScript code, the SampleDataEvent handler does the reverse operation of converting the string of hexadecimal values back to normalized float-point values.

Here’s the ActionScript code for the FlashAudioOutput class:

package
{
	import flash.display.Sprite;
	import flash.events.SampleDataEvent;
	import flash.external.ExternalInterface;
	import flash.media.Sound;

	/**
	 * A class to allow real-time audio generation in Javascript in browsers that
	 * don't support the HTML5 real-time audio API. Calls out to a generateAudio
	 * function at regular intervals to get a buffer of audio in the form of
	 * a string of 16-bit signed hexadecimal interleaves (LRLR...) samples.
	 */
	public class FlashAudioOutput extends Sprite
	{
		private const NUM_CHANNELS:uint = 2;			// Audio output is stereo
		private const CHUNK_LEN:uint = 8192;			// Number of stereo sample pairs per SampleDataEvent
		private const SAMPLE_RATE:Number = 44100;		// Sample rate (Hz)

		public function FlashAudioOutput()
		{
			var isAvailable:Boolean = ExternalInterface.available;
			if (!isAvailable)
			{
				trace("ExternalInterface not available!");
				return;
			}

			// Create Sound object for and start real-time audio 'callbacks'
			var snd:Sound = new Sound;
			snd.addEventListener(SampleDataEvent.SAMPLE_DATA, onSampleData);
			snd.play();
		}

		/**
		 * Called at regular intervals to request new audio
		 * data to be sent to the output.
		 */
		private function onSampleData(ev:SampleDataEvent):void
		{
			const BITS_PER_SAMPLE:uint = 16;		// Number of bits per sample
			const HEX_DIGITS_PER_SAMPLE:uint = 4;	// Number of hexadecimal digits per 16-bit sample
			const SHRT_MAX:Number = 32767.0;		// Maximum signed 16-bit number
			const SCALE:Number = 1.0/SHRT_MAX;		// For scaling from 16-bit signed int to normalized [-1,1] range
			const TWO_TO_16TH:Number = 65536;		// 2^16. For two's complement conversion

			// Call out to the Javascript audio generation function
			var hexBuf:String = ExternalInterface.call("generateAudio", CHUNK_LEN);

			var expectedLength:uint = NUM_CHANNELS * CHUNK_LEN * HEX_DIGITS_PER_SAMPLE;
			if (hexBuf.length != expectedLength)
				trace("hexString not expected length", hexBuf.length, expectedLength);

			var pos:uint = 0;
			for (var i:uint = 0; i < NUM_CHANNELS*CHUNK_LEN; i++)
			{
				// Get next 16-bit, 4 digit hex number
				var sub:String = hexBuf.substr(pos, HEX_DIGITS_PER_SAMPLE);
				pos += HEX_DIGITS_PER_SAMPLE;

				// Convert from 16-bit two's complement hexadecimal
				// to floating point number in range [-1,1]
				var val:Number = parseInt(sub, BITS_PER_SAMPLE);
				if (val > SHRT_MAX)
					val -= TWO_TO_16TH;
				val *= SCALE;

				// Write to output
				ev.data.writeFloat(val);
			}
		}
	}
}

And here’s the relevant bit of Javascript code:

			var phase = 0.0;
			var phaseInc = 2*Math.PI*440/44100;

			// Audio generation function. Called at regular intervals by the audio output SWF.
			function generateAudio(n)
			{
				// Generate a sine tone
				// Exercise for developer: make more interesting sound πŸ™‚
				var x = new Array(2*n);
				var pos = 0;
				for (i = 0; i < n; i++)
				{
					var val = 0.1*Math.sin(phase);
					x[pos++] = val;	// Left
					x[pos++] = val;	// Right
					phase += phaseInc;
				}

				return convertToHexString(x);
			}

			//
			// Converts an array of floating point sample values into 16-bit hexadecimal
			// two's complement representation. Input values are expected to be in
			// [-1,1] range and are clipped if they exceed it.
			function convertToHexString(x)
			{
				SHRT_MAX = 32767.0;
				TWO_TO_16TH = 65536;
				BITS_PER_SAMPLE = 16;
				HEX_DIGITS_PER_SAMPLE = 4;

				n = x.length;
				hexBuf = "";

				// For each sample
				for (i = 0; i < n; i++)
				{
					val = x[i];

					// Clip to range [-1,1]
					if (val > 1)
						val = 1;
					else if (val < -1)
						val = -1;

					// Scale and convert to two's complement form
					val = Math.floor(SHRT_MAX*val);
					if (val < 0)
						val += TWO_TO_16TH;

					// Convert to hexadecimal
					hex = val.toString(BITS_PER_SAMPLE);

					// Append to string, adding leading zeros if necessary
					pad = HEX_DIGITS_PER_SAMPLE - hex.length;
					while (pad--)
						hexBuf += "0";
					hexBuf += hex;
				}
				return hexBuf;
			}

A running implementation of the above can be found here.

Have fun with the code. Feel free to use it however you like. Comments are always welcome.

Note: to copy the code without the line numbers, just mouse over the top-right part are of each listing. Some icons will appear, one of which is for “copy to clipboard”. Click that, then paste into whatever editor you use for Flash development.

 

Advertisements

About Gerry Beauregard

I'm a Singapore-based Canadian software engineer, inventor, musician, and occasional triathlete. My current work and projects mainly involve audio technology for the web and iOS. I'm the author of AudioStretch, an audio time-stretching/pitch-shifting app for musicians. Past jobs have included writing speech recognition software for Apple, creating automatic video editing software for muvee, and designing ASICs for Nortel. I hold a Bachelor of Applied Science (Electrical Engineering) from Queen's University and a Master of Arts in Electroacoustic Music from Dartmouth College.
This entry was posted in Audio, Flash, Programming. Bookmark the permalink.

4 Responses to A Flash audio output for real-time Javascript audio generation

  1. Alain Mazy says:

    Hi Gerry,

    It’s funny, we try to make + / – the same things at the same time .. πŸ™‚

    2 years ago, it was time stretching (Master Tempo), and there is the transmission of audio data between Flash and JS.

    The past week, I try every means to transmit audio data (byteArray) between Flash and JS, except that my goal was to relieve my Flash application (non-blocking) in creating a SWF Process, a way to create a multithreaded (and multicore) in flash by OS relay.

    Except that I needed to send 20MB of data :S And all my attempts have ended in failure because of the slow convertion (stringify). I tried everything! (AMF3, Array [float], int, string, etc. ..)

    Yet, if we send a byteArray directly (without convertion) to JS, he receives much at the speed of lightning! but the format is unknown, it is [object Object]. JS should be able to convert directly in FloatArray (for example) .. Conversely, in AS3, it should be cast [object Object] in ByteArray .. I submitted a post on the forum:
    http://forums.adobe.com/message/4537845#4537845

    Until Flash is really multithreaded, delegate heavy tasks such as FFT at JS is probably a good idea .. it would increase the resolution without blocking Flash. like this:
    https://github.com/jsantell/dancer.js/blob/master/lib/fft.js

    Also, I wonder if using bytesArray converted Flash Feature Premium (memory domain) by Azoth for example, would not ultimately faster than linked lists .. to try πŸ™‚

    two years ago, I experienced the linked lists, it’s nice, but I do not win AVVAIS much, for now, I spend all Premium features in Flash and I earn at least 50% in speed ..

    Good luck! πŸ™‚

  2. alama32 says:

    Hi Gerry, Yes, i use base64encode (Azoth version for more speed), but last few days there has been more, the worker! can be shared between a byteArray Process 2 SWF. there is therefore no longer transfer.

    See my tutorial (sorry in frensh) but i make a new website (fr/en).
    http://www.covergraph.com/blog/?p=551

  3. alama32 says:

    PS: i have tried pass vector at JS, but it’s more slow than base64Encode direct byteArray because one loop for byteArray.readFloat() and after, Stringify (Json). Otherwise, in JS decode base64 into Array(float)..

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s