A Public, Cross-Platform DSP API?


Porting DSP code to take advantage of platform-specific ‘native’ DSP libraries is a pain. Surely there’s a better way.

Over the years, I’ve worked on DSP projects for many different platforms. As much as possible I try to find a library of essential DSP functions that are well-optimized for the target platform. For example, when I was doing a lot of development on Windows, I used Intel IPP (Integrated Performance Primitives). In recent years, I’ve been developing a lot for iOS, so I now use Apple’s Accelerate/vDSP Framework.

The advantage of using a ‘native’ DSP library is that it can be wicked fast. The FFTs in Intel IPP and in the Apple Accelerate framework are substantially faster than any of the C versions you’ll find on the web. They’re also very nicely documented. Unfortunately they’re also totally non-portable. If you write some nice code using Intel IPP, for example, porting it to some other platform where IPP is not supported will be a lot of work. Even within just the Windows world, you can’t even count on using Intel IPP alone, as it’s not supported on ARM.

A while ago, I needed to port a bunch of DSP code I’d written for iOS to make it run on a Linux server and on Android tablets. I ended up writing my own vDSP-like library – optimistically dubbed “OpenDSP” – all in standard C++, using signatures that are nearly identical to Apple’s own functions. For example, the Accelerate framework has function vDSP_vmul(), which “multiplies vector A by vector B and leaves the result in vector C”. My own implementation looks like this:

void DSP_vmul(
	const float x1[],
	DSP_Stride stride1,
	const float x2[],
	DSP_Stride stride2,
	float y[],
	DSP_Stride strideResult,
	DSP_Length n)
{
	while (n--)
	{
		*y = *x1 * *x2;
		x1 += stride1;
		x2 += stride2;
		y += strideResult;
	}
}

I wrote my own implementations for the couple-of-dozen vDSP functions I needed, including several of the FFT ones. I put them all into a single file (OpenDSP.cpp), and in the accompanying OpenDSP.h header, I mapped the vDSP_abc function names to my DSP_abc, but only if the target was a non-Apple platform.

#ifdef __APPLE__

// If you're building for an Apple platform (Mac or iOS), then you really should use
// the Accelerate framework. It's really fast!

#include < Accelerate/Accelerate.h >

#else

// If you're *not* building for an Apple platform, we use #defines to map vDSP function
// names to the corresponding OpenDSP function names. This allows you to write cross-platform
// signal processing code that's blazingly fast on Apple platforms, and uses Apple's vDSP
// functions, and run the same code on other platforms as well.

#define vDSP_vmul  DSP_vmul

// Similarly for vDSP_vclr, vadd, vsmul, create_fftsetup, fft_zrip, etc...

#endif

This approach worked very well, in that I can now compile code using vDSP functions on any platform, not just Apple’s.

I’ve considered making my OpenDSP library public, but here’s the problem: the vDSP documentation, and therefore presumably the API itself, is covered by Apple’s copyright. So if I were to release the library publicly, Apple could plausibly come after me for copyright infringement. After all, Oracle sued Google over the Java API, and ultimately won the case.

Personally, I’d rather focus my brain power on developing interesting apps, not on learning platform-specific APIs. Ideally there would be a single public-domain API for basic DSP functions, with a generic C/C++ implementation that could be compiled for any platform. Platform owners would be free to create their own super-optimised implementations, or in the mean time developers could map them to existing platform-specific libraries (Intel IPP, Apple Accelerate/vDSP, ARM Ne10, etc.).

Does anyone know whether such an API already exists? If not, would anyone be interested in helping to define one?

Posted in Audio, DSP, FFT, iOS, Programming | 2 Comments

Downloading audio and video from YouTube


Users of my AudioStretch app sometimes ask me whether you can use YouTube content in it, for example to work out how to play a solo from some jazz concert video. Short answer: you can’t. AudioStretch doesn’t have the ability to grab content directly from YouTube, and even if it were technically possible to add such a feature, it’s probably not allowed.

That said, there are services to download content from YouTube and get it into audio or video files on your PC (or Mac). Once you’ve got the file on your PC, you can copy it into Dropbox. Then in Dropbox on your iPad, you can choose the file and ‘export’ it to AudioStrech (or other app) via the iOS “Open In…” mechanism. So here are a couple of services you can check out.

YouTube-mp3.org lets you download the audio track of a YouTube video as an mp3 file. Just take the URL from YouTube, copy/paste it into YouTube-mp3.org, and in a few seconds an mp3 will be ready for download.

If you want to download a video (not just the audio) from YouTube, go to the YouTube video, then change the start of the URL from http://www.youtube.com to ssyoutube.com, e.g. change:

http://www.youtube.com/watch?v=KNd1lXq5MQA

to

http://ssyoutube.com/watch?v=KNd1lXq5MQA

That second link redirects to http://en.savefrom.net, and from there you’ll be able to download the video as an mp4. Note that you can you can open the audio track mp4 videos in AudioStretch via “Open In…” from Dropbox, just as you would load up mp3 files.

Disclaimer: I’m unsure of the legality of the above services, and have no idea whether they’re entirely safe, etc. so use them at your own risk! And also, please remember that much material on YouTube is not in the public domain, and probably shouldn’t have been uploaded to YouTube in the first place. If you’re just downloading a song to work it out in AudioStretch or some other time-stretching app, and not redistributing it, that may be OK (at least morally, if not legally). But if you really like a song, support the artist by buying a copy of it from a legitimate source. If your favourite artist puts out educational videos or books for learning their stuff, buy them. You’d be surprised how little most recording artists earn (and most app developers, for that matter). Musicians and other creative people often love doing what they do, but need to buy food and pay the rent too!

Posted in Uncategorized | Leave a comment

The Expert (Short Comedy Sketch)


Classic! I’ve been in so many meetings like this…

Video | Posted on by | Leave a comment

Real-Time Audio Generation Using WASAPI on Windows 8.1


Most of my audio coding work these days is in Flash or iOS, but I always keep an eye on other possible platforms to develop audio apps for. One of the biggies is of course Windows 8, so I figured I should at least get familiar with the basics of doing interactive real-time audio generation in a Windows 8 app, using Microsoft audio APIs.

Much to my surprise I couldn’t find a good demo app (at least when I searched a few months ago). The nearest I could find was an MSDN WASAPI sample, which is fine, but doesn’t do real-time audio generation, i.e. it doesn’t generate audio that’s calculated continuously, on-the-fly, and sent to the audio output with relatively little latency.

So I created my own (full code here), using the MSDN WASAPI sample as a starting point, stripping it down to the bare essentials for interactive audio generation. It’s a simple continuous sine tone generator, with the sine frequency adjustable via a slider. It looks like this:

SinePlayer

My code sets up a real-time audio output that calls an audio generation callback function at regular intervals. The fiddly WASAPI stuff is encapsulated in C++ AudioOutput class with a very simple interface. Basically you create the AudioOutput, initialize it with a static callback function and an object pointer.

	// Create and initialize a WASAPI renderer
	m_audioOutput = Make<AudioOutput>();
	if (m_audioOutput)
		m_audioOutput->Init(AudioOutputCallback, this);

In that static callback function, you can use the object pointer to call a non-static method of an object. In my demo, the callback function is implemented in MainPage.xaml.cpp. MainPage.xaml has a slider that sets the instantaneous frequency of the sine tone. Here’s the static callback function and the redirection to MainPage’s class method:

void MainPage::AudioOutputCallback(
	float32 *output, 
	int n, 
	int numChannels,
	int sampleRate,
	Platform::Object^ user)
{
	MainPage^ mainPage = safe_cast<MainPage^>(user);
	assert(mainPage);
	mainPage->GenerateAudio(output, n, numChannels, sampleRate);
}

And here’s MainPage’s GenerateAudio function:

void MainPage::GenerateAudio(
	float32 *output, 
	int n, 
	int numChannels, 
	int sampleRate)
{
	// Compute the phase increment for the current frequency
	assert(m_frequency != 0);
	double phaseInc = 2*M_PI*m_frequency/sampleRate;

	// Generate the samples
	for (int i = 0; i < n; i++)
	{
		float32 x = float(0.1 * sin(m_phase));
		for (int ch = 0; ch < numChannels; ch++)
			*output++ = x;
		m_phase += phaseInc;
	}

	// Bring phase back into range [0, 2pi]
	m_phase = fmod(m_phase, 2*M_PI);
}

Again, you can download a zip with the full project source code. It builds fine in Microsoft Visual Studio Express 2013 for Windows, and runs fine on Windows 8.1 running under Bootcamp on my MacBook Pro. I make no claims that it will build or run on any other configuration! If you find this code useful and adapt the code for your own projects, no attribution is necessary… but of course it’s always welcome, as are thank you notes in comments. Enjoy!

Posted in Uncategorized | Leave a comment

100 AppStore Reviews for AudioStretch


Reached an interesting milestone a few days ago: 100 user reviews for AudioStretch. The response has been overwhelmingly positive, averaging about 4.5 out of 5 stars across all versions.

Here’s a screenshot of the most recent reviews as they appear on AppAnnie:

100Reviews

Posted in Uncategorized | Leave a comment

Indiloop “Mix for the Masses” contest


If you’ve got a good ear and enjoy playing around with music tools, check out Indiloop. Now’s a great time to sign up, as Indiloop’s having a remix contest, with a top prize of $2000. The task: using Indiloop, create a remix using ‘stems’ from three artists.

header

No need to buy anything. Signing up to Indiloop is free. All the raw musical material is available on Indiloop. It’s all browser based, so no special software required, just a reasonably fast PC or Mac, and a web browser that supports Flash.

My connection to Indiloop: I developed the Flash real-time audio signal processing “engine” that does all the pitch-shifting, time-stretching, and mixing. Basically the fiddly behind-the-scenes code that makes real-time remixing possible in a web browser. A team of very talented folks at Indiloop HQ in Vancouver does everything else – UI design, web and database programming, support, music licensing, legal, finance. Little startups with big ambitions are always team efforts!

Posted in Uncategorized | Leave a comment

Indiloop


Here’s a little promo video for Indiloop, a Vancouver-based company that has a super-cool online music remixing/mashup service. It’s been up and running on the web for a while, and an iPad version is coming soon. I wrote all the audio signal processing code for both versions – in Flash for the web, and native iOS code for iPad.

Indiloop was chosen as one of the finalists for the MidemLab startup competition at MIDEM, a major music industry conference that takes place every year in Cannes, France.

Turns out that a startup I did some audio programming for a couple of years ago, Paris-based Weezic, was also selected as a Midemlab finalist this year.

I’m hoping both Indiloop and Weezic will win in their respective categories!

Posted in Uncategorized | Leave a comment