Optical Theremin - Demo

At Over The Air I demonstrated what I considered a novel use for one of Android's sensors. I wanted to create a Theremin - a type of musical instrument which is played by moving one's hand over it - changing pitch and tone by moving nearer or further away.

My first attempt used the proximity sensor. However, on all the Android phones I tried the sensor's accuracy was binary - it could sense if something was close by, but not say how close.

So, what else could I use to detect how near or far a hand was from the screen? I decided to co-opt the Light Sensor. This is normally used to automatically adjust the brightness of the screen - making it easier to see in strong light.

When the light sensor is uncovered, the total lux (that's the measure of light) may be 100. As a hand moves closer to it, that value will dip until it reaches 0 (or, on my phone, 4).

We can then represent that light value as a sound - essentially transforming lx into Hz!

This is what is sounds like

Beautiful, I'm sure you agree! You can hear an interview where I discuss this app with the BBC's Jamillah Knowles on the Outriders Podcast (22m 50s in).

If you want to have a play with it, the Optical Theremin Demo is in the Google App Store. Do note, it was coded in a couple of sleep deprived hours, crashes when you exit, and can produce "music" which scares children and animals. You have been warned!

Use The Source, Luke!

I've included the full source below, but I'd like to pick out two points which may be of interest.

Getting The Lux Value

Firstly, we need to register a listener for the light sensor.

@Override public void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
mSensorManager = (SensorManager) getSystemService(SENSOR_SERVICE);
mLightSensor = mSensorManager.getDefaultSensor(Sensor.TYPE_LIGHT);
mSensorManager.registerListener(this, mLightSensor, SensorManager.SENSOR_DELAY_FASTEST);
}

Every time the light sensor changes, this method will be called. It takes the light value and performs a simple mathematical transformation on it (adds 10, multiplies by 5). I found that this gave the most pleasing sound - but you can adjust it to your tastes

@Override public void onSensorChanged(SensorEvent event){
if (event.sensor.getType()==Sensor.TYPE_LIGHT){
    mLux = event.values[0];
    freqOfTone = (mLux +10) * 5;
    }
}

Cum on Feel the Noize

So, how do we get Android to generate a tone? I faffed around with this audio generating code from StackOverflow until I could successfully generate a tone.

Essentially, this creates a WAV of a tone and gets it ready to play.

However, this sounded rather boring, so I added some reverb.

audioTrack.attachAuxEffect(EnvironmentalReverb.PARAM_DECAY_TIME);

And that's it!

Download the Optical Theremin Demo App - or use the source to create something much more melodious.

Full Source

package mobi.shkspr.android.theremin;

import java.util.Random;

import android.app.Activity;
import android.hardware.Sensor;
import android.hardware.SensorEvent;
import android.hardware.SensorEventListener;
import android.hardware.SensorManager;
import android.media.AudioFormat;
import android.media.AudioManager;
import android.media.AudioTrack;
import android.media.audiofx.EnvironmentalReverb;
import android.os.Bundle;
import android.os.Handler;
import android.util.Log;
import android.widget.TextView;

    public class TheraminActivity
	extends Activity
	implements SensorEventListener{
	// originally from http://marblemice.blogspot.com/2010/04/generate-and-play-tone-in-android.html
	// and modified by Steve Pomeroy steve@staticfree.info

	private final int duration = 5; // seconds
	private final int sampleRate = 8000;
	private final int numSamples = duration * sampleRate;
	private final double sample[] = new double[numSamples];
	private double freqOfTone = 440; // hz

	private final byte generatedSnd[] = new byte[2 * numSamples];

	private SensorManager mSensorManager;
	private Sensor mLightSensor;
	private float mLux = 0.0f;
	private String tLux = "Lux is ";

	public AudioTrack audioTrack;

	Handler handler = new Handler();

	@Override public void onCreate(Bundle savedInstanceState) {

	    super.onCreate(savedInstanceState);


	    mSensorManager = (SensorManager) getSystemService(SENSOR_SERVICE);
	    mLightSensor = mSensorManager.getDefaultSensor(Sensor.TYPE_LIGHT);

	    mSensorManager.registerListener(this, mLightSensor, SensorManager.SENSOR_DELAY_FASTEST);

	}

	@Override public void onSensorChanged(SensorEvent event){
	    if (event.sensor.getType()==Sensor.TYPE_LIGHT){
		mLux = event.values[0];
		String luxStr = String.valueOf(mLux);
		TextView tv = new TextView(this);
		tv.setText(tLux);
		setContentView(tv);
		Random r = new Random();
		freqOfTone = (mLux +10) * 5;
	    }

	}
	@Override protected void onResume() {
	    super.onResume();
	    final Thread thread = new Thread(new Runnable() {
		public void run() {

		    for (int i = 0; i < 300; i++)   {
			genTone();

			audioTrack = new AudioTrack(
					AudioManager.STREAM_MUSIC,
					sampleRate,
					AudioFormat.CHANNEL_OUT_MONO,
					AudioFormat.ENCODING_PCM_16BIT,
					numSamples,
					AudioTrack.MODE_STATIC);


			    try {

				playSound();
				Thread.sleep(505);
			    } catch (IllegalStateException e) {

			    } catch (InterruptedException e) {
				// TODO Auto-generated catch block
				e.printStackTrace();
			    }
		    }
		}
	    });

	    thread.start();
	}

	void genTone(){ // fill out the array
	    tLux = "Frequency is " + freqOfTone;
	    //Log.d("LUXTAG", "Lux value: " + tLux);

	    for (int i = 0; i < numSamples; ++i) {
		sample[i] = Math.sin(2 * Math.PI * i /(sampleRate/freqOfTone));
	    }

	// convert to 16 bit pcm sound array
	// assumes the sample buffer is normalised.
	    int idx = 0; for (final double dVal : sample) {
		// scale to maximum amplitude
		final short val = (short) ((dVal * 32767)); // in 16 bit wav PCM, first byte is the low order byte
		generatedSnd[idx++] = (byte) (val & 0x00ff);
		generatedSnd[idx++] = (byte) ((val & 0xff00) >>> 8);

	    }
	}

	void playSound(){
	    genTone();
	    try {		   audioTrack.attachAuxEffect(EnvironmentalReverb.PARAM_DECAY_TIME);
		audioTrack.write(generatedSnd, 0, generatedSnd.length); audioTrack.play();
	    } catch (IllegalStateException e) {
		audioTrack.release();
	    }
	}

	@Override
	public void onAccuracyChanged(Sensor sensor, int accuracy) {
	    // TODO Auto-generated method stub
	}

	@Override
	public void onPause() {
	    super.onPause();
	    audioTrack.stop();
	    audioTrack.flush();
	    audioTrack.release();
	}

	@Override
	public void onStop() {
	    super.onStop();
	    audioTrack.stop();
	    audioTrack.flush();
	    audioTrack.release();
	}

	@Override
	public void onDestroy() {
	    super.onDestroy();
	    audioTrack.flush();
	    audioTrack.stop();
	    audioTrack.flush();
	    audioTrack.release();
	}
    }