Sonification and Simple Auditory Displays with Arduino

Sonification and Simple Auditory Displays with Arduino

Stand-alone Arduino projects built by sighted people often use LEDs and LCDs as output methods. The color and placement of an LED might indicate a state or activity, while quantitative data like numbers or a simple graph are often presented on an LCD.

Of course, neither of these methods works well for blind makers or users. Instead, we prefer non-visual interface options such as sound, speech, or haptic feedback. I recently posted an article on using servos for quantitative haptic feedback (servometers) , so now I'd like to take a step back to introduce some simple accessible Arduino output techniques using sound. Other output methods such as vibration and text-to-speech are also extremely useful and articles on those topics are forthcoming.

What is Sonification and Auditory Display

An auditory display is exactly what it sounds like – the use of sound to represent information. There are lots of ways of doing this including alarms, alerts, transformation of data into sound, etc. Sonification is a type of auditory display that uses sound (not speech) to represent quantitative information. Usually sonification uses a specific set of sounds and varies the qualities of the sound depending on the information being represented. Since it is really easy to manipulate the pitch of a tone, many sonification techniques have used pitch to represent information. In other words, by associating a particular variable with the pitch of an output sound, we can make an auditory display that represents what is happening with that variable. If the pitch goes up it lets us know that the value of the variable increased. If the pitch goes down then we know the variable decreased. Of course, pitch is not the only sound parameter that can be used, but it's extremely easy to implement and control with the Arduino. It's also relatable for most users and it's classic, so that's what I'll be discussing in this article. Other parameters for sonification include timbre characteristics, pulse rate, and many others. Sound level (volume) is generally not recommended as a display parameter because users usually like to control that. You can learn much more about sonification and Auditory Displays from Wikipedia , or with a little additional Googling.

Making Sound With Arduino

You can buy a number of fancy shields for playing sounds and music with an Arduino, but we are going to confine the discussion to making sound with the classic Arduino Uno (available from Amazon) without any additional hardware other than an unamplified 8Ω speaker (also available from Amazon) . You can also use a piezoelectric buzzer, but the resulting sound quality is rather disappointing. Although buzzers are certainly loud enough (generally too loud), the quality of the sound they produce can be rather hard on the ears. They are also not reliable when reproducing pitch. I strongly prefer using a small, 8Ω speaker, sometimes with a 100Ω resistor in series with it if you are worried about drawing too much current.

There are two simple functions that are generally used to produce sound from an Arduino. These are analogWrite() and tone() . Both of these functions use Pulse Width Modulation (PWM), so in trying out the following examples, be sure that one wire of your speaker is connected to a digital pin that can support PWM (i.e., 3, 5, 6, 9, 10, or 11 on an Arduino Uno). The other speaker wire should be connected to GROUND.

Simple Beep Sonification with analogWrite()

The analogWrite() function lets you use a speaker to make audible beeps of various durations and intensities. This is really useful for providing simple auditory feedback for lots of projects. For example, you can use one, two, or three quick beeps to let the user distinguish between three different states or modes. You can make a long beep to mean one thing and a short beep to mean another. Beep rate can be used as an auditory display for a smoothly varying value with the time between beeps corresponding to the value (or its inverse) being represented like a Geiger counter.

The analogWrite() function is part of the core Arduino library. When used it tells the processor to start alternating the pin state between low (0V) and high (5V) at about 1000 times per second depending on which pin you are using. It continues to alternate low and high at this rate until you tell the pin to do something else. This alternation between low and high becomes a tone when you connect a speaker (or piezoelectric buzzer) to the output pin.

analogWrite() takes two arguments: the pin number and a number between 0 and 255. The pin number obviously tells the Arduino which pin to use, but the second argument tells the Arduino how much time to spend in the high state and how much time to spend in the low state. The percentage of time that the pin spends in the high state is called the Duty Cycle. A value of 0 for this second argument corresponds to a duty cycle of 0% (none of the time is spent in the high state), while a value of 255 tells the pin to spend all of its time in the high state (100% duty cycle). In the middle of the range at 128, the pin spends equal amounts of time in the high state and the low state (a 50% duty cycle). The cool thing about this is that, for sound, it gives you digital control of how loud the sound is. Low values for the duty cycle are quieter and high values are louder. Above approximately 10% duty cycle the loudness no longer increases, but using values below about 30 lets you adjust the sound level quite nicely. This is particularly useful if you are using a piezoelectric buzzer instead of a speaker because they can be extremely loud and obnoxious at full volume.

Sonification Example with analogWrite()

The following sketch illustrates how to use analogWrite to make a beep whose volume can be controlled while the auditory display demo uses the rate of the beeps to sonify the sensor reading.

/**********
 * This sketch demonstrates how to use analogWrite() to make audio beeps,
 * how to implement a simple, beep-rate auditory display,
 * and how to adjust sound volume.
 * For details on analogWrite(), see:    
 *   https://www.arduino.cc/en/Reference/AnalogWrite
 * Note that pins 5-6 on Uno and similar boards will give you a tone close to 1KHz
 * while other PWM pins will give you something closer to 500 Hz.
 * 
 * Wiring:
 *   Speaker -- 1 lead to pin 9, the other to ground.
 *   10K volume pot -- 1 end to 5V, 1 end to ground, arm to A0.
 *   10K sensor pot -- 1 end to 5V, 1 end to ground, arm to A1.
 *   
 * By Josh Miele for the Blind Arduino Project, 09/17/2018
*//////////

//pin assignments
const byte volPotPin = 0;   //pot to control volume
const byte sensorPotPin = 1;   //pot to simulate a sensor reading
const byte speakerPin = 9;  //attach an 8 ohm speaker to this PWM pin
//Sound parameters
int maxVol = 20;   //maximum volume
int beepDur = 50;  //Beep duration in milliseconds 

//Delay timing parameters
int minDelay = 25;   //Minimum delay in milliseconds
int maxDelay = 1000-beepDur;  //max delay corresponds to 1 beep per second

void setup() {
  pinMode(speakerPin, OUTPUT);
}   //end of setup

void loop() {
  //read from volPot to set the duty cycle of the beep
  //use map to get the duty cycle param into the right range.
  int dutyCycle = map(analogRead(volPotPin), 0, 1023, 0, maxVol);
  delay(1);   //for stability)

  //read sensor pot to simulate a varying parameter
  //This could be any dynamic value from sensors, serial, or cloud.
  //Use map to scale the input to the right range of delays
  //Note that this should be exponential, but...
  int delayTime = map(analogRead(sensorPotPin), 0, 1023, minDelay, maxDelay);
  
  //Make the beep
  analogWrite(speakerPin, dutyCycle);   //Turn on the sound
  delay(beepDur);  //Wait for the length of a beep
  digitalWrite(speakerPin, LOW);   //Turn off the sound
  delay(delayTime);  //between minDelay and maxDelay
}   //end of main loop
  

Notes About Designing Auditory Displays with analogWrite()

If you are planning to implement a simple auditory display using beeps, here are a few pointers. Generally speaking people are good at distinguishing between long beeps and short beeps (think of Morse Code), but are not so great at estimating beep duration, so it's not recommended to use beep duration as a display parameter. On the other hand, people are pretty good at hearing very slight differences in timing between beeps(e.g., tempo). This means that varying the delay between beeps makes a pretty good auditory display. As mentioned above, using sound level as a display parameter is discouraged for a variety of reasons. The sketch above has a couple of flaws, most notably, the use of delay prevents the auditory display from responding immediately to changes in the sensor value. This can definitely be dealt with, but it would make the code much more complicated, so it is left as an exercise to the student. The other issue with the sketch is that we vary the delay linearly with the sensor input, but really it should be exponential in order to make the display more usable. More about that in the next section.

Sonification with Tone()

analogWrite() is great for just making an unspecified beep, but it provides no control over the pitch of the sound. For that we need to use tone() . This built-in function allows us to use the same speaker or piezoelectric buzzer to play a tone at a particular frequency, so we can now use pitch as a quantitative auditory display parameter.

The tone() function accepts three arguments: pin (the pin the speaker is attached to), frequency (in Hertz), and duration (in milliseconds). If you omit the duration parameter, the tone plays until you actively turn it off with another function like noTone() or digitalWrite(). For example, if you have your speaker attached to digital pin 9 and you want to play a middle A for 1 second, the command would look like this:

tone(9, 440, 1000);

Some Notes on Frequency, Pitch, and Perception

It's important to understand the difference between frequency and pitch. If all you want is a tone that gets higher and lower as you vary a parameter, then it's fine to simply use linear frequency. Increasing frequency makes a tone sound higher, and decreasing frequency makes it sound lower, but this simple approach leaves most of the pitch change at the bottom of the range. It is as if you have a speedometer that gives you great sensitivity at slow speeds, but at higher speeds it gets less and less accurate. To make the most use of the auditory display, we should have the pitch change equally distributed over the range of frequencies we want to use.

Frequency is the number of cycles per second of the sound. It's measured in hertz (Hz). 1Hz is one cycle per second. The audible range of frequencies for humans is approximately 20Hz to 20,000Hz, with most people losing some range at the top with age.

Another important thing to know is that we perceive pitch in cycles called octaves. The notes of any scale will repeat as we play up through the scale to the next octave, and the next, and the next. Every note has a fundamental frequency associated with it (e.g., middle A if 440Hz). Doubling that frequency gives you the fundamental frequency of that same note in the octave above, while half of that frequency gives you the fundamental frequency of that same note in the octave below. For example, 220, 440, and 880 all have the pitch of A, and are in three consecutive octaves, but the frequency difference between them is far from equal.

Putting it in mathematical terms, there is an exponential (or logarithmic) relationship between frequency and pitch.

This matters because humans are extremely used to hearing tones as pitches. Unless the exponential relationship is used, differences at higher frequencies will sound much smaller than differences at lower frequencies. For example, imagine we hear a tone at 100Hz. If we increase that frequency by 100Hz up to 200Hz, it has gone through an entire octave of change. On the other hand, if we start with a tone of 800Hz and increase the frequency by the same 100Hz up to 900Hz, the pitch only increases by a little bit – much less than an octave. This means that the same amount of increase in frequency sounds much smaller when it's a higher frequency than when it is a low frequency. Using the exponential relationship fixes this problem.

Putting it a different way, and using some musical terminology, if you want the pitch of your sound to change by a fixed interval for a fixed input, you need to include the exponential factor.

In fact, it turns out that most of our perceptions follow this exponential relationship, including our ability to hear tempo changes. This means that in our previous example we really should have used an exponential function instead of map to control the delay. This is psychology's well-known Webber Law which basically says that as stimuli get bigger (e.g., intensity, frequency, duration, etc.), you need a correspondingly larger difference in that stimulus in order to notice it as much.

Pitch Sonification Example with tone()

In the following example, tone provides output for a value being read from an analog input. In this case the input is connected to a potentiometer for demonstration purposes, but we could just as easily use a sensor of some kind, or be reading a value from a serial or cloud-based device. The point is that we're using tone() to represent a dynamically varying, real-time input. Note that this is not that different from the PitchFollower example sketch included with the Arduino IDE. Unlike that example, though, this version offers two different pitch Sonification methods – simple frequency variation, and a more sophisticated (better) linear variation of pitch with input value. This sketch also provides an easy way of turning off the sound to preserve your sanity.

/*
 * Simple Sonification 
 * This sketch demonstrates the basics of using tone() for sonification.
 * It includes a quick-and-dirty qualitative approach using the value of 
 * analogRead() as the frequency. It also illustrates a more 
 * sophisticated approach, using the exponential function to keep 
 * display pitch linear with input value.
 * Wiring:
 * 8 ohm Speaker -- one lead goes to pin 9, the other goes to ground
 * 10K linear Pot -- One end goes to 5v, the other end goes to ground. 
 *     The arm of the pot goes to A0.
 * Josh Miele for the Blind Arduino Project -- 10/24/2018
 */

//Specify pin assignments
const byte speakerPin = 9;   //Needs to be a PWM pin
const byte potPin = 0;   //any analog pin will do

//Set up some variables for the exponential sonification
float minVal = 10;   //Minimum sonified value
float maxVal = 1023;   //Maximum value from analogRead()
float octaves = 3.0;    //how many octaves in the display range?
int baseFrq = 100;     //Lowest display frequency

void setup() {
  pinMode(speakerPin, OUTPUT);
}   //end of setup routine

void loop() {
  //Read the pot and play the tone.
  int potVal = analogRead(potPin);  //we get vals between 0 and 1023

  //if the sensor value is above 10, then we play a tone
  if (potVal > minVal) {
    /* There are two different sonification versions below.
     *  Only one should be used at a time, but try experimenting 
     *  with each by uncommenting it and commenting the other.
     *  The first version simpley uses the output of analogRead()
     *     as the frequency value. This has most pitch change in the 
     *     bottom of the range. 
     * The second version uses an exponential function to 
     *     output equal pitch change for equal input change
     *     throughout the range of analogRead() values.
     *     Try also experimenting with different values of octaves and baseFrq
    */
    tone(speakerPin, potVal);   //Frq is exactly equal to sensor input
//  tone(speakerPin, baseFrq*pow(2, (octaves*(potVal-minVal)/(maxVal-minVal)) ) );  //exponential sonification
  } else {
    //turn off the sound when the pot is all the way down.
    noTone(speakerPin);   //To preserve your sanity!
  }
  
  delay(10);        //Just for stability...
}   //end of main loop

Applying Simple Sonification

There are lots of ways to use tone() and analogWrite() to represent data. After reading this article you can probably imagine how we have used similar approaches to build auditory continuity testers, obstacle detectors, carpenters levels, volt meters, and many other devices. You can combine these simple Sonification techniques to create pitch displays that (for example) pulse in a particular region of the display to indicate critical values. Other displays use tones that are audible only when the input value changes, but are silent otherwise. The simple techniques described above can be used as the building blocks to form some extremely sophisticated auditory display tools. Please let us know how you use them!