MUSIC VISUALIZATION IN iTunes – PART III
This third part of the music visualization using iTunes series goes out of the box; out of the computer’s box that is. More or less the same information as in Part I will be displayed but this time not on the computer screen but on a separate LCD display. Often I am doing something on the computer when the whole screen real estate is in use. There is no screen space to display the iTunes visualization. You ask, how about on a second monitor? Well, that is already filled with stuff too.
The bird’s eye overview of the project is this; an iTunes plugin, via a serial port, using an USB cable, communicates with an Arduino Uno microcontroller. Connected to the Arduino is an graphical LCD display (Lumex LCM-S12864GSF). It can display 128 x 64 pixels providing screen real estate to display the artist, album and song title along with the spectrograph for both left and right channels. The mating of the Arduino and the display was accomplished by a custom PCB that provides a brightness control via a potentiometer.
The ensuing discussion of the design will be subdivided into hardware and software components.
Hardware
The hardware side of the project is quite simple; the Arduino Uno board is connected via a standard USB cable to the computer. Whereas the LCD display is attached to the Arduino using a custom milled PCB panel. The connections between the microcontroller and the LCD display follow the pinout connections outlined in the glcd library [1]. This library was used since the LCD display is equipped with an KS0108 controller chip, so the choice of leveraging on an existing library was obvious. The corresponding connections between the Arduino and the LCD are shown on Figure 1 below.
The connector board contains a 220 Ohm resistor and a 5K Ohm potentiometer to allow control of the screen’s brightness as per the LUMEX data sheet. Beyond that there isn’t much in the way of hardware for this project. The software aspect will be more complex than before; there is a part running on the computer and another one on the microcontroller.
Software
The software aspect of this project is comprised of an iTunes plugin feeding data to the microcontroller, which in turn drives the graphical LCD display. The iTunes plugin leverages on the previously outlined setup provided by the SDK. It opens a serial port at the initialization of the plugin and sends data packets to the Arduino. The information sent for display on the LCD screen is the artist’s name, the album and the song title. Results of the FFT are sent for both left and right channel. The data packets have the following structure;
-Left channel frequency intensities as ten integers in the range from 0 to 8 preceded by a ‘#’ to denote the start of the left channel data packet, for example
#0246321010
-Right channel frequency intensities as ten integers in the range from 0 to 8 preceded by a ‘$’ to denote the start of the left channel data packet, as
$1357320000
– The artist’s name, as a string, preceded by a ‘*’,
*Bob Marley
– The album’s title, as a string, preceded by a ‘^’,
^Songs of Freedom
– The song’s title, as a string, preceded by a ‘{’,
{Judge Not
The symbols selected as data packet designators were chose in the hope that neither the artist nor the album or song title contains them. Of course these cover 90-some percent of the cases, in the remaining 10 there will be hiccups since the Arduino-side interpreter will mis-recognize some packets. Perhaps a more judicious selection of symbols would yield a better solution. I leave that to you, my dear reader. In addition, the number of characters to display, as seen in the title photograph above, is limited by the available space on the LCD display. Currently a 5×7 pixel font is set up yielding up to 17 usable characters per line past the “Art:”, “Alb:” and “Sng:”. Thus, the iTunes plugin truncates the text strings to 17 characters plus an end-of-a-line �. The spectrograph will be made up of ten bars for each channel. There can be up to eight vertical segments per bar.
Following is the source code for the iTunes plugin. One aspect of the plugin has to be clarified. The current version of iTunes, which is 11 as of writing this article, updates the display more frequently than versions prior to it. This causes a slight problem by sending data packets for the Arduino too often, saturating its buffer and eventually slowing the LCD update to a crawl. To remedy this situation, via a global counter, every second display refresh sends the messages out. To further resolve the situation, the serial output buffer is flushed before sending a new set of messages.
void DrawVisualRS232OutputAudioSpectrogram3(VisualPluginData *visualPluginData)
{
// to slow down the communication to avoid saturating the Arduino’s
// receive buffer, every second time refreshing the display will actually send a
// message out
if (globalRS232OutputCounter>1) {
// flush the serial port output buffer
tcflush(port,TCIOFLUSH);
CGRect drawRect;
CGPoint drawAreaExtent;
// this shouldn't happen but let's be safe
if (visualPluginData->destView==NULL) {
return;
}
drawRect=[visualPluginData->destView bounds];
drawAreaExtent.x=drawRect.size.width;
drawAreaExtent.y=drawRect.size.height;
// fill background
[[NSColor blackColor] set];
NSRectFill(drawRect);
// draw gray border
CGRectborderRect;
borderRect.origin.x=5.0;
borderRect.origin.y=5.0;
borderRect.size.width=drawAreaExtent.x-10.0;
borderRect.size.height=drawAreaExtent.y-10.0;
[[NSColor grayColor] setStroke];
DrawRoundedRect(borderRect,5.0,5.0,3.0);
// determine the locations of frequency marks
// to print five of these on the LCD display even though ten
// bars will be drawn per channel
int frequencyMarks[10]={0,2000,4000,6000,8000,10000,12000,
14000,16000,18000};
int frequencyMarksLocations[10];
int counter=0;
// compute the spacing between the FFT values, in Hz
float oneIndexSpacing=visualPluginData->trackInfo.sampleRateFloat/512.0;
for (int i=0;i<256;++i) {
if ((int)(i*oneIndexSpacing)>frequencyMarks[counter] && counter<10) {
frequencyMarksLocations[counter]=i;
++counter;
}
}
// left channel
int values[10]={0,0,0,0,0,0,0,0,0,0};
counter=0;
for (int i=0;i<256;++i) {
if (i==frequencyMarksLocations[counter] && counter<10) {
float average=0.0;
int lowerLimit;
int upperLimit;
if (frequencyMarksLocations[counter]==1) {
average=0.0;
lowerLimit=0;
upperLimit=(int)frequencyMarksLocations[counter+1]*0.5;
for (int j=lowerLimit;j<upperLimit;++j) {
average+=visualPluginData->renderData.spectrumData[0][j];
}
average/=(upperLimit-lowerLimit);
values[counter]=(int)((average/256.0)*32.0);
// up to 8 are allowed per bar
if (values[counter]>8) {
values[counter]=8;
}
}
else if (counter<9) {
average=0.0;
lowerLimit=(int)((frequencyMarksLocations[counter]
+frequencyMarksLocations[counter-1])*0.5);
upperLimit=(int)((frequencyMarksLocations[counter
+1]+frequencyMarksLocations[counter])*0.5);
for (int j=lowerLimit;j<upperLimit;++j) {
average+=visualPluginData->renderData.spectrumData[0][j];
}
average/=(upperLimit-lowerLimit);
values[counter]=(int)((average/256.0)*32.0);
// up to 8 are allowed per bar
if (values[counter]>8) {
values[counter]=8;
}
}
else {
}
++counter;
}
}
NSString *toSendValuesLeft=[NSString
stringWithFormat:@"#%i%i%i%i%i%i%i%i%i%i",
values[0],values[1],values[2],values[3],values[4],
values[5],values[6],values[7],values[8],values[9]];
const char *toCSendValuesLeft=[toSendValuesLeft UTF8String];
//.......... communicate to the Arduino
write(port,toCSendValuesLeft,strlen(toCSendValuesLeft));
//..........
// right channel
counter=0;
for (int i=0;i<256;++i) {
if (i==frequencyMarksLocations[counter] && counter<10) {
float average=0.0;
int lowerLimit;
int upperLimit;
if (frequencyMarksLocations[counter]==1) {
average=0.0;
lowerLimit=0;
upperLimit=(int)frequencyMarksLocations[counter+1]*0.5;
for (int j=lowerLimit;j<upperLimit;++j) {
average+=visualPluginData->renderData.spectrumData[1][j];
}
average/=(upperLimit-lowerLimit);
values[counter]=(int)((average/256.0)*32.0);
// up to 8 are allowed per bar
if (values[counter]>8) {
values[counter]=8;
}
}
else if (counter<9) {
average=0.0;
lowerLimit=(int)((frequencyMarksLocations[counter]
+frequencyMarksLocations[counter-1])*0.5);
upperLimit=(int)((frequencyMarksLocations[counter
+1]+frequencyMarksLocations[counter])*0.5);
for (int j=lowerLimit;j<upperLimit;++j) {
average+=visualPluginData->renderData.spectrumData[0][j];
}
average/=(upperLimit-lowerLimit);
values[counter]=(int)((average/256.0)*32.0);
// up to 8 are allowed per bar
if (values[counter]>8) {
values[counter]=8;
}
}
else {
}
++counter;
}
}
NSString *toSendValuesRight=[NSString
stringWithFormat:@"$%i%i%i%i%i%i%i%i%i%i",
values[0],values[1],values[2],values[3],values[4],
values[5],values[6],values[7],values[8],values[9]];
const char *toCSendValuesRight=[toSendValuesRight UTF8String];
//........... communicate to the Arduino
write(port,toCSendValuesRight,strlen(toCSendValuesRight));
//...........
// song title, performer etc.
NSString *theArtist=NULL;
if (visualPluginData->trackInfo.artist[0]!=0) {
theArtist=[NSString stringWithCharacters:&visualPluginData->trackInfo.
artist[1] length:visualPluginData->trackInfo.artist[0]];
}
NSString *toSendArtist=[NSString stringWithFormat:@"*%@",theArtist];
const char *toCSendArtist=[toSendArtist UTF8String];
char toCSendArtistF[18];
for (int i=0;i<17;++i) {
toCSendArtistF[i]=' ';
}
toCSendArtistF[17]='�';
int length=[toSendArtist length];
if (length>17) {
length=17;
}
for (int i=0;i<length;++i) {
toCSendArtistF[i]=toCSendArtist[i];
}
//.................. communicate to the Arduino
write(port,toCSendArtistF,strlen(toCSendArtistF));
//..................
NSString *theAlbum=NULL;
if (visualPluginData->trackInfo.album[0]!=0) {
theAlbum=[NSString stringWithCharacters:&visualPluginData->trackInfo
.album[1] length:visualPluginData->trackInfo.album[0]];
}
NSString *toSendAlbum=[NSString stringWithFormat:@"^%@",theAlbum];
const char *toCSendAlbum=[toSendAlbum UTF8String];
char toCSendAlbumF[18];
for (int i=0;i<17;++i) {
toCSendAlbumF[i]=' ';
}
toCSendAlbumF[17]='�';
length=[toSendAlbum length];
if (length>17) {
length=17;
}
for (int i=0;i<length;++i) {
toCSendAlbumF[i]=toCSendAlbum[i];
}
//............ communicate to the Arduino
write(port,toCSendAlbumF,strlen(toCSendAlbumF));
//............
NSString *theSongTitle=NULL;
if ( visualPluginData->trackInfo.name[0]!=0) {
theSongTitle=[NSString
stringWithCharacters:&visualPluginData->trackInfo.name[1]
length:visualPluginData->trackInfo.name[0]];
}
NSString *toSendSongTitle=[NSString stringWithFormat:@"{%@",theSongTitle];
const char *toCSendSongTitle=[toSendSongTitle UTF8String];
char toCSendSongTitleF[18];
for (int i=0;i<17;++i) {
toCSendSongTitleF[i]=' ';
}
toCSendSongTitleF[17]='�';
length=[toSendSongTitle length];
if (length>17) {
length=17;
}
for (int i=0;i<length;++i) {
toCSendSongTitleF[i]=toCSendSongTitle[i];
}
//........... communicate to the Arduino
write(port,toCSendSongTitleF,strlen(toCSendSongTitleF));
//...........
globalRS232OutputCounter=0;
}
globalRS232OutputCounter++;
}
The setup of the serial communication port is done in the following function
int open_port(void)
{
int fd; // File descriptor for the port
// the location of the port varies between computers and operating systems
// please see where your Arduino is connected...
fd=open("/dev/cu.usbmodem1d11", O_RDWR | O_NOCTTY | O_NDELAY);
if (fd==-1) {
perror("open_port: Unable to open /dev/cu.usbmodem1d11 ");
}
else {
fcntl(fd, F_SETFL, 0);
}
struct termios options;
tcgetattr(fd, &options);
// set the baud rate
cfsetispeed(&options, B115200);
cfsetospeed(&options, B115200);
options.c_cflag |= (CLOCAL | CREAD);
tcsetattr(fd, TCSANOW, &options);
return (fd);
}
A global declaration of the port opening function and the port itself are done at the top of the input file as such;
int open_port(void);
int port;
As explained earlier, to reduce the frequency of sending data packets to the Arduino, the following counter variable is used in the DrawVisualRS232OutputAudioSpectro-gram3 function above.
int globalRS232OutputCounter=0;
The open_port(void) function is called at the time of activating the plugin within iTunes:
OSStatus ActivateVisual( VisualPluginData * visualPluginData, VISUAL_PLATFORM_VIEW destView, OptionBits options )
{
....
port=open_port();
....
}
As before, the drawRect function takes care of the screen update mechanism:
-(void)drawRect:(NSRect)dirtyRect
{
....
DrawVisualRS232OutputAudioSpectrogram3(_visualPluginData);
....
}
This more or less concludes the iTunes plugin side of the software. The receiving end on the Arduino is contained in the following sketch.
// include the glcd library #include // include the fonts #include <fonts/allFonts.h> // storage for the display char inputStringArtistC[17]; char inputStringAlbumC[17]; char inputStringSongC[17]; char inputStringLeftBar[11]; char inputStringRightBar[11]; int barValuesLeft[10]; int barValuesRight[10]; int inByte=0; char myCmd[128]; int inputSize=0; // setup chores void setup() { // initialize glcd GLCD.Init(); // select the font for the default text area GLCD.SelectFont(System5x7); // start the serial port Serial.begin(115200); } // print the artist on the LCD display void printArtist() { GLCD.CursorTo(0,0); GLCD.print("Art:"); GLCD.println(inputStringArtistC); } // print the album on the LCD display void printAlbum() { GLCD.CursorTo(0,1); GLCD.print("Alb:"); GLCD.println(inputStringAlbumC); } // print the song on the LCD display void printSong() { GLCD.CursorTo(0,2); GLCD.print("Sng:"); GLCD.println(inputStringSongC); } // draw the left channel’s bars void drawLeftBar() { char oneDigit[2]; oneDigit[1]='�'; oneDigit[0]=inputStringLeftBar[0]; barValuesLeft[0]=atoi(oneDigit); oneDigit[0]=inputStringLeftBar[1]; barValuesLeft[1]=atoi(oneDigit); oneDigit[0]=inputStringLeftBar[2]; barValuesLeft[2]=atoi(oneDigit); oneDigit[0]=inputStringLeftBar[3]; barValuesLeft[3]=atoi(oneDigit); oneDigit[0]=inputStringLeftBar[4]; barValuesLeft[4]=atoi(oneDigit); oneDigit[0]=inputStringLeftBar[5]; barValuesLeft[5]=atoi(oneDigit); oneDigit[0]=inputStringLeftBar[6]; barValuesLeft[6]=atoi(oneDigit); oneDigit[0]=inputStringLeftBar[7]; barValuesLeft[7]=atoi(oneDigit); oneDigit[0]=inputStringLeftBar[8]; barValuesLeft[8]=atoi(oneDigit); oneDigit[0]=inputStringLeftBar[9]; barValuesLeft[9]=atoi(oneDigit); int barStartX; int barStartY; for (int i=0;i<10;++i) { barStartX=i*6; for (int j=0;j<8;++j) { barStartY=56-(j+1)*3; GLCD.FillRect(barStartX,barStartY,5,2,WHITE); } for (int j=0;j<barValuesLeft[i];++j) { barStartY=56-(j+1)*3; GLCD.FillRect(barStartX,barStartY,4,1); } } } // draw the right channel’s bars void drawRightBar() { char oneDigit[2]; oneDigit[1]='�'; oneDigit[0]=inputStringRightBar[0]; barValuesRight[0]=atoi(oneDigit); oneDigit[0]=inputStringRightBar[1]; barValuesRight[1]=atoi(oneDigit); oneDigit[0]=inputStringRightBar[2]; barValuesRight[2]=atoi(oneDigit); oneDigit[0]=inputStringRightBar[3]; barValuesRight[3]=atoi(oneDigit); oneDigit[0]=inputStringRightBar[4]; barValuesRight[4]=atoi(oneDigit); oneDigit[0]=inputStringRightBar[5]; barValuesRight[5]=atoi(oneDigit); oneDigit[0]=inputStringRightBar[6]; barValuesRight[6]=atoi(oneDigit); oneDigit[0]=inputStringRightBar[7]; barValuesRight[7]=atoi(oneDigit); oneDigit[0]=inputStringRightBar[8]; barValuesRight[8]=atoi(oneDigit); oneDigit[0]=inputStringRightBar[9]; barValuesRight[9]=atoi(oneDigit); int barStartX; int barStartY; for (int i=0;i<10;++i) { barStartX=64+i*6; for (int j=0;j<8;++j) { barStartY=56-(j+1)*3; GLCD.FillRect(barStartX,barStartY,5,2,WHITE); } for (int j=0;j<barValuesLeft[i];++j) { barStartY=56-(j+1)*3; GLCD.FillRect(barStartX,barStartY,4,1); } } } // main loop void loop() { char c; int messageLength=50; int bufferLength=63; // print frequency axis labels - left GLCD.DrawString("0",0,57); GLCD.DrawString("4",12,57); GLCD.DrawString("8",24,57); GLCD.DrawString("12",36,57); GLCD.DrawString("16",50,57); // print frequency axis labels - right GLCD.DrawString("0",64,57); GLCD.DrawString("4",76,57); GLCD.DrawString("8",88,57); GLCD.DrawString("12",100,57); GLCD.DrawString("16",114,57); inputSize=0; // when characters arrive over the serial port... if (Serial.available()>messageLength) { inputSize=Serial.available(); for (int i=0;i<inputSize;i++) { myCmd[i]=Serial.read(); } // say what you got: Serial.println(myCmd); // parse out the left channel // find the beginning of a valid message (#) int messageStart=0; int messageFound=0; for (int i=0;i<inputSize;i++) { if (myCmd[i]=='#') { messageStart=i+1; messageFound=1; break; } } if (messageFound==1 && (messageStart+10)<bufferLength) { int counter=0; for (int i=0;i<11;++i) { inputStringLeftBar[i]=' '; } for (int i=messageStart;i<(messageStart+10);++i) { inputStringLeftBar[counter]=myCmd[i]; counter++; } inputStringLeftBar[counter]='�'; Serial.println(inputStringLeftBar); drawLeftBar(); } // parse out the right channel // find the beginning of a valid message ($) messageStart=0; messageFound=0; for (int i=0;i<inputSize;i++) { if (myCmd[i]==' messageStart=i+1; messageFound=1; break; } } if (messageFound==1 && (messageStart+10)<bufferLength) { int counter=0; for (int i=0;i<11;++i) { inputStringRightBar[i]=' '; } for (int i=messageStart;i<(messageStart+10);++i) { inputStringRightBar[counter]=myCmd[i]; counter++; } inputStringRightBar[counter]='�'; Serial.println(inputStringRightBar); drawRightBar(); } // parse out the artist // find the beginning of a valid message (*) messageStart=0; messageFound=0; for (int i=0;i<inputSize;i++) { if (myCmd[i]=='*') { messageStart=i+1; messageFound=1; break; } } if (messageFound==1 && (messageStart+16)<bufferLength) { int counter=0; for (int i=0;i<16;++i) { inputStringArtistC[i]=' '; } for (int i=messageStart;i<(messageStart+16);++i) { inputStringArtistC[counter]=myCmd[i]; counter++; } inputStringArtistC[counter]='�'; Serial.println(inputStringArtistC); printArtist(); } // parse out the album // find the beginning of a valid message (^) messageStart=0; messageFound=0; for (int i=0;i<inputSize;i++) { if (myCmd[i]=='^') { messageStart=i+1; messageFound=1; break; } } if (messageFound==1 && (messageStart+16)<bufferLength) { int counter=0; for (int i=0;i<16;++i) { inputStringAlbumC[i]=' '; } for (int i=messageStart;i<(messageStart+16);++i) { inputStringAlbumC[counter]=myCmd[i]; counter++; } inputStringAlbumC[counter]='�'; Serial.println(inputStringAlbumC); printAlbum(); } // parse out the song // find the beginning of a valid message ({) messageStart=0; messageFound=0; for (int i=0;i<inputSize;i++) { if (myCmd[i]=='{') { messageStart=i+1; messageFound=1; break; } } if (messageFound==1 && (messageStart+16)<bufferLength) { int counter=0; for (int i=0;i<16;++i) { inputStringSongC[i]=' '; } for (int i=messageStart;i<(messageStart+16);++i) { inputStringSongC[counter]=myCmd[i]; counter++; } inputStringSongC[counter]='�'; Serial.println(inputStringSongC); printSong(); } } }
And that is about it…
I hope you enjoyed reading this part of the music visualization using iTunes series. All the mistakes contained in these articles are mine. When, not if, you find any, please let me know so I can correct them and update these webpages!
References
1. glcd library for KS0108 controller chip-equipped graphical LCD displays. http://playground.arduino.cc/Code/GLCDks0108
Note: Apple, iTunes, Arduino and other hardware/software is a trademark of their respective owners.