{"id":376,"date":"2013-11-24T22:24:59","date_gmt":"2013-11-24T22:24:59","guid":{"rendered":"http:\/\/localhost:8888\/?page_id=376"},"modified":"2022-07-06T18:40:19","modified_gmt":"2022-07-06T18:40:19","slug":"music-visualization-part-i","status":"publish","type":"page","link":"https:\/\/www.amzsaki.com\/?page_id=376","title":{"rendered":"Music visualization – Part I."},"content":{"rendered":"

MUSIC VISUALIZATION IN\u00a0iTunes\u00a0– PART I<\/strong><\/p>\n

\"iTunesViz01a\"<\/p>\n

Music is inherently enjoyed by our brains via our ears. To me this enjoyment can be heightened by actually seeing the music being played. The delightful vibration and cascade of frequencies adds to the experience. It appears that I am a sucker for visual things.<\/p>\n

This first part of the three-part series on music visualization, by a non-musically inclined author, is devoted to using the iTunes SDK to plot the spectrogram of the music being played. First, I would like to express my gratitude to the nice folks at Apple for providing the general public with such a software development kit.\u00a0 Although the download location of the SDK is not that obvious, you can find it on Apple\u2019s website after a little bit of searching.<\/p>\n

The following brief document outlines the salient features of displaying a spectrograph as a visualization within iTunes. It is assumed that the dear reader possesses enough knowledge to compile & link the plugin and can place the resulting file into the appropriate folder. As of writing this document, the plugin bundle has to be moved to\u00a0your_username\/Library\/iTunes\/iTunes Plug-ins\/<\/em>\u00a0folder. Of course, this location may be different with an update to iTunes or OSX. In the Xcode project, the file called iTunesPlugInMac.mm<\/em> will be modified. The project, as is, provides all the necessary mechanisms for initialization.<\/p>\n

The modifications, in order, will be made to the following functions;\u00a0DrawVisual<\/em>\u00a0and drawRect<\/em>. DrawVisual<\/em> will take care of plotting the spectrograph and the drawRect<\/em> function will select what visualization to draw. Parts II and III of this mini-series will have different visualizations, so which to display can be selected in drawRect.<\/p>\n

The DrawVisual<\/em> function enables us to draw onto a canvas in response to the music being played. To keep things minimal, the background will be cleared to black, a nice grey border with rounded corners will be added as well. The spectrograph for both left and right channels will occupy the bottom half of the screen while the upper part will show the album cover, the performer, song and album title. It is kept simple.<\/p>\n

The SDK provides the frequencies retrieved from a FFT. The information gathered from the Net lists the FFT as a 512-coefficient for the two channels. The values for each element of the array have entries in the range 0-255. The SDK has access to this information in the\u00a0visualPluginData->renderData.spectrumData<\/em> data structure.<\/p>\n

Next, I\u2019ll present the DrawVisual<\/em> function with annotated source code.<\/p>\n

void DrawVisual(VisualPluginData *visualPluginData)\n{\n\n  \/\/ definition of the rectangle to draw into\n  CGRect drawRect;\n  \/\/ a start point of an entity to draw\n  CGPointstartPoint;\n  \/\/ the extent of a bar(graph) for visualization - allowing rescaling the\n  \/\/ visualization window\n  CGPointbarExtent;\n  \/\/ extent of area to draw into\n  CGPoint drawAreaExtent;\n\n  \/\/ this shouldn't happen but let's be safe\n  if (visualPluginData->destView==NULL) {\n    return;\n  }\n\n  \/\/ obtain the extent of the current widow to draw into\n  drawRect=[visualPluginData->destView bounds];\n  drawAreaExtent.x=drawRect.size.width;\n  drawAreaExtent.y=drawRect.size.height;\n\n  \/\/ fill background\n  [[NSColor blackColor] set];\n  NSRectFill(drawRect);\n\n  \/\/ draw gray border\n  CGRectborderRect;\n  borderRect.origin.x=5.0;\n  borderRect.origin.y=5.0;\n  borderRect.size.width=drawAreaExtent.x-10.0;\n  borderRect.size.height=drawAreaExtent.y-10.0;\n\n  [[NSColor grayColor] setStroke];\n  DrawRoundedRect(borderRect,5.0,5.0,3.0);\n\n  \/\/ the maximum number of channels one can show\n  int numSpectrumChannelsToShow=256;\n  \/\/ padding (empty space) between the segments of the bar graph, both\n  \/\/ horizontally and vertically\n  float barPaddingX=2.0;\n  float barPaddingY=2.0;\n  \/\/ height of each individual segment\n  float barSegmentHeight=5.0;\n  \/\/ determine the locations of frequency marks\n  \/\/ human hearing can detect sound in the 20Hz to 20kHz range\n  \/\/ so I decided that we will show ten equally spaced values as labels \n  \/\/ within this range\n  int frequencyMarks[10]={0,2000,4000,6000,8000,10000,12000,\n                                           14000,16000,18000};\n  NSArray *labelArray=[[NSArray alloc] \n      initWithObjects:@\"0Hz\",@\"2kHz\",@\"4kHz\",@\"6kHz\",\n                      @\"8kHz\",@\"10kHz\",@\"12kHz\",@\"14kHz\",\n                      @\"16kHz\",@\"18kHz\",nil];\n  int frequencyMarksLocations[10];\n  int counter=0;\n\n  \/\/ retrieve the sampling frequency of the music being played\n  \/\/ and establish what is the actual spacing (in Hz) between the values\n  float oneIndexSpacing=visualPluginData->trackInfo.sampleRateFloat\/512.0;\n\n  \/\/ store the location (index) where each of the label\u2019s frequencies are found\n  for (int i=0;i<numSpectrumChannelsToShow;++i) { \n    if ((int)(i*oneIndexSpacing)>frequencyMarks[counter] && counter<10) {\n      frequencyMarksLocations[counter]=i;\n      ++counter;\n    }\n  }\n\n  \/\/ create a font and set up the attributes for the labels\n  NSFont *labelFont=[NSFont fontWithName:@\"Arial\" size:12.0];\n  NSDictionary *labelAttrs=[NSDictionary dictionaryWithObjectsAndKeys:\n                           [NSColor whiteColor],\n                           NSForegroundColorAttributeName, labelFont,   \n                           NSFontAttributeName, NULL];\n\n  \/\/ display the left channel labels\n  counter=0;\n  for (int i=0;i<frequencyMarksLocations[9];++i) {        \n    startPoint.x=20.0;\n    startPoint.y=45.0;\n\n    barExtent.x=(drawAreaExtent.x*0.5-startPoint.x*2.0\n                     -barPaddingX*frequencyMarksLocations[9])\/\n                     frequencyMarksLocations[9];\n\n     if (barExtent.x<1.0) {\n       barExtent.x=1.0;\n       barPaddingX=0.0;\n     }\n\n     startPoint.x+=i*(barExtent.x+barPaddingX);\n     startPoint.y=startPoint.y;\n\n     if (i==frequencyMarksLocations[counter] && counter<10) {\n       NSString *frequencyLabel=NULL;\n       frequencyLabel=[labelArray objectAtIndex:counter];\n\n       if (frequencyLabel!=NULL) {\n         [frequencyLabel drawAtPoint:startPoint withAttributes:labelAttrs];\n       }\n\n       ++counter;\n     }      \n   }\n\n   \/\/ pick another font and set up the attributes for the channel labels\n   NSFont *channelLabelFont=[NSFont fontWithName:@\"Arial\" size:20.0];\n   NSDictionary *channelLabelAttrs=[NSDictionary dictionaryWithObjectsAndKeys:\n                 [NSColor whiteColor], NSForegroundColorAttributeName,\n                 channelLabelFont, NSFontAttributeName, NULL];\n\n   startPoint.x=drawAreaExtent.x*0.18; \n   startPoint.y=15.0;\n\n   \/\/ display the label\n   NSString *leftChannelLabel=@\"Left Channel\";\n   [leftChannelLabel drawAtPoint:startPoint withAttributes:channelLabelAttrs];\n\n   \/\/ left channel bar graphs\n   for (int i=0;i<frequencyMarksLocations[9];++i) {\n     startPoint.x=20.0;\n     startPoint.y=60.0;\n\n     barExtent.x=(drawAreaExtent.x*0.5-startPoint.x*2.0\n                     -barPaddingX*frequencyMarksLocations[9])\/\n                     frequencyMarksLocations[9];\n\n    if (barExtent.x<1.0) { \n      barExtent.x=1.0; barPaddingX=0.0; \n    } \n    startPoint.x+=i*(barExtent.x+barPaddingX); \n    startPoint.y=startPoint.y; \n    float factor=(visualPluginData->renderData.spectrumData[0][i]\/256.0);\n\n    float available_space=drawAreaExtent.y*0.5-startPoint.y-20.0;\n    barExtent.y=factor*available_space;\n\n    \/\/ determine the number of segments (vertically) to display for this entry\n    int numSegments=(int)(factor*available_space)\/(barPaddingY\n                                                +barSegmentHeight);\n    \/\/ determine the maximum number of segments one can display\n    int numMaxSegments=(int)(available_space)\/(barPaddingY\n                                                +barSegmentHeight);\n\n    \/\/ draw each segment\n    for (int j=0;j<numSegments;++j) {\n\n      CGFloatred;\n      CGFloatgreen;\n      CGFloatblue;\n\n      float inputValue=((float)j\/(float)numMaxSegments)*255.0;\n\n      \/\/ compute the color corresponding to this value\n      GetColorGreenYellowRed(inputValue,0.0,255.0,&red,&green,&blue);\n\n      [[NSColor colorWithDeviceRed:red green:green blue:blue alpha:1] \n                             set];\n\n      float segmentStartY=startPoint.y+j*(barPaddingY+barSegmentHeight);\n\n      drawRect=NSMakeRect(startPoint.x,segmentStartY,barExtent.x,\n                                                          barSegmentHeight);\n\n      \/\/ draw the segment\n      DrawRectFilled(drawRect);\n    }\n  }\n\n  \/\/ display the right channel labels\n  counter=0;\n  for (int i=0;i<frequencyMarksLocations[9];++i) {\n    startPoint.x=20.0;\n    startPoint.y=45.0;\n\n    barExtent.x=(drawAreaExtent.x*0.5-startPoint.x*2.0\n                     -barPaddingX*frequencyMarksLocations[9])\/\n                     frequencyMarksLocations[9];\n\n    if (barExtent.x<1.0) {\n      barExtent.x=1.0;\n      barPaddingX=0.0;\n    }\n\n    startPoint.x+=i*(barExtent.x+barPaddingX);\n    startPoint.y=startPoint.y;\n\n    CGPoint startPointR;\n\n    startPointR.x=startPoint.x+drawAreaExtent.x*0.5;\n    startPointR.y=startPoint.y;\n\n    if (i==frequencyMarksLocations[counter] && counter<10) {\n      NSString *frequencyLabel=NULL;\n      frequencyLabel=[labelArray objectAtIndex:counter];\n\n      if (frequencyLabel!=NULL) {\n        [frequencyLabel drawAtPoint:startPointR withAttributes:labelAttrs];\n      }\n\n      ++counter;\n    }\n  }\n\n  startPoint.x=drawAreaExtent.x*0.68; \n  startPoint.y=15.0;\n\n  \/\/ display the right channel label\n  NSString *rightChannelLabel=@\"Right Channel\";\n  [rightChannelLabel drawAtPoint:startPoint withAttributes:channelLabelAttrs];\n\n  \/\/ right channel bar graphs\n  for (int i=0;i<frequencyMarksLocations[9];++i) {\n    startPoint.x=20.0;\n    startPoint.y=60.0;\n\n    barExtent.x=(drawAreaExtent.x*0.5-startPoint.x*2.0\n                     -barPaddingX*frequencyMarksLocations[9])\/\n                     frequencyMarksLocations[9];\n\n    if (barExtent.x<1.0) { \n      barExtent.x=1.0; barPaddingX=0.0; \n    } \n    startPoint.x+=i*(barExtent.x+barPaddingX); \n    startPoint.y=startPoint.y; \n    float factor=(visualPluginData->renderData.spectrumData[1][i]\/256.0);\n    float available_space=drawAreaExtent.y*0.5-startPoint.y-20.0;\n    barExtent.y=factor*available_space;\n\n    \/\/ determine the number of segments (vertically) to display for this entry\n    int numSegments=(int)(factor*available_space)\/(barPaddingY\n                                        +barSegmentHeight);\n    \/\/ determine the maximum number of segments one can display\n    int numMaxSegments=(int)(available_space)\/(barPaddingY\n                                        +barSegmentHeight);\n\n    \/\/ draw each segment\n    for (int j=0;j<numSegments;++j) { \n      CGFloatred; \n      CGFloatgreen; \n      CGFloatblue; \n\n      float inputValue=((float)j\/(float)numMaxSegments)*255.0;                          \/\/ compute the color corresponding to this value \n      GetColorGreenYellowRed(inputValue,0.0,255.0,&red,&green,&blue); \n      [[NSColor colorWithDeviceRed:red green:green blue:blue alpha:1]                            set]; \n      float segmentStartY=startPoint.y+j*(barPaddingY+barSegmentHeight); \n      drawRect=NSMakeRect(startPoint.x+drawAreaExtent.x*0.5,                                 segmentStartY,barExtent.x,barSegmentHeight);                       \n      \/\/ draw the segment DrawRectFilled(drawRect); \n    } \n  } \n\n  \/\/ display song title, performer etc. \n  startPoint=CGPointMake(30.0,drawAreaExtent.y*0.5+20.0); \n  NSFont *stringFont=[NSFont fontWithName:@\"Arial\" size:14.0]; \n  NSDictionary *attrs=[NSDictionary dictionaryWithObjectsAndKeys:                            [NSColor whiteColor],NSForegroundColorAttributeName, stringFont,\n                            NSFontAttributeName, NULL]; \n  NSString *theTrackNumber=NULL; \n\n  if (visualPluginData->trackInfo.trackNumber!=0) {\n    theTrackNumber=[NSString stringWithFormat:@\"%i\",\n                           visualPluginData->trackInfo.trackNumber];\n  }\n  if (theTrackNumber!=NULL) {\n    [theTrackNumber drawAtPoint:startPoint withAttributes:attrs];\n  }\n\n  startPoint.x+=20.0;\n\n  NSString *theNumTracks=NULL;\n\n  if (visualPluginData->trackInfo.numTracks!=0) {\n    theNumTracks=[NSString stringWithFormat:@\"\/%i\",\n            visualPluginData->trackInfo.numTracks];\n  }\n  else {\n    theNumTracks=[NSString stringWithFormat:@\"\/?\"];\n  }\n\n  if (theNumTracks!=NULL) {\n    [theNumTracks drawAtPoint:startPoint withAttributes:attrs];\n  }\n\n  startPoint.x+=30.0;\n\n  NSString *theArtist=NULL;\n\n  if (visualPluginData->trackInfo.artist[0]!=0) {\n    theArtist=[NSString stringWithCharacters:\n                    &visualPluginData->trackInfo.artist[1]\n                    length:visualPluginData->trackInfo.artist[0]];\n  }\n\n  if (theArtist!=NULL) {\n    [theArtist drawAtPoint:startPoint withAttributes:attrs];\n  }\n\n  startPoint.x+=[theArtist length]*8.0+8.0;\n\n  NSString *theAlbum=NULL;\n\n  if (visualPluginData->trackInfo.album[0]!=0) {\n    theAlbum=[NSString stringWithCharacters:\n           &visualPluginData->trackInfo.album[1]\n           length:visualPluginData->trackInfo.album[0]];\n  }\n\n  if (theAlbum!=NULL) {\n    [theAlbum drawAtPoint:startPoint withAttributes:attrs];\n  }\n\n  startPoint.x+=[theAlbum length]*8.0+8.0;\n\n  \/\/ if we have a song title, draw it (prefer the stream title\n  \/\/ over the regular name if we have it)\n  NSString *theSongTitle=NULL;\n\n  if (visualPluginData->streamInfo.streamTitle[0]!=0) {\n    theSongTitle=[NSString stringWithCharacters:\n            &visualPluginData->streamInfo.streamTitle[1]\n            length:visualPluginData->streamInfo.streamTitle[0]];\n  }\n  else if ( visualPluginData->trackInfo.name[0]!=0) {\n    theSongTitle=[NSString stringWithCharacters:\n            &visualPluginData->trackInfo.name[1]\n            length:visualPluginData->trackInfo.name[0]];\n  }\n\n  if (theSongTitle!=NULL) {\n    [theSongTitle drawAtPoint:startPoint withAttributes:attrs];\n  }\n\n  startPoint.x+=[theSongTitle length]*8.0+8.0;\n\n  NSString *theYear=NULL;\n\n  if (visualPluginData->trackInfo.year!=0) {\n    theYear=[NSString stringWithFormat:@\"(%i)\",\n              visualPluginData->trackInfo.year];\n  }\n  else {\n    theYear=[NSString stringWithFormat:@\"(?)\"];\n  }\n\n  if (theYear!=NULL) {\n    [theYear drawAtPoint:startPoint withAttributes:attrs];\n  }\n\n  startPoint.x=30.0;\n\n  \/\/ draw the artwork\n  if (visualPluginData->currentArtwork!=NULL) {\n    startPoint.y+=20.0;\n    NSSize aSize;\n    aSize.width=(drawAreaExtent.y*0.5-60.0);\n    aSize.height=(drawAreaExtent.y*0.5-60.0);\n\n    [visualPluginData->currentArtwork setSize:aSize];\n    [visualPluginData->currentArtwork drawAtPoint:startPoint\n                                             fromRect:NSZeroRect\n                                            operation:NSCompositeSourceOver\n                                             fraction:1.0];\n  }\n}<\/span><\/pre>\n
Other miscellaneous functions utilized in the above function:<\/p>\n
\/\/ interpolate the color cube (in the R, G and B space) to obtain the color\n\/\/ the lowest value (0) is green and the highest (1) is red\nvoid GetColorGreenYellowRed(CGFloat d, CGFloat dmin, CGFloat dmax, CGFloat *r, CGFloat *g, CGFloat *b)\n{\n  CGFloat ds;\n\n  if (d<dmin) { \n    d=dmin; \n  }      \n  if (d>dmax) {\n    d=dmax;\n  }\n\n  ds=dmax-dmin;\n\n  if (d<(dmin+0.5*ds)) { \n    *r=0.0+(d-dmin)\/(ds*0.5); \n    *g=1.0; \n    *b=0.0; \n  } \n  if (d>=(dmin+0.5*ds)) {\n    *r=1.0;\n    *g=1.0-(d-ds*0.5)\/(ds*0.5);\n    *b=0.0;\n  }\n}\n\n\/\/ draw a rounded rectangle\nvoid DrawRoundedRect(NSRect rect, CGFloat x, CGFloat y, CGFloat lineWidth)\n{\n    NSBezierPath* thePath=[NSBezierPath bezierPath];\n\n    [thePath setLineWidth:lineWidth];\n    [thePath appendBezierPathWithRoundedRect:rect xRadius:x yRadius:y];\n    [thePath stroke];\n}\n\n\/\/ draw a filled rectangle\nvoid DrawRectFilled(NSRect rect)\n{  \n  NSRectFill(rect);\n  [NSBezierPath strokeRect:rect];\n}<\/span><\/pre>\n
Note that these functions are driven by the following, already implemented function:<\/p>\n
-(void)drawRect:(NSRect)dirtyRect\n{\n  if ( _visualPluginData != NULL ) {\n    DrawVisual(_visualPluginData);\n  }\n}<\/span><\/pre>\n
And that is about it! I hope this short tutorial, if I can call it that, would enable you to create some awesome visualizations! In the next part we will look at another way to display the same frequency information.<\/p>\n
Note: Apple, iTunes and other hardware\/software is trademark of their respective owners.<\/p>\n","protected":false},"excerpt":{"rendered":"
MUSIC VISUALIZATION IN\u00a0iTunes\u00a0– PART I Music is inherently enjoyed by our brains via our ears. To me this enjoyment can<\/p>\n
Continue readingMusic visualization – Part I.<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"parent":298,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-376","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/www.amzsaki.com\/index.php?rest_route=\/wp\/v2\/pages\/376","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.amzsaki.com\/index.php?rest_route=\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.amzsaki.com\/index.php?rest_route=\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.amzsaki.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.amzsaki.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=376"}],"version-history":[{"count":6,"href":"https:\/\/www.amzsaki.com\/index.php?rest_route=\/wp\/v2\/pages\/376\/revisions"}],"predecessor-version":[{"id":3538,"href":"https:\/\/www.amzsaki.com\/index.php?rest_route=\/wp\/v2\/pages\/376\/revisions\/3538"}],"up":[{"embeddable":true,"href":"https:\/\/www.amzsaki.com\/index.php?rest_route=\/wp\/v2\/pages\/298"}],"wp:attachment":[{"href":"https:\/\/www.amzsaki.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=376"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}