Advanced search

On other websites (5645)

• Managing Music Playback Channels

  30 June 2013, by Multimedia Mike — General

  My Game Music Appreciation site allows users to interact with old video game music by toggling various channels, as long as the underlying synthesizer engine supports it.

  
  
  

  Users often find their way to the Nintendo DS section pretty quickly. This is when they notice an obnoxious quirk with the channel toggling feature: specifically, one channel doesn’t seem to map to a particular instrument or track.

  When it comes to computer music playback methodologies, I have long observed that there are 2 general strategies: Fixed channel and dynamic channel allocation.

  Fixed Channel Approach
  One of my primary sources of computer-based entertainment used to be watching music. Sure I listened to it as well. But for things like Amiga MOD files and related tracker formats, there was a rich ecosystem of fun music playback programs that visualized the music. There exist music visualization modes in various music players these days (such as iTunes and Windows Media Player), but those largely just show you a single wave form. These files were real time syntheses based on multiple audio channels and usually showed some form of analysis for each channel. My personal favorite was Cubic Player:

  
  
  

  Most of these players supported the concept of masking individual channels. In doing so, the user could isolate, study, and enjoy different components of the song. For many 4-channel Amiga MOD files, I observed that the common arrangement was to use the 4 channels for beat (percussion track), bass line, chords, and melody. Thus, it was easy to just listen to, e.g., the bass line in isolation.

  MODs and similar formats specified precisely which digital audio sample to play at what time and on which specific audio channel. To view the internals of one of these formats, one gets the impression that they contain an extremely computer-centric view of music.

  Dynamic Channel Allocation Algorithm
  MODs et al. enjoyed a lot of popularity, but the standard for computer music is MIDI. While MOD and friends took a computer-centric view of music, MIDI takes, well, a music-centric view of music.

  There are MIDI visualization programs as well. The one that came with my Gravis Ultrasound was called PLAYMIDI.EXE. It looked like this…

  
  
  

  … and it confused me. There are 16 distinct channels being visualized but some channels are shown playing multiple notes. When I dug into the technical details, I learned that MIDI just specifies what notes need to be played, at what times and frequencies and using which instrument samples, and it was the MIDI playback program’s job to make it happen.

  Thus, if a MIDI file specifies that track 1 should play a C major chord consisting of notes C, E, and G, it would transmit events “key-on C; delta time 0; key-on E; delta time 0; key-on G; delta time …; [other commands]“. If the playback program has access to multiple channels (say, up to 32, in the case of the GUS), the intuitive approach would be to maintain a pool of all available channels. Then, when it’s time to process the “key-on C” event, fetch the first available channel from the pool, mark it as in-use, play C on the channel, and return that channel to the pool when either the sample runs its course or the corresponding “key-off C” event is encountered in the MIDI command stream.

  About That Game Music
  Circling back around to my game music website, numerous supported systems use the fixed channel approach for playback while others use dynamic channel allocation approach, including evey Nintendo DS game I have so far analyzed.

  Which approach is better? As in many technical matters, there are trade-offs either way. For many systems, the fixed channel approach is necessary because for many older audio synthesis systems, different channels had very specific purposes. The 8-bit NES had 5 channels: 2 square wave generators (used musically for melody/treble), 1 triangle wave generator (usually used for bass line), a noise generator (subverted for all manner of percussive sounds), and a limited digital channel (was sometimes assigned richer percussive sounds). Dynamic channel allocation wouldn’t work here.

  But the dynamic approach works great on hardware with 16 digital channels available like, for example, the Nintendo DS. Digital channels are very general-purpose. What about the SNES, with its 8 digital channels? Either approach could work. In practice, most games used a fixed channel approach: Games might use 4-6 channels for music while reserving the remainder for various in-game sound effects. Some notable exceptions to this pattern were David Wise’s compositions for Rare’s SNES games (think Battletoads and the various Donkey Kong Country titles). These clearly use some dynamic channel approach since masking all but one channel will give you a variety of instrument sounds.

  Epilogue
  There! That took a long time to explain but I find it fascinating for some reason. I need to distill it down to far fewer words because I want to make it a FAQ on my website for “Why can’t I isolate specific tracks for Nintendo DS games?”

  Actually, perhaps I should remove the ability to toggle Nintendo DS channels in the first place. Here’s a funny tale of needless work: I found the Vio2sf engine for synthesizing Nintendo DS music and incorporated it into the program. It didn’t support toggling of individual channels so I figured out a way to add that feature to the engine. And then I noticed that most Nintendo DS games render that feature moot. After I released the webapp, I learned that I was out of date on the Vio2sf engine. The final insult was that the latest version already supports channel toggling. So I did the work for nothing. But then again, since I want to remove that feature from the UI, doubly so.

• OpenCV and Network Cameras -or- How to spy on the neighbors?

  16 May 2014, by Alexander

  ---

  A bit of context; this program was built originally to work with USB cameras - but because of the setup between where the cameras needs to be and where the computer is it makes more sense to switch to cameras run over a network. Now I’m trying to convert the program to accomplish this, but my efforts thus far have met with poor results. I’ve also asked this same question over on the OpenCV forums. Help me spy on my neighbors! (This is with their permission, of course!) :D

  ---

  I’m using:

  • OpenCV v2.4.6.0
  • C++
  • D-Link Cloud Camera 7100 (Installer is DCS-7010L, according to the instructions.)

  ---

  I am trying to access the DLink camera’s video feed through OpenCV.

  I can access the camera through it’s IP address with a browser without any issues. Unfourtunately; my program is less cooperative. When attempting to access the camera the program gives the OpenCV-generated error:

  warning: Error opening file (../../modules/highgui/src/cap_ffmpeg_impl.hpp:529)

  This error occurs with just about everything I try that doesn’t somehow generate more problems.

  For reference - the code in OpenCV’s cap_ffmpeg_impl.hpp around line 529 is as follows:

  522    bool CvCapture_FFMPEG::open( const char* _filename )
  
  523    {
  
  524        unsigned i;
  
  525        bool valid = false;
  
  526
  
  527        close();
  
  528
  
  529    #if LIBAVFORMAT_BUILD >= CALC_FFMPEG_VERSION(52, 111, 0)
  
  530        int err = avformat_open_input(&ic, _filename, NULL, NULL);
  
  531    #else
  
  532        int err = av_open_input_file(&ic, _filename, NULL, 0, NULL);
  
  533    #endif
  
  ...
  
  616    }

  ...for which I have no idea what I’m looking at. It seems to be looking for the ffmpeg version - but I’ve already installed the latest ffmpeg on that computer, so that shouldn’t be the issue.

  This is the edited down version I tried to use as per Sebastian Schmitz’s recommendation:

   1    #include <fstream>                            // File input/output
  
   2    #include <iostream>                           // cout / cin / etc
  
   3    #include                       // Windows API stuff
  
   4    #include                             // More input/output stuff
  
   5    #include <string>                         // "Strings" of characters strung together to form words and stuff
  
   6    #include <cstring>                            // "Strings" of characters strung together to form words and stuff
  
   7    #include <streambuf>                      // For buffering load files
  
   8    #include <array>                          // Functions for working with arrays
  
   9    #include <opencv2></opencv2>imgproc/imgproc.hpp>        // Image Processor
  
  10    #include <opencv2></opencv2>core/core.hpp>          // Basic OpenCV structures (cv::Mat, Scalar)
  
  11    #include <opencv2></opencv2>highgui/highgui.hpp>        // OpenCV window I/O
  
  12    #include "opencv2/calib3d/calib3d.hpp"
  
  13    #include "opencv2/features2d/features2d.hpp"
  
  14    #include "opencv2/opencv.hpp"
  
  15    #include "resource.h"                     // Included for linking the .rc file
  
  16    #include                             // For sleep()
  
  17    #include <chrono>                         // To get start-time of program.
  
  18    #include <algorithm>                      // For looking at whole sets.
  
  19
  
  20    #ifdef __BORLANDC__
  
  21      #pragma argsused
  
  22    #endif
  
  23
  
  24    using namespace std;                      // Standard operations. Needed for most basic functions.
  
  25    using namespace std::chrono;              // Chrono operations. Needed getting starting time of program.
  
  26    using namespace cv;                           // OpenCV operations. Needed for most OpenCV functions.
  
  27
  
  28    string videoFeedAddress = "";
  
  29    VideoCapture videoFeedIP = NULL;
  
  30    Mat clickPointStorage; //Artifact from original program.
  
  31
  
  32    void displayCameraViewTest()
  
  33    {
  
  34      VideoCapture cv_cap_IP;
  
  35      Mat color_img_IP;
  
  36      int capture;
  
  37      IplImage* color_img;
  
  38      cv_cap_IP.open(videoFeedAddress);
  
  39      Sleep(100);
  
  40      if(!cv_cap_IP.isOpened())
  
  41      {
  
  42          cout &lt;&lt; "Video Error: Video input will not work.\n";
  
  43          cvDestroyWindow("Camera View");
  
  44          return;
  
  45      }
  
  46      clickPointStorage.create(color_img_IP.rows, color_img_IP.cols, CV_8UC3);
  
  47      clickPointStorage.setTo(Scalar(0, 0, 0));
  
  48      cvNamedWindow("Camera View", 0); // create window
  
  49      IplImage* IplClickPointStorage = new IplImage(clickPointStorage);
  
  50      IplImage* Ipl_IP_Img;
  
  51      
  
  52      for(;;)
  
  53      {
  
  54          cv_cap_IP.read(color_img_IP);
  
  55          IplClickPointStorage = new IplImage(clickPointStorage);
  
  56          Ipl_IP_Img = new IplImage(color_img_IP);
  
  57          cvAdd(Ipl_IP_Img, IplClickPointStorage, color_img);
  
  58          cvShowImage("Camera View", color_img); // show frame
  
  59          capture = cvWaitKey(10); // wait 10 ms or for key stroke
  
  60          if(capture == 27 || capture == 13 || capture == 32){break;} // if ESC, Return, or space; close window.
  
  61      }
  
  62      cv_cap_IP.release();
  
  63      delete Ipl_IP_Img;
  
  64      delete IplClickPointStorage;
  
  65      cvDestroyWindow("Camera View");
  
  66      return;
  
  67    }
  
  68    
  
  69    int main()
  
  70    {
  
  71      while(1)
  
  72      {
  
  73          cout &lt;&lt; "Please Enter Video-Feed Address: ";
  
  74          cin >> videoFeedAddress;
  
  75          if(videoFeedAddress == "exit"){return 0;}
  
  76          cout &lt;&lt; "\nvideoFeedAddress: " &lt;&lt; videoFeedAddress &lt;&lt; endl;
  
  77          displayCameraViewTest();
  
  78          if(cvWaitKey(10) == 27){return 0;}
  
  79      }
  
  80      return 0;
  
  81    }
  
  </algorithm></chrono></array></streambuf></cstring></string></iostream></fstream>

  Using added ’cout’s I was able to narrow it down to line 38: "cv_cap_IP.open(videoFeedAddress);"

  No value I enter for the videoFeedAddress variable seems to get a different result. I found THIS site that lists a number of possible addresses to connect to it. Since there exists no 7100 anywhere in the list & considering that the install is labeled "DCS-7010L" I used the addresses found next to the DCS-7010L listings. When trying to access the camera most of them can be reached through the browser, confirming that they reach the camera - but they don’t seem to affect the outcome when I use them in the videoFeedAddress variable.

  I’ve tried many of them both with and without username:password, the port number (554), and variations on ?.mjpg (the format) at the end.

  I searched around and came across a number of different "possible" answers - but none of them seem to work for me. Some of them did give me the idea for including the above username:password, etc stuff, but it doesn’t seem to be making a difference. Of course, the number of possible combinations is certainly rather large- so I certainly have not tried all of them (more direction here would be appreciated). Here are some of the links I found:

  1. This is one of the first configurations my code was in. No dice.
  2. This one is talking about files - not cameras. It also mentions codecs - but I wouldn’t be able to watch it in a web browser if that were the problem, right? (Correct me if I’m wrong here...)
  3. This one has the wrong error code/points to the wrong line of code!
  4. This one mentions compiling OpenCV with ffmpeg support - but I believe 2.4.6.0 already comes with that all set and ready! Otherwise it’s not that different from what I’ve already tried.
  5. Now THIS one appears to be very similar to what I have, but the only proposed solution doesn’t really help as I had already located a list of connections. I do not believe this is a duplicate, because as per THIS meta discussion I had a lot more information and so didn’t feel comfortable taking over someone else’s question - especially if I end up needing to add even more information later.

  Thank you for reading this far. I realize that I am asking a somewhat specific question - although I would appreciate any advice you can think of regarding OpenCV & network cameras or even related topics.

  ---

  TLDR: Network Camera and OpenCV are not cooperating. I’m unsure if
  it’s the address I’m using to direct the program to the camera or the
  command I’m using - but no adjustment I make seems to improve the
  result beyond what I’ve already done! Now my neighbors will go unwatched!

• OpenCV and Network Cameras -or- How to spy on my neighbors?

  12 May 2014, by Alexander

  ---

  A bit of context; this program was built originally to work with USB cameras - but because of the setup between where the cameras needs to be and where the computer is it makes more sense to switch to cameras run over a network. Now I’m trying to convert the program to accomplish this, but my efforts thus far have met with poor results. I’ve also asked this same question over on the OpenCV forums. Help me spy on my neighbors! (Totally with their permission, of course!) :D

  ---

  I’m using:

  • OpenCV v2.4.6.0
  • C++
  • D-Link Cloud Camera 7100 (Installer is DCS-7010L, according to the instructions.)

  ---

  I am trying to access the DLink camera’s video feed through OpenCV.

  I can access the camera through it’s IP address with a browser without any issues. Unfourtunately; my program is less cooperative. When attempting to access the camera the program gives the OpenCV-generated error:

  warning: Error opening file (../../modules/highgui/src/cap_ffmpeg_impl.hpp:529)

  This error occurs with just about everything I try that doesn’t somehow generate more problems.

  For reference - the code in OpenCV’s cap_ffmpeg_impl.hpp around line 529 is as follows:

  522    bool CvCapture_FFMPEG::open( const char* _filename )
  
  523    {
  
  524        unsigned i;
  
  525        bool valid = false;
  
  526
  
  527        close();
  
  528
  
  529    #if LIBAVFORMAT_BUILD >= CALC_FFMPEG_VERSION(52, 111, 0)
  
  530        int err = avformat_open_input(&ic, _filename, NULL, NULL);
  
  531    #else
  
  532        int err = av_open_input_file(&ic, _filename, NULL, 0, NULL);
  
  533    #endif
  
  ...
  
  616    }

  ...for which I have no idea what I’m looking at. It seems to be looking for the ffmpeg version - but I’ve already installed the latest ffmpeg on that computer, so that shouldn’t be the issue.

  This is the edited down version I tried to use as per Sebastian Schmitz’s recommendation:

   1    #include <fstream>                            // File input/output
  
   2    #include <iostream>                           // cout / cin / etc
  
   3    #include                       // Windows API stuff
  
   4    #include                             // More input/output stuff
  
   5    #include <string>                         // "Strings" of characters strung together to form words and stuff
  
   6    #include <cstring>                            // "Strings" of characters strung together to form words and stuff
  
   7    #include <streambuf>                      // For buffering load files
  
   8    #include <array>                          // Functions for working with arrays
  
   9    #include <opencv2></opencv2>imgproc/imgproc.hpp>        // Image Processor
  
  10    #include <opencv2></opencv2>core/core.hpp>          // Basic OpenCV structures (cv::Mat, Scalar)
  
  11    #include <opencv2></opencv2>highgui/highgui.hpp>        // OpenCV window I/O
  
  12    #include "opencv2/calib3d/calib3d.hpp"
  
  13    #include "opencv2/features2d/features2d.hpp"
  
  14    #include "opencv2/opencv.hpp"
  
  15    #include "resource.h"                     // Included for linking the .rc file
  
  16    #include                             // For sleep()
  
  17    #include <chrono>                         // To get start-time of program.
  
  18    #include <algorithm>                      // For looking at whole sets.
  
  19
  
  20    #ifdef __BORLANDC__
  
  21      #pragma argsused
  
  22    #endif
  
  23
  
  24    using namespace std;                      // Standard operations. Needed for most basic functions.
  
  25    using namespace std::chrono;              // Chrono operations. Needed getting starting time of program.
  
  26    using namespace cv;                           // OpenCV operations. Needed for most OpenCV functions.
  
  27
  
  28    string videoFeedAddress = "";
  
  29    VideoCapture videoFeedIP = NULL;
  
  30    Mat clickPointStorage; //Artifact from original program.
  
  31
  
  32    void displayCameraViewTest()
  
  33    {
  
  34      VideoCapture cv_cap_IP;
  
  35      Mat color_img_IP;
  
  36      int capture;
  
  37      IplImage* color_img;
  
  38      cv_cap_IP.open(videoFeedAddress);
  
  39      Sleep(100);
  
  40      if(!cv_cap_IP.isOpened())
  
  41      {
  
  42          cout &lt;&lt; "Video Error: Video input will not work.\n";
  
  43          cvDestroyWindow("Camera View");
  
  44          return;
  
  45      }
  
  46      clickPointStorage.create(color_img_IP.rows, color_img_IP.cols, CV_8UC3);
  
  47      clickPointStorage.setTo(Scalar(0, 0, 0));
  
  48      cvNamedWindow("Camera View", 0); // create window
  
  49      IplImage* IplClickPointStorage = new IplImage(clickPointStorage);
  
  50      IplImage* Ipl_IP_Img;
  
  51      
  
  52      for(;;)
  
  53      {
  
  54          cv_cap_IP.read(color_img_IP);
  
  55          IplClickPointStorage = new IplImage(clickPointStorage);
  
  56          Ipl_IP_Img = new IplImage(color_img_IP);
  
  57          cvAdd(Ipl_IP_Img, IplClickPointStorage, color_img);
  
  58          cvShowImage("Camera View", color_img); // show frame
  
  59          capture = cvWaitKey(10); // wait 10 ms or for key stroke
  
  60          if(capture == 27 || capture == 13 || capture == 32){break;} // if ESC, Return, or space; close window.
  
  61      }
  
  62      cv_cap_IP.release();
  
  63      delete Ipl_IP_Img;
  
  64      delete IplClickPointStorage;
  
  65      cvDestroyWindow("Camera View");
  
  66      return;
  
  67    }
  
  68    
  
  69    int main()
  
  70    {
  
  71      while(1)
  
  72      {
  
  73          cout &lt;&lt; "Please Enter Video-Feed Address: ";
  
  74          cin >> videoFeedAddress;
  
  75          if(videoFeedAddress == "exit"){return 0;}
  
  76          cout &lt;&lt; "\nvideoFeedAddress: " &lt;&lt; videoFeedAddress &lt;&lt; endl;
  
  77          displayCameraViewTest();
  
  78          if(cvWaitKey(10) == 27){return 0;}
  
  79      }
  
  80      return 0;
  
  81    }
  
  </algorithm></chrono></array></streambuf></cstring></string></iostream></fstream>

  Using added ’cout’s I was able to narrow it down to line 38: "cv_cap_IP.open(videoFeedAddress);"

  No value I enter for the videoFeedAddress variable seems to get a different result. I found THIS site that lists a number of possible addresses to connect to it. Since there exists no 7100 anywhere in the list & considering that the install is labeled "DCS-7010L" I used the addresses found next to the DCS-7010L listings. When trying to access the camera most of them can be reached through the browser, confirming that they reach the camera - but they don’t seem to affect the outcome when I use them in the videoFeedAddress variable.

  I’ve tried many of them both with and without username:password, the port number (554), and variations on ?.mjpg (the format) at the end.

  I searched around and came across a number of different "possible" answers - but none of them seem to work for me. Some of them did give me the idea for including the above username:password, etc stuff, but it doesn’t seem to be making a difference. Of course, the number of possible combinations is certainly rather large- so I certainly have not tried all of them (more direction here would be appreciated). Here are some of the links I found:

  1. This is one of the first configurations my code was in. No dice.
  2. This one is talking about files - not cameras. It also mentions codecs - but I wouldn’t be able to watch it in a web browser if that were the problem, right? (Correct me if I’m wrong here...)
  3. This one has the wrong error code/points to the wrong line of code!
  4. This one mentions compiling OpenCV with ffmpeg support - but I believe 2.4.6.0 already comes with that all set and ready! Otherwise it’s not that different from what I’ve already tried.
  5. Now THIS one appears to be very similar to what I have, but the only proposed solution doesn’t really help as I had already located a list of connections. I do not believe this is a duplicate, because as per THIS meta discussion I had a lot more information and so didn’t feel comfortable taking over someone else’s question - especially if I end up needing to add even more information later.

  Thank you for reading this far. I realize that I am asking a somewhat specific question - although I would appreciate any advice you can think of regarding OpenCV & network cameras or even related topics.

  ---

  TLDR: Network Camera and OpenCV are not cooperating. I’m unsure if
  it’s the address I’m using to direct the program to the camera or the
  command I’m using - but no adjustment I make seems to improve the
  result beyond what I’ve already done! Now my neighbors will go unwatched!