Using ImageCorrelate to Find Features and Detect Text in Images with Mathematica

When working with images, there are often times when we need to extract components from the image. Many times we already know what elements are important, we just do not know which are in a given image. An example of this in my research is extracting timestamps from microscope images. Here we will go over an algorithm I wrote in Mathematica to extract  the timestamp from microscope images when text recognition fails.

If you do not have Mathematica at home, it is well worth the price for a home copy.

The Problem

Videos and Images from our microscope have a timestamp (and other metadata) digitally imposed on top of the image (see below). It takes a long time to go through and write down which frames correspond to what real time and event, which is needed for proper analysis of the experiment.

Sample Microscope Image with overlaid MetaData

Sample Microscope Image with overlaid MetaData

Here we will focus on extracting the timestamp, but this process can be used for any object where you know what it looks like (at least on average). The timestamp extraction is a nice place to start since the numbers always look the same (they do not rotate or transform), and the format is constant. In the sample shown, the timestamp is always in the same place with a format of HH: MM: SS and date formatted as MM-DD- YY. All that will change is the actual numbers. 

1. Kernel Image (What you are extracting)

The first step is to define a set of kernel images. These are the objects that you will be looking for in the image. For the time stamp, we are looking for the numbers 0-9, so our kernel will be a set of images for each of these:

2. Using ImageCorrelate

Next we will use Mathematica's built in ImageCorrelate function to find where any of the kernel images may be in the image. Since the timestamp is always in the same place in our images, we can crop them for accuracy and time savings. We will use the following image as our sample:

Now we can use ImageCorrelate for each kernel image. We also apply ColorNegate and Binarize in order to obtain a single point where an arbitrary kernel image is most probably located. The threshold level in Binarize is the most important parameter, so make sure you play around with this value to get it working in your case. For the digits in the timestamp, we place a very high threshold of 0.989 (out of 1) to make sure we are very accurate. If we did not do this, we would have false positives, especially for the digit 1, which you can imagine would give a positive location on either the left or right side of 0 (and others).

(*Apply ImageCorrelate,and make maxima white points*)
imageCorrelations = (ImageCorrelate[image, #,  CosineDistance]//ColorNegate//Binarize[#, 0.989] &) &/@ kernelImages

What this does is, for each image in the set of kernel images, we find the CosineDistance of each point in the image we are testing, and then convert the maxima to for each into a single point. This will return a list of images with white points where each kernel was found based on the prescribed threshold.

3. Identifying Locations

Now we need to find where each of the numbers are located so we can reconstruct the timestamp. To do this, we will use the MorphologicalComponents function to identify where the pixels are located for each white dot we made at a maximum in the ImageCorrelate method. 

(* Use MorpologicalComponents to identify locations of white points *)
results = MorphologicalComponents[#] & /@ imageCorrelations;

Once we have the location of these points, we convert them to a single point for their center. We will only consider the horizontal direction in this case, but in a more general case, it is easy to look at both directions. We also add a mapping rule, so we can convert them back to the numbers each kernel represents.

(* Find where each copy is located, only caring about the horizontal location in this case *)

locationOfKernels = Table["", {x, 1, Length[kernelImages]}];

Do[
 locationOfKernels[[i]] = N@Mean[#[[All, 2]]] & /@ (Position[results[[i]], #] & /@ Range[1, Max[results[[i]]]]);
 locationOfKernels[[i]] = MapThread[Rule, {{(locationOfKernels[[i]])}, {i - 1}}];
 , {i, 1, Length[kernelImages]}]

Which will return a list with a sub list for each kernel image. Since we know the order of the kernel images, we design it so that the location for kernel image maps do the numerical value of the digit we detected. In our example here, we get:

{
  {{764.} -> 0}
  , {{1240.47, 31.} -> 1}
  , {{86.9167, 238.5, 843.308, 994.} -> 2}
  , {{1298., 466., 541.824} -> 3}
  , {{} -> 4}
  , {{313.} -> 5}
  , {{} -> 6}
  , {{} -> 7}
  , {{1069.} -> 8}
  , {{} -> 9}
}
  

4. Reconstructing the DateString

Now all we have left to do is use some logic to reconstruct the DateString. We can use our a priori knowledge of the format displayed (i.e., HH:MM:SS MM-DD-YY). First we break up the results so that each number found is its own sub-list.

numbersByLocation = Flatten[Thread[{locationOfKernels[[#, 1, 1]], locationOfKernels[[#, 1, 2]]}] & /@ Range[1, Length[locationOfKernels]], 1];

Then we sort the list by locations, and break it up into sets of two-digitl numbers, which can be combined using FromDigits.

dateString = FromDigits /@ Partition[(#[[2]] & /@ SortBy[numbersByLocation, First]), 2]

Which returns a list of:

{12, 25, 33, 2, 28, 13}

And can also be converted in Mathematica's standard format ({y,m,d,h,m,s}):

standardDateString = {#[[6]], #[[4]], #[[5]], #[[1]], #[[2]], #[[3]]} &@dateString
DateString[standardDateString]

Giving us our date!

"Thu 28 Feb 13 12:25:33"

Questions?

Let me know by leaving a comment or connecting:
YouTube Channelhttps://www.youtube.com/user/NicholasMSchneider
Twitterhttps://twitter.com/NMSchneider 
Facebookhttp://facebook.com/Schneider.NicholasM
LinkedInhttp://www.linkedin.com/in/nmschneider/
Google+https://plus.google.com/+NicholasSchneiderPennRIT/

Comment

Nicholas M Schneider

Nicholas M Schneider is a 2010 graduate from the Kate Gleason College of Engineering who is now a Doctoral Candidate at the University of Pennsylvania. Originally from an obscure town south of Buffalo, New York, he attended the Rochester Institute of Technology where he received concurrent Bachelor of Science and Master of Science degrees. While there he had a number of Co-ops including a six month stay as a Design Engineer at Lockheed Martin and Research positions with Dr. Satish Kandlikar. Nicholas currently works with Dr. Haim H Bau in the field dubbed “in situ electron microscopy of liquid systems” where he studies applications in energy and biological systems. Outside of the lab, Nicholas Schneider is a Graduate Associate in Rodin College House and enjoys running (he ran his second Philly Marathon this past November), cooking, baking, reading, and justifying his coffee addiction by making it a hobby.