Using animation callbacks to stabilize video

Overview

This is a sample Python script that utilizes Opticks, the Python Scripting Extension and the OpenCV Extension Opticks extensions. This script performs video stabilization.

Setting up user.py

The user.py script will load every time you start Opticks with the Python extension. It is a convenient way to configure your Python environment. You can locate the script by loading Opticks and bringing up Tools->Options. Look in Scripting->Python Engine->Python User Configuration File Location option. Edit this file, creating a new one if it does not already exist. Add the following to your user.py. This will add the location of stabilize.py to your Python Path so you can import the module. This location can be used as a temporary working directory for writing your own scripts. After editing user.py save it to disk and restart Opticks.

Loading the data and setting up Opticks

The technique presented here is not a very robust stabilization technique and is meant to be a demonstration only. It will
not successfully stabilize more than about 5 frames of data in the sample data set so we will load just a few frames.

1. Select File->Import Data... and navigate to hicam_catch-gray.re.ice.h5.
2. Select Options dialog.
3. On the Subset tab, highlight only bands 56 through 60 to import just a subset of the data.
4. Select Ok then Open
5. Once the data loads, select View->Create Animation
6. Play through the video frames once or twice and notice that the sensor platform is moving so the background moves relative to the camera.

The data must be loaded in-memory or on-disk (not on-disk read-only) since we will modify the frames in place. This is more efficient then generating a new, stabilized video. Also, ensure the data is loaded in BSQ interleave. BSQ is the most efficient interleave for working with video and the example code expects that you have loaded BSQ data. In-memory BSQ will be the default options so you should not need to make any changes to the import options.

The Script

We'll look at the stabilize.py script and discuss it in this section. We will detect the sensor motion and remove it to stabilize the background. There are a number of utility functions which are put together to stabilize the image. We'll discuss each below. The kdtree.py file comes from the python-kdtree project. It is a very basic kd-tree data structure used to locate points which are geometrically close to a location. This implementation is very basic and easy to understand but is not very efficient. It should not be used for production code.

This preamble imports needed modules inluding OpenCV and Numpy.

The calc_pts() function uses the SURF algorithm to extracts good points in the image to track. These are likely to be strong and distinguishable in future frames. See the SURF website for more information on the algorithm. The ExtractSURF() function returns much more data than we need for this example so we throw out everything except the (x,y) coordinates at the center of the features.

Once we have a list of features in two consecutive frames, we attempt to match the features up. We'll iterate through all the features in one frame and locate the geometrically closest feature in the other frame. This is not a very good method and is likely to generate a lot of outliers but we'll use the RANSAC algorithm to clean up the data.

The RANSAC algorithm is used to determine the affine (1^st^ order polynomial) translation needed to line the second frame up with the first frame. RANSAC is a good algorithm when there are outliers in the location correspondences. Briefly, the algorithm randomly selects the number of correspondences needed to calculate an affine transform - three - and generates the transform matrix. All of the remaining points are transformed using this matrix. If the transformed location is the same as the expected location of the corresponding feature (within an error threshold) then we consider the transform matrix as a good match to that point. We repeat this a number of times and take the transform matrix with the most matches. This function demonstrates use of the OpenCV library from Python and Opticks. I won't explain all the details so you should reference the OpenCV manual for details on all of the OpenCV function calls.

This function takes a Numpy array and uses OpenCV to warp the image in that array using a specified transformation matrix. This is used to align one frame to another once we have a transform matrix calculated using RANSAC.

This utility function takes frame data which is multi-byte and rescales it to one byte data since the OpenCV SURF implementation requires one byte data.

This function is used to correct a frame of video. It calculates SURF feature points for the frame, aligns them to the SURF feature points in the base frame, calculates the affine transform needed to align the frames, and then it aligns them. The re.update() on line 74 is necessary to refresh the Opticks layer so that the new data is displayed. If you are processing multiple frames at one time, instead of processing a frame and then displaying it, this method should be called after all the frames are processed as it is more computationally efficient.

This function calculates the SURF points for the base frame (frame 0) and indexes them using a kd-tree. It then registers an animation callback which will execute processframe() each time a new frame of the video is displayed. This assumes we have created a 1fps animation using View->Create Animation.

Bug Workaround The code on lines 87-90 is only necessary due to a deficiency in version 1.2.0 of the Python interface. This requires that we pass frame times even if you don't need them.

Finally, the cleanup() function unregisters the animation callback. We call this once we have stabilized all the frames.

Using the code

Now we'd like to stabilize the video we loaded earlier. Open the Python scripting window and type:

This will import the script. If there are any errors locating the module, make sure you setup your user.py file to point to the directory where stabilize.py and kdtree.py live and that you restarted Opticks so the changes take effect.

Next we need to access the raster data for the video and the animation. We'll initialize the stabilization routines as well. Make sure the video is set to frame 0 then type the following into the scripting window to initialize the active animation and the active video.

Now we can play through the animation once to stabilize it. Ensure the the animation is set to play once through the animation then begin playback.

When that is complete, detach the stabilization callback, set the animation to replay continuously, and then play the animation. Notice that the background is stabilized.

Finally, you'll notice that the grayscale intensities change. This is because when we align the frames, there is missing data on the edges which is filled in with black pixels. These are throwing off the statistics used to display the frames. We can fix this by opening the Histogram Window, right clicking in the window and selecting Bad Values.... Add 0 to the bad values list, select the check box to set this for all frames, and then hit{{Ok}}. This will ignore the padding around the edges and the frames intensities should be the same.