Processing movie (AEV, MP4) in KNIME

[s.roughley]

I’ve been trying to see if there is a way of loading movies (AEV or MP4 format) into knime, and then being able to access each frame as an individual image in a stack?

I’ve tried a few ‘obvious’ things but not had any joy - is it possible?

Thanks

Steve

[Andrew_Steel]

Hi @s.roughley,

since there is no video node you could use opencv-python in a python node.

Best Regards
Andrew

[christian.birkhold]

You can try to load the videos with the Image Reader. The image reader can load n-dimensional images (for RGB videos X,Y,Channel,Time would be the dimensions of the image). Depending on the codec of the videos, it could work.

[s.roughley]

Thanks Christian - sadly this easy approach did not work. @Andrew_Steel - I will take a look at the OpenCV library - I think there is also a Java version so may be able to write a node to do the job

Thanks

Steve

[s.roughley]

So resurrecting this thread and investigating @Andrew_Steel suggestion, opencv will do the job. The following python code saves every nth image to a temp folder, having scaled it down. The scale factor, output format (png or jpg) and n are determined by user inputs:

# Copy input to output
output_table_1 = input_table_1.copy()

import cv2
from io import BytesIO

imgCol = []
i = 0

for x in range(len(input_table_1['Path'])):
    cap = cv2.VideoCapture(input_table_1['Path'][x])
    images = []
    while(cap.isOpened()):
        ret, frame = cap.read()
        if ret == False:
            break
        outPath = flow_variables['temp_dir_path_location']+"/" + input_table_1['outName'][x] + "_" + str(i) +"." + flow_variables['imgFormat']
        i = i + 1
        if i % flow_variables['nthFrame'] == 0:
            newDim = (int(frame.shape[1] * flow_variables['thumbnailScale'] / 100), int(frame.shape[0] * flow_variables['thumbnailScale'] / 100))
            frame = cv2.resize(frame, newDim, interpolation=cv2.INTER_AREA) 
            cv2.imwrite(outPath, frame)
            images.append(outPath)
        #is_success, buffer = cv2.imencode(".png", frame)
        #io_buf = BytesIO(buffer)
        #images = io_buf.getvalue()
    imgCol.append(images)


output_table_1['frames']=imgCol

Notes:

1. The commentated out code was me trying to write the blobs directly to the output table which KNIME can’t do yet!
2. Apologies to the pythonistas out there who could probably make this much more elegant - I inhabit the java world by day and cobbled this together from various examples and looking up things that are probably elementary in python-speak

Thanks again,

Steve

PS - anyone want to flick through several thousand thumbnails to spot the interesting wildlife that the webcam caught in amidst the videos of blowing leaves, rain etc etc?!

[badger101]

Hi Steve, thanks for sharing the solution. If you have a paid Cloud Vision account, and just in case if you want to detect animals caught in your wildlife webcam, it’s possible with Knime.

A free version account of the API can process up to 1000 images.

Here’s a Medium article you might find interesting. It’s for both color detection & label detection.

Use Google Vision with KNIME for Image Feature Mining

@s.roughley

[s.roughley]

Thanks @badger101 - at the last run of this workflow it found around 70,000 movie clips to process!

I’ve augmented it a bit with an added piece where I highlight the differences between the 0th frame and each other, along with the number of distinct change areas, the total changing area and the proportion of the image changing using opencv (I did this in a second python snippet using the frames output from the first - I’m sure it will be more efficient to simply merge into one snippet eventually!):

(This is in a python snippet with 2 input ports - the first has the 0th frame for the movie, the 2nd all the others)

# Copy input to output
output_table_1 = input_table_2.copy()

import cv2
import imutils
from skimage.metrics import structural_similarity as compare_ssim
import numpy as np

#Load the base image
imgBase = cv2.imread(input_table_1['frames'][0])
imgBaseGrey = cv2.cvtColor(imgBase, cv2.COLOR_BGR2GRAY)

minContourSize = flow_variables['minDiffArea']
scores = []
numDiffs = []
areas = []
comps = []
diffs = []
thresholds = []
totalAreas = []
pcChanges = []

for i in range(len(input_table_2['frames'])):
	imgComp = cv2.imread(input_table_2['frames'][i])
	imgCompGrey = cv2.cvtColor(imgComp, cv2.COLOR_BGR2GRAY)
	
	# compute the Structural Similarity Index (SSIM) between the two
	# images, ensuring that the difference image is returned
	(score, diff) = compare_ssim(imgBaseGrey, imgCompGrey, full=True)
	diff = (diff * 255).astype("uint8")
	scores.append(score)
	
	# threshold the difference image, followed by finding contours to
	# obtain the regions of the two input images that differ
	thresh = cv2.threshold(diff, 0, 255,
		cv2.THRESH_BINARY_INV | cv2.THRESH_OTSU)[1]
	cnts = cv2.findContours(thresh.copy(), cv2.RETR_EXTERNAL,
		cv2.CHAIN_APPROX_SIMPLE)
	cnts = imutils.grab_contours(cnts)
	
	# Create a mask and draw the contour bounding boxes on it
	mask = np.zeros(imgComp.shape[:2],dtype=np.uint8)
	for c in cnts:
		if cv2.contourArea(c) > minContourSize:
			(x, y, w, h) = cv2.boundingRect(c)
			cv2.rectangle(mask, (x, y), (x + w, y + h), (255), -1)
	# Now re-countour on the mask
	cnts = cv2.findContours(mask, cv2.RETR_EXTERNAL,
		cv2.CHAIN_APPROX_SIMPLE)
	cnts = imutils.grab_contours(cnts)
	
	numDiffs.append(len(cnts))

	# loop over the contours
	imgAreas = []
	for c in cnts:
		imgAreas.append(cv2.contourArea(c))
		cv2.drawContours(imgComp, [c], 0, (0, 0, 255), 2)
	areas.append(imgAreas)
	totalArea = sum(imgAreas)
	totalAreas.append(totalArea)
	pcChanges.append(100.0 * totalArea / np.prod(imgComp.shape[:2]))
	    
	# show the output images
	outPath = input_table_2['frames'][i].replace("."+flow_variables['imgFormat'], "_")
	compPath = outPath +"comp."+flow_variables['imgFormat']
	cv2.imwrite(compPath, imgComp)
	comps.append(compPath)


output_table_1['Scores']=scores
output_table_1['Number of Diffs'] = numDiffs
output_table_1['Areas'] = areas
output_table_1['Total Area'] = totalAreas
output_table_1['% Change'] = pcChanges
output_table_1['Boxed'] = comps

The inspiration for the above came from 2 really helpful sources:
Firstly, Image Difference with OpenCV and Python - PyImageSearch showed how to compare and add a simple bounding rectangle - but sometimes there were overlapping rectangles, and rectangles within other rectangles.
The second showed how to combine the rectangles neatly and relatively efficiently - python - OpenCV join contours when rectangle overlaps another rect - Stack Overflow

I can then filter the number of frames I have to look at - those with large numbers of changing areas or a high overall %age change are generally things like the light catching the background or the wind blowing bushes…

Who knows, maybe I will add in something like this next - Detecting animals in the backyard — practical application of deep learning. | by Gaiar Baimuratov | Towards Data Science ?!

Steve