Matlab to Knime image transfer

KNIME Analytics Platform
1

Hi,

for my current workflow modelling, I'm trying to pass image data from MATLAB to KNIME, using the MATLAB Snippet node. The image stack in MATLAB is consisting of ~1000 frames with a spatial resolution of 640x480 pixel. I've started experimenting with only 100 frames of them. The data is in MATLAB converted into a 2D dataset that contains 640x480=307200 rows and 100 (number of images) columns. The number format is double. All in all the resulting dataset is ~300MB in MATLAB workspace. My problem is, that the execution of the MATLAB snippet node is very slow in KNIME, I need about two minutes until the node execution is finished, from which only 1 minute is the actual execution time of the MATLAB script (many file I/Os). So the actual transfer time or overhead for this 300MB dataset seems to be 1 minute which seems to be a lot of time for transferring this small image stack of 100 frames, especially since my actual aim is to transfer ten times more image data. Is there any chance to speed up the whole process? I've already changed the memory policy from the MATLAB snippet node to "keep all in memory" but this has not a big impact. Directly reading the files is unfortunately not possible, since they are in HDF5 format and KNIME is unable to read it.

Any help would be nice!

2

I've never got the MATLAB script node to work successfully, but it does appear to launch a new MATLAB instance on each run - are you sure it's not that process that's taking the extra time, rather than the data transfer? If so I think Mathworks has some tips on how to speed up MATLAB startup.

If you can't convert your files to a different format in MATLAB, you could use (for example) a Python script node to read the HDF5 file using the h5py package which should be part of the Anaconda Python distribution.

 

3

Hi, thank you for your answer. The MATLAB node is not starting a new instance everytime, only on it's first execution. Then it always uses the already opened MATLAB instance. So I'm pretty sure that the time is needed to transfer the data which is a bit annoying... Python would be an alternative for reading hdf5 but I do have indeed a mixed set of scripts for image processing, some are in MATLAB some are in python and it would have been nice, if I could have used KNIME to automate processes and to transfer data between the single scripts. Unfortunately the slow data transfer from and to KNIME is really a big problem for my workflow...

4