I have just found an old posting about somebody else asking for HDF5 support, but without any answer. Since HDF5 is supported by Matlab, ImageJ, C, and IDL, is there any support by KNIME available to open HDF5 files with KNIME?
Regards,
gerhard
1 Like
Hi Gerhard,
this is a tricky question. We could easily (and we already did that internally actually for some formats) write a dedicated HDF5 opener for KNIME given a format. However, HDF5 iteself is not really a format definition, rather a container where arbitrary formats can be defined (e.g. HDF5 SPIM Format for the BigDataViewer, HDF5 format of Ilastik etc etc). One could write a generic HDF5 Viewer where you can select some data sets which are then interpreted as e.g. images under the assumption that the format is e.g. raw flat iteration order data etc. But this node doesn't exist, yet.
Which format do you want to read in in particular?
Best,
Christian
Hi Christian,
well it's based on the situation we have in the company. We test all our cameras under different conditions and store the image data, the EMVA1288 processed results, the test conditions an everything in huge HDF5 files, because we have decided a couple of years ago, that instead have everything separately stored and remember the paths, it is easier to archive that way, whenever server are changed. Now, sometime it occurs that a year later we would like to know some statistics, which have not been foreseen before, let's say hwo many image sensor sin the cameras had bad clusters with a size larger than x. Then we would have to access a range of these HDF5 files, open, readout this parameter or evaluate it, and process this to create an overview graph or table.
I just learned last week about Knime, but thought it could be elegantly used for such a task. That's why the question arose.
Best,
gerhard
Hi Gerhard,
yes, this sounds reasonable and like a use-case for KNIME. Writing the corresponding HDF5 reader for your format using libraries like https://wiki-bsse.ethz.ch/pages/viewpage.action;jsessionid=3F148352DFB9C0CFDCB0986C566A3806?pageId=26609113 is straightforward. Here is an example where we read another HDF5 format: https://github.com/knime-ip/knip-ilastik/tree/hdf5Reader/org.knime.knip.ilastik/src/org/knime/knip/ilastik/nodes/hdf5reader.
I hope this helps you getting started. Feel free to reach out to me here if you need more information or help to set things up.
Best,
Christian
PS: You could also use the KNIME Python integration to access the HDF5 file, which maybe easier as a first step.
Hi Christian, hi Gerhard,
wouldn't it be better to have SCIFIO properly supporting those hdf5 files? I did some preliminary work on a h5 reader implemented as a SCIFIO format on the recent hackathon. I didn't find the time to continue this yet, but it's on my list, because hdf5 support for files created by Huygens Pro or ilastik would benefit both ImageJ and KNIME worlds.
Cheers,
Jan
Hi Jan,
since I have no experience with SCIFIO I can't answer the question properly. Generally I would support and appreciate the HDF5 format, because i am still convinced, that HDF5 is y very powerful way of storing different types of data in the same file and use a good internal structure.
Cheers,
Gerhard
Hi, all,
Is there any progress about this subject ?
I’m particularly interested in this part. Can you give more details about this part ?
Did you mean to write python script to read HDF5 or you already have python node for hdf5 reading ?
I can not find the node in the current version of knime with Python integration installed.
Cheers,
Tong
Hi @TongLI,
As @christian.dietz has tried to explain in his first post, HDF5 is a very flexible container for data and not a classical format with defined syntax. He was suggesting to use the h5py library (or an alternative one) with our Python Integration, much like @DaveK is showing in this workflow:
Best,
Stefan
2 Likes
Thank you for your explanation. I understood what crhistian wanted to say.
However, it’s always possible to write a node that do the basic job as load and proposing all the internal paths, isn’t it ?
Its several lines of code. I wanted to create one with Python, but it seems impossible to write a normal node (such as file loader that is Java-based) with Python.
Edit: I just saw the workflow of davek. Fianlly, it’s more complex than I thought. 
Best
Tong
Well, it is complex because of the flexible nature of HDF5. You could hardcode a lot of the information, if you are only looking to open only a specific kind of HDF5…
Hi,
if you dont mind me asking, is there a plan to create an hdf5-reader node in the near future?