List and read .table files from the server ignoring empty ones

Hello,

I have created a bunch of .table files on the server. I listed the files and tried to loop over and read them one by one, but apparently some can be empty, have no rows at all, which makes the Table Reader to fail stating ‘Cannot read file! This file is either not in KNIME table format or corrupted!’ The try-catch procedure does not work in such a configuration issue. I then tried to use the File Meta Info in order to identify the files in the list that appear too small to be meaningful, but haven’t been successful extracting the meta info (all variables shown as missing), I guess because the files are on the server. Any suggestions how to solve the issue?

Hi @pzkor

Reading empty KNIME tables from the server works fine for me.
How did you create the .table files? Please note that simply changing the ending doesn’t work, they need to be created using the Table Writer node.

Best,
Alice

1 Like

Hi Alice,

Thanks for your response. Let me share an example with you. The workbook is set to read two of such files generated by the Table Writer node based on a Twitter query. One works and the other one doesn’t.Table file not working.knwf (24.0 KB)

Hi @pzkor

would you maybe have the two files for me as well?

Hi! They should be in the /data folder of the workflow.

Thanks, got mislead by the path set in the reader node. I see what you mean. So this is consistent behavior, that you encounter with any table that is 0 kb?

Yes, I don’t fully understand it though because I had controlled for empty tables. image

In any case, it was not a good idea to create a file for each individual search term and I have now changed the workflow so that a file is only written at the end.

But I have thousands of historically extracted files on the server which I want to load in. And somehow automatically leave aside those that are causing problems. As said earlier, I thought the options are an error management by a Try Catch or identifying the 0KB files from the list before starting to loop over, but I don’t know how to get either of them to work.

Have you tried the List Files/Folders node to get the list of files on the server, followed by a Files/Folders Meta Info node, filter out the 0kb files and then read in the other files in a loop?

Hi @Alice_Krebs

Yes I have. It runs, but the variables are all shown as missing. I am listing the files on the server from Desktop.
image image

Which KNIME version are you on? And why does it have “EXT: …” in the path?

I am on Desktop 4.2.2. The EXT gets created by the String to URI node. The File Meta Info node only accepts URI type references.

As I am connecting to files on the server, the string I am converting into a URI is of the form:

knime://knime.server.name/folder/file.table

Hi @pzkor
Ah, I see.
I downloaded 4.2.2 to understand what’s going on, and I see your pain! As a quick fix I can recommend you to upgrade to 4.3 if that is suitable for you. That’s the version I worked on at first, and the new file handling spares all the URI trouble. Simply two nodes, done. Let me know if that could work for you.

Ok, that’s good to know!

I don’t dare to upgrade the Desktop until the server has been upgraded and we won’t be doing that until the first patch version is out. But assuming that is just a matter of a couple of weeks, I can pause this work in here for now.

Thanks for your help!