List Files/Folders missing files

Good morning

Version: KNIME 4.3

One of my colleagues has spotted some odd behaviour with the List Files/Folders node when using a windows based network share.

Example path: \data\myfoldershare\sub folder\

The List Files/Folders node find 12 files in this folder but should find 79.

If I run a Java snippet with a simple File#listFiles() call then all 79 files are found, so our initial thought of it being a network/VPN glitch doesn’t look right.

If we copy the files locally the List Files/Folders node finds them all.

If there’s something I can do to provider a more useful bug report let me know, not sure how to provide you a reproducible example in this scenario. But if you tell me what class to look in I’m happy to step through the code.

Cheers

Sam

P.S. This component finds all the files: List Files – KNIME Hub

I did some digging, the problem seems to be that some of the files return false to the BasicFileAttributes#isRegularFile() call which means they don’t pass the filtering in the visitFile call in

Seems all the files that are missing return true to BasicFileAttributes#isOther and seem to show a ‘size on disk’ of 0 bytes.

Though I have no problem reading or copying these files. Looks like it may be more to do with something in our IT infrastructure than an issue in KNIME.

Cheers

Sam

1 Like

From further investigation the files filtered out have the ReparsePoint attribute. I don’t know why some files have this attribute but presumably it’s something to do with the underlying filesystem on our network share.

I think my conclusion is we’d like to have an option in the node to allow files that return true to isOther() to pass the filter and be listed. Maybe extending this so also optionally include symbolic links?

Cheers

Sam

Hi @swebb,

you are right “special” folders/files are currently not listed by the List Files/Folders node.
We have a ticket to add an option to enable the listing of these objects.
Added a +1 there.

Cheers,
Julian

3 Likes

Thanks Julian :), we’ll use a workaround in the meantime.

Hi Sam,
we have added a new option to the Filter options that allows you to list also special files. THe option is enabled by default for newly dragged nodes.

This is available in the nightly build and it would be great if you could try it and let us know if it fixes your problem. If no problems come up with this new option it will be released with the 4.3.3 release.
Thanks
Tobias

4 Likes

Hi Tobais

Thanks for the update, I can confirm the nightly build is finding the files with the ‘l’ attribute now :slight_smile:

KNIME 4.4:

KNIME 4.3:

image

Cheers

Sam

3 Likes

Thanks a lot for the prompt testing :slight_smile:

Quick flow up. Where you able to read those files?

Best,
Mark

I didn’t check that but I just have and KNIME still reads them (it always could it just wouldn’t list them).

The CSV seems to load in fine. Though I did observe that whatever the node is doing when the dialog says scanning seems to cause a change in the flags set on the files. For example:

  1. List the directory

  1. Run the List Files node or put the path into the dialog and let it ‘scan’

  2. List the directory again

So by the time it gets to reading the file it’s no longer perceived as a special file (and indeed KNIME 4.3 list files now finds them after they’ve been touched by the node from nightly).

I remain none the wiser as to what’s actually going on behind the scenes in the file share but KNIME can now list the files so I’m happy :).

Cheers

Sam

Interesting and thx again for your quick help. Very much appreciated!

Nice this also works for files that are synced by MS Azure File Sync and set to cloud only.
I Wasn’t able to list those files in 4.3.2

3 Likes

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.