One of my colleagues has spotted some odd behaviour with the List Files/Folders node when using a windows based network share.
Example path: \data\myfoldershare\sub folder\
The List Files/Folders node find 12 files in this folder but should find 79.
If I run a Java snippet with a simple File#listFiles() call then all 79 files are found, so our initial thought of it being a network/VPN glitch doesn’t look right.
If we copy the files locally the List Files/Folders node finds them all.
If there’s something I can do to provider a more useful bug report let me know, not sure how to provide you a reproducible example in this scenario. But if you tell me what class to look in I’m happy to step through the code.
I did some digging, the problem seems to be that some of the files return false to the BasicFileAttributes#isRegularFile() call which means they don’t pass the filtering in the visitFile call in
Seems all the files that are missing return true to BasicFileAttributes#isOther and seem to show a ‘size on disk’ of 0 bytes.
Though I have no problem reading or copying these files. Looks like it may be more to do with something in our IT infrastructure than an issue in KNIME.
From further investigation the files filtered out have the ReparsePoint attribute. I don’t know why some files have this attribute but presumably it’s something to do with the underlying filesystem on our network share.
I think my conclusion is we’d like to have an option in the node to allow files that return true to isOther() to pass the filter and be listed. Maybe extending this so also optionally include symbolic links?
you are right “special” folders/files are currently not listed by the List Files/Folders node.
We have a ticket to add an option to enable the listing of these objects.
Added a +1 there.
Hi Sam,
we have added a new option to the Filter options that allows you to list also special files. THe option is enabled by default for newly dragged nodes.
This is available in the nightly build and it would be great if you could try it and let us know if it fixes your problem. If no problems come up with this new option it will be released with the 4.3.3 release.
Thanks
Tobias
The CSV seems to load in fine. Though I did observe that whatever the node is doing when the dialog says scanning seems to cause a change in the flags set on the files. For example:
So by the time it gets to reading the file it’s no longer perceived as a special file (and indeed KNIME 4.3 list files now finds them after they’ve been touched by the node from nightly).
I remain none the wiser as to what’s actually going on behind the scenes in the file share but KNIME can now list the files so I’m happy :).