List Files/Folders missing files

swebb · February 19, 2021, 11:04am

Good morning

Version: KNIME 4.3

One of my colleagues has spotted some odd behaviour with the List Files/Folders node when using a windows based network share.

Example path: \data\myfoldershare\sub folder\

The List Files/Folders node find 12 files in this folder but should find 79.

If I run a Java snippet with a simple File#listFiles() call then all 79 files are found, so our initial thought of it being a network/VPN glitch doesn’t look right.

If we copy the files locally the List Files/Folders node finds them all.

If there’s something I can do to provider a more useful bug report let me know, not sure how to provide you a reproducible example in this scenario. But if you tell me what class to look in I’m happy to step through the code.

Cheers

Sam

P.S. This component finds all the files: List Files – KNIME Hub

swebb · February 19, 2021, 12:35pm

I did some digging, the problem seems to be that some of the files return false to the BasicFileAttributes#isRegularFile() call which means they don’t pass the filtering in the visitFile call in

github.com

knime/knime-base/blob/master/org.knime.filehandling.core/src/org/knime/filehandling/core/defaultnodesettings/filechooser/FilterVisitor.java

/*
 * ------------------------------------------------------------------------
 *
 *  Copyright by KNIME AG, Zurich, Switzerland
 *  Website: http://www.knime.com; Email: contact@knime.com
 *
 *  This program is free software; you can redistribute it and/or modify
 *  it under the terms of the GNU General Public License, Version 3, as
 *  published by the Free Software Foundation.
 *
 *  This program is distributed in the hope that it will be useful, but
 *  WITHOUT ANY WARRANTY; without even the implied warranty of
 *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
 *  GNU General Public License for more details.
 *
 *  You should have received a copy of the GNU General Public License
 *  along with this program; if not, see <http://www.gnu.org/licenses>.
 *
 *  Additional permission under GNU GPL version 3 section 7:
 *

This file has been truncated. show original

Seems all the files that are missing return true to BasicFileAttributes#isOther and seem to show a ‘size on disk’ of 0 bytes.

Though I have no problem reading or copying these files. Looks like it may be more to do with something in our IT infrastructure than an issue in KNIME.

Cheers

Sam

swebb · February 19, 2021, 3:14pm

From further investigation the files filtered out have the ReparsePoint attribute. I don’t know why some files have this attribute but presumably it’s something to do with the underlying filesystem on our network share.

I think my conclusion is we’d like to have an option in the node to allow files that return true to isOther() to pass the filter and be listed. Maybe extending this so also optionally include symbolic links?

Cheers

Sam

julian.bunzel · February 19, 2021, 3:24pm

Hi @swebb,

you are right “special” folders/files are currently not listed by the List Files/Folders node.
We have a ticket to add an option to enable the listing of these objects.
Added a +1 there.

Cheers,
Julian

swebb · February 19, 2021, 3:28pm

Thanks Julian :), we’ll use a workaround in the meantime.

tobias.koetter · March 31, 2021, 8:07am

Hi Sam,
we have added a new option to the Filter options that allows you to list also special files. THe option is enabled by default for newly dragged nodes.

This is available in the nightly build and it would be great if you could try it and let us know if it fixes your problem. If no problems come up with this new option it will be released with the 4.3.3 release.
Thanks
Tobias

swebb · March 31, 2021, 10:09am

Hi Tobais

Thanks for the update, I can confirm the nightly build is finding the files with the ‘l’ attribute now

KNIME 4.4:

KNIME 4.3:

Cheers

Sam

tobias.koetter · March 31, 2021, 11:11am

Thanks a lot for the prompt testing

Mark_Ortmann · March 31, 2021, 4:53pm

Quick flow up. Where you able to read those files?

Best,
Mark

swebb · March 31, 2021, 5:21pm

I didn’t check that but I just have and KNIME still reads them (it always could it just wouldn’t list them).

The CSV seems to load in fine. Though I did observe that whatever the node is doing when the dialog says scanning seems to cause a change in the flags set on the files. For example:

List the directory

Run the List Files node or put the path into the dialog and let it ‘scan’
List the directory again

So by the time it gets to reading the file it’s no longer perceived as a special file (and indeed KNIME 4.3 list files now finds them after they’ve been touched by the node from nightly).

I remain none the wiser as to what’s actually going on behind the scenes in the file share but KNIME can now list the files so I’m happy :).

Cheers

Sam

Mark_Ortmann · March 31, 2021, 6:13pm

Interesting and thx again for your quick help. Very much appreciated!

dijkstran · July 9, 2021, 8:41am

Nice this also works for files that are synced by MS Azure File Sync and set to cloud only.
I Wasn’t able to list those files in 4.3.2

system · January 7, 2022, 8:42pm

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.