List Files/Folders extremely slow

KNIME Analytics Platform
1

Hi there,

did you also experience that the “List Files/Folders” node are working extremely slow when it’s applied to network folders or folders with a bigger amount of files?

I just did a comparison with python (started within a python source node).

for root, dirs, files in os.walk(InPath, topdown=False):
   for name in files:
       if name.endswith('.txt'):
           csvs_full.append(os.path.join(root, name))
           csvs.append(name)

this simple os.walk statement is 20 to 100 times faster (seconds compared to hours in KNIME natively)

Andreas

1 Like
2

The perceived regression might be related to the network type (smb share?) and it’s connection speed you can utilize. I.e. recently found that the CAT cable I bought were utter garbage. Whilst rated for 1 Gb/s they barely managed 100 Mb/s.

To trouble shoot that I’d recommend providing more details and eventually testing to write a test file maybe with that tool:

Best
Mike

2 Likes