KNIME SSH/SFTP-related Nodes Very Slow

Hi there,

I am having some quite severe performance problems with transferring files from KNIME to various SSH-connected endpoints (e.g. Egnyte, OVH) (i.e. SSH Connector node, with e.g. Transfer Files node, but also applies to CSV Writer etc too). When comparing the speed vs Filezilla SFTP to the same destination location, the differences are huge.

  • a single 3.7MB file: 2min 40s (KNIME) vs 2secs (Filezilla)
  • a single 65MB file: timeout error (KNIME) vs 1min (Filezilla)

In the List Files/Folders node, if I try to point this to a folder with 232 sub-folders and 18,000 files inside, the time for a “Files and Folders” search varies from 10mins to 20mins depending on the endpoint. Both of these seem extremely long, for a simple index listing? a Folders-only search takes 1.5min.

Additionally, I have noted severe unresponsiveness when intereacting with the Transfer Files node, when connected to these endpoints. The element of the UI where it shows the expected number of files detected at the user-provided directory, shows as “scanning” for extremely long periods, if the amount of folders is large. In fact, related to this, in the Transfer Files node, if I choose a working folder which is at higher/root level, and then click to “Browse”, I can see not only 2-3mins of unresponsiveness via the node’s UI, even when the node UI becomes responsive again, I see continued lack of responsiveness within the KNIME GUI itself for a period afterwards.

Since I have compared various destination endpoints, as well as the same SFTP process versus Filezilla, the problem seems to lie with KNIME - but I don’t see any parameters I can try to improve this.

Any advice would be much appreciated,
Best regards,
Steven

Hi @srussell

Would you be able to share an example workflow (or a screenshot of it) to illustrate how you are using the problematic nodes? Screenshots showing the configuration window of the mentioned nodes would be even more helpful to see what settings are used.

What Analytics Platform (AP) version are you using in your setup?

Is there a proxy/firewall/VPN in place in this network?

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.