CSV reader from variable using new SSH Connector

[austinrecords]

Hello, I’m new to Knime and think it’s great. I’m attempting something on Knime that we commonly do where I work using scheduled scripts. I’m attempting to read a CSV file from an SFTP server, then do something with the data afterwards. However, I’m unable to get the CSV reader to use the flow variable that contains the path to the file on sftp. I’ve done a lot of searching, but I can only find examples of this for the legacy nodes, not the new ones. The closest example I can find is on the KNIME file handling guide, but it doesn’t show all the configurations I need. So I’m trying to figure this out myself.

In each of the 3 attempts above, I’m successfully getting the path to the CSV on the sftp server in the Path variable:

In the CSV reader, I can find the flow variable and select it here. But that’s as far as I can get:

Can someone tell me what I’m missing?

[elsamuel]

Welcome to the forum @austinrecords.

Can you share the workflow? At the very least, can show us how the CSV Reader nodes are configured? Did you select the option to read from a Custom/KNIME URL?

As an alternative, if you click on the ‘…’ in the CSV Reader node icon, a menu will pop up with the option to add a File System Connection port. Try connecting this directly to the SSH connector node and then configuring the CSV Reader node appropriately.

[austinrecords]

Thank you for responding, Elsamuel.

@elsamuel

Can you share the workflow? At the very least, can show us how the CSV Reader nodes are configured? Did you select the option to read from a Custom/KNIME URL?

Unfortunately, I can’t share the workflow because I’m connecting to work & company-related things. We read in a lot of CSVs from various disparate 3rd party systems. I’m trying to evaluate the efficacy of replacing the ETL & Data Science things we currently do programmatically with Knime.

I’ve played around with a lot of different options for the CSV readers. But here is how I have them currently. Notice I’m getting the Path variable which populates the File field:

@elsamuel

As an alternative, if you click on the ‘…’ in the CSV Reader node icon, a menu will pop up with the option to add a File System Connection port. Try connecting this directly to the SSH connector node and then configuring the CSV Reader node appropriately.

I’ve attempted this as well even specifying the file name as you can see here:

It results in the error: “Execute failed: The specified file Reports/FY2019_Annual_Review_Extract_20220205_06_00_39_AM.csv does not exist.”. I’ve tried removing the “Reports” part of the path as well, no luck.

Again, perhaps I’m just missing it. But this is the only thing in Knime so far I’ve had trouble finding a good example on. Each time I search, I only see examples for older legacy nodes, not the new ones.

[Thyme]

Have you tried to connect the nodes this way?

The CSV Reader gets the filename via a Flow Variable and is pointed to the remote location via the File System Connection.
The CSV Reader configuration window should look like this:

[austinrecords]

That was exactly what I needed, Thyme. Thank you.