Flow Variables - Questions from a newbie

Hello all (Hallo an alle), 

I am starting to use KNIME and I really feel that it will be helpful in some of the data analysis tasks that I have been doing manually. However I have several nodes that need to be filled in with the same value, and I am thinking that flow variables are the solution to this problem.

I understand how flow variables should be used, but I cannot figure out how to implement what I need given the current documentation. 

 

So some questions:

1. I have a CSV reader at the beginning of my workflow and a CSV writer at the end with the processed data. I would like to have the name of the processed data to be in the same location as the imput CSV but with a modification to the name. Such as myData.csv as the input and myData_processed.csv as the output filename. In the flow variable dialog in the CSV reader I created a flow variable "input_filename" for the input URL. How do I use this variable in the CSV writer so that I can create the modified output filename?

 

2. I need to select a subset of the data from the file based on a timestamp. I have implemented a workflow to do this, but I want to use one of the date_time values to normalize the times so that I have a column with elapsed time. I have one node to "Extract Time Window" with a beginning and end date/time. I want to use the beginning date/time as the value to normalize my time data in a "Time Window" node. When I define a flow variable in the configure dialog, I get an error message in the console :
"WARN  Extract Time Window Errors overwriting node settings with flow variables: Can't export value "1370892610000" as variable "start_time", out of range"

What should I do to create a timestamp flow variable that I can use in another node?

 

3. Is there anyway to see the value of a flow variable once a node has been executed? At least that way I can trace its value as it is used on other nodes.

 

Not sure if these questions have been answered before, if so, if someone could point me to the relevant references that would be great. A quick Google search did not find the answers with sufficient detail to help?

 

Thanks in advance,

 

Scott

 

Hi Scott,

1- You can use the flow variables tab in CSV Writer, but you might want to modify the path with the Java Edit Variable node.

2- Unfortunately the time nowadays with granularity of milliseconds from 1970 do not fit within an int and longs/dates are not supported as flow variable types. So I guess you cannot use these time-stamps as flow variables unless the receiver node expects the time as a String type.

3- If the executed node has an output port, it will show the flow variables in each output table in the Flow Variables tab, although you can also use the KNIME Node Monitor view (View/Other.../KNIME views/KNIME Node Monitor) to show the variable on the selected node.

Hope this helps. Cheers, gabor

For 1. Use a variable to table row to get variable into a cell. Then manipulate this to contain the processed tag, ie. use string replacer and replace .csv for _processed.csv Now use table row to variable to convert this cell back into a variable. To inject this variable back into your workflow, use the inject variable node. Now when you get to the csv writer node, put a dummy filename in the write filename box, then goto flowvariables tab, find the write filename section and choose the variable from the drop down box.

for 3. When a node is run, right click on it and choose bottom option to view table and choose properties tab to see current value of variable.

For 2. I have also had problems use date/times as flow variables without success so cannot help here.

hope this was of some help.

simon

1 Like

Hello Scott,

 

Here's how I deal with 1:

  • Use a File Upload node from the Quick Form extension.
  • Enter a value in the "Variable Name" field (I usually use "fileName").
  • Use the variable port to link the File Upload node to the Reader of interest, in your case the CSV reader.
  • Use a Java Edit Variable (simple) from the Flow Control extension in the Variables section.
  • Overwrite Variable (String) by using something like this in the Method Body section, where "fileName" is the Variable Name specified in the File Upload name (the "S" is automatically added when you double click on the variable in the Flow Variable List pane on the left):
return $${SfileName}$$.replace(".csv","_processed.csv");
  • Use the variable port to link to your CSV Writer.
  • Enter a dummy value in the file name section of the CSV Writer.
  • Select your File Name variable by clicking on the button next to this section.

One of the neat things about this approach is that your variable is available throughout the whole workflow, which means that you can plug your Java Edit Variable (simple) node anywhere. Just connect the variable port back into any node in the workflow and you get the modified file name in the rest of the workflow.

 

For your timestamps, I'd still use Java, but, unlike Simon, I'm a java fan... ;-)

 

HTH,

Fred

I followed your advise and converted the input filename into an output filename through a Java Edit Variable node.

I ran into an issue that seems to me to be an error in KNIME.

When I defined the flow variable for the input filename (through the flow variable dialog in the CSV Reader node) and looked at the value after browsing for the file, it was stored as :

file:<path to the file>/<filename>

This string was stored in my flow variable for the input filename. So of course when I modified this string by appending a new prefix the new flow variable was stored as

file:<path to the file>/<filename>_processed.csv

Using this value for the output filename resulted in the creation of a directory "file:" off of my home directory. I think that this is happening because it is storing the variable as a URL and using the incorrect value of "file:/" as the protocol reference. (it should be "file://" if it is being used in this way). However it should not be part of the value of the flow variable. I worked around the issue by using a substring function call in the Java Edit Variable node to eliminate the "file:" prefix.

Is there a bug report section where I can open this problem? I am on a Mac OSX 10.8.4 and KNIME 2.7.4

 

Thanks for you help,

Scott