I am attempting to create a workflow utilizing an External Tool (Labs) node that executes a program which produces multiple files in SDF, DAT and TXT (tab delimited) formats. The variable flow outPort of the External Tool (Labs) is connected to a Flow Variable If Switch that determines which respective file reader should be used.
Based on the command line directives, the fields of the output files can differ with each run. That is, each run may produce files with a different number of fields, such that each field may or may not appear in the output file.
My Issue: the File Reader node seems to have problems handling data in files specified in the "Variable Settings" when the number of fields or the data type(s) of the field differ from the contents of the required file specified in the valid URL. When I attempt to execute the workflow and the internal structure of the output file does not conform to the file specified in the (required) valid URL text box I receive one of two errors:
1. Execute Failed: Wrong data format...
2. Execute Failed: New line in quoted string (or closing quote missing). In line 27.
It seems that the columns the file reader scan results are controlled by the file pointed to in the valid URL text box in the File Reader rather than the file identified by the selected flow variable in the Variable Settings dialogue (that was intended to replace the valid URL and ???also be scanned???).
Could someone please offer a suggestion to help me understand and work through/around this issue?
you can you the CSV Reader node to read csv formatted files (also tab separated) with different number of columns, which would solve problem 1. However, new line characters are not allowed in a quoted string.
I have the same problem as adamclee. When I reset the file reader variable "data url" to a new file in batch mode, I get the "too many/ too few data fields" error. I tried using CSV reader too but it does not have all the options file reader provides and I get the error" Encountered duplicate row". I have no idea how to fix this problem and would really appreciate it if someone could help me.
The File Reader is not the appropriate node to read files with different content - different to what was set in the node's dialog. The File Reader expects the kind of data (e.g. number of columns and the data type) that was specified in the dialog to be read from the file. It does not scan the file and re-sets those settings during execute. The CSV Reader does re-analyze the content of the file everytime it is executed. It is the node of choice to read files with different content. It has less options than the file reader unfortunately.
The only other option I see is to read in the file with one cell per line and to do the splitting in subsequent nodes (like Cell Splitter and String Manipulation nodes).
Thanks a lot Peter, my other question is that if I can use CSV reader to read space delimited files? If yes, what expression should I use as the column delimiter in the node? I tried using "\s " and "\t" but none of them worked.
simply use whitespace " " as column separator. If you are using "\n" as row separator make sure that you have a \n after your last line. If there is no \n after your last line with data, select "Support Short Lines".