Hello everyone,
while doing some testing i realized that the CSV Reader is not always reading all the columns even if default is “any unknown new column” is on:
Note the setting is like the follwing: the original file and changed are the same. So the upper node that reads the new file first reads the file without a new column. in Second step is to reset the Reader and read the “changed2” but first change the name to “changed” - So that the reader reads a file with the same name as when the column was not there (unfortunately one needs to change the name of the ‘changed’ file too, but that’s the only way that path and filename stay the same).
If one plays with this it often happened that the the collumn was somethimes even shown in the “preview” but not in the read table.Chance that it works or not are 50:50 at least on my machine.
I can confirm your observation. It’s expected behavior of the CSV Reader node. In it is default configuration, the node will read only the columns it saw during configuration.
If I am not mistaken, you are looking for a way of making CSV Reader to reflect changes in number or type of columns it results after it has been configured. There is a dedicated setting, i.e., Support changing file schemas, in the advanced settings tab for that. See screen shot below.
Hi @temesgen-dadi,
hey i had the last days time to test this a little. It is quite a lot going on in the background there in the settings.xml. I tested it and i now think by default i would use support changing file schemas on on.
Since there is a lot happening in the .xml and i am not aware of everything. Are there things i should be aware besides that the transformations tab doesn’t work?
One line in the settings.xml caught my eye tho, does anybody know what: “skip empty columns**” does? I could not find it in the Readers tabs.
Hi Daniel,
well yeah, what i wrote was easy to be misunderstood. What i meant is more the entry: “skip_empty_columns” is still in the settings.xml but deactivated if SCFS is off (as default). If SCFS is activated then it is not in the settings.xml plus that i was not able to find the setting in the advanced settings tab - but that might be just me not finding it.
But all in it is just a bit strange, since the feature seems to be always deactivated no matter what you do.
More important to me is the question if when support changing files schemas is on, if there are other things to be considered other than that the transformations tab is deactivated? Since it is not that transparent what else might be affected from this…And a lot depends on how the reader reads the files…
