I've got a lot of XML files which I want to extract at once in a Knime flow. Each of the files is about 500MB in size. My computer has 28GB of RAM memory and I've made sure Knime has 24GB available.
When extracting one file in order to test my flow everything works like a charm. But as soon as I try to execute a loop which will extract all XML files a single folder, I get an out of memory issue (at the 6th iteration already).
Does anyone know what I can possibly do to make this loop more efficient in memory usage?
The first thing I've tried is to cache all the files during the loop, but this makes absolutely no difference.
The second thing I've tried is to use an Xpath in the XML Reader Node, but for some reason I can't get this to work. I've used the namespaces and Xpath that works in the Xpath node. But this results in an empty data table when I use this in the XML Reader Node, or I get an error message:
Execute failed: NAMESPACE_ERR: An attempt is made to create or change an object in a way which is incorrect with regard to namespaces.
This is what my settings for the XML Reader Node look like when I get this error message. Incorporating the root namespace doesn't solve the issue.
I'm sorry but due too privacy issues I can't show you a part of the XML of accompanying XSD file. I hope this is enough information...
I've also tried to find this error, but only get a few discusions about this subject on a Java forum, which unfortunately isn't really helpful for me.