Knime keeps freezing on me. The used ram just keeps growing as I rerun the same workflow. Even after closing the workflow, the ram is still showing close to 2GB with NOTHING open in knime. I tried limiting the memory in the knime.ini file and it only made it worse. I also changed all or most of the nodes to "write to disk", which seemed to help a little but not enough. Any suggestions?
The memory indicator in the OS is not really useful because Java doesn't return memory to the OS once it has claimed it. Therefore the OS memory consumption usually only grows. More interesting is the memory bar (heap status) at the bottom right of the KNIME application window. It shows how much memory really is used and still available.
Fresh open KNIME, CSV reader for file with 600K rows, knime fails and freezes growing to 3.4GB. Tableau opens it no problem and didn't increase memory much at all. Unfortunately Tableau sucks for data manipulation and cant even concatenate files. I need to combine two CSVs with 600K rows each, pretty simple on a PC with KNIME.
600k rows shouldn't be a problem at all. How many columns does the CSV file have? What is the error message the CSV Reader reports? Did you also try using the File Reader node?
Using the file reader node, if I set the delimiter to comma it also fails growing ram to 3.4GB and freezing knime. If I leave it to "self detect" settings it works, it detects and sets comma as the delimiter, and doesn't grow and crash knime. But why would me setting comma as the delimiter crash knime but knime setting comma as the delimiter be ok?
In other words, I got it working but it seems like there are bugs in knime.
On a separate note, xls reader has worse issues where small files can be read but big ones always fail or freeze. I use CSV reader or file reader to avoid that problem.
If self-detection of settings works and manually setting not, then you are not setting all options correctly. I'd compare your manual with the auto-detected settings.
Reading XLS file is a problem in general due to the used format and data structures in Excel. The Excel Reader in 3.3 should be much better than the previous versions, though.
The only setting I set was the delimiter as comma. Auto-detect set a comma as well.