I am running a simple workflow that uses a CSV reader to read 4 million rows of data from a CSV file. It runs slower than I would like it to run. I ran the “top” command and I see that only 2% of the machine’s memory is being used. I wonder if if increasing the memory available to the job would help. How do we make KNIME Server use more memory for a job?
Hi Carlos,
The KNIME Server executor takes it’s memory settings from the knime.ini file in the /config directory, or if that doesn’t exist the knime.ini file in the executor directory. Increasing that value to half the total available on the machine (assuming no additional workloads run on the machine), is recommended.
Best,
Jon
Hi Jon,
I have 60 GB of memory on my Linux machine. This machine is 100% dedicated to KNIME Server. So I will configure knime.ini to allow KNIME Server to use 30 GB of memory.
What is the exact line I need to put in the knime.ini file?
I see a knime.ini file in the executor directory, but not in the config directory. The knime.ini file has the following line in it:
-Xmx29G
Is my KNIME Server already then configured properly regarding memory? If so, then my CSV Reader issue is probably due to something else.
Thanks,
Carlos -
That sounds like it is configured to use all available memory. It’s probably quite an IO intensive operation, so you may want to check that there is a reasonable amount of temp space available (and that it’s running on a fast drive).
Jon,
I’ll check to see if we have a fast drive.
Regarding the temp folder, where is that located? Is that inside of the KNIME Server installation folder? The full path to our KNIME Server installation is /applications/knime_server. I see a temp folder in /applications/knime_server//workflow_repository/temp: Is this the temp folder that you are referring to?
Carlos -
Hi Carlos,
The temp files will be written to the jobs directory which is part of the workflow_repository/ directory.
Best,
Jon