Knime optimization

Hello

I have problem with optimization.

Problem: I have 190 columns with nearly 20 million record. Spread across different subsheet .

While reading the file in knime using excel file reader.its taking nearly few hrs time to execute .

Can anyone help me how to reduce time or optimize it.

Hi @saai,

welcome to the Knime community. The performance regression you experience can be caused by many factors. Let me ask a few questions to get a better picture about your situation:

  1. Are these Excel files located on a network drive?
  2. Are these excel files and / or your OS as well as Knime stored on an SSD or actual spinning hard disk?
  3. How much memory does your system have at its disposal?
  4. How much memory did you allocate to Knime by adjusting your knime.ini in the knime application folder? Please check the following document for futher instructions.
  1. Can you share your content of your knime.ini please?
  2. How much data in MG or GB are you trying to read using? Meaning, how large are the Excel files in question?
  3. Do you read explicit files or index a folder and read all files contained within it possibly including files you actually do not want to read et all?
  4. What kind of data are you dealing with, can you share a sample? Reason I am asking is that in one cell you can store more data than in an entire Excel file if done wrong (no offense) but the Excel might also contain images.
  5. Do you need to read the entire contents of the Excel files or just specific sheets?
  6. Can you share your workflow and some sample data for testing purposes please?

Best
Mike

3 Likes