# Best Memory Optimization Practices

**URL:** <https://forum.knime.com/t/best-memory-optimization-practices/32775>\
**Category:** KNIME Analytics Platform\
**Created:** [May 6, 2021, 8:13pm UTC](https://forum.knime.com/t/best-memory-optimization-practices/32775 "2021-05-06T20:13:30Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![AndrewDeacon](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/a/a183cd/32.png) [@AndrewDeacon](https://forum.knime.com/u/AndrewDeacon)\
**Post date:** [May 6, 2021, 8:13pm UTC](https://forum.knime.com/t/best-memory-optimization-practices/32775/1 "2021-05-06T20:13:30Z")

</div>

I have a very large workflow with lots of nodes and data. Are there any “best practices” I can utilize to help with my memory usage? All of a sudden I am starting to get the Java Heap error. Cleaning up the amount of rows and columns before that doesn’t seem to make too much of an impact. Other suggestions?

---

<div class="post-metadata">

**Author:** ![elsamuel](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/e/b4bc9f/32.png) [@elsamuel](https://forum.knime.com/u/elsamuel)\
**Post date:** [May 6, 2021, 8:32pm UTC](https://forum.knime.com/t/best-memory-optimization-practices/32775/2 "2021-05-06T20:32:20Z")

</div>

Have you seen the suggestions on this page: [Optimizing KNIME workflows for performance | KNIME](https://www.knime.com/blog/optimizing-knime-workflows-for-performance) ?

---

<div class="post-metadata">

**Author:** ![AndrewDeacon](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/a/a183cd/32.png) [@AndrewDeacon](https://forum.knime.com/u/AndrewDeacon)\
**Post date:** [May 6, 2021, 8:47pm UTC](https://forum.knime.com/t/best-memory-optimization-practices/32775/3 "2021-05-06T20:47:02Z")

</div>

Yeah. That seems to be focused more on execution speed. I am not too concerned about the length of time it takes to execute. I am more focused on it being able to churn through all the data. (Obviously the knime.ini file is optimized).

---

<div class="post-metadata">

**Author:** ![AndrewDeacon](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/a/a183cd/32.png) [@AndrewDeacon](https://forum.knime.com/u/AndrewDeacon)\
**Post date:** [May 6, 2021, 9:35pm UTC](https://forum.knime.com/t/best-memory-optimization-practices/32775/4 "2021-05-06T21:35:52Z")

</div>

For example, do workflows that are not open but have “completed” nodes in them effect the amount of space?

---

<div class="post-metadata">

**Author:** ![AndrewDeacon](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/a/a183cd/32.png) [@AndrewDeacon](https://forum.knime.com/u/AndrewDeacon)\
**Post date:** [May 6, 2021, 9:39pm UTC](https://forum.knime.com/t/best-memory-optimization-practices/32775/5 "2021-05-06T21:39:13Z")

</div>

And if not, does knime make it easy for me to split a workflow up into several different workflows that are called, run, and once the data is extracted are closed out. I don’t need to review all the data at all the stages.

---

<div class="post-metadata">

**Author:** ![mlauber71](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mlauber71/32/73590_2.png) [@mlauber71](https://forum.knime.com/u/mlauber71)\
**Post date:** [May 7, 2021, 6:40am UTC](https://forum.knime.com/t/best-memory-optimization-practices/32775/6 "2021-05-07T06:40:29Z")

</div>

I have a collection about optimization. Columnar storage and garbage collection might be a thing (you might want to test them first). Also you could force some nodes to be written too disk.

Also a strategically placed Cache node might help.

> **[A meta collection about KNIME and performance and performance tuning and some...](https://hub.knime.com/mlauber71/spaces/Public/latest/_knime_performance_and_tuning~iMkc4nou4_2G6OQM)**
>
> A meta collection about KNIME and performance and performance tuning and some problems Please be aware: there might not be the one perfect setting for every ta…

Indeed one way could be to split the job into sub workflows

> **[Call sub workflow from a mother KNIME workflow – mlauber71](https://hub.knime.com/mlauber71/spaces/Public/latest/kn_example_call_workflow/a_001_main_workflow~uPGki0nVinczB7SZ)**
>
> Call sub workflow from a mother KNIME workflow. Communicate between them via JSON files

But in the end KNIME is a graphic platform based on Java so more power/RAM is always welcome

---

<div class="post-metadata">

**Author:** ![AndrewDeacon](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/a/a183cd/32.png) [@AndrewDeacon](https://forum.knime.com/u/AndrewDeacon)\
**Post date:** [May 7, 2021, 12:54pm UTC](https://forum.knime.com/t/best-memory-optimization-practices/32775/7 "2021-05-07T12:54:13Z")

</div>

Perfect. Thanks. Exactly what I am looking for. Just want to make sure I am using all the nodes most effectively for large datasets and workflows.

---

<div class="post-metadata">

**Author:** ![AndrewDeacon](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/a/a183cd/32.png) [@AndrewDeacon](https://forum.knime.com/u/AndrewDeacon)\
**Post date:** [May 7, 2021, 4:53pm UTC](https://forum.knime.com/t/best-memory-optimization-practices/32775/8 "2021-05-07T16:53:08Z")

</div>

Between the Columnar storage, the cache nodes, and the trash nodes everything is working quite well now. Do you think I would benefit from pooling nodes into even more metanodes? If the metanode isn’t open, does Knime hold all the values of the nodes inside it in memory?

Also, does it make sense that I would have a Cache node that would give me the Java Heap Error when writing to disk?

Thanks for the help.

---

<div class="post-metadata">

**Author:** ![mlauber71](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mlauber71/32/73590_2.png) [@mlauber71](https://forum.knime.com/u/mlauber71)\
**Post date:** [May 7, 2021, 5:08pm UTC](https://forum.knime.com/t/best-memory-optimization-practices/32775/9 "2021-05-07T17:08:15Z")

</div>

I think with a [component](https://docs.knime.com/latest/analytics_platform_components_guide/index.html) (similar to a metanode but with much more options) you could tell it to write to disk, that might save some RAM. Writing to disk of course comes with a price tag. You would need time to process it from memory to disk and maybe back, but it could result in a more stable workflow.

As long as a workflow is open KNIME would keep the data ‘ready’ that could either be in memory or on disk. So if you still have issues it might be an option to split the workflow into sub-workflows (which obviously increases the handling costs thru calls and planning). As a side note: with the [KNIME server](https://hub.knime.com/mlauber71/spaces/Public/latest/_knime_server_hub_general_praise~yXCPyGfb1tNnaa85#externalresources) you could orchestrate a cascade of workflows easily 🙂

A Cache node also can either be held in memory (faster) or could be told to write its content to disk. The cache per se would not save memory it would collect all the previous transactions into one place which sometimes could make a workflow more stable and might be placed before you export data to an outside file or you write it into a database.

The columnar storage has some further options to tweak (you might want to be careful playing with that). But a word of advise: I have seen problems with the underlying parquet format and KNIME.

 ![image](https://forum-cdn.knime.com/uploads/default/original/3X/a/3/a39ac17ffbc564cf5ea20be7f8e64b319660b8c8.png)

---

<div class="post-metadata">

**Author:** ![AndrewDeacon](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/a/a183cd/32.png) [@AndrewDeacon](https://forum.knime.com/u/AndrewDeacon)\
**Post date:** [May 7, 2021, 8:19pm UTC](https://forum.knime.com/t/best-memory-optimization-practices/32775/10 "2021-05-07T20:19:47Z")

</div>

Interesting. I’ve also noticed that I am occasionally getting that Java Heap error inside a particular metanode and outside the node, in the main workflow, memory isn’t even close to being an issue.

---

<div class="post-metadata">

**Author:** ![AndrewDeacon](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/a/a183cd/32.png) [@AndrewDeacon](https://forum.knime.com/u/AndrewDeacon)\
**Post date:** [May 11, 2021, 1:13pm UTC](https://forum.knime.com/t/best-memory-optimization-practices/32775/11 "2021-05-11T13:13:31Z")

</div>

So, even more interesting, I noticed that I didn’t have problem with some “more complicated” nodes and had severe lag with “easier” nodes so I did some digging and found that if my RowID was a mess (usually from several Joiners, etc) then it was struggling. I added in some well-placed RowId nodes and now have very little lag throughout the system and do not get the Java Heap error.

---

<div class="post-metadata">

**Author:** ![ScottF](https://forum-cdn.knime.com/user_avatar/forum.knime.com/scottf/32/61316_2.png) [@ScottF](https://forum.knime.com/u/ScottF)\
**Post date:** [May 11, 2021, 3:24pm UTC](https://forum.knime.com/t/best-memory-optimization-practices/32775/12 "2021-05-11T15:24:36Z")

</div>

Sorry for the trouble here. We actually have a ticket in the system to address the problems introduced by complicated RowIDs (AP-15125). I will add a +1 from you on that ticket.

---

<div class="post-metadata">

**Author:** ![AndrewDeacon](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/a/a183cd/32.png) [@AndrewDeacon](https://forum.knime.com/u/AndrewDeacon)\
**Post date:** [May 11, 2021, 8:13pm UTC](https://forum.knime.com/t/best-memory-optimization-practices/32775/13 "2021-05-11T20:13:22Z")

</div>

Perfect. Good to know I am not going crazy.

---

<div class="post-metadata">

**Author:** ![system](https://forum-cdn.knime.com/uploads/default/original/1X/ab3ccf34482a0329361734a18199390177204f15.png) [@system](https://forum.knime.com/u/system)\
**Post date:** [November 10, 2021, 8:14am UTC](https://forum.knime.com/t/best-memory-optimization-practices/32775/14 "2021-11-10T08:14:18Z")

</div>

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.
