Improve Knime performance and stability on weak computers

[mlauber71]

@R72 thank you for this lengthy description of your Optimization process. A lot of issues sound familiar.

I would like to add these remarks.

If sounds have a lengthy process and there is a risk of crash you can try and store a list of already processed items or batches and continue there:

Automatically Continue Loop Execution upon Failure -- KNIME Server

@rrembert3 could you write the sent mail addresses to a file and if something goes wrong you resume from the last position. The CSV writer would allow to add data without header to a file of the file would exist https://forum.knime.com/t/re-start-a-loop-at-a-given-row/57133/4?u=mlauber71 https://hub.knime.com/-/spaces/-/latest/~dr1pFsDBpmwazSQh/

Then if you want to collect results you can append them to a CSV file. The writer offers the option to continue a file without headers or you collect several parquet files into a big one. You could combine this with an approach to restore a process

knime assign failed, request status: data overflow incoming data too big

@RVC2023 one way to get data from and to R is to use Parquet files. You can use them in one go or you can use chunks which is especially useful if you have large data. [image] The large file can be split into smaller parts. If they have the same structure you could also write them in a loop and later import them back as one file: [image]

Then you can read further hints. But you have mentioned quite a few:

Mastering KNIME: Unlocking Peak Performance with Expert Tips and Smart Settings