TextClassifierLearner received memory warning ... at 30% of dedicated MEM usage

Dear KNIMEler,
while running a text classification WF (KNIME 3.5.3, all free extensions) for finding the best settings (n-gram length (char), border padding) I got the problem that the ‘TextClassifierLearner’ stopped working with the message:

ERROR TextClassifierLearner 0:142:158:0:134 Execute failed: Received memory warning, cancelled training after 8507 rows. Reduce number of training documents, change feature settings, or increase KNIME memory.

My WF - doing cross-validation within a parameter optimization loop - consists of the following nodes (please note the brackets indicate a ‘Metanode’):
‘ParameterOptimization Loop Start’->‘Java Edit Variable’->[MetaNode for Cross validation.]-’>Loop End’

The metanode - fed by a CSVReader (2 columns: 1 category column, 1 text column) - consists of:
‘X-Partitioner’-> ‘TextClassifierLearner’/TextclassifierPredictor’->X-Aggregator->Scorer’. The JavaEditVariable node is used to create the Flow-Variables used in the learner node. The CSV Reader is reading 10.000 lines - there are 6 categories and the text has max 60 chars per lline; 10 folds are set in the X-Partitioner.

When this happens the memory (heap) usage of KNIME (lower border) shows a usage of max ~5GB (out of 24GB assigned to KNIME). Furthermore,
(i) simply ‘executing’ the just stopped ‘TextClassifierLearner’ works!
(ii) using the ‘TextClassifierLearner’ without any loops - but with 100.000 lines - also works!
(iii) reducing to 5 folds works!

Any idea why low memory is indicated?
Erich

Hi Erich,

sorry for the trouble – the memory warning thingie was implemented a bit in a hurry and I’ve also noticed that it behaves over-sensitive sometimes. (on the other hand, I wanted to avoid locking up KNIME when memory is getting full due to some crazy n-gram feature settings like – I’ve made quite some frustrating experiences in that regard :wink: )

I can release and update version during the next days which allows to disable the warning mechanism (towards the weekend).

Best regards,
Philipp

Hi Philipp,

thanks for your fast reply. I’m happy to test your adaptions when available.

Thx
Erich

Hi Erich,

in the recent Palladian Nodes update, the TextClassifierLearner has a checkbox in the “Expert settings” section called “Do not stop on memory warnings”.

Please let me know if that helps to solve you issue!

– Philipp

Hi Philipp,
thanks for the quick fix.
Any idea when this update will be available?

Best
Erich

Hi Erich,

it should actually be already on the 3.5 update site, however seems that it didn’t go through. Let me double check this again.

It is for sure already on the on nightly build update site: https://www.knime.com/community

Best,
Philipp

Hi Philipp,

Yes, your checkbox on ‘Expert settings’ solved my problem!

Thx for this perfect support:)

Best
Erich