Dear KNIMEler,
while running a text classification WF (KNIME 3.5.3, all free extensions) for finding the best settings (n-gram length (char), border padding) I got the problem that the ‘TextClassifierLearner’ stopped working with the message:
ERROR TextClassifierLearner 0:142:158:0:134 Execute failed: Received memory warning, cancelled training after 8507 rows. Reduce number of training documents, change feature settings, or increase KNIME memory.
My WF - doing cross-validation within a parameter optimization loop - consists of the following nodes (please note the brackets indicate a ‘Metanode’):
‘ParameterOptimization Loop Start’->‘Java Edit Variable’->[MetaNode for Cross validation.]-‘>Loop End’
The metanode - fed by a CSVReader (2 columns: 1 category column, 1 text column) - consists of:
‘X-Partitioner’-> ‘TextClassifierLearner’/TextclassifierPredictor’->X-Aggregator->Scorer’. The JavaEditVariable node is used to create the Flow-Variables used in the learner node. The CSV Reader is reading 10.000 lines - there are 6 categories and the text has max 60 chars per lline; 10 folds are set in the X-Partitioner.
When this happens the memory (heap) usage of KNIME (lower border) shows a usage of max ~5GB (out of 24GB assigned to KNIME). Furthermore,
(i) simply ‘executing’ the just stopped ‘TextClassifierLearner’ works!
(ii) using the ‘TextClassifierLearner’ without any loops - but with 100.000 lines - also works!
(iii) reducing to 5 folds works!
Any idea why low memory is indicated?
Erich
)