Dear KNIMER´s,
I always get this error message even though I have the column data cache enabled:
I´m using KNIME 4.7.0 and KNIME Columnar Table Backend 4.7.0.v202211291058
Any pointers?
Kind regards,
Paul
@goodvirus can you tell us your operating system and maybe a log in debug mode. I am collecting hints about problems with parquet and arrow data formats.
Also: can you try and uninstall and reinstall the extension.
Hi @mlauber71,
thanks for the fast reply. I´m running Windows 10 Enterprise with 32 Gb ram an i7 processor.
Uninstalling the extension is not possible, I think because its part of the default KNIME installation now:
“KNIME Columnar Table Backend” cannot be fully uninstalled because other installed software requires it. The parts that are not required will be uninstalled.
But I found that there is an update, so I updated to " KNIME Columnar Table Backend 4.7.1.v202301311311, but still no difference.
An there seems to be no information in the log about that problem:
log.txt (42.7 KB)
Kind regards,
Paul
Hi @goodvirus,
the error is interesting. We have seen something similar on Mac because there the amount of “free” RAM is usually close to zero (because OSX caches a lot of things while the memory is not directly required by other processes).
When you select “Use default values” we do try to estimate a good size of the cache based on the amount of free RAM. If the amount of free RAM is zero, then this error can be shown - because if the estimated cache size is zero we disable the cache.
Could it be that your RAM is completely full? Does this error also show up if you restart the computer and start KNIME before starting a lot of other programs?
How much RAM did you allocate to the JVM for KNIME (configured the -Xmx setting in the knime.ini), is there anything left for the system? E.g. if you allocate 32GB to the JVM while having 32GB of RAM, the estimated cache size will also be zero.
If any of these are the reasons for this error message to show up, we should definitely improve the message to give better hints 
Hope that helps,
Carsten
2 Likes
Ah and @mlauber71 good suggestion about re-installing the extension. However, as @goodvirus correctly assessed, the columnar backend is part of the default install and is used also by other plugins (e.g. the Python Integration) and thus it’s hard to uninstall.
2 Likes
@carstenhaubold not sue if this is related but I still fear there are some systematic problems with the implementation of these data storage formats at least on Apple silicon machines (AP-19918, AP-19894). Mine is currently unusable and I use the intel one (missing out on the 20%+ speed improvement) - but there are more reports about components that should be installed which are not or might not show up or are unable to uninstall. Might not be related but I was noticing a pattern.
Also “RDKit” and similar messages seem to pop up a lot. Tried to uninstall and reinstall related packages but to no avail.
No, these things are not related. But the naming of the features is misleading, unfortunately:
KNIME’s Apache Parquet or ORC “Column Storage” formats are features of the “default” table backend, whereas the “Columnar Backend” is a complete rewrite of KNIME’s table backend (which uses Apache Arrow in turn). The problems you are seeing @mlauber71 are that the Apache Parquet and ORC Column Storage Features are not working properly with Apple silicon, while our “Columnar Backend” is definitely working on Apple silicon - otherwise the new Python Integration wouldn’t work on Apple silicon KNIME builds.
@goodvirus’ issue was on a Windows machine when using the “Columnar Backend”, so the Apple silicon incompatibilities shouldn’t be a problem here.
Hope that resolves the confusion a little.
The RDKit errors however are definitely there in the logs, did you try to re-install / update that extension @goodvirus? We have updated the RDKit extension for KNIME 4.7 quite substantially.
3 Likes
Yes they do - I was just wondering if there might be more to it components not being able to be installed, and also not showing up on the startup graphic (like they would). Sticking to Intel/Rosetta for the moment on Mac. Windows running just fine.
Hi @carstenhaubold,
i think this was the problem. I had -Xmx29000m as a setting, because I wanted KNIME to use most of my ram. I reduced it and the error is not happening anymore. The question now is, how do I give KNIME as much memory as possible, but also have the column backend?
2 Likes
I don’t have RDKit installed as far a I know. This are my installed extensions:
| Geospatial Analytics Extension for KNIME | 1.0.0.202212100353 | sdl.harvard.features.geospatial.feature.group | SDL, Harvard, Cambridge US |
|---|---|---|---|
| KNIME Amazon Cloud Connectors | 4.7.0.v202301311059 | org.knime.features.cloud.aws.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME Amazon Machine Learning Integration | 4.7.0.v202211082351 | org.knime.features.cloud.aws.mlservices.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME Analytics Platform | 4.7.1.v202301311353 | org.knime.desktop.product | null |
| KNIME Azure Cloud Connectors | 4.7.0.v202211161540 | org.knime.features.ext.azure.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME Chromium Embeded Framework (CEF) Browser | 4.7.1.v202301311311 | org.knime.features.browser.cef.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME Columnar Table Backend | 4.7.1.v202301311311 | org.knime.features.core.columnar.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME Connectors for Common Databases | 4.7.0.v202211120527 | org.knime.features.dbdrivers.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME Data Generation | 4.7.0.v202211082353 | org.knime.features.datageneration.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME Database | 4.7.0.v202211231356 | org.knime.features.database.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME Ensemble Learning Wrappers | 4.7.0.v202206271101 | org.knime.features.ensembles.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME Excel Support | 4.7.0.v202211171209 | org.knime.features.ext.poi.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME Expressions | 4.7.0.v202211082355 | org.knime.features.expressions.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME External Tool Support | 4.7.0.v202211151738 | org.knime.features.ext.exttool.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME External Tool Support (Labs) | 4.7.0.v202206271101 | org.knime.features.exttool.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME H2O Machine Learning Integration | 4.7.0.v202211220919 | org.knime.features.ext.h2o.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME Hub Integration | 4.15.0.v202211281224 | com.knime.features.workbench.hubview.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME Image Processing | 1.8.3.202212100549 | org.knime.knip.feature.feature.group | University of Konstanz / KNIME |
| KNIME Interactive R Statistics Integration | 4.7.0.v202211090002 | org.knime.features.r.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME JavaScript Views | 4.7.0.v202211091556 | org.knime.features.js.views.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME JavaScript Views (Labs) | 4.7.0.v202207221134 | org.knime.features.js.views.labs.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME Javasnippet | 4.7.0.v202211082357 | org.knime.features.javasnippet.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME JSON-Processing | 4.7.0.v202212030000 | org.knime.features.json.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME Machine Learning Interpretability Extension | 4.7.0.v202211082358 | org.knime.features.mli.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME Math Expression (JEP) | 4.7.0.v202208041429 | org.knime.features.ext.jep.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME Network Mining | 4.7.0.v202211082359 | org.knime.features.network.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME Network Mining distance matrix support | 4.7.0.v202206271101 | org.knime.features.network.distmatrix.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME NGS tools | 0.2.300.qualifier | org.pasteur.pf2.ngs.feature.feature.group | Plate-forme 2 - Transcriptome et Epigenome, Institut Pasteur, Paris. |
| KNIME Office 365 Connectors | 4.7.1.v202301311356 | org.knime.features.ext.office365.filehandling.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME Parallel Chunk Loop Nodes | 4.7.0.v202211090007 | org.knime.features.virtual.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME Personal Productivity Tools | 4.7.1.v202301312109 | org.knime.features.personalproductivity.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME Plotly | 4.7.0.v202211090000 | org.knime.features.js.plotly.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME Public Server Access | 4.7.1.v202302021000 | com.knime.features.enterprise.client.exampleserver.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME Python 2 Integration (legacy) | 4.7.1.v202301311311 | org.knime.features.python2.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME Python Extension Development (Labs) | 4.7.1.v202301311311 | org.knime.features.python3.nodes.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME Python Scripting extension | 4.3.0.v202301240921 | de.mpicbg.tds.knime.scripting.python.feature.feature.group | Max Planck Institute of Molecular Cell Biology and Genetics (MPI-CBG), Dresden, Germany |
| KNIME Quick Forms | 4.7.1.v202301311311 | org.knime.features.js.quickforms.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME REST Client Extension | 4.7.0.v202210211111 | org.knime.features.rest.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME ServerSpace | 4.16.1.v202301311407 | com.knime.features.explorer.serverspace.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME Shapefile Support | 1.6.0.v202212100552 | de.bund.bfr.knime.esri.feature.feature.group | Christian Thöns |
| KNIME Statistics Nodes | 4.7.0.v202206271529 | org.knime.features.stats.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME Statistics Nodes (Labs) | 4.7.0.v202211090004 | org.knime.features.stats2.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME Textprocessing | 4.7.0.v202211181006 | org.knime.features.ext.textprocessing.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME Textprocessing German Language Pack | 4.7.0.v202206271102 | org.knime.features.ext.textprocessing.language.german.feature.group | KNIME Gmbh, Konstanz, Germany |
| KNIME Timeseries nodes | 4.7.0.v202209020834 | org.knime.features.timeseries.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME XGBoost Integration | 4.7.0.v202211082337 | org.knime.features.xgboost.feature.group | KNIME AG, Zurich, Switzerland |
| KNIME XML-Processing | 4.7.0.v202211082337 | org.knime.features.xml.feature.group | KNIME AG, Zurich, Switzerland |
| Palladian for KNIME | 2.7.0.202211091145 | ws.palladian.nodes.feature.feature.group | palladian.ws |
| Redfield BERT Nodes | 1.0.2.202212230230 | se.redfield.bert.feature.feature.group | Redfield AB |
| Redfield NLP Nodes | 1.1.2.202212230250 | se.redfield.textprocessing.feature.feature.group | Redfield AB |
| Vernalis KNIME Nodes | 1.36.1.v202212051049 | com.vernalis.knime.feature.feature.group | Vernalis Research Ltd, Cambridge, UK |
I think the errors show up during loading of the Vernalis extension, it could be that it depends on RDKit (the log says An error occurred while automatically activating bundle com.vernalis.knime.chem.mmp)
1 Like
Hi @carstenhaubold ,
I just installed RDKit and the error is gone. But interesting, that KNIME didn’t automatically installed all the decencies when I installed vernalis.
Kind regards,
Paul
1 Like
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.