# Columnar Backend: Data Preview takes much more time to display or never finishes **URL:** \ **Category:** Feedback & Ideas\ **Tags:** bug\ **Created:** [August 9, 2023, 4:16pm UTC](https://forum.knime.com/t/columnar-backend-data-preview-takes-much-more-time-to-display-or-never-finishes/71605 "2023-08-09T16:16:56Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![mwiegand](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mwiegand/32/66896_2.png) [@mwiegand](https://forum.knime.com/u/mwiegand)\ **Post date:** [August 9, 2023, 4:16pm UTC](https://forum.knime.com/t/columnar-backend-data-preview-takes-much-more-time-to-display-or-never-finishes/71605/1 "2023-08-09T16:16:56Z")
Hi, In a workflow about identifying duplicated images which contains tables with images (about 800 MB to 1 GB), their corresponding binary object and the binary object as a string, I am unable to preview the data when I have the new columnar backend enabled. ![image](https://forum-cdn.knime.com/uploads/default/original/3X/5/c/5cf58e6d6e5199357679b5224d504825eeeafc3a.png) ![image](https://forum-cdn.knime.com/uploads/default/original/3X/e/d/edb6e0f67d18ae11f6004c8b19ba5c1b60e59dd5.png) About 30 seconds in ![image](https://forum-cdn.knime.com/uploads/default/original/3X/0/4/04fcb6ffb7f0a5eb1065b35b49ab9071ac632e87.png) About 90 seconds in ![image](https://forum-cdn.knime.com/uploads/default/original/3X/1/c/1c86c0e45368b9c9f8f9c2f5c8b7f41661b08a52.png) About 600 seconds in ![image](https://forum-cdn.knime.com/uploads/default/original/3X/c/c/cc2401fb4dc5ce20f8fa1f1f6a048d7f3be42399.png) The data preview actually never finished loading and the preview windows I was unable to close requiring me to force kill Knime. I am tryig to create a test workflow which I will upload, if sucessfull, later. But due to the nature of Knime being required to get force quit, this is rather “challenging”. PS: The preview window suddenly closed. It seems my request to close it got registered and magically executed. PPS: A presumably very positive change about the columnar backend is the roughly 100 % higher disc read throughput. Whilst neither a quantitative nor qualitative anylsis, I only managed to get about 800 MB throughput peak with the default row based backend. The columnar one manages to read \>1.6 GB / sec. ![image](https://forum-cdn.knime.com/uploads/default/original/3X/9/8/98b0e075e75cfebd9b1b92f6274e97a424a4dd30.png) However, displaying the table data seems to take considerably more time. After saving and restarting I did another test and it took about 5 Minutes to display the data. A test workflow indicated that it seems to correlate with the amount of the data which the table holds. Whilist the images are online accessible, it would stress my bill a bit too much. I could however provide you an AWS S3 download link for testing purposes if you “pinky” promise to not … joke aside. Let me know how’d like to proceed. Best Mike ---
**Author:** ![carstenhaubold](https://forum-cdn.knime.com/user_avatar/forum.knime.com/carstenhaubold/32/33947_2.png) [@carstenhaubold](https://forum.knime.com/u/carstenhaubold)\ **Post date:** [August 10, 2023, 9:36am UTC](https://forum.knime.com/t/columnar-backend-data-preview-takes-much-more-time-to-display-or-never-finishes/71605/2 "2023-08-10T09:36:02Z")
Hi Mike! Thanks again for all your reports and your patience, highly appreciated! This is very interesting. I’ll send you a PM to get the workflow + data so we can investigate. Best, Carsten ---
**Author:** ![mwiegand](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mwiegand/32/66896_2.png) [@mwiegand](https://forum.knime.com/u/mwiegand)\ **Post date:** [August 10, 2023, 10:17am UTC](https://forum.knime.com/t/columnar-backend-data-preview-takes-much-more-time-to-display-or-never-finishes/71605/3 "2023-08-10T10:17:04Z")
You are mpst welcome. I am preparing the data atm… Please await my DM in about an hour or so … depending on fast my little falls and keeps sleeping 😅 ---
**Author:** ![mwiegand](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mwiegand/32/66896_2.png) [@mwiegand](https://forum.knime.com/u/mwiegand)\ **Post date:** [August 10, 2023, 10:53am UTC](https://forum.knime.com/t/columnar-backend-data-preview-takes-much-more-time-to-display-or-never-finishes/71605/4 "2023-08-10T10:53:54Z")
Update: I have to rectify my initial “complaint”. Whilst the issue seems to have been caused by the columnar backend - please note that I took utmost care by resetting, regenerating saving the workflow, closing and repoening - the issue was caused too while back testing again. As follows the test workflow but w/o the sensitive customer data (shared via PM with the Knime team). Please note that I possibly tracked the issue down to lots of string data being present (1.3 GB) while investigating about a general Knime instability: > [@Binary Object causes Knime instability](https://forum.knime.com/t/binary-object-causes-knime-instability/71416/7): > > Hi @wiswedel, hi all, I think the funky characters are kind of normal as these, I suppose to recall, are some sort of control but not really text characters. When these strings are displayed, they’ll get interpreted no matter what. Though, I tried copy & pasting the string into Sublime which can / should be able to cope with that and still see them. Anyways, about the purpose, I am working on a a simple appraoch to extract the XML Meta Data. These are easily extractable when the binary object … > **[Columnar Backend Data Preview does not load – mwiegand](https://hub.knime.com/mwiegand/spaces/Public/Bugs/Columnar%20Backend%20Data%20Preview%20does%20not%20load~e46sKFiCZU-dnbmc/current-state)** > > Solutions for data science: find workflows, nodes and components, and collaborate in spaces. ---
**Author:** ![mwiegand](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mwiegand/32/66896_2.png) [@mwiegand](https://forum.knime.com/u/mwiegand)\ **Post date:** [August 10, 2023, 11:29am UTC](https://forum.knime.com/t/columnar-backend-data-preview-takes-much-more-time-to-display-or-never-finishes/71605/5 "2023-08-10T11:29:42Z")
Sorry for the “spamming” but I got yet again a new symptom. Trying to open another workflow after switching back to the default row based backend, worth to note that I did not reset and calculated the data, loading the workflow does never finish. I was on a walk for about 20 minutes with my little one and it seems hung. Little to no IO, constant CPU usage despite (presumably) nothing happening in the background. ![image](https://forum-cdn.knime.com/uploads/default/original/3X/0/d/0dc7874b8016e88638ed2c265b7f109d968af5a8.png) ---
**Author:** ![carstenhaubold](https://forum-cdn.knime.com/user_avatar/forum.knime.com/carstenhaubold/32/33947_2.png) [@carstenhaubold](https://forum.knime.com/u/carstenhaubold)\ **Post date:** [August 11, 2023, 7:51am UTC](https://forum.knime.com/t/columnar-backend-data-preview-takes-much-more-time-to-display-or-never-finishes/71605/6 "2023-08-11T07:51:51Z")
Hi @mwiegand, As you have noticed yourself, the preview of the table is busy rendering very long strings. Thanks @mwiegand for sending me your workflow. I could observe the behavior as well. When I tried @wiswedel’s fix in the current nightly the table preview loads much faster. It is still noticeable when scrolling that loading and displaying the long strings takes some time – which makes sense as it still needs to display a lot of data – but it does not get stuck. For the time being, it would be best to filter out the string column that contains the binary representation of the image before looking at the table view. I’d argue that this column is not very human readable anyways… > Trying to open another workflow after switching back to the default row based backend, worth to note that I did not reset and calculated the data, loading the workflow does never finish. I’m not sure whether I got this right. Is step 2 what you were saying? 1. you had workflow A open which used the columnar backend 2. you re-configure this workflow to use the row-based backend (this **does** reset all nodes) 3. you try to open workflow B, but this never finishes? Best, Carsten ---
**Author:** ![mwiegand](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mwiegand/32/66896_2.png) [@mwiegand](https://forum.knime.com/u/mwiegand)\ **Post date:** [August 11, 2023, 10:31am UTC](https://forum.knime.com/t/columnar-backend-data-preview-takes-much-more-time-to-display-or-never-finishes/71605/7 "2023-08-11T10:31:53Z")
Changing Knime‘s backen from default row based to new columnar backend never reset any workflow for me. Not the open ones while changing the setting nor after saving, close and reopen the very same workflow and also not those I had not open for a while. I was always required to manually reset, thought. The only discernible difference I believe to have recognized is a progress bar, left to the heap space indicator, which I believe indicates the amount of consumed memory for the columnar backend. Worth to note that this barely increased regardless of the data and amount I was processing or inspecting via preview. Knime set the amount of allocated memory to about 13 GB which I never adjusted. I will check the nighty build later today. Currently I am on a walk so my little son falls asleep. ---
**Author:** ![mwiegand](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mwiegand/32/66896_2.png) [@mwiegand](https://forum.knime.com/u/mwiegand)\ **Post date:** [August 11, 2023, 11:07am UTC](https://forum.knime.com/t/columnar-backend-data-preview-takes-much-more-time-to-display-or-never-finishes/71605/8 "2023-08-11T11:07:33Z")
I tried to try the nighty build but ran into another issue I reported being caused by prev. Knime versions but this time it also struck for the latest: > [@Cannot launch prev. Knime 4.7.6](https://forum.knime.com/t/cannot-launch-prev-knime-4-7-6/71526/2): > > Update: I downloaded the most recent nighty build and the very same issue of not being able to start Knime appeared again. !SESSION 2023-08-11 12:53:58.416 ----------------------------------------------- eclipse.buildId=unknown java.version=1.8.0\_371 java.vendor=Oracle Corporation BootLoader constants: OS=win32, ARCH=x86\_64, WS=win32, NL=en\_GB Command-line arguments: -os win32 -ws win32 -arch x86\_64 !ENTRY org.eclipse.equinox.simpleconfigurator 4 0 2023-08-11 12:53:58.926 !MESSAGE F… ---
**Author:** ![carstenhaubold](https://forum-cdn.knime.com/user_avatar/forum.knime.com/carstenhaubold/32/33947_2.png) [@carstenhaubold](https://forum.knime.com/u/carstenhaubold)\ **Post date:** [August 11, 2023, 11:32am UTC](https://forum.knime.com/t/columnar-backend-data-preview-takes-much-more-time-to-display-or-never-finishes/71605/9 "2023-08-11T11:32:21Z")
> Changing Knime‘s backen from default row based to new columnar backend never reset any workflow for me. Not the open ones while changing the setting nor after saving, close and reopen the very same workflow and also not those I had not open for a while. Oh! Now I see the confusion. There are two settings to switch the table backend: one per-workflow, and one in KNIME’s preferences which will apply to _all workflows that will be created after changing the setting_. I was thinking of this dialog in the screenshot which configures the _current_ workflow only. To get there you (currently in classic UI only) right click on the current workflow in the explorer → `Configure` → select the table backend. This _should_ reset your workflow. ![Screenshot 2023-08-11 at 13.30.59](https://forum-cdn.knime.com/uploads/default/original/3X/8/5/85eb50d0d27b9049a83fa1a4f42b3913fdf20f8c.png) Changing the global preference setting will _not_ impact any workflows that have been created before. ---
**Author:** ![mwiegand](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mwiegand/32/66896_2.png) [@mwiegand](https://forum.knime.com/u/mwiegand)\ **Post date:** [August 13, 2023, 12:00pm UTC](https://forum.knime.com/t/columnar-backend-data-preview-takes-much-more-time-to-display-or-never-finishes/71605/10 "2023-08-13T12:00:38Z")
Thanks for pointing that misconfiguration out but it is rather confusing that the app settings are “locally” overwritten and there seems to be no indication / note in the preferences nor notification upon opening a workflow with contradicting, app vs. workflow, settings. I think it would be good idea to: In order to profide greatest flexibility but make users aware of possible disjoints to 1. Add a hint in the application settings which instructs users to also change the workflow settings 2. Add a notification upon opening a workflow with settings that disjoint those of the app 3. Add an option in the app settings to force overwrite / apply the columnar or classic row based backend Upon reprocessing the data it seems the Hash Calculator is facing a serious performance regression. It processed for +15 minutes and barely reached 80 %. ![image](https://forum-cdn.knime.com/uploads/default/original/3X/3/7/370d99b2de9ff197d0354f76cd85ebfb4770440e.png) Anyways, focussing on the main challenge. Even after switching to the columnar backend, reprocessing and saving. It takes several minutes to display a table which contains the data of just ten images. ![image](https://forum-cdn.knime.com/uploads/default/original/3X/6/b/6bdd1c4cf0ba4dfea14c5d9f352c573d71eacf2e.png) I alos removed the column with the string representation of the binary object and the preview still froze. Guess, albeit the binary object seems capped as well, it still impairs the UI responsiveness. ---
**Author:** ![s.roughley](https://forum-cdn.knime.com/user_avatar/forum.knime.com/s.roughley/32/366_2.png) [@s.roughley](https://forum.knime.com/u/s.roughley)\ **Post date:** [August 24, 2023, 11:44am UTC](https://forum.knime.com/t/columnar-backend-data-preview-takes-much-more-time-to-display-or-never-finishes/71605/11 "2023-08-24T11:44:07Z")
I have seen this issue before with lots of long strings. unlike some of the chemistry column types which can contain long strings and which use a factory to determined whether to use a Blob-based version of the cell or an in-memory version depending on the length of the string, String columns are always ‘in memory’ - maybe a factory class (I think there is one!) which could mimic the chemistry types (e.g. SdfValue implementations) is called for? Steve ---
**Author:** ![mwiegand](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mwiegand/32/66896_2.png) [@mwiegand](https://forum.knime.com/u/mwiegand)\ **Post date:** [August 26, 2023, 1:41pm UTC](https://forum.knime.com/t/columnar-backend-data-preview-takes-much-more-time-to-display-or-never-finishes/71605/12 "2023-08-26T13:41:57Z")
Adding to this I traced back another possible issue. Though, I am not quite certain to which degree it is related. What I did was improving my RegEx as it was not quite accurate. There were still some remaining rows which could not be converted to XML. Attempting to look at just one cell was always impossible as the preview never rendered. It then tried to save the extracted string as a file to actually be able to inspect the data. However, the presumably string seems to still be binary. The following screenshot can be understood as follows: 1. Files to Binary Object Node reads images 2. Binary Object is converted to a string with the purpose of extracting the XMP Meta Data 3. Line Breaks are removed which simplifies the RegEx 4. Filtering for just one file and one column 5. Saving as CSV or TXT (not gz compressed!) In Sublime both the CSV and TXT are not of the type text. However, the column spec of the Extract Table Spec node tells otherwise. Reading one of the saved files via a hex editor, seems to prove that the data, whilst it should be text, is not. ![image](https://forum-cdn.knime.com/uploads/default/original/3X/f/d/fd03d793541f5e1e2d6d0514846a4dbdd25b1771.jpeg) Opening the csv in Excel apparently does not quite work: ![image](https://forum-cdn.knime.com/uploads/default/original/3X/6/9/69d5df772f5e50373a1b553dba9204c51d61cea3.png) ![image](https://forum-cdn.knime.com/uploads/default/original/3X/3/5/35d1d07ffdb463d1716c629ef5b55448d051d729.png) I can reliably reproduce the issue and have created a workflow. In order to execute it, you must [download the image](https://new.bellevue-hotel.de/wp-content/uploads/2014/12/IMG_0704-HDR.jpg) first as it’s of \>6 MB in size. Some minor adjustments to the paths are required as well. I will upload the workflow soon (Knime is currently unresponsive). The point is that this seems to cause Knime data preview, node configurations and sometimes Knime itself to snap, requiring a forced kill via task manager. Edit: had to rebuild the workflow as Knime, well, crashed because of this issue. Here is the workflow: > **[Binary Data not converted to String - Loading Table Preview impossible –...](https://hub.knime.com/mwiegand/spaces/Public/latest/Bugs/Binary%20Data%20not%20converted%20to%20String%20-%20Loading%20Table%20Preview%20impossible~3Ey8duwnT01GLZmz)** > > Solutions for data science: find workflows, nodes and components, and collaborate in spaces. Best Mike