UTF-8 characters not rendering in table view

[Aaron_Hart]

অস্ট্রেলিয়া is apparently the Bengali translation of Australia. If you pasted this text into the table creator it can not be rendered by KNIME even though the underlying data is retained. This text is visible in other places on my local system including several text editors. This is a bit distracting as I would like to be able to visually confirm that this data is at least present and appears to be correct (not that I can read Bengali of course).

Edit: Adding an actual question: are there any eclipse settings that can be used to get the text to render properly or is this an issue internal to KNIME?

[quaeler]

What version of KNIME on which OS are you using? This appears to work fine for me under macOS 10.13 :- /

[swebb]

You could try adding the following line to your knime.ini file found in the program directory (in case someone stumbles across this in the future)

-Dfile.encoding=UTF-8

[Aaron_Hart]

Good reason to use a mac but sadly I’m on windows 10

Also, thanks for checking!

[Aaron_Hart]

@swebb

-Dfile.encoding=UTF-8

I thought that might work as well but no dice. There are a lot of font preferences in eclipse, I wonder if any of them are used in the KNIME table output renderer?

[mlauber71]

You could try to set different encodings under the preferences. Maybe that could help.

Why does the xpath node get incorrect data in the Indian language?

I changed the HTML parsing and retrieving part to some Palladian nodes and used your XPath definition and now it might work (although I cannot read the words). Maybe you want to check if this is what you need. Make sure you use UTF-8 as encoding. [image] [image] [image] [image] [image] kn_example_xpath_indian_words.knwf (198.0 KB)

[swebb]

Hrm, I thought that’s how we resolved the issue when we were having a similar problem reading UTF-8 strings out of a database.

I’ll double check that’s the property we set with my colleague tomorrow.

Did you re-read in the content after you’d made the change (not that I think that would necessarily be required).

But actually mlauber71’s solution looks better.

[Aaron_Hart]

Yeah, I double checked the setting was in my .ini file and confirmed the setting from the “About KNIME” > “Configuration” window. I tried re-executing everything to no avail as well. I followed mlauber71’s link (thanks btw) but had already tried that and to me it seems like these settings are specifically for files. I just would like to display the data correctly. Files are not a problem for us. The fact that it works by default on Mac seems like a good hint. Either something in java or related to system fonts maybe?

Does anyone know how to change table fonts in KNIME?

[mlauber71]

I can reproduce the problem on a Windows 7 machine. The view in KNIME does not show Bengali letters although it can display Hindi letters. The processing does work in both cases in files, when I write it into a workflow description.

Also the exports into Excel and CSV (UTF-16 - UTF-8 not tested) does work as expected. It seems to be a display problem within KNIME data on Windows (7). On MacOS it seems to be OK but I will have another look later.

Maybe someone with a Linux machine can check the examples.

You could write to KNIME support about that or @ScottF could put it on a list …

kn_example_utf8_hindi_bengali.knwf (21.3 KB)

[izaychik63]

On Windows 10 I see same squares if past example text in the Table Creator field.

[mwiegand]

Hi,

I am teviving this old topic with the hope to get an answer to the presumably same issue but on Mac OSX.

Java Snippet: Convert Unicode to String not working all the time

Hi, I am testing something out and struggle to identify possible reasons why some Unicode characters like “\u1E900” which translates to 𞤀 is converted via the following mechanism to Ẑ0 instead? Test Data Code Point in 4-Digit Hexadecimal Unicode Character UTF-8 Result \u1E900 𞤀 Ẑ0 \u1E922 𞤢 Ẓ2 \u1E922 𞤢 Ẓ2 \u1E900 𞤀 Ẑ0 \u1E901 𞤁 Ẑ1 \u1E923 𞤣 Ẓ3 \u1E923 𞤣 Ẓ3 \u0050 P P \u0070 p p \u0070 p p \u0050 P P Java Code // system imports import org.knime.base.node.jsnipp…

I am currently testing the entire Unicode character set and have noticed that StringEscapeUtils.unescapeJava doesn’t convert all characters properly. I doubt it’s a font issue as the characters extracted from the Unicode namelists such as this are displayed correctly. The source is UTF-8 encoded which likely eliminates an encoding mismatch.

In a very old external post, which might not be valid anymore, it was once stated:

With ICU4J you can use com.ibm.icu.impl.Utility.unescape(String s) to convert the literal string to utf8 string. However, java string internally doesn’t use utf8 encoding, instead it uses UTF-16 (Big Endian) to present unicode characters. To fully convert the string from utf8 literal to java unicode representation, you need to decode it with ISO-8859-1 then read the bytes back to string using encoding UTF-8.

I wonder if you have an idea @mlauber71 as you are one of the most skilled cracks around

Cheers
Mike

[mwiegand]

FYI – I got this resolved in the linked post of mine. Though, I used the help of ChatGPT. Interesting experience … still feel a bit filthy.