Filtering out Chinese, Korean, Japanese, Thai characters with Python Script Node or Rule Engine using regular expression

[xli]

Hi all, I was trying to filter data records that contain any Asian characters and later on use a translator on them…I tried with Python Script with exact same code as what I do in Spyder, which works in Spyder but not in the Python Script in KNIME.

here is the code:
output_table_1 = input_table_1.copy()
def contain_chinese(check_str):
for ch in check_str:
if ‘\u4e00’ <= ch <= ‘\u9fa5’:
return True
return False

def contain_korean(check_str):
for ch in check_str:
if ‘\uac00’ <= ch <= ‘\ud7a3’:
return True
return False

def contain_japanese(check_str):
for ch in check_str:
if ‘\u0800’ <= ch <= ‘\u4e00’:
return True
return False

output_table_1 = output_table_1[output_table_1[‘Customer Name’].apply(lambda x : contain_chinese(x))]

And I also tried using regular expression but it doesn’t seems to work well neither…
here (example for Japanese), it would give me results that totally does not contain Japanese characters: ^.([\u0800-\u4e00]).

Therefore I end by with filtering out all latin characters with this Rule Engine Row Filter but it would still give me latin characters
here is what I put and i chose exclude TRUE matches: $Customer Name$ MATCHES “^[a-zA-Z0-9_?@&.,，~()（）^:;/=+~'’ àáâãäåæçèéêëìíîïðñòóôõöøùúûüýþÿ-]+” => TRUE

If anyone can give some advice, it would be really appreciated!
Thank you in advance!

[mlauber71]

@xli you might want to check if you will have to double escape the Unicode codes.

Weird Character Appearing

@badger101 a very strange case indeed. It you want to spot such things best copy them into an advanced editor like notepad ++ or visual studio code or something. You might have to escape the unicode string in the RegEx syntax with KNIME. regexReplace($$CURRENTCOLUMN$$, "[\\u200E]" , "") Another approach can be to eliminate all strange characters and only leave the ones you explicitly allow (there was a debate about this before): regexReplace($$CURRENTCOLUMN$$, "[^\\w\\d\\s.,!@#$%&*()=+~-]","…

There are several other threads and examples about RegEx that might help.

How to deal with unicode / special / invisible characters?

Hi @mwiegand Looking up this specific character, it falls into a class of unicode characters termed “Formatting”. I was recently investigating how to strip emoji and other characters from strings, and in the process discovered some quite useful info. In regex, there are a set of regex patterns describing unicode “classes” or “categories”. Once you know the class you wish to find or filter, you can then use these patterns. The pattern for this particular Formatting class is \p{Cf} , so if you …

[xli]

@mlauber71

escape

hi thanks for your reply.
I tried double escape but it still doesnt work. It’s not like it doesnt work at all, with the rule engine row filter, I did filtered some records with Asian form, but it would also give records that are all in English.

[xli]

hi thanks for your reply.
I tried double escape but it still doesnt work. It’s not like it doesnt work at all, with the rule engine row filter, I did filtered some records with Asian form, but it would also give records that are all in English.