Hi there, once a while I happen to encounter a very nasty data issue, caused by copy & paste i.e. form Excel (iso-1252?) to a web editor (utf-8). Issues can manifest in corrupt data being saved (broken CSV) to processing in Knime coming to a complete halt. In this particular case it’s an Zero Width Space Unicode Character <0x200b> which only becomes visible when copy & pasting the cell at fault into an editor like sublime. Though, you must be very lucky picking the exact cell or tediously reduce the data set to gradually narrow it down. [Bildschirmfoto 2022-10-14 um 18.01.32] The question is, how would you deal with that issue? I even struggle to properly phrase it. Is it an encoding discrepancy or something entirely else? Thanks for your feedback! Best Mike

Hi @mwiegand Looking up this specific character, it falls into a class of unicode characters termed “Formatting”. I was recently investigating how to strip emoji and other characters from strings, and in the process discovered some quite useful info. In regex, there are a set of regex patterns describing unicode “classes” or “categories”. Once you know the class you wish to find or filter, you can then use these patterns. The pattern for this particular Formatting class is \p{Cf} , so if you replace your regex “SUCCESS” node to look for that class instead, you will find SUCCESS! :wink: [image] I had a go at putting together a component for filtering based on these classes, with some success. It can be found here, and is useful if the aim is simply to remove characters of particular classes. [image] String Emoji and Character Class Filter – takbb String Character Class and Emoji Filter Filter out emoji and other classes of characters from a string using a built in Regular Expression. This component can … (If the config screen for the component only initially has a small number of classes listed, try executing the component to “refresh” it. I probably need to fix something, but not quite sure where.) Some information about the different unicode categories, such as the “formatting” category can be found here: [image] Find all Unicode Characters from Hieroglyphs to Dingbats – Unicode Compart All Unicode Symbols with Names and Descriptions on One Page

How to deal with unicode / special / invisible characters?

KNIME Analytics Platform

bruno29a October 16, 2022, 5:11am 12

Hi @mwiegand here’s other threads about “invisible” characters:

2 Likes