# How to deal with unicode / special / invisible characters? **URL:** \ **Category:** KNIME Analytics Platform\ **Created:** [October 14, 2022, 4:09pm UTC](https://forum.knime.com/t/how-to-deal-with-unicode-special-invisible-characters/48095 "2022-10-14T16:09:07Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![mwiegand](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mwiegand/32/66896_2.png) [@mwiegand](https://forum.knime.com/u/mwiegand)\ **Post date:** [October 14, 2022, 4:09pm UTC](https://forum.knime.com/t/how-to-deal-with-unicode-special-invisible-characters/48095/1 "2022-10-14T16:09:07Z")
Hi there, once a while I happen to encounter a very nasty data issue, caused by copy & paste i.e. form Excel (iso-1252?) to a web editor (utf-8). Issues can manifest in corrupt data being saved (broken CSV) to processing in Knime coming to a complete halt. In this particular case it’s an Zero Width Space Unicode Character `<0x200b>` which only becomes visible when copy & pasting the cell at fault into an editor like sublime. Though, you must be very lucky picking the exact cell or tediously reduce the data set to gradually narrow it down. ![Bildschirmfoto 2022-10-14 um 18.01.32](https://forum-cdn.knime.com/uploads/default/original/3X/1/8/1829855adc8c9849efa0f2f875a42dbc1f45987b.jpeg) The question is, how would you deal with that issue? I even struggle to properly phrase it. Is it an encoding discrepancy or something entirely else? Thanks for your feedback! Best Mike ---
**Author:** ![badger101](https://forum-cdn.knime.com/user_avatar/forum.knime.com/badger101/32/45755_2.png) [@badger101](https://forum.knime.com/u/badger101)\ **Post date:** [October 14, 2022, 6:42pm UTC](https://forum.knime.com/t/how-to-deal-with-unicode-special-invisible-characters/48095/2 "2022-10-14T18:42:41Z")
Hi @mwiegand , here’s a similar issue with a possibility of a similar solution: > [@Weird Character Appearing](https://forum.knime.com/t/weird-character-appearing/47511/8): > > @badger101 a very strange case indeed. It you want to spot such things best copy them into an advanced editor like notepad ++ or visual studio code or something. You might have to escape the unicode string in the RegEx syntax with KNIME. regexReplace($$CURRENTCOLUMN$$, "[\\u200E]" , "") Another approach can be to eliminate all strange characters and only leave the ones you explicitly allow (there was a debate about this before): regexReplace($$CURRENTCOLUMN$$, "[^\\w\\d\\s.,!@#$%&\*()=+~-]","… ---
**Author:** ![mwiegand](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mwiegand/32/66896_2.png) [@mwiegand](https://forum.knime.com/u/mwiegand)\ **Post date:** [October 14, 2022, 7:26pm UTC](https://forum.knime.com/t/how-to-deal-with-unicode-special-invisible-characters/48095/3 "2022-10-14T19:26:53Z")
Thanks for the suggestion but my case is slightly different as the character I am talking about is not visible per se. I tried search and replace but no approach can actually find the characters in question. Here is the workflow. Not, though, I could only reproduce it since I found another occasion but I never was able to deliberately cause that kind of data issue by manually creating the text myself. > **[Unable to deal with invinsible inucode characters – mwiegand](https://hub.knime.com/mwiegand/spaces/Public/latest/Bugs/Unable%20to%20deal%20with%20invinsible%20inucode%20characters~TBN1BdZ7wJ8uLHbu)** > > Solutions for data science: find workflows, nodes and components, and collaborate in spaces. Best Mike ---
**Author:** ![ArjenEX](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/a/97f17d/32.png) [@ArjenEX](https://forum.knime.com/u/ArjenEX)\ **Post date:** [October 14, 2022, 7:37pm UTC](https://forum.knime.com/t/how-to-deal-with-unicode-special-invisible-characters/48095/4 "2022-10-14T19:37:48Z")
It’s better to take the other approach as mentioned in the second part of the linked post: only allow certain chars and remove the rest. In its most basic form: `regexReplace($Text$, "[^A-Za-z]", "SUCCESS")` ![image](https://forum-cdn.knime.com/uploads/default/original/3X/0/f/0ffc5bdc483d03e9084cc7cabe0a8455e122faf0.png) ---
**Author:** ![mwiegand](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mwiegand/32/66896_2.png) [@mwiegand](https://forum.knime.com/u/mwiegand)\ **Post date:** [October 15, 2022, 11:50am UTC](https://forum.knime.com/t/how-to-deal-with-unicode-special-invisible-characters/48095/5 "2022-10-15T11:50:51Z")
Thanks for your suggestion @ArjenEX. Albeit it shows that Regex can target these sort of characters this RegEx is slightly too generic creating the possibility of false positives. Wonder, if matching 1. Exact RegEx for unicode doesn’t work means it’s interpreted 2. Searching for RegEx any whitespace `\s` doesn’t work 3. What possible character the zero with space could match to in RegEx However, that would not help to circumvent any non-compliant character being present. Maybe the issue must be tackled from a different angle … will give that some thought over the weekend. Best Mike ---
**Author:** ![mlauber71](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mlauber71/32/73590_2.png) [@mlauber71](https://forum.knime.com/u/mlauber71)\ **Post date:** [October 15, 2022, 1:12pm UTC](https://forum.knime.com/t/how-to-deal-with-unicode-special-invisible-characters/48095/6 "2022-10-15T13:12:14Z")
@mwiegand can you provide an example of such a special character in a UTF-8 file or something (edit: I might check your example). And the I think like in the thread @badger101 mentioned that maybe defining a positive list of characters and eliminating all the rest would be a good way forward. Since if your data might throw such Unicode characters at you more often you might easily miss one by just going after the one. I tried to put together a collection of such characters for English but have not yet found a sufficient string for: European characters with reasonable special characters (Blanks, punctuations …) but no hidden blanks, ascii blocks and so on … that would be the challenge by going with a positive list. ---
**Author:** ![takbb](https://forum-cdn.knime.com/user_avatar/forum.knime.com/takbb/32/57077_2.png) [@takbb](https://forum.knime.com/u/takbb)\ **Post date:** [October 15, 2022, 2:19pm UTC](https://forum.knime.com/t/how-to-deal-with-unicode-special-invisible-characters/48095/8 "2022-10-15T14:19:45Z")
Hi @mwiegand Looking up this specific character, it falls into a class of unicode characters termed “Formatting”. I was recently investigating how to strip emoji and other characters from strings, and in the process discovered some quite useful info. In regex, there are a set of regex patterns describing unicode “classes” or “categories”. Once you know the class you wish to find or filter, you can then use these patterns. The pattern for this particular Formatting class is `\p{Cf}` , so if you replace your regex “SUCCESS” node to look for that class instead, you will find SUCCESS! 😉 ![image](https://forum-cdn.knime.com/uploads/default/original/3X/6/d/6d99dce091802a6fc1e67b0323e296d2a73f1820.png) I had a go at putting together a component for filtering based on these classes, with some success. It can be found here, and is useful if the aim is simply to remove characters of particular classes. > **[String Emoji and Character Class Filter – takbb](https://hub.knime.com/takbb/spaces/Public/latest/Components/String%20Emoji%20and%20Character%20Class%20Filter~owbALkyOgnCFmZNR)** > > String Character Class and Emoji Filter Filter out emoji and other classes of characters from a string using a built in Regular Expression. This component can … (If the config screen for the component only initially has a small number of classes listed, try executing the component to “refresh” it. I probably need to fix something, but not quite sure where.) Some information about the different unicode categories, such as the “formatting” category can be found here: > **[Find all Unicode Characters from Hieroglyphs to Dingbats – Unicode Compart](https://www.compart.com/en/unicode/category/Cf)** > > All Unicode Symbols with Names and Descriptions on One Page ---
**Author:** ![DiaAzul](https://forum-cdn.knime.com/user_avatar/forum.knime.com/diaazul/32/48297_2.png) [@DiaAzul](https://forum.knime.com/u/DiaAzul)\ **Post date:** [October 15, 2022, 2:24pm UTC](https://forum.knime.com/t/how-to-deal-with-unicode-special-invisible-characters/48095/9 "2022-10-15T14:24:19Z")
@mwiegand You can use the string replacer node to remove Unicode characters. You need to enter the value as a escaped Unicode character `\u200b` and check the ‘use backslash as escape character’ checkbox. This will detect the character as required. ![Screenshot 2022-10-15 151051](https://forum-cdn.knime.com/uploads/default/original/3X/9/b/9becbb2e124515b8c54822f075af645c5061b307.png) My previous (deleted) post was pointing the finger at KNIME not identifying Unicode characters correctly. Whilst, KNIME does identify escaped Unicode characters in the String Replacer node, it does not identify Unicode characters in the String Manipulation node. If you enter the formula `string(“\u200b) in the string manipulation node then it returns the text “\200b” rather than the Unicode character. Note, that if you enter string(”\n") in the string manipulation node then it does escape the character correctly. It is only Unicode support that is missing in the String Manipulation node - which is a bit of a nuisance. DiaAzul ---
**Author:** ![mlauber71](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mlauber71/32/73590_2.png) [@mlauber71](https://forum.knime.com/u/mlauber71)\ **Post date:** [October 15, 2022, 8:25pm UTC](https://forum.knime.com/t/how-to-deal-with-unicode-special-invisible-characters/48095/10 "2022-10-15T20:25:10Z")
@DiaAzul have you tried a double escape? It does work in the [String Manipulation (Multi Column) – KNIME Community Hub](https://kni.me/n/XHEmLNBRrffGaxF4) > [@Weird Character Appearing](https://forum.knime.com/t/weird-character-appearing/47511/8): > > @badger101 a very strange case indeed. It you want to spot such things best copy them into an advanced editor like notepad ++ or visual studio code or something. You might have to escape the unicode string in the RegEx syntax with KNIME. regexReplace($$CURRENTCOLUMN$$, "[\\u200E]" , "") Another approach can be to eliminate all strange characters and only leave the ones you explicitly allow (there was a debate about this before): regexReplace($$CURRENTCOLUMN$$, "[^\\w\\d\\s.,!@#$%&\*()=+~-]","… ---
**Author:** ![DiaAzul](https://forum-cdn.knime.com/user_avatar/forum.knime.com/diaazul/32/48297_2.png) [@DiaAzul](https://forum.knime.com/u/DiaAzul)\ **Post date:** [October 15, 2022, 9:52pm UTC](https://forum.knime.com/t/how-to-deal-with-unicode-special-invisible-characters/48095/11 "2022-10-15T21:52:49Z")
@mlauber71 , You are correct to the extent that, if you use `regexReplace` and double escape the Unicode character `\\u200b`, the character will be substituted by the replacement text. However, the normal `replace` function does not work with either single or double escaped Unicode characters. Double escaping in strings is not a good implementation within KNIME. Personally, I would consider this an undesirable implementation and regex strings should be consistent with other programming languages (i.e. only one backslash is required to escape a character). It is also inconsistent with other fields and nodes in KNIME where double escaping is not required. As a tool which is aimed at the low-code community lack of consistency within the software and with other software tools is a nuisance which is difficult to explain to new users. ---
**Author:** ![bruno29a](https://forum-cdn.knime.com/user_avatar/forum.knime.com/bruno29a/32/28802_2.png) [@bruno29a](https://forum.knime.com/u/bruno29a)\ **Post date:** [October 16, 2022, 5:11am UTC](https://forum.knime.com/t/how-to-deal-with-unicode-special-invisible-characters/48095/12 "2022-10-16T05:11:25Z")
Hi @mwiegand here’s other threads about “invisible” characters: > [@Remove hidden characters](https://forum.knime.com/t/remove-hidden-characters/35119): > > Hello, I have hidden characters in the data set I got and some special characters. I was able to remove the carriage returns with the string replacer node but I was not able to remove other hidden characters like the one after the string Pa.m3.mol-1 in my attached example. Please let me know if there is way to remove my hidden character and/or how to remove hidden characters at all. Thank you for your help. Claude. [Example\_with\_hidden\_characters.xlsx](https://forum.knime.com/uploads/short-url/eOw6iKHRvN2lY89vgIFS5AblqtW.xlsx) (8.5 KB) > [@Remove special characters from the text](https://forum.knime.com/t/remove-special-characters-from-the-text/35457): > > This never happened before but ones again I’m searching for your help in solving very simple task. I have a text parsed by Tika Parser URL input. The source is pdf file. The file itself is rather simple but there’re lists like: • Advocate for change and work with the company to adopt • Partnered with a local domestic violence shelter to provide • Participated in an EMEA-wide Diversity Dialogue new policies and services. And like „ Reducing our environmental footprint. [the character isn’t pr… ---
**Author:** ![mwiegand](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mwiegand/32/66896_2.png) [@mwiegand](https://forum.knime.com/u/mwiegand)\ **Post date:** [October 17, 2022, 8:51am UTC](https://forum.knime.com/t/how-to-deal-with-unicode-special-invisible-characters/48095/13 "2022-10-17T08:51:43Z")
A wonderful good morning! I love this community …the nastier the problem, the more likely you are finding some enthusiasts with great ideas. To conclude / recap: 1. @takbb found it’s a class of unicode characters called “Formatting” Note: I found with his help this [comprehensive overview](https://www.regular-expressions.info/unicode.html) (section Unicode Categories) 2. Several interesting issues related to Knime nodes processing these characters differently / inconsistently were noted by @DiaAzul (namely String Manipulation Node) **Solutions identified** The shared component provides a generic by @takbb seems to provide a quite generic and versatile approach: [String Emoji and Character Class Filter – KNIME Hub](https://hub.knime.com/takbb/spaces/Public/latest/Components/String%20Emoji%20and%20Character%20Class%20Filter~owbALkyOgnCFmZNR) For more specific use cases his solution pointed out by searching for the character class `\p{Cf}` works like a charm as well. Thanks to all for your time and feedback! Best Mike ---
**Author:** ![system](https://forum-cdn.knime.com/uploads/default/original/1X/ab3ccf34482a0329361734a18199390177204f15.png) [@system](https://forum.knime.com/u/system)\ **Post date:** [October 24, 2022, 8:52am UTC](https://forum.knime.com/t/how-to-deal-with-unicode-special-invisible-characters/48095/14 "2022-10-24T08:52:19Z")
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed. ---
**Author:** ![ScottF](https://forum-cdn.knime.com/user_avatar/forum.knime.com/scottf/32/61316_2.png) [@ScottF](https://forum.knime.com/u/ScottF)\ **Post date:** [January 22, 2024, 3:01pm UTC](https://forum.knime.com/t/how-to-deal-with-unicode-special-invisible-characters/48095/15 "2024-01-22T15:01:11Z")
---
**Author:** ![ScottF](https://forum-cdn.knime.com/user_avatar/forum.knime.com/scottf/32/61316_2.png) [@ScottF](https://forum.knime.com/u/ScottF)\ **Post date:** [February 5, 2024, 2:19pm UTC](https://forum.knime.com/t/how-to-deal-with-unicode-special-invisible-characters/48095/16 "2024-02-05T14:19:39Z")
Hi all, we have a new blog post out this week that is related to this thread: > **[Invisible Unicode characters: How to find and remove them | KNIME](https://www.knime.com/blog/how-to-find-remove-invisible-unicode-characters/)** > > Get a quick data cleaning fix for finding and removing invisible unicode characters with KNIME. Happy reading! ---
**Author:** ![takbb](https://forum-cdn.knime.com/user_avatar/forum.knime.com/takbb/32/57077_2.png) [@takbb](https://forum.knime.com/u/takbb)\ **Post date:** [February 5, 2024, 3:26pm UTC](https://forum.knime.com/t/how-to-deal-with-unicode-special-invisible-characters/48095/17 "2024-02-05T15:26:09Z")
Thanks @ScottF. Nice article 🙂 This topic came up on another post recently, with the problem of identifying “spaces”. It mentions some additional components that can assist: > [@Table difference Finder - incorrect differences in column name identified](https://forum.knime.com/t/table-difference-finder-incorrect-differences-in-column-name-identified/76779/6): > > OK, @taylorpeter55 , you have a Unicode non-breaking space character \u00A0 in one of the outputs at the point where you have a regular space (Unicode \u0020) in the other. I suspect this is there courtesy of Excel. This is between the words “Amount” and “Currency”. You can read the text downwards in the first column in each table. From the workflow, it looks like you are still on KNIME 4.7. There is a String Cleaner node in KNIME 5.2 but I don’t think that would help here, because y… In particular, it seems that Excel likes to add “non breaking spaces”, and these can be “fixed” instead of “stripped” using Unicode Space Replace: > **[Unicode Space Replace – takbb](https://hub.knime.com/takbb/spaces/Public/Components/Unicode%20Space%20Replace~13SqtWGfsGk4CnmK/current-state?pk_vid=ae85398c035cb50517071462430655e1)** > > Replaces various Unicode "space" characters found in a String column with a regular space @takbb Brian Bates 24 Jan 2024 ---
**Author:** ![mwiegand](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mwiegand/32/66896_2.png) [@mwiegand](https://forum.knime.com/u/mwiegand)\ **Post date:** [February 7, 2024, 11:01am UTC](https://forum.knime.com/t/how-to-deal-with-unicode-special-invisible-characters/48095/18 "2024-02-07T11:01:03Z")
Lovely to see this super annoying but interesting topic being shed some light on.