New problem requiring solution and probably it is of interest to other KNIME users as well.
I have a workflow which to the above of its main task allows translation of the text with the use of Google Translate API.
Google Translate has a limit on the size of the text it translates by API, so I made a Loop which is sending the text with a limited number of characters. After I made several mistakes, e.g. it appeared that Chinese characters must be treated as two characters not like one and the text in Chinese must be 2 times shorter then in European or Arabic, I made a workflow which works almost fine. I was happy but it was a fullish happiness – I cut the text no matter boundaries of words, so mistakes appear in translation. Not too often in fact, but they are boring.
So, the question:
How to split the text with the loop to have no more then like 1000 characters to process, but in case this 1000 splits a word in half, process only the text till the end of the last complete word? Yes, obviously the next request should start not with the 1001 character but with a character following the last translated word.
Sorry for long and not very clear explanation – even in my native language it sounds a bit strange, but I hope you understand me.
Interesting question (as usual with your posts :- )!
This is a quick reply but please let me know if it is not clear enough to further develop it.
What I think you need is to know where the different word separators appear, i.e. create a list of blank separator indexes and then separate words based on a modulo 1000 of these indexes (and not the hard 1000 cut off). To achieve this, I suggest the following trick:
use a “cell splitter” node to split your sentence into a list of words
ungroup using the ungroup node
with the “string manipulation” node, calculate the length of every word and add 1 (the separator has disapeared but needs to be count too )
Use the “moving aggregation” node to calculate the cumulative sum of word length
At this point you know when you reach the “less than but nearest” 1000 cut-off based on “cummulative sum word length” and you just need to use modulo 1000 for your words to split it into chunks of “less than but nearest” 1000 characters without chopping any word !
The spirit of the idea is here. I’m explaining this by heart without KNIME on hands but it should work. Otherwise, please come back and I’ll explain it better
Maybe a coding node is also an option for you. (Note: I probably have to sort out the missing values first if the text is not exactly divisible by 1000) I used @aworker dataset as example (Thanks for that!) here and reduced the spilt to 100 words
Sorry for a very-very late reply – I got COVID and was unable to work for several days, then missed deadlines did not give me a chance to test proposed solutions. My apologies.
Now I am back and fully operational. @aworker aworker – your approach works perfect with a plain text. This is certainly a solution of the problem. I had to deal with some extra challenges specific to my task (when split by space the text loses paragraphs and/or the node fails). Also (I do not know why in fact; may be something is wrong on my side) html formatted text leads to fail of the cell split node. I will have to search for the right approach to this problem… @Daniel_Weikert Daniel_Weikert – I am sorry that I did not ask as soon as you posted your solution. Can you please share the workflow or at least the code in python that you used?
I’m really sorry you got Covid and hope you are now feeling much better and fully recovered.
Thanks for your feedback. My implementation was certainly just a beginning of a possible solution that no doubts need to be adapted to fullfil your needs, for instance if your text is html tagged as you said.
I do not understand what you mean by “(when split by space the text loses paragraphs and/or the node fails).” Paragraphs should not be lost because of space splitting or the nodes fail. What node in particular is failing? Would you need extra help, could you please share your workflow or the data where it is failling ?
)!
)
!
