Clean data sentence by sentence

Hi all,

I have a CSV file which has 2 rows and 7 columns.  However I am only interested in preprocessing the last column (the blue color).

Example of input file is attached in inputproduct.jpg.

How should I do the cleaning/preprocessing text sentence by sentence?

I have used the KNIME to do that for me, but it gives me the terms in separated lines. 

Example of cleaned data file is attached in outputproduct.jpg.

I want the Term as String Column to be put in one line/sentence, for instance

pleasant maneuvering coupe drove

aware dishonest nearly cabin otherwise

Can KNIME do that for me?

Many thanks.

Fizah.

If You want to (pre)process Your documents sentence by sentence You can use the "Sentence Extractor" node right after the Parser node (or Strings to Document node). This node will extract all sentences of all documents as string column. After that (create a title for each sentence e.g. "" and filter old documnt column) use the "Strings to Document" node to convert the sentence strings into documents. Now You have a document for each sentence. These documents can than be preprocessed the regular way, meaning that You can use the "BoW creator" followed by any preprocessing node. If You want to get the text of the preprocessed documents after the preprocessing is done, You can group by the Document Column ("GroupBy" node) and than use the "Document Data Extractor" to extract the Fulltext of the document, which is now preprocessed, as string column.

[Any Parser]->[Sentence Extractor]->[Java Snippet (to create a title string e.g. "")]->[Column Filter (to filter the old Document column)]->[Strings To Document (create a Doc for each sentence)]->[BoW]->[Preprocessing nodes e.g. Stopword filter, ...]->[GroupBy (group on Document column)]->[Document Data Extractor (extract Fulltext as string)]

Hope this helps.

Cheers, Kilian

If You just need the preprocessed sentences (at the end of your preprocessing procedure) as a string column, there is also a more easy way to do this. After You red the documents with any parser node use the "BoW" node than directly use any preprocessing node e.g. Stopword filter etc. (with the setting deep preprocessing, which is switched on be default anyway). Now that You have the preprocessed documents group by the Document column, and after that use the Sentence extractor.

[Any Parser]->[BoW]->[Preprocessing nodes e.g. Stopword filter, ...]->[GroupBy (group on Document column)]->[Sentence Extractor]

The result of this is among others one string column, containing the sentences of all preprocessed documents.

Kilian

can you help me giving dataset for 20newsgroup  in knime tech.

or can you provide me dataset  for  textpreprocessing which contains only

text documents.

 

how to read a word file, pdf in knime tech.  and how to read multiple files ?