Table row to variable loop start / Loop End -/- mysql

Hi,

I don't understand why the "Table row to variable loop start" run very fast at the begin's run and along running, the running velocity slowed down when I go further. 

Example : the first 10 minutes the node process 5000 Datas and the 10 next minutes, the node process only 1500 Datas

Is it a RAM problem ? I am not sure.

How can I improve the process velocity ?

Thanks for answer,

Thierry Crettol

Dear Thierry:

Can't tell without looking at a workflow. What do you do with the data in the workflow? Do you collect all of it at the end? Do you see a constant throughput if you discard all data within the loop and concatenate empty tables instead? (Just for the sake of diagnosing this issue.)

- Bernd

Hi,

Thank you for your answer.

I give you a workflow to show you what happens. To run the workflow you need to create :

- a database named "testKnime" and add the following table to mysql :


CREATE TABLE `feed` (
  `id` int(11) NOT NULL,
  `title` text NOT NULL,
  `description` text NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

The result of the given workflow :

0 min : 0
4 min : 4110 rows processed
5 min : 4600 rows processed
6 min : 5156 rows processed
7 min : 5600 rows processed
8 min : 5980 rows processed
9 min : 6416 rows processed
10 min : 6800 rows processed
18 min : 9377 rows processed

 

So, you can find the workflow on this url :

http://ringtarget.com/workflow_group.zip

 

Thanks for your help and Happy new Year !!! :)

Thierry Crettol

It slows down from the "Variable to Table row" or from the "concatenate" Node. Beacause if I remove Column in the "Variable to Table row" Node it runs very faster !

 

It slows down from the "Variable to Table row" or from the "concatenate" Node. Beacause if I remove Column in the "Variable to Table row" Node it runs very faster !

The solution is maybe to use a ROWID colunn ...
And after concatenate with the row that match ROWID column

(Didn't prepare the DB but had a look at the workflow!)

If you have trouble with RowIDs then it might make sense to "Generate new RowIDs" in the Loop End node?

Hi,

Oki, thanks, I tried another way, using a ROW_ID key.

Instead of the Column appender, I made a new workflow with the Joiner node. It go faster.

See you.

Thierry Crettol