parallel loop ?

Friendly greetings !

I'm a new user (since yesterday) and wrote this loop that read ~50Millions lines in 30 CSV files, do some simple processing and insert the result in a mysql database. (for easier processing later, and for learning purpose).

 

 

 

 

 

This loop could be parallelized without problem, but it isn't. (the bottleneck is the CVS Reader, CPU-Bound). Is there a way to explicitely parallelize this loop ?

 

Thank you very much, have fun :)

The image is gone :(

here is the url : http://fractals.s3.amazonaws.com/vrac/csvloop.jpg

ho nooooooo. i have only the last table in the database angry

Did you check the "Append table" option in the Database Writer?

i'm not sure what happened. but now it works.

(i rewrote the workflow many times)