consolidate multiple csv files & upload to db

KNIME Analytics Platform
1

Hi, i am new to KNIME. My requirement is to consolidate multiple csv files and upload the data to mysql database. i am able to consolidate using the below workflow.

image

I have 1 challenge here. Excel has 1048576 rows only, but i have data more than 11lakh. Therefore i have split the multiple file into 2 groups and consolidating. It worked fine, but i have challenge updating the consolidated 2 excel file to db.

Could some one help what is the optimized way of consolidating these file and upload to db?

Regards,
Pavan.

2

Hi,
I would try 2 options:

  1. Instead of Excel (with known limitations), write output to CSV file using “CSV writer” node and then upload data from the CSV file into DB.
  2. Using properly configured “Database Writer” node, write output to DB directly in KNIME workflow.

Martin K.

3

Hi Martin,

Thanks for your suggestions, i will definitely try them.

Regards,
Pavan.