I am trying to import 8Lakh rows with 70 columns from a csv file using csv reader but it is getting failed after loading 35% what could be the issue and how can we solve it. i don’t want to decrease the number of rows.
Welcome to KNIME community!
That is 800.000 rows, right? Don’t think KNIME should have a problem with it. Do you get some error message?
Hi @vijaykumar! In general, it is recommended to use the File Reader node instead of the CSV Reader node (because it is more flexible). In many cases, the problem is a ’ or " in some of your cells. You can deactivate this behaviour in the File Reader node under “Advanced” -> “Quote support”.
As @agaunt has suggested you first should try the File Reader. If you still experience problems with CSV import you could try and use the Readr package in R.
The Actian nodes used to be really good for handling big files, but they are still not working with the latest version of Knime:
I am getting when i have tried uploading Raw data but i have done the cleaning in Jupyter and uploaded the processed data and it is working fine. I think this area Knime should work and make the product more robust.
Glad it worked out with Jupyter. Did you know that:
- You can use the Python code found in Jupyter notebooks from the Python scripting nodes in KNIME
- You can execute KNIME workflows directly from within Python. If you are working within a Jupyter notebook you can also get a (static) view of the workflow in the notebook.
Here is a blog post where you can read more about it:
Tnx for a feedback.
one of the things I would do here is using streaming. You can find more in our blog post about optimizing the speed: https://www.knime.com/blog/optimizing-knime-workflows-for-performance
PS: and as a small hint: check out the nightly built if you want to see a faster KNIME
This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.