Maximum data Knime can accept from a CSV file?

[vijaykumar]

I am trying to import 8Lakh rows with 70 columns from a csv file using csv reader but it is getting failed after loading 35% what could be the issue and how can we solve it. i don’t want to decrease the number of rows.

[ipazin]

Hi there!

Welcome to KNIME community!

That is 800.000 rows, right? Don’t think KNIME should have a problem with it. Do you get some error message?

Br,
Ivan

[agaunt]

Hi @vijaykumar! In general, it is recommended to use the File Reader node instead of the CSV Reader node (because it is more flexible). In many cases, the problem is a ’ or " in some of your cells. You can deactivate this behaviour in the File Reader node under “Advanced” -> “Quote support”.

[mlauber71]

As @agaunt has suggested you first should try the File Reader. If you still experience problems with CSV import you could try and use the Readr package in R.

Error with File Reader node while running in a loop

I had a similar problem where there were some ‘dirty’ CSV files out of a large number of files and they were too large to just ‘edit’ them in Notepad++ (and that also was not a very nice process). I built a workflow using the R package readr, which seems to me more tolerant towards problematic CSV files and gives you some control how to handle mismatching quotes and so on. You might have to toy around with the settings of the read_delim function. [46] kn_example_import_csv_r_readr.knwf (52.…

String to document Error

The text file looks a bit challenging. Large lines with various special characters, text with commas, quotation marks and then , as column separator … what could possibly go wrong … but as usually Readr saves the day … (hopefully) [image] [image] kn_example_r_readr_string_document.knwf (396.3 KB)

[evert.homan_scilifelab.se]

The Actian nodes used to be really good for handling big files, but they are still not working with the latest version of Knime:

https://docs.actian.com/dataflow/6.6/index.html#page/InstallConfig%2FInstalling_DataFlow_Extensions_in_KNIME.htm%23wwconnect_header

[vijaykumar]

I am getting when i have tried uploading Raw data but i have done the cleaning in Jupyter and uploaded the processed data and it is working fine. I think this area Knime should work and make the product more robust.

[ipazin]

Hi there!

Glad it worked out with Jupyter. Did you know that:

1. You can use the Python code found in Jupyter notebooks from the Python scripting nodes in KNIME
2. You can execute KNIME workflows directly from within Python. If you are working within a Jupyter notebook you can also get a (static) view of the workflow in the notebook.

Here is a blog post where you can read more about it:
https://www.knime.com/blog/knime-and-jupyter#footnote

Tnx for a feedback.

Br,
Ivan

[Iris]

Hi @vijaykumar

one of the things I would do here is using streaming. You can find more in our blog post about optimizing the speed: https://www.knime.com/blog/optimizing-knime-workflows-for-performance

Cheers, Iris

PS: and as a small hint: check out the nightly built if you want to see a faster KNIME