CSV Reader: too few data elements problem

#1

I am having some issues with the CSV reader trying to read a CSV output from an ERP system.

I get the warning “Didn’t get any values for columns with indexes #46, 47, 49, 51, 52, 53. Please verify column types.” This warning confuses me because I only have 20 columns in the data.

Then it fails to execute with the error: too few data elements (line: 2 (1637)

When I enable short lines I get the same warning, but it does execute. Though for some reason it gets rid of my column headings and puts in a ton of blank columns at the end.

Is there anything I can do about this? I am trying to build a tool that I can leave with a client and having to rename 19 columns and drop 30 or so others seems like an opportunity for errors down the line.

Thanks in advance,
Eric

0 Likes

#2

Hi @ewhulbert -

Have you tried using the File Reader node in place of the CSV Reader node? It’s a little bit better about handling data with inherent inconsistencies.

If you’re still having trouble, feel free to post a short snippet of your dataset, if you can. Then maybe we can work through the parsing issues together. Even an anonymized version of the data with similar formatting would probably work, if there are confidentiality issues.

4 Likes

#3

Sometimes with more complicated files the R package Readr might help.

R - Readr - read CSV files

4 Likes

#4

Thanks gents. I had tried the file reader and it was not working. It would either put everything as one line or just give an error. But then I fixed a different problem that csv reader was having, a random " in a variable, and now file reader works great. I had no re-checked file reader after that fix before asking, rookie move there.

I figured out some of the problem with the extra columns too. There appears to be an error where part of a row is duplicated and it extends to a couple of extra unnamed columns.

2 Likes