I want to read very large Data (from a Database consisting of 15Mio. entries) using the SDF Reader Node. (I already increased my heap space inner the knime.ini)
Now, if I open/look up the file which was read in, there are some entries lost (especially in one specific column). There only appears a "?" instead of the real entry. I´m sure that there are no values missing in the data(base) which is read by the SDF Reader.
Could you please help me with this problem? did anybody observe similar problems reading large data?
This doesn't have to do with the file size or the available memory. Can you verify that there is really a value for each molecule in your data? Also empty properties may show up as missing values.
You could also check whether the column types (guessed during the node configuration step!?!) are reasonable. It might happen that the first k molecules in your dataset contain only numbers in some property, so the node will try to generate a numerical column and then fail if suddenly alphabetic characters appear.
I´m sure, that there are no empty properties in the data! the columntypes are defined in the configuration settings (here double) therefore not guessed. I also found out that it might not have to do whith the values by themselves, as e.g. there is a negative value missing for one molecule but in another row an negative value (for the same column) coud be read ... ? furthermore there only are doubles in that column (in the database I want to be read) and the missing values don´t appear from a specific row on...