Hello, I’m having an issue with a DB Reader which is taking multiple hours loading but never fully loading. The number of rows going into it is about 300,000 so I expect it to take some time but not all day. This happens both in the server and locally.
Hi @ORice36 ,
is your DB hosted locally or remotely? Depending on your system, Windows vs. OS X vs. Linux, you might want to ping or tracert to identify if there are any network bottle necks.
Best
Mike
4 Likes
Hi @ORice36 , in addition to network performance, another likely reason for sluggish reading of data is the query itself. The DB Reader simply actions the query that been built by the other DB nodes that precede it. Do you have any idea of the complexity of the generated query and how well it is performing on the database? What other db nodes are you using prior to the DB Reader?
It might be you need some additional indexes, or it requires other optimisation.
Also how much data is there in each row? “300,000 rows” doesn’t tell us much about the quantity of data being returned, and it could be multiple columns each with large data items.
Another factor is the amount of memory on your system, that is made avaliable to KNIME. A lack of memory could slow things down as the amount of data increases.
You could also look at the Fetch Size config on the DB Reader and adjust that. See the node documentation.
1 Like
@ORice36: What do you mean taking multiple hours? It is taking time to get to 100% and turned Green or It turns green and your KNIME modern UI freezes?
Both have separate solutions. I connect remotely my workplace DB and for million it takes seconds turns the light to green. Of course, when I work with large number of rows I keep the KNIME UI in Classic mode.
I have no idea, which issue you are referring to? Also let me know when you run same query on the same the Table/View/Function with DBeaver (also has UI of eclipse and base of JavaDBC very similar performance you will get in KNIME) how much time that takes (Note: If your node not turning into green and stuck then try DBeaver solution and let us know)?
1 Like