DB to DB CDC/Streaming

I know KNIME has the streaming capability, but what if we want to constantly stream data from one DB to another DB all day, every day?

@elohbeck I think technically you could do that, question would be how to secure the running of the server/system and how to handle disconnects or crashes. I theme to use the knime sever and build my Workflows in a way that they would check what tasks have already been done and then continue the work (after a restart).


@mlauber71 thanks for sending - this looks promising. To your point, I’m wondering how to keep it running continuously, and how to keep it from inserting duplicate data. Continuous sync and reconcile…

I think this very much depends on your databases and their settings. For example what kind of unique ids they would enforce and if you would have to catch a failing process.

One way to do it would be to always check for ids that have not been transferred yet and just transfer them. I think you would have to put some effort into planning and testing.

1 Like

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.