Sticky Data and Sticky Chart Nodes

Node Development
1

What would it take for me to develop a “Sticky Data” node?

What is a Sticky Data node? Great question!

A Sticky Data node is a node that holds onto its last output tables even after the node is reset. When the node is run again, the output table data is updated and the node will hold onto this new output even if it is later reset.

Why have a Sticky Data node?

My actual problem is this. I develop workflows that take a long time to run but want to see intermediate results. I also want to develop “run forever” workflows for students so they can observe real-time emerging patterns from agent-based models.

Unfortunately the KNIME charts are blank 99% of the time waiting for the upstream workflow to finish running. Then you might see a quick flash from the chart as its input data is updated. But the workflow then loops, the nodes are reset, and the chart returns to a blank state.

The ideal would be to have both Sticky Data nodes and Sticky Chart nodes. But it may be possible to just develop a single Sticky Data node that does nothing other than pass through input data and then hold onto it. A regular Chart node could then sit downstream from the Sticky Data node, and the Sticky Data node could fool the Chart node into thinking the data is always ready.

I can foresee numerous complications with such an approach, but if Sticky Data nodes are possible then this might be a elegant solution to a difficult problem.

2

My first guess would be that you might be able to do something with the #loadInternals() and #saveInternals in the NodeModel - either to save the incoming table in the chart node (or a rendered view of it), or to save the output table in your ‘sticky data’ node. I’m not sure if somewhere deep in the KNIME core there is something that will delete any saved internals during a node reset though…

Steve

3

I think you can approach this problem by separating the model execution and the visualization into two different workflows:

  1. The modeling workflow runs in a loop and writes intermediate results to disk (e.g. via a Table Writer)
  2. The visualization workflow reads this file and visualizes it, you can add a check to only refresh the visualization when the input data changed, and use a Wait node to wait a bit before looking again.
    (you can use the File Meta Info node to check the last modified date)

best,
Gabriel

3 Likes
4

Very cool! Thanks for the suggestion!!

1 Like