i am working on a dataset of roughly 4 Mio rows where I need to extract all (roughly 900k) distinct networks. In my understanding, there are two ways to extract these subnetworks: either (1) by using the Network To Row node, or (2) by using the SubGraph Extractor node.
Option (1) is beyond the capacities of my computer, as it consumes too much ram and too much processing power to send just the whole dataset through it.
Option (2) could in theory, in combination with a Node Neighbor Extractor node, circumvent the limited power of my machine when being coupled with a Recursive Loop. (As in the screenshot - sorry, I can’t just upload the workflow as the data is rather sensitive). Yet when trying to execute the loop, I get the following error message:
“no network view found in cache for id…”
It thus seems that the SubGraph Extractor node (as well as many other network mining nodes) expects receiving always the same network with the same ID. Is there a way to tell the nodes to deal with the updated network?
the network ports only hand over a pointer to the network in a global cache but not the network itself. When a network node is reset the network it has been created is removed from the cache. So as soon as the loop body is reset the network that you pass on to the next round via the delegation loop is removed from the cache which results in your error message. Try to replace the Model to Cell and Cell to model node with the Network to Row and Row to Network node.
Thank you for the reply, Tobias!
This at least solves the mystery. I’ll thus need to upgrade my machine in order to pperform the network operations, as I tried to resolve to the looping option just in order to avoid the memory-intensive network to row and row to network nodes.
ah maybe I should have mentioned that you can disable the caching of network cells
When you use the Network to Row and Row to Network node the networks are cached by default in memory to increase the performance. You can disable this via the preferences. Just go to File->Preferences and in the Preferences Dialog go to KNIME->Network where you can disable the caching.
Sorry about that