Workflow Dokumentation - Node IDs in exact execution order

Hi,

my aim is to create an automated workflow documentation. Therefore, it would be nice to have all the node IDs of the workflow in a table in the EXACT order they are executed, together with the “custom description”. I can isolate the “custom description” from the settings.xml file and the node IDs from the knime.workflow and/or Workflow Summary using the XPath node but I failed to isolate the exact sequence of the nodes as they are executed in the workflow. I also tried the “Timer Info” node the same (even after sorting for execution time). A similar question was posted in the forum: Get the whole workflow flow information - #2 by Ema and How to write a documentation for a KNIME workflow? - #9 by Patrick1974

So my question…

Is there a way to isolate node IDs/Labels in the exact sequence of executed nodes of a workflow? E.g.:

Node 1 (executed) → Node 2(branch - exectuded) → Node 3 (not executed) → Node 4 (executed)

Table:

Node Id Node Name Description
Node 1 Excel Reader qrwqeqwe
Node 2 If Switch wqegesrz
Node 4 Column Filter qeqwewqe

The problem might be associated with the fact that my workflow is quite big with several branches and the node “composition” has changed over time.

The workflow “Interactive Workflow Metadata exploration” here (Interactive Workflow Metadata exploration – KNIME Hub) is pretty close … But still I cannot get the correct order displayed in the interactive table.

I read that there is more information provided when using the KNIME server (Compliance and Explainability with Metadata Mapping | KNIME) and Workflow summary (incl. automation) but for now I work on a local machine.

Would be nice if someone could help me out.

Many thanks in advance.

Best

A.

1 Like

Hello @Anjo,

you can parse KNIME log to get exact sequence of executed nodes of a workflow as there is timestamp within it and messages like:
2021-06-11 13:28:57,898 ... Data Generator 0:19 has new state: EXECUTING
...
2021-06-11 13:28:58,952 ... Data Generator 0:19 has new state: EXECUTED

Additionally in workflow.knime file (located within workflow directory) there is information regarding connections between nodes so you can map branches to nodes if needed.

Br,
Ivan

5 Likes

Hi Ivan,

many thanks for your input.

Just to be sure, with KNIME log you refer to the knime.log file in folder …\knime-workspace.metadata\knime. I am asking because when I open this file and search for similar entries (e.g. EXECUTUNG or EXECUTED) I cannot find anything besides warnings. Do I have to change the “level of monitoring” (e.g. WARN, ERRORS etc.) to increase the depth of information for the log file somewhere? And how do you differentiate between different workflows within the file (just by date and time?). I hae to admit I will have to have a deeper look at the file.

The good old KNIME question. Is there maybe an example workflow for parsing the KNIME log? So far I could only Google search find workflows for parsing server logs.

One additional thing: as Patrick1947 explained in his forum entry: How to write a documentation for a KNIME workflow? - #9 by Patrick1974

… at the moment I am only able to correctly parse the “workflow.knime” file with the xpath node when:

… especially the part “Replacing the first config-tag” was neccessary to parse all nodes but I was just wondering if there is a configuration of the XPATH node to do it without manipulation of the workflow.knime file.

Many thanks again.

I will give an update on far I have come with the new information.

Best
A

Okay, I found out, you already explained the part with the “level of monitoring” here: How to get knime.log updated for each run of a workflow - #2 by ipazin

3 Likes

Hi @Anjo,

regarding the possibility of fetching the required information from the workflow.knime file mentioned by @ipazin, you may be interested in the following demo workflow I’ve uploaded to the hub

The demo reads its own workflow.knime file, and interprets the xml to fetch the list of nodes, and the destination connection for each of them. It then “tree walks” that information by a recursive loop to determine the sequence for the execution of the nodes.

You can modify the “open file or folder” component to supply the absolute path to any worklflow.knime file. The output from this may be of assistance to you.

2 Likes

Hi @takbb,

the workflow looks quiet complex but that sounds pretty much to what I was looking for. I will definitely try it out!

Thank you very much for the hint to your workflow. I will post an update here when I managed to get something running.

Best A.

Hi @anjo, yes it does seem complex, but that’s simply because of the structure of the data it is intepreting from the XML file.

The workflow can be broken down into the following sections:
1 - read the XML file into a table
2 - derive the set of nodes
3 - derive the set of connections
4 - determine which nodes each node connects to (both in and out)
5 - walk the connections, to determine sequence

I hope it helps with what you need.

2 Likes

Hello @Anjo,

Seems you got all other answers (and more!) already :slight_smile:

Good luck!
Ivan

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.