Knime Logging Mechanism

Hi Everyone,

Does Knime has capability to report workflow logs with the below information ?

  1. Workflow start time/end time
  2. Numer of records processed at each node
  3. Any other valid details
  4. Success Failure messages

A practical usecase would be to build a dashboard for Workflow monitoring.

I would like to load these details into a control table for audit purposes. Please help.

Thanks
Sudheer

1 Like

Hi Sudheer,

We are working at the moment on a feature where that kind of information can be exported as xml from a workflow. Besides the features you mentioned, this will also include node settings, information about the environment the workflow was run on, which plug-ins where installed, etc.

Best regards,
Jeany

2 Likes

Hi Jeany,

Thanks for the response. If the solution is in labs, when is it expected to release ? Until then do you think of any alternative ?

Thanks
Sudheer

Hi Sudheer,

We expect it to come with the next release in July. Exporting an executed workflow and saving it gives you already a lot of information as the data is stored with the exported workflow. In addition, you can also save the log file (View->Open KNIME log) which gives you information about the error messages, in which nodes and when they occurred, etc.

Hope that helps,
Jeany

Hi Jeany,

Exporting each workflow everytime is not a feasible option and logs don’t give number of records processed at each nodes. What is the alternate solution available until july ?

Thanks
sudheer

Hi there @sudheerkopparapu,

did you think about implementing certain points in a workflow and then export that information to a file or even better to a database? Points 1 and 4 should be quite easy to do. Point 3 implementation depends on what valid details you need and that leaves point 2. Although not complex to implement for a single node adds complexity and execution time to whole workflow if you wish to have this kind of info for every node. I can imagine this being a requirement but not sure how valuable this information actually is :confused:

Br,
Ivan

Hi Ipazin,

Could you pleas give me pointers on 1 and 4.
Regarding 2, we need checkpoints at very few nodes. So need pointers on this as well

Thanks
Sudheer

Hi there @sudheerkopparapu,

For 1 you can use Date&Time Configuration node both when starting your workflow and ending it with option Use execution time checked. It will give you start and end timestamp in flow variables.

For 4 use Try/Catch sequences. If you workflow is small you can use only one and if bigger you can use one sequence for each logical part of your workflow. See this example.

For 2 you can use Extract Table Dimension node which will give you number of rows and columns both in table format and as flow variables. The name of node which processed those records needs to be hard-coded somehow (use String Configuration or simple Table Creator node.

Additionally you can use Extract Context Properties to get workflow name and have something like this as reporting records:

WorkflowName Start CheckPoint1 CheckPoint2 End Status Note
test 2020-04-21T16:01:47.075 10.123 5.987 2020-04-21T16:16:15.278 Success -
transfer 2020-04-21T16:16:50.64 65.432 - - Failure Error msg

It requires a bit of effort but is most certainly achievable! Also there are topics where similar was discussed so give forum search a try and you might find additional valuable information!

Br,
Ivan

2 Likes

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.