Get notification of KNIME Server scheduled workflow failures?

monitoring
rest
#1

My team now uses KNIME Server to automate a significant amount of work – we have several workflows that execute many times a day, and users that depend on the data lagging by no more than several hours. (When it breaks, we hear complaints from our users and have to debug it quickly).

Is there a way to get KNIME Server to notify us when one of these key workflows fails to execute? We see the ability to get notifications every time a workflow completes, however, that would spam our inbox with hundreds of false positive messages each day since it’s mostly working.

I can imagine complex email filters (if I can figure out what to filter for), or creating workflows that create logs and other workflows to monitor those logs along with programming the schedule so if something fails we’d have to predict when it was supposed to run and didn’t finish, etc. What would be the simplest way get this kind of notification? Is there anything built in to KNIME Server, or if not, what’s the easiest way to get this information?

Thank you!

0 Likes

#2

Hi @YetAnotherKnimeUser,

this is not possible built in, there you either get all notifications or none. But we have this on our roadmap and will review this again for next year.

You can already built this yourself with our REST API. (Included in KNIME Server Medium and Large)

With it you can built a workflow getting all executed jobs from the server and sending you can email if any did not successfully execute.

Best wishes, Iris

2 Likes

#3

Thanks, Iris! Assuming we did have access to the REST API, how would we build this? Are there nodes in KNIME to do these actions other than the “KNIME Server Connection”? Or would we have to hand-code the REST API calls?

I found this article about using the REST API from the command line but not much else, at least, nothing about how to access the REST API from within KNIME.

Thank you!

0 Likes

#4

Hi @YetAnotherKnimeUser

I would do the following (but it is definitely only one possible solution)

  1. Set all schedules to not discard the jobs after execution.
  2. Get a list of all jobs: GET http://localhost:8080/knime/rest/v4/jobs
    ( 3. Get a list of all scheduled-jobs: GET http://localhost:8080/knime/rest/v4/scheduled-jobs not necessary)
  3. filter the jobs list to only contain the ones with a scheduler-id
  4. Search for all jobs which are neither EXECUTING nor EXECUTED
  5. Send a email with remaining jobs (this can include the failure status)
  6. Either discard those jobs, or write them into table. This table you can save on the server. In the next execution of this workflow, I would first check this table and send the email only if there are new instances.

I am working on something similar right now, so if you are interested, I can share an example workflow.

Best, Iris

1 Like

#5

Hi Iris,
That was very helpful! Thank you!

Yes, if you were able to share an example workflow, that would be hugely helpful to us – this would be accessing KNIME server metadata via the REST API in a way we have not done before, any starter workflows would be greatly appreciate.

Thanks,
Aaron

0 Likes

#6

Hi @YetAnotherKnimeUser
The workflow is here:
SendEmailsForFailesScheduledJobs.knwf (83.8 KB)

I thought it make sense to have one email per failing workflow. The part what is missing is to which email the error get send, but I guess this can be implemented using the String Manipulation or some rules based on how your usernames and emails adresses are related.

Cheers, Iris

0 Likes

#7

Thank you, Iris! We’ll be trying this out shortly.

0 Likes

#8

Hi @Iris – Is there any chance you have an update on whether this might be available in a current version, or in a version coming soon? Just checking – thank you! We love the idea that this might be built-into the server in the near future.

0 Likes

#9

Hi @YetAnotherKnimeUser

yes it was released with KNIME 4.9. it is so awesome! I do not say this because I work for KNIME, this features makes me happy everyday :slight_smile: Now I only get emails when my workflows fail and if there are no emails I do not need to worry.

You can find a screenshot how this looks now here.

https://www.knime.com/whats-new-in-knime-40#scheduling-improvements

1 Like

#10

That’s great news, @Iris! I’m looking forward to using it – we’re upgrading to v4 now. Thanks for the superfast reply too!

2 Likes

closed #11

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

0 Likes