I’m working with KNIME Analytics (will be moving to KNIME Server sometime this March), and I’ve gotten a basic workflow set up to extract data from the database, parse it into JSON, and then submit it to a Web Service via POST Request.
Right now, I’m working on taking the response data and putting into the database.
The issue I’m having with this is that this Web Service can take some time to finish processing a request, and I can’t predict how long. Even if I make the request as small as possible, there might be other jobs queued up before mine on the Web Service end. So it could take 5 seconds… or 10 minutes (so far that’s the longest it’s taken–but, in theory, it could take longer).
If you run this via program - and I have an existing PHP program that does that - the Web Service returns a “job-has-been-queued” response with a URL that you can ping with a GET or POST request. It returns a JSON response, and you can tell if it’s completed by the is_finished flag.
So, in theory, this is pretty simple: send the POST request, then extract the Job Location URL from the initial “job-has-been-queued” response, and ping that every 5 seconds (or whatnot) until the is_finished flag is true. Then the JSON response data can be processed.
Right now, I have set a 5 minute wait on my GET Request, but if it’s not complete by then, the workflow cannot proceed. When I was testing this, I discovered that if the Web Service wasn’t complete yet, I could click on the GET Request node and Execute it manually until the job is done. But this isn’t a great solution, since many jobs can finish in less than 5 minutes – and the possibility of manual execution isn’t something I want to leave as-is.
While I can see KNIME has several Wait functions, I’m struggling to set this up in my KNIME workflow. My goal is to be able to run this entire process automatically (the user only has to input an ID value to extract data from the database), so the user can walk away while everything is processing–no manual steps.
Can anyone point me in the right direction?