Wait/Repeat for POST/GET Request

I’m working with KNIME Analytics (will be moving to KNIME Server sometime this March), and I’ve gotten a basic workflow set up to extract data from the database, parse it into JSON, and then submit it to a Web Service via POST Request.

Right now, I’m working on taking the response data and putting into the database.

The issue I’m having with this is that this Web Service can take some time to finish processing a request, and I can’t predict how long. Even if I make the request as small as possible, there might be other jobs queued up before mine on the Web Service end. So it could take 5 seconds… or 10 minutes (so far that’s the longest it’s taken–but, in theory, it could take longer).

If you run this via program - and I have an existing PHP program that does that - the Web Service returns a “job-has-been-queued” response with a URL that you can ping with a GET or POST request. It returns a JSON response, and you can tell if it’s completed by the is_finished flag.

So, in theory, this is pretty simple: send the POST request, then extract the Job Location URL from the initial “job-has-been-queued” response, and ping that every 5 seconds (or whatnot) until the is_finished flag is true. Then the JSON response data can be processed.

Right now, I have set a 5 minute wait on my GET Request, but if it’s not complete by then, the workflow cannot proceed. When I was testing this, I discovered that if the Web Service wasn’t complete yet, I could click on the GET Request node and Execute it manually until the job is done. But this isn’t a great solution, since many jobs can finish in less than 5 minutes – and the possibility of manual execution isn’t something I want to leave as-is.

While I can see KNIME has several Wait functions, I’m struggling to set this up in my KNIME workflow. My goal is to be able to run this entire process automatically (the user only has to input an ID value to extract data from the database), so the user can walk away while everything is processing–no manual steps.

Can anyone point me in the right direction?

Hi,

as far as I understand, your challenge is how to build the “ping every x seconds”?

I’d suggest to build a loop starting with a Generic Loop Start and terminated with a Variable Condition Loop End. Within the loop, add a Wait node, followed by a GET Request, check whether you receive the “has finished” response, and in this case terminate the loop (else wise just continue looping).

That’s the rough sketch. Hope it helps.

– Philipp

2 Likes

Hello Philipp:
Thanks for your reply.

This sketch works well, but Variable Condition Loop End doesn’t provide me with the ability to select a variable from the JSON response. I’m only allowed to select “Available flow variables” (which are currentIteration and knime.workspace – not from the output of the GET Request). The examples I’ve found all seem to use currentIteration (e.g., loop 100 times)

Is there some way to get the variable condition loop end to take a variable from the output?

Without knowing the JSON in detail, I’d do the following:

  • use a JSON Path node to extract the relevant property from your response
  • use a Table Row to Variable to create a flow variable
  • input this flow variable into the loop end node

Does this work? :slight_smile:

Best,
Philipp

1 Like

Hello Philipp:
This worked perfectly for me! Thank you!

1 Like