Loop Variables for API

KNIME Analytics Platform
16

@badger101

Went back and fourth a few times between different setups but ended up with these two that stuck. They are producing the same output.

image
image1233×833 41.1 KB

General notes:

  • Recursive Loop is the way to go here. It’s tricky to master but it has to be used since the loop needs to “remember” what was just queried and what the URL of the next GET request should subsequently be.
  • You need to properly control the exit of the loop. In this case, whenever the nextPageToken is not retrievable it means no further pagination is available. Using a pretty simple evaluation with a Column Expression node.
 if (column("nextPageToken") == null) {
    true
} else {
    false
}

With this boolean you can make the Recursive Loop End exit variable controlled. It will override the maximum number of iterations and exit accordingly.

image

  • I would personally stay away from JSON to Table as much as possible. It’s quite a system breaker if you have a properly sized JSON. If you only need to retrieve the nextPageToken, then a JSON Path query works a lot faster (there is only one instances of it in the JSON). Retrieval can be achieved via $['nextPageToken']

image
image1174×835 29.5 KB

  • Same applies for the actual content btw. I figured it’s not very likely that you need every single property. I would only get what you need. For example, I opted for the text and the date. Ensure that you have the List property enabled. Put an Ungroup node after it and it’s all done.

image
image1411×1179 79.5 KB

  • The slight difference between option A and B is mainly around how you would like to manage both streams (pagination and non-pagination). The “A-way” is to to leave the If-switch uncontrolled and set its ports to Both. The top stream handles the pagination or is blocked by the empty table switch otherwise. The bottom stream always takes care of the first page or all comments. I didn’t touch the creation of the new url etc.
    The “B-way” separates this more strictly. It evaluates if the nextPageToken is present or not and assigns a single port accordingly. You can then control all flows (pagination - first page, pagination - other pages, non-pagination) individually.

  • I didn’t fully stick to your original setup but these two methods require fever nodes, for example taking out the 3rd GET Request that you have.

  • Execution time:
    Option A - 192 ms.
    Option B - 227 ms.
  • Other than that I think you’ll find your away around.

WF: Pagination YouTube API version A_EX.knwf (123.6 KB)

Hope this helps!

PS: Apply your own token again first :wink:

6 Likes