Hello Knime Community,
I receive Data via a Get Request. Due to restrictions not all Data can be retrieved, so beside Data the Request also delievers an URL to the next set of Data. I like to implement something like:
Send Request with initial URL
Extract URL if available (save initial url and extrated url)
Send Request with extrated URL, back to 2
No URL - variable loop over the collected URLs and retrieve the Data
I think the Variable Condition Loop Node would be the best fit?
Or maybe it is possible to do it in one loop?
@sven-abx , you probably want to use the Recursive Loop in this case, and you basically end the loop if there is no URL extracted anymore.
I don’t think you can send back the extracted URL to the next iteration with other loops.
thanks for your reply. looks like the recursive loop node is the right choice, but the loop end node failes with “Execute failed: Cell count in row “Row0#1” is not equal to length of column names array: 17 vs. 13”.
I think the problem is “Row#1” and is switches from Row0 (Varible with URL) to “Row0#1” when the URL is empty/not found.
Try this in the “Recursive End loop” :
Notice that, only the last iteration will be available at the end of the loop.
thanks for your help
@SamirAbida. currently i try to figure out how to handle the link( #n) after every itereation to check if a link is extracted or not.
Could you share some example (even dummy data) that I can understand your problem ?
sorry, I have now data to share. I talking about:
First iteration retrieves
@odata.nextLink, the second iteration is @odata.nextLink( #1), third will be @odata.nextLink( #2) and so on.
did you have any progress? In case not maybe these two topics can help you out:
Hoping someone can help. I’m trying to analyse data from youtube’s data api over REST.
I’ve found the channel I want, and within the workflow I’ve retreived videos and got the details. so the basic stuff is working fine. Where I’m coming unstuck is with the pagination of results.
I want to return data about all the videos in the channel (~1750). this means i need to return multiple pages. There is a ‘next page token’ returned with the first result, but for the life of me I can’t work o…
I need your help.
I have an API with pagination. And send request with parameter (date). Server returns first page with some IDs and url for next page.
So I don’t know how make such loop which can do:
get page by url
if page is empty - break the loop
if page have data write it and start new iteration with new page url
What kind of loop I need to choose?
thanks for you reply. The first one looks promising and I was building something simliar.
Thanks and have a nice day,
This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.