Looping over API/JSON

[sven-abx]

Hello Knime Community,

I receive Data via a Get Request. Due to restrictions not all Data can be retrieved, so beside Data the Request also delievers an URL to the next set of Data. I like to implement something like:

1. Send Request with initial URL
2. Extract URL if available (save initial url and extrated url)
3. Send Request with extrated URL, back to 2
4. No URL - variable loop over the collected URLs and retrieve the Data

I think the Variable Condition Loop Node would be the best fit?

Or maybe it is possible to do it in one loop?

BR,
Sven

[bruno29a]

Hi @sven-abx , you probably want to use the Recursive Loop in this case, and you basically end the loop if there is no URL extracted anymore.

I don’t think you can send back the extracted URL to the next iteration with other loops.

[sven-abx]

hi @bruno29a,

thanks for your reply. looks like the recursive loop node is the right choice, but the loop end node failes with “Execute failed: Cell count in row “Row0#1” is not equal to length of column names array: 17 vs. 13”.

I think the problem is “Row#1” and is switches from Row0 (Varible with URL) to “Row0#1” when the URL is empty/not found.

BR,
Sven

[SamirAbida]

Hello @sven-abx,

Try this in the “Recursive End loop” :

Notice that, only the last iteration will be available at the end of the loop.

Br,
Samir

[sven-abx]

hi,

thanks for your help @SamirAbida. currently i try to figure out how to handle the link(#n) after every itereation to check if a link is extracted or not.

br,
sven

[SamirAbida]

Hi @sven-abx,

Could you share some example (even dummy data) that I can understand your problem ?

Thanks,
Br,
Samir

[sven-abx]

Hi @SamirAbida,

sorry, I have now data to share. I talking about:

First iteration retrieves @odata.nextLink, the second iteration is @odata.nextLink(#1), third will be @odata.nextLink(#2) and so on.

br,
sven

[ipazin]

Hello @sven-abx,

did you have any progress? In case not maybe these two topics can help you out:

Paginating Youtube REST results

Hi All Hoping someone can help. I’m trying to analyse data from youtube’s data api over REST. I’ve found the channel I want, and within the workflow I’ve retreived videos and got the details. so the basic stuff is working fine. Where I’m coming unstuck is with the pagination of results. I want to return data about all the videos in the channel (~1750). this means i need to return multiple pages. There is a ‘next page token’ returned with the first result, but for the life of me I can’t work o…

Loop through api with pagination

Hello. I need your help. I have an API with pagination. And send request with parameter (date). Server returns first page with some IDs and url for next page. So I don’t know how make such loop which can do: get page by url if page is empty - break the loop if page have data write it and start new iteration with new page url What kind of loop I need to choose?

Br,
Ivan

[sven-abx]

hi @ipazin,
thanks for you reply. The first one looks promising and I was building something simliar.

Thanks and have a nice day,

BR,
Sven