Hoping someone can help. I’m trying to analyse data from youtube’s data api over REST.
I’ve found the channel I want, and within the workflow I’ve retreived videos and got the details. so the basic stuff is working fine. Where I’m coming unstuck is with the pagination of results.
I want to return data about all the videos in the channel (~1750). this means i need to return multiple pages. There is a ‘next page token’ returned with the first result, but for the life of me I can’t work out the exact sequence of nodes to return the basic query first, then manipulate the string to include the next page token. At present I’m thinking of chunk-loop and then variable loop, but to be honest I’m not sure that will work.
My sequence is based on the youtube API example in the server - I could upload a workflow but to all intents and purposes it’s that one, but with looping over the search results to take into account the pagination.
I’ve looked around, and found some code examples, but nothing that helps here. Any thoughts gratefully received.
sorry for the late response. If you did not solve the problem yourself by now, have a look at the recursive loop start. This should enable you to pass the next page token to the loop beginning and let it run another iteration till you reached the very last page of your results.
i do have a similar problem. I think the recursive loop might work but also don t really get it finished.
the problem is that is do:
a first call which gives the first set and the “next page link”
I would then need to use the next page link for the second call
but the GET node would need to know when to take the first inital link and when the “next page link”
so i would need some sort of switch that indicates that a next page link is present and switches.
this is what i intended to do, a first GET to do the initial call, then the loop start. the XPATH extracts the information from the first set and (simplified) the next page link, which is passed back to the loop start.
the GET in the loop would do the second call then. but how can i switch the second get on/off if a next page link is present ?
You might not even need two GET Request nodes. You could configure GET Request (node 805) to use a column as input and the very first URL you provide via Table Creator (or String Config + Variable to Table Row). With XPath you extract all the information you need plus the next page URL. The interesting values you provide as top input of your Recursive Loop End. For the bottom input of Recursive Loop End node you only provide the next page URL (make sure to use the same column name as in the initial input data table) if there is any. If there is non, you can configure a stopping condition for Recursive Loop End by using the option to end via flow variable.
Here is a solution that works really well. I will try to share a workflow but the article covers the solution pretty well. Let me know if you have any questions:
Hey @TardisPilot
really really smart and nice solution for the problem.
I used it for my problem and i think it might work, final tests missing but I’m confident.
Really appreciate it.
@Marten_Pfannenschmidt maybe it would be nice to also include Tardis’ solution as a tutorial or something.
