Pagination on Get Request node

KNIME Analytics Platform
1

Hello everyone,

I’m new to KNIME and need to use JIRA REST API to analyze some data. I’m using the Get request node to extract the data from Jira and works well.

The problem is that I need do pagination on JSON output.

https://xxx.atlassian.net/rest/api/2/search?jql=sprint%20in%20futureSprints()&startAt=3&maxResults=100

If I open the REST API URL in the browser, I’m able to navigate in the JSON pagination with the parameter startAt=0… startAt=3 and so on.

The REST API header give me this:

{"expand":"schema,names","startAt":3,"maxResults":100,"total":1660,"issues":[{"expand":"operations,versionedRepresentations,editmeta,changelog,renderedFields","id":"171417","self":"https://xxx.atlassian.net/rest/api/2/issue/171417","key":"xxx","fields":…

I think the Chunk Loop Start is one option to do that pagination, but I don’t know how to pass the parameter startAt=0

Anyone can give-me one example to do that?

Thanks in advance

2

Maybe you can provide us with a workflow that has a working extraction and/or a working URL that you know contains information you want. The URLs in the text seem not to be working and their special structure might be compromised by the limitations of this forum. So maybe attach them as a file or even better a KNIME workflow.

If it is just the “startAt=0” part you might create a loop that replaces the 0 with 1,2,3,4. If you get the maximum number of pages somewhere you might even set up a dynamic workflow determining the necessary number of loops.

1 Like
3

KNIME_project6.knwf (10.0 KB)
Hi mlauber71, many thanks for your help.

Unfortunately, I can’t give publicly the endpoints and credentials of our enterprise Jira Account.

However, I created a dummy account in Github and a Knime Workflow that do a very similar approach.

In this case I need to change the webservice pagination changing the flag page=1…2…(n).

https://api.github.com/user/repos?page=1&per_page=100

4

Not really sure about the results, but the structure of a loop could look something like this. You might have to adapt it to your needs.

image
image980×548 151 KB

kn_example_http_get_pagination.knwf (50.6 KB)

5

Hi Both,

another idea would be to use the Recursive Loops for this. It is very helpful if you get the next page in the JSON as a result. Because than you can continue the loop until there is not more next page available.

Cheers, Iris

1 Like
6

Many thanks mlauber71 and Iris, I already resolved the problem with something very similar.

Have a happy new year!

1 Like
7

Hi Peter29, I have the same issue with the same application you are connecting. Could you share the solution with us? Txs!

8

Hi Peter,

I am facing the same issue too. Could you please share the solution.
Thank you!

9

Hello @Peter29,
I am facing a similar problem. Can you provide your solution?

Thank you in advance
Mareike

10

Hello @marryah,

if haven’t seen there are couple of topics dealing with this. For example check this one as it has references to others as well:

Br,
Ivan

1 Like