GET Request just shows 1000 records

[mauuuuu5]

Hi everyone I am trying to query the information from this website www.datos.gov.co more specifically this set of information. In order to query the information the website provides:

• A JSON link: https://www.datos.gov.co/resource/37ii-v4q2.json

• An API ID

• API Password

• Token API

I managed to use the GET Request node but it is just querying 1000 results.

In the attached workflow I have hard coded a dummy email and password with the API’s information if someone can help me to query more records

Thank you

API Datos.knwf (1.1 MB)

[ArjenEX]

Hi @mauuuuu5

Such a limit is very common by design to avoid that the API is flooded with large data requests. The page you mentioned regarding the dataset links to Socrata API documentation. In there, it has a section about Paging through Data.

https://dev.socrata.com/docs/paging.html#2.1

The default limit of 1000 records is mentioned there.

It also gives some examples on how the use the limit and offset.

[mauuuuu5]

Thank you I managed to query 5.000 at once with this:

https://www.datos.gov.co/resource/jbjy-vk9h.json?$limit=5000&$offset=0

and then managed to query 100.000

Thank you

[knimediger]

@mauuuuu5

The intention of this limit is to avoid too heavy load on the API.
So it is good practise to request small bunches of data and to concatenate them if required.
The parameter offset is for that purpose.