Hello,
I have been trying to configure an API profile which will be able to update the API URI with the next page number (i.e. ...page_number=0..., page_number=1...) For the total records that are present.
There is a field 'Total' returned by the query which can be used for this, and also a maximum records per page/response number.
Offset-based pagination doesn't seem to work as it keep querying the same page.
Page based pagination also doesn't work as if selected a page with no records the API doesn't return 400 response, or even an empty page (as the total items field is present all the time)
Has any one faced something similar?
Maybe some screenshots would help.
What about cursor based pagination?
I find the same issue with offset/limit pagination; Matillion does not loop through the pages as claimed, at least from what I am experiencing. Had to build my own iteration.
Hello All
Unfortunately, there are no "global" API structure or paging standards, what we provide at present is a close as we could get to "standard" paging patterns.
Determining which is the paging strategy actually being used by your API can be tricky at times, not to mention, some strategies are very similar but just use different parameter names or patterns.
We are very aware of this issue at Matillion, and have been working on a solution for some time to get around this.
Quite often APIs bare no similarity to other APIs, so if you could provide the name of the API you are working with, we could potentially help you come up with a solution.
Regards
Ben Smiles
Matillion Product Team
@BenSmilesMatillionProductOwner
I am also interested with possible pagination solution. Seeking your advice on the feasible solution approach.
For example for the follow Braze API to get campaigns ids list.
<base_url>/campaigns/list?page=1
The page of campaigns to return, defaults to 0 (returns the first set of up to 100).
Reference: GET: Export Campaigns List (braze.com)
Hello All
I have had a look at the Braze API example, and have seen it uses a very limited version of paging (page parameter only), which isn't supported.
This could potentially be solved by creating a "page" parameter and then use a variable in API Extract / API Query to iterate over the parameter. That's my feeling anyway.
With regards to the other APIs, I can't really give too much of a solution without looking at the API documentation.
Regards
Ben Smiles
Matillion Product Team