API Query Pagination. Looking for possible solutions

Matillion ETL
1

Hi everyone,

 

Below is the set URIs for the API endpoints. I want to call this using the API Query component in Matilltion.

 

  • URL: https://ecommercestore.com/products?access_token=abcde&responsetype=json&startseq=0&endseq=99
  • URL: https://ecommercestore.com/products?access_token=abcde&responsetype=json&startseq=99&endseq=198
  • URL: https://ecommercestore.com/products?access_token=abcde&responsetype=json&startseq=198&endseq=297
  • URL: https://ecommercestore.com/products?access_token=abcde&responsetype=json&startseq=297&endseq=396
  • URL: https://ecommercestore.com/products?access_token=abcde&responsetype=json&startseq=396&endseq=495
  • URL: https://ecommercestore.com/products?access_token=abcde&responsetype=json&startseq=495&endseq=594
  • URL: https://ecommercestore.com/products?access_token=abcde&responsetype=json&startseq=594&endseq=693
  • URL: https://ecommercestore.com/products?access_token=abcde&responsetype=json&startseq=693&endseq=792
  • URL: https://ecommercestore.com/products?access_token=abcde&responsetype=json&startseq=792&endseq=891
  • URL: https://ecommercestore.com/products?access_token=abcde&responsetype=json&startseq=891&endseq=990

 

In the above sequence of API URLs, the parameters "startseq" and "endseq" need to be increment in step sizes of 99 for every call until there is no response. How can I configure this pagination when creating the API profile to reach the outcome?

 

Any other alternate strategies also are appreciated.

 

Thanks,

P

2

Hi @PG881302​, it appears the paging strategy that you have shown isn't natively supported. There may be some custom approach that could be implemented within the API profile but I think your best bet on getting help with that would be Matillion Support. Without know how Matillion is creating the request under the API profile it would be hard to figure this out on your own.

So, I would start with support but if that goes no where, I know you can accomplish this with a Python script. I would investigate this approach as a last resort because although it's doable it can introduce technical debt and code complexity. I hope this helps.

3

Hi @Bryan​

Thanks for your response. I will follow your advice and keep this thread updated on how this can be done on the platform.

-P