I am having issue with Api Query Pagination, when enabled the url get parameters get duplicated multiple time for each page

Matillion ETL
1

Example in first call api url looks like this:

 

http://my_api_end_point/path?param1=value1&param2=value2

 

second page

http://my_api_end_point/path?page=2&param1=value1&param2=value2&param1=value1&param2=value2

 

page 3

http://my_api_end_point/path?page=3&param1=value1&param2=value2&param1=value1&param2=value2&param1=value1&param2=value2

 

and so on ....

 

Any guide would be highly appreciated.

2

Hi @SureshKatwal​,

Since Matillion's documentation as some conflicting approaches, lets make sure you are using the most supported model of paging. Are you following this method for paging or are you on an older version of Matillion that doesn't have this model for paging? https://documentation.matillion.com/docs/6753241

3

Hello SureshKatwal (Customer)

 

Thank you for getting in touch via our Community.

 

I can confirm Matillion does support Page-Based paging since, if I'm not mistaken, release 1.51. So you should be able to resolve this issue simple by upgrading to our latest version. I believe upgrading should solve your issue. Please see attached screenshot.

 

0694G00000E1ZPLQA3_0D74G000007ikxASAQ
0694G00000E1ZPLQA3_0D74G000007ikxASAQ1096×699 43.5 KB

 

Regards

Matillion Product Team

4

Thanks @Bryan​, I have followed that same guide line you mentioned. Some more details about my matillion environment:

  • Version: 1.48.4 (build 596) AMI Version: 1.48.4.
  • Running on aws for snowflake
  • Api has Page based pagination with 200 response code and empty data set for nth+1 call
  • Matillion has pagination feature in query profile management but it does not have option for Page based pagination so updated RSD file manually
  • I am using Advanced Mode on Api Query Component

Here is sample RSD file:

<api:script xmlns:api="http://apiscript.com/ns?v1" xmlns:xs="http://www.w3.org/2001/XMLSchema">

<api:info title="actions" desc="Generated schema file." xmlns:other="http://apiscript.com/ns?v1">

<attr name="ACTION_TITLE" xs:type="string" readonly="false" other:xPath="action_title" />

<input name="rows@next" desc="Identifier used for paging" />

<input name="param1" dec="some param" />

<input name="param2" dec="another param" />

</api:info>

<api:set attr="BackwardsCompatibilityMode" value="true" />

<api:check attr="_input.rows@next">

<api:set attr="URI" value="https://API_END_POINT/api/stats/4?page=[_input.rows@next]" />

<api:else>

<api:set attr="_input.rows@next" value="1" />

<api:set attr="URI" value="https://API_END_POINT/api/stats/4" />

</api:else>

</api:check>

<api:set attr="RepeatElement" value="/dates/items/" />

<api:set attr="paramname#" value="param1" />

<api:set attr="paramvalue#" value="[_input.param1]" />

<api:set attr="paramname#" value="param2" />

<api:set attr="paramvalue#" value="[_input.param2]" />

<api:set attr="paramname#" value="type" />

<api:set attr="EnablePaging" value="TRUE" />

<api:script method="GET">

<api:set attr="method" value="GET"/>

<api:call op="jsonproviderGet">

<api:set attr="rows@next" value="[_input.rows@next | add(1)]" />

<api:push/>

</api:call>

</api:script>

</api:script>

5

Hi @Bryan​ & @SureshKatwal​

Just a quick note on the subject of the documentation, I have raised this with the team who will look into it and make any adjustments to ensure things are clearer.

Thanks Joe

6

Thanks, upgrading to the latest version did the trick.

 

Also, had to add numbers on end of params like this

 

<api:set attr="paramname#1" value="param1" />

<api:set attr="paramvalue#1" value="[_input.param1]" />

<api:set attr="paramname#2" value="param2" />

<api:set attr="paramvalue#2" value="[_input.param2]" />