I'm pulling from an endpoint that allows the 'If-Modified-Since' header to be set with a date to filter results.
I created a profile with the If-Modified-Since as a Header->connection type and populate it with the correct formatted date.
When I sample or run the component it says success but returns no values. If I remove the If-Modified-Since header then it returns all values.
On further inspection, I noticed that the response headers contain the 18 records that I am expecting for my sample but the component says 'Zero rows staged.'
I have attached the log file (debug level=3).
The response has some XML type fields, could these be throwing off Matillion's parser somehow?
If anyone could have a look at this it would be greatly appreciated. I've learned a lot about the cdata rds scripting language that is used in the API Query through this exercise but still not quite getting how to setup these requests correctly...
Cheers!
Steve
Actually I have discovered that Pagination seems to be the issue. If you look at the log above, you can see that 18 records are returned in the first request, then it attempts to execute the page. However the "Link" header field is blank.
When I am setting up the API Query profile, I get the expected 18 values in the initial SEND while building the headers/parameters etc. However if I implement the "Link Header" paging strategy then the final request is returning no values similar to what the log above is showing.
This leads me to believe the Link Header paging strategy is not working. If I query all records of the "payment" endpoint above, the API returns the following in the "Link" header field:
<https://healthcarels.simprosuite.com/api/v1.0/companies/0/customerPayments/?display=all&pageSize=100&columns=ID%2CPayment%2CNotes%2CInvoices%2CDateModified&page=2>; rel="next", <https://healthcarels.simprosuite.com/api/v1.0/companies/0/customerPayments/?display=all&pageSize=100&columns=ID%2CPayment%2CNotes%2CInvoices%2CDateModified&page=129>; rel="last", <https://healthcarels.simprosuite.com/api/v1.0/companies/0/customerPayments/?display=all&pageSize=100&columns=ID%2CPayment%2CNotes%2CInvoices%2CDateModified&page=1>; rel="first"
Is this value appropriate to use with Matillion's Link Header pagination strategy or should I be using another paging strategy?
Thanks!
So in using the "Link Header" strategy, if the API returns a blank "Link" header then Matillion is still attempting to execute the page even though there is no next page URL in the Link header?