Trying to utilize the API Extract component to load variant (JSON) rows into Snowflake from an API that returns an array of JSON objects (see sample below). Setting up pagination in the extract profile, I select offset paging. The endpoint supports standard limit/offset query parameters to support paging. The repeating element is set automatically since the response is an array. I set the Record Count to an id field that exists on each object.
Running the component results in the following error:
Unable to complete stage: For offset paging the API response must be a JSON object.
If I disable pagination on the profile endpoint the component works and loads an object per row as expected. The test endpoint I'm using is not indicative of the data volume that other endpoints will provide so pagination is a requirement.
The error doesn't really provide me any clues since I know the response JSON is valid. Any ideas on what could be causing this issue would be appreciated.
Sample response:
[
{
"building": {
"id": "cb838b4e-f615-455c-b698-d64d56ab87d8"
},
"id": "78fefcec-b9f5-42e0-9bce-70063d9874a4",
"number": "401",
"occupancies": [
{
"maxOccupancy": 10,
"type": "classroom"
}
],
"roomCharacteristics": [
{
"id": "56378532-4005-40ac-b58f-b69fd27fb1df"
}
],
"roomTypes": [
{
"detail": {
"id": "b10c358d-5e9d-448d-a48e-57a798eefdff"
},
"type": "classroom"
}
],
"site": {
"id": "b1156e7a-cea5-4e24-9fa2-bf8a367f2c61"
},
"title": "401 Tech"
},
{
"building": {
"id": "cb838b4e-f615-455c-b698-d64d56ab87d8"
},
"id": "fa020ea5-6f18-47b2-8b2a-1b592fc67d8d",
"number": "102",
"occupancies": [
{
"maxOccupancy": 15,
"type": "classroom"
}
],
"roomTypes": [
{
"detail": {
"id": "b10c358d-5e9d-448d-a48e-57a798eefdff"
},
"type": "classroom"
}
],
"site": {
"id": "b1156e7a-cea5-4e24-9fa2-bf8a367f2c61"
},
"title": "102 Tech"
}
]