I had some questions about how I can most effectively extract task history data using the Matillion API. The issue I am running into is that the response I am getting is huuuuuuge. It appears that I am getting component-by-component info on each pipeline run, which is causing some issues. I just want data at the pipeline level, and would much prefer not to extract component level data and then filter afterwards. I have tried: ?limit=1 , ?since= , filter/by/… and others.
I have spoken to the team, and we recommend that you use the API Query component in Matillion ETL to access the Task History API. This component allows you to filter the data returned in the API response and apply filters to any column you need. For more information, read API Query - Matillion Docs . For a general example of extracting and loading task history into a target data platform, you could also give this a read: Task History - Matillion Docs .