Orchestration Job, Transformation Job and its status status, runtime, table loaded by Transformation job, row inserted into table, update and rejected information etc., from matillion repository
Hi @User16246047157757097411 ,
It sounds like you are looking for the data out of Job History. Are you looking to store this data in a database perhaps in Snowflake, Redshift, etc...? If so, you would need to leverage the Matillion API to accomplish this. We do this today using Query Profile methods that wrap the Task API to retrieve the info which is then pushed into a table. We keep track of the last load so that we know to only grab task history from the last time the job ran. I hope this helps. If I missed the mark on your question please let me know and I will try to help where I can. Thanks for posting!
We have created an set of Audit jobs that write this information into an audit table. We have to export the relevant row counts and pass this back into the audit job. We have pre-job, post-job and error audit jobs.
It would be nice if there was an easier way to do this.
Does anyone have a same job or any screenshots to follow? I am in need to capture the job audit information to store in an audit log table.
hi @Bryan,
May be we are using the same way that you have mentioned.
We are using Matillion API profile in the API query component and then extract the data from 2 Data sources available namely, "Run History Summary" and "Run History Detail".
Hi @colin.thomas1616066511354, may be you can give it a try. We found this as the easiest way. We've scheduled this job to run once in a day.
Thank you,
Kumar